Andy's Thoughts

New blog site...

2011-07-26T14:09:00.000-04:00

I have transferred all my original posts to a new site and improved blog powered by WordPress. I was very happy with my Google Blogger site, but with some upcoming changes, I needed my own dedicated site… so welcome to PaulTechnologies.com!

If you were following my old site here on Blogger, please subscribe to my new blog. Along with my random posts on virtualization, I will also have articles and whitepapers I have been publishing, some best practices, and a soon-to-be-released program for auditing Citrix XenApp (cleverly called XenAppAudit) that I have been developing.

I’m excited about these changes and I hope you find value there as well… please check out my new site and drop me a note or a comment if there are any particular topics you would like to see covered!

This will be my last post here at sbcengine.blogspot.com, thanks for reading and I hope to have you follow me to http://paultechnologies.com...

Thanks!
-Andy

Technical Deep Dive: XenApp Load Evaluators

2011-06-27T17:09:00.000-04:00

One of the often overlooked features of XenApp is truly understanding the load evaluators. As a consultant, I commonly see environments using only the Default Load Evaluator. If I am lucky, they might be using the Advanced Load Evaluator. Rarely do I find organizations actively monitoring or customizing their load evaluators.

Load Evaluators have not changed much since Presentation Server days, but amazingly they are not commonly optimized. Every environment and every workload is different, so whichever load evaluator is implemented may vary, but they should be customized and monitored.

Read more on my full article over at TheGenerationV: http://www.thegenerationv.com/2011/06/technical-deep-dive-xenapp-load.html

Also, be on the look out for my new programming endeavor.. the XenApp Audit Tool. I have incorporated the QueryDS analysis into my application. Coming soon!

Are you ready for VDI?

2011-04-26T10:08:00.001-04:00

More companies are investigating the benefits of implementing a Virtual Desktop Infrastructure (VDI). If you are one of those companies then you need to ask yourself, “Am I ready for VDI?” The first step is identifying exactly what VDI means to your organization. VDI actually encompasses several types of delivery, but is most commonly associated with what Gartner classifies as hosted virtual desktops (HVD). VDI can also include implementations of server-based computing (SBC), hosted shared desktops (HSD) and client-side virtualization. For the sake of simplicity, we will focus on the hosted virtual desktop model for VDI since that is the most common assumption of a VDI deployment.

As a virtualization architect, I have been involved in numerous VDI implementations ranging from proof of concepts to full production lifecycles. In the process of design and implementation engagements, one thing is clear: many companies are not ready for VDI. Some simply do not need VDI, but think they want it. Others may need it, but have a long road to travel to get there. The question then becomes how to determine if VDI is right for you and how to make it a reality. This is not about deciding which solution is right for you, but understanding what should go into that decision making process.

Want to read more? Check out the full article at http://www.virtual-strategy.com/

Citrix Synergy 2011

2011-04-07T11:17:00.000-04:00

If you are thinking about going to Synergy this year, I will be assisting in moderating the XenClient learning labs (SYN409D) during Summit and speaking as part of the XenDesktop team during Synergy. This is an exciting opportunity for me! If you are planning on attending, check out the 5 part WOW TO HOW Series, I will be speaking on Part 3: Storage Infrastructure (SYN305)

For more information on Synergy, or to register, follow this link: Synergy 2011

To see more details on my session, follow this link: SYN305

XenDesktop 5 Deep Dive: Machine Creation Services (MCS) on vSphere 4.1

2011-03-07T22:35:00.001-05:00

As promised, I have published my Deep Dive around MCS over at The GenerationV Site: http://tinyurl.com/4tzpfts. Check it out!

The data is abstracted, but all the source came from a real-world pilot. The question posed was, "What is the impact of MCS on storage and how exactly does it work on vSphere?" So, I rolled up my sleeves, dug in, and figured I would share my findings.

In short, MCS works as design and can greatly save on storage space by using a linked snapshot as well as save on administration by rapidly deploying machines. The downside becomes when you need to move machines to different storage. Not always a common event, but one to consider.

Personally, I still prefer PVS for an enterprise-scale deployment, but MCS has it's place.

Other resources:

XenDesktop Design Handbook Success Kit - One stop shop for XD Resources
Citrix XenDesktop Architecure Series - Citrix Community Blog for XD Architecture
Dan Feller's Blog- Great Architectural Resource for XenDesktop

It's been awhile...

2011-02-28T14:44:00.001-05:00

WOW. It's been awhile since posting, but I have a lot of topics in the pipeline. Since joining GlassHouse, I figured I would have more time to write, but the inverse has turned out to be true.

However, thanks to some recent requests, I am back on the blog. Here are some things to look for in coming weeks:

VDI preparedness -- what you need before you jump into virtual desktops
Deep Dive into MCS -- XenDesktop 5's new Machine Creation Services
Deep Dive into WI on NetScaler -- the new Web Interface integrated directly onto NetScaler VPX
Deep Dive into XenDesktop 5 -- notes from the field on what's new and improved plus evolving best practices
7 Rules of Consulting -- written as a series, my own personal take on being a consultant

Joining GlassHouse Technologies!

2010-07-08T21:56:00.000-04:00

I'm happy to announce that I have joined the team at GlassHouse Technologies as a Senior Virtualization Consultant. With this role, I will be tasked with helping manage virtualization projects, with a main focus on Citrix Technologies.

My main focus will be on expanding our XenDesktop / Virtual Desktop offerings, and will leverage all of our corporate expertise on Datacenter Management, Virtualization, Storage and Security.

Citrix closes the gap with VMWare -- XenServer 5.6 Beta, Part 1

2010-04-23T12:35:00.000-04:00

Most of the complaints I've heard in regards to XenServer vs. VMWare is the fact that XenServer does not allow an overcommitment of memory. If you have a host server with 16 GB of RAM, you can only allocate up to 16 GB of RAM for the guests (yes, I know, technically less since Xen requires some resources... but you get the point).

Personally, I'm not a fan of over-allocating memory. I actually like that with XenServer, I do not have to worry about it. I've seen too many VMWare farms suffer because of assigning too much memory then having to constantly swap resources. I think if you engineer the solution correctly, and size your servers properly, memory allocation should be kept in check.

However, seeing this as a fault when comparing virtualization host platforms, Citrix has released the latest Beta of XenServer 5.6. One of the new features IS allowing your to over-allocate memory... they call it Dynamic Memory Control. I have to admit, it is pretty cool and very easy to use. The XenCenter display makes it VERY intuitive... all the better to see WHERE your resources are allocated, especially when squeezing memory.

To read more technical details, see my full article at: http://www.thegenerationv.com/2010/04/xenserver-56-preview-part-1-dynamic.html

Louisville Citrix Users Group -- Next Meeting 4-8-2010

2010-03-09T23:10:00.000-05:00

The next meeting for LCUG has been announced...

PVS Discussion, networking, pizza and refreshments... what more could you want?

SEE YOU THERE!
What: Provisioning Server Discussion and Trilogy Case Study
When: Thursday, April 8, 2010 6:00 PM
Where: Peak 10, 752 Barrett Avenue, Louisville, KY 40204

Learn more at: http://www.meetup.com/citrixlouisville/

Direct Booting a VHD in XenServer

2010-02-20T00:53:00.002-05:00

I've run across several instances where I needed to direct boot a VHD in XenServer. One example was migrating from Hyper-V to XenServer. Another was updating the network stack, XenTools, and PVS Tools on existing vDisks. Regardless, the following is the three-step process I took...

Step 1: Create an EXT-3 drive
The standard install of XenServer create a local storage repository using LVM format. You can destroy this and create an EXT partition instead. Please note, this will also destroy ANY VMs on that partition, so proceed with caution. In my XenServer farms where I am using shared storage, I like to create at least one host with an EXT drive for flexibility.

Step 2: Import the VHD
Using WinSCP, copy your VHD files to your new EXT Storage Repository.

Step 3: Mount that Drive!
Create a new VM, note the the UUID of the drive, then kill it and rename your copied VHD to that UUID and fire it up!

Sound easy? It is! For more details, check out my step-by-step guide over the TheGenerationV.com! (shameless plug, I know)

Just be careful not to place the VHD file in the wrong volume, namely the root partition! (oops)

Special thanks to:

Duane Bradley and Geoff Green, MTM Technologies ... they helped me with a nice guide the first time I had to create an EXT-3 drive to import images.
Trevor Mansell for his Citrix blog post on direct VHD boots for PVS 5.1

Recovering from XenServer Root Drive Full -- Oppsie!

2010-02-01T12:18:00.002-05:00

I was trying to import a VHD straight into XenServer the other day. I was moving a VM from HyperV to XenServer 5.5, so instead of starting from scratch, I thought I would simply copy the VHD from HyperV to XS using WinSCP. Following the directions from Trevor Mansell's Citrix Blog

All went well until my XenServer died on me...
Of course, we all have those Homer moments (slap on head, "DOH!"). I realized, AFTER THE FACT, that I was actually copying the VHD to the ROOT volume, not the SR I intended too. The XenServer was still running, but I could not manage the server via CLI nor XenCenter. Trying to launch the xsconsole, nothing would happen.

As several of you may have guessed off the bat, and I realized after the fact, I copied the file not to the local SR, but to the kernel install partition. OOPS!

When that partition filled up, the console crashed. BOOM! (well, more of a whimper really, but still...)

I was able to recover by doing the following process:

PuTTY into the XS
navigate to the SR location I attempted to copy the VHD to, then delete the .FILEPART
(for not linux folk, RM is the remove [delete] command)
then run the xe-toolstack-restart command
Everything was happy after that

I did not attempt to reboot the XenServer for fear it might not revive. Luckily, the toolstack came up after clearing the space. Hopefully, you won't encounter this... but if by accident you do, now you know how to recover.

The Future of Computing? BYOC!

2009-11-08T00:40:00.000-05:00

Check out my latest article, and in depth look at Bring Your Own Computer to Work Policies...

A growing trend in technology firms is Bring Your Own Computer (BYOC or BYOPC). Of course organizations such as Microsoft, Google, Citrix and Cisco are embracing this -- which is to be expected. However, as workforces become increasingly mobile, and virtualization methodologies are further adapted in enterprises, BYOC will inevitably grow.

In the simplest terms, Virtualization is about decoupling. In server virtualization (vSphere, XenServer, HyperV), we decouple the server OS from the server hardware. With application virtualization (XenApp, AppV), we decouple software programs from the OS. With SAN and NAS storage, we long ago decoupled data storage from operating machines. With the advent of client hypervisors, we will being doing the same with PCs. All of these pieces are decoupled to create a more robust and flexible infrastructure. It is only logical, then, to decouple the enterprise organization from the desktop machine itself.

Read the full article HERE over at TheGenerationV.com!

Deploying Windows 7 with XenDesktop 3

2009-11-04T23:20:00.001-05:00

I've been running Windows 7 since May. I have to say, I've very impressed. Microsoft got it right. They took the best of XP and Vista (and MACs), plus some good common sense features to make a great product. I've run it on three systems, ranging from a brand new Quad-Core 4gb RAM laptop to an old single-core 1gb ram nc6120.

As a virtualization engineer, I'm seeing a lot of customers evaluating virtual desktop deployments as a way to distribute Windows 7 in their enterprise. It would be easy to argue Windows 7 is a reason they are embracing VDI; of course, it is just as easy to say they are using Windows 7 as the excuse for VDI. Either way, both are gathering steam.

Citrix is releasing XenDesktop 4 this month. This version is designed with full Windows 7 support in mind, but what if you want (or need) to use XenDesktop 3 -- maybe you are doing a Proof of Concept, maybe you already have the infrastructure for XD3, whatever the reason, there are a few minor issues.

#1 -- Use the latest Virtual Desktop Agent (VDA). This is the component that is installed on the virtual desktop to interact with the Desktop Delivery Controller. The original VDA 3 will not install on Windows 7 (but you can trick it through Vista compatibility mode). The latest VDA will install, with the warning that it is not fully supported. You can get version 3.0.3075 (XDE300VDA004) at CTX122134.

#2 -- Printers do not AutoCreate. When testing Windows 7 during a Proof of Concept, we kept encountering Error 1116 in the Event Log. The AddPrinter() process failed with Error Code 0x5, which translated into Access Denied.

This is due to the CPSVC Service Account failing in the Windows 7 environment. A short term fix is to run the service as LOCAL SYSTEM account, which will have the proper access. This is resolved in XenDesktop 4 (CTX122744 #21).

It took awhile of fighting it, but once we got Windows 7 running smoothly on XenDesktop 3 infrastructure, the client was so happy we destroyed the XP and Vista PoC images. Their goal is to go all Win7, all Virtual.

Windows NLB on Hyper-V R2

2009-10-28T00:15:00.013-04:00

Today I had the luxury of setting up a Unicast NLB on a couple of Windows 2008 servers (for Citrix Web Interface 5.2). The servers themselves are guest VMs running on Hyper-V R2. I noticed that when trying to create the NLB, it would not converge. The NLB wizard was erroring out, plus the entire NIC would go offline, losing its static IP association. I would also see Error 12289.

I mean, I've done NLBs plenty of times before... unicast or multicast... 2003 or 2008... really no difference... its not that hard, but it would not work!!! Arrrgggg!!! After bashing my head for a few minutes thinking I was insane, I did some digging.

I've done this on Physical Servers (check), VMWare ESX guests (check), and XenServer guests (check)... no issues. So I figured it was Hyper-V related... and viola! NLB creates a virtual MAC for the balanced servers to listen on, however the v-switch in Hyper-V does not allow it by default.

In R1, I found out, you had to statically assign the NLB MAC as the VM MAC. Luckily, in R2, they made this easier (along with a lot of other nice enhancements). In Hyper-V R2 / SCVMM R2, set the NIC of the VM to enable spoofing.

Once I set this, the NLB could properly configure, bind, and converge.

Of note: Although I could have left the MAC at dynamic, I chose to set both VMs to static, but inheriting their current MAC address. I'm just not a big fan of MACs changing. :-) Also, the VMs were OFFLINE when I made the changes through SCVMM console.

More reading:
check out Microsoft's John Howard's post on MAC spoofing.
You can also read Rob's Blog

Optimizing XenApp on VMWare ESX

2009-10-19T23:03:00.002-04:00

There are numerous benefits for running Citrix XenApp on XenServer; including single vendor support, built-in optimizations, and integration features. However, what if you are working in a VMWare ESX environment? As a consultant or an internal engineer, you cannot always dictate the virtualization environment. The following are some tried and true best practices for optimizing XenApp on VMWare ESX.

Read More, view the full article at GenV: http://www.thegenerationv.com/2009/10/optimizing-xenapp-on-vmware-esx.html

Louisville-Area Citrix Users Group Meeting Oct 21

2009-10-11T23:35:00.005-04:00

Hey guys, I hope all is well! I have not been very active lately due to numerous computer problems (and like most IT geeks, I tend not to want to work on my own PC issues...)

As I mentioned previously, There is a new Louisville-Area Citrix Users Group. Here is a link: http://www.meetup.com/citrixlouisville/

The first meeting is Wednesday, October 21st at the BBC Tap Room.

Chris Bacon is helping sponsor, so if nothing else, come down and get a few free brews on Citrix's tab! I'm planning on going, so I hope to see some of you there. Feel free to pass this on to anyone who may be interested.

TheGenerationV.com

2009-09-11T22:38:00.005-04:00

I'm happy to announce that I am joining the team over at TheGenerationV.com.

I will still be maintaining my personal blog here as time allows, of course, but will also be writing case studies and analysis articles for GenerationV.

TheGenerationV.com is dedicated to hosting discussions and technical articles about next generation technology. TheGenerationV.com was founded on July 10, 2009 by Rick Rohne, however, it is made up of some of the most talented Systems Integrators in the country.

I'm looking forward to sharing my thoughts on virtualization with this team of engineers. Please come check it out.

Louisville-Area Citrix Users Group

2009-08-14T11:52:00.004-04:00

Chris Morgan with University of Louisville School of Dentistry has started a Louisville-Area Citrix Users Group. He and I had been discussing this for months with our regional Citrix sales management team. I'm glad to say he has taken the initiative and created a MeetUp site to facilitate this new group.

If you are a Citrix user, administrator or engineer; using Presentation Server, XenApp, XenServer, XenDesktop, NetScaler, or any other Citrix Technologies; or if you are interested in learning more, please sign up at http://www.meetup.com/citrixlouisville/

Also, as new group, we are looking for topic suggestions and vendor sponsorships. Feel free to submit any ideas you may have.

Now With MTM!

2009-07-05T23:05:00.005-04:00

I'm ecstatic to announce I have joined the team at MTM Techonologies as a Systems Consultant. I had the pleasure of working with MTM as a customer for many years. During this time, I had the opportunity to get to know them top to bottom. When the time came for me to make a change, this is the team I knew I wanted to join.

I will be focusing on delivering solutions for our customers by levering my experience and expertise in Citrix Technologies, VMWare, and Microsoft Technologies. Of course, I will not be alone, I am a member of a much larger team with diverse backgrounds which allows us to deliver complete solutions.

MTM Technologies is a leading national provider of innovative IT solutions and services to Global 2000 and mid-size companies. Partnered with the top-tier technology providers and with offices from coast to coast, our unique solution-based methodology, combined with our experience and expertise, has enabled thousands of industry-leading organizations to effectively address their business needs, optimize the returns on their IT investments, mitigate risk, and focus on growth and profitability

•Awarded Citrix Multi-Location Partner of the Year
•Awarded Citrix Strategic Training Partner of the Year
•Member of Citrix XenDesktop Partner Design Council
•Member of AppSense Partner Design Council

HP NCU vs Symantec End Point Security

2009-07-04T10:38:00.012-04:00

I ran into an odd situation at a client site last week. We were implementing HP Blades, but were having problems with the teaming software. BL460 G1 blades with the NC737i network cards, Windows 2008 Enterprise SP1. Although both NICs were present, the HP Network Configuration Utility (NCU) was BLANK on 9 out of 12 servers. 2 servers were 2003 and were fine, the rest 2008 and all had the same symptoms, save 1.

So I went through the normal process of spending a day updating all drivers, firmwares, etc on the blades, the chassis, and using all of the latest Paqs from HP for the detected hardware.

After not getting any decent help from teir 1 support at HP, and not finding much luck on the forums, I asked for escalation and created a new forum entry... which netted fast results.

The full thread is here, but following is a quick summary: LMM recommened Remove/disable the tefer2 from the protocol stack or the network devices on device manager

TO FIX:

Device Manager / Show Hidden Devices
DISABLE the Teefer2 driver for the active NICs
(note, this will kill network communication)

Run the NCU
Create, configure and commit the TEAM
When the team is built, it creates a new virtual NIC

From the HP L2 Engineer, I got this answer:

This issue occurs when teefer2 driver is installed on a ProLiant Server running Microsoft Windows. Symantec Endpoint Protection 11 includes some new features as Firewall, Intrusion Prevention, Proactive Threat Scanning and Device Application Control. By default it will install the Teefer2 driver.

The Teefer2 driver is responsible for capturing all network traffic entering or leaving a particular interface (via the associated miniport driver), so that the packets may be passed to the personal firewall component of the SEP 11.0 client for analysis. This driver is listed under Local Area connection drivers/protocols and hidden under device manager and it will not work with NCU.

The moral of the story: Forums, blogs, and knowledge bases are generally going to get you going faster. If you can't find what you need, pepper all resources available until you get it working.

Quest vWorkspace

2009-06-10T22:18:00.004-04:00

Some time back, Quest Software purchased Provision Networks. They have sense re-branded and re-released their Virtual Access Suite (VAS) product line as vWorkspace.

Earlier this year, I got the opportunity to work with Patrick Rouse from Quest, he is a Microsoft MVP and a well published writer. This was a Proof of Concept to determine if Quest vWorkspace 6.x could replace our Citrix XenApp 4.5 farm. After a thorough testing and analysis, we determined Quest could replace Citrix, and possibly surpass the XenApp offerings in key areas. The next step, beyond a PoC, is a pilot roll out, which has not been completed yet due to time constraints.

However, I wanted to mention a few other things from Quest. I REALLY like this product, the company, and the engineering staff. Everyone I have met at Quest has been personable, knowledgeable and willing to answer all my questions (or hunt down the answer for me). I feel they provide an Enterprise solution, but with a start-up's flexibility and resolution time. They maintain a comprehensive blog for information release as well.

Along with ease of installation, configuration, and management; their universal printing solution and profile management make maintaining an enterprise deployment very simple. Like everything, the devil is in the details, but Quest and their staff make vWorkspace robust yet not overly complex.

Of course, vWorkspace is not just application virtualization. If it was, it would still rock. It does much more. vWorkspace is really designed to be an all-in-one broker for your virtual platforms... this includes App Delivery (via Terminal Services) as well as VDI.

My background has been primarily Citrix/Terminal Services, which is how I was first introduced to Quest. However, as I have branched more into ESX and VDI, I see more opportunity for Quest in the marketplace. Their VDI manager integrates nicely with VMWare ESX or Hyper-V (plus several other hypervisors), allowing for permanent or temporary desktops to be served on demand from templates. Since the management interface is the same, you can manage your TS APPS and your VDI from the same location, using the same toolset. As a matter of fact, you can even manage your existing Citrix servers from the vWorkspace console as well!

If you are in the market, I highly recommend checking out vWorkspace. It should be considered against Citrix XenApp, XenDesktop, or ESX View.

Additional Reading:

Check out Sean Clark's blog on his vWorkspace training.
SYS-CON Announces Virtualization Journal Readers' Choice Awards Winners:
Best Desktop Virtualization Platforms
Best VDI - Virtual Desktop Infrastructure

Latest and Greatest: My Citrix on VMWare

2009-06-06T12:50:00.004-04:00

This may be my last entry on Citrix/VMWare for the time being.

After 1 month of production runs, our 1 vCPU 2GB RAM VMs are churning nicely, averaging around 32 concurrent users. Running at 10 VMs per DELL 2950 Host, we see that Host Memory is stable, Host CPU is well utilized. There is enough capacity that another 2 VMs could potentially be added with no adverse effects, netting approx 380 users per HOST... solid density.

The only questionable item we have seen is paging on the guest level. To combat this, we established another host running 10 VMs configured with 3gb each and a seperate host running 10 VMs with 4gb each.

Early results indicate the 3gb is our "sweet spot." With the same number of users, the 4gb VMs were capping their utilization at 2.8-3.2 GB Used. Also, considering the hosts only have 48gb, consuming 30-36gb (10-12 VMs @ 3gb each) still allows overhead for the host, for vmotion, and any auxilliary or even additional VMs without the risk of oversubscription. The 4gb VMs, although with slightly more horsepower, forces the host to limit to 10 or 11 vms at most, since we want to avoid over subscription.

In summary:
10 x 2gb VM -- increased CPU due to paging, no other problems. Potential for 15+ VMs
10 x 3gb VM -- less paging. Potential for 12-15 VMs, based on total resource consumption
10 x 4gb VM -- solid performance, but peaked memory just over 3gb. 10-11 VMs max per host

Please note, this is within our framework with our specific custom application. One caveat of the 4gb servers is we did NOT increase the load evaluator, so the number of users was still capped. Increase the user load may allow for more users per VM to leverage the additional RAM.

Early Returns: 3gb looks to be the optimal configuration for this environment based on user load, vm performance, and esxtop results. It would also maintian the focus of scaling out instead of scaling up.

Follow up: Unfortunately, this will be my last post for this specific case study. I am moving on to different projects with a different company. However, I will post new case studies and update this study if and when I receive any additional results.

Rolling out updated Citrix VM Template

2009-04-21T19:40:00.001-04:00

I have updated our Citrix VM Template:

Added CD, left disconnected
Updated Tools, included Memory Manager (aka Balloon Driver) -- after reviewing this time and again with various engineers, I have come to several conclusions:

a lot of the info out there is still based on ESX 2.x
The Balloon Driver is a safety net, which should not be normally called
on when designed properly
when in doubt, and until proven otherwise, go with the standard package

Tweaked the service controls
Added PowerShell 1.0 (I'm becoming a big fan of PS scripting)

With this updated Template, I am deploying 8 new VMs for Production Load. These are deploying tonight, sanity check tomorrow, full load by Thurs hopefully.
My plan is to have these 8 machines, plus 2 existing production Citrix VMs (10 total) on the same isolated ESX 3.5 U3 Host, all running on iSCSI SAN LUNs. This should tell us how close to accurate our forecasts are. Our goal is to run approximately 30-35 users per VM, 10-12 VMs per host... this would give us a user density of at least 300 users per physical host, but as high as 450, so we should have some built-in growth and scalability.
However, forecasted numbers are just an educated guess. By loading a dedicated host full, we can monitor the impact on the front and back ends. Our concern is being CPU bound on the host more so than being Memory Bound. The ESX host in question is a Dual Quad-Core Intel (8 cores) with 32gb RAM. Each VM will have 1 vCPU and 2gb RAM.
To monitor this, we will be using:

ESXtop on the ESX Host to monitor host-level performance
vCENTER real-time stats to graphically display and monitor the individual VMs
Citrix Load Evaluators (# of sessions) to monitor user distribution
End-User Experience of performance

With the ESXTop and vCenter, we will specifically look at:

%READY (the amount of time a VM was waiting on processor)
%EUSED (how much of the physical processor power the VM is consuming)
%MEM (how much memory is used by a VM)
Host CPU Load Average
Memory Balloon bytes (to watch if the balloon driver is getting called, and determine if it is helping or hurting performance)

Based upong the VM stats and the ESX Host stats we can determine if we have adequately sized our virtual and physical servers. This will help us understand if we have added capacity, or if we are oversubscribing. For more on ESX Top, see this pdf from VMWare. Sure it is ESX2, but the information is still valid.
More to come later, but if you have direct experience, drop me a line and let me know what you have encountered....

An Interesting Day with Citrix Load Evaluators

2009-04-21T19:16:00.001-04:00

Ever have one of those days? Well today was it for me! We run XenApp 4.5 in 2 geographic production Zones. I get an urgent page from support that EVERYONE hitting one of our Zones was getting "... servers are reporting full load and cannot accept your connection ..." EEK!

For our Physical Servers, since our full migration to 4.5 a year ago, we have use an 80/20 evaluator (which I have mentioned before), but to review:

CPU Utilization: No load at <= 20%, Full Load > 80%
Memory Utilization: No load at <= 20%, Full Load > 80%

Pretty straight forward. Just an hour before the emergency, I checked my QFARM /ZONELOAD and saw the servers were running with load values between 7000 and 8000, the highest at 8140. Ok, running hot and heavy, but manageable. Then BAM!

In a closer analysis, the servers were all on the tipping point of that 80% threshold for memory consumption. So one server running a 7000 with the load split evenly between Memory and CPU, while another server could be running at 7000 with the load entirely Memory bound. All it takes is one more heavy process to tip the scales. This is what happened to us. Of course, as soon as one server reports full load, the others get hammered, creating a domino effect.

In looking at the servers, they were running between 85% and 93% physical memory utilization and 90% page file utilization. OUCH.

To ease the immediate burden, we changed the load evaluator from 80/20 to DEFAULT (100 Users Max plus Load Throttling). Although this does not fix the memory consumption, it at least allows new connections. They may suffer from lack of memory, but a little is better than nothing in a pinch.

What are we doing about it? We are ramping up our Citrix to VMs process, deploying 8 new VMs (more on that later). This will help us spread the load across more units. We have decided to keep the 80/20 rule in place on the physical servers for now since it has worked well for 18 months. We feel a resource-based evaluator (Mem, CPU) is better than an arbitrary one (User Count). We have also discussed using the built-in ADVANCED evaluator or creating a Memory only load evaluator, but have decided to keep status-quo.

What to take away from this? Understand your environment and applications to make sure you have a proper load evaluator in place. Also understand that looking at /ZONELOAD or /LOAD counts is not enough (this was our mistake). You need to look at the different components of the load counter to make sure you are not operating under a false sense of security.

Use the Load Monitor! I have to admit, since going to XenApp 4.5, I live in the Citrix Access Management Console (AMC) and rarely if ever go into the older Presentation Server Console (PSC). In the PSC, under the servers, you can look at the Load Monitor, which shows you exactly how the load value is calculated. So if you are using 2 criteria, say CPU and RAM 80/20 like me, you can see that a 70% load might be 67% Memory Usages and 27% CPU Usage. In this case, you do not have 3000 load to give before hitting the full 10000 mark, you only have 13% Memory to give before you hit the magic 80% mark, forcing a 10000 load report.

More on Citrix Load Evaluators

2009-04-17T21:39:00.001-04:00

Well, most of feed back I received from other industry players confirmed my initial suspicions... counters based on Windows Perf Counters where Windows is a VM are not reliable. A local area architect, who is not married to any one technology, summed it up nicely:

"You have touched on something that is inherent to the virtual world: the performance counters are always wrong. Exactly how wrong, that is debatable. In the past, I have used empirical numbers to control load. That means keying off of something like user count, memory utilization, or page swaps. It isn't an exact science, but some testing should give you the right mix."

Based on my research, combined with various feedback points, I set the following:

Load Throttling – High
Server User Load – Report Full Load at 45 Users45 Users

This is really just a tweak of the DEFAULT load evaluator, but works for what I am after.

Now, you may ask, “Why 45?” Well, our physical servers typically run with load values ranging from 6200 to 7400 based off Memory and CPU Utilization. This generally supports 65-70 sessions. When creating our specs for the VMs, we targeted 30 users, but the actual running values have gone as high as 55 before completely tanking.

So, if the average load eval is around 7000, 30 users (our target) is 70% of 43… round up to 45. This would keep our VMs running at 30 users in a comparative fashion to the physicals, while allowing a 50% cushion for spikes.

Using vCENTER to monitor vCPU and vRAM performance, and WMI/SNMP to monitor/trend the load value and # terminal server session has shown positive results. Prior to the load evaluator change, the VMs would peg the vCPU and become sluggish or completely unresponsive, user sessions would climb as high as 55. Since the change, the vCPU will occasionally spike to 100%, but is averaging around 75%. Users sessions are averaging around 32 users. Right on target.

As more VMs are added, and the load is distributed more evenly, that will be the true test. My next goal is to start collecting vCenter stats using PowerShell and writing the results to a SQL database. I will use that to correlate and trend the actual values for capacity planning.

I will also soon be building out a full ESX Server hosting only XenApp workloads. This will let me see how many of my optimized VMs I can efficiently run on one host. This number should range between 10 and 15, I'm anticipating a comfort level around 12, but time will tell. This should net me 300 to 350 users per HOST.

Citrix Load Evaluator Rules in a VM

2009-04-10T21:29:00.001-04:00

I mentioned in my previous post that one of my VMs was pegging at 100% of the vCPU, but the Windows Task Manager was showing only spikes (although a lot of them), and my Citrix load value for that server was in the 7200 range. Of course, the system was crawling and unusable for customers.

This highlights a concern I have had for some time... how to measure the load of Citrix VMs using the traditional load evaluators. The reason for the concern is that most of the criteria available is based on Windows coutners... and it is commonly know that GuestOS counters are inaccurate in a VM. From VMWare:

The problems seen as a result of usage of traditional in-guest performance measurements come from three areas:
They are unaware of work being performance by the virtualization software, they will not have complete information on the resources being used by the virtualization software. This includes memory management, scheduling, and other support processes like the service console in ESX.
The way in which guest OSes account time is different and ineffective in a virtual machine.
Their visibility into available CPU resources is based on the fraction of the CPU that they have been provided by the virtualization software

I currently use a traditional 80/20 Evaluator (80% or more = Full Load, 20% or less = No Load) for both RAM and CPU. However, in a VM, these are virtualized, so this count CANNOT be accurate. That is an inherent flaw in the system. It also explains why my servers which are scaled to handle around 30 or so users are running as high as 55 users. I would expect some variance, maybe 20-25%, but 80% does not seem right.

I think the fault lies within the flawed measurement. The only counter that would be worthwhile is the # of connected users... but that brings back a flaw from older monitoring that only measures users and not activity.

So, which flawed system to use? Next week, I plan to implement a # USERS based load evaluator, then use ESX TOP and vCENTER to monitor the true CPU and MEMORY Utilization against the actual number of users to find the sweetspot. I think I will set the FULL LOAD at 40, so an 80% load will be 30 users, which is my target, but allowing enough cushion for more users in a pinch.

If you are running Citrix in a ESX environment, I'd love to hear form you on what you use for load evaluation and resource monitoring.

Update on My Citrix VMWare Machines

2009-04-10T21:18:00.002-04:00

Ok, if you have been following my blog, you know I have been researching and building the best possible Citrix VMs to run on my ESX Hosts. Granted, XenServer might be better for XenApp, but we work with what we have, and I have ESX 3.5.

I currently have 4 production VMs running, 2 in each production Zone. The first two in DataCenter1 went live about a month ago... and churn nicely. They will routinely show a load evaluation of 10000 (Full Load), and the 1 vCPU Utilization is fairly high (based on vCenter stats), but no complaints from users. Initial test = PASS.

A week ago, using the same template and automation scripts, I deployed 2 VMs to DataCenter2. One of these VMs became unusable. The load evaluator was showing around a 72% usage, but vCenter was showing the vCPU pegged and not letting up. An RPD session was very sluggish, to the point of being unusable. The next day, the other VM in that zone blew up with licensing errors. I have since removed those from production for futher testing. Second test = FAIL.

What is strange, same template, same processes, same relative user base and load, drastically different results. Granted, 4 machines is not a larger sample size. I plan on some load testing next week, will post results on that later.

Updates on ESX Templates and Citrix on VMWare

2009-03-10T21:00:00.001-04:00

From my previous posts, you can see I am digging in on ESX best practices, specifically for Terminal Services/Citrix.

I will be posting an updated white paper, entailing my best practices, methods, and results soon.

However, in the mean time, Project VRC is now LIVE. Great info comparing ESX, XenServer, Hyper-V, and Bare Metal. According to their tests, they recommend 2 vCPU instead of 1 vCPU... and they present a compelling argument.

For now, I am moving forward with 1 vCPU and 2 GB RAM to start a baseline, but will add a 1 vCPU 4GB RAM and a 2 vCPU and 4GB Ram for comparision. I'll include the results in my summary paper once it is available. If you want the prelim finding, just email me.

In the meantime, check out Project VRC

Creating and Deploying a Citrix Template on VMWare ESX

2009-02-21T14:41:00.001-05:00

Similar to my other related posts, this is specific to my operating environment and may not fit all. However, the best practices referenced here should help in most environments.

Please note, we use the drive remapping (C --> M) as well as direct addressing.

Creating the Citrix GOLD Template

Base OS configuration:

Create the base VM per my Citrix on VMWare article.
Update OS (Windows 2003 R2 SP2) to the latest updates
Windows Server Configuration: Using Add/Remove Programs, Windows Components:
If installed, remove Remove .NET Frameworks 2.0 & 3.0
Application Server: Message Queuing (default)
Management and Monitoring Tools: Simple Network Management Protocol
Uncheck Internet Explorer Enhanced Security Configuration
Terminal Server / Full Security / Use Auto Discovery / Per Device Mode
Windows System Configuration: Modify the System Properties:
Advanced > Performance Settings > Data Execution Prevention: essential programs
Advanced > Performance Settings > Virtual Memory: Initial & Maximum: 6144MB
Advanced > Environment Variables: update paths for custom applications
Advanced > Error Reporting: Uncheck Force Queue Mode
Remote > Check Enable Remote Desktop
Copy App Install sets to local Data Drive\INSTALLS

Application Configuration:

Run DriveReMap.exe from INSTALLS\PS 4.5 INSTALL directory, assign C to M (will require a reboot)
Supporting Applications: Use the content copied to the INSTALLS folder
Framework 2.0.50727.42
Framework 3.0
Frameworks 3.5 SP1
Install SNMP Informant & Configure SNMP Settings (if used)
Copy any standard tools (including PSKILL, PSEXEC, UPTIME)
Install the any viewer Applications: Office, Notepad++, Acrobat, etc.
Install Citrix Presentation Server 4.5 for Windows 2003 (assume an existing Farm)
Apply Approved Citrix Patches for Presentation Server 4.5
Hotfix Roll Up #3
Configure Citrix Services (deviation from defaults are listed):
Citrix ActiveSync Service: DISABLED
Citrix Print Manager Service: Automatic. Set to Run as (DOMAIN ACCOUNT)
Citrix XML Service: DISABLED
Check for any additional Windows Updates
Apply Citrix Registry Changes, as needed, such as:
Seamless Windows
Citrix Universal Printer
System Page Pool Manager
System Beep
Modify Printers and Print Drivers
Remove any unwanted printers and drivers
Install any required native print drivers
Delete Contents of INSTALLS directory
Delete Service Pack Uninstall Files
Clear all Event Logs
Empty Recycle Bin
Stop the Following Services and set to Manual Startup:
Citrix IMAService
Citrix SMA Service
Citrix Print Manager Service
Shut down VM

Prepare and Deploy Template

Clone VM to Template
Deploy VM from Template, select Customize
Using SysPrep from customization, assign Machine Name and join Domain
Assign Static IP and AltAddr (if necessary)
Move computer account to proper OU
Run DSMAINT RECREATELHC
Set IMA, SMA, and CPSVC to Automatic
Reboot Server
Verify Farm Connectivity (QFARM)
Verify Remote Desktop Connectivity
Assign Election Preference and Zone in Presentation Server Console
Assign Load Evaluator, Hierarchy, and Published Desktop in Access Management Console
Verify ICA Connectivity, local and remote
Test/verify pre-configured applications
Perform any final application or system updates
Publish necessary applications

Some final thoughts: if you are using multiple geographic zones, I recommend having a separately prepared template for each Zone with the proper database connection and zone name pre-configured.

Citrix on VMWare ESX

2009-02-21T13:32:00.001-05:00

Although these guidelines are created using the following environment, these standards should apply to just about any Terminal Services-based virtual environment (XenApp 5, Quest vWorkspace 6, 2X, Windows 2008 TS, etc.):

VMWare ESX 3.5 Update 2
Virtual Center 2.5
Windows 2003 R2 SP2 Standard Edition
Citrix Presentation/XenApp Server 4.5

Virtual Machine Settings

1 vCPU
2 GB RAM
HDD1: 15 GB vmdk on VMFS (used for OS)
HDD2: 15 GB vmdk on VMFS (used for data, increase as necessary)
Delete the CD/DVD and FLOPPY drives from the VM
Disable Sound
VMTools: Do NOT install the Balloon Driver

Virtual Machine, Operating System

Windows 2003 R2 SP2 x86 STANDARD
Latest service packs and critical updates
6 GB Page File on Data Drive
Disable any unused services
Citrix Presentation Server/XenApp Server 4.5 RollUp 3
Standard applications & toolsets

Further explanations & considerations:

Create a template, then deploy from a SysPrep'd Template. The template should follow best practices, including aligning the drives, registry settings, etc.
Windows will scan the CD and Floppy drives, even if they are empty, by removing the drives, you reduce OS overhead. These can be added back later when needed, but when virtualizing, every little bit helps.
Not installing the balloon driver (knows as the Memory Controller in the install set) will reduced the memory swapping which wreaks havoc on Citrix in a VM environment.
Because of I/O demands, placing the VMDK files on a RAID10 set is optimal.
Consolidate similar workloads where possible, i.e., have a cluster of ESX hosts dedicated to ONLY your Citrix Farm. This will optimize VMWare's native memory sharing as well as streamlining I/O.

Finally, as a general Rule of Thumb 2 Citrix VMs = 1 Physical Citrix Server. Obviously, this will vary based upon the actual workloads and application demands. As your farm grows, Scale out, not up…

Additional Reading:

For more info on templates and other settings, see my other article on VMWare Best Practices.
Vincent Vlieghe's most excellent article, detailing many of these settings.
The Ether's take on optimizing Citrix on ESX

ESX Virtual Machine Best Practices

2009-02-21T12:13:00.002-05:00

My primary focus is server-based computing, specifically Citrix Presentation/XenApp Server. With a stronger push for virtualization in most market segments, I've seen mixed results for virtualizing Citrix on VMWare.

Several vendors and consultants have recommended using XenServer (now from Citrix, formerly from Xen Source) which is better at handling XenApp workloads. My question as been, WHY? What are the key differences and why the performance difference? The key is Memory Sharing and how VMWare allows overallocation of resources.

Anyway, after a lot of reading, research, and meetings, I've put together a few best-practices. This article will focus on general VMWare ESX best practices. I'll also be posting an article specifically for optimizing Citrix on VMWare, and then more articles pertaining to rapid deployment.

With out further ado... here are my thoughts on ESX Best Practices:

USE TEMPLATES

Templates allow for rapid deployment of virtual machines from a stock image. The idea is to build a "GOLD" template for your basic servers, then deploy the templates and add any extra software for the specific function of that server.
Keep Templates small (1 vCPU, 512 RAM, 2 vHDD - seperate OS and Data drives). It is easier to grow the virtual machine after deployment.
At a minimum, these Templates should include each flavor of OS you need, including all the latest server packs and updates, VMTools, as well as any standard tool sets (7Zip, Notepad++, Acrobat, PSTools, Resource Kits, etc.)
Using Virtual Center, import SysPrep to allow customization on deployment. This allows you to package the base image, rename and IP the machines at deployment.
Templates should include the other best practices listed here.
More info on Templates is available in this whitepaper.

ALIGN DISKS

A common practice in high I/O systems, such as SQL and Exchange, is to align the disk partitions to the physical drives. This practice is critical for optimal VM performance.
Properly aligned drives increase throughput and decrease latency.
For Windows guest OS, use DISKPART to create a 64k Partition, which aligns with the 64k partition of the hard disk
Format the Windows drives using 32k allocation unit sizes
This optimally uses the space and prevents the data clusters from spanning multiple blocks, which then span drive chunks.
I recommend aligning the drives on TEMPLATES (above), so you can set it and forget it.
VMWare has a nice whitepaper published here, regarding partition alignment.

OTHER CONSIDERATIONS

For Terminal Server Apps, disable the Balloon Driver from the VMTools. This is critical for Citrix, but useful for any TS deployment to avoid Memory swaps and high CPU hits. More on this here.
Use separate OS and Data drives, place the PAGE File on the Data Drive
If possible, use Separate LUNS or Drive Arrays for the OS and Data drives
OS and small data drives (<200>Data stores (File shares, Exchange, SQL) and large drives should use RDM
Adjust the Disk timeout valie to the storage vendor's recommended: HKLM/SYSTEM/CurrentControlSet/Services/Disk/TimeoutValue = REG_DWORD Hex value 3c (60)
Disable Last Time Access Atrribute for NTFS, this setting that keeps track of the last time a file was accessed. Removing the necessity for the system to keep reading and writing this information may speed up performance. The command is: fsutil behavior set disablelastaccess 1

In summary, each environment is different. These are the steps I have identified for use in my environment. They may or may not be best for yours. If you apply these steps, you should see a relative performance increase in your VMs. Ideally, apply all of these to your master Templates, then deploy away.

Additional Resources:

Disk Alignment -- http://www.vmware.com/pdf/esx3_partition_align.pdf
ESX Templates -- http://www.vmware.com/pdf/vc_2_templates_usage_best_practices_wp.pdf
ESX Memory -- http://www.vmware.com/pdf/esx3_memory.pdf

Jack-of-all-Trades, Master of None?

2009-01-17T23:21:00.001-05:00

I've always considered myself a Jack-of-all-Trades, but a master of none. I have purposely tried to walk a fine line between business and technology. That's one reason I chose to pursue an MBA instead of just another certification. It gives me portability beyond just IT, as well as better upward mobility on the corporate latter.

I also think being well rounded allows me to act as a better SOLUTION Architect. Having been a programmer, a DBA, a systems engineer, a network administrator, a trainer, and a manager ... I tend to look at problems differently than others that may be more specialized that I am. I take more a holistic approach.

However, for all the positives this may present, there are negatives. I find that when pressed with high-level problems to solve, I tend to lean toward a "working solution" versus a true root cause analysis. Although this usually suffices, some problems do creep back up. The problem is, there are many issues that are simply beyond me. Granted, I can troubleshoot and Google with the best of us ... but if I were more specialized, perhaps I would be faster to the fix.

That being said, I have found myself over the past several years focusing more and more on server-based computing ... thus this blog. I've been managing Citrix systems for about 7 years now ... ranging from Metaframe XP up through XenApp 5.0. During this time, I have also examined MS Terminal Services 2000, 2005, and 2008; as well as Ericom PowerConnect, 2X, and (Quest) Provision Networks.

During this same time, I've seen my SQL and Programming skills wane. I try to keep up, but not at the same level. If you don't do it everyday, its hard to stay current on the latest techniques. I'd be happy to write sql queries and do database and table design... but I would not be the candidate for enterprise grade optimization.

I've backed into Systems from Programming/DBA work... where once I considered my self merely an admin and not an engineer, I fully embrace myself as an Engineer now. Even so, I try not to lock myself too tightly into one technology or service. Each to their own... but I find I work best when wearing multiple hats.

All of this works great in a corporate environment, where we are expected to wear many hats, and perform multiple roles. Especially in a down economy, most technology departments expect their staff to do more with less. Now, as a consultant, you are expected to be the top of your game... if you bring in a database guru, do you care if they know the OS?

For me, I'll keep going with the flow... if it leads to a true specialty, so be it. For now, I'm happy where I am and what I do. What about you?

For other thoughts, check Computer Contractors - Does it Pay to Specialize, by Joshua Feinberg.

SBC Resources

2009-01-16T00:38:00.002-05:00

Here are some good links/resources for Server-Based Computing:

Doug Brown's site: http://www.dabcc.com/ ... great resource with original content and site aggregation
http://www.msterminalservices.org/ One of the best resources for all things SBC.
Patrick Rouse's site, some good articles and a nice little FAQ: http://www.sessioncomputing.com/
Brian Madden. One of the single most useful sites for SBC, ever. If you do anyting with TS or Citrix, you will want to reference his site.
Ctl-Alt-Delete, great TS/Citrix utilities.
ThinComputing, Michel Roth's site
http://www.thomaskoetzing.de/, my resource for scripting and web interface customizations... very insightful
http://sbc.vanbragt.net/mambo/ -- wonderful take on SBC

Blogs and other useful links:

I'll update the list as I think of other useful resources. If you have something I missed, please post a comment.

Degree vs. Certification

2009-01-15T23:56:00.001-05:00

A common question in IT is, which is better -- a certification or a degree? Unfortunately, there is no right answer. In my mind, it is a combination of both... but mostly is about whether you can do the job or not. Personally, I have a double master's degree (MBA, MS IT), but very few certs (A+, MCP). I chose to focus on college and project management, and have found little need for industry certs. I fill the role of a Sr. Systems Engineer... even though I may not be an MCSE, go figure!

When I was in college, I questioned my COBOL professor why I was even in school, learning either dead languages or items that would be obsolete by the time I graduated. He told me the degree showed both a commitment as well as a foundation of knowledge companies would look for. I understand that now, but still do not completely agree.

When looking at hiring managers, I've meet certification snobs, degree snobs, and those who focus solely on skills/roles. I personally do not put a lot of stock in either degrees or certs when I am hiring ... simply for the reason those are only pieces of paper.

The Certification Snob. I had an encounter a few years ago in a training class. An individual was in Product Management, and had worked at IBM for 20+ years. He was contemplating MBA or MCSE. I felt neither would add anything he did not already have, but between the two, I felt MBA was better from a marketing/sales perspective. Another attendee vehemently disagreed. When he heard I was only A+, he told me I was not qualified to do my job...

The Degree Snob. Another encounter I had was as a hiring manager. Another up level manager's first question was "What degree does the candidate have..." Nevermind the person had 15 years in the industry and was highly recommended, if there was no degree, he had no interest. The lack of a four year degree was a deal breaker, and the search went on...

Beware of Blank Paper! Even if someone has a degree or a certification, it does not make them qualified. I have personally seen MBA's who could not do simple math or have an intelligent conversation. I've also had Cisco Network Engineers who could not implement a new router on a new T1 circuit. The paper only means they have passed the necessary assignments or tests, it does not mean they really know what they are doing...

So, which one? The true answer, is "It Depends." It depends on your background, your skills, your time table, and the company's needs. I recommend:
Get a cert when... your company will pay for training and tests (but not college classes), you need a rapid resume or job boost, you already have the degree, want to specialize/focus in specific technologies
Get a degree when... your company has a tuition program (take advantage), you are looking for long term, you do not want to pidgeon-hole yourself in a specific technology, prefer to work in theories and general concepts

Andy Says... Get Both. The investment in time and money will pay off in the long run.

Other reading. For more information, check out this article from Greg Shields over at WindowsServer.com. Also, Vault has a nice comparative analysis here.

Welcome

2009-01-15T23:40:00.001-05:00

Welcome All!

After researching numerous blogs, I've decided to jump in and start my own. The primary purpose will be to document and comment on information technologies. These posts will be a way for me to combine various tid-bits of data from across the web and multiple resources... hopefully it will be useful for others as well.

My current interests lie in overall System/Solution Architecture and are not dedicated to any specific technology. I am currently a MS Systems Engineer specializing in Citrix XenApp. However, I believe in using the best tools available at any time, regardless of vendor.

If you have any questions, or specific topics you would like to see, please comment.