Virtualization Eh

I’ve recently been having some great conversations around different storage protocols and their performance differences/capabilities.  In that light, I wanted take this opportunity to poll the larger community as well.

Please let me know which is your favorite protocol and why (in the comments).

An excellent reference/comparison was done by Cormac Hogan and can be downloaded here.

Once again, VMware has pulled together an incredible team of Performance resources to revise and then release a new Performance Best Practices guide for vSphere 5.1.  Kudos to the team!

Available here: Performance Best Practices for VMware vSphere 5.1

There is a lot of great info in here so it’s hard to pick highlights:

Availability of the C1E halt state typically provides a reduction in power consumption with little or no impact on performance.  When “Turbo Boost” is enabled, the availability of C1E can sometimes even increase the performance of certain single-threaded workloads.  We therefore recommend that you enable C1E in BIOS.  However, for a very few workloads that are highly sensitive to I/O latency, especially those with low CPU utilization, C1E can reduce performance.  In these cases, you might obtain better performance by disabling C1E in BIOS. [page 15, Power Management BIOS Settings]

This really acknowledges the fact that performance tuning is an ‘art’ because it involves understanding your workloads, what they’re doing and changing parameters that have trade-off affects.  In this case, as long as applications hosted on this physical asset are not sensitive to latency, leaving C1E enabled could give you a small performance advantage.  However, if you don’t know what a particular workload requires, you may be inadvertently impacting it.  Mark’s rule: “Unless you know what you’re doing, defaults are best!”  Believe it or not, a lot of time is spent deciding what defaults values should be based on experience, averages, workload safety, etc.

ESXi 5.1introduces virtual hardware version 9.  By creating virtual machines using this hardware version, or upgrading existing virtual machines to this version, a number of additional capabilities become available.  Some of these, such as support for virtual machines with up to 64 vCPUs and support for virtual GPU hardware acceleration, can improve performance for some workloads.  This hardware version is not compatible with versions of ESXi prior to 5.1, however, and thus if a cluster of ESXi hosts will contain some hosts running pre-5.1 versions of ESXi, the virtual machines running on hardware version 9 will be constrained to run only on the ESXi 5.1 hosts.  This could limit vMotion choices for Distributed Resource Scheduling (DRS) or Distributed Power Management (DPM). [page 17, ESXi General Considerations]

I know many people excited to start upgrading their clusters.  Keep things like this in mind when planning for cluster wide migrations.  When using mixed versions of software and virtual hardware there will be limitations.  Be sure those are managed timely.

Be careful when using CPU affinity on systems with hyper-threading.  Because the two logical processors share most of the processor resources, pinning vCPUs, whether from different virtual machines or from a single SMP virtual machine, to both logical processors on one core (CPUs 0 and 1, for example) could cause poor performance. [page 21, Hyper-Threading]

I’ll go even a step further to suggest pinning is a bad operational practice and will cause you issues.  Leave pinning to the benchmark experts and teams but in day-to-day operational life the value you’d receive if often negated by the level of management required to ensure that workload is not impacting other services within your infrastructure.

For the best iSCSI performance, enable jumbo frames when possible.  In addition to supporting jumbo frames for software iSCSI, ESXi 5.1 now also supports jumbo frames for hardware iSCSI.  Using jumbo frames with iSCSI can reduce packet-processing overhead, thus improving the CPU efficiency of storage I/O. [page 32, iSCSI and NFS Recommendations]

Yes enabling jumbo frames can aggravate the network admin but it is valuable for NFS and iSCSI, especially with 10Gb links.  Take your network admin out for lunch or beverages one day – breaking down the virtualization/network silo is the next operational challenge (see software defined datacenter).

So download this guide, get to know it intimately, it’s one of you core performance resources.

Its been quiet recently…very quite…because something BIG was being created…

Finally I’m able to share some of my excitement with the virtualization community around VMware’s next major release – vSphere 5.  You all know I have a passion for performance so, today we start a new journey around education on just what is possible on this Cloud Infrastructure.  Here’s the start:

What’s New in VMware vSphere 5.0: Performance Whitepaper

Some very cool technical highlights of the ‘Monster’ VM:

I used to say one could confidently virtualize 99% of their workloads, but after today’s release, I’m now suggesting one can virtualize 99.9% of workloads.  Over the coming weeks you’ll start to see a stream of performance data demonstrating this enormous capability.

“And one more thing” Eric just posted:

vSphere 5 can virtualize itself and 64 bit guests – Sweet!

I’m excited to highlight that VMware, specifically Chethan Kumar, has released an updated Performance Troubleshooting guide for vSphere. The document was originally penned by Hal Rosenberg and has been referred to as the ‘Performance Troubleshooting Bible.’  Both of these fine gentleman work as performance engineers within VMware and have strove to identify common issues and potential resolutions in an easy-to-use framework.

This new edition has been updated specifically for vSphere 4.1 and should be part of every virtual administrators core collection. Whatever you call it, all virtual administrators should not only review this guide for the cool performance information contained within, but it should be used as Step #1 for troubleshooting any performance related issue.  Using it will help you diagnose all the most common issues seen in the field.

Abstract:

This document provides step-by-step approach for troubleshooting most common performance problems in vSphere-based virtual environments. The steps discussed in the document use performance data and charts readily available in the vSphere Client and esxtop to aid the troubleshooting flows. Each performance troubleshooting flow has two parts:

1. How to identify the problem using specific performance counters.
2. Possible causes of the problem and solutions to solve it.

I reference this guide at the end of all my presentations and will be updating my links to the new edition.  Performance is often seen as a dark art but in reality is often misunderstood. I hope resources, like this guide, will help take everyone’s knowledge to the next level through awareness and practical application.

The document is available here: Troubleshooting Performance Related Problems in vSphere 4.1 Environments

If you have any feedback, feel free to drop me a line and I’ll pass it along.

I’m very excited to be presenting at VMware’s Partner Exchange 2011 in Orlando.  In this session I’ll be covering why I’m confident you can virtualize your Tier 1 apps on the vSphere platform, practices & configurations to consider and some common troubleshooting steps.

Session Details:

• Location: Coronado DE
• Time/Date: Thursday Feb 10 at 2:00pm

So for all you VMware partners out there, I hope you can attend and reach out to me while you’re down there so we can chat performance.

For more information on PEX 2011 and Registration check out this link.

Remember:  Performance is not a barrier.

I was on a road trip recently and had a number of discussions about consolidation ratios as a measure of success.  This worried me.  Especially considering that these were large virtual adopters and I was talking with these clients about moving their Tier 1 applications to their virtual platforms.  It made me realize that a number of organizations are still using that ‘old’ ratio as a key performance indicator (KPI) for measuring how successful they are at virtualization.

Yes – back in the day consolidation ratios were a common measure often used for capacity management, generating ROI models and generally bragging rights.  This was fine when the only services being migrated were “low hanging fruit” or “extremely” underutilized servers.  One could expect 10:1, 25:1 even 50+:1 as a ratio.

But can we keep up those high ratios as we migrate Tier 1 services to virtual platforms?

Easy answer: No!

For the obvious reason that these larger workloads are actually using the hardware resources underneath them (being physical or virtual) so when we start to stack them side by side there are fewer resources to go around.  But this is not a bad thing.  A lower ratio does not mean you’ve failed.  It seems an obvious message but one I wanted to re-iterate as some seem to miss it.

Lets make sure our vision and focus for virtualization goes deeper than consolidation.  Don’t forget about the many other cool things virtualization brings like:  DR enablement, unlocking multicore platforms, simpler management and capacity in the cloud.

Merry Christmas Everyone!