A few weeks back I was called in to help a customer who was experiencing problems completing Jetstress testing for an Exchange 2010 deployment. It wasn’t an issue of Jetstress reporting failed tests. Rather, they were unable to get through most of their tests without the Jetstress application actually crashing (JetstressWin.exe has stopped working). They would see the following after the Jetstress testing completed but before it could write any log files to disk.
The only Jetstress related error in the Application log was an ESE error with Event ID 482:
JetstressWin (3584) Instance3584.6: An attempt to write to the file “F:\DB\Jetstress006001.edb” at offset 63087017984 (0x0000000eb0478000) for 32768 (0x00008000) bytes failed over 0 seconds with system error 1117 (0x0000045d): “The request could not be performed because of an I/O device error.”. The write operation will fail with error –1022 (0xfffffc02). If this error persists then the file may be damaged and may need to be restored from a previous backup.
During the process of Jetstress completing a test run, it generates a large amount of I/O as it flushes anything in cache to disk. It was at this point that the Jetstress application was crashing. This behavior is normal but it’s an important clue because of the high disk I/O generated.
The customer was using vSphere 4.1 and the Exchange 2010 Mailbox servers were each configured with PVSCSI virtual SCSI controllers using VMDK files. As it turns out, they were hit with the PVSCI bug described in this VMware KB:
Windows 2008 R2 virtual machine using a paravirtual SCSI adapter reports the error: Operating system error 1117 encountered http://kb.vmware.com/kb/2004578
The interesting thing to note here is that although Exchange is specifically called out here in the KB, it doesn’t mention that it may cause the application (in this case Jetstress) to crash. The crashing led the team to troubleshoot Jetstress initially, thinking something was wrong with Jetstress and the various DLLs it requires to run.
At the end of the day the issue was resolved by following the instructions in the KB and changing the virtual SCSI driver to LSI Logic SAS. After making that change there were no subsequent issues with Jetstress.
In case you haven’t read the KB linked above, I want to note that this issue is resolved in all versions of vSphere from 4.1 to 5.0. You’ll need to install the updates described in the KB if you want to use the PVSCSI driver and vSphere 4.1 through 5.0 (it is resolved in vSphere 5.1).
Hopefully this helps anyone who might be experiencing this issue. I also hope it doesn’t dissuade anyone from using the PVSCSI driver for their business critical applications, as it can deliver better performance with lower CPU utilization when high I/O workloads are virtualized.
Microsoft first released the Server Virtualization Validation Program back in 2008 to help validate their products running on server virtualization technologies (both Microsoft and 3rd party). Followed soon after that release was a tool called the SVVP Support Policy Wizard which made it very easy to simply plug in your application, OS version, and desired hypervisor and out would come a support statement.
Recently Microsoft removed the SVVP Support Policy Wizard and replaced it with a more generic support stance for its operating systems and applications. If you go to the site for the SVVP wizard (http://www.windowsservercatalog.com/svvp.aspx?svvppage=svvpwizard.htm) you are instead redirected to page called “Application Support in Virtual Environments Policy.”
The page lists out all of the Microsoft products that are supported on either Hyper-V or third party hypervisors validated in the SVVP. The key point they make is in the very first paragraph (emphasis mine):
All the below listed Microsoft Server-focused application versions, as well as all later versions of those applications, are supported on Hyper-V and Server Virtualization Validation Program (SVVP) validated products, so long as the virtualization product supports the correct operating system version and platform architecture(s) required by a specific application.
To make the point more clearly, they go on to say:
Any virtualization product listed in the catalog that supports the correct Windows Server operating system Version and Platform Architecture is supported for the applications listed below.
If there are application specific requirements, those are listed in a separate link next to the specific applications. If not, then you can feel confident that the application is supported.
Even better, they go on to state that features like memory ballooning, VM fault tolerance, or live migration are out of the scope of the SVVP since those features operate “without the knowledge or cooperation of the operating system or applications executing within the virtual machine..” Microsoft goes on to say the following:
However, unless otherwise stated in articles or documents for Windows Server operating systems or for Microsoft Server-focused applications, Microsoft does not preclude their use by customers.
Microsoft is saying that features like memory ballooning, live migration, etc., are fully supported to be used with the applications listed in the SVVP unless otherwise stated. They do state that if you run into an issue with these features you may have to reach out to the hypervisor vendor directly for support. In other words – if memory ballooning broke your Exchange server, call VMware. Remember though that certain features, like virtual machine snapshots, are still not going to be supported for some Microsoft applications.
I think this is very good news for organizations looking to virtualize mission critical applications. While the Support Policy Wizard was great for finding out specific details about specific versions, in my opinion it is much better to simply show that all of the applications are supported so long as the hypervisor is on the SVVP and the hypervisor supports the right OS requirements for the application. After all, I doubt there are many organizations looking at the SVVP to make sure that they can run Exchange 5.5 on Windows 2000 in a virtual machine.
And even better that this makes it clear that features like memory ballooning or vMotion are supported unless the application specifically states that it is not. These are the kinds of changes we need to see to help organizations drive the virtualization of their mission critical applications. I give Microsoft credit here for helping to clarify their support policies.
Those of you who follow me on Twitter have probably seen me say this a few times – 2012 will be the year of SQL virtualization, specifically SQL on vSphere. I figured it was about time to back that up with some facts and some opinions on just why I am so adamant about that statement.
I spend a lot of time talking to organizations and helping them virtualize their mission critical applications. In the last few years, many organizations have had success virtualizing Exchange 2010. A big reason why is due to the architectural changes in Exchange that have reduced I/O requirements as well as introduced new availability features. Combine that with the fact that Microsoft has relaxed their support policies around virtualizing Exchange and it is no surprise that many organizations have chosen to virtualize 100% of their Exchange 2010 environment.
I think that it is now Microsoft SQL that will be the next big mission critical applications that organizations will choose to virtualize. I’ve already spoken with many large organizations myself this year alone who are interested in taking on large scale SQL virtualization and consolidation projects, and others which are looking to build SQL as a Service platforms as well.
Here are a few of the reasons why I think this is the year of SQL virtualization.
With the release of SQL 2012, Microsoft has introduced new features that will make it much easier to virtualize SQL while still maintaining high availability. Specifically, the new SQL 2012 AlwaysOn Availability Groups (AAGs) can provide nearly the same level of availability as a traditional SQL cluster without the strict requirements for virtualizing a traditional Windows Failover Cluster (MSCS). AAGs can also have multiple passive copies (previously not possible with database mirroring), and the passive nodes can actually be used for read requests to offload processing from the active node.
So you get the availability of a cluster, along with multiple passive copies of the database, but you don’t have to go through the difficulties of virtualizing a cluster. Despite the fact that Microsoft changed to a per-core licensing model for SQL 2012 I still see this as a big driver for organizations to move to virtualize their SQL 2012 environment.
Even with the move to per-core licensing in SQL 2012, organizations can still save a lot of money by virtualizing SQL. That is because when organizations move to the per-core model and license SQL 2012 Enterprise Edition, they can virtualize an unlimited number of SQL 2012 (or lower) instances on those cores. I’ve personally worked with a number of organizations that either use the older Server+CAL licensing model or have a large number of CPU licenses that they can move into the virtual infrastructure and see significant savings.
There’s a lot more to the licensing story that I won’t get into here, but you can read my previous post on the subject.
Underutilized SQL servers
VMware has a large collection of data on production servers from its Capacity Planner data warehouse. This data shows that the vast majority of SQL servers are actually very underutilized and require very little CPU, memory, or disk I/O. Many people assume that SQL is difficult to virtualize because of the heavy performance requirements, but VMware’s data (in addition to my own observations at many customers) has shown that not to be the case.
Combine under-utilized SQL servers with advantageous licensing if you heavily consolidate your SQL workloads, and it is no surprise that many organizations are eager to virtualize SQL. There may be heavy hitting SQL servers in your environment and those can be virtualized too, but it is likely that many of them are actually pretty light.
VMware is getting serious about mission critical apps
Earlier this year, VMware released a new competency for their partners called the Virtualizing Business Critical Applications (VBCA) competency. VMware partners can go to Partner Central and receive free training and education to help them be successful with virtualizing business critical applications like SQL and Exchange. If you work for a VMware partner and have not yet seen the training and information available for the VBCA competency, log on to Partner Central and go check it out (after you finish reading this post, of course).
VMware clearly sees the value in helping their partners be successful with virtualizing mission critical applications like SQL. With VMware helping their partners to deliver successful projects, it can only help drive adoption of virtualized SQL (and Exchange, etc.) even higher.
Combine all of those things together with the scalability of vSphere 5, and you have a recipe for SQL virtualization. Many organizations have already started down this path and others are just starting, and I believe that this year will be the year that more organizations choose to virtualize SQL than before. And I believe it’s about time….
Microsoft announced on the Exchange Team blog today that they now support running Jetstress 2010 inside a virtual machine. You can read more about it and a little bit of the background on this one at the Exchange Team blog post here.
If you aren’t familiar with Jetstress, here’s a two second version: Jetstress is a tool from Microsoft that simulates the disk activity of an Exchange Mailbox server. It is used to simulate the load on the storage system used to host Exchange mailbox databases, and measures key attributes like disk I/O latency and total IOPS.
In my experience with Jetstress I’ve seen exactly what Microsoft has seen in the past. Namely, results from Jetstress (particularly I/O latency numbers) that do not match what is reported by either the ESXi host or the storage itself. So despite what Microsoft says here about the fact that Jetstress is now supported when run in a virtual machine, I still think it makes sense to independently validate the results.
As recently as last year I had a customer run Jetstress on their Exchange 2010 virtual machine that was running on vSphere 4.1 and the results were less than what we expected. Once we ran it again and measured the I/O latency in both esxtop as well as from the storage array management software we could clearly see the numbers were within the acceptable range.
Why would the numbers be inaccurate? Jetstress relies on in-guest timers to measure things like I/O latency, and so like any other in-guest timer they are susceptible to clock drift issues when the host’s CPUs become heavily loaded. A more common tool for testing Exchange performance, Load Generator 2010, relies on a separate client machine and so the numbers produced from that are generally considered to be more accurate (though it doesn’t test the same thing as Jetstress).
I can see why Microsoft has made this change – combine the fact that you can easily get 10 core CPUs with hypervisors that have improved considerably over the years and you are unlikely to see many issues anymore. For the most part you’re likely to see accurate test results unless the ESXi host itself is under heavy CPU contention.
Bottom line: I still recommend that results from Jetstress be validated outside the virtual machine using either esxtop or stats from the storage array, preferably both. If all three match up then you can feel confident that the data you’re seeing from Jetstress is accurate.
Yes, it’s 2012 and we’re still talking about whether or not organizations should consider running a Microsoft Windows Failover Cluster (sometimes referred to as MSCS clustering) in a vSphere environment. I know this topic has been written about before by others but I wanted to share some of my own thoughts and experiences around this topic. My focus these days is helping organizations virtualize their mission critical applications, and in that pursuit the topic of guest clustering comes up often.
What is supported?
To start with, one common misconception is that guest clustering is not supported at all in vSphere. If anyone out there still believes this (and I’ve spoken to many organizations over the years that do), I’d like to state definitively that this is not true. Guest clustering is absolutely supported by both VMware and Microsoft provided you follow the guidelines from both companies in order to maintain support.
One of the best KB articles VMware has released on this subject can be found here. It does a great job of summarizing the various supported configurations and goes into some application specific clustering types as well. I keep this KB article handy and use it frequently in discussions with customers. The following table lists the supported configurations:
I know many folks are very much against the idea of virtualizing a Microsoft cluster and will argue vehemently against its use. I can tell you I am most definitely not in that camp – I believe that the needs and requirements of the business should dictate whether or not clustering is used.
Why would you use guest clustering?
Ok so now you’ve seen what’s supported and after looking at that table you’re probably wondering if it’s even worth it. Is virtualizing a cluster kind of a pain? Yep, you’ll get no argument from me that it’s more difficult. It prevents you from using vMotion or DRS in Fully Automated mode, forces you to use RDMs, and only fiber channel is supported to name a few. The largest restrictions are when using “shared disk” clusters, or clusters that require dedicated storage that is shared amongst the cluster nodes.
With all of these restrictions, why would you use guest clustering in the first place?
Guest Clustering vs. vSphere HA
Let’s be clear about one thing: vSphere HA does not provide the same level of availability as guest clustering. vSphere HA is an awesome feature that can be used in combination with guest clustering, but HA is not application aware and can only protect against hardware and operating system failure.
Application-aware high availability
The biggest reason I see customers using guest clustering is to provide high availability at the application level beyond what native vSphere features can provide. SQL and older versions of Exchange are two commonly clustered guests. In particular I’ve worked with a lot of customers recently who have physical SQL clusters running SQL 2005 and 2008, and they are working towards bringing them into vSphere as part of larger SQL consolidation and SQL as a Service projects.
Guest OS patching
Many organizations still use clusters so that they can patch the underlying OS or application without causing an outage to end users. As covered in the next section, newer versions of Microsoft applications have functionality that can provide this benefit without clustering. I expect we’ll see clusters used just for this purpose decline as applications improve.
For many organizations, the availability that clustering provides is more important than the vSphere features they lose by implementing it. It all comes down to what is important to the business.
Alternatives to guest clustering
Of course there are alternatives to clustering that can provide the same or similar levels of availability without the restrictions. Here are some examples.
Newer versions of Exchange support technologies that do not require shared disks to provide high availability. Exchange 2010 in particular supports Database Availability Groups, which can provide HA down to the database level but does not require any shared disk clustering. That means it can support vMotion, DRS, and all the other great vSphere features. You can read more about the Exchange 2010/vSphere goodness here.
SQL Database Mirroring/SQL 2012
SQL has had database mirroring for years, and mirroring can provide similar levels of availability to clusters without requiring shared storage. And with the release of SQL 2012 earlier this month, Microsoft has improved upon that concept with SQL “AlwaysOn” technology. AlwaysOn technology has some similarities to Database Availability Groups in Exchange 2010 by providing multiple database copies and recoverability at the database level.
With the popularity of storage arrays offering NAS capabilities, the use of file clusters seems to be declining. These arrays have multiple controllers and redundancy across the platform that can provide the same or better availability than a traditional file cluster.
Use In-Guest iSCSI
Do you want to combine the enterprise features of vSphere with the availability of guest clustering? One way to consider doing this is by using in-guest iSCSI to present storage to clustered virtual machines. Using this method gets around the VMware policy of only supporting fiber channel while still allowing features like vMotion to be used. This method is by no means an easy solution – you may need to adjust cluster heartbeat timeouts to allow vMotion to operate successfully, networking at the vSphere level becomes more complicated, using multipathing software is more complicated, vendor support may be more complicated, etc. Again it all comes down to the requirements of the business.
Have I convinced any of you that guest clustering is not the worst thing in the world? I really hope so, because I don’t think we should ever be so closed minded that we immediately dismiss something because it is difficult to implement. I am definitely in the camp of folks who prefer to avoid using guest clustering so that I can take advantage of all the great features in vSphere. But I am most definitely not in the camp of people who dismiss it entirely.
For me, it comes down to what are the needs of the business and how can those needs be met. There are situations where guest clustering is required, and I don’t think we should be telling organizations to keep those servers physical. If you want to virtualize mission critical applications then you should be prepared to consider all possible configurations to meet the needs of the business.
I’ve seen a theme at several customers who are virtualizing mission critical applications on vSphere: The isolated vSphere cluster used just for that application. I’ve seen organizations do this many times when virtualizing Exchange 2010, often dedicating two or three vSphere hosts just for Exchange and related components (domain controller, virtual load balancer, etc).
There are several (understandable) reasons why organizations choose to go down this road. The most common I’ve heard are:
1) “I don’t want my Exchange/SQL/SharePoint guys messing with the rest of the VMs on my production cluster(s).”
2) “I don’t want any other workload taking resources away from my Exchange/SQL/SharePoint VMs.”
I can see why folks make both of these arguments and can understand their logic. Many vSphere admins live in the Hosts and Clusters view where they see all virtual machines rather than those organized into folders (where granular permissions can be set more easily). And some are concerned that setting reservations on virtual machines to guarantee access to resources will have a negative impact on available slots in a cluster.
I don’t believe that either of these arguments are enough to require a dedicated vSphere cluster. A few thoughts are below, starting with addressing the two most common reasons I listed above.
1) VMs can be grouped into application specific folders, and granular permissions can be set to only grant access to those folders or VMs.
2) Reservations aren’t a bad thing or something to be scared of, they just need to be factored into any design. And using the “Percentage of Cluster Resources'” policy gets around worrying about individual slot sizes (though total resource assignments still need to be considered and calculated).
3) Fewer hosts in a cluster gives VMware features like DRS or HA fewer options when trying to initiate a failover or migrate virtual machines based on resource consumption. If an application is truly mission critical, I’d rather have more options to optimize availability and performance rather than fewer.
4) Let’s be honest – vSphere isn’t free or cheap, so dedicating hosts to a specific application is likely to result in heavily underutilized hosts and higher costs. Aren’t underutilized servers and cost reduction some of the reasons why we got into this virtualization business in the first place? If you made dedicated clusters for all of your mission critical applications you would end up with application silos that greatly reduce the efficiency of virtualization.
At the end of the day a design decision like this should always come down to the requirements. There may be very valid requirements, like compliance, security, or extremely large applications that make an isolated vSphere cluster a necessity. Assuming those requirements are not necessary in your environment I would strongly consider whether there is really a reason to use a dedicated cluster for your mission critical applications.
If you agree/disagree/think I’m crazy - feel free to let me know in the comments..
As I discussed last week, Microsoft has updated their guidance and support regarding hypervisor high availability and live migration technologies with Exchange 2010. This is welcome news for anyone looking to virtualize Exchange 2010 on vSphere, as it now allows you to take advantage of vMotion and HA on all Exchange servers, even those in a Database Availability Group (DAG).
Microsoft’s language on this change in support is causing a little bit of confusion on exactly what they do and do not support. See the following section below taken from the Exchange 2010 System Requirements page on TechNet (bolded emphasis mine):
Exchange server virtual machines, including Exchange Mailbox virtual machines that are part of a Database Availability Group (DAG), can be combined with host-based failover clustering and migration technology as long as the virtual machines are configured such that they will not save and restore state on disk when moved or taken offline. All failover activity must result in a cold start when the virtual machine is activated on the target node. All planned migration must either result in shut down and a cold start or an online migration that utilizes a technology such as Hyper-V live migration.
The bolded section seems to be the part that is confusing some people into thinking that some VMware features are not supported. But think about it for a second – does that description really sound like any VMware feature other than a simple Suspend? vMotion doesn’t save anything to disk during a migration and HA doesn’t save anything during a failure/restart of a VM. So what is Microsoft describing here?
Remember back when Hyper-V was released and Microsoft was touting a feature called Quick Migration to compete with VMware's vMotion? It was called Quick Migration and not Live Migration because the virtual machine was paused briefly and the state of the virtual machine was saved to disk during the migration. From the Quick Migration with Hyper-V (opens a .doc) whitepaper:
For a planned migration, quick migration saves the state of a running guest virtual machine (memory of original server to disk/shared storage), moves the storage connectivity from one physical server to another, and then restores the guest virtual machine onto the second server (disk/shared storage to memory on the new server).
It’s clear that the language in the TechNet article applies only to Hyper-V Quick Migration and not vMotion or even Hyper-V’s newer Live Migration feature. Microsoft probably felt they had a large enough installed base of Hyper-V RTM that clarifying this language was important.
Rumor has it that enough people are confused by this language that Microsoft is planning on clarifying it soon. At the end of the day this language doesn’t apply to those of us virtualizing Exchange 2010 on vSphere. Both HA and vMotion are now supported and that’s all that matters to us.
This Saturday, Microsoft published a new white paper entitled “Best Practices for Virtualizing Exchange Server 2010 with Windows Server 2008 R2 Hyper V” that provides a lot of great info that is applicable to VMware as well. One of the most important things in this entire document is a change in policy regarding supporting virtualized Exchange 2010 with Database Availability Groups (DAG) in combination with hypervisor high availability and live migration. Previously Microsoft did not support the use of high availability or live migration even on its own Hyper-V platform. In the VMware world this of course means HA and vMotion. The whitepaper states the following:
Exchange server virtual machines, including Exchange Mailbox virtual machines that are part of a Database Availability Group (DAG), can be combined with host-based failover clustering and migration technology as long as the virtual machines are configured such that they will not save and restore state on disk when moved or taken offline. All failover activity must result in a cold start when the virtual machine is activated on the target node. All planned migration must either result in shut down and a cold start or an online migration that utilizes a technology such as Hyper-V live migration
This new document, as well as a post on the MS Exchange Team blog, confirms the new support stance. The Technet page has been updated as well. Note that you must be running Exchange 2010 SP1 in order to support these features.
Folks may know that I’m a big proponent of virtualizing mission critical, tier-1 applications like Exchange 2010. I’ve written about it here, touched on it here, and commented to TechTarget on the subject here and most recently here. It’s clearly an important subject to me and I applaud Microsoft for introducing this change. I think this will help to convince organizations that it is safe to virtualize Exchange 2010 on all hypervisors.
But..there must always be a but..
Remember that just because Microsoft now officially supports something doesn’t actually change anything in terms of functionality. Did VMware HA and vMotion work properly in combination with Exchange 2010 and DAGs before this policy change? VMware HA – sure, it just wasn’t officially supported. VMware vMotion – umm, hang on a minute there.
The DAG ultimately relies on Windows Failover Clustering to work, and WFC is notoriously finicky about even brief drops in network connectivity and loss of heartbeat. When performing a live migration using vMotion there is usually at least one ping dropped, and in my experience that single drop is often enough to cause databases to failover to other nodes in the DAG.
Does this mean that even though Microsoft supports vMotion now that you still can’t use it? Of course not, but it does require a slight change in your design to increase the cluster heartbeat timeout value to allow for the brief network interruption.
The values that need to change are the following:
SameSubnetDelay: The value (in milliseconds) of the cluster heartbeat frequency. By default, this value is 1,000 milliseconds.
SameSubnetThreshold: The value represents the amount of missed heartbeats that will be tolerated before a failover event occurs. By default this value is 5, so combined with the above value that means 5 seconds of lost heartbeats will result in a cluster failover by default.
Five seconds seems like enough time for a vMotion to complete, but in practice I’ve seen databases failover at multiple clients when using the default heartbeat values. Luckily you can change these values very easily by using PowerShell. The following commands show how to raise the timeout to 10 seconds (the Microsoft recommended max) from the default of 5, taken directly from the Microsoft whitepaper:
Depending on your environment you may not need to make this change, so always test first before implementing any cluster wide change like this. Make sure you have enough bandwidth on your hosts to account for migrating an Exchange VM that may have 32GB of RAM or more. And of course always stick with configurations that are supported by Microsoft.
I’m happy that Microsoft made this change, and hope that it signals a trend towards more virtualization friendly licensing in the future.
Over the long Thanksgiving weekend I decided to do some testing of one of the coolest new features in vSphere 4.1 - vStorage APIs for Array Integration. My original thought was to see if the performance benefits of using VAAI would justify more heavily using the eagerzeroedthick VMDK format because of the faster deployment times. I'll get to the results of that testing in a second, but first some background.
VAAI is a technology that allows the ESX/ESXi host to offload certain storage functions directly to the storage array rather than processing the data itself. A typical operation such as deploying a VM from template requires the ESX/ESXi host to read the data from the template via whatever storage protocol is in use (fiber, iSCSI, etc) and then write that data to the storage when cloning the VM. That isn't the most efficient use of resources, and it is compounded when cloning multiple VMs at once as those read/write operations become redundant.
By leveraging VAAI, those operations are offloaded to the storage array and so it eliminates much of those redundant reads/writes. As a result these operations complete much faster and with reduced CPU overhead to manage the process. In order to use VAAI you'll need both vSphere Enterprise as well as a storage array that supports it. Although the number of supported arrays is small that number will most certainly grow.
For my testing I used a Dell EqualLogic PS5000E running the 5.0.2 firmware which fully supports VAAI. My original thought was to see how much quicker deploying eagerzeroedthick VMDKs was with VAAI compared to without VAAI. Using eagerzeroedthick disks helps with performance of the VM by zeroing out all of the blocks in advance instead of when they are first accessed. This format is required for VMware Fault Tolerance and is recommended for high I/O servers such as Exchange and SQL.
To the results:
There has been a lot of drama recently between Microsoft and VMware with respect to virtualizing Exchange 2010. The background and details are summarized well in this article (in which I'm quoted). I think this situation brings up some important points about virtualizing critical Tier 1 applications - vendor support is very important, and disregarding vendor requirements for support can lead to problems when you need support the most.
There are a number of seemingly strange support requirements from Microsoft and VMware which often leads people to get confused on what is supported or how to configure their environments. A few of them are listed below: