PVSCSI Bug Causing Exchange 2010 Jetstress to Crash
A few weeks back I was called in to help a customer who was experiencing problems completing Jetstress testing for an Exchange 2010 deployment. It wasn’t an issue of Jetstress reporting failed tests. Rather, they were unable to get through most of their tests without the Jetstress application actually crashing (JetstressWin.exe has stopped working). They would see the following after the Jetstress testing completed but before it could write any log files to disk.
The only Jetstress related error in the Application log was an ESE error with Event ID 482:
JetstressWin (3584) Instance3584.6: An attempt to write to the file “F:\DB\Jetstress006001.edb” at offset 63087017984 (0x0000000eb0478000) for 32768 (0x00008000) bytes failed over 0 seconds with system error 1117 (0x0000045d): “The request could not be performed because of an I/O device error.”. The write operation will fail with error –1022 (0xfffffc02). If this error persists then the file may be damaged and may need to be restored from a previous backup.
During the process of Jetstress completing a test run, it generates a large amount of I/O as it flushes anything in cache to disk. It was at this point that the Jetstress application was crashing. This behavior is normal but it’s an important clue because of the high disk I/O generated.
The customer was using vSphere 4.1 and the Exchange 2010 Mailbox servers were each configured with PVSCSI virtual SCSI controllers using VMDK files. As it turns out, they were hit with the PVSCI bug described in this VMware KB:
Windows 2008 R2 virtual machine using a paravirtual SCSI adapter reports the error: Operating system error 1117 encountered http://kb.vmware.com/kb/2004578
The interesting thing to note here is that although Exchange is specifically called out here in the KB, it doesn’t mention that it may cause the application (in this case Jetstress) to crash. The crashing led the team to troubleshoot Jetstress initially, thinking something was wrong with Jetstress and the various DLLs it requires to run.
At the end of the day the issue was resolved by following the instructions in the KB and changing the virtual SCSI driver to LSI Logic SAS. After making that change there were no subsequent issues with Jetstress.
In case you haven’t read the KB linked above, I want to note that this issue is resolved in all versions of vSphere from 4.1 to 5.0. You’ll need to install the updates described in the KB if you want to use the PVSCSI driver and vSphere 4.1 through 5.0 (it is resolved in vSphere 5.1).
Hopefully this helps anyone who might be experiencing this issue. I also hope it doesn’t dissuade anyone from using the PVSCSI driver for their business critical applications, as it can deliver better performance with lower CPU utilization when high I/O workloads are virtualized.
2012 – The Year of SQL Virtualization
Those of you who follow me on Twitter have probably seen me say this a few times – 2012 will be the year of SQL virtualization, specifically SQL on vSphere. I figured it was about time to back that up with some facts and some opinions on just why I am so adamant about that statement.
I spend a lot of time talking to organizations and helping them virtualize their mission critical applications. In the last few years, many organizations have had success virtualizing Exchange 2010. A big reason why is due to the architectural changes in Exchange that have reduced I/O requirements as well as introduced new availability features. Combine that with the fact that Microsoft has relaxed their support policies around virtualizing Exchange and it is no surprise that many organizations have chosen to virtualize 100% of their Exchange 2010 environment.
I think that it is now Microsoft SQL that will be the next big mission critical applications that organizations will choose to virtualize. I’ve already spoken with many large organizations myself this year alone who are interested in taking on large scale SQL virtualization and consolidation projects, and others which are looking to build SQL as a Service platforms as well.
Here are a few of the reasons why I think this is the year of SQL virtualization.
SQL 2012
With the release of SQL 2012, Microsoft has introduced new features that will make it much easier to virtualize SQL while still maintaining high availability. Specifically, the new SQL 2012 AlwaysOn Availability Groups (AAGs) can provide nearly the same level of availability as a traditional SQL cluster without the strict requirements for virtualizing a traditional Windows Failover Cluster (MSCS). AAGs can also have multiple passive copies (previously not possible with database mirroring), and the passive nodes can actually be used for read requests to offload processing from the active node.
So you get the availability of a cluster, along with multiple passive copies of the database, but you don’t have to go through the difficulties of virtualizing a cluster. Despite the fact that Microsoft changed to a per-core licensing model for SQL 2012 I still see this as a big driver for organizations to move to virtualize their SQL 2012 environment.
Licensing
Even with the move to per-core licensing in SQL 2012, organizations can still save a lot of money by virtualizing SQL. That is because when organizations move to the per-core model and license SQL 2012 Enterprise Edition, they can virtualize an unlimited number of SQL 2012 (or lower) instances on those cores. I’ve personally worked with a number of organizations that either use the older Server+CAL licensing model or have a large number of CPU licenses that they can move into the virtual infrastructure and see significant savings.
There’s a lot more to the licensing story that I won’t get into here, but you can read my previous post on the subject.
Underutilized SQL servers
VMware has a large collection of data on production servers from its Capacity Planner data warehouse. This data shows that the vast majority of SQL servers are actually very underutilized and require very little CPU, memory, or disk I/O. Many people assume that SQL is difficult to virtualize because of the heavy performance requirements, but VMware’s data (in addition to my own observations at many customers) has shown that not to be the case.
Combine under-utilized SQL servers with advantageous licensing if you heavily consolidate your SQL workloads, and it is no surprise that many organizations are eager to virtualize SQL. There may be heavy hitting SQL servers in your environment and those can be virtualized too, but it is likely that many of them are actually pretty light.
VMware is getting serious about mission critical apps
Earlier this year, VMware released a new competency for their partners called the Virtualizing Business Critical Applications (VBCA) competency. VMware partners can go to Partner Central and receive free training and education to help them be successful with virtualizing business critical applications like SQL and Exchange. If you work for a VMware partner and have not yet seen the training and information available for the VBCA competency, log on to Partner Central and go check it out (after you finish reading this post, of course).
VMware clearly sees the value in helping their partners be successful with virtualizing mission critical applications like SQL. With VMware helping their partners to deliver successful projects, it can only help drive adoption of virtualized SQL (and Exchange, etc.) even higher.
In Conclusion…
Combine all of those things together with the scalability of vSphere 5, and you have a recipe for SQL virtualization. Many organizations have already started down this path and others are just starting, and I believe that this year will be the year that more organizations choose to virtualize SQL than before. And I believe it’s about time….
Microsoft Jetstress 2010 now supported in virtual machines
Microsoft announced on the Exchange Team blog today that they now support running Jetstress 2010 inside a virtual machine. You can read more about it and a little bit of the background on this one at the Exchange Team blog post here.
If you aren’t familiar with Jetstress, here’s a two second version: Jetstress is a tool from Microsoft that simulates the disk activity of an Exchange Mailbox server. It is used to simulate the load on the storage system used to host Exchange mailbox databases, and measures key attributes like disk I/O latency and total IOPS.
In my experience with Jetstress I’ve seen exactly what Microsoft has seen in the past. Namely, results from Jetstress (particularly I/O latency numbers) that do not match what is reported by either the ESXi host or the storage itself. So despite what Microsoft says here about the fact that Jetstress is now supported when run in a virtual machine, I still think it makes sense to independently validate the results.
As recently as last year I had a customer run Jetstress on their Exchange 2010 virtual machine that was running on vSphere 4.1 and the results were less than what we expected. Once we ran it again and measured the I/O latency in both esxtop as well as from the storage array management software we could clearly see the numbers were within the acceptable range.
Why would the numbers be inaccurate? Jetstress relies on in-guest timers to measure things like I/O latency, and so like any other in-guest timer they are susceptible to clock drift issues when the host’s CPUs become heavily loaded. A more common tool for testing Exchange performance, Load Generator 2010, relies on a separate client machine and so the numbers produced from that are generally considered to be more accurate (though it doesn’t test the same thing as Jetstress).
I can see why Microsoft has made this change – combine the fact that you can easily get 10 core CPUs with hypervisors that have improved considerably over the years and you are unlikely to see many issues anymore. For the most part you’re likely to see accurate test results unless the ESXi host itself is under heavy CPU contention.
Bottom line: I still recommend that results from Jetstress be validated outside the virtual machine using either esxtop or stats from the storage array, preferably both. If all three match up then you can feel confident that the data you’re seeing from Jetstress is accurate.
Virtual machine snapshots and tier-1 apps: Not always supported
After seeing a discussion in the vExpert forums and in my own experience in talking to customers, I came to the conclusion that many people aren’t aware of some of the support restrictions around virtualizing tier-1 Microsoft applications.
The one support requirement that many folks aren’t aware of is the use of virtual machine snapshots on Microsoft applications, particularly Exchange or SQL. For both applications, Microsoft is explicit in stating that virtual machine snapshots of any kind are not supported.
Why does this matter? Many popular virtual machine backup products (purposely not naming any) rely on snapshots to take clean backups of running VMs. That means using these products to backup Exchange or SQL VMs may be unsupported.
Here are the official Microsoft support statements for Exchange and SQL:
Microsoft Exchange 2010 System Requirements:
Some hypervisors include features for taking snapshots of virtual machines. Virtual machine snapshots capture the state of a virtual machine while it's running. This feature enables you to take multiple snapshots of a virtual machine and then revert the virtual machine to any of the previous states by applying a snapshot to the virtual machine. However, virtual machine snapshots aren't application aware, and using them can have unintended and unexpected consequences for a server application that maintains state data, such as Exchange. As a result, making virtual machine snapshots of an Exchange guest virtual machine isn't supported.
Support policy for SQL Server running in a hardware virtualization environment (covers all versions):
Virtualization Snapshots for Hyper-V or for any virtualization vendor are not supported to use with SQL Server in a virtual machine. It is possible that you may not encounter any problems when using snapshots and SQL Server, but Microsoft will not provide technical support to SQL Server customers for a virtual machine that was restored from a snapshot.
Pretty clear, right? The SQL policy actually explains this the best. They’re saying that you may not have any trouble using virtual machine snapshots, but if you restore a VM using a backup product that relies on snapshots then they will not support any issues with that restore.
What about virtual machine backup products that use an application aware VSS writer? I’ve spoken to Microsoft about this in particular and the answer I got back matches what their policy states. That is, if you’re using a virtual machine backup product that relies on snapshots, that solution is unsupported and you may be redirected to the backup vendor for support. This is true whether or not the virtual machine backup uses an application aware VSS writer, as some popular backup products do.
When you are virtualizing your mission critical applications like Exchange or SQL, support is extremely important. So what are your options?
1) If you have a Microsoft Premier support agreement, then Microsoft will make reasonable efforts to support you even if you run in unsupported configurations. If your organization has a Premier support agreement then you’ll most likely get support if you use a backup product that relies on virtual machine snapshots. Always confirm with your Microsoft rep in advance to ensure they will support the configuration.
2) Talk to your Microsoft representative and try to get the support statement clarified. Show them your configuration and get confirmation that it is or is not supported.
3) Rely on the backup vendor for first level support if your restored VM is having issues.
4) Use a different type of backup product for Exchange or SQL virtual machines.
As I said, mission critical applications have more strict requirements for support and I believe that you should make all attempts to follow these requirements. These applications are usually too important to an organization to risk a lengthy outage due to a miscommunication or disagreement about what is or is not supported.
Guest clustering on vSphere: A familiar topic revisited
Yes, it’s 2012 and we’re still talking about whether or not organizations should consider running a Microsoft Windows Failover Cluster (sometimes referred to as MSCS clustering) in a vSphere environment. I know this topic has been written about before by others but I wanted to share some of my own thoughts and experiences around this topic. My focus these days is helping organizations virtualize their mission critical applications, and in that pursuit the topic of guest clustering comes up often.
What is supported?
To start with, one common misconception is that guest clustering is not supported at all in vSphere. If anyone out there still believes this (and I’ve spoken to many organizations over the years that do), I’d like to state definitively that this is not true. Guest clustering is absolutely supported by both VMware and Microsoft provided you follow the guidelines from both companies in order to maintain support.
One of the best KB articles VMware has released on this subject can be found here. It does a great job of summarizing the various supported configurations and goes into some application specific clustering types as well. I keep this KB article handy and use it frequently in discussions with customers. The following table lists the supported configurations:
I know many folks are very much against the idea of virtualizing a Microsoft cluster and will argue vehemently against its use. I can tell you I am most definitely not in that camp – I believe that the needs and requirements of the business should dictate whether or not clustering is used.
Why would you use guest clustering?
Ok so now you’ve seen what’s supported and after looking at that table you’re probably wondering if it’s even worth it. Is virtualizing a cluster kind of a pain? Yep, you’ll get no argument from me that it’s more difficult. It prevents you from using vMotion or DRS in Fully Automated mode, forces you to use RDMs, and only fiber channel is supported to name a few. The largest restrictions are when using “shared disk” clusters, or clusters that require dedicated storage that is shared amongst the cluster nodes.
With all of these restrictions, why would you use guest clustering in the first place?
Guest Clustering vs. vSphere HA
Let’s be clear about one thing: vSphere HA does not provide the same level of availability as guest clustering. vSphere HA is an awesome feature that can be used in combination with guest clustering, but HA is not application aware and can only protect against hardware and operating system failure.
Application-aware high availability
The biggest reason I see customers using guest clustering is to provide high availability at the application level beyond what native vSphere features can provide. SQL and older versions of Exchange are two commonly clustered guests. In particular I’ve worked with a lot of customers recently who have physical SQL clusters running SQL 2005 and 2008, and they are working towards bringing them into vSphere as part of larger SQL consolidation and SQL as a Service projects.
Guest OS patching
Many organizations still use clusters so that they can patch the underlying OS or application without causing an outage to end users. As covered in the next section, newer versions of Microsoft applications have functionality that can provide this benefit without clustering. I expect we’ll see clusters used just for this purpose decline as applications improve.
For many organizations, the availability that clustering provides is more important than the vSphere features they lose by implementing it. It all comes down to what is important to the business.
Alternatives to guest clustering
Of course there are alternatives to clustering that can provide the same or similar levels of availability without the restrictions. Here are some examples.
Exchange 2010
Newer versions of Exchange support technologies that do not require shared disks to provide high availability. Exchange 2010 in particular supports Database Availability Groups, which can provide HA down to the database level but does not require any shared disk clustering. That means it can support vMotion, DRS, and all the other great vSphere features. You can read more about the Exchange 2010/vSphere goodness here.
SQL Database Mirroring/SQL 2012
SQL has had database mirroring for years, and mirroring can provide similar levels of availability to clusters without requiring shared storage. And with the release of SQL 2012 earlier this month, Microsoft has improved upon that concept with SQL “AlwaysOn” technology. AlwaysOn technology has some similarities to Database Availability Groups in Exchange 2010 by providing multiple database copies and recoverability at the database level.
File Clusters
With the popularity of storage arrays offering NAS capabilities, the use of file clusters seems to be declining. These arrays have multiple controllers and redundancy across the platform that can provide the same or better availability than a traditional file cluster.
Use In-Guest iSCSI
Do you want to combine the enterprise features of vSphere with the availability of guest clustering? One way to consider doing this is by using in-guest iSCSI to present storage to clustered virtual machines. Using this method gets around the VMware policy of only supporting fiber channel while still allowing features like vMotion to be used. This method is by no means an easy solution – you may need to adjust cluster heartbeat timeouts to allow vMotion to operate successfully, networking at the vSphere level becomes more complicated, using multipathing software is more complicated, vendor support may be more complicated, etc. Again it all comes down to the requirements of the business.
Conclusions
Have I convinced any of you that guest clustering is not the worst thing in the world? I really hope so, because I don’t think we should ever be so closed minded that we immediately dismiss something because it is difficult to implement. I am definitely in the camp of folks who prefer to avoid using guest clustering so that I can take advantage of all the great features in vSphere. But I am most definitely not in the camp of people who dismiss it entirely.
For me, it comes down to what are the needs of the business and how can those needs be met. There are situations where guest clustering is required, and I don’t think we should be telling organizations to keep those servers physical. If you want to virtualize mission critical applications then you should be prepared to consider all possible configurations to meet the needs of the business.
Storage DRS of Exchange 2010 Workloads
On September 15th, VMware released a new whitepaper entitled “Microsoft Exchange Server 2010 Performance on vSphere 5.” If you are interested in virtualizing Exchange 2010 (or really any mission critical application) I’d recommend giving it a read.
One of the interesting things they’ve done in this test aside from scale up/scale out testing is to formally test vMotion and Storage vMotion during LoadGen. LoadGen, if you are unfamiliar, is a Microsoft tool used to simulate user activity in an Exchange environment and measure user response time.
I won’t spoil the ending of the report, but as expected the vMotion and Storage vMotion operations did not cause exceptions that would cause the LoadGen test to fail. In fact, the Storage vMotion had almost no impact whatsoever.
This brings up an interesting point – since VMware was able to prove that Storage vMotion caused no impact even during an Exchange performance test, it may be a good candidate for Storage DRS in vSphere 5. Microsoft has improved the I/O consumption of Exchange 2010 so much that it actually consumes very little I/O at all (relatively speaking compared to previous versions). Still, Exchange is a mission critical application and heavily utilized Exchange environments could benefit from Storage DRS migrating particular mailbox database VMDKs to less utilized storage to optimize the user experience. And as this whitepaper shows there is very little impact during the migration.
There may be reasons why organizations would chose to exclude their Exchange VMs from automatic Storage vMotion with Storage DRS. Some of those reasons could be around compliance, acceptance of this new vSphere 5 feature, concerns about performance, and possibly the impact on backup/DR strategies. Assuming those could be mitigated, I believe this testing shows that Exchange VMs actually make good candidates for Storage DRS.
I’m curious of what others think, so feel free to leave a comment.
Clearing up confusion regarding HA/vMotion support for Exchange 2010
As I discussed last week, Microsoft has updated their guidance and support regarding hypervisor high availability and live migration technologies with Exchange 2010. This is welcome news for anyone looking to virtualize Exchange 2010 on vSphere, as it now allows you to take advantage of vMotion and HA on all Exchange servers, even those in a Database Availability Group (DAG).
Microsoft’s language on this change in support is causing a little bit of confusion on exactly what they do and do not support. See the following section below taken from the Exchange 2010 System Requirements page on TechNet (bolded emphasis mine):
Exchange server virtual machines, including Exchange Mailbox virtual machines that are part of a Database Availability Group (DAG), can be combined with host-based failover clustering and migration technology as long as the virtual machines are configured such that they will not save and restore state on disk when moved or taken offline. All failover activity must result in a cold start when the virtual machine is activated on the target node. All planned migration must either result in shut down and a cold start or an online migration that utilizes a technology such as Hyper-V live migration.
The bolded section seems to be the part that is confusing some people into thinking that some VMware features are not supported. But think about it for a second – does that description really sound like any VMware feature other than a simple Suspend? vMotion doesn’t save anything to disk during a migration and HA doesn’t save anything during a failure/restart of a VM. So what is Microsoft describing here?
Remember back when Hyper-V was released and Microsoft was touting a feature called Quick Migration to compete with VMware's vMotion? It was called Quick Migration and not Live Migration because the virtual machine was paused briefly and the state of the virtual machine was saved to disk during the migration. From the Quick Migration with Hyper-V (opens a .doc) whitepaper:
For a planned migration, quick migration saves the state of a running guest virtual machine (memory of original server to disk/shared storage), moves the storage connectivity from one physical server to another, and then restores the guest virtual machine onto the second server (disk/shared storage to memory on the new server).
It’s clear that the language in the TechNet article applies only to Hyper-V Quick Migration and not vMotion or even Hyper-V’s newer Live Migration feature. Microsoft probably felt they had a large enough installed base of Hyper-V RTM that clarifying this language was important.
Rumor has it that enough people are confused by this language that Microsoft is planning on clarifying it soon. At the end of the day this language doesn’t apply to those of us virtualizing Exchange 2010 on vSphere. Both HA and vMotion are now supported and that’s all that matters to us.
vSphere and Exchange admins can live in harmony – Microsoft finally supports HA and vMotion
This Saturday, Microsoft published a new white paper entitled “Best Practices for Virtualizing Exchange Server 2010 with Windows Server 2008 R2 Hyper V” that provides a lot of great info that is applicable to VMware as well. One of the most important things in this entire document is a change in policy regarding supporting virtualized Exchange 2010 with Database Availability Groups (DAG) in combination with hypervisor high availability and live migration. Previously Microsoft did not support the use of high availability or live migration even on its own Hyper-V platform. In the VMware world this of course means HA and vMotion. The whitepaper states the following:
Exchange server virtual machines, including Exchange Mailbox virtual machines that are part of a Database Availability Group (DAG), can be combined with host-based failover clustering and migration technology as long as the virtual machines are configured such that they will not save and restore state on disk when moved or taken offline. All failover activity must result in a cold start when the virtual machine is activated on the target node. All planned migration must either result in shut down and a cold start or an online migration that utilizes a technology such as Hyper-V live migration
This new document, as well as a post on the MS Exchange Team blog, confirms the new support stance. The Technet page has been updated as well. Note that you must be running Exchange 2010 SP1 in order to support these features.
Folks may know that I’m a big proponent of virtualizing mission critical, tier-1 applications like Exchange 2010. I’ve written about it here, touched on it here, and commented to TechTarget on the subject here and most recently here. It’s clearly an important subject to me and I applaud Microsoft for introducing this change. I think this will help to convince organizations that it is safe to virtualize Exchange 2010 on all hypervisors.
But..there must always be a but..
Remember that just because Microsoft now officially supports something doesn’t actually change anything in terms of functionality. Did VMware HA and vMotion work properly in combination with Exchange 2010 and DAGs before this policy change? VMware HA – sure, it just wasn’t officially supported. VMware vMotion – umm, hang on a minute there.
The DAG ultimately relies on Windows Failover Clustering to work, and WFC is notoriously finicky about even brief drops in network connectivity and loss of heartbeat. When performing a live migration using vMotion there is usually at least one ping dropped, and in my experience that single drop is often enough to cause databases to failover to other nodes in the DAG.
Does this mean that even though Microsoft supports vMotion now that you still can’t use it? Of course not, but it does require a slight change in your design to increase the cluster heartbeat timeout value to allow for the brief network interruption.
The values that need to change are the following:
SameSubnetDelay: The value (in milliseconds) of the cluster heartbeat frequency. By default, this value is 1,000 milliseconds.
SameSubnetThreshold: The value represents the amount of missed heartbeats that will be tolerated before a failover event occurs. By default this value is 5, so combined with the above value that means 5 seconds of lost heartbeats will result in a cluster failover by default.
Five seconds seems like enough time for a vMotion to complete, but in practice I’ve seen databases failover at multiple clients when using the default heartbeat values. Luckily you can change these values very easily by using PowerShell. The following commands show how to raise the timeout to 10 seconds (the Microsoft recommended max) from the default of 5, taken directly from the Microsoft whitepaper:
Import-module FailoverClusters
(Get-Cluster).SameSubnetThreshold=10
(Get-Cluster).SameSubnetDelay=1000
Depending on your environment you may not need to make this change, so always test first before implementing any cluster wide change like this. Make sure you have enough bandwidth on your hosts to account for migrating an Exchange VM that may have 32GB of RAM or more. And of course always stick with configurations that are supported by Microsoft.
I’m happy that Microsoft made this change, and hope that it signals a trend towards more virtualization friendly licensing in the future.
Recent Posts
- New Book Coming Soon: Virtualizing Microsoft Business Critical Applications on VMware vSphere
- Cloning Windows Server 2012 Domain Controllers on vSphere 5
- vSphere Home Lab Upgrade–Synology DS1812+
- Honored to be an EMC Elect
Recent Comments
- Glenn Drawdy on New Book Coming Soon: Virtualizing Microsoft Business Critical Applications on VMware vSphere
- Matt Liebowitz on New Book Coming Soon: Virtualizing Microsoft Business Critical Applications on VMware vSphere
- Michael Engemann on New Book Coming Soon: Virtualizing Microsoft Business Critical Applications on VMware vSphere
- Matt Liebowitz on New SQL 2012 Licensing and Its Impact On Virtualization