Isolated clusters for mission critical applications?
I’ve seen a theme at several customers who are virtualizing mission critical applications on vSphere: The isolated vSphere cluster used just for that application. I’ve seen organizations do this many times when virtualizing Exchange 2010, often dedicating two or three vSphere hosts just for Exchange and related components (domain controller, virtual load balancer, etc).
There are several (understandable) reasons why organizations choose to go down this road. The most common I’ve heard are:
1) “I don’t want my Exchange/SQL/SharePoint guys messing with the rest of the VMs on my production cluster(s).”
2) “I don’t want any other workload taking resources away from my Exchange/SQL/SharePoint VMs.”
I can see why folks make both of these arguments and can understand their logic. Many vSphere admins live in the Hosts and Clusters view where they see all virtual machines rather than those organized into folders (where granular permissions can be set more easily). And some are concerned that setting reservations on virtual machines to guarantee access to resources will have a negative impact on available slots in a cluster.
I don’t believe that either of these arguments are enough to require a dedicated vSphere cluster. A few thoughts are below, starting with addressing the two most common reasons I listed above.
1) VMs can be grouped into application specific folders, and granular permissions can be set to only grant access to those folders or VMs.
2) Reservations aren’t a bad thing or something to be scared of, they just need to be factored into any design. And using the “Percentage of Cluster Resources'” policy gets around worrying about individual slot sizes (though total resource assignments still need to be considered and calculated).
3) Fewer hosts in a cluster gives VMware features like DRS or HA fewer options when trying to initiate a failover or migrate virtual machines based on resource consumption. If an application is truly mission critical, I’d rather have more options to optimize availability and performance rather than fewer.
4) Let’s be honest – vSphere isn’t free or cheap, so dedicating hosts to a specific application is likely to result in heavily underutilized hosts and higher costs. Aren’t underutilized servers and cost reduction some of the reasons why we got into this virtualization business in the first place? If you made dedicated clusters for all of your mission critical applications you would end up with application silos that greatly reduce the efficiency of virtualization.
At the end of the day a design decision like this should always come down to the requirements. There may be very valid requirements, like compliance, security, or extremely large applications that make an isolated vSphere cluster a necessity. Assuming those requirements are not necessary in your environment I would strongly consider whether there is really a reason to use a dedicated cluster for your mission critical applications.
If you agree/disagree/think I’m crazy - feel free to let me know in the comments..
Playing with VMware Zimbra for Android
I’m not much of an app user to be honest, and I’m definitely not an app reviewer. I use an HTC Evo (which runs Android) and don’t have that many applications installed. By far the most important application my phone is the email client connecting to my corporate Exchange server.
When I saw that VMware had released a Fling called VMware Zimbra for Android I was eager to try it out. The email client that comes by default on the Evo is ok but has a number of drawbacks so I was eager to see what the Zimbra app could do. I thought I’d share some thoughts on this app in case anyone else was interested in trying it out.
Before I start – I know you can install custom ROMs on the Evo which lets you use stock Android without the HTC applications. And I’m sure that the email clients available in the Android market offer features above the stock HTC email client but they aren’t free.
First off, I thought I’d list some of the things that annoy me about the HTC email client that is included with the Evo.
1) You can’t multi-select messages to perform bulk actions, such as delete or mark as read.
2) When someone sends you a calendar appointment you can’t actually see what time the appointment is for. You can only accept/decline it. How this made it out of any QA/QC check I will never understand. Here’s what it looks like:
3) Tasks configured in Outlook/Exchange do not show up.
There are others annoyances but those are the big ones that I don’t like. I have been using the Zimbra client for the last few weeks and while it isn’t perfect it definitely has some nice benefits.
For starters, you can easily select multiple messages and perform actions on them (mark as read, delete, file into folder). Not only that, but you can also select which folders to sync on a regular basis. In the HTC app you need to go into the folder and then select Sync for new messages to come in (other than in the Inbox). And when typing a recipient in the To: field it will automatically search the Exchange GAL as you type. The HTC app can search the GAL too but it requires many more taps and is not automatic.
Tasks also show up and tasks with reminders will fire off a notification as well. In addition, you can actually see the time in a calendar appointment you receive in email. This screenshot is from the same appointment I sent myself in the above screenshot:
I really like this application a lot and have started using it more and more. It has good responsiveness and great features as compared to the default HTC email app. Unfortunately there are still some things that aren’t so great in this application, at least for me.
1) You can’t control how often it syncs, only how far back. It only syncs new emails as items arrive and you have no option to change it. Not the end of the world, but I like how HTC lets me set a sync schedule based on the time of day (great way to save battery).
2) The widgets that come with it are pretty basic, especially the calendar widget. The HTC calendar widget has lots of different configurations, including a single item view and a full scrollable agenda view. The Zimbra widget is a simple square that doesn’t fit as well on the screen as the HTC widget. Also, the Mail icon for the HTC mail client shows how many unread emails you have (similar to an iPhone). The Zimbra client is static and does not show unread messages. See the two screenshots below to see what I’m talking about (If you can look past my adorable son):
3) Saving attachments from an email is a little annoying because it puts attachments into their own folder under /mnt/sdcard/vmwarezimbra/attachments/<randomnumber>. In my case my wife sends me a picture of my son almost every day that I save to the phone and the HTC email client saves it into the default Downloads folder. That makes it really easy to find all of the pictures. Zimbra saves it into that random folder (in this case the folder 518) so the image gallery list gets cluttered quickly.
The only one that really impacts me is the image saving but I can probably work around it. Now that I’ve been using the Zimbra client exclusively for a while I’ve come to like it a lot. I’m going to continue to use it going forward and hope they can make some updates to get around the issue I’ve found.
My Next Step….
Whenever any virtualization blogger makes a post with a title like this it’s usually obvious what the post is about. So you can probably figure out where I’m going with this.
I’ve decided that it’s time to take on a new challenge in my career. To that end I’ve accepted an Advisory Consultant position with EMC and will be starting in early August. I’ll be able to work with the technology that I love with a focus on two key areas: Tier-1 application virtualization and VDI. EMC is a great company with huge exposure to virtualization, and the fact that many people in the virtualization industry that I respect already work there made this an opportunity I couldn’t pass up.
For the past ten years I’ve worked at a consulting company in New York City called Kraft & Kennedy. I was exposed to virtualization very early on (way back in 2001 with ESX 1.5) and was hooked ever since. I was able to learn the technology and eventually focus my career in that direction. They allowed me to attend VMworld, blog, tweet, and enabled me to become successful and for that I’ll always be grateful. It’s a great company with lots of very smart people and I’ve enjoyed my career there.
I’m hoping that having a stronger focus on virtualization will allow me to blog more frequently so you should see more content here soon. It helps that now I’ll only be writing for one blog - I was previously splitting my blogging duties between here and Kraft Kennedy’s Technology Blog.
I’m really excited about the move to EMC and am very much looking forward to getting started!
What being a vExpert means to me
VMware announced the list of VMware vExpert awards today and I was happy find out that I was among those who received the award. This is the second year in a row for me which made it extra sweet.
The vExpert award is given to those who go above and beyond to promote VMware and virtualization in the community. That includes the VMTN forums, blogging, tweeting, presenting and conferences, and many other things. More info on the vExpert award here: http://www.vmware.com/communities/vexpert/
To me the best part about winning this award is the honor in being included among a list of people in our industry that I truly respect. We have one of the best communities in all of IT and to be named in a list with some of the top people in our industry is truly an honor.
Thanks to John Troyer and the team of people who put this award together this year. I know it is a lot of hard work but know that the award means a great deal to us and we really appreciate it. In fact I still have the email from John from 6/4/2010 telling me I won the vExpert in 2010 sitting in my Inbox. I couldn’t bring myself to delete it or folder it. It was a reminder of the recognition and a motivator to keep working hard for next year.
If it’s true that you are judged by the company that you keep then I think I’ll be just fine…
Fix Partition Alignment with VMware Converter 5.0 (Beta)
I saw the news today that VMware released a public beta of VMware vCenter Converter 5.0 and was interested in the new features. Having used VMware’s P2V tools since ESX 2.x and the days of the P2V Assistant I’m always interested to see how the product matures to make the P2V process easier.
Probably the most anticipated feature is the following (from the Release Notes):
Optimized disk and partition alignment and cluster size change.
There are many other posts out there that talk about the value of aligning partitions for performance reasons so I won’t go into that here. Suffice to say that it’s a good thing to get your partitions aligned. All Windows operating systems since Windows 2008/Vista have been aligned by default, but other versions of Windows were not. When you undergo a large P2V project and bring in older systems, they will come into the environment unaligned and potentially hurt performance.
With the new version of VMware Converter, you can now align partitions on the fly as part of your P2V process. There are other tools/products out there that support this feature as well, but they are either tied to particular storage vendors or the products are not free.
Thankfully the new version of Converter will bring this functionality to everyone. This feature is enabled by default and will automatically align partitions during the conversion. When selecting the disks to convert, the "Create optimized partition layout” option is selected by default:
You should always first check with your storage vendor to confirm their recommended partition alignment. Here is how Converter does it (from the product documentation):
Creating an optimized partition layout aligns the partitions on a 1MB boundary for Windows sources and on a 64KB boundary for Linux sources.
A retweet by Forbes Guthrie asked the following: Does that mean converting an existing VM will fix disk alignment issue? My first thought was “why the heck not” but I wanted to test it to confirm. I downloaded Converter 5.0 Beta and ran a V2V conversion of an existing Windows 2003 VM.
Here is how the partition alignment looked before the conversion, showing the unaligned 32KB offset:
I ran the V2V conversion with the default option selected that you see above. After the conversion completed, here is how the partition alignment looked:
Sure enough the V2V conversion fixed the partition alignment to the proper 1MB offset as seen above. Also strangely enough Diskpart is reporting both volumes now have 0 bytes free each. I haven’t seen this happen after a P2V/V2V before and I’m not even sure it is anything to worry about. Both My Computer and Disk Management report the free space correctly so I’m not really worried about it but it’s definitely bizarre.
I’m not at all surprised by the results since if it works for a P2V there is no reason why it shouldn’t work for a V2V. This is good news for those of us who have P2V projects with older systems or are still dealing with older VMs. This could be a relatively easy fix if alignment issues are causing performance problems in your environment. After all, when virtualizing systems performance is important and every little bit helps.
Also, as always this is still a beta product so features might change and support is likely non-existent. I would wait until this product is officially released and supported before using this against production systems.
Clearing up confusion regarding HA/vMotion support for Exchange 2010
As I discussed last week, Microsoft has updated their guidance and support regarding hypervisor high availability and live migration technologies with Exchange 2010. This is welcome news for anyone looking to virtualize Exchange 2010 on vSphere, as it now allows you to take advantage of vMotion and HA on all Exchange servers, even those in a Database Availability Group (DAG).
Microsoft’s language on this change in support is causing a little bit of confusion on exactly what they do and do not support. See the following section below taken from the Exchange 2010 System Requirements page on TechNet (bolded emphasis mine):
Exchange server virtual machines, including Exchange Mailbox virtual machines that are part of a Database Availability Group (DAG), can be combined with host-based failover clustering and migration technology as long as the virtual machines are configured such that they will not save and restore state on disk when moved or taken offline. All failover activity must result in a cold start when the virtual machine is activated on the target node. All planned migration must either result in shut down and a cold start or an online migration that utilizes a technology such as Hyper-V live migration.
The bolded section seems to be the part that is confusing some people into thinking that some VMware features are not supported. But think about it for a second – does that description really sound like any VMware feature other than a simple Suspend? vMotion doesn’t save anything to disk during a migration and HA doesn’t save anything during a failure/restart of a VM. So what is Microsoft describing here?
Remember back when Hyper-V was released and Microsoft was touting a feature called Quick Migration to compete with VMware's vMotion? It was called Quick Migration and not Live Migration because the virtual machine was paused briefly and the state of the virtual machine was saved to disk during the migration. From the Quick Migration with Hyper-V (opens a .doc) whitepaper:
For a planned migration, quick migration saves the state of a running guest virtual machine (memory of original server to disk/shared storage), moves the storage connectivity from one physical server to another, and then restores the guest virtual machine onto the second server (disk/shared storage to memory on the new server).
It’s clear that the language in the TechNet article applies only to Hyper-V Quick Migration and not vMotion or even Hyper-V’s newer Live Migration feature. Microsoft probably felt they had a large enough installed base of Hyper-V RTM that clarifying this language was important.
Rumor has it that enough people are confused by this language that Microsoft is planning on clarifying it soon. At the end of the day this language doesn’t apply to those of us virtualizing Exchange 2010 on vSphere. Both HA and vMotion are now supported and that’s all that matters to us.
vSphere and Exchange admins can live in harmony – Microsoft finally supports HA and vMotion
This Saturday, Microsoft published a new white paper entitled “Best Practices for Virtualizing Exchange Server 2010 with Windows Server 2008 R2 Hyper V” that provides a lot of great info that is applicable to VMware as well. One of the most important things in this entire document is a change in policy regarding supporting virtualized Exchange 2010 with Database Availability Groups (DAG) in combination with hypervisor high availability and live migration. Previously Microsoft did not support the use of high availability or live migration even on its own Hyper-V platform. In the VMware world this of course means HA and vMotion. The whitepaper states the following:
Exchange server virtual machines, including Exchange Mailbox virtual machines that are part of a Database Availability Group (DAG), can be combined with host-based failover clustering and migration technology as long as the virtual machines are configured such that they will not save and restore state on disk when moved or taken offline. All failover activity must result in a cold start when the virtual machine is activated on the target node. All planned migration must either result in shut down and a cold start or an online migration that utilizes a technology such as Hyper-V live migration
This new document, as well as a post on the MS Exchange Team blog, confirms the new support stance. The Technet page has been updated as well. Note that you must be running Exchange 2010 SP1 in order to support these features.
Folks may know that I’m a big proponent of virtualizing mission critical, tier-1 applications like Exchange 2010. I’ve written about it here, touched on it here, and commented to TechTarget on the subject here and most recently here. It’s clearly an important subject to me and I applaud Microsoft for introducing this change. I think this will help to convince organizations that it is safe to virtualize Exchange 2010 on all hypervisors.
But..there must always be a but..
Remember that just because Microsoft now officially supports something doesn’t actually change anything in terms of functionality. Did VMware HA and vMotion work properly in combination with Exchange 2010 and DAGs before this policy change? VMware HA – sure, it just wasn’t officially supported. VMware vMotion – umm, hang on a minute there.
The DAG ultimately relies on Windows Failover Clustering to work, and WFC is notoriously finicky about even brief drops in network connectivity and loss of heartbeat. When performing a live migration using vMotion there is usually at least one ping dropped, and in my experience that single drop is often enough to cause databases to failover to other nodes in the DAG.
Does this mean that even though Microsoft supports vMotion now that you still can’t use it? Of course not, but it does require a slight change in your design to increase the cluster heartbeat timeout value to allow for the brief network interruption.
The values that need to change are the following:
SameSubnetDelay: The value (in milliseconds) of the cluster heartbeat frequency. By default, this value is 1,000 milliseconds.
SameSubnetThreshold: The value represents the amount of missed heartbeats that will be tolerated before a failover event occurs. By default this value is 5, so combined with the above value that means 5 seconds of lost heartbeats will result in a cluster failover by default.
Five seconds seems like enough time for a vMotion to complete, but in practice I’ve seen databases failover at multiple clients when using the default heartbeat values. Luckily you can change these values very easily by using PowerShell. The following commands show how to raise the timeout to 10 seconds (the Microsoft recommended max) from the default of 5, taken directly from the Microsoft whitepaper:
Import-module FailoverClusters
(Get-Cluster).SameSubnetThreshold=10
(Get-Cluster).SameSubnetDelay=1000
Depending on your environment you may not need to make this change, so always test first before implementing any cluster wide change like this. Make sure you have enough bandwidth on your hosts to account for migrating an Exchange VM that may have 32GB of RAM or more. And of course always stick with configurations that are supported by Microsoft.
I’m happy that Microsoft made this change, and hope that it signals a trend towards more virtualization friendly licensing in the future.
VMware conducting vCenter high availability survey
This week on VMware's Uptime Blog I noticed a new post asking for customer feedback about vCenter high availability requirements. To that end, VMware has posted a survey for customers to fill out to get more information. To be clear, the survey addresses high availability of the vCenter server itself and does not relate to VMware's HA feature that is included in vSphere.
I'm happy to see VMware addressing this topic as it's one I've been a little concerned about for a while. Julian Wood posted a long article over on his blog that discusses this issue in great detail, and Stu over at vinternals also discusses it in this post. The short version is this - as vCenter has matured over the years, more and more products and features have been integrated and ultimately rely on vCenter to work. Applications like View, AppSpeed, CapacityIQ, etc., all rely on vCenter to work and so protecting it has become very important.
I remember when vCenter (then VirtualCenter) first came out for ESX 2. There was little concern about VirtualCenter availability back then because you'd really only lose vMotion if the server went down. This was of course before DRS, HA, DPM, etc. Now that vCenter is a much more critical component of a vSphere environment, it is becoming increasingly important to ensure its uptime.
Using tools like VMware HA and vCenter Heartbeat can really help to improve availability, but is it enough? It sounds like VMware is curious what customers think, and so they've asked customers to fill out this survey. I'd encourage everyone to fill out the poll honestly so that VMware can understand their customer's needs and work to improve vCenter in the future.
VMware Tools Shrink Option Unavailable?
In my recent post on using svMotion to reclaim disk space, I talked about using SDelete to zero out blocks for free space reclamation. Another way to do this is to use the Shrink option which is part of the VMware Tools. Duncan's post here gives details on using the VMware Tools Shrink option so I won't go into much detail, but it essentially accomplishes the same goal as SDelete.
I saw a post earlier this week in the VMTN forums that mentioned the Shrink option wasn't available and so I wanted to clear one thing up. The Shrink option is only available in the VMware Tools if the guest is using thick provisioned disks. If you're trying to use the Shrink option on a thin provisioned VM (or if you have snapshots on a thick provisioned disk), you'll see the following message in the VMware Tools Shrink tab:
Shrink is disabled for this virtual machine. Shrinking is disabled for virtual disks not used in persistent mode and other factors. For more information, see the documentation for your VMware product.
VMware Tools Shrink Disabled
On a VM that is thick provisioned, you'll see a Shrink tab that looks like the following:
VMware Tools Shrink Enabled
Hope this helps someone who is confused about why they can't use the Shrink option in VMware Tools.
Reclaiming disk space with Storage vMotion and Thin Provisioning
I've seen a number of posts on the VMTN forums as well as had discussions with folks regarding reclaiming disk space with Storage vMotion so I thought it made sense to do a quick write-up on how this process works.
The situation is unfortunately all too common - virtual machines are deployed and treated like a physical machine in terms of resource all0cation. In the physical world it is common to see VMs with multiple processors, lots of RAM, and at least 73GB disks (of which maybe 25GB is actually used). Folks often have a hard time letting go of that configuration, and so they'll bring it into the virtual world. While it is easy and relatively painless to reduce CPU and memory, reducing the size of a virtual disk to reclaim some space on your VMFS partition isn't as straightforward. Still easy, just not straightforward.
You have two solutions to the problem depending on how you want to reclaim the space:
1) Use VMware Converter to V2V the VM and resize the disk. This will reduce the size of the disk both in the guest and on the VMFS partition, but requires a brief amount of downtime to swap between the original VM and the converted VM.
2) Use Storage vMotion along with thin provisioning in vSphere to resize the disk on the VMFS partition but leave it the original size in the guest. The size of the disk will grow as data is written to the VM.
Most of the time option 2 works without a problem. But what about the case where you previously had data on the drive and deleted it and wish to reclaim the space. In that case the Storage vMotion will likely not be able to reclaim the space. Why? Because the blocks that had been previously written to the drive need to be zeroed out first.
The easiest way to do that is to use a tool like SDelete. SDelete is capable of scanning a drive and zeroing out blocks from deleted files so that they can be reclaimed as free space. Use the following command to zero out the free space:
Sdelete.exe -c c:\ <-- Replace the c:\ with the appropriate drive letter.
In this example we have a VM that has a thick provisioned 25GB C: drive and nearly all of the space on disk was consumed. I deleted some old data and now show 15.7GB free inside the VM. I then ran SDelete as listed above and would like to convert the VM to thin provisioned and reclaim some space. Here is how it looks now:
25GB Consumed + 2GB for VMkernel swap
I then right click on the VM, choose Migrate, and select that I want to Change Datastore. Once you get to the Disk Format screen, remember to select the "Thin provisioned format" option as seen below:
Thin provisioned format
After the Storage vMotion is complete, the disk is now thin provisioned and takes up much less space:
If you're going to use thin provisioning in vSphere, make sure you have the proper alerts setup in vCenter to notify you when space is getting tight. It's a great way to better utilize disk space as well as reclaim space without downtime.
Recent Posts
- 2012 – The Year of SQL Virtualization
- Microsoft Jetstress 2010 now supported in virtual machines
- Virtual machine snapshots and tier-1 apps: Not always supported
- Guest clustering on vSphere: A familiar topic revisited
Recent Comments
- EGarbuzov on vSphere and Exchange admins can live in harmony – Microsoft finally supports HA and vMotion
- virtualTodd on 2012 – The Year of SQL Virtualization
- Matt Liebowitz on 2012 – The Year of SQL Virtualization
- Jase McCarty on 2012 – The Year of SQL Virtualization