Low Cost Disaster Recovery Options For VMware

For years now Corner Edge Solutions has been using VMware exclusively for all of our new server installs, and even for hardware upgrades by virtualizing the original server install and configuration to a new physical server with VMware.  This goes for even a small, one server setup as well.  We have felt this is a great way to increase reliability as well as improve disaster recovery times.

While having two complete setups of VMware is probably cost prohibitive, with the small footprint of VMware ESXi, you can use a simple workstation or even a laptop as a cold spare DR backup.  As you’ll see below, I have easily installed VMware on my laptop, and with a large enough hard drive, and enough memory, I could run a small to medium office server environment setup on my one laptop, or even a mid-range business desktop.  The cost of these is usually around $1500-2500, and when you consider the cost of a second server may come in at $5000 or more, this is a great low-budget way to have your office back up and working quickly in the event of a major disaster.

To do this simply, just power off the VM’s on a schedule that fits your DR needs and copy the files from the main DataStore and upload them the the DataStore on your backup setup.  You will want to make sure your backup processor is a 64-bit proc with VT enabled if you are running 64-bit VM’s, you have enough storage space for the foreseeable future, and definitely install as much memory as your budget and workstation will allow, and that should be it.

You clearly shouldn’t expect the same performance of this setup as you would get from a true server, but it would get people back online and running again while you work on the main server.

Here is a quick picture I took of my laptop running VMware ESXi just for fun.  I had installed ESXi on a USB stick, and booted to that when I powered on my laptop.  This install was originally done on a PowerEdge 2950, and without any modifications to the install, it came up just fine on an Dell Latitude E6500.  Simply carry a USB flash drive and a large external storage drive and you can have a backup ESXi server wherever you go.

VMware ESXi Server Keeps Running with Failed RAID Array

We at Corner Edge Solutions LOVE VMware.  It’s not too hard to tell that based on our blog, but this past week we found a new reason to fall in love all over again.  One of our ESXi 4 servers in a cluster had a double drive failure on our RAID 5 array, which would have completely crashed a server had it been a typical setup, but since it is running VMware ESXi with all the VM DataStores on a iSCSI storage device, we had ZERO impact on our environment.  ESXi is the lightweight version of the original ESX server which runs entirely in memory, not requiring disk access once it has been loaded at startup.

Since this machine was part of a cluster, we simply migrated the VMs on the failed server to the other working ESXi server through vSphere vCenter Server.  The working VMware server was able to overcommit the available physical memory by almost 50% with room to spare.  We then took down the server with the bad drives to rebuild.  We also took this opportunity to install the OS onto a USB flash drive, which installed internally to the server, and remove the remaining two working hard drives to run a completely diskless server configuration.  With a small amount of configuring to VMware, the newly rebuilt server was ready to join the cluster again and the VMs were then evenly distributed throughout the cluster, all the while never having to power anything off.  That means never having to send out maintenance notices to customers that their hosted servers will be offline, and keeping out uptime in tact.  The whole process took only about 5 hours as well.  When was the last time a total failure on a system RAID drive, and nothing went down, and everything was upgraded and repaired in 5 hours?

High CPU Usage in VMware Guest may NOT be CPU Problem

Hi CPU usage in the VMware guest may not be caused from an overloaded CPU.  It may be from a DataStore that is too slow.  Check the performance tab in your VMware client and see if that too is showing high CPU utilization.  If it isn’t, consider the speed of your DataStore.

We recently moved a VM from a RAID 5 with 3 slower SATA drives to a RAID 10 with 6 SAS 15k drives and with no other changes to the VM configuration and the machine performance has increased dramatically.

The problem appears to be caused by a backup writing files to the disk, which backs up the processor causing it to run at near 100% utilization waiting to be able to move on to its next task.  This will probably be most noticeable with database servers like Exchange and SQL.

Take this into consideration also when you are spec’ing out a new VMware server, as upgrading the DataStore disks to faster ones is not a simple job.  Include growth into your design; it’s always better to have an under-utilized server that is fast than one that is over-utilized and slow.

Data Store Size Limits in VMware ESX and ESXi

Well, you took the leap and are now virtualized.  You’re now doing more than ever, and data size is growing rapidly.  Time to add a new virtual hard drive to your machine, but wait…  I said 500 GB, why is it only 256 GB.  Well, you hit a limitation of the data store in VMware under your current default configuration.  Check to see what your data store block size is.  here’s where to find it:

Highlighted you will see the data store block size.

When data stores are created, their default block size is 1 MB, which gives you a maximum virtual hard drive size maximum of 256 GB.  So how do you get larger VHD’s?

Hopefully you are reading this and have a brand new ESX/ESXi setup, in which case you can just delete the data store and recreate it, choosing a different block size.  If you already have machines running on the data store, you have a project ahead of you, because deleting the data store will format all data on that drive, and you will have to start from scratch, or be creative before you make the change (there are some ideas of how to work around this below).

If you have the disk space to cover 2TB, then I would go with the maximum of 8MB block size to give you a maximum virtual HDD size of 2TB.  There is no noticeable I/O performance difference by using maximum size, so use the largest size to mazimize your storage.  Here is a quick reference of what block size you can choose and what the maximum VHD that will give you:

Block Size     Max VHD size

1MB                256GB
2MB               512GB
4MB               1TB
8MB               2TB

Already have servers running?  How do you fix it?

If you already have the data store in use, and can afford some downtime for a maintenance window, here is a workaround you could do, asuming you have more than one ESX(i) server at your disposal.  You can power the VM down and use the free VMware vCenter Converter (http://www.vmware.com/products/converter/) to move the virtual machine from one ESX(i) server to another.  Figure on about 1 minute per gig of hard drive size when moving it, with a GB network.  Once the VM is moved to its new location, power it an and make sure all is working well before you delete from disk the VM on the original ESX(i) server.  Once all the VMs are moved off the ESX server, you can go ahead and remove the data store and create a new one using the new block size.

If you hapen to have your VMs in a cluster with vMotion, this task is even easier, as you can change the location of the datastore through the migrate option.  If you dont have any other ESX servers, you could probably do it to VMware Servers, but at that point, you would probably be better just adding multiple drives to the VMs, it would be a lot less work.

Here is a nice reference guide from VMware with this and other importand configuration information: http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_config_max.pdf