Low Cost Disaster Recovery Options For VMware

For years now Corner Edge Solutions has been using VMware exclusively for all of our new server installs, and even for hardware upgrades by virtualizing the original server install and configuration to a new physical server with VMware.  This goes for even a small, one server setup as well.  We have felt this is a great way to increase reliability as well as improve disaster recovery times.

While having two complete setups of VMware is probably cost prohibitive, with the small footprint of VMware ESXi, you can use a simple workstation or even a laptop as a cold spare DR backup.  As you’ll see below, I have easily installed VMware on my laptop, and with a large enough hard drive, and enough memory, I could run a small to medium office server environment setup on my one laptop, or even a mid-range business desktop.  The cost of these is usually around $1500-2500, and when you consider the cost of a second server may come in at $5000 or more, this is a great low-budget way to have your office back up and working quickly in the event of a major disaster.

To do this simply, just power off the VM’s on a schedule that fits your DR needs and copy the files from the main DataStore and upload them the the DataStore on your backup setup.  You will want to make sure your backup processor is a 64-bit proc with VT enabled if you are running 64-bit VM’s, you have enough storage space for the foreseeable future, and definitely install as much memory as your budget and workstation will allow, and that should be it.

You clearly shouldn’t expect the same performance of this setup as you would get from a true server, but it would get people back online and running again while you work on the main server.


Here is a quick picture I took of my laptop running VMware ESXi just for fun.  I had installed ESXi on a USB stick, and booted to that when I powered on my laptop.  This install was originally done on a PowerEdge 2950, and without any modifications to the install, it came up just fine on an Dell Latitude E6500.  Simply carry a USB flash drive and a large external storage drive and you can have a backup ESXi server wherever you go.


VMware ESXi Server Keeps Running with Failed RAID Array

We at Corner Edge Solutions LOVE VMware.  It’s not too hard to tell that based on our blog, but this past week we found a new reason to fall in love all over again.  One of our ESXi 4 servers in a cluster had a double drive failure on our RAID 5 array, which would have completely crashed a server had it been a typical setup, but since it is running VMware ESXi with all the VM DataStores on a iSCSI storage device, we had ZERO impact on our environment.  ESXi is the lightweight version of the original ESX server which runs entirely in memory, not requiring disk access once it has been loaded at startup.

Since this machine was part of a cluster, we simply migrated the VMs on the failed server to the other working ESXi server through vSphere vCenter Server.  The working VMware server was able to overcommit the available physical memory by almost 50% with room to spare.  We then took down the server with the bad drives to rebuild.  We also took this opportunity to install the OS onto a USB flash drive, which installed internally to the server, and remove the remaining two working hard drives to run a completely diskless server configuration.  With a small amount of configuring to VMware, the newly rebuilt server was ready to join the cluster again and the VMs were then evenly distributed throughout the cluster, all the while never having to power anything off.  That means never having to send out maintenance notices to customers that their hosted servers will be offline, and keeping out uptime in tact.  The whole process took only about 5 hours as well.  When was the last time a total failure on a system RAID drive, and nothing went down, and everything was upgraded and repaired in 5 hours?

High CPU Usage in VMware Guest may NOT be CPU Problem

Hi CPU usage in the VMware guest may not be caused from an overloaded CPU.  It may be from a DataStore that is too slow.  Check the performance tab in your VMware client and see if that too is showing high CPU utilization.  If it isn’t, consider the speed of your DataStore.

We recently moved a VM from a RAID 5 with 3 slower SATA drives to a RAID 10 with 6 SAS 15k drives and with no other changes to the VM configuration and the machine performance has increased dramatically.

The problem appears to be caused by a backup writing files to the disk, which backs up the processor causing it to run at near 100% utilization waiting to be able to move on to its next task.  This will probably be most noticeable with database servers like Exchange and SQL.

Take this into consideration also when you are spec’ing out a new VMware server, as upgrading the DataStore disks to faster ones is not a simple job.  Include growth into your design; it’s always better to have an under-utilized server that is fast than one that is over-utilized and slow.

Can’t Delete DataStore on New VMware ESXi Install even with embedded SDcard

Like many of you out there, when I am creating a VM, there are times I want a virtual hard drive larger than 256GB.  Unfortunately VMware defaults the DataStore block size to 1MB, which limits you to a virtual HDD of only that size.  In a previous post (here), I walked you through deleting and recreating the DataStore to get the larger block size needed for a VHDD of up to 2TB.  But what happens when you can’t delete the DataStore to recreate it.  This is the error I am talking about: “Error during the configuration of the host: DestroyVmfsDatastore: can’t delete partition 3 on lun vmhba1:0:0″

It is no surprise when you install VMware, it installs some other information on the drives for the OS, but I was surprised to see files on the local hard drives when I was installing VMware on the embedded SDcard we typically specify with our servers.  Despite installing the actual OS on a separate flash drive, the install actually wrote some data to one of the drives on the server.

Compare DataStore1 which has VMware ESXi installed on it to DataStore 2 which has nothing but VM’s on it.  You can see there are a lot more partitions, which is why the DataStore won’t delete.  If VMware was installed on an embedded flash drive you would see less partitions on DataStore1, but still a lot more than on DataStore2.

The easiest way to avoid this is if you are installing to an embedded SD card is to not attach your hard drives or configure your virtual disks if you are using a RAID card until after the ESXi installation.  If the drives aren’t there, then the installation can’t write files to them.

If, however you have already installed VMware ESXi to the flash drive and don’t want to reinstall it again with the drives disconnected, the easiest way it from inside the RAID card utility at startup, just re-initialize the offending drive, which will delete all data that has been written to it and then you will be able to create the DataStore with the larger block size.

If you do not have the option of installing to an embedded flash drive, then my recommendation would be to create a 1GB virtual disk in your RAID card first, install VMware there, and then create the remaining disks in the RAID utility after.  Then, when you are creating the DataStore in VMware, you can specify the block size then.

I hope this helps take away some of the confusion on why you cant delete a DataStore on a clean install with no VM’s even created.

How to virtualize an old computer

As developers and systems engineers we use an immense amount of software tools and applications.  Transfering to using a new computer seems like it can take months before you have “all you need” for our daily tasks.

When I took delivery of my new laptop in January I struggled with this issue.  My old machine had been in place for 3 years and now I was faced with starting fresh.  64-bit Vista on my laptop also presented a challenge for old applications I used.  Not to mention that old PCanywhere, old VPN connections, etc…

SO, I decided to virtualize my old laptop.  I can litterally run that computer in VMware faster than I could before, and I don’t have to worry about old applications or old VPN configurations for customers, they all just work!

So how do you do it?  Simply install the free Vmware converter on the machine you want to virtualize and point it to your nearest VMWare workstaion or server.  I highly recommend this, it was a truely painless process.

Fore more information check out these sites:

VMware converter site

How to run VMWARE converter step by step

Data Store Size Limits in VMware ESX and ESXi

Well, you took the leap and are now virtualized.  You’re now doing more than ever, and data size is growing rapidly.  Time to add a new virtual hard drive to your machine, but wait…  I said 500 GB, why is it only 256 GB.  Well, you hit a limitation of the data store in VMware under your current default configuration.  Check to see what your data store block size is.  here’s where to find it:

Highlighted you will see the datastore block size.

Highlighted you will see the data store block size.

When data stores are created, their default block size is 1 MB, which gives you a maximum virtual hard drive size maximum of 256 GB.  So how do you get larger VHD’s?

Hopefully you are reading this and have a brand new ESX/ESXi setup, in which case you can just delete the data store and recreate it, choosing a different block size.  If you already have machines running on the data store, you have a project ahead of you, because deleting the data store will format all data on that drive, and you will have to start from scratch, or be creative before you make the change (there are some ideas of how to work around this below).

If you have the disk space to cover 2TB, then I would go with the maximum of 8MB block size to give you a maximum virtual HDD size of 2TB.  There is no noticeable I/O performance difference by using maximum size, so use the largest size to mazimize your storage.  Here is a quick reference of what block size you can choose and what the maximum VHD that will give you:

Block Size     Max VHD size

1MB                256GB
2MB               512GB
4MB               1TB
8MB               2TB

Already have servers running?  How do you fix it?

If you already have the data store in use, and can afford some downtime for a maintenance window, here is a workaround you could do, asuming you have more than one ESX(i) server at your disposal.  You can power the VM down and use the free VMware vCenter Converter (http://www.vmware.com/products/converter/) to move the virtual machine from one ESX(i) server to another.  Figure on about 1 minute per gig of hard drive size when moving it, with a GB network.  Once the VM is moved to its new location, power it an and make sure all is working well before you delete from disk the VM on the original ESX(i) server.  Once all the VMs are moved off the ESX server, you can go ahead and remove the data store and create a new one using the new block size.

If you hapen to have your VMs in a cluster with vMotion, this task is even easier, as you can change the location of the datastore through the migrate option.  If you dont have any other ESX servers, you could probably do it to VMware Servers, but at that point, you would probably be better just adding multiple drives to the VMs, it would be a lot less work.

Here is a nice reference guide from VMware with this and other importand configuration information: http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_config_max.pdf

SysinfoException… a cryptic Vmware ESX 3.5 Vmotion error translated

Corner Edge has just completed a MAJOR implementation of a Vmware ESX cluster. We have greatly improved our data center capabilities and offerings.

This time around I want to add another tidbit to the online body of knowledge.

I recently received the following error while configuring Vmotion on our Vmware ESX 3.5 server cluster:

Error during the configuration of the host: SysinfoException: Node (VSI_NODE_net_tcpip_plumb) ; Status(bad0005)= Already exists; Message= Unable to Set

Now what is THAT supposed to mean? Vmware has amazing technology, but that “amazing” thing hasn’t yet reached their error messages.

It turns out that if your iSCSI traffic is configured with a specific network (Example:, and you try to configure Vmotion ports within the same network space (Example:, the Virtual Center wizard fails with the error above.

“Best practices” dictate that iSCSI traffic and Vmotion traffic should be completely separate from each other. But even if they are physically separated, and you may want use the same network space… well, you can’t.

Solution: Make the network address space for iSCSI (Example: 10.0.10.x) and Vmotion (Example: 10.0.20.x) different.

Oh and a better error message could be:

Wow buddy, you may think you’re really smart using the same network for iSCSI and Vmotion, but might as well forget it. I refuse to work that way. Make them separate networks!