Automatically converting machines to thin provisioned format

I haven’t been posting too much here recently I’m afraid. A lot of the things that I’m currently working on are pretty specific to the environment here, and are not particularly useful (or indeed, interesting) to anyone else.

One of the things I’ve been doing might be more generally useful. We needed to convert around 700 of our machines to thin-provisioned format. When migrating to the new environment we’d stuck to traditional “thick” machines, as there was a lot of upheaval, and we didn’t have the necessary monitoring in place. Now that things have settled, we were looking to take advantage of the thin format to save us some space. We had already run SDELETE on the (non-persistent) machines during the migration, so we were ready to go.

The conversion process is fairly simple: a new option during a storage vMotion. However, the idea of doing this 700 times did not appeal to me, so I wrote the script below to automate the process.

# Convert Machines To Thin Provisioned
# Ben Neise 16/03/10

# Create an empty array
$arrMachinesToBeConverted = @()

Write-Host "Getting virtual machine objects" -ForegroundColor Blue
Write-Host ""

# Select the broad category of VMs we're looking at, for example all the machines in a specific blue folder
$objVMs = Get-Folder "Projects" | Get-VM | Sort-Object

Write-Host "Generating list of candidates" -ForegroundColor Blue
Write-Host ""

# Loop through all the virtual machines selected
ForEach ($objVM in $objVMs){

	Write-Host "Investigating " -NoNewline
	Write-Host $objVM -ForegroundColor Blue -NoNewline

	# Skip the rest of the loop if the machine is thin provisioned
	If ($objVM | Get-HardDisk | Where-Object {$_.StorageFormat -like "Thin"}){
		Write-Host " - disks are already thin provisioned" -ForegroundColor DarkGray
		continue
	}

	# Skip the rest of the loop is the machine is switched on and non-persistent
	If ($objVM | Where-Object {$_.PowerState -ne "PoweredOff"} | Get-HardDisk | Where-Object {$_.Persistence -notlike "IndependentPersistent"}){
		Write-Host " - switched on and non-persistent" -ForegroundColor DarkGray
		continue
	}

	# Skip the rest of the loop if the machine is powered-on, and has snapshots
	If ($objVM | Where-Object {$_.PowerState -ne "PoweredOff" -and ($objVM | Get-Snapshot)}) {
		Write-Host " - switched on and with snapshots" -ForegroundColor DarkGray
		continue
	}

	# Skips the rest of the loop if the machine has a shared drive and is not set up as fault tolerant (indicating that it's a Linked Clone)
	# Thanks to Keshav Attrey for this method - http://www.vmdev.info/?p=546)
	$viewVM = $objVM | Get-View -Property Name,Summary,Config.Hardware.Device
	$unshared = $viewVM.Summary.Storage.Unshared
	$committed = $viewVM.Summary.Storage.Committed
	$ftInfo = $viewVM.Summary.Config.FtInfo
	If (($unshared -ne $committed) -and (($ftInfo -eq $null) -or ($ftInfo.InstanceUuids.Length -le 1))){
		Write-Host "The machine is a linked clone" -ForegroundColor DarkGray
		continue
	}

	Write-Host "Added to the list of machines to be converted"
	$arrMachinesToBeConverted += $objVM
}

Write-Host "Starting Storage vMotions" -ForegroundColor Blue
Write-Host  $arrMachinesToBeConverted.Count -ForegroundColor Blue -NoNewline
Write-Host " machines to be converted" -ForegroundColor DarkGray
Write-Host ""

ForEach ($objVM in $arrMachinesToBeConverted | Sort-Object){

	# Get the biggest datastore
	$objBiggestDatastore = Get-Datastore | Sort-Object -Property FreeSpaceMB -Descending
	# Select the datastore from the top of the previously generated list (index 0) and remove the preceeding "Datastore-" from it's ID to give us the MOID
	$strTargetDatastore = ($objBiggestDatastore[0].Id).replace('Datastore-','')	

	# Let the user know what's going on
	Write-Host "Migrating machine " -NoNewline
	Write-Host $objVM -ForegroundColor Blue -NoNewline
	Write-Host ", to Thin Provisioned format on datastore " -NoNewline
	Write-Host $objBiggestDatastore[0] -ForegroundColor Blue -NoNewline
	Write-Host ""

	# Get the view of the VM we're moving
	$viewVM = Get-View -VIObject $objVM
	# Remove the preceding "HostSystem-" from the VM's Host's ID, giving us the Host's MOID
	$strTargetHost = ($objVM.host.id).replace('HostSystem-','')
	# Create a task specification for the relocation
	$specRelocate = New-Object VMware.Vim.VirtualMachineRelocateSpec
	# Add the target datastore to the specification using the MOID
	$specRelocate.datastore = New-Object VMware.Vim.ManagedObjectReference
	$specRelocate.datastore.type = "Datastore"
	$specRelocate.datastore.value = $strTargetDatastore
	# Add the host to the specification using the MOID
	$specRelocate.host = New-Object VMware.Vim.ManagedObjectReference
	$specRelocate.host.type = "HostSystem"
	$specRelocate.host.value = $strTargetHost
	# This is the specification property that makes the disk Thin Provisioned
	$specRelocate.transform = "sparse"
	# Create the task where the previously specified task is applied to the view of the target VM
	$task = $viewVM.RelocateVM_Task($specRelocate, $priority)
	# Start the task, and wait for it to complete before continuing
	Get-VIObjectByVIView $task | Wait-Task | Out-Null

}

It works through the collection of virtual machine objects in $objVMs, and adds them to an array if they pass certain criteria. They must be:-

• Not be currently thin provisioned
• Not switched on and non-persistent
• Not switched on with snapshots
• Not a linked clone (thanks to Keshav Attrey for his method of finding linked clones)

It then starts a separate loop going through each machine object in the array. I could have integrated the migration task into the first loop, but this approach means you can add a human “sanity-check” if you’ve got certain machines you don’t want to migrate.

The script finds the datastore with the largest amount of free space, and then the actual migration is done using code which I generated using Onyx. The free space on the datastores is re-evaluated before every move. This makes the script quite slow, but this isn’t something you want to rush.

vSphere bug with DRS, StandBy and non-persistent hard drives

We’ve been in touch with VMware recently about an issue we were experiencing in vSphere 4, where machines in standby could not be powered on. VMware have now confirmed that this is a bug, and that there will be a fix in R2.

While it’s fairly specific to our use-case, I thought I’d share the details in case anyone else runs into this.

First of all, this bug will only affect you if the following conditions are met:

• You are using VMware vSphere 4.0 (or 4.0 Update 1)
• Guest OS power-saving settings cause the virtual machine to enter standby
• One or more of the guest’s hard drives are set to “independent non-persistent”
• DRS is enabled on the virtual machine’s cluster

The machine enters standby as normal. The issue arises when you try to power the virtual machine back on: if DRS has allocated the machine to another host based on load the machine will not resume, and gives an error similar to the following:-

“Virtual Machine is configured to use a device that prevents the operation: Device ‘Hard disk 1′ is disk which is not in persistent mode. Device ‘Hard disk 1′ which is not in persistent mode”.

You cannot manually migrate the machine (even back to the original host). You cannot change the power-state on the machine, edit the virtual machine settings, or delete the machine.

If this has happened to you, the only way we’ve found to get the machine back up-and-running seems to be to remove the machine from inventory, then create a new virtual machine with the same specifications, and add the old machine’s VMDK.

Fortunately, there are a couple of workarounds. You can either disable power-saving settings in the guest OS or change the guest power management settings from “Suspend the virtual machine” to “Put the guest OS into standby mode and leave the virtual machine powered on” (you can automate this as described in my previous post).

Changing the guest power-management settings means that when the guest enters standby, although vSphere shows the machine as “powered-on”, VMware Tools is not running, which can cause problems (i.e., when trying to gracefully shut down a batch of machines).

This was also my first time working with the VMware vSphere support and I was impressed. They quickly replicated the problem and confirmed that it was indeed a bug. As most people nowadays tend to use snapshots rather than non-persistent drives, and few users virtualise desktop operating systems (which are more likely to have power-saving settings on by default) I can understand why this particular set of circumstances went untested.

Changing StandByAction using PowerShell script created with help from Onyx

We’re currently having some issues caused by the convergence of vSphere 4.0, IndependentNonPersistent drives, StandBy and DRS (I’ll post more on that later).  As a workaround, we needed to modify 228 machines so that they did not go into hibernation. You can do this though the vSphere Client by right clicking the virtual machine, click Edit Settings, go to the Options Tab, then select Power Management, and changing the radio button. We were wanting to change from “Suspend the virtual machine” to “Put the guest OS into standby mode and leave the virtual machine powered on”.

To do this the machines need to be powered down. We had an imminent maintenance window, but it wouldn’t allow us the time to make this change manually (even if we wanted to), this necessitated some automation. Unfortunately I had no idea how to go about editing this setting using the PowerCLI, even after a little search through the VMware PowerCLI community.

This seemed like the perfect opportunity to try out Project Onyx.

Carter Shanklin’s video does a good job of explaining how to Onyx up and running, and it worked exactly as described (even on my Windows 7 machine).

1. Download the Onyx files and extract to a folder
2. Run the Onyx executable

1. Click the Connect button, and connect to your VirtualCenter server.
2. Once that’s launched, start vSphere client, but instead of connecting to your VirtualCenter server, connect to http://localhost:1545 (Carter actually says 1445 in the video, but you can see on screen that he’s using 1545). Use your normal credentials.
3. Ignore the warning about unencrypted traffic (as Carter explains, the unencrypted traffic is local-only, the network traffic is still encrypted)
4. Click the Start button on Onyx
5. In vSphere client make whatever changes it is that you’re wanting to record.
6. Click the Pause button on Onyx, and you’ll see in the window a script has been created.
7. Copy this into your favourite PowerShell editor, and modify until it’s suitable for your purposes.

The original capture from the Onyx Window

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeVersion = "2009-11-27T09:16:04.570821Z"
$spec.powerOpInfo = New-Object VMware.Vim.VirtualMachineDefaultPowerOpInfo
$spec.powerOpInfo.defaultPowerOffType = "soft"
$spec.powerOpInfo.defaultSuspendType = "hard"
$spec.powerOpInfo.defaultResetType = "soft"
$spec.powerOpInfo.standbyAction = "checkpoint"

$_this = Get-View -Id 'VirtualMachine-vm-1074'
$_this.ReconfigVM_Task($spec)

A second capture changing the setting back to isolate the exact line that makes the changes

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeVersion = "2009-11-27T09:16:33.872017Z"
$spec.powerOpInfo = New-Object VMware.Vim.VirtualMachineDefaultPowerOpInfo
$spec.powerOpInfo.defaultPowerOffType = "soft"
$spec.powerOpInfo.defaultSuspendType = "hard"
$spec.powerOpInfo.defaultResetType = "soft"
$spec.powerOpInfo.standbyAction = "powerOnSuspend"

$_this = Get-View -Id 'VirtualMachine-vm-1074'
$_this.ReconfigVM_Task($spec)

And a finished script, which will run it against all machines in a specified blue folder comment/uncomment one of the $specVM.powerOpInfo.standbyAction lines to choose which option you want.

$objVMs = Get-Folder "Folder Name" | Get-VM
ForEach ($objVM in $objVMs){
	$specVM = New-Object VMware.Vim.VirtualMachineConfigSpec
	$specVM.powerOpInfo = New-Object VMware.Vim.VirtualMachineDefaultPowerOpInfo
	$specVM.powerOpInfo.standbyAction = "checkpoint" 			# Put the guest OS into StandBy Mode and leave the Virtual Machine powered On
	#$specVM.powerOpInfo.standbyAction = "powerOnSuspend" 		# Suspend the Virtual Machine
	$viewVM = Get-View -Id $objVM.Id
	$viewVM.ReconfigVM_Task($specVM)
}

I was actually surprised at how easy this was; and I think it’s going to make me a bit more adventurous with what I attempt to do via the PowerCLI.

Alpha build of Project Onyx

Carter Shanklin has announced that VMware have released an Alpha build of the long-anticipated Project Onyx. This is a script recorder for vSphere Client, which is designed to allow scripting of things which are awkward or difficult to achieve using the VMware PowerCLI APIs alone.

Downloading this at the moment, although I don’t suspect I’ll have time to look at it for a while.