In order to try make it easier to find the owner of a machine, we implemented a custom attribute “Infrastructure Consultant”, which the deployer should fill complete. Inevitably, despite the best of intentions, this is occasionally missed, and we end up with machine of unknown provenace.

The following script sorts this by finding machines where the custom attribute is empty, then populating it with a best guess, based on the machine’s event log.

It looks for three types of events:-

• “Deploying…” covers machines which have been deployed from another template
• “Creating…” covers machines which have been imported via VMware Converter
• “Clone of…” covers machines cloned from existing machines

This seems to cover everything on our environment. If you find something else, then it should be simple enough to add it.

# Name of the custom attribute which we are wanting to check/update
$strCAInfrastructureConsultant = "Infrastructure Consultant"

# Loop through all the machines
ForEach ($objVM in (Get-VM | Sort-Object Name)){
	Write-Host "Checking " -NoNewline
	Write-Host $objVM -ForegroundColor Blue
	# If the specified custom attribute is empty
	If ($objVM.CustomFields.Item($strCAInfrastructureConsultant) -eq ""){
		# Find the username of the person who created the machine. As this is returned in Domain\Username format, we split it, and take the second portion
		$strInfrastructureConsultant = ((@(($objVM | Get-ViEvent | Where-Object {$_.FullFormattedMessage -match "Deploying*" -or $_.FullFormattedMessage -match "Creating*" -or $_.FullFormattedMessage -match "Clone of*"} | Select-Object Username)))[0].Username).Split("\")[1]
		Write-Host "Adding " -NoNewline -ForegroundColor DarkGray
		Write-Host $strInfrastructureConsultant -NoNewline -ForegroundColor White
		Write-Host " as Infrastructure Consultant" -ForegroundColor DarkGray
		# Write that username to the custom attribute
		($objVM | Get-View).setCustomValue($strCAInfrastructureConsultant,$strInfrastructureConsultant)
	}
}

This doesn’t take a long time to run, and will hopefully catch all those occasions where we forget to complete the custom attributes on the machine. It could of course be easily modified to check/update any other custom attribute.

Eric Gray‘s done some great stuff with using VMware vCenter alarms with PowerShell to automatically migrate machines. And Carter Shanklin’s got a great description of a different method, but I was after something which didn’t involve making any modifications to the vCenter server. Not everyone here is familiar with the PowerShell /PowerCLI, and I didn’t want to put something in place which would require this knowledge to support (or more likely, require me to be dragged off whatever I’m doing to take a look!).

I decided to use a scheduled task, running from the reporting server (which already runs the scheduled tasks for the maintenance and reporting scripts). This would periodically check the datastores, and tweet if they exceeded a certain threshold. The team supporting the servers can easily set up their phones for Twitter alerts from the specific user

I spent some time trying to do this using pure PowerShell, using the techniques described by Mike Ormond and Joe Pruitt, but I could not get it to work via our proxy (rather annoyingly it worked intermittently). As a workaround, I ended up using Blue Onion‘s command-line twitter client TweetC. This stores the twitter username and password in an INI file, and can be called from within a PowerShell script.

# Load the PowerCLI
Add-PSSnapin VMware.VimAutomation.Core

# Your VI server
$strVIServer = "yourVIServer"

# Percentage free space at which you would like a message
$intThreshold = 10

# Connect to the VI server
Connect-VIServer $strVIServer

# Loop through the datastores
ForEach ($objDatastore in (Get-Datastore | Sort-Object FreeSpaceMB | Select-Object Name, @{Name="PercentageFreeSpace";Expression={[math]::round(($_.FreeSpaceMB / $_.CapacityMB * 100), 1)}})){
If ($objDatastore.PercentageFreeSpace -lt $intThreshold){
$objName = $objDatastore.Name
$objFreeSpace = $objDatastore.PercentageFreeSpace
E:\Tools\tweetC\tweetc.exe "$objName has only $objFreeSpace % remaining"
}
}

# Disconnect from the VI server
Disconnect-VIServer -Confirm:$false

As the screenshot below shows, this works a treat (I adjusted the thresholds for testing).

The Twitter account is protected (to save it from getting followed by porn-bots). I can now set it up for device alerts, which should hopefully give me a head’s up on any potential problems. I could also implement other alerts in this method. For example, you could ping a critical machine, and tweet if it’s not available:

If ((Get-WmiObject -Class Win32_PingStatus -Filter "Address='CriticalMachineHostname'").StatusCode -ne 0){
E:\Tools\tweetC\tweetc.exe "Critical machine not reposnding to pings!"
}

Ideally I’d like to revisit this, and remove the requirement for TweetC, but in the meantime it seems to be working great.

Online Training – Automating vSphere with the VIX API from Eric Sloof NTPRO.NL on Vimeo.

I can’t wait to try this out, although I think I’m going to have to do a little Visual Basic study first.

One of the things I’ve been doing might be more generally useful. We needed to convert around 700 of our machines to thin-provisioned format. When migrating to the new environment we’d stuck to traditional “thick” machines, as there was a lot of upheaval, and we didn’t have the necessary monitoring in place. Now that things have settled, we were looking to take advantage of the thin format to save us some space. We had already run SDELETE on the (non-persistent) machines during the migration, so we were ready to go.

The conversion process is fairly simple: a new option during a storage vMotion. However, the idea of doing this 700 times did not appeal to me, so I wrote the script below to automate the process.

# Convert Machines To Thin Provisioned
# Ben Neise 16/03/10

# Create an empty array
$arrMachinesToBeConverted = @()

Write-Host "Getting virtual machine objects" -ForegroundColor Blue
Write-Host ""

# Select the broad category of VMs we're looking at, for example all the machines in a specific blue folder
$objVMs = Get-Folder "Projects" | Get-VM | Sort-Object

Write-Host "Generating list of candidates" -ForegroundColor Blue
Write-Host ""

# Loop through all the virtual machines selected
ForEach ($objVM in $objVMs){

	Write-Host "Investigating " -NoNewline
	Write-Host $objVM -ForegroundColor Blue -NoNewline

	# Skip the rest of the loop if the machine is thin provisioned
	If ($objVM | Get-HardDisk | Where-Object {$_.StorageFormat -like "Thin"}){
		Write-Host " - disks are already thin provisioned" -ForegroundColor DarkGray
		continue
	}

	# Skip the rest of the loop is the machine is switched on and non-persistent
	If ($objVM | Where-Object {$_.PowerState -ne "PoweredOff"} | Get-HardDisk | Where-Object {$_.Persistence -notlike "IndependentPersistent"}){
		Write-Host " - switched on and non-persistent" -ForegroundColor DarkGray
		continue
	}

	# Skip the rest of the loop if the machine is powered-on, and has snapshots
	If ($objVM | Where-Object {$_.PowerState -ne "PoweredOff" -and ($objVM | Get-Snapshot)}) {
		Write-Host " - switched on and with snapshots" -ForegroundColor DarkGray
		continue
	}

	# Skips the rest of the loop if the machine has a shared drive and is not set up as fault tolerant (indicating that it's a Linked Clone)
	# Thanks to Keshav Attrey for this method - http://www.vmdev.info/?p=546)
	$viewVM = $objVM | Get-View -Property Name,Summary,Config.Hardware.Device
	$unshared = $viewVM.Summary.Storage.Unshared
	$committed = $viewVM.Summary.Storage.Committed
	$ftInfo = $viewVM.Summary.Config.FtInfo
	If (($unshared -ne $committed) -and (($ftInfo -eq $null) -or ($ftInfo.InstanceUuids.Length -le 1))){
		Write-Host "The machine is a linked clone" -ForegroundColor DarkGray
		continue
	}

	Write-Host "Added to the list of machines to be converted"
	$arrMachinesToBeConverted += $objVM
}

Write-Host "Starting Storage vMotions" -ForegroundColor Blue
Write-Host  $arrMachinesToBeConverted.Count -ForegroundColor Blue -NoNewline
Write-Host " machines to be converted" -ForegroundColor DarkGray
Write-Host ""

ForEach ($objVM in $arrMachinesToBeConverted | Sort-Object){

	# Get the biggest datastore
	$objBiggestDatastore = Get-Datastore | Sort-Object -Property FreeSpaceMB -Descending
	# Select the datastore from the top of the previously generated list (index 0) and remove the preceeding "Datastore-" from it's ID to give us the MOID
	$strTargetDatastore = ($objBiggestDatastore[0].Id).replace('Datastore-','')	

	# Let the user know what's going on
	Write-Host "Migrating machine " -NoNewline
	Write-Host $objVM -ForegroundColor Blue -NoNewline
	Write-Host ", to Thin Provisioned format on datastore " -NoNewline
	Write-Host $objBiggestDatastore[0] -ForegroundColor Blue -NoNewline
	Write-Host ""

	# Get the view of the VM we're moving
	$viewVM = Get-View -VIObject $objVM
	# Remove the preceding "HostSystem-" from the VM's Host's ID, giving us the Host's MOID
	$strTargetHost = ($objVM.host.id).replace('HostSystem-','')
	# Create a task specification for the relocation
	$specRelocate = New-Object VMware.Vim.VirtualMachineRelocateSpec
	# Add the target datastore to the specification using the MOID
	$specRelocate.datastore = New-Object VMware.Vim.ManagedObjectReference
	$specRelocate.datastore.type = "Datastore"
	$specRelocate.datastore.value = $strTargetDatastore
	# Add the host to the specification using the MOID
	$specRelocate.host = New-Object VMware.Vim.ManagedObjectReference
	$specRelocate.host.type = "HostSystem"
	$specRelocate.host.value = $strTargetHost
	# This is the specification property that makes the disk Thin Provisioned
	$specRelocate.transform = "sparse"
	# Create the task where the previously specified task is applied to the view of the target VM
	$task = $viewVM.RelocateVM_Task($specRelocate, $priority)
	# Start the task, and wait for it to complete before continuing
	Get-VIObjectByVIView $task | Wait-Task | Out-Null

}

It works through the collection of virtual machine objects in $objVMs, and adds them to an array if they pass certain criteria. They must be:-

• Not be currently thin provisioned
• Not switched on and non-persistent
• Not switched on with snapshots
• Not a linked clone (thanks to Keshav Attrey for his method of finding linked clones)

It then starts a separate loop going through each machine object in the array. I could have integrated the migration task into the first loop, but this approach means you can add a human “sanity-check” if you’ve got certain machines you don’t want to migrate.

The script finds the datastore with the largest amount of free space, and then the actual migration is done using code which I generated using Onyx. The free space on the datastores is re-evaluated before every move. This makes the script quite slow, but this isn’t something you want to rush.

“The trust relationship between this workstation and the primary domain failed.”

The problem is detailed in this KB Article. What happens is that every 30 days (by default) the client initiates a computer password change on the domain controller. This computer password is used to authenticate the computer as the computer object in AD, and is distinct from the user’s password. When the non-persistent machine resets, the passwords go out of synchronization and domain authentication fails.

This can be fixed, as per Microsoft’s KB article, by disabling the client-initiated computer password changes; this can be done using Local or Group Policy, by script, or by directly editing the registry.

Using local, or group policy

Set the key shown below to Disabled

Using REGEDIT

Set the below value to 1

Using Windows shell script

 :: Set registry key to disable computer password expiry

REG ADD "HKLM\SYSTEM\CurrentControlSet\Services\Netlogon\Parameters" /v DisablePasswordChange /t REG_DWORD /d 1 /f

If you’ve got access to the Domain Controller, you can also set a GPO so that Domain Controller: Refuse Machine Password Changes is Enabled. This is in Windows Settings à Securiy Settings à Local Policies à Security Options (the same location as the Domain Member: Disable Machine Account Password Changes).

Also, if you need to rejoin machines that have already fallen off the domain, you can miss the reboot after removing it from the domain, so:

1. Shut the machine down
2. Make the drives Persistent
3. Start the machine and log in
4. Remove the machine from the domain
5. Add the machine to the domain
6. Reboot
7. Shut-down and make Non-Peristent

Skipping the middle reboot saves a couple of minutes (which adds up if you have a lot to do). The above processes can also be scripted through the use of PowerCLI with Invoke-VMCommand and either NETDOM (for XP/Vista) or PowerShell for Windows 7.

At first, we manually changed these assignments using the Disk Management MMC before setting the drives to non-persistent. Lately, I’ve been using a DISKPART script, which speeds up the process, but still requires logging into each machine.

I saw Arnim van Lieshout’s post on Invoke-VMScript yesterday, and realised that this could be used to run the DISKPART script on multiple machines. In order to get the TXT file used by DISKPART onto the machine, I’m also using the Copy-VMGuestFile command found in PowerCLI 4.0 Update 1.

# Change drive letter assignment from D to Q

# Gets the specific VMs we're after as an object
$objVMs = Get-Folder "Sequencers" | Get-VM | Sort-Object Name

# Assign the command line required for DISKPART to a variable
$strScript = "IF EXIST D:\ DISKPART /S C:\DiskPart_Change_C_To_Q.txt"

# Loop through the VMs
ForEach ($objVM in $objVMs){
# Let the user know
Write-Host Copying file to $objVM
Copy-VMGuestFile -Source "C:\Tools\DiskPart_Change_C_To_Q.txt" -Destination "c:\" -LocalToGuest -VM $objVM -HostUser root -HostPassword password -GuestUser Administrator -GuestPassword password
Write-Host Changing disk partitions on $objVM
Invoke-VMScript $strScript -vm $objVM -HostUser root -HostPassword password -GuestUser Administrator -GuestPassword password -ScriptType "bat"
}</pre>
The TXT file contained the following:-
<pre>SELECT VOLUME 2
ASSIGN LETTER=Q

This requires root credentials for the host, and administrator rights on the target machine, but, as Arnim notes, will work in the absence of client network connectivity.

This is a bit of a hack,: leaving the TXT file behind on the C drive; using root credentials rather than an account with the least effective permissions; and having the script contain the credentials in plain-text. However, I’m enthusiastic about using this solution in a more structured way in future. I’m already thinking about using it to defragment and SDelete our thick-provisioned virtual machines before converting them to Thin provisioned disks.

Linked clones allow multiple machines to share a common read-only “base” VMDK file, with each machine generating their own delta (REDO). Under normal usage circumstances, the REDO would continue to grow throughout the life of the machine; however as our machines have non-persistent hard drives, they reset to a clean state when powered-down. This makes our environment ideally suited to taking advanatage of the functionality offered by Linked Clones. They can either be created manually (by moving and renaming files on the datastore), or via the APIs, you can get more information on them in this White Paper from VMware.

Our training machines are functionally identical to our production machines, and similarly consist of three types – Capture, Packaging and Verification. These are 11, 8, and 8 GB respectively. The usage patterns are slightly different, as – unlike “live” projects which have a steady stream of work, trainees tend to come in in large batches. This means that the training environment either needs to be continuously large, but mostly idle, or it needs to be regularly redeployed then stripped back.

The benefits achieved via the implementation of Linked Clones in this project resulted in roughly the same ratio of space saving as our proof-of-concepts, but as the number of machines involved was greater, the differences are more pronounced. Also this is the first time we’ve exceeded 8 machines sharing the same VMDK, which is a notable milestone as it is only possible if we limit the number of possible hosts that the machines can run on (there is a VMFS limitation of 8 hosts accessing a VMDK concurrently). As we have DRS enabled, this meant reducing the number of hosts in each cluster to 8 or less.

We deployed twenty-five machines each, of  the three different builds used in a project. All were Windows XP virtual machines

The three machines being used as the parents had their slack space on the drives was cleaned using SDelete, then the machine was converted to Thin Provisioned using Storage vMotion. It was switched off, and a snapshot was created. This snapshot will form the base for the parent’s clones.

The machines were deployed using the a script similar to the one at the bottom of this post, and it took just over an hour to deploy and customize all 75 machines. This was considerably faster than the time it would have taken to deploy 75 machines using the normal “Deploy from template”  method.

Here are the data:-

• Estimated space used if deployed traditionally: 675 GB
  • Capture: 11 GB per machine
  • Packaging: 8 GB per machine
  • Verification: 8 GB per machine

• Estimated Space if Deployed as Thin Provisioned, but not Linked Clones: 238.75 GB
  • Capture: 3.34 GB per machine
  • Packaging: 3.72 GB per machine
  • Verification: 2.49 GB per machine

• Space used in current configuration: 19.3 GB
  • Capture 3.34 GB for parent, plus 0.13 GB per Linked Clone
  • Packaging 3.72 GB for parent, plus 0.13 GB per Linked Clone
  • Verification 2.49 GB for parent, plus 0.13 GB per Linked Clone

And in graph-format, for extra impact:-

All size estimates are based on the machines in a powered-down state. When powered on, a swap file (equal to the size of the assigned RAM) is created, and (assuming the machines are non-persistent) REDO files are created on all types of machines.

I’ve been on a few of the machines and they don’t appear to suffer from any noticeable performance degradation, although the true test won’t come until we get considerable concurrent use.

I’m tentatively declaring this a huge success. Rather than the training environment using 240 GB between training engagaments, it’s now down to a svelte 20GB, with no reduction in functionality.

Below is a script similar to the one I used to deploy the linked clones. The actual “meat”, which deploys the machines was based on Hal Rottenberg’s New-LinkedClone.ps1 script. As far as possible, I’ve tried to strip out stuff that’s specific to our environment (we use the Custom Attributes as an asset management database and to track which machines were deployed from which templates). There’s probably going to be stuff in there that doesn’t make much sense, but if you’ve got a bit of an understanding of PowerShell, you should be able to cut and keep the bits you want.

# Script to deploy linked clones

# List of custom attributes which you're wanting to copy from the template or parent to the newly created machine
# (Machines deployed from templates no longer inherit CAs in vSphere 4.0)
# These help us track provenance, and provide information to the user
$arrStrAttributesToCopy = @(
"AD Object Location",
"Customisation",
"Infrastructure Consultant",
"Logon Administrator Name",
"Logon Administrator Password",
"Logon User Name",
"Logon User Password",
"Mobilisation Consultant",
"Project",
"Role",
)

# Name of the Custom Attribute on the parent which contains the name of the customisation to use
$CustomFieldName = "Customisation"

Function DeployLinkedClone ($strSourceVM, $intToBeDeployed, $intStartDeployingAtNumber, $CustomFieldName){
# Bases the name of the machine on the second part of the string split by spaces. This assumes that the template follows the standard naming convention of "Tmpl [Name] x.x"
$strMachinePrefix = ($strSourceVM.split(' ')[1])
$objVM = Get-VM $strSourceVM
$viewVM = $objVM | Get-View
$objCustomization = Get-OSCustomizationSpec ($objVM.CustomFields.Item($CustomFieldName))
# Ensure that the machines does not have a non persistent HD
If ($objVM | Get-HardDisk | Where-Object {$_.Persistence -like "IndependentNonPersistent"}){
Write-Host $objTemplate has a non-persistent HD!
}
# If the customisation, as specified in the parent's custom attribute does not exist, then quit.
If (!$objCustomization){
Write-Host Customisation ($objVM.CustomFields.Item($CustomFieldName)) not found. Exiting.
Break
}
$i = 1
Do {
# Convert the single digit integer (i.e., "1") into a double digit (i.e., "01")
$strMachineNumber = ("{0:0#}" -f $intStartDeployingAtNumber)
# Concatenate the machine name prefix (from the template name) with the double-digit integer, which is incrememted on each loop
$strMachineBeingDeployed = $strMachinePrefix+$strMachineNumber
# Check that the machine doesn't already exist
If ((Get-VM -Name $strMachineBeingDeployed -ErrorAction SilentlyContinue)){
Write-Host "Machine $strMachineBeingDeployed already exists!"
Break
}
# Let the user know what's going on
Write-Host ""
Write-Host "Deploying new linked-clone " -NoNewline
Write-Host $strMachineBeingDeployed -ForegroundColor Blue -NoNewline
Write-Host ", from template " -NoNewline
Write-Host $strSourceVM -ForegroundColor Blue -NoNewline
Write-Host ", using customisation " -NoNewline
Write-Host $objCustomization -ForegroundColor Blue -NoNewline
Write-Host ", on the same Host as the parent" -NoNewline
Write-Host ""
# Create the new machine using all these variables
$objFolder = $viewVM.parent
$specClone = New-Object Vmware.Vim.VirtualMachineCloneSpec
# Get the most recent snapshot attached to the machine
$specClone.Snapshot = $viewVM.Snapshot.CurrentSnapshot
# Create an object to represent the location of the clone
$specClone.Location = New-Object Vmware.Vim.VirtualMachineRelocateSpec
# This is the move-type that specifies the new disk backing (which is the bit that makes a linked clone)
$specClone.Location.DiskMoveType = "createNewChildDiskBacking"
# Run the task with the specified parameters
$task = $viewVM.CloneVM_Task($objFolder, $strMachineBeingDeployed, $specClone)
Get-VIObjectByVIView $task | Wait-Task | Out-Null
# Get the object for the machine which was just deployed
$objTargetVM = Get-VM $strMachineBeingDeployed
# Apply the customisation specification to the newly created clone
Set-VM -VM $objTargetVM -OSCustomizationSpec $objCustomization -Confirm:$false
# Start the clone
Start-VM -VM $objTargetVM
# Get the view (needed for writing custom attributes)
$viewTarget = $objTargetVM | Get-View
# Loop through each of the custom attributes which are to be copied
ForEach ($arrStrAttributeToCopy in $arrStrAttributesToCopy){
# Read the attribute from the source template
$objAttribute = $objVM.CustomFields.Item($arrStrAttributeToCopy)
# Apply the attribute to the machine object
$viewTarget.setCustomValue($arrStrAttributeToCopy,$objAttribute)
}
# Set the "Template" custom attribute to the parent templates
$arrStrAttributeToCopy = "Template"
$viewTarget.setCustomValue($arrStrAttributeToCopy,$strSourceTemplate)
# Increment the number used for naming the machines
$intStartDeployingAtNumber ++
# Increment the number used to count the number of machines deployed
$i ++
}
# Continue to loop while the number of machines deployed is less than the number required
While ($i -le $intToBeDeployed)
}

# Get the current time (for timing how long the script took to run)
$dteStart = Get-Date

# Name of source VM, should be persistent, should have a snapshot and the customisation specified in the nominated custom attribute
$strSourceVM = "Tmpl Capture 1.0"
# Number to be deployed
$intToBeDeployed = 25
# Number to start deploying from
$intStartDeployingAtNumber = 1
DeployLinkedClone $strSourceVM $intToBeDeployed $intStartDeployingAtNumber $CustomFieldName

# Name of source VM, should be persistent, should have a snapshot and the customisation specified in the nominated custom attribute
$strSourceVM = "Tmpl Packaging 1.0"
# Number to be deployed
$intToBeDeployed = 25
# Number to start deploying from
$intStartDeployingAtNumber = 1
DeployLinkedClone $strSourceVM $intToBeDeployed $intStartDeployingAtNumber $CustomFieldName

# Name of source VM, should be persistent, should have a snapshot and the customisation specified in the nominated custom attribute
$strSourceVM = "Tmpl Verification 1.0"
# Number to be deployed
$intToBeDeployed = 25
# Number to start deploying from
$intStartDeployingAtNumber = 1
DeployLinkedClone $strSourceVM $intToBeDeployed $intStartDeployingAtNumber $CustomFieldName

$dteEnd = Get-Date
$dteDiff = New-TimeSpan $dteStart $dteEnd
$timeTaken = [math]::round($dteDiff.totalMinutes, 2)
Write-Host ""
Write-Host "It took" $timeTaken "minutes for these machines to deploy"

# End of script

I passed the VCP on vSphere 4 just before Christmas with 463/500. At the time the deadline for taking the exam without having to the attend the What’s New was 31st December 2009, but that’s now been extended to the 31st January. It was nice to have it out of the way before Christmas though.

The exam seemed no more difficult than the VCP on VI 3; I think they even reused a few of the questions. I found the following useful:-

• Simon Long’s VCP vSphere 4 Practice Exams
• vSphere 4 Configuration Maximums
• Barry Coombs’ VMware vSphere cue-cards

Those sites also contain numerous links to other resources, so I’m sure you’ll find something which will suit your revision style.

Next on my to-do list is the Microsoft 70-431 Microsoft SQL Server 2005 – Implementation and Maintenance.

While it’s fairly specific to our use-case, I thought I’d share the details in case anyone else runs into this.

First of all, this bug will only affect you if the following conditions are met:

• You are using VMware vSphere 4.0 (or 4.0 Update 1)
• Guest OS power-saving settings cause the virtual machine to enter standby
• One or more of the guest’s hard drives are set to “independent non-persistent”
• DRS is enabled on the virtual machine’s cluster

The machine enters standby as normal. The issue arises when you try to power the virtual machine back on: if DRS has allocated the machine to another host based on load the machine will not resume, and gives an error similar to the following:-

“Virtual Machine is configured to use a device that prevents the operation: Device ‘Hard disk 1′ is disk which is not in persistent mode. Device ‘Hard disk 1′ which is not in persistent mode”.

You cannot manually migrate the machine (even back to the original host). You cannot change the power-state on the machine, edit the virtual machine settings, or delete the machine.

If this has happened to you, the only way we’ve found to get the machine back up-and-running seems to be to remove the machine from inventory, then create a new virtual machine with the same specifications, and add the old machine’s VMDK.

Fortunately, there are a couple of workarounds. You can either disable power-saving settings in the guest OS or change the guest power management settings from “Suspend the virtual machine” to “Put the guest OS into standby mode and leave the virtual machine powered on” (you can automate this as described in my previous post).

Changing the guest power-management settings means that when the guest enters standby, although vSphere shows the machine as “powered-on”, VMware Tools is not running, which can cause problems (i.e., when trying to gracefully shut down a batch of machines).

This was also my first time working with the VMware vSphere support and I was impressed. They quickly replicated the problem and confirmed that it was indeed a bug. As most people nowadays tend to use snapshots rather than non-persistent drives, and few users virtualise desktop operating systems (which are more likely to have power-saving settings on by default) I can understand why this particular set of circumstances went untested.

To do this the machines need to be powered down. We had an imminent maintenance window, but it wouldn’t allow us the time to make this change manually (even if we wanted to), this necessitated some automation. Unfortunately I had no idea how to go about editing this setting using the PowerCLI, even after a little search through the VMware PowerCLI community.

This seemed like the perfect opportunity to try out Project Onyx.

Carter Shanklin’s video does a good job of explaining how to Onyx up and running, and it worked exactly as described (even on my Windows 7 machine).

1. Download the Onyx files and extract to a folder
2. Run the Onyx executable



3. Click the Connect button, and connect to your VirtualCenter server.
4. Once that’s launched, start vSphere client, but instead of connecting to your VirtualCenter server, connect to http://localhost:1545 (Carter actually says 1445 in the video, but you can see on screen that he’s using 1545). Use your normal credentials.
5. Ignore the warning about unencrypted traffic (as Carter explains, the unencrypted traffic is local-only, the network traffic is still encrypted)
6. Click the Start button on Onyx
7. In vSphere client make whatever changes it is that you’re wanting to record.
8. Click the Pause button on Onyx, and you’ll see in the window a script has been created.
9. Copy this into your favourite PowerShell editor, and modify until it’s suitable for your purposes.

The original capture from the Onyx Window

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeVersion = "2009-11-27T09:16:04.570821Z"
$spec.powerOpInfo = New-Object VMware.Vim.VirtualMachineDefaultPowerOpInfo
$spec.powerOpInfo.defaultPowerOffType = "soft"
$spec.powerOpInfo.defaultSuspendType = "hard"
$spec.powerOpInfo.defaultResetType = "soft"
$spec.powerOpInfo.standbyAction = "checkpoint"

$_this = Get-View -Id 'VirtualMachine-vm-1074'
$_this.ReconfigVM_Task($spec)

A second capture changing the setting back to isolate the exact line that makes the changes

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.changeVersion = "2009-11-27T09:16:33.872017Z"
$spec.powerOpInfo = New-Object VMware.Vim.VirtualMachineDefaultPowerOpInfo
$spec.powerOpInfo.defaultPowerOffType = "soft"
$spec.powerOpInfo.defaultSuspendType = "hard"
$spec.powerOpInfo.defaultResetType = "soft"
$spec.powerOpInfo.standbyAction = "powerOnSuspend"

$_this = Get-View -Id 'VirtualMachine-vm-1074'
$_this.ReconfigVM_Task($spec)

And a finished script, which will run it against all machines in a specified blue folder comment/uncomment one of the $specVM.powerOpInfo.standbyAction lines to choose which option you want.

$objVMs = Get-Folder "Folder Name" | Get-VM
ForEach ($objVM in $objVMs){
	$specVM = New-Object VMware.Vim.VirtualMachineConfigSpec
	$specVM.powerOpInfo = New-Object VMware.Vim.VirtualMachineDefaultPowerOpInfo
	$specVM.powerOpInfo.standbyAction = "checkpoint" 			# Put the guest OS into StandBy Mode and leave the Virtual Machine powered On
	#$specVM.powerOpInfo.standbyAction = "powerOnSuspend" 		# Suspend the Virtual Machine
	$viewVM = Get-View -Id $objVM.Id
	$viewVM.ReconfigVM_Task($specVM)
}

I was actually surprised at how easy this was; and I think it’s going to make me a bit more adventurous with what I attempt to do via the PowerCLI.