Hyper-V Manager cannot connect–make sure RPC service is running

My Hyper-V Server 2008R2 lab environment is so stable I rarely fire up the Hyper-V MMC snap-in. The last time I opened it was four months ago when I switched from booting it from a USB stick to booting it from an IDE drive. I opened it today and it gave me the typical “Cannot connect to the RPC service on computer 'Servername'. Make sure your RPC service is running.” error. My quick troubleshooting steps follow:

1. I double checked to make sure the IP address and host name were still specified in the \etc\hosts file. It was.
2. I attempted to remote into the Hyper-V server directly using Remote Desktop and was met with the “Your password has expired” message. I reset the password, and voila – the Hyper-V manager MMC connected to my server and displayed the current state of my VM’s.

Hyper-V Lab Experiment; Attack of the Snapshots

This is a brief chronicle of the effect of re-creating a Virtual Machine from a Virtual Hard Drive that had Snapshots taken and not merged back into the parent image.

Background

In August 2010, I created a Hyper-V lab with a Windows 7 development VM and a Server 2008R2 testing VM. The VM hard disks, virtual machines, and snapshots were stored on the SATA hard disk. The Hyper-V Server 2008R2 host was installed on a bootable USB pen drive.

Apparently around November 2010 and over the course of 2011 I created a few snapshots of the Windows 7 VM.

Fast forward to a couple weeks ago

My Hyper-V environment lost power and when it restarted the BIOS failed to recognize the Kingston USB pen drive that Hyper-V was booting from. Result: My lab was down. This happened one other time, but after some fiddling with the BIOS, I  was able to get the USB drive to be recognized.

In order to prevent this from happening in the future, I replaced the USB pen drive by installing Hyper-V Server 2008R2 on an IDE drive, plugged it into my lab box and booted. I created a new Virtual Machine using the existing Window 7 Virtual Hard Drive and powered up my development VM. Upon a quick glance everything looked fine.

Missing Files and Programs

A few days later while logged into my Dev VM, I noticed several of my files and directories were missing, Microsoft Office wasn’t installed, and SQL Server said the trial had expired. What?!?

A quick check of the Windows Event log and there was an entry that Windows was not shut down properly on November 11, 2010, but was back up and running on Feb 12, 2012. Suddenly I was missing over a year of development activity; not really. 

Don’t Panic!

November 11, 2010 was the date the first snapshot was taken. All the activity since then was in the snapshot differencing disks (3 files with a .AVHD extension).

I had a Windows Image Backup and regular full backups, but instead of restoring from backup, I decided to have a little fun with the Hyper-V management console instead.

A little research and I found this article about merging differencing disks. It isn’t the best reference you can find because it doesn’t have screenshots of the process like this one. I actually backed up the parent VHD and child AVHD files and renamed the AVHD files VHD, but it appears the renaming process is unnecessary. Otherwise after using the Hyper-V management console to merge each of the AVHD files to their parent drives, I powered up the dev VM after merging and everything was back to the state prior to the initial power outage. Success!

Moral of the Story

If you use a Hyper-V snapshot to test a Windows Update or some other configuration change that you want to be able to roll back out of, make sure to merge the snapshot back into the VM. Don’t leave it in a snapshotted state. Especially not with multiple active snapshots.

Reference: Hyper-V: Avoid using differencing disks on virtual machines that run server workloads in a production environment.

photo credit: pnoeric / CC BY-SA 2.0

Physical to Virtual using SysInternals Disk2VHD

The hard drive controller was regularly failing on my source control server. What to do? The server was due for an upgrade soon as its hardware was more or less obsolete and the performance was less than satisfactory. While the controller was still mostly working, I used SysInternals Disk2VHD to create a virtual copy of the physical server. In the interim while waiting on the new Hyper-V server to permanently place it on, I can temporarily run the server in VirtualPC 2007 and with three times more RAM than was available on the physical server.

The process was simple, fast, and seamless. I just had to re-activate the OS after the conversion. Just make sure you activate it before the expiration date which I believe was only 3 days on my copy or it will brick your ability to manage the server from the terminal.

While waiting for copying and converting processes to finish I ran across a few related posts that I thought I might want to reference in the future.

• Are VHD's Compatible between Hyper-V and Virtual Server and Virtual PC
• Vittorio Pavesi's post to Convert VHD from Hyper-V to Virtual PC which describes how to overcome the compatibility limitation that prevents using a Hyper-V created VHD in Virtual PC.

Iomega StorCenter ix2 Network Attached Storage for Hyper-V backup

Running a virtualized environment like Hyper-V has some limitations that you don’t have with a physical server environment. Physical USB ports and DVD drives are shared by multiple virtual servers and they can’t all access them at the same time. Currently Hyper-V doesn’t support using external USB drives. If that is your typical backup and restore strategy, as is often the case for small businesses, you need another option like a Network Attached Storage (NAS) unit.

In my lab environment I selected the Iomega StorCenter ix2-200 1 TB Network Attached Storage. After a lot of research and pouring over user submitted reviews, I found that this cost effective unit had a proportionately smaller number of bad reviews compared to the other units I researched.

Noteable features that appealed to me were

• RAID 1 with automatic RAID rebuild (two 500Gb SATA drives mirrored for redundancy)
• User replaceable hard drives (I was going to easily outgrow the 500Gb available storage space)
• One touch backup of the backup to a connected external USB drive
• Ability to connect additional external USB drives for network accessible storage

The not so good

The web interface stopped working once with a “device not available” error or simply a HTTP server error web page after an extremely long browser hang. The drive itself continued to store and serve my backup images and shared files, but not being able to access the web interface got me wondering what else wasn’t functioning (like the RAID controller possibly?).

The web interface is the easiest way to check the amount of free space and perform firmware upgrades. The Iomega support website provided no useful help at all. With some research I found that other people were having similar issues with the ix4 model and a reboot of the NAS unit restored the web interface. Unfortunately for them, they were using the device to store their running virtual hard drives since the device is VMware certified. They had to first power down all the virtual servers (VM’s) before power cycling the NAS and power each of the VM’s back up - which is a serious inconvenience for the administrator and any users of the virtual servers.

A simple power cycle corrected my issue, but hopefully it will not be a regular occurrence as I don’t have physical access to my lab at all times.

Managing Hyper-V Server without the MMC

Why might someone need or want to manage Hyper-V without the Machine Management Console snap-in? Maybe you want to use XP, Server 2003, or a Vista/ Windows 7 Home Premium or lower edition which is not supported by the Hyper-V MMC. This post is most relevant to a Hyper-V™ Server 2008 R2 installation, not Server 2008 R2 with the Hyper-V role.

Four options* for managing the Hyper-V environment:

• PowerGui with the Hyper-V powerpack (Vista, Winows 7, Server 2008 clients)
• PowerShell Management Library for Hyper-V
• Microsoft System Center Virtual Machine Manager (SC VMM) which can be used to manage both Hyper-V and VMware environments
• Hyper-V Management Console by utharam (the evaluation version threw a config error for me when I tried to run it)

* I will leave it as a homework assignment for you to determine which option may work best for you – if any.
photo credit: homard.net / CC BY 2.0

Hyper-V Poster

Slightly off topic, but Microsoft released a complex yet easy to read and informative Server 2008 R2 Hyper-V component architecture poster. A small snapshot of the poster is represented in the image below.

Why I Almost Switched from Hyper-V to VMware vSphere

I gave Hyper-V the good ol’ college try. I had a Windows 7 instance and a Windows Server 2008R2 Web edition instance installed. A few of the limitations that I didn’t like about Hyper-V:

• Management Console only supported on Windows 7 Ultimate or Vista Ultimate (not Home Premium or XP – pretty limiting for a home virtual environment)
• I couldn’t allocate any more RAM in total than was physically present on the Hypervisor – even if the instances were off. That doesn’t make it very easy to spin up a development or testing instance for quick testing and spin down
• The hypervisor management console locked up causing me to hard boot both server instances and the hypervisor to get it running again
• If I remoted or logged directly into the hypervisor (Server 2008R2 core install) itself and accidentally closed the command line window and cscript window, I couldn’t find a way to open them again without shutting down the server instances and power cycling the hypervisor. Yikes! that isn’t a very good production option.

The “Almost” in the title is because VMware vSphere and also Citrix XenServer both had compatibility problems with my hardware. Hyper-V worked right out of the box.

Hyper-V Could not initialize memory: there is not enough space on the Disk

I received this error when I tried to increase the memory of the single freshly installed VM (Server 2008R2) on my Hyper-V 2008R2 host from 1024Mb to 2048Mb. This is a new clean 4Gb machine with almost a terabyte of free disk space so I was surprised by the error. The issue comes from how memory is allocated if the VM state is saved and how I installed Hyper-V on a bootable 8Gb USB drive. I found the clue to the answer in Robert Larson’s article on Hyper-V File Storage and Permissions

Background

In my setup Hyper-V itself is running on the USB drive. The USB drive simply contains a .vhd file that holds the bootable Hyper-V image. The image itself takes up about 6.5Gb which leaves 1.5Gb free.

The Problem

When a VM is initialized it writes a .bin file the size of the amount of RAM allocated to the VM. The default path to that is on the Hyper-V host drive, in my case the USB drive. I had already pointed the Virtual Hard Disk and Snapshot paths to the 1Tb SATA drive because I knew they wouldn’t fit on the USB drive (duh), but I didn’t realize that the location of the VM configuration would need space for all the VM memory dumps.

The Fix

Change the Virtual Machine path to use the 1Tb SATA drive also.

Hyper-V on a USB drive

In preparation for setting up a virtualized development environment on some new iron, I ran across this article series on Hyper-V virtualization in a flash – part 1, and part 2. I was planning on installing the Hyper-V on a separate hard drive, but I think I’ll try this first.

Sadly, my virtualization needs have outgrown my previous process of using client based virtualization like Sun VirtualBox and Microsoft Virtual PC.