Hyper-V MAC conflicts

So here's an interesting one I came across recently;

When you run (in this case) HP Teaming on your NICs with Hyper-V on Server 2008 R2 you will get the following error... a lot.

Port {Teaming NIC GUID} was prevented from using MAC address {MAC of a VM} because it is pinned to port {GUID of VMs NIC}.

The real symptom is erratic pings and/or connectivity to your VMs. It will show mostly as 1 or 2 dropped packets every now and then.

Reason is, after 2008 R2 the networking in Hyper-V was secured to prevent MAC spoofing (A huge vulnerability up to this point). Problem is, the HP Teaming NIC want to effectively spoof all the MACs behind it, so it can control the load balancing etc... tsk tsk, what to do, what to do...

Solution is simple, thankfully there's a real easy tick box to turn that shiz off;

(This is from within System Centre Virtual Machine Manager, but same setting is there in Hyper-V Manager)

Hopefully this saves someone some pain.

EDIT: Please post your mileage on this one if you do come across it. Not 100% sure that it's the final answer in my particular problem. I may be looking at deciding NIC teaming (With HP at least) is not workable.

As it turns out this didn't solve our particular problem at all.

In our case the error being reported around the MAC addresses jump between ports was symptomatic of a loop in the network (Split Horizon)

Hyper-V switching seems to be very sensitive to this and there was no other evidence of this on the network, however when the offending device (Cisco Airport) was removed the problem vanished immediately.

A great success! So if you see this MAC address changing very often then check your network for loops (via segment isolation)

Hyper-V NIC Teaming

In a previous post I wrote about my experiences with Hyper-V on Server 2008 R2, and that teaming didn't work.

Well the good news is it works. The bad news is it feels like a house of cards.

Working with HP DL380 G5 and the very latest Proliant Support Pack (PSP 7.6) I managed to setup Hyper-V (on 2008 R2 Core) for Live Migration on teamed NICs. The storage is fibre attached and I would imagine that iSCSI teams are still not possible, but if anyone manages to do this let me know.

Anyway, onto the details.

For an exisiting Hyper-V server:

• Do one host at a time Live Migrating everything off it before starting
• Delete your Virtual Networks through Hyper-V Manager
• Install the PSP, downloading and updating NIC firmware as required
• Reboot
• Download the HP Network Configuration Utility (NCU) for 2008 R2 and install
• Reboot
• Create the team using the util C:\Program Files\HP\NCU\hpteam.cpl
• Reboot
• Set the IP info for your net interface (sconfig)
• Restart Hyper-V and Cluster services
• Recreate your Virtual Networks in Hyper-V manager.

Note: The last step is that hardest and takes ages to process so BE PATIENT.

also if it does fail then you have to break your team, reboot, recreate your team and try again.

I'd recommend turning off Windows Updates and avoid doing them for the moment. There is talk of windows updates breaking the teaming. Your own testing will be required on this one.

EDIT; I found a hiccup with teaming and MAC address. Have a read just so you are aware; http://anicegameof.blogspot.com/2010/04/hyper-v-mac-conflicts.html

Hyper-V Live Migration

Well Microsoft Hyper-V has been around for a little while now and has been playing the catchup game. However the release of Server 2008 R2 sees the introduction of features previously only available from other vendors.

One such feature is Live Migration, the ability to move a Virtual Machine from one Host to another without turning the guest off. Coupled with a case study deployment of Direct Access I got the opportunity to complete an implementation of Live Migration with great success.

Things I have learnt;

• There's a lot of useless information around for Hyper-V - I'll try not to add to it :)
• Hyper-V and Hyper-V R2 are quite different, dont expect them to behave the same.
• DO NOT team network cards - As of today (04/09/09) teaming is not supported, and when it is, it will be up to the vendor to provide and support any teaming of NICs.
  Update: This has changed. Check this post for details.
• You'll need more NICs - Hyper-V loves network cards. 2 teamed for production (when it actually works), 1 for heart beat, 1 for live migration traffic and 1+ for iSCSI, if you use it.
• You'll have to rethink your SAN - Cluster Shared Storage require a witness disk for the quorim data, and you can't store anything else on it. So plan to setup a new vDisk from your SAN at around 250 -> 500MB.
• You'll want more resources - One of the really appealing aspects to Live Migration is the ability to setup an N+1 High{ish} Availability Cluster, but that means the nearly 90% utalization you are currently running your poor, innocent hosts at, isn't going to do. Plan / Budget for it.

Server 2008 core

It is a huge pain. I do like working on core though, it feels (excuse the pun) hard-core, but it does make managing the hosts hard. Unless you are going to implement Virtual Machine Manager (which I do recommend btw) you aren't going to be able to do all the thing you want to within core. Troubleshooting is also made substantially harder.

The big appeal is of course the smaller footprint Core has - though dont expect some magic performance boost by just going to core.

That being said, I'd implement Hyper-V on 2008 Core whenever I can - it just seems appropriate.

As for your guests - You'll need to re setup their networking, moving to R2 will install new networking hardware with nice fresh DHCP settings [maybe an oversight from Microsoft but no biggie]. Just remember to record them before hard.

Micosoft Cluster Shared Storage

I've kept this seperate as it is required by Hyper-V Live Migration but IS NOT part of Hyper-V.

This means you should consider it seperately and carefully.

You'll need a static IP and hostname for your Cluster when you create it, you'll need to configure the Witness disk mentioned earlier and you'll need to seperate the heart beat, production and Live Migration network all setup in Cluster Manager.

TIP: Moving an already NTFS formatted disk to Cluster Shared Storage does not destroy the data. Be sure you have backups though.

Installation Order

Do things one at a time and test them carefully and you wont go wrong. - Measure twice, cut once and all that.

1. SAN - Create the Witness Disk, Virtual Machine storage, LUNs etc
2. Server 2008 R2 - Install the base OS, if at all possible stick with the Windows Drivers (I'd only use vendor drivers after careful testing) *See below for Core
3. Networking - Assign your IPs, Name your NICs, test connectivity between all hosts and SAN [iSCSI]
4. Storage - Connect to the SAN, check all the drives appear and make sure they have the same drive letters on all hosts.
5. Clustering - Install MPIO and Clustering. Create your cluster, give it its IP and hostname, check your hosts and networking appears corrently. Configure your witness disk.
6. Hyper-V - Install Hyper-V, configure all networking, dont create Guests!
7. Live Migration - Enable Cluster Shared Storage and add your disks (No, not the witness disk)
8. Add/create your Virtual Machine using Cluster Manager.
9. Check their configuration using Hyper-V manager.
10. Start your Guests using Hyper-V manager.
11. Migrate your Guests using Cluster Manager.

Well thats all for now, have fun with Virtualization. It's powerful technology but there's a lot of room to break things, so please be careful.

Footnote about Server 2008 Core:

There's a lot of good info online about the basic setup of core as well as a few mini GUI tools to download. Check out http://www.petri.co.il/configuring-windows-server-2008-networking-settings.htm

One thing I'd recommend is get remote management working and do as much as you can from your Windows 7 management machine.

Update: As for this past weekend (12/10/09) I've managed to get NIC teaming working with Hyper-V. Have a read.