Wednesday, April 21, 2010

Hyper-V MAC conflicts

So here's an interesting one I came across recently;

When you run (in this case) HP Teaming on your NICs with Hyper-V on Server 2008 R2 you will get the following error... a lot.

Port {Teaming NIC GUID} was prevented from using MAC address {MAC of a VM} because it is pinned to port {GUID of VMs NIC}.

The real symptom is erratic pings and/or connectivity to your VMs. It will show mostly as 1 or 2 dropped packets every now and then.

Reason is, after 2008 R2 the networking in Hyper-V was secured to prevent MAC spoofing (A huge vulnerability up to this point). Problem is, the HP Teaming NIC want to effectively spoof all the MACs behind it, so it can control the load balancing etc... tsk tsk, what to do, what to do...

Solution is simple, thankfully there's a real easy tick box to turn that shiz off;
(This is from within System Centre Virtual Machine Manager, but same setting is there in Hyper-V Manager)



















Hopefully this saves someone some pain.

EDIT: Please post your mileage on this one if you do come across it. Not 100% sure that it's the final answer in my particular problem. I may be looking at deciding NIC teaming (With HP at least) is not workable.

As it turns out this didn't solve our particular problem at all.
In our case the error being reported around the MAC addresses jump between ports was symptomatic of a loop in the network (Split Horizon)
Hyper-V switching seems to be very sensitive to this and there was no other evidence of this on the network, however when the offending device (Cisco Airport) was removed the problem vanished immediately.

A great success! So if you see this MAC address changing very often then check your network for loops (via segment isolation)


3 comments:

  1. We ran into this very issue trying to present a virtual NIC for HP's Virtual SAN Appliance software through a NIC team, and because of your post we looked through our connectivity to the virtual NIC and saw it was dropping packets.

    Once we saw the dropped packets and understood this was because of the teaming configuration, we played with the NIC team settings and changed it from Switched Assisted Load Balancing to another settting that worked in our enviornment.

    I'm not sure we would have looked at the connectivity to the virtual NICs w/o your post so thank you VERY much, and now we stopped getting the warnings and performance has increased.

    ReplyDelete
  2. This didn't help us. We don't have Spoofing enabled on the VMs, and the System Event log is literally full of these events, there are no other system events. We're not getting dropped packets either. I'm using Broadcom teaming with BACS 3 using Load Balancing.

    ReplyDelete
    Replies
    1. Im having the same issue... "Port 'A Team GUID' was prevented from using MAC address 'VM MAC Address' because it is pinned to port 'Virtual Switch GUID'. Im using Broadcom 5709c NIC's configured for SLB with failover, 2 node cluster with VMQ enabled, the error only appears on 1 team of one node which is configured slightly differently (no choice unfortunately).

      I believe this warning is directly related to the Broadcom "HyperVMode" registry key used in conjunction with VMQ on SLB teams, as this registry key is required to show VM Macs in the switch arp tables, by advertising them from the physical nic i.e. spoofing and also get VMQ's to be advertised to the host. Enabling VMQ on the VM seems to extend the problem, as many people say disabling the network performance optimizations through VMM or setting VMQOffloadWeight to 0 (same thing as disbling VMQ) resolves the issue totally.

      The difference between my two nodes is the working one uses an LACP team.

      Equaly, I think allowing MAC Spoofing will fix the problem on broadcom NIC's, but I havent had time to test it yet =)

      I will post more if I get time!

      Delete