Friday, March 12, 2010

XenServer 5.5u2 and Ghost IPs

Came across a weird issue with Citrix XenServer 5.5 update 2 this week that I figured I'd share. While the circumstances that bring this bug to life may be rare, it can cause a number of problems so I figured I'd post some details and how I fixed it.

In our situation we were testing a number of Network related changes using some Fiber over Ethernet cards (10-40gb network anyone?). Sadly I apparently can not mention the name right now no matter how sweet they are because we are only doing a proof of concept with the company.

During this testing we were moving back and forth between the embedded NICs (Dell Server Broadcom NICs) and virtual NICs from these virtual fiber NICs as the management interface (with NICs being removed, re-added frequently). At one point of troubleshooting I ran a standard Linux ifconfig -a and I noticed that a number of NICs had stale IP addresses assigned, even though these NICs were removed from as the primary management interface and xe pif-param-list uuid=(eth0's UUID) showed the interface had no IP assigned.

Strange behavior indeed, checking the 2 other servers we were testing on they had the same issue, a lot of stale data, even after a reboot of one host. It seems the Xen software was not passing down it's IP address removals (or cleaning up after itself) using the xsconsole, the only way to fix the issue was to clear the IP manually using the following command:
xe pif-reconfigure-ip mode=none uuid=(UUID of ghost IP nic)

The issue appears to only occur when you selected a new management NIC using xsconsole, setting it via the shell always cleared the old IP and set the new IP (likely because the steps include the above mode=none command), so going forward if I change the primary management interface I now use the command shell vs xsconsole:
xe host-management-disable

xe pif-reconfigure-ip mode=none uuid=(old_mgmt_nic_uuid)

xe pif-reconfigure-ip mode=static IP=(server_IP) netmask=(netmask) gateway=(gw_IP) uuid=(new_mgmt_nic_uuid)

xe host-management-reconfigure pif-uuid=(new_mgmt_nic_uuid)

No comments:

Post a Comment