In a digital age life boat of end-users, customers, colleagues, and those unable to perform work the last occupant should be the Administrator.
Most modern server-class systems offer a means of remote access, such as iLO or iDRAC. This affords the Administrator an ability to gain remote control of the physical hardware when other access methods are suddenly unavailable. Unfortunately it is common for the vendors to charge extra for such technology and in the case of XenServer, if the management interface should fail, well the anxiety of regaining control is understandable. This is a rare field where the philosophical observation of control being an illusion is completely irrelevant.
While VMs may be running without complaints from end-users, the Administrator is seeking to find why ping tests act as if the host is dead, XenCenter will not synchronize with the host, and SSH connection attempts fail. There is something going on within the stand-alone XenServer host (or pool of hosts) despite power lights for the physical hardware showing the host is up and sending and receiving data.
Putting any permutations of “what could be the issue” aside, those who have leveraged iSCSI or NFS-Storage with the proper network setup have one ace up their sleeve. This trump card is that of the Storage Interface’s IP address: used to transmit data to and from NFS-based or iSCSI-based SAN/NAS devices. Even in segmented local area networks, designed for the XenServer deployment, if the Adminstrator can ping the IP address of the storage interface from another XenServer (or from their workstation), the chances are good that using SSH to login as the local system user (root) will work.
It is not a security flaw nor is it a design flaw. It is simple networking as the SSH daemon within XenServer listens for secure connections across all physical, IP-configured interfaces. So, by pointing the SSH client to the storage interface’s IP address, control for the Administrator can be established for troubleshooting.
As an example – even considering a different IP Scheme/Subnet such as a 172.x.x.x network for storage – the Administrator needs only to remember the Storage Interface IP addresses:
In my own lab environment I have split management, VM traffic, and storage into separate networks. Still, dom0’s network routing kernel is and has to be aware of these to function. On a rare typo on my part and in combination with zero motivation to head “to the office upstairs”, I realized I could regain control and correct my issue by simply connecting to 172.x.x.x via SSH (despite my other internal networks being 10.0.x.x) and presto – I share this with you fellow XenServer aficionados.
So, never give up and I hope this trick I self-taught myself while helping many clients with seemingly isolated hardware comes in handy.