High Availability & HA Lizard


Featured Image from the Google Image archives…

After persuading you I am not riding on the coattails of a GENIUS for self promotion, I now have to direct all XenServer Administrators to another fine piece of work by none other than Tobias Kreidl.  Or this Tobias Kreidl, but they are one in the same.

No longer being with Citrix doesn’t mean I’ve dropped my favourite type-1 hypervisor of choice to quickly roll out my computing needs, so in my new life as my own XenServer Administrator, I can’t do anything else but share this grand slam of a write-up regarding none other than… High Availability.  Yup, I said it.

Without spoiling too much, his long standing research into HA Lizard is well documented along with the typical pain points of the Tennable-based “HA” code, limitations, and various mathematical outcomes that can, sans running HA Lizard, lead to anything from incorrect host fencing all the way to increasing a XenServer pool’s host count to smooht out the nuanced behaviors when one runs less than three XenServer hosts in a pool with High Availability activated.


Another shining example of a community member going out of their own way to help all of us Xenophiles with success in our own deployments and it can be read here:


I need to dig up the screen scrape, but as a final note I find it fascinating how time sharing, Burroughs mainframes, and their MCP OS had this built into job execution with multi-processor process control: ensuring a single job would fence as to avoid other data terminal operators from losing their compute cycles:

Procedures can be invoked in four ways – normal, call, process, and run.

The normal invocation invokes a procedure in the normal way any language invokes a routine, by suspending the calling routine until the invoked procedure returns.

The call mechanism invokes a procedure as a coroutine. Coroutines have partner tasks, where control is explicitly passed between the tasks by means of a CONTINUE instruction. These are synchronous processes.

The process mechanism invokes a procedure as an asynchronous task and in this case a separate stack is set up starting at the lexical level of the processed procedure. As an asynchronous task, there is no control over exactly when control will be passed between the tasks, unlike coroutines. Note also that the processed procedure still has access to the enclosing environment and this is a very efficient IPC (Inter Process Communication) mechanism. Since two or more tasks now have access to common variables, the tasks must be synchronized to prevent race conditions, which is handled by the EVENT data type, where processes can WAIT on an event until they are caused by another cooperating process. EVENTs also allow for mutual exclusion synchronization through the PROCURE and LIBERATE functions. If for any reason the child task dies, the calling task can continue – however, if the parent process dies, then all child processes are automatically terminated. On a machine with more than one processor, the processes may run simultaneously. This EVENT mechanism is a basic enabler for multiprocessing in addition to multitasking.


I love computing.

— JK Benedict | @xenfomation

Dundee is now XenServer 7.0!


The image above – from Sega’s Zillion – shaped mine and my brother’s lives.

I often make reference – internally or externally – to these hosts, but no one has ever asked “Whom do you speak of?”

Now you know.

A big congratulations is in order to XenServer PM, Dev, QA, Tim, and every single one of you who have participated from Dundee’s first Alpha drop to its official release as XenServer 7.0!

I direct much thanks to Tim for various reasons.  Mostly to pick on him as, despite being at Synergy amongst Tobias, David, Carl, and so many other great hooligans, he found a way to yet keep all of us up-to-date via XenServer.org:

It was a little over a year ago when I introduced a project code named Dundee to this community. In the intervening year, we’ve had a number pre-release builds; all introducing ever greater capabilities into what I’m now happy to announce as XenServer 7. As you would expect from a major version number, XenServer 7 makes some rather significant strides forward, and defines a significant new capability.

Tim’s typing marches on with melodic precision and before ending on my favorite part – the download – we get this luscious piece of data to salivate over:

No major release would be complete without a requisite bump in performance, and XenServer 7 is no exception. Host memory limits have been bumped to 5TB per host, with a corresponding bump to 1.5TB per VM; OS willing of course. Host CPU count has been increased to 288 cores, and guest virtual CPU count has increased to 32; again OS willing. Disk scalability has also increased with support for up to 255 virtual block devices per VM and 4096 VBDs per host, all while supporting up to 20,000 VDIs per SR. Since XenServer often is deployed in Microsoft Windows environments, Active Directory support for role based authentication is a key requirement, and with XenServer 7, we’ve improved overall AD performance to support very large AD forests with a resulting improvement in login times.

Naturally, I was eager to quit working with internal builds and bite into the real deal yesterday evening – and that I did and am still going!  Along with run of the mill basic testing, I have already staged up a bit more for subsequent entries.

Still, an engineer is only as good as the documents he can validate to be true, so a special thanks to Mike Meyer (@AspnSys)!  While I was readily able to pull down XenServer 7.0 as any client, partner, or individual would, I could not find the documentation at support.citrix.com!

This is where Mike stepped into a Tweet I had on the wire – to assure me the docs were indeed under docs.citrix.com, albeit it wasn’t where we expected it quite to be.

Lots of moving parts during Synergy, so hey — it saved me from calling people as to prevent my own panic with ensuring each of you has the right resources for this release.  Thanks, Mike!

By now, XenServer 7.0 and components can be downloaded directly from




Such a backstory!

Well, it is finally behind us and already I have began my own local testing:  the usual stuff that consumers (such as myself) would tackle.  Sure, I had a head start with the release materials this morning, but colleagues, partners, and clients come first on a daily basis.

Sleep can be overrated, especially with a release hot of the presses.

Now that I’ve been home since 19:00 EST, I wanted to blog my lesser escapades with XenServer 7.0 and what I’ve done with my trinity of machines known as…

JJ, Apple, and Champ!

Champ Steps It Up…

With Apple on SAN duty to my brother and JJ running my home resources, I picked on good champol’ Champ.  He’s sturdy, a bit bulky, but powerful and with the power at my hand, I literally had to remind myself Golden Rule #2 with XenServer:

Download The Latest XenCenter Before Upgrading ANY XenServer Host(s)

No, seriously.  People think I make this up and understandably so.  It’s a client utility, right?  Yes, it really is for the most part, but it drops slap-dash “client bit” for a more aggressive “go-between” during migrations, server status reports, and most of all…

Upgrades (especially if a Rolling Pool upgrade is performed or if one is migrating VMs back-and-forth among pool members).

“What is Golden Rule #1 regarding XenServer?” you may ask.

Download The Product Documentation & Read It!

Seriously, now.  Don’t get sensitive on me!  I am just as guilty as skimming through instructions, literature, and studying.  I am trying to advise, to help, and to work through this with you.
Now, let’s take a quick recap as to ensure we are ready to approach an existing installation or a fresh, brand new Type 1 Hypervisor Environment:

oldfloppySave the Plastic!  Boot from USB!

Indeed, if you have the opportunity to, use a USB Thumb drive and a free utility – such as Rufus for Windows or UNetbootin (or DD) for Linux – to take the ISO file and explode it onto a USB thumb drive (1GB or larger).

If you don’t know how to use these tools or would like more information, specifically regarding XenServer 7.0 with Rufus, UNetbootin, or DD, please let me know!

Test 1: XenServer Health Check

After downloading the latest XenCenter and subsequently installing it (or upgrading my existing XenCenter), I found all of my XenServer hosts to be clearly visible.  Of course – per documentation – the latest XenCenter is not backwards compatible with pre-6.x versions of XenServer.

Sure, I had to disable (again) the RDP pop-up option that is apparently set back by default, but hey — not really a big deal as this is what made me smile:


As part of a long standing “call home” feature, which is truly for the customer experience, I was happy to see a feature I had tested before present itself in its anticipated glory!  The Health Check utility!

You don’t have to enroll, but if you have support or use https://cis.citrix.com on any basis to check the status of a pool or a stand-alone host, this nifty utility will allow you to schedule automatic uploads of server status reports as so you can gain insight into your XenServer deployment: updates, preventative patches, etc.

The great part of this feature is that while Champ was currently running 6.5 SP1 and JJ is still running 6.2 SP1, I could enroll them both for automatic uploads or I could still run manual server status reports… with a great option to simultaneously send a copy to CIS (along with a Citrix Support case number)… and that makes me – an employee and customer – quite content.

As a quick example of the normal server status report process, all looks the same until the final step where it not only saves the report for myself (locally), but it asks if it should also upload the report to CIS (this test comes from JJ, but is also the same for 6.5 SP1 and 7.0 XenServer hosts):


I like it.  I’m biased as I contribute as much as I can to the CIS team, so again, great work to everyone involved.

Test 2: The Upgrade

Champ had a few VMs I had been testing with: each VMs virtual disks contained within Local storage for my own nefarious purposes:


I don’t want to lose these guys, however if you have read the Installation Guide for XenServer 7.0, you will know that one of the new features is GPT partitioning.  It offers a larger capacity for the root installation along with a separate partition for /var/log/.  In short, a better disk layout to prevent system logs and patches from filling up the XenServer root partition.

Per the documentation, if VMs or any VHD/VDI are held on Local storage the upgrade process won’t use GPT partitioning (https://docs.citrix.com/content/dam/docs/en-us/xenserver/xenserver-7-0/downloads/xenserver-7-0-installation-guide.pdf, Page 12):


(Now you can see why I am such a fan of documentation!)

Instead of the new 18GB disk layout with /var/log/ (4GB) split into its own partition, I temporarily had an upgraded XenServer 7.0 host ready to rock, however it still had a root partition size of 4GB:

[root@CHAMP ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1       3.5G  2.2G  1.1G  67% /
devtmpfs        319M     0  319M   0% /dev
tmpfs           329M   36K  329M   1% /dev/shm
tmpfs           329M  976K  328M   1% /run
tmpfs           329M     0  329M   0% /sys/fs/cgroup
xenstore        329M     0  329M   0% /var/lib/xenstored
/dev/loop0       55M   55M     0 100% /var/xen/xc-install
tmpfs            66M     0   66M   0% /run/user/0

So, in planning for a fresh install, I exported those VMs to USB-based storage and deleted the originals from Champ: freeing local storage of any VDI/VHD files, etc as to ensure that on my next effort – a clean install – I get the benefit of GPT partitions and I can import my VMs back to Local storage.

I get my Xen and my Virtualization, too…

Quick Note!

With the release of XenServer 6.5, on installation the bare metal host’s total RAM is taken into consideration.  The purpose of this is to set – for new installs only – the dom0 memory allocated (instead of the old 752M limit) to a higher value.

The same holds true with XenServer 7.0 on fresh installation.  The breakdown of total host RAM to how much DOM0 memory is allocated on a fresh install is as follows:

Total Host Ram    -->     DOM0 Memory Allocation
24GB (or less)            752M
24 - 47GB                 2048M
48 - 63GB                 3072M
64 - 1024                 4096M

For XenServer deployments running 6.5 or greater, at least 8192M is recommended if the memory caching feature is licensed/enabled.

Test 3: The Fresh Install

After exporting my VMs, I completed a fresh install.  Upon re-connecting, I enrolled Champ into the Health Checker and immediately checked the disk partitioning for XenServer’s root:

[root@CHAMP ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        18G  1.8G   16G  11% /
devtmpfs        319M     0  319M   0% /dev
tmpfs           329M   36K  329M   1% /dev/shm
tmpfs           329M  992K  328M   1% /run
tmpfs           329M     0  329M   0% /sys/fs/cgroup
xenstore        329M     0  329M   0% /var/lib/xenstored
/dev/loop0       55M   55M     0 100% /var/xen/xc-install
/dev/sda5       4.0G  138M  3.7G   4% /var/log
tmpfs            66M     0   66M   0% /run/user/0

And Presto!  I now have a 22GB GPT partition setup: 18GB dedicated to literal /root and 4GB dedicated to /var/log/!

Within XenCenter, one of my favourite features is that of the “Open SSH Console” as Putty comes with XenServer/XenCenter 7.0.  It works for not only the control domain/dom0, but also for Linux guests when an SSH port is detected:


Much like the “Remote Desktop” button for Windows guest VMs (or Linux VMs with XRDP installed), we simply click it and login:


I did find that in /var/log/, the messages file was at 0 bytes — not growing.  This has been reported, so for me, the work around was simply to add the following to /etc/rsyslog.d/xenserver.conf between local 5 / v6d logging:

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

This was taken from /etc/rsyslog.conf and is something I am actively investigating as once this is added to /etc/rsyslog.d/xenserver.conf, for the changes to take affect we simply can execute:

[root@CHAMP log]# service rsyslog restart
.... and we start to see /var/log/messages growing:
[root@CHAMP log]# ls -la /var/log/messages
-rw-------  1 root root      602 May 25 04:38 messages

Since logrotate is responsible for the rotation, as well as other configurable options to apply to Syslog-based file handling, one may notice that /etc/ has two logrotate configuration files:

[root@CHAMP log]# ls -la /etc/logrotate*conf
-rw-r--r-- 1 root root 591 May 25 04:14 /etc/logrotate.conf
-rw-r--r-- 1 root root 849 May 11 06:05 /etc/logrotate-xenserver.conf

The reason to point this out is that the larger of the two, /etc/logrotate-xenserver.conf, does not have compression or other options enabled.  So, running logrotate from the command line (manually), I have found the former config file to be optimum:

[root@CHAMP log]# logrotate /etc/logrotate.conf

This version of the config /etc/logrotate.conf contains the following options:

# see "man logrotate" for details 

# The compressed files are deleted by another script, based on total disc use.
# Therefore for logrotate we specify a large number.
rotate 9999

# create new (empty) log files after rotating old ones

# Use sharedscripts for efficiency: if we rotate several files that are written
# by the same process, send it SIGHUP just once. For syslogd this causes a
# restart, which the man page says is expensive.

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    create 0664 root utmp
        minsize 1M
    rotate 1

/var/log/btmp {
    create 0600 root utmp
    rotate 1

# system-specific logs may be also be configured here.
[root@CHAMP log]# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files daily

# keep one months worth of backlogs
rotate 31

# create new (empty) log files after rotating old ones

# compress log files

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    create 0664 root utmp
        minsize 1M
    rotate 1

/var/log/btmp {
    create 0600 root utmp
    rotate 1

# system-specific logs may be also be configured here.

Not only does this have sane options, but the head of the file clearly specifies something I believe many XenServer administrators already know:

# The compressed files are deleted by another script, based on total disc use.
# Therefore for logrotate we specify a large number.
rotate 9999

There are two such background scripts which run – all the time – to prevent Syslog data files from growing out of control.  I don’t mean “check periodically”.  I truly mean we have checks and balances to prevent any sort of flooding as it relates to Syslog data so that, in an odd situation, the files never grow wildly out of control…. especially if logs are being forwarded to another host.

Similar checks have been in place before, but with the 4GB total installation size, these scripts were rather ineffective, but until I do more research into this final implementation of system logging I want to hold off on any speculation.

To me, it simply looks like an appearance of “duplicate” configuration files, where as they probably are more or less for reference.

I will certainly be testing my log generator today as it is quite about time to hit the office, anyway.

Test 4: VMs and Stuff

This will be updated later today as with lsof and other tools being installed within dom0, I am quite certain I can get some nifty “Tobias Style” metrics out of a SOHO XenServer 7.0 system with a few Windows and Linux guest VMs!

Test 5: Miscellaneous



In Summary…

XenServer rocks and I thank everyone from my colleagues to YOU for allowing me to work with something so amazing!


–jkbs | @xenfomation | XenServer.org | My Blog



Virtualizing: Because I CAN…



To all of you who read the first of this series, Virtualizing: Because I CAN…, thank you very much for the feedback and shares.  During a weekend filled with family, I rushed to pump out a quick article that truly goes far back in my work with Tim & Trolle.  After a bit of a challenge of Twitter, I rushed into it.

I wanted to take a second to be a bit more clear about what I’ve done, what I do, and why I do it.  First, the ground rules:

  1. I get the stuff working as far as I can…
  2. I explain why I did it and how…
  3. Just because I do it doesn’t mean you should in production…
  4. Yes, where the environments I am dealing can be shared, I will setup XVA exports to share, share, share

The whole principle of this is to take work I’ve done (and still do) for utility operating systems and share how I got it running in XenServer… along with the outcome.  This includes fresh distros, as well, but for the next week, I’ll be focusing in on a true “ERA” (of sorts).

Lastly, as I’ve said…

Just because I do it doesn’t mean you should in production and these articles – for clients, partners, colleagues, and the XenServer community – are strictly science experiments.  Don’t ask for support on Xenix, etc.  That’s all I’m asking.

Oh, and I PROMISE, if you read all the way through you’ll find something that every XenServer user has seen after upgrading tools…

Alright.  I feel better.  Time for coffee and more writing.  #BritsAreAwesome #CambridgeMadeXen #UnionJack #GiveMeAnAmstrad

PCs, Clones, and Ports: Part 1

Depending on your level of disdain, love, or preference of utility – as we are returning to an age where two technological giants divorced and left us as the kids in the middle… with Personal Computers and software you didn’t have to rewind.

Ladies and Gentlemen, I present to you Mr. and Mrs. Ex….


Ah, yes.  The tragic lovers who completely continued their honeymoon so distastefully in and through poor Gary Kildall’s death, began the suffocation of CP/M and what CP/M could have become, but always kept the doors open to show us the fisticuffs.

Looking back now, I often wonder which one of them spot the setup?  I know it wasn’t Apple (despite the commonality betwixt these THREE companies being Mr. Bill Gates).

My bet is on Microsoft and for nothing more than the same reason they were able to date Apple…

“The hardware exists and we already have the ability to code on it… and code on the competition’s platform… and the compatibles… and we build a dynsasty of software every architecture must have for the consumer….”  Oh, the places you went, returned to, and tried to go into.

Fast Forward: XenServer, DOS, and … Amstrad?

I’m not heading in any chronological order due to the nature of IBM, IBM compatibles, and the reams of software that is avaialable.  Let’s take this quick example:


And if you hit F2, we get…


… and no one could remeber to exit PC-BASIC by typing “system”.  And wouldn’t you know it?  Like fine wine, it has aged and can be used on Mac, Windows, and Linux.  Take a peek at https://sourceforge.net/projects/pcbasic/ or, start your own software archive like me by  visiting http://deger.republika.pl/Download_MS_Basic_versions.htm.

Not familiar enough?  What about this?


Well, even if you still don’t recognize these items, I welcome you to the world of IBM/PC/MS-DOS… from XenServer.  Yup, I dug out an old VM export for a “DOS Collection” project where my kids, friends, and especially ME could immerse ourselves in the glorious environment that is generally called “DOS”: code page 437, FAT, basic, and where many people first absorbed that which was Personal Computing.

The Environment Details…

With respect to XenServer, I’ve had this VM since the 6.0 days.  With each revision, it just runs more and more smoothly and while it looks and feels like MS-DOS 7.1, it existed first as MS-DOS 5.x.  During the XenServer 6.2 days, I did some finess work with the installer and it’s a hybrid MS-DOS 5/6/7 release with plenty of opensource utilities stacked on it.

The reasons I took such odd approach was based on four key reasons:

  1. Jim Hall’s FreeDos, which I spent a short time testing and contributing to (those guys are beyond intelligent, trust me)
  2. These later versions could handle my favorite games, still compile code in primitive forms of my favorite languages, BUT drivers… DRIVERS… were becoming more cohesive
  3. DOSBox exists, but is limited as an emulator/almost type-2 hypervisor

Using the “Other” template, I set aside 1 vCPU and 256MB of RAM.  Considering the software I had, I looked back over pre-GUI OS/2, older versions of MS-DOS (pre 4.x) and IBM/PC-DOS that existed from the long lived IBM/Microsoft relationship and came to a single conclusion to work around my virtual hard disk needs:

I needed to master three ISOs.  One with MS-DOS 7.1.  One with MS-DOS 5 and 6.  And a final disk with a host of extender utilities, FAT improvement, code-fixed DOS utilities (format, fdisk, etc).  After all, this is a virtual machine and I don’t have to torture myself with the days before drivers became a bit more standardized, memory management wasn’t as manually intensive (ie, gaming versus word processing), and so forth.

After I mastered the ISOs and slapped them onto my local SR, I booted off of MS-DOS 7.1 and “broke” out of the installer.  That gave me the prompt I needed – with CD-ROM/Mouse support – do mount the utility ISO and it was there I was able to use updated versions of fdisk/format to achieve this:


After the disk was formatted, I basically had all the individual compents I could have authored into my own installation ISO, but instead I hand picked the core of MS-DOS, installed DOS extenders, helper utilities, multiple device drivers (including many variants of mice, long/short filename, linkers, basic VGA drivers, etc).

I crossed my fingers, rebooted the VM, and oh me oh my.  I was in heaven as a packet driver could be installed later (or even Windows 3.x, 95, or 98), but I went about my DOS business… which included running AN AMSTRAD CPC EMULATOR WITHIN DOS WITHIN XENSERVER!

Yes, I need to get Wattcom installed, but for now — enjoy these shots from XenServer and if you have an Amstrad, feel free to let me know as in the states… SLIM.  EXPENSIVE.  NOT GUARANTEED TO WORK.  PICKINGS.

True Emulation in Virtualization…

Since I couldn’t get an Amstrad, let alone two, for my father and I to have our old school code battles on, I found NO$CPC — it does not disappoint, I promise.  The game/code you are seeing is my own improvements onto Conway’s Game of Life.



Indeed, I WAS READY…


And off we go despite not having time to add code for more than 10 generations…


So, I took a few minutes to play a game.  Battle Chess, oh yes.  This thing won’t run for 10 minutes in DOSBox without crashing – which I can understand – but in XenServer… no issues!  I LOST!



Of course I went on and on to play Mechwarrior, Indianapolis 500, but then I decided to hack out a version of something we all see when tools are successfully installed and a VM is rebooted.  The IDE/Language was 80/20 Software’s “ASIC” (almost basic).  Powerful – especially with extended libraries for INT21 ops, Chipset Access, etc.  However… this is for all true XenServer fans…


Look familiar?


Conclusion of Part 1

Yup, this was quite a long piece, but the fact I was running a mixed DOS environment, writing and compiling code, emulating a completely different architecture within the virtualized environment… well, I had to share.

Next up?  OS/2, PC-DOS?  eComStation and some Windows 3.11?  We shall see!

–jkbs | @xenfomation | XenServer.org | My Blog

Virtualizing: Because I CAN…


… and I spent far too much time, money, and effort working with countless permutations of hardware, software, peripherals, and so on just to have “The Industry” force me to upgrade.

Yup.  The following series goes out to many, many individuals (you know who you are) and it aims to show what I can do with some time and XenServer.  It is a fun challenge, but for me, it is really about having the environments, tools, and instant access to things old and new.

I’ve edited this page 7 hours after it’s original post because I was in a rush.  I’ve added more screenshots (below).

So, from the abstract to the useless re-branding of other operating systems, I will be presenting old and new OSes I’ve virtualized on XenServer.  Sometimes I might dive into the gory details and sometimes, like this morning, I may just gloss over the fun.

Something Old… Something Old-ish…

I sat down and spent some time with several operating systems: breathing life back into the tried, true, and sometimes (in retrospect) very annoying to use.  I’ve always been a fan of Window Maker and OpenStep, so I hit those first.

Eh?  OpenStep?  Yes.  Specifically 3.x or 4.x from back in the day.  It is amazing to see how much of Steve Jobs’ NEXT interface “things” ended up into Mac OSes upon his return.

The OpenStep version I pumped up on XenServer still needs some work.  Irradic mouse control (relative positioning), tweaking of VESA/Basic VGA drivers, and … networking.  It was a fun little project and with the display being in black and white, I had to laugh.

Oh.  Before I begin, YES — I used a generic template withi 512MB RAM, 4GB virtual disk, VGA enabled (not Cirrus), and also disabled any HVM things like parallel port and serial port.  XenServer 6.5 SP1.

The screenshots…








As you can see by this edit, I added things I mucked around with for a while: preferences, the text editor, terminal, and so on.  No doubt an interesting trip down memory lane, but until I can make some more time to tweak my OpenStep Guest, it’s more for show than anything.

The interesting thing was that for as little that was being done, well, OpenStep is just a CPU greedy as it was on bare metal.  Overall it was sluggish, but I shall return and tweak this VM.

In the same vein of the “User Experience”, I finally built a template for Window Maker Live: a Debian 8 niche distro built to mimic OpenStep and its user interface.

I have to say that this VM is staying up and running!

I assigned it one CPU and 2GB of RAM, but it never squeaked above 40% of that amount and as such, I’ll probably lower the RAM to 1GB.  The installation was blazing fast, updates were minimal, and guest tools installed without a hitch for obvious reasons.

I’d like to think that this experience is what I had with OpenStep back in the day, but hey — it’s all in fun and all possible because of XenServer!





In Conclusion…

Laugh if you will, but there is more to come.  XenServer can rock the latest OSes as well as the legacy ones.  For any business, that is pretty darn powerful because not everyone can drop OS and Core Products just to use the latest version of X, Y, and Zed.

Next up?  That is a secret, but I will be covering fresh distros (as well).  Feel free to send a request and I’ll see what I can do!

–jkbs | @xenfomation | XenServer.org | My Blog


XenServer & Hotfixes


** Updated 19-APR-2016 with emphasis on XenServer 6.x hotfix resources **

Not too many clock cycles ago, The Great Mr. (thank you for the permission to cite you, sir!) tweeted a sage piece of advice.  It is without saying that, since his tweet, I have pondered this topic as one who supports XenServer and one who, as an end-user, consumes such things “opensource”…


(The direct link to this can be found by clicking here – Thanks, Lorscheider!)

He is absolutely correct.  If we shift context to all of supported versions of XenServer 6.x, he is now absolutely correct.


Because of CTX138115 – an article a current Citrix Manager and myself used to help maintain over a year ago.

Being an Enterprise company, Citrix and appropriate resources now manage this powerful resource in an effort to provide XenServer Administrators the shortest path to updating their XenServer hosts by stacking updates, specifying when to reboot, and so forth.

I am happy Lorscheider brought up XenServer 6.5’s “from vanilla-to-fully-patched” approach because I have worked with many individuals that apply a single hotfi, reboot, download the next hotfix, install, reboot, etc.

It isn’t required and this document is to help save XenServer Administrators time!

Most Common Use Case…

Be it during my working hours, via Twitter for auxilliary support, or even as a recommendation to the opensource community (hashtag XenServer.org!),  many individuals are running XenServer 6.x at a certain hotfix level.

Despite any issue a question arises:

“I know I am behind on hotfixes, so which ones should I install?”

Well, thanks to CTX138115, we can point everyone from internal colleagues to external users to this article as so they can identify which updates, drivers, order, and when to reboot all in a single shot:

XenServer 6.5


XenServer 6.2


XenServer 6.1


XenServer 6.0.2


XenServer 6.0


Now, back to Lorscheider’s post…

I’ve Just Installed XenServer 6.5

Congrats!  Now, you have to patch it to the minimum supported release-level, XenServer 6.5 SP1.  While we are at it, let’s go ahead and fully patch it!

“But HOW,” you ask.

My answer is CTX138115 :Recommended Updates for XenServer 6.x Hotfixes ….

Based on this article, we have a concise list of what to apply to XenServer 6.5 with only one reboot:

Apply the following hotfixes for XenServer 6.5:

  1. CTX142355 – XenServer 6.5.0 Service Pack 1 – XS65ESP1 should be installed by all customers running XenServer 6.5.
    Installation of XS65ESP1 will be required for future functional hotfixes.
  2. CTX202481 – Hotfix XS65ESP1012 – For XenServer 6.5.0 Service Pack 1
  3. CTX204053 – Hotfix XS65ESP1021 – For XenServer 6.5.0 Service Pack 1  (Includes XS65ESP1005 and XS65ESP1013)
  4. CTX205228 – Hotfix XS65ESP1022 – For XenServer 6.5.0 Service Pack 1
  5. CTX207337 – Hotfix XS65ESP1024 – For XenServer 6.5.0 Service Pack 1 (Includes XS65ESP1003 , XS65ESP1010 , XS65ESP1018 )
  6. CTX208513 – Hotfix XS65ESP1025 – For XenServer 6.5.0 Service Pack 1 (All customers who are affected by the issue described in CTX208403: Citrix XenServer Security Update for CVE-2016-0800 should install this hotfix).
  7. CTX209355 – Hotfix XS65ESP1026 – For XenServer 6.5.0 Service Pack 1  (Includes XS65ESP1023)
  8. Reboot

And it doesn’t just stop there.

From XenServer 6.0 to 6.5, CTX138115 contains each realease’s order of hotfixes to apply, when to reboot, and so forth.  This article is a must for new or existing Administrators as we all know it is best to apply/update XenTools after patching.

So for now, there you go.  If you are looking to patch an old or new XenServer, check out CTX138115 :Recommended Updates for XenServer 6.x Hotfixes instead of installing one patch, rebooting, and repeating that painful order!

There will be a second part to this for XenServer 6.1 clients as well as Dev-Ops!

–jkbs | @xenfomation | XenServer.org | My Blog

iSCSI or NFS: The Other Management Interface


In a digital age life boat of end-users, customers, colleagues, and those unable to perform work the last occupant should be the Administrator.

Most modern server-class systems offer a means of remote access, such as iLO or iDRAC. This affords the Administrator an ability to gain remote control of the physical hardware when other access methods are suddenly unavailable. Unfortunately it is common for the vendors to charge extra for such technology and in the case of XenServer, if the management interface should fail, well the anxiety of regaining control is understandable. This is a rare field where the philosophical observation of control being an illusion is completely irrelevant.

While VMs may be running without complaints from end-users, the Administrator is seeking to find why ping tests act as if the host is dead, XenCenter will not synchronize with the host, and SSH connection attempts fail. There is something going on within the stand-alone XenServer host (or pool of hosts) despite power lights for the physical hardware showing the host is up and sending and receiving data.

Putting any permutations of “what could be the issue” aside, those who have leveraged iSCSI or NFS-Storage with the proper network setup have one ace up their sleeve. This trump card is that of the Storage Interface’s IP address: used to transmit data to and from NFS-based or iSCSI-based SAN/NAS devices. Even in segmented local area networks, designed for the XenServer deployment, if the Adminstrator can ping the IP address of the storage interface from another XenServer (or from their workstation), the chances are good that using SSH to login as the local system user (root) will work.

It is not a security flaw nor is it a design flaw. It is simple networking as the SSH daemon within XenServer listens for secure connections across all physical, IP-configured interfaces. So, by pointing the SSH client to the storage interface’s IP address, control for the Administrator can be established for troubleshooting.

As an example – even considering a different IP Scheme/Subnet such as a 172.x.x.x network for storage – the Administrator needs only to remember the Storage Interface IP addresses:

storageSince has gone unresponsive, the two storage IPs of and (if pingable) should be the next logical targets to attempt an SSH session to as to begin troubleshooting.

In my own lab environment I have split management, VM traffic, and storage into separate networks. Still, dom0’s network routing kernel is and has to be aware of these to function.  On a rare typo on my part and in combination with zero motivation to head “to the office upstairs”, I realized I could regain control and correct my issue by simply connecting to 172.x.x.x via SSH (despite my other internal networks being 10.0.x.x) and presto – I share this with you fellow XenServer aficionados.

So, never give up and I hope this trick I self-taught myself while helping many clients with seemingly isolated hardware comes in handy.

–jkbs | @Xenfomation | XenServer.org

Ghosts of NetScape and XenServer


Sometime after the Druids built a Henge and found much needed rest (some 694224000 seconds after 01-JAN-1970), communications over the wire needed some form of encryption.

Enter NetScape circa the early 1990s and their development of Secure Sockets Layer (SSL). Fast forward to 2015 and the irony that you never used SSL Version 1 will no longer escape you as it was never released. The reason? Too many security holes.

Seem familiar? If not, it should for SSL and TLS (the successor to SSL) are about as open and old as the Alamo sans doors. Sufficed to say, 2014 was an interesting year with successive exploits, vulnerabilities and general cipher cracking, though all of us survived.

None of this came as a shock to myself. Well, that’s not entirely true. What shocked and still shocks me are the horrid names given to the Common Vulnerability Exploits (CVEs) documented at Mitre.org. Part of this lack of shock was my realization that something so immersed in our world, something so heavily used and seemingly taken for granted – like toast – would never be broken.

Another part of my glazed, bored look on this subject is due to previous work lives relating to or brushing against an entire industry. If you haven’t heard of said industry, it is known as Security or IT Security.  You see, many projects, books, paperwork and heavy research nights have passed where my title of “Many Titles” had me locked in this industry. Interesting, complex and sometimes fun, it leads to the following disclaimer:

My days spent heavily amidst the ever ambiguous “Security Field” are gone. If the door is locked, secured, but I am missing a house I know I’ve been hacked. Thus, this post represents my own views and ideas… no one else and for no particular reason. If you are looking for hardcore security advice, use The Google to find a CISSP/Security Firm.

With SSL, TLS, etc providing cryptography over the wire and eventually making its way into the application layer, questions will surface regarding countless areas of IT infrastructure having a gaping security hole.

As for XenServer, I can exploits from last year and so on either don’t affect XenServer OR have been mitigated via hotfixes, best practices and more. Still, how can you tell if your XenServers are using “BLAH BLAH” Cipher with WEAK encryption standards?

XenServer uses SSL based technologies and upon installation (unless a private certificate is purchased), it generates its own self-signed certificate. Since this is so, OpenSSL utilites come with the XenServer bundle and these can be used in turn to get an idea of what type of encryption levels, standards and more are in use.

Introduce your friendly openssl command: found at a XenServer host’s command line near you!

A good example to start with is to see if SSLv2 is being utilized on your XenServer host(s)

From the command line, execute the following command to establish a client-based connection to XenServer on port 443. This literally tries to engage in a server/client discussion and will tell us:

  • If a particular protocol/cipher is used
  • If a connection to the host is viable (or not) with said protocol/cipher
  • And if you are unsure: call Citrix Support

So, from the command line:

openssl s_client -connect YOUR_XS_HOST_IP -ssl2 | less

The output is grand (hence the less option), but SSLv2 is disabled and I see these important line items indicating it will not be used for host/client encryption:


no peer certificate available

No client certificate CA names sent

SSL handshake has read 450 bytes and written 53 bytes

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported

Other examples of the openssl utility can be found at OpenSSL.org. Check it out and if you are curious about your Citrix products, also look here to sort by product and version for documents related to security notices, advisories and more.

–jkbs | @Xenfomation | XenServer.org