Run CentOS 7 or RHEL 7 in PV mode with Citrix XenServer

Due to relying on crappy hardware in my lab I had a need to run some VMs for a new project (more on that later) on a box which did not support hardware VT. Most hypervisors won’t touch anything more than 32bit single core VMs without hardware VT – fortunately the project requirements specified the use of Citrix XenServer which is a little bit more flexible in that area. In XenServer Windows VMs have to be run in HVM mode (hardware virtualisation) but Linux VMs can be run in PV (software virtualisation) under certain conditions. PV uses bootloaders like pygrub to boot the Linux kernel rather than emulating a complete BIOS etc.

CentOS/RHEL 7 installation media has a very unusual structure, the image is partitioned like a hard drive rather than the traditional ISO9660 file layout. The OS also standardises on LVM and XFS to move away from more archaic structures which hold technology back. None of these things work nicely with Xen’s PV abilities, so Citrix recommend that you run CentOS/RHEL 7 VMs in HVM. No good for me though!

There are a number of hurdles to overcome if you want to run CentOS 7 in PV, but it’s absolutely possible. Step 1 is getting the installer to boot. The installation media is partitioned as above, and the bootloader passes parameters to the kernel to tell it where to find the rest of the installer (on the second partition of the media). If you use PV mode’s pygrub to boot the installation kernel it doesn’t know about any of those parameters so the installation will fail. In fact, you won’t even get to the installation wizard! Fortunately you can just lift the parameters out of the grub.cfg file in the ISO and paste them into XenServer:

utf8 nogpt noipv6 inst.stage2=hd:LABEL=CentOS\x207\x20x86_64

Really it’s the “inst.stage2” parameter which is the key here. OK great, now our installer’s kernel can find the 2nd stage of the installation media. However we still need to get it to stop using XFS and use something that pygrub can understand, even if it’s only for the /boot partition. So when the installer starts, just go in to the hard drive layout settings and custom partitioning… wait, it’s not there in the text mode installer! From RedHat’s own documentation:

If you install Red Hat Enterprise Linux in text mode, you can only use the default partitioning schemes described in this section. You cannot add or remove partitions or file systems beyond those that the installation program automatically adds or removes.

Alright, so we have to use the graphical installer. You can trigger this by adding the graphical keyword to the kernel parameters when you create the VM. It won’t work though, as the Xen PV emulated console isn’t capable of displaying graphics. My solution to this problem was CentOS’s VNC installation feature, whereby it will start an X server with the graphical installer running in a VNC session. You need to add more kernel parameters to specify the IP address settings, like this:

ip=<ipaddress>::<defaultgateway>:<netmask>:<hostname>:<nicid>:none nameserver=<dnsserver>

Yes there are 2 colons between the IP address and the default gateway, that’s not a typo. All of these IP settings are to be used by the VM, so set them accordingly. nicid is the OS assigned device name of the network card, usually eth0. Some example settings:

ip=192.168.0.1::192.168.0.254:255.255.255.0:controller:eth0:none nameserver=192.168.1.254

So my complete set of kernel parameters was:

graphical utf8 nogpt noipv6 inst.stage2=hd:LABEL=CentOS\x207\x20x86_64 ip=192.168.0.1::192.168.0.254:255.255.255.0:controller:eth0:none nameserver=192.168.1.254

Follow these steps to set up the VM:

  1. Create your VM using the CentOS 6 (64-bit) template in XenServer – remember it needs 3GB of RAM or the installer will get upset!
  2. At the bootloader parameters, you will need to enter the kernel parameters discussed above, obviously setting the IP address information suitably for your environment
  3. Once the installer has booted you can grab VNCViewer and point it at the IP address specified in the kernel parameters. You’ll need to add :1 on the end to specify which VNC service to connect to
  4. When you’re connected with VNC, you should see the normal CentOS/RHEL graphical installer. The changes you need to make are all in the hard drive partitioning settings. I changed /boot and / to ext4 and used traditional MBR partitioning rather than LVM. You may be able to use LVM and only change /boot to ext4, I didn’t have time to try various different configurations.
  5. Complete the installation and hey presto, x64 CentOS 7 running in PV mode in XenServer!