Setting up an Isolated Driver Domain in Xen

Disclaimer : This article assumes that you have some experience in compiling kernels and are familiar with installing and using Xen.

Xen has a very useful feature of Driver Domains. This post will not talk about the theory, but instead focus on setting up a Driver Domain.
Information about driver domains can be found on the Xen-wiki .

So, as mentioned on the wiki page, we basically need 3 things to setup a driver domain :
1. PCI Passthrough
2. Backend Driver
3. Hotplug scripts

One of the easiest ways to create a Driver Domain is the boot the Dom0 kernel as a DomU.
This article talks about setting up a driver domain for a PCI device. The process described below does not apply for creating a driver domain for block devices.

There are some config options that have to be taken care of.

1. Backend Driver :

The back-end driver for the device should be compiled. In this post, we will be creating a driver domain for a PCI device. Hence the PCI backend driver should be present in the DomU that is to be used as the driver domain.
Since we will be using the same kernel for Dom0 and DomU, we compile the PCI backend driver in the kernel configuration. "pciback" is the PCI backend driver. It can be compiled either as a module or built into the kernel.
I have compiled pciback into the kernel. This makes the process of passthrough extremely easy.




2. PCI Passthrough :
PCI passthrough is a feature that allows DomU to gain direct access to the physical device.
The first thing that has to be done is to compile the Dom0 kernel with PCI Passthrough enabled. This step is optional.

Compile and install the Dom0 kernel.

The first step for acheiving PCI passthrough is to find the PCI ID of the device.
The PCI ID can be found using "lspci". Once the PCI ID has been found out, follow these steps :


If pciback is built into the kernel, then add the following options to the Dom0 kernel command line, in the bootloader's config file :

pciback.permissive pciback.hide=(03:00.0)

Here, 03:00.0 is the PCI ID of the device. ( A NIC on my machine). This hides the device from Dom0. If the device is not hidden from Dom0, then Dom0 initialises the PCI device...which we do not want.

The next step is to modify the DomU's configuration file. Add the following line to expose the PCI device to the DomU that is to e used as the driver domain.

pci = [ '03:00.0' ].

Now, start the DomU. Remember to use the same Dom0 kernel for this DomU.

Once the DomU is booted, issue "lspci" from within the DomU. The PCI device should appear.

If lscpi shows the PCI device, then it has been successfully pass-throughed !

3. Hotplug scripts :

Now, to use the DomU containing the PCI device as a driver domain, we need hotplug scripts. Hotplug scripts allow other DomUs' frontend drivers to attach themselves with the backend.

The easiest way to get the hotplug scripts in the Driver Domain is to get the xen source code and do a "make install-tools" within the Driver Domain.
This should install the required hotplug scripts in the Driver Domain.

This completes the process of setting up a driver domain ! :-)

For a NIC, run the network-bridge script from /etc/xen/scripts to bring up Xen's virtual bridge in the Driver Domain.

Now, to let the other DomUs know about the driver domain, edit the configuration file of the DomU.

vif = [ 'mac=some_mac_id, backend=id_of_the_driver_domain' ]

the "backend" option identifies the Driver Domain for the DomU.
Now boot the other DomUs.
It can be verified that the vifs from the DomUs are connected to the bridge in the Driver domain,
run

$brctl show

in the driver domain.

Xen-tools and ubuntu Dom0

Installing Xen as described here does not install the xen-tools properly. Even if the hypervisor and Dom0 are up perfectly, there is nothing more that can be done without xen-tools.

$ sudo xm list

Traceback (most recent call last):
File "/usr/sbin/xm", line 5, in
from xen.xm import main
ImportError: No module named xen.xm

Thanks to Shriniket and Hormazd for reproducing the error :-)

As described here, the solution to this problem involves editing the Config.mk file.

The Config.mk file is in the source root directory.

The following lines (line no. 38 and 39) have to be changed from :

PYTHON      ?= python
PYTHON_PREFIX_ARG ?= --prefix="$(PREFIX)"

to :

PYTHON      = python
PYTHON_PREFIX_ARG = 

Save the changes. Then,

# make clean

# make install-xen

# make install-tools

# make install-stubdom


Doing # make clean and # make install-tools might be sufficient.
Haven't tried it.


Everything should be working fine now.

Tuning Config.mk results Xen packages to be placed
into /usr/local/lib/python2.6/dist-packages due to
Changeset 19594 in xen-3.4-testing.hg.
Otherwise, Xen packages would go to
/usr/lib/python2.6/site-packages,
which is not default location for python 2.6 on Ubuntu 9.04
( as opposed to  Fedora 11 ).
Thus you won’t be able to start xend in Dom0.

Cheers !

Problems with getting Xen working ...

IF you are looking for how to install Xen, you can find it here.

Trying to get Xen working , ran into some problems ...

1. There is this peculiar problem that I faced with all the kernels that I tried as Dom0.

Gave up waiting for root device. Common problems:
-Boot args (cat /proc/cmdline)
-Check root delay = (did the system wait long enough?)
-Check root=(did the system wait for the right device?)
-Missing modules (cat /proc/modules; ls /dev)
ALERT! /dev/disk/by-uuid/xxxxx-xxxx-xxxxxxxxxxxx does not exist. Dropping to shell!

Busybox v1.10.2 (Ubuntu 1:1.10.2-2ubuntu7) built-in shell (ash)
Enter 'help' for a list of built-in commands.

The hypervisor was booting fine, but the Dom0 refused to recognise my root device and would drop me into busybox.

The solution to this problem is to pass an extra parameter to the kernel, indicating that it should wait longer for the root device. (SCSI disks have this problem).
So now, you need to edit your /boot/grub/menu.lst. The parameter to be passed is "rootdelay". rootdelay=90 has worked for me. You may experiment with lower/higer values.

$sudo gedit /boot/grub/menu.lst

#######Snippet - menu.lst ###########

title Xen 3.4.0 / Ubuntu 9.04, kernel 2.6.18.8
uuid fe8adb1a-7b82-40c4-b7d9-e35921c1155f
kernel /boot/xen-3.4.0.gz
module /boot/vmlinuz-2.6.26-2-xen-686 root=LABEL=root ro console=tty0 rootdelay=90
module /boot/initrd.img-2.6.26-2-xen-686

2. The second problem that I faced was with the 2.6.18 xenified kernel. The kernel boots into busybox and the keyboard becomes totally unusable. None of the keys works.
Although I did notice that the quickplay buttons do generate an interrupt ( it shows a message on the screen when I press a quickplay button), the rest of the keys are useless.

Could not fix this :-(

3. The 3rd problem that I faced was with the 2.6.26 Debian xenified kernel, along with the first one.

ata1.00: qc timeout (cmd 0xec)
ata1.00: failed to IDENTIFY (i/o error, err_mask= 0x4)

A little googling led me to think this was an ACPI problem. So I tried passing acpi=off in the kernel cmdline, but did not work out.
Some more googling led me in various directions. Tried atleast 5 different parameters.

The one parameter that finally worked was pci=nomsi . Found it here, and oddly enough as a solution to a different problem !

Apparently this is occurs due to problem of the Debian kernel on some motherboards and is consistently reproducable.

So, the final kernel command line that works perfectly is :

/boot/vmlinuz-2.6.26-2-xen-686 root=LABEL=root ro console=tty0 rootdelay=90 pci=nomsi

Hope this helps !

Cheers ! :-)

Installing Xen 3.4.1

Recently, there was need to install Xen on my laptop for project work.

Xen 3.3 is present in the ubuntu repositories and installation is very straight forward.

If you plan to use Xen 3.3, then the task is extremly easy.

Steps :

1.Installing the Xen hypervisor.

sudo apt-get install ubuntu-xen-desktop

2. Installing a Dom0 kernel. There are various options available.
i) You can either use the 2.6.18.8 xenified kernel available on the official website. Get it here.
ii) You can download Debian xenified kernel from here and the modules for the kernel from here.
iii) You can use download a vanilla kernel from here and patch it to work as a Dom0. You can get the patches here. Patches for version 2.6.29-6 are stable.
iv) You can download a vanilla kernel and configure it as a Dom0 using PVOPS. (There are still quite a few issues with this.)

3. Done.

Installing Xen 3.4.1 :

Detailed steps :

I am describing how to install Xen 3.4.1 from source and using the Debian xenified kernel (2.6.26) as Dom0. I compiled the 2.6.18.8 xenified kernel, but my keyboard would become unusable. I also tried patching the vanilla kernel, but ran into some problems. So I settled for the Debian xenified kernel.

Step 1: Satisfying the dependencies.

Special thanks to Furquan Shaikh, my senior, for providing me a comprehensive list of all the dependencies.

$ sudo apt-get install gcc make binutils zlib1g-dev pyhton-dev python2.6-dev python-opnessl libncurses5-dev libssl-dev xorg-dev bridge-utils iproute udev tetex-base tetex-bin latex-make texi2html texinfo texlive texlive-base gawk gettext-kde

Alternatively, you can automate the process using :

$ sudo apt-get build-dep package-anme

For more info on this, click here and here
Thanks to CDK :-)

will install all the required dependencies.



Step 2 : Downloading the latest Xen source. You can download the source from here.

Step 3 : Extract the source into a suitable directory. Then

i) # make install-xen
ii) # make install-tools
iii)# make install-stubdom

If all the dependencies were satisfied, step 3 should not give any problems at all.

Step 4 : The packages for Dom0 Debian kernel can be obtained from here and here.

Once downloaded, install the kernel using :

$ sudo dpkg -i linux-image-2.6.26-2-xen-686_2.6.26-19_i386.deb linux-modules-2.6.26-2-xen-686_2.6.26-19_i386.deb

This installs the Dom0 kernel and also updates the grub entries.

This completes installation of the hypervisor and the Dom0. Since the Dom0 kernel is 2.6.26, make sure that your root partition is not ext4. ext4 support was included since kernel 2.6.28.

Now, reboot your computer, you should boot into the Dom0 kernel if everything went right. :-)

I ran into some problems while booting the Dom0 kernel. I am posting them here.