Configuring User Mode Linux [UML] and
An environment to Linux kernel development and debugging
A Virtual
Machine is a term that has different meanings in
Computer Science. This page refers to how to configure
and build a virtual machine with the meaning of a
fake computer created by software, being executed in the
form of application in the user space, inside of a real GNU/Linux
system. Such software that simulates the fake computer
described in this document uses
operating system level virtualization; its made by 2
components:
1) An operating system executing in the user space - an
appropriate kernel (and User-Mode-Linux
is such project that develops this kernel); and
2) Programs in user space - available in a file
system.
This document describes the steps to build such
components and how to integrate them, getting at the end
a virtual machine GNU/Linux inside of the main one
running in the computer. We used in the building of the
file system a Debian tool name 'debbootstrap', but such
tool can be installed in others systems as well -
described at the end of this
page, allowing this doc be used by any GNU/Linux
distro. After the creation of the virtual machine, is
described how used it to develop and debug the Linux
kernel, and the develop environment with the steps to
integrate the kernel changes with the tests and the debug
phase.
This document is organized in 4 parts:
- Part I : Creating and configuring an UML kernel
- Part II : Creating and configuring a file system
- Part III : The virtual machine: using it and the environment to develop and debug the Linux kernel
- Part IV : Solutions to eventual problems
The last update of this document dates from April 14, 2007 - attention to dates, because configurations changes with time and the documentations becomes inapplicable (and there is already a lot of documentations that in the Internet about configuring a virtual machine using UML that are inapplicable). Contributions are always welcome, if you find bugs/suggestions and still time and disposal, send it through baroni@ime.usp.br
Thanks to everybody
that helped, the professor Roberto
Hirata from IME-USP that
in the Operating System course at the institute
allowed and stimulated the writing of this
document while I was his student-assistant in
the course, to
Install the git-core package (debian users:
apt-get install git-core). Go to the desired
directory to put the kernel sources
(/usr/src/ traditionally):
Inside of the directory .git/refs/tags/ there
are files that point to marks in the tree,
corresponding to the state of the tree when in
the release of the versions:
Go the the directory:
Clean the old configurations (when exist - in
the case you want 'clean' the tree after make
different configurations and compile it several
times, taking the tree back to its original
state):
Compile the modules:
Create a directory (here, named "root_fs") and,
to keep related things together, create such
system inside of the uml kernel sources:
The kernel needs a __file__ of a file system,
from where it will access. To make this one,
first create a file with 300MB (leaving ~100MB
of free space) in the directory of the kernel:
To configure a network connection in the virtual
machine we'll use the tun/tap method. Tun/tap is
a connection that associates an interface with a
IP or ETHERNET address to a device
(/dev/net/tun), in such a way that when a
process writes a package to such interface
address this will be write at the /dev/net/tun
device, that will send the package to the kernel
network management layer, that will manage it as
a normal network packet coming from that
address associated. This way, tun/tap presents
as a useful resource to communicate processes in
the user space - see Documentation/networking/tuntap.txt
in the kernel tree to know more. It's needed to
enable the support to tun/tap at the main system
kernel (see that checking if "CONFIG_TUN=y" in
its .config file, otherwise you should recompile
your kernel with such option).
A program is needed to be used, to configure the
device /dev/net/tun at the programming level,
using IOCTL calls. We'll use an available
program from the cvs of the UML project -
'tunctl'. Get it:
There will be 3 distinct interfaces:
The network connection between the virtual
machine and the main system happens through the
tap0 interface: when some packet is write at the
eth0 of the virtual machine, it is send to the
tun/tap interface at the main system, and
vice-versa.
The network configurations in a Debian system
are defined by the default in the file
/etc/network/interface, that is read by the
/etc/init.d/networking script executed by the
command '$ /etc/init.d/networking <start |
restart | stop>' or by the 'ifup' and
'ifdown' programs. The interfaces (1) and (3) of
the main system are defined editing such file as
follows: (change the 'user' by some restrict
user in the system):
Now to the virtual machine system, mount your
file system and configure the interface (2)
editing your configuration file
(/mnt/vmfs/etc/network/interfaces) defined as
follows:
If desire to open the access of the virtual
machine (192.168.1.2) to the Internet through
the network interface of the main system with
the Internet IP ($EXTERN_IP here), use the
iptables to create a network addresses table
(NAT):
Now that the UML kernel and the file system were created,
enable the tap0 interface of the main system to be
ready to the connection with the virtual machine:
And so the virtual machine will start its execution,
displaying the same screen that normal boot of a Linux
kernel in a computer, and a xterm window will be opened
with the login prompt. Access it as root (no password
needed - set it with 'passwd' command after login)
The Fedora Core 5 has presented a bug when running the
UML kernel - freezing this one, right after it mounted
its file system. If that is your case, create a new
kernel to the main system (get the sources, compile
and install it).
You must have a kernel in the main system newer than
2.6.16-rc6 - if not, the following error message at
startup time is got: "Kernel panic - not syncing:
handle_trap - failed to wait at end of syscall, errno =
0, status = 2943".
To shutdown the virtual machine, run:
There are 3 main ways to make changes in the Linux
kernel:
Start the development, change or apply patchs to
test in the UML kernel sources. After the
changes, compile your UML kernel again:
If you are modifying some module, will be needed
to compile the module and copy it to the
file system of the virtual machine:
Create a directory somewhere (e.g.:
~/devel/my_modules), and so start creating an
source file to write the module. As example will
be used the file
"procinfo.c" to build an extern module that
prints information of the processes running in
the system.
If you wanna debug errors, or the execution flow of the
Linux kernel, you can use the GNU gdb. To debug modules
additional steps are needed, and so, described in separated
sections:
Start the UML kernel at gdb:
To debug modules as soon as they are loaded and
follow its execution we need to add its
symbols in the gdb symbol table right in the
moment that it is loaded and pointed by the
global 'modules' pointer, that represents a
list of modules loaded in the kernel, and point
the location that it was loaded (memory
address). To make these proceedings
automatically, 3 files were created (you must
change them properly with respect to the paths
of the files used):
1) umlgdb_dispatcher:
will open an xterm to start a gdb running the
kernel, and send a SIGINT to the kernel to stop
it and bring the control back to the gdb when
typed <ENTER>. <CTRL-C> will kill
gdb and the kernel, exiting.
Some commands of the GNU gdb:
The debootstrap program is part of the Debian system,
it's some shell scripts that use the glibc (and so, must be
installed), and the method to install it in another
systems is as follows:
This section describes how to fix errors at the kernel
compile time. For example, if you was using the
2.6.20-rc1 kernel, you may get the following error:
Part I - Creating and
configuring an UML kernel
$ cd /usr/src/
Download the tree:
git-clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6-linus-gittree
This will create the linux-2.6-linus-git/ tree
(in /usr/src/). This tree has the full historic
of the kernel development since 2.6.11, and is
possible to see what changed in each file
executing: '$ gitk kernel/fork.c' (for example,
to see the changes made in such file, or just '$
gitk' to see the full historic of all files). To
keep the sources of the tree updated with its
development, run the command: '$ git-pull'
inside of such directory regularly.
$ cd /usr/src/linux-2.6-linus-gittree
See the existent marks:
$ ls .git/refs/tags
Pick a desired tree more recent that have UML
support - to see witch one is this, look at the
address of the project relative its releases:
http://user-mode-linux.sourceforge.net/work/current/2.6/
and look for the last version (at the date that
this doc was made - April 02 2007 - the last
version with available patches is 2.6.21-rc1),
and so we'll get such one.
$ git-archive --format=tar --prefix=v2.6.21-rc1.uml/ v2.6.21-rc1 | (cd ../ && tar xf -)
This will extract the last candidate tree to the
stable release (at the moment of this writing)
and top of development - generating the
directory v2.6.21-rc1.uml at the directory
below (/usr/src/v2.6.21-rc1.uml).
$ make mrproper
$ make mrproper ARCH=um
To generate an appropriate .config to the UML
(the .config file is the one that defines the
configurations of the kernel):
$ make defconfig ARCH=um
To configure the kernel, run its graphic
interface with the command below. Install the
ncurses-dev (apt-get libncurses5-dev if apt is
available). Important: don't change the
configuration about the kind of the processor
(leave the default one), and also enable the
option to compile the kernel with frame
pointers.
$ make menuconfig ARCH=um
And so try to compile the kernel:
$ make linux ARCH=um
If you got some error, see how to fix errors trying to compile the
kernel at the end of this doc
$ make modules ARCH=um
This will generate the executable kernel files "linux"
and "vmlinux" at the root of kernel sources directory.
You can try to run the kernel now, waiting for it to
stop its execution with error when it try to mount the
file system with the command below:
$ ./vmlinux
If you get some error saying about file system missing,
everything is ok, there is no more thing to configure to
the kernel than its file system - next part. Despite
that, if you got the following error below:
PROT_EXEC mmap in /dev/shm/...failed: Operation not permitted
It's because the security restrictions of the system
that do not allow to be executed in shared memory. Such
restriction is resolved configuring another temporary
file to be used: first, create such directory:
$ mkdir /tmp/uml
After that, change the permissions making the user owner of
the kernel (run '$ man chmod' to know more) the owner
of such directory created (change 'user' by the user
owner of the kernel)
$ chown user.user /tmp/uml
Give full permissions to such directory:
$ chmod 777 /tmp/uml
And finally export the system variable that says what is
the temporary directory:
$ export TMPDIR=/tmp/uml
Part II - Creating and configuring a file
system
Now build a file system to be used with the UML kernel.
Following below the steps needed.
$ cd /usr/src/v2.6.21-rc1.uml/
$ mkdir root_fs
With the debootstrap install a Debian system in
such directory with the command below (if using
a Debian system at the main machine, install it
via 'apt-get install debootstrap', or see how to
download and configure the deboostrap at the end of this
document). This will create a Debian system
testing version in the directory "root_fs/" (of
167Mb)(at the moment, etch is the testing
release, but see at www.debian.org/releases/ if
it changes or if you want another one):
$ debootstrap etch root_fs/ http://ftp.debian.org/debian/
If using an AMD-64 processor in your computer,
substitute the above command by the below one:
$ debootstrap --arch amd64 sarge root_fs/ http://amd64.debian.net/debian-amd64/
$ cd /usr/src/v2.6.21-rc1.uml/
$ dd if=/dev/zero of=linux_fs.ext3 bs=1M count=200
Associate a loopback device with the file
system's file. See what devices are available
with the command '$ losetup -f' (running as
root). The use of the loopback device is
necessary to the layout of the file system (its
meta data) be correctly inserted in the file
system's file:
$ losetup /dev/loop1 linux_fs.ext3
Create a file system in the device (that was
associated with the file):
$ mkfs.ext3 -j /dev/loop1
Mount the file system of the file in a directory
to access it:
$ mkdir /mnt/vmfs
$ mount -t ext3 -o loop linux_fs.ext3 /mnt/vmfs
Copy the files of the Debian distro to your
file system created in the file:
$ cp -dpR root_fs/* /mnt/vmfs/
Now install the modules of the UML kernel to the
appropriated directories (/lib/modules) in the
file system to be used by it (still inside of
the UML kernel directory):
$ make modules_install INSTALL_MOD_PATH=/mnt/vmfs/ ARCH=um
Now configure the system starting from some of
its configuration files. First, edit the
/mnt/vmfs/etc/initab file with a text editor,
leaving only the first line:
1:2345:respawn:/sbin/getty 38400 tty1
Now edit the file /mnt/vmfs/etc/fstab:
#<file system> <mount point> <type> <options> <dump> <pass>
/dev/ubd0 / ext3 defaults 0 0
/dev/ubd1 none swap sw 0 0
proc /proc proc defaults 0 0
This says that the files /dev/ubd{0,1} of the
main system will be used by the UML kernel as
base to mount its file system. Such files
can be created automatically at the moment of
its use by devfs - in the main computer
system, configure the udev inteface, setting the
ubdX devices editing the file
/etc/udev/links.conf (in the Debian system - in
others see the udev manual of your GNU/Linux
distro), as above:
M ubd0 b 98 0
M ubd1 b 98 16
M ubd2 b 98 32
M ubd3 b 98 48
M ubd4 b 98 64
M ubd5 b 98 80
M ubd6 b 98 96
M ubd7 b 98 112
or can also be created manually if desired -
go the directory where the files
associated with devices are located ("/dev/")
and create the files with the command below:
$ cd /dev
$ for i in 0 1 2 3 4 5 6 7; do mknod ubd$i b 98 $[ $i * 16 ]; done
Create a file named 'swap_file' to be used as
swap device by the UML (of 200M here):
$ dd if=/dev/zero of=swap_file count=200 bs=1M
$ mkswap swap_file
Finally umount the file system designated to the
virtual machine:
$ umount /mnt/vmfs
Disassociate it from the loopback device:
$ losetup -d /dev/loop1
Make the owner of the file system's file the
same one of the UML kernel ('user' may be the
owner of the kernel):
$ chown user.user linux_fs.ext3
This way, the linux_fs.ext3 file will be the
file system.
$ mkdir /usr/src/uml-project/
$ cd /usr/src/uml-project/
$ cvs -d:pserver:anonymous@user-mode-linux.cvs.sourceforge.net:/cvsroot/user-mode-linux login (no password)
$ cvs -d:pserver:anonymous@user-mode-linux.cvs.sourceforge.net:/cvsroot/user-mode-linux checkout tools
$ cd tools/tunctl
$ make
1) eth0 interface in the main system: the
internal network of the main system. To
illustration this, we'll use the address
192.168.0.1 at the network 192.168.0.[1-7],
netmask 255.255.255.248, using as gateway some
another interface (e.g. eth1, eth0:1, ...) having
some valid Internet IP.
2) eth0 interface in the virtual machine with
address 192.168.1.2 at network
192.168.1.[1-7], netmask 255.255.255.248,
using as gateway the interface that follows (3):
3) tap0 interface in the main system with
address 192.168.1.1, at the network
192.168.1.[1-7], netmask 255.255.255.248,
having as gateway the above (2) interface.
#/etc/network/interfaces
auto lo eth0
iface lo inet loopback
iface eth0 inet static
address 192.168.0.1
network 192.168.0.248
netmask 255.255.255.248
broadcast 192.168.0.248
gateway 192.168.0.1
iface tap0 inet static
address 192.168.1.1
netmask 255.255.255.248
pre-up /usr/src/uml-project/tools/tunctl/tunctl -u $( cat /etc/passwd | grep user | cut -f 3 -d :)
down /usr/src/uml-project/tools/tunctl/tunctl -d tap /dev/net/tun
gateway 192.168.1.2
This way, when you desire to enable the tap0
interface in the main system (that must be done
before to start the virtual machine execution),
run 'ifup tap0', that so such file will
be read and the definitions will be made and the
commands preceding by 'up/down/pre-up' will be
executed properly.
$ mount -t ext3 -o loop linux_fs.ext3 /mnt/vmfs
#/etc/network/interfaces - (uml virtual machine "vm001")
auto lo eth0
iface lo inet loopback
iface eth0 inet static
address 192.168.1.2
network 192.168.1.248
netmask 255.255.255.248
broadcast 192.168.1.248
gateway 192.168.1.1
Edit the /etc/hosts file of the virtual machine
(in /mnt/vmfs/etc/hosts):
#/etc/hosts - network configuration (uml virtual machine "vm001")
127.0.0.1 localhost
192.168.1.2 vm001
192.168.0.1 motherhost
And setting the hostname (named "vm001" here):
$ echo "vm001" > /mnt/vmfs/etc/hostname
Umount the file system destined to the virtual
machine:
$ umount /mnt/vmfs
$ echo 1 > /proc/sys/net/ipv4/ip_forward
$ iptables -t nat -A POSTROUTING -s 192.168.1.2 -d ! 192.168.0.0/29 -o eth0 -j MASQUERADE
$ iptables -I FORWARD -s 192.168.1.2 -m state --state ! INVALID -j ACCEPT
And if desired to access it through the Internet
(e.g. ssh service running at the virtual machine
port 22), open a port in the main system (e.g.
1020 here) and redirect the connections going to
this port coming from Internet to the virtual
machine at port 22 (set EXTERN_IP properly):
$ iptables -t nat -I PREROUTING -d ${EXTERN_IP} -p tcp --dport 1020 -j DNAT --to 192.168.1.2:22
$ iptables -I FORWARD -p tcp -d 192.168.1.2 --dport 22 -j ACCEPT
Part III - The virtual machine:
running it, and the environment to develop and debug the
Linux kernel
This part describes the proceedings to use the virtual
machine made with the UML kernel and the file system
created.
Such virtual machine has been used a lot by developers
of the Linux kernel to test their changes, what in
another way (when using as the kernel of the main system)
each bug mostly breaks the whole system, bringing
the needed to reboot the machine every time. Besides
that, it's also possible to debug the kernel, and follow
the execution flow of the Linux operating system :)!
Let's see how:
$ ifup tap0
Finally execute the kernel and start the virtual
machine with the command:
$ ./linux ubd0=linux_fs.ext3 ubd1=swap_file mem=128M con0=fd:0,fd:1 con=xterm ssl=xterm eth0=tuntap,tap0 umid=vm001
If you had created the files /dev/ubdX in the main
system manually, say to disable the use of the devfs
inserting the 'devfs=none' string in the command line
above.
$ shutdown -h now
$ make linux ARCH=um
$ make modules ARCH=um
$ mount -t ext3 -o loop linux_fs.ext3 /mnt/vmfs/
$ make modules_install INSTALL_MOD_PATH=/mnt/vmfs/ ARCH=um
$ umount /mnt/vmfs/
To this case, you must have in mind to what
kernel version this module will be used, and
have available the sources of such kernel (named
'target' here). To compile the source code of
the module and improve the process, we create a
Makefile as follows (change the KERNELDIR value
to point to the path of the 'target' kernel
sources)
# Makefile
obj-m := procinfo.o
module-objs := procinfo.o
KERNELDIR = /usr/src/v2.6.21-rc1.uml/
default:
make -C $(KERNELDIR) M=$(shell pwd) modules ARCH=um
clean:
rm -f rm -f procinfo.o procinfo.ko procinfo.mod.c procinfo.mod.o Module.symvers
And so, compile it:
$ make
Mount the file system used by the UML kernel to
copy the module created
$ mount -t ext3 -o loop linux_fs.ext3 /mnt/vmfs/
$ cp /path/to/the/module/created/procinfo.ko /mnt/vmfs/usr/src/
$ umount /mnt/vmfs/
After start the virtual machine, insert the module with
'insmod /usr/src/procinfo.ko' and remove it with 'rmmod
procinfo.ko' (as root).
If your virtual machine freezes, see if there isn't
instances of the virtual machine hanging in the system
as zombie processes, what can gives error message about
resource unavailable (errno 11). Kill the processes
after restart the virtual machine with any possible
crash:
$ pkill -9 linux
$ gdb ./linux
It's needed to inform the debugger to ignore the
signals SIGUSR1 and SIGSEGV received:
(gdb) handle SIGSEGV pass nostop noprint
(gdb) handle SIGUSR1 pass nostop noprint
Now insert a breakpoint at 'start_kernel' kernel
function:
(gdb) b start_kernel
And start the execution of the kernel in the GNU
debugger:
(gdb) r ubd0=linux_fs.ext3 ubd1=swap_file mem=128M con0=fd:0,fd:1 con=xterm ssl=xterm eth0=tuntap,tap0 umid=vm001
If you desire to stop the execution of the
kernel in the gdb, a lt;CTRL-C> irá
will kill the own gdb because the terminal mode
is RAW, so, the method to be used is to open
another terminal (xterm) and send a SIGINT
signal (asking for an INTerruption of its
execution) to the main thread of the UML kernel
(change '$(whoami)' by the user running the
kernel):
$ kill -INT $(cat /home/$(whoami)/.uml/vm001/pid)
And now we can continue the debug in the gdb
(see below some examples of commands). Type 'c'
(continue) to continue the normal execution of
the kernel.
Again, kill the previous pending processes
always as needed by some crash:
$ pkill -9 linux gdb port-helper
2) umlgdb_wrapper:
load the module symbols automatically as soon as
it is inserted with the 'insmod' command in the
virtual machine and stops the kernel in the gdb
at the point of the call to the module init
function. Such file was based in the original
Chandan Kudige 'umlgdb' tool, available in the
'tools' directory of the UML project.
3) .gdbinit: to be put in
the UML kernel directory
(/usr/src/v2.6.21-rc1.uml if following the part
I); it will be read automatically by the gdb
always that this one be executed in this directory.
To start the UML kernel under gdb, gives
execution permission to the umlgdb_dispatcher
file, and run it:
$ chmod +x umlgdb_dispatcher
$ ./umlgdb_dispatcher
What happens is that when some user insert a
module with 'insmod' command inside of the
virtual machine, the gdb will break, load the
module symbol table at the memory address that
the the module was loaded, and stop with a
's'(tep) command in the gdb prompt at the line
that calls 'mod->init()', stepping to the first
line of code of the module in it's init
function when the user type <ENTER> in the
gdb prompt - a
screenshot displaying the execution of the
umlgdb_dispatcher..
Insert a breakpoint in a line of some file ...............................[b]reak x.c:105
Insert a breakpoint in a function.........................................[b]reak some_function
Insert a temporary breakpoint in a function .............................tbreak another_function
List the breakpoints made ................................................[i]nfo [b]reakpoints
Disable a breakpoint (the 2nd listed) ....................................disable 2
Don't stop at the 2nd breakpoint at the next 8 times .....................ignore 2 8
Delete the 2nd breakpoint ................................................[d]elete 2
Execute just the actual line of the program .............................[s]tep
If the next line is a function, run it, and stop at next line ............[n]ext
Run 'next' to all functions calls until the end of the actual function....finish
Continue the execution (up to the next breakpoint)........................[c]continue
List 10 or more line of codes around the actual line .....................[l]ist
Exit from GDB ............................................................[q]uit
To get more information about debug and UML kernel under
gdb, see:
Part IV - Solutions to eventual problems
Installing debootstrap in
systems others than Debian
$ cd /usr/local/src/
$ wget http://ftp.debian.org/debian/pool/main/d/debootstrap/debootstrap-udeb_0.3.3_i386.udeb
$ ar -x debootstrap-udeb_0.3.3_i386.udeb
$ cd /
$ zcat /usr/local/src/work/data.tar.gz | tar xv
Errors at UML kernel compile
time
[...]
CC mm/slab.o
mm/slab.c:3557: error: conflicting types for ‘kmem_ptr_validate’
include/linux/slab.h:58: error: previous declaration of ‘kmem_ptr_validate’ was here
make[1]: ** [mm/slab.o] Erro 1
make: ** [mm] Erro 2
$
So let's look if such error was already fixed,
checking the patches - get the 'patches.tar'
file available at:
$ wget http://user-mode-linux.sourceforge.net/work/current/2.6/2.6.20-rc1/patches.tar
Extracting will create a 'patches/' directory
with the patches inside:
$ tar xv patches.tar
See if there is some patch that has something
related to the same file in the error message -
slab.c:
$ cd patches/
$ grep -n Index: * | grep slab.c
fastcall:4:Index: linux-2.6.17/mm/slab.c
Checking the file at the line number pointed,
and analyzing the error, wee see that this patch
('fastcall' file) refers to the fix of the
error (mm/slab.c:3557: error: conflicting types
for #kmem_ptr_validate#):
$ cat patches/fastcall
# There's a mismatch between the definition and declaration of
# kmem_ptr_validate. This removes the "fastcall" from the definition.
Index: linux-2.6.17/mm/slab.c
===================================================================
--- linux-2.6.17.orig/mm/slab.c 2006-12-14 23:01:47.000000000 -0500
+++ linux-2.6.17/mm/slab.c 2006-12-14 23:07:00.000000000 -0500
@@ -3553,7 +3553,7 @@ EXPORT_SYMBOL(kmem_cache_zalloc);
*
* Currently only used for dentry validation.
*/
-int fastcall kmem_ptr_validate(struct kmem_cache *cachep, const void *ptr)
+int kmem_ptr_validate(struct kmem_cache *cachep, const void *ptr)
{
unsigned long addr = (unsigned long)ptr;
unsigned long min_addr = PAGE_OFFSET;
$
So, apply such patch at the kernel sources directory:
$ cd /usr/src/v2.6.20-rc1.uml
$ cat patches/fastcall | patch -p1
And try to recompile the kernel:
$ make linux ARCH=um
If you get another error, do the same check in the
'patches.tar' file, and apply them as necessary, and
continue to compile the kernel.
