Using QEMU for CS 45 Linux Kernel Projects

Contents

What is QEMU?
Setting up QEMU on the PCs
Running QEMU (logging in, staring qemu, staring an OS, shutting down an OS, exiting qemu)
Compiling, Installing and Booting your OS changes from with in QEMU
Debugging the kernel
Connecting to a QEMU VM from a remote machine
Troubleshooting QEMU and Linux

QEMU

QEMU is a processor emulator that we are using to boot and run the Linux 2.6.18 kernel. You will do your kernel development on the lab machines (modifying the kernel source and rebuilding it), and then you will install, boot, and test your modified kernel on the qemu virtual machine (VM). As a result, if something goes wrong with your modified kernel you do not crash the entire machine but just the qemu VM. The QEMU Homepage

Setting Up QEMU on the PCs

What you need to do to set up qemu on a PC so that you and your project partner can start working.

Logging In

You and your partner will be assigned one of the PCs in the CS Lab on which you will store your virtual qemu machine and kernel source in a subdirectory in /local that only you and your partner can access. Even though you can only run qemu on the machine on which you are assigned (since it is the only one, and should be the only one, with your qemu machine in /local), you can run ssh into your CS lab machine from any other lab machine and run qemu remotely.

For example, if the PC's name is cream, then you would type the following to connect (-X enables X forwarding, use -Y if ssh'ing from MAC-OSX): 

    % ssh -X cream
The -X option enables X forwarding in a secure way.

Steps for setting up your QEMU environment

First, get your qemu machine assignment and the qemu's root password from me or from one of the sysadmins.

Next, only one of you or your partner should do the following (you and your partner are sharing the same qemu files and kernel source, so it is up to you to coordinate changes that you make to it):

  1. ssh into the PC to which you have been assigned: 
      
    % ssh -X my_pc
  2. Make a working_dir directory in /local that you and your partner will share. For example, if my working_dir is named tia_n_pal: 
      
    % mkdir /local/tia_n_pal 
    % chmod 700 /local/tia_n_pal

  3. Copy over the linux kernel and the Debian image file (located in /scratch/vm into your /local/working_dir subdirectory 
        % cp /scratch/vm/linux-source-2.6.18.tar.bz2  /local/tia_n_pal/. 
    % cp /scratch/vm/qemu/qemuDebianEtch-newlibc.img   /local/tia_n_pal/. 

		# really just copy everything from here
    % cp /scratch/vm/qemu/*  /local/tia_n_pal/.

  4. Run easyfacl.py to set acls on your shared directory so that you and your partner can share its contents while keeping them private (easyfacl.py will prompt you to enter information about what and with whom you want to share): 
      % easyfacl.py

    Enter a space separated list of users: newhall pal 
    Enter a pathname (relative or full): /local/tia_n_pal 
    These commands will be entered
    setfacl -R -d -m user:newhall:rwX,user:pal:rwX,user  /local/tia_n_pal
    setfacl -R -m user:newhall:rwX,user:pal:rwX,user  /local/tia_n_pal
    Should I do this? (Y/n)y
    acls are set up
    press Return

    A note about copying files into directories with acls (vs. creating files in the directory): on directories with acls if you create a new file or directory in the acl directory the default acls apply to the new file or directory (this is great). If, however, you copy an existing file into a directory with acls, the acls do not apply to the copied file (this is unfortunate). There are two things you can do to set acls on a file you copy into the acl directory:

    1. copy the file into the acl directory, then explicitly add acls to the file: 
        cd /local/me_and_pal/.
  cp ~/myprivatestuff/private_foo   .
  setfacl -m user:yourname:rwx,user:your_pals_username:rwx private_foo
    2. use touch to create an empty new file in your acl directory, then overwrite it with the file you are copying: 
        cd /local/me_and_pal/.
  touch private_foo                   # creates private_foo with acls set!
  cp ~/myprivatestuff/private_foo  .

      See Safe File Sharing for Group Projects for more information on setting, listing, and removing ACLs, and easyfacl.py for more information on easyfacl.py)

  5. Now you are ready to run qemu.

Running QEMU

Starting QEMU:

  1. Log into the PC on which your qemu machine is located. For example: 
        % ssh -X cream

  2. Set up tun/tap network interface (you will be asked to enter your password): 
       % sudo tunctl -b -u your_user_name

	 # I'd do:
   % sudo tunctl -b -u newhall

   ## to see results (you should see an entry for tap0):
   % ifconfig -a
    The first time you run sudo, you will see warning message and you will be asked to enter your password.

  3. Start qemu on the PC: 
        % cd /local/you_n_partner/

    ## to start qemu with graphics mode enabled (the prefered way to run) 
    % qemu -m 256 -net nic -net tap,ifname=tap0,script=/etc/qemu-ifup qemuDebianEtch-newlibc.img 

    ## or, to start qemu without graphics mode:
    % qemu -nographic -m 256 -net nic -net tap,ifname=tap0,script=/etc/qemu-ifup qemuDebianEtch-newlibc.img
    The qemu window pops up, and you can select which kernel to boot (the default is 1.6.18-withkdb, for cs45 assignments you will add and boot your own versions of the kernel, but leave this one as an option). If you run ifconfig -a in another window, you will see that the tap0 entry now has been assigned IP 172.20.0.1

    CNTRL-ALT   frees your cursor from the QEMU window
    CNTRL-ALT-1   gives you your booted kernel's prompt
    CNTRL-ALT-2   gives you the qemu prompt (you will likely only use this when you quit qemu)
    CNTRL-PageUp   to scroll up in the qemu terminal window
    CNTRL-PageDown   to scroll up in the qemu terminal window

    Note: if you run in nographic mode, you can't see the grub menu and kdb does not work. However, you can remotely login from a machine that does not have X11 (like Windows) and run qemu using the nographic mode.

Starting an OS inside of QEMU:

When you boot up Linux inside qemu, you get a grub menu of different kernels to boot. You can use the arrow keys to scroll through the list and then hit the Enter key to choose your kernel. If you don't select one, after a few seconds the kernel at the top of the list is booted.

There are two accounts, one for root and one for user cs45. The first time you log in, log in as root (we will give you the default root password). Once you do this, change your root password to something else (run the passwd command to do this), then and change cs45's password as well ("passwd cs45"). You can add new users to the system by running the adduser command (you may want to add a user for you and your partner).

Shutting down an OS inside of QEMU:

To shut down your kernel from inside QEMU, do the following:
  1. login as root
  2. type sync; sync; shutdown -h now
To reboot your kernel from inside QEMU, do the following:
  1. login as root
  2. type sync; sync; reboot

Shutting down QEMU:

  1. first shut down your kernel

  2. then exit qemu
    If you are running in graphics mode, then from the ctrl-alt-2 qemu window, type "quit" after "Power down" to exit the qemu process. If you are running in non-graphics mode, then from another window logged into my_pc type "pkill qemu" to send qemu a SIGKILL signal.

  3. After exiting qemu, turn off the tap interface: 
       % sudo tunctl -d tap0
    From the ctrl-alt-2 qemu window, type "quit" after "Power down" to exit the qemu process. You could also kill it from your shell ("pkill qemu" sends it a SIGKILL signal).

    If you forget to turn off the tap interface, either you or your partner can run this command to turn it off before re-starting it next time you run qemu.

Compiling, Installing and Booting your Kernel changes in QEMU

You will compile and build your changes to the kernel on the CS machines, and then install your changes on your qemu machine.

These steps only need to be done once:

  1. copy the kernel source into your /local directory: 
    % cp /scratch/vm/linux-source-2.6.18.tar.bz2  /local/you_n_partner/.
  2. unzip and untar the file: 
     % cd /local/you_n_partner
 % tar xjf linux-source-2.6.18.tar.bz2
  3. apply kdb and asm-i386 bug fix patches: 
    % cd  linux-source-2.6.18
% patch -p1 < /scratch/vm/qemu/kdb-v4.4-2.6.18-common-2
% patch -p1 < /scratch/vm/qemu/kdb-v4.4-2.6.18-i386-2
% patch -p1 < /scratch/vm/qemu/asm-i386.patch
  4. copy the starting point config file into .config in linux-source-2.6.18 and run make menuconfig: 
    % cd linux-source-2.6.18 
% cp /scratch/vm/qemu/config-qemu-2.6.18-kdb-noinitrd .config
% make menuconfig 	# just choose Exit with the arrow keys, then Save
  5. follow Option 1 below for building the kernel.

Building the kernel

 There are two ways to build the kernel and which one you use depends on what you have changed when you build:

Option 1: building kernel image (and kernel header) packages

Build this way the first time you build for each assignment (additionally, any time you add or remove header or source files to the kernel you may want to rebuild your kernel packages) 
    % cd /local/you_n_partner/linux-source-2.6.18 
    % fakeroot make-kpkg clean

    #
    # NOTE: change "whatever" after --append-to-version to something more 
    #       meaningful to you.  for example, I might append -proj2 to my 
    #       kernel version with my project 2 solution
		#       (we don't use the --initrd way of building the package because
		#        it is incompatable with qemu's direct linux boot)
		#
		#       the build will take several minutes
    #
    %  fakeroot make-kpkg --revision=1.0 --append-to-version -whatever kernel_image

    #
    # if you add new kernel header files, or change existing ones, you need 
    # to build a kernel headers package containing your changed files:
    #
    % fakeroot make-kpkg --revision=1.0 --append-to-version -whatever kernel_image kernel_headers

    make-kpkg creates files named:
    ------------------------------ 
      ../linux-image-2.6.18-whatever_1.0_i386.deb 
      ../linux-headers-2.6.18-whatever_1.0_i386.deb

Option 2: rebuilding the kernel image only

Use this option for subsequent builds that do not involve adding new interface functions or header files: 
    % cd /local/you_n_partner/linux-source-2.6.18
    #
    # you don't always need to do this, but if you have changed include files
    # or things don't seem to be getting built correctly do:
    #
    % make clean

    # do this only if you have added or removed #includes from existing
    # source or header files (you need to rebuild the dependencies in
    # this case):
    #
    % make dep

    # to re-build the kernel image
    #
    % make bzImage

    the kernel image is in:
    -----------------------
    /local/you_n_partner/linux-source-2.6.18/arch/i386/boot/bzImage

Three ways to install and boot your kernel

Which one you choose depends on which option you used to build it.

Option 1: Installing kernel packages:

    # (a) from your CS machine, copy your kernel packages to your qemu machine:
    #
    % scp linux-image-2.6.18-whatever_1.0_i386.deb root@172.20.0.2:.

    # copy over kernel headers package:
    #
    % scp linux-headers-2.6.18-whatever_1.0_i386.deb root@172.20.0.2:.

    # (b) Now, as root on your qemu machine install the packages:
    #     from root's home directory (or wherever you scp the files)

    % dpkg -i linux-image-2.6.18-whatever_1.0_i386.deb
       Setting up linux-image-2.6.18-whatever (1.0) ...
       Searching for GRUB installation directory ... found: /boot/grub .
       Testing for an existing GRUB menu.list file... found: /boot/grub/menu.lst .
       Searching for splash image... none found, skipping...
       Found kernel: /boot/vmlinuz-2.6.18-whatever
       Updating /boot/grub/menu.lst ... done

    % dpkg -i linux-headers-2.6.18-whatever_1.0_i386.deb

    # (c) reboot your qemu machine and choose your -whatever kernel
    #     from the grub menu
    #
    %  sync; sync; reboot


    # (d) verify that the new version of your kernel rebooted
    #
    % uname -a
      Linux freeoszoo 2.6.18-whatever #1 Mon Aug 27 13:39:01 EDT 2007 i686 GNU/Linux

    # note: If this is a re-install of a kernel package that you 
    # have already installed (i.e. the same -whatever flag as an 
    # installed kernel package), you need to first remove the 
    # old package(s), before you do the dpkg -i of the new ones:

    % dpkg -r linux-image-2.6.18-whatever

    # You can use the -I option to dpkg to list info about the package file, 
    # including its name (used in the -r option)

Option 2: Installing just a new kernel image file (bzImage):

Copy the new bzImage file to your qemu machine and replace it with the appropriate vmlinuz file in /boot: 
    # (a) from your CS machine copy bzImage to your qemu machine 
    # from /local/your_user_name/qemu/kernel-source-2.6.8/arch/i386/boot/
    #
    % scp bzImage root@172.20.0.2:.

    # (b) on your qemu machine replace your old image with the new one:
    #
    %  mv /root/bzImage /boot/vmlinuz-2.6.18-whatever

    # (c) then upadate the grub loader 
    #
    %  update-grub 

    # (c) reboot your qemu machine and choose your -whatever kernel
    #     from the grub memu
    #
    %  sync; sync; reboot

Option 3: Direct Linux Boot (for bzImage only):

This is the easiest way to get qemu to boot your modfied kernel, however, it is not always the best way to run qemu. You can get qemu to directly boot a kernel image from the local file system using the -kernel command line option (you can add -nographic to the end of this command too): 

%  qemu -kernel linux-source-2.6.18/arch/i386/boot/bzImage -hda qemuDebianEtch-newlibc.img -append "root=/dev/hda1"
Important Note about Modifying Kernel Source Files: As you make changes to kernel source and header files, you should copy over these changed files into a private subdirectory of your home directory (like /home/your_user_name/cs45/projectX/). We do periodic backups of /home directories but do not backup /local on the CS Lab machines (i.e. your changes to files in /local/me_n_pal/linux-source-2.6.18/ are not backed up). By copying over your kernel files that you modify into your /home, you can take advantage of our file backup system to recover lost changes you make to these files (see Retrieving Lost Files). Only copy kernel source and header files that you actually modify into your home directory (don't copy all of the kernel source into your home directory, as you will run out of disk space if you do, and you can always grab a new copy of the kernel source from the CS45 starting point).

Debugging the Kernel

You can add debugging printk calls to kernel code. printk output goes to the console and to the file: /var/log/kern.log. If your kernel crashes, it is likely that some of the printk output has not yet been written to kern.log. As a result, you may want to jot down the last printk output on the console before rebooting. Also, be very careful about where you put printk calls; you don't want them on a code path that is executed many times per second.

Additionally, there are two debuggers that you can use to debug your kernel. The first is kdb, a kernel debugger that is included in the kernel build. You can invoke the kdb debugger at any point in your kernel's execution. Additionally, if your kernel panics, you will get the kdb prompt. The second is gdb. You can attach gdb to the kernel executable if you run qemu with the -s option.

Using gdb:

To use gdb, use the '-s' command line option to qemu: 

% qemu -s <the rest of the quemu command depending on how you run it>
qemu will wait for gdb to attach to the vmlinux process: 
 (qemu) Waiting gdb connection on port 1234
In another window, start gdb on the vmlinux executable (from the linux-source-2.6.18 subdirectory): 
%  gdb vmlinux
then, connect to qemu: 
 (gdb) target remote localhost:1234
to let qemu continue execution: 
 (gdb) cont
You can use CNTL-C to interrupt the kernel and get back the gdb prompt

see the GDB Guide for more information on using gdb.

Using kdb:

kdb is the kernel debugger. You must start qemu in graphics mode for kdb to work. And you need to be in the ctrl-alt-1 terminal. To use kdb At any point in the execution, you can hit the pause key twice to get the kdb prompt. Typing help with give you a list of commands. Also, if your kernel calls panic(), it will switch you into kdb mode.

When you get the kdb prompt, type help to see the set of debugging commands. A few useful ones: 

help:       list all commands
bt:         backtrace (stack) of the current process
btp pid:    bactrace of process with given pid
ps -A:      list all processes in the system
md  vaddr:  display memory contents at given virtual address
pid pid:    switch to a different process with given pid
bp  vaddr:  set breakpoint at given virtual address
bpl:        display breakpoints for current process
bc  bpnum:  clear breakpoint
go:         continue kernel from where we interupted
ss:         single step (at instruction level)
Also, see KDB Guide

Connecting to the QEMU VM using ssh or scp from the host PC

We have set up networking on QEMU so that you can only connect to/from the virtual machine to/from the host PC from which you are running qemu. When qemu starts, a private network connection is created between the PC and the qemu VM. This can be used to ssh and scp to/from the PC and qemu.

Because we do not have X installed on the qemu VM, you will likely want to ssh into your qemu VM from the machine on which you are running qemu. This is a useful way to work as it allows you to have multiple windows logged into your qemu machine, and you can leave the main qemu console for seeing printk output while working in other windows.

From the PC you can remotely log into your qemu machine using ssh (you can log in as root or as some other user account you created on your qemu machine): 

% ssh some_qemu_usr@172.20.0.2

You can copy files to/from your PC from/to the qemu machine using scp: 

# copy file foo.c from the PC to qemu machine in some_qemu_usr's proj2 subdirectory: 
% scp foo.c some_qemu_usr@172.20.0.2:/home/qemu_usr/proj2/.

# copy foo.c from qemu machine to your private cs45 subdirectory on CS machines
% scp foo.c your_usr_name@172.20.0.1:/home/your_usr_name/cs45/.

Details

Below is a detailed description of how to find IP addresses for connecting to/from the qemu VM and the lab machine on which you are running qemu.

After booting Linux on qemu, you can run /sbin/ifconfig from within the qemu window to get your VM's IP addresses. For example: 

% /sbin/ifconfig

eth0      Link encap:Ethernet  HWaddr 52:54:00:12:34:56  
          inet addr:172.20.0.2  Bcast:172.20.0.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:3765 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1980 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:5556513 (5.2 MiB)  TX bytes:155055 (151.4 KiB)
          Interrupt:9 Base address:0xc100 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:12 errors:0 dropped:0 overruns:0 frame:0
          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
The eth0 entry contains the IP address of your qemu machine: inet addr:172.20.0.2 (you use this IP to connect from the PC to your qemu machine).

Similarly, on the PC running qemu, you can run /sbin/ifconfig. For example: 

% /sbin/ifconfig -a

eth0      Link encap:Ethernet  HWaddr 00:07:E9:62:68:A5  
          inet addr:130.58.68.47  Bcast:130.58.68.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1869144 errors:0 dropped:0 overruns:0 frame:0
          TX packets:802064 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:2469282803 (2.2 GiB)  TX bytes:171113790 (163.1 MiB)
          Base address:0xdf40 Memory:fcfe0000-fd000000 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:52765 errors:0 dropped:0 overruns:0 frame:0
          TX packets:52765 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:34671242 (33.0 MiB)  TX bytes:34671242 (33.0 MiB)

tun0      Link encap:Ethernet  HWaddr 00:FF:D8:62:ED:EB  
          inet addr:172.20.0.1  Bcast:172.20.255.255  Mask:255.255.0.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2004 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3794 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:159434 (155.6 KiB)  TX bytes:5560520 (5.3 MiB)

On the PC, the entry tun0 gives the private qemu network IP for the PC (172.20.0.1 in this example). The qemu machine can ssh or scp to the PC using this IP.

Remember: you can only connect to your qemu machine from the PC on which you are running qemu.

Troubleshooting QEMU and Linux

I'll post answers to qemu and Linux questions here.