Using VirtualBox

for CS45 Linux Kernel Labs


VirtualBox is x86 and AMD64/Intel64 virutalization software. You will use virtual box for labs that involve modifying and testing the Linux. Modifying the kernel source code and compiling it will be done directly on CS machines. You will install, boot, and test your modified kernel on the virtualbox virtual machine (VM). One benefit of using VirtualBox is that if your kernel crashes you do not crash the entire machine, just the VirtualBox VM.

The VirtualBox Homepage has more information about virtualbox and about using virtualbox.

Setting Up VirtualBox on the CS lab machines
This section describes what you need to do to set up virtualbox on a CS lab machine so that you and your project partner can start working. You first need:
  1. A specific assigned CS machine on which you will run virtual box and do all your Linux kernel development.
  2. A special shared user account that you and your partner will use for just this joint lab work.

Logging In

You and your partner will be assigned a user account that you can share for CS45 work, and a specific CS Lab machine on which you will compile your Linux kernel and run virtualbox.

You will store the virtualbox machine image and the Linux kernel source code in a subdirectory in /local on your assigned machine. Even though you can only run virtualbox on the machine on which you are assigned (since it is the only one, and should be the only one, with your virtualbox machine in /local), you can ssh into your CS lab machine from any other CS lab machine (or anywhere else) and run virtualbox remotely.

For example, if the PC's name is ginger, then you would type the following to connect (-X enables X forwarding, use -Y if ssh'ing from MAC-OSX):

ssh -X your_account_name@ginger 
The -X option enables X forwarding in a secure way.

If you ssh in from outside cs, I'd recommend running virtualbox in non-gui mode and not doing X forwarding, as it can be very slow.

Steps for setting up your VirtualBox environment

One of you or your partner:

  1. ssh into the CS machine to which you have been assigned using your shared account (ex. ginger and cs45X), and choose a password for your shared account together so that both of you can remember it:
        % ssh -X cs45X@ginger
  2. Make a working_dir directory in /local that you and your partner will share. For example, if my working_dir is named tia_n_pal:
        % mkdir /local/tia_n_pal 
        % chmod 700 /local/tia_n_pal

  3. Copy over the virtualbox image files from /local/vm (a copy is also available on every CS in /scratch/vm, but it will take longer to copy over than the one in /local) into your /local/working_dir subdirectory:
        % cp -r /scratch/vm/ubuntu-12.04.3-NOINITRD  /local/tia_n_pal/. 

  4. Now you are ready to run virtualbox.

Running VirtualBox

Starting VirtualBox:

  1. Log into the PC on which your virtualbox machine is located using your shared account (ex: ginger, cs45X):
        % ssh -X cs45X@ginger 

  2. Start virtualbox:
        $ cd /local/you_n_partner/
        $ virtualbox 
        (ignore the "Error opening file for reading: Permission denied" error)
  3. The very first time you run virtualbox, you need to explicitly add the VM image you want to run. To do this:
      From the Machine menu choose Add then browse through the files systems to select this file to open:

  4. In the VM Mananger window, select the VM you want to run (there is only one choice) and then choose the start icon to start it running.

    The starting point .vbox VM contains a single linux kernel to boot. As you add more kernels, you can select the particular kernel you want to boot from the grub menu using the arrow keys and Enter.

running VirtualBox in non-gui mode

You can also run virtualbox in non-graphics mode. This is useful when remotely connecting from outside of the cs subnet, or when you want to start up virtualbox once, and then remotely log in over a period of time. In the later case, you will need to run virtualbox within a screen session as well as in non-graphics mode. Here are the steps:
$ screen    # start a screen session
$ vboxheadless -s "ubuntu-12.04.3-NOINITRD"  # run virtualbox in non-gui mode

Cntrl-a d          # detach from the screen session (machine keeps running)
note: this will always boot the default kernel (specified in the grub menu) (and it takes about 1 minute to boot). If the default kernel is not the one you want to boot, you can change the default kernel:
(1) in /etc/default/grub, set GRUB_DEFAULT to the number kernel entry you want:
    GRUB_DEFAULT=0    # the default kernel is the one in entry 0
                      # change it to be the entry you want as the default (0,1,2,...)

   you can list the kernels in which entry:
    grep menuentry /boot/grub/grub.cfg

(2) then run:
    sudo update-grub

Starting an OS inside of VirtualBox:

When virtualbox is run in gui mode, and you first boot up the system, you will see the grub menu with a list of kernels to boot. Use the arrow keys to scroll through the list and then hit the Enter key to choose your kernel. If you don't select one, after 10 seconds the default kernel (at the top of the list) is booted. If you run in non-gui mode, the default kernel boots.

After the kernel boots in your VM you can log into the console window. However, it will be easier to work on your virtual machine by ssh'ing into it. To do this specify port number 10022 and the CSmachine on which you are running virtualbox. There is one user account (swatcs) with the starting point VM, its password (swatcs.45):

ssh -p 10022 swatcs@ginger   

The virtualbox image is set up with a single user account: swatcs. The password is swatcs.45. You can change any of your user account passwords, by running the passwd command. You can also add new users to the system using the adduser command.

Shutting down or rebooting an OS:

To shut down your kernel:
$ sudo sync
$ sudo sync
$ sudo halt -p     # halt and power down
To reboot your kernel:
$ sudo sync
$ sudo sync
$ sudo reboot

Shutting down VirtualBox:

First shut down your kernel, then exit virtualbox using the menu options. If you run in non-gui mode, use CNTRL-C to exit vboxheadless after halting your kernel.

Compiling, Installing and Booting your Kernel changes in VirtualBox
You will compile and build your changes to the Linux kernel on your CS machine in /local/, and then install your built kernel on the virtualbox VM.

Getting a copy of the Linux Source

These steps only need to be done once (or whenever you want to grab a fresh copy of the code):

  1. copy the kernel source into your /local directory:
     $ cp /local/vm/linux- /local/you_n_partner/. 
  2. unzip and untar the file:
     $ cd /local/you_n_partner
     $ tar xjf linux- 
  3. copy the starting point config file into .config in linux- and run make menuconfig:
     $ cd linux- 
     $ cp /scratch/vm/config-noinitrd .config
     $ make menuconfig 	# just choose Exit with the arrow keys
  4. follow Option 1 below for building the kernel the first time.

Building the kernel

There are two options for building the kernel: If you have problems with the Option 2 build or install, I recommend doing an Option 1 build and remove and re-install the kernel packages.

Option 1: Building kernel image (and kernel header) packages

  1. Set this environment variable once in the shell you are compiling in (it will use a parallel compile and result in a faster total compile time):
    export CONCURRENCY_LEVEL=4    # anything higher than 4 doesn't help 
  2. Do this the first time, and maybe other builds, but you do not always have to do this on every build (it is like a 'make clean'):
    # NOTE: make-kpkg clean wipes out everything that has been compiled before.
    #  For the very first build you need to do this step, but for subsequent 
    #  builds you may want to not do it, and only if package builds are 
    #  failing in odd ways try doing a make-kpkg clean to re-build 
    #  everything from scratch ("from scratch" takes ~5 minutes)
    $ cd /local/you_n_partner/linux-
    $ fakeroot make-kpkg clean
  3. build the kernel packages you want to build (typically kernel_headers and/or usually kernel_image).

    # NOTE: in the linux .config file is a variable that is set to 
    #       append a local name to your kernel version (in this example
    #       it is set to "-lab3", you can change this in the .config file
    #       for other lab assignment builds 
    #         CONFIG_LOCALVERSION="-lab3"
    #       Anytime you change .config, you need to re-run make menuconfig   
    #  (a clean build will take about 5 minutes):
    $  fakeroot make-kpkg --revision=1.0 kernel_image
    # if you add new kernel header files, or change existing ones, you need 
    # to build a kernel headers package that contains your changed files too:
    $  fakeroot make-kpkg --revision=1.0  kernel_headers
    make-kpkg creates files named:
    You also can list both kernel_image and kernel_headers in a single make-kpkg command line.
    $  fakeroot make-kpkg --revision=1.0  kernel_image kernel_headers
NOTE: if this is a rebuild of a previous package, then you will need to remove the previous package before installing the new one:
# To remove a previous version of an installed package, on your VM
dpkg -r linux-image-

Option 2: Recompiling the kernel bzImage file

Set this environment variable once in the shell you are compiling in (it will use a parallel compile and result in a faster total compile time):
export CONCURRENCY_LEVEL=4    # anything higher than 4 doesn't help 

From the top-level linux source directory (cd /local/you_n_partner/linux-, do:

make bzImage
If the build is failing in odd ways (not due to errors in files you have changed), then you can try doing a make clean (and make dep) and try make bzImage again. However, it may be easier to do an Option 1 rebuild at this point.
# you don't always need to do this, but if you have changed include files  
# things don't seem to be getting built correctly do:
$ make clean

# do this only if you have added or removed #includes from existing
# source or header files (you need to rebuild the dependencies in
# this case):
$ make dep
Remember that if you change .h files you need to copy them over to your VM if test programs there need to use these .h files. You can do this by hand, but again, it is often easier to just rebuild the kernel_headers package, remove the old one, and install the new version.

Installing and booting your kernel

How you install your kernel changes depends on how you built it (Option 1 (kernel package build) or Option 2 (bzImage build)).

Option 1: Installing kernel packages:

# (a) from your CS machine, copy the kernel packages to your VM:
#     (I'm using ginger as an example CS machine name)
$ scp -P 10022 linux-image- swatcs@ginger:.
$ scp -P 10022 linux-headers- swatcs@ginger:.

# (b) Next, from within your VM, install the packages:
#     from root's home directory (or wherever you scp the files) 
$ sudo dpkg -i linux-image-
$ sudo dpkg -i linux-headers- 
note: ignore this error message when installing the headers package (it is not really an error):
Error! Your kernel headers for 2.6.34-lab3 cannot be found.

# (c) reboot your VM and choose your new kernel from the grub menu
$  sudo sync; sudo sync; sudo reboot

# (d) login and verify that the new version of your kernel rebooted 
#     check both the name of the kernel (does it have the -lab3 extention)
#     and the build date (is it a new build or old one)
$ uname -a

# note: If this is a re-install of a kernel package 
# that you have already installed (i.e. the same -lab3 flag as an 
# installed kernel package), you need to first remove the 
# old package(s), before you do the dpkg -i of the new ones: 
$ dpkg -r linux-image- 

# Use the -I option to dpkg to list info about the package file, 
# including its name (used in the -r option)
$ dpkg -I linux-image- 

Option 2: Installing bzImage and files:

# (a) from your CS machine, copy the kernel bzImage and files
      to your VM: (I'm using ginger as an example CS machine name)
$ cd /local/you_n_partner/linux-2.6.32
$ scp -P 10022 swatcs@ginger:.
$ scp -P 10022 arch/x86_64/boot/bzImage swatcs@ginger:.

# (b) ssh into your VM and install the bzImage and files
#     (replace the vmlinux... executable in /boot with the bzImage 
#      be careful that you are replacing the right one...using mv -i will 
#      give you a chance abort before overwriting the wrong one)
$ ssh -p 1022 swatcs@ginger
$ ls -l   # verify dates on bzImage and files you scp'ed over
$ sudo mv -i bzImage /boot/vmlinux-
$ sudo mv -i /boot/

# (c) run update-grub
$ sudo update-grub

# (d) sync, halt and then reboot into your new kernel
$ sudo sync
$ sudo sync
$ sudo reboot  

# if rebooting is failing, first try booting into the cs45 kernel and then do:
$ sudo sync
$ sudo sync
$ sudo halt -p  # turn off and then back on the VM and boot your kernel

An Important Note about Modifying Kernel Source Files:
We do not back-up files in each machine's /local directory. As a result, you should periodically copy files into your private cs45 subdirectory in /home, which we do back-up ( Retrieving Lost Files).

I suggest that you use git to do this:

  1. one of you should create a shared bare git repository (and share it with three users: you, your partner, and your CS45 shared account user).
  2. You and your partner should clone a copy in your private /home/you/cs45 subdirectories.
  3. Your shared CS45 account user should clone a copy into /local on your assigned machine.
As you work, periodically add, commit, and push changes to the linux kernel files you modify (and also to any user-level programs you write as part of a lab assignment).

Make sure to include in your git repo only the linux source and header files that you modify for the current lab assignment, and not the entire linux kernel source tree, which will eat up your account quota. These are the files that you will submit for each lab assignment, and the ones that you cannot recover. If you accidentally delete other kernel files (ones you aren't changing), you can always get a copy of them from the linux source starting point. If you lose files you have modified, then can then get a back-up from your git repo.

ssh'ing and scp'ing from the host PC to your VM
Use ssh to log in to your VM from the assigned CS lab machine from which you are running virtualbox:
$ ssh -p 10022 swatcs@ginger
Use scp to transfer files between your VM and your CS machine. Here are some examples (and yes, it is -P for scp and -p for ssh):
$ scp -P 10022  file.c swatcs@ginger:.
$ scp -P 10022  *.c swatcs@ginger:.          # copy a bunch of .c files
$ scp -P 10022  -r dirname swatcs@ginger:.   # recurively copy a directory
From your VM you can scp to your CS user account. Here are some examples:
$ scp file username@machinename:/home/username/cs45/labs/lab2/.
$ scp file cs45username@machinename:/local/you_n_partner/lab2/.

Debugging the Kernel
You do not have valgrind to help you debug at the kernel level, but you can use the kernel's kgdb support to remotely attach gdb to your running kernel and debug it. You can also add debugging statements to your kernel using printk.


A useful way to see what your kernel code is doing is to add debugging printk statements to print out some execution state as as it runs. printk is similar to printf, except remember that floating point values are not supported at kernel level.

printk output goes to the console and to log files on your VM in /var/log/kern.log and /var/log/syslog. You can open them in vim to printk output. Also, running tail -f on one of these files will let you see printk output as your kernel runs:

$ tail -f /var/log/kern.log
If your kernel crashes, it is likely that some of the printk output has not yet been written to the log files. As a result, you may want to jot down the last printk output on the console before rebooting. Also, be very careful about where you put printk calls; you don't want them on a code path that is executed many times per second.

printk output should also show up on the console window. If not, run this command, and then it should:

sudo dmesg -n 7


You can use the linux kernel's built-in support for kgdb to attach gdb to your running kernel and debug it. The default config file you copied over already builds the kernel image with kgdb support enabled. You need to follow some steps on boot to connect gdb running on the host machine (your CS machine) to the linux kernel VM running in VirtualBox. (this link contains some nice screen dumps showing some of these steps: KGDB with VirtualBox).

Steps to attach gdb to your VM:

  1. Run virtualbox, but do not boot a VM (or halt your VM if already running), and then configure the serial port:

    Click on your virtual machine instance in the VirtualBox Manager window (your VM should not be running), and choose Settings. Select the "Serial Ports" tab, choose Port 1, and enter the following and the choose OK:

    (1) Check the "Enable Serial Port" box
    (2) Choose COM1 for "Port Number"
    (3) Choose HostPipe for "Port Mode"
    (4) Check the "Create Pipe" box
    (5) Enter /local/me_and_pal/serial in the "Port/File Path" box  
        (where /local/me_and_pal/ is the name of your subdirectory /local on
         on your machine in which your virtualbox image and linux source are) 

  2. Start the boot of your VM, and at the grub menu choose the kernel version you want, and then enter e to edit its information. Scroll down to the linux line and append the following to it (and these are zeros not ohs):
    kgdboc=ttyS0,115200 kgdbwait
    # The result will look something like:
    linux /boot/vmlinux/.... \
    ...  rw kgdboc=ttySO,115200 kgdbwait
    Then hit F10 to continue to boot. Your VM will continue booting but not finish, printing a message:
    kgdb: Waiting for connection from remote gdb..." message

  3. At this point, in a terminal on the host machine (your CS machine) run:
    socat -d -d /local/me_and_pal/serial PTY:
    # you should see output like this:
    2014/01/31 14:13:24 socat[27748] N opening connection to AF=1 "/local/me_n_pal/serial"
    2014/01/31 14:13:24 socat[27748] N successfully connected from local address AF=1 "qp\xEE\x7E"
    2014/01/31 14:13:24 socat[27748] N successfully connected via 
    2014/01/31 14:13:24 socat[27748] N PTY is /dev/pts/16
    2014/01/31 14:13:24 socat[27748] N starting data transfer loop with FDs [3,3] and [4,4]
    Note: the /dev/pts/16 part will likely be different, and use the one that yours lists as the target in gdb.

    Do not terminate socat; it needs to be running in the background to use the pseudo terminal to connect gdb.

  4. in another terminal on the host machine (your lab machine) run gdb and attacke to the remote target:
    $ cd /local/me_and_pal/linux-
    $ gdb ./vmlinux
    Reading symbols from /local/me_and_pal/linux-
    (gdb) target remote /dev/pts/16       # or whatever the number is from socat 
    Remote debugging using /dev/pts/16
    kgdb_breakpoint () at kernel/kgdb.c:1718
    1718            wmb(); /* Sync point after breakpoint */
    At this point you can use gdb commands to debug your kernel code: set breakpoints in your kernel functions, step and next through the execution of code, print variables, ...

Troubleshooting VirtualBox and Linux
I'll post answers to virtualbox and Linux questions here.