Computer Laboratory

CHERI

Qemu-CHERI

Qemu-CHERI is an adaptation of the popular Qemu ISA emulator to implement the CHERI-MIPS instruction set. Qemu-CHERI boots CheriBSD and passes all of the CheriBSD MIPS and CheriABI tests. It implements 256-bit capabilities, as well as CHERI-128 (ISAv5, ISAv6) compressed capabilities, and "magic" 128-bit capabilities (in which architectural 256-bit capabilities can be compressed to 128 bits without loss when loaded and stored from memory). This latter mode permits easier diagnosis of software bugs associated with 128-bit-sized capabilities vs. those associated specifically with capability compression (e.g., changed alignment requirements). More information on Qemu itself can be found on the Qemu Wiki.

Building and Installing qemu-system-cheri

On FreeBSD, qemu-system-cheri can be installed using the package command:

# pkg install qemu-cheri

For 128-bit capabilities use the following instead:

# pkg install qemu-cheri128

It can also be installed using FreeBSD ports:

# cd /usr/ports/emulators/qemu-cheri; make install

For 128-bit capabilities use:

# cd /usr/ports/emulators/qemu-cheri128; make install

The latest version can be built from the source in the CTSRD-CHERI Github repository. FreeBSD users will need to use gmake instead of make when building Qemu manually:

# git clone http://github.com/CTSRD-CHERI/qemu
# cd qemu
# ./configure --target-list=cheri-softmmu --prefix=/usr/local --disable-linux-user --disable-linux-aio --disable-kvm --disable-xen --extra-cflags=-g
# make
# make install

For compressed 128-bit capabilities, add the "-DCHERI_128" flag to CFLAGS:

[...]
# ./configure --target-list=cheri-softmmu --prefix=/usr/local --disable-linux-user --disable-linux-aio --disable-kvm --disable-xen --extra-cflags="-g -DCHERI_128"

For "magic" 128-bit capabilities, add "-DCHERI_MAGIC128" flag to CFLAGS:

[...]
# ./configure --target-list=cheri-softmmu --prefix=/usr/local --disable-linux-user --disable-linux-aio --disable-kvm --disable-xen --extra-cflags="-g -DCHERI_MAGIC128"

Building CheriBSD Kernel and Disk Image

To build a suitable kernel and disk image follow the detailed instructions in the CHERI Programmer's Guide. More information can also be found on the CheriBSD web page. Use the CHERI_MALTA64 (or CHERI128_MALTA64) kernconf to build the kernel. In short, on a FreeBSD host:

  1. Install dependencies, if not already installed:
    # pkg install git gettext cmake ninja
  2. Build CHERI clang cross compiler (also see Chapter 2 of CHERI Programmer's Guide):
    % git clone git://github.com/CTSRD-CHERI/llvm.git
    % cd llvm/tools
    % git clone git://github.com/CTSRD-CHERI/clang.git
    % cd ..
    % mkdir Build
    % cd Build
    % cmake -G Ninja -DCMAKE_BUILD_TYPE:STRING=Release -DBUILD_SHARED_LIBS:BOOL=ON -DLLVM_DEFAULT_TARGET_TRIPLE:STRING=cheri-unknown-freebsd ..
    % ninja

    or for 128-bit capabilities add "-DLLVM_CHERI_IS_128=ON" to the cmake options:

    % cmake -G Ninja -DCMAKE_BUILD_TYPE:STRING=Release -DLLVM_CHERI_IS_128=ON -DBUILD_SHARED_LIBS:BOOL=ON -DLLVM_DEFAULT_TARGET_TRIPLE:STRING=cheri-unknown-freebsd ..

    Remove the incompatible std*.h and limits.h files included with Clang/LLVM:

    % rm lib/clang/3.*/include/std* lib/clang/3.*/include/limits.h
  3. Build CheriBSD kernel and disk image (also see Chapter 7 of CHERI Programmer's Guide):
    % git clone git://github.com/CTSRD-CHERI/cheribsd.git
    % cd cheribsd
    % git submodule init
    % git submodule update
    % setenv MAKEOBJDIRPREFIX /var/tmp/obj
    % make CHERI=256 CHERI_CC=/path/to/cheri-clang -j16 buildworld
    % make CHERI=256 KERNCONF=CHERI_MALTA64 -j16 buildkernel
        

    Where /path/to/cheri-clang is the path to the clang cross compiler built above.

    Or for 128-bit capabilities, use "CHERI=128" and "KERNCONF=CHERI128_MALTA64":

    % git clone git://github.com/CTSRD-CHERI/cheribsd.git
    % git submodule init
    % git submodule update
    % cd cheribsd
    % setenv MAKEOBJDIRPREFIX /var/tmp/obj
    % make CHERI=128 CHERI_CC=/path/to/cheri-clang -j16 buildworld
    % make CHERI=128 KERNCONF=CHERI128_MALTA64 -j16 buildkernel

    As root, create a disk image:

    # setenv MAKEOBJDIRPREFIX /var/tmp/obj
    # mkdir /var/tmp/root
    # make CHERI=256 CHERI_CC=/path/to/cheri-clang installworld DESTDIR=/var/tmp/root
    # make CHERI=256 KERNCONF=CHERI_MALTA64 installkernel DESTDIR=/var/tmp/root
    # make CHERI=256 CHERI_CC=/path/to/cheri-clang distribution DESTDIR=/var/tmp/root
    # echo '/dev/ada0 / ufs rw 1 1' > /var/tmp/root/etc/fstab
    # makefs -M 2g -B be /var/tmp/disk.img /var/tmp/root

    Again, use "CHERI=128" and "KERNCONF=CHERI128_MALTA64" for 128-bit capabilities:

    # setenv MAKEOBJDIRPREFIX /var/tmp/obj
    # mkdir /var/tmp/root
    # make CHERI=128 CHERI_CC=/path/to/cheri-clang installworld DESTDIR=/var/tmp/root
    # make CHERI=128 KERNCONF=CHERI128_MALTA64 installkernel DESTDIR=/var/tmp/root
    # make CHERI=128 CHERI_CC=/path/to/cheri-clang distribution DESTDIR=/var/tmp/root
    # echo '/dev/ada0 / ufs rw 1 1' > /var/tmp/root/etc/fstab
    # makefs -M 2g -B be /var/tmp/disk.img /var/tmp/root
  4. Copy the disk image (/var/tmp/disk.img) and kernel (/var/tmp/root/boot/kernel) to the host system that has qemu-system-cheri installed. Pre-built disk images and CheriBSD kernels can be found at: https://www.cl.cam.ac.uk/research/security/ctsrd/mips64-packages/qemu-cheri-images/

Running CheriBSD with Qemu-Cheri

CheriBSD Startup and Regression Testing

Using the kernel and disk image created above you can boot CheriBSD into multiuser and run the CheriBSD regression tests. Use the 'malta' system model (-M malta) with 2G of system memory (-m 2048):

$ qemu-system-cheri -M malta -kernel ./kernel -nographic -hda ./disk.img -m 2048
[...]
Copyright (c) 1992-2015 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-CURRENT #0 199c896(master)-dirty: Wed Jan  6 16:38:14 UTC 2016
    [Javascript required]:/var/tmp/obj/mips.mips64/home/sson/src/cheribsd/sys/CHERI_MALTA64 mips
gcc version 4.2.1 20070831 patched [FreeBSD]
Preloaded elf kernel "kernel" at 0xffffffff806bbb10.
CHERI: compiled for 256-bit capabilities
real memory  = 2147483648 (2097152K bytes)
[...]
FreeBSD/mips (cheri) (ttyu0)
login:

At this point you can login as the root user (no password) and run the CheriBSD regression tests:

login: root
[...]
root@:~ # cheritest -a
[...]
TEST: test_string_memmove_c: Test explicit capability memmove
PASS: test_string_memmove_c
SUMMARY: passed 142 failed 1 (1 expected)
root@:~ # cheriabitest -a
[...]
TEST: test_string_memmove_c: Test explicit capability memmove
PASS: test_string_memmove_c
SUMMARY: passed 142 failed 1 (1 expected)
root@:~ #

Note that Qemu can be exited using the key sequence "control-a x" at any time.

Instruction, Register and Memory Tracing

Qemu-Cheri also has support for simple instruction, register and memory tracing. This is very useful for debugging and can be turned on (and off) by the command-line when Qemu is started or via the Qemu monitor. Note that turning on instruction tracing will add a lot of overhead to the Qemu emulation and can generate very large trace log files.

In addition to the disassembled instructions the trace includes changes to registers and memory. Here is a small sample of what the trace log looks like (e.g. start of an exception handler):

0xffffffff80000194:  csetdefault        c30
    Write C00|v:1 s:0 p:7fffffff b:0000000000000000 l:ffffffffffffffff
             |o:0000000000000000 t:0
[...]
0xffffffff8051d0ec:  sd a3,296(k1)
    Memory Write [c000000000143588] = 000000016004f5a0
0xffffffff8051d0f0:  cgetcause  k0
    Write k0 = 00000000000006ff
0xffffffff8051d0f4:  daddiu     t4,k1,608
    Write t4 = c0000000001436c0
0xffffffff8051d0f8:  csc        c28,t4,192(c30)
    Cap Memory Write [c0000000001436c0] = v:1 tps:00000000ffff00fa
    c:0000000000000000 b:0000000000000000 l:0000010000000000

Starting Instruction Tracing on Start Up

To start instruction tracing on start up add "-D <logfile> -d instr" to the Qemu command-line. For example:

$ qemu-system-cheri -M malta -kernel ./kernel -nographic -hda ./disk.img -m 2048 -D /var/tmp/instr.log -d instr

Starting and Stopping Instruction Tracing via Qemu Monitor

Instruction tracing can also be started and stopped using the Qemu Monitor. To do this toggle into the Qemu Monitor using the "control-a c" key sequence. At the Qemu Monitor prompt, to start instruction tracing, do:

(qemu) logfile /var/tmp/instr.log
(qemu) log instr

Toggle back to the console with another "control-a c" key sequence.

To stop instruction tracing using the Qemu Monitor:

(qemu) log none

qemu-system-cheri also can create instruction traces that are compatible with the libcheritrace format that is used by the tracedump utility and CheriVis. To generate libcheritrace instruction traces use '-d cvtrace' instead of '-d instr'. For example:

$ qemu-system-cheri -M malta -kernel ./kernel -nographic -hda ./disk.img -m 2048 -D /var/tmp/cvtrace.bin -d cvtrace
[...]
$ tracedump -t /var/tmp/cvtrace.bin
[...]