Prazor/VHLS Temporary User Manual

References

The Prazor Virtual Platform is a simulator is written using SystemC TLM 2.0 sockets and can model a number of CPU architectures (x86_64, ARM, MIPS, OpenRISC).

The Zynq model within Prazor covers the ARM cores, L1 and L2 caches, GIC, SCU, DRAM, SD Card and all peripherals and timers needed to boot linux with UART input and output. The Ethernet is currently missing, as is the Epiphany chip, Neon, USB, the FPGA and TCM. Floating point is being added. Jazelle cannot be added since it is secret. All of these components are modelled as TLM 2.0 blocking SystemC classes with loose timing and quantum keeper.

Prazor does not directly model the FPGA fabric (programmable logic) of the Zynq. Instead, one route is to use HLS to generate the FPGA design and to use a SystemC output from the HLS tool for the ESL model (ie. link it into the Prazor binary).

Computer Laboratory Local Use

Static snapshots, called pvp_nn, of the simulator code are also installed on the Computer Lab file server and can copied or linked to

/usr/groups/han/clteach/btlm/current

If updates are needed, revised/updated copies will be maintained at

/usr/groups/han/clteach/btlm/pvp_xx

A first step is to run the pre-built simulator with a pre-compiled binary image and then look at the structure of what you have run. Later steps will be to modify both halves and confirm proficiency with this. This is mostly a matter of getting your environment variables set up correctly.

Making a simple first run of the pre-built vhls binary.

The virtual platform can be run from the command line.

At the Computer Laboratory you will need the following set ups:

export CLTEACH=/usr/groups/han/clteach
export TARCH=ARM32
export PRAZOR=$CLTEACH/btlm/current

The pre-built simulator is here:

export VHLS=/usr/groups/han/clteach/btlm/current/vhls/src/vhls

The pre-built hello-world program as an ELF binary for ARM is here:

/usr/groups/han/clteach/btlm/current/vhls/images/hellow-world/hello

It is easiest to run under make, but a manual command line run should also work. You'll need something like

   $VHLS \
       -dram-system-ini /usr/groups/han/clteach/btlm/current/vhls/src/dramsim2/dist/system.ini.example \
       -dram-device /usr/groups/han/clteach/btlm/current/vhls/src/dramsim2/dist/ini/DDR3_micron_8M_8B_x16_sg15.ini \
       -cores 1 -tracelevel 0 -global-qk-ns 1 -no-caches -image ./hello -name vv \
       -- red yellow green blue well done milos 

The command line is divided into two sections using the -- separator. Three sections with two of these separators are sometimes used. The part before the first separator is processed by the Prazor simulator. The part afterwards is passed in to the running program as its argv command line. Generally, this program consists of a wrapper (energyshim.c) that splits the command line again at the second -- separator with the final part being used by the application proper and the earlier part controlling logging instrumentation.

You will get a UART output spool file written to your working directory and also an energy use report from the TLM POWER3 library if included in the platform version.

Before open public release, please :

1. Get an xparch user ID and password from DJG or MP

2. Login via the web interface (password required) to    http://phabricator.xparch.com
   2a and change your password.
   2b Install your public key on phabricator

3. Use ssh-agent to put your private key in scope and then

   git clone ssh://vcs@phabricator.xparch.com:22/diffusion/P/vhls.git


4. cd to vhls and follow the README.md

Owing to the nature of git, it is possible for local users to clone their own git repo from the checkout in /usr/groups/han/clteach/btlm/current using the appropriate git command (someone please let me know what it is) or just doing a simple cp -ra of that folder so that you get the hidden hit repo management files.

Manual Linking Makefile

The Prazor Vitual platform is normally built with automake and the platform of interest is then loaded as a DLL (.so) file. But this can be very tricky to use if you have not used it before and so a simple manual Makefile is given here that you can adapt if you prefer. This makefile also helpfully gives you a stong idea of the overall structure and what automake is achieving.

#
# VHLS Prazor hand-crafed Makefile for SystemC TLM Zynq-style models.
#
# People often have problems with automake, so a simple manual Makefile is all we really need.
# This version assumes the C++ src files have already been compiled to .o form
# This version without TLM POWER3 and Speedo
#

VHLSDIR=/usr/groups/han/clteach/btlm/pvp10/vhls

CPP=g++

CPPFLAGS=-O2 -std=c++0x \
   -DTARCH=ARM32 \
   -DVHLS_STATIC=1 \
   -DSC_CPLUSPLUS=199701L -DSC_DISABLE_API_VERSION_CHECK=1 \
   -I/usr/groups/han/clteach/systemc/systemc-current/include \
   -I/usr/groups/han/clteach/systemc/systemc-current/include/tlm_core/tlm_2 \
   -I/usr/groups/han/clteach/boost/boost_1_48_0  \
   -I$(VHLSDIR)/src

LDFLAGS = -L$(SYSTEMC)/lib-x86_64 -lsystemc -lpthread

# Simulator infrastructure
PRAZOR_SRC= \
	$(VHLSDIR)/src/tenos/argv_backdoor.o \
	$(VHLSDIR)/src/tenos/cpu_busaccess.o \
	$(VHLSDIR)/src/tenos/generic_branch_predictor.o \
	$(VHLSDIR)/src/tenos/io_backdoor.o \
	$(VHLSDIR)/src/tenos/MpHash.o \
	$(VHLSDIR)/src/tenos/tenos.o \

# The platform of interest
PARALLELLA= \
	$(VHLSDIR)/src/platform/arm/zynq/parallella/zynq.o 

# Generic ARM IP blocks
ARM_A9_ZYNQ = \
	$(VHLSDIR)/src/arm/arm_abt.o \
	$(VHLSDIR)/src/arm/arm_ccache.o \
	$(VHLSDIR)/src/arm/armcore_tlm.o \
	$(VHLSDIR)/src/arm/arm_cortex_a9.o \
	$(VHLSDIR)/src/arm/arm_cp14.o \
	$(VHLSDIR)/src/arm/arm_cp15.o \
	$(VHLSDIR)/src/arm/armdis.o \
	$(VHLSDIR)/src/arm/armisa.o \
	$(VHLSDIR)/src/arm/arm_L2Cpl310.o \
	$(VHLSDIR)/src/arm/arm_mmu.o \
	$(VHLSDIR)/src/arm/arm_scu.o \
	$(VHLSDIR)/src/arm/armthumb.o \
	$(VHLSDIR)/src/arm/arm_timers.o \
	$(VHLSDIR)/src/arm/gic_arm_tlm.o \
	$(VHLSDIR)/src/arm/sclr_arm_tlm.o 

# Static memories and caches
MEMORYSRC= \
	$(VHLSDIR)/src/memories/base_mmu_tlm.o \
	$(VHLSDIR)/src/memories/ccache.o \
	$(VHLSDIR)/src/memories/dram64_cbg.o \
	$(VHLSDIR)/src/memories/generic_tlm_mem.o \
	$(VHLSDIR)/src/memories/memloaders.o \
	$(VHLSDIR)/src/memories/scu.o \
	$(VHLSDIR)/src/memories/secondary_cache_with_directory.o \
	$(VHLSDIR)/src/memories/sram64_cbg.o 

# DRAM simulator
DRAM2SIMSRC= \
	$(VHLSDIR)/src/dramsim2/dramsim_sc_wrapper.o \
	$(VHLSDIR)/src/dramsim2/dist/Bank.o \
	$(VHLSDIR)/src/dramsim2/dist/BankState.o \
	$(VHLSDIR)/src/dramsim2/dist/BusPacket.o \
	$(VHLSDIR)/src/dramsim2/dist/CommandQueue.o \
	$(VHLSDIR)/src/dramsim2/dist/IniReader.o \
	$(VHLSDIR)/src/dramsim2/dist/MemoryController.o \
	$(VHLSDIR)/src/dramsim2/dist/MemorySystem.o \
	$(VHLSDIR)/src/dramsim2/dist/Rank.o \
	$(VHLSDIR)/src/dramsim2/dist/SimulatorObject.o \
	$(VHLSDIR)/src/dramsim2/dist/TraceBasedSim.o \
	$(VHLSDIR)/src/dramsim2/dist/Transaction.o 

# GDB stub via RSP Protocol
GDBSRC= \
	$(VHLSDIR)/src/gdbrsp/gdbrsp.o \
	$(VHLSDIR)/src/gdbrsp/GdbServerSC.o \
	$(VHLSDIR)/src/gdbrsp/RspConnection.o \
	$(VHLSDIR)/src/gdbrsp/RspPacket.o \
	$(VHLSDIR)/src/gdbrsp/Utils.o \
	$(VHLSDIR)/src/gdbrsp/vhls_soc_debug.o

# Bus Components
BUSSRC= \
	$(VHLSDIR)/src/bus/busmux64.o

# Input and Output Devices
IO= \
	$(VHLSDIR)/src/io/sdio_cbg.o \
	$(VHLSDIR)/src/io/uart64_cbg.o


all:
	$(CPP) -o vhls $(CPPFLAGS) $(LDFLAGS) $(PRAZOR_SRC) $(IO)  $(BUSSRC) \
              $(MEMORYSRC) $(GDBSRC) $(DRAM2SIMSRC) \
              $(ARM_A9_ZYNQ) $(PARALLELLA) $(VHLSDIR)/src/vhls.cpp \
              $(LDFLAGS)
	echo "The excutable file vhls should now exist in the current directory"
	ls -l
# eof

You will find a copy of this Makefile in

/usr/groups/han/clteach/btlm/pvp10/ManualMake

Using Automake to Compile Prazor

You will first need a copy of the source files

Set ups

You only need to set BOOST, LDFLAGS, TLM_POWER3 and CXXFLAGS for compiling the simulator. For running it you just need the PRAZOR and TARCH settings as mentioned above.

The following setup is generally what is needed:

export CLTEACH=/usr/groups/han/clteach
export BOOST=$CLTEACH/boost/boost_1_48_0
export BOOST_ROOT=$BOOST
export SYSTEMC=$CLTEACH/systemc/systemc-current
export TLM_POWER3=$CLTEACH/tlm-power3
export LDFLAGS="-L$SYSTEMC/lib-linux64 -L/usr/local/lib -L$TLM_POWER3/src/.libs"
export CXXFLAGS="-I$SYSTEMC/include/ -I$BOOST_ROOT -I$SYSTEMC/include/tlm_core/tlm_2 -I$TLM_POWER3/include -g -O2"
export TARCH=ARM32
export PRAZOR=$CLTEACH/btlm/current

Plugging your own IP Block Models into the Virtual Platform

As said, one purpose of building your own copy of the simulator is to explore the performance of a program on different architectures, such as with different cache sizes. You also need to recompile the simulator to include your own IP blocks.

New blocks should be instantiated inside the platform configuration C++ file, such as

          /usr/groups/han/clteach/btlm/current/vhls/src/platform/arm/zynq/parallella/zynq{.cpp,.h}

You need to get to the point where you have your own copy of this file where you have made modifications. You can either make links to the existing versions of all the other files in the simulator or have your own copies of them too.

To instantiate a simple peripheral for programmed I/O access, first look at the way an existing I/O device is wired in, such as the UART. You can see it is connected to the I/O bus with the following line

             BUSMUX64_BIND(busmux0, UARTS[uu]->port0, start, UART_SPACING)

Using a base address of 0x43c00000 will mean the self-same code can run on the virtual ARM as the real ARM.

There may be code relating to ETHERNET_CRC in the platform file: if so, this is a placeholder that is otherwise unused and you can remove it or else adapt it as a basis of your own.

Note that Prazor uses an extended generic payload, not the default one provided in SystemC. Hence, your device will need to instantiate a simple_target_socket where an extra type argument is passed to the socket constructor to that defines the payload, such as

    tlm_utils::simple_target_socket port0;

DRAM Data Sheets.

The -cores n flag specifies how many CPU cores are created.

The -self-starting-cores n flag specifies how many CPU cores are in run mode at reset time. For Zynq linux this option should not be used, thus leaving the setting at its default of 1 because the linux kernel uses a store to the system controller block from the first CPU to enable the subsequent core(s).

The -image flag specifies an ELF binary file to be loaded into the DRAM model.

The -tracelevel n flag turns on a given level of tracing. n is in the range 0 to 9 with 0 being all tracing off. Multiple tracelevel flags can be used, separated by -watch flags.

The -watch B +N flag takes a hex base address and a hex length N to define a region to watch. This region is watched at the tracelevel of the previous tracelevel flag. Typically you might set tracelevel 9, then define some watch regions, then set tracelevel back to zero so that no tracing is done outside the watched regions.

The -name vv flag defines the name to be used as root segment in the name of output files. This is useful to distinguish output for sequences of experiments run by the same makefile in a common folder.

The -dram-system-ini filename.ini sets up the DRAM simulator parameters. Just use the standard file provided always.

The -dram-device datasheet.ini sets up the DRAM type in use. Each type of DRAM has different energy and performance details. For the Parallella card, please use XXXX? TBD.

The -wait-for-debugger causes the simulation not to start until a GDB session is remotely attached.

The -no-caches flag is implemented in abench1.h to not instantiate the caches and to connect the ARM cores directly to the DRAM. A variant is perhaps needed to skip either L1 or L2 but not both. With the -no-caches flag the caches are removed from the design and their static power disappears. The processor will run at the speed allowable by the main store, which will be slow if this is DRAM without caches.

The -no-harvard flag is implemented in abench1.h to only instantiate a single L1 cache per core, not split I and D caches. Variants to easily control the cache size and associativity could be easily implemented.

A 'backdoor' is an artefact not present on the real hardware that can be used on the simulator for access to a simulator feature, such as logging or exiting the simulator.

Some backdoors are available, for access to simulator argv and argc and getenv.

Some SWI instructions, using codes not used for linux, are hardwired as backdoors for writing a character without going via a UART or other output model, exiting the simulator and getting core instruction counts without using a platform-specific PMU set up.

 
// ARM trigger instruction trace - print next 100 instructions
   asm volatile (" swi #203":: );                             

Prazor Simulator Structure

For the P35 course, the file that defines the architecure you are modelling is

$PRAZOR/src/platform/arm/zynq/parallella/zync{.h,.cpp}    

You should take a good look at that file since you must make minor edits to it when you add your own IP blocks to the system.

Prazor Loose Timing

Prazor uses Loose Timing. The LT quatum is set on the command line. If it is set to a value close to a bus-cycle period, then transaction order is guaranteed and it degenerates to an approximately-timed model.

Q. When I use sc_delay_ += std_delay, I get:

    Assertion `unused_delay == SC_ZERO_TIME' failed.

A. The prazor model does not normally use the textbook sc_delay value. That is why it is named sc_delay_ throughout, to indicate that the standard field should not be directly used. Instead Prazor is normally compiled to use prazor_gp_t which has its own delay field called ltd of type lt_delay. The advantage of this type is it allows forks and joins in the loosely-timed trajectory, with a MAX function being applied correctly at the joins. The coding style used throughout is to use the AUGMENT_LT_DELAY macro in place of the manual addition you have used. The macro augments the correct delay variable according to the coding style used.

Compiling Application Programs Bare Metal

For simplicity and for clear results, rather than booting an operating system on to the simulator as well as a application of interest, it is often better to run the application 'bare metal'. For bare-metal operation you statically link a few tiny device drivers or backdoor drivers with the application and just load the combined binary into the simulator.

The gdb debugger can be connected to a running Prazor simulation for single stepping, inspecting and changing memory and setting break and watchpoints. This uses the RSP protocol over a TCP remote connection.

You should start the simulator with the -wait-debugger option. The simulator will print which port it is listening on and this is commonly localhost:9600.

Load your binary ELF file into the debugger using the gdb file command before remotely connecting.

  $ /usr/groups/han/clteach/arm-gdb/gdb-7.8/gdb
  GNU gdb (GDB) 7.8
  (gdb) target remote :9600
  Remote debugging using :9600
  0x00000000 in ?? ()
  (gdb) file dfsin % The path to your binary ELF image
  Reading symbols from dfsin...done.
  (gdb) cont
  Control-C
  (gdb) where
  [Remote target] #1 stopped.
  (gdb) where
  #0  puthex64 (leadingz=leadingz@entry=1, d0=125116699948, f=)
      at /home/djg11/d320/prazor-virtual-platform/vhls/src/crt/prlibc/prlibc.c:1052
  #1  0x00041ce8 in locked_printf (format=0x4516c "input=%016llx expected=%016llx output=%016llx ok=%i\n", 
      format@entry=0x44fc0  ",2", poi=..., poi@entry=...) at /home/djg11/d320/prazor-virtual-platform/vhls/src/crt/prlibc/prlibc.c:274
  #2  0x00041f68 in printf (format=0x4516c "input=%016llx expected=%016llx output=%016llx ok=%i\n")
      at /home/djg11/d320/prazor-virtual-platform/vhls/src/crt/prlibc/prlibc.c:342
  #3  0x000418b4 in bm_main (verbosef=verbosef@entry=7) at dfsin.c:177
  #4  0x00043af0 in main (argc=7, argv=0xfffe0004) at ../energyshim.c:200
  #0  mul64To128 (a=12105675798371893248, b=7613535337020653568, z0Ptr=0x41134 , z1Ptr=0x3bff50) at softfloat-macros:186
  #1  0x00173a24 in ?? ()
  (gdb) info registers
  r0             0x0      0
  r1             0x100    256
  r2             0xe0001000       3758100480
  r3             0x9      9
  r4             0x2189552c       562648364
  r5             0x1d     29
  r6             0xa      10
  r7             0x452b0  283312
  r8             0xe801c  950300
  r9             0x1      1
  r10            0x0      0
  r11            0x1000000        16777216
  r12            0x45198  283032
  sp             0x3bff20 0x3bff20
  lr             0x43954  276820
  pc             0x430b4  0x430b4 
  cpsr           0x8000005f       2147483743
  (gdb)  x/4x 0x44dc4
  0x44dc4 <__udivdi3+208>:        0xe0a55005      0xe2533001      0x1afffff8      0xe0948000
  (gdb)  x/4i 0x44dc4
  => 0x44dc4 <__udivdi3+208>:     adc     r5, r5, r5
     0x44dc8 <__udivdi3+212>:     subs    r3, r3, #1
     0x44dcc <__udivdi3+216>:     bne     0x44db4 <__udivdi3+192>
     0x44dd0 <__udivdi3+220>:     adds    r8, r4, r0
  (gdb) print num_keys  -- display a global variable
  $1 = 1000
  (gdb) up
  #4  0x00041620 in estimateDiv128To64 (a1=0, b=, a0=4815960295168657408) at softfloat-macros:217
  217       z = (b0 << 32 <= a0) ? LIT64 (0xFFFFFFFF00000000) : (a0 / b0) << 32;
  (gdb) print z -- try to print a local var (but it was in a register)
  $2 = 
  (gdb) print a0  -- print a local var that still exists
  $3 = 4815960295168657408
  (gdb) info threads
    Id   Target Id         Frame   -- We only have one CPU core (sim commands line was -cores 1) 
   * 1    Remote target     0x00041620 in estimateDiv128To64 (a1=0, b=, a0=4815960295168657408) at softfloat-macros:217

Start gdb and give the command "target remote localhost:9600" You can leave out the machine name or the localhost name if you run the simulator and debugger on the same machine.

Ideally you need to use an ARM version of gdb to connect. Otherwise you will see registers with x86 names being displayed.

A copy of gdb for ARM gdb is installed at the Computer Laboratory here:

/usr/groups/han/clteach/arm-gdb/gdb-7.8/gdb

Switching between cores: the threads commands of gdb have been adapted/abused to enable you to connect to different cores with a multiprocessor. Further details of how to switch between cores and the the spEEDO energy API will be added here ...

Build gdb for ARM with this arg to configure
./configure --target=arm-linux-gnu

Useful gdb commands:
  target remote localhost:9600
  info registers                -- show register contents
  x/16i $pc                     -- display next 16 instructions disassembled
  x/32x $sp                      -- display memory from sp upwards for 32 words
  file /home/djg11/d320/prazor/trunk/chstone/dfsin/dfsin -- load the symbol table from an ELF binary for symbolic debugging
-- If it says no debugging symbols then it is stripped binary
  load                          -- download a binary (or the current binary) - better to load from the command line -image arg?
  cont                          -- continue execution
  break 0x44cca                 -- set a break point
  where                         -- give stack backtrace trace
  up                            -- change current stack frame by moving up one stack frame (towards caller) 
  down                          -- the reverse, move down one stack frame
  print num_keys                -- display a global variable or local var in current stack frame
  info threads -- show threads (may show hardware cores instead using Prazor)
  thread 2 -- switch thread/core


  set $pc = 0x485               -- perform a jump (not working prior to 18th feb 2015).
  stepi                         -- run one instruction only (not working 18th feb 2015 but being fixed).

// useful additional commands for debugging the debugger:
 set debug remote 1
 set debug target 1

Building Code that can run on both the simulator and the real cards

The gcc C compiler defaults to generating Thumb-2 machine code and hardware floating point. The Thumb modes and hardware floating point on the simulator do not currently work (are just being debugged and should work in the near future), so meanwhile, to run the same binary code on both systems requires coercing gcc to avoid to only use the old ARM32 mode. The relevant flags are:

   -marm -mfloat-abi=soft

With these flags, code compiled on the workstatstion or on the real card is interchangeable and .o files can freely be copied backwards and forwards using scp.

However, note the installed libraries, libc and libgcc, on the Parallella cards also use Thumb mode so, for detailed performance comparison (and to avoid linker errors about VPF) you should avoid using these and instead use your own compiled versions of these too (or the ones on the links in the detailed documentation at the bottom of this section).

The difference between application binaries targeted running bare metal on the simulator and running on linux on the real card are mainly to do with console I/O. The I/O paths are very different and so performance comparions should not be made. Also, they are incompatible and you need to swap the system calls used on the real card with direct calls to the UART device driver as used on the simulator.

The best way to redirect the I/O is to link the same .o application and library files with some slightly different I/O shims, as in the barelift shim example below. Essentially you want to replace uart64_driver.o with bareliftshim.o. You should link using the ld program (not using gcc as a linker) since this will give you complete control over which kickoff code and libraries are included. Also, always check what you have made using objdump -d as a disassembler.

For example, to run the dfsin standard test one would link as follows using the same binaries as ran on the simulator except for the barelift pair.

        LIBGCC=/home/linaro/djg11/libgcc-nothumb.a
        ld -o a.out bareliftcrt.o bareliftshim.o dfsin.o prlibc.o $(LIBGCC)
        objdump -d ./a.out > dis
        # If you see any 16 bit (4 hex digit) opcodes in the disassembly they you are using thumb code.
        ./a.out

Once Thumb modes are working we can compare energy and performance with and without them. (It is working now - thanks Milos - April 2015).

Running S/W on a Physical Zynq Card

Running Linux on the Prazor Parallella/Zynq Simulator

Video Screen Capture of Interactive Linux Shell Session

Energy Budgets and Modelling: Modelled versus Measured Comparison and Results

There are various sources of discrepancy between the Prazor energy results and the measurements on parcard1.

Firstly, the implementation of the sPEEDO interface on Prazor returns the core's energy account and the whole system energy account. And on the physical probe attached to parcard1 it returns the whole system energy and the energy used by the Zynq core. So the figures reported would not directly agree even if both sides were completely accurate. Note: The Zynq core contains both processors, the L2 cache + GIC + SCU and the programmable FPGA logic.

Secondly, the Prazor system has features missing from its model, notably the DRAM driving pads and the Ethernet PHY energy.

Ethernet Power

The Ethernet PHY on the real card takes a lot of energy that is not modelled (currently) in Prazor. This has four modes of operation that take different amounts of power. If you unplug the 100 Mbps Ethernet cable and log in via the UART instead the power saving is about 160 mW. The actual measurements of the 5 volt supply current were:

    Ethernet unplugged   385mA
    10  Mb/s             410mA
    100 Mb/s             425mA
    1 GB/sec             470mA

The power consumed by the PHY can be computed by multiplying by 0.001 to convert to amps, then multiply by 5 to convert to Watts and finally take of 20 percent to account for the SMPSU efficiency inconverty from 5 to 1.8 volts.

Another 40mW may be accounted by the LED indicators which will go off when the Ethernet is unplugged.

The Ethernet MAC hard macro is quoted in the Xilinx datasheet as taking 15 mW.

DRAM Power