Figure 1b is used to show that Hadoop MapReduce is capable of interfering with the behavour of memcached and that QJump is capable of resolving this interference.

Figure 1b

Software Required

In addition to these general software requirements, we also used:

Patches

To obtain more detailed results from memaslap, we patched it to collect statistics in memory and dump them to standard output when complete. The patch file should be applied to the clients sub-directory of the libmemcached 1.0.15 source tree listed above. To build memaslap, run "./configure --enable-memaslap" followed by "make && sudo make install".

Physical Configuration

The basic physical network configuration is described here. In this experiment the following configuration was used:

Sofware Configuration

The full Hadoop Map-Reduce configuration can be found here. In experiments without QJump running, memcached and memaslap were configued as follows:

/usr/bin/memcached -m 64 -p 11211 -u memcache
../clients/memaslap -s 10.10.0.6:11211 -S 1s -B -T2 -c 128 > out

When QJump was enabled, it was configured for memcached and memasalp to run at piority level 3 with a rate limit of 5Gb/s:

bytesq=256 timeq=5 p4rate=100

To run memaslap and memcached at prioirty 3 with QJump enabled, they were run with our application utility:

./qjau.py -p 3 -c "../clients/memaslap -s 10.10.0.6:11211 -S 1s -B -T2 -c 128 > out"
./qjau.py -p 3 -c "/usr/bin/memcached -m 64 -p 11211 -u memcache"

Raw Data

Our original unprocessed datasets can be found here (93MB). There are 15 files in the archive. Each file contains a short header followed by rows of the form:

{GET | SET | TOTAL} , REQUEST ID, LATENCY

The file names are of the form:

Processing and plotting the results

The processing and plotting scripts can be found here. To produce the figure, we ran the following bash script:

for i in {0..4} ; do ./do_hist.sh memcachedBS301_hadoop_P3_ $i ; done
for i in {0..4} ; do ./do_hist.sh memcachedBS301_hadoop_ $i ; done
for i in {0..4} ; do ./do_hist.sh memcachedBS301_ $i ; done

cat out_memcachedBS301_[0-4].lats > out_memcachedBS301_all.lats
cat out_memcachedBS301_hadoop_[0-4].lats > out_memcachedBS301_hadoop_all.lats
cat out_memcachedBS301_hadoop_P3_[0-4].lats > out_memcachedBS301_hadoop_P3_all.lats

python plot_memcached_latency_cdfs.py memcachedBS301_all.lats idle memcachedBS301_hadoop_all.lats contended memcachedBS301_hadoop_P3_all.lats QJump