Figure 1c
Figure 1c (Page 2) is used as a motivational experiment to show that Hadoop MapReduce is capable of interfering with the behavour of Naiad. It also show that QJump is capable of resolving this interference.
Figure 1c
Software Required
In addition to these general software requirements, we also used:
- QJump kernel module
- QJump application utility
- MSR Naiad 0.2.3 (running the patched example latency test)
- Hadoop Map-Reduce 4.5.0 (installed from Cloudera packages, instructions here)
Patches
To obtain detailed results from the Naiad latency test, we patched it to output raw statistics to standard output when complete. The patch file should be applied to the /Naiad/Examples/Naiad/latency.cs.
Physical Configuration
The basic physical network configuration is described here. In this experiment the following configuration was used:
- H1-H1 & H7 ran the (patched) Naiad nodes
- The Hadoop Map-Reduce cluster ran on H4-H6, H8-H10 and H12
Sofware Configuration
The full Hadoop Map-Reduce configuration can be found here. In experiments without QJump running, Naiad was configured as follows:
e=0; p=0; ~/Naiad/Examples/bin/Release/Examples.exe latency -t 4 -p $$p -n 4 -h 10.10.0.10:2101 10.10.0.13:2101 10.10.0.7:2101 10.10.0.6:2101 > ../data/naiad_BSFSHT301_$${p}_$${e}.bs
Where "$$p" is the host number in the range of 0-3 and "$$e" is the experiment number.
When QJump was enabled, it was configured for Naiad to run at piority level 7 with a rate limit of 5Gb/s:
bytesq=256 timeq=5 p0rate=100
To run Naiad with QJump enabled, it was run with our application utility:
e=0; p=0; ./qjau.py -p 3 -c "~/Naiad/Examples/bin/Release/Examples.exe latency -t 4 -p $$p -n 4 -h 10.10.0.10:2101 10.10.0.13:2101 10.10.0.7:2101 10.10.0.6:2101 > ../data/naiad_BSFSHT301_HADOOP$${p}_$${e}.bs"
Where "$$p" is the host number in the range of 0-3 and "$$e" is the experiment number.
Raw Data
Our original unprocessed datasets can be found here (929kB). There are 63 files in the archive. Each file contains 10,001 lines. Each line contains a value in miliseconds for the syncronization latency. The file names are of the form:
- naiad_BSFSHT301_[0-3]_[0-9].bs - Naiad running the latency synchronization example. 10 trails were run and data was collected from each of 4 machines.
- naiad_BSFSHT301_HADOOP_TCP0_[0-3]_[0-1].bs - Naiad running the latency synchronization example on a network shared with Hadoop. 5 trails were run and data was collected from each of 4 machines.
- naiad_BSFSHT301_HADOOP_TCP7_[0-3]_[0-4].bs - Naiad running the latency synchronization example on a nstwork shared with Hadoop, with QJump enabled. 5 trails were run and data was collected from each of 4 machines.
Processing and plotting the results
The processing and plotting scripts can be found here. To plot the results, run:
python plot_naiad_latency_cdfs.py \
../data/naiad_BSFSHT301_0_1.bs alone \
../data/naiad/naiad_BSFSHT301_HADOOP_TCP0_0_1.bs hadoop \
../data/naiad/naiad_BSFSHT301_HADOOP_TCP7_1_1.bs qjump \
figure1c