Getting Started

Downloading

Development versions of Skywriting are stored on Github, with the main repository available here:

$ git checkout http://github.com/mrry/skywriting.git

Check this repository out, and then compile and install it using PyPi:

$ python setup.py install

This will install the Python libraries, as well as the helper scripts sw-master, sw-worker and sw-job, which are explained below.

TODO: java setup

Concepts

Skywriting uses a master/worker architecture, and so you will need to set up a single master instance, and as many workers as you have machines/cores in your cluster. Communication between these instances happens using HTTP POST and a simple JSON RPC format.

Master Setup

The sw-master command sets up a master server listening on a port, using the fully-qualified domain name of the host.

$ sw-master [-p port]

Note that it does not listen on localhost by default, and so you should use the FQDN to address the node. It defaults to port 8080, and do not forget to open up a hole in your firewall so that other worker nodes in the cluster can also see the master.

Worker Setup

Each worker runs on its own process and port, where it listens for jobs to be dispatched to it by the master process. Workers need to be able to see the master port, and also to receive connections from the master server, so ensure your firewall is suitably configured. Workers never directly communication with each other.

To get started with a single worker, type in:

$ sw-worker -m <master uri> -p <port>

Job Submission

Now that you have a master and some workers, it is time to submit a job for parallel processing. There is a script to get you started in src/sw/helloworld.sw:

function hello(who) {
    return "hello " + who;
}

greeting = spawn(hello,["world"]);
return *greeting;

This illustrates the Javascript-like syntax of Skywriting, and spawns a task which return a static greeting string. Of course, you could return the string directly without the intermediate function, but using one lets us test that your master/worker setup is functioning correctly.

Submit the job by running:

$ sw-job -m <master uri> <script file>

Currently a lot of debugging information gets emitted, this will be quietened as development settles down.

Interactive Console

There is also an interactive console you can use to evaluate Skywriting scripts directly. The sw-console script connects to a master, just like a worker does, and drops you into an interactive shell.

$ sw-console --master http://click.local:8080
(Cmd) function hello() { return "hello world"; }
<mrry.mercator.runtime.references.SWNullReference instance at 0x101369b48>
(Cmd) x=spawn(hello,[]);
<mrry.mercator.runtime.references.SWNullReference instance at 0x1013698c0>
(Cmd) return *x;
hello world

The console itself is a wrapper to the job submission interface, so you still need to have workers connected to run the interactive commands as you type them in. If a worker is not available, the console will block until the job is completed.

Computer Laboratory