Skywriting language reference

Skywriting contains many built-in functions, and various additional functions via the standard library. This page provides a reference to the interface of every Skywriting function.

Job management functions

The core job management functions are the primitive interface for interacting with a CIEL cluster.

exec()

Executes an external piece of code, and waits for the result.

result = exec(executor, args, num_outputs);

Argument	Description
`executor`	The name of the CIEL executor to invoke.
`args`	A structure containing arguments for the executor. (Usually a dictionary.)
`num_outputs`	The number of outputs to expect.
`result`	A list containing `num_outputs` references.

spawn()

Creates a new task to compute a Skywriting function.

result = spawn(f, args);

Argument	Description
`f`	The Skywriting function to compute.
`args`	A list of parameters to pass to `f`.
`result`	A future reference to the result of `f(args)`.

spawn_exec()

Creates a new task to execute and external piece of code.

result = spawn_exec(executor, args, num_outputs);

Argument	Description
`executor`	The name of the CIEL executor to invoke.
`args`	A structure containing arguments for the executor. (Usually a dictionary.)
`num_outputs`	The number of outputs to expect.
`result`	A list containing `num_outputs` references. N.B. These references are futures.

ref()

Creates a reference from a URL.

result = ref(url);

Argument	Description
`url`	A string containing a URL. N.B. The `ref()` function presently only supports the `swbs://` URL scheme. For other URL schemes, use the `grab()` function.
`result`	A reference to the object at the given URL.

Utility functions

get_key()

Returns the value associated with the given key in dict, or default if that key is not found in dict.

result = get_key(dict, key, default);

Argument	Description
`dict`	The dictionary to be queried.
`key`	The key to be accessed.
`default`	A value to return in the case that `key` is not found in `dict`.
`result`	The value associated with `key` in `dict`, or `default` if `key` is not found in `dict`.

has_key()

Returns true if the given key is found in dict, otherwise false.

result = has_key(dict, key);

Argument	Description
`dict`	The dictionary to be queried.
`key`	The key to be accessed.
`result`	`true` if `key` is found in `dict`, otherwise `false`.

int()

Returns the integer value of the given input (typically a string).

result = int(input);

Argument	Description
`input`	The value to be converted.
`result`	The integer value of `input`. If `input` cannot be converted, an error will be generated.

len()

Returns the length of the given list.

result = len(list);

Argument	Description
`list`	A list object.
`result`	The number of elements in `list`.

map()

Applies a function to each element of the given list, and returns the resulting list. Returns a list

include "mapreduce";
result = map(f, list);

Argument	Description
`f`	A single-parameter Skywriting function, anonymous function or lambda expression.
`list`	A list object.
`result`	A list containing `f(list[i])` for each element in `list`.

mapreduce()

Applies the MapReduce algorithm to the given list. The given mapper is applied to each element in the given list, generating num_outputs outputs per element. These are then “shuffled” so that reducer i receives the ith output of each mapper. The result is a list containing the results of applying reducer to the collated num_outputs outputs of the mappers.

N.B. This function does not create tasks, partition outputs between reducers, or perform sorting on the inputs to reducers. To create tasks, include spawn() or spawn_exec() in the definition of mapper and reducer. Partitioning and sorting should be implemented in mapper and reducer respectively.

include "mapreduce";
result = mapreduce(list, mapper, reducer, num_outputs);

Argument	Description
`list`	A list object.
`mapper`	A single-parameter Skywriting function, anonymous function or lambda expression that returns a list of `num_outputs` results.
`reducer`	A single-parameter Skywriting function, anonymous function or lambda expression. N.B. This function must take a single parameter, which is a list, having the same length as `list`.
`num_outputs`	The number of outputs of each `mapper`, and hence the number of outputs of the overall `mapreduce()` function.
`result`	A list of length `num_outputs`.

range()

Returns a list of numbers within a given range.

result = range(stop);
result = range(start, stop);

Argument	Description
`start`	The first number in the range. (Optional, defaults to `0`.)
`stop`	The first number outside the range.
`result`	A list of `stop - start` integers, containing `[start, start + 1, …, stop - 1].

Wrapper functions

The wrapper functions are convenience functions for invoking executors.

environ()

Convenience function for spawning a task using the environ executor.

include "environ";
result = environ(input_refs, command_line, num_outputs);

Argument	Description
`input_refs`	A list of references to be used as inputs.
`command_line`	The command line to run, as a list of strings.
`num_outputs`	The number of outputs to expect.
`result`	A list containing `num_outputs` future references.

grab()

Fetches the given URL into the cluster, and returns a reference to the resulting value.

include "grab";
result = grab(url);

Argument	Description
`url`	The URL to fetch. N.B. Currently only the `http://` URL scheme is supported.
`result`	A reference to the object at the given URL.

java()

Convenience function for spawning a task using the java executor.

include "java";
result = java(class_name, input_refs, args,
              jar_refs, num_outputs);

Argument	Description
`input_refs`	A list of references to be used as inputs.
`command_line`	The command line to run, as a list of strings.
`num_outputs`	The number of outputs to expect.
`result`	A list containing `num_outputs` future references.

stdinout()

Convenience function for spawning a task using the stdinout executor

include "stdinout";
result = stdinout(input_refs, command_line);
result = stdinout_stream(input_refs, command_line);

Argument	Description
`input_refs`	A list of references to be used as inputs.
`command_line`	The command line to run, as a list of strings.
`result`	A list containing `num_outputs` future references.