next up previous contents index
Next: 4.4 Application Program Interfaces Up: 4. Miscellaneous Concepts Previous: 4.2 Condor's Checkpoint Mechanism   Contents   Index


4.3 Computing On Demand (COD)

Computing On Demand (COD) extends Condor's high throughput computing abilities to include a method for running short-term jobs on instantly-available resources. Support for COD was added to Condor in version 6.5.2.

The motivation for COD extends Condor's job management to include interactive, compute-intensive jobs, giving these jobs immediate access to the compute power they need over a relatively short period of time. COD provides computing power on demand, switching predefined resources from working on Condor jobs to working on the COD jobs. These COD jobs (applications) cannot use the batch scheduling functionality of Condor, since the COD jobs require interactive response-time. Many of the applications that are well-suited to Condor's COD capabilities involve a cycle: application blocked on user input, computation burst to compute results, block again on user input, computation burst, etc. When the resources are not being used for the bursts of computation to service the application, they should continue to execute long-running batch jobs.

Here are examples of applications that may benefit from COD capability:

The way Condor helps these kinds of applications is to provide an infrastructure to use Condor batch resources for the types of compute nodes described above. Condor does NOT provide tools to parallelize existing GUI applications. The COD functionality is an interface to allow these compute nodes to interact with long-running Condor batch jobs. The user provides both the compute node applications and the interactive master application that controls them. Condor only provides a mechanism to allow these interactive (and often parallelized) applications to seamlessly interact with the Condor batch system.

4.3.1 Overview of How COD Works

The resources of a Condor pool (nodes) run jobs. When a high-priority COD job appears at a node, the lower-priority (currently running) batch job is suspended. The COD job runs immediately, while the batch job remains suspended. When the COD job completes, the batch job instantly resumes execution.

Administratively, an interactive COD application puts claims on nodes. While the COD application does not need the nodes (to run the COD jobs), the claims are suspended, allowing batch jobs to run.

4.3.2 Authorizing Users to Create and Manage COD Claims

Claims on nodes are assigned to users. A user with a claim on a resource can then suspend and resume a COD job at will. This gives the user a great deal of power on the claimed resource, even if it is owned by another user. Because of this, it is essential that users allowed to claim COD resources can be trusted not to abuse this power. Users are authorized to have access to the privilege of creating and using a COD claim on a machine. This privilege is granted when the Condor administrator places a given user name in the VALID_COD_USERS list in the Condor configuration for the machine (usually in a local configuration file).

In addition, the tools to request and manage COD claims require that the user issuing the commands be authenticated. Use one of the strong authentication methods described in section 3.6.1 ``Security Configuration'' on page [*]. If one of these methods cannot be used, then file system authentication may be used when directly logging in to that machine (to be claimed) and issuing the command locally.

4.3.3 Defining a COD Application

To run an application on a claimed COD resource, an authorized user defines characteristics of the application. Examples of characteristics are the executable or script to use, the directory to run the application in, command-line arguments, and files to use for standard input and output. COD users specify a ClassAd that describes these characteristics for their application. There are two ways for a user to define a COD application's ClassAd:

  1. in the Condor configuration files of the COD resources
  2. when they use the condor_ cod command-line tool to launch the application itself

These two methods for defining the ClassAd can be used together. For example, the user can define some attributes in the configuration file, and only provide a few dynamically defined attributes with the condor_ cod tool.

Regardless of how the COD application's ClassAd is defined, the application's executable and input data must be pre-staged at the node. This is a current limitation of Condor's support for COD that will eventually go away. For now, there is no mechanism to transfer files for a COD application, and all I/O must be performed locally or onto a network file system that is accessible by a node.

The following three sections detail defining the attributes. The first lists the attributes that can be used to define a COD application. The second describes how to define these attributes in a Condor configuration file. The third explains how to define these attributes using the condor_ cod tool. COD Application Attributes

Attributes for a COD application are either required or optional. The following attributes are required:

This attribute defines the full path to the executable program to be run as a COD application. Since Condor does not currently provide any mechanism to transfer files on behalf of COD applications, this path should be a valid path on the machine where the application will be run. It is a string attribute, and must therefore be enclosed in quotation marks ("). There is no default.

IWD is an acronym for Initial Working Directory. It defines the full path to the directory where a given COD application are to be run. Unless the application changes its current working directory, any relative path names used by the application will be relative to the IWD. If any other attributes that define file names (for example, In, Out, and so on) do not contain a full path, the IWD will automatically be pre-pended to those filenames. It is a string attribute, and must therefore be enclosed in quotation marks ("). There is no default.

If the condor_ startd daemon is executing as root on the resource where a COD application will run, the user must also define Owner to specify what user name the application will run as. (On Windows, the condor_ startd daemon always runs as an Administrator service, which is equivalent to running as root on UNIX platforms). If the user specifies any COD application attributes with the condor_ cod_activate command-line tool, the Owner attribute will be defined as the user name that ran condor_ cod_activate. However, if the user defines all attributes of their COD application in the Condor configuration files, and does not define any attributes with the condor_ cod_activate command-line tool (both methods are described below in more detail), there is no default and Owner must be specified in the configuration file. Owner must contain a valid user name on the given COD resource. It is a string attribute, and must therefore be enclosed in quotation marks (").

The following list of attributes are optional:

This string defines the path to the file on the COD resource that should be used as standard input (stdin) for the COD application. This file (and all parent directories) must be readable by whatever user the COD application will run as. If not specified, the default is /dev/null.

This string defines the path to the file on the COD resource that should be used as standard output (stdout) for the COD application. This file must be writable (and all parent directories readable) by whatever user the COD application will run as. If not specified, the default is /dev/null. It is a string attribute, and must therefore be enclosed in quotation marks (").

This string defines the path to the file on the COD resource that should be used as standard error (stderr) for the COD application. This file must be writable (and all parent directories readable) by whatever user the COD application will run as. If not specified, the default is /dev/null. It is a string attribute, and must therefore be enclosed in quotation marks (").

This string defines environment variables to set for a given COD application. Each environment variable has the form NAME=value. Multiple variables are delimited with a semicolon. An example: Env = "PATH=/usr/local/bin:/usr/bin;TERM=vt100" It is a string attribute, and must therefore be enclosed in quotation marks (").

This string attribute defines the list of arguments to be supplied to the program on the command-line. The arguments are delimited (separated) by space characters. There is no default. If the JobUniverse corresponds to the Java universe, the first argument must be the name of the class containing main. It is a string attribute, and must therefore be enclosed in quotation marks (").

This attribute defines what Condor job universe to use for the given COD application. At this point, the only supported universes are vanilla and Java. This attribute must be an integer, with vanilla using the value 5, and Java the value 10. If JobUniverse is not specified, the vanilla universe is used by default. For more information about the Condor job universes, see section 2.4.1 on page [*].

This string attribute is only used if JobUniverse is 10 (the Java universe). If a given COD application is a Java program, specify the JAR files that the program requires with this attribute. There is no default. It is a string attribute, and must therefore be enclosed in quotation marks ("). Multiple file names may be delimited with either commas or whitespace characters, and therefore, file names can not contain spaces.

This attribute specifies what signal should be sent whenever the Condor system needs to gracefully shutdown the COD application. It can either be specified as a string containing the signal name (for example KillSig = "SIGQUIT"), or as an integer (KillSig = 3) The default is to use SIGTERM.

This string specifies a file name for a log file that the condor_ starter daemon can write with entries for relevant events in the life of a given COD application. It is similar to the UserLog file specified for regular Condor jobs with the Log setting in a submit description file. However, certain attributes that are placed in the regular UserLog file do not make sense in the COD environment, and are therefore omitted. The default is not to write this log file. It is a string attribute, and must therefore be enclosed in quotation marks (").

If the StarterUserLog attribute is defined, the default format is a human-readable format. However, Condor can write out this log in an XML representation, instead. To enable the XML format for this UserLog, the StarterUserLogUseXML boolean is set to TRUE. The default if not specified is FALSE.

NOTE: If any path attribute (Cmd, In, Out,Err, StarterUserLog) is not a full path name, Condor automatically prepends the value of IWD.

The final set of attributes define an identification for a COD application. The job ID is made up of both the ClusterId and ProcId attributes (as described below). This job ID is similar to the job ID that is created whenever a regular Condor batch job is submitted. For regular Condor batch jobs, the job ID is assigned automatically by the condor_ schedd whenever a new job is submitted into the persistent job queue. However, since there is no persistent job queue for COD, the usual mechanism to identify the jobs does not exist. Moreover, commands that require the job ID for batch jobs such as condor_ q and condor_ rm do not exist for COD. Instead, the claim ID is the unique identifier for COD jobs and COD-related commands.

When using COD, the job ID is only used to identify the job in various log messages and in the COD-specific output of condor_ status. The COD job ID is part of the information included in all events written to the StarterUserLog regarding a given job. The COD job ID is also used in the Condor debugging logs described in section 3.3.4 on page [*] For example, in the condor_ starter daemon's log file for COD jobs (called StarterLog.cod by default) or in the condor_ startd daemon's log file (called StartLog by default).

These COD IDs are optional. The job ID is useful to define where it helps a user with accounting or debugging of their own application. In this case, it is the user's responsibility to ensure uniqueness, if so desired.

This integer defines the cluster identifier for a COD job. The default value is 1. The ClusterId can also be defined with the condor_ cod_activate command-line tool using the -cluster option.

This integer defines the process identifier (within a cluster) for a COD job. The default value is 0. The ProcId can also be defined with the condor_ cod_activate command-line tool using the -cluster option.

NOTE: The cluster and proc identifiers can also be specified as command-line arguments to the condor_ cod_activate tool when spawning a given COD application. See section 4.3.4 below for details on using condor_ cod_activate. Defining Attributes in the Condor Configuration Files

To define COD attributes in the Condor configuration file for a given application, the user selects a keyword to uniquely name ClassAd attributes of the application. This case-insensitive keyword is used as a prefix for the various configuration file attribute names. When a user wishes to spawn a given application, the keyword is given as an argument to the condor_ cod tool and the keyword is used at the remote COD resource to find attributes which define the application.

Any of the ClassAd attributes described in the previous section can be specified in the configuration file with the keyword prefix followed by an underscore character ("_").

For example, if the user's keyword for a given fractal generation application is ``FractGen'', the resulting entries in the Condor configuration file may appear as:

FractGen_Cmd = "/usr/local/bin/fractgen"
FractGen_Iwd = "/tmp/cod-fractgen"
FractGen_Out = "/tmp/cod-fractgen/output"
FractGen_Err = "/tmp/cod-fractgen/error"
FractGen_Args = "mandelbrot -0.65865,-0.56254 -0.45865,-0.71254"

In this example, the executable may create other files. The Out and Err attributes specified in the configuration file are only for standard output and standard error redirection.

When the user wishes to spawn an instance of this application, they use the -keyword option of FractGen in the command-line of the condor_ cod_activate command.

NOTE: If a user is defining all attributes of their COD application in the Condor configuration files, and the condor_ startd daemon on the COD resource they are using is running as root, the user must also define Owner to be the user that the COD application should run as (see section 4.3.3 above). Defining Attributes with the condor_ cod Tool

COD users may define attributes dynamically (at the time they spawn a COD application). In this case, the user writes the ClassAd attributes into a file, and the file name is passed to the condor_ cod_activate tool using the -jobad command-line option. These attributes are read by the condor_ cod tool and passed through the system onto the condor_ starter daemon which spawns the COD application. If the file name given is -, the condor_ cod tool will read from standard input (stdin).

Users should not add a keyword prefix when defining attributes with the condor_ cod_activate tool. The attribute names can be used in the file directly.

WARNING: The current syntax for this file is not the same as the syntax in the file used with condor_ submit.

NOTE: Users should not define the Owner attribute when using condor_ cod_activate on the command line, since Condor will automatically insert the correct value based on what user runs the condor_ cod_activate command and how that user authenticates to the COD resource. If a user defines an attribute that does not match the authenticated identity, Condor treats this case as an error, and it will fail to launch the application.

4.3.4 Managing COD Resource Claims

Separate commands are provided by Condor to manage COD claims on batch resources. Once created, each COD claim has a unique identifying string, called the claim ID. Most commands require a claim ID to specify which claim you wish to act on. These commands are the means by which COD applications interact with the rest of the Condor system. They should be issued by the controller application to manage its compute nodes. Here is a list of the commands:

Create a new COD claim on a given resource.

Spawn a specific application on a specific COD claim.

Suspend a running application within a specific COD claim.

Resume a suspended application on a specific COD claim.

Shut down an application, but hold onto the COD claim for future use.

Destroy a specific COD claim, and shut down any job that is currently running on it.

To issue these commands, a user or application invokes the condor_ cod tool. A command may be specified as the first argument to this tool, as

condor_cod request -name
or the condor_ cod tool can be installed in such a way that the same binary is used for a set of names, as
condor_cod_request -name

Other than the command name itself (which must be included in full) additional options supported by each tool can be abbreviated to the shortest unambiguous value. For example, -name can also be specified as -n. However, for a command like condor_ cod_activate that supports both -classad and -cluster, the user must use at least -cla or -clu. If the user specifies an ambiguous option, the condor_ cod tool will exit with an error message.

In addition, there is now a -cod option to condor_ status.

The following sections describe each option in greater detail. Request

A user must be granted authorization to a create COD claims on a specific machine. In addition, when the user uses these COD claims, the application binary or script they wish to run (and any input data) must be pre-staged on the machine. Therefore, a user cannot simply request a COD claim at random.

The user specifies the resource on which to make a COD claim. This is accomplished by specifying the name of the condor_ startd daemon desired by invoking condor_ cod_request with the -name option and the resource name (usually the host name). For example:

condor_cod_request -name

If the condor_ startd daemon desired belongs to a different Condor pool than the one where executing the COD commands, use the -pool option to provide the name of the central manager machine of the other pool. For example:

condor_cod_request -name -pool

An alternative is to provide the IP address and port number where the condor_ startd daemon is listening with the -addr option. This information can be found in the condor_ startd ClassAd as the attribute StartdIpAddr or by reading the log file when the condor_ startd first starts up. For example:

condor_cod_request -addr "<>"

If neither -name or -addr are specified, condor_ cod_request attempts to connect to the condor_ startd daemon running on the local machine (where the request command was issued).

If the condor_ startd daemon to be used for the COD claim is an SMP machine and has multiple virtual machines, specify which resource on the machine to use for COD by providing the full name of the resource, not just the host name. For example:

condor_cod_request -name

A constraint on what virtual machine is desired may be provided, instead of specifying it by name. For example, to run on machine, not caring which virtual machine is used, so long as it the machine is not currently running a job, use something like:

condor_cod_request -name -requirements 'State!="Claimed"'

In general, be careful with shell quoting issues, so that your shell is not confused by the ClassAd expression syntax (in particular if the expression includes a string). The safest method is to enclose any requirement expression within single quote marks (as shown above).

Once a given condor_ startd daemon has been contacted to request a new COD claim, the condor_ startd daemon checks for proper authorization of the user issuing the command. If the user has the authority, and the condor_ startd daemon finds a resource that matches any given requirements, the condor_ startd daemon creates a new COD claim and gives it a unique identifier, the claim ID. This ID is used to identify COD claims when using other commands. If condor_ cod_request succeeds, the claim ID for the new claim is printed out to the screen. All other commands to manage this claim require the claim ID to be provided as a command-line option.

When the condor_ startd daemon assigns a COD claim, the ClassAd describing the resource is returned to the user that requested the claim. This ClassAd is a snap-shot of the output of condor_status -long for the given machine. If condor_ cod_request is invoked with the -classad option (which takes a file name as an argument), this ClassAd will be written out to the given file. Otherwise, the ClassAd is printed to the screen. The only essential piece of information in this ClassAd is the Claim ID, so that is printed to the screen, even if the whole ClassAd is also being written to a file.

The claim ID as given after listing the machine ClassAd appears as this example:

ID of new claim is: "<>#1073352104#4"
When using this claim ID in further commands, include the quote marks as well as all the characters in between the quote marks.

NOTE: Once a COD claim is created, there is no persistent record of it kept by the condor_ startd daemon. So, if the condor_ startd daemon is restarted for any reason, all existing COD claims will be destroyed and the new condor_ startd daemon will not recognize any attempts to use the previous claims. Activate

Once a user has created a valid COD claim and has the claim ID, the next step is to spawn a COD job using the claim. The way to do this is to activate the claim, using the condor_ cod_activate command. Once a COD application is active on a COD claim, the COD claim will move into the Running state, and any batch Condor job on the same resource will be suspended. Whenever the COD application is inactive (either suspended, removed from the machine, or if it exits on its own), the state of the COD claim changes. The new state depends on why the application became inactive. The batch Condor job then resumes.

To activate a COD claim, first define attributes about the job to be run in either the local configuration of the COD resource, or in a separate file as described in this manual section. Invoke the condor_ cod_activate command to launch a specific instance of the job on a given COD claim ID. The options given to condor_ cod_activate vary depending on if the job attributes are defined in the configuration file or are passed via a file to the condor_ cod_activate tool itself. However, the -id option is always required by condor_ cod_activate, and this option should be followed by a COD claim ID that the user acquired via condor_ cod_request.

If the application is defined in the configuration files for the COD resource, the user provides the keyword (described in section 4.3.3) that uniquely identifies the application's configuration attributes. To continue the example from that section, the user would spawn their job by specifying -keyword FractGen, for example:

condor_cod_activate -id "<claim_id>" -keyword FractGen
Substitute the <claim_id> with the valid Cod Claim Id. Using the same example as given above, this example would be:
condor_cod_activate -id "<>#1073352104#4" -keyword FractGen

If the job attributes are placed into a file to be passed to the condor_ cod_activate tool, the user must provide the name of the file using the -jobad option. For example, if the job attributes were defined in a file named cod-fractgen.txt, the user spawns the job using the command:

condor_cod_activate -id "<claim_id>" -jobad cod-fractgen.txt
Alternatively, if the filename specified with -jobad is -, the condor_ cod_activate tool reads the job ClassAd from standard input (stdin).

Regardless of how the job attributes are defined, there are other options that condor_ cod_activate accepts. These options specify the job ID for the application to be run. The job ID can either be specified in the job's ClassAd, or it can be specified on the command line to condor_ cod_activate. These options are -cluster and -proc. For example, to launch a COD job with keyword foo as cluster 23, proc 5, or 23.5, the user invokes:

condor_cod_activate -id "<claim_id>" -key foo -cluster 23 -proc 5
The -cluster and -proc arguments are optional, since the job ID is not required for COD. If not specified, the job ID defaults to 1.0. Suspend

Once a COD application has been activated with condor_ cod_activate and is running on a COD resource, it may be temporarily suspended using condor_ cod_suspend. In this case, the claim state becomes Suspended. Once a given COD job is suspended, if there are no other running COD jobs on the resource, a Condor batch job can use the resource. By suspending the COD application, the batch job is allowed to run. If a resource is idle when a COD application is first spawned, suspension of the COD job makes the batch resource available for use in the Condor system. Therefore, whenever a COD application has no work to perform, it should be suspended to prevent the resource from being wasted.

The interface of condor_ cod_suspend supports the single option -id, to specify the COD claim ID to be suspended. For example:

condor_cod_suspend -id "<claim_id>"

If the user attempts to suspend a COD job that is not running, condor_ cod_suspend exits with an error message. The COD job may not be running because it is already suspended or because the job was never spawned on the given COD claim in the first place. Resume

Once a COD application has been suspended with condor_ cod_suspend, it can be resumed using condor_ cod_resume. In this case, the claim state returns to Running. If there is a regular batch job running on the same resource, it will automatically be suspended if a COD application is resumed.

The condor_ cod_resume tool supports only the -id option to specify the COD claim ID the user wishes to resume. For example:

condor_cod_resume -id "<claim_id>"

If the user attempts to resume a COD job that is not suspended, condor_ cod_resume exits with an error message. Deactivate

If a given COD application does not exit on its own and needs to be removed manually, invoke the condor_ cod_deactivate command to kill the job, but leave the COD claim ID valid for future COD jobs. The user must specify the claim ID they wish to deactivate using the -id option. For example:

condor_cod_deactivate -id "<claim_id>"

By default, condor_ cod_deactivate attempts to gracefully cleanup the COD application and give it time to exit. In this case the COD claim goes into the Vacating state and the condor_ starter process controlling the job will send it the KillSig defined for the job (SIGTERM by default). This allows the COD job to catch the signal and do whatever final work is required to exit cleanly.

However, if the program is stuck or if the user does not want to give the application time to clean itself up, the user may use the -fast option to tell the condor_ starter to quickly kill the job and all its descendants using SIGKILL. In this case the COD claim goes into the Killing state. For example:

condor_cod_deactivate -id "<claim_id>" -fast

In either case, once the COD job has finally exited, the COD claim will go into the Idle state and will be available for future COD applications. If there are no other active COD jobs on the same resource, the resource would become available for batch Condor jobs. Whenever the user wishes to spawn another COD application, they can reuse this idle COD claim by using the same claim ID, without having to go through the process of running condor_ cod_request.

If the user attempts a condor_ cod_deactivate request on a COD claim that is neither Running nor Suspended, the condor_ cod tool exits with an error message. Release

If users no longer wish to use a given COD claim, they can release the claim with the condor_ cod_release command. If there is a COD job running on the claim, the job will first be shut down (as if condor_ cod_deactivate was used), and then the claim itself is removed from the resource and the claim ID is destroyed. Further attempts to use the claim ID for any COD commands will fail.

The condor_ cod_release command always prints out the state the COD claim was in when the request was received. This way, users can know what state a given COD application was in when the claim was destroyed.

Like most COD commands, condor_ cod_release requires the claim ID to be specified using -id. In addition, condor_ cod_release supports the -fast option (described above in the section about condor_ cod_deactivate). If there is a job running or suspended on the claim when it is released with condor_cod_release -fast, the job will be immediately killed. If -fast is not specified, the default behavior is to use a graceful shutdown, sending whatever signal is specified in the KillSig attribute for the job (SIGTERM by default).

4.3.5 Limitations of COD Support in Condor

Condor's support for COD has a few limitations.

The following items are all limitations we plan to remove in future releases of Condor:

None of the above items are fundamentally difficult to add and we hope to address them relatively quickly. If you run into one of these limitations, and it is a barrier to using COD, please contact with the subject ``COD limitation'' to gain quick help.

The following list are more fundamental limitations that we do not plan to address:

next up previous contents index
Next: 4.4 Application Program Interfaces Up: 4. Miscellaneous Concepts Previous: 4.2 Condor's Checkpoint Mechanism   Contents   Index