Runtime library interface

Next: Marshalling code library Up: MSRPC2 User Manual Previous: Stub InternalsClient

Runtime library interface

This chapter describes the structure of the MSRPC2 runtime system which supports the MSRPC2 system for all protocols which involve Inter-Process Communication (IPC). The runtime system implements the RPC mechanism which forms the basis of MSRPC2, including service, session and protocol specific communication management. This chapter defines the interface to the MSRPC2 runtime library, and describes its functionality. The goal of this is to facilitate the addition of new functionality to to the runtime library by those unfamiliar with its structure.

Introduction

Before describing the MSRPC2 runtime system, a few definitions and some motivation are required. These are given below. Thereafter the runtime system interface is described, first by addressing the IPC protocol independent layer, and thereafter a specific implementation of a protocol dependent layer - for MSNL.

Like MSNL, MSRPC2 aims to provide improved performance by reducing multiplexing and by relying to some extent on the properties of the underlying communication facilities. Since MSNL offers a transport which is fairly reliable, MSRPC2 exploits this by neglecting to implement a reliable transport protocol on top of MSNL. Errors arising at the network level are left to the user to sort out, and the semantics of the RPC mechanism reflect this.

MSRPC2 is a simple and efficient RPC mechanism for distributed systems programming. It originates from MSRPC, a lightweight RPC system based on SUN RPC, implemented by the authors at the Computer Laboratory. Whilst the RPC mechanism offered by MSRPC was efficient, its library interface, the stub generation utility and the interface description language it used were unfriendly and lacked the requisite functionality for a tool which could be used for distributed systems programming. A complete redesign of the system, combined with a complete re-implementation of the runtime system have resulted in the new system, MSRPC2.

The MSRPC2 library has been written to run on both Wanda and Unix. The only difference in the code between the two versions is the platform specific protocol interface. The bulk of the code is platform independent and makes use of a portable threads package, named ``pthreads'', and standardised as the POSIX 1003.4a standard.

The MSRPC2 runtime system was written to support a lightweight protocol stack. The initial implementation supported only MSNL as the transport protocol, but recently a TCP transport has been added. MSRPC2 relies on the use of virtual circuit based transport protocols (MSNL and TCP) to reduce the per RPC overhead. State associated with the binding between client and server is maintained at the endpoints, and the connection identifier is the only identifier needed to uniquely specify a communicating (client, server) tuple. The semantics of the RPC system are dependent on the protocol being used. The use of MSNL has the following consequences:

MSNL is an unreliable protocol, and hence cells and packets may be lost by the network. Because ATM networks aim to deliver very high reliability, however, such losses are expected to occur infrequently. MSRPC2 thus implements an unreliable protocol, which does not guarantee that an RPC will be executed once only in the absence of failures. On the contrary, the only semantics supported for MSNL are At Most Once (AMO) semantics. Options exist to request the runtime system to re-try calls which fail due to loss or time out.
MSNL is an ATM protocol and fragments large packets into cells. By keeping the per RPC overhead to a minimum, MSRPC2 permits a typical RPC call and its arguments to fit into a single ATM cell. The overhead per MSRPC2 call is only eight octets of data for both the call and the returned results.
MSNL permits connections to request different quality of service parameters (QOS) at connection setup time. Currently the QOS used for all MSRPC2 connections is ``best-effort'', however it is possible to support RPC connections which offer enhanced QOS. The MSRPC2 interface reference permits a service to specify its QOS requirements, so that data transferred across its interface will be properly delivered by the network.

For TCP the reliable communication protocol means that an Exactly Once semantics can be supported.

The following sections describe the components of the MSRPC2 runtime library and utilities in their current state.

Library Interface

The MSRPC2 library presents an interface to clients and servers which permits the stubs to invoke RPC calls on remote (i.e. not same domain) servers, and servers to export their interfaces.

Client

The client stubs are responsible for binding to the server, and management of the communication with the server for a given session. The user (i.e. the stubs) are presented with (dare I say it) an object-oriented binding structure which represents all state and methods associated with the binding. To create a binding, the prospective client calls the function:

bool_t
   clnt_bind(cl, ifref, m_alloc, m_free, exc_disp, exc)
clntSession             **cl;
MSRPC2InterfaceRef      *ifref;
PFC                     m_alloc;
PFV                     m_free;
caddr_t                 exc_disp;
excpt_t                 *exc;

Here the supplied pointer to the interface reference should be a pointer to the interface reference which refers to the service to which the client wishes to bind. The memory management functions m_alloc, m_free are used by the MSRPC2 library on behalf of the client, to allocate any memory required for RPC arguments or results. The standard C library functions may be supplied for this purpose. The function exc_disp is a pointer to the stub level exception handler which will manage the dispatching of all RPC library exceptions to the client. The exception pointer is supplied to permit the user to establish why the binding operation failed, if the return code is FALSE. A return code of TRUE indicates success, and the variable cl will point to a client session structure which contains all state for the binding, as well as stub level methods for accessing the underlying RPC protocol. The session structure has the following fields:

typedef struct _clntSession {
   LIBS bool_t  (*call_init)    _((struct _clntSession *self, MSDR **m, 
                                   int32 procno, excpt_t *exc));
   LIBS bool_t  (*call)         _((struct _clntSession *self, excpt_t *exc));
   LIBS bool_t  (*announce)     _((struct _clntSession *self, excpt_t *exc, 
                                  bool_t isretry));
   LIBS bool_t  (*control)      _((struct _clntSession *self, int32 op, 
                                   caddr_t param));
   LIBS void    (*destroy)      _((struct _clntSession *self));
   LIBS caddr_t                 private; /* private proto  dependent state */
   LIBS caddr_t                 exc_disp; /* opaque exception dispatch state */
} clntSession;

The specification LIBS ensures that the fields are write only by the libraries, and compile time errors will occur if any session fields are modified outside the library. The session structure is the client's only access to the runtime library. Each method is described in more detail below, using the MSNL transport service as an example. Note that the session structure is transport protocol independent, and represents the session-layer functions of the RPC runtime system. Protocol specific data is stored on the structure as a private opaque field private which can be accessed and interpreted by the methods associated with the session.

The methods, whilst presenting a protocol independent interface, are protocol specific, and are initialised to the appropriate methods the protocol required for the binding, when the binding is created by clnt_bind. At bind time the protocol specific binding function is called to establish a connection of the appropriate type to the remote service instance. The required information for the protocol is stored in the interface reference passed to the library. For MSNL the binding function is:

bool_t
   clntmsnl_bind(cl, ref, m_alloc, m_free, exc_disp, exc)
clntSession             *cl;
MSRPC2InterfaceRef      *ref;
PFC                     m_alloc;
PFV                     m_free;
caddr_t                 exc_disp;
excpt_t                 *exc;

This function returns TRUE if it succeeds, and FALSE if the binding failed. In this case the exception block has been set to indicate the cause. This function initiates communication with the remote service instance. If the interface reference requires the client to check the identity of the service, then when the connection has been established, the client waits for a message from the server which contains the interface identifier. This is checked against the identifier in the interface reference. If the two do not match then the binding is destroyed. If no check is requested then this sequence is not performed.

Once the client has established binding with the remote service instance, it can send RPC requests and obtain results. Note that each session is single threaded. It is not possible to multiplex RPC calls onto a single session. Before making an RPC, the client stubs must request the library to initialise state required for a call. It uses the method call_init on the binding structure for this purpose. For MSNL this function is:

bool_t
   clntmsnl_call_init(cl, msdrs, procno, exc)
clntSession     *cl;
MSDR            **msdrs;
int32           procno;
excpt_t         *exc;

This function initialises the marshalling state required for the session, in the variable msdrs. To do this the library requests an I/O buffer from the I/O buffer system on the platform. For wanda the I/O buffer system which forms part of the IPC system in wanda is used. For unix, a library of I/O buffer management functions is supplied as a part of the RPC library. This permits I/O buffers to be used universally throughout the MSRPC2 library, giving a platform independent interface. The integer procno is the number representing the operation in the MSRPC2 interface which is to be invoked. This is defined by the interface compiler. This function returns TRUE on success, and FALSE, with exception, on failure.

Once it has successfully initialised the runtime system, the client stubs can make an RPC call. To do this the stubs call either the method announce or the method call to make the RPC. announce is used for announcement operations, whilst call is used for operations which return results. When either of these methods is invoked, the library assumes that the arguments to the call have already been marshalled. Marshalling is done by the stubs, using the initialised msdrs supplied for this purpose by the library as a result of the call_init operation.

bool_t
   clntmsnl_announce(cl, exc, isretry)
clntSession     *cl;
excpt_t         *exc;
bool_t           isretry;

The MSNL versions of call and announce make an RPC call on half of the client. The underlying MSNL service layer is used to transmit the I/O buffer containing the marshalled arguments to the server. The client library then blocks in a receive until either a timeout expires or results are received from the server. In the case of an announcement, no results are expected, and the library returns immediately after making the call. The timeout can be set using the control operation described below. Received results from a call are partially unmarshalled within the library, to determine whether the call executed successfully or not. In the latter case the client's exception handler is called with the exception, in the former, the call simply returns TRUE and the stubs can unmarshal the results.

The results or exceptions are passed to the stub code in the msdrs which is the marshalling object associated with the call. This will permit easy unmarshalling of the results. If the call fails then the exception structure contains a reason for the failure, and a pointer to the msdrs which can be used to unmarshal the arguments to the exception, returned by the server.

bool_t
   clntmsnl_call(cl, exc)
clntSession     *cl;
excpt_t         *exc;

Several properties of the underlying protocol may be controlled via the library interface. Each binding object has a method control which can be used to set these parameters. These parameters may differ according to the transport protocol being used. For MSNL the function used is:

bool_t
   clntmsnl_control(cl, request, info)
clntSession     *cl;
int32           request;
caddr_t         info;

The parameters which may be set are:

MSRPC2_CLSET_TIMEOUT - sets the timeout before the exception is raised which declares that the RPC has failed.
MSRPC2_CLGET_TIMEOUT - retrieves the current value of the timeout.
MSRPC2_CLSET_BUFSIZE - sets the size of I/O buffer to use for the session. This should be fixed. It is a hangover from msrpc, which requires that the maximum buffer size be specified before the session is set up. Currently the limit is set at 1500 bytes, which is the size of the largest I/O buffer on a default wanda kernel.
MSRPC2_CLGET_BUFSIZE - retrieves the current value of the buffer size been used by the MSDR library.
MSRPC2_CLSET_RETRIES - this is a feature which is supported by MSNL to permit the library to make several retries of a failed RPC request before declaring that the call has failed. The value of this variable will be used to determine how many retries are made.
MSRPC2_CLGET_RETRIES - retrieves the current value of the count.

A client should terminate its session when it no longer requires it. This should be done by calling the session's destroy method. For MSNL this is:

void
   clntmsnl_destroy(cl)
clntSession     *cl;

Server

The server side of the library is far more complicated. A service offer, corresponding to an instance of a middl interface, is made by calling the function

bool_t
   svc_offer(svc_int, ifref, intf, m_alloc, m_free, cons, dest, exc)
svcInterface            **svc_int;
MSRPC2InterfaceRef      *ifref;
caddr_t                 intf;
PFC                     m_alloc;
PFV                     m_free;
svcBindingConstructor   cons;
svcBindingDestructor    dest;
excpt_t                 *exc;

This is a protocol independent function, and takes as argument a pointer to an interface reference in which the protocol field must be set. This allows the library to determine which protocol to support for this service instance. Also passed as arguments are memory allocation and deallocation routines, which will be used by the library to allocate memory for all RPC arguments and results which require it. The argument svc_int is modified by the library and on return points to the interface instance which will be listening for client binding requests. The argument intf is a pointer used the library to refer to the stub level state representing the interface instance. It is not interpreted by the library. The methods cons,dest are user-level methods which are supplied to the library to permit validation of new binding requests and notification of binding termination. When a new client binding request is made, the user level constructor function will be called, to allow the user to vet the client. If this function returns FALSE, the new client binding request will be rejected. If it returns TRUE then the client will be accepted. The destructor will be called when the binding is broken, for any reason. This allows the server to free any per-client binding state which it has attached to the binding object.

The MSNL version of the offer function is used to initiate the service. It creates an offer, by forking off a thread which listens for new client binding requests. This thread has as argument a pointer to the section object created for the interface. This is known as the rendezvous thread, and it is responsible for creating all new bindings. The interface object created contains list of all bindings which belong to this interface instance. When the interface is withdrawn, this list is used to ensure that all bindings for the interface are terminated appropriately.

bool_t
   svcmsnl_offer(svc_int, ifref, intf, m_alloc, m_free, constr, destr, exc)
svcInterface            **svc_int;
MSRPC2InterfaceRef      *ifref;
caddr_t                 intf;
PFC                     m_alloc;
PFV                     m_free;
svcBindingConstructor   constr;
svcBindingDestructor    destr;
excpt_t                 *exc;

The offer routine returns a pointer to an svcSession . This is the server-side representation of the interface/binding. In the case of an interface, it represents the interface instance. For each new client binding, a new svcSession is created. This maintains all per session state for the binding. It also has a private field which is for use by the protocol specific code which manages the connection with the client. The notation LIBS indicates that the fields may only be modified by the library. The notation BOTH indicates that the stub level code and the library may modify the fields.

struct _svcSession {
   LIBS void            (*destroy)      _((svcSession *self)); 
   LIBS bool_t          (*control)      _((struct _svcSession *self, 
                                           u_int32 op, caddr_t param));
   LIBS bool_t          (*recv)         _((svcSession *self, 
                                           u_int32 *procno, MSDR **m,
                                           excpt_t *exc));
   LIBS bool_t          (*reply_init)   _((svcSession *self, MSDR **m,
                                           u_int32 raises));
   LIBS bool_t          (*reply)        _((svcSession *self));
   LIBS void            (*excpt_intnl)  _((svcSession *self, 
                                           excpt_t *exc));
   BOTH svcBindingDispatcher dispatch; 
   LIBS bool_t          alive;          /* used by svc_run for all protos */
   LIBS caddr_t         private;        /* protocol dependent state */
   LIBS svcSession      *next;          /* chain for linked list of sessions */
   LIBS svcType         type;           /* rendezvous or connected */
   LIBS svcList         *svc_list;      /* list of all sessions for service */
   LIBS pthread_t       thread;         /* the thread managing the session */
   LIBS caddr_t         binding;        /* stub per session state block */
   BOTH MSRPC2InterfaceRef      *ifref; /* stores state on endian proto etc */
   LIBS svcBindingDestructor    destr;  /* stub state destructor */
};

Note that each binding is single threaded. For each binding a thread is forked to manage and perform all RPC requests for the associated client. No other thread should act on behalf of the client, otherwise the library will be unable to withdraw the interface.

When a new binding is created, the rendezvous thread calls the internal function

bool_t
   svcmsnl_intnl_create(session, sock, ifref, m_alloc, m_free, exc)
svcSession              *session;
int32                   sock;
MSRPC2InterfaceRef      *ifref;
PFC                     m_alloc;
PFV                     m_free;
excpt_t                 *exc;

to create a new binding object for the binding. It also calls up into the user level code by invoking the binding constructor for the interface. The session object is initialised with all appropriate state for the interface to which it refers, and the methods are initialised to the protocol specific methods for managing the RPC session specified in the interface reference.

Each binding is managed by a single thread. Each thread, on starting, calls the function (protocol independent) svc_run which contains an infinite loop. On each iteration, the binding's recv method is called. For a rendezvous thread, the recv method is initialised to the function which is responsible for forking off new sessions when new binding requests are received. For MSNL this is the function

bool_t /* ALWAYS FALSE */
   svcmsnl_rendezvous(session, procp, msdrs, exc)
svcSession      *session;
u_int32         *procp;         /* DUMMY argument, never used */
MSDR            **msdrs;        /* DUMMY argument, never used */
excpt_t         *exc;

This function waits for connection requests on the MSNL sap associated with the interface reference. When a request is received, if the binding constructor returns TRUE, a new binding is created. This involves forking a new thread, with the recv method to the real receive method for the protocol. In the case of MSNL this is svcmsnl_recv. The other methods will also be initialised to the appropriate values.

bool_t
   svcmsnl_recv(session, procp, msdrs, exc)
svcSession      *session;
u_int32         *procp;
MSDR            **msdrs;
excpt_t         *exc;

This function waits for RPC calls from its associated client. The function receives an I/O buffer from the MSNL layer managing the communication with the client. This function unmarshals the arguments required by the stub layer code, and then calls the dispatch function associated with the binding. This function then unmarshals the arguments and calls the user level code. If an error occurs within this function, an exception is raised. Exceptions are marshalled and returned to the calling user. This includes both user level exceptions and library exceptions. If this function returns value of TRUE to the caller (svc_run ) then the dispatch function will be invoked. If not then the associated exception, if there is one will be returned. If the user-level dispatcher returns TRUE, then the call succeeded, otherwise it failed.

If the invocation is expected to return results then the stub level code must call the function reply_init on the binding to initialised appropriate state in the library for the reply. The MSDR structure will be initialised with an I/O buffer in preparation for the reply. For MSNL this function is

bool_t
   svcmsnl_reply_init(session, msdrs, raises)
svcSession      *session;
MSDR            **msdrs;
u_int32         raises;

Once the reply state has been initialised, the stub level code can use the MSDR object to marshal either the results or the exception arguments. Note that it is the responsibility of the stub code to do all marshalling. The stub code may then call the reply function for the session. This will transmit the marshalled results or exception and arguments to the client.

bool_t
   svcmsnl_reply(session)
svcSession      *session;

Similar to the client side, the server is provided with an interface which permits the properties of the session to be modified on a per binding basis. This is the control function, which for MSNL has the following form. All arguments info for the time functions are a struct timeval *, otherwise they are unsigned integers.

bool_t
   svcmsnl_control(session, request, info)
svcSession      *session;
u_int32         request;
caddr_t         info;

The parameters which may be modified/retrieved include

MSRPC2_SVCSET_TIMEOUT - set the timeout which the server uses before declaring the session to have died.
MSRPC2_SVCGET_TIMEOUT - get the timeout above.
MSRPC2_SVCSET_BUFSIZE - set the buffer size used by the MSDR library when initialised an MSDR object. The default value is 1500 bytes.
MSRPC2_SVCGET_BUFSIZE - get the buffer size above.

Next: Marshalling code library Up: MSRPC2 User Manual Previous: Stub InternalsClient

Simon Crosby and Richard Hayton