[Openais] Adding a service

[Steven Dake]

On Fri, 2004-07-23 at 14:13, Mark Haverkamp wrote:
> On Mon, 2004-07-19 at 14:00, Steven Dake wrote:
> 
> [ ... ]
> 
> > 
> > The gmi code does both membership and ordered delivery (and soon
> > authentication and encryption) of messages with virtual synchrony
> > semantics.  Their integration is required for various reasons relating
> > to virtual synchrony.
> > 
> > The idea is that AIS services use the service_handler structure and
> > services provided by the openais executive to communicate with other
> > processors and the APIs users.  Processor to processor communication
> > occurs through the gmi layer and delivery function.
> > 
> > The gmi delivery function basically calls the correct message handler
> > for the message identifier.  This delivery function then generally does
> > the work of the ais service, responding back to the api if "itself" was
> > the processor that sourced the api request.
> > 
> > The GMI code also calls a configuration change handler whenever a
> > configuration change occurs (ie: the membership changes).  This is the
> > optimal mechanism to get membership information as a ais service (vs
> > using the SA Forum membership API).  This is required vs implementing
> > against the SA Forum membership api because the membership api doesn't
> > guarantee VS, hence, correct operation in a partition or merge may not
> > be guaranteed.
> > 
> > Regards
> > -steve
> 
> I've been looking at integrating the event service into aisexec.
> Here's what I want to do. (Hopefully you can tell me if its OK, or 
> there is a better way that I haven't seen).
> 
> Doing an saEvtInitialize will set up the connection between the app and
> the event service.  The event service would set up some data for keeping
> track of things like open channels, subscriptions, etc.  I don't think
> that I want to send and receive those kinds of requests over the fd that
> I got with the initial connection because that could interfere with
> events that may be coming in on that fd.  So, I had though that I'd
> create a different connection for requests like channel operations,
> etc.  The connection would be terminated after the request had
> completed.  The problem is that, how do I make the correspondence
> between a request on a new connection and the one that really represents
> the saEvtInitialize?  A solution that I have come up with, would send a
> handle back to the library when the saEvtIntialize is done.  This handle
> would be sent on the request connections to make the correlation to the
> real instance data on the server.
> 
> Would this work OK or is there a better way to do this?
> 
Mark

This can pose some security risk (since someone could fake a message
with an invalid handle or somebody else's handle and cause havok).  If
you can get it to work, though, we can always resolve the problems later
with a little rework..

I could be wrong, but I assume you are talking about async events, where
you must dispatch a message when an async message occurs and at the same
time be able to handle responses from regular requests on the same fd. 
There is an API for doing this kind of operation, although a quick look
through the devmap shows it is not described :(  I would recommend one
FD for every initialize call since the fd can be used to securely track
state for a particular user invocation.

1. First have a look at exec/amf.c::saAmfInitialize.

This function creates a queue  to store responses that are not to be
handled by the syncronous function, but instead meant to be handled by
the dispatch (async) function.

    /*
     * An inq is needed to store async messages while waiting for a
     * sync response
     */
    error = saQueueInit (&amfInstance->inq, 512, sizeof (void *));
    if (error != SA_OK) {
        goto error_put_destroy;
    }

2. Next have a look at exec/amf.c::saAmfProtectionGroupTrackStart.

This function must ensure that it gets a particular response, even when
it may receive a request for a dispatch (async call).  To solve this,
the function queues the message on amfInstance->inq.  It will only
return a message in &req_amf_protectiongrouptrackstart once a message
with MESSAGE_RES_AMF_PROTECTIONGROUPTRACKSTART defined in header->id of
the response is received.

    error = saSendRetry (amfInstance->fd,
&req_amf_protectiongrouptrackstart,
        sizeof (struct req_amf_protectiongrouptrackstart),
MSG_NOSIGNAL);
    if (error != SA_OK) {
        goto error_unlock;
    }

^^^^^^ This code sends the request


    error = saRecvQueue (amfInstance->fd, &message,
        &amfInstance->inq, MESSAGE_RES_AMF_PROTECTIONGROUPTRACKSTART);

^^^^^^^^ This is the undocumented API which waits for a particular
response.  It will wait until a message with the header
MESSAGE_RES_AMF_PROTECTIONGROUPTRACKSTART is received.  Any other
message it queues for the dispatch function to read the inq.

3. Finally have a look at the exec/amf/saAmfDispatch function.

        saQueueIsEmpty(&amfInstance->inq, &empty);
        if (empty == 0) {
            /*
             * Queue is not empty, read data from queue
             */
            saQueueItemGet (&amfInstance->inq, (void *)&queue_msg);
            msg = *queue_msg;
            memcpy (&dispatch_data, msg, msg->size);
            saQueueItemRemove (&amfInstance->inq);
        } else {
           /*
             * Queue empty, read response from socket
             */
            error = saRecvRetry (amfInstance->fd, &dispatch_data.header,
                sizeof (struct message_header), MSG_WAITALL |
MSG_NOSIGNAL);
            if (error != SA_OK) {
                goto error_unlock;
            }
            if (dispatch_data.header.size > sizeof (struct
message_header)) {
                error = saRecvRetry (amfInstance->fd,
&dispatch_data.data,
                    dispatch_data.header.size - sizeof (struct
message_header),
                    MSG_WAITALL | MSG_NOSIGNAL);
                if (error != SA_OK) {
                    goto error_unlock;
                }
            }
        }

This code basically checks if the queue is empty, then reads from the
queue if there is a request, otherwise it reads from the socket.

You might ask why doesn't the poll (not shown) block if there are
messages in the queue but none in the socket.  It doesn't block because
every time a saRecvQueue queues a message, it sends a request to the
executive (activate poll) which then sends a dummy message back to the
library (activate poll) which keeps poll from blocking.  The dummy
message is ignored by the dispatch function.  

Not a great approach (the activate poll stuff).  I have an idea to fix
it though.  Before a poll is ever done, the inq could be checked to see
if it is empty.  If there are messages on the inq, the dispatch function
would not call poll, but instead indicate to the dispatch function to
dispatch messages.

Fortunately most of this activate poll mess is hidden from the library
developer in saRecvQueue (this does the activate poll stuff).  The
develoepr simply has to be aware that the activate poll message is
coming and ignore it appropriately.

Thanks!
-steve