[Openais] RE: exporting openais GMI

Steven Dake sdake at mvista.com
Wed Jul 7 17:18:16 PDT 2004


Atul,

On Wed, 2004-07-07 at 16:52, Sabharwal, Atul wrote:
> Steve,
> I got started with gmi.h and poll.h. Gmi interface is pretty straight
> forward but I am
> Not clear on the polling API's.  From aisexec code, I figured I have to
> do
> A poll_create, poll_dispatch_add and a poll_run.
> 

1. poll_create creates an instance (or handle) which is returned to the
user of the API.  This is used when using any of the poll APIs to
reference the particular poll.  This allows poll to be run with more
then one instance in an application (although I haven't tested that and
it is probably broken).

2. poll_destroy releases an instance and all its data for a poll_handle.

3. poll_dispatch_add adds a function to be called when the pollable
descriptor specified by events is met.  Here is how it works:

dispatch_fn is called by the poll abstraction when the fd described in
the call to dispatch_add's "revent" (read poll man page) & events is
nonzero.  Basically, if you set "events" to POLLIN, and input arrives on
the socket, dispatch_fn will be called with revents set to the events on
the file descriptor returned from the poll system call (and fd, and data
as passed to dispatch_add).

The fd should be a socket but files may work too.  The data is user
defineable data which is passed to the callback (for passing instance
information).

4. poll_dispatch_modify modifes the poll callback information without
deleting and reading.

5. poll_dispatch_delete deletes the poll data specified by the fd so it
is no longer monitored)

6. poll_timer_add adds a timer function that is called msec_in_future
milliseconds.  "data" is passed to timer_fn when it is called (after the
timeout has expired.  timer_handle_out is an address to a
poll_timer_handle used by timer_delete to delete the timer before it has
expired.

7.  poll_timer_delete deletes a timer function specified by timer_handle

8. poll_run starts the main loop processing.  Any timers that should
expire are expired at the appropriate times by calling the timer
function specified in poll_timer_add.  Any file descriptors that
revent's are set are dispatched with the dispatch functions.

9. poll_stop is not yet implemented.

The main function should look something like:
poll_create
gmi_create with the poll handle returned by poll_create
Add any functions that trigger on a listen'ed bind'ed socket
Add any functions that should be called when a pollable fd is ready to
be read
gmi_join
poll_run

then as you are running:
in your functions that use the bind'ed fd to accept, do the accept call
in your functions that should read from an accept'ed fd, do the read
call
in your functions that are pending write, do your write calls
If you need timers, use poll_timer_create and poll_timer_delete to call
functions at the specified times.

it is important to keep in mind that the gmi is completely abstracted
and nothing must be done other then create and join.  The
dispatch_add/dispatch_remove is only for sockets you want to listen to. 
If you don't care to accept connections from other clients, you don't
have to use those functions at all.  The AIS needs this functionality to
accept connections from APIs which is why it is there.

Timers are not perfectly accurate but close.  It is generally the
minimum time poll may sleep, whith is kernel jiffies + 1, whatever time
unit that is.  On 2.6, this works out to 2 msec as the minimum timer. 
Timers may expire late if the system is heavily loaded.  Timers of zero
timeout can be added, but be careful about recursion.

Thanks
-steve

> If you could elaborate more in this API and the function handlers, it
> would help
> Me out.
> 
> Thanks,
> 
> Atul
> 
> -------------------------------------------------------------
> P.S:  All opinions are my personal opinion(s) & responsibility and do
> not represent the view of my employer ( Intel Corporation ).
>  
> 
> >-----Original Message-----
> >From: Steven Dake [mailto:sdake at mvista.com] 
> >Sent: Wednesday, June 30, 2004 12:19 PM
> >To: Sabharwal, Atul
> >Subject: exporting openais GMI
> >
> >Atul
> >
> >As we discussed in the conf call this morning, it is possible 
> >to use the
> >group messaging interface (which does virtual synchrony) for your
> >purposes without waiting for the SAF AIS event and messaging 
> >interface. 
> >This would atleast help you prototype what you want to do.
> >
> >I have exported the group messaging interface and poll abstraction as a
> >library for use in programs without using AIS in the latest bk for
> >openais.
> >
> >To see the programming interfaces, look at exec/gmi.h and exec/poll.h. 
> >It should be pretty self explinatory, but if you have questions, please
> >ask.  Also, the code in the AIS executive can be viewed to see how the
> >APIs are used.
> >
> >Thanks
> >-steve
> >
> >On Tue, 2004-06-29 at 13:43, Sabharwal, Atul wrote:
> >> Steve,
> >> 
> >> Is there some documentation on how to use Open AIS ?
> >> 
> >> Thanks,
> >> 
> >> Atul
> >> 
> >> -------------------------------------------------------------
> >> P.S:  All opinions are my personal opinion(s) & responsibility and do
> >> not represent the view of my employer ( Intel Corporation ).
> >>  
> >> 
> >> >-----Original Message-----
> >> >From: Steven Dake [mailto:sdake at mvista.com] 
> >> >Sent: Tuesday, June 29, 2004 12:24 PM
> >> >To: Sabharwal, Atul
> >> >Cc: John Cherry; cgl_specs at lists.osdl.org
> >> >Subject: RE: [cgl_specs] Cluster wide event logging
> >> >
> >> >On Tue, 2004-06-29 at 10:19, Sabharwal, Atul wrote:
> >> >> Couple of questions ::
> >> >> 
> >> >> >The implementation can be either a "log server" or a "log 
> >> >collector". 
> >> >> >If it is a "log collector", the notion of cluster time is very
> >> >> >important. 
> >> >> 
> >> >> 1. What is the difference between log server and log 
> >> >collector ? Are you
> >> >> 
> >> >>    thinking that the local syslogd is the log server and 
> >the central
> >> >>    log server is the log collector ?
> >> >> 2. W.r.t clustered time, is NTP a good choice ?
> >> >> 3. Ordering of syslog events is a bigger problem with UDP 
> >datagrams.
> >> >>    Are you looking for cluster wide ordering of events ?
> >> >
> >> >One interesting idea is to use the AIS spec in your implementation. 
> >> >This solves your UDP datagram unreliability and ordering problems.
> >> >
> >> >I personally believe losing syslog messages is definately not ideal,
> >> >especially when those losses don't occur in the overload 
> >case.  In the
> >> >overload case, it would be possible to drop events at the sender, vs
> >> >having them dropped during transmission or reordered.
> >> >
> >> >> 4. What about reliability concerns ? There is RFC 3195 
> >which uses TCP
> >> >>    for reliable delivery although not currently implemented 
> >> >in current
> >> >>    Linux version. Would this be a big concern ?
> >> >> --
> >> >> Atul
> >> >> >
> >> >> >Atul,
> >> >> >
> >> >> >The Cluster-wide Log collection requirement was 
> >> >specifically worded to
> >> >> >NOT dictate an implementation.  It states...
> >> >> >
> >> >> >"OSDL CGL specifies that carrier grade Linux shall provide a
> >> >> >cluster-wide logging mechanism.  A cluster-wide log shall 
> >> >contain node
> >> >> >identification, message type, and cluster time 
> >identification.  This
> >> >> >cluster-wide log may be implemented as a central log or as the
> >> >> >collection of specific node logs."
> >> >> >
> >> >> >Logging (error and event) has been a topic for discussion 
> >> >on lkml and
> >> >> >other initiative lists.  Any implementation will need 
> >> >buy-in from the
> >> >> >the kernel development community and have generic value in 
> >> >> >non-clustered
> >> >> >systems as well.
> >> >> >
> >> >> >> 
> >> >> >> Q1.  Is the log server in the compute cluster or not ?
> >> >> >
> >> >> >The implementation can be either a "log server" or a "log 
> >> >collector". 
> >> >> >If it is a "log collector", the notion of cluster time is very
> >> >> >important.  Each node in the cluster must have the same notion 
> >> >> >of time. 
> >> >> >If it is a "log server", it would need the notion of 
> >> >cluster membership
> >> >> >if it leverages group messaging.  In this case, it would be an 
> >> >> >advantage
> >> >> >to be in the context of the cluster.  However, this 
> >requirement was
> >> >> >written in the context of cluster management and could be 
> >> >> >implemented as
> >> >> >a function of a cluster management node.  Terence may have 
> >> >> >more input on
> >> >> >this.
> >> >> >
> >> >> >> Q2.  Is node failover required for the log server ?
> >> >> >
> >> >> >Yes, if implemented as a service within the cluster.  No, 
> >> >if it is the
> >> >> >collection of logs or the function of a cluster management node.
> >> >> >
> >> >> >> Q3.  Do we need to support a cluster log viewer ?
> >> >> >
> >> >> >While not part of the requirement, the value of the logged 
> >> >information
> >> >> >is in the ability to extract, view, and interpret the 
> >> >informtation.  A
> >> >> >log viewer that could correlate the events in the cluster 
> >> >(whether by
> >> >> >time or event sequence) would be valuable.
> >> >> >
> >> >> >Again, Terence may be able to shed more light on this 
> >> >requirement, but
> >> >> >the requirement is basically that a mechanism exists to 
> >> >> >correlate events
> >> >> >and errors in a cluster such that analysis of order-based 
> >> >> >cluster events
> >> >> >is possible.
> >> >> >
> >> >> >John Cherry
> >> >> >CGL Roadmap Coordinator
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >> >
> >
> >
> >




More information about the Openais mailing list