[Openais] membership change peculiarity

Mark Haverkamp markh at osdl.org
Thu Sep 30 10:42:20 PDT 2004


Steve, 

I've been trying debug the case where the aisexec programs can't sync up
on a membership sometimes.  I ran into something peculiar.  I had the
program compiled with -DEBUG and scripted the output from each of the
nodes.  Once they were all running I interrupted one of them.  Something
strange happened.  One of the nodes seemed to think that it was the only
one for a while instead of there still being three nodes.  After a while
things settled and formed a three node membership. Does this sound
right?  Here are some excerpts from the console where I killed cl008.

peculiarity 
cl017:
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(3): Token being retransmitted.
L(3): Token loss in OPERATIONAL.
L(4): entering GATHER state.
L(4): SENDING attempt join because this node is ring rep.
L(5): GATHER timeout:
L(5): host 0 attempted to join 192.168.1.17
L(4): I am the only member.
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17
L(4): Members Left:
L(4): 	192.168.1.8
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(4): Members Joined:
L(5): Evt conf change
L(5): m 1, j 0, l 3
***L(5): cluster node at 192.168.1.8 down   <--- TRUE
***L(5): cluster node at 192.168.1.18 down  <--- FALSE
***L(5): cluster node at 192.168.1.19 down  <--- FALSE
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17 
L(4): Members Left:
L(4): Members Joined:
L(5): Evt conf change
L(5): m 1, j 0, l 0
L(4): Send EVT_CONF_CHANGE
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
L(5): Delivering MCAST message with seqid 10 to pending delivery queue
Queue empty[0] 0 queues seqid 10
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 1 checked in 0, mn 80bb428
L(4): I am oldest
L(5): No messages to send
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
Queue empty[0] 1 queues seqid 10
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): Delivering MCAST message with seqid 11 to pending delivery queue
Queue empty[0] 0 queues seqid 11
L(5): Remote channel operation request
L(4): Receive EVT_CONF_DONE from 1101a8c0
L(4): All services unplugged, unplugging processor
L(4): All processors unplugged, allowing messages to be transmitted.
Queue empty[0] 1 queues seqid 11
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 10
L(4): Got attempt join from 192.168.1.18
L(4): entering GATHER state.
L(4): SENDING attempt join because this node is ring rep.
L(5): ATTEMPT JOIN: state gather
L(4): Got attempt join from 192.168.1.19
L(5): ATTEMPT JOIN: state gather
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.18
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.18
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 1 192.168.1.19
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.18
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 1 192.168.1.19
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(4): CONSENSUS reached!
L(5): This node responsible for sending the FORM token.
Got membership form token
conf_desc_list 1
highest seq 0 11
setting barrier seq to 11
setting barrier seq to 12
Got membership form token
L(5): Swallowing FORM token in EVS state.
FORM CONF ENTRIES 1
EVS STATE group arut 11 gmi arut 11 highest 11 barrier 12 starting group arut 11
L(5): releasing messages up to and including 10
EVS STATE group arut 12 gmi arut 12 highest 11 barrier 12 starting group arut 12
L(4): EVS recovery of messages complete, transitioning to operational.
L(5): releasing messages up to and including 11
CONFCHG ENTRIES 1
calling recovery
New queue for ip 192.168.1.18
New queue for ip 192.168.1.19
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(4): Members Left:
L(4): Members Joined:
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(5): MCASTING MESSAGE
L(5): Message size is 608
L(5): mcasted message added to pending queue
L(5): Evt conf change
L(5): m 3, j 2, l 0
L(5): Send set evt ID 0 to 192.168.1.18
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
L(5): Send set evt ID 0 to 192.168.1.19
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
L(4): Send EVT_CONF_CHANGE
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
L(5): Delivering MCAST message with seqid 1 to pending delivery queue
Queue empty[0] 1 queues seqid 0
Queue empty[1] 0 queues seqid 1
Queue empty[2] 1 queues seqid 0 lowest so far 0
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1201a8c0 members 3 checked in 0, mn 80bf4c0
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 0
L(5): Delivering MCAST message with seqid 2 to pending delivery queue
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 0 queues seqid 2
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1301a8c0 members 3 checked in 1, mn 80bfb68
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): releasing messages up to and including 0
L(5): Delivering MCAST message with seqid 3 to pending delivery queue
Queue empty[0] 0 queues seqid 3
Queue empty[1] 1 queues seqid 1 lowest so far 3
Queue empty[2] 1 queues seqid 2 lowest so far 3
L(4): got nodejoin message 192.168.1.17
Queue empty[0] 1 queues seqid 3
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 4 to pending delivery queue
Queue empty[0] 0 queues seqid 4
Queue empty[1] 1 queues seqid 1 lowest so far 4
Queue empty[2] 1 queues seqid 2 lowest so far 4
L(5): Remote channel operation request
L(5): Received Set event ID OP from 1101a8c0 to 0 for 1201a8c0 my addr 1101a8c0 base 1101a8c000000000
Queue empty[0] 1 queues seqid 4
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 5 to pending delivery queue
Queue empty[0] 0 queues seqid 5
Queue empty[1] 1 queues seqid 1 lowest so far 5
Queue empty[2] 1 queues seqid 2 lowest so far 5
L(5): Remote channel operation request
L(5): Received Set event ID OP from 1101a8c0 to 0 for 1301a8c0 my addr 1101a8c0 base 1101a8c000000000
Queue empty[0] 1 queues seqid 5
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 6 to pending delivery queue
Queue empty[0] 0 queues seqid 6
Queue empty[1] 1 queues seqid 1 lowest so far 6
Queue empty[2] 1 queues seqid 2 lowest so far 6
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 3 checked in 2, mn 80bb428
L(4): I am oldest
L(5): No messages to send
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
Queue empty[0] 1 queues seqid 6
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): releasing messages up to and including 0
L(5): Delivering MCAST message with seqid 7 to pending delivery queue
Queue empty[0] 0 queues seqid 7
Queue empty[1] 1 queues seqid 1 lowest so far 7
Queue empty[2] 1 queues seqid 2 lowest so far 7
L(5): Remote channel operation request
L(4): Receive EVT_CONF_DONE from 1101a8c0
L(4): All services unplugged, unplugging processor
Queue empty[0] 1 queues seqid 7
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(4): All processors unplugged, allowing messages to be transmitted.

cl018:
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
already received token 2261 2261
L(4): Got attempt join from 192.168.1.17
L(4): rep is 192.168.1.8, not handling attempt join.
L(5): not the rep for this ring, not handling join.
L(5): Delivering MCAST message with seqid 10 to pending delivery queue
Queue empty[0] 1 queues seqid 8
Queue empty[1] 0 queues seqid 10
Queue empty[2] 1 queues seqid 4 lowest so far 8
Queue empty[3] 1 queues seqid 6 lowest so far 8
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 4 checked in 4, mn 80bfc20
Queue empty[0] 1 queues seqid 8
Queue empty[1] 1 queues seqid 10
Queue empty[2] 1 queues seqid 4
Queue empty[3] 1 queues seqid 6
L(5): Delivering MCAST message with seqid 11 to pending delivery queue
Queue empty[0] 1 queues seqid 8
Queue empty[1] 0 queues seqid 11
Queue empty[2] 1 queues seqid 4 lowest so far 8
Queue empty[3] 1 queues seqid 6 lowest so far 8
L(5): Remote channel operation request
L(4): Receive EVT_CONF_DONE from 1101a8c0
L(4): All services unplugged, unplugging processor
Queue empty[0] 1 queues seqid 8
Queue empty[1] 1 queues seqid 11
Queue empty[2] 1 queues seqid 4
Queue empty[3] 1 queues seqid 6
L(3): Token loss in OPERATIONAL.
L(4): entering GATHER state.
L(4): SENDING attempt join because this node is ring rep.
L(4): Got attempt join from 192.168.1.19
L(5): ATTEMPT JOIN: state gather
L(4): Got attempt join from 192.168.1.17
L(5): ATTEMPT JOIN: state gather
L(5): GATHER timeout:
L(5): host 0 attempted to join 192.168.1.17
L(5): host 1 attempted to join 192.168.1.18
L(5): host 2 attempted to join 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.19
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.19
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.19
L(5): Join from rep 1 192.168.1.17
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.19
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 1 192.168.1.17
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(4): CONSENSUS reached!
Got membership form token
Got membership form token
conf_desc_list 1
highest seq 0 11
setting barrier seq to 11
setting barrier seq to 12
EVS STATE group arut 11 gmi arut 11 highest 11 barrier 12 starting group arut 11
L(5): releasing messages up to and including 9
EVS STATE group arut 12 gmi arut 12 highest 11 barrier 12 starting group arut 12
L(4): EVS recovery of messages complete, transitioning to operational.
L(5): releasing messages up to and including 11
CONFCHG ENTRIES 4
calling recovery
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(4): Members Left:
L(4): 	192.168.1.8
L(4): Members Joined:
L(5): Evt conf change
L(5): m 3, j 0, l 1
L(5): cluster node at 192.168.1.8 down
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(4): Members Left:
L(4): Members Joined:
L(5): Evt conf change
L(5): m 3, j 0, l 0
L(4): Send EVT_CONF_CHANGE
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
L(5): releasing messages up to and including 0
L(5): Delivering MCAST message with seqid 1 to pending delivery queue
Queue empty[0] 1 queues seqid 0
Queue empty[1] 0 queues seqid 1
Queue empty[2] 1 queues seqid 0 lowest so far 0
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1201a8c0 members 3 checked in 0, mn 80bd8d0
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 0
L(5): Delivering MCAST message with seqid 2 to pending delivery queue
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 0 queues seqid 2
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1301a8c0 members 3 checked in 1, mn 80bd658
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 3 to pending delivery queue
Queue empty[0] 0 queues seqid 3
Queue empty[1] 1 queues seqid 1 lowest so far 3
Queue empty[2] 1 queues seqid 2 lowest so far 3
L(4): got nodejoin message 192.168.1.17
Queue empty[0] 1 queues seqid 3
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 4 to pending delivery queue
Queue empty[0] 0 queues seqid 4
Queue empty[1] 1 queues seqid 1 lowest so far 4
Queue empty[2] 1 queues seqid 2 lowest so far 4
L(5): Remote channel operation request
L(5): Received Set event ID OP from 1101a8c0 to 0 for 1201a8c0 my addr 1201a8c0 base 1201a8c000000000
Queue empty[0] 1 queues seqid 4
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 5 to pending delivery queue
Queue empty[0] 0 queues seqid 5
Queue empty[1] 1 queues seqid 1 lowest so far 5
Queue empty[2] 1 queues seqid 2 lowest so far 5
L(5): Remote channel operation request
L(5): Received Set event ID OP from 1101a8c0 to 0 for 1301a8c0 my addr 1201a8c0 base 1201a8c000000000
Queue empty[0] 1 queues seqid 5
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 6 to pending delivery queue
Queue empty[0] 0 queues seqid 6
Queue empty[1] 1 queues seqid 1 lowest so far 6
Queue empty[2] 1 queues seqid 2 lowest so far 6
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 3 checked in 2, mn 80bfc20
Queue empty[0] 1 queues seqid 6
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): releasing messages up to and including 0
L(5): Delivering MCAST message with seqid 7 to pending delivery queue
Queue empty[0] 0 queues seqid 7
Queue empty[1] 1 queues seqid 1 lowest so far 7
Queue empty[2] 1 queues seqid 2 lowest so far 7
L(5): Remote channel operation request
L(4): Receive EVT_CONF_DONE from 1101a8c0
L(4): All services unplugged, unplugging processor
Queue empty[0] 1 queues seqid 7
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): releasing messages up to and including 6
L(4): All processors unplugged, allowing messages to be transmitted.


cl019:
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(5): releasing messages up to and including 9
L(3): Token being retransmitted.
L(5): releasing messages up to and including 9
L(4): Got attempt join from 192.168.1.17
L(4): rep is 192.168.1.8, not handling attempt join.
L(5): not the rep for this ring, not handling join.
L(5): Delivering MCAST message with seqid 10 to pending delivery queue
Queue empty[0] 1 queues seqid 8
Queue empty[1] 0 queues seqid 10
Queue empty[2] 1 queues seqid 4 lowest so far 8
Queue empty[3] 1 queues seqid 6 lowest so far 8
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 4 checked in 4, mn 80bfce0
Queue empty[0] 1 queues seqid 8
Queue empty[1] 1 queues seqid 10
Queue empty[2] 1 queues seqid 4
Queue empty[3] 1 queues seqid 6
L(5): Delivering MCAST message with seqid 11 to pending delivery queue
Queue empty[0] 1 queues seqid 8
Queue empty[1] 0 queues seqid 11
Queue empty[2] 1 queues seqid 4 lowest so far 8
Queue empty[3] 1 queues seqid 6 lowest so far 8
L(5): Remote channel operation request
L(4): Receive EVT_CONF_DONE from 1101a8c0
L(4): All services unplugged, unplugging processor
Queue empty[0] 1 queues seqid 8
Queue empty[1] 1 queues seqid 11
Queue empty[2] 1 queues seqid 4
Queue empty[3] 1 queues seqid 6
L(4): Got attempt join from 192.168.1.18
L(4): rep is 192.168.1.8, not handling attempt join.
L(3): Token loss in OPERATIONAL.
L(4): entering GATHER state.
L(4): SENDING attempt join because this node is ring rep.
L(4): Got attempt join from 192.168.1.17
L(5): ATTEMPT JOIN: state gather
L(5): GATHER timeout:
L(5): host 0 attempted to join 192.168.1.17
L(5): host 1 attempted to join 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.18
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): JOIN in commit
L(5): Gather list
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 0 192.168.1.18
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(5): Join from rep 1 192.168.1.17
L(5): 	member 0 192.168.1.17
L(5): 	member 1 192.168.1.18
L(5): 	member 2 192.168.1.19
L(4): CONSENSUS reached!
Got membership form token
Got membership form token
conf_desc_list 1
highest seq 0 11
setting barrier seq to 11
setting barrier seq to 12
EVS STATE group arut 11 gmi arut 11 highest 11 barrier 12 starting group arut 11
L(5): releasing messages up to and including 9
EVS STATE group arut 12 gmi arut 12 highest 11 barrier 12 starting group arut 12
L(4): EVS recovery of messages complete, transitioning to operational.
L(5): releasing messages up to and including 11
CONFCHG ENTRIES 4
calling recovery
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(4): Members Left:
L(4): 	192.168.1.8
L(4): Members Joined:
L(5): Evt conf change
L(5): m 3, j 0, l 1
L(5): cluster node at 192.168.1.8 down
L(4): CLM CONFIGURATION CHANGE
L(4): New Configuration:
L(4): 	192.168.1.17
L(4): 	192.168.1.18
L(4): 	192.168.1.19
L(4): Members Left:
L(4): Members Joined:
L(5): Evt conf change
L(5): m 3, j 0, l 0
L(4): Send EVT_CONF_CHANGE
L(5): MCASTING MESSAGE
L(5): Message size is 272
L(5): mcasted message added to pending queue
L(5): Delivering MCAST message with seqid 1 to pending delivery queue
Queue empty[0] 1 queues seqid 0
Queue empty[1] 0 queues seqid 1
Queue empty[2] 1 queues seqid 0 lowest so far 0
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1201a8c0 members 3 checked in 0, mn 80bdb00
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 0
L(5): releasing messages up to and including 0
L(5): Delivering MCAST message with seqid 2 to pending delivery queue
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 0 queues seqid 2
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1301a8c0 members 3 checked in 1, mn 80bb428
Queue empty[0] 1 queues seqid 0
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 3 to pending delivery queue
Queue empty[0] 0 queues seqid 3
Queue empty[1] 1 queues seqid 1 lowest so far 3
Queue empty[2] 1 queues seqid 2 lowest so far 3
L(4): got nodejoin message 192.168.1.17
Queue empty[0] 1 queues seqid 3
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 4 to pending delivery queue
Queue empty[0] 0 queues seqid 4
Queue empty[1] 1 queues seqid 1 lowest so far 4
Queue empty[2] 1 queues seqid 2 lowest so far 4
L(5): Remote channel operation request
L(5): Received Set event ID OP from 1101a8c0 to 0 for 1201a8c0 my addr 1301a8c0 base 1301a8c000000000
Queue empty[0] 1 queues seqid 4
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 5 to pending delivery queue
Queue empty[0] 0 queues seqid 5
Queue empty[1] 1 queues seqid 1 lowest so far 5
Queue empty[2] 1 queues seqid 2 lowest so far 5
L(5): Remote channel operation request
L(5): Received Set event ID OP from 1101a8c0 to 0 for 1301a8c0 my addr 1301a8c0 base 1301a8c000000000
Queue empty[0] 1 queues seqid 5
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): Delivering MCAST message with seqid 6 to pending delivery queue
Queue empty[0] 0 queues seqid 6
Queue empty[1] 1 queues seqid 1 lowest so far 6
Queue empty[2] 1 queues seqid 2 lowest so far 6
L(5): Remote channel operation request
L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 3 checked in 2, mn 80bfce0
Queue empty[0] 1 queues seqid 6
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): releasing messages up to and including 0
L(5): Delivering MCAST message with seqid 7 to pending delivery queue
Queue empty[0] 0 queues seqid 7
Queue empty[1] 1 queues seqid 1 lowest so far 7
Queue empty[2] 1 queues seqid 2 lowest so far 7
L(5): Remote channel operation request
L(4): Receive EVT_CONF_DONE from 1101a8c0
L(4): All services unplugged, unplugging processor
Queue empty[0] 1 queues seqid 7
Queue empty[1] 1 queues seqid 1
Queue empty[2] 1 queues seqid 2
L(5): releasing messages up to and including 6
L(4): All processors unplugged, allowing messages to be transmitted.

-- 
Mark Haverkamp <markh at osdl.org>




More information about the Openais mailing list