[Openais] Re: membership change peculiarity

Steven Dake sdake at mvista.com
Thu Sep 30 12:23:38 PDT 2004


Mark,

This looks like a bug in the membership algorithm.  A bugzilla entry is
in order.  How repeatable is this problem?  Did the node think it was
the only one, right after you killed off node 8, or was it some time
(more then 2 seconds) later?

The way it works is when cl017 detects a token loss, it should enter the
gather state.  Then it sends an attempt join (multicast, but not
reliable or ordered).  Every rep of a ring should respond with an
attempt join.  In this case, node 8 is special, because it is always
likely to be the ring rep (until it is killed, in which case 17 is the
ring rep).  The procesor with the smallest IP address is chosen as the
ring rep.  Of importance is that non-reps do not take part in building
the membership for the membership algorithm.

The fact that it is now 17 that fails to produce the desired gathers
that doesn't detect the new ring could cause some speculation that the
membership algorithm fails to form on token loss in the operational
state when the rep is killed off.  If it were some other node, I'd say
it is likely it just didn't get picked up because the processor failed
to communicate.

We just don't have enough information from the logs to detect what the
algorithm is doing incorrectly, though.  We are in need of some more
debug output...

I'll get to these problems as soon as I can; I'm working on the cluster
membership library/executive (clm) code trying to fix up a few bugs and
increase code coverage...

Regards
-steve

On Thu, 2004-09-30 at 10:42, Mark Haverkamp wrote:
> Steve, 
> 
> I've been trying debug the case where the aisexec programs can't sync up
> on a membership sometimes.  I ran into something peculiar.  I had the
> program compiled with -DEBUG and scripted the output from each of the
> nodes.  Once they were all running I interrupted one of them.  Something
> strange happened.  One of the nodes seemed to think that it was the only
> one for a while instead of there still being three nodes.  After a while
> things settled and formed a three node membership. Does this sound
> right?  Here are some excerpts from the console where I killed cl008.
> 
> peculiarity 
> cl017:
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(3): Token being retransmitted.
> L(3): Token loss in OPERATIONAL.
> L(4): entering GATHER state.
> L(4): SENDING attempt join because this node is ring rep.
> L(5): GATHER timeout:
> L(5): host 0 attempted to join 192.168.1.17
> L(4): I am the only member.
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17
> L(4): Members Left:
> L(4): 	192.168.1.8
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(4): Members Joined:
> L(5): Evt conf change
> L(5): m 1, j 0, l 3
> ***L(5): cluster node at 192.168.1.8 down   <--- TRUE
> ***L(5): cluster node at 192.168.1.18 down  <--- FALSE
> ***L(5): cluster node at 192.168.1.19 down  <--- FALSE
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17 
> L(4): Members Left:
> L(4): Members Joined:
> L(5): Evt conf change
> L(5): m 1, j 0, l 0
> L(4): Send EVT_CONF_CHANGE
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> L(5): Delivering MCAST message with seqid 10 to pending delivery queue
> Queue empty[0] 0 queues seqid 10
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 1 checked in 0, mn 80bb428
> L(4): I am oldest
> L(5): No messages to send
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> Queue empty[0] 1 queues seqid 10
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): Delivering MCAST message with seqid 11 to pending delivery queue
> Queue empty[0] 0 queues seqid 11
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_DONE from 1101a8c0
> L(4): All services unplugged, unplugging processor
> L(4): All processors unplugged, allowing messages to be transmitted.
> Queue empty[0] 1 queues seqid 11
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 10
> L(4): Got attempt join from 192.168.1.18
> L(4): entering GATHER state.
> L(4): SENDING attempt join because this node is ring rep.
> L(5): ATTEMPT JOIN: state gather
> L(4): Got attempt join from 192.168.1.19
> L(5): ATTEMPT JOIN: state gather
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.18
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.18
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 1 192.168.1.19
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.18
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 1 192.168.1.19
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(4): CONSENSUS reached!
> L(5): This node responsible for sending the FORM token.
> Got membership form token
> conf_desc_list 1
> highest seq 0 11
> setting barrier seq to 11
> setting barrier seq to 12
> Got membership form token
> L(5): Swallowing FORM token in EVS state.
> FORM CONF ENTRIES 1
> EVS STATE group arut 11 gmi arut 11 highest 11 barrier 12 starting group arut 11
> L(5): releasing messages up to and including 10
> EVS STATE group arut 12 gmi arut 12 highest 11 barrier 12 starting group arut 12
> L(4): EVS recovery of messages complete, transitioning to operational.
> L(5): releasing messages up to and including 11
> CONFCHG ENTRIES 1
> calling recovery
> New queue for ip 192.168.1.18
> New queue for ip 192.168.1.19
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(4): Members Left:
> L(4): Members Joined:
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(5): MCASTING MESSAGE
> L(5): Message size is 608
> L(5): mcasted message added to pending queue
> L(5): Evt conf change
> L(5): m 3, j 2, l 0
> L(5): Send set evt ID 0 to 192.168.1.18
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> L(5): Send set evt ID 0 to 192.168.1.19
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> L(4): Send EVT_CONF_CHANGE
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> L(5): Delivering MCAST message with seqid 1 to pending delivery queue
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 0 queues seqid 1
> Queue empty[2] 1 queues seqid 0 lowest so far 0
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1201a8c0 members 3 checked in 0, mn 80bf4c0
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 0
> L(5): Delivering MCAST message with seqid 2 to pending delivery queue
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 0 queues seqid 2
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1301a8c0 members 3 checked in 1, mn 80bfb68
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): releasing messages up to and including 0
> L(5): Delivering MCAST message with seqid 3 to pending delivery queue
> Queue empty[0] 0 queues seqid 3
> Queue empty[1] 1 queues seqid 1 lowest so far 3
> Queue empty[2] 1 queues seqid 2 lowest so far 3
> L(4): got nodejoin message 192.168.1.17
> Queue empty[0] 1 queues seqid 3
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 4 to pending delivery queue
> Queue empty[0] 0 queues seqid 4
> Queue empty[1] 1 queues seqid 1 lowest so far 4
> Queue empty[2] 1 queues seqid 2 lowest so far 4
> L(5): Remote channel operation request
> L(5): Received Set event ID OP from 1101a8c0 to 0 for 1201a8c0 my addr 1101a8c0 base 1101a8c000000000
> Queue empty[0] 1 queues seqid 4
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 5 to pending delivery queue
> Queue empty[0] 0 queues seqid 5
> Queue empty[1] 1 queues seqid 1 lowest so far 5
> Queue empty[2] 1 queues seqid 2 lowest so far 5
> L(5): Remote channel operation request
> L(5): Received Set event ID OP from 1101a8c0 to 0 for 1301a8c0 my addr 1101a8c0 base 1101a8c000000000
> Queue empty[0] 1 queues seqid 5
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 6 to pending delivery queue
> Queue empty[0] 0 queues seqid 6
> Queue empty[1] 1 queues seqid 1 lowest so far 6
> Queue empty[2] 1 queues seqid 2 lowest so far 6
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 3 checked in 2, mn 80bb428
> L(4): I am oldest
> L(5): No messages to send
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> Queue empty[0] 1 queues seqid 6
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): releasing messages up to and including 0
> L(5): Delivering MCAST message with seqid 7 to pending delivery queue
> Queue empty[0] 0 queues seqid 7
> Queue empty[1] 1 queues seqid 1 lowest so far 7
> Queue empty[2] 1 queues seqid 2 lowest so far 7
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_DONE from 1101a8c0
> L(4): All services unplugged, unplugging processor
> Queue empty[0] 1 queues seqid 7
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(4): All processors unplugged, allowing messages to be transmitted.
> 
> cl018:
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> already received token 2261 2261
> L(4): Got attempt join from 192.168.1.17
> L(4): rep is 192.168.1.8, not handling attempt join.
> L(5): not the rep for this ring, not handling join.
> L(5): Delivering MCAST message with seqid 10 to pending delivery queue
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 0 queues seqid 10
> Queue empty[2] 1 queues seqid 4 lowest so far 8
> Queue empty[3] 1 queues seqid 6 lowest so far 8
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 4 checked in 4, mn 80bfc20
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 1 queues seqid 10
> Queue empty[2] 1 queues seqid 4
> Queue empty[3] 1 queues seqid 6
> L(5): Delivering MCAST message with seqid 11 to pending delivery queue
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 0 queues seqid 11
> Queue empty[2] 1 queues seqid 4 lowest so far 8
> Queue empty[3] 1 queues seqid 6 lowest so far 8
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_DONE from 1101a8c0
> L(4): All services unplugged, unplugging processor
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 1 queues seqid 11
> Queue empty[2] 1 queues seqid 4
> Queue empty[3] 1 queues seqid 6
> L(3): Token loss in OPERATIONAL.
> L(4): entering GATHER state.
> L(4): SENDING attempt join because this node is ring rep.
> L(4): Got attempt join from 192.168.1.19
> L(5): ATTEMPT JOIN: state gather
> L(4): Got attempt join from 192.168.1.17
> L(5): ATTEMPT JOIN: state gather
> L(5): GATHER timeout:
> L(5): host 0 attempted to join 192.168.1.17
> L(5): host 1 attempted to join 192.168.1.18
> L(5): host 2 attempted to join 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.19
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.19
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.19
> L(5): Join from rep 1 192.168.1.17
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.19
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 1 192.168.1.17
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(4): CONSENSUS reached!
> Got membership form token
> Got membership form token
> conf_desc_list 1
> highest seq 0 11
> setting barrier seq to 11
> setting barrier seq to 12
> EVS STATE group arut 11 gmi arut 11 highest 11 barrier 12 starting group arut 11
> L(5): releasing messages up to and including 9
> EVS STATE group arut 12 gmi arut 12 highest 11 barrier 12 starting group arut 12
> L(4): EVS recovery of messages complete, transitioning to operational.
> L(5): releasing messages up to and including 11
> CONFCHG ENTRIES 4
> calling recovery
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(4): Members Left:
> L(4): 	192.168.1.8
> L(4): Members Joined:
> L(5): Evt conf change
> L(5): m 3, j 0, l 1
> L(5): cluster node at 192.168.1.8 down
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(4): Members Left:
> L(4): Members Joined:
> L(5): Evt conf change
> L(5): m 3, j 0, l 0
> L(4): Send EVT_CONF_CHANGE
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> L(5): releasing messages up to and including 0
> L(5): Delivering MCAST message with seqid 1 to pending delivery queue
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 0 queues seqid 1
> Queue empty[2] 1 queues seqid 0 lowest so far 0
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1201a8c0 members 3 checked in 0, mn 80bd8d0
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 0
> L(5): Delivering MCAST message with seqid 2 to pending delivery queue
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 0 queues seqid 2
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1301a8c0 members 3 checked in 1, mn 80bd658
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 3 to pending delivery queue
> Queue empty[0] 0 queues seqid 3
> Queue empty[1] 1 queues seqid 1 lowest so far 3
> Queue empty[2] 1 queues seqid 2 lowest so far 3
> L(4): got nodejoin message 192.168.1.17
> Queue empty[0] 1 queues seqid 3
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 4 to pending delivery queue
> Queue empty[0] 0 queues seqid 4
> Queue empty[1] 1 queues seqid 1 lowest so far 4
> Queue empty[2] 1 queues seqid 2 lowest so far 4
> L(5): Remote channel operation request
> L(5): Received Set event ID OP from 1101a8c0 to 0 for 1201a8c0 my addr 1201a8c0 base 1201a8c000000000
> Queue empty[0] 1 queues seqid 4
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 5 to pending delivery queue
> Queue empty[0] 0 queues seqid 5
> Queue empty[1] 1 queues seqid 1 lowest so far 5
> Queue empty[2] 1 queues seqid 2 lowest so far 5
> L(5): Remote channel operation request
> L(5): Received Set event ID OP from 1101a8c0 to 0 for 1301a8c0 my addr 1201a8c0 base 1201a8c000000000
> Queue empty[0] 1 queues seqid 5
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 6 to pending delivery queue
> Queue empty[0] 0 queues seqid 6
> Queue empty[1] 1 queues seqid 1 lowest so far 6
> Queue empty[2] 1 queues seqid 2 lowest so far 6
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 3 checked in 2, mn 80bfc20
> Queue empty[0] 1 queues seqid 6
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): releasing messages up to and including 0
> L(5): Delivering MCAST message with seqid 7 to pending delivery queue
> Queue empty[0] 0 queues seqid 7
> Queue empty[1] 1 queues seqid 1 lowest so far 7
> Queue empty[2] 1 queues seqid 2 lowest so far 7
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_DONE from 1101a8c0
> L(4): All services unplugged, unplugging processor
> Queue empty[0] 1 queues seqid 7
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): releasing messages up to and including 6
> L(4): All processors unplugged, allowing messages to be transmitted.
> 
> 
> cl019:
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(5): releasing messages up to and including 9
> L(3): Token being retransmitted.
> L(5): releasing messages up to and including 9
> L(4): Got attempt join from 192.168.1.17
> L(4): rep is 192.168.1.8, not handling attempt join.
> L(5): not the rep for this ring, not handling join.
> L(5): Delivering MCAST message with seqid 10 to pending delivery queue
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 0 queues seqid 10
> Queue empty[2] 1 queues seqid 4 lowest so far 8
> Queue empty[3] 1 queues seqid 6 lowest so far 8
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 4 checked in 4, mn 80bfce0
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 1 queues seqid 10
> Queue empty[2] 1 queues seqid 4
> Queue empty[3] 1 queues seqid 6
> L(5): Delivering MCAST message with seqid 11 to pending delivery queue
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 0 queues seqid 11
> Queue empty[2] 1 queues seqid 4 lowest so far 8
> Queue empty[3] 1 queues seqid 6 lowest so far 8
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_DONE from 1101a8c0
> L(4): All services unplugged, unplugging processor
> Queue empty[0] 1 queues seqid 8
> Queue empty[1] 1 queues seqid 11
> Queue empty[2] 1 queues seqid 4
> Queue empty[3] 1 queues seqid 6
> L(4): Got attempt join from 192.168.1.18
> L(4): rep is 192.168.1.8, not handling attempt join.
> L(3): Token loss in OPERATIONAL.
> L(4): entering GATHER state.
> L(4): SENDING attempt join because this node is ring rep.
> L(4): Got attempt join from 192.168.1.17
> L(5): ATTEMPT JOIN: state gather
> L(5): GATHER timeout:
> L(5): host 0 attempted to join 192.168.1.17
> L(5): host 1 attempted to join 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.18
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): JOIN in commit
> L(5): Gather list
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 0 192.168.1.18
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(5): Join from rep 1 192.168.1.17
> L(5): 	member 0 192.168.1.17
> L(5): 	member 1 192.168.1.18
> L(5): 	member 2 192.168.1.19
> L(4): CONSENSUS reached!
> Got membership form token
> Got membership form token
> conf_desc_list 1
> highest seq 0 11
> setting barrier seq to 11
> setting barrier seq to 12
> EVS STATE group arut 11 gmi arut 11 highest 11 barrier 12 starting group arut 11
> L(5): releasing messages up to and including 9
> EVS STATE group arut 12 gmi arut 12 highest 11 barrier 12 starting group arut 12
> L(4): EVS recovery of messages complete, transitioning to operational.
> L(5): releasing messages up to and including 11
> CONFCHG ENTRIES 4
> calling recovery
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(4): Members Left:
> L(4): 	192.168.1.8
> L(4): Members Joined:
> L(5): Evt conf change
> L(5): m 3, j 0, l 1
> L(5): cluster node at 192.168.1.8 down
> L(4): CLM CONFIGURATION CHANGE
> L(4): New Configuration:
> L(4): 	192.168.1.17
> L(4): 	192.168.1.18
> L(4): 	192.168.1.19
> L(4): Members Left:
> L(4): Members Joined:
> L(5): Evt conf change
> L(5): m 3, j 0, l 0
> L(4): Send EVT_CONF_CHANGE
> L(5): MCASTING MESSAGE
> L(5): Message size is 272
> L(5): mcasted message added to pending queue
> L(5): Delivering MCAST message with seqid 1 to pending delivery queue
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 0 queues seqid 1
> Queue empty[2] 1 queues seqid 0 lowest so far 0
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1201a8c0 members 3 checked in 0, mn 80bdb00
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 0
> L(5): releasing messages up to and including 0
> L(5): Delivering MCAST message with seqid 2 to pending delivery queue
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 0 queues seqid 2
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1301a8c0 members 3 checked in 1, mn 80bb428
> Queue empty[0] 1 queues seqid 0
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 3 to pending delivery queue
> Queue empty[0] 0 queues seqid 3
> Queue empty[1] 1 queues seqid 1 lowest so far 3
> Queue empty[2] 1 queues seqid 2 lowest so far 3
> L(4): got nodejoin message 192.168.1.17
> Queue empty[0] 1 queues seqid 3
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 4 to pending delivery queue
> Queue empty[0] 0 queues seqid 4
> Queue empty[1] 1 queues seqid 1 lowest so far 4
> Queue empty[2] 1 queues seqid 2 lowest so far 4
> L(5): Remote channel operation request
> L(5): Received Set event ID OP from 1101a8c0 to 0 for 1201a8c0 my addr 1301a8c0 base 1301a8c000000000
> Queue empty[0] 1 queues seqid 4
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 5 to pending delivery queue
> Queue empty[0] 0 queues seqid 5
> Queue empty[1] 1 queues seqid 1 lowest so far 5
> Queue empty[2] 1 queues seqid 2 lowest so far 5
> L(5): Remote channel operation request
> L(5): Received Set event ID OP from 1101a8c0 to 0 for 1301a8c0 my addr 1301a8c0 base 1301a8c000000000
> Queue empty[0] 1 queues seqid 5
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): Delivering MCAST message with seqid 6 to pending delivery queue
> Queue empty[0] 0 queues seqid 6
> Queue empty[1] 1 queues seqid 1 lowest so far 6
> Queue empty[2] 1 queues seqid 2 lowest so far 6
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_CHANGE from 1101a8c0 members 3 checked in 2, mn 80bfce0
> Queue empty[0] 1 queues seqid 6
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): releasing messages up to and including 0
> L(5): Delivering MCAST message with seqid 7 to pending delivery queue
> Queue empty[0] 0 queues seqid 7
> Queue empty[1] 1 queues seqid 1 lowest so far 7
> Queue empty[2] 1 queues seqid 2 lowest so far 7
> L(5): Remote channel operation request
> L(4): Receive EVT_CONF_DONE from 1101a8c0
> L(4): All services unplugged, unplugging processor
> Queue empty[0] 1 queues seqid 7
> Queue empty[1] 1 queues seqid 1
> Queue empty[2] 1 queues seqid 2
> L(5): releasing messages up to and including 6
> L(4): All processors unplugged, allowing messages to be transmitted.




More information about the Openais mailing list