[Openais] Re: defect 73 partial fix

Mark Haverkamp markh at osdl.org
Tue Oct 19 11:06:35 PDT 2004


On Mon, 2004-10-18 at 15:49 -0700, Steven Dake wrote:
> Mark
> 
> Here is a partial fix for #73.  Basically I classify this bug as the
> fact that openais behaves poorly if the executive is started and stopped
> with the debugger.  In all cases we want this to work properly by
> kicking the processor out of the membership and making correct conensus
> decisions.
> 
> I find that the current gmi doesn't do this very well (yet).
> 
> Try out the partial fix and let me know if it improves things.

It may have helped.  It didn't help the gdb problem, but it seemed to
help the problem where time advances on a node (i.e. call ntpdate to
update the time).  It did seem to recover from this after a while.  It
did act funny though.  The node that I updated the time on, thought that
it had a token timeout and went through some config change cycles.  At
one point it thought that it was the only node, but the others were
still there and sending messages.  This confused cl017 because it was
receiving messages from nodes that it didn't know about.  After a while,
it came out of it and saw the other nodes again.  I've included the log
from cl017.


> 
> During this process, I noticed that part of the membership algorithm is
> not implemented properly (the failed processor conensus).  I'll work
> that out next and see if that gets us better consensus behavior.
> 
> Thanks
> -steve
> 


Oct 18 15:57:50 cl017 [NOTICE  ] [GMI  ] All processors unplugged, allowing messages to be transmitted.

Updated time here:

Oct 19 10:49:05 cl017 [WARNING ] [GMI  ] Token loss in OPERATIONAL.
Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] entering GATHER state.
Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] SENDING attempt join because this node is ring rep.
Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.8
Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] CONSENSUS reached!
Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] New Configuration:
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] Members Left:
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] Members Joined:
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] New Configuration:
Oct 19 10:49:05 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
Oct 19 10:49:05 cl017 [NOTICE  ] [GMI  ] All services unplugged, unplugging processor
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.18
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.8
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.19
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
Oct 19 10:49:07 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] New Configuration:
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] Members Left:
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] Members Joined:
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
Oct 19 10:49:07 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
Oct 19 10:49:07 cl017 [WARNING ] [GMI  ] Token being retransmitted.
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.19
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] rep is 192.168.1.8, not handling attempt join.
Oct 19 10:49:08 cl017 [WARNING ] [GMI  ] Token loss in OPERATIONAL.
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] entering GATHER state.
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] SENDING attempt join because this node is ring rep.
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] Got attempt join from 192.168.1.18
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] CONSENSUS reached!
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] New Configuration:
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] Members Left:
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] Members Joined:
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] New Configuration:
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
Oct 19 10:49:08 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.17
Oct 19 10:49:08 cl017 [NOTICE  ] [GMI  ] All services unplugged, unplugging processor

These are evt messages saying that it received an event from a node 
that isn't in the cluster. As you can see above, cl017 thinks that it is the 
only node.

Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:08 cl017 last message repeated 3 times
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:08 cl017 last message repeated 2 times
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:08 cl017 last message repeated 11 times
Oct 19 10:49:08 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:09 cl017 last message repeated 106 times
Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:09 cl017 last message repeated 736 times
Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:09 cl017 last message repeated 10 times
Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:09 cl017 last message repeated 30 times
Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:09 cl017 last message repeated 11 times
Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:09 cl017 last message repeated 34 times
Oct 19 10:49:09 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:11 cl017 last message repeated 279 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:11 cl017 last message repeated 12 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:11 cl017 last message repeated 11 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:11 cl017 last message repeated 48 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:11 cl017 last message repeated 11 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:11 cl017 last message repeated 49 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:11 cl017 last message repeated 11 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:11 cl017 last message repeated 28 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:11 cl017 last message repeated 106 times
Oct 19 10:49:11 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:12 cl017 last message repeated 477 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:12 cl017 last message repeated 11 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:12 cl017 last message repeated 45 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:12 cl017 last message repeated 11 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:12 cl017 last message repeated 49 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:12 cl017 last message repeated 11 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:12 cl017 last message repeated 40 times
Oct 19 10:49:12 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:12 cl017 last message repeated 181 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:13 cl017 last message repeated 239 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:13 cl017 last message repeated 11 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:13 cl017 last message repeated 37 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:13 cl017 last message repeated 11 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:13 cl017 last message repeated 47 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:13 cl017 last message repeated 11 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:13 cl017 last message repeated 45 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:13 cl017 last message repeated 11 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:13 cl017 last message repeated 41 times
Oct 19 10:49:13 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:13 cl017 last message repeated 162 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 224 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 2 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 8 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 2 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 11 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 4 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 20 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 14 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 34 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 11 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 40 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 11 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:14 cl017 last message repeated 36 times
Oct 19 10:49:14 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:14 cl017 last message repeated 120 times
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:15 cl017 last message repeated 15 times
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:15 cl017 last message repeated 11 times
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:15 cl017 last message repeated 26 times
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:15 cl017 last message repeated 11 times
Oct 19 10:49:15 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:16 cl017 last message repeated 12 times
Oct 19 10:49:16 cl017 [SECURITY] [GMI  ] Received message has invalid digest... ignoring.
Oct 19 10:49:16 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:16 cl017 last message repeated 11 times
Oct 19 10:49:16 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:16 cl017 last message repeated 11 times
Oct 19 10:49:16 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:16 cl017 last message repeated 25 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 12 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 42 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 12 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 87 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 3 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 7 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 57 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 30 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 29 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 34 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 30 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 29 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 30 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 last message repeated 30 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:17 cl017 last message repeated 11 times
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:17 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:19 cl017 last message repeated 765 times
Oct 19 10:49:19 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:19 cl017 last message repeated 294 times
Oct 19 10:49:19 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:19 cl017 last message repeated 11 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:20 cl017 last message repeated 41 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:20 cl017 last message repeated 11 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:20 cl017 last message repeated 47 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:20 cl017 last message repeated 11 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:20 cl017 last message repeated 41 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:20 cl017 last message repeated 11 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:20 cl017 last message repeated 36 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:20 cl017 last message repeated 11 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:20 cl017 last message repeated 39 times
Oct 19 10:49:20 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:20 cl017 last message repeated 119 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 16 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 2 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 8 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 2 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 11 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 7 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 11 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 11 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 30 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 11 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 38 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 11 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 42 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:21 cl017 last message repeated 11 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:21 cl017 last message repeated 49 times
Oct 19 10:49:21 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 last message repeated 146 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 last message repeated 10 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 last message repeated 10 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 last message repeated 9 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 last message repeated 3 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:22 cl017 last message repeated 6 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:22 cl017 last message repeated 10 times
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:22 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.18
Oct 19 10:49:23 cl017 last message repeated 16 times
Oct 19 10:49:23 cl017 [NOTICE  ] [EVT  ] No cluster node data for 192.168.1.19
Oct 19 10:49:23 cl017 last message repeated 11 times
Oct 19 10:49:23 cl017 [NOTICE  ] [GMI  ] EVS recovery of messages complete, transitioning to operational.
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] CLM CONFIGURATION CHANGE
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] New Configuration:
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.17
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.18
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.19
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] Members Left:
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] Members Joined:
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] ^I192.168.1.8
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.17
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.18
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.19
Oct 19 10:49:23 cl017 [NOTICE  ] [CLM  ] got nodejoin message 192.168.1.8
Oct 19 10:49:23 cl017 [NOTICE  ] [GMI  ] All services unplugged, unplugging processor
Oct 19 10:49:23 cl017 [NOTICE  ] [EVT  ] Channel Chan1, total 2, local 1
Oct 19 10:49:23 cl017 [NOTICE  ] [GMI  ] All processors unplugged, allowing messages to be transmitted.

Back to normal again.....



-- 
Mark Haverkamp <markh at osdl.org>




More information about the Openais mailing list