[Openais] TOTEM deadlock
Hans Feldt
Hans.Feldt at ericsson.com
Thu Nov 9 01:01:42 PST 2006
SW version: trunk 1291 with the ipc.c patch
I get deadlock when running a 4 processor UML cluster. I use the =
uml_switch daemon for networking. I can see in the uml_switch console =
that it drops packets (a real switch would behave the same in an =
overload situation).
The test case I am running is fairly simple, start 4 nodes with a one =
second delay between each. AMF has SW configured on each node. The last =
thing that happens is that AMF (p02) is synchronising the last joined =
processor (p05) and the token and/or messages are dropped by uml_switch =
and TOTEM does not recover from that.
The error does not occur every time, it seems to be timing dependent.
Without the ipc.c patch, I usually get segmentation faults instead.
Regards,
Hans
-------------- next part --------------
Nov 9 9:03:34.621162 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6ad, type 16
Nov 9 9:03:34.621304 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6f8, type 16
Nov 9 9:03:34.621449 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c743, type 9
Nov 9 9:03:34.855388 [totemsrp.c:2296] Retransmit List: 1d 1e 1f 20 21 22 =
Nov 9 9:03:34.858666 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c011, type 13
Nov 9 9:03:34.860013 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c232, type 9
Nov 9 9:03:34.860162 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c011, type 13
Nov 9 9:03:34.860301 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c232, type 9
Nov 9 9:03:34.860416 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c467, type 13
Nov 9 9:03:34.860554 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c011, type 9
Nov 9 9:03:34.860678 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c246, type 13
Nov 9 9:03:34.860802 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c011, type 9
Nov 9 9:03:34.860933 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c246, type 13
Nov 9 9:03:34.861053 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c011, type 9
Nov 9 9:03:34.861183 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c246, type 13
Nov 9 9:03:34.861313 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 560 bytes, ptr 0x4021c467, type 9
Nov 9 9:03:34.861437 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 540 bytes, ptr 0x4021c011, type 13
Nov 9 9:03:34.861566 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 298 bytes, ptr 0x4021c232, type 1
Nov 9 9:03:34.861680 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 366 bytes, ptr 0x4021c361, type 4
Nov 9 9:03:34.861810 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 592 bytes, ptr 0x4021c011, type 5
Nov 9 9:03:34.861936 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 1687 bytes, ptr 0x4021c011, type 6
Nov 9 9:03:34.862091 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6ad, type 16
Nov 9 9:03:34.862219 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6f8, type 16
Nov 9 9:03:34.862341 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 1687 bytes, ptr 0x4021c011, type 6
Nov 9 9:03:34.862554 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6ad, type 16
Nov 9 9:03:34.862689 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6f8, type 16
Nov 9 9:03:34.862813 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 1687 bytes, ptr 0x4021c011, type 6
Nov 9 9:03:34.862927 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6ad, type 16
Nov 9 9:03:34.863055 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6f8, type 16
Nov 9 9:03:34.863170 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 592 bytes, ptr 0x4021c743, type 5
Nov 9 9:03:35.641902 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:35.642949 [totemsrp.c:1240] Saving state aru 2e high seq recei=
ved 2e
Nov 9 9:03:35.644444 [totemsrp.c:2819] Storing new sequence id for ring 14
Nov 9 9:03:35.645315 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.664426 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.666455 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:35.668333 [totemsrp.c:2819] Storing new sequence id for ring 18
Nov 9 9:03:35.670981 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.672228 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:35.676527 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:35.676820 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.678696 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:35.679089 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:35.679302 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.679507 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.679721 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:35.679925 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.680129 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:35.680360 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:35.680600 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.680827 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.681061 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:35.681326 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:35.681565 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:35.681781 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:35.682596 [totemsrp.c:3953] Sending initial ORF token
Nov 9 9:03:36.151389 [amfcluster.c:0249] >timer_function_cluster_assign_w=
orkload_tmo: =
Nov 9 9:03:36.723248 [totemsrp.c:1266] Restoring instance->my_aru 2e my h=
igh seq received 2e
Nov 9 9:03:36.723678 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:36.724165 [totemsrp.c:2819] Storing new sequence id for ring 1c
Nov 9 9:03:36.724442 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.725275 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.737943 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:36.738873 [totemsrp.c:2819] Storing new sequence id for ring 20
Nov 9 9:03:36.741939 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.743251 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:36.743881 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:36.744135 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.744381 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:36.752672 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:36.753511 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.754087 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.754361 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:36.754574 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.754776 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:36.754994 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:36.755226 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.755479 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.755736 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:36.756063 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:36.756296 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:36.756508 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:36.757328 [totemsrp.c:3953] Sending initial ORF token
Nov 9 9:03:37.773507 [totemsrp.c:1266] Restoring instance->my_aru 2e my h=
igh seq received 2e
Nov 9 9:03:37.773958 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:37.774542 [totemsrp.c:2819] Storing new sequence id for ring 24
Nov 9 9:03:37.774818 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.775216 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.793015 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:37.794425 [totemsrp.c:2819] Storing new sequence id for ring 28
Nov 9 9:03:37.795474 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.809154 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:37.809882 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:37.810466 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.818237 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:37.820067 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:37.820410 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.820516 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.820628 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:37.820978 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.821089 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:37.821321 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:37.821426 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.821527 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.821634 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:37.821838 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:37.821941 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:37.822042 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:37.822412 [totemsrp.c:3953] Sending initial ORF token
Nov 9 9:03:38.848251 [totemsrp.c:1266] Restoring instance->my_aru 2e my h=
igh seq received 2e
Nov 9 9:03:38.849827 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:38.851357 [totemsrp.c:2819] Storing new sequence id for ring 2c
Nov 9 9:03:38.852340 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.862375 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.863972 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:38.864848 [totemsrp.c:2819] Storing new sequence id for ring 30
Nov 9 9:03:38.867832 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.869218 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:38.869930 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:38.870215 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.870685 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:38.871037 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:38.871546 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.875158 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.875706 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:38.876130 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.876503 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:38.876937 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:38.877343 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.877728 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.878068 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:38.878429 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:38.878805 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:38.879256 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:38.880529 [totemsrp.c:3953] Sending initial ORF token
Nov 9 9:03:39.882422 [totemsrp.c:1360] The token was lost in state 4 from=
timer 0x81896f0
Nov 9 9:03:39.883144 [totemsrp.c:1266] Restoring instance->my_aru 2e my h=
igh seq received 2e
Nov 9 9:03:39.885804 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:39.890841 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:39.891951 [totemsrp.c:2819] Storing new sequence id for ring 34
Nov 9 9:03:39.894970 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:39.896520 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:39.897246 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:39.897499 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.897606 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:39.897710 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:39.897811 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.897910 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.898013 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:39.898118 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.898230 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:39.898348 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:39.898471 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.898575 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.898682 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:39.898879 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:39.898980 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:39.899079 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:39.915004 [totemsrp.c:3953] Sending initial ORF token
Nov 9 9:03:40.943283 [totemsrp.c:1360] The token was lost in state 4 from=
timer 0x81896f0
Nov 9 9:03:40.944781 [totemsrp.c:1266] Restoring instance->my_aru 2e my h=
igh seq received 2e
Nov 9 9:03:40.955014 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:40.959070 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:40.960003 [totemsrp.c:2819] Storing new sequence id for ring 38
Nov 9 9:03:40.964367 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:40.964983 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:40.965663 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:40.965897 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.966417 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:40.966822 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:40.970245 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.970681 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.971063 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:40.971402 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.971769 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:40.972252 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:40.972617 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.972962 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.973298 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:40.973829 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:40.974224 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:40.974642 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:40.975937 [totemsrp.c:3953] Sending initial ORF token
Nov 9 9:03:42.004767 [totemsrp.c:1360] The token was lost in state 4 from=
timer 0x81896f0
Nov 9 9:03:42.005485 [totemsrp.c:1266] Restoring instance->my_aru 2e my h=
igh seq received 2e
Nov 9 9:03:42.013550 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:42.014592 [totemsrp.c:2665] Creating commit token because I am=
the rep.
Nov 9 9:03:42.015470 [totemsrp.c:2819] Storing new sequence id for ring 3c
Nov 9 9:03:42.018408 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:42.020385 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:42.020882 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:42.021115 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.021686 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:42.022099 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:42.023303 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.023759 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.024175 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:42.024532 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.024855 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:42.025217 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:42.025545 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.025908 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.026254 [totemsrp.c:1835] copying all old ring messages from=
1b-2e.
Nov 9 9:03:42.027089 [totemsrp.c:1874] Originated 20 messages in RECOVERY.
Nov 9 9:03:42.027765 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c 2d 2e =
Nov 9 9:03:42.028406 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:42.044793 [totemsrp.c:3953] Sending initial ORF token
Terminated
-------------- next part --------------
Nov 9 9:03:34.895204 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 1687 bytes, ptr 0x402
Nov 9 9:03:34.895311 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c
Nov 9 9:03:34.895414 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c
Nov 9 9:03:35.635975 [totemsrp.c:1360] The token was lost in state 1 from=
timer 0x81896f0
Nov 9 9:03:35.638167 [totemnet.c:1034] Receive multicast socket recv buff=
er size (202752 bytes).
Nov 9 9:03:35.638605 [totemnet.c:1040] Transmit multicast socket send buf=
fer size (202752 bytes).
Nov 9 9:03:35.642640 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.645253 [totemsrp.c:1240] Saving state aru 2c high seq recei=
ved 2c
Nov 9 9:03:35.646291 [totemsrp.c:2819] Storing new sequence id for ring 14
Nov 9 9:03:35.649228 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.662275 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.667447 [totemsrp.c:2819] Storing new sequence id for ring 18
Nov 9 9:03:35.667853 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.672815 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:35.673761 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:35.674182 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.674550 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:35.674670 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:35.674785 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.674884 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.674984 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:35.675088 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.675185 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:35.675284 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:35.675383 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.675480 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.675753 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:35.675921 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:35.676022 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:35.676128 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:36.661262 [amfcluster.c:0249] >timer_function_cluster_assign_w=
orkload_tmo: =
Nov 9 9:03:36.726965 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:36.728730 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.732231 [totemsrp.c:2819] Storing new sequence id for ring 1c
Nov 9 9:03:36.733026 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.736979 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.741869 [totemsrp.c:2819] Storing new sequence id for ring 20
Nov 9 9:03:36.743201 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.744293 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:36.752693 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:36.753536 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.754109 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:36.754378 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:36.754591 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.754792 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.755011 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:36.755243 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.755442 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:36.755694 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:36.756022 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.756261 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.756475 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:36.756715 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:36.756883 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:36.757064 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:37.784371 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:37.786600 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.789536 [totemsrp.c:2819] Storing new sequence id for ring 24
Nov 9 9:03:37.790187 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.792638 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.795398 [totemsrp.c:2819] Storing new sequence id for ring 28
Nov 9 9:03:37.795988 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.810386 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:37.816475 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:37.816905 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.817013 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:37.817118 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:37.817227 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.817329 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.817434 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:37.817549 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.817676 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:37.817816 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:37.817941 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.818060 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.818189 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:37.818345 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:37.818448 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:37.818547 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:38.848232 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:38.850871 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.852725 [totemsrp.c:2819] Storing new sequence id for ring 2c
Nov 9 9:03:38.855490 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.859462 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.865404 [totemsrp.c:2819] Storing new sequence id for ring 30
Nov 9 9:03:38.868412 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.870137 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:38.870749 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:38.871122 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.874443 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:38.875234 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:38.875777 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.876197 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.876557 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:38.876997 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.877393 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:38.877782 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:38.878113 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.878474 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.878861 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:38.879309 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:38.879639 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:38.879968 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:39.885690 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:39.888701 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:39.892884 [totemsrp.c:2819] Storing new sequence id for ring 34
Nov 9 9:03:39.896081 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:39.908531 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:39.910751 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:39.920304 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.920771 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:39.921102 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:39.921211 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.921318 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.921419 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:39.921535 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.921633 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:39.921737 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:39.921846 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.921965 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.922074 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:39.922214 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:39.922328 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:39.922449 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:40.946623 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:40.947460 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:40.960347 [totemsrp.c:2819] Storing new sequence id for ring 38
Nov 9 9:03:40.962689 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:40.965827 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:40.966499 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:40.966910 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.970392 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:40.970801 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:40.971130 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.971461 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.971844 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:40.972329 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.972680 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:40.973022 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:40.973358 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.973768 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.974171 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:40.974666 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:40.975041 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:40.975368 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:42.008156 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:42.010618 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:42.016036 [totemsrp.c:2819] Storing new sequence id for ring 3c
Nov 9 9:03:42.018789 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:42.021046 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:42.021768 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:42.022195 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.023409 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:42.023835 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:42.024234 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.024586 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.024911 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:42.025267 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.025598 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:42.025973 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:42.026308 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.027068 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.027745 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:42.028478 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:42.029015 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 =
Nov 9 9:03:42.029719 [totemsrp.c:1878] Not Originated for recovery: =
Terminated
-------------- next part --------------
Nov 9 9:03:34.629680 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c011, type 16
Nov 9 9:03:34.629797 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c05c, type 16
Nov 9 9:03:34.629907 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 366 bytes, ptr 0x4021c0a7, type 4
Nov 9 9:03:34.630017 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 592 bytes, ptr 0x4021c21a, type 5
Nov 9 9:03:35.643011 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.646339 [totemsrp.c:1240] Saving state aru 1a high seq recei=
ved 28
Nov 9 9:03:35.648712 [totemsrp.c:2819] Storing new sequence id for ring 14
Nov 9 9:03:35.649639 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.664605 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.670303 [totemsrp.c:2819] Storing new sequence id for ring 18
Nov 9 9:03:35.671483 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.676799 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:35.678908 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:35.679153 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.679362 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:35.679574 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:35.679781 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.679983 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.680194 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:35.680433 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.680665 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:35.680894 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:35.681125 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.681387 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.681635 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:35.681871 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:35.682070 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:35.682249 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Nov 9 9:03:36.114820 [amfcluster.c:0249] >timer_function_cluster_assign_w=
orkload_tmo: =
Nov 9 9:03:36.719615 [totemsrp.c:1360] The token was lost in state 4 from=
timer 0x81896f0
Nov 9 9:03:36.720774 [totemsrp.c:1266] Restoring instance->my_aru 1a my h=
igh seq received 28
Nov 9 9:03:36.721519 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.730547 [totemsrp.c:2819] Storing new sequence id for ring 1c
Nov 9 9:03:36.731145 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.735739 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.740245 [totemsrp.c:2819] Storing new sequence id for ring 20
Nov 9 9:03:36.741020 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.744926 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:36.745315 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:36.746690 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.746925 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:36.747129 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:36.747344 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.747598 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.747836 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:36.748071 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.748274 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:36.748499 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:36.748718 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.749044 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.749271 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:36.749509 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:36.749713 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:36.749885 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Nov 9 9:03:37.773646 [totemsrp.c:1266] Restoring instance->my_aru 1a my h=
igh seq received 28
Nov 9 9:03:37.774517 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.788224 [totemsrp.c:2819] Storing new sequence id for ring 24
Nov 9 9:03:37.793055 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.793818 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.807071 [totemsrp.c:2819] Storing new sequence id for ring 28
Nov 9 9:03:37.807941 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.811302 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:37.811570 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:37.811692 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.811817 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:37.811921 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:37.812024 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.812122 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.812328 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:37.812430 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.812533 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:37.812652 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:37.812783 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.812895 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.813012 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:37.813206 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:37.813309 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:37.813409 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Nov 9 9:03:38.845003 [totemsrp.c:1360] The token was lost in state 4 from=
timer 0x81896f0
Nov 9 9:03:38.845705 [totemsrp.c:1266] Restoring instance->my_aru 1a my h=
igh seq received 28
Nov 9 9:03:38.848268 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.855507 [totemsrp.c:2819] Storing new sequence id for ring 2c
Nov 9 9:03:38.856459 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.859685 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.867320 [totemsrp.c:2819] Storing new sequence id for ring 30
Nov 9 9:03:38.868783 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.871009 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:38.874618 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:38.875283 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.875839 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:38.876254 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:38.876605 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.877050 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.877449 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:38.877829 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.878156 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:38.878522 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:38.878916 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.879359 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.879684 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:38.880013 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:38.880321 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:38.880630 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Nov 9 9:03:39.885672 [totemsrp.c:1266] Restoring instance->my_aru 1a my h=
igh seq received 28
Nov 9 9:03:39.888683 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:39.894945 [totemsrp.c:2819] Storing new sequence id for ring 34
Nov 9 9:03:39.896504 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:39.910914 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:39.914588 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:39.915071 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.915876 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:39.916112 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:39.916318 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.916480 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.916583 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:39.916684 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.916858 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:39.916971 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:39.917082 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.917186 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.917297 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:39.917438 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:39.917539 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:39.917641 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Nov 9 9:03:40.954939 [totemsrp.c:1266] Restoring instance->my_aru 1a my h=
igh seq received 28
Nov 9 9:03:40.957209 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:40.962033 [totemsrp.c:2819] Storing new sequence id for ring 38
Nov 9 9:03:40.963212 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:40.966794 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:40.970533 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:40.970914 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.971275 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:40.971616 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:40.972003 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.972475 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.972832 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:40.973160 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.973486 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:40.973907 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:40.974312 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.974732 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.975086 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:40.975409 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:40.975717 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:40.976033 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Nov 9 9:03:42.008649 [totemsrp.c:1266] Restoring instance->my_aru 1a my h=
igh seq received 28
Nov 9 9:03:42.010818 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:42.017904 [totemsrp.c:2819] Storing new sequence id for ring 3c
Nov 9 9:03:42.019391 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:42.022068 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:42.023481 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:42.023896 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.024287 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:42.024633 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:42.024961 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.025312 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.025651 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:42.026023 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.026355 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:42.027203 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:42.027842 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.028447 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.028997 [totemsrp.c:1835] copying all old ring messages from=
1b-28.
Nov 9 9:03:42.029790 [totemsrp.c:1874] Originated 12 messages in RECOVERY.
Nov 9 9:03:42.030339 [totemsrp.c:1877] Originated for recovery: 1d 1e 1f =
20 21 22 23 24 25 26 27 28 =
Nov 9 9:03:42.030872 [totemsrp.c:1878] Not Originated for recovery: 1b 1c =
Terminated
-------------- next part --------------
Nov 9 9:03:34.880530 [amf.c:1731] >message_handler_req_exec_amf_sync_data=
: Component 'Comp-2' deserialised
Nov 9 9:03:34.880631 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6ad, type 16
Nov 9 9:03:34.880734 [amf.c:1739] >message_handler_req_exec_amf_sync_data=
: Healthcheck 'key1' deserialised
Nov 9 9:03:34.880842 [amf.c:1682] >message_handler_req_exec_amf_sync_data=
: rec 70 bytes, ptr 0x4021c6f8, type 16
Nov 9 9:03:34.880952 [amf.c:1739] >message_handler_req_exec_amf_sync_data=
: Healthcheck 'key0' deserialised
Nov 9 9:03:35.639954 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.647264 [totemsrp.c:1240] Saving state aru 2c high seq recei=
ved 2c
Nov 9 9:03:35.647840 [totemsrp.c:2819] Storing new sequence id for ring 14
Nov 9 9:03:35.648273 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.660458 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:35.668866 [totemsrp.c:2819] Storing new sequence id for ring 18
Nov 9 9:03:35.669792 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:35.677747 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:35.678015 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:35.678134 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.678254 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:35.678385 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:35.678507 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.678881 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.699052 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:35.699177 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.699276 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:35.699377 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:35.699476 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:35.699713 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:35.699838 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:35.700021 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:35.700132 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:35.700230 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:36.721688 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:36.722254 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.730140 [totemsrp.c:2819] Storing new sequence id for ring 1c
Nov 9 9:03:36.731590 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.734132 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:36.740546 [totemsrp.c:2819] Storing new sequence id for ring 20
Nov 9 9:03:36.741263 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:36.745470 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:36.746031 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:36.746270 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.746480 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:36.746673 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:36.746896 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.747112 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.747324 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:36.747578 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.747817 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:36.748055 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:36.748258 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:36.748474 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:36.748701 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:36.749084 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:36.749305 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:36.749543 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:37.770260 [totemsrp.c:1360] The token was lost in state 4 from=
timer 0x81896f0
Nov 9 9:03:37.771317 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:37.773723 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.788395 [totemsrp.c:2819] Storing new sequence id for ring 24
Nov 9 9:03:37.788880 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.791680 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:37.807515 [totemsrp.c:2819] Storing new sequence id for ring 28
Nov 9 9:03:37.808324 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:37.814018 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:37.814297 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:37.814433 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.814553 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:37.814676 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:37.814809 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.814927 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.815031 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:37.815132 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.815236 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:37.815337 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:37.815442 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:37.815555 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:37.815671 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:37.815884 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:37.815986 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:37.816086 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:38.846562 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:38.847263 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.854362 [totemsrp.c:2819] Storing new sequence id for ring 2c
Nov 9 9:03:38.854922 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.858050 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:38.865984 [totemsrp.c:2819] Storing new sequence id for ring 30
Nov 9 9:03:38.866487 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:38.871847 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:38.872398 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:38.872544 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.872657 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:38.872783 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:38.872902 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.873017 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.873134 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:38.873242 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.873339 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:38.873438 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:38.873537 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:38.873639 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:38.873751 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:38.873960 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:38.874064 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:38.874162 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:39.883675 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:39.884226 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:39.893459 [totemsrp.c:2819] Storing new sequence id for ring 34
Nov 9 9:03:39.894023 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:39.911610 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:39.912088 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:39.912347 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.912468 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:39.912572 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:39.912673 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.912773 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.912874 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:39.912978 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.913092 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:39.913214 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:39.913339 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:39.913468 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:39.913589 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:39.913794 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:39.913898 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:39.913998 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:40.946619 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:40.952183 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:40.960813 [totemsrp.c:2819] Storing new sequence id for ring 38
Nov 9 9:03:40.961333 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:40.967818 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:40.968092 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:40.968209 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.968324 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:40.968427 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:40.968534 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.968645 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.968758 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:40.968869 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.968981 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:40.969087 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:40.969215 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:40.969330 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:40.969440 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:40.969580 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:40.969682 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:40.969796 [totemsrp.c:1878] Not Originated for recovery: =
Nov 9 9:03:42.006324 [totemsrp.c:1266] Restoring instance->my_aru 2c my h=
igh seq received 2c
Nov 9 9:03:42.006997 [totemsrp.c:1662] entering GATHER state.
Nov 9 9:03:42.016536 [totemsrp.c:2819] Storing new sequence id for ring 3c
Nov 9 9:03:42.017155 [totemsrp.c:1698] entering COMMIT state.
Nov 9 9:03:42.023116 [totemsrp.c:1732] entering RECOVERY state.
Nov 9 9:03:42.044632 [totemsrp.c:1766] position [0] member 192.168.0.2:
Nov 9 9:03:42.044940 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.045179 [totemsrp.c:1776] aru 2e high delivered 2e received =
flag 0
Nov 9 9:03:42.045299 [totemsrp.c:1766] position [1] member 192.168.0.3:
Nov 9 9:03:42.045564 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.045672 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.045781 [totemsrp.c:1766] position [2] member 192.168.0.4:
Nov 9 9:03:42.045894 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.046003 [totemsrp.c:1776] aru 1a high delivered 1a received =
flag 0
Nov 9 9:03:42.046122 [totemsrp.c:1766] position [3] member 192.168.0.5:
Nov 9 9:03:42.046232 [totemsrp.c:1770] previous ring seq 16 rep 192.168.0=
.2
Nov 9 9:03:42.046337 [totemsrp.c:1776] aru 2c high delivered 2c received =
flag 0
Nov 9 9:03:42.046506 [totemsrp.c:1835] copying all old ring messages from=
1b-2c.
Nov 9 9:03:42.046727 [totemsrp.c:1874] Originated 18 messages in RECOVERY.
Nov 9 9:03:42.046832 [totemsrp.c:1877] Originated for recovery: 1b 1c 1d =
1e 1f 20 21 22 23 24 25 26 27 28 29 2a 2b 2c =
Nov 9 9:03:42.046930 [totemsrp.c:1878] Not Originated for recovery: =
Terminated
More information about the Openais
mailing list