[Openais] Corosync segfaults after restarting network service (Centos-5.5)
Benjamin Ash
bash at intelerad.com
Wed Mar 30 09:39:45 PDT 2011
Hi,
It seems that Corosync is segfaulting whenever I restart the network.
steps to reproduce:
1) start corosync with the following service configuration in /etc/corosync/service.d:
service {
# Load the Pacemaker Cluster Resource Manager
name: pacemaker
ver: 0
}
2) service network restart
3) wait a few seconds for corosync to segfault
version info:
Corosync Cluster Engine, version '1.3.0'
Copyright (c) 2006-2009 Red Hat, Inc.
Please see the backtrace and fplay output in the attached files.
Thanks,
-ben
--
Benjamin Ash, RHCE
Technical Manager, Client Services
Intelerad Medical Systems Inc.
Central Station Building
895 de la Gauchetiere Street W., Suite 400
Montreal, QC Canada H3B 4G1
tel: +1.514.931.6222 x7311
fax: +1.514.931.4653
bash at intelerad.com
www.intelerad.com
This email or any attachments may contain confidential or legally privileged information intended for the sole use of the addressees. Any use, redistribution, disclosure, or reproduction of this information, except as intended, is prohibited. If you received this email in error, please notify the sender and remove all copies of the message, including any attachments.
-------------- next part --------------
Starting replay: head [26219] tail [0]
rec=[1] Log Message=Corosync Cluster Engine ('1.3.0'): started and ready to provide service.
rec=[2] Log Message=Corosync built-in features: nss rdma
rec=[3] Log Message=Successfully read main configuration file '/etc/corosync/corosync.conf'.
rec=[4] Log Message=Token Timeout (1000 ms) retransmit timeout (238 ms)
rec=[5] Log Message=token hold (180 ms) retransmits before loss (4 retrans)
rec=[6] Log Message=join (50 ms) send_join (0 ms) consensus (1200 ms) merge (200 ms)
rec=[7] Log Message=downcheck (1000 ms) fail to recv const (50 msgs)
rec=[8] Log Message=seqno unchanged const (30 rotations) Maximum network MTU 1402
rec=[9] Log Message=window size per rotation (50 messages) maximum messages per rotation (17 messages)
rec=[10] Log Message=send threads (0 threads)
rec=[11] Log Message=RRP token expired timeout (238 ms)
rec=[12] Log Message=RRP token problem counter (2000 ms)
rec=[13] Log Message=RRP threshold (10 problem count)
rec=[14] Log Message=RRP mode set to none.
rec=[15] Log Message=heartbeat_failures_allowed (0)
rec=[16] Log Message=max_network_delay (50 ms)
rec=[17] Log Message=HeartBeat is Disabled. To enable set heartbeat_failures_allowed > 0
rec=[18] Log Message=Initializing transport (UDP/IP Multicast).
rec=[19] Log Message=Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
rec=[20] Log Message=you are using ipc api v2
rec=[21] Log Message=Receive multicast socket recv buffer size (262142 bytes).
rec=[22] Log Message=Transmit multicast socket send buffer size (262142 bytes).
rec=[23] Log Message=The network interface [10.93.2.160] is now up.
rec=[24] Log Message=Created or loaded sequence id 4432.10.93.2.160 for this ring.
rec=[25] Log Message=debug: pcmk_user_lookup: Cluster user root has uid=0 gid=0
rec=[26] Log Message=info: process_ais_conf: Reading configure
rec=[27] Log Message=info: config_find_init: Local handle: 7739444317642555395 for logging
rec=[28] Log Message=info: config_find_next: Processing additional logging options...
rec=[29] Log Message=info: get_config_opt: Found 'on' for option: debug
rec=[30] Log Message=info: get_config_opt: Found 'yes' for option: to_logfile
rec=[31] Log Message=info: get_config_opt: Found '/var/log/cluster/corosync.log' for option: logfile
rec=[32] Log Message=info: get_config_opt: Found 'yes' for option: to_syslog
rec=[33] Log Message=info: get_config_opt: Defaulting to 'daemon' for option: syslog_facility
rec=[34] Log Message=info: config_find_init: Local handle: 5650605097994944516 for service
rec=[35] Log Message=info: config_find_next: Processing additional service options...
rec=[36] Log Message=info: get_config_opt: Defaulting to 'pcmk' for option: clustername
rec=[37] Log Message=info: get_config_opt: Defaulting to 'no' for option: use_logd
rec=[38] Log Message=info: get_config_opt: Defaulting to 'no' for option: use_mgmtd
rec=[39] Log Message=info: pcmk_startup: CRM: Initialized
rec=[40] Log Message=Logging: Initialized pcmk_startup
rec=[41] Log Message=info: pcmk_startup: Maximum core file size is: 18446744073709551615
rec=[42] Log Message=debug: pcmk_user_lookup: Cluster user hacluster has uid=103 gid=105
rec=[43] Log Message=info: pcmk_startup: Service: 9
rec=[44] Log Message=info: pcmk_startup: Local hostname: imsha1.dyn.intelerad
rec=[45] Log Message=info: pcmk_update_nodeid: Local node id: 2684509450
rec=[46] Log Message=info: update_member: Creating entry for node 2684509450 born on 0
rec=[47] Log Message=info: update_member: 0x671ad0 Node 2684509450 now known as imsha1.dyn.intelerad (was: (null))
rec=[48] Log Message=info: update_member: Node imsha1.dyn.intelerad now has 1 quorum votes (was 0)
rec=[49] Log Message=info: update_member: Node 2684509450/imsha1.dyn.intelerad is now: member
rec=[50] Log Message=info: spawn_child: Forked child 4809 for process stonithd
rec=[51] Log Message=debug: pcmk_user_lookup: Cluster user hacluster has uid=103 gid=105
rec=[52] Log Message=info: spawn_child: Forked child 4810 for process cib
rec=[53] Log Message=info: spawn_child: Forked child 4811 for process lrmd
rec=[54] Log Message=debug: pcmk_user_lookup: Cluster user hacluster has uid=103 gid=105
rec=[55] Log Message=info: spawn_child: Forked child 4812 for process attrd
rec=[56] Log Message=debug: pcmk_user_lookup: Cluster user hacluster has uid=103 gid=105
rec=[57] Log Message=info: spawn_child: Forked child 4813 for process pengine
rec=[58] Log Message=debug: pcmk_user_lookup: Cluster user hacluster has uid=103 gid=105
rec=[59] Log Message=info: spawn_child: Forked child 4814 for process crmd
rec=[60] Log Message=Service engine loaded: Pacemaker Cluster Manager 1.0.9
rec=[61] Log Message=Service engine loaded: corosync extended virtual synchrony service
rec=[62] Log Message=Service engine loaded: corosync configuration service
rec=[63] Log Message=Service engine loaded: corosync cluster closed process group service v1.01
rec=[64] Log Message=Service engine loaded: corosync cluster config database access v1.01
rec=[65] Log Message=Service engine loaded: corosync profile loading service
rec=[66] Log Message=Service engine loaded: corosync cluster quorum service v0.1
rec=[67] Log Message=Compatibility mode set to whitetank. Using V1 and V2 of the synchronization engine.
rec=[68] Log Message=entering GATHER state from 15.
rec=[69] Log Message=Creating commit token because I am the rep.
rec=[70] Log Message=Saving state aru 0 high seq received 0
rec=[71] Log Message=Storing new sequence id for ring 1154
rec=[72] Log Message=entering COMMIT state.
rec=[73] Log Message=got commit token
rec=[74] Log Message=entering RECOVERY state.
rec=[75] Log Message=position [0] member 10.93.2.160:
rec=[76] Log Message=previous ring seq 4432 rep 10.93.2.160
rec=[77] Log Message=aru 0 high delivered 0 received flag 1
rec=[78] Log Message=Did not need to originate any messages in recovery.
rec=[79] Log Message=got commit token
rec=[80] Log Message=Sending initial ORF token
rec=[81] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru 0
rec=[82] Log Message=install seq 0 aru 0 high seq received 0
rec=[83] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
rec=[84] Log Message=install seq 0 aru 0 high seq received 0
rec=[85] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
rec=[86] Log Message=install seq 0 aru 0 high seq received 0
rec=[87] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
rec=[88] Log Message=install seq 0 aru 0 high seq received 0
rec=[89] Log Message=retrans flag count 4 token aru 0 install seq 0 aru 0 0
rec=[90] Log Message=Resetting old ring state
rec=[91] Log Message=recovery to regular 1-0
rec=[92] Log Message=Delivering to app 1 to 0
rec=[93] Log Message=notice: pcmk_peer_update: Transitional membership event on ring 4436: memb=0, new=0, lost=0
rec=[94] Log Message=notice: pcmk_peer_update: Stable membership event on ring 4436: memb=1, new=1, lost=0
rec=[95] Log Message=info: pcmk_peer_update: NEW: imsha1.dyn.intelerad 2684509450
rec=[96] Log Message=debug: pcmk_peer_update: Node 2684509450 has address r(0) ip(10.93.2.160)
rec=[97] Log Message=info: pcmk_peer_update: MEMB: imsha1.dyn.intelerad 2684509450
rec=[98] Log Message=debug: send_cluster_id: Local update: id=2684509450, born=0, seq=4436
rec=[99] Log Message=info: update_member: Node imsha1.dyn.intelerad now has process list: 00000000000000000000000000013312 (78610)
rec=[100] Log Message=This node is within the primary component and will provide service.
rec=[101] Log Message=entering OPERATIONAL state.
rec=[102] Log Message=A processor joined or left the membership and a new membership was formed.
rec=[103] Log Message=mcasted message added to pending queue
rec=[104] Log Message=Delivering 0 to 1
rec=[105] Log Message=Delivering MCAST message with seq 1 to pending delivery queue
rec=[106] Log Message=debug: pcmk_cluster_id_callback: Node update: imsha1.dyn.intelerad (1.0.10)
rec=[107] Log Message=confchg entries 1
rec=[108] Log Message=Barrier Start Received From -1610457846
rec=[109] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[110] Log Message=Synchronization barrier completed
rec=[111] Log Message=Synchronization actions starting for (dummy CLM service)
rec=[112] Log Message=mcasted message added to pending queue
rec=[113] Log Message=Delivering 1 to 2
rec=[114] Log Message=Delivering MCAST message with seq 2 to pending delivery queue
rec=[115] Log Message=mcasted message added to pending queue
rec=[116] Log Message=releasing messages up to and including 1
rec=[117] Log Message=Delivering 2 to 3
rec=[118] Log Message=Delivering MCAST message with seq 3 to pending delivery queue
rec=[119] Log Message=confchg entries 1
rec=[120] Log Message=Barrier Start Received From -1610457846
rec=[121] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[122] Log Message=Synchronization barrier completed
rec=[123] Log Message=Committing synchronization for (dummy CLM service)
rec=[124] Log Message=Synchronization actions starting for (dummy AMF service)
rec=[125] Log Message=releasing messages up to and including 2
rec=[126] Log Message=mcasted message added to pending queue
rec=[127] Log Message=releasing messages up to and including 3
rec=[128] Log Message=Delivering 3 to 4
rec=[129] Log Message=Delivering MCAST message with seq 4 to pending delivery queue
rec=[130] Log Message=confchg entries 1
rec=[131] Log Message=Barrier Start Received From -1610457846
rec=[132] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[133] Log Message=Synchronization barrier completed
rec=[134] Log Message=Committing synchronization for (dummy AMF service)
rec=[135] Log Message=Synchronization actions starting for (dummy CKPT service)
rec=[136] Log Message=mcasted message added to pending queue
rec=[137] Log Message=releasing messages up to and including 4
rec=[138] Log Message=Delivering 4 to 5
rec=[139] Log Message=Delivering MCAST message with seq 5 to pending delivery queue
rec=[140] Log Message=confchg entries 1
rec=[141] Log Message=Barrier Start Received From -1610457846
rec=[142] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[143] Log Message=Synchronization barrier completed
rec=[144] Log Message=Committing synchronization for (dummy CKPT service)
rec=[145] Log Message=Synchronization actions starting for (dummy EVT service)
rec=[146] Log Message=entering GATHER state from 11.
rec=[147] Log Message=mcasted message added to pending queue
rec=[148] Log Message=got commit token
rec=[149] Log Message=Saving state aru 6 high seq received 6
rec=[150] Log Message=Storing new sequence id for ring 1158
rec=[151] Log Message=entering COMMIT state.
rec=[152] Log Message=got commit token
rec=[153] Log Message=entering RECOVERY state.
rec=[154] Log Message=TRANS [0] member 10.93.2.160:
rec=[155] Log Message=position [0] member 10.93.2.80:
rec=[156] Log Message=previous ring seq 4436 rep 10.93.2.80
rec=[157] Log Message=aru d high delivered d received flag 1
rec=[158] Log Message=position [1] member 10.93.2.160:
rec=[159] Log Message=previous ring seq 4436 rep 10.93.2.160
rec=[160] Log Message=aru 6 high delivered 5 received flag 1
rec=[161] Log Message=Did not need to originate any messages in recovery.
rec=[162] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 0, aru ffffffff
rec=[163] Log Message=install seq 0 aru 0 high seq received 0
rec=[164] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 1, aru 0
rec=[165] Log Message=install seq 0 aru 0 high seq received 0
rec=[166] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 2, aru 0
rec=[167] Log Message=install seq 0 aru 0 high seq received 0
rec=[168] Log Message=token retrans flag is 0 my set retrans flag0 retrans queue empty 1 count 3, aru 0
rec=[169] Log Message=install seq 0 aru 0 high seq received 0
rec=[170] Log Message=retrans flag count 4 token aru 0 install seq 0 aru 0 0
rec=[171] Log Message=Resetting old ring state
rec=[172] Log Message=recovery to regular 1-0
rec=[173] Log Message=Delivering to app 6 to 6
rec=[174] Log Message=Delivering 5 to 6
rec=[175] Log Message=Delivering MCAST message with seq 6 to pending delivery queue
rec=[176] Log Message=notice: pcmk_peer_update: Transitional membership event on ring 4440: memb=1, new=0, lost=0
rec=[177] Log Message=info: pcmk_peer_update: memb: imsha1.dyn.intelerad 2684509450
rec=[178] Log Message=notice: pcmk_peer_update: Stable membership event on ring 4440: memb=2, new=1, lost=0
rec=[179] Log Message=info: update_member: Creating entry for node 1342332170 born on 4440
rec=[180] Log Message=info: update_member: Node 1342332170/unknown is now: member
rec=[181] Log Message=info: pcmk_peer_update: NEW: .pending. 1342332170
rec=[182] Log Message=debug: pcmk_peer_update: Node 1342332170 has address r(0) ip(10.93.2.80)
rec=[183] Log Message=info: pcmk_peer_update: MEMB: .pending. 1342332170
rec=[184] Log Message=info: pcmk_peer_update: MEMB: imsha1.dyn.intelerad 2684509450
rec=[185] Log Message=debug: pcmk_peer_update: 1 nodes changed
rec=[186] Log Message=info: send_member_notification: Sending membership update 4440 to 0 children
rec=[187] Log Message=debug: send_cluster_id: Local update: id=2684509450, born=4440, seq=4440
rec=[188] Log Message=info: update_member: 0x671ad0 Node 2684509450 ((null)) born on: 4440
rec=[189] Log Message=This node is within the primary component and will provide service.
rec=[190] Log Message=entering OPERATIONAL state.
rec=[191] Log Message=A processor joined or left the membership and a new membership was formed.
rec=[192] Log Message=mcasted message added to pending queue
rec=[193] Log Message=Delivering 0 to 1
rec=[194] Log Message=Delivering MCAST message with seq 1 to pending delivery queue
rec=[195] Log Message=debug: pcmk_cluster_id_callback: Node update: imsha1.dyn.intelerad (1.0.10)
rec=[196] Log Message=confchg entries 2
rec=[197] Log Message=Barrier Start Received From -1610457846
rec=[198] Log Message=Barrier completion status for nodeid 1342332170 = 0.
rec=[199] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[200] Log Message=Received ringid(10.93.2.80:4440) seq 1
rec=[201] Log Message=Received ringid(10.93.2.80:4440) seq 2
rec=[202] Log Message=Delivering 1 to 2
rec=[203] Log Message=Delivering MCAST message with seq 2 to pending delivery queue
rec=[204] Log Message=debug: pcmk_cluster_id_callback: Node update: imsha2.dyn.intelerad (1.0.10)
rec=[205] Log Message=info: update_member: 0x67c0a0 Node 1342332170 (imsha2.dyn.intelerad) born on: 4432
rec=[206] Log Message=info: update_member: 0x67c0a0 Node 1342332170 now known as imsha2.dyn.intelerad (was: (null))
rec=[207] Log Message=info: update_member: Node imsha2.dyn.intelerad now has process list: 00000000000000000000000000013312 (78610)
rec=[208] Log Message=info: update_member: Node imsha2.dyn.intelerad now has 1 quorum votes (was 0)
rec=[209] Log Message=info: send_member_notification: Sending membership update 4440 to 0 children
rec=[210] Log Message=confchg entries 2
rec=[211] Log Message=Barrier Start Received From 1342332170
rec=[212] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[213] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[214] Log Message=Synchronization barrier completed
rec=[215] Log Message=Synchronization actions starting for (dummy CLM service)
rec=[216] Log Message=Received ringid(10.93.2.80:4440) seq 3
rec=[217] Log Message=Delivering 2 to 3
rec=[218] Log Message=Delivering MCAST message with seq 3 to pending delivery queue
rec=[219] Log Message=mcasted message added to pending queue
rec=[220] Log Message=releasing messages up to and including 2
rec=[221] Log Message=Delivering 3 to 4
rec=[222] Log Message=Delivering MCAST message with seq 4 to pending delivery queue
rec=[223] Log Message=Received ringid(10.93.2.80:4440) seq 4
rec=[224] Log Message=Received ringid(10.93.2.80:4440) seq 5
rec=[225] Log Message=Delivering 4 to 5
rec=[226] Log Message=Delivering MCAST message with seq 5 to pending delivery queue
rec=[227] Log Message=confchg entries 2
rec=[228] Log Message=Barrier Start Received From 1342332170
rec=[229] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[230] Log Message=Barrier completion status for nodeid -1610457846 = 0.
rec=[231] Log Message=mcasted message added to pending queue
rec=[232] Log Message=releasing messages up to and including 3
rec=[233] Log Message=Delivering 5 to 6
rec=[234] Log Message=Delivering MCAST message with seq 6 to pending delivery queue
rec=[235] Log Message=confchg entries 2
rec=[236] Log Message=Barrier Start Received From -1610457846
rec=[237] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[238] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[239] Log Message=Synchronization barrier completed
rec=[240] Log Message=Committing synchronization for (dummy CLM service)
rec=[241] Log Message=Synchronization actions starting for (dummy AMF service)
rec=[242] Log Message=Received ringid(10.93.2.80:4440) seq 6
rec=[243] Log Message=releasing messages up to and including 5
rec=[244] Log Message=mcasted message added to pending queue
rec=[245] Log Message=releasing messages up to and including 6
rec=[246] Log Message=Delivering 6 to 7
rec=[247] Log Message=Delivering MCAST message with seq 7 to pending delivery queue
rec=[248] Log Message=confchg entries 2
rec=[249] Log Message=Barrier Start Received From -1610457846
rec=[250] Log Message=Barrier completion status for nodeid 1342332170 = 0.
rec=[251] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[252] Log Message=Received ringid(10.93.2.80:4440) seq 7
rec=[253] Log Message=Received ringid(10.93.2.80:4440) seq 8
rec=[254] Log Message=Delivering 7 to 8
rec=[255] Log Message=Delivering MCAST message with seq 8 to pending delivery queue
rec=[256] Log Message=confchg entries 2
rec=[257] Log Message=Barrier Start Received From 1342332170
rec=[258] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[259] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[260] Log Message=Synchronization barrier completed
rec=[261] Log Message=Committing synchronization for (dummy AMF service)
rec=[262] Log Message=Synchronization actions starting for (dummy CKPT service)
rec=[263] Log Message=releasing messages up to and including 8
rec=[264] Log Message=Received ringid(10.93.2.80:4440) seq 9
rec=[265] Log Message=Delivering 8 to 9
rec=[266] Log Message=Delivering MCAST message with seq 9 to pending delivery queue
rec=[267] Log Message=confchg entries 2
rec=[268] Log Message=Barrier Start Received From 1342332170
rec=[269] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[270] Log Message=Barrier completion status for nodeid -1610457846 = 0.
rec=[271] Log Message=mcasted message added to pending queue
rec=[272] Log Message=Delivering 9 to a
rec=[273] Log Message=Delivering MCAST message with seq a to pending delivery queue
rec=[274] Log Message=confchg entries 2
rec=[275] Log Message=Barrier Start Received From -1610457846
rec=[276] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[277] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[278] Log Message=Synchronization barrier completed
rec=[279] Log Message=Committing synchronization for (dummy CKPT service)
rec=[280] Log Message=Synchronization actions starting for (dummy EVT service)
rec=[281] Log Message=Received ringid(10.93.2.80:4440) seq a
rec=[282] Log Message=mcasted message added to pending queue
rec=[283] Log Message=releasing messages up to and including 9
rec=[284] Log Message=Delivering a to b
rec=[285] Log Message=Delivering MCAST message with seq b to pending delivery queue
rec=[286] Log Message=Received ringid(10.93.2.80:4440) seq b
rec=[287] Log Message=Received ringid(10.93.2.80:4440) seq c
rec=[288] Log Message=Delivering b to c
rec=[289] Log Message=Delivering MCAST message with seq c to pending delivery queue
rec=[290] Log Message=releasing messages up to and including a
rec=[291] Log Message=Received ringid(10.93.2.80:4440) seq d
rec=[292] Log Message=Delivering c to d
rec=[293] Log Message=Delivering MCAST message with seq d to pending delivery queue
rec=[294] Log Message=mcasted message added to pending queue
rec=[295] Log Message=releasing messages up to and including c
rec=[296] Log Message=Delivering d to e
rec=[297] Log Message=Delivering MCAST message with seq e to pending delivery queue
rec=[298] Log Message=Received ringid(10.93.2.80:4440) seq e
rec=[299] Log Message=releasing messages up to and including d
rec=[300] Log Message=mcasted message added to pending queue
rec=[301] Log Message=releasing messages up to and including e
rec=[302] Log Message=Delivering e to f
rec=[303] Log Message=Delivering MCAST message with seq f to pending delivery queue
rec=[304] Log Message=confchg entries 2
rec=[305] Log Message=Barrier Start Received From -1610457846
rec=[306] Log Message=Barrier completion status for nodeid 1342332170 = 0.
rec=[307] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[308] Log Message=Received ringid(10.93.2.80:4440) seq f
rec=[309] Log Message=Received ringid(10.93.2.80:4440) seq 10
rec=[310] Log Message=Delivering f to 10
rec=[311] Log Message=Delivering MCAST message with seq 10 to pending delivery queue
rec=[312] Log Message=confchg entries 2
rec=[313] Log Message=Barrier Start Received From 1342332170
rec=[314] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[315] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[316] Log Message=Synchronization barrier completed
rec=[317] Log Message=Committing synchronization for (dummy EVT service)
rec=[318] Log Message=Synchronization actions starting for (corosync cluster closed process group service v1.01)
rec=[319] Log Message=mcasted message added to pending queue
rec=[320] Log Message=Received ringid(10.93.2.80:4440) seq 11
rec=[321] Log Message=Delivering 10 to 11
rec=[322] Log Message=Delivering MCAST message with seq 11 to pending delivery queue
rec=[323] Log Message=Received ringid(10.93.2.80:4440) seq 12
rec=[324] Log Message=Delivering 11 to 12
rec=[325] Log Message=Delivering MCAST message with seq 12 to pending delivery queue
rec=[326] Log Message=mcasted message added to pending queue
rec=[327] Log Message=releasing messages up to and including 10
rec=[328] Log Message=Delivering 12 to 14
rec=[329] Log Message=Delivering MCAST message with seq 13 to pending delivery queue
rec=[330] Log Message=Delivering MCAST message with seq 14 to pending delivery queue
rec=[331] Log Message=chosen downlist from node r(0) ip(10.93.2.80)
rec=[332] Log Message=Received ringid(10.93.2.80:4440) seq 13
rec=[333] Log Message=Received ringid(10.93.2.80:4440) seq 14
rec=[334] Log Message=Received ringid(10.93.2.80:4440) seq 15
rec=[335] Log Message=Delivering 14 to 15
rec=[336] Log Message=Delivering MCAST message with seq 15 to pending delivery queue
rec=[337] Log Message=confchg entries 2
rec=[338] Log Message=Barrier Start Received From 1342332170
rec=[339] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[340] Log Message=Barrier completion status for nodeid -1610457846 = 0.
rec=[341] Log Message=mcasted message added to pending queue
rec=[342] Log Message=releasing messages up to and including 12
rec=[343] Log Message=Delivering 15 to 16
rec=[344] Log Message=Delivering MCAST message with seq 16 to pending delivery queue
rec=[345] Log Message=confchg entries 2
rec=[346] Log Message=Barrier Start Received From -1610457846
rec=[347] Log Message=Barrier completion status for nodeid 1342332170 = 1.
rec=[348] Log Message=Barrier completion status for nodeid -1610457846 = 1.
rec=[349] Log Message=Synchronization barrier completed
rec=[350] Log Message=Committing synchronization for (corosync cluster closed process group service v1.01)
rec=[351] Log Message=Received ringid(10.93.2.80:4440) seq 16
rec=[352] Log Message=Received ringid(10.93.2.80:4440) seq 17
rec=[353] Log Message=Delivering 16 to 17
rec=[354] Log Message=Delivering MCAST message with seq 17 to pending delivery queue
rec=[355] Log Message=mcasted message added to pending queue
rec=[356] Log Message=releasing messages up to and including 15
rec=[357] Log Message=Delivering 17 to 18
rec=[358] Log Message=Delivering MCAST message with seq 18 to pending delivery queue
rec=[359] Log Message=Completed service synchronization, ready to provide service.
rec=[360] Log Message=Received ringid(10.93.2.80:4440) seq 18
rec=[361] Log Message=releasing messages up to and including 17
rec=[362] Log Message=releasing messages up to and including 18
rec=[363] Log Message=Received ringid(10.93.2.80:4440) seq 19
rec=[364] Log Message=Delivering 18 to 19
rec=[365] Log Message=Delivering MCAST message with seq 19 to pending delivery queue
rec=[366] Log Message=WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
rec=[367] Log Message=Msg[13] (dest=local:crmd, from=imsha2.dyn.intelerad:crmd.7415, remote=true, size=175): <create_request_adv origin="post_cache_update" t="crmd" version="3.0.1" subt="request" ref
rec=[368] Log Message=releasing messages up to and including 19
rec=[369] Log Message=Received ringid(10.93.2.80:4440) seq 1a
rec=[370] Log Message=Delivering 19 to 1a
rec=[371] Log Message=Delivering MCAST message with seq 1a to pending delivery queue
rec=[372] Log Message=Received ringid(10.93.2.80:4440) seq 1b
rec=[373] Log Message=Delivering 1a to 1b
rec=[374] Log Message=Delivering MCAST message with seq 1b to pending delivery queue
rec=[375] Log Message=WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
rec=[376] Log Message=Msg[22] (dest=local:cib, from=imsha2.dyn.intelerad:cib.7411, remote=true, size=1087): <copy __name__="cib_command" t="cib" cib_clientid="60c36199-13b7-41a4-b2c9-49d5e44c3ca9" c
rec=[377] Log Message=releasing messages up to and including 1b
rec=[378] Log Message=info: pcmk_ipc: Recorded connection 0x689e40 for attrd/4812
rec=[379] Log Message=debug: process_ais_message: Msg[0] (dest=local:ais, from=imsha1.dyn.intelerad:attrd.4812, remote=true, size=5): 4812
rec=[380] Log Message=Received ringid(10.93.2.80:4440) seq 1c
rec=[381] Log Message=Delivering 1b to 1c
rec=[382] Log Message=Delivering MCAST message with seq 1c to pending delivery queue
rec=[383] Log Message=WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
rec=[384] Log Message=Msg[14] (dest=local:crmd, from=imsha2.dyn.intelerad:crmd.7415, remote=true, size=175): <create_request_adv origin="post_cache_update" t="crmd" version="3.0.1" subt="request" ref
rec=[385] Log Message=releasing messages up to and including 1c
rec=[386] Log Message=Received ringid(10.93.2.80:4440) seq 1d
rec=[387] Log Message=Delivering 1c to 1d
rec=[388] Log Message=Delivering MCAST message with seq 1d to pending delivery queue
rec=[389] Log Message=WARN: route_ais_message: Sending message to local.cib failed: unknown (rc=-2)
rec=[390] Log Message=Msg[23] (dest=local:cib, from=imsha2.dyn.intelerad:cib.7411, remote=true, size=717): <copy __name__="cib_command" t="cib" cib_clientid="60c36199-13b7-41a4-b2c9-49d5e44c3ca9" c
rec=[391] Log Message=releasing messages up to and including 1d
rec=[392] Log Message=info: pcmk_ipc: Recorded connection 0x68ffc0 for stonithd/4809
rec=[393] Log Message=debug: process_ais_message: Msg[0] (dest=local:ais, from=imsha1.dyn.intelerad:stonithd.4809, remote=true, size=5): 4809
rec=[394] Log Message=info: pcmk_ipc: Recorded connection 0x694320 for cib/4810
rec=[395] Log Message=info: pcmk_ipc: Sending membership update 4440 to cib
rec=[396] Log Message=debug: process_ais_message: Msg[0] (dest=local:ais, from=imsha1.dyn.intelerad:cib.4810, remote=true, size=5): 4810
rec=[397] Log Message=Received ringid(10.93.2.80:4440) seq 1e
rec=[398] Log Message=Delivering 1d to 1e
rec=[399] Log Message=Delivering MCAST message with seq 1e to pending delivery queue
rec=[400] Log Message=WARN: route_ais_message: Sending message to local.crmd failed: unknown (rc=-2)
rec=[401] Log Message=Msg[15] (dest=local:crmd, from=imsha2.dyn.intelerad:crmd.7415, remote=true, size=232): <create_request_adv origin="join_make_offer" t="crmd" version="3.0.1" subt="request" refer
rec=[402] Log Message=Received ringid(10.93.2.80:4440) seq 1f
rec=[403] Log Message=Delivering 1e to 1f
rec=[404] Log Message=Delivering MCAST message with seq 1f to pending delivery queue
rec=[405] Log Message=releasing messages up to and including 1e
rec=[406] Log Message=releasing messages up to and including 1f
rec=[407] Log Message=Received ringid(10.93.2.80:4440) seq 20
rec=[408] Log Message=Delivering 1f to 20
rec=[409] Log Message=Delivering MCAST message with seq 20 to pending delivery queue
rec=[410] Log Message=releasing messages up to and including 20
rec=[411] Log Message=info: pcmk_ipc: Recorded connection 0x698680 for crmd/4814
rec=[412] Log Message=info: pcmk_ipc: Sending membership update 4440 to crmd
rec=[413] Log Message=debug: process_ais_message: Msg[0] (dest=local:ais, from=imsha1.dyn.intelerad:crmd.4814, remote=true, size=5): 4814
rec=[414] Log Message=mcasted message added to pending queue
rec=[415] Log Message=Delivering 20 to 21
rec=[416] Log Message=Delivering MCAST message with seq 21 to pending delivery queue
rec=[417] Log Message=Received ringid(10.93.2.80:4440) seq 21
rec=[418] Log Message=releasing messages up to and including 21
rec=[419] Log Message=mcasted message added to pending queue
rec=[420] Log Message=Delivering 21 to 22
rec=[421] Log Message=Delivering MCAST message with seq 22 to pending delivery queue
rec=[422] Log Message=Received ringid(10.93.2.80:4440) seq 22
rec=[423] Log Message=Received ringid(10.93.2.80:4440) seq 23
rec=[424] Log Message=Delivering 22 to 23
rec=[425] Log Message=Delivering MCAST message with seq 23 to pending delivery queue
rec=[426] Log Message=Received ringid(10.93.2.80:4440) seq 24
rec=[427] Log Message=Delivering 23 to 24
rec=[428] Log Message=Delivering MCAST message with seq 24 to pending delivery queue
rec=[429] Log Message=releasing messages up to and including 22
rec=[430] Log Message=releasing messages up to and including 24
rec=[431] Log Message=mcasted message added to pending queue
rec=[432] Log Message=Delivering 24 to 25
rec=[433] Log Message=Delivering MCAST message with seq 25 to pending delivery queue
rec=[434] Log Message=Received ringid(10.93.2.80:4440) seq 25
rec=[435] Log Message=releasing messages up to and including 25
rec=[436] Log Message=Received ringid(10.93.2.80:4440) seq 26
rec=[437] Log Message=Delivering 25 to 26
rec=[438] Log Message=Delivering MCAST message with seq 26 to pending delivery queue
rec=[439] Log Message=releasing messages up to and including 26
rec=[440] Log Message=Received ringid(10.93.2.80:4440) seq 27
rec=[441] Log Message=Delivering 26 to 27
rec=[442] Log Message=Delivering MCAST message with seq 27 to pending delivery queue
rec=[443] Log Message=Received ringid(10.93.2.80:4440) seq 28
rec=[444] Log Message=Delivering 27 to 28
rec=[445] Log Message=Delivering MCAST message with seq 28 to pending delivery queue
rec=[446] Log Message=Received ringid(10.93.2.80:4440) seq 29
rec=[447] Log Message=Delivering 28 to 29
rec=[448] Log Message=Delivering MCAST message with seq 29 to pending delivery queue
rec=[449] Log Message=releasing messages up to and including 29
rec=[450] Log Message=Received ringid(10.93.2.80:4440) seq 2a
rec=[451] Log Message=Delivering 29 to 2a
rec=[452] Log Message=Delivering MCAST message with seq 2a to pending delivery queue
rec=[453] Log Message=releasing messages up to and including 2a
rec=[454] Log Message=Received ringid(10.93.2.80:4440) seq 2b
rec=[455] Log Message=Delivering 2a to 2b
rec=[456] Log Message=Delivering MCAST message with seq 2b to pending delivery queue
rec=[457] Log Message=releasing messages up to and including 2b
rec=[458] Log Message=mcasted message added to pending queue
rec=[459] Log Message=mcasted message added to pending queue
rec=[460] Log Message=Delivering 2b to 2d
rec=[461] Log Message=Delivering MCAST message with seq 2c to pending delivery queue
rec=[462] Log Message=Delivering MCAST message with seq 2d to pending delivery queue
rec=[463] Log Message=Received ringid(10.93.2.80:4440) seq 2c
rec=[464] Log Message=Received ringid(10.93.2.80:4440) seq 2d
rec=[465] Log Message=releasing messages up to and including 2d
rec=[466] Log Message=Received ringid(10.93.2.80:4440) seq 2e
rec=[467] Log Message=Delivering 2d to 2e
rec=[468] Log Message=Delivering MCAST message with seq 2e to pending delivery queue
rec=[469] Log Message=releasing messages up to and including 2e
rec=[470] Log Message=Received ringid(10.93.2.80:4440) seq 2f
rec=[471] Log Message=Delivering 2e to 2f
rec=[472] Log Message=Delivering MCAST message with seq 2f to pending delivery queue
rec=[473] Log Message=Received ringid(10.93.2.80:4440) seq 30
rec=[474] Log Message=Delivering 2f to 30
rec=[475] Log Message=Delivering MCAST message with seq 30 to pending delivery queue
rec=[476] Log Message=Received ringid(10.93.2.80:4440) seq 31
rec=[477] Log Message=Delivering 30 to 31
rec=[478] Log Message=Delivering MCAST message with seq 31 to pending delivery queue
rec=[479] Log Message=Received ringid(10.93.2.80:4440) seq 32
rec=[480] Log Message=Delivering 31 to 32
rec=[481] Log Message=Delivering MCAST message with seq 32 to pending delivery queue
rec=[482] Log Message=Received ringid(10.93.2.80:4440) seq 33
rec=[483] Log Message=Delivering 32 to 33
rec=[484] Log Message=Delivering MCAST message with seq 33 to pending delivery queue
rec=[485] Log Message=Received ringid(10.93.2.80:4440) seq 34
rec=[486] Log Message=Delivering 33 to 34
rec=[487] Log Message=Delivering MCAST message with seq 34 to pending delivery queue
rec=[488] Log Message=Received ringid(10.93.2.80:4440) seq 35
rec=[489] Log Message=Delivering 34 to 35
rec=[490] Log Message=Delivering MCAST message with seq 35 to pending delivery queue
rec=[491] Log Message=releasing messages up to and including 35
rec=[492] Log Message=Received ringid(10.93.2.80:4440) seq 36
rec=[493] Log Message=Delivering 35 to 36
rec=[494] Log Message=Delivering MCAST message with seq 36 to pending delivery queue
rec=[495] Log Message=Received ringid(10.93.2.80:4440) seq 37
rec=[496] Log Message=Delivering 36 to 37
rec=[497] Log Message=Delivering MCAST message with seq 37 to pending delivery queue
rec=[498] Log Message=releasing messages up to and including 37
rec=[499] Log Message=Received ringid(10.93.2.80:4440) seq 38
rec=[500] Log Message=Delivering 37 to 38
rec=[501] Log Message=Delivering MCAST message with seq 38 to pending delivery queue
rec=[502] Log Message=Received ringid(10.93.2.80:4440) seq 39
rec=[503] Log Message=Delivering 38 to 39
rec=[504] Log Message=Delivering MCAST message with seq 39 to pending delivery queue
rec=[505] Log Message=Received ringid(10.93.2.80:4440) seq 3a
rec=[506] Log Message=Delivering 39 to 3a
rec=[507] Log Message=Delivering MCAST message with seq 3a to pending delivery queue
rec=[508] Log Message=Received ringid(10.93.2.80:4440) seq 3b
rec=[509] Log Message=Delivering 3a to 3b
rec=[510] Log Message=Delivering MCAST message with seq 3b to pending delivery queue
rec=[511] Log Message=Received ringid(10.93.2.80:4440) seq 3c
rec=[512] Log Message=Delivering 3b to 3c
rec=[513] Log Message=Delivering MCAST message with seq 3c to pending delivery queue
rec=[514] Log Message=Received ringid(10.93.2.80:4440) seq 3d
rec=[515] Log Message=Delivering 3c to 3d
rec=[516] Log Message=Delivering MCAST message with seq 3d to pending delivery queue
rec=[517] Log Message=Received ringid(10.93.2.80:4440) seq 3e
rec=[518] Log Message=Delivering 3d to 3e
rec=[519] Log Message=Delivering MCAST message with seq 3e to pending delivery queue
rec=[520] Log Message=Received ringid(10.93.2.80:4440) seq 3f
rec=[521] Log Message=Delivering 3e to 3f
rec=[522] Log Message=Delivering MCAST message with seq 3f to pending delivery queue
rec=[523] Log Message=Received ringid(10.93.2.80:4440) seq 40
rec=[524] Log Message=Delivering 3f to 40
rec=[525] Log Message=Delivering MCAST message with seq 40 to pending delivery queue
rec=[526] Log Message=releasing messages up to and including 3e
rec=[527] Log Message=releasing messages up to and including 40
rec=[528] Log Message=mcasted message added to pending queue
rec=[529] Log Message=Delivering 40 to 41
rec=[530] Log Message=Delivering MCAST message with seq 41 to pending delivery queue
rec=[531] Log Message=Received ringid(10.93.2.80:4440) seq 41
rec=[532] Log Message=mcasted message added to pending queue
rec=[533] Log Message=Delivering 41 to 42
rec=[534] Log Message=Delivering MCAST message with seq 42 to pending delivery queue
rec=[535] Log Message=Received ringid(10.93.2.80:4440) seq 42
rec=[536] Log Message=releasing messages up to and including 41
rec=[537] Log Message=releasing messages up to and including 42
rec=[538] Log Message=Received ringid(10.93.2.80:4440) seq 43
rec=[539] Log Message=Delivering 42 to 43
rec=[540] Log Message=Delivering MCAST message with seq 43 to pending delivery queue
rec=[541] Log Message=releasing messages up to and including 43
rec=[542] Log Message=Received ringid(10.93.2.80:4440) seq 44
rec=[543] Log Message=Delivering 43 to 44
rec=[544] Log Message=Delivering MCAST message with seq 44 to pending delivery queue
rec=[545] Log Message=releasing messages up to and including 44
rec=[546] Log Message=Received ringid(10.93.2.80:4440) seq 45
rec=[547] Log Message=Delivering 44 to 45
rec=[548] Log Message=Delivering MCAST message with seq 45 to pending delivery queue
rec=[549] Log Message=releasing messages up to and including 45
rec=[550] Log Message=Received ringid(10.93.2.80:4440) seq 46
rec=[551] Log Message=Delivering 45 to 46
rec=[552] Log Message=Delivering MCAST message with seq 46 to pending delivery queue
rec=[553] Log Message=releasing messages up to and including 46
rec=[554] Log Message=Received ringid(10.93.2.80:4440) seq 47
rec=[555] Log Message=Delivering 46 to 47
rec=[556] Log Message=Delivering MCAST message with seq 47 to pending delivery queue
rec=[557] Log Message=Received ringid(10.93.2.80:4440) seq 48
rec=[558] Log Message=Delivering 47 to 48
rec=[559] Log Message=Delivering MCAST message with seq 48 to pending delivery queue
rec=[560] Log Message=Received ringid(10.93.2.80:4440) seq 49
rec=[561] Log Message=Delivering 48 to 49
rec=[562] Log Message=Delivering MCAST message with seq 49 to pending delivery queue
rec=[563] Log Message=Received ringid(10.93.2.80:4440) seq 4a
rec=[564] Log Message=Delivering 49 to 4a
rec=[565] Log Message=Delivering MCAST message with seq 4a to pending delivery queue
rec=[566] Log Message=Received ringid(10.93.2.80:4440) seq 4b
rec=[567] Log Message=Delivering 4a to 4b
rec=[568] Log Message=Delivering MCAST message with seq 4b to pending delivery queue
rec=[569] Log Message=releasing messages up to and including 48
rec=[570] Log Message=releasing messages up to and including 4b
rec=[571] Log Message=mcasted message added to pending queue
rec=[572] Log Message=mcasted message added to pending queue
rec=[573] Log Message=mcasted message added to pending queue
rec=[574] Log Message=Delivering 4b to 4e
rec=[575] Log Message=Delivering MCAST message with seq 4c to pending delivery queue
rec=[576] Log Message=Delivering MCAST message with seq 4d to pending delivery queue
rec=[577] Log Message=Delivering MCAST message with seq 4e to pending delivery queue
rec=[578] Log Message=Received ringid(10.93.2.80:4440) seq 4c
rec=[579] Log Message=Received ringid(10.93.2.80:4440) seq 4d
rec=[580] Log Message=Received ringid(10.93.2.80:4440) seq 4e
rec=[581] Log Message=releasing messages up to and including 4e
rec=[582] Log Message=mcasted message added to pending queue
rec=[583] Log Message=Received ringid(10.93.2.80:4440) seq 4f
rec=[584] Log Message=Delivering 4e to 4f
rec=[585] Log Message=Delivering MCAST message with seq 4f to pending delivery queue
rec=[586] Log Message=Received ringid(10.93.2.80:4440) seq 50
rec=[587] Log Message=Delivering 4f to 50
rec=[588] Log Message=Delivering MCAST message with seq 50 to pending delivery queue
rec=[589] Log Message=Received ringid(10.93.2.80:4440) seq 51
rec=[590] Log Message=Delivering 50 to 51
rec=[591] Log Message=Delivering MCAST message with seq 51 to pending delivery queue
rec=[592] Log Message=mcasted message added to pending queue
rec=[593] Log Message=Delivering 51 to 53
rec=[594] Log Message=Delivering MCAST message with seq 52 to pending delivery queue
rec=[595] Log Message=Delivering MCAST message with seq 53 to pending delivery queue
rec=[596] Log Message=Received ringid(10.93.2.80:4440) seq 52
rec=[597] Log Message=Received ringid(10.93.2.80:4440) seq 53
rec=[598] Log Message=Received ringid(10.93.2.80:4440) seq 54
rec=[599] Log Message=Delivering 53 to 54
rec=[600] Log Message=Delivering MCAST message with seq 54 to pending delivery queue
rec=[601] Log Message=Received ringid(10.93.2.80:4440) seq 55
rec=[602] Log Message=Delivering 54 to 55
rec=[603] Log Message=Delivering MCAST message with seq 55 to pending delivery queue
rec=[604] Log Message=Received ringid(10.93.2.80:4440) seq 56
rec=[605] Log Message=Delivering 55 to 56
rec=[606] Log Message=Delivering MCAST message with seq 56 to pending delivery queue
rec=[607] Log Message=releasing messages up to and including 51
rec=[608] Log Message=Received ringid(10.93.2.80:4440) seq 57
rec=[609] Log Message=Delivering 56 to 57
rec=[610] Log Message=Delivering MCAST message with seq 57 to pending delivery queue
rec=[611] Log Message=Received ringid(10.93.2.80:4440) seq 58
rec=[612] Log Message=Delivering 57 to 58
rec=[613] Log Message=Delivering MCAST message with seq 58 to pending delivery queue
rec=[614] Log Message=Received ringid(10.93.2.80:4440) seq 59
rec=[615] Log Message=Delivering 58 to 59
rec=[616] Log Message=Delivering MCAST message with seq 59 to pending delivery queue
rec=[617] Log Message=releasing messages up to and including 56
rec=[618] Log Message=releasing messages up to and including 59
rec=[619] Log Message=mcasted message added to pending queue
rec=[620] Log Message=mcasted message added to pending queue
rec=[621] Log Message=Delivering 59 to 5b
rec=[622] Log Message=Delivering MCAST message with seq 5a to pending delivery queue
rec=[623] Log Message=Delivering MCAST message with seq 5b to pending delivery queue
rec=[624] Log Message=Received ringid(10.93.2.80:4440) seq 5a
rec=[625] Log Message=Received ringid(10.93.2.80:4440) seq 5b
rec=[626] Log Message=releasing messages up to and including 5b
rec=[627] Log Message=Received ringid(10.93.2.80:4440) seq 5c
rec=[628] Log Message=Delivering 5b to 5c
rec=[629] Log Message=Delivering MCAST message with seq 5c to pending delivery queue
rec=[630] Log Message=Received ringid(10.93.2.80:4440) seq 5d
rec=[631] Log Message=Delivering 5c to 5d
rec=[632] Log Message=Delivering MCAST message with seq 5d to pending delivery queue
rec=[633] Log Message=releasing messages up to and including 5d
rec=[634] Log Message=mcasted message added to pending queue
rec=[635] Log Message=mcasted message added to pending queue
rec=[636] Log Message=Delivering 5d to 5f
rec=[637] Log Message=Delivering MCAST message with seq 5e to pending delivery queue
rec=[638] Log Message=Delivering MCAST message with seq 5f to pending delivery queue
rec=[639] Log Message=Received ringid(10.93.2.80:4440) seq 5e
rec=[640] Log Message=Received ringid(10.93.2.80:4440) seq 5f
rec=[641] Log Message=releasing messages up to and including 5f
rec=[642] Log Message=Received ringid(10.93.2.80:4440) seq 60
rec=[643] Log Message=Delivering 5f to 60
rec=[644] Log Message=Delivering MCAST message with seq 60 to pending delivery queue
rec=[645] Log Message=Received ringid(10.93.2.80:4440) seq 61
rec=[646] Log Message=Delivering 60 to 61
rec=[647] Log Message=Delivering MCAST message with seq 61 to pending delivery queue
rec=[648] Log Message=Received ringid(10.93.2.80:4440) seq 62
rec=[649] Log Message=Delivering 61 to 62
rec=[650] Log Message=Delivering MCAST message with seq 62 to pending delivery queue
rec=[651] Log Message=releasing messages up to and including 62
rec=[652] Log Message=mcasted message added to pending queue
rec=[653] Log Message=Delivering 62 to 63
rec=[654] Log Message=Delivering MCAST message with seq 63 to pending delivery queue
rec=[655] Log Message=Received ringid(10.93.2.80:4440) seq 63
rec=[656] Log Message=releasing messages up to and including 63
rec=[657] Log Message=mcasted message added to pending queue
rec=[658] Log Message=mcasted message added to pending queue
rec=[659] Log Message=mcasted message added to pending queue
rec=[660] Log Message=Delivering 63 to 66
rec=[661] Log Message=Delivering MCAST message with seq 64 to pending delivery queue
rec=[662] Log Message=Delivering MCAST message with seq 65 to pending delivery queue
rec=[663] Log Message=Delivering MCAST message with seq 66 to pending delivery queue
rec=[664] Log Message=Received ringid(10.93.2.80:4440) seq 64
rec=[665] Log Message=Received ringid(10.93.2.80:4440) seq 65
rec=[666] Log Message=Received ringid(10.93.2.80:4440) seq 66
rec=[667] Log Message=releasing messages up to and including 66
rec=[668] Log Message=Received ringid(10.93.2.80:4440) seq 67
rec=[669] Log Message=Delivering 66 to 67
rec=[670] Log Message=Delivering MCAST message with seq 67 to pending delivery queue
rec=[671] Log Message=Received ringid(10.93.2.80:4440) seq 68
rec=[672] Log Message=Delivering 67 to 68
rec=[673] Log Message=Delivering MCAST message with seq 68 to pending delivery queue
rec=[674] Log Message=releasing messages up to and including 68
rec=[675] Log Message=mcasted message added to pending queue
rec=[676] Log Message=Delivering 68 to 69
rec=[677] Log Message=Delivering MCAST message with seq 69 to pending delivery queue
rec=[678] Log Message=Received ringid(10.93.2.80:4440) seq 69
rec=[679] Log Message=releasing messages up to and including 69
rec=[680] Log Message=mcasted message added to pending queue
rec=[681] Log Message=Delivering 69 to 6a
rec=[682] Log Message=Delivering MCAST message with seq 6a to pending delivery queue
rec=[683] Log Message=Received ringid(10.93.2.80:4440) seq 6a
rec=[684] Log Message=releasing messages up to and including 6a
rec=[685] Log Message=Received ringid(10.93.2.80:4440) seq 6b
rec=[686] Log Message=Delivering 6a to 6b
rec=[687] Log Message=Delivering MCAST message with seq 6b to pending delivery queue
rec=[688] Log Message=Received ringid(10.93.2.80:4440) seq 6c
rec=[689] Log Message=Delivering 6b to 6c
rec=[690] Log Message=Delivering MCAST message with seq 6c to pending delivery queue
rec=[691] Log Message=releasing messages up to and including 6c
rec=[692] Log Message=Received ringid(10.93.2.80:4440) seq 6d
rec=[693] Log Message=Delivering 6c to 6d
rec=[694] Log Message=Delivering MCAST message with seq 6d to pending delivery queue
rec=[695] Log Message=Received ringid(10.93.2.80:4440) seq 6e
rec=[696] Log Message=Delivering 6d to 6e
rec=[697] Log Message=Delivering MCAST message with seq 6e to pending delivery queue
rec=[698] Log Message=Received ringid(10.93.2.80:4440) seq 6f
rec=[699] Log Message=Delivering 6e to 6f
rec=[700] Log Message=Delivering MCAST message with seq 6f to pending delivery queue
rec=[701] Log Message=releasing messages up to and including 6f
rec=[702] Log Message=Received ringid(10.93.2.80:4440) seq 70
rec=[703] Log Message=Delivering 6f to 70
rec=[704] Log Message=Delivering MCAST message with seq 70 to pending delivery queue
rec=[705] Log Message=Received ringid(10.93.2.80:4440) seq 71
rec=[706] Log Message=Delivering 70 to 71
rec=[707] Log Message=Delivering MCAST message with seq 71 to pending delivery queue
rec=[708] Log Message=releasing messages up to and including 71
rec=[709] Log Message=Received ringid(10.93.2.80:4440) seq 72
rec=[710] Log Message=Delivering 71 to 72
rec=[711] Log Message=Delivering MCAST message with seq 72 to pending delivery queue
rec=[712] Log Message=Received ringid(10.93.2.80:4440) seq 73
rec=[713] Log Message=Delivering 72 to 73
rec=[714] Log Message=Delivering MCAST message with seq 73 to pending delivery queue
rec=[715] Log Message=releasing messages up to and including 73
rec=[716] Log Message=Received ringid(10.93.2.80:4440) seq 74
rec=[717] Log Message=Delivering 73 to 74
rec=[718] Log Message=Delivering MCAST message with seq 74 to pending delivery queue
rec=[719] Log Message=Received ringid(10.93.2.80:4440) seq 75
rec=[720] Log Message=Delivering 74 to 75
rec=[721] Log Message=Delivering MCAST message with seq 75 to pending delivery queue
rec=[722] Log Message=releasing messages up to and including 75
rec=[723] Log Message=mcasted message added to pending queue
rec=[724] Log Message=mcasted message added to pending queue
rec=[725] Log Message=Delivering 75 to 77
rec=[726] Log Message=Delivering MCAST message with seq 76 to pending delivery queue
rec=[727] Log Message=Delivering MCAST message with seq 77 to pending delivery queue
rec=[728] Log Message=Received ringid(10.93.2.80:4440) seq 76
rec=[729] Log Message=Received ringid(10.93.2.80:4440) seq 77
rec=[730] Log Message=releasing messages up to and including 77
rec=[731] Log Message=Received ringid(10.93.2.80:4440) seq 78
rec=[732] Log Message=Delivering 77 to 78
rec=[733] Log Message=Delivering MCAST message with seq 78 to pending delivery queue
rec=[734] Log Message=Received ringid(10.93.2.80:4440) seq 79
rec=[735] Log Message=Delivering 78 to 79
rec=[736] Log Message=Delivering MCAST message with seq 79 to pending delivery queue
rec=[737] Log Message=Received ringid(10.93.2.80:4440) seq 7a
rec=[738] Log Message=Delivering 79 to 7a
rec=[739] Log Message=Delivering MCAST message with seq 7a to pending delivery queue
rec=[740] Log Message=releasing messages up to and including 7a
rec=[741] Log Message=mcasted message added to pending queue
rec=[742] Log Message=mcasted message added to pending queue
rec=[743] Log Message=Delivering 7a to 7c
rec=[744] Log Message=Delivering MCAST message with seq 7b to pending delivery queue
rec=[745] Log Message=Delivering MCAST message with seq 7c to pending delivery queue
rec=[746] Log Message=Received ringid(10.93.2.80:4440) seq 7b
rec=[747] Log Message=Received ringid(10.93.2.80:4440) seq 7c
rec=[748] Log Message=mcasted message added to pending queue
rec=[749] Log Message=mcasted message added to pending queue
rec=[750] Log Message=releasing messages up to and including 7c
rec=[751] Log Message=Delivering 7c to 7e
rec=[752] Log Message=Delivering MCAST message with seq 7d to pending delivery queue
rec=[753] Log Message=Delivering MCAST message with seq 7e to pending delivery queue
rec=[754] Log Message=Received ringid(10.93.2.80:4440) seq 7d
rec=[755] Log Message=Received ringid(10.93.2.80:4440) seq 7e
rec=[756] Log Message=Received ringid(10.93.2.80:4440) seq 7f
rec=[757] Log Message=Delivering 7e to 7f
rec=[758] Log Message=Delivering MCAST message with seq 7f to pending delivery queue
rec=[759] Log Message=Received ringid(10.93.2.80:4440) seq 80
rec=[760] Log Message=Delivering 7f to 80
rec=[761] Log Message=Delivering MCAST message with seq 80 to pending delivery queue
rec=[762] Log Message=Received ringid(10.93.2.80:4440) seq 81
rec=[763] Log Message=Delivering 80 to 81
rec=[764] Log Message=Delivering MCAST message with seq 81 to pending delivery queue
rec=[765] Log Message=releasing messages up to and including 81
rec=[766] Log Message=Received ringid(10.93.2.80:4440) seq 82
rec=[767] Log Message=Delivering 81 to 82
rec=[768] Log Message=Delivering MCAST message with seq 82 to pending delivery queue
rec=[769] Log Message=Received ringid(10.93.2.80:4440) seq 83
rec=[770] Log Message=Delivering 82 to 83
rec=[771] Log Message=Delivering MCAST message with seq 83 to pending delivery queue
rec=[772] Log Message=Received ringid(10.93.2.80:4440) seq 84
rec=[773] Log Message=Delivering 83 to 84
rec=[774] Log Message=Delivering MCAST message with seq 84 to pending delivery queue
rec=[775] Log Message=releasing messages up to and including 84
rec=[776] Log Message=mcasted message added to pending queue
rec=[777] Log Message=mcasted message added to pending queue
rec=[778] Log Message=Delivering 84 to 86
rec=[779] Log Message=Delivering MCAST message with seq 85 to pending delivery queue
rec=[780] Log Message=Delivering MCAST message with seq 86 to pending delivery queue
rec=[781] Log Message=Received ringid(10.93.2.80:4440) seq 85
rec=[782] Log Message=Received ringid(10.93.2.80:4440) seq 86
rec=[783] Log Message=releasing messages up to and including 86
rec=[784] Log Message=Received ringid(10.93.2.80:4440) seq 87
rec=[785] Log Message=Delivering 86 to 87
rec=[786] Log Message=Delivering MCAST message with seq 87 to pending delivery queue
rec=[787] Log Message=Received ringid(10.93.2.80:4440) seq 88
rec=[788] Log Message=Delivering 87 to 88
rec=[789] Log Message=Delivering MCAST message with seq 88 to pending delivery queue
rec=[790] Log Message=releasing messages up to and including 88
rec=[791] Log Message=mcasted message added to pending queue
rec=[792] Log Message=mcasted message added to pending queue
rec=[793] Log Message=Delivering 88 to 8a
rec=[794] Log Message=Delivering MCAST message with seq 89 to pending delivery queue
rec=[795] Log Message=Delivering MCAST message with seq 8a to pending delivery queue
rec=[796] Log Message=Received ringid(10.93.2.80:4440) seq 89
rec=[797] Log Message=Received ringid(10.93.2.80:4440) seq 8a
rec=[798] Log Message=releasing messages up to and including 8a
rec=[799] Log Message=Received ringid(10.93.2.80:4440) seq 8b
rec=[800] Log Message=Delivering 8a to 8b
rec=[801] Log Message=Delivering MCAST message with seq 8b to pending delivery queue
rec=[802] Log Message=Received ringid(10.93.2.80:4440) seq 8c
rec=[803] Log Message=Delivering 8b to 8c
rec=[804] Log Message=Delivering MCAST message with seq 8c to pending delivery queue
rec=[805] Log Message=Received ringid(10.93.2.80:4440) seq 8d
rec=[806] Log Message=Delivering 8c to 8d
rec=[807] Log Message=Delivering MCAST message with seq 8d to pending delivery queue
rec=[808] Log Message=Received ringid(10.93.2.80:4440) seq 8e
rec=[809] Log Message=Delivering 8d to 8e
rec=[810] Log Message=Delivering MCAST message with seq 8e to pending delivery queue
rec=[811] Log Message=releasing messages up to and including 8e
rec=[812] Log Message=mcasted message added to pending queue
rec=[813] Log Message=mcasted message added to pending queue
rec=[814] Log Message=Delivering 8e to 90
rec=[815] Log Message=Delivering MCAST message with seq 8f to pending delivery queue
rec=[816] Log Message=Delivering MCAST message with seq 90 to pending delivery queue
rec=[817] Log Message=Received ringid(10.93.2.80:4440) seq 8f
rec=[818] Log Message=Received ringid(10.93.2.80:4440) seq 90
rec=[819] Log Message=Received ringid(10.93.2.80:4440) seq 91
rec=[820] Log Message=Delivering 90 to 91
rec=[821] Log Message=Delivering MCAST message with seq 91 to pending delivery queue
rec=[822] Log Message=Received ringid(10.93.2.80:4440) seq 92
rec=[823] Log Message=Delivering 91 to 92
rec=[824] Log Message=Delivering MCAST message with seq 92 to pending delivery queue
rec=[825] Log Message=Received ringid(10.93.2.80:4440) seq 93
rec=[826] Log Message=Delivering 92 to 93
rec=[827] Log Message=Delivering MCAST message with seq 93 to pending delivery queue
rec=[828] Log Message=releasing messages up to and including 90
rec=[829] Log Message=releasing messages up to and including 93
rec=[830] Log Message=mcasted message added to pending queue
rec=[831] Log Message=mcasted message added to pending queue
rec=[832] Log Message=Delivering 93 to 95
rec=[833] Log Message=Delivering MCAST message with seq 94 to pending delivery queue
rec=[834] Log Message=Delivering MCAST message with seq 95 to pending delivery queue
rec=[835] Log Message=Received ringid(10.93.2.80:4440) seq 94
rec=[836] Log Message=Received ringid(10.93.2.80:4440) seq 95
rec=[837] Log Message=releasing messages up to and including 95
rec=[838] Log Message=Received ringid(10.93.2.80:4440) seq 96
rec=[839] Log Message=Delivering 95 to 96
rec=[840] Log Message=Delivering MCAST message with seq 96 to pending delivery queue
rec=[841] Log Message=Received ringid(10.93.2.80:4440) seq 97
rec=[842] Log Message=Delivering 96 to 97
rec=[843] Log Message=Delivering MCAST message with seq 97 to pending delivery queue
rec=[844] Log Message=Received ringid(10.93.2.80:4440) seq 98
rec=[845] Log Message=Delivering 97 to 98
rec=[846] Log Message=Delivering MCAST message with seq 98 to pending delivery queue
rec=[847] Log Message=Received ringid(10.93.2.80:4440) seq 99
rec=[848] Log Message=Delivering 98 to 99
rec=[849] Log Message=Delivering MCAST message with seq 99 to pending delivery queue
rec=[850] Log Message=releasing messages up to and including 97
rec=[851] Log Message=releasing messages up to and including 99
rec=[852] Log Message=mcasted message added to pending queue
rec=[853] Log Message=mcasted message added to pending queue
rec=[854] Log Message=Delivering 99 to 9b
rec=[855] Log Message=Delivering MCAST message with seq 9a to pending delivery queue
rec=[856] Log Message=Delivering MCAST message with seq 9b to pending delivery queue
rec=[857] Log Message=Received ringid(10.93.2.80:4440) seq 9a
rec=[858] Log Message=Received ringid(10.93.2.80:4440) seq 9b
rec=[859] Log Message=Received ringid(10.93.2.80:4440) seq 9c
rec=[860] Log Message=Delivering 9b to 9c
rec=[861] Log Message=Delivering MCAST message with seq 9c to pending delivery queue
rec=[862] Log Message=Received ringid(10.93.2.80:4440) seq 9d
rec=[863] Log Message=Delivering 9c to 9d
rec=[864] Log Message=Delivering MCAST message with seq 9d to pending delivery queue
rec=[865] Log Message=releasing messages up to and including 9b
rec=[866] Log Message=releasing messages up to and including 9d
rec=[867] Log Message=sendmsg(ucast) failed (non-critical):
rec=[868] Log Message=sendmsg(ucast) failed (non-critical):
rec=[869] Log Message=mcasted message added to pending queue
rec=[870] Log Message=mcasted message added to pending queue
rec=[871] Log Message=Delivering 9d to 9f
rec=[872] Log Message=Delivering MCAST message with seq 9e to pending delivery queue
rec=[873] Log Message=Delivering MCAST message with seq 9f to pending delivery queue
rec=[874] Log Message=releasing messages up to and including 9f
rec=[875] Log Message=FAILED TO RECEIVE
rec=[876] Log Message=entering GATHER state from 6.
rec=[877] Log Message=mcasted message added to pending queue
rec=[878] Log Message=entering GATHER state from 0.
Finishing replay: records found [878]
-------------- next part --------------
Core was generated by `corosync'.
Program terminated with signal 6, Aborted.
#0 0x00000036d4630265 in raise () from /lib64/libc.so.6
(gdb) thread apply all bt
Thread 8 (Thread 4805):
#0 0x00000036d46cb186 in poll () from /lib64/libc.so.6
#1 0x000000000040994f in prioritized_timer_thread (data=<value optimized out>) at timer.c:127
#2 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 7 (Thread 4806):
#0 0x00000036d4e0cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x0000000000405ef0 in corosync_exit_thread_handler (arg=0x0) at main.c:200
#2 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 6 (Thread 4808):
#0 0x00000036d4e0e1c1 in nanosleep () from /lib64/libpthread.so.0
#1 0x00002aaaabdb1f88 in ?? () from /usr/libexec/lcrso/pacemaker.lcrso
#2 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#3 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 5 (Thread 4815):
#0 0x00000036d4e0cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x00002aaaaaef75a8 in ipc_sem_wait (conn=<value optimized out>) at ../include/corosync/coroipc_ipc.h:199
#2 pthread_ipc_consumer (conn=<value optimized out>) at coroipcs.c:681
#3 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 4 (Thread 4816):
#0 0x00000036d4e0cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x00002aaaaaef75a8 in ipc_sem_wait (conn=<value optimized out>) at ../include/corosync/coroipc_ipc.h:199
#2 pthread_ipc_consumer (conn=<value optimized out>) at coroipcs.c:681
#3 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 3 (Thread 4817):
#0 0x00000036d4e0cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x00002aaaaaef75a8 in ipc_sem_wait (conn=<value optimized out>) at ../include/corosync/coroipc_ipc.h:199
#2 pthread_ipc_consumer (conn=<value optimized out>) at coroipcs.c:681
#3 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 2 (Thread 4819):
#0 0x00000036d4e0cd01 in sem_wait () from /lib64/libpthread.so.0
#1 0x00002aaaaaef75a8 in ipc_sem_wait (conn=<value optimized out>) at ../include/corosync/coroipc_ipc.h:199
#2 pthread_ipc_consumer (conn=<value optimized out>) at coroipcs.c:681
#3 0x00000036d4e0673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000036d46d3f6d in clone () from /lib64/libc.so.6
Thread 1 (Thread 4803):
#0 0x00000036d4630265 in raise () from /lib64/libc.so.6
#1 0x00000036d4631d10 in abort () from /lib64/libc.so.6
#2 0x00000036d46296e6 in __assert_fail () from /lib64/libc.so.6
#3 0x00002aaaaaad1212 in memb_consensus_agreed (instance=0x2aaaabcb4010) at totemsrp.c:1194
#4 memb_state_consensus_timeout_expired (instance=0x2aaaabcb4010) at totemsrp.c:1490
#5 0x00002aaaaaac1fe2 in timerlist_expire (handle=7417221384171945984) at tlist.h:309
#6 poll_run (handle=7417221384171945984) at coropoll.c:523
#7 0x00000000004072fb in main (argc=<value optimized out>, argv=<value optimized out>, envp=<value optimized out>)
at main.c:1813
(gdb)
More information about the Openais
mailing list