[Openais] clvmd + openais "Cannot allocate memory"

[Quentin Smith]

[please preserve the Cc in replies]

Hi-

I'm trying to debug a problem that has surfaced recently in our clvmd 
2.02.56 + openais 1.1.1 + corosync 1.1.2 cluster. We use clvmd to manage 
volumes on a shared iSCSI storage array. We have four nodes in our 
cluster, and after running for a while we start getting errors when trying 
to perform LVM operations such as:

root at babylon-four:~# lvs
   cluster request failed: Cannot allocate memory
   Can't get lock for xenvg
   Skipping volume group xenvg
   cluster request failed: Cannot allocate memory
   Can't get lock for babylon-four

and

root at citadel-station:~# lvcreate -L 1G -n quentin-test-2 xenvg
   Error locking on node 1280050a: Cannot allocate memory
   Error locking on node 1380050a: Cannot allocate memory
   Error locking on node 1180050a: Cannot allocate memory
   Error locking on node 1080050a: Cannot allocate memory
   Aborting. Failed to activate new LV to wipe the start of it.

We had been able to clear these errors up by restarting corosync and 
clvmd, but now that we've added another two nodes to our cluster, we're 
getting these errors very frequently. Is there a buffer somewhere that we 
need to increase the size of? I've attached our current corosync.conf to 
the end of this message.

(Needless to say, the servers all have plenty of RAM free...)

Thanks,
--Quentin

root at babylon-four:~# cat /etc/corosync/corosync.conf
compatibility: whitetank

totem {
 	version: 2
 	secauth: off
 	rrp_mode: passive
 	interface {
 		ringnumber: 0
 		bindnetaddr: 10.5.128.0
 		mcastaddr: 239.255.252.0
 		mcastport: 5405
 	}
 	interface {
 		ringnumber: 1
 		bindnetaddr: 10.6.128.0
 		mcastaddr: 239.255.252.1
 		mcastport: 5405
 	}
}

logging {
 	to_syslog: yes
 	syslog_facility: daemon
 	timestamp: on
 	debug: off
}

amf {
 	mode: disabled
}