2:23 p.m.
On Wed, 2019-05-29 at 13:49 +0000, Stolte, Felix wrote:
Hi,
is anyone running an active-passive nfs-ganesha cluster with cephfs backend and using the
rados_kv recovery backend? My setup runs fine, but takeover is giving me a headache. On
takeover I see the following messages in ganeshas log file:
Note that there are significant problems with the rados_kv recovery
backend. In particular, it does not properly handle the case where the
server crashes during the grace period. The rados_ng and rados_cluster
backends do handle those situations properly.
29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 :
ganesha.nfsd-9793[dbus_heartbeat] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE,
duration 5
29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat]
nfs_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip 10.0.0.5
29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat]
rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2
29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat]
rados_kv_read_recov_clids_takeover :CLIENT ID :EVENT :Failed to takeover
29/05/2019 15:38:26 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[reaper]
nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
The result is clients hanging for up to 2 Minutes. Has anyone ran into the same problem?
Ceph Version: 12.2.11
nfs-ganesha: 2.7.3
If I had to guess, the hanging is probably due to state that is being
held by the other node's MDS session that hasn't expired yet. Ceph v12
doesn't have the client reclaim interfaces that make more instantaneous
failover possible. That's new in v14 (Nautilus). See pages 12 and 13
here:
https://static.sched.com/hosted_files/cephalocon2019/86/Rook-Deployed%20N...
ganesha.conf (identical on both nodes besides nodeid in rados_kv:
NFS_CORE_PARAM {
Enable_RQUOTA = false;
Protocols = 3,4;
}
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
NFS_krb5 {
Active_krb5 = false;
}
NFSv4 {
Only_Numeric_Owners = true;
RecoveryBackend = rados_kv;
Grace_Period = 5;
Lease_Lifetime = 5;
Yikes! That's _way_ too short a grace period and lease lifetime. Ganesha
will probably exit the grace period before the clients ever realize the
server has restarted, and they will fail to reclaim their state.
Minor_Versions = 1,2;
}
RADOS_KV {
ceph_conf = '/etc/ceph/ceph.conf';
userid = "ganesha";
pool = "cephfs_metadata";
namespace = "ganesha";
nodeid = "cephgw-k2-1";
}
Any hint would be appreciated.
I consider ganesha's dbus-based takeover mechanism to be broken by
design, as it requires the recovery backend to do things that can't be
done atomically. If a crash occurs at the wrong time, the recovery
database can end up trashed and no one can reclaim anything.
If you really want an active/passive setup then I'd move away from that
and just have whatever clustering software you're using start up the
daemon on the active node after ensuring that it's shut down on the
passive one. With that, you can also use the rados_ng recovery backend,
which is more resilient in the face of multiple crashes.
In that configuration you would want to have the same config file on
both nodes, including the same nodeid so that you can potentially take
advantage of the RECLAIM_RESET interface to kill off the old session
quickly after the server restarts.
You also need a much longer grace period.
Cheers,
--
Jeff Layton <jlayton(a)redhat.com>