On Mon, 2021-07-19 at 08:58 -0400, Kaleb Keithley wrote: It was developed using k8s, but in principle you should be able to use the rados_cluster recovery backend with pacemaker too. What version of ganesha and ceph cluster are you using? What OS is all of this running on? ??? ??? This looks like a terrible idea. That gives you a lot less breathing room when things truly do go wrong. This is almost certainly just papering over a real problem. A 5 min timeout suggests that it's the Ceph MDS that's timing out state from the old client, and that sounds like something is not working right. Looking further...I think there may be a problem with FSAL_CEPH, the rados_ng recovery backend and active/passive deployments: When ganesha starts up, FSAL_CEPH tells the Ceph MDS to drop any state previously held by its "nodeid". This lets the ceph client (ganesha) get on with the business of reclaim as soon as it starts up. The MDS will give up on waiting for this eventually, but it takes...5 mins. The nodeid defaults to the system hostname (a'la gethostname()). In most pacemaker-style clusters, the hostnames of the different physical nodes are different, so that request to drop old state is probably not working right here. With the rados_cluster backend, you can also configure a nodeid to use. If you set up a trivial rados_cluster with one (active/passive) host, I expect it'll work more like you want. So, I'd set this in the NFSv4 block: RecoveryBackend = rados_cluster; ...and also configure both servers in your active/passive cluster with this in the RADOS_KV block: nodeid = some_nodename; You'll also need to "ganesha-rados-grace add" the nodeid that you put in that block. Longer term, maybe we should allow rados_ng and rados_kv to set a static nodeid too? That would also probably fix this for people who just want a simple active/passive cluster. Basically we'd just need to add a get_nodeid operation for them. -- Jeff Layton <jlayton(a)redhat.com&gt;

On Mon, Jul 19, 2021 at 09:37:43AM -0400, Jeff Layton wrote: Using `rados_cluster` and sharing a `nodeid`... NFSv4 { RecoveryBackend = rados_cluster; Minor_Versions = 1,2; Grace_Period = 60; } [...] RADOS_KV { UserId = "nfsmeta"; pool = "cephfs.tank.meta"; namespace = "ganesha"; nodeid = "nfs.storage"; } ...seems to have resolved the problem; failover times now seem to track the `Grace_Period` setting. Was I misusing `rados_ng`? All the docs I found said "use rados_ng for active/passive configurations". -- Lars Kellogg-Stedman <lars(a)redhat.com&gt; | larsks @ {irc,twitter,github} http://blog.oddbit.com/ | N1LKS

On Tue, 2021-07-20 at 09:55 -0400, Jeff Layton wrote: It was pretty simple to do, though I haven't done any real testing with it so far. If you feel inclined to try it out, here's the patch: https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/521021 With that you should be able to use rados_ng, as long as you ensure that you set the same nodeid on both hosts. That said, there is nothing wrong with using rados_cluster in this configuration either, and that should allow you to eventually scale-out to multiple active nodes seamlessly if you choose. I sort of wonder if we ought to phase-out rados_ng altogether and just clearly document how to use rados_cluster in an active/passive configuration. -- Jeff Layton <jlayton(a)redhat.com&gt;