kpkim has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1261098?usp=email ) Change subject: SAL: keep a recovery record while another clientid still uses it ...................................................................... SAL: keep a recovery record while another clientid still uses it The fs recovery backends name a client's record after the client address and the client owner ("<client addr>-(<len>:<owner>)"). The server address the client connected to is not part of the name. When one node serves several floating addresses (for example after a takeover and a later rebalance), a client that mounted more than one of those addresses gets one clientid per address, and all of them map to the same record in the node's recovery directory. When one of those addresses moves to another node, its clientid stops renewing and the reaper expires it. nfs_client_id_expire() then calls nfs4_rm_clid(), which removes the shared record although the client's other clientid on this node is still confirmed and holds state. If this node fails next, the node that takes over its addresses loads no record for that client. With clid_count at zero, nfs_try_lift_grace() ends grace on the next reaper pass, and the client's reclaim of its locks fails with NFS4ERR_NO_GRACE. Applications see EIO on the lost lock. We hit this with three nodes, ctdb-managed floating addresses and the fs backend on a shared directory: the second of three node power cuts lost an fcntl lock this way, and the sequence reproduces reliably once the takeover node has left its own grace period. Before removing the record, look for another confirmed clientid with the same cid_recov_tag. If one exists, keep the record and only drop this clientid's tag. With recovery_backend_ipbased the records live in per-server-address directories, so a clientid only counts as sharing the record when the server address matches too. The last clientid using the record still removes it as before. The scan takes each hash partition lock and then the other clientid's cid_mutex, the same order the reaper uses, and runs before this clientid's own cid_mutex is taken. Reported in https://github.com/nfs-ganesha/nfs-ganesha/issues/1443 Change-Id: I46b3c1c0648712dc0b6aae282c28747d71a62d38 Signed-off-by: Kyeongpyo Kim <hgichon@gmail.com> --- M src/SAL/nfs4_clientid.c M src/SAL/nfs4_recovery.c M src/include/sal_functions.h 3 files changed, 83 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/98/1261098/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1261098?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I46b3c1c0648712dc0b6aae282c28747d71a62d38 Gerrit-Change-Number: 1261098 Gerrit-PatchSet: 1 Gerrit-Owner: kpkim <hgichon@gmail.com>