kpkim has uploaded this change for review.

View Change

SAL: keep a recovery record while another clientid still uses it

The fs recovery backends name a client's record after the client address
and the client owner ("<client addr>-(<len>:<owner>)"). The server address
the client connected to is not part of the name.

When one node serves several floating addresses (for example after a
takeover and a later rebalance), a client that mounted more than one of
those addresses gets one clientid per address, and all of them map to the
same record in the node's recovery directory. When one of those
addresses moves to another node, its clientid stops renewing and the
reaper expires it. nfs_client_id_expire() then calls nfs4_rm_clid(),
which removes the shared record although the client's other clientid on
this node is still confirmed and holds state.

If this node fails next, the node that takes over its addresses loads no
record for that client. With clid_count at zero, nfs_try_lift_grace()
ends grace on the next reaper pass, and the client's reclaim of its
locks fails with NFS4ERR_NO_GRACE. Applications see EIO on the lost
lock. We hit this with three nodes, ctdb-managed floating addresses and
the fs backend on a shared directory: the second of three node power
cuts lost an fcntl lock this way, and the sequence reproduces reliably
once the takeover node has left its own grace period.

Before removing the record, look for another confirmed clientid with the
same cid_recov_tag. If one exists, keep the record and only drop this
clientid's tag. With recovery_backend_ipbased the records live in
per-server-address directories, so a clientid only counts as sharing the
record when the server address matches too. The last clientid using the
record still removes it as before.

The scan takes each hash partition lock and then the other clientid's
cid_mutex, the same order the reaper uses, and runs before this
clientid's own cid_mutex is taken.

Reported in https://github.com/nfs-ganesha/nfs-ganesha/issues/1443

Change-Id: I46b3c1c0648712dc0b6aae282c28747d71a62d38
Signed-off-by: Kyeongpyo Kim <hgichon@gmail.com>
---
M src/SAL/nfs4_clientid.c
M src/SAL/nfs4_recovery.c
M src/include/sal_functions.h
3 files changed, 83 insertions(+), 0 deletions(-)

git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/98/1261098/1

To view, visit change 1261098. To unsubscribe, or for help writing mail filters, visit settings.

Gerrit-MessageType: newchange
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I46b3c1c0648712dc0b6aae282c28747d71a62d38
Gerrit-Change-Number: 1261098
Gerrit-PatchSet: 1
Gerrit-Owner: kpkim <hgichon@gmail.com>