April 2021 - Devel - NFS Ganesha List Archives

Change in ...nfs-ganesha[next]: Allow EXPORT pseudo path to be changed during export update
by Frank Filz (GerritHub) 20 Feb '24

20 Feb '24

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/490334 ) Change subject: Allow EXPORT pseudo path to be changed during export update ...................................................................... Allow EXPORT pseudo path to be changed during export update This also fully allows adding or removing NFSv4 support from an export since we can now handle the PseudoFS swizzing that occurs. Note that an explicit PseudoFS export may be removed or added, though you can not change it from export_id 0 because we currently don't allow changing the export_id. Note that this patch doesn't handle DBUS add or remove export though that is an option to improve. I may add them to this patch (it wouldn't be that hard) but I want to get this reviewed as is right now. There are implications to a client of changing the PseudoFS. I have tested moving an export in the PseudoFS with a client mounted. The client will be able to continue accessing the export, though it may see an ESTALE error if it navigates out of the export. The current working directory will go bad and the pwd comment will fail indicating a disconnected mount. I have also seen referencing .. from the root of the export wrapping around back to the root (I believe this is how disconnected mounts are set up). FSAL_PSEUDO lookups and create handles (PUTFH or any use of an NFSv3 handle where the inode isn't cached) which fail during an export update are instead turned into ERR_FSAL_DELAY which turns into NFS4ERR_DELAY or NFS3ERR_JUKEBOX to force the client to retry under the completed update. Change-Id: I507dc17a651936936de82303ff1291677ce136be Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/FSAL_PSEUDO/handle.c M src/MainNFSD/libganesha_nfsd.ver M src/Protocols/NFS/nfs4_pseudo.c M src/include/export_mgr.h M src/include/nfs_proto_functions.h M src/support/export_mgr.c M src/support/exports.c 7 files changed, 560 insertions(+), 203 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/34/490334/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/490334 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I507dc17a651936936de82303ff1291677ce136be Gerrit-Change-Number: 490334 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

2 2

Monitoring in Ganesha?
by Bjorn Leffler 22 Nov '21

22 Nov '21

Apart from the counters that you can access through dbus, is there any other monitoring built into Ganesha? I'm thinking of adding it with this higher level plan: - Exporting metrics from Ganesha to Prometheus. - Aggregate data in Prometheus. - Display monitoring consoles and graphs with Grafana. - Package up Prometheus, Grafana and the preconfigured rules/dashboards as a Docker image. - This makes it straightforward to deploy monitoring alongside a Gansha binary. My rough coding plan for the code is to: - Add a USE_MONITORING directive to the CMakeLists.txt files. - Add a build dependency to the Prometheus C client <https://github.com/digitalocean/prometheus-client-c>. - Create a src/monitoring directory for the new source files and templates. - Increment counters and timers throughout the code. - Use histograms to compute latency percentiles, heatmaps, etc. Is this a good idea? Any objections or suggestions? Thanks, Bjorn

3 6

Change in ...nfs-ganesha[next]: v4_recovery: off-by-one error, v4_recov_dir is truncated
by Kaleb KEITHLEY (GerritHub) 29 Apr '21

29 Apr '21

Kaleb KEITHLEY has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/515560 ) Change subject: v4_recovery: off-by-one error, v4_recov_dir is truncated ...................................................................... v4_recovery: off-by-one error, v4_recov_dir is truncated noticed log entries with invalid path name, e.g. ... fs_rm_revoked_handles :CLIENT ID :EVENT :opendir /var/lib/nfs/ganesha/v4reco/::ffff:192.168.122.102... and ... fs_rm_clid_impl :CLIENT ID :EVENT :Failed to remove client recovery dir (/var/lib/nfs/ganesha/v4reco/::ffff:192... Inspection of running process shows that v4_recov_dir_len is 27, but "/var/lib/nfs/ganesha/v4recov" is 28 chars in length. As a result, at lines 375-376 of fs_recovery.c (in 3.x) one char too few are copied from the initial parent_path (i.e. v4_recov_dir) when called from fs_rm_clid(). And at line 892 it kludges around the mistake by copying v4_recov_dir_len + 1 chars. Also change to unsigned int for v4_recov_dir_len and v4_old_dir_len which can never be negative. (Could have used size_t I suppose.) Change-Id: If25ef835ea459b3e8c796e38fc940890f27b58a7 Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> --- M src/SAL/recovery/recovery_fs.c 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/60/515560/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/515560 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: If25ef835ea459b3e8c796e38fc940890f27b58a7 Gerrit-Change-Number: 515560 Gerrit-PatchSet: 1 Gerrit-Owner: Kaleb KEITHLEY <kaleb(a)redhat.com> Gerrit-MessageType: newchange

1 0

Change in ...nfs-ganesha[next]: Improve export-config man page
by Frank Filz (GerritHub) 27 Apr '21

27 Apr '21

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/515366 ) Change subject: Improve export-config man page ...................................................................... Improve export-config man page Change-Id: Idbfe5a89ac3da935d674bac0db40d9fbf8eee5b4 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/doc/man/ganesha-export-config.rst 1 file changed, 193 insertions(+), 6 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/66/515366/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/515366 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Idbfe5a89ac3da935d674bac0db40d9fbf8eee5b4 Gerrit-Change-Number: 515366 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

1 0

Change in ...nfs-ganesha[next]: selinux: additional policy for RHEL8.4
by Kaleb KEITHLEY (GerritHub) 27 Apr '21

27 Apr '21

Kaleb KEITHLEY has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/515361 ) Change subject: selinux: additional policy for RHEL8.4 ...................................................................... selinux: additional policy for RHEL8.4 various fusefs_t, new in RHEL8.4 Change-Id: Ib41f89174550756aa8edb2a923b7ca5312505e8d Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> --- M src/selinux/ganesha.te 1 file changed, 9 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/61/515361/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/515361 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ib41f89174550756aa8edb2a923b7ca5312505e8d Gerrit-Change-Number: 515361 Gerrit-PatchSet: 1 Gerrit-Owner: Kaleb KEITHLEY <kaleb(a)redhat.com> Gerrit-MessageType: newchange

1 0

Announce Push of V4-dev.58
by Frank Filz 23 Apr '21

23 Apr '21

Branch next Tag:V4-dev.58 Merge Highlights * Fix stall when readdir races with I/O and a new mdcache entry created Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 576e3ba Frank S. Filz V4-dev.58 aadb9cf Frank S. Filz Fix stall when readdir races with I/O and a new mdcache entry created

1 0

Re: Failed to grab state owner mutex in _state_del_locked
by Sriram Patil 22 Apr '21

22 Apr '21

Hi, With full debug logs for RW LOCK, I could see that the lock gets destroyed as part of free_client_id. 2021-04-22T13:09:39Z : epoch 6080e5a2 : w1hs3r0302.vsanstfsad.local : ganesha.nfsd-180[::ffff:172.30.100.232] [svc_440] 349 :free_client_id :RW LOCK :Destroy mutex 0x7fa390179148 (&clientid->cid_owner.so_mutex) at /build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_clientid.c:349 …… …… 2021-04-22T13:19:44Z : epoch 6080e5a2 : w1hs3r0302.vsanstfsad.local : ganesha.nfsd-180[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22, acquiring mutex 0x7fa390179148 (&owner->so_mutex) at /build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 I think the culprit is “free_client_id” function. In this function the “cid_owner->so_mutex” is destroyed without worrying about the refcount PTHREAD_MUTEX_destroy(&clientid->cid_mutex); PTHREAD_MUTEX_destroy(&clientid->cid_owner.so_mutex); if (clientid->cid_minorversion == 0) PTHREAD_MUTEX_destroy(&clientid->cid_cb.v40.cb_chan.mtx); put_gsh_client(clientid->gsh_client); I see that there have been many changes in export CRUD workflows so it would be difficult to simply backport some of these changes to 2.8.4. Thanks, Sriram From: Sriram Patil <sriramp(a)vmware.com> Date: Tuesday, April 20, 2021 at 12:53 PM To: Matthew DeVore <matvore(a)google.com> Cc: Solomon Boulos <boulos(a)google.com>, Frank Filz <ffilzlnx(a)mindspring.com>, dang(a)redhat.com <dang(a)redhat.com>, devel(a)lists.nfs-ganesha.org <devel(a)lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: Failed to grab state owner mutex in _state_del_locked Thanks Matthew. I checked the fixes. Unfortunately, are not related to state owner mutex. I am re-running the tests on our side with full debug logging to see if I can debug the issue further. Thanks, Sriram From: Matthew DeVore <matvore(a)google.com> Date: Tuesday, April 20, 2021 at 12:39 PM To: Sriram Patil <sriramp(a)vmware.com> Cc: Solomon Boulos <boulos(a)google.com>, Frank Filz <ffilzlnx(a)mindspring.com>, dang(a)redhat.com <dang(a)redhat.com>, devel(a)lists.nfs-ganesha.org <devel(a)lists.nfs-ganesha.org> Subject: Re: [NFS-Ganesha-Devel] Re: Failed to grab state owner mutex in _state_del_locked 2021年4月20日(火) 11:42 Sriram Patil <sriramp(a)vmware.com<mailto:sriramp@vmware.com>>: Hi Solomon, Thanks for CC’ing Matthew. I saw a couple of review requests from him but they were abandoned. So, not sure which ones are valid patches. You can see the ones that were accepted with `git log --author=matvore@` My last pthread-related fix was 28e457 - I guess you need Ganesha v4 for that - `git tag --contains 28e457` shows for me: V4-dev.41 V4-dev.42 V4-dev.43 V4-dev.44 Thanks, Sriram From: Solomon Boulos <boulos(a)google.com<mailto:boulos@google.com>> Date: Tuesday, April 20, 2021 at 11:39 AM To: Frank Filz <ffilzlnx(a)mindspring.com<mailto:ffilzlnx@mindspring.com>> Cc: Matthew DeVore <matvore(a)google.com<mailto:matvore@google.com>>, Sriram Patil <sriramp(a)vmware.com<mailto:sriramp@vmware.com>>, dang(a)redhat.com<mailto:dang@redhat.com> <dang(a)redhat.com<mailto:dang@redhat.com>>, devel(a)lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org> <devel(a)lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org>> Subject: Re: [NFS-Ganesha-Devel] Re: Failed to grab state owner mutex in _state_del_locked Weren’t there a bunch of fixes from devore (cc’ed) where the union of mutex and something else (condition variable?) weren’t used consistently? That was the first thing that popped in my mind for EINVAL. On Tue, Apr 20, 2021 at 11:33 Frank Filz <ffilzlnx(a)mindspring.com<mailto:ffilzlnx@mindspring.com>> wrote: There are some possibly related fixes in V3.4 that aren’t in V2.8.4 but they don’t look like they might be relevant. I do know that delegations have not been well tested lately, so if you’re using delegations, there might well be a lock problem. Frank From: Sriram Patil [mailto:sriramp@vmware.com<mailto:sriramp@vmware.com>] Sent: Tuesday, April 20, 2021 10:33 AM To: devel(a)lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org> Cc: Frank Filz <ffilzlnx(a)mindspring.com<mailto:ffilzlnx@mindspring.com>>; dang(a)redhat.com<mailto:dang@redhat.com> Subject: [NFS-Ganesha-Devel] Failed to grab state owner mutex in _state_del_locked Hi, Recently we have been observing a ganesha abort because it receives EINVAL when trying to lock the state owner lock (owner->so_mutex). 2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local : ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22, acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at /build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 I modified some macros and printed RW LOCK activities whenever mutex name is “&owner->so_mutex”. In this, I observed that the lock is never destroyed. So, this EINVAL error is confusing. The EINVAL is observed when removing the export. The previous log for the lock is in DELEG RETURN. Could the mutex have been destroyed in the free_client_id function (nfs4_clientid.c)? In that case, it won't use that string to destroy it, but instead use `&clientid->cid_owner.so_mutex` FWIW I found it effective to log all the destroys/creates (-N NIV_FULL_DEBUG) and then grep the log for the hex address (0x7f2fc4007958 in this case) to know all activity on it. 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4 :Request 3: opcode 8 is OP_DELEGRETURN 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn :NFS4 LOCK :Entering NFS v4 DELEGRETURN handler ----------------------------------------------------- 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on 0x7f18d800be70 fd 90 (::ffff:172.30.72.54:720<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.30.72.…>) xid=2141597608 returned XPRT_IDLE 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn :NFS4 LOCK :Successful exit 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW LOCK :Acquired mutex 0x7f18b4017058 (&owner->so_mutex) at /build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 …… ….. 2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22, acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at /build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 I am not very familiar with the NFSv4 state owner code. But does this look like some known issue? Note: We are using ganesha 2.8.4 Thanks, Sriram _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org<mailto:devel-leave@lists.nfs-ganesha.org>

1 0

Re: Failed to grab state owner mutex in _state_del_locked
by Matthew DeVore 20 Apr '21

20 Apr '21

2021年4月20日(火) 11:42 Sriram Patil <sriramp(a)vmware.com>: > Hi Solomon, > > > > Thanks for CC’ing Matthew. I saw a couple of review requests from him but > they were abandoned. So, not sure which ones are valid patches. > You can see the ones that were accepted with `git log --author=matvore@` My last pthread-related fix was 28e457 - I guess you need Ganesha v4 for that - `git tag --contains 28e457` shows for me: V4-dev.41 V4-dev.42 V4-dev.43 V4-dev.44 > > > Thanks, > > Sriram > > > > *From: *Solomon Boulos <boulos(a)google.com> > *Date: *Tuesday, April 20, 2021 at 11:39 AM > *To: *Frank Filz <ffilzlnx(a)mindspring.com> > *Cc: *Matthew DeVore <matvore(a)google.com>, Sriram Patil < > sriramp(a)vmware.com>, dang(a)redhat.com <dang(a)redhat.com>, > devel(a)lists.nfs-ganesha.org <devel(a)lists.nfs-ganesha.org> > *Subject: *Re: [NFS-Ganesha-Devel] Re: Failed to grab state owner mutex > in _state_del_locked > > Weren’t there a bunch of fixes from devore (cc’ed) where the union of > mutex and something else (condition variable?) weren’t used consistently? > That was the first thing that popped in my mind for EINVAL. > > > > On Tue, Apr 20, 2021 at 11:33 Frank Filz <ffilzlnx(a)mindspring.com> wrote: > > There are some possibly related fixes in V3.4 that aren’t in V2.8.4 but > they don’t look like they might be relevant. > > > > I do know that delegations have not been well tested lately, so if you’re > using delegations, there might well be a lock problem. > > > > Frank > > > > *From:* Sriram Patil [mailto:sriramp@vmware.com] > *Sent:* Tuesday, April 20, 2021 10:33 AM > *To:* devel(a)lists.nfs-ganesha.org > *Cc:* Frank Filz <ffilzlnx(a)mindspring.com>; dang(a)redhat.com > *Subject:* [NFS-Ganesha-Devel] Failed to grab state owner mutex in > _state_del_locked > > > > Hi, > > > > Recently we have been observing a ganesha abort because it receives EINVAL > when trying to lock the state owner lock (owner->so_mutex). > > > > 2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local : > ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK > :Error 22, acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at > /build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 > > > > I modified some macros and printed RW LOCK activities whenever mutex name > is “&owner->so_mutex”. In this, I observed that the lock is never > destroyed. So, this EINVAL error is confusing. The EINVAL is observed when > removing the export. The previous log for the lock is in DELEG RETURN. > > Could the mutex have been destroyed in the free_client_id function (nfs4_clientid.c)? In that case, it won't use that string to destroy it, but instead use `&clientid->cid_owner.so_mutex` FWIW I found it effective to log all the destroys/creates (-N NIV_FULL_DEBUG) and then grep the log for the hex address (0x7f2fc4007958 in this case) to know all activity on it. > > > 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : > ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4 > :Request 3: opcode 8 is OP_DELEGRETURN > > > > 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : > ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn > :NFS4 LOCK :Entering NFS v4 DELEGRETURN handler > ----------------------------------------------------- > > > > 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : > ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on > 0x7f18d800be70 fd 90 (::ffff:172.30.72.54:720 > <https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.30.72.…>) > xid=2141597608 returned XPRT_IDLE > > > 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : > ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn > :NFS4 LOCK :Successful exit > > 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : > ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW > LOCK :Acquired mutex 0x7f18b4017058 (&owner->so_mutex) at > /build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 > > …… > > ….. > > 2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : > ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK > :Error 22, acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at > /build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397 > > > > I am not very familiar with the NFSv4 state owner code. But does this look > like some known issue? > > > > Note: We are using ganesha 2.8.4 > > > > Thanks, > > Sriram > > > > > > _______________________________________________ > Devel mailing list -- devel(a)lists.nfs-ganesha.org > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > >

2 1

Re: Failed to grab state owner mutex in _state_del_locked
by Frank Filz 20 Apr '21

20 Apr '21

There are some possibly related fixes in V3.4 that aren't in V2.8.4 but they don't look like they might be relevant. I do know that delegations have not been well tested lately, so if you're using delegations, there might well be a lock problem. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Tuesday, April 20, 2021 10:33 AM To: devel(a)lists.nfs-ganesha.org Cc: Frank Filz <ffilzlnx(a)mindspring.com>; dang(a)redhat.com Subject: [NFS-Ganesha-Devel] Failed to grab state owner mutex in _state_del_locked Hi, Recently we have been observing a ganesha abort because it receives EINVAL when trying to lock the state owner lock (owner->so_mutex). 2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local : ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22, acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at /build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/ nfs4_state.c:397 I modified some macros and printed RW LOCK activities whenever mutex name is "&owner->so_mutex". In this, I observed that the lock is never destroyed. So, this EINVAL error is confusing. The EINVAL is observed when removing the export. The previous log for the lock is in DELEG RETURN. 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4 :Request 3: opcode 8 is OP_DELEGRETURN 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn :NFS4 LOCK :Entering NFS v4 DELEGRETURN handler ----------------------------------------------------- 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on 0x7f18d800be70 fd 90 (::ffff:172.30.72.54:720) xid=2141597608 returned XPRT_IDLE 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn :NFS4 LOCK :Successful exit 2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW LOCK :Acquired mutex 0x7f18b4017058 (&owner->so_mutex) at /build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nf s4_state.c:397 .. ... 2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local : ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22, acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at /build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nf s4_state.c:397 I am not very familiar with the NFSv4 state owner code. But does this look like some known issue? Note: We are using ganesha 2.8.4 Thanks, Sriram

3 3

Change in ...nfs-ganesha[next]: Fix stall when readdir races with I/O and a new mdcache entry created
by Frank Filz (GerritHub) 20 Apr '21

20 Apr '21

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/514964 ) Change subject: Fix stall when readdir races with I/O and a new mdcache entry created ...................................................................... Fix stall when readdir races with I/O and a new mdcache entry created If readdir causes a new mdcache entry to be created and that races with I/O on another instance of the cache entry, which thus requires merge_share to resolve the share reservation, the need to take the obj_lock for write blocks while the I/O completes while holding the lock for read. In this case, there's actually no need to do anything in merge_share because all the counters in the readdir's copy of the obj handle are 0. Since that copy of the obj handle MUST not accessible to any other thread, it can be accessed without holding the lock. So we move acquiring and releasing the lock into merge_share and check that the dupe_share has non-zero counters before taking the lock. Change-Id: I372713360a5c795f77b1365c263c66302ec2490a Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/FSAL_CEPH/handle.c M src/FSAL/FSAL_GLUSTER/handle.c M src/FSAL/FSAL_GPFS/file.c M src/FSAL/FSAL_KVSFS/kvsfs_handle.c M src/FSAL/FSAL_LIZARDFS/handle.c M src/FSAL/FSAL_MEM/mem_handle.c M src/FSAL/FSAL_RGW/handle.c M src/FSAL/FSAL_VFS/file.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c M src/FSAL/commonlib.c M src/include/FSAL/fsal_commonlib.h 11 files changed, 53 insertions(+), 53 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/64/514964/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/514964 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I372713360a5c795f77b1365c263c66302ec2490a Gerrit-Change-Number: 514964 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

1 0