11:32 a.m.
There are some possibly related fixes in V3.4 that aren't in V2.8.4 but they
don't look like they might be relevant.
I do know that delegations have not been well tested lately, so if you're
using delegations, there might well be a lock problem.
Frank
From: Sriram Patil [mailto:sriramp@vmware.com]
Sent: Tuesday, April 20, 2021 10:33 AM
To: devel(a)lists.nfs-ganesha.org
Cc: Frank Filz <ffilzlnx(a)mindspring.com>; dang(a)redhat.com
Subject: [NFS-Ganesha-Devel] Failed to grab state owner mutex in
_state_del_locked
Hi,
Recently we have been observing a ganesha abort because it receives EINVAL
when trying to lock the state owner lock (owner->so_mutex).
2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK
:Error 22, acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at
/build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/
nfs4_state.c:397
I modified some macros and printed RW LOCK activities whenever mutex name is
"&owner->so_mutex". In this, I observed that the lock is never
destroyed.
So, this EINVAL error is confusing. The EINVAL is observed when removing the
export. The previous log for the lock is in DELEG RETURN.
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4
:Request 3: opcode 8 is OP_DELEGRETURN
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn :NFS4
LOCK :Entering NFS v4 DELEGRETURN handler
-----------------------------------------------------
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on
0x7f18d800be70 fd 90 (::ffff:172.30.72.54:720) xid=2141597608 returned
XPRT_IDLE
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn
:NFS4 LOCK :Successful exit
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW
LOCK :Acquired mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nf
s4_state.c:397
..
...
2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK
:Error 22, acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nf
s4_state.c:397
I am not very familiar with the NFSv4 state owner code. But does this look
like some known issue?
Note: We are using ganesha 2.8.4
Thanks,
Sriram
11:39 a.m.
New subject: Failed to grab state owner mutex in _state_del_locked
Weren’t there a bunch of fixes from devore (cc’ed) where the union of mutex
and something else (condition variable?) weren’t used consistently? That
was the first thing that popped in my mind for EINVAL.
On Tue, Apr 20, 2021 at 11:33 Frank Filz <ffilzlnx(a)mindspring.com> wrote:
There are some possibly related fixes in V3.4 that aren’t in V2.8.4
but
they don’t look like they might be relevant.
I do know that delegations have not been well tested lately, so if you’re
using delegations, there might well be a lock problem.
Frank
*From:* Sriram Patil [mailto:sriramp@vmware.com]
*Sent:* Tuesday, April 20, 2021 10:33 AM
*To:* devel(a)lists.nfs-ganesha.org
*Cc:* Frank Filz <ffilzlnx(a)mindspring.com>; dang(a)redhat.com
*Subject:* [NFS-Ganesha-Devel] Failed to grab state owner mutex in
_state_del_locked
Hi,
Recently we have been observing a ganesha abort because it receives EINVAL
when trying to lock the state owner lock (owner->so_mutex).
2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK
:Error 22, acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at
/build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
I modified some macros and printed RW LOCK activities whenever mutex name
is “&owner->so_mutex”. In this, I observed that the lock is never
destroyed. So, this EINVAL error is confusing. The EINVAL is observed when
removing the export. The previous log for the lock is in DELEG RETURN.
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4
:Request 3: opcode 8 is OP_DELEGRETURN
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn
:NFS4 LOCK :Entering NFS v4 DELEGRETURN handler
-----------------------------------------------------
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on
0x7f18d800be70 fd 90 (::ffff:172.30.72.54:720) xid=2141597608 returned
XPRT_IDLE
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn
:NFS4 LOCK :Successful exit
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW
LOCK :Acquired mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
……
…..
2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK
:Error 22, acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
I am not very familiar with the NFSv4 state owner code. But does this look
like some known issue?
Note: We are using ganesha 2.8.4
Thanks,
Sriram
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
11:42 a.m.
New subject: Failed to grab state owner mutex in _state_del_locked
Hi Solomon,
Thanks for CC’ing Matthew. I saw a couple of review requests from him but they were
abandoned. So, not sure which ones are valid patches.
Thanks,
Sriram
From: Solomon Boulos <boulos(a)google.com>
Date: Tuesday, April 20, 2021 at 11:39 AM
To: Frank Filz <ffilzlnx(a)mindspring.com>
Cc: Matthew DeVore <matvore(a)google.com>, Sriram Patil <sriramp(a)vmware.com>,
dang(a)redhat.com <dang(a)redhat.com>, devel(a)lists.nfs-ganesha.org
<devel(a)lists.nfs-ganesha.org>
Subject: Re: [NFS-Ganesha-Devel] Re: Failed to grab state owner mutex in
_state_del_locked
Weren’t there a bunch of fixes from devore (cc’ed) where the union of mutex and something
else (condition variable?) weren’t used consistently? That was the first thing that popped
in my mind for EINVAL.
On Tue, Apr 20, 2021 at 11:33 Frank Filz
<ffilzlnx@mindspring.com<mailto:ffilzlnx@mindspring.com>> wrote:
There are some possibly related fixes in V3.4 that aren’t in V2.8.4 but they don’t look
like they might be relevant.
I do know that delegations have not been well tested lately, so if you’re using
delegations, there might well be a lock problem.
Frank
From: Sriram Patil [mailto:sriramp@vmware.com<mailto:sriramp@vmware.com>]
Sent: Tuesday, April 20, 2021 10:33 AM
To: devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org>
Cc: Frank Filz <ffilzlnx@mindspring.com<mailto:ffilzlnx@mindspring.com>>;
dang@redhat.com<mailto:dang@redhat.com>
Subject: [NFS-Ganesha-Devel] Failed to grab state owner mutex in _state_del_locked
Hi,
Recently we have been observing a ganesha abort because it receives EINVAL when trying to
lock the state owner lock (owner->so_mutex).
2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22,
acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at
/build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
I modified some macros and printed RW LOCK activities whenever mutex name is
“&owner->so_mutex”. In this, I observed that the lock is never destroyed. So, this
EINVAL error is confusing. The EINVAL is observed when removing the export. The previous
log for the lock is in DELEG RETURN.
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4 :Request 3:
opcode 8 is OP_DELEGRETURN
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn :NFS4 LOCK
:Entering NFS v4 DELEGRETURN handler
-----------------------------------------------------
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on 0x7f18d800be70
fd 90
(::ffff:172.30.72.54:720<https://nam04.safelinks.protection.outlook.com/?url=http%3A%2F%2F172.30.72.54%3A720%2F&data=04%7C01%7Csriramp%40vmware.com%7Cf7787e2b327f46fbde9208d9042bacf2%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637545407911295122%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aEiDqYSF5oSVQ2NTdd7JHYjn4cL%2FNpy0lVcZOPsF7k0%3D&reserved=0>)
xid=2141597608 returned XPRT_IDLE
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn :NFS4 LOCK
:Successful exit
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW LOCK :Acquired
mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
……
…..
2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22,
acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
I am not very familiar with the NFSv4 state owner code. But does this look like some known
issue?
Note: We are using ganesha 2.8.4
Thanks,
Sriram
_______________________________________________
Devel mailing list --
devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org>
To unsubscribe send an email to
devel-leave@lists.nfs-ganesha.org<mailto:devel-leave@lists.nfs-ganesha.org>
11:41 a.m.
New subject: Failed to grab state owner mutex in _state_del_locked
Hi Frank,
Let me try to run our tests by disabling delegations to confirm this. May be I can get
enable RW LOCKS debugging to pin point exactly which op/thread is destroying the lock.
This could also be some ref counting issue.
Thanks,
Sriram
From: Frank Filz <ffilzlnx(a)mindspring.com>
Date: Tuesday, April 20, 2021 at 11:32 AM
To: Sriram Patil <sriramp(a)vmware.com>, devel(a)lists.nfs-ganesha.org
<devel(a)lists.nfs-ganesha.org>
Cc: dang(a)redhat.com <dang(a)redhat.com>
Subject: RE: [NFS-Ganesha-Devel] Failed to grab state owner mutex in _state_del_locked
There are some possibly related fixes in V3.4 that aren’t in V2.8.4 but they don’t look
like they might be relevant.
I do know that delegations have not been well tested lately, so if you’re using
delegations, there might well be a lock problem.
Frank
From: Sriram Patil [mailto:sriramp@vmware.com]
Sent: Tuesday, April 20, 2021 10:33 AM
To: devel(a)lists.nfs-ganesha.org
Cc: Frank Filz <ffilzlnx(a)mindspring.com>; dang(a)redhat.com
Subject: [NFS-Ganesha-Devel] Failed to grab state owner mutex in _state_del_locked
Hi,
Recently we have been observing a ganesha abort because it receives EINVAL when trying to
lock the state owner lock (owner->so_mutex).
2021-03-28T10:09:12Z : epoch 605fa7ae : w1hs3i1902.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22,
acquiring mutex 0x7f2fc4007958 (&owner->so_mutex) at
/build/mts/release/bora-17422501/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
I modified some macros and printed RW LOCK activities whenever mutex name is
“&owner->so_mutex”. In this, I observed that the lock is never destroyed. So, this
EINVAL error is confusing. The EINVAL is observed when removing the export. The previous
log for the lock is in DELEG RETURN.
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 775 :process_one_op :NFS4 :Request 3:
opcode 8 is OP_DELEGRETURN
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 76 :nfs4_op_delegreturn :NFS4 LOCK
:Entering NFS v4 DELEGRETURN handler
-----------------------------------------------------
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [svc_181] 1377 :free_nfs_request :DISP :SVC_DECODE on 0x7f18d800be70
fd 90 (::ffff:172.30.72.54:720) xid=2141597608 returned XPRT_IDLE
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 129 :nfs4_op_delegreturn :NFS4 LOCK
:Successful exit
2021-04-20T04:07:12Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[::ffff:172.30.72.54] [svc_165] 397 :_state_del_locked :RW LOCK :Acquired
mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
……
…..
2021-04-20T04:57:00Z : epoch 607e2d18 : w1hs3r0313.vsanstfsad.local :
ganesha.nfsd-90[none] [dbus_heartbeat] 397 :_state_del_locked :RW LOCK :Error 22,
acquiring mutex 0x7f18b4017058 (&owner->so_mutex) at
/build/mts/release/sb-45847366/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_state.c:397
I am not very familiar with the NFSv4 state owner code. But does this look like some known
issue?
Note: We are using ganesha 2.8.4
Thanks,
Sriram
1101
days inactive
1101
days old
3 comments
3 participants
participants (3)
-
Frank Filz -
Solomon Boulos -
Sriram Patil