For NLM UNLOCK - connection to rpc.statd not getting closed
Hi, Not sure if my previous mail was sent, hence re-sending the mail. I used the latest ganesha code and ran the following test from a NFSv3 client in a script: ct=0 while [ $ct -lt 4096 ]; do flock -x mylock echo 1 >> myfile let ct=$ct+1 done After running for 1000+ times the client got error "No locks available" ganesha.log had the following trace: ganesha.nfsd-9328[svc_103] nsm_connect :NLM :CRIT :connect to statd failed: RPC: Unknown protocol /var/log/messages showed "Too many open files" message. It looks like for NLM LOCK requests connection to rpc.statd were created but not closed for NLM UNLOCK request. After analyzing the code, it seems this happens because for NLM LOCK request the 'xprt->xp_refcnt' is ref'ed twice. But while handling NLM UNLOCK request the 'xprt->xp_refcnt' is un-ref'ed only once, and thus svc_vc_destroy_it() doesn't get called and connection to rpc.statd is not closed. More details about the code analysis is below. Can you please check about this issue ? Thank you. I am not sure why are we incrementing 'xprt->xp_refcnt' twice in svc_xprt_lookup() ? For NLM LOCK request the code path is: -------------------------------------------------------------- nlm4_Lock() -> ...... -> nsm_connect() -> ....... -> makefd_xprt() -> svc_xprt_lookup() 137 SVCXPRT * 138 svc_xprt_lookup(int fd, svc_xprt_setup_t setup) 139 { ...... ...... 173 (*setup)(&xprt); /* zalloc, xp_refcnt = 1 */ --> leads to call to svc_vc_xprt_setup() 174 xprt->xp_fd = fd; 175 xprt->xp_flags = SVC_XPRT_FLAG_INITIAL; 176 177 /* Get ref for caller */ 178 SVC_REF(xprt, SVC_REF_FLAG_NONE); Here, at line 173 function svc_vc_xprt_setup() is called which sets 'xprt->xp_refcnt = 1' Then at line 178, SVC_REF increments 'xprt->xp_refcnt' by 1. Thus, when handling NLM LOCK request 'xprt->xp_refcnt = 2' is set. For NLM UNLOCK request the code path is: ------------------------------------------------------------------- nlm4_Unlock() -> ...... -> nsm_disconnect -> ..... -> clnt_vc_destroy() -> svc_release_it() 410 static inline void svc_release_it(SVCXPRT *xprt, u_int flags, 411 const char *tag, const int line) 412 { 413 int32_t refs = atomic_dec_int32_t(&xprt->xp_refcnt); ...... ...... 425 if (likely(refs > 0)) { 426 /* normal case */ 427 return; 428 } 429 430 /* enforce once-only semantic, trace others */ 431 xp_flags = atomic_postset_uint16_t_bits(&xprt->xp_flags, 432 SVC_XPRT_FLAG_RELEASING); ...... 439 /* Releasing last reference */ 440 (*(xprt)->xp_ops->xp_destroy)(xprt, flags, tag, line); Here, at line 413 'xprt->xp_refcnt' gets decremented and becomes 'xprt->xp_refcnt = 1'. But as 'xprt->xp_refcnt != 0' the function returns from line 427. And thus it doesn't proceed with closure of connection. Thanks, Madhu Thorat IBM Pune.
Can you try with this patch applied to ntirpc? Daniel On 4/22/19 4:42 AM, Madhu Thorat wrote:
Hi,
Not sure if my previous mail was sent, hence re-sending the mail.
I used the latest ganesha code and ran the following test from a NFSv3 client in a script: ct=0 while [ $ct -lt 4096 ]; do flock -x mylock echo 1 >> myfile let ct=$ct+1 done
After running for 1000+ times the client got error "No locks available"
ganesha.log had the following trace: ganesha.nfsd-9328[svc_103] nsm_connect :NLM :CRIT :connect to statd failed: RPC: Unknown protocol
/var/log/messages showed "Too many open files" message. It looks like for NLM LOCK requests connection to rpc.statd were created but not closed for NLM UNLOCK request.
After analyzing the code, it seems this happens because for NLM LOCK request the 'xprt->xp_refcnt' is ref'ed twice. But while handling NLM UNLOCK request the 'xprt->xp_refcnt' is un-ref'ed only once, and thus svc_vc_destroy_it() doesn't get called and connection to rpc.statd is not closed.
More details about the code analysis is below. Can you please check about this issue ? Thank you. I am not sure why are we incrementing 'xprt->xp_refcnt' twice in svc_xprt_lookup() ?
For NLM LOCK request the code path is: -------------------------------------------------------------- nlm4_Lock() -> ...... -> nsm_connect() -> ....... -> makefd_xprt() -> svc_xprt_lookup()
137 SVCXPRT * 138 svc_xprt_lookup(int fd, svc_xprt_setup_t setup) 139 { ...... ...... 173 (*setup)(&xprt); /* zalloc, xp_refcnt = 1 */ --> leads to call to svc_vc_xprt_setup() 174 xprt->xp_fd = fd; 175 xprt->xp_flags = SVC_XPRT_FLAG_INITIAL; 176 177 /* Get ref for caller */ 178 SVC_REF(xprt, SVC_REF_FLAG_NONE);
Here, at line 173 function svc_vc_xprt_setup() is called which sets 'xprt->xp_refcnt = 1' Then at line 178, SVC_REF increments 'xprt->xp_refcnt' by 1. Thus, when handling NLM LOCK request 'xprt->xp_refcnt = 2' is set.
For NLM UNLOCK request the code path is: ------------------------------------------------------------------- nlm4_Unlock() -> ...... -> nsm_disconnect -> ..... -> clnt_vc_destroy() -> svc_release_it()
410 static inline void svc_release_it(SVCXPRT *xprt, u_int flags, 411 const char *tag, const int line) 412 { 413 int32_t refs = atomic_dec_int32_t(&xprt->xp_refcnt); ...... ...... 425 if (likely(refs > 0)) { 426 /* normal case */ 427 return; 428 } 429 430 /* enforce once-only semantic, trace others */ 431 xp_flags = atomic_postset_uint16_t_bits(&xprt->xp_flags, 432 SVC_XPRT_FLAG_RELEASING); ...... 439 /* Releasing last reference */ 440 (*(xprt)->xp_ops->xp_destroy)(xprt, flags, tag, line);
Here, at line 413 'xprt->xp_refcnt' gets decremented and becomes 'xprt->xp_refcnt = 1'. But as 'xprt->xp_refcnt != 0' the function returns from line 427. And thus it doesn't proceed with closure of connection.
Thanks, Madhu Thorat IBM Pune.
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Hi,
Not sure if my previous mail was sent, hence re-sending the mail.
I used the latest ganesha code and ran the following test from a NFSv3 client in a script: ct=0 while [ $ct -lt 4096 ]; do flock -x mylock echo 1 >> myfile let ct=$ct+1 done
After running for 1000+ times the client got error "No locks available"
ganesha.log had the following trace: ganesha.nfsd-9328[svc_103] nsm_connect :NLM :CRIT :connect to statd failed: RPC: Unknown protocol
/var/log/messages showed "Too many open files" message. It looks like for NLM LOCK requests connection to rpc.statd were created but not closed for NLM UNLOCK request.
After analyzing the code, it seems this happens because for NLM LOCK request the 'xprt->xp_refcnt' is ref'ed twice. But while handling NLM UNLOCK request the 'xprt->xp_refcnt' is un-ref'ed only once, and thus svc_vc_destroy_it() doesn't get called and connection to rpc.statd is not closed.
More details about the code analysis is below. Can you please check about this issue ? Thank you. I am not sure why are we incrementing 'xprt->xp_refcnt' twice in svc_xprt_lookup() ?
For NLM LOCK request the code path is: -------------------------------------------------------------- nlm4_Lock() -> ...... -> nsm_connect() -> ....... -> makefd_xprt() -> svc_xprt_lookup()
137 SVCXPRT * 138 svc_xprt_lookup(int fd, svc_xprt_setup_t setup) 139 { ...... ...... 173 (*setup)(&xprt); /* zalloc, xp_refcnt = 1 */ --> leads to call to svc_vc_xprt_setup() 174 xprt->xp_fd = fd; 175 xprt->xp_flags = SVC_XPRT_FLAG_INITIAL; 176 177 /* Get ref for caller */ 178 SVC_REF(xprt, SVC_REF_FLAG_NONE);
Here, at line 173 function svc_vc_xprt_setup() is called which sets 'xprt->xp_refcnt = 1' Then at line 178, SVC_REF increments 'xprt->xp_refcnt' by 1. Thus, when handling NLM LOCK request 'xprt->xp_refcnt = 2' is set.
For NLM UNLOCK request the code path is: ------------------------------------------------------------------- nlm4_Unlock() -> ...... -> nsm_disconnect -> ..... -> clnt_vc_destroy() -> svc_release_it()
410 static inline void svc_release_it(SVCXPRT *xprt, u_int flags, 411 const char *tag, const int
Hi Daniel, Thank you for the patch. Wondering if you got a chance to see the last mail? If the code change in the shared link don't look appropriate please suggest another fix for this issue. Thank you. Thanks, Madhu Thorat IBM-India Software Labs, Pune. From: "Madhu P Punjabi" <madhu.punjabi@in.ibm.com> To: dang@redhat.com Cc: devel@lists.nfs-ganesha.org Date: 04/24/2019 12:42 AM Subject: [NFS-Ganesha-Devel] Re: For NLM UNLOCK - connection to rpc.statd not getting closed Hi Daniel, Thank you for the patch. I tried using it but the connection to rpc.statd remains open. Do you intend to use the flag 'CLNT_FLAG_LOCAL' something like in the following link ? On using it like this the connection gets closed in clnt_vc_destroy (). Please check the changes. Thank you. https://paste.fedoraproject.org/paste/YNfYhK5xJxjp2SS8StWKvg Thanks, Madhu Thorat. ----- Original message ----- From: Daniel Gryniewicz <dang@redhat.com> To: devel@lists.nfs-ganesha.org Cc: Subject: [NFS-Ganesha-Devel] Re: For NLM UNLOCK - connection to rpc.statd not getting closed Date: Tue, Apr 23, 2019 8:21 PM Can you try with this patch applied to ntirpc? Daniel On 4/22/19 4:42 AM, Madhu Thorat wrote: line)
412 { 413 int32_t refs = atomic_dec_int32_t(&xprt->xp_refcnt); ...... ...... 425 if (likely(refs > 0)) { 426 /* normal case */ 427 return; 428 } 429 430 /* enforce once-only semantic, trace others */ 431 xp_flags = atomic_postset_uint16_t_bits(&xprt->
xp_flags,
432 SVC_XPRT_FLAG_RELEASING); ...... 439 /* Releasing last reference */ 440 (*(xprt)->xp_ops->xp_destroy)(xprt, flags, tag, line);
Here, at line 413 'xprt->xp_refcnt' gets decremented and becomes 'xprt->xp_refcnt = 1'. But as 'xprt->xp_refcnt != 0' the function returns from line 427. And thus it doesn't proceed with closure of connection.
Thanks, Madhu Thorat IBM Pune.
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
This fixes it for me. Can you test and review? https://github.com/nfs-ganesha/ntirpc/pull/173 Daniel On 4/29/19 7:11 AM, Madhu P Punjabi wrote:
Hi Daniel,
Thank you for the patch. Wondering if you got a chance to see the last mail? If the code change in the shared link don't look appropriate please suggest another fix for this issue. Thank you.
Thanks, Madhu Thorat IBM-India Software Labs, Pune.
Inactive hide details for "Madhu P Punjabi" ---04/24/2019 12:42:35 AM---Hi Daniel,"Madhu P Punjabi" ---04/24/2019 12:42:35 AM---Hi Daniel,
From: "Madhu P Punjabi" <madhu.punjabi@in.ibm.com> To: dang@redhat.com Cc: devel@lists.nfs-ganesha.org Date: 04/24/2019 12:42 AM Subject: [NFS-Ganesha-Devel] Re: For NLM UNLOCK - connection to rpc.statd not getting closed
------------------------------------------------------------------------
Hi Daniel,
Thank you for the patch. I tried using it but the connection to rpc.statd remains open. Do you intend to use the flag 'CLNT_FLAG_LOCAL' something like in the following link ? On using it like this the connection gets closed in clnt_vc_destroy(). Please check the changes. Thank you.
_https://paste.fedoraproject.org/paste/YNfYhK5xJxjp2SS8StWKvg_
Thanks, Madhu Thorat.
----- Original message ----- From: Daniel Gryniewicz <dang@redhat.com> To: devel@lists.nfs-ganesha.org Cc: Subject: [NFS-Ganesha-Devel] Re: For NLM UNLOCK - connection to rpc.statd not getting closed Date: Tue, Apr 23, 2019 8:21 PM
Can you try with this patch applied to ntirpc?
Daniel
Hi,
Not sure if my previous mail was sent, hence re-sending the mail.
I used the latest ganesha code and ran the following test from a NFSv3 client in a script: ct=0 while [ $ct -lt 4096 ]; do flock -x mylock echo 1 >> myfile let ct=$ct+1 done
After running for 1000+ times the client got error "No locks available"
ganesha.log had the following trace: ganesha.nfsd-9328[svc_103] nsm_connect :NLM :CRIT :connect to statd failed: RPC: Unknown protocol
/var/log/messages showed "Too many open files" message. It looks like for NLM LOCK requests connection to rpc.statd were created but not closed for NLM UNLOCK request.
After analyzing the code, it seems this happens because for NLM LOCK request the 'xprt->xp_refcnt' is ref'ed twice. But while handling NLM UNLOCK request the 'xprt->xp_refcnt' is un-ref'ed only once, and thus svc_vc_destroy_it() doesn't get called and connection to rpc.statd is not closed.
More details about the code analysis is below. Can you please check about this issue ? Thank you. I am not sure why are we incrementing 'xprt->xp_refcnt' twice in svc_xprt_lookup() ?
For NLM LOCK request the code path is: -------------------------------------------------------------- nlm4_Lock() -> ...... -> nsm_connect() -> ....... -> makefd_xprt() -> svc_xprt_lookup()
137 SVCXPRT * 138 svc_xprt_lookup(int fd, svc_xprt_setup_t setup) 139 { ...... ...... 173 (*setup)(&xprt); /* zalloc, xp_refcnt = 1 */ --> leads to call to svc_vc_xprt_setup() 174 xprt->xp_fd = fd; 175 xprt->xp_flags = SVC_XPRT_FLAG_INITIAL; 176 177 /* Get ref for caller */ 178 SVC_REF(xprt, SVC_REF_FLAG_NONE);
Here, at line 173 function svc_vc_xprt_setup() is called which sets 'xprt->xp_refcnt = 1' Then at line 178, SVC_REF increments 'xprt->xp_refcnt' by 1. Thus, when handling NLM LOCK request 'xprt->xp_refcnt = 2' is set.
For NLM UNLOCK request the code path is: ------------------------------------------------------------------- nlm4_Unlock() -> ...... -> nsm_disconnect -> ..... -> clnt_vc_destroy() -> svc_release_it()
410 static inline void svc_release_it(SVCXPRT *xprt, u_int flags, 411 const char *tag, const int
On 4/22/19 4:42 AM, Madhu Thorat wrote: line)
412 { 413 int32_t refs = atomic_dec_int32_t(&xprt->xp_refcnt); ...... ...... 425 if (likely(refs > 0)) { 426 /* normal case */ 427 return; 428 } 429 430 /* enforce once-only semantic, trace others */ 431 xp_flags = atomic_postset_uint16_t_bits(&xprt->xp_flags, 432 SVC_XPRT_FLAG_RELEASING); ...... 439 /* Releasing last reference */ 440 (*(xprt)->xp_ops->xp_destroy)(xprt, flags, tag, line);
Here, at line 413 'xprt->xp_refcnt' gets decremented and becomes 'xprt->xp_refcnt = 1'. But as 'xprt->xp_refcnt != 0' the function returns from line 427. And thus it doesn't proceed with closure of connection.
Thanks, Madhu Thorat IBM Pune.
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
participants (3)
-
Daniel Gryniewicz -
Madhu P Punjabi -
Madhu Thorat