segfault when mounting with vers=4.0
Hi, I'm using nfs-ganesha 2.6.2 on Ubuntu 16.04. When a client mounts with "-t nfs4 -o sec=krb5,vers=4.0" ganesha.nfsd crashes in nfs_rpc_create_chan_v40(): Thread 11 "ganesha.nfsd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff269c700 (LWP 1118)] nfs_rpc_create_chan_v40 (clientid=0x7fffec0097b0, flags=<optimized out>) at /root/src/nfs-ganesha-2.6.2/src/MainNFSD/nfs_rpc_callback.c:591 591 if (AUTH_FAILURE(chan->auth)) { (gdb) (gdb) print chan->auth $1 = (AUTH *) 0x0 The cause seems to be the following: nfs_rpc_create_chan_v40() calls nfs_rpc_callback_setup_gss(), which calls gssd_refresh_krb5_machine_credential(). For some reason which I don't yet understand gssd_refresh_krb5_machine_credential() fails in my setup, and that makes nfs_rpc_callback_setup_gss() to return without setting chan->auth. What would be a proper fix for this problem? Rolf -- Rolf Anders ............................ http://www.rz.uni-augsburg.de Universität Augsburg, Rechenzentrum ............. Tel. (0821) 598-2030 86135 Augsburg .................................. Fax. (0821) 598-2028
On 08/06/2018 09:30 AM, Rolf Anders wrote:
Hi,
I'm using nfs-ganesha 2.6.2 on Ubuntu 16.04. When a client mounts with "-t nfs4 -o sec=krb5,vers=4.0" ganesha.nfsd crashes in nfs_rpc_create_chan_v40():
Thread 11 "ganesha.nfsd" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff269c700 (LWP 1118)] nfs_rpc_create_chan_v40 (clientid=0x7fffec0097b0, flags=<optimized out>) at /root/src/nfs-ganesha-2.6.2/src/MainNFSD/nfs_rpc_callback.c:591 591 if (AUTH_FAILURE(chan->auth)) { (gdb) (gdb) print chan->auth $1 = (AUTH *) 0x0
The cause seems to be the following: nfs_rpc_create_chan_v40() calls nfs_rpc_callback_setup_gss(), which calls gssd_refresh_krb5_machine_credential(). For some reason which I don't yet understand gssd_refresh_krb5_machine_credential() fails in my setup, and that makes nfs_rpc_callback_setup_gss() to return without setting chan->auth.
What would be a proper fix for this problem?
Rolf
Hi, Rolf. Looks like a bug. I have a potential fix, can you try it? You will need to apply auth-dummy-ganesha.patch to Ganesha, and auth-dummy-ntirpc.patch to ntirpc. You will likely have to re-run cmake to get ntirpc to regenerate it's exports. Thanks, Daniel
On Tue, Aug 07, 2018 at 10:11:51AM -0400, Daniel Gryniewicz wrote:
Looks like a bug. I have a potential fix, can you try it? You will need to apply auth-dummy-ganesha.patch to Ganesha, and auth-dummy-ntirpc.patch to ntirpc. You will likely have to re-run cmake to get ntirpc to regenerate it's exports.
thank you for the patches. They solve the segfault in nfs_rpc_create_chan_v40(), but later another one occurs in clnt_vc_call(): Thread 15 "ganesha.nfsd" received signal SIGSEGV, Segmentation fault. 0x00007ffff617ba52 in clnt_vc_call (cc=0x7fffee5b2300) at /root/src/patch/libntirpc-1.6.2/src/clnt_vc.c:301 301 (cc->cc_auth->ah_cred.oa_flavor == RPCSEC_GSS) (gdb) print cc->cc_auth $8 = (AUTH *) 0x0 (gdb) bt #0 0x00007ffff617ba52 in clnt_vc_call (cc=0x7fffee5b2300) at /root/src/patch/libntirpc-1.6.2/src/clnt_vc.c:301 #1 0x00007ffff61793d3 in clnt_req_wait_reply (cc=0x7fffee5b2300) at /root/src/patch/libntirpc-1.6.2/src/clnt_generic.c:609 #2 0x00005555555a49e0 in rpc_cb_null (chan=0x7fffee5321b0, locked=false) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_rpc_callback.c:672 #3 0x00005555555a6976 in nfs_test_cb_chan (clientid=0x7fffee532100) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_rpc_callback.c:1326 #4 0x00005555555ec557 in nfs4_op_setclientid_confirm (op=0x7fffee550020, data=0x7fffee1fba70, resp=0x7fffee5500c0) at /root/src/patch/nfs-ganesha-2.6.2/src/Protocols/NFS/nfs4_op_setclientid_confirm.c:455 #5 0x00005555555c6672 in nfs4_Compound (arg=0x7ffff13c6f08, req=0x7ffff13c6800, res=0x7fffee552280) at /root/src/patch/nfs-ganesha-2.6.2/src/Protocols/NFS/nfs4_Compound.c:752 #6 0x00005555555c316d in nfs_rpc_process_request (reqdata=0x7ffff13c6800) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_worker_thread.c:1329 #7 0x00005555555c3956 in nfs_rpc_valid_NFS (req=0x7ffff13c6800) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_worker_thread.c:1554 #8 0x00007ffff6195506 in svc_vc_decode (req=0x7ffff13c6800) at /root/src/patch/libntirpc-1.6.2/src/svc_vc.c:815 #9 0x00005555555b6214 in nfs_rpc_decode_request (xprt=0x7ffff401b400, xdrs=0x7ffff404e980) ---Type <return> to continue, or q <return> to quit---q Due to the authentication failure chan->auth is set to NULL in nfs_rpc_create_chan_v40(). Eventually chan->auth is passed to clnt_vc_call() in the cc_auth field of a struct clnt_req. Rolf -- Rolf Anders ............................ http://www.rz.uni-augsburg.de Universität Augsburg, Rechenzentrum ............. Tel. (0821) 598-2030 86135 Augsburg .................................. Fax. (0821) 598-2028
I'm confused, since the point of the of the patch was that chan->auth is never NULL. It should always have an auth (whether good or an error) in it. Can you debug and see where chan->auth is set to NULL? Daniel On 08/09/2018 12:18 PM, Rolf Anders wrote:
On Tue, Aug 07, 2018 at 10:11:51AM -0400, Daniel Gryniewicz wrote:
Looks like a bug. I have a potential fix, can you try it? You will need to apply auth-dummy-ganesha.patch to Ganesha, and auth-dummy-ntirpc.patch to ntirpc. You will likely have to re-run cmake to get ntirpc to regenerate it's exports.
thank you for the patches. They solve the segfault in nfs_rpc_create_chan_v40(), but later another one occurs in clnt_vc_call():
Thread 15 "ganesha.nfsd" received signal SIGSEGV, Segmentation fault. 0x00007ffff617ba52 in clnt_vc_call (cc=0x7fffee5b2300) at /root/src/patch/libntirpc-1.6.2/src/clnt_vc.c:301 301 (cc->cc_auth->ah_cred.oa_flavor == RPCSEC_GSS) (gdb) print cc->cc_auth $8 = (AUTH *) 0x0
(gdb) bt #0 0x00007ffff617ba52 in clnt_vc_call (cc=0x7fffee5b2300) at /root/src/patch/libntirpc-1.6.2/src/clnt_vc.c:301 #1 0x00007ffff61793d3 in clnt_req_wait_reply (cc=0x7fffee5b2300) at /root/src/patch/libntirpc-1.6.2/src/clnt_generic.c:609 #2 0x00005555555a49e0 in rpc_cb_null (chan=0x7fffee5321b0, locked=false) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_rpc_callback.c:672 #3 0x00005555555a6976 in nfs_test_cb_chan (clientid=0x7fffee532100) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_rpc_callback.c:1326 #4 0x00005555555ec557 in nfs4_op_setclientid_confirm (op=0x7fffee550020, data=0x7fffee1fba70, resp=0x7fffee5500c0) at /root/src/patch/nfs-ganesha-2.6.2/src/Protocols/NFS/nfs4_op_setclientid_confirm.c:455 #5 0x00005555555c6672 in nfs4_Compound (arg=0x7ffff13c6f08, req=0x7ffff13c6800, res=0x7fffee552280) at /root/src/patch/nfs-ganesha-2.6.2/src/Protocols/NFS/nfs4_Compound.c:752 #6 0x00005555555c316d in nfs_rpc_process_request (reqdata=0x7ffff13c6800) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_worker_thread.c:1329 #7 0x00005555555c3956 in nfs_rpc_valid_NFS (req=0x7ffff13c6800) at /root/src/patch/nfs-ganesha-2.6.2/src/MainNFSD/nfs_worker_thread.c:1554 #8 0x00007ffff6195506 in svc_vc_decode (req=0x7ffff13c6800) at /root/src/patch/libntirpc-1.6.2/src/svc_vc.c:815 #9 0x00005555555b6214 in nfs_rpc_decode_request (xprt=0x7ffff401b400, xdrs=0x7ffff404e980) ---Type <return> to continue, or q <return> to quit---q
Due to the authentication failure chan->auth is set to NULL in nfs_rpc_create_chan_v40(). Eventually chan->auth is passed to clnt_vc_call() in the cc_auth field of a struct clnt_req.
Rolf
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
On Thu, Aug 09, 2018 at 12:42:48PM -0400, Daniel Gryniewicz wrote:
I'm confused, since the point of the of the patch was that chan->auth is never NULL. It should always have an auth (whether good or an error) in it. Can you debug and see where chan->auth is set to NULL?
With the patch nfs_rpc_callback_setup_gss() creates an AUTH structure via authnone_ncreate_dummy() and sets its ah_error.re_status to RPC_SYSTEMERROR. Afterwards nfs_rpc_create_chan_v40() checks this status with AUTH_FAILURE and sets chan->auth to NULL: chan->auth = nfs_rpc_callback_setup_gss(chan, &clientid->cid_credential); [...] if (AUTH_FAILURE(chan->auth)) { err = rpc_sperror(&chan->auth->ah_error, "failed"); LogDebug(COMPONENT_NFS_CB, "%s", err); gsh_free(err); AUTH_DESTROY(chan->auth); chan->auth = NULL; return EINVAL; } Rolf -- Rolf Anders ............................ http://www.rz.uni-augsburg.de Universität Augsburg, Rechenzentrum ............. Tel. (0821) 598-2030 86135 Augsburg .................................. Fax. (0821) 598-2028
On 08/09/2018 02:17 PM, Rolf Anders wrote:
On Thu, Aug 09, 2018 at 12:42:48PM -0400, Daniel Gryniewicz wrote:
I'm confused, since the point of the of the patch was that chan->auth is never NULL. It should always have an auth (whether good or an error) in it. Can you debug and see where chan->auth is set to NULL?
With the patch nfs_rpc_callback_setup_gss() creates an AUTH structure via authnone_ncreate_dummy() and sets its ah_error.re_status to RPC_SYSTEMERROR. Afterwards nfs_rpc_create_chan_v40() checks this status with AUTH_FAILURE and sets chan->auth to NULL:
chan->auth = nfs_rpc_callback_setup_gss(chan, &clientid->cid_credential); [...] if (AUTH_FAILURE(chan->auth)) { err = rpc_sperror(&chan->auth->ah_error, "failed");
LogDebug(COMPONENT_NFS_CB, "%s", err); gsh_free(err); AUTH_DESTROY(chan->auth); chan->auth = NULL; return EINVAL; }
Gotcha, thanks. I'll look into it. Daniel
Although I'd have done it somewhat differently (setup the dummy in ntirpc itself as I had in other places in the code), Dan's proposed solution was OK. What I don't understand now: why is it still trying to clnt_call, when it has already been told the chan is EINVAL? We set fields to NULL to catch exactly these kind of errors.
On Fri, Aug 10, 2018 at 06:50:41AM -0400, William Allen Simpson wrote:
What I don't understand now: why is it still trying to clnt_call, when it has already been told the chan is EINVAL?
I don't know if that's the reason, but I noticed that nfs_rpc_get_chan() explicitly ignores the return value of nfs_rpc_create_chan_v40(): if (!chan->clnt) (void)nfs_rpc_create_chan_v40(clientid, flags); Rolf -- Rolf Anders ............................ http://www.rz.uni-augsburg.de Universität Augsburg, Rechenzentrum ............. Tel. (0821) 598-2030 86135 Augsburg .................................. Fax. (0821) 598-2028
Also, the subject is somewhat misleading, so I've corrected it. It's probably not just 4.0 support. We've known that the ntirpc GSS implementation was buggy since at least late 2014 (when I joined the project), and probably since 2013 errors were introduced. We discussed some of these issues at Fall Bake-a-thon in 2014. It doesn't handle async calls well. It requires contiguous buffers, which requires much special casing both in the epoll receives and client callbacks. Back in March/April, I'd indicated that the next thing to work on was GSS for async zero copy. But Matt forbade it, and asked my boss to stop me from changing "his" code. While I'm around to help debug existing problems, since then I've not allocated time to write new code. Could somebody please look at that code before shipping V2.7?
On 08/09/2018 02:28 PM, Daniel Gryniewicz wrote:
On 08/09/2018 02:17 PM, Rolf Anders wrote:
On Thu, Aug 09, 2018 at 12:42:48PM -0400, Daniel Gryniewicz wrote:
I'm confused, since the point of the of the patch was that chan->auth is never NULL. It should always have an auth (whether good or an error) in it. Can you debug and see where chan->auth is set to NULL?
With the patch nfs_rpc_callback_setup_gss() creates an AUTH structure via authnone_ncreate_dummy() and sets its ah_error.re_status to RPC_SYSTEMERROR. Afterwards nfs_rpc_create_chan_v40() checks this status with AUTH_FAILURE and sets chan->auth to NULL:
chan->auth = nfs_rpc_callback_setup_gss(chan, &clientid->cid_credential); [...] if (AUTH_FAILURE(chan->auth)) { err = rpc_sperror(&chan->auth->ah_error, "failed");
LogDebug(COMPONENT_NFS_CB, "%s", err); gsh_free(err); AUTH_DESTROY(chan->auth); chan->auth = NULL; return EINVAL; }
Gotcha, thanks. I'll look into it.
Daniel
Okay, here's a replacement ganesha patch. Leave ntirpc patched with the old patch. Can you try this? Daniel
On Fri, Aug 10, 2018 at 08:57:04AM -0400, Daniel Gryniewicz wrote:
Okay, here's a replacement ganesha patch. Leave ntirpc patched with the old patch. Can you try this?
This patch solves the problem. Thank you. Rolf -- Rolf Anders ............................ http://www.rz.uni-augsburg.de Universität Augsburg, Rechenzentrum ............. Tel. (0821) 598-2030 86135 Augsburg .................................. Fax. (0821) 598-2028
participants (3)
-
Daniel Gryniewicz -
Rolf Anders -
William Allen Simpson