Yeah, sadly libntirpc (and ntirpc implementations generally) actually aren't able to establish connections as clients. I haven't looked at the GSS auth code, but will do so. I don't think there's any reason it would "require" CLIENT (it's just more bytes that need to be on the wire). But my experience with the CLIENT functions from the V3 proxy was that I couldn't even get it to establish connections. (see FSAL_PROXY_V3/rpc.c). Would you mind pasting your in-progress code? (if you got it to work, that would be *fantastic*) On Thu, Sep 17, 2020 at 10:21 AM Jo Seaton <jo(a)petagene.com&gt; wrote: > > Hi all, > > I've been working on re-adding GSS/Kerberos authentication support to the PROXY (V4) FSAL, with a mind to eventually also adding Kerberos delegation support for completeness. I've been having some issues I was hoping to get some feedback on. > > The first issue I've been having is to do with the GSS code in libntirpc. > I'm calling authgss_ncreate_default with a valid CLIENT * and service name, and hopefully a reasonable struct rpc_gss_sec. This fails, because the call to gss_verify_mic inside authgss_refresh fails. This appears to be because authgss_verify fills a relevant buffer (gd->gc_wire_verf) with <empty>, which originally comes from cc_verf in the clnt code (via the start of authgss_validate). Specifically I'm looking at clnt_req's cc_verf which gets used for AUTH_VALIDATE in clnt_generic.c, and always seems to have the same "_null_auth" value - which seems surprising to me! If anyone can give me some insight into what exactly cc_verf is supposed to contain that might help me fix it. Working around it by ignoring the result of gss_verify_mic does seem to work OK. > > The second issue is to do with the structure of the PROXY FSAL. > It appears that it largely handles requests "manually", calling the relevant xdr_* functions, and reading/writing to sockets itself. The GSS auth code on the other hand, seems to require use of CLIENT *, which in my understanding means handing responsibility for the socket to that CLIENT *. These two approaches appear incompatible to me. I've made some reasonable progress rewriting PROXY with clnt_req_* functions, similar to nfs_rpc_callback.c, but if anyone has any feedback on a) why the original approach (ffilz suggests those functions didn't used to be threadsafe?) b) the most sensible thing to do now, it would be very much appreciated. > > Anyway any feedback is very welcome, I'm very new to both Ganesha and GSS/Kerberos. > > Many thanks, > > Jo > > _______________________________________________ > Devel mailing list -- devel(a)lists.nfs-ganesha.org > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org

I could not figure it out after multiple days of sadness. I had hoped to

AFAICT, there is no way to use clnt_call or rpc_call from within a

I *think* it's because of something the main NFSD does in setting up the

Hi both, thanks for the responses, Solomon - I had a hell of a time initially with the client functions that turned out to be because some libntirpc initialisation hadn't been done yet. I have an unpleasant hack to delay PROXY exports until that's happened that solves the issue for me - maybe you had the same? Concerning use of client functions, here's my initialisation (rpc_sock has already been opened): rpc->clnt = clnt_vc_ncreatef(rpc->rpc_sock, &raddr, > proxyv4_exp->info.srv_prognum, (rpcvers_t) 4, > proxyv4_exp->info.srv_sendsize, proxyv4_exp->info.srv_recvsize, 0); > rpc->auth = authgss_ncreate_default(rpc->clnt, &quot;nfs(a)nfs-storage.blah.net&quot;, > &rpcsec_gss_data); > and here's the RPC calls: > struct clnt_req *cc; > enum clnt_stat stat; > cc = gsh_malloc(sizeof(*cc)); > clnt_req_fill(cc, clnt, au, CB_COMPOUND, > (xdrproc_t) xdr_COMPOUND4args, (caddr_t)args, > (xdrproc_t) xdr_COMPOUND4res, (caddr_t)res); > > stat = clnt_req_setup(cc, tout); > if (stat == RPC_SUCCESS) { > cc->cc_refreshes = 1; > > > > > stat = CLNT_CALL_WAIT(cc); > } > clnt_req_release(cc); > This appears to roughly work - I have some issues but I'm not done yet. My auth_gss.c hack is just this: > (in authgss_verify:) > maj_stat = > gss_verify_mic(&min_stat, gd->ctx, &signbuf, &checksum, &qop_state); > if (maj_stat != GSS_S_COMPLETE || qop_state != gd->sec.qop) { > gss_log_status("gss_verify_mic", maj_stat, min_stat); > if (maj_stat == GSS_S_CONTEXT_EXPIRED) { > gd->established = false; > authgss_destroy_context(auth); > } > /* return (false); */ // here > > (in authgss_refresh:) > maj_stat = > > gss_verify_mic(&min_stat, gd->ctx, &bufin, &bufout, > > &qop_state); > > maj_stat = GSS_S_COMPLETE; // and here > I'm happy to post my entire code when I get back, but I'm away for the next week (sorry!). Solomon - does this mean you'd be happy with an approach like the above if it works? I'm not opposed to using the existing PROXY approach, but writing an equivalent of the libntirpc GSS code just for PROXY sounds like a good way to make bugs. Many thanks, Jo On Thu, Sep 17, 2020 at 8:11 PM Matt Benjamin <mbenjami(a)redhat.com&gt; wrote: > Hi, > > It's really not the case (at least, unless we recently broke > something) that libntirpc cannot act as clients. There might be some > specific issue with GSS, though. > > Matt > > On Thu, Sep 17, 2020 at 1:47 PM Solomon Boulos via Devel > <devel(a)lists.nfs-ganesha.org&gt; wrote: > > > > Yeah, sadly libntirpc (and ntirpc implementations generally) actually > aren't able to establish connections as clients. I haven't looked at the > GSS auth code, but will do so. I don't think there's any reason it would > "require" CLIENT (it's just more bytes that need to be on the wire). > > > > But my experience with the CLIENT functions from the V3 proxy was that > I couldn't even get it to establish connections. (see FSAL_PROXY_V3/rpc.c). > Would you mind pasting your in-progress code? (if you got it to work, that > would be *fantastic*) > > > > On Thu, Sep 17, 2020 at 10:21 AM Jo Seaton <jo(a)petagene.com&gt; wrote: > >> > >> Hi all, > >> > >> I've been working on re-adding GSS/Kerberos authentication support to > the PROXY (V4) FSAL, with a mind to eventually also adding Kerberos > delegation support for completeness. I've been having some issues I was > hoping to get some feedback on. > >> > >> The first issue I've been having is to do with the GSS code in > libntirpc. > >> I'm calling authgss_ncreate_default with a valid CLIENT * and service > name, and hopefully a reasonable struct rpc_gss_sec. This fails, because > the call to gss_verify_mic inside authgss_refresh fails. This appears to be > because authgss_verify fills a relevant buffer (gd->gc_wire_verf) with > <empty>, which originally comes from cc_verf in the clnt code (via the > start of authgss_validate). Specifically I'm looking at clnt_req's cc_verf > which gets used for AUTH_VALIDATE in clnt_generic.c, and always seems to > have the same "_null_auth" value - which seems surprising to me! If anyone > can give me some insight into what exactly cc_verf is supposed to contain > that might help me fix it. Working around it by ignoring the result of > gss_verify_mic does seem to work OK. > >> > >> The second issue is to do with the structure of the PROXY FSAL. > >> It appears that it largely handles requests "manually", calling the > relevant xdr_* functions, and reading/writing to sockets itself. The GSS > auth code on the other hand, seems to require use of CLIENT *, which in my > understanding means handing responsibility for the socket to that CLIENT *. > These two approaches appear incompatible to me. I've made some reasonable > progress rewriting PROXY with clnt_req_* functions, similar to > nfs_rpc_callback.c, but if anyone has any feedback on a) why the original > approach (ffilz suggests those functions didn't used to be threadsafe?) b) > the most sensible thing to do now, it would be very much appreciated. > >> > >> Anyway any feedback is very welcome, I'm very new to both Ganesha and > GSS/Kerberos. > >> > >> Many thanks, > >> > >> Jo > >> > >> _______________________________________________ > >> Devel mailing list -- devel(a)lists.nfs-ganesha.org > >> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > > > > _______________________________________________ > > Devel mailing list -- devel(a)lists.nfs-ganesha.org > > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > Ann Arbor, Michigan 48103 > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 <(734)%20821-5101> > fax. 734-769-8938 <(734)%20769-8938> > cel. 734-216-5309 <(734)%20216-5309> > >

I could not figure it out after multiple days of sadness. I had hoped to just use rpc_call!

AFAICT, there is no way to use clnt_call or rpc_call from within a Ganesha FSAL (and luckily, I looked at the V4 proxy and said "Oh, they just rolled their own, too").

I *think* it's because of something the main NFSD does in setting up the listening code (via svc_create or similar). I tried wading through the libntirpc code to understand if there's also something particularly different about that.

Yeah, sadly libntirpc (and ntirpc implementations generally) actually aren't able to establish connections as clients. I haven't looked at the GSS auth code, but will do so. I don't think there's any reason it would "require" CLIENT (it's just more bytes that need to be on the wire). But my experience with the CLIENT functions from the V3 proxy was that I couldn't even get it to establish connections. (see FSAL_PROXY_V3/rpc.c). Would you mind pasting your in-progress code? (if you got it to work, that would be *fantastic*) On Thu, Sep 17, 2020 at 10:21 AM Jo Seaton <jo(a)petagene.com <mailto:jo@petagene.com> > wrote: > > Hi all, > > I've been working on re-adding GSS/Kerberos authentication support to the PROXY (V4) FSAL, with a mind to eventually also adding Kerberos delegation support for completeness. I've been having some issues I was hoping to get some feedback on. > > The first issue I've been having is to do with the GSS code in libntirpc. > I'm calling authgss_ncreate_default with a valid CLIENT * and service name, and hopefully a reasonable struct rpc_gss_sec. This fails, because the call to gss_verify_mic inside authgss_refresh fails. This appears to be because authgss_verify fills a relevant buffer (gd->gc_wire_verf) with <empty>, which originally comes from cc_verf in the clnt code (via the start of authgss_validate). Specifically I'm looking at clnt_req's cc_verf which gets used for AUTH_VALIDATE in clnt_generic.c, and always seems to have the same "_null_auth" value - which seems surprising to me! If anyone can give me some insight into what exactly cc_verf is supposed to contain that might help me fix it. Working around it by ignoring the result of gss_verify_mic does seem to work OK. > > The second issue is to do with the structure of the PROXY FSAL. > It appears that it largely handles requests "manually", calling the relevant xdr_* functions, and reading/writing to sockets itself. The GSS auth code on the other hand, seems to require use of CLIENT *, which in my understanding means handing responsibility for the socket to that CLIENT *. These two approaches appear incompatible to me. I've made some reasonable progress rewriting PROXY with clnt_req_* functions, similar to nfs_rpc_callback.c, but if anyone has any feedback on a) why the original approach (ffilz suggests those functions didn't used to be threadsafe?) b) the most sensible thing to do now, it would be very much appreciated. > > Anyway any feedback is very welcome, I'm very new to both Ganesha and GSS/Kerberos. > > Many thanks, > > Jo > > _______________________________________________ > Devel mailing list -- devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org> _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org>

Hi All, Apologies for the delay, I am back. Please find (very experimental!) code here: https://github.com/jseaton/nfs-ganesha/commit/5ad302b815acc4b99bd111753c9... https://github.com/jseaton/ntirpc/commit/c7a3c5b87435ddca4a62220e13b535bd... note: copies and such will hang sporadically for unclear reasons - I suspect I need to remove (or add...) some locking, but I might be wrong. Also assumes a krb5-supporting remote. @Solomon - sorry, I can't confirm for other protocols, but feel free to have a go! @Frank - I'd be interested in that if we can work out what the issue is. I'm only fairly sure nfs_Init_svc is the problem, I don't understand the internals of that yet. I'm on #ganesha (as fiiiii) UK hours if anyone wants to chat about this. Many thanks, Jo On Fri, Sep 18, 2020 at 6:10 PM Frank Filz <ffilzlnx(a)mindspring.com&gt; wrote: > Interesting discussion. If you’re able to join us in IRC, we hang out on > Freenode #ganesha > > > > It might make sense to break up nfs_Init_svc so whatever is needed by the > PROXY FSALs can be set up before loading exports but prevents the server > from opening up to connections before the exports are processed. > > > > Frank > > > > *From:* Jo Seaton [mailto:jo@petagene.com] > *Sent:* Friday, September 18, 2020 9:56 AM > *To:* Solomon Boulos <boulos(a)google.com&gt; > *Cc:* Matt Benjamin <mbenjami(a)redhat.com>; Ganesha-devel < > devel(a)lists.nfs-ganesha.org&gt; > *Subject:* [NFS-Ganesha-Devel] Re: Seeking assistance with Ganesha PROXY > FSAL / libntirpc issues > > > > Aha! Yes, I've been reading the V4 code so I totally missed this, but > this is exactly the same error! It went away as soon as I managed to delay > the clnt stuff till after nfs_Init_svc. > > > > The actual method I use is pretty horrible - I make the FSAL export init > fail, and then later (inside nfs_rpc_cb_init_ccache, yuck!) trigger > re-parsing the relevant bit of config file, so reloading the export after > all the init I need has happened. Obviously at the very least I'd like to > contain this nastiness to the FSAL. Any prettier ideas very welcome because > just urgh. > > > > Anyway, I'm happy to share code + try stuff out for you, but that will > have to wait a week! > > > > Thanks for getting back to me about this, it's nice to not be alone in > this code :) > > > > Jo > > > > On Fri, Sep 18, 2020 at 5:29 PM Solomon Boulos <boulos(a)google.com&gt; wrote: > > Absolutely. I didn't want to write it :). > > > > From the review on > https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/487439 > > > > > I could not figure it out after multiple days of sadness. I had hoped > to just use rpc_call! > > > > AFAICT, there is no way to use clnt_call or rpc_call from within a > Ganesha FSAL (and luckily, I looked at the V4 proxy and said "Oh, they just > rolled their own, too"). > > > I *think* it's because of something the main NFSD does in setting up > the listening code (via svc_create or similar). I tried wading through the > libntirpc code to understand if there's also something particularly > different about that. > > > > I'm surprised your rpc->clnt = clnt_vc_ncreatef works around it. Can you > try the same for NLM / PORTMAP / whatever? I can't find the simple example > I went back and forth with folks on that code, but it was basically "No, I > can't even issue a MNT_NULL with clnt_call / rpc_call". My comment on > proxyv3_call though describes the error: > > > > /* > * NOTE(boulos): proxyv3_call is basically rpc_call redone by hand, > because > * ganesha's NFSD hijacks the RPC setup to the point where we can't issue > our > * own NFS-related rpcs as a simple client via clnt_ncreate (internally, > * svc_exprt_lookup explodes saying "fd %d max_connections 0 exceeded"). > */ > > > > Are you saying you did something else either for the FSAL loading / > similar to wait until later such that clnt_ncreate will succeed? (so that > max_connections can be >0). Either way, I intentionally left the proxy_v3 > code ready though to rip out and look just like clnt_call / rpc_call, so > that it can all be replaced if we can fix this :). > > > > > > On Fri, Sep 18, 2020 at 9:14 AM Jo Seaton <jo(a)petagene.com&gt; wrote: > > Hi both, thanks for the responses, > > > > Solomon - I had a hell of a time initially with the client functions that > turned out to be because some libntirpc initialisation hadn't been done > yet. I have an unpleasant hack to delay PROXY exports until that's happened > that solves the issue for me - maybe you had the same? > > > > Concerning use of client functions, here's my initialisation (rpc_sock > has already been opened): > > > > rpc->clnt = clnt_vc_ncreatef(rpc->rpc_sock, &raddr, > proxyv4_exp->info.srv_prognum, (rpcvers_t) 4, > proxyv4_exp->info.srv_sendsize, proxyv4_exp->info.srv_recvsize, 0); > rpc->auth = authgss_ncreate_default(rpc->clnt, &quot;nfs(a)nfs-storage.blah.net&quot;, > &rpcsec_gss_data); > > > > and here's the RPC calls: > > struct clnt_req *cc; > enum clnt_stat stat; > cc = gsh_malloc(sizeof(*cc)); > clnt_req_fill(cc, clnt, au, CB_COMPOUND, > (xdrproc_t) xdr_COMPOUND4args, (caddr_t)args, > (xdrproc_t) xdr_COMPOUND4res, (caddr_t)res); > > stat = clnt_req_setup(cc, tout); > if (stat == RPC_SUCCESS) { > cc->cc_refreshes = 1; > > > > > stat = CLNT_CALL_WAIT(cc); > } > clnt_req_release(cc); > > > > This appears to roughly work - I have some issues but I'm not done yet. > > > > My auth_gss.c hack is just this: > > (in authgss_verify:) > > maj_stat = > gss_verify_mic(&min_stat, gd->ctx, &signbuf, &checksum, &qop_state); > > if (maj_stat != GSS_S_COMPLETE || qop_state != gd->sec.qop) { > gss_log_status("gss_verify_mic", maj_stat, min_stat); > if (maj_stat == GSS_S_CONTEXT_EXPIRED) { > gd->established = false; > authgss_destroy_context(auth); > } > /* return (false); */ // here > > > > (in authgss_refresh:) > > maj_stat = > > gss_verify_mic(&min_stat, gd->ctx, &bufin, &bufout, > > &qop_state); > > maj_stat = GSS_S_COMPLETE; // and here > > > > I'm happy to post my entire code when I get back, but I'm away for the > next week (sorry!). > > > > Solomon - does this mean you'd be happy with an approach like the above > if it works? I'm not opposed to using the existing PROXY approach, but > writing an equivalent of the libntirpc GSS code just for PROXY sounds like > a good way to make bugs. > > > > Many thanks, > > Jo > > > > On Thu, Sep 17, 2020 at 8:11 PM Matt Benjamin <mbenjami(a)redhat.com&gt; > wrote: > > Hi, > > It's really not the case (at least, unless we recently broke > something) that libntirpc cannot act as clients. There might be some > specific issue with GSS, though. > > Matt > > On Thu, Sep 17, 2020 at 1:47 PM Solomon Boulos via Devel > <devel(a)lists.nfs-ganesha.org&gt; wrote: > > > > Yeah, sadly libntirpc (and ntirpc implementations generally) actually > aren't able to establish connections as clients. I haven't looked at the > GSS auth code, but will do so. I don't think there's any reason it would > "require" CLIENT (it's just more bytes that need to be on the wire). > > > > But my experience with the CLIENT functions from the V3 proxy was that > I couldn't even get it to establish connections. (see FSAL_PROXY_V3/rpc.c). > Would you mind pasting your in-progress code? (if you got it to work, that > would be *fantastic*) > > > > On Thu, Sep 17, 2020 at 10:21 AM Jo Seaton <jo(a)petagene.com&gt; wrote: > >> > >> Hi all, > >> > >> I've been working on re-adding GSS/Kerberos authentication support to > the PROXY (V4) FSAL, with a mind to eventually also adding Kerberos > delegation support for completeness. I've been having some issues I was > hoping to get some feedback on. > >> > >> The first issue I've been having is to do with the GSS code in > libntirpc. > >> I'm calling authgss_ncreate_default with a valid CLIENT * and service > name, and hopefully a reasonable struct rpc_gss_sec. This fails, because > the call to gss_verify_mic inside authgss_refresh fails. This appears to be > because authgss_verify fills a relevant buffer (gd->gc_wire_verf) with > <empty>, which originally comes from cc_verf in the clnt code (via the > start of authgss_validate). Specifically I'm looking at clnt_req's cc_verf > which gets used for AUTH_VALIDATE in clnt_generic.c, and always seems to > have the same "_null_auth" value - which seems surprising to me! If anyone > can give me some insight into what exactly cc_verf is supposed to contain > that might help me fix it. Working around it by ignoring the result of > gss_verify_mic does seem to work OK. > >> > >> The second issue is to do with the structure of the PROXY FSAL. > >> It appears that it largely handles requests "manually", calling the > relevant xdr_* functions, and reading/writing to sockets itself. The GSS > auth code on the other hand, seems to require use of CLIENT *, which in my > understanding means handing responsibility for the socket to that CLIENT *. > These two approaches appear incompatible to me. I've made some reasonable > progress rewriting PROXY with clnt_req_* functions, similar to > nfs_rpc_callback.c, but if anyone has any feedback on a) why the original > approach (ffilz suggests those functions didn't used to be threadsafe?) b) > the most sensible thing to do now, it would be very much appreciated. > >> > >> Anyway any feedback is very welcome, I'm very new to both Ganesha and > GSS/Kerberos. > >> > >> Many thanks, > >> > >> Jo > >> > >> _______________________________________________ > >> Devel mailing list -- devel(a)lists.nfs-ganesha.org > >> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > > > > _______________________________________________ > > Devel mailing list -- devel(a)lists.nfs-ganesha.org > > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > > > > -- > > Matt Benjamin > Red Hat, Inc. > 315 West Huron Street, Suite 140A > <https://www.google.com/maps/search/315+West+Huron+Street,+Suite+140A+Ann+... > Ann Arbor, Michigan 48103 > <https://www.google.com/maps/search/315+West+Huron+Street,+Suite+140A+Ann+... > > http://www.redhat.com/en/technologies/storage > > tel. 734-821-5101 <(734)%20821-5101> > fax. 734-769-8938 <(734)%20769-8938> > cel. 734-216-5309 <(734)%20216-5309> > >