Issues seen with krb5i and krb5p mounts
Hi all, There are a few clients eg- Ubuntu 18.04.3 (4.15.0-55-generic) and RH7.8 (3.10.0-1127.el7.x86_64) for which we have observed... simple command like 'dd' either hangs or returns EIO. This is happening only on krb5i and krb5p mounts. It seems to happen for file sizes eg- 100MB and larger mostly. But sometimes even a 30 MB file sees failures. A client eg- RH7.6 (3.10.0-957.el7.x86_64) does not seem to hit this issue...so might be with more recent kernels? We fixed the issue with check-in https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/490802 The idea was to let clients know that Ganesha denied the request VS just dropping the request. This fix did seem to help and hangs/errors stopped completely... but for larger file sizes eg- 1000MB we started seeing "Permission Denied" errors. This was different than the EIO errors seen earlier. Reason could be we are now sending an "AUTH DENIED" error so clients translate it to this new error. We tried to add more logging into Ganesha and observe that these particular clients seem to send a lot of requests together. When we process same, the sequence no. is pretty much out or order and we drop the requests outside the sequence window, as per the RFC 2203 Section 7.2.1. The sequence window that we have is 32. Testing these clients with kNFS does not hit the issue...The kNFS sequence window seems to be larger and is 128. So, tried to increase the sequence window as well to 128 for ganesha. That does not seem to help fix the issue. We also have below additional 'seqmask' check and many of the requests went into that category as well and were dropped. "libntirpc/src/svc_auth_gss.c": } else if (offset >= gd->win || (gd->seqmask & (1 << offset))) { *no_dispatch = true; goto gd_free; } Also observed that now these clients sent many requests above the 128 window...which we would again drop. Wondering what is the proper way to fix this and any idea on what these clients are doing different. Thanks and regards, Trishali. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
We are also seeing the same issue. Thought that after this client patch (it went into 4.20) https://www.spinics.net/lists/linux-nfs/msg69633.html Client would have started to honor the RPCSEC_GSS window but we are hitting this issue even on kernel versions 5.xx The only solution which mitigates it somewhat is to limit the in-flight RPCs that the client can have by tweaking tcp_max_slot_table_entries (for example to 16 or 32) and along with that change the GSS window size in Ganesha to 256. Max_slot_table_entries guarantees the in-flight rpcs, but it doesn't guarantee the sequence limits. For example, if some of the requests take unusually long time than the16 in-flight RPCs can have the sequence numbers which might not fit into 256.
Raised a pull request for increasing window size: https://github.com/nfs-ganesha/ntirpc/pull/215
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i, krb5p mounts where as it goes through during krb5 mounts. Environment: Distro: CentOs 8.2.2004 Kernel version: 4.18.0-193 Ganesha version: V3-Stable krb version: krb5 Securtiy Type: krb5p FSAL: VFS Steps to reproduce: 1) Compiled and installed nfs-ganesha V3-stable branch 2) Enabled VFS – libfsalvfs.so and exported ext3 file system 3) Enabled krb5p security in the conf file 4) Triggered “# iozone -a “ from nfs4 mount After running for sometime – it got hung. Is there any JIRA tracking this issue? any suggested workarounds?
Sorry, didn't see this until today. Malahal, does this sound like it might be the issue with the GSS window? Frank
-----Original Message----- From: chakragithub@gmail.com [mailto:chakragithub@gmail.com] Sent: Thursday, October 8, 2020 1:42 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i, krb5p mounts where as it goes through during krb5 mounts.
Environment: Distro: CentOs 8.2.2004 Kernel version: 4.18.0-193 Ganesha version: V3-Stable krb version: krb5
Securtiy Type: krb5p FSAL: VFS
Steps to reproduce: 1) Compiled and installed nfs-ganesha V3-stable branch 2) Enabled VFS – libfsalvfs.so and exported ext3 file system 3) Enabled krb5p security in the conf file 4) Triggered “# iozone -a “ from nfs4 mount After running for sometime – it got hung.
Is there any JIRA tracking this issue? any suggested workarounds? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Looks like it based on the issue with krb5i and krb5p. On Fri, Oct 9, 2020 at 7:52 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Sorry, didn't see this until today.
Malahal, does this sound like it might be the issue with the GSS window?
Frank
-----Original Message----- From: chakragithub@gmail.com [mailto:chakragithub@gmail.com] Sent: Thursday, October 8, 2020 1:42 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i, krb5p mounts where as it goes through during krb5 mounts.
Environment: Distro: CentOs 8.2.2004 Kernel version: 4.18.0-193 Ganesha version: V3-Stable krb version: krb5
Securtiy Type: krb5p FSAL: VFS
Steps to reproduce: 1) Compiled and installed nfs-ganesha V3-stable branch 2) Enabled VFS – libfsalvfs.so and exported ext3 file system 3) Enabled krb5p security in the conf file 4) Triggered “# iozone -a “ from nfs4 mount After running for sometime – it got hung.
Is there any JIRA tracking this issue? any suggested workarounds? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Do you want to share your results with the fix/workaround? Thanks Frank From: Malahal Naineni [mailto:malahal@gmail.com] Sent: Friday, October 9, 2020 10:24 AM To: Frank Filz <ffilzlnx@mindspring.com> Cc: chakragithub@gmail.com; nfs-ganesha <devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts Looks like it based on the issue with krb5i and krb5p. On Fri, Oct 9, 2020 at 7:52 PM Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > wrote: Sorry, didn't see this until today. Malahal, does this sound like it might be the issue with the GSS window? Frank
-----Original Message----- From: chakragithub@gmail.com <mailto:chakragithub@gmail.com> [mailto:chakragithub@gmail.com <mailto:chakragithub@gmail.com> ] Sent: Thursday, October 8, 2020 1:42 AM To: devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i, krb5p mounts where as it goes through during krb5 mounts.
Environment: Distro: CentOs 8.2.2004 Kernel version: 4.18.0-193 Ganesha version: V3-Stable krb version: krb5
Securtiy Type: krb5p FSAL: VFS
Steps to reproduce: 1) Compiled and installed nfs-ganesha V3-stable branch 2) Enabled VFS – libfsalvfs.so and exported ext3 file system 3) Enabled krb5p security in the conf file 4) Triggered “# iozone -a “ from nfs4 mount After running for sometime – it got hung.
Is there any JIRA tracking this issue? any suggested workarounds? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org>
Frank, Malahal Thanks for responding....i tried testing the same use case with older stable versions. Till v2.5 stable ganesha, it worked fine. But from v2.6Stable - it was not hang but became very slow after some point of time and eventually it exited. In the logs i see checksum failures in rpc process request. **************************** 09/10/2020 T08:22:17.567082-0700 17095[none] [svc_913] 769 :nfs_rpc_process_request :SVCAUTH_CHECKSUM failed for Program 100003, Version 4, Function 1, xid=3473413753, SVCXPRT=0x7f8924101d00, fd=26 09/10/2020 T08:22:17.660648-0700 17095[none] [svc_917] 769 :nfs_rpc_process_request :SVCAUTH_CHECKSUM failed for Program 100003, Version 4, Function 1, xid=3540522617, SVCXPRT=0x7f8924101d00, fd=26 **************************** Let me know if you need any other info -
Yes, it works fine with Ganesha2.5 and started having issues with Ganesha2.7 (probably started in ganesha2.6 but we haven't used it). The fix is in libntirpc, here. I don't see the main repo updated with these patches (my repo might be out of date), so I am supplying a link to our repo: https://github.com/ganltc/ntirpc.git, the following 2 commits: fd787f4e9 Increase the sequence window for krb5i and krb5p b890bfb1c Make RPCSEC_GSS window size function of ioq_thrd_max Hope this helps! On Fri, Oct 9, 2020 at 11:18 PM Chakra Divi <chakragithub@gmail.com> wrote:
Frank, Malahal
Thanks for responding....i tried testing the same use case with older stable versions.
Till v2.5 stable ganesha, it worked fine. But from v2.6Stable - it was not hang but became very slow after some point of time and eventually it exited. In the logs i see checksum failures in rpc process request.
**************************** 09/10/2020 T08:22:17.567082-0700 17095[none] [svc_913] 769 :nfs_rpc_process_request :SVCAUTH_CHECKSUM failed for Program 100003, Version 4, Function 1, xid=3473413753, SVCXPRT=0x7f8924101d00, fd=26 09/10/2020 T08:22:17.660648-0700 17095[none] [svc_917] 769 :nfs_rpc_process_request :SVCAUTH_CHECKSUM failed for Program 100003, Version 4, Function 1, xid=3540522617, SVCXPRT=0x7f8924101d00, fd=26 **************************** Let me know if you need any other info - _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Thanks Malahal, ntirpc window size fixes are working. But we are seeing a performance degradation on v3 when compared with v2.5. iozone taking 50% more time to complete on the same setup with v3 version. IO operations with bigger block sizes - seems like having the impact. With 3.3 – after fix real 36m37.985s user 0m1.974s sys 1m33.561s v2.5 – working version real 23m21.187s user 0m1.797s sys 1m16.537s Please let me if you need anything more from my setup like traces...to get hold of the issue
Assuming that these numbers are with KRB5I and KRB5P, then the window size is a multiple of the number of threads. The old code might be using a fixed number of a bigger window size. How many rpc max threads are you using? The default is 200, I think and it should have a big window size though. Do you see any perf issues with plain KRB5 or with AUTH_SYS between 2.5 and 3.3. Generally speaking 3.3 should have similar or better perf compared to ganesha2.5 with AUTH_SYS. Regards, Malahal. On Mon, Oct 12, 2020 at 2:33 PM Chakra Divi <chakragithub@gmail.com> wrote:
Thanks Malahal, ntirpc window size fixes are working. But we are seeing a performance degradation on v3 when compared with v2.5. iozone taking 50% more time to complete on the same setup with v3 version. IO operations with bigger block sizes - seems like having the impact.
With 3.3 – after fix real 36m37.985s user 0m1.974s sys 1m33.561s
v2.5 – working version real 23m21.187s user 0m1.797s sys 1m16.537s
Please let me if you need anything more from my setup like traces...to get hold of the issue _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Thanks Malahal, i have seen similar performance between 2.5 and 3.3 for AUTH_SYS and krb5 mounts. iozone tool took similar time on both versions. v3.3 - sys - 9m15.870s v2.5 – sys - 9m20.688s v3.3 – krb5 - 9m32.907s v2.5 – krb5 - 9m18.325s I have not the number of threads - so it is default : thrd_max 200 gss_seq_win 800 - window size is based on your fix. Do you think tuning window size would fix this issue ?
with krb5i - i'm not seeing not much degradation - when compared with v2.5. v2.5 - krb5i - 14m6.888s v3 - krb5i - 18m55.072s with krb5p - v3 took more than 50% time than v2.5
Ok, I'm beginning to fear that using gss_iov for krb5p is causing problems, try this quick ntirpc patch and see if it makes a difference: diff --git a/src/authgss_prot.c b/src/authgss_prot.c index b01578f..409da6e 100644 --- a/src/authgss_prot.c +++ b/src/authgss_prot.c @@ -267,7 +267,7 @@ xdr_rpc_gss_wrap(XDR *xdrs, xdrproc_t xdr_func, void *xdr_ptr, * If it's privacy, and NEWBUF is supported (because xdrs is a vector) * then NEWBUF will have allocated the new buffer. */ - vector = (svc == RPCSEC_GSS_SVC_INTEGRITY) || XDR_NEWBUF(xdrs); + vector = (svc == RPCSEC_GSS_SVC_INTEGRITY); /* Marshal rpc_gss_data_t (sequence number + arguments). * If it's a vector, the response has been marshalled into a new
-----Original Message----- From: Chakra Divi [mailto:chakragithub@gmail.com] Sent: Tuesday, October 13, 2020 9:26 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
with krb5i - i'm not seeing not much degradation - when compared with v2.5.
v2.5 - krb5i - 14m6.888s v3 - krb5i - 18m55.072s
with krb5p - v3 took more than 50% time than v2.5 _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Frank, Thanks for the response, tried your patch disabling usage of gss_iov for krb5p but no luck - iozone tool stuck at 4k block size iops...i could see the following messages in ganesha logs. Is there any other place that needs to be changed to disable gss_iov ? ********* 14/10/2020 T02:05:43.505432-0700 6554[::ffff:10.10.108.68] [svc_35] 0 :rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set dead) 14/10/2020 T02:05:43.505508-0700 6554[::ffff:10.10.108.68] [svc_100] 0 :rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set dead) 14/10/2020 T02:05:43.505577-0700 6554[::ffff:10.10.108.68] [svc_87] 0 :rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set dead) 14/10/2020 T02:05:43.505602-0700 6554[::ffff:10.10.108.68] [svc_64] 0 :rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set dead) ********* Regards, Chakra On Tue, Oct 13, 2020 at 10:15 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Ok, I'm beginning to fear that using gss_iov for krb5p is causing problems, try this quick ntirpc patch and see if it makes a difference:
diff --git a/src/authgss_prot.c b/src/authgss_prot.c index b01578f..409da6e 100644 --- a/src/authgss_prot.c +++ b/src/authgss_prot.c @@ -267,7 +267,7 @@ xdr_rpc_gss_wrap(XDR *xdrs, xdrproc_t xdr_func, void *xdr_ptr, * If it's privacy, and NEWBUF is supported (because xdrs is a vector) * then NEWBUF will have allocated the new buffer. */ - vector = (svc == RPCSEC_GSS_SVC_INTEGRITY) || XDR_NEWBUF(xdrs); + vector = (svc == RPCSEC_GSS_SVC_INTEGRITY);
/* Marshal rpc_gss_data_t (sequence number + arguments). * If it's a vector, the response has been marshalled into a new
-----Original Message----- From: Chakra Divi [mailto:chakragithub@gmail.com] Sent: Tuesday, October 13, 2020 9:26 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
with krb5i - i'm not seeing not much degradation - when compared with v2.5.
v2.5 - krb5i - 14m6.888s v3 - krb5i - 18m55.072s
with krb5p - v3 took more than 50% time than v2.5 _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Frank, i have added your patch on top of Malahal Window size changes - and observed hang issue. hope this is right ? Let me know if you need any other info - to root cause the perf issue Chakra
Frank, Any other info that can help me to proceed on this perf issue ? Chakra
-----Original Message----- From: Chakra Divi [mailto:chakragithub@gmail.com] Sent: Sunday, October 18, 2020 9:35 PM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Frank,
Any other info that can help me to proceed on this perf issue ?
Unfortunately I'm not sure. I've done some work with Kerberos but it hasn't been much of a priority. Maybe someone else can suggest some tracing you could enable or add to try and find where it's hanging up. Frank
Frank, Thanks for responding. Do you think disabling gss_iov for krb5p will solve this ? If yes, i will proceed with debugging on top of your patch Should i raise a bug for this issue ? will that be helpful ? Chakra
participants (6)
-
Ashish Sangwan -
Chakra Divi -
chakragithub＠gmail.com
-
Frank Filz -
Malahal Naineni -
Trishali Nayar