8:34 a.m.
Hi all,
There are a few clients eg- Ubuntu 18.04.3 (4.15.0-55-generic) and RH7.8
(3.10.0-1127.el7.x86_64) for which we have observed... simple command like
'dd' either hangs or returns EIO. This is happening only on krb5i and krb5p
mounts. It seems to happen for file sizes eg- 100MB and larger mostly. But
sometimes even a 30 MB file sees failures.
A client eg- RH7.6 (3.10.0-957.el7.x86_64) does not seem to hit this
issue...so might be with more recent kernels?
We fixed the issue with check-in
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/490802 The idea was to
let clients know that Ganesha denied the request VS just dropping the
request.
This fix did seem to help and hangs/errors stopped completely... but for
larger file sizes eg- 1000MB we started seeing "Permission Denied" errors.
This was different than the EIO errors seen earlier. Reason could be we are
now sending an "AUTH DENIED" error so clients translate it to this new
error.
We tried to add more logging into Ganesha and observe that these particular
clients seem to send a lot of requests together. When we process same, the
sequence no. is pretty much out or order and we drop the requests outside
the sequence window, as per the RFC 2203 Section 7.2.1. The sequence window
that we have is 32.
Testing these clients with kNFS does not hit the issue...The kNFS sequence
window seems to be larger and is 128.
So, tried to increase the sequence window as well to 128 for ganesha. That
does not seem to help fix the issue.
We also have below additional 'seqmask' check and many of the requests went
into that category as well and were dropped.
"libntirpc/src/svc_auth_gss.c":
} else if (offset >= gd->win || (gd->seqmask & (1 << offset)))
{
*no_dispatch = true;
goto gd_free;
}
Also observed that now these clients sent many requests above the 128
window...which we would again drop.
Wondering what is the proper way to fix this and any idea on what these
clients are doing different.
Thanks and regards,
Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Trishali Nayar
IBM Systems
ETZ, Pune.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
12:17 p.m.
We are also seeing the same issue. Thought that after this client patch (it went into
4.20) https://www.spinics.net/lists/linux-nfs/msg69633.html
Client would have started to honor the RPCSEC_GSS window but we are hitting this issue
even on kernel versions 5.xx
The only solution which mitigates it somewhat is to limit the in-flight RPCs that the
client can have by tweaking tcp_max_slot_table_entries (for example to 16 or 32) and along
with that change the GSS window size in Ganesha to 256. Max_slot_table_entries guarantees
the in-flight rpcs, but it doesn't guarantee the sequence limits. For example, if some
of the requests take unusually long time than the16 in-flight RPCs can have the sequence
numbers which might not fit into 256.
1:31 p.m.
Raised a pull request for increasing window size:
https://github.com/nfs-ganesha/ntirpc/pull/215
3:41 a.m.
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i, krb5p
mounts where as it goes through during krb5 mounts.
Environment:
Distro: CentOs 8.2.2004
Kernel version: 4.18.0-193
Ganesha version: V3-Stable
krb version: krb5
Securtiy Type: krb5p
FSAL: VFS
Steps to reproduce:
1) Compiled and installed nfs-ganesha V3-stable branch
2) Enabled VFS – libfsalvfs.so and exported ext3 file system
3) Enabled krb5p security in the conf file
4) Triggered “# iozone -a “ from nfs4 mount
After running for sometime – it got hung.
Is there any JIRA tracking this issue? any suggested workarounds?
9:22 a.m.
Sorry, didn't see this until today.
Malahal, does this sound like it might be the issue with the GSS window?
Frank
-----Original Message-----
From: chakragithub(a)gmail.com [mailto:chakragithub@gmail.com]
Sent: Thursday, October 8, 2020 1:42 AM
To: devel(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i,
krb5p mounts where as it goes through during krb5 mounts.
Environment:
Distro: CentOs 8.2.2004
Kernel version: 4.18.0-193
Ganesha version: V3-Stable
krb version: krb5
Securtiy Type: krb5p
FSAL: VFS
Steps to reproduce:
1) Compiled and installed nfs-ganesha V3-stable branch
2) Enabled VFS – libfsalvfs.so and exported ext3 file system
3) Enabled krb5p security in the conf file
4) Triggered “# iozone -a “ from nfs4 mount
After running for sometime – it got hung.
Is there any JIRA tracking this issue? any suggested workarounds?
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to
devel-leave(a)lists.nfs-ganesha.org
12:24 p.m.
Looks like it based on the issue with krb5i and krb5p.
On Fri, Oct 9, 2020 at 7:52 PM Frank Filz <ffilzlnx(a)mindspring.com> wrote:
Sorry, didn't see this until today.
Malahal, does this sound like it might be the issue with the GSS window?
Frank
> -----Original Message-----
> From: chakragithub(a)gmail.com [mailto:chakragithub@gmail.com]
> Sent: Thursday, October 8, 2020 1:42 AM
> To: devel(a)lists.nfs-ganesha.org
> Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
>
> Hi, We also encountered this issue on fslavfs. iozone tool is hung
during krb5i,
> krb5p mounts where as it goes through during krb5 mounts.
>
> Environment:
> Distro: CentOs 8.2.2004
> Kernel version: 4.18.0-193
> Ganesha version: V3-Stable
> krb version: krb5
>
> Securtiy Type: krb5p
> FSAL: VFS
>
> Steps to reproduce:
> 1) Compiled and installed nfs-ganesha V3-stable branch
> 2) Enabled VFS – libfsalvfs.so and exported ext3 file system
> 3) Enabled krb5p security in the conf file
> 4) Triggered “# iozone -a “ from nfs4 mount
> After running for sometime – it got hung.
>
> Is there any JIRA tracking this issue? any suggested workarounds?
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send
an email to
> devel-leave(a)lists.nfs-ganesha.org
12:41 p.m.
Do you want to share your results with the fix/workaround?
Thanks
Frank
From: Malahal Naineni [mailto:malahal@gmail.com]
Sent: Friday, October 9, 2020 10:24 AM
To: Frank Filz <ffilzlnx(a)mindspring.com>
Cc: chakragithub(a)gmail.com; nfs-ganesha <devel(a)lists.nfs-ganesha.org>
Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Looks like it based on the issue with krb5i and krb5p.
On Fri, Oct 9, 2020 at 7:52 PM Frank Filz <ffilzlnx(a)mindspring.com
<mailto:ffilzlnx@mindspring.com> > wrote:
Sorry, didn't see this until today.
Malahal, does this sound like it might be the issue with the GSS window?
Frank
-----Original Message-----
From: chakragithub(a)gmail.com <mailto:chakragithub@gmail.com>
[mailto:chakragithub@gmail.com <mailto:chakragithub@gmail.com> ]
Sent: Thursday, October 8, 2020 1:42 AM
To: devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org>
Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Hi, We also encountered this issue on fslavfs. iozone tool is hung during krb5i,
krb5p mounts where as it goes through during krb5 mounts.
Environment:
Distro: CentOs 8.2.2004
Kernel version: 4.18.0-193
Ganesha version: V3-Stable
krb version: krb5
Securtiy Type: krb5p
FSAL: VFS
Steps to reproduce:
1) Compiled and installed nfs-ganesha V3-stable branch
2) Enabled VFS – libfsalvfs.so and exported ext3 file system
3) Enabled krb5p security in the conf file
4) Triggered “# iozone -a “ from nfs4 mount
After running for sometime – it got hung.
Is there any JIRA tracking this issue? any suggested workarounds?
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
<mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to
devel-leave(a)lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org>
12:47 p.m.
Frank, Malahal
Thanks for responding....i tried testing the same use case with older stable versions.
Till v2.5 stable ganesha, it worked fine. But from v2.6Stable - it was not hang but became
very slow after some point of time and eventually it exited. In the logs i see checksum
failures in rpc process request.
****************************
09/10/2020 T08:22:17.567082-0700 17095[none] [svc_913] 769 :nfs_rpc_process_request
:SVCAUTH_CHECKSUM failed for Program 100003, Version 4, Function 1, xid=3473413753,
SVCXPRT=0x7f8924101d00, fd=26
09/10/2020 T08:22:17.660648-0700 17095[none] [svc_917] 769 :nfs_rpc_process_request
:SVCAUTH_CHECKSUM failed for Program 100003, Version 4, Function 1, xid=3540522617,
SVCXPRT=0x7f8924101d00, fd=26
****************************
Let me know if you need any other info -
2:37 p.m.
Yes, it works fine with Ganesha2.5 and started having issues with
Ganesha2.7 (probably started in ganesha2.6 but we haven't used it). The fix
is in libntirpc, here. I don't see the main repo updated with these patches
(my repo might be out of date), so I am supplying a link to our repo:
https://github.com/ganltc/ntirpc.git, the following 2 commits:
fd787f4e9 Increase the sequence window for krb5i and krb5p
b890bfb1c Make RPCSEC_GSS window size function of ioq_thrd_max
Hope this helps!
On Fri, Oct 9, 2020 at 11:18 PM Chakra Divi <chakragithub(a)gmail.com> wrote:
Frank, Malahal
Thanks for responding....i tried testing the same use case with older
stable versions.
Till v2.5 stable ganesha, it worked fine. But from v2.6Stable - it was not
hang but became very slow after some point of time and eventually it
exited. In the logs i see checksum failures in rpc process request.
****************************
09/10/2020 T08:22:17.567082-0700 17095[none] [svc_913] 769
:nfs_rpc_process_request :SVCAUTH_CHECKSUM failed for Program 100003,
Version 4, Function 1, xid=3473413753, SVCXPRT=0x7f8924101d00, fd=26
09/10/2020 T08:22:17.660648-0700 17095[none] [svc_917] 769
:nfs_rpc_process_request :SVCAUTH_CHECKSUM failed for Program 100003,
Version 4, Function 1, xid=3540522617, SVCXPRT=0x7f8924101d00, fd=26
****************************
Let me know if you need any other info -
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
4:01 a.m.
Thanks Malahal, ntirpc window size fixes are working. But we are seeing a performance
degradation on v3 when compared with v2.5.
iozone taking 50% more time to complete on the same setup with v3 version. IO operations
with bigger block sizes - seems like having the impact.
With 3.3 – after fix
real 36m37.985s
user 0m1.974s
sys 1m33.561s
v2.5 – working version
real 23m21.187s
user 0m1.797s
sys 1m16.537s
Please let me if you need anything more from my setup like traces...to get hold of the
issue
1:20 p.m.
Assuming that these numbers are with KRB5I and KRB5P, then the window size
is a multiple of the number of threads. The old code might be using a fixed
number of a bigger window size. How many rpc max threads are you using? The
default is 200, I think and it should have a big window size though.
Do you see any perf issues with plain KRB5 or with AUTH_SYS between 2.5 and
3.3. Generally speaking 3.3 should have similar or better perf compared to
ganesha2.5 with AUTH_SYS.
Regards, Malahal.
On Mon, Oct 12, 2020 at 2:33 PM Chakra Divi <chakragithub(a)gmail.com> wrote:
Thanks Malahal, ntirpc window size fixes are working. But we are
seeing a
performance degradation on v3 when compared with v2.5.
iozone taking 50% more time to complete on the same setup with v3
version. IO operations with bigger block sizes - seems like having the
impact.
With 3.3 – after fix
real 36m37.985s
user 0m1.974s
sys 1m33.561s
v2.5 – working version
real 23m21.187s
user 0m1.797s
sys 1m16.537s
Please let me if you need anything more from my setup like traces...to get
hold of the issue
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
7:41 a.m.
Thanks Malahal, i have seen similar performance between 2.5 and 3.3 for AUTH_SYS and krb5
mounts. iozone tool took similar time on both versions.
v3.3 - sys - 9m15.870s
v2.5 – sys - 9m20.688s
v3.3 – krb5 - 9m32.907s
v2.5 – krb5 - 9m18.325s
I have not the number of threads - so it is default : thrd_max 200 gss_seq_win 800 -
window size is based on your fix. Do you think tuning window size would fix this issue ?
11:26 a.m.
with krb5i - i'm not seeing not much degradation - when compared with v2.5.
v2.5 - krb5i - 14m6.888s
v3 - krb5i - 18m55.072s
with krb5p - v3 took more than 50% time than v2.5
11:45 a.m.
Ok, I'm beginning to fear that using gss_iov for krb5p is causing problems, try this
quick ntirpc patch and see if it makes a difference:
diff --git a/src/authgss_prot.c b/src/authgss_prot.c
index b01578f..409da6e 100644
--- a/src/authgss_prot.c
+++ b/src/authgss_prot.c
@@ -267,7 +267,7 @@ xdr_rpc_gss_wrap(XDR *xdrs, xdrproc_t xdr_func, void *xdr_ptr,
* If it's privacy, and NEWBUF is supported (because xdrs is a vector)
* then NEWBUF will have allocated the new buffer.
*/
- vector = (svc == RPCSEC_GSS_SVC_INTEGRITY) || XDR_NEWBUF(xdrs);
+ vector = (svc == RPCSEC_GSS_SVC_INTEGRITY);
/* Marshal rpc_gss_data_t (sequence number + arguments).
* If it's a vector, the response has been marshalled into a new
-----Original Message-----
From: Chakra Divi [mailto:chakragithub@gmail.com]
Sent: Tuesday, October 13, 2020 9:26 AM
To: devel(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
with krb5i - i'm not seeing not much degradation - when compared with v2.5.
v2.5 - krb5i - 14m6.888s
v3 - krb5i - 18m55.072s
with krb5p - v3 took more than 50% time than v2.5
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to
devel-leave(a)lists.nfs-ganesha.org
4:09 a.m.
Frank,
Thanks for the response, tried your patch disabling usage of gss_iov
for krb5p but no luck - iozone tool stuck at 4k block size iops...i
could see the following messages in ganesha logs.
Is there any other place that needs to be changed to disable gss_iov ?
*********
14/10/2020 T02:05:43.505432-0700 6554[::ffff:10.10.108.68] [svc_35] 0
:rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set
dead)
14/10/2020 T02:05:43.505508-0700 6554[::ffff:10.10.108.68] [svc_100] 0
:rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set
dead)
14/10/2020 T02:05:43.505577-0700 6554[::ffff:10.10.108.68] [svc_87] 0
:rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set
dead)
14/10/2020 T02:05:43.505602-0700 6554[::ffff:10.10.108.68] [svc_64] 0
:rpc :svc_vc_reply: 0x7f960c101510 fd 13 SVCAUTH_WRAP failed (will set
dead)
*********
Regards,
Chakra
On Tue, Oct 13, 2020 at 10:15 PM Frank Filz <ffilzlnx(a)mindspring.com> wrote:
>
> Ok, I'm beginning to fear that using gss_iov for krb5p is causing problems, try
this quick ntirpc patch and see if it makes a difference:
>
> diff --git a/src/authgss_prot.c b/src/authgss_prot.c
> index b01578f..409da6e 100644
> --- a/src/authgss_prot.c
> +++ b/src/authgss_prot.c
> @@ -267,7 +267,7 @@ xdr_rpc_gss_wrap(XDR *xdrs, xdrproc_t xdr_func, void *xdr_ptr,
> * If it's privacy, and NEWBUF is supported (because xdrs is a vector)
> * then NEWBUF will have allocated the new buffer.
> */
> - vector = (svc == RPCSEC_GSS_SVC_INTEGRITY) || XDR_NEWBUF(xdrs);
> + vector = (svc == RPCSEC_GSS_SVC_INTEGRITY);
>
> /* Marshal rpc_gss_data_t (sequence number + arguments).
> * If it's a vector, the response has been marshalled into a new
>
> > -----Original Message-----
> > From: Chakra Divi [mailto:chakragithub@gmail.com]
> > Sent: Tuesday, October 13, 2020 9:26 AM
> > To: devel(a)lists.nfs-ganesha.org
> > Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
> >
> > with krb5i - i'm not seeing not much degradation - when compared with v2.5.
> >
> > v2.5 - krb5i - 14m6.888s
> > v3 - krb5i - 18m55.072s
> >
> > with krb5p - v3 took more than 50% time than v2.5
> > _______________________________________________
> > Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email
to
> > devel-leave(a)lists.nfs-ganesha.org
>
8:26 a.m.
Frank,
i have added your patch on top of Malahal Window size changes - and observed hang issue.
hope this is right ?
Let me know if you need any other info - to root cause the perf issue
Chakra
12:10 p.m.
-----Original Message-----
From: Chakra Divi [mailto:chakragithub@gmail.com]
Sent: Sunday, October 18, 2020 9:35 PM
To: devel(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Devel] Re: Issues seen with krb5i and krb5p mounts
Frank,
Any other info that can help me to proceed on this perf issue ?
Unfortunately I'm not sure. I've done some work with Kerberos but it hasn't
been much of a priority.
Maybe someone else can suggest some tracing you could enable or add to try and find where
it's hanging up.
Frank
6:32 a.m.
Frank,
Thanks for responding.
Do you think disabling gss_iov for krb5p will solve this ? If yes, i will proceed with
debugging on top of your patch
Should i raise a bug for this issue ? will that be helpful ?
Chakra
1521
days inactive
1676
days old
18 comments
6 participants
participants (6)
-
Ashish Sangwan -
Chakra Divi -
chakragithub＠gmail.com
-
Frank Filz -
Malahal Naineni -
Trishali Nayar