Re: Ganesha crash while doing Delegation recall
by Frank Filz
If you can narrow down which operations are the cause of the performance degradation that might help. 8k I/Os could be I/O impact or could be metadata impact or just operation latency.
Frank
From: Sriram Patil [mailto:sriramp@vmware.com]
Sent: Tuesday, May 3, 2022 2:49 PM
To: Frank Filz <ffilzlnx(a)mindspring.com>; Sunil Bhargo <sbhargo(a)vmware.com>; devel(a)lists.nfs-ganesha.org
Cc: dang(a)redhat.com
Subject: [NFS-Ganesha-Devel] Re: Ganesha crash while doing Delegation recall
We use custom FSAL layer, which cannot be open sourced unfortunately. Let me try out 4.0.5 and see if the perf regressions are still there.
Thanks,
Sriram
From: Frank Filz <ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> >
Date: Tuesday, May 3, 2022 at 1:57 PM
To: Sriram Patil <sriramp(a)vmware.com <mailto:sriramp@vmware.com> >, Sunil Bhargo <sbhargo(a)vmware.com <mailto:sbhargo@vmware.com> >, devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> <devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> >
Cc: dang(a)redhat.com <mailto:dang@redhat.com> <dang(a)redhat.com <mailto:dang@redhat.com> >
Subject: RE: [NFS-Ganesha-Devel] Re: Ganesha crash while doing Delegation recall
⚠ External Email
You are using FSAL_VFS? I think there is some stuff in 4.0.x that might help, we will be tagging 4.1 in the next few weeks, so if you want to try 4.0.5 and see how that fares that would be good information. It would be good to understand what might have caused such a big performance regression.
Frank
From: Sriram Patil [mailto:sriramp@vmware.com]
Sent: Tuesday, May 3, 2022 12:02 PM
To: Frank Filz <ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> >; Sunil Bhargo <sbhargo(a)vmware.com <mailto:sbhargo@vmware.com> >; devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org>
Cc: dang(a)redhat.com <mailto:dang@redhat.com>
Subject: [NFS-Ganesha-Devel] Re: Ganesha crash while doing Delegation recall
Hi Frank,
Thanks for the response. I also though disabling delegations is a safer bet at this point. But I wanted to check regarding latest version. We tried upgrading to version 3.5 a few months back and then observer about 15-25% regression in our perf tests for 8k RW on NFSV3 as well as NFSv4.1. Were there any perf fixes that went in after 3.5?
This perf regression is holding us back from upgrading.
Thanks,
Sriram
From: Frank Filz <ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> >
Date: Friday, April 29, 2022 at 4:15 PM
To: Sunil Bhargo <sbhargo(a)vmware.com <mailto:sbhargo@vmware.com> >, devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> <devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> >
Cc: dang(a)redhat.com <mailto:dang@redhat.com> <dang(a)redhat.com <mailto:dang@redhat.com> >, Sriram Patil <sriramp(a)vmware.com <mailto:sriramp@vmware.com> >
Subject: RE: [NFS-Ganesha-Devel] Ganesha crash while doing Delegation recall
⚠ External Email
V2.8 is pretty old and no longer supported. Additionally, I don’t know if delegations have been tested or supported recently at all so I doubt there’s any fixes but you could certainly try a newer release, or look through the git history.
Frank
From: Sunil Bhargo [mailto:sbhargo@vmware.com]
Sent: Thursday, April 28, 2022 6:21 AM
To: devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org>
Cc: ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> ; dang(a)redhat.com <mailto:dang@redhat.com> ; Sriram Patil <sriramp(a)vmware.com <mailto:sriramp@vmware.com> >
Subject: [NFS-Ganesha-Devel] Ganesha crash while doing Delegation recall
Hi,
Currently seeing the following crash and it is a recurrent crash. We are using ganesha 2.8.4. Can someone please help in telling if this is a known issue and whether it is fixed in later releases ?
(gdb) bt
#0 0x00007fcc26df6570 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x00007fcc2760906c in svc_rqst_expire_insert () from /lib/libntirpc.so.1.8
#2 0x00007fcc275f2e19 in clnt_req_callback () from /lib/libntirpc.so.1.8
#3 0x00007fcc27855c3e in nfs_rpc_call () from /lib/libganesha_nfsd.so.2.8
#4 0x00007fcc27856a13 in nfs_rpc_cb_single () from /lib/libganesha_nfsd.so.2.8
#5 0x00007fcc278837a4 in delegrecall_one () from /lib/libganesha_nfsd.so.2.8
#6 0x00007fcc27884422 in delegrecall_impl () from /lib/libganesha_nfsd.so.2.8
#7 0x00007fcc27886900 in queue_delegrecall () from /lib/libganesha_nfsd.so.2.8
#8 0x00007fcc278e6da9 in fridgethr_start_routine () from /lib/libganesha_nfsd.so.2.8
#9 0x00007fcc26df3f87 in start_thread () from /lib/libpthread.so.0
#10 0x00007fcc268755bf in clone () from /lib/libc.so.6
This seems like a channel is removed while it is being accessed by the backchannel.
Thanks,
Sunil Bhargo
_____
⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.
10 months, 3 weeks
Re: Ganesha crash while doing Delegation recall
by Frank Filz
You are using FSAL_VFS? I think there is some stuff in 4.0.x that might help, we will be tagging 4.1 in the next few weeks, so if you want to try 4.0.5 and see how that fares that would be good information. It would be good to understand what might have caused such a big performance regression.
Frank
From: Sriram Patil [mailto:sriramp@vmware.com]
Sent: Tuesday, May 3, 2022 12:02 PM
To: Frank Filz <ffilzlnx(a)mindspring.com>; Sunil Bhargo <sbhargo(a)vmware.com>; devel(a)lists.nfs-ganesha.org
Cc: dang(a)redhat.com
Subject: [NFS-Ganesha-Devel] Re: Ganesha crash while doing Delegation recall
Hi Frank,
Thanks for the response. I also though disabling delegations is a safer bet at this point. But I wanted to check regarding latest version. We tried upgrading to version 3.5 a few months back and then observer about 15-25% regression in our perf tests for 8k RW on NFSV3 as well as NFSv4.1. Were there any perf fixes that went in after 3.5?
This perf regression is holding us back from upgrading.
Thanks,
Sriram
From: Frank Filz <ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> >
Date: Friday, April 29, 2022 at 4:15 PM
To: Sunil Bhargo <sbhargo(a)vmware.com <mailto:sbhargo@vmware.com> >, devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> <devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> >
Cc: dang(a)redhat.com <mailto:dang@redhat.com> <dang(a)redhat.com <mailto:dang@redhat.com> >, Sriram Patil <sriramp(a)vmware.com <mailto:sriramp@vmware.com> >
Subject: RE: [NFS-Ganesha-Devel] Ganesha crash while doing Delegation recall
⚠ External Email
V2.8 is pretty old and no longer supported. Additionally, I don’t know if delegations have been tested or supported recently at all so I doubt there’s any fixes but you could certainly try a newer release, or look through the git history.
Frank
From: Sunil Bhargo [mailto:sbhargo@vmware.com]
Sent: Thursday, April 28, 2022 6:21 AM
To: devel(a)lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org>
Cc: ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> ; dang(a)redhat.com <mailto:dang@redhat.com> ; Sriram Patil <sriramp(a)vmware.com <mailto:sriramp@vmware.com> >
Subject: [NFS-Ganesha-Devel] Ganesha crash while doing Delegation recall
Hi,
Currently seeing the following crash and it is a recurrent crash. We are using ganesha 2.8.4. Can someone please help in telling if this is a known issue and whether it is fixed in later releases ?
(gdb) bt
#0 0x00007fcc26df6570 in pthread_mutex_lock () from /lib/libpthread.so.0
#1 0x00007fcc2760906c in svc_rqst_expire_insert () from /lib/libntirpc.so.1.8
#2 0x00007fcc275f2e19 in clnt_req_callback () from /lib/libntirpc.so.1.8
#3 0x00007fcc27855c3e in nfs_rpc_call () from /lib/libganesha_nfsd.so.2.8
#4 0x00007fcc27856a13 in nfs_rpc_cb_single () from /lib/libganesha_nfsd.so.2.8
#5 0x00007fcc278837a4 in delegrecall_one () from /lib/libganesha_nfsd.so.2.8
#6 0x00007fcc27884422 in delegrecall_impl () from /lib/libganesha_nfsd.so.2.8
#7 0x00007fcc27886900 in queue_delegrecall () from /lib/libganesha_nfsd.so.2.8
#8 0x00007fcc278e6da9 in fridgethr_start_routine () from /lib/libganesha_nfsd.so.2.8
#9 0x00007fcc26df3f87 in start_thread () from /lib/libpthread.so.0
#10 0x00007fcc268755bf in clone () from /lib/libc.so.6
This seems like a channel is removed while it is being accessed by the backchannel.
Thanks,
Sunil Bhargo
_____
⚠ External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.
10 months, 3 weeks