NFSv3 Filebench tests hang
We are trying to execute Filebench tests with NFS Ganesha and it most of the time hangs (filebench process stuck in D state) and does not complete the test. I have observed the following errors in ganesha.log: 2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1 2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU thread. 2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388 :lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the LRU thread can close them. I was searching in the ganesha community for any similar issues and found these: https://bugzilla.redhat.com/show_bug.cgi?id=1713261 which is fixed by Frank in : https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/ This patch is not in present in our repo yet and I was thinking of cherry-picking this change into our FSAL_VFS which is causing the issue. Frank/Soumya, Can you confirm if this fix is going to resolve this issue and if I cherry pick this alone or is there other changes required to resolve this issue please?
On 2/28/20 3:10 PM, des@vmware.com wrote:
We are trying to execute Filebench tests with NFS Ganesha and it most of the time hangs (filebench process stuck in D state) and does not complete the test.
Are you using NFSv3 mount?
I have observed the following errors in ganesha.log:
2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1 2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU thread. 2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388 :lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the LRU thread can close them.
I was searching in the ganesha community for any similar issues and found these: https://bugzilla.redhat.com/show_bug.cgi?id=1713261
which is fixed by Frank in : https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/
This patch is not in present in our repo yet and I was thinking of cherry-picking this change into our FSAL_VFS which is causing the issue.
Frank/Soumya, Can you confirm if this fix is going to resolve this issue and if I cherry pick this alone or is there other changes required to resolve this issue please?
yes. AFAIK this alone should most likely fix issue with open_fd_count going negative. Thanks, Soumya
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Yes its NFSv3 mounts. Thanks Soumya for confirming. On 28/02/20, 3:18 PM, "Soumya Koduri" <skoduri@redhat.com> wrote: On 2/28/20 3:10 PM, des@vmware.com wrote: > We are trying to execute Filebench tests with NFS Ganesha and it most of the time hangs (filebench process stuck in D state) and does not complete the test. Are you using NFSv3 mount? > > I have observed the following errors in ganesha.log: > > 2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1 > 2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU thread. > 2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388 :lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the LRU thread can close them. > > I was searching in the ganesha community for any similar issues and found these: > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.r... > > which is fixed by Frank in : > https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview.ger... > > This patch is not in present in our repo yet and I was thinking of cherry-picking this change into our FSAL_VFS which is causing the issue. > > Frank/Soumya, > Can you confirm if this fix is going to resolve this issue and if I cherry pick this alone or is there other changes required to resolve this issue please? yes. AFAIK this alone should most likely fix issue with open_fd_count going negative. Thanks, Soumya > _______________________________________________ > Devel mailing list -- devel@lists.nfs-ganesha.org > To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org >
What version of Ganesha are you using? I'm not sure if there are other related patches that might be relevant, but recent releases have been primarily stability (bug) fixes with the features being mostly on the smaller end. Frank
-----Original Message----- From: des@vmware.com [mailto:des@vmware.com] Sent: Friday, February 28, 2020 1:41 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] NFSv3 Filebench tests hang
We are trying to execute Filebench tests with NFS Ganesha and it most of the time hangs (filebench process stuck in D state) and does not complete the test.
I have observed the following errors in ganesha.log:
2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd- 34[::ffff:172.30.0.111] [svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1 2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd- 34[::ffff:172.30.0.111] [svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU thread. 2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388 :lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the LRU thread can close them.
I was searching in the ganesha community for any similar issues and found these: https://bugzilla.redhat.com/show_bug.cgi?id=1713261
which is fixed by Frank in : https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/
This patch is not in present in our repo yet and I was thinking of cherry-picking this change into our FSAL_VFS which is causing the issue.
Frank/Soumya, Can you confirm if this fix is going to resolve this issue and if I cherry pick this alone or is there other changes required to resolve this issue please? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
We are using V2.8.2 but last we pulled was 7 months ago. Frank, your fix I mentioned above did fix the Filebench hang in fileserver workload. But now we are seeing new issue in the 'videoserver' workload. We are seeing a dump of these errors in ganesha log: 2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121] [svc_3546] 288 :vdfs_filehandle_open :FSAL :vdfs_open failed: could not get attributes: Input/output error (5) 2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121] [svc_3546] 84 :posix2fsal_error :FSAL :Mapping 5 to ERR_FSAL_IO, rlim_cur=65536 rlim_max=65536 There are like 200k lines of these errors within few minutes. I think all operations are failing with this error. So this means Ganesha's LRU threads could not close the FDs faster and hence hitting the underlying filesystem resource limits? ulimit -n on our server is 65536, and ganesha.conf is configured with : CACHEINODE { LRU_Run_Interval = 60; FD_Limit_Percent = 75; FD_HWMark_Percent = 50; FD_LWMark_Percent = 20; Entries_HWMark = 65536; Reaper_Work_Per_Lane = 100; } So we should have hit the high water mark and before underlying filesystem and closed the FDs right?
I'm not sure what's happening here. Your LRU_Run_Interval is 60, which leaves a window if a workload starts at the wrong time that opens many FDs per second, but when we pass the HWMark (which is 50% for you, so lots of space) we'll wake up the LRU thread, and it should run. You could change a few of the log messages in lru_run() and/or lru_run_lane() to LogEvent(), so that they show up in the log without needing to turn on FULL_DEBUG (which will likely slow things down enough to affect the test). This should tell you what the LRU is doing during that time. Several fixes went in post 2.8 that you might be interested in, Several of which may apply here: aa07c9b9886b5c7f5fce32138e58f71345a3696e MDCACHE: fix lru_run thread don't get scheduling when thread_delay adjusted to zero a49352d63b407010ac0d5817a8772787c7691b73 MDCACHE - Put ref on root It may just be best to update to 2.8.3 and try again. Daniel On 3/4/20 5:08 AM, des@vmware.com wrote:
We are using V2.8.2 but last we pulled was 7 months ago.
Frank, your fix I mentioned above did fix the Filebench hang in fileserver workload.
But now we are seeing new issue in the 'videoserver' workload.
We are seeing a dump of these errors in ganesha log:
2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121] [svc_3546] 288 :vdfs_filehandle_open :FSAL :vdfs_open failed: could not get attributes: Input/output error (5) 2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121] [svc_3546] 84 :posix2fsal_error :FSAL :Mapping 5 to ERR_FSAL_IO, rlim_cur=65536 rlim_max=65536
There are like 200k lines of these errors within few minutes. I think all operations are failing with this error.
So this means Ganesha's LRU threads could not close the FDs faster and hence hitting the underlying filesystem resource limits?
ulimit -n on our server is 65536, and ganesha.conf is configured with :
CACHEINODE { LRU_Run_Interval = 60; FD_Limit_Percent = 75; FD_HWMark_Percent = 50; FD_LWMark_Percent = 20; Entries_HWMark = 65536; Reaper_Work_Per_Lane = 100; }
So we should have hit the high water mark and before underlying filesystem and closed the FDs right?
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Thanks Daniel, I will evaluate the fixes you mentioned and try it out. We are in end of a release cycle and we did not want to upgrade ganesha at this stage. But will surely pull 2.8.3 to our development branch.
participants (5)
-
Daniel Gryniewicz -
Deepthi Shivaramu -
des＠vmware.com
-
Frank Filz -
Soumya Koduri