1:40 a.m.
We are trying to execute Filebench tests with NFS Ganesha and it most of the time hangs
(filebench process stuck in D state) and does not complete the test.
I have observed the following errors in ganesha.log:
2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111]
[svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1
2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111]
[svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU
thread.
2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388
:lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the
LRU thread can close them.
I was searching in the ganesha community for any similar issues and found these:
https://bugzilla.redhat.com/show_bug.cgi?id=1713261
which is fixed by Frank in :
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/
This patch is not in present in our repo yet and I was thinking of cherry-picking this
change into our FSAL_VFS which is causing the issue.
Frank/Soumya,
Can you confirm if this fix is going to resolve this issue and if I cherry pick this alone
or is there other changes required to resolve this issue please?
1:47 a.m.
On 2/28/20 3:10 PM, des(a)vmware.com wrote:
We are trying to execute Filebench tests with NFS Ganesha and it most
of the time hangs (filebench process stuck in D state) and does not complete the test.
Are you using NFSv3 mount?
I have observed the following errors in ganesha.log:
2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111]
[svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1
2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111]
[svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU
thread.
2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388
:lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the
LRU thread can close them.
I was searching in the ganesha community for any similar issues and found these:
https://bugzilla.redhat.com/show_bug.cgi?id=1713261
which is fixed by Frank in :
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/
This patch is not in present in our repo yet and I was thinking of cherry-picking this
change into our FSAL_VFS which is causing the issue.
Frank/Soumya,
Can you confirm if this fix is going to resolve this issue and if I cherry pick this
alone or is there other changes required to resolve this issue please?
yes. AFAIK this alone should most likely fix issue with open_fd_count
going negative.
Thanks,
Soumya
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
1:49 a.m.
Yes its NFSv3 mounts. Thanks Soumya for confirming.
On 28/02/20, 3:18 PM, "Soumya Koduri" <skoduri(a)redhat.com> wrote:
On 2/28/20 3:10 PM, des(a)vmware.com wrote:
We are trying to execute Filebench tests with NFS Ganesha and it most
of the time hangs (filebench process stuck in D state) and does not complete the test.
Are you using NFSv3 mount?
I have observed the following errors in ganesha.log:
2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111]
[svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1
2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111]
[svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU
thread.
2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388
:lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the
LRU thread can close them.
I was searching in the ganesha community for any similar issues and found these:
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzill...
which is fixed by Frank in :
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Freview....
This patch is not in present in our repo yet and I was thinking of cherry-picking this
change into our FSAL_VFS which is causing the issue.
Frank/Soumya,
Can you confirm if this fix is going to resolve this issue and if I cherry pick this
alone or is there other changes required to resolve this issue please?
yes. AFAIK this alone should most likely fix issue with open_fd_count
going negative.
Thanks,
Soumya
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
11:26 a.m.
What version of Ganesha are you using? I'm not sure if there are other related patches
that might be relevant, but recent releases have been primarily stability (bug) fixes with
the features being mostly on the smaller end.
Frank
-----Original Message-----
From: des(a)vmware.com [mailto:des@vmware.com]
Sent: Friday, February 28, 2020 1:41 AM
To: devel(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Devel] NFSv3 Filebench tests hang
We are trying to execute Filebench tests with NFS Ganesha and it most of the
time hangs (filebench process stuck in D state) and does not complete the test.
I have observed the following errors in ganesha.log:
2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-
34[::ffff:172.30.0.111] [svc_598] 405 :fsal_close :FSAL :open_fd_count is
negative: -1 2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-
34[::ffff:172.30.0.111] [svc_731] 2144 :mdcache_lru_fds_available :INODE LRU
:FD Hard Limit Exceeded, waking LRU thread.
2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none]
[cache_lru] 1388 :lru_run :INODE LRU :Futility count exceeded. Client load is
opening FDs faster than the LRU thread can close them.
I was searching in the ganesha community for any similar issues and found
these:
https://bugzilla.redhat.com/show_bug.cgi?id=1713261
which is fixed by Frank in :
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/
This patch is not in present in our repo yet and I was thinking of cherry-picking
this change into our FSAL_VFS which is causing the issue.
Frank/Soumya,
Can you confirm if this fix is going to resolve this issue and if I cherry pick this
alone or is there other changes required to resolve this issue please?
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to
devel-leave(a)lists.nfs-ganesha.org
2:08 a.m.
We are using V2.8.2 but last we pulled was 7 months ago.
Frank, your fix I mentioned above did fix the Filebench hang in fileserver workload.
But now we are seeing new issue in the 'videoserver' workload.
We are seeing a dump of these errors in ganesha log:
2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121]
[svc_3546] 288
:vdfs_filehandle_open :FSAL :vdfs_open failed: could not get attributes: Input/output
error (5)
2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121]
[svc_3546] 84
:posix2fsal_error :FSAL :Mapping 5 to ERR_FSAL_IO, rlim_cur=65536 rlim_max=65536
There are like 200k lines of these errors within few minutes. I think all operations are
failing with this error.
So this means Ganesha's LRU threads could not close the FDs faster and hence hitting
the underlying filesystem
resource limits?
ulimit -n on our server is 65536, and ganesha.conf is configured with :
CACHEINODE { LRU_Run_Interval = 60; FD_Limit_Percent = 75; FD_HWMark_Percent = 50;
FD_LWMark_Percent = 20; Entries_HWMark = 65536; Reaper_Work_Per_Lane = 100; }
So we should have hit the high water mark and before underlying filesystem and closed the
FDs right?
5:53 a.m.
I'm not sure what's happening here. Your LRU_Run_Interval is 60, which
leaves a window if a workload starts at the wrong time that opens many
FDs per second, but when we pass the HWMark (which is 50% for you, so
lots of space) we'll wake up the LRU thread, and it should run. You
could change a few of the log messages in lru_run() and/or
lru_run_lane() to LogEvent(), so that they show up in the log without
needing to turn on FULL_DEBUG (which will likely slow things down enough
to affect the test). This should tell you what the LRU is doing during
that time. Several fixes went in post 2.8 that you might be interested
in, Several of which may apply here:
aa07c9b9886b5c7f5fce32138e58f71345a3696e
MDCACHE: fix lru_run thread don't get scheduling when thread_delay
adjusted to zero
a49352d63b407010ac0d5817a8772787c7691b73
MDCACHE - Put ref on root
It may just be best to update to 2.8.3 and try again.
Daniel
On 3/4/20 5:08 AM, des(a)vmware.com wrote:
We are using V2.8.2 but last we pulled was 7 months ago.
Frank, your fix I mentioned above did fix the Filebench hang in fileserver workload.
But now we are seeing new issue in the 'videoserver' workload.
We are seeing a dump of these errors in ganesha log:
2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121]
[svc_3546] 288
:vdfs_filehandle_open :FSAL :vdfs_open failed: could not get attributes: Input/output
error (5)
2020-03-03T21:08:53Z : epoch 5e5bfd3c : fsvm23 : ganesha.nfsd-33[::ffff:172.30.0.121]
[svc_3546] 84
:posix2fsal_error :FSAL :Mapping 5 to ERR_FSAL_IO, rlim_cur=65536 rlim_max=65536
There are like 200k lines of these errors within few minutes. I think all operations are
failing with this error.
So this means Ganesha's LRU threads could not close the FDs faster and hence hitting
the underlying filesystem
resource limits?
ulimit -n on our server is 65536, and ganesha.conf is configured with :
CACHEINODE { LRU_Run_Interval = 60; FD_Limit_Percent = 75; FD_HWMark_Percent = 50;
FD_LWMark_Percent = 20; Entries_HWMark = 65536; Reaper_Work_Per_Lane = 100; }
So we should have hit the high water mark and before underlying filesystem and closed the
FDs right?
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
9:30 p.m.
Thanks Daniel, I will evaluate the fixes you mentioned and try it out.
We are in end of a release cycle and we did not want to upgrade ganesha at this stage.
But will surely pull 2.8.3 to our development branch.
1513
days inactive
1519
days old
6 comments
5 participants
participants (5)
-
Daniel Gryniewicz -
Deepthi Shivaramu -
des＠vmware.com
-
Frank Filz -
Soumya Koduri