Also, I just found one recently introduced in next (and possibly
backported to IBM's 2.5?). Fix is here:
On 9/25/19 1:37 AM, Malahal Naineni wrote:
Hi Bill, Madhu did find an issue with ganesha2.5 that we are using
where it does NOT do an unref in certain readdir code path. We will
patch our version soon.
On Thu, Sep 19, 2019 at 9:29 PM Daniel Gryniewicz <dang(a)redhat.com
So, it's not impossible, based on the workload, but it may also be a
For global FDs (All NFSv3 and stateless NFSv4), we obviously cannot
when the client closes the FD, and opening/closing all the time
large performance hit. So, we cache open FDs.
All handles in MDCACHE live on the LRU. This LRU is divided into 2
levels. Level 1 is more active handles, and they can have open FDs.
Various operation can demote a handle to level 2 of the LRU. As
this transition, the global FD on that handle is closed. Handles that
are actively in use (have a refcount taken on them) are not eligible
this transition, as the FD may be being used.
We have a background thread that runs, and periodically does this
demotion, closing the FDs. This thread runs more often when the number
of open FDs is above FD_HwMark_Percent of the available number of FDs,
and runs constantly when the open FD count is above FD_Limit_Percent of
the available number of FDs.
So, a heavily used server could definitely have large numbers of FDs
open. However, there have also, in the past, been bugs that would
either keep the FDs from being closed, or would break the accounting
they were closed, but Ganesha still thought they were open). You
say what version of Ganesha you're using, so I can't tell if one of
those bugs apply.
On 9/19/19 10:19 AM, Billich Heinrich Rainer (ID SD) wrote:
> Is it usual to see 200’000-400’000 open files for a single ganesha
> process? Or does this indicate that something ist wrong?
> We have some issues with ganesha (on spectrum scale protocol nodes)
> reporting NFS3ERR_IO in the log. I noticed that the affected nodes
> have a large number of open files, 200’000-400’000 open files per
> (and 500 threads and about 250 client connections). Other nodes have
> 1’000 – 10’000 open files by ganesha only and don’t show the issue.
> If someone could explain how ganesha decides which files to keep
> and which to close that would help, too. As NFSv3 is stateless the
> client doesn’t open/close a file, it’s the server to decide when to
> close it? We do have a few NFSv4 clients, too.
> Are there certain access patterns that can trigger such a large
> of open file? Maybe traversing and reading a large number of
> Thank you,
> I did count the open files by counting the entries in /proc/<pid of
> ganesha>/fd/ . With several 100k entries I failed to do a ‘ls
> list all the symbolic links, hence I can’t relate the open files to
> different exports easily.
> Heinrich Billich
> ETH Zürich
> Tel.: +41 44 632 72 56
> heinrich.billich(a)id.ethz.ch <mailto:firstname.lastname@example.org>
> Support mailing list -- support(a)lists.nfs-ganesha.org
> To unsubscribe send an email to
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org