Nfs-Ganesha 2.6.3 - NFSv3 - Share hangs after couple of days
I have been dumping several TBs of data over the last few days. I noticed that my copy operations have been hanging for several hours. My ganesha.log is flooded with the below log. [cache_lru] lru_run :INODE LRU :CRIT :Futility count exceeded. The LRU thread is unable to make progress in reclaiming FDs, will try harder. I turned on full debug logs and I see the following logs. Looks like it is not closing any FDs. 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :F_DBG :LRU awakes. 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :F_DBG :lru entries: 51444 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :DEBUG :Open FDs over high water mark, reapring aggressively. 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[chunk_lru] chunk_lru_run :INODE LRU :F_DBG :LRU awakes, lru chunks used: 337 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :DEBUG :Reaping up to 50 entries from lane 0 . . . . 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run_lane :INODE LRU :DEBUG :Actually processed 50 entries on lane 0 closing 0 descriptors I checked the source code and looks like FD is not closed because of this block. What could cause this ref-count to go above 2? This is my own FSAL. Could the FSAL be not decreasing the ref-count? I dont seem to have this problem with NFSv4. FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c /* check refcnt in range */ if (unlikely(refcnt > 2)) { /* This unref is ok to be done without a valid op_ctx * because we always map a new entry to an export before * we could possibly release references in * mdcache_new_entry. */ QUNLOCK(qlane); mdcache_lru_unref(entry); goto next_lru; } I can attach the full log file in the next email if required. I don't have logs at the full-debug level when it was working fine.
Your FSAL should not be taking refcounts (in fact, it should not even have access to MDCACHE entries in order to take refcounts). There have been FD leaks in the past (where we didn't correctly track FDs, and counted more than we should). There have also been entry refcount leaks in the past, which would trigger the code you quoted. 2.6 is a pretty old release, at this point. I would recommend that you upgrade to 2.8 (or even to the upcoming 3.0) Daniel On 10/24/19 10:57 AM, Satish Chandra Kilaru wrote:
I have been dumping several TBs of data over the last few days. I noticed that my copy operations have been hanging for several hours. My ganesha.log is flooded with the below log.
[cache_lru] lru_run :INODE LRU :CRIT :Futility count exceeded. The LRU thread is unable to make progress in reclaiming FDs, will try harder.
I turned on full debug logs and I see the following logs. Looks like it is not closing any FDs.
24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :F_DBG :LRU awakes. 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :F_DBG :lru entries: 51444 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :DEBUG :Open FDs over high water mark, reapring aggressively. 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[chunk_lru] chunk_lru_run :INODE LRU :F_DBG :LRU awakes, lru chunks used: 337 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :DEBUG :Reaping up to 50 entries from lane 0 . . . . 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run_lane :INODE LRU :DEBUG :Actually processed 50 entries on lane 0 closing 0 descriptors
I checked the source code and looks like FD is not closed because of this block. What could cause this ref-count to go above 2? This is my own FSAL. Could the FSAL be not decreasing the ref-count? I dont seem to have this problem with NFSv4.
FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c /* check refcnt in range */ if (unlikely(refcnt > 2)) { /* This unref is ok to be done without a valid op_ctx * because we always map a new entry to an export before * we could possibly release references in * mdcache_new_entry. */ QUNLOCK(qlane); mdcache_lru_unref(entry); goto next_lru; }
I can attach the full log file in the next email if required. I don't have logs at the full-debug level when it was working fine. _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Thank you. My FSAL is not directly accessing those refcounts. I was wondering if it was indirectly affecting those. Any known issues in this area fixed in 2.7 and/or 2.8? --Satish On Thu, Oct 24, 2019 at 12:33 PM Daniel Gryniewicz <dang@redhat.com> wrote:
Your FSAL should not be taking refcounts (in fact, it should not even have access to MDCACHE entries in order to take refcounts). There have been FD leaks in the past (where we didn't correctly track FDs, and counted more than we should). There have also been entry refcount leaks in the past, which would trigger the code you quoted.
2.6 is a pretty old release, at this point. I would recommend that you upgrade to 2.8 (or even to the upcoming 3.0)
Daniel
I have been dumping several TBs of data over the last few days. I noticed that my copy operations have been hanging for several hours. My ganesha.log is flooded with the below log.
[cache_lru] lru_run :INODE LRU :CRIT :Futility count exceeded. The LRU
On 10/24/19 10:57 AM, Satish Chandra Kilaru wrote: thread is unable to make progress in reclaiming FDs, will try harder.
I turned on full debug logs and I see the following logs. Looks like it
is not closing any FDs.
24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net :
24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :F_DBG :lru entries: 51444 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :DEBUG :Open FDs over high water mark, reapring aggressively. 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[chunk_lru] chunk_lru_run :INODE LRU :F_DBG :LRU awakes, lru chunks used: 337 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :DEBUG :Reaping up to 50 entries from lane 0 . . . . 24/10/2019 13:20:50 : epoch 5da91d15 : paplbur04.telecom.tcnz.net : ganesha.nfsd-101379[cache_lru] lru_run_lane :INODE LRU :DEBUG :Actually
ganesha.nfsd-101379[cache_lru] lru_run :INODE LRU :F_DBG :LRU awakes. processed 50 entries on lane 0 closing 0 descriptors
I checked the source code and looks like FD is not closed because of
this block. What could cause this ref-count to go above 2? This is my own FSAL. Could the FSAL be not decreasing the ref-count? I dont seem to have this problem with NFSv4.
FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c /* check refcnt in range */ if (unlikely(refcnt > 2)) { /* This unref is ok to be done without a valid
op_ctx
* because we always map a new entry to an
export before
* we could possibly release references in * mdcache_new_entry. */ QUNLOCK(qlane); mdcache_lru_unref(entry); goto next_lru; }
I can attach the full log file in the next email if required. I don't
have logs at the full-debug level when it was working fine.
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
-- Please Donate to www.wikipedia.org
participants (2)
-
Daniel Gryniewicz -
Satish Chandra Kilaru