I have been testing Ganesha 2.5.4 code with default mdcache settings. It starts showing issues after prolonged I/O runs.

Once it exhausts all the allowed fds, its kind of gets stuck returning ERR_FSAL_DELAY for every client op.

A snapshot of the mdcache

open_fd_count = 4055

lru_state = {

entries_hiwat = 100000,

entries_used = 323,

chunks_hiwat = 100000,

chunks_used = 9,

fds_system_imposed = 4096,

fds_hard_limit = 4055,

fds_hiwat = 3686,

fds_lowat = 2048,

futility = 109,

per_lane_work = 50,

biggest_window = 1638,

prev_fd_count = 4055,

prev_time = 1529013538,

fd_state = 3

}

[cache_lru] lru_run :INODE LRU :INFO :After work, open_fd_count:4055 entries used count:327 fdrate:0 threadwait=9

[cache_lru] lru_run :INODE LRU :INFO :lru entries: 327 open_fd_count:4055

[cache_lru] lru_run :INODE LRU :INFO :lru entries: 327open_fd_count:4055

[cache_lru] lru_run :INODE LRU :INFO :After work, open_fd_count:4055 entries used count:327 fdrate:0 threadwait=90

I have killed the NFS clients, so no new I/O is being received. But even after a couple of hours I don't see lru_run making any progress, thereby open_fd_count remains a 4055 and even a single file open won't be served. So basically the server is in stuck state.

I have these changes patched over 2.5.4 code

e2156ad3feac841487ba89969769bf765457ea6e Replace cache_fds parameter and handling with better logic

667083fe395ddbb4aa14b7bbe7e15ffca87e3b0b MDCACHE - Change and lower futility message

37732e61985d919e6ca84dfa7b4a84163080abae Move open_fd_count from MDCACHE to FSALs (https://review.gerrithub.io/#/c/391267/)

Any suggestions how to resolve this ?