Announce Push of V5.3.2
by Frank Filz
Branch next
Tag:V5.3.2
Merge Highlights
* [GPFS] Change handle_size to 48 if GPFS returned handle with size 40
* Add LogEventLimited to trace in fsal_common_is_referral
* Handle granted upcalls when they are not in the blocked list
* Add an option to print op id as part of a log entry
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
850c6fbb9 Frank S. Filz V5.3.2
a99efb78d Shahar Hochma Add an option to print op id as part of a log entry
137110141 Malahal Naineni Handle granted upcalls when they are not in the
blocked list
986bddc8e Madhu Thorat Add LogEventLimited to trace in
fsal_common_is_referral
d70dc1a42 Madhu Thorat [GPFS] Change handle_size to 48 if GPFS returned
handle with size 40
2 months, 3 weeks
[M] Change in ...nfs-ganesha[next]: MDCACHE: We need to reap chunks and implement a low water mark
by Frank Filz (GerritHub)
Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556283?usp=email )
Change subject: MDCACHE: We need to reap chunks and implement a low water mark
......................................................................
MDCACHE: We need to reap chunks and implement a low water mark
We shouldn't cache directories forever. We should reap directory
chunks at some interval and rate.
Add a low water mark that we will reap down to.
Change-Id: I5683e9f8be41dc7d92a43a0c6e35ac98a3c0c4f2
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
---
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_ext.h
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_read_conf.c
M src/config_samples/config.txt
M src/doc/man/ganesha-cache-config.rst
6 files changed, 116 insertions(+), 11 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/83/556283/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556283?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I5683e9f8be41dc7d92a43a0c6e35ac98a3c0c4f2
Gerrit-Change-Number: 556283
Gerrit-PatchSet: 1
Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>
Gerrit-MessageType: newchange
2 months, 3 weeks
[S] Change in ...nfs-ganesha[next]: MDCACHE: Fix run interval for chunk lru
by Frank Filz (GerritHub)
Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556253?usp=email )
Change subject: MDCACHE: Fix run interval for chunk lru
......................................................................
MDCACHE: Fix run interval for chunk lru
If chunks_used > chunks_hiwat, wait_ratio could be negative which
would break if time_t is unsigned. Also improve comment for when
new_thread_wait winds up at 0.
Change-Id: I4a4a8a74daeda95b53c6202f095fd79b72be82cf
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
---
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c
1 file changed, 12 insertions(+), 4 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/53/556253/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556253?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I4a4a8a74daeda95b53c6202f095fd79b72be82cf
Gerrit-Change-Number: 556253
Gerrit-PatchSet: 1
Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>
Gerrit-MessageType: newchange
2 months, 4 weeks
[M] Change in ...nfs-ganesha[next]: MDCACHE: make directory chunks references to mdcache entries long term
by Frank Filz (GerritHub)
Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556252?usp=email )
Change subject: MDCACHE: make directory chunks references to mdcache entries long term
......................................................................
MDCACHE: make directory chunks references to mdcache entries long term
Also, rename mdcache_dir_entry_t "entry" to "mde_entry" for ease of
auditing the use of the mde_entry.
Change-Id: Iacc688bb8421e73619e5c7d767f6bde1b4fde884
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
---
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_export.c
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_int.h
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h
7 files changed, 51 insertions(+), 30 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/52/556252/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556252?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: Iacc688bb8421e73619e5c7d767f6bde1b4fde884
Gerrit-Change-Number: 556252
Gerrit-PatchSet: 1
Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>
Gerrit-MessageType: newchange
2 months, 4 weeks
[S] Change in ...nfs-ganesha[next]: IBM-only: Validate client_record before using it.
by Name of user not set (GerritHub)
yogendra858(a)yahoo.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556189?usp=email )
Change subject: IBM-only: Validate client_record before using it.
......................................................................
IBM-only: Validate client_record before using it.
Signed-off-by: Malahal Naineni <malahal(a)us.ibm.com>
(cherry picked from commit ccad258b2e501934f5bd977239bc38fd0942c1df)
Change-Id: Ia3d668b33f6f9a7f2d09c805007c8f3a376b323b
Signed-off-by: Yogendra Charya <Yogendra.Charya(a)ibm.com>
---
M src/Protocols/NFS/nfs4_op_setclientid_confirm.c
1 file changed, 18 insertions(+), 3 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/89/556189/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556189?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: Ia3d668b33f6f9a7f2d09c805007c8f3a376b323b
Gerrit-Change-Number: 556189
Gerrit-PatchSet: 1
Gerrit-Owner: yogendra858(a)yahoo.com
Gerrit-CC: Malahal <malahal(a)gmail.com>
Gerrit-MessageType: newchange
2 months, 4 weeks
[S] Change in ...nfs-ganesha[next]: Handle granted upcalls when they are not in the blocked list
by Name of user not set (GerritHub)
Attention is currently required from: Malahal.
Hello Malahal,
I'd like you to do a code review.
Please visit
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556129?usp=email
to review the following change.
Change subject: Handle granted upcalls when they are not in the blocked list
......................................................................
Handle granted upcalls when they are not in the blocked list
When we CANCEL a blocked lock request, we also send UNLOCK request. This
way the upcall processing code can ignore locks that are not in blocked
lock list.
Change-Id: Id02ba5b6ebcf0dcc40599a8a187846b69a039080
Signed-off-by: Malahal Naineni <malahal(a)us.ibm.com>
---
M src/SAL/state_lock.c
1 file changed, 38 insertions(+), 4 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/29/556129/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556129?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: Id02ba5b6ebcf0dcc40599a8a187846b69a039080
Gerrit-Change-Number: 556129
Gerrit-PatchSet: 1
Gerrit-Owner: skmprabhu252(a)gmail.com
Gerrit-Reviewer: Malahal <malahal(a)gmail.com>
Gerrit-Attention: Malahal <malahal(a)gmail.com>
Gerrit-MessageType: newchange
3 months
Re: MDCACHE LRU reaper thread issues
by Frank Filz
What Ganesha version are you using?
Could you open a github issue for this?
Thanks
Frank
From: Deepak Arumugam Sankara Subramanian [mailto:deepakarumugam.s@nutanix.com]
Sent: Saturday, June 24, 2023 1:25 PM
To: devel(a)lists.nfs-ganesha.org; Frank Filz <ffilzlnx(a)mindspring.com>
Subject: [NFS-Ganesha-Devel] MDCACHE LRU reaper thread issues
ISSUE
We recently hit some issues where the number of ganesha mdcache entries grew to 6-9 Million when the mdcache hiwater mark was at 450,000.
ROOT CAUSE
Upon further debugging we found that readdir was not playing well with garbage collection. The major issue comes from the fact that we have a temporary pointer from the dirent to the inode mdcache entry.
Today when readdir fills up a chunk it puts a temporary reference from dirent to the inode cache entry. This happens inside mdc_readdir_chunk_object
new_dir_entry->entry = new_entry;
Then we clear it out inside mdcache_readdir_chunked,
if (has_write && dirent->entry) {
/* If we get here, we have the write lock, have an
* entry, and took a ref on it above. The dirent also
* has a ref on the entry. Drop that ref now. This can
* only be done under the write lock. If we don't have
* the write lock, then this was not the readdir that
* took the ref, and another readdir will drop the ref,
* or it will be dropped when the dirent is cleaned up.
* */
mdcache_lru_unref(dirent->entry, LRU_FLAG_NONE);
dirent->entry = NULL;
}
But this means each readdir can hold a refcount on 128 entries in the inode cache at a time.
This leads to a scenario where the following happens
1. We have multiple readdirs going in parallel each reading different directories with 1 Million entries. Each readdir ends up holding a refcount of 2 on 128 entries - some of which end up being the entries at the LRU edge. Because of this the reaper thread is not able to find an entry to reap
2. There is also another bug where we are not hitting the clearing out code path in mdcache_readdir_chunked. We don't have an RCA on where this bug is - but we often see systems where all the LRU edges of the inode cache queues are filled with a refcount of 2 even when there is no load on the system. And all the entries are being referenced by their dirents which is the cause of the refcount.
POSSIBLE RESOLUTIONS/QUESTIONS
1. Can we remove this temporary pointer from dirent to the normal mdcache entry? This optimization is only helpful in the cache miss code path. I think this optimization accomplishes two things,
(a) We don't have to do a lookup from dirent to inode entry in the readdir cache miss code path. But since we are already in the readdircache miss code path I don't think the performance penalty will even be visible.
(b) It prevents the inode cache entries from being flushed out when a readdir is in progress. I wonder if we should just use the LRU way to accomplish this - which brings me to the second question
2. Can we always move an entry to the MRU of L2 even during scans? Today this is a bit of an inconsistent behavior - we insert new entries created during readdir at the MRU of L2 but we don't adjust old entries to the MRU. Not sure if this is an intended behaviour or bug
Inside mdc_readdir_chunk_object we call mdcache_new_entry with LRU_FLAG_NONE
status = mdcache_new_entry(export, sub_handle, attrs_in, false, NULL,
false, &new_entry, NULL, LRU_FLAG_NONE);
if its a inode cache hit it follows this path
status = mdcache_find_keyed_reason(&key, entry, flags);
a. s.. if (likely(*entry)) {
fsal_status_t status;
/* Initial Ref on entry */
mdcache_lru_ref(*entry, flags);
since flags is not LRU_REQ_INITIAL this entry doesn't get adjusted to the MRU of L2 or LRU of L1
3. but if its a inode cache miss it follows this path
s.. mdcache_lru_insert(nentry, flags);
if flags has LRU_FLAG_NONE mdcache_lru_insert inserts the entry into the MRU of L2
We would really appreciate your comments on the questions/resolutions,
Thanks,
Deepak
3 months
[S] Change in ...nfs-ganesha[next]: Add an option to print op id as part of a log entry
by Name of user not set (GerritHub)
shaharhoch(a)gmail.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556046?usp=email )
Change subject: Add an option to print op id as part of a log entry
......................................................................
Add an option to print op id as part of a log entry
This will add a unique identifier to the log entry. This can allow us to
track the flow of a specific operation (with the same op_ctx).
An example of a log print with the option enabled:
13/06/2023 T07:13:11.196878+0000 : epoch 64881707 : buildvm-shaharhoch : ganesha.nfsd-10230[main] /home/shaharhoch/ganesha/src/support/exports.c:963 :export_check_options :RW LOCK :F_DBG :op_id=4 :Unlocked 0xaeb6f8 (&exp->exp_lock) at /home/shaharhoch/ganesha/src/support/exports.c:963
Change-Id: Iebb0d999189f58b2b62a01dc8aae05b7f5b20096
Signed-off-by: Shahar Hochma <shaharhoch(a)google.com>
---
M src/config_samples/config.txt
M src/config_samples/logging.txt
M src/doc/man/ganesha-log-config.rst
M src/log/log_functions.c
4 files changed, 18 insertions(+), 0 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/46/556046/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556046?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: Iebb0d999189f58b2b62a01dc8aae05b7f5b20096
Gerrit-Change-Number: 556046
Gerrit-PatchSet: 1
Gerrit-Owner: shaharhoch(a)gmail.com
Gerrit-MessageType: newchange
3 months