June 2023 - Devel - NFS Ganesha List Archives

[S] Change in ...nfs-ganesha[next]: Add an option to print op id as part of a log entry
by Name of user not set (GerritHub) 25 Jun '23

25 Jun '23

shaharhoch(a)gmail.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556046?usp=email ) Change subject: Add an option to print op id as part of a log entry ...................................................................... Add an option to print op id as part of a log entry This will add a unique identifier to the log entry. This can allow us to track the flow of a specific operation (with the same op_ctx). An example of a log print with the option enabled: 13/06/2023 T07:13:11.196878+0000 : epoch 64881707 : buildvm-shaharhoch : ganesha.nfsd-10230[main] /home/shaharhoch/ganesha/src/support/exports.c:963 :export_check_options :RW LOCK :F_DBG :op_id=4 :Unlocked 0xaeb6f8 (&exp->exp_lock) at /home/shaharhoch/ganesha/src/support/exports.c:963 Change-Id: Iebb0d999189f58b2b62a01dc8aae05b7f5b20096 Signed-off-by: Shahar Hochma <shaharhoch(a)google.com> --- M src/config_samples/config.txt M src/config_samples/logging.txt M src/doc/man/ganesha-log-config.rst M src/log/log_functions.c 4 files changed, 18 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/46/556046/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556046?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Iebb0d999189f58b2b62a01dc8aae05b7f5b20096 Gerrit-Change-Number: 556046 Gerrit-PatchSet: 1 Gerrit-Owner: shaharhoch(a)gmail.com Gerrit-MessageType: newchange

1 0

[XS] Change in ...nfs-ganesha[next]: Add op_id to op_context
by Name of user not set (GerritHub) 25 Jun '23

25 Jun '23

shaharhoch(a)gmail.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556045?usp=email ) Change subject: Add op_id to op_context ...................................................................... Add op_id to op_context The op id will be a counter that continuesly increases Change-Id: Iafd69164b53110043e7ea6a9826fe67b260bcb6c Signed-off-by: Shahar Hochma <shaharhoch(a)google.com> --- M src/FSAL/commonlib.c M src/include/fsal_api.h 2 files changed, 3 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/45/556045/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/556045?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Iafd69164b53110043e7ea6a9826fe67b260bcb6c Gerrit-Change-Number: 556045 Gerrit-PatchSet: 1 Gerrit-Owner: shaharhoch(a)gmail.com Gerrit-MessageType: newchange

1 0

MDCACHE LRU reaper thread issues
by Deepak Arumugam Sankara Subramanian 24 Jun '23

24 Jun '23

ISSUE We recently hit some issues where the number of ganesha mdcache entries grew to 6-9 Million when the mdcache hiwater mark was at 450,000. ROOT CAUSE Upon further debugging we found that readdir was not playing well with garbage collection. The major issue comes from the fact that we have a temporary pointer from the dirent to the inode mdcache entry. Today when readdir fills up a chunk it puts a temporary reference from dirent to the inode cache entry. This happens inside mdc_readdir_chunk_object new_dir_entry->entry = new_entry; Then we clear it out inside mdcache_readdir_chunked, if (has_write && dirent->entry) { /* If we get here, we have the write lock, have an * entry, and took a ref on it above. The dirent also * has a ref on the entry. Drop that ref now. This can * only be done under the write lock. If we don't have * the write lock, then this was not the readdir that * took the ref, and another readdir will drop the ref, * or it will be dropped when the dirent is cleaned up. * */ mdcache_lru_unref(dirent->entry, LRU_FLAG_NONE); dirent->entry = NULL; } But this means each readdir can hold a refcount on 128 entries in the inode cache at a time. This leads to a scenario where the following happens 1. We have multiple readdirs going in parallel each reading different directories with 1 Million entries. Each readdir ends up holding a refcount of 2 on 128 entries - some of which end up being the entries at the LRU edge. Because of this the reaper thread is not able to find an entry to reap 2. There is also another bug where we are not hitting the clearing out code path in mdcache_readdir_chunked. We don't have an RCA on where this bug is - but we often see systems where all the LRU edges of the inode cache queues are filled with a refcount of 2 even when there is no load on the system. And all the entries are being referenced by their dirents which is the cause of the refcount. POSSIBLE RESOLUTIONS/QUESTIONS 1. Can we remove this temporary pointer from dirent to the normal mdcache entry? This optimization is only helpful in the cache miss code path. I think this optimization accomplishes two things, (a) We don't have to do a lookup from dirent to inode entry in the readdir cache miss code path. But since we are already in the readdircache miss code path I don't think the performance penalty will even be visible. (b) It prevents the inode cache entries from being flushed out when a readdir is in progress. I wonder if we should just use the LRU way to accomplish this - which brings me to the second question 2. Can we always move an entry to the MRU of L2 even during scans? Today this is a bit of an inconsistent behavior - we insert new entries created during readdir at the MRU of L2 but we don't adjust old entries to the MRU. Not sure if this is an intended behaviour or bug Inside mdc_readdir_chunk_object we call mdcache_new_entry with LRU_FLAG_NONE status = mdcache_new_entry(export, sub_handle, attrs_in, false, NULL, false, &new_entry, NULL, LRU_FLAG_NONE); if its a inode cache hit it follows this path status = mdcache_find_keyed_reason(&key, entry, flags); * s.. if (likely(*entry)) { fsal_status_t status; /* Initial Ref on entry */ mdcache_lru_ref(*entry, flags); since flags is not LRU_REQ_INITIAL this entry doesn't get adjusted to the MRU of L2 or LRU of L1 3. but if its a inode cache miss it follows this path s.. mdcache_lru_insert(nentry, flags); if flags has LRU_FLAG_NONE mdcache_lru_insert inserts the entry into the MRU of L2 We would really appreciate your comments on the questions/resolutions, Thanks, Deepak

1 0

Announce Push of V5.3.1
by Frank Filz 23 Jun '23

23 Jun '23

Branch next Tag:V5.3.1 Merge Highlights * uid2grp - fix placement of PTHREAD_MUTEX_destroy * GLUSTER: fixup new fsal_fd handling * CHECKPATCH - we stll prefer 80 column lines * Fix uo scripts/runcp.sh - macro no longer used * CHECKPATCH - fix up warnings and errors that have crept in * Several fixes to multilock test tool plus new test cases * FSAL_MDCACHE/mdcache_handle: Fixed rename atomicity * init_fds_limit(): Log EVENT to print current set FD limit * nfs_read_conf: Set the PNFS_MDS & PNFS_DS default to false like written * Fix possible race when populating pseudo fs * Fix crash @ free_client_record() Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: c3dd75362 Frank S. Filz V5.3.1 232521210 Prabhu Murugesan Fix crash @ free_client_record() c67f9a5e8 Shahar Hochma Fix possible race when populating pseudo fs d440c2b41 Assaf Yaari nfs_read_conf: Set the PNFS_MDS & PNFS_DS default to false like written in the docs and comments b3c9bd840 Madhu Thorat init_fds_limit(): Log EVENT to print current set FD limit eb15ca16c Lior Suliman multilock: Fixed a bug in the sleep command 117b325ae Lior Suliman FSAL_MDCACHE/mdcache_handle: Fixed rename atomicity 5879883d6 Lior Suliman multilock: Added scenarios to sample_tests 6729c56e3 Lior Suliman multilock: Added SO_REUSEADDR to the multilock console server 2292e0c75 Lior Suliman multilock: Allow the usage of the full UINT64 range for locks 99d09f50b Frank S. Filz CHECKPATCH - fix up warnings and errors that have crept in 4253855e6 Frank S. Filz Fix uo scripts/runcp.sh - macro no longer used 9ab836d97 Frank S. Filz CHECKPATCH - we stll prefer 80 column lines 7e445649b Frank S. Filz GLUSTER: fixup new fsal_fd handling b5f8eaebe Frank S. Filz uid2grp - fix placement of PTHREAD_MUTEX_destroy

1 0

[M] Change in ...nfs-ganesha[next]: CHECKPATCH - fix up warnings and errors that have crept in
by Frank Filz (GerritHub) 22 Jun '23

22 Jun '23

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555985?usp=email ) Change subject: CHECKPATCH - fix up warnings and errors that have crept in ...................................................................... CHECKPATCH - fix up warnings and errors that have crept in Change-Id: Id15b1daf654bcdf1ef4875ff51d10ec7f43ca827 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h M src/MainNFSD/nfs_rpc_callback.c M src/Protocols/NFS/nfs4_op_close.c M src/Protocols/NFS/nfs_proto_tools.c M src/RPCAL/rpc_tools.c M src/include/9p.h M src/include/9p_req_queue.h M src/include/atomic_utils.h M src/support/nfs4_acls.c 10 files changed, 48 insertions(+), 19 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/85/555985/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555985?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Id15b1daf654bcdf1ef4875ff51d10ec7f43ca827 Gerrit-Change-Number: 555985 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

1 0

[S] Change in ...nfs-ganesha[next]: Fix uo scripts/runcp.sh - macro no longer used
by Frank Filz (GerritHub) 22 Jun '23

22 Jun '23

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555984?usp=email ) Change subject: Fix uo scripts/runcp.sh - macro no longer used ...................................................................... Fix uo scripts/runcp.sh - macro no longer used Change-Id: Ia6ce0404f2575b5804749ef5ab3837a3fe3ea3e9 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/scripts/runcp.sh 1 file changed, 0 insertions(+), 10 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/84/555984/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555984?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ia6ce0404f2575b5804749ef5ab3837a3fe3ea3e9 Gerrit-Change-Number: 555984 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

1 0

[XS] Change in ...nfs-ganesha[next]: CHECKPATCH - we stll prefer 80 column lines
by Frank Filz (GerritHub) 22 Jun '23

22 Jun '23

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555983?usp=email ) Change subject: CHECKPATCH - we stll prefer 80 column lines ...................................................................... CHECKPATCH - we stll prefer 80 column lines The kernel has relaxed to allow 100 column. Gerrithub code review shows a line at 80 columns and most of our code is 80 columns so let's keep it. Change-Id: I7f67afca0cd3458488d2340c225b3cb7fa1011f9 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/scripts/checkpatch.conf 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/83/555983/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555983?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I7f67afca0cd3458488d2340c225b3cb7fa1011f9 Gerrit-Change-Number: 555983 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

1 0

[XS] Change in ...nfs-ganesha[next]: multilock: Fixed a bug in the sleep command
by Lior Suliman (GerritHub) 22 Jun '23

22 Jun '23

Attention is currently required from: Frank Filz. Hello Frank Filz, I'd like you to do a code review. Please visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555979?usp=email to review the following change. Change subject: multilock: Fixed a bug in the sleep command ...................................................................... multilock: Fixed a bug in the sleep command The timeend - time(NULL) happened to be negative and that called the pselect syscall with a negative number and the tool finished with an error. The case that ran was `simple_block`, when it got to the 2 second sleep command it did it in two iterations and in the second iteration the tv_secs was calculated to be minus 1 resulting in an error from the pselect call. Output: Line 34: sleep 2 Waiting for clients About to sleep for 2 secs Waiting for clients About to sleep for -1 secs <- Error ... err: select failed with ERRNO 22 "Invalid argument" Receive failed ERRNO 22 "Invalid argument" Unexpected response 1 errors FAIL Change-Id: Ifddc3fe27ad72f21db6122a5084186051efc4fd2 Signed-off-by: Lior Suliman <liorsu(a)google.com> --- M src/tools/multilock/ml_console.c 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/79/555979/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555979?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ifddc3fe27ad72f21db6122a5084186051efc4fd2 Gerrit-Change-Number: 555979 Gerrit-PatchSet: 1 Gerrit-Owner: Lior Suliman <liorsu(a)gmail.com> Gerrit-Reviewer: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-Attention: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

1 0

[XS] Change in ...nfs-ganesha[next]: nfs_read_conf: Set the PNFS_MDS & PNFS_DS default to false like writt...
by Assaf Yaari (GerritHub) 22 Jun '23

22 Jun '23

Assaf Yaari has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555955?usp=email ) Change subject: nfs_read_conf: Set the PNFS_MDS & PNFS_DS default to false like written in the docs and comments ...................................................................... nfs_read_conf: Set the PNFS_MDS & PNFS_DS default to false like written in the docs and comments Change-Id: I8035ed52e935e4f61a24164bbfdb71552a765421 Signed-off-by: Assaf Yaari <assafy(a)google.com> --- M src/support/nfs_read_conf.c 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/55/555955/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555955?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I8035ed52e935e4f61a24164bbfdb71552a765421 Gerrit-Change-Number: 555955 Gerrit-PatchSet: 1 Gerrit-Owner: Assaf Yaari <assafy(a)google.com> Gerrit-MessageType: newchange

1 0

[XS] Change in ...nfs-ganesha[next]: nfs4_op_exchange_id: pnfs server flags should not be empty
by Assaf Yaari (GerritHub) 22 Jun '23

22 Jun '23

Assaf Yaari has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555954?usp=email ) Change subject: nfs4_op_exchange_id: pnfs server flags should not be empty ...................................................................... nfs4_op_exchange_id: pnfs server flags should not be empty In case the client sent pNFS flags in the EXCHANGE_ID request and the server has no pnfs_mds, EXCHGID4_FLAG_USE_NON_PNFS should be returned. See section 13.1 in RFC8881. Change-Id: Ib760cf26a6cb3b9d10c382eb8b03f36a223c79cc Signed-off-by: Assaf Yaari <assafy(a)google.com> --- M src/Protocols/NFS/nfs4_op_exchange_id.c 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/54/555954/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/555954?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ib760cf26a6cb3b9d10c382eb8b03f36a223c79cc Gerrit-Change-Number: 555954 Gerrit-PatchSet: 1 Gerrit-Owner: Assaf Yaari <assafy(a)google.com> Gerrit-MessageType: newchange

1 0