nfs4_op_readdir returning NFS4ERR_DELAY due to error in mdcache_avl_insert_ck
by Sandeep Nashikkar
Hi
We are having a samba server exporting a locally mounted ganesha share and we have our own FSAL implementation.
Samba share consistently get stuck during readdir operation on NFS export in particular sequence of operations from samba on ganesha export.
These are particular set of operations before it gets stuck:
1. There are around 160 files in the folder and readdir output looks like
.
..
file1
file2
file3
file4
.....and so on
2. file2 and file3 in the previous readdir output are deleted.
3. readdir_ex calls from FSAL now return following one by one
.
..
file1
file4
4. After it returns file4, readdir call from ganesha returns NFS4ERR_DELAY to client with following error in ganesha logs:
12/01/2021 03:24:32 : epoch 5ffc6dd5 : <machine> : ganesha.nfsd-20352[svc_168] mdcache_avl_insert_ck :NFS READDIR :DEBUG :NFS READDIR: DEBUG: Already existent when inserting dirent 0x7f0b6c0d8c40 for <filename> on entry=0x7f0b6802dfb0 FSAL cookie=4, duplicated directory cookies make READDIR unreliable.
1. ls -l command on the directory over local mount also gets stuck for few mins.
2. Previous readdir calls logs between (before files were deleted In #2) reveal that the file4 had cookie=6 and that's where ganesha throws an error
12/01/2021 03:24:08 : epoch 5ffc6dd5 : OneCSUnixMa.idcprodcert.loc : ganesha.nfsd-20352[svc_173] mdcache_avl_insert_ck :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Insert dirent 0x7f0b68020cb0 for collection-101-3540931298411264490.wt on entry=0x7f0b6802dfb0 FSAL cookie=6
So couple of questions:
1. What is the cookie value we should send to avoid this error occurring during insertion in AVL tree? We currently return the offset of the entry in cookie.
2. Do we need to invalidate the mdcache of ganesha using up-call whenever there is a change in the files inside the underlying dirent due to rename or delete operation?
Thanks
Sandeep
3 years, 11 months
Change in ...nfs-ganesha[next]: Clean up log config
by Frank Filz (GerritHub)
Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508876 )
Change subject: Clean up log config
......................................................................
Clean up log config
Make Default_Log_Level useful. Now if components are not specified, they will
be set to the Default_Log_Level. The Default_Log_Level is preset from the -N
option.
If component ALL is specified along with other components, any component not
explicitly mentioned will be set to the level ALL was specified as. If ALL is
removed, Default_Log_Level takes over, thus ALL can be unset also.
If a component was explicitly set, and then removed, the component will be
reset to Default_Log_Level.
Log level changes are always logged.
RPC_Debug_Level is cleaned up so it is set as now documented.
The logging documentation is improved.
Change-Id: I39648fa626b6307d47151ac171806e2b1aeb0edc
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
---
M src/config_samples/config.txt
M src/doc/man/ganesha-log-config.rst
M src/include/log.h
M src/log/log_functions.c
4 files changed, 136 insertions(+), 51 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/76/508876/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508876
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I39648fa626b6307d47151ac171806e2b1aeb0edc
Gerrit-Change-Number: 508876
Gerrit-PatchSet: 1
Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>
Gerrit-MessageType: newchange
3 years, 11 months
Change in ...nfs-ganesha[next]: Deprecate set log levels from environment
by Frank Filz (GerritHub)
Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508875 )
Change subject: Deprecate set log levels from environment
......................................................................
Deprecate set log levels from environment
JUSTIFICATION:
- I'm not sure anyone uses this
- It's confusing
- For early debug, using -N FULL_DEBUG is fine
- Using the LOG config is pretty easy
Change-Id: I24dafb24fc96bba5e3459cdc034a049cbc694ac2
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
---
M src/MainNFSD/nfs_init.c
M src/include/log.h
M src/log/log_functions.c
3 files changed, 0 insertions(+), 50 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/75/508875/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508875
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I24dafb24fc96bba5e3459cdc034a049cbc694ac2
Gerrit-Change-Number: 508875
Gerrit-PatchSet: 1
Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>
Gerrit-MessageType: newchange
3 years, 11 months
Change in ...nfs-ganesha[next]: Pull up ntirpc
by Daniel Gryniewicz (GerritHub)
Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508820 )
Change subject: Pull up ntirpc
......................................................................
Pull up ntirpc
Pull up fix for async I/O from FSALs
Change-Id: Ia8521c5cdaadcaaf1f2ba1d87a0bbbae0fb4e05c
Signed-off-by: Daniel Gryniewicz <dang(a)redhat.com>
---
M src/libntirpc
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/20/508820/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508820
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: Ia8521c5cdaadcaaf1f2ba1d87a0bbbae0fb4e05c
Gerrit-Change-Number: 508820
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com>
Gerrit-MessageType: newchange
3 years, 11 months
Issue with calling done_cb from a different thread for fsal_read2
by Chakra Divi
Hi all,
I'm working to create an async model in fsal layer for my filesystem. As part of this, called "done_cb" for a read_req from my own thread in fsal layer. This causes hang and nfs4 stopped responding. Here is pseudo code for the read callback. i could see a similar implementation in FSAL_MEM (mem_handle.c) layer
void fsal_read2_cb(void *cbArg, int err) {
struct test_wa *wa = (struct test_wa*)cbArg;
fsal_errors_t fsal_error = posix2fsal_error(err);
struct root_op_context root_op_context;
init_root_op_context(&root_op_context, wa->ctx_export, wa->fsal_export, 0, 0, UNKNOWN_REQUEST);
wa->done_cb(wa->obj_hdl, fsalstat(fsal_error, err), wa->read_arg, wa->caller_arg);
release_root_op_context();
free(wa);
return;
}
Shouldn't we call done_cb from a custom thread ? Can you please let me know if i'm missing something and right way to do this - Thanks for the help
Regards,
Chakra
3 years, 11 months
Change in ...nfs-ganesha[next]: Fix op_ctx handling for async FSALs
by Daniel Gryniewicz (GerritHub)
Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508819 )
Change subject: Fix op_ctx handling for async FSALs
......................................................................
Fix op_ctx handling for async FSALs
When we suspend a request, we cannot touch the contents of op_ctx,
because the callback may have already run, and may have freed the
request (and therefore the op_ctx). All we can do is NULL out op_ctx,
so that future calls don't get a save_op_ctx set wrongly.
Change-Id: I139f68032bfe3233e13b96c8e5bf544258681a68
Signed-off-by: Daniel Gryniewicz <dang(a)redhat.com>
---
M src/FSAL/commonlib.c
M src/MainNFSD/nfs_worker_thread.c
M src/Protocols/NFS/nfs4_Compound.c
3 files changed, 11 insertions(+), 4 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/19/508819/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508819
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I139f68032bfe3233e13b96c8e5bf544258681a68
Gerrit-Change-Number: 508819
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com>
Gerrit-MessageType: newchange
3 years, 11 months
Change in ...nfs-ganesha[next]: FSAL_MEM - Don't submit async without any threads
by Daniel Gryniewicz (GerritHub)
Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508818 )
Change subject: FSAL_MEM - Don't submit async without any threads
......................................................................
FSAL_MEM - Don't submit async without any threads
The error check handled the case, and inline writes occurred, but the
log was filled with error from fridgethr_submit()
Change-Id: I1d3110e7d2467b2131ea2e29e9490ce2909bc2c0
Signed-off-by: Daniel Gryniewicz <dang(a)redhat.com>
---
M src/FSAL/FSAL_MEM/mem_handle.c
1 file changed, 7 insertions(+), 5 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/18/508818/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508818
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I1d3110e7d2467b2131ea2e29e9490ce2909bc2c0
Gerrit-Change-Number: 508818
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com>
Gerrit-MessageType: newchange
3 years, 11 months
Announce Push of V4-dev.47
by Frank Filz
Branch next
Tag:V4-dev.47
Merge Highlights
* ProxyV3: Pass the *original* credentials to backends.
* FSAL_RGW - Fix version check
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
b32386c Frank S. Filz V4-dev.47
7dfacf4 Daniel Gryniewicz FSAL_RGW - Fix version check
a0fa07f Solomon Boulos ProxyV3: Pass the *original* credentials to backends.
3 years, 11 months