January 2021 - Devel - Nfs Ganesha List Archives

by Bjorn Leffler

Apart from the counters that you can access through dbus, is there any other monitoring built into Ganesha? I'm thinking of adding it with this higher level plan: - Exporting metrics from Ganesha to Prometheus. - Aggregate data in Prometheus. - Display monitoring consoles and graphs with Grafana. - Package up Prometheus, Grafana and the preconfigured rules/dashboards as a Docker image. - This makes it straightforward to deploy monitoring alongside a Gansha binary. My rough coding plan for the code is to: - Add a USE_MONITORING directive to the CMakeLists.txt files. - Add a build dependency to the Prometheus C client <https://github.com/digitalocean/prometheus-client-c>. - Create a src/monitoring directory for the new source files and templates. - Increment counters and timers throughout the code. - Use histograms to compute latency percentiles, heatmaps, etc. Is this a good idea? Any objections or suggestions? Thanks, Bjorn

1 year, 10 months

3
6
0 / 0

Issue with Ganesha not processing UDP NULL proc requests from AMD

by anirban.chalcogen＠gmail.com

Hi Daniel or Frank, We spoke about this issue on weekly call just over a week ago. I remember Daniel's advice on that call was to dig into the Ganesha core dump and try to extract the communication channel for each fd from the svc_rqst_epoll_loop() calls and verify that the UDP listen socket for NFS is being epoll'd on. I'm not sure however, the data is present in that stack that would actually allow us to make that call. (gdb) bt full #0 0x00007f3c46616eb3 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f3c4876f04c in svc_rqst_epoll_loop (sr_rec=0x4ae8c10) at /<redacted>/libntirpc/src/svc_rqst.c:1013 cc = 0x7f3c46d13eb4 <pthread_cond_timedwait@(a)GLIBC_2.3.2+516> n = 0x0 ts = {tv_sec = 2750901, tv_nsec = 954488941} timeout_ms = 29000 expire_ms = -1544065341 n_events = 32572 __func__ = "svc_rqst_epoll_loop" #2 0x00007f3c4876f199 in svc_rqst_run_task (wpe=0x4ae8c10) at /<redacted>libntirpc/src/svc_rqst.c:1065 sr_rec = 0x4ae8c10 finished = false __func__ = "svc_rqst_run_task" #3 0x00007f3c48777dca in work_pool_thread (arg=0x7f3c300008c0) at /<redacted>/libntirpc/src/work_pool.c:181 wpt = 0x7f3c300008c0 pool = 0x7f3c48993940 <svc_work_pool> have = 0x0 ts = {tv_sec = 1608803814, tv_nsec = 833467718} rc = 0 spawn = false __func__ = "work_pool_thread" #4 0x00007f3c46d0fea5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f3c466168dd in clone () from /lib64/libc.so.6 No symbol table info available. I am assuming that you would need to chenck the xp_type member in the SVCXPRT structure? However, the members of the latter are lost in the stack since during rearm, we obtain the sr_rec as follows: int svc_rqst_rearm_events(SVCXPRT *xprt) { struct rpc_dplx_rec *rec = REC_XPRT(xprt); struct svc_rqst_rec *sr_rec = (struct svc_rqst_rec *)rec->ev_p; Here, ev_p is a pointer and it seems to me we will not be able to obtain rec from *sr_rec with a container_of() kind macro since ev_p is not being referenced. Do you have another way to ascertain if the FDs polled have the UDP descriptor in them? Thanks, Anirban

2 years, 7 months

2
9
0 / 0

No merge this week - no submissions

by Frank Filz

There were no patch submissions for this week, so no merge. Frank

2 years, 7 months

1
0
0 / 0

Ganesha 3.5

by Daniel Gryniewicz

Since Ceph Pacific has come out, with the changes that break building Ganesha 3.4 and earlier against it, I've released a 3.5 with just the fix to build against Ceph Pacific. There's no associated ntirpc release. Daniel

2 years, 8 months

1
0
0 / 0

Announce Push of V4-dev.49

by Frank Filz

Branch next Tag:V4-dev.49 Merge Highlights * LOG config reload changes to be more sensible * LOG remove unused methods of log level setting/changing Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 4e0b839 Frank S. Filz V4-dev.49 3357036 Frank S. Filz Remove SIGUSR1 and SIGUSR2 log level manipulation 09d9624 Frank S. Filz Correct signal documentation for -h command line option d2c8b4f Frank S. Filz Clean up log config 157b6a0 Frank S. Filz Remove setting log levels from environment

2 years, 8 months

1
0
0 / 0

Issue with Ganesha not processing UDP NULL proc requests from AMD

by anirban.chalcogen＠gmail.com

Hi Daniel or Frank, We spoke of this issue a little over a week ago in our weekly calls. On our last call, we showed you the core backtrace that was obtained for Ganesha. You suggested that we might have to dig into the core dump to ascertain whether the UDP socket is being added to the set of FDs to be epoll'd. However, I'm not sure if that data is not lost in the epoll_wait() stack. Here's for example the full backtrace of a stack with epoll_wait(). (gdb) bt full #0 0x00007f3c46616eb3 in epoll_wait () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f3c4876f04c in svc_rqst_epoll_loop (sr_rec=0x4ae8c10) at /usr/src/debug/nfs-ganesha-2.7.5-ibm056.02/libntirpc/src/svc_rqst.c:1013 cc = 0x7f3c46d13eb4 <pthread_cond_timedwait@(a)GLIBC_2.3.2+516> n = 0x0 ts = {tv_sec = 2750901, tv_nsec = 954488941} timeout_ms = 29000 expire_ms = -1544065341 n_events = 32572 __func__ = "svc_rqst_epoll_loop" #2 0x00007f3c4876f199 in svc_rqst_run_task (wpe=0x4ae8c10) at /usr/src/debug/nfs-ganesha-2.7.5-ibm056.02/libntirpc/src/svc_rqst.c:1065 sr_rec = 0x4ae8c10 finished = false __func__ = "svc_rqst_run_task" #3 0x00007f3c48777dca in work_pool_thread (arg=0x7f3c300008c0) at /usr/src/debug/nfs-ganesha-2.7.5-ibm056.02/libntirpc/src/work_pool.c:181 wpt = 0x7f3c300008c0 pool = 0x7f3c48993940 <svc_work_pool> have = 0x0 ts = {tv_sec = 1608803814, tv_nsec = 833467718} rc = 0 spawn = false __func__ = "work_pool_thread" #4 0x00007f3c46d0fea5 in start_thread () from /lib64/libpthread.so.0 No symbol table info available. #5 0x00007f3c466168dd in clone () from /lib64/libc.so.6 No symbol table info available. (gdb) My idea was if we could trace back to the SVCXPRT instance from the *sr_rec because xp_type member in the former has the type of the transport channel. However, if you see how we obtain the sr_rec form the SVCXPRT, int svc_rqst_rearm_events(SVCXPRT *xprt) { struct rpc_dplx_rec *rec = REC_XPRT(xprt); struct svc_rqst_rec *sr_rec = (struct svc_rqst_rec *)rec->ev_p; int code = EINVAL; Here, the ev_p is not being referenced so we cannot use a container_of() kind macro to easily obtain the rec structure and trace outwards. Is there another way we could determine the transport channel from the core backtrace? Thanks, Anirban

2 years, 8 months

1
1
0 / 0

Announce Push of V4-dev.48

by Frank Filz

Branch next Tag:V4-dev.48 Merge Highlights * FSAL_MEM - Don't submit async without any threads * Fix op_ctx handling for async FSALs * Pull up ntirpc fix for async I/O from FSALs * Indicate USE_NFS_RDMA cmake option is not currently supported Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 9a728a0 Frank S. Filz V4-dev.48 a09c311 Frank S. Filz Indicate USE_NFS_RDMA cmake option is not currently supported 036638b Daniel Gryniewicz Pull up ntirpc 4cdea72 Daniel Gryniewicz Fix op_ctx handling for async FSALs 1911c75 Daniel Gryniewicz FSAL_MEM - Don't submit async without any threads

2 years, 8 months

1
0
0 / 0

Re: nfs4_op_readdir returning NFS4ERR_DELAY due to error in mdcache_avl_insert_ck

by Daniel Gryniewicz

Cookies need to not change. It doesn't matter what they are, they just need to be stable. If a cookie changes, the parent directory needs to be invalidated with an invalidate UP call, with the FSAL_UP_INVALIDATE_DIR_CHUNKS and FSAL_UP_INVALIDATE_DIR_POPULATED flags set. You can just send invalidate calls and leave your setup the same as it is now, but this will result in bad directory listing performance for large directories, since the cache will be constantly invalidated. A better solution is to come up with some form of cookie that is stable in the face of a remove. Storing the dirents in some form of tree with the tree's key used as the cookie would work (that's what we do for whence-is-name FSALs). Then again, performance of such a setup can't be great, so maybe it doesn't matter? Daniel On 1/14/21 9:03 AM, Sandeep Nashikkar via Devel wrote: > Hi > > We are having a samba server exporting a locally mounted ganesha share > and we have our own FSAL implementation. > > Samba share consistently get stuck during readdir operation on NFS > export in particular sequence of operations from samba on ganesha export. > > These are particular set of operations before it gets stuck: > > 1. There are around 160 files in the folder and readdir output looks like > . > .. > file1 > file2 > file3 > file4 > â€¦..and so on > 2. file2 and file3 in the previous readdir output are deleted. > 3. readdir_ex calls from FSAL now return following one by one > . > .. > file1 > file4 > 4. After it returns file4, readdir call from ganesha returns > NFS4ERR_DELAY to client with following error in ganesha logs: > > 12/01/2021 03:24:32 : epoch 5ffc6dd5 : <machine> : > ganesha.nfsd-20352[svc_168] mdcache_avl_insert_ck :NFS READDIR :DEBUG > :NFS READDIR: DEBUG: Already existent when inserting dirent > 0x7f0b6c0d8c40 for <filename> on entry=0x7f0b6802dfb0 FSAL cookie=4, > duplicated directory cookies make READDIR unreliable. > > 5. ls -lÂ command on the directory over local mount also gets stuck for > few mins. > 6. Previous readdir calls logs between (before files were deleted In > #2) reveal that the file4 had cookie=6 and thatâ€™s where ganesha > throws an error > 12/01/2021 03:24:08 : epoch 5ffc6dd5 : OneCSUnixMa.idcprodcert.loc : > ganesha.nfsd-20352[svc_173] mdcache_avl_insert_ck :NFS READDIR > :F_DBG :NFS READDIR: FULLDEBUG: Insert dirent 0x7f0b68020cb0 for > collection-101-3540931298411264490.wt on entry=0x7f0b6802dfb0 FSAL > cookie=6 > > So couple of questions: > > 1. What is the cookie value we should send to avoid this error > occurring during insertion in AVL tree? We currently return the > offset of the entry in cookie. > 2. Do we need to invalidate the mdcache of ganesha using up-call > whenever there is a change in the files inside the underlying dirent > due to rename or delete operation? > > Thanks > > Sandeep > > > _______________________________________________ > Devel mailing list -- devel(a)lists.nfs-ganesha.org > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org >

2 years, 8 months

1
0
0 / 0

Change in ...nfs-ganesha[next]: Remove SIGUSR1 and SIGUSR2 log level manipulation

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508918 ) Change subject: Remove SIGUSR1 and SIGUSR2 log level manipulation ...................................................................... Remove SIGUSR1 and SIGUSR2 log level manipulation I don't think these are used and it's much easier to use the LOG config and SIGHUP or DBUS. These signals are sort of confusing and a legacy from before Ganesha had good logging control. Change-Id: I324f406dd21bd068d1e5504ffd0ebe30676c31fe Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/log/log_functions.c 1 file changed, 6 insertions(+), 61 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/18/508918/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508918 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I324f406dd21bd068d1e5504ffd0ebe30676c31fe Gerrit-Change-Number: 508918 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 / 0

Change in ...nfs-ganesha[next]: Correct signal documentation for -h command line option

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508917 ) Change subject: Correct signal documentation for -h command line option ...................................................................... Correct signal documentation for -h command line option Change-Id: I19ff8034ffc41cbb7323653246174e68159535cc Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/MainNFSD/nfs_main.c 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/17/508917/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/508917 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I19ff8034ffc41cbb7323653246174e68159535cc Gerrit-Change-Number: 508917 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

2 years, 8 months

1
0
0 / 0

2023

2022

2021

2020

2019

2018

Devel January 2021