February 2020 - Devel - NFS Ganesha List Archives

Change in ...nfs-ganesha[next]: MDCACHE - Add MDCACHE {} config block
by Daniel Gryniewicz (GerritHub) 05 Nov '20

05 Nov '20

Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/454929 Change subject: MDCACHE - Add MDCACHE {} config block ...................................................................... MDCACHE - Add MDCACHE {} config block Add a config block name MDCACHE that is a copy of CACHEINODE. Both can be configured, but MDCACHE will override CACHEINODE. This allows us to deprecate CACHEINODE. Change-Id: I49012723132ae6105b904a60d1a96bb2bf78d51b Signed-off-by: Daniel Gryniewicz <dang(a)fprintf.net> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_read_conf.c M src/config_samples/ceph.conf M src/config_samples/config.txt M src/config_samples/ganesha.conf.example M src/doc/man/ganesha-cache-config.rst M src/doc/man/ganesha-config.rst 6 files changed, 31 insertions(+), 7 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/29/454929/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/454929 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I49012723132ae6105b904a60d1a96bb2bf78d51b Gerrit-Change-Number: 454929 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com> Gerrit-MessageType: newchange

3 2

lseek gets bad offset from nfs client with ganesha/gluster which supports SEEK
by Kinglong Mee 14 Sep '20

14 Sep '20

The latest ganesha/gluster supports seek according to, https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-41#section-15.11 From the given sa_offset, find the next data_content4 of type sa_what in the file. If the server can not find a corresponding sa_what, then the status will still be NFS4_OK, but sr_eof would be TRUE. If the server can find the sa_what, then the sr_offset is the start of that content. If the sa_offset is beyond the end of the file, then SEEK MUST return NFS4ERR_NXIO. For a file's filemap as, Part 1: HOLE 0x0000000000000000 ---> 0x0000000000600000 Part 2: DATA 0x0000000000600000 ---> 0x0000000000700000 Part 3: HOLE 0x0000000000700000 ---> 0x0000000001000000 SEEK(0x700000, SEEK_DATA) gets result (sr_eof:1, sr_offset:0x70000) from ganesha/gluster; SEEK(0x700000, SEEK_HOLE) gets result (sr_eof:0, sr_offset:0x70000) from ganesha/gluster. If an application depends the lseek result for data searching, it may enter infinite loop. while (1) { next_pos = lseek(fd, cur_pos, seek_type); if (seek_type == SEEK_DATA) { seek_type = SEEK_HOLE; } else { seek_type = SEEK_DATA; } if (next_pos == -1) { return ; cur_pos = next_pos; } The lseek syscall always gets 0x70000 from nfs client for those two cases, but, if underlying filesystem is ext4/f2fs, or the nfs server is knfsd, the lseek(0x700000, SEEK_DATA) gets ENXIO. I wanna to know, should I fix the ganesha/gluster as knfsd return ENXIO for the first case? or should I fix the nfs client to return ENXIO for the first case? thanks, Kinglong Mee

4 8

Crash in dupreq
by katcherw＠gmail.com 03 Apr '20

03 Apr '20

We saw a crash in version 2.8.2 running an iozone crash: Program terminated with signal 11, Segmentation fault. #0 0x00007ff028dce240 in atomic_sub_uint32_t (var=0x104, sub=1) at /usr/src/debug/nfs-ganesha-2.8.2/include/abstract_atomic.h:384 384 return __atomic_sub_fetch(var, sub, __ATOMIC_SEQ_CST); Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-12.el7.x86_64 dbus-libs-1.11.16-1.fc26.x86_64 elfutils-libelf-0.158-3.el7.x86_64 elfutils-libs-0.158-3.el7.x86_64 glibc-2.17-55.el7.x86_64 gssproxy-0.3.0-9.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-8.el7.x86_64 libacl-2.2.51-12.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libblkid-2.23.2-16.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-10.el7.x86_64 libgcrypt-1.5.3-4.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-9.el7.x86_64 libselinux-2.5-11.el7.x86_64 libuuid-2.23.2-16.el7.x86_64 libwbclient-4.8.3-4.el7.x86_64 pcre-8.32-12.el7.x86_64 samba-client-libs-4.8.3-4.el7.x86_64 samba-winbind-modules-4.8.3-4.el7.x86_64 sssd-client-1.11.2-65.el7.x86_64 systemd-libs-219-42.el7_4.1.x86_64 userspace-rcu-0.7.16-1.el7.x86_64 xz-libs-5.1.2-8alpha.el7.x86_64 zlib-1.2.7-13.el7.x86_64 (gdb) bt #0 0x00007ff028dce240 in atomic_sub_uint32_t (var=0x104, sub=1) at /usr/src/debug/nfs-ganesha-2.8.2/include/abstract_atomic.h:384 #1 0x00007ff028dce265 in atomic_dec_uint32_t (var=0x104) at /usr/src/debug/nfs-ganesha-2.8.2/include/abstract_atomic.h:405 #2 0x00007ff028dd1b65 in dupreq_entry_put (dv=0x0) at /usr/src/debug/nfs-ganesha-2.8.2/RPCAL/nfs_dupreq.c:912 #3 0x00007ff028dd4630 in nfs_dupreq_rele (req=0x7fec44a269f0, func=0x7ff029159d60 <nfs3_func_desc+672>) at /usr/src/debug/nfs-ganesha-2.8.2/RPCAL/nfs_dupreq.c:1382 #4 0x00007ff028d74bc5 in free_args (reqdata=0x7fec44a269f0) at /usr/src/debug/nfs-ganesha-2.8.2/MainNFSD/nfs_worker_thread.c:693 #5 0x00007ff028d77783 in nfs_rpc_process_request (reqdata=0x7fec44a269f0) at /usr/src/debug/nfs-ganesha-2.8.2/MainNFSD/nfs_worker_thread.c:1509 #6 0x00007ff028d77a74 in nfs_rpc_valid_NFS (req=0x7fec44a269f0) at /usr/src/debug/nfs-ganesha-2.8.2/MainNFSD/nfs_worker_thread.c:1601 #7 0x00007ff028b2632d in svc_vc_decode (req=0x7fec44a269f0) at /usr/src/debug/nfs-ganesha-2.8.2/libntirpc/src/svc_vc.c:829 #8 0x00007ff028b227bf in svc_request (xprt=0x7fee0c202ad0, xdrs=0x7fec3c987a70) at /usr/src/debug/nfs-ganesha-2.8.2/libntirpc/src/svc_rqst.c:793 #9 0x00007ff028b2623e in svc_vc_recv (xprt=0x7fee0c202ad0) at /usr/src/debug/nfs-ganesha-2.8.2/libntirpc/src/svc_vc.c:802 #10 0x00007ff028b22740 in svc_rqst_xprt_task (wpe=0x7fee0c202cf0) at /usr/src/debug/nfs-ganesha-2.8.2/libntirpc/src/svc_rqst.c:774 #11 0x00007ff028b23048 in svc_rqst_epoll_loop (wpe=0x108d570) at /usr/src/debug/nfs-ganesha-2.8.2/libntirpc/src/svc_rqst.c:1089 #12 0x00007ff028b2baff in work_pool_thread (arg=0x7fed44052070) at /usr/src/debug/nfs-ganesha-2.8.2/libntirpc/src/work_pool.c:184 #13 0x00007ff0270c8df3 in start_thread () from /lib64/libpthread.so.0 #14 0x00007ff0267cd3dd in clone () from /lib64/libc.so.6 (gdb) frame 5 #5 0x00007ff028d77783 in nfs_rpc_process_request (reqdata=0x7fec44a269f0) at /usr/src/debug/nfs-ganesha-2.8.2/MainNFSD/nfs_worker_thread.c:1509 1509 free_args(reqdata); (gdb) p *reqdata $2 = { svc = { rq_xprt = 0x7fee0c202ad0, rq_clntname = 0x0, rq_svcname = 0x0, rq_xdrs = 0x7fec3c987a70, rq_u1 = 0x0, rq_u2 = 0x0, rq_cksum = 14781944753697519387, rq_auth = 0x7ff028d45cd0 <svc_auth_none>, rq_ap1 = 0x0, rq_ap2 = 0x0, ... Notice rq_u1 == 0. nfs_rpc_process_request is calling free_args which is calling nfs_dupreq_rele. There is a comment at the beginning of nfs_dupreq_rele: * We assert req->rq_u1 now points to the corresponding duplicate request * cache entry (dv). But this assumption doesn't seem to be true, and eventually this will be dereferenced leading to a crash. We're trying to reproduce the problem with NIV_FULL_DEBUG. Has anyone seen this problem before, or know what would cause rq_u1 to be 0? Should there be a check for NULL in nfs_dupreq_rele in addition to DUPREQ_NOCACHE? Bill

6 7

NFSv3 Filebench tests hang
by des＠vmware.com 04 Mar '20

04 Mar '20

We are trying to execute Filebench tests with NFS Ganesha and it most of the time hangs (filebench process stuck in D state) and does not complete the test. I have observed the following errors in ganesha.log: 2020-02-25T02:43:49Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_598] 405 :fsal_close :FSAL :open_fd_count is negative: -1 2020-02-25T02:46:27Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[::ffff:172.30.0.111] [svc_731] 2144 :mdcache_lru_fds_available :INODE LRU :FD Hard Limit Exceeded, waking LRU thread. 2020-02-25T02:46:28Z : epoch 5e543710 : fsvm23 : ganesha.nfsd-34[none] [cache_lru] 1388 :lru_run :INODE LRU :Futility count exceeded. Client load is opening FDs faster than the LRU thread can close them. I was searching in the ganesha community for any similar issues and found these: https://bugzilla.redhat.com/show_bug.cgi?id=1713261 which is fixed by Frank in : https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/455570/ This patch is not in present in our repo yet and I was thinking of cherry-picking this change into our FSAL_VFS which is causing the issue. Frank/Soumya, Can you confirm if this fix is going to resolve this issue and if I cherry pick this alone or is there other changes required to resolve this issue please?

5 6

Crash related to avltree_remove(..) with 2.7.6 when cleaning a directory chunk
by Madhu P Punjabi 03 Mar '20

03 Mar '20

2 2

nfs-ganesha exits with "Error: couldn't complete write to the log file " error on screen
by Satish Chandra Kilaru 03 Mar '20

03 Mar '20

Error: couldn't complete write to the log file /tmp/ganesha.log status=13 (Permission denied) message was: 27/02/2020 13:37:28 : epoch 5e583698 : m4hcadev1.commvault.com : ganesha.nfsd-32262[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha, built at Oct 29 2019 17:39:50 on localhost it successfully created ganesha.log as below. But failed to write to it with Permission Denied error. -rw-rw-r--. 1 root root 0 Feb 27 13:31 ganesha.log What could be the problem?

2 3

Announce Push of V4-dev.7
by Frank Filz 27 Feb '20

27 Feb '20

Branch next Tag:V4-dev.7 Merge Highlights * main(){fatal_die: } - Fix leaks in nfs_main.c. * Fix ambiguous description in nfs_init.h. * Allow to build with DBUS an no GSSAPI * Provision to get MDCACHE resource utilization Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 869c6f8 Frank S. Filz V4-dev.7 b5428a8 Sachin Punadikar Provision to get MDCACHE resource utilization c95574f Daniel Gryniewicz Allow to build with DBUS an no GSSAPI 6d7feb3 Xi Jinyu Fix ambiguous description in nfs_init.h. 0660aea Xi Jinyu main(){fatal_die: } - Fix leaks in nfs_main.c.

1 0

Change in ...nfs-ganesha[next]: Supporting protocol update as part of UpdateExport
by Sriram Patil (GerritHub) 27 Feb '20

27 Feb '20

Sriram Patil has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/486007 ) Change subject: Supporting protocol update as part of UpdateExport ...................................................................... Supporting protocol update as part of UpdateExport UpdateExport DBUS does not handle protocol update. For example, when the protocol is changes from NFSv3 to NFSv4 for an export, after UpdateExport none of hte mounts worked. This fix allows updating the protocols as part of UpdateExport. Updating to NFSv4 was the problem because the new path was not added to pseudo FS Change-Id: I6a78ec85fd2a48064161420d9459ecedebfad68f Signed-off-by: Sriram Patil <sriramp(a)vmware.com> --- M src/include/export_mgr.h M src/support/export_mgr.c M src/support/exports.c 3 files changed, 44 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/07/486007/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/486007 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I6a78ec85fd2a48064161420d9459ecedebfad68f Gerrit-Change-Number: 486007 Gerrit-PatchSet: 1 Gerrit-Owner: Sriram Patil <sriramp(a)vmware.com> Gerrit-MessageType: newchange

1 0

回复：Re: ganesha hang more than five hours
by QR 26 Feb '20

26 Feb '20

Not yet. Because the docker did not enable core dump. Will try to create a full backtrace for this, thanks. -------------------------------- ----- 原始邮件 ----- 发件人：Daniel Gryniewicz <dgryniew(a)redhat.com> 收件人：devel(a)lists.nfs-ganesha.org 主题：[NFS-Ganesha-Devel] Re: ganesha hang more than five hours 日期：2020年02月25日 21点20分 No, definitely not a known issue. Do you have a full backtrace of one (or several, if they're different) hung threads? Daniel On 2/24/20 9:28 PM, QR wrote: > Hi Dang, > > Ganesha hangs more than five hours. It seems that 198 svc threads hang > on nfs3_readdirplus. > Is there a known issue about this? Thanks in advance. > > Ganesha server info > ganesha version: V2.7.6 > FSAL : In house > nfs client info > nfs version : nfs v3 > client info : CentOS 7.4 > > _______________________________________________ > Devel mailing list -- devel(a)lists.nfs-ganesha.org > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org

1 0

ganesha hang more than five hours
by QR 25 Feb '20

25 Feb '20

Hi Dang, Ganesha hangs more than five hours. It seems that 198 svc threads hang on nfs3_readdirplus.Is there a known issue about this? Thanks in advance. Ganesha server info ganesha version: V2.7.6 FSAL : In housenfs client info nfs version : nfs v3 client info : CentOS 7.4

2 1