August 2019 - Devel - NFS Ganesha List Archives

Change in ...nfs-ganesha[next]: MDCACHE - Add MDCACHE {} config block
by Daniel Gryniewicz (GerritHub) 05 Nov '20

05 Nov '20

Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/454929 Change subject: MDCACHE - Add MDCACHE {} config block ...................................................................... MDCACHE - Add MDCACHE {} config block Add a config block name MDCACHE that is a copy of CACHEINODE. Both can be configured, but MDCACHE will override CACHEINODE. This allows us to deprecate CACHEINODE. Change-Id: I49012723132ae6105b904a60d1a96bb2bf78d51b Signed-off-by: Daniel Gryniewicz <dang(a)fprintf.net> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_read_conf.c M src/config_samples/ceph.conf M src/config_samples/config.txt M src/config_samples/ganesha.conf.example M src/doc/man/ganesha-cache-config.rst M src/doc/man/ganesha-config.rst 6 files changed, 31 insertions(+), 7 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/29/454929/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/454929 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I49012723132ae6105b904a60d1a96bb2bf78d51b Gerrit-Change-Number: 454929 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com> Gerrit-MessageType: newchange

3 2

lseek gets bad offset from nfs client with ganesha/gluster which supports SEEK
by Kinglong Mee 14 Sep '20

14 Sep '20

The latest ganesha/gluster supports seek according to, https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-41#section-15.11 From the given sa_offset, find the next data_content4 of type sa_what in the file. If the server can not find a corresponding sa_what, then the status will still be NFS4_OK, but sr_eof would be TRUE. If the server can find the sa_what, then the sr_offset is the start of that content. If the sa_offset is beyond the end of the file, then SEEK MUST return NFS4ERR_NXIO. For a file's filemap as, Part 1: HOLE 0x0000000000000000 ---> 0x0000000000600000 Part 2: DATA 0x0000000000600000 ---> 0x0000000000700000 Part 3: HOLE 0x0000000000700000 ---> 0x0000000001000000 SEEK(0x700000, SEEK_DATA) gets result (sr_eof:1, sr_offset:0x70000) from ganesha/gluster; SEEK(0x700000, SEEK_HOLE) gets result (sr_eof:0, sr_offset:0x70000) from ganesha/gluster. If an application depends the lseek result for data searching, it may enter infinite loop. while (1) { next_pos = lseek(fd, cur_pos, seek_type); if (seek_type == SEEK_DATA) { seek_type = SEEK_HOLE; } else { seek_type = SEEK_DATA; } if (next_pos == -1) { return ; cur_pos = next_pos; } The lseek syscall always gets 0x70000 from nfs client for those two cases, but, if underlying filesystem is ext4/f2fs, or the nfs server is knfsd, the lseek(0x700000, SEEK_DATA) gets ENXIO. I wanna to know, should I fix the ganesha/gluster as knfsd return ENXIO for the first case? or should I fix the nfs client to return ENXIO for the first case? thanks, Kinglong Mee

4 8

Re: [Nfs-ganesha-devel] 2.7.3 with CEPH_FSAL Crashing
by Daniel Gryniewicz 10 Oct '19

10 Oct '19

This is not one I've seen before, and a quick look at the code looks strange. The only assert in that bit is asserting the parent is a directory, but the parent directory is not something that was passed in by Ganesha, but rather something that was looked up internally in libcephfs. This is beyond my expertise, at this point. Maybe some ceph logs would help? Daniel On 7/15/19 10:54 AM, David C wrote: > This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. > > > Hi All > > I'm running 2.7.3 using the CEPH FSAL to export CephFS (Luminous), it > ran well for a few days and crashed. I have a coredump, could someone > assist me in debugging this please? > > (gdb) bt > #0 0x00007f04dcab6207 in raise () from /lib64/libc.so.6 > #1 0x00007f04dcab78f8 in abort () from /lib64/libc.so.6 > #2 0x00007f04d2a9d6c5 in ceph::__ceph_assert_fail(char const*, char > const*, int, char const*) () from /usr/lib64/ceph/libceph-common.so.0 > #3 0x00007f04d2a9d844 in ceph::__ceph_assert_fail(ceph::assert_data > const&) () from /usr/lib64/ceph/libceph-common.so.0 > #4 0x00007f04cc807f04 in Client::_lookup_name(Inode*, Inode*, UserPerm > const&) () from /lib64/libcephfs.so.2 > #5 0x00007f04cc81c41f in Client::ll_lookup_inode(inodeno_t, UserPerm > const&, Inode**) () from /lib64/libcephfs.so.2 > #6 0x00007f04ccadbf0e in create_handle (export_pub=0x1baff10, > desc=<optimized out>, pub_handle=0x7f0470fd4718, > attrs_out=0x7f0470fd4740) at > /usr/src/debug/nfs-ganesha-2.7.3/FSAL/FSAL_CEPH/export.c:256 > #7 0x0000000000523895 in mdcache_locate_host (fh_desc=0x7f0470fd4920, > export=export@entry=0x1bafbf0, entry=entry@entry=0x7f0470fd48b8, > attrs_out=attrs_out@entry=0x0) > at > /usr/src/debug/nfs-ganesha-2.7.3/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1011 > #8 0x000000000051d278 in mdcache_create_handle (exp_hdl=0x1bafbf0, > fh_desc=<optimized out>, handle=0x7f0470fd4900, attrs_out=0x0) at > /usr/src/debug/nfs-ganesha-2.7.3/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1578 > #9 0x000000000046d404 in nfs4_mds_putfh > (data=data@entry=0x7f0470fd4ea0) at > /usr/src/debug/nfs-ganesha-2.7.3/Protocols/NFS/nfs4_op_putfh.c:211 > #10 0x000000000046d8e8 in nfs4_op_putfh (op=0x7f03effaf1d0, > data=0x7f0470fd4ea0, resp=0x7f03ec1de1f0) at > /usr/src/debug/nfs-ganesha-2.7.3/Protocols/NFS/nfs4_op_putfh.c:281 > #11 0x000000000045d120 in nfs4_Compound (arg=<optimized out>, > req=<optimized out>, res=0x7f03ec1de9d0) at > /usr/src/debug/nfs-ganesha-2.7.3/Protocols/NFS/nfs4_Compound.c:942 > #12 0x00000000004512cd in nfs_rpc_process_request > (reqdata=0x7f03ee5ed4b0) at > /usr/src/debug/nfs-ganesha-2.7.3/MainNFSD/nfs_worker_thread.c:1328 > #13 0x0000000000450766 in nfs_rpc_decode_request (xprt=0x7f02180c2320, > xdrs=0x7f03ec568ab0) at > /usr/src/debug/nfs-ganesha-2.7.3/MainNFSD/nfs_rpc_dispatcher_thread.c:1345 > #14 0x00007f04df45d07d in svc_rqst_xprt_task (wpe=0x7f02180c2538) at > /usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:769 > #15 0x00007f04df45d59a in svc_rqst_epoll_events (n_events=<optimized > out>, sr_rec=0x4bb53e0) at > /usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:941 > #16 svc_rqst_epoll_loop (sr_rec=<optimized out>) at > /usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:1014 > #17 svc_rqst_run_task (wpe=0x4bb53e0) at > /usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:1050 > #18 0x00007f04df465123 in work_pool_thread (arg=0x7f044c0008c0) at > /usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/work_pool.c:181 > #19 0x00007f04dda05dd5 in start_thread () from /lib64/libpthread.so.0 > #20 0x00007f04dcb7dead in clone () from /lib64/libc.so.6 > > Package versions: > > nfs-ganesha-2.7.3-0.1.el7.x86_64 > nfs-ganesha-ceph-2.7.3-0.1.el7.x86_64 > libcephfs2-14.2.1-0.el7.x86_64 > librados2-14.2.1-0.el7.x86_64 > > I notice in my Ceph log I have a bunch of slow requests around the time > it went down, I'm not sure if it's a symptom of Ganesha segfaulting or > if it was a contributing factor. > > Thanks, > David > > > _______________________________________________ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >

4 9

GPFS LogCrit on lock_op2
by Frank Filz 20 Sep '19

20 Sep '19

On the call, I mentioned I would look at bypassing permission check for the file owner for the open_func call in fsal_find_fd with open_for_locks. It turns out there is a difference between FSAL_GPFS and FSAL_VFS FSAL_VFS makes the ultimate call to open_by_handle as root, and therefor even a non-owner of the file will not be an issue in opening the file read/write. GPFS calls GPFSFSAL_open which calls fsal_set_credentials so if the permissions do not allow read/write when open_for_locks occurs, then the file will instead be opened in the same mode as the OPEN stateid. I think it would be good to evaluate when GPFSFSAL_open actually needs to be called, and whether open_func should make a more direct call to fsal_internal_handle2fd. Frank

3 4

Announce Push of V2.9-dev.6
by Frank Filz 30 Aug '19

30 Aug '19

Branch next Tag:V2.9-dev.6 Release Highlights * 3 updates relating to rados-grace * GPFS: fix alloc_handle to not access unintilaized space * During IP giveback mark all clients as stale * cmake: include CheckSymbolExists on cmake-3.15.x Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: c41f628 Frank S. Filz V2.9-dev.6 f721168 Kaleb S. KEITHLEY cmake: include CheckSymbolExists on cmake-3.15.x 128be7f Suhrud Patankar During IP giveback mark all clients as stale 9b66e95 Malahal Naineni GPFS: fix alloc_handle to not access unintilaized space f1f5be2 Jeff Layton rados_grace: remove superfluous warning in rados_grace_dump d3ba007 Jeff Layton ganesha-rados-grace: add --cephconf and --userid command-line options 146dada Jeff Layton doc: clarify that ganesha-rados-grace does not consult ganesha.conf

1 0

Change in ...nfs-ganesha[next]: cmake: include CheckSymbolExists on cmake-3.15.x
by Kaleb KEITHLEY (GerritHub) 30 Aug '19

30 Aug '19

Kaleb KEITHLEY has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/466942 ) Change subject: cmake: include CheckSymbolExists on cmake-3.15.x ...................................................................... cmake: include CheckSymbolExists on cmake-3.15.x check_symbol_exists must be explicitly included starting in 3.15.0 see https://github.com/nfs-ganesha/nfs-ganesha/issues/454 and see https://github.com/nfs-ganesha/nfs-ganesha/pull/455 N.B. At the time of this writing, even bleeding edge distributions like fedora-32/rawhide, Ubuntu eoan (19/10), and Debian bullseye/sid are still using cmake-3.14.x or cmake-3.13.x. Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> Change-Id: I8fbd2ecd8b6430de12e663827027f36ff7743ad3 --- M src/cmake/modules/FindCEPHFS.cmake 1 file changed, 3 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/42/466942/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/466942 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I8fbd2ecd8b6430de12e663827027f36ff7743ad3 Gerrit-Change-Number: 466942 Gerrit-PatchSet: 1 Gerrit-Owner: Kaleb KEITHLEY <kaleb(a)redhat.com> Gerrit-MessageType: newchange

1 0

No community call next week
by Frank Filz 29 Aug '19

29 Aug '19

Let's skip the community call next week. I realized that everything we will be doing to get our oldest off to her first day of kindergarten will make it hard for me to break away for even a shortened call. Catch you all online (I'll be online within 30 minutes after the normal end time of the call - i.e. about 8:30 PDT). Thanks Frank

1 0

request from clients
by Alok Sinha 28 Aug '19

28 Aug '19

I am new to Ganesha and file system. I seeing a problem and need advice. On a client machine , I fork multiple processes each reading a different file in a same directory. I expect that Ganesha will see requests for files coming in parallel. In real world, I see that fsal_lookup sees one file at a time. Is this expected? For Ganesha, i see that requests are serial while I expect or rather want to force parallel. -alok

3 3

nfs-ganesha clients freeze, nfs ganesha daemon dies
by Erik Jacobson 28 Aug '19

28 Aug '19

Hello. I'm statring a new thread for this problem. I have a 3x3 Gluster Volume and I'm trying to use Ganesha for NFS services. One of the 9 server nodes, I have enabled Ganesha NFS server on one. The volume is being used being host clients with NFS roots. A long separate thread shows how I got to this point but what works on the client side is: RHEL 7.6 aarch64 4.14.0-115.el7a.aarch64 RHEL 7.6 x86_64 3.10.0-957.el7.x86_64 OverlayFS - NFS v4 underdir with TMPFS overlay. The Ganesha server has Allow_Numeric_Owners = True; Only_Numeric_Owners = True; Disable_ACL = TRUE; Disable_ACL is required for the aarch64 overlay to properly read non-root files. (However, Disable_ACL must be false for aarch64 if you are using NFS v3 strangely). The x86_64 node fully boots through full init/systemd startup to the login prompt. When I startup the aarch64 node, it gets various degrees of done... then both NFS clients freeze up 100%. Restarting nfs-ganesha gets them going for a moment, then they freeze again. It turned out in some cases the nfs-ganesha daemon was present during the freeze but no longer serving the nodes. However, a more common case (and the captured one) is nfs-ganesha is gone. I will attach a tarball with a bunch of information on the problem including the config file I used, debugging logs, and some traces. Ganesha 2.8.2 - Ganesha, Gluster servers x86_64 Since the aarch64 node causes Ganesha to crash early, and the debug log can get to 2GB quickly, I set up a test case as follows: Tracing starts... - x86_64 fully nfs-root-booted, it comes up fine. * Actively using nfs for root during tests below - aarch64 node - boot to the miniroot env (a "fat" initrd that has more tools and from which we do the NFS mount) - It stops before switching control to the init start to run the tests like below. - cp'd /dev/null to ganesha log here - started the tcpdump to the problem node - Ran the following. Ganesha died at 'wc -l', also notice the Input/Output error on the first attempt: bash-4.2# bash reset4.sh + umount /a umount: /a: not mounted + umount /root_ro_nfs umount: /root_ro_nfs: not mounted + umount /rootfs.rw + mount -o ro,nolock 172.23.255.249:/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel /root_ro_nfs + mount -t tmpfs -o mpol=interleave tmpfs /rootfs.rw + mkdir /rootfs.rw/upperdir + mkdir /rootfs.rw/work + mount -t overlay overlay -o lowerdir=/root_ro_nfs,upperdir=/rootfs.rw/upperdir,workdir=/rootfs.rw/work /a bash-4.2# chroot /a chroot: failed to run command '/bin/sh': Input/output error bash-4.2# chroot /a sh: no job control in this shell sh-4.2# ls /usr/bin|wc -l - When the above froze and ganesha died, I stopped tcpdump and collected the pieces in to a tarball. See attached. Erik

5 14

Change in ...nfs-ganesha[next]: rados_grace: remove superfluous warning in rados_grace_dump
by Jeff Layton (GerritHub) 28 Aug '19

28 Aug '19

Jeff Layton has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/466614 ) Change subject: rados_grace: remove superfluous warning in rados_grace_dump ...................................................................... rados_grace: remove superfluous warning in rados_grace_dump This routine is only called from ganesha-rados-grace and when the read operation fails, this just clutters up the output. The caller should be responsible for handling this error. Change-Id: I7a522e488750a9ae7610e50756107820c7e5590f Signed-off-by: Jeff Layton <jlayton(a)redhat.com> --- M src/support/rados_grace.c 1 file changed, 1 insertion(+), 3 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/14/466614/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/466614 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I7a522e488750a9ae7610e50756107820c7e5590f Gerrit-Change-Number: 466614 Gerrit-PatchSet: 1 Gerrit-Owner: Jeff Layton <jlayton(a)redhat.com> Gerrit-MessageType: newchange

1 0