September 2018 - Devel - NFS Ganesha List Archives

lseek gets bad offset from nfs client with ganesha/gluster which supports SEEK
by Kinglong Mee 14 Sep '20

14 Sep '20

The latest ganesha/gluster supports seek according to, https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-41#section-15.11 From the given sa_offset, find the next data_content4 of type sa_what in the file. If the server can not find a corresponding sa_what, then the status will still be NFS4_OK, but sr_eof would be TRUE. If the server can find the sa_what, then the sr_offset is the start of that content. If the sa_offset is beyond the end of the file, then SEEK MUST return NFS4ERR_NXIO. For a file's filemap as, Part 1: HOLE 0x0000000000000000 ---> 0x0000000000600000 Part 2: DATA 0x0000000000600000 ---> 0x0000000000700000 Part 3: HOLE 0x0000000000700000 ---> 0x0000000001000000 SEEK(0x700000, SEEK_DATA) gets result (sr_eof:1, sr_offset:0x70000) from ganesha/gluster; SEEK(0x700000, SEEK_HOLE) gets result (sr_eof:0, sr_offset:0x70000) from ganesha/gluster. If an application depends the lseek result for data searching, it may enter infinite loop. while (1) { next_pos = lseek(fd, cur_pos, seek_type); if (seek_type == SEEK_DATA) { seek_type = SEEK_HOLE; } else { seek_type = SEEK_DATA; } if (next_pos == -1) { return ; cur_pos = next_pos; } The lseek syscall always gets 0x70000 from nfs client for those two cases, but, if underlying filesystem is ext4/f2fs, or the nfs server is knfsd, the lseek(0x700000, SEEK_DATA) gets ENXIO. I wanna to know, should I fix the ganesha/gluster as knfsd return ENXIO for the first case? or should I fix the nfs client to return ENXIO for the first case? thanks, Kinglong Mee

4 8

IO Error on NFS v4.0 but not on v4.1
by Boris Faure 05 Oct '18

05 Oct '18

Hi everyone, I'm running a simple test with ganesha-next (but the issue exists at least in latest V2.5.5-stable): $ sudo mount -t nfs -o v4.0,users,nolock,soft,timeo=2,retrans=3 127.0.0.1:/ /mnt/nfs $ dd if=/dev/zero of=/mnt/nfs/dump/pw.1 bs=1024k count=2048 conv=fdatasync dd: error writing '/mnt/nfs/dump/pw.1': Input/output error 1616+0 records in 1615+0 records out 1693450240 bytes (1,7 GB, 1,6 GiB) copied, 8,28076 s, 205 MB/s $ sudo mount -t nfs -o users,nolock,soft,timeo=2,retrans=3 127.0.0.1:/ /mnt/nfs $ dd if=/dev/zero of=/mnt/nfs/dump/pw.1 bs=1024k count=2048 conv=fdatasync 2048+0 records in 2048+0 records out 2147483648 bytes (2,1 GB, 2,0 GiB) copied, 6,27708 s, 342 MB/s The second mount is using nfs v4.1. I don't have anything special in my configuration. If I do the dd with a blocksize of 512k, it works fine. If I set rsize/wsize to 524288 as mount options, the dd with bs=1024k still fails. The only log line I get is the following: 25/09/2018 13:47:25 : epoch 5baa2036 : caipirinha : ganesha.nfsd-2381[svc_182] rpc :TIRPC :EVENT :svc_ioq_flushv() writev failed (104) Thanks for your help. Best Regards -- Boris Faure Software Engineer -- <https://www.scality.com/download-idc-marketscape/?utm_source=ScalityEmail&u…>

4 9

Change in ffilz/nfs-ganesha[next]: FSAL_GLUSTER: set "cache-posix-acl" option when export supports acl
by GerritHub 29 Sep '18

29 Sep '18

From Kinglong Mee <kinglongmee(a)gmail.com>: Kinglong Mee has uploaded this change for review. ( https://review.gerrithub.io/427305 Change subject: FSAL_GLUSTER: set "cache-posix-acl" option when export supports acl ...................................................................... FSAL_GLUSTER: set "cache-posix-acl" option when export supports acl Change-Id: I9d1b9fb22a30b44f1dbaa90e1f7e5f342a6bb0f3 Signed-off-by: Kinglong Mee <mijinlong(a)open-fs.com> --- M src/FSAL/FSAL_GLUSTER/export.c 1 file changed, 11 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/05/427305/1 -- To view, visit https://review.gerrithub.io/427305 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I9d1b9fb22a30b44f1dbaa90e1f7e5f342a6bb0f3 Gerrit-Change-Number: 427305 Gerrit-PatchSet: 1 Gerrit-Owner: Kinglong Mee <kinglongmee(a)gmail.com>

1 0

Change in ffilz/nfs-ganesha[next]: namespace cleanup: some poorly named and/or non-static symbols
by GerritHub 28 Sep '18

28 Sep '18

From <kaleb(a)redhat.com>: kaleb(a)redhat.com has uploaded this change for review. ( https://review.gerrithub.io/427287 Change subject: namespace cleanup: some poorly named and/or non-static symbols ...................................................................... namespace cleanup: some poorly named and/or non-static symbols In Gluster are seeing a collision between the variable named 'options' in the gluster xlators and the variable named 'options' in ganesha.nfsd. ISTR a recent change made in Fedora to RT linker that affect this but am unable to find a reference to the discussion at this time. It is unclear to me whether this is a side effect of the introduction of the -z defs linker flags in Fedora 28. This probably affects RHEL 8 too. In a nutshell, references within a gluster xlator .so used to always be resolved to the "closest" definition and/or resolved at static link time; either way to the one in the xlator .so. Now though we see them resolved at runtime to the first definition of the symbol; in this case the one in ganesha.nfsd. In addition to 'options' there are a number of other variables in ganesha.nfsd that a) should be static, or b) should be renamed so as not to pollute the namespace. I picked a few that stood out. There may be others. Note too that gluster is working to clean up its own scribbling in the namespace. And, e.g., 'options' are exported from xlator .so files so that each xlators' options can be enumerated for display. Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> Change-Id: I0a5893ac4d54fce84e71f38997e1370f39d42ad3 --- M src/FSAL/FSAL_PROXY/handle.c M src/MainNFSD/9p_dispatcher.c M src/MainNFSD/9p_rdma_callbacks.c M src/MainNFSD/nfs_admin_thread.c M src/MainNFSD/nfs_init.c M src/MainNFSD/nfs_lib.c M src/MainNFSD/nfs_main.c M src/MainNFSD/nfs_rpc_callback.c M src/MainNFSD/nfs_rpc_dispatcher_thread.c M src/MainNFSD/nfs_worker_thread.c M src/Protocols/NFS/nfs4_Compound.c M src/SAL/nfs4_clientid.c M src/SAL/nfs4_state_id.c M src/include/nfs_core.h M src/include/nfs_lib.h M src/log/log_functions.c M src/support/client_mgr.c M src/support/export_mgr.c M src/support/server_stats.c 19 files changed, 114 insertions(+), 109 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/87/427287/1 -- To view, visit https://review.gerrithub.io/427287 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I0a5893ac4d54fce84e71f38997e1370f39d42ad3 Gerrit-Change-Number: 427287 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <kaleb(a)redhat.com>

1 0

Change in ffilz/nfs-ganesha[next]: FSAL_GLUSTER: for seek2, offset == filesize is "beyond the end of file"
by GerritHub 28 Sep '18

28 Sep '18

From Frank Filz <ffilzlnx(a)mindspring.com>: Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/427278 Change subject: FSAL_GLUSTER: for seek2, offset == filesize is "beyond the end of file" ...................................................................... FSAL_GLUSTER: for seek2, offset == filesize is "beyond the end of file" Change-Id: I5df7fdd166f1ec929033717c11575a65f7a8c9f2 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/FSAL_GLUSTER/handle.c 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/78/427278/1 -- To view, visit https://review.gerrithub.io/427278 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I5df7fdd166f1ec929033717c11575a65f7a8c9f2 Gerrit-Change-Number: 427278 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>

1 0

Scalability issue with VFS FSAL and large amounts of read i/o in flight
by Kropelin, Adam 28 Sep '18

28 Sep '18

Hello, I am observing a scalability issue with recent-ish versions of nfs-ganesha (including -next) when NFS clients have a significant amount of in-flight read requests. My test setup has a ganesha server with a single export on the VFS FSAL. I have multiple Linux clients, all mounting that export with NFSv4.0. On the clients I run a simple read workload using dd: 'dd if=/mnt/test/testfile of=/dev/null bs=1M'. All clients read the same 1 GB file. Each client is bandwidth-limited to 1 Gbps while the server has 10 Gbps available. A single client achieves ~100 MB/sec. Adding a second client brings the aggregate throughput up to ~120 MB/sec. A third client gets the aggregate to ~130 MB/sec, and it pretty much plateaus at that point. Clearly this is well below the aggregate bandwidth the server is capable of. Additionally, and this is the behavior that made me originally discover this issue in production, while the clients are performing their read test, the server becomes extremely slow to respond to mount requests. By "extremely slow" I mean it takes 60 seconds or more to perform a simple mount while 8 clients are running the read test. I've ruled out external bottlenecks -- disk i/o on the server is essentially zero during the test (as would be expected since that 1 GB file will most certainly be in page cache). The server shows no significant CPU load at all. Using the in-kernel NFS server with the same clients I can easily saturate the 10 Gpbs network link from 8-10 clients with no effect on mount times, so network is not a bottleneck here. Other things of interest: * -next and V2.5 both exhibit the issue, but V2.2 does not * By observation on the wire I see that the Linux NFS client is submitting 16 or more 1 MB READ RPCs at once. If I prevent that behavior by adding 'iflag=direct' to the dd command, suddenly scalability is back where it should be. Something about having a lot of read i/o in flight seems to matter here. * I grabbed several core dumps of ganesha during a period where 8 clients were hitting it. Every single thread is idle (typically pthread_cond_wait'ing for some work) except for one rpc worker which is in writev. This is true repeatedly throughout the test. It is as if somehow a single rpc worker thread is doing all of the network i/o to every client. Thanks in advance for any ideas... --Adam

8 36

Change in ffilz/nfs-ganesha[next]: Usage: fix typo '<epoch<'
by GerritHub 27 Sep '18

27 Sep '18

From <zhbingyin(a)sina.com>: zhbingyin(a)sina.com has uploaded this change for review. ( https://review.gerrithub.io/427181 Change subject: Usage: fix typo '<epoch<' ...................................................................... Usage: fix typo '<epoch<' Change-Id: Ifade2d3c60b1f6aebb14571c18ece25fd68d2a03 Signed-off-by: Bingyin Zhang <zhbingyin(a)sina.com> --- M src/MainNFSD/nfs_main.c 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/81/427181/1 -- To view, visit https://review.gerrithub.io/427181 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Ifade2d3c60b1f6aebb14571c18ece25fd68d2a03 Gerrit-Change-Number: 427181 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <zhbingyin(a)sina.com>

1 0

Change in ffilz/nfs-ganesha[next]: FSAL_GLUSTER: Drop redundant information in construct_handle
by GerritHub 27 Sep '18

27 Sep '18

From <zhbingyin(a)sina.com>: zhbingyin(a)sina.com has uploaded this change for review. ( https://review.gerrithub.io/427177 Change subject: FSAL_GLUSTER: Drop redundant information in construct_handle ...................................................................... FSAL_GLUSTER: Drop redundant information in construct_handle Drop useless argument 'len' and variable 'buffxstat'; Change-Id: Ib235152331ead24cf1f27446bc91305a45bded15 Signed-off-by: Bingyin Zhang <zhbingyin(a)sina.com> --- M src/FSAL/FSAL_GLUSTER/export.c M src/FSAL/FSAL_GLUSTER/gluster_internal.c M src/FSAL/FSAL_GLUSTER/gluster_internal.h M src/FSAL/FSAL_GLUSTER/handle.c 4 files changed, 10 insertions(+), 15 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/77/427177/1 -- To view, visit https://review.gerrithub.io/427177 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Ib235152331ead24cf1f27446bc91305a45bded15 Gerrit-Change-Number: 427177 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <zhbingyin(a)sina.com>

1 0

Change in ffilz/nfs-ganesha[next]: MDCACHE: fallocate needs to invalidate attributes
by GerritHub 27 Sep '18

27 Sep '18

From Frank Filz <ffilzlnx(a)mindspring.com>: Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/427167 Change subject: MDCACHE: fallocate needs to invalidate attributes ...................................................................... MDCACHE: fallocate needs to invalidate attributes Since this may affect filesize (and will affect space used), we need to invalidate attributes so they will be refreshed. Change-Id: I3c0a9ea01c3b3c2e6341c6a6d286564dc0a0c83d Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_file.c 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/67/427167/1 -- To view, visit https://review.gerrithub.io/427167 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I3c0a9ea01c3b3c2e6341c6a6d286564dc0a0c83d Gerrit-Change-Number: 427167 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>

1 0

Change in ffilz/nfs-ganesha[next]: FSAL_GLUSTER: Allow seek2 on any open file not just open for read
by GerritHub 27 Sep '18

27 Sep '18

From Frank Filz <ffilzlnx(a)mindspring.com>: Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/427154 Change subject: FSAL_GLUSTER: Allow seek2 on any open file not just open for read ...................................................................... FSAL_GLUSTER: Allow seek2 on any open file not just open for read Change-Id: Ic2666e0c96cf71224be4af06f28c5d9719cabcf6 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/FSAL_GLUSTER/handle.c 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/54/427154/1 -- To view, visit https://review.gerrithub.io/427154 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Ic2666e0c96cf71224be4af06f28c5d9719cabcf6 Gerrit-Change-Number: 427154 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>

1 0