March 2019 - Devel - NFS Ganesha List Archives

Ganesha crashed in mdcache_clean_dirent_chunk
by Ashish Sangwan 08 Mar '19

08 Mar '19

(gdb) bt #0 0x00007f0e1193759b in raise () from /lib64/libpthread.so.0 #1 0x0000000000448dd4 in crash_handler (signo=11, info=0x7f0e0d3fcdf0, ctx=0x7f0e0d3fccc0) at /usr/src/debug/nfs-ganesha-2.7.1/MainNFSD/nfs_init.c:246 #2 <signal handler called> #3 0x000000000054777e in mdcache_clean_dirent_chunk (chunk=0x7f0df5883470) at /usr/src/debug/nfs-ganesha-2.7.1/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:448 #4 0x00000000005386c2 in lru_clean_chunk (chunk=0x7f0df5883470) at /usr/src/debug/nfs-ganesha-2.7.1/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:2078 #5 0x000000000053883b in mdcache_lru_unref_chunk (chunk=0x7f0df5883470) at /usr/src/debug/nfs-ganesha-2.7.1/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:2097 #6 0x00000000005369ac in chunk_lru_run_lane (lane=3) at /usr/src/debug/nfs-ganesha-2.7.1/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1509 #7 0x0000000000536d46 in chunk_lru_run (ctx=0x7f0e0dc0f080) at /usr/src/debug/nfs-ganesha-2.7.1/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1563 #8 0x0000000000508af9 in fridgethr_start_routine (arg=0x7f0e0dc0f080) at /usr/src/debug/nfs-ganesha-2.7.1/support/fridgethr.c:550 #9 0x00007f0e1192fe25 in start_thread () from /lib64/libpthread.so.0 #10 0x00007f0e11237bad in clone () from /lib64/libc.so.6 (gdb) f 3 #3 0x000000000054777e in mdcache_clean_dirent_chunk (chunk=0x7f0df5883470) at /usr/src/debug/nfs-ganesha-2.7.1/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:448 448 glist_for_each_safe(glist, glistn, &chunk->dirents) { (gdb) info locals glist = 0x0 glistn = 0x0 parent = 0x7f0dd8c68200 I can see that in the above case, the content lock on the parent is not held. A parallel rmdir/rename on the dirent pointed by the glist could set the glist->next to NULL triggering the crash. In chunk_lru_run_lane() we are releasing qlock and than reacquire it in mdcache_lru_unref_chunk() Without the content lock held in the above scenario, mdc_try_get_cached() could sneak in, get the qlock for bumping the chunk's lru and return the cached dirent. Also, in 2.6, mdcache_clean_dirent_chunk() was always called with content lock held from both the callers: lru_reap_chunk_impl and lru_remove_chunk. Probably that's why this crash was not seen with 2.6. Does this seems to be a plausible scenario? Ashish

2 1

Change in ...nfs-ganesha[next]: MDCACHE - Take content lock when freeing chunk
by Daniel Gryniewicz (GerritHub) 08 Mar '19

08 Mar '19

Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447484 Change subject: MDCACHE - Take content lock when freeing chunk ...................................................................... MDCACHE - Take content lock when freeing chunk The chunk cleanup involves a number of fields which are protected by the content lock, so make sure the content lock is held when freeing a chunk. Change-Id: I324e361063b027a9dd6e31cca49c3803d5adc934 Signed-off-by: Daniel Gryniewicz <dang(a)fprintf.net> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h 4 files changed, 53 insertions(+), 20 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/84/447484/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447484 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I324e361063b027a9dd6e31cca49c3803d5adc934 Gerrit-Change-Number: 447484 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com> Gerrit-MessageType: newchange

1 0

Change in ...nfs-ganesha[next]: FSAL_MEM - Pass object, not it's address
by Daniel Gryniewicz (GerritHub) 08 Mar '19

08 Mar '19

Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447483 Change subject: FSAL_MEM - Pass object, not it's address ...................................................................... FSAL_MEM - Pass object, not it's address myself is allocated from the heap, not the stack, so pass it, not it's address. Change-Id: I9371b081b1c9867373db888dc7d520953821818a Signed-off-by: Daniel Gryniewicz <dang(a)fprintf.net> --- M src/FSAL/FSAL_MEM/mem_export.c 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/83/447483/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447483 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I9371b081b1c9867373db888dc7d520953821818a Gerrit-Change-Number: 447483 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com> Gerrit-MessageType: newchange

1 0

Work pool vs fridgethr
by fanzi2009＠hotmail.com 08 Mar '19

08 Mar '19

Hi, I found work pool and fridgethr(I'm also confused about this name) have similar function. They all implement the function for producer/consumer. Is there any difference between them? What does fridgethr mean? Thanks, Marvin

2 1

A question about sending reply
by 891873395＠qq.com 08 Mar '19

08 Mar '19

Hi： I have some confusion about sending reply to client。In the process of sending the response，fouction svc_ioq_flushv() would be called. Here, use the writev() system call to send reply via sockfd. I think we're using non-blocking socket here, but comment is "/* blocking write */" and the transport is finish when return < 0. Why? result = writev(xprt->xp_fd, wiov, iw); remaining -= result; if (result == fbytes) { wiov += iw - 1; iw = 0; continue; } if (unlikely(result < 0)) { __warnx(TIRPC_DEBUG_FLAG_ERROR, "%s() writev failed (%d)\n", __func__, errno); cfconn_set_dead(xprt); break; } I think when the return code is less than 0, we should judge whether the error code is EAGAIN.

2 1

Change in ...nfs-ganesha[next]: ganesha_mgr gets a new option to show the filesystem cache
by Name of user not set (GerritHub) 08 Mar '19

08 Mar '19

ntrishal(a)in.ibm.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447424 Change subject: ganesha_mgr gets a new option to show the filesystem cache ...................................................................... ganesha_mgr gets a new option to show the filesystem cache The get_fscache option is added to the ganesha_mgr utility. It will show the major and minor device id's of the mounted filesystems. Change-Id: If18b7a45918079ac3ccba1a2a1c5855fb1d17b53 Signed-off-by: Trishali Nayar <ntrishal(a)in.ibm.com> --- M src/FSAL/commonlib.c M src/MainNFSD/nfs_init.c M src/include/FSAL/fsal_commonlib.h M src/scripts/ganeshactl/Ganesha/ganesha_mgr_utils.py M src/scripts/ganeshactl/ganesha_mgr.py 5 files changed, 208 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/24/447424/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447424 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: If18b7a45918079ac3ccba1a2a1c5855fb1d17b53 Gerrit-Change-Number: 447424 Gerrit-PatchSet: 1 Gerrit-Owner: ntrishal(a)in.ibm.com Gerrit-MessageType: newchange

1 0

Change in ...nfs-ganesha[next]: RFC: [do not merge] (lib)ganesha_nfsd as a DSO
by Name of user not set (GerritHub) 07 Mar '19

07 Mar '19

kaleb(a)redhat.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447380 Change subject: RFC: [do not merge] (lib)ganesha_nfsd as a DSO ...................................................................... RFC: [do not merge] (lib)ganesha_nfsd as a DSO Build ganesha as a DSO, i.e. libganesha_nfsd.so. Take Two-dot-two, addressing comments from the first go-around. (not building src/tools to see if this will get past CEA-HPC because tools is trying to link with -lrados_recov for some reason.) The ganesha.nfsd daemon consists of just nfs_main:main() and is linked with the DSO. Then all the FSALs (except FSAL_PSEUDO and FSAL_MDCACHE*) are linked with libganesha_nfsd to resolve functions in ganesha.nfsd itself. This will be required if, e.g., Fedora, decides to require default linking with '-z defs'. They tried to do it for F29 but retreated when they found that too many applications broke. In the end this may all be just an exercise in "truth and beauty." *FSAL_PSEUDO and FSAL_MDCACHE are special; they are included in the DSO so do not need to be (and can't be) linked with the DSO. A few other things that could be done are 1) link libganesha_nfsd.so with a .sym file to explicitly control which symbols are exposed or visible to the outside world. 2) build with -flto (Link Time Optimization) Annecdotes claim a 5-10% speed improvement is typical, and some applications have seen as much as 45-50% boost. Of course we have to do the experiment to see what we really achieve. 3) don't link ganesha.nfsd with -bdynamic as it's no longer needed Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> Change-Id: I31095bf24f80ae24984f26e242be5ded01dd03d2 --- M src/CMakeLists.txt M src/FSAL/CMakeLists.txt M src/FSAL/FSAL_CEPH/CMakeLists.txt M src/FSAL/FSAL_GLUSTER/CMakeLists.txt M src/FSAL/FSAL_GPFS/CMakeLists.txt M src/FSAL/FSAL_MEM/CMakeLists.txt M src/FSAL/FSAL_PROXY/CMakeLists.txt M src/FSAL/FSAL_RGW/CMakeLists.txt M src/FSAL/FSAL_VFS/panfs/CMakeLists.txt M src/FSAL/FSAL_VFS/vfs/CMakeLists.txt M src/FSAL/FSAL_VFS/xfs/CMakeLists.txt M src/FSAL/Stackable_FSALs/CMakeLists.txt M src/MainNFSD/CMakeLists.txt M src/SAL/recovery/recovery_fs_ng.c M src/config_parsing/CMakeLists.txt M src/nfs-ganesha.spec-in.cmake M src/support/murmur3.c 17 files changed, 561 insertions(+), 51 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/80/447380/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447380 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I31095bf24f80ae24984f26e242be5ded01dd03d2 Gerrit-Change-Number: 447380 Gerrit-PatchSet: 1 Gerrit-Owner: kaleb(a)redhat.com Gerrit-MessageType: newchange

1 0

Deploying a Ceph+NFS Server Cluster with Rook
by Jeff Layton 06 Mar '19

06 Mar '19

I had several people ask me to put together some instructions on how to deploy a Ceph+NFS cluster from scratch, and the new functionality in Ceph and rook.io make this quite easy. I wrote a Ceph community blog post that walks the reader through the process: https://ceph.com/community/deploying-a-cephnfs-server-cluster-with-rook/ I don't think that site has a way to post comments, but I'm happy to answer questions about it via email. -- Jeff Layton <jlayton(a)poochiereds.net>

1 0

ACLs not being read
by katcherw＠gmail.com 06 Mar '19

06 Mar '19

Hi, I am having a problem with ACLs in NFS Ganesha v2.6.3. Based on what I believe to be the cause of the problem, I think the problem will exist in the tip too. I am using a proprietary fsal. The symptom of the problem is simple. If I create a file and quickly read the ACLs, nothing is returned: # echo test > test.txt # nfs4_getfacl test.txt # If I restart ganesha, I see the correct ACLs: # nfs4_getfacl test.txt A::OWNER@:rwatTnNcCy A::GROUP@:rtncy A::EVERYONE@:rtncy Also, if I create a file, and wait over 60 seconds, then I can read the correct ACLs. This is all working fine in v2.5.1. I think I see the problem in the code. In my case, the nfs client first sends a GETATTR request with the ACL bit set in the request mask. But then it sends another one without the ACL bit set. During this second request, when ACLs are not requested, the entry in mdcache is updated in mdcache_refresh_attrs. In 2.5.1, mdcache_refresh_attrs, after the subfsal is called to get the attributes, it does: if (entry->attrs.acl != NULL) { /* We used to have an ACL... */ if (need_acl) { /* We requested update of an existing ACL, release the * old one. */ nfs4_acl_release_entry(entry->attrs.acl); } else { /* The ACL wasn't requested, move it into the * new attributes so we will retain it and make * it such that the entry attrs DO request the * ACL. */ attrs.acl = entry->attrs.acl; attrs.valid_mask |= ATTR_ACL; entry->attrs.request_mask |= ATTR_ACL; } /* ACL was released or moved to new attributes. */ entry->attrs.acl = NULL; } Here, the second path is taken. Basically it moves the ACLs to attrs, but it sets the entry's request mask bit. Soon after, fsal_copy_attrs is called which copies the ACLs back to the entry. It does this because the entry's request mask ACL bit is set. If it was not set, the ACLs would not be copied. Unfortunately, this is not happening in 2.6.3: In 2.6.3, this functionality was moved into mdc_update_attr_cache: if (entry->attrs.acl != NULL) { /* We used to have an ACL... */ if (attrs->acl != NULL) { /* We got an ACL from the sub FSAL whether we asked for * it or not, given that we had an ACL before, and we * got a new one, update the ACL, so release the old * one. */ nfs4_acl_release_entry(entry->attrs.acl); } else { /* A new ACL wasn't provided, so move the old one * into the new attributes so it will be preserved * by the fsal_copy_attrs. */ attrs->acl = entry->attrs.acl; attrs->valid_mask |= ATTR_ACL; } /* NOTE: Because we already had an ACL, * entry->attrs.request_mask MUST have the ATTR_ACL bit set. * This will assure that fsal_copy_attrs below will copy the * selected ACL (old or new) into entry->attrs. */ /* ACL was released or moved to new attributes. */ entry->attrs.acl = NULL; } else if (attrs->acl != NULL) { /* We didn't have an ACL before, but we got a new one. We may * not have asked for it, but receive it anyway. */ entry->attrs.request_mask |= ATTR_ACL; } We are taking the path where it says "A new ACL wasn't provided". It is moving the existing ACL into attrs, which we would like to be copied later by fsal_copy_attrs, just like in 2.5.1. But note in 2.6, we are never setting the ACL bit in the entry's request mask, only in attrs. Therefore, fsal_copy_attrs will not copy the ACL back to the entry, and the ACL will be lost forever. Even worse, is the trusted ACL flag in mde_flags is set, so now the caller thinks there are no ACLs at all! The logic looks the same in the current 2.8 code to me. Please let me know if my analysis is correct or not, and if so, if there is any possible workaround. Bill

3 9

Change in ...nfs-ganesha[next]: Fix when use glusterfs, copy read only file error. Bug URL: https...
by burningkernel (GerritHub) 04 Mar '19

04 Mar '19

burningkernel has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447012 Change subject: Fix when use glusterfs, copy read only file error. Bug URL: https://bugzilla.redhat.com/show_bug.cgi?id=1543996 https://github.com/nfs-ganesha/nfs-ganesha/issues/262 ...................................................................... Fix when use glusterfs, copy read only file error. Bug URL: https://bugzilla.redhat.com/show_bug.cgi?id=1543996 https://github.com/nfs-ganesha/nfs-ganesha/issues/262 Reference link: https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/426589/3/src/FSAL/FSAL_CE… Signed-off-by: Jingbo Cao <burningkernel(a)gmail.com> Change-Id: Ic5967785b66081ca5afa37f17bdc2d52321583ae --- M src/FSAL/FSAL_GLUSTER/handle.c 1 file changed, 4 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/12/447012/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/447012 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ic5967785b66081ca5afa37f17bdc2d52321583ae Gerrit-Change-Number: 447012 Gerrit-PatchSet: 1 Gerrit-Owner: burningkernel <burningkernel(a)gmail.com> Gerrit-MessageType: newchange

1 0