August 2018 - Devel - NFS Ganesha List Archives

Fwd: Ganesha 2.5, crash /segfault while executing nlm4_Unlock
by Sachin Punadikar 19 Sep '18

19 Sep '18

---------- Forwarded message ---------- From: Sachin Punadikar <punadikar.sachin(a)gmail.com> Date: Tue, Jun 26, 2018 at 3:57 PM Subject: Ganesha 2.5, crash /segfault while executing nlm4_Unlock To: nfs-ganesha-devel <nfs-ganesha-devel(a)lists.sourceforge.net> Hi All, Recently a crash was reported by customer for Ganesha 2.5. (gdb) where #0 0x00007f475872900b in pthread_rwlock_wrlock () from /lib64/libpthread.so.0 #1 0x000000000041eac9 in fsal_obj_handle_fini (obj=0x7f4378028028) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL/commonlib.c:192 #2 0x000000000053180f in mdcache_lru_clean (entry=0x7f4378027ff0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:589 #3 0x0000000000536587 in _mdcache_lru_unref (entry=0x7f4378027ff0, flags=0, func=0x5a9380 <__func__.23209> "cih_remove_checked", line=406) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1921 #4 0x0000000000543e91 in cih_remove_checked (entry=0x7f4378027ff0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_hash.h:406 #5 0x0000000000544b26 in mdc_clean_entry (entry=0x7f4378027ff0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:235 #6 0x000000000053181e in mdcache_lru_clean (entry=0x7f4378027ff0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:592 #7 0x0000000000536587 in _mdcache_lru_unref (entry=0x7f4378027ff0, flags=0, func=0x5a70af <__func__.23112> "mdcache_put", line=190) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1921 #8 0x0000000000539666 in mdcache_put (entry=0x7f4378027ff0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.h:190 #9 0x000000000053f062 in mdcache_put_ref (obj_hdl=0x7f4378028028) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/FSAL /Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1709 #10 0x000000000049bf0f in nlm4_Unlock (args=0x7f4294165830, req=0x7f4294165028, res=0x7f43f001e0e0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/Prot ocols/NLM/nlm_Unlock.c:128 #11 0x000000000044c719 in nfs_rpc_execute (reqdata=0x7f4294165000) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/Main NFSD/nfs_worker_thread.c:1290 #12 0x000000000044cf23 in worker_run (ctx=0x3c200e0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/Main NFSD/nfs_worker_thread.c:1562 #13 0x000000000050a3e7 in fridgethr_start_routine (arg=0x3c200e0) at /usr/src/debug/nfs-ganesha-2.5.3-ibm013.00-0.1.1-Source/supp ort/fridgethr.c:550 #14 0x00007f4758725dc5 in start_thread () from /lib64/libpthread.so.0 #15 0x00007f4757de673d in clone () from /lib64/libc.so.6 A closer look at the backtrace indicates that there was cyclic flow of execution as below: nlm4_Unlock -> mdcache_put_ref -> mdcache_put -> _mdcache_lru_unref -> mdcache_lru_clean -> fsal_obj_handle_fini and then mdc_clean_entry -> cih_remove_checked -> (purposely coping next flow on below line) -> _mdcache_lru_unref -> mdcache_lru_clean -> fsal_obj_handle_fini (currently crashing here) Do we see any code issue here ? Any hints on how to RCA this issue ? Thanks in advance. -- with regards, Sachin Punadikar -- with regards, Sachin Punadikar

4 7

Announce Push of V2.7-rc4
by Frank Filz 31 Aug '18

31 Aug '18

Branch next Tag:V2.7-rc4 Release Highlights * Various grace period management fixes * Fix uinitialized xdr_attrs_args when meeting error at nfs4 readdir callback * Fix uinitialized end_of_file/fsal_stable at nfs/9p * Fix bad assign logical caused use after freed * FreeBSD: fix compilation on FreeBSD 12 * FSAL_MDCACHE: remove limitation on Reaper_Work_Per_Lane * Set op_ctx in state_blocked_lock_caller * Set sbd_blocked_cookie correctly * MDCACHE: fix a couple issues in readdir chunking Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 97f1b8b Frank S. Filz V2.7-rc4 b9c94ee Daniel Gryniewicz MDCACHE - Handle reloading first chunk dd5e7c5 Daniel Gryniewicz MDCACHE - Take lock when bumping chunk 8bef6b0 Sachin Punadikar Set sbd_blocked_cookie correctly cce90c8 Sachin Punadikar Set op_ctx in state_blocked_lock_caller 6b0b125 Fatih Acar FSAL_MDCACHE: remove limitation on Reaper_Work_Per_Lane 65df2a5 Fatih Acar FreeBSD: fix compilation on FreeBSD 12 44bc0ea Kinglong Mee Fix bad assign logical caused use after freed 222ab63 Kinglong Mee Fix uinitialized end_of_file/fsal_stable at nfs/9p d25e4b5 Kinglong Mee Fix uinitialized xdr_attrs_args when meeting error at nfs4 readdir callback 369d043 Jeff Layton rados_grace: move nfs_in_grace check to nfs_maybe_start_grace fcc50c4 Jeff Layton NFS: don't check for grace period in nfs4_chk_clid a17c8d6 Jeff Layton NLM: don't check for a grace period in nlm4_Unshare 2872d04 Jeff Layton NLM: don't check for grace period in nlm4_Unlock f7f45c4 Jeff Layton SAL: recovery_backend API cleanups

1 0

Change in ffilz/nfs-ganesha[next]: rados_grace: move nfs_in_grace check to nfs_maybe_start_grace
by GerritHub 31 Aug '18

31 Aug '18

From Jeff Layton <jlayton(a)redhat.com>: Jeff Layton has uploaded this change for review. ( https://review.gerrithub.io/424267 Change subject: rados_grace: move nfs_in_grace check to nfs_maybe_start_grace ...................................................................... rados_grace: move nfs_in_grace check to nfs_maybe_start_grace The maybe_start_grace routines should not need to do this themselves. Change-Id: I1d6d08aeaf0c919be980569d584bf8d5bf87e3d8 Signed-off-by: Jeff Layton <jlayton(a)redhat.com> --- M src/SAL/nfs4_recovery.c M src/SAL/recovery/recovery_rados_cluster.c 2 files changed, 1 insertion(+), 5 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/67/424267/1 -- To view, visit https://review.gerrithub.io/424267 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I1d6d08aeaf0c919be980569d584bf8d5bf87e3d8 Gerrit-Change-Number: 424267 Gerrit-PatchSet: 1 Gerrit-Owner: Jeff Layton <jlayton(a)redhat.com>

1 0

Change in ffilz/nfs-ganesha[next]: MDCACHE - Handle reloading first chunk
by GerritHub 31 Aug '18

31 Aug '18

From Daniel Gryniewicz <dang(a)redhat.com>: Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/424264 Change subject: MDCACHE - Handle reloading first chunk ...................................................................... MDCACHE - Handle reloading first chunk When the first chunk needs to be reloaded, due to missing entries, we need to remove the chunk and pass back the correct dirent. Change-Id: I0fd4c519c7461f16e5e4699385a2a8d88ec11e25 Signed-off-by: Daniel Gryniewicz <dang(a)redhat.com> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c 1 file changed, 5 insertions(+), 5 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/64/424264/1 -- To view, visit https://review.gerrithub.io/424264 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I0fd4c519c7461f16e5e4699385a2a8d88ec11e25 Gerrit-Change-Number: 424264 Gerrit-PatchSet: 1 Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com>

1 0

Change in ffilz/nfs-ganesha[next]: Set sbd_blocked_cookie correctly
by GerritHub 31 Aug '18

31 Aug '18

From Sachin Punadikar <psachin(a)in.ibm.com>: Sachin Punadikar has uploaded this change for review. ( https://review.gerrithub.io/424260 Change subject: Set sbd_blocked_cookie correctly ...................................................................... Set sbd_blocked_cookie correctly In "state_add_grant_cookie", the sbd_blocked_cookie field initialized in the begining. On failure of "do_lock_op", allocated cookie is freed, but sbd_blocked_cookie value kept as is, leading to double free condition. Also on freeing cookie, its entry from hash table should be removed. Change-Id: Ieabab40ffd06218b61d056541c0b08c297cd5dd4 Signed-off-by: Sachin Punadikar <psachin(a)in.ibm.com> --- M src/SAL/state_lock.c 1 file changed, 9 insertions(+), 4 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/60/424260/1 -- To view, visit https://review.gerrithub.io/424260 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Ieabab40ffd06218b61d056541c0b08c297cd5dd4 Gerrit-Change-Number: 424260 Gerrit-PatchSet: 1 Gerrit-Owner: Sachin Punadikar <psachin(a)in.ibm.com>

1 0

Change in ffilz/nfs-ganesha[next]: NFS: don't check for grace period in nfs4_chk_clid
by GerritHub 31 Aug '18

31 Aug '18

From Jeff Layton <jlayton(a)redhat.com>: Jeff Layton has uploaded this change for review. ( https://review.gerrithub.io/424258 Change subject: NFS: don't check for grace period in nfs4_chk_clid ...................................................................... NFS: don't check for grace period in nfs4_chk_clid We will validate cid_allow_reclaim flag at the point where we are actually reclaiming. There are also some ways that we can go back into grace and load a new recovery db after coming out of it with certain dbus commands. In that situation we could conceivably have a client establish a session outside of the grace period and then issue reclaim requests against it after the grace period has restarted (maybe a merging of state held by two servers in the cluster?). Drop this check from the code since we always check cid_allow_reclaim in conjunction with grace period checks at the time of the reclaim. Change-Id: I7cfd5ad6cd74c9358e9ec86710d0a31ad1ffa883 Signed-off-by: Jeff Layton <jlayton(a)redhat.com> --- M src/SAL/nfs4_recovery.c 1 file changed, 0 insertions(+), 3 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/58/424258/1 -- To view, visit https://review.gerrithub.io/424258 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: I7cfd5ad6cd74c9358e9ec86710d0a31ad1ffa883 Gerrit-Change-Number: 424258 Gerrit-PatchSet: 1 Gerrit-Owner: Jeff Layton <jlayton(a)redhat.com>

1 0

Change in ffilz/nfs-ganesha[next]: NLM: don't check for a grace period in nlm4_Unshare
by GerritHub 31 Aug '18

31 Aug '18

From Jeff Layton <jlayton(a)redhat.com>: Jeff Layton has uploaded this change for review. ( https://review.gerrithub.io/424257 Change subject: NLM: don't check for a grace period in nlm4_Unshare ...................................................................... NLM: don't check for a grace period in nlm4_Unshare Checking for the grace period is pointless. Unshare is releasing state, and that can't cause a conflict. It is true that an Unshare does get a reclaim flag in its arguments, but I chalk that up to poor protocol design rather than some deeper meaning. Change-Id: Id84038b738db6ea2643af6eecc5b0eb01422b29c Signed-off-by: Jeff Layton <jlayton(a)redhat.com> --- M src/Protocols/NLM/nlm_Unshare.c 1 file changed, 0 insertions(+), 15 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/57/424257/1 -- To view, visit https://review.gerrithub.io/424257 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Id84038b738db6ea2643af6eecc5b0eb01422b29c Gerrit-Change-Number: 424257 Gerrit-PatchSet: 1 Gerrit-Owner: Jeff Layton <jlayton(a)redhat.com>

1 0

Change in ffilz/nfs-ganesha[next]: NLM: don't check for grace period in nlm4_Unlock
by GerritHub 31 Aug '18

31 Aug '18

From Jeff Layton <jlayton(a)redhat.com>: Jeff Layton has uploaded this change for review. ( https://review.gerrithub.io/424256 Change subject: NLM: don't check for grace period in nlm4_Unlock ...................................................................... NLM: don't check for grace period in nlm4_Unlock There is no need to check for the grace period when processing an unlock. The grace period exists to prevent clients from acquiring conflicting state that other clients have yet to reclaim. An unlock, however, can't cause a conflict since we're releasing state that is genuinely held by the client. Change-Id: Iad2f36b9579f868d7715636f43d1c2ce2479a12b Signed-off-by: Jeff Layton <jlayton(a)redhat.com> --- M src/Protocols/NLM/nlm_Unlock.c 1 file changed, 0 insertions(+), 8 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/56/424256/1 -- To view, visit https://review.gerrithub.io/424256 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Iad2f36b9579f868d7715636f43d1c2ce2479a12b Gerrit-Change-Number: 424256 Gerrit-PatchSet: 1 Gerrit-Owner: Jeff Layton <jlayton(a)redhat.com>

1 0

Change in ffilz/nfs-ganesha[next]: FSAL_MDCACHE: remove limitation on Reaper_Work_Per_Lane
by GerritHub 31 Aug '18

31 Aug '18

From <fatih(a)gandi.net>: fatih(a)gandi.net has uploaded this change for review. ( https://review.gerrithub.io/424229 Change subject: FSAL_MDCACHE: remove limitation on Reaper_Work_Per_Lane ...................................................................... FSAL_MDCACHE: remove limitation on Reaper_Work_Per_Lane We encountered cases where the first 2000 entries in the LRU of each lane are all referenced. In consequence the reaper thread does not reap further unreferenced entries because it stops iterating too early. Raising Reaper_Work_Per_Lane above 2k fixed the issue. Change-Id: Ia1da12a21a42bf984c9f11424815080b45129a91 Signed-off-by: Fatih Acar <fatih.acar(a)gandi.net> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_read_conf.c M src/config_samples/config.txt M src/doc/man/ganesha-cache-config.rst 3 files changed, 3 insertions(+), 3 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/29/424229/1 -- To view, visit https://review.gerrithub.io/424229 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Ia1da12a21a42bf984c9f11424815080b45129a91 Gerrit-Change-Number: 424229 Gerrit-PatchSet: 1 Gerrit-Owner: Anonymous Coward <fatih(a)gandi.net>

1 0

Change in ffilz/nfs-ganesha[next]: FSAL: eliminate fsal_grace boolean
by GerritHub 30 Aug '18

30 Aug '18

From Jeff Layton <jlayton(a)redhat.com>: Jeff Layton has uploaded this change for review. ( https://review.gerrithub.io/424122 Change subject: FSAL: eliminate fsal_grace boolean ...................................................................... FSAL: eliminate fsal_grace boolean The fsal_staticfsinfo_t struct has a boolean called fsal_grace that will offload grace period handling to the FSAL. Nothing defaults to this being on, but GPFS has an option that allows it to be enabled. Removing this boolean simplifies the code significantly, and it's not clear that it's actually being used. This patch removes it, under the assumption that it's no longer useful. Cc: Malahal Naineni <malahal(a)gmail.com> Change-Id: Ic4af1f814e86175553a3a57eedc771b17947dee2 Signed-off-by: Jeff Layton <jlayton(a)redhat.com> --- M src/FSAL/FSAL_GPFS/main.c M src/FSAL/commonlib.c M src/FSAL/fsal_config.c M src/Protocols/NFS/nfs4_op_lock.c M src/Protocols/NFS/nfs4_op_open.c M src/Protocols/NLM/nlm_Lock.c M src/Protocols/NLM/nlm_Share.c M src/include/fsal_types.h 8 files changed, 5 insertions(+), 35 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/22/424122/1 -- To view, visit https://review.gerrithub.io/424122 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Ic4af1f814e86175553a3a57eedc771b17947dee2 Gerrit-Change-Number: 424122 Gerrit-PatchSet: 1 Gerrit-Owner: Jeff Layton <jlayton(a)redhat.com> Gerrit-CC: Malahal <malahal(a)gmail.com>

1 0