April 2023 - Devel - Nfs Ganesha List Archives

[S] Change in ...nfs-ganesha[next]: Change btfs to btrfs in two log messages

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553280?usp=email ) Change subject: Change btfs to btrfs in two log messages ...................................................................... Change btfs to btrfs in two log messages Change-Id: Ibee7a186bfbc54f74d63c364bc7b3e60f8076525 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/localfs.c 1 file changed, 12 insertions(+), 2 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/80/553280/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553280?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ibee7a186bfbc54f74d63c364bc7b3e60f8076525 Gerrit-Change-Number: 553280 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

4 months, 4 weeks

1
0
0 / 0

Announce Push of V5.1

by Frank Filz

Branch next Tag:V5.1 Merge Highlights * Empty commit to note that 5.0 was mistakenly labeled 5.1 in CMakeLists.txt * requires ntirpc 5.0 (pullup was in 5.0) * Add asserts for refcount decrements below 0 * Add atomic_inc_unless_0_int32_t * Use atomic_inc_unless_0_int32_t in several places to avoid incrementing refcount from 0 * Use it in hold_state_owner to avoid incrementing refcount from 0 avoiding use of the hash latch. * CMakefiles: specific the Python to Python3 * fix: broken v4_recov_dir_len when using fs_ng backend * Fix for fail to unmount fs after removing all exports of that fs * rgw: version 1.2.1 changed api version (rgw_readdir2 returns int) * rgw: rgw_xattr fixes Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 38ae53541 Frank S. Filz V5.1 ac2fe689f Matt Benjamin rgw: rgw_xattr fixes a99478410 Matt Benjamin rgw: version 1.2.1 changed api version (rgw_readdir2 returns int) a5f509122 Yogendra Charya Fix for fail to unmount fs after removing all exports of that fs fed5f6796 Kaleb S. KEITHLEY cmake: minium version for ntirpc d40b50f8d Shuoran Liu fix: broken v4_recov_dir_len when using fs_ng backend 5bd819773 Vicente Cheng CMakefiles: specific the Python to Python3 1012c5950 Frank S. Filz Change hold_state_owner to use atomic_inc_unless_0_int32_t 7f482f4a6 Frank S. Filz Use atomic_inc_unless_0_int32_t instead of inc and check 278da4e25 Frank S. Filz Add atomic_inc_unless_0_int32_t 28b8e373b Frank S. Filz Add assert for all refcount decrements

5 months

1
0
0 / 0

[M] Change in ...nfs-ganesha[next]: Change hold_state_owner to use atomic_inc_unless_0_int32_t

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553234 ) Change subject: Change hold_state_owner to use atomic_inc_unless_0_int32_t ...................................................................... Change hold_state_owner to use atomic_inc_unless_0_int32_t Also change the name to hold_state_owner_ref (which makes it easier to audit code for proper use of refcounts). By using atomic_inc_unless_0_int32_t, we can detect the case where the refcount is already 0, so rather than trying to process this owner we skip it (for now). This means that hold_state_owner_ref doesn't need to take the hash latch (which caused a lock order inversion) to protect while we check for 0 refcount. This lock inversion was introduced in commit c8e18281a4cf0b535f731cf0fcd0dcc5566c2ef0 as an attempt to avoid the incrementing refcount from 0. This addresses the deadlock caused by the lock inversion reported in github issue #934. Change-Id: Ia3babaad97a3e430091dc3b23e6e21d095418ac2 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/SAL/nfs4_clientid.c M src/SAL/state_misc.c M src/include/sal_functions.h 3 files changed, 49 insertions(+), 47 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/34/553234/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553234 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Ia3babaad97a3e430091dc3b23e6e21d095418ac2 Gerrit-Change-Number: 553234 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

5 months

1
0
0 / 0

[S] Change in ...nfs-ganesha[next]: Use atomic_inc_unless_0_int32_t instead of inc and check

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553233 ) Change subject: Use atomic_inc_unless_0_int32_t instead of inc and check ...................................................................... Use atomic_inc_unless_0_int32_t instead of inc and check Instead of incrementing and checking if 1, we use the new atomic_inc_unless_0_int32_t to only increment if not 0. Change-Id: I147727d17c4017b5abc9953227f75af3312069fc Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/SAL/nlm_owner.c M src/SAL/nlm_state.c M src/SAL/state_misc.c 3 files changed, 19 insertions(+), 10 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/33/553233/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553233 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I147727d17c4017b5abc9953227f75af3312069fc Gerrit-Change-Number: 553233 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

5 months

1
0
0 / 0

[M] Change in ...nfs-ganesha[next]: Add atomic_inc_unless_0_int32_t

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553232 ) Change subject: Add atomic_inc_unless_0_int32_t ...................................................................... Add atomic_inc_unless_0_int32_t This is used to increment a refcount when we don't want to take a reference if the refcount is 0. Change-Id: Iddab43ee415065af74af3fd9da17bc8c38b72d9b Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/include/abstract_atomic.h 1 file changed, 62 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/32/553232/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553232 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Iddab43ee415065af74af3fd9da17bc8c38b72d9b Gerrit-Change-Number: 553232 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

5 months

1
0
0 / 0

[S] Change in ...nfs-ganesha[next]: Add assert for all refcount decrements

by Frank Filz (GerritHub)

Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553231 ) Change subject: Add assert for all refcount decrements ...................................................................... Add assert for all refcount decrements Change-Id: Iedd75e14fd105aa73f9d45125c874d7d5e645d16 Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> --- M src/FSAL/FSAL_MEM/mem_handle.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c M src/Protocols/NFS/nfs4_Compound.c M src/RPCAL/nfs_dupreq.c M src/SAL/nfs41_session_id.c M src/SAL/nfs4_clientid.c M src/SAL/nfs4_state_id.c M src/SAL/nlm_owner.c M src/SAL/nlm_state.c M src/SAL/state_lock.c 10 files changed, 37 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/31/553231/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553231 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Iedd75e14fd105aa73f9d45125c874d7d5e645d16 Gerrit-Change-Number: 553231 Gerrit-PatchSet: 1 Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com> Gerrit-MessageType: newchange

5 months

1
0
0 / 0

Re: Client id expiration - multiple clients using same hostname and deadlock issues.

by Frank Filz

Please open a github issue for this. We definitely need to not crash or deadlock. It sounds like as long as we don't fail miserably, the client can't expect anything. I'm not sure if we can handle the situation more gracefully and still do the right thing in cases where this situation is caused by a client whose IP address has changed for some reason or other things that might make an actual single client appear to the server as multiple clients. Frank From: Deepak Arumugam Sankara Subramanian [mailto:deepakarumugam.s@nutanix.com] Sent: Tuesday, April 25, 2023 2:04 PM To: devel(a)lists.nfs-ganesha.org Cc: Pradeep Thomas <pradeep.thomas(a)nutanix.com>; Pragyashree Gogoi <pragyashree.gogoi(a)nutanix.com>; Sagar Singh <sagar.singh(a)nutanix.com>; Gaurav Gangalwar <gaurav.gangalwar(a)nutanix.com>; Andrew Lin <andrew.lin(a)nutanix.com> Subject: [NFS-Ganesha-Devel] Client id expiration - multiple clients using same hostname and deadlock issues. Hi , We recently ran into some issues in client-id expiration code paths. We've had multiple issues where a CREATE_SESSION tries to expire a clientid when the clientid is being used by other rpcs like OPEN, EXCHANGE_ID etc.You typically don't expect a client to send a OPEN/EXCHANGE_ID rpc while the server is processing a CREATE_SESSSION rpc unless the client is misbehaving. We found that in our labs/test-beds multiple clients had the same hostname set and ganesha was mapping them to the same client record. Now although the clients were misbehaving we feel that the server should handle/fail gracefully. In these issues the server usually runs into an unexpected assert,segfault and crashes or it runs into a deadlock and hangs forever . Q1: We need some recommendations on whats the right thing to do when multiple clients use the same owner_id. The RFC says this about co_ownerid The string should be unique so that multiple clients do not present the same string. The consequences of two clients presenting the same string range from one client getting an error to one client having its leased state abruptly and unexpectedly cancelled. but we wanted to know if one of these responses is better than other. These are some more details on the individual issues raised from setups having the same hostname Deadlock: Two client ids say c1 and c2 from 2 different clients were associated with the same co_ownerid and same record cr1 1. Client 1 -> thread 1 was doing a open. It was inside the get_state_owner function trying to get a owner for the open state. It acquired ht_owner->partitions[15].lock created a new open owner and was trying to hold the mutex on cr1 aka nfs4_owner->so_clientrec->cid_mutex so it could add the owner to the client record cr1. 2. Client2 -> thread 2 was doing a create session. It was inside nfs_clientid_expire function. It was parsing the open owner list inside c1 and trying to 'delete' each owner (the second while(true) loop). It was holding nfs4_owner->so_clientrec->cid_mutex and trying to get ht_owner->partitions[15].lock Q2: If we were to fix this deadlock, whats the recommended order for acquiring the locks - should we acquire the client_rec->cid_mutex first and then acquire ht_owner table lock or vice versa This looks like it could be a common deadlock pattern we might've encountered before since we have 2 structures referencing each other each with their own locks I can update more details on the other failures(asserts, segfaults etc). Let me know if that is needed Thanks, Deepak

5 months

2
1
0 / 0

[S] Change in ...nfs-ganesha[next]: config: this is nfs-ganesha 5.0

by Kaleb KEITHLEY (GerritHub)

Kaleb KEITHLEY has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553174 ) Change subject: config: this is nfs-ganesha 5.0 ...................................................................... config: this is nfs-ganesha 5.0 GANESHA_VERSION is used all over the place Change-Id: I29c3d3ec9afe107b115d13f84f0eef4216e8d231 Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> --- M src/CMakeLists.txt 1 file changed, 13 insertions(+), 1 deletion(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/74/553174/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/553174 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I29c3d3ec9afe107b115d13f84f0eef4216e8d231 Gerrit-Change-Number: 553174 Gerrit-PatchSet: 1 Gerrit-Owner: Kaleb KEITHLEY <kaleb(a)redhat.com> Gerrit-MessageType: newchange

5 months

1
0
0 / 0

Client id expiration - multiple clients using same hostname and deadlock issues.

by Deepak Arumugam Sankara Subramanian

Hi , We recently ran into some issues in client-id expiration code paths. We've had multiple issues where a CREATE_SESSION tries to expire a clientid when the clientid is being used by other rpcs like OPEN, EXCHANGE_ID etc.You typically don't expect a client to send a OPEN/EXCHANGE_ID rpc while the server is processing a CREATE_SESSSION rpc unless the client is misbehaving. We found that in our labs/test-beds multiple clients had the same hostname set and ganesha was mapping them to the same client record. Now although the clients were misbehaving we feel that the server should handle/fail gracefully. In these issues the server usually runs into an unexpected assert,segfault and crashes or it runs into a deadlock and hangs forever . Q1: We need some recommendations on whats the right thing to do when multiple clients use the same owner_id. The RFC says this about co_ownerid The string should be unique so that multiple clients do not present the same string. The consequences of two clients presenting the same string range from one client getting an error to one client having its leased state abruptly and unexpectedly cancelled. but we wanted to know if one of these responses is better than other. These are some more details on the individual issues raised from setups having the same hostname Deadlock: Two client ids say c1 and c2 from 2 different clients were associated with the same co_ownerid and same record cr1 1. Client 1 -> thread 1 was doing a open. It was inside the get_state_owner function trying to get a owner for the open state. It acquired ht_owner->partitions[15].lock created a new open owner and was trying to hold the mutex on cr1 aka nfs4_owner->so_clientrec->cid_mutex so it could add the owner to the client record cr1. 2. Client2 -> thread 2 was doing a create session. It was inside nfs_clientid_expire function. It was parsing the open owner list inside c1 and trying to 'delete' each owner (the second while(true) loop). It was holding nfs4_owner->so_clientrec->cid_mutex and trying to get ht_owner->partitions[15].lock Q2: If we were to fix this deadlock, whats the recommended order for acquiring the locks - should we acquire the client_rec->cid_mutex first and then acquire ht_owner table lock or vice versa This looks like it could be a common deadlock pattern we might've encountered before since we have 2 structures referencing each other each with their own locks I can update more details on the other failures(asserts, segfaults etc). Let me know if that is needed Thanks, Deepak

5 months

1
0
0 / 0

UDP mount in nfs-ganesha

by Sagar Singh

Hi, We are trying to work on a UDP mount for ganesha, and need some feedback. For NLM lock recovery we use the node local ip address of ganesha server-node. It works perfectly fine with the TCP mount. But with the UDP mount we have seen that local rq_xprt->xp_local.nb.buf points to valid local-ip, however rq_xprt->xp_local.ss have ip address ADDR_ANY. (gdb) p *((struct sockaddr_in*)&reqdata->svc->rq_xprt->xp_local.ss) $2 = {sin_family = 2, sin_port = 264, sin_addr = {s_addr = 0}, sin_zero = "\000\000\000\000\000\000\000"} (gdb) p *((struct sockaddr_in*)reqdata->svc->rq_xprt->xp_local.nb.buf) $3 = {sin_family = 3, sin_port = 0, sin_addr = {s_addr = 2237796106}, sin_zero = "\n\017b\205\000\000\000"} Which causes problems in lock recovery and other functionalities. We have made some changes in the initialization code of local ip: https://github.com/nfs-ganesha/ntirpc/pull/273/files Which seems to resolve the issue of the local address being ADDR_ANY and lock recovery works fine. We are not sure that :- 1. Will this fix work with all kinds of clients? 2. Will this break any existing functionalities? Any feedback will be greatly appreciated. Thanks, Sagar Singh

5 months

2
1
0 / 0

2023

2022

2021

2020

2019

2018

Devel April 2023