[S] Change in ...nfs-ganesha[next]: Fix for NLM CLOSE_WAIT Socket Leak & negative AUTH refcount in NSM op...
by Rajesh Prasad (GerritHub)
Rajesh Prasad has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238734?usp=email )
Change subject: Fix for NLM CLOSE_WAIT Socket Leak & negative AUTH refcount in NSM operations
......................................................................
Fix for NLM CLOSE_WAIT Socket Leak & negative AUTH refcount in NSM operations
NLM CLOSE_WAIT Issue - Fix in nlm_owner.c file
When using NFSv3 with NLM (Network Lock Manager), TCP callback connections
from NFS Ganesha to clients get stuck in CLOSE_WAIT state and are never
properly closed. This leads to socket exhaustion over time.
netstat command shows multiple CLOSE_WAIT connections
The remote peer (NLM client) sent FIN to close the connection. NFS Ganesha
received the FIN and ACKed it but Ganesha never close socket FD.
The socket remains in CLOSE_WAIT indefinitely until Ganesha restart.
Negative AUTH refcount Issue - Fix in nsm.c file
This change fixes a critical bug where AUTH_DESTROY() causes negative
reference counts (ah_refcnt) when called on authnone auth structures.
ROOT CAUSE:
1. authnone_ncreate() returns a shared singleton but does NOT initialize
ah_refcnt (it remains 0 from static storage initialization)
2. AUTH_DESTROY() calls auth_put() which decrements: 0 - 1 = -1
3. Multiple AUTH_DESTROY() calls lead to: -1, -2, -3, ..., -120, ..
The fix removes all AUTH_DESTROY() calls for nsm_auth since it's a
shared singleton that should persist for the process lifetime.
Change-Id: I45ec10b59e0728bdc387eff555121d208600c885
Signed-off-by: Rajesh Prasad <raprasad(a)redhat.com>
---
M src/Protocols/NLM/nsm.c
M src/SAL/nlm_owner.c
2 files changed, 17 insertions(+), 3 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/34/1238734/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238734?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I45ec10b59e0728bdc387eff555121d208600c885
Gerrit-Change-Number: 1238734
Gerrit-PatchSet: 1
Gerrit-Owner: Rajesh Prasad <raprasad(a)redhat.com>
1 month
Announce Push of V9.15
by Frank Filz
Branch next
Tag:V9.15
NOTE: This merge includes and ntirpc pullup - please update your submodule.
NOTE: This is a huge merge, instead of the usual detailed list of
highlights,
a much abbreviated set is given.
Merge Highlights
* ntirpc pullup
* foundation patches for upcomming integrated NSM patch set
* added Cluster_Members config (other features may use this also)
* some refinement of rpc_tools and ip_uitls, reduce number of includes of
gsh_rpc.h
* numerous bug fixes including:
* several refcount leaks and use after free
* Fix TOCTOU races in NFSv4 OPEN, OPEN_DOWNGRADE, CLOSE, LOCK and LOCKU
operations.
* NFSv4: fix OPEN_DOWNGRADE validation for BOTH after separate opens
* Added bounds checks
* NFSv4: fix OPEN_DOWNGRADE validation for BOTH after separate opens
* Validate CLAIM_DELEGATE* stateids in open4_claim_deleg()
* Added new metrics support QOS feature
* main: Add progress logging for initialization steps
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
adb062bb3 Frank S. Filz V9.15
90ba38388 Sreedhar Agraharam Added new metrics support QOS feature
baa956ab2 Nishant Puri SAL: recover gracefully from zombie lock-owner
so_related_owner mismatch
3735f868a Christoph Martin Fix python install problem, see
https://github.com/nfs-ganesha/nfs-ganesha/pull/1393
77d965248 Jagnya Tripathy NFSv4: fix OPEN_DOWNGRADE validation for BOTH
after separate opens
2dbfd4f4c Lior Suliman main: Add progress logging for initialization steps
4357cecdd Shahar Hochma Fix dir chunk ref count leak
5a38655d3 Yoni Couriel nfs4_owner: Fixed refcount leak.
a42eeefa9 Yoni Couriel state_lock: Fixed refcount leak.
fda79a3de Ofir Vainshtein Refactor: Add bounds-checked fattr comparison in
nfs4_Fattr_cmp
a7fdf6962 Ofir Vainshtein Fix: Prevent releasing uninitialized op_context in
LAYOUTRETURN
9f74a3483 Ofir Vainshtein Add bounds check for NFSv4 attribute index.
75b713eb1 Matt Benjamin FSAL_RGW: fix a buffer overrun in module options
parsing
1a6dd2640 Suhas Athani Validate CLAIM_DELEGATE* stateids in
open4_claim_deleg()
63a0710e5 Inna Batenkov Fix TOCTOU races in NFSv4 OPEN, OPEN_DOWNGRADE,
CLOSE, LOCK and LOCKU operations.
6bd2cb238 Inna Batenkov Remove call to mdcache_get_parent when the parent
object cache is disabled.
20a130ba5 Inna Batenkov Conditionally retrieve ACLs in access check
7f7431886 Sachin Punadikar GPFS: Fix pynfs failure for RDLK1
58563fbb1 Oscar Ou Fix use-after-free in rados_kv_connect error paths
a3c0e4d0d Frank S. Filz ntirpc pullup
c772f5ade Frank S. Filz Client list normalization - all have str
d83d30c06 Frank S. Filz NLM: Add config to not unmonitor on clean shutdown
8283bcd1f Frank S. Filz Add cluster_members config parameter
062d50f5d Frank S. Filz Better document use of add_client and use correct
field for HAProxy_Hosts
64a037423 Frank S. Filz Explicitly include uid2grp.h
ca0b05aaa Frank S. Filz Trim gsh_rpc.h from many places, also a few other
redundant includes
2d67c3428 Frank S. Filz ip_utils: add is_af_local and is_inaddrany
957ea36a3 Frank S. Filz Move a few more bits from gsh_rpc.h to ip_utils.h
dae6359a8 Frank S. Filz rpc_tools: use sockaddr_t instead of netbuf for
copy_xprt_addr
fae5f8f9d Frank S. Filz Make netconfig available earlier
85c1733fa Frank S. Filz Move definitions of NFS_options and NFS_program to
gsh_config.h
1934b060e Frank S. Filz display.h - more documentation in header file
5c44a9ed2 Frank S. Filz enhance display_opaque_value_max with flags and
additional set of non-printable
dd00d814b Frank S. Filz SAL: move obj_is_junction function to more sensible
place
348f17c5b Frank S. Filz Additional Debug
3cefd170e Frank S. Filz Add libganesha_monitoring.so to .gitignore
1 month
[M] Change in ...nfs-ganesha[next]: Optimisation to avoid same-client lock-path delegation recall in FSAL_UP
by Suhas Athani (GerritHub)
Suhas Athani has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238566?usp=email )
Change subject: Optimisation to avoid same-client lock-path delegation recall in FSAL_UP
......................................................................
Optimisation to avoid same-client lock-path delegation recall in FSAL_UP
This patch is proposed as an optimised solution for patch -
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1235527
This change fixes recall24 by preventing spurious NFS delegation recall
when a backend upcall arrives during same-client lock/open-mode transitions.
With Ceph delegation callbacks, lock-driven reopen activity could trigger
up_async_delegrecall, leading to delegrecall_one / CB_RECALL even when all
active locks belong to the same client that owns the delegation.
Added a guard in FSAL_UP (delegrecall_impl_per_state) to skip recall when
all tracked active NFSv4 locks on the file belong to the same clientid as
the delegation owner. In that case, delegation state is restored to
DELEG_GRANTED and CB_RECALL is not issued.
Optimization:
- Add lock_clientid_hint on struct state_file, maintained in state_lock.c
when locks are added, removed, or wiped.
- First lock records that client’s clientid; a lock from a different client
sets the hint to UINT64_MAX , clearing the last lock resets the hint to 0.
- delegrecall_skip_for_same_client_locks() uses this hint to check instead
of iterating file.lock_list.
This test case is from nfstest delegation suite -
https://github.com/kofemann/nfstest/blob/master/test/nfstest_delegation
Change-Id: I1e33457561db047b311d310e5e1238d4d6f69065
Signed-off-by: Suhas Athani <Suhas.Athani(a)ibm.com>
---
M src/FSAL_UP/fsal_up_top.c
M src/SAL/state_lock.c
M src/include/sal_data.h
M src/include/sal_functions.h
4 files changed, 129 insertions(+), 37 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/66/1238566/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238566?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I1e33457561db047b311d310e5e1238d4d6f69065
Gerrit-Change-Number: 1238566
Gerrit-PatchSet: 1
Gerrit-Owner: Suhas Athani <Suhas.Athani(a)ibm.com>
1 month
[XL] Change in ...nfs-ganesha[next]: Implement Internal NSM
by Frank Filz (GerritHub)
Frank Filz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238558?usp=email )
Change subject: Implement Internal NSM
......................................................................
Implement Internal NSM
Add recovery database for recovery_fs and NSM code to support Ganesha
taking over from rpc.statd. Ganesha will send SM_NOTIFY from the database
and process SM_NOTIFY sent from clients.
Change-Id: I8bcb3144cf1bea063c2f4800fe623d4b48045459
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
---
M src/CMakeLists.txt
M src/MainNFSD/libganesha_nfsd.ver
M src/MainNFSD/nfs_admin_thread.c
M src/MainNFSD/nfs_init.c
M src/MainNFSD/nfs_rpc_dispatcher_thread.c
M src/MainNFSD/nfs_worker_thread.c
M src/Protocols/NLM/CMakeLists.txt
M src/Protocols/NLM/nlm_util.c
M src/Protocols/NLM/nsm.c
M src/Protocols/XDR/xdr_nsm.c
M src/SAL/nfs4_recovery.c
M src/SAL/nlm_owner.c
M src/SAL/recovery/recovery_fs.c
M src/SAL/state_lock.c
M src/include/gsh_recovery.h
M src/include/nfs_init.h
M src/include/nfs_proto_data.h
M src/include/nfs_proto_functions.h
M src/include/nsm.h
M src/include/sal_data.h
M src/include/sal_functions.h
M src/support/nfs_read_conf.c
22 files changed, 1,668 insertions(+), 33 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/58/1238558/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238558?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I8bcb3144cf1bea063c2f4800fe623d4b48045459
Gerrit-Change-Number: 1238558
Gerrit-PatchSet: 1
Gerrit-Owner: Frank Filz <ffilzlnx(a)mindspring.com>
1 month
[S] Change in ...nfs-ganesha[next]: SAL: recover gracefully from zombie lock-owner so_related_owner mismatch
by Nishant Puri (GerritHub)
Nishant Puri has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238516?usp=email )
Change subject: SAL: recover gracefully from zombie lock-owner so_related_owner mismatch
......................................................................
SAL: recover gracefully from zombie lock-owner so_related_owner mismatch
When a kernel NFSv4 client crashes or its lease expires while holding a
byte-range lock, the lock-owner ID it generated via the IDA allocator
(ida_simple_get) may be recycled for a completely new process. When that
new process opens the same file and requests a lock, the server calls
create_nfs4_owner() and finds an existing STATE_LOCK_OWNER_NFSV4 entry in
ht_nfs4_owner whose opaque owner bytes match -- but whose so_related_owner
points to the old (now-stale) open owner rather than the new one.
Before this change the code unconditionally logged a CRIT message and
returned NULL, which propagated to the client as NFS4ERR_RESOURCE.
Because the client retried indefinitely the file appeared permanently
"zombie locked" from that perspective.
The root cause on the client side is IDA-based lock-owner ID recycling
(fixed in upstream Linux kernels by switching to atomic64_inc_return, but
still present in RHEL 8 / kernel 4.18 based deployments).
The fix distinguishes two cases at the point of mismatch:
1. so_lock_list is non-empty (zombie still holds active POSIX locks):
This is a genuine conflict. Keep the existing CRIT log and return
NULL so the client receives NFS4ERR_RESOURCE and can retry cleanly.
2. so_lock_list is empty (zombie has no active POSIX locks):
The lock owner is a harmless zombie whose POSIX-lock state was
already released (e.g. after LOCKU without a subsequent CLOSE or
FREE_STATEID). Re-associate so_related_owner to the new open owner
instead of failing, allowing the incoming LOCK request to succeed.
Reference counts are maintained correctly: the stale open-owner reference
is decremented and the new one is incremented, both under so_mutex.
Change-Id: I9b76a2582b9066b196b01c75a2777a6dd335ee9a
Signed-off-by: Nishant Puri <npuri(a)redhat.com>
---
M src/SAL/nfs4_owner.c
1 file changed, 33 insertions(+), 7 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/16/1238516/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238516?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I9b76a2582b9066b196b01c75a2777a6dd335ee9a
Gerrit-Change-Number: 1238516
Gerrit-PatchSet: 1
Gerrit-Owner: Nishant Puri <npuri(a)redhat.com>
1 month
[XS] Change in ...nfs-ganesha[next]: Fix dir chunk ref count leak
by Name of user not set (GerritHub)
shaharhoch(a)google.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238513?usp=email )
Change subject: Fix dir chunk ref count leak
......................................................................
Fix dir chunk ref count leak
A recent change added a skip to a failure flow, but it called `mdcache_lru_unref_chunk` before the skip, but if we goto skip, we also unref the chunk there.
This causes us to to dec-ref the chunk twice and release it while other dirs still reference to it.
This can cause us memory corruption.
Change-Id: I685b8e6977deeaa670cd80ebed041e79c8039252
Signed-off-by: Shahar Hochma <shaharhoch(a)google.com>
---
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c
1 file changed, 2 insertions(+), 2 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/13/1238513/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1238513?usp=email
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email
Gerrit-MessageType: newchange
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I685b8e6977deeaa670cd80ebed041e79c8039252
Gerrit-Change-Number: 1238513
Gerrit-PatchSet: 1
Gerrit-Owner: shaharhoch(a)google.com
1 month, 1 week