Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None`
Hi — a design question rather than a bug report, and I'd like to check whether we're leaning on behaviour that is intended before we build on it. ### Setup - NFS-Ganesha **V6.5**, built with `USE_FSAL_VFS=ON`, and notably `USE_RADOS_RECOV=OFF` / `USE_RADOS_GRACE=OFF`, so the rados recovery backends are not available to us. - One Ganesha instance per volume, running in a container, single server (no active/active, no clustering). - The container exports a single block-backed filesystem, and the export is the **root** of that filesystem: ``` EXPORT { Export_Id = 1; Path = /data/export; Pseudo = /; Access_Type = RW; Squash = No_Root_Squash; SecType = sys; Protocols = 4; FSAL { Name = VFS; } } ``` ### Problem `RecoveryBackend = fs` defaults its `RecoveryRoot` to `/var/lib/nfs/ganesha`, which in our container is ephemeral — it is discarded on every restart, so `clid_count` is always 0 on start-up and no client can reclaim (`nfs4_op_open.c` returns `NFS4ERR_NO_GRACE` for `CLAIM_PREVIOUS`). The only durable storage the server has is the exported filesystem itself, which follows the server when it moves. So we would like `RecoveryRoot` to live on that filesystem — but since we export its root, the recovery directory would sit inside the exported tree, visible and writable by clients. ### What we're considering Add a second export covering just the recovery directory, with no access: ``` EXPORT { Export_Id = 2; Path = /data/export/.ganesha; Pseudo = /.ganesha; Access_Type = None; Protocols = 4; FSAL { Name = VFS; } } ``` ### What we observed (V6.5, FSAL_VFS) Both exports load: ``` Export ... 1 pseudo (/) path (/data/export) perms (... no_root_squash, RWrw, ...) Export ... 2 pseudo (/.ganesha) path (/data/export/.ganesha) perms (... ----, ...) NFSv4 pseudo file system successfully initialized ``` From a client — including **root** under `No_Root_Squash`: - `ls -a` on the share does **not** list `.ganesha` at all - `stat`/`cat` on that path return `ENOENT`, not `EACCES` - `rm -rf <share>/* <share>/.[!.]*` removes everything else and leaves the recovery data untouched - the parent export continues to serve reads and writes normally Controlled by removing only the second export and restarting: the same directory (same inode, mode 0700) then appears in `ls -a` and its contents are readable over NFS. So the nested export is unambiguously what makes it unreachable. ### Questions 1. **Is this a supported use of nested exports**, or are we relying on an accident? Is there an intended mechanism for excluding a subdirectory from an export that we've missed? 2. **Is the READDIR filtering intentional and stable?** We expected `EACCES` on traversal and instead the junction is omitted from the listing entirely, which is nicer for us — but we don't want to depend on it if it's incidental. 3. **Any correctness concern with `RecoveryRoot` living inside an exported tree?** The server writes its own client database into a path that is nominally part of an export, albeit one no client can enter. Is there re-entrancy, FSAL, or grace-period interaction we should worry about? 4. For a single-server, one-instance-per-volume deployment where the recovery directory travels with the exported filesystem, **is `fs` or `fs_ng` preferable?** We read `recovery_fs_ng.c` as adding atomic generation swaps via `rename()` + `symlink()`; our backing filesystem is ext4, so both are viable.
What a clever trick. I really hadn't given much thought to doing this. But yes it works. And yes, it is utilizing functionality in an intended and supported way. And yes, the "invisible" directory is intentional. That aspect was desired by those interested in multi-tenancy where you don't want the other tenants to have any idea of the location or even existence of other tenants. One tiny caveat - Ganesha has no subtree checking for handles so handle guessing can be used to access files not intended to be visible. As to fs vs fs_ng, I'm insufficiently knowledgeable about the changes for ng so I can't offer much advice there. If fs_ng works given testing, then that is probably the best way to go. Frank -----Original Message----- From: purkaitabhinandan@gmail.com [mailto:purkaitabhinandan@gmail.com] Sent: Tuesday, August 18, 2026 9:42 PM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None` Hi — a design question rather than a bug report, and I'd like to check whether we're leaning on behaviour that is intended before we build on it. ### Setup - NFS-Ganesha **V6.5**, built with `USE_FSAL_VFS=ON`, and notably `USE_RADOS_RECOV=OFF` / `USE_RADOS_GRACE=OFF`, so the rados recovery backends are not available to us. - One Ganesha instance per volume, running in a container, single server (no active/active, no clustering). - The container exports a single block-backed filesystem, and the export is the **root** of that filesystem: ``` EXPORT { Export_Id = 1; Path = /data/export; Pseudo = /; Access_Type = RW; Squash = No_Root_Squash; SecType = sys; Protocols = 4; FSAL { Name = VFS; } } ``` ### Problem `RecoveryBackend = fs` defaults its `RecoveryRoot` to `/var/lib/nfs/ganesha`, which in our container is ephemeral — it is discarded on every restart, so `clid_count` is always 0 on start-up and no client can reclaim (`nfs4_op_open.c` returns `NFS4ERR_NO_GRACE` for `CLAIM_PREVIOUS`). The only durable storage the server has is the exported filesystem itself, which follows the server when it moves. So we would like `RecoveryRoot` to live on that filesystem — but since we export its root, the recovery directory would sit inside the exported tree, visible and writable by clients. ### What we're considering Add a second export covering just the recovery directory, with no access: ``` EXPORT { Export_Id = 2; Path = /data/export/.ganesha; Pseudo = /.ganesha; Access_Type = None; Protocols = 4; FSAL { Name = VFS; } } ``` ### What we observed (V6.5, FSAL_VFS) Both exports load: ``` Export ... 1 pseudo (/) path (/data/export) perms (... no_root_squash, RWrw, ...) Export ... 2 pseudo (/.ganesha) path (/data/export/.ganesha) perms (... ----, ...) NFSv4 pseudo file system successfully initialized ``` From a client — including **root** under `No_Root_Squash`: - `ls -a` on the share does **not** list `.ganesha` at all - `stat`/`cat` on that path return `ENOENT`, not `EACCES` - `rm -rf <share>/* <share>/.[!.]*` removes everything else and leaves the recovery data untouched - the parent export continues to serve reads and writes normally Controlled by removing only the second export and restarting: the same directory (same inode, mode 0700) then appears in `ls -a` and its contents are readable over NFS. So the nested export is unambiguously what makes it unreachable. ### Questions 1. **Is this a supported use of nested exports**, or are we relying on an accident? Is there an intended mechanism for excluding a subdirectory from an export that we've missed? 2. **Is the READDIR filtering intentional and stable?** We expected `EACCES` on traversal and instead the junction is omitted from the listing entirely, which is nicer for us — but we don't want to depend on it if it's incidental. 3. **Any correctness concern with `RecoveryRoot` living inside an exported tree?** The server writes its own client database into a path that is nominally part of an export, albeit one no client can enter. Is there re-entrancy, FSAL, or grace-period interaction we should worry about? 4. For a single-server, one-instance-per-volume deployment where the recovery directory travels with the exported filesystem, **is `fs` or `fs_ng` preferable?** We read `recovery_fs_ng.c` as adding atomic generation swaps via `rename()` + `symlink()`; our backing filesystem is ext4, so both are viable. _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
On Wed, 19 Aug 2026 15:59:21 -0700, "Frank Filz" <ffilzlnx@mindspring.com> wrote:
One tiny caveat - Ganesha has no subtree checking for handles so handle guessing can be used to access files not intended to be visible.
We are planning to use FSID_Override=True to force per-export Filesystem_ID's into handles. We can then specify a unique Filesystem_ID per export to avoid this issue. We still need to understand the negatives connotations of doing this.
As to fs vs fs_ng, I'm insufficiently knowledgeable about the changes for ng so I can't offer much advice there.
If fs_ng works given testing, then that is probably the best way to go.
When I originally had to make this decision, I found comments in fs_ng indicating that it might be incomplete. These days, fs at least supports RecoveryBackendIPBased=True, so should be conceptually simpler to use. I'm obviously interested in further discussion of these ideas... :-) peace & happiness, martin
Thanks Martin. The FSID_Override idea is the first thing I've seen that would actually close the hole Frank mentioned, so I'm interested in how it goes. Issue is we're on 6.5 and neither of the things you mention seems to be there. As far as I can tell fsid_override is only parsed in `nfs_read_conf.c` from V15.1, and `recovery_fs.c` doesn't reference `recovery_backend_ipbased` until V15.1 either, nothing at V15. Neither string is in our binary. So: which version are you doing FSID_Override on, next or a backport onto something older? Does turning it on change handles that clients already hold? We'd be enabling it under live mounts, so I want to know whether to expect a round of ESTALE or whether it only affects handles issued afterwards. If you haven't got to the negatives yet, that's the one I'd look at first. And how are you picking the Filesystem_Id per export? We create an export per volume, dynamically, so I need something that generates them rather than a number in a config file. Curious whether you're deriving it from something stable or just counting. On RecoveryBackendIPBased, which release were you on when you used it? If it only landed in V15.1 and you had it before then, you're presumably carrying a backport, and I'd like to know, because we may end up wanting the same. We hit the same problem, but solved it by patch rather than config since 6.5 gives us no option: we drop the client's source address from the recovery tag entirely. Under a container network that address flips between the bridge and the VXLAN interface depending on whether client and server share a node, so moving the server rekeyed every client, not just the one that moved. Tags are stable across nodes now. One more thing, do you remember what the fs_ng comments actually said, or what you ran into? We've taken your advice and stayed on fs, but I'd rather know the real gap than avoid it on reputation. cheers, Abhinandan
Sorry I have not updated the Wiki with the latest supported version... We switched to a model where we really only support the next branch at the latest level. In truth, we will help folks several major versions back from that (currently 15.1 and several versions back from that), but if someone hits a problem that seems like it might have been addressed, we would ask them to try the latest version. The major version number bumps whenever we (I) merge a significant feature patch. RecoveryBackendIPBased should be a few versions back from 15.1, but definitely nowhere close to 6.5. Frank -----Original Message----- From: purkaitabhinandan@gmail.com [mailto:purkaitabhinandan@gmail.com] Sent: Wednesday, August 19, 2026 9:06 PM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None` Thanks Martin. The FSID_Override idea is the first thing I've seen that would actually close the hole Frank mentioned, so I'm interested in how it goes. Issue is we're on 6.5 and neither of the things you mention seems to be there. As far as I can tell fsid_override is only parsed in `nfs_read_conf.c` from V15.1, and `recovery_fs.c` doesn't reference `recovery_backend_ipbased` until V15.1 either, nothing at V15. Neither string is in our binary. So: which version are you doing FSID_Override on, next or a backport onto something older? Does turning it on change handles that clients already hold? We'd be enabling it under live mounts, so I want to know whether to expect a round of ESTALE or whether it only affects handles issued afterwards. If you haven't got to the negatives yet, that's the one I'd look at first. And how are you picking the Filesystem_Id per export? We create an export per volume, dynamically, so I need something that generates them rather than a number in a config file. Curious whether you're deriving it from something stable or just counting. On RecoveryBackendIPBased, which release were you on when you used it? If it only landed in V15.1 and you had it before then, you're presumably carrying a backport, and I'd like to know, because we may end up wanting the same. We hit the same problem, but solved it by patch rather than config since 6.5 gives us no option: we drop the client's source address from the recovery tag entirely. Under a container network that address flips between the bridge and the VXLAN interface depending on whether client and server share a node, so moving the server rekeyed every client, not just the one that moved. Tags are stable across nodes now. One more thing, do you remember what the fs_ng comments actually said, or what you ran into? We've taken your advice and stayed on fs, but I'd rather know the real gap than avoid it on reputation. cheers, Abhinandan _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Hi Abhinandan, On Thu, 20 Aug 2026 04:05:51 -0000, purkaitabhinandan@gmail.com wrote:
Thanks Martin. The FSID_Override idea is the first thing I've seen that would actually close the hole Frank mentioned, so I'm interested in how it goes.
Issue is we're on 6.5 and neither of the things you mention seems to be there. As far as I can tell fsid_override is only parsed in `nfs_read_conf.c` from V15.1, and `recovery_fs.c` doesn't reference `recovery_backend_ipbased` until V15.1 either, nothing at V15. Neither string is in our binary.
So: which version are you doing FSID_Override on, next or a backport onto something older?
We're on something older than you. We're on V5.9 with >160 cherry-picks/backports, mostly bug fixes.
Does turning it on change handles that clients already hold? We'd be enabling it under live mounts, so I want to know whether to expect a round of ESTALE or whether it only affects handles issued afterwards. If you haven't got to the negatives yet, that's the one I'd look at first.
We're doing loosely clustered NFS-Ganesha but we don't expect this type of change to work across online upgrades. One limitation is that you can't change the value of Filesystem_ID while NFS-Ganesha is up... or FSID_override either, I think.
And how are you picking the Filesystem_Id per export? We create an export per volume, dynamically, so I need something that generates them rather than a number in a config file. Curious whether you're deriving it from something stable or just counting.
My plan is to pick a Filesystem_ID major for all exports for a site and set the minor to be equal to the Export_ID. This seems to be a common approach. I haven't thought through multitenancy, but my starting point would be a tenant ID as major and continue with Export_ID for minor. It is a nice and dumb approach... but needs to be carefully considered.
On RecoveryBackendIPBased, which release were you on when you used it? If it only landed in V15.1 and you had it before then, you're presumably carrying a backport, and I'd like to know, because we may end up wanting the same.
Yeah, again, we've cherry-picked/backported to V5.9.
We hit the same problem, but solved it by patch rather than config since 6.5 gives us no option: we drop the client's source address from the recovery tag entirely. Under a container network that address flips between the bridge and the VXLAN interface depending on whether client and server share a node, so moving the server rekeyed every client, not just the one that moved. Tags are stable across nodes now.
One more thing, do you remember what the fs_ng comments actually said, or what you ran into? We've taken your advice and stayed on fs, but I'd rather know the real gap than avoid it on reputation.
I think it was this that stopped me from digging deeper: #ifndef FIXME return; #else /** * More investigation/design is required to determine the best * implementation for the remainder of this method. In the interim, * Coverity complains that this is dead code because of the early * return above. Hence the remainder of this method is ifdef'ed * out to avoid the Coverity complaint. * * @todo: FIXME: make the rest of this work */ :-) I think, after seeing this, and the lack of comments anywhere describing how fs_ng is meant to be better, I cut my losses early and decided to see if we could make something work using fs. That was quite tricky... but now trivial with RecoveryBackendIPBased. peace & happiness, martin
How does FSID_Override prevent handle guessing? You can take the inode portion of a handle, and wrap it in export ID and FSID that match an export you have access to. The open_by_handle_at system call will then happily access the file or directory. Good to know about recovery_fs_ng. We should take a look at that code and figure out if it makes any sense and either make it usable or discard it. Frank -----Original Message----- From: Martin Schwenke [mailto:martin@meltin.net] Sent: Wednesday, August 19, 2026 8:20 PM To: devel@lists.nfs-ganesha.org Cc: Frank Filz <ffilzlnx@mindspring.com>; purkaitabhinandan@gmail.com Subject: Re: [NFS-Ganesha-Devel] Re: Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None` On Wed, 19 Aug 2026 15:59:21 -0700, "Frank Filz" <ffilzlnx@mindspring.com> wrote:
One tiny caveat - Ganesha has no subtree checking for handles so handle guessing can be used to access files not intended to be visible.
We are planning to use FSID_Override=True to force per-export Filesystem_ID's into handles. We can then specify a unique Filesystem_ID per export to avoid this issue. We still need to understand the negatives connotations of doing this.
As to fs vs fs_ng, I'm insufficiently knowledgeable about the changes for ng so I can't offer much advice there.
If fs_ng works given testing, then that is probably the best way to go.
When I originally had to make this decision, I found comments in fs_ng indicating that it might be incomplete. These days, fs at least supports RecoveryBackendIPBased=True, so should be conceptually simpler to use. I'm obviously interested in further discussion of these ideas... :-) peace & happiness, martin
On Thu, 20 Aug 2026 17:57:44 -0700, "Frank Filz" <ffilzlnx@mindspring.com> wrote:
How does FSID_Override prevent handle guessing?
You can take the inode portion of a handle, and wrap it in export ID and FSID that match an export you have access to. The open_by_handle_at system call will then happily access the file or directory.
Good point! That is still possible... To secure it properly, we would have to use a subdirectory mount of the cluster filesystem per export. With Lustre we would then need the distinct FSID on each export to allow NFS-Ganesha to find the "filesystem" associated with the export to get the mount_fd to do open_by_handle_at(). I know this will be secure because resolving a handle from a subsequent subdirectory mount doesn't work (ESTALE) if we use the mount_fd for the first subdirectory mount... because all the subdirectory mounts have the same default FSID. That's the behaviour we see without FSID_Override=True.
Good to know about recovery_fs_ng. We should take a look at that code and figure out if it makes any sense and either make it usable or discard it.
Yeah, as per my other reply, it wasn't a very educated guess. I took a fairly quick look and gave up early... peace & happiness, martin
Thanks, both of you. I came in with four fairly narrow questions and got much more than that out of it, which I appreciate. Quick update on where we landed. We've moved off 6.5 onto 15.1 and dropped the patch we were carrying. The config that replaces it turned out to be RecoverySkipIp rather than RecoveryBackendIPBased — I had assumed those were two names for the same thing, and that was my own misreading rather than anything either of you said. As far as I can tell SkipIp is what drops the client address from the tag in nfs4_create_clid_name, while IPBased groups the database into ip_<addr> directories keyed on cr_server_addr. Two different halves, and I only worked that out after testing. Martin, I did try to follow your lead on IPBased before I understood that, so it may be worth mentioning why it didn't fit us — not a criticism of the advice, which is clearly right for your setup. You've got a service IP that clients follow, so whichever node takes the IP reads that IP's directory and everyone reclaims. We don't have a floating address at all: clients reach a per-volume ClusterIP, and the server ends up recording its own pod IP as cr_server_addr, which changes whenever the pod is rescheduled. So IPBased would have sharded our database by the one address guaranteed to move. We're running it off with SkipIp on instead. Otherwise we're staying on fs, and keeping the nested Access_Type = None export for the recovery directory. Frank, your point about wrapping an inode in an export ID and FSID you do have access to was the useful correction there — we've stopped treating that export as any kind of boundary and now describe it as simply keeping the directory out of the way. And Martin, thanks for coming back on fs_ng; the FIXME is exactly what I was after, rather than avoiding it on reputation alone. If any of that reading looks off to either of you, I'd be glad to hear it — particularly the SkipIp/IPBased split, and whether fs plus a denied nested export is a combination you'd expect to keep working. No rush, and no need to reply if it all sounds reasonable. Otherwise I'll stop taking up list time. Thanks again, genuinely helpful. cheers, Abhinandan On Fri, 21 Aug 2026 at 08:22, Martin Schwenke <martin@meltin.net> wrote:
On Thu, 20 Aug 2026 17:57:44 -0700, "Frank Filz" <ffilzlnx@mindspring.com> wrote:
How does FSID_Override prevent handle guessing?
You can take the inode portion of a handle, and wrap it in export ID and FSID that match an export you have access to. The open_by_handle_at system call will then happily access the file or directory.
Good point! That is still possible...
To secure it properly, we would have to use a subdirectory mount of the cluster filesystem per export. With Lustre we would then need the distinct FSID on each export to allow NFS-Ganesha to find the "filesystem" associated with the export to get the mount_fd to do open_by_handle_at().
I know this will be secure because resolving a handle from a subsequent subdirectory mount doesn't work (ESTALE) if we use the mount_fd for the first subdirectory mount... because all the subdirectory mounts have the same default FSID. That's the behaviour we see without FSID_Override=True.
Good to know about recovery_fs_ng. We should take a look at that code and figure out if it makes any sense and either make it usable or discard it.
Yeah, as per my other reply, it wasn't a very educated guess. I took a fairly quick look and gave up early...
peace & happiness, martin
I think you’re good. And the implementation bits you are relying on are stable and won’t go away. You have a novel idea of using an export to hide an internal use directory, but the idea of having different access (including hiding) directories via exports is an intended feature of Ganesha exports. Some filesystems implement sub-volumes in a way that even improves the security to eliminate the risk of handle guessing. Frank From: Abhinandan Purkait [mailto:purkaitabhinandan@gmail.com] Sent: Thursday, August 20, 2026 9:00 PM To: Martin Schwenke <martin@meltin.net> Cc: Frank Filz <ffilzlnx@mindspring.com>; devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None` Thanks, both of you. I came in with four fairly narrow questions and got much more than that out of it, which I appreciate. Quick update on where we landed. We've moved off 6.5 onto 15.1 and dropped the patch we were carrying. The config that replaces it turned out to be RecoverySkipIp rather than RecoveryBackendIPBased — I had assumed those were two names for the same thing, and that was my own misreading rather than anything either of you said. As far as I can tell SkipIp is what drops the client address from the tag in nfs4_create_clid_name, while IPBased groups the database into ip_<addr> directories keyed on cr_server_addr. Two different halves, and I only worked that out after testing. Martin, I did try to follow your lead on IPBased before I understood that, so it may be worth mentioning why it didn't fit us — not a criticism of the advice, which is clearly right for your setup. You've got a service IP that clients follow, so whichever node takes the IP reads that IP's directory and everyone reclaims. We don't have a floating address at all: clients reach a per-volume ClusterIP, and the server ends up recording its own pod IP as cr_server_addr, which changes whenever the pod is rescheduled. So IPBased would have sharded our database by the one address guaranteed to move. We're running it off with SkipIp on instead. Otherwise we're staying on fs, and keeping the nested Access_Type = None export for the recovery directory. Frank, your point about wrapping an inode in an export ID and FSID you do have access to was the useful correction there — we've stopped treating that export as any kind of boundary and now describe it as simply keeping the directory out of the way. And Martin, thanks for coming back on fs_ng; the FIXME is exactly what I was after, rather than avoiding it on reputation alone. If any of that reading looks off to either of you, I'd be glad to hear it — particularly the SkipIp/IPBased split, and whether fs plus a denied nested export is a combination you'd expect to keep working. No rush, and no need to reply if it all sounds reasonable. Otherwise I'll stop taking up list time. Thanks again, genuinely helpful. cheers, Abhinandan On Fri, 21 Aug 2026 at 08:22, Martin Schwenke <martin@meltin.net <mailto:martin@meltin.net> > wrote: On Thu, 20 Aug 2026 17:57:44 -0700, "Frank Filz" <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > wrote:
How does FSID_Override prevent handle guessing?
You can take the inode portion of a handle, and wrap it in export ID and FSID that match an export you have access to. The open_by_handle_at system call will then happily access the file or directory.
Good point! That is still possible... To secure it properly, we would have to use a subdirectory mount of the cluster filesystem per export. With Lustre we would then need the distinct FSID on each export to allow NFS-Ganesha to find the "filesystem" associated with the export to get the mount_fd to do open_by_handle_at(). I know this will be secure because resolving a handle from a subsequent subdirectory mount doesn't work (ESTALE) if we use the mount_fd for the first subdirectory mount... because all the subdirectory mounts have the same default FSID. That's the behaviour we see without FSID_Override=True.
Good to know about recovery_fs_ng. We should take a look at that code and figure out if it makes any sense and either make it usable or discard it.
Yeah, as per my other reply, it wasn't a very educated guess. I took a fairly quick look and gave up early... peace & happiness, martin
Hi, We ran into what looks like a bug in the fs recovery backend on V15.1. We have a one-line change that fixes it for us, and I'd like to check with you whether it is the right fix before we rely on it. In fs_reclaim_complete() (src/SAL/recovery/recovery_fs.c) the marker name is copied into an uninitialised path[PATH_MAX] without its terminator: path[dstpos] = '/'; memcpy(path + dstpos + 1, reclaim_complete_marker, marker_len); fd = creat(path, 0700); so creat() picks up whatever followed on the stack. On our builds that is a display_client_id_rec() dump from an earlier frame, giving entries like v4recov/(27:Linux NFSv4.1 node-1)/reclaim_completenf = (nil) {NULL} unconf = ... fs_check_reclaim_complete() then looks for the exact name and misses it, so check_clid() refuses CLAIM_PREVIOUS for every NFSv4.1 client: locks are dropped without the client noticing, and grace always runs to Grace_Period because reclaim_completes never reaches the clid count. It reads as intermittent, since whether the trailing bytes happen to start with a NUL is luck. Measured on three nodes, one fcntl(F_WRLCK) holder each, NFSv4.1 only: across three restarts the marker name predicted the outcome every time, and the client-visible I/O pause was ~118s instead of ~34s. What we are running is marker_len + 1 in that memcpy, so the terminator comes with it — the bounds check just above already reserves the byte. That gives us 3/3 reclaims and grace lifting early. Is that the correct fix in your view, or would you rather see it done another way — an snprintf here would match the reader side. Also happy to hear if you think we are misreading the cause. Thanks, Abhinandan On Fri, 21 Aug 2026 at 21:16, Frank Filz <ffilzlnx@mindspring.com> wrote:
I think you’re good. And the implementation bits you are relying on are stable and won’t go away. You have a novel idea of using an export to hide an internal use directory, but the idea of having different access (including hiding) directories via exports is an intended feature of Ganesha exports. Some filesystems implement sub-volumes in a way that even improves the security to eliminate the risk of handle guessing.
Frank
*From:* Abhinandan Purkait [mailto:purkaitabhinandan@gmail.com] *Sent:* Thursday, August 20, 2026 9:00 PM *To:* Martin Schwenke <martin@meltin.net> *Cc:* Frank Filz <ffilzlnx@mindspring.com>; devel@lists.nfs-ganesha.org *Subject:* [NFS-Ganesha-Devel] Re: Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None`
Thanks, both of you. I came in with four fairly narrow questions and got much more than that out of it, which I appreciate.
Quick update on where we landed. We've moved off 6.5 onto 15.1 and dropped the patch we were carrying. The config that replaces it turned out to be RecoverySkipIp rather than RecoveryBackendIPBased — I had assumed those were two names for the same thing, and that was my own misreading rather than anything either of you said. As far as I can tell SkipIp is what drops the client address from the tag in nfs4_create_clid_name, while IPBased groups the database into ip_<addr> directories keyed on cr_server_addr. Two different halves, and I only worked that out after testing.
Martin, I did try to follow your lead on IPBased before I understood that, so it may be worth mentioning why it didn't fit us — not a criticism of the advice, which is clearly right for your setup. You've got a service IP that clients follow, so whichever node takes the IP reads that IP's directory and everyone reclaims. We don't have a floating address at all: clients reach a per-volume ClusterIP, and the server ends up recording its own pod IP as cr_server_addr, which changes whenever the pod is rescheduled. So IPBased would have sharded our database by the one address guaranteed to move. We're running it off with SkipIp on instead.
Otherwise we're staying on fs, and keeping the nested Access_Type = None export for the recovery directory. Frank, your point about wrapping an inode in an export ID and FSID you do have access to was the useful correction there — we've stopped treating that export as any kind of boundary and now describe it as simply keeping the directory out of the way. And Martin, thanks for coming back on fs_ng; the FIXME is exactly what I was after, rather than avoiding it on reputation alone.
If any of that reading looks off to either of you, I'd be glad to hear it — particularly the SkipIp/IPBased split, and whether fs plus a denied nested export is a combination you'd expect to keep working. No rush, and no need to reply if it all sounds reasonable. Otherwise I'll stop taking up list time. Thanks again, genuinely helpful.
cheers, Abhinandan
On Fri, 21 Aug 2026 at 08:22, Martin Schwenke <martin@meltin.net> wrote:
On Thu, 20 Aug 2026 17:57:44 -0700, "Frank Filz" <ffilzlnx@mindspring.com> wrote:
How does FSID_Override prevent handle guessing?
You can take the inode portion of a handle, and wrap it in export ID and FSID that match an export you have access to. The open_by_handle_at system call will then happily access the file or directory.
Good point! That is still possible...
To secure it properly, we would have to use a subdirectory mount of the cluster filesystem per export. With Lustre we would then need the distinct FSID on each export to allow NFS-Ganesha to find the "filesystem" associated with the export to get the mount_fd to do open_by_handle_at().
I know this will be secure because resolving a handle from a subsequent subdirectory mount doesn't work (ESTALE) if we use the mount_fd for the first subdirectory mount... because all the subdirectory mounts have the same default FSID. That's the behaviour we see without FSID_Override=True.
Good to know about recovery_fs_ng. We should take a look at that code and figure out if it makes any sense and either make it usable or discard it.
Yeah, as per my other reply, it wasn't a very educated guess. I took a fairly quick look and gave up early...
peace & happiness, martin
Thanks Frank, that's what I needed. The multi-tenancy reasoning is the reassuring part, since it means the invisible junction is something someone maintains rather than something I'd be getting away with. So we'll go ahead: RecoveryRoot on the exported filesystem, hidden behind the nested export, staying on fs. Two things I got stuck on afterwards, both about versions. Martin suggested FSID_Override, and mentioned that fs supports RecoveryBackendIPBased these days. Neither seems to exist in what we build. `recovery_fs.c` doesn't mention `recovery_backend_ipbased` at V15 and mentions it ten times at V15.1, and `nfs_read_conf.c` only parses `fsid_override` from V15.1. Neither string is in our 6.5 binary. Against that, the wiki says 6.5 is the current supported release, the newest stable branch is V6-stable, and the V15.1 tag looks identical to next - your announce mail says "Branch next" as well. So which is it, from a packager's point of view? Is 6.5 still what you'd want a downstream shipping, with V7 to V15 being development snapshots? Is a new stable branch coming, and from which tag? And does V6-stable take backports at all - would RecoveryBackendIPBased be welcome there, or is the answer simply to move off 6.5? I ask because we already carry a local patch for this. On 6.5 a client can't reclaim once we reschedule the server, because `fs_create_clid_name()` puts the client's source address at the front of the tag, and under a container network that address depends on whether the client happens to land on the same node as the server. Same client, bridge address when co-located, VXLAN address when not. Moving the server rekeyed every client, not just the one that moved, so no amount of durable storage would have helped. We drop the address: - (void)snprintf(clientid->cid_recov_tag, total_size, - "%s-(%s:%s)", str_client_addr, cidstr_lenx, - cidstr); + (void)snprintf(clientid->cid_recov_tag, total_size, "(%s:%s)", + cidstr_lenx, cidstr); Tags come out byte-identical wherever the server runs; I checked across three nodes covering each co-location combination. It covers fs_ng too, since `fs_ng_backend.add_clid` is `fs_add_clid`. It does need client hostnames to be unique, which is fine on Kubernetes. Reading V15.1, that looks like `RecoveryBackendIPBased = false`. Is it the same thing, or does the option do more than change the tag? And is there anything on 6.5 that reads the tag back and expects the old `<addr>-(len:id)` shape - takeover, the reaper, `fs_read_recov_clids_recover()`? That's the part I'm least sure of, and the patch is already in our images, so I'd rather ask than find out. Happy to put it on Gerrit against V6-stable if it's any use to anyone else. Last thing, your caveat about handles. Martin's fix needs V15.1, so it isn't available to us either. Is there anything on 6.5 that stops a handle for a file under the hidden export working through the parent export, or is upgrading the only answer? And how worried should I be? For FSAL_VFS the handle is the kernel's, inode plus generation, so guessing feels theoretical to me, but you'd know better. Worst case I can see is someone wiping the recovery entries, which only puts us back to no reclaim at all. Thanks, Abhinandan
participants (4)
-
Abhinandan Purkait -
Frank Filz -
Martin Schwenke -
purkaitabhinandan＠gmail.com