What a clever trick. I really hadn't given much thought to doing this. But yes it works. And yes, it is utilizing functionality in an intended and supported way. And yes, the "invisible" directory is intentional. That aspect was desired by those interested in multi-tenancy where you don't want the other tenants to have any idea of the location or even existence of other tenants. One tiny caveat - Ganesha has no subtree checking for handles so handle guessing can be used to access files not intended to be visible. As to fs vs fs_ng, I'm insufficiently knowledgeable about the changes for ng so I can't offer much advice there. If fs_ng works given testing, then that is probably the best way to go. Frank -----Original Message----- From: purkaitabhinandan(a)gmail.com [mailto:purkaitabhinandan@gmail.com] Sent: Tuesday, August 18, 2026 9:42 PM To: devel(a)lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None` Hi — a design question rather than a bug report, and I'd like to check whether we're leaning on behaviour that is intended before we build on it. ### Setup - NFS-Ganesha **V6.5**, built with `USE_FSAL_VFS=ON`, and notably `USE_RADOS_RECOV=OFF` / `USE_RADOS_GRACE=OFF`, so the rados recovery backends are not available to us. - One Ganesha instance per volume, running in a container, single server (no active/active, no clustering). - The container exports a single block-backed filesystem, and the export is the **root** of that filesystem: ``` EXPORT { Export_Id = 1; Path = /data/export; Pseudo = /; Access_Type = RW; Squash = No_Root_Squash; SecType = sys; Protocols = 4; FSAL { Name = VFS; } } ``` ### Problem `RecoveryBackend = fs` defaults its `RecoveryRoot` to `/var/lib/nfs/ganesha`, which in our container is ephemeral — it is discarded on every restart, so `clid_count` is always 0 on start-up and no client can reclaim (`nfs4_op_open.c` returns `NFS4ERR_NO_GRACE` for `CLAIM_PREVIOUS`). The only durable storage the server has is the exported filesystem itself, which follows the server when it moves. So we would like `RecoveryRoot` to live on that filesystem — but since we export its root, the recovery directory would sit inside the exported tree, visible and writable by clients. ### What we're considering Add a second export covering just the recovery directory, with no access: ``` EXPORT { Export_Id = 2; Path = /data/export/.ganesha; Pseudo = /.ganesha; Access_Type = None; Protocols = 4; FSAL { Name = VFS; } } ``` ### What we observed (V6.5, FSAL_VFS) Both exports load: ``` Export ... 1 pseudo (/) path (/data/export) perms (... no_root_squash, RWrw, ...) Export ... 2 pseudo (/.ganesha) path (/data/export/.ganesha) perms (... ----, ...) NFSv4 pseudo file system successfully initialized ``` From a client — including **root** under `No_Root_Squash`: - `ls -a` on the share does **not** list `.ganesha` at all - `stat`/`cat` on that path return `ENOENT`, not `EACCES` - `rm -rf <share>/* <share>/.[!.]*` removes everything else and leaves the recovery data untouched - the parent export continues to serve reads and writes normally Controlled by removing only the second export and restarting: the same directory (same inode, mode 0700) then appears in `ls -a` and its contents are readable over NFS. So the nested export is unambiguously what makes it unreachable. ### Questions 1. **Is this a supported use of nested exports**, or are we relying on an accident? Is there an intended mechanism for excluding a subdirectory from an export that we've missed? 2. **Is the READDIR filtering intentional and stable?** We expected `EACCES` on traversal and instead the junction is omitted from the listing entirely, which is nicer for us — but we don't want to depend on it if it's incidental. 3. **Any correctness concern with `RecoveryRoot` living inside an exported tree?** The server writes its own client database into a path that is nominally part of an export, albeit one no client can enter. Is there re-entrancy, FSAL, or grace-period interaction we should worry about? 4. For a single-server, one-instance-per-volume deployment where the recovery directory travels with the exported filesystem, **is `fs` or `fs_ng` preferable?** We read `recovery_fs_ng.c` as adding atomic generation swaps via `rename()` + `symlink()`; our backing filesystem is ext4, so both are viable. _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org

Thanks Martin. The FSID_Override idea is the first thing I've seen that would actually close the hole Frank mentioned, so I'm interested in how it goes. Issue is we're on 6.5 and neither of the things you mention seems to be there. As far as I can tell fsid_override is only parsed in `nfs_read_conf.c` from V15.1, and `recovery_fs.c` doesn't reference `recovery_backend_ipbased` until V15.1 either, nothing at V15. Neither string is in our binary. So: which version are you doing FSID_Override on, next or a backport onto something older? Does turning it on change handles that clients already hold? We'd be enabling it under live mounts, so I want to know whether to expect a round of ESTALE or whether it only affects handles issued afterwards. If you haven't got to the negatives yet, that's the one I'd look at first. And how are you picking the Filesystem_Id per export? We create an export per volume, dynamically, so I need something that generates them rather than a number in a config file. Curious whether you're deriving it from something stable or just counting. On RecoveryBackendIPBased, which release were you on when you used it? If it only landed in V15.1 and you had it before then, you're presumably carrying a backport, and I'd like to know, because we may end up wanting the same. We hit the same problem, but solved it by patch rather than config since 6.5 gives us no option: we drop the client's source address from the recovery tag entirely. Under a container network that address flips between the bridge and the VXLAN interface depending on whether client and server share a node, so moving the server rekeyed every client, not just the one that moved. Tags are stable across nodes now. One more thing, do you remember what the fs_ng comments actually said, or what you ran into? We've taken your advice and stayed on fs, but I'd rather know the real gap than avoid it on reputation. cheers, Abhinandan

Hi Abhinandan, On Thu, 20 Aug 2026 04:05:51 -0000, purkaitabhinandan(a)gmail.com wrote: We're on something older than you. We're on V5.9 with >160 cherry-picks/backports, mostly bug fixes. We're doing loosely clustered NFS-Ganesha but we don't expect this type of change to work across online upgrades. One limitation is that you can't change the value of Filesystem_ID while NFS-Ganesha is up... or FSID_override either, I think. My plan is to pick a Filesystem_ID major for all exports for a site and set the minor to be equal to the Export_ID. This seems to be a common approach. I haven't thought through multitenancy, but my starting point would be a tenant ID as major and continue with Export_ID for minor. It is a nice and dumb approach... but needs to be carefully considered. Yeah, again, we've cherry-picked/backported to V5.9. I think it was this that stopped me from digging deeper: #ifndef FIXME return; #else /** * More investigation/design is required to determine the best * implementation for the remainder of this method. In the interim, * Coverity complains that this is dead code because of the early * return above. Hence the remainder of this method is ifdef'ed * out to avoid the Coverity complaint. * * @todo: FIXME: make the rest of this work */ :-) I think, after seeing this, and the lack of comments anywhere describing how fs_ng is meant to be better, I cut my losses early and decided to see if we could make something work using fs. That was quite tricky... but now trivial with RecoveryBackendIPBased. peace & happiness, martin

On Thu, 20 Aug 2026 17:57:44 -0700, "Frank Filz" <ffilzlnx(a)mindspring.com&gt; wrote: Good point! That is still possible... To secure it properly, we would have to use a subdirectory mount of the cluster filesystem per export. With Lustre we would then need the distinct FSID on each export to allow NFS-Ganesha to find the "filesystem" associated with the export to get the mount_fd to do open_by_handle_at(). I know this will be secure because resolving a handle from a subsequent subdirectory mount doesn't work (ESTALE) if we use the mount_fd for the first subdirectory mount... because all the subdirectory mounts have the same default FSID. That's the behaviour we see without FSID_Override=True. Yeah, as per my other reply, it wasn't a very educated guess. I took a fairly quick look and gave up early... peace & happiness, martin

I think you’re good. And the implementation bits you are relying on are stable and won’t go away. You have a novel idea of using an export to hide an internal use directory, but the idea of having different access (including hiding) directories via exports is an intended feature of Ganesha exports. Some filesystems implement sub-volumes in a way that even improves the security to eliminate the risk of handle guessing. Frank From: Abhinandan Purkait [mailto:purkaitabhinandan@gmail.com] Sent: Thursday, August 20, 2026 9:00 PM To: Martin Schwenke <martin(a)meltin.net&gt; Cc: Frank Filz <ffilzlnx(a)mindspring.com>; devel(a)lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Guidance: keeping `RecoveryRoot` inside an exported tree via a nested export with `Access_Type = None` Thanks, both of you. I came in with four fairly narrow questions and got much more than that out of it, which I appreciate. Quick update on where we landed. We've moved off 6.5 onto 15.1 and dropped the patch we were carrying. The config that replaces it turned out to be RecoverySkipIp rather than RecoveryBackendIPBased — I had assumed those were two names for the same thing, and that was my own misreading rather than anything either of you said. As far as I can tell SkipIp is what drops the client address from the tag in nfs4_create_clid_name, while IPBased groups the database into ip_<addr> directories keyed on cr_server_addr. Two different halves, and I only worked that out after testing. Martin, I did try to follow your lead on IPBased before I understood that, so it may be worth mentioning why it didn't fit us — not a criticism of the advice, which is clearly right for your setup. You've got a service IP that clients follow, so whichever node takes the IP reads that IP's directory and everyone reclaims. We don't have a floating address at all: clients reach a per-volume ClusterIP, and the server ends up recording its own pod IP as cr_server_addr, which changes whenever the pod is rescheduled. So IPBased would have sharded our database by the one address guaranteed to move. We're running it off with SkipIp on instead. Otherwise we're staying on fs, and keeping the nested Access_Type = None export for the recovery directory. Frank, your point about wrapping an inode in an export ID and FSID you do have access to was the useful correction there — we've stopped treating that export as any kind of boundary and now describe it as simply keeping the directory out of the way. And Martin, thanks for coming back on fs_ng; the FIXME is exactly what I was after, rather than avoiding it on reputation alone. If any of that reading looks off to either of you, I'd be glad to hear it — particularly the SkipIp/IPBased split, and whether fs plus a denied nested export is a combination you'd expect to keep working. No rush, and no need to reply if it all sounds reasonable. Otherwise I'll stop taking up list time. Thanks again, genuinely helpful. cheers, Abhinandan On Fri, 21 Aug 2026 at 08:22, Martin Schwenke <martin(a)meltin.net <mailto:martin@meltin.net> > wrote: On Thu, 20 Aug 2026 17:57:44 -0700, "Frank Filz" <ffilzlnx(a)mindspring.com <mailto:ffilzlnx@mindspring.com> > wrote: Good point! That is still possible... To secure it properly, we would have to use a subdirectory mount of the cluster filesystem per export. With Lustre we would then need the distinct FSID on each export to allow NFS-Ganesha to find the "filesystem" associated with the export to get the mount_fd to do open_by_handle_at(). I know this will be secure because resolving a handle from a subsequent subdirectory mount doesn't work (ESTALE) if we use the mount_fd for the first subdirectory mount... because all the subdirectory mounts have the same default FSID. That's the behaviour we see without FSID_Override=True. Yeah, as per my other reply, it wasn't a very educated guess. I took a fairly quick look and gave up early... peace & happiness, martin

Thanks Frank, that's what I needed. The multi-tenancy reasoning is the reassuring part, since it means the invisible junction is something someone maintains rather than something I'd be getting away with. So we'll go ahead: RecoveryRoot on the exported filesystem, hidden behind the nested export, staying on fs. Two things I got stuck on afterwards, both about versions. Martin suggested FSID_Override, and mentioned that fs supports RecoveryBackendIPBased these days. Neither seems to exist in what we build. `recovery_fs.c` doesn't mention `recovery_backend_ipbased` at V15 and mentions it ten times at V15.1, and `nfs_read_conf.c` only parses `fsid_override` from V15.1. Neither string is in our 6.5 binary. Against that, the wiki says 6.5 is the current supported release, the newest stable branch is V6-stable, and the V15.1 tag looks identical to next - your announce mail says "Branch next" as well. So which is it, from a packager's point of view? Is 6.5 still what you'd want a downstream shipping, with V7 to V15 being development snapshots? Is a new stable branch coming, and from which tag? And does V6-stable take backports at all - would RecoveryBackendIPBased be welcome there, or is the answer simply to move off 6.5? I ask because we already carry a local patch for this. On 6.5 a client can't reclaim once we reschedule the server, because `fs_create_clid_name()` puts the client's source address at the front of the tag, and under a container network that address depends on whether the client happens to land on the same node as the server. Same client, bridge address when co-located, VXLAN address when not. Moving the server rekeyed every client, not just the one that moved, so no amount of durable storage would have helped. We drop the address: - (void)snprintf(clientid->cid_recov_tag, total_size, - "%s-(%s:%s)", str_client_addr, cidstr_lenx, - cidstr); + (void)snprintf(clientid->cid_recov_tag, total_size, "(%s:%s)", + cidstr_lenx, cidstr); Tags come out byte-identical wherever the server runs; I checked across three nodes covering each co-location combination. It covers fs_ng too, since `fs_ng_backend.add_clid` is `fs_add_clid`. It does need client hostnames to be unique, which is fine on Kubernetes. Reading V15.1, that looks like `RecoveryBackendIPBased = false`. Is it the same thing, or does the option do more than change the tag? And is there anything on 6.5 that reads the tag back and expects the old `<addr>-(len:id)` shape - takeover, the reaper, `fs_read_recov_clids_recover()`? That's the part I'm least sure of, and the patch is already in our images, so I'd rather ask than find out. Happy to put it on Gerrit against V6-stable if it's any use to anyone else. Last thing, your caveat about handles. Martin's fix needs V15.1, so it isn't available to us either. Is there anything on 6.5 that stops a handle for a file under the hidden export working through the parent export, or is upgrading the only answer? And how worried should I be? For FSAL_VFS the handle is the kernel's, inode plus generation, so guessing feels theoretical to me, but you'd know better. Worst case I can see is someone wiping the recovery entries, which only puts us back to no reclaim at all. Thanks, Abhinandan