11:02 p.m.
Thanks Frank, that's what I needed. The multi-tenancy reasoning is the
reassuring part, since it means the invisible junction is something
someone maintains rather than something I'd be getting away with. So
we'll go ahead: RecoveryRoot on the exported filesystem, hidden behind
the nested export, staying on fs.
Two things I got stuck on afterwards, both about versions.
Martin suggested FSID_Override, and mentioned that fs supports
RecoveryBackendIPBased these days. Neither seems to exist in what we
build. `recovery_fs.c` doesn't mention `recovery_backend_ipbased` at V15 and
mentions it ten times at V15.1, and `nfs_read_conf.c` only parses
`fsid_override` from V15.1. Neither string is in our 6.5 binary. Against
that, the wiki says 6.5 is the current supported release, the newest
stable branch is V6-stable, and the V15.1 tag looks identical to next -
your announce mail says "Branch next" as well.
So which is it, from a packager's point of view? Is 6.5 still what you'd
want a downstream shipping, with V7 to V15 being development snapshots?
Is a new stable branch coming, and from which tag? And does V6-stable
take backports at all - would RecoveryBackendIPBased be welcome there,
or is the answer simply to move off 6.5?
I ask because we already carry a local patch for this. On 6.5 a client
can't reclaim once we reschedule the server, because
`fs_create_clid_name()` puts the client's source address at the front of
the tag, and under a container network that address depends on whether
the client happens to land on the same node as the server. Same client,
bridge address when co-located, VXLAN address when not. Moving the
server rekeyed every client, not just the one that moved, so no amount
of durable storage would have helped. We drop the address:
- (void)snprintf(clientid->cid_recov_tag, total_size,
- "%s-(%s:%s)", str_client_addr, cidstr_lenx,
- cidstr);
+ (void)snprintf(clientid->cid_recov_tag, total_size, "(%s:%s)",
+ cidstr_lenx, cidstr);
Tags come out byte-identical wherever the server runs; I checked across
three nodes covering each co-location combination. It covers fs_ng too,
since `fs_ng_backend.add_clid` is `fs_add_clid`. It does need client
hostnames to be unique, which is fine on Kubernetes.
Reading V15.1, that looks like `RecoveryBackendIPBased = false`. Is it the
same thing, or does the option do more than change the tag? And is there
anything on 6.5 that reads the tag back and expects the old
`<addr>-(len:id)` shape - takeover, the reaper,
`fs_read_recov_clids_recover()`? That's the part I'm least sure of, and
the patch is already in our images, so I'd rather ask than find out.
Happy to put it on Gerrit against V6-stable if it's any use to anyone
else.
Last thing, your caveat about handles. Martin's fix needs V15.1, so it
isn't available to us either. Is there anything on 6.5 that stops a
handle for a file under the hidden export working through the parent
export, or is upgrading the only answer? And how worried should I be?
For FSAL_VFS the handle is the kernel's, inode plus generation, so
guessing feels theoretical to me, but you'd know better. Worst case I
can see is someone wiping the recovery entries, which only puts us back
to no reclaim at all.
Thanks,
Abhinandan