NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
1:01 p.m.
Hi,
I configured five unrestriced LXC containers on Proxmox and also set lxc.seccomp.profile
to empty.
This is my configuration:
root@nfsshares-a:~# egrep -v '^[[:blank:]]*#|^[[:blank:]]*$'
/etc/ganesha/ganesha.conf
NFS_CORE_PARAM
{
Enable_NLM = false;
Enable_RQUOTA = false;
Protocols = 3,4;
mount_path_pseudo = true;
}
NFS_KRB5
{
Active_krb5 = false;
}
NFSv4
{
RecoveryBackend = rados_ng;
Minor_Versions = 1,2;
}
MDCACHE {
Dir_Chunk = 0;
}
EXPORT_DEFAULTS {
SecType = "sys";
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
}
CEPH
{
}
RADOS_KV
{
UserId = "ganesharecov";
}
RADOS_URLS
{
UserId = "ganeshaurls";
watch_url = "rados://nfs-ganesha/ganesha-namespace/conf-nfsshares";
}
%url rados://nfs-ganesha/ganesha-namespace/conf-nfsshares
root@nfsshares-a:~#
And this is an example export. The other exports just have different clients for the other
paths.
root@proxmox07:~# cat /tmp/export-1
EXPORT {
FSAL {
secret_access_key = "AQDm7eFfJWvhIBAAuI++d0SNkdylNkMx5t/N8w==";
user_id = "ganesha";
name = "CEPH";
filesystem = "cephfs";
}
CLIENT {
clients = 10.20.56.254;
squash = "no_root_squash";
access_type = "RW";
}
pseudo = "/";
squash = "no_root_squash";
access_type = "NONE";
path = "/";
export_id = 1;
transports = "TCP";
protocols = 3, 4;
}
Mounting filesystems works fine, but as soon as I restart the nfs-ganesha daemon I get a
stale file handle on the client.
Looking at
root@nfsshares-a:~# ganesha-rados-grace --userid ganesharecov --pool nfs-ganesha --ns
namespace-ganesha dump
cur=1 rec=0
======================================================
nfsshares-a E
nfsshares-b E
nfsshares-c E
nfsshares-d E
nfsshares-e E
root@nfsshares-a:~#
The nodes never loose the E state.
These are the permissions on ceph:
root@proxmox07:~# cat /etc/pve/priv/ceph.client.ganesha*.keyring
[client.ganesha]
key = <removed>
caps mds = "allow r path=/, allow rw path=/vol"
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=cephfs_data, allow rw pool=nfs-ganesha"
[client.ganesharecov]
key = <removed>
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=nfs-ganesha"
[client.ganeshaurls]
key = <removed>
caps mon = "allow r"
caps osd = "allow class-read object_prefix rbd_children, allow rw
pool=nfs-ganesha"
root@proxmox07:~#
Which service account is used to remove the E state on ceph? Do I have a permission
problem?
Best regards
Rainer
Log from a restart:
----
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-239[main] main :MAIN
:EVENT :ganesha.nfsd Starting: Ganesha Version 3.4
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] rados_ng_init
:CLIENT ID :EVENT :Rados kv store init done
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] lower_my_caps
:NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management
in FSAL
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] lower_my_caps
:NFS STARTUP :EVENT :currenty set capabilities are: =
cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed
(-1765328160:22)
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main]
nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start :NFS
STARTUP :EVENT :-------------------------------------------------
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start :NFS
STARTUP :EVENT : NFS SERVER INITIALIZED
21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start :NFS
STARTUP :EVENT :-------------------------------------------------
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-236[main] main :MAIN
:EVENT :ganesha.nfsd Starting: Ganesha Version 3.4
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] rados_ng_init
:CLIENT ID :EVENT :Rados kv store init done
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] lower_my_caps
:NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management
in FSAL
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] lower_my_caps
:NFS STARTUP :EVENT :currenty set capabilities are: =
cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed
(-1765328160:22)
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main]
nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start :NFS
STARTUP :EVENT :-------------------------------------------------
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start :NFS
STARTUP :EVENT : NFS SERVER INITIALIZED
22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start :NFS
STARTUP :EVENT :-------------------------------------------------
root@nfsshares-a:~#
1:23 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
I changed the NFSv4 recovery backend from rados_ng to rados_cluster, as all five
nfs-ganesha processes will be up all the time.
And I found my mistake: I thought the rados_kv would also use the default namespace
"ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the
grace DB at an incorrect location.
I played around with the nodeid parameter for some time as I expected my configuration
error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say
something in the startup log like
nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace
with nodeid <nodeid>
That would have been very helpful and time saving!
So NFSv4 recovery works fine over the 5 nodes.
NFSv3 "recovery" also works - EXCEPT (!!!):
I am mounting a cephfs .snap directory on NFS-root booted systems like this:
10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file
handle.
If I mount the RW image directly like this
10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
the IPv4 takeover just works.
Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover?
Thanks
Rainer
6:07 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
Several issues.
We've had problems with snapshot directories in the past; they don't
seem to follow normal semantics, so it may be that they don't work with
recovery.
Second, I'm not sure NFSv3 recovery was ever certified with RADOS
recovery and Ceph. My memory is that it was only NFSv4 recovery. I may
be wrong about this, though.
CCing Jeff, who wrote this code, as he would likely know.
Daniel
On 1/23/21 4:23 AM, rainer.stumbaum(a)gmail.com wrote:
I changed the NFSv4 recovery backend from rados_ng to rados_cluster,
as all five nfs-ganesha processes will be up all the time.
And I found my mistake: I thought the rados_kv would also use the default namespace
"ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the
grace DB at an incorrect location.
I played around with the nodeid parameter for some time as I expected my configuration
error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say
something in the startup log like
nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace
with nodeid <nodeid>
That would have been very helpful and time saving!
So NFSv4 recovery works fine over the 5 nodes.
NFSv3 "recovery" also works - EXCEPT (!!!):
I am mounting a cephfs .snap directory on NFS-root booted systems like this:
10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file
handle.
If I mount the RW image directly like this
10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
the IPv4 takeover just works.
Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover?
Thanks
Rainer
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
8:38 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
On Mon, 2021-01-25 at 09:07 -0500, Daniel Gryniewicz wrote:
Several issues.
We've had problems with snapshot directories in the past; they don't
seem to follow normal semantics, so it may be that they don't work with
recovery.
Second, I'm not sure NFSv3 recovery was ever certified with RADOS
recovery and Ceph. My memory is that it was only NFSv4 recovery. I may
be wrong about this, though.
CCing Jeff, who wrote this code, as he would likely know.
Daniel
On 1/23/21 4:23 AM, rainer.stumbaum(a)gmail.com wrote:
> I changed the NFSv4 recovery backend from rados_ng to rados_cluster, as all five
nfs-ganesha processes will be up all the time.
>
> And I found my mistake: I thought the rados_kv would also use the default namespace
"ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the
grace DB at an incorrect location.
>
> I played around with the nodeid parameter for some time as I expected my
configuration error to be there. It would be extremly helpfull to have rados_ng and
rados_cluster to say something in the startup log like
> nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using
rados://nfs-ganesha/grace with nodeid <nodeid>
> That would have been very helpful and time saving!
>
That's a good idea -- happy to review a patch for this, or if you open a
bug I can take a look at that when I get a chance.
> So NFSv4 recovery works fine over the 5 nodes.
>
> NFSv3 "recovery" also works - EXCEPT (!!!):
> I am mounting a cephfs .snap directory on NFS-root booted systems like this:
> 10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type
nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
>
> As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale
file handle.
>
Yeah, not too surprising aince you're poking around in snapshots.
libcephfs is a bit hobbled as an API for lookups of snapped inodes. See:
https://github.com/nfs-ganesha/nfs-ganesha/blob/4e0b839f74608ce7005e533ed...
If it's not already in cache, then we don't really have a mechanism to
look up a snapshotted inode at the moment. This could probably be fixed,
but it would need to be patched into libcephfs first and then you'd need
to fix ganesha to use the new API.
> If I mount the RW image directly like this
> 10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
> the IPv4 takeover just works.
>
> Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP
failover?
>
--
Jeff Layton <jlayton(a)redhat.com>
9:36 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
Hi Jeff,
already created a pull request for the people coming behind me, see here:
https://github.com/nfs-ganesha/nfs-ganesha/pull/678
So I need to have a look into nfs-kernel-server now for that functionality then. I can do
little bug fixes but have no idea about Ceph nor Ganesha internals unfortunately...
Best regards and thanks for the quick response.
Rainer
12:13 p.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
On Mon, 2021-01-25 at 17:36 +0000, rainer.stumbaum(a)gmail.com wrote:
Hi Jeff,
already created a pull request for the people coming behind me, see here:
https://github.com/nfs-ganesha/nfs-ganesha/pull/678
>
https://github.com/nfs-ganesha/nfs-ganesha/blob/4e0b839f74608ce7005e533ed...
So I need to have a look into nfs-kernel-server now for that functionality then. I can do
little bug fixes but have no idea about Ceph nor Ganesha internals unfortunately...
Best regards and thanks for the quick response.
Rainer
Understood. I went ahead and opened a tracker ticket cephfs:
https://tracker.ceph.com/issues/48991
It's probably not too difficult a project for ceph if you wanted to dive
in. It looks like the protocol is already set up to do this, it's just
the API never had the snapid plumbed in.
If you don't want to tackle it though, I completely understand. It may
just be a while before anyone has the cycles to do so.
Cheers,
--
Jeff Layton <jlayton(a)kernel.org>
6:52 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
On Sat, 2021-01-23 at 09:23 +0000, rainer.stumbaum(a)gmail.com wrote:
I changed the NFSv4 recovery backend from rados_ng to rados_cluster,
as all five nfs-ganesha processes will be up all the time.
And I found my mistake: I thought the rados_kv would also use the default namespace
"ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the
grace DB at an incorrect location.
I played around with the nodeid parameter for some time as I expected my configuration
error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say
something in the startup log like
nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace
with nodeid <nodeid>
That would have been very helpful and time saving!
So NFSv4 recovery works fine over the 5 nodes.
NFSv3 "recovery" also works - EXCEPT (!!!):
I am mounting a cephfs .snap directory on NFS-root booted systems like this:
10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file
handle.
If I mount the RW image directly like this
10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs
(ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
the IPv4 takeover just works.
Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover?
Hi Rainer,
I started working on some patches for this. We have to extend libcephfs
first, and then use the new functionality from ganesha. I have some
draft patches in my trees, but what I don't currently have is a reliable
reproducer for this.
Do you a sequence of steps that always results in an -ESTALE in your
current setup? I'd like to make sure the patches I have fix it.
Thanks,
--
Jeff Layton <jlayton(a)poochiereds.net>
10:16 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
Hi Jeff,
so the minimal config would be like this:
- CephFS
- two NFS-Ganesha nodes (node a: 10.20.56.240, node b:10.20.56.241) with keepalived
installed and a VIP of 10.20.56.2
- One client able to mount NFS shares in that 10.20.56.0/24 network
Test 1:
Mount a NFS share:
mount -o
ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys
10.20.56.2:/vol/test /tmp/nfs-test1
Failover from a to b by rebooting. That should work.
Test 2:
Create a snapshot on the Ceph node:
cd /vol/test ; mkdir .snap/12345
Mount the snapshot:
mount -o
ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys
10.20.56.2:/vol/test/.snap/12345 /tmp/nfs-test2
ls /tmp/nfs-test2
...should work
Failover from a to b by rebooting
ls /tmp/nfs-test2 should give the ESTALE.
All the best
Rainer
5:14 a.m.
New subject: NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
On Tue, 2021-02-02 at 18:16 +0000, rainer.stumbaum(a)gmail.com wrote:
Hi Jeff,
so the minimal config would be like this:
- CephFS
- two NFS-Ganesha nodes (node a: 10.20.56.240, node b:10.20.56.241) with keepalived
installed and a VIP of 10.20.56.2
- One client able to mount NFS shares in that 10.20.56.0/24 network
Test 1:
Mount a NFS share:
mount -o
ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys
10.20.56.2:/vol/test /tmp/nfs-test1
Failover from a to b by rebooting. That should work.
Test 2:
Create a snapshot on the Ceph node:
cd /vol/test ; mkdir .snap/12345
Mount the snapshot:
mount -o
ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys
10.20.56.2:/vol/test/.snap/12345 /tmp/nfs-test2
ls /tmp/nfs-test2
...should work
Failover from a to b by rebooting
ls /tmp/nfs-test2 should give the ESTALE.
Thanks. I was able to come up with one that was a bit easier with NFSv4:
- mount the mount
- create a snapshot: mkdir /mnt/ganesha/cephfs/.snap/1
- tail -f /mnt/ganesha/cephfs/.snap/1/testfile
- kill ganesha and restart it
...at that point, the tail command gets back ESTALE when it tries to
recover the stateid. I have a PR that adds a new interface to libcephfs:
https://github.com/ceph/ceph/pull/39294
...once that's merged I'll submit the corresponding patch for ganesha
(it's pretty small):
https://github.com/jtlayton/nfs-ganesha/commit/6f297bc33062de841c7c43e6be...
Feel free to test these out if you're able to hand-build ceph and
ganesha, otherwise they should trickle out to distros eventually.
Cheers,
--
Jeff Layton <jlayton(a)poochiereds.net>
1171
days inactive
1184
days old
8 comments
5 participants
participants (5)
-
Daniel Gryniewicz -
Jeff Layton -
Jeff Layton -
Jeff Layton -
rainer.stumbaum＠gmail.com