NFSv3 active-passive cluster with FSAL CephFS and keepalived - Stale file handle after restart and also failover
Hi, I configured five unrestriced LXC containers on Proxmox and also set lxc.seccomp.profile to empty. This is my configuration: root@nfsshares-a:~# egrep -v '^[[:blank:]]*#|^[[:blank:]]*$' /etc/ganesha/ganesha.conf NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 3,4; mount_path_pseudo = true; } NFS_KRB5 { Active_krb5 = false; } NFSv4 { RecoveryBackend = rados_ng; Minor_Versions = 1,2; } MDCACHE { Dir_Chunk = 0; } EXPORT_DEFAULTS { SecType = "sys"; Squash = No_Root_Squash; Attr_Expiration_Time = 0; } CEPH { } RADOS_KV { UserId = "ganesharecov"; } RADOS_URLS { UserId = "ganeshaurls"; watch_url = "rados://nfs-ganesha/ganesha-namespace/conf-nfsshares"; } %url rados://nfs-ganesha/ganesha-namespace/conf-nfsshares root@nfsshares-a:~# And this is an example export. The other exports just have different clients for the other paths. root@proxmox07:~# cat /tmp/export-1 EXPORT { FSAL { secret_access_key = "AQDm7eFfJWvhIBAAuI++d0SNkdylNkMx5t/N8w=="; user_id = "ganesha"; name = "CEPH"; filesystem = "cephfs"; } CLIENT { clients = 10.20.56.254; squash = "no_root_squash"; access_type = "RW"; } pseudo = "/"; squash = "no_root_squash"; access_type = "NONE"; path = "/"; export_id = 1; transports = "TCP"; protocols = 3, 4; } Mounting filesystems works fine, but as soon as I restart the nfs-ganesha daemon I get a stale file handle on the client. Looking at root@nfsshares-a:~# ganesha-rados-grace --userid ganesharecov --pool nfs-ganesha --ns namespace-ganesha dump cur=1 rec=0 ====================================================== nfsshares-a E nfsshares-b E nfsshares-c E nfsshares-d E nfsshares-e E root@nfsshares-a:~# The nodes never loose the E state. These are the permissions on ceph: root@proxmox07:~# cat /etc/pve/priv/ceph.client.ganesha*.keyring [client.ganesha] key = <removed> caps mds = "allow r path=/, allow rw path=/vol" caps mon = "allow r" caps osd = "allow class-read object_prefix rbd_children, allow rw pool=cephfs_data, allow rw pool=nfs-ganesha" [client.ganesharecov] key = <removed> caps mon = "allow r" caps osd = "allow class-read object_prefix rbd_children, allow rw pool=nfs-ganesha" [client.ganeshaurls] key = <removed> caps mon = "allow r" caps osd = "allow class-read object_prefix rbd_children, allow rw pool=nfs-ganesha" root@proxmox07:~# Which service account is used to remove the E state on ceph? Do I have a permission problem? Best regards Rainer Log from a restart: ---- 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-239[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.4 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] rados_ng_init :CLIENT ID :EVENT :Rados kv store init done 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 21/01/2021 10:48:08 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22) 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor. 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED 21/01/2021 10:48:22 : epoch 60095be8 : nfsshares-a : ganesha.nfsd-246[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-236[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.4 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] rados_ng_init :CLIENT ID :EVENT :Rados kv store init done 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 22/01/2021 19:47:22 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+ep 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:22) 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor. 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED 22/01/2021 19:47:49 : epoch 600b2bca : nfsshares-a : ganesha.nfsd-249[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- root@nfsshares-a:~#
I changed the NFSv4 recovery backend from rados_ng to rados_cluster, as all five nfs-ganesha processes will be up all the time. And I found my mistake: I thought the rados_kv would also use the default namespace "ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the grace DB at an incorrect location. I played around with the nodeid parameter for some time as I expected my configuration error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say something in the startup log like nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace with nodeid <nodeid> That would have been very helpful and time saving! So NFSv4 recovery works fine over the 5 nodes. NFSv3 "recovery" also works - EXCEPT (!!!): I am mounting a cephfs .snap directory on NFS-root booted systems like this: 10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2) As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file handle. If I mount the RW image directly like this 10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2) the IPv4 takeover just works. Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover? Thanks Rainer
Several issues. We've had problems with snapshot directories in the past; they don't seem to follow normal semantics, so it may be that they don't work with recovery. Second, I'm not sure NFSv3 recovery was ever certified with RADOS recovery and Ceph. My memory is that it was only NFSv4 recovery. I may be wrong about this, though. CCing Jeff, who wrote this code, as he would likely know. Daniel On 1/23/21 4:23 AM, rainer.stumbaum@gmail.com wrote:
I changed the NFSv4 recovery backend from rados_ng to rados_cluster, as all five nfs-ganesha processes will be up all the time.
And I found my mistake: I thought the rados_kv would also use the default namespace "ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the grace DB at an incorrect location.
I played around with the nodeid parameter for some time as I expected my configuration error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say something in the startup log like nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace with nodeid <nodeid> That would have been very helpful and time saving!
So NFSv4 recovery works fine over the 5 nodes.
NFSv3 "recovery" also works - EXCEPT (!!!): I am mounting a cephfs .snap directory on NFS-root booted systems like this: 10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file handle.
If I mount the RW image directly like this 10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2) the IPv4 takeover just works.
Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover?
Thanks Rainer _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
On Mon, 2021-01-25 at 09:07 -0500, Daniel Gryniewicz wrote:
Several issues.
We've had problems with snapshot directories in the past; they don't seem to follow normal semantics, so it may be that they don't work with recovery.
Second, I'm not sure NFSv3 recovery was ever certified with RADOS recovery and Ceph. My memory is that it was only NFSv4 recovery. I may be wrong about this, though.
CCing Jeff, who wrote this code, as he would likely know.
Daniel
On 1/23/21 4:23 AM, rainer.stumbaum@gmail.com wrote:
I changed the NFSv4 recovery backend from rados_ng to rados_cluster, as all five nfs-ganesha processes will be up all the time.
And I found my mistake: I thought the rados_kv would also use the default namespace "ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the grace DB at an incorrect location.
I played around with the nodeid parameter for some time as I expected my configuration error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say something in the startup log like nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace with nodeid <nodeid> That would have been very helpful and time saving!
That's a good idea -- happy to review a patch for this, or if you open a bug I can take a look at that when I get a chance.
So NFSv4 recovery works fine over the 5 nodes.
NFSv3 "recovery" also works - EXCEPT (!!!): I am mounting a cephfs .snap directory on NFS-root booted systems like this: 10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file handle.
Yeah, not too surprising aince you're poking around in snapshots. libcephfs is a bit hobbled as an API for lookups of snapped inodes. See: https://github.com/nfs-ganesha/nfs-ganesha/blob/4e0b839f74608ce7005e533eda14... If it's not already in cache, then we don't really have a mechanism to look up a snapshotted inode at the moment. This could probably be fixed, but it would need to be patched into libcephfs first and then you'd need to fix ganesha to use the new API.
If I mount the RW image directly like this 10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2) the IPv4 takeover just works.
Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover?
-- Jeff Layton <jlayton@redhat.com>
Hi Jeff, already created a pull request for the people coming behind me, see here: https://github.com/nfs-ganesha/nfs-ganesha/pull/678
https://github.com/nfs-ganesha/nfs-ganesha/blob/4e0b839f74608ce7005e533eda14...
So I need to have a look into nfs-kernel-server now for that functionality then. I can do little bug fixes but have no idea about Ceph nor Ganesha internals unfortunately... Best regards and thanks for the quick response. Rainer
On Mon, 2021-01-25 at 17:36 +0000, rainer.stumbaum@gmail.com wrote:
Hi Jeff,
already created a pull request for the people coming behind me, see here: https://github.com/nfs-ganesha/nfs-ganesha/pull/678
https://github.com/nfs-ganesha/nfs-ganesha/blob/4e0b839f74608ce7005e533eda14...
So I need to have a look into nfs-kernel-server now for that functionality then. I can do little bug fixes but have no idea about Ceph nor Ganesha internals unfortunately...
Best regards and thanks for the quick response. Rainer
Understood. I went ahead and opened a tracker ticket cephfs: https://tracker.ceph.com/issues/48991 It's probably not too difficult a project for ceph if you wanted to dive in. It looks like the protocol is already set up to do this, it's just the API never had the snapid plumbed in. If you don't want to tackle it though, I completely understand. It may just be a while before anyone has the cycles to do so. Cheers, -- Jeff Layton <jlayton@kernel.org>
On Sat, 2021-01-23 at 09:23 +0000, rainer.stumbaum@gmail.com wrote:
I changed the NFSv4 recovery backend from rados_ng to rados_cluster, as all five nfs-ganesha processes will be up all the time.
And I found my mistake: I thought the rados_kv would also use the default namespace "ganesha-namespace" but instead uses NULL. So I was maintaining and dumping the grace DB at an incorrect location.
I played around with the nodeid parameter for some time as I expected my configuration error to be there. It would be extremly helpfull to have rados_ng and rados_cluster to say something in the startup log like nfs4_recovery_init :CLIENT ID :EVENT :rados_cluster init using rados://nfs-ganesha/grace with nodeid <nodeid> That would have been very helpful and time saving!
So NFSv4 recovery works fine over the 5 nodes.
NFSv3 "recovery" also works - EXCEPT (!!!): I am mounting a cephfs .snap directory on NFS-root booted systems like this: 10.20.56.2:/vol/diskless/.snap/00225/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2)
As soon as I failover with the 10.20.56.2 IP to another nfs-ganesha I get the stale file handle.
If I mount the RW image directly like this 10.20.56.2:/vol/diskless/debian10-amd64-srv on /run/initramfs/rofs type nfs (ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys,local_lock=all,addr=10.20.56.2) the IPv4 takeover just works.
Is it possible to mount a .snap Ceph snapshot directory and survice a NFSv3 IP failover?
Hi Rainer, I started working on some patches for this. We have to extend libcephfs first, and then use the new functionality from ganesha. I have some draft patches in my trees, but what I don't currently have is a reliable reproducer for this. Do you a sequence of steps that always results in an -ESTALE in your current setup? I'd like to make sure the patches I have fix it. Thanks, -- Jeff Layton <jlayton@poochiereds.net>
Hi Jeff, so the minimal config would be like this: - CephFS - two NFS-Ganesha nodes (node a: 10.20.56.240, node b:10.20.56.241) with keepalived installed and a VIP of 10.20.56.2 - One client able to mount NFS shares in that 10.20.56.0/24 network Test 1: Mount a NFS share: mount -o ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys 10.20.56.2:/vol/test /tmp/nfs-test1 Failover from a to b by rebooting. That should work. Test 2: Create a snapshot on the Ceph node: cd /vol/test ; mkdir .snap/12345 Mount the snapshot: mount -o ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys 10.20.56.2:/vol/test/.snap/12345 /tmp/nfs-test2 ls /tmp/nfs-test2 ...should work Failover from a to b by rebooting ls /tmp/nfs-test2 should give the ESTALE. All the best Rainer
On Tue, 2021-02-02 at 18:16 +0000, rainer.stumbaum@gmail.com wrote:
Hi Jeff,
so the minimal config would be like this: - CephFS - two NFS-Ganesha nodes (node a: 10.20.56.240, node b:10.20.56.241) with keepalived installed and a VIP of 10.20.56.2 - One client able to mount NFS shares in that 10.20.56.0/24 network
Test 1: Mount a NFS share: mount -o ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys 10.20.56.2:/vol/test /tmp/nfs-test1
Failover from a to b by rebooting. That should work.
Test 2: Create a snapshot on the Ceph node: cd /vol/test ; mkdir .snap/12345 Mount the snapshot: mount -o ro,relatime,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=600,acregmax=600,acdirmin=600,acdirmax=600,hard,nocto,nolock,noacl,proto=tcp,port=2049,timeo=100,retrans=360,sec=sys 10.20.56.2:/vol/test/.snap/12345 /tmp/nfs-test2 ls /tmp/nfs-test2 ...should work Failover from a to b by rebooting ls /tmp/nfs-test2 should give the ESTALE.
Thanks. I was able to come up with one that was a bit easier with NFSv4: - mount the mount - create a snapshot: mkdir /mnt/ganesha/cephfs/.snap/1 - tail -f /mnt/ganesha/cephfs/.snap/1/testfile - kill ganesha and restart it ...at that point, the tail command gets back ESTALE when it tries to recover the stateid. I have a PR that adds a new interface to libcephfs: https://github.com/ceph/ceph/pull/39294 ...once that's merged I'll submit the corresponding patch for ganesha (it's pretty small): https://github.com/jtlayton/nfs-ganesha/commit/6f297bc33062de841c7c43e6bea2b... Feel free to test these out if you're able to hand-build ceph and ganesha, otherwise they should trickle out to distros eventually. Cheers, -- Jeff Layton <jlayton@poochiereds.net>
participants (5)
-
Daniel Gryniewicz -
Jeff Layton -
Jeff Layton -
Jeff Layton -
rainer.stumbaum＠gmail.com