May 2019 - Support - NFS Ganesha List Archives

NFS-Ganesha availability probleme / FSAL Gluster
by Dominique Rousseau 18 Jun '19

18 Jun '19

Hi, I'm not sure my setup is "the right way", but we encounter problems on nfs-ganesha setup with glusterfs FSAL glusterfs version is 3.13.2-1 ( on Debian 9 ) nfs-ganesha is 2.5.3-1 We have 2 gluster nodes ( called gluster1 an gluster2 ), both with nfs-ganesha installed. Some NFS clients connect to gluster1, others to gluster2 We are using ACL on gluster (with ext4 on the bricks) and they are activated in ganesha conf We had the first problem 2 days ago, on gluster1, the second yesterday on gluster2 In the minutes befores the NFS stops responding, the only suspect log I get is : 27/05/2019 17:16:00 : epoch 5cebf468 : gluster1 : ganesha.nfsd-1877[work-123] posix_acl_2_fsal_acl :FSAL :WARN :Cannot retrieve permission set 27/05/2019 17:16:00 : epoch 5cebf468 : gluster1 : ganesha.nfsd-1877[work-123] posix_acl_2_fsal_acl :FSAL :WARN :Cannot retrieve permission set ( and so, thousands of lines ) the same, for second problem : 28/05/2019 18:18:16 : epoch 5cecdfa1 : gluster2 : ganesha.nfsd-11510[work-229] posix_acl_2_fsal_acl :FSAL :WARN :Cannot retrieve permission set 28/05/2019 18:18:16 : epoch 5cecdfa1 : gluster2 : ganesha.nfsd-11510[work-229] posix_acl_2_fsal_acl :FSAL :WARN :Cannot retrieve permission set ( and so, thousands of lines ) Restart of the nfs-ganesha process seems to be enough to restore access The ganesha.conf file is like this : NFS_Core_Param { #Use supplied name other tha IP In NSM operations NSM_Use_Caller_Name = true; #Copy lock states into "/var/lib/nfs/ganesha" dir Clustered = false; #Use a non-privileged port for RQuota Rquota_Port = 875; } EXPORT{ Export_Id = 1 ; # Export ID unique to each export Path = "/alternc-html"; # Path of the volume to be exported. Eg: "/test_volume" FSAL { name = GLUSTER; hostname = "127.0.0.1"; # IP of one of the nodes in the trusted pool volume = "alternc-html"; # Volume name. Eg: "test_volume" } Access_type = RW; # Access permissions Squash = No_root_squash; # To enable/disable root squashing Disable_ACL = FALSE; # To enable/disable ACL Pseudo = "/alternc-html"; # NFSv4 pseudo path for this export. Eg: "/test_volume_pseudo" Protocols = "3"; ##,"4" ; # NFS protocols supported Transports = "UDP","TCP" ; # Transport protocols supported SecType = "sys"; # Security flavors supported }

1 1

Configuring a NFS-GANESHA server as a NFS proxy
by iyerramji＠gmail.com 13 Jun '19

13 Jun '19

Hi, I am following this wiki https://github.com/nfs-ganesha/nfs-ganesha/wiki/PROXY It says this - > ./configure --with-fsal=PROXY but I dont see the configure file in your source code. Which file are you referring to ? Thanks Ramji

6 5

Re: [ceph-users] Nfs-ganesha with rados_kv backend
by Jeff Layton 29 May '19

29 May '19

On Wed, 2019-05-29 at 13:49 +0000, Stolte, Felix wrote: > Hi, > > is anyone running an active-passive nfs-ganesha cluster with cephfs backend and using the rados_kv recovery backend? My setup runs fine, but takeover is giving me a headache. On takeover I see the following messages in ganeshas log file: > Note that there are significant problems with the rados_kv recovery backend. In particular, it does not properly handle the case where the server crashes during the grace period. The rados_ng and rados_cluster backends do handle those situations properly. > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 5 > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] nfs_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip 10.0.0.5 > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2 > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] rados_kv_read_recov_clids_takeover :CLIENT ID :EVENT :Failed to takeover > 29/05/2019 15:38:26 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE > > The result is clients hanging for up to 2 Minutes. Has anyone ran into the same problem? > > Ceph Version: 12.2.11 > nfs-ganesha: 2.7.3 > If I had to guess, the hanging is probably due to state that is being held by the other node's MDS session that hasn't expired yet. Ceph v12 doesn't have the client reclaim interfaces that make more instantaneous failover possible. That's new in v14 (Nautilus). See pages 12 and 13 here: https://static.sched.com/hosted_files/cephalocon2019/86/Rook-Deployed%20NFS… > ganesha.conf (identical on both nodes besides nodeid in rados_kv: > > NFS_CORE_PARAM { > Enable_RQUOTA = false; > Protocols = 3,4; > } > > CACHEINODE { > Dir_Chunk = 0; > NParts = 1; > Cache_Size = 1; > } > > NFS_krb5 { > Active_krb5 = false; > } > > NFSv4 { > Only_Numeric_Owners = true; > RecoveryBackend = rados_kv; > Grace_Period = 5; > Lease_Lifetime = 5; Yikes! That's _way_ too short a grace period and lease lifetime. Ganesha will probably exit the grace period before the clients ever realize the server has restarted, and they will fail to reclaim their state. > Minor_Versions = 1,2; > } > > RADOS_KV { > ceph_conf = '/etc/ceph/ceph.conf'; > userid = "ganesha"; > pool = "cephfs_metadata"; > namespace = "ganesha"; > nodeid = "cephgw-k2-1"; > } > > Any hint would be appreciated. I consider ganesha's dbus-based takeover mechanism to be broken by design, as it requires the recovery backend to do things that can't be done atomically. If a crash occurs at the wrong time, the recovery database can end up trashed and no one can reclaim anything. If you really want an active/passive setup then I'd move away from that and just have whatever clustering software you're using start up the daemon on the active node after ensuring that it's shut down on the passive one. With that, you can also use the rados_ng recovery backend, which is more resilient in the face of multiple crashes. In that configuration you would want to have the same config file on both nodes, including the same nodeid so that you can potentially take advantage of the RECLAIM_RESET interface to kill off the old session quickly after the server restarts. You also need a much longer grace period. Cheers, -- Jeff Layton <jlayton(a)poochiereds.net>

1 0

Re: [ceph-users] Nfs-ganesha with rados_kv backend
by Jeff Layton 29 May '19

29 May '19

On Wed, 2019-05-29 at 13:49 +0000, Stolte, Felix wrote: > Hi, > > is anyone running an active-passive nfs-ganesha cluster with cephfs backend and using the rados_kv recovery backend? My setup runs fine, but takeover is giving me a headache. On takeover I see the following messages in ganeshas log file: > Note that there are significant problems with the rados_kv recovery backend. In particular, it does not properly handle the case where the server crashes during the grace period. The rados_ng and rados_cluster backends do handle those situations properly. > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 5 > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] nfs_start_grace :STATE :EVENT :NFS Server recovery event 5 nodeid -1 ip 10.0.0.5 > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] rados_kv_traverse :CLIENT ID :EVENT :Failed to lst kv ret=-2 > 29/05/2019 15:38:21 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[dbus_heartbeat] rados_kv_read_recov_clids_takeover :CLIENT ID :EVENT :Failed to takeover > 29/05/2019 15:38:26 : epoch 5cee88c4 : cephgw-e2-1 : ganesha.nfsd-9793[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE > > The result is clients hanging for up to 2 Minutes. Has anyone ran into the same problem? > > Ceph Version: 12.2.11 > nfs-ganesha: 2.7.3 > If I had to guess, the hanging is probably due to state that is being held by the other node's MDS session that hasn't expired yet. Ceph v12 doesn't have the client reclaim interfaces that make more instantaneous failover possible. That's new in v14 (Nautilus). See pages 12 and 13 here: https://static.sched.com/hosted_files/cephalocon2019/86/Rook-Deployed%20NFS… > ganesha.conf (identical on both nodes besides nodeid in rados_kv: > > NFS_CORE_PARAM { > Enable_RQUOTA = false; > Protocols = 3,4; > } > > CACHEINODE { > Dir_Chunk = 0; > NParts = 1; > Cache_Size = 1; > } > > NFS_krb5 { > Active_krb5 = false; > } > > NFSv4 { > Only_Numeric_Owners = true; > RecoveryBackend = rados_kv; > Grace_Period = 5; > Lease_Lifetime = 5; Yikes! That's _way_ too short a grace period and lease lifetime. Ganesha will probably exit the grace period before the clients ever realize the server has restarted, and they will fail to reclaim their state. > Minor_Versions = 1,2; > } > > RADOS_KV { > ceph_conf = '/etc/ceph/ceph.conf'; > userid = "ganesha"; > pool = "cephfs_metadata"; > namespace = "ganesha"; > nodeid = "cephgw-k2-1"; > } > > Any hint would be appreciated. I consider ganesha's dbus-based takeover mechanism to be broken by design, as it requires the recovery backend to do things that can't be done atomically. If a crash occurs at the wrong time, the recovery database can end up trashed and no one can reclaim anything. If you really want an active/passive setup then I'd move away from that and just have whatever clustering software you're using start up the daemon on the active node after ensuring that it's shut down on the passive one. With that, you can also use the rados_ng recovery backend, which is more resilient in the face of multiple crashes. In that configuration you would want to have the same config file on both nodes, including the same nodeid so that you can potentially take advantage of the RECLAIM_RESET interface to kill off the old session quickly after the server restarts. You also need a much longer grace period. Cheers, -- Jeff Layton <jlayton(a)redhat.com>

1 0

Error messages
by David C 17 May '19

17 May '19

Hi All I recently put an nfs-ganesha CEPH_FSAL deployment into production, so far so good but I'm seeing some errors in the logs I didn't see when testing and was hoping someone could shed some light on what they mean. I haven't had any adverse behaviour reported from the clients (apart from a potential issue with slow 'ls' operations which I'm investigating). Versions: libcephfs2-13.2.2-0.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64 Ceph cluster is 12.2.10 Log errors: "posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY" I'm seeing this one frequently although seems to spam the log with 20 or so occurrences in a second. "15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : > ganesha.nfsd-1990[svc_1653] posix2fsal_error :FSAL :INFO :Mapping 5 to > ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576 > 15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : > ganesha.nfsd-1990[svc_1653] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error > in nfs4_mds_putfh converted to NFS4ERR_IO but was set non-retryable" I've only seen a few occurrences of this one 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : > ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO > :nfs4_gid_to_name failed with code -2. > 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : > ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup > for 1664 failed, using numeric group This one doesn't seem too serious, my guess is there are accounts on the clients with gids/uids that the server can't look up. The server is using SSSD to bind to AD if that helps. Export: { > Export_ID=100; > Protocols = 4; > Transports = TCP; > Path = /; > Pseudo = /ceph/; > Access_Type = RW; > Squash = No_root_squash; > Attr_Expiration_Time = 0; > Disable_ACL = FALSE; > Manage_Gids = TRUE; > Filesystem_Id = 100.1; > FSAL { > Name = CEPH; > } > } Thanks, David

2 2

rpc_clnt_ping_timer_expired
by Valerio Luccio 14 May '19

14 May '19

Hello, I tried doing a google search, but didn't come up with much useful info. I have 4 CentOS 7 servers running glusterfs 5.3 that manage a 12 brick Distributed-Replicate volume. I run nfs-ganesha 2.7.1.-1 on all 4. I have another 3 Linux servers tha mount the gluster volume using glusterfs (no problem there) and I distribute the mounts to use different gluster servers. I also have a legacy OSX server that mounts the volume via NFS from the 4th gluster server and does a Samba reshare (my users use the OSX to mount the data disk). The problem comes with this last mount. Every once in a while the ganesha-gfapi.log file will show a number of rpc_clnt_ping_timer_expired errors claiming that the servers have not responded in 42 seonds, and what's most confusing to me is that it's not the OSX server, but rather the other gluster servers including the same server on which it is running. When this happens my OSX server freezes and I have to restart ganesha-nfs. I'm attaching the configuration files. Thanks, -- Valerio Luccio (212) 998-8736 Center for Brain Imaging 4 Washington Place, Room 157 New York University New York, NY 10003 "In an open world, who needs windows or gates ?"

1 0