February 2019 - Support - NFS Ganesha List Archives

Many ceph clusters from one ganesha server (rgw)
by asmolik＠cloudferro.com 01 Mar '19

01 Mar '19

Hello, Is it possible to connect ganesha and serve to clients more than one ceph cluster with fsal_rgw? I can see that there is a global rgw config part with one ceph.conf in ganesha.conf, so I guess it's not as simple as adding another fsal to the config, but I'm wondering if it's at all possible - maybe with some changes to the code. Is there maybe any other way to serve many rgw's so that they are visible as mountpoint to the clients? Aleksander Smolik

3 3

KRB5, nfs-ganesha and directories appearing with ownership as nobody/nobody on client side.
by TomK 28 Feb '19

28 Feb '19

Hey All, Wondering if there would be a cross domain issue if KRB5 is enabled on NFS Ganesha and thereby causing an issue if the NFS Ganesha servers are on say, domain aaa.123.xyz while the clients are on domain bbb.123.xyz? Users get a permission denied on clients when accessing the remote mount and while checking I see it's owned by nobody / nobody rather then their user and group. The client is on a different DNS domain and ownership comes through with nobody/nobody. Hence clients get a permission denied accessing the folder I'm thinking. However the ONLY thing different that stands out ATM is the domains being different. Wondering if anyone ran into a similar experience and could share their solution? Do I need to add the realm to the /etc/krb5.conf config on the NFS Ganesha servers perhaps? -- Cheers, Tom K. ------------------------------------------------------------------------------------- Living on earth is expensive, but it includes a free trip around the sun.

4 8

glusterfs + ctdb + nfs-ganesha , unplug the network cable of serving node, takes around ~20 mins for IO to resume
by Liu, Dan 25 Feb '19

25 Feb '19

Hi all We did some failover/failback tests on 2 nodes（A and B） with architecture 'glusterfs + ctdb(public address) + nfs-ganesha'。 file is replica 2 volume. 1st: During write, unplug the network cable of serving node A ->NFS Client took a few seconds to recover to conitinue writing. After some minutes, plug the network cable of serving node A ->NFS Client also took a few seconds to recover to conitinue writing. 2nd: During write, unplug the network cable of serving node A ->NFS Client took 20 minutes to recover to conitinue writing. It is too slow for clients to accept the recovery time。 Why is it failed to fast recovery and took 20 minutes to recovery successfully. Is there anyone knows the reason? We are looking forward to your reply. Thanks. ------------------------------------------------------------------------------------------------------------- The following is some test logs and configuration. cat /etc/ganesha/ganesha.conf NFS_Core_Param { NSM_Use_Caller_Name = true; Clustered = false; MNT_Port = 20048; NLM_Port = 32803; Rquota_Port = 8750; } EXPORT{ Export_Id=181; Path=/file/gv_nfs; FSAL { name = GLUSTER; hostname=localhost; volume="file"; volpath="gv_nfs"; } Pseudo=/file/gv_nfs; Protocols = 3, 4; Transports = TCP; SecType = sys; CLIENT { Clients = *.*.*.*; Access_type = rw; Squash = root_squash; Anonymous_Uid = 99; Anonymous_Gid = 99; } } NFSv4{ Lease_Lifetime = 15; Grace_Period = 30; } 以上、よろしくお�いいたします。 -------------------------------------------------- ************************************************** Liu Dan PF Dept Nanjing Fujitsu Nanda Software Tech.Co.,Ltd.(FNST) TEL：+86+25-86630566-8512 FUJITSU INTERNAL：79955-8512 EMail: liud.fnst(a)cn.fujitsu.com<mailto:liud.fnst@cn.fujitsu.com> ************************************************** --------------------------------------------------

1 0

Crash in libntirpc code (Ganesha 2.3 version) - should be applicable to latest code as well
by Sachin Punadikar 18 Feb '19

18 Feb '19

Hi All, Customer reported a crash in Ganesha 2.3. Jan 22 20:40:52 xxxxx kernel: ganesha.nfsd[xxxxx]: unhandled signal 11 at 00003ffcf8005200 nip 00003ffcf8005200 lr 00003fff7a265e1c code 30002 Program terminated with signal 11, Segmentation fault. (gdb) where #0 0x00003ffcf8005200 in ?? () #1 0x00003fff7a265e1c in xdr_ioq_uv_release (uv=0x3ffd389df700) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/xdr_ioq.c:206 #2 0x00003fff7a267340 in xdr_ioq_release (ioqh=0x3ffd385f8fc8) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/xdr_ioq.c:720 #3 0x00003fff7a2673dc in xdr_ioq_destroy (xioq=0x3ffd385f8f00, qsize=424) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/xdr_ioq.c:729 #4 0x00003fff7a267470 in xdr_ioq_destroy_internal (xdrs=0x3ffd385f8f00) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/xdr_ioq.c:742 #5 0x00003fff7a268480 in svc_ioq_callback (wpe=0x3ff9880025e0) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/svc_ioq.c:222 #6 0x00003fff7a269400 in work_pool_thread (arg=0x3ffa180008c0) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/work_pool.c:196 #7 0x00003fff7a2dc2bc in .start_thread () from /lib64/libpthread.so.0 #8 0x00003fff7a0fb304 in .__clone () from /lib64/libc.so.6 (gdb) frame 1 #1 0x00003fff7a265e1c in xdr_ioq_uv_release (uv=0x3ffd389df700) at /usr/src/debug/nfs-ganesha-2.3.2-ibm59-0.1.1-Source/libntirpc/src/xdr_ioq.c:206 206 uv->u.uio_release(&uv->u, UIO_FLAG_NONE); When I checked Ganesha 2.3/2.5 code (also the latest community code), I am unable to find code for function uio_release. Am I missing anything here ? Or the code is really missing this function ? -- with regards, Sachin Punadikar

1 0

NLM locking issue: lock vanishes on client but stays on ganesha server
by Billich Heinrich Rainer (PSI) 11 Feb '19

11 Feb '19

Hello, I observe a NLM/NFS v3 locking issue with nfs ganesha: While a lock is no longer visible on the client in /proc/locks or with ‘lslocks’ the lock still is present on the ganesha server in /proc/locks. Further lock requests on the client fail or only succeed after some 10 seconds and a second try. I don’t see the same on a second server, hence I assume that a reboot will fix the issue for the moment – but this won’t help in the long run. I will open a support call with IBM as this is part of CES/SpectrumScale. Still if you note something that resembles a known bug or issue please let me know. In the logs I see the following - 2019-02-04 19:41:57 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm_send_async :NLM :MAJ :Cannot create NLM async tcp connection to client ::ffff:129.129.99.62 2019-02-04 19:41:57 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm4_send_grant_msg :NLM :MAJ :GRANTED_MSG RPC call failed with return code -1. Removing the blocking lock 2019-02-04 19:42:17 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm_send_async :NLM :MAJ :Cannot create NLM async tcp connection to client ::ffff:129.129.99.62 2019-02-04 19:42:17 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm4_send_grant_msg :NLM :MAJ :GRANTED_MSG RPC call failed with return code -1. Removing the blocking lock 2019-02-03 06:51:31 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm_send_async :NLM :CRIT :NLM async Client procedure call 10 failed with return code 4 : RPC: Success 2019-02-03 09:23:07 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm_send_async :NLM :CRIT :NLM async Client procedure call 10 failed with return code 4 : RPC: Success 2019-02-03 09:23:07 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm_send_async :NLM :MAJ :Cannot create NLM async tcp connection to client ::ffff:129.129.99.48 2019-02-03 09:23:07 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm4_send_grant_msg :NLM :MAJ :GRANTED_MSG RPC call failed with return code -1. Removing the blocking lock 2019-02-03 09:23:42 : epoch 00020089 : xbl-ces-2.psi.ch : gpfs.ganesha.nfsd-15508[State_Async] nlm_send_async :NLM :MAJ :Cannot create NLM async tcp connection to client ::ffff:129.129.99.48 and about 20 sockets in state CLOSE_WAIT, all used by ganesha. The message about “Cannot create NLM async tcp connection” is unique on the server which shows the issue. Several clients are affected, hence I don’t suspect an issue on the client side. It happens also with different files. I don’t see anything limited on the ganesha server – ~300 tcp sockets in total, plenty of free memory, about 300 ganesha threads. Thank you, Heiner Billich -- Paul Scherrer Institut Heiner Billich System Engineer Scientific Computing Science IT / High Performance Computing WHGA/106 Forschungsstrasse 111 5232 Villigen PSI Switzerland Phone +41 56 310 36 02 heiner.billich(a)psi.ch<mailto:heiner.billich@psi.ch> https://www.psi.ch

1 0

NFS4 reexporter question
by Becker, Jeffrey C. (ARC-TN)[InuTeq, LLC] 08 Feb '19

08 Feb '19

Hello, I've built nfs-ganesha v2.7.1, and am trying to use the PSEUDO FSAL to reexport an nfs4 mounted directory from the ganesha server to a client. I used the base ganesha.conf (uncommenting out stuff I needed), and followed the examples in /usr/share/doc/packages/ganesha/config_samples/. The mount command on the client succeeds but when I do an ls in the mountpoint dir, there's nothing there. One issue is the client is also mounting the filesystem directly from the same place the ganesha server is, but I didn't think that mattered. I can send config files if needed. Please help me get this working. Thanks. -Jeff

5 17

Some errors starting ganesha - ceph
by Oscar Segarra 05 Feb '19

05 Feb '19

Hi, I'm running nfs-ganesha from docker container. It seems work perfectly, nevertheless when I start the server I get the following messages: Bootstrapping Ganesha NFS config, Bootstrapping CA cert, Skipping boostraping idmap config, Starting rpcbind, Starting rpc.statd, Starting sssd, Starting dbus, Starting Ganesha NFS, 28/01/2019 23:15:17 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 2.7.1, 28/01/2019 23:15:17 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed, 28/01/2019 23:15:17 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper., 28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized., 28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90, 28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL, 28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip, *28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs,* 28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized, *28/01/2019 23:15:18 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists,* *28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:0),* 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor., 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] rpc :TIRPC :EVENT :svc_rqst_hook_events: 0x4b7cb80 fd 1024 xp_refcnt 1 sr_rec 0x4af76d0 evchan 2 ev_refcnt 3 epoll_fd 37 control fd pair (35:36) hook failed (9), *28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nsm_connect :NLM :CRIT :connect to statd failed: RPC: Unknown protocol,* *28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nsm_unmonitor_all :NLM :CRIT :Unmonitor all nsm_connect failed,* 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED, 28/01/2019 23:15:20 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------, *28/01/2019 23:16:50 : epoch 5c4f7ef5 : vdicube_pub_ceph_nfs : ganesha.nfsd-25[reaper] nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE* I'd like to fix them in order to have a clean execution. Is this possible? Any help will be welcome, Thanks a lot. Ó

4 6