August 2021 - Support - NFS Ganesha List Archives

ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole 07 Jan '25

07 Jan '25

Hi, I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date Centos 7 with nfs-ganesha 3.5. After following the Ceph install guide https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha I am able to create a NFS 4.1 Datastore in vmware using the ip address of all three nodes. Everything appears to work OK.. The issue however is that for some reason esxi is creating thick provisioned eager zeroed disks instead of thin provisioned disks on this datastore, whether I am migrating, cloning, or creating new vms. Even running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in a thick eager zeroed vmdk file. This should not be possible on an NFS datastore, because vmware requires a VAAI NAS plugin to accomplish thick provisioning over NFS before it can thick provision disks. Linux clients to the same datastore can create thin qcow2 images, and when looking at the images created by esxi from the linux hosts you can see that the vmdks are indeed thick: ls -lsh total 81G 512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog 40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk 40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk 512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk 512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk 0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd 2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx but the qcow2 files from the linux hosts are thin as one would expect: qemu-img create -f qcow2 big_disk_2.img 500G ls -lsh total 401K 200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img 200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img 512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm These ls -lsh results are the same from esx, linux nfs clients and from cephfs kernel client. What is happening here? Are there undocumented VAAI features in nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I want thin provisioned disks. ceph nfs export ls dev-nfs-cluster --detailed [ { "export_id": 1, "path": "/Development-Datastore", "cluster_id": "dev-nfs-cluster", "pseudo": "/Development-Datastore", "access_type": "RW", "squash": "no_root_squash", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "dev-nfs-cluster1", "fs_name": "dev_cephfs_vol", "sec_label_xattr": "" }, "clients": [] } ] rpm -qa | grep ganesha nfs-ganesha-ceph-3.5-1.el7.x86_64 nfs-ganesha-rados-grace-3.5-1.el7.x86_64 nfs-ganesha-rados-urls-3.5-1.el7.x86_64 nfs-ganesha-3.5-1.el7.x86_64 centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch rpm -qa | grep ceph python3-cephfs-15.2.7-0.el7.x86_64 nfs-ganesha-ceph-3.5-1.el7.x86_64 python3-ceph-argparse-15.2.7-0.el7.x86_64 python3-ceph-common-15.2.7-0.el7.x86_64 cephadm-15.2.7-0.el7.x86_64 libcephfs2-15.2.7-0.el7.x86_64 ceph-common-15.2.7-0.el7.x86_64 ceph -v ceph version 15.2.7 (<ceph_uuid>) octopus (stable) The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm disks. -- Robert Toole rtoole(a)tooleweb.ca 403 368 5680

4 5

The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
by Desert 31 Aug '21

31 Aug '21

system version: centos8 nfs-ganesha version: 3.5-3 log: Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4. Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648 bytes. Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0 dumped core. Stack trace of thread 1066334: #0 0x00007fe484c6e37f n/a (n/a) #1 0x000000003789c20b n/a (n/a)

3 3

Performance of RGW_FSAL vs CEPH_FSAL
by ethan.deson＠oracle.com 30 Aug '21

30 Aug '21

Is the RGW_FSAL using the S3 API under the hood (aka sending S3 http requests)? Someone on my team is conjecturing that RGW_FSAL will be markedly slower than CEPH_FSAL on that basis. Does anyone have a sense of the average performance characteristics of these two FSALs?

2 1

The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
by Desert 30 Aug '21

30 Aug '21

system version: centos8 nfs-ganesha version: 3.5-3 log: Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4. Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648 bytes. Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0 dumped core. Stack trace of thread 1066334: #0 0x00007fe484c6e37f n/a (n/a) #1 0x000000003789c20b n/a (n/a)

1 0

NFS Ganesha is Read Only Filesystem
by ethan.deson＠oracle.com 26 Aug '21

26 Aug '21

I have a ceph filesystem I'm trying to export using nfs ganesha. Ceph indicates that the cluster is fine: ceph status cluster: id: 108a355b-1e9b-493d-91a0-ab898a67aa51 health: HEALTH_OK ... mds: cephfs:1 {0=unique=up:active} But when I try to mount the filesystem over nfs with nfs ganesha I get: $ mount -t nfs -o nfsvers=4.1,noauto,soft,sync localhost:/ test $ touch test/i touch: cannot touch ‘test/i’: Read-only file system And there are errors in ganesha.log Anything obvious wrong with my configuration? { Export_Id = 25; Path = "/"; Pseudo = "/"; Access_Type = RW; Protocols = 4; Transports = TCP; SecType = sys; Squash = No_Root_Squash; FSAL { Name = "CEPH"; Filesystem = "cephfs"; User_Id = "testuser"; Secret_Access_Key = "<removed secret for post>"; } } CEPH { Ceph_Conf = "/etc/ceph/ceph.conf"; } 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.4 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper. 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized. 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] create_export :FSAL :CRIT :Unable to init Ceph handle for /. 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/) to (/) 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:14): 1 validation errors in block FSAL 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:14): Errors processing block (FSAL) 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:1): 1 validation errors in block EXPORT 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] config_errs_to_log :CONFIG :CRIT :Config File (/etc/ganesha/ganesha.conf:1): Errors processing block (EXPORT) 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,35,36,37+eip 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] gsh_dbus_pkginit :DBUS :CRIT :dbus_bus_get failed (Failed to connect to socket /run/dbus/system_bus_socket: No such file or directory) 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] gsh_dbus_register_path :DBUS :CRIT :dbus_connection_register_object_path called with no DBUS connection 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:99) 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor. 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[dbus] gsh_dbus_thread :DBUS :CRIT :DBUS not initialized, service thread exiting 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[dbus] gsh_dbus_thread :DBUS :EVENT :shutdown 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] rpc :TIRPC :EVENT :svc_rqst_hook_events: 0x5615da3f7070 fd 1024 xp_refcnt 1 sr_rec 0x5615da183cc0 evchan 2 ev_refcnt 3 epoll_fd 20 control fd pair (18:19) direction in hook failed (9) 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nsm_connect :NLM :EVENT :connect to statd failed: RPC: Unknown protocol 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nsm_unmonitor_all :NLM :EVENT :Unmonitor all nsm_connect failed 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_start :NFS STARTUP :EVENT :------------------------------------------------- 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED 26/08/2021 02:24:34 : epoch 6126fb62 : otec-blockstorage-imaging-red-1 : ganesha.nfsd-1221[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------

3 5

Fwd: [Gluster-users] Ganesha+Gluster strange issue: incomplete directory reads
by Kaleb Keithley 26 Aug '21

26 Aug '21

---------- Forwarded message --------- From: Ivan Rossi <rouge2507(a)gmail.com> Date: Thu, Aug 26, 2021 at 1:01 PM Subject: [Gluster-users] Ganesha+Gluster strange issue: incomplete directory reads To: <gluster-users(a)gluster.org> Hello list, Ganesha (but not Gluster) newbie here. This is the first time I have to set-up Ganesha to serve a Gluster volume, but it seems i stumbled on a weird issue. Hope it is due to my inexperience with Ganesha. I need to serve a Gluster volume to some old production VMs that cannot install a recent Gluster client. Thus they need to access the Gluster data using the standard Linux NFS client. Since native Gluster NFS server is gone, I had to go the Ganesha route. Volume served is used for bioinformatics analysis, each subdirectory containing on the order of a thousand files, a few of them large (think 50 Gb each) Now the issue: When the volume is mounted on the client (using *NFSv3*) directory reads SOMETIMES return an INCOMPLETE list of files. Problem goes away if you redo the read in a different way as if the first directory metadata read did not complete successfully but it is then cached anyway. Problem does not manifests if there are few files in the directory or they are all small (think < 1 GB) Direct access to the files is OK eve if they did not show up in the ls. E.g. : mount -t nfs ganesha:/srv/glfs/work /mnt/ ls /mnt/47194616IMS187mm10 | wc -l # wrong result ls: reading directory /mnt/47194616IMS187mm10: Input/output error 304 # right ( NB ls-l returns one line more than plain ls) ls -l /mnt/47194616IMS187mm10 | wc -l 668 # after 'ls -l' now even plain ls returns the expected number of files ls /mnt/47194616IMS187mm10 | wc -l 667 Furthermore i see the Input/output message only because of the pip to wc, if i just run plain ls, in a terminal it fails silently returning a partial list. If the client mounts the volume using *NFSv4* everything looks as expected. mount -t nfs -o vers=4.0 ganesha:/work /mnt/ ls /mnt/47194616IMS187mm10 | wc -l 667 but as you can guess my confidence in using Ganesha in production is somewhat shaking ATM. My feeling is that it is a Ganesha problem or something lacking in the Ganesha configuration for Gluster. My Ganesha configuration is basically just defaults. No failover conf either. My Gluster setup has nothing strange, I am just serving a R3 volume and defaults are just fine to get a fast volume given the hardware. Furthermore the volume looks fine from the Gluster clients. I am using Gluster 8.4 and Ganesha 3.4 on Debian 10 (buster). Packages coming from the Gluster and Ganesha repos, not the debian one. Has anyone seen anything similar before? Did I stumble on a bug? Any advice or common wisdom to share? Ivan Rossi ________ Community Meeting Calendar: Schedule - Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC Bridge: https://meet.google.com/cpu-eiue-hvk Gluster-users mailing list Gluster-users(a)gluster.org https://lists.gluster.org/mailman/listinfo/gluster-users -- Kaleb

1 0

Ref Leak in Chunk Cache
by rosrake＠amazon.com 18 Aug '21

18 Aug '21

Hi, I am running ganesha 3.5. I have a directory structure such that it has 25000 directories with each one of them contain 500 files. The ChunkSize is set to 1000 and Entries-HWMark is set to 50000 in the ganesha configuration. I run a script to list all the 25000 directories. I ran the list command three times. No other clients were interacting with server. Through the stats tool, I retrieved the chunk count value. 1st Run: Chunk Count: 58335 2nd Run: Chunk Count: 69792 3rd Run: Chunk Count: 73270 From my understanding, the Chunk Cache stores the directory information about a directory. So with 25000 directories with 500 files each, ideally the chunk count should be around 25000. One for each directory. And on subsequent runs, it should not increase. I have not configured the Chunk_HWMark value. So it is set to the default 100k. I suspect there is some chunk ref leak that is causing the chunk count to grow. Setup Summary Ganesha: 3.5 Cache Configuration: MDCACHE { Dir_Chunk = 1000; Entries_HWMark = 500000; } VFS: Custom

2 2