October 2021 - Support - NFS Ganesha List Archives

ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole 06 Jan '25

06 Jan '25

Hi, I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date Centos 7 with nfs-ganesha 3.5. After following the Ceph install guide https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha I am able to create a NFS 4.1 Datastore in vmware using the ip address of all three nodes. Everything appears to work OK.. The issue however is that for some reason esxi is creating thick provisioned eager zeroed disks instead of thin provisioned disks on this datastore, whether I am migrating, cloning, or creating new vms. Even running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in a thick eager zeroed vmdk file. This should not be possible on an NFS datastore, because vmware requires a VAAI NAS plugin to accomplish thick provisioning over NFS before it can thick provision disks. Linux clients to the same datastore can create thin qcow2 images, and when looking at the images created by esxi from the linux hosts you can see that the vmdks are indeed thick: ls -lsh total 81G 512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog 40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk 40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk 512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk 512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk 0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd 2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx but the qcow2 files from the linux hosts are thin as one would expect: qemu-img create -f qcow2 big_disk_2.img 500G ls -lsh total 401K 200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img 200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img 512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm These ls -lsh results are the same from esx, linux nfs clients and from cephfs kernel client. What is happening here? Are there undocumented VAAI features in nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I want thin provisioned disks. ceph nfs export ls dev-nfs-cluster --detailed [ { "export_id": 1, "path": "/Development-Datastore", "cluster_id": "dev-nfs-cluster", "pseudo": "/Development-Datastore", "access_type": "RW", "squash": "no_root_squash", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "dev-nfs-cluster1", "fs_name": "dev_cephfs_vol", "sec_label_xattr": "" }, "clients": [] } ] rpm -qa | grep ganesha nfs-ganesha-ceph-3.5-1.el7.x86_64 nfs-ganesha-rados-grace-3.5-1.el7.x86_64 nfs-ganesha-rados-urls-3.5-1.el7.x86_64 nfs-ganesha-3.5-1.el7.x86_64 centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch rpm -qa | grep ceph python3-cephfs-15.2.7-0.el7.x86_64 nfs-ganesha-ceph-3.5-1.el7.x86_64 python3-ceph-argparse-15.2.7-0.el7.x86_64 python3-ceph-common-15.2.7-0.el7.x86_64 cephadm-15.2.7-0.el7.x86_64 libcephfs2-15.2.7-0.el7.x86_64 ceph-common-15.2.7-0.el7.x86_64 ceph -v ceph version 15.2.7 (<ceph_uuid>) octopus (stable) The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm disks. -- Robert Toole rtoole(a)tooleweb.ca 403 368 5680

4 5

Announce Push of V4-rc2
by Frank Filz 29 Oct '21

29 Oct '21

Branch next Tag:V4-rc2 Ganesha V4.0 is in sight! Merge Highlights * NLM - fix issues with SM_NOTIFY handling * NLM - allow NLM to send messages to IPv4 only clients * NLM - additional and cleaned up debugging * PROXY_V4 - fix wrong type used for return value * KVSFS - cleanup supported bits * CEPH - fix stuck readdir * Always initialize rwlock when creating acl object * Do not read a fsal_filesystem if type is incomplete * Fix flood of "fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral" messages. Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: ef07e96e2 Frank S. Filz V4-rc2 4f2bf3953 Nuzhat Fathima Shaik Fix flood of "fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral" messages. 7e5502b09 Matthew DeVore Do not read a fsal_filesystem if type is incomplete b5b55dc39 Matthew DeVore Always initialize rwlock when creating acl object a5cc08308 fenghua Zhen Fix readdir stuck. When ceph-radosgw returns part of the dirs results, ganesha marks the last result as DIR_TERMINATE and rescan, caused an endless loop. 0eab09fa0 Frank S. Filz Cleanup some bits of what is supported in FSAL_KVSFS a5c0e66a9 Frank S. Filz PROXY_V4: proxyv4_rpc_read_reply returns an int, don't store in an enum c58c033cf Frank S. Filz NLM: for async response, if no IPv6 address is available convert IPv4 address 08c11eff7 Frank S. Filz Fix up SM_NOTIFY caller_name 5df7769e7 Frank S. Filz NLM: Use IPv4 address instead of encapsulated for caller_name dc03cb3dc Frank S. Filz Remove nlm_owner reference count held by nlm_state 38b5e8dd6 Frank S. Filz nsm client always has a caller_name, don't use ssc_client in hash aa5e540d4 Frank S. Filz Improve nsm_client debugging f7711bbc7 Frank S. Filz Fixup a wrong log component, nlm_util should be COMPONENT_NLM 8b38e8b96 Frank S. Filz Remove unused parameter from get_nsm_client eb283dee7 Frank S. Filz Move convert_ipv6_to_ipv4 to rpc_tools.c and add is_loopback

1 0

NFSv3 NLM service conflict with linux kernel client
by Olivier Garaud 26 Oct '21

26 Oct '21

Hi, Do you know if there is a workaround to the NLM problem we can experience when a client is running on the same host as Ganesha ? The linux kernel lock manager registers NLM to the portmapper overriding the registration done by ganesha. A more complete description of the problem can be found here: https://ganltc.github.io/problem-determination-guide-for-spectrum-scale-nfs… §NFS client or application hang due to NLM locks. Thanks for your help, Olivier

2 5

New to NFS Ganesha - seeking general info on tuning and benchmarking
by zlaymonk＠gmail.com 07 Oct '21

07 Oct '21

Hi, We have a 2-node NFS ganesha cluster (using CTDB), that I recently inherited administration for. Most of our internal users are satisfied with the performance, but one particular user has had all sorts of performance related issues affecting their adoption and usage of the NFS shares. From initial analysis, this seems to be a problem of loading (because it occurs during intensive reads/writes of large datasets, such as during rsync transfers or backup reads). So far, I have not yet been able to establish whether the problem is due to the number of files or the volume of data, and I am now setting up test client nodes to try and reproduce the client's issues. I googled around seeking information on tuning (kernel, NFS, OS, etc), but cannot seem to find any at all. Please, can someone share some insight into how to go about trying to optimize the performance of ganesha. Also, would appreciate any insight /suggestion on how best to benchmark the performance. Thanks in advance

2 1