May 2024 - Support - NFS Ganesha List Archives

ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole 07 Jan '25

07 Jan '25

Hi, I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date Centos 7 with nfs-ganesha 3.5. After following the Ceph install guide https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha I am able to create a NFS 4.1 Datastore in vmware using the ip address of all three nodes. Everything appears to work OK.. The issue however is that for some reason esxi is creating thick provisioned eager zeroed disks instead of thin provisioned disks on this datastore, whether I am migrating, cloning, or creating new vms. Even running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in a thick eager zeroed vmdk file. This should not be possible on an NFS datastore, because vmware requires a VAAI NAS plugin to accomplish thick provisioning over NFS before it can thick provision disks. Linux clients to the same datastore can create thin qcow2 images, and when looking at the images created by esxi from the linux hosts you can see that the vmdks are indeed thick: ls -lsh total 81G 512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog 40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk 40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk 512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk 512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk 0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd 2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx but the qcow2 files from the linux hosts are thin as one would expect: qemu-img create -f qcow2 big_disk_2.img 500G ls -lsh total 401K 200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img 200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img 512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm These ls -lsh results are the same from esx, linux nfs clients and from cephfs kernel client. What is happening here? Are there undocumented VAAI features in nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I want thin provisioned disks. ceph nfs export ls dev-nfs-cluster --detailed [ { "export_id": 1, "path": "/Development-Datastore", "cluster_id": "dev-nfs-cluster", "pseudo": "/Development-Datastore", "access_type": "RW", "squash": "no_root_squash", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "dev-nfs-cluster1", "fs_name": "dev_cephfs_vol", "sec_label_xattr": "" }, "clients": [] } ] rpm -qa | grep ganesha nfs-ganesha-ceph-3.5-1.el7.x86_64 nfs-ganesha-rados-grace-3.5-1.el7.x86_64 nfs-ganesha-rados-urls-3.5-1.el7.x86_64 nfs-ganesha-3.5-1.el7.x86_64 centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch rpm -qa | grep ceph python3-cephfs-15.2.7-0.el7.x86_64 nfs-ganesha-ceph-3.5-1.el7.x86_64 python3-ceph-argparse-15.2.7-0.el7.x86_64 python3-ceph-common-15.2.7-0.el7.x86_64 cephadm-15.2.7-0.el7.x86_64 libcephfs2-15.2.7-0.el7.x86_64 ceph-common-15.2.7-0.el7.x86_64 ceph -v ceph version 15.2.7 (<ceph_uuid>) octopus (stable) The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm disks. -- Robert Toole rtoole(a)tooleweb.ca 403 368 5680

4 5

Announce Push of V5.9
by Frank Filz 21 May '24

21 May '24

Branch next Tag:V5.9 Sorry, this patch should not have been back ported. Merge Highlights * Revert "Require librgw 1.2.1" Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 7b6f5b543 Frank S. Filz V5.9 8aae04dca Frank S. Filz Revert "Require librgw 1.2.1"

1 0

Announce Push of V5.8
by Frank Filz 17 May '24

17 May '24

Branch: V%-stable Tag:V5.8 NOTE: This release includes an ntirpc pullup, please update your submodule. Collected bug fixes from V5.7 through V6-dev.13 Merge Highlights * Fixes to POSIX ACL handling * Introduction of SaunaFS FSAL * Memory leak fixes * Delegation fixes * ntirpc V5.8 fixes * fd_lru and other fsal_fd related fixes * FSAL_PSEUDO set change, mtime, ctime on create directory * Some logging changes Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com> Contents: 96adc44d9 Frank S. Filz V5.8 7bb7e8ef7 JAGNYA DATTA TRIPATHY rtpref and wtpref values should not exceed rtmax and wtmax in fsinfo e8a5a2bd0 Deepak Arumugam Sankara Subramanian Release refcount on entry before restarting readdir 505e0ab8a Rojin George Handle parallel delegation recalls for the same fsal_objs f305617fe Rubén Alcolea Núñez Add SaunaFS FSAL d93544d8a Daniel Gryniewicz Require librgw 1.2.1 4cfd08de6 Dhairya Parmar doc: fix cmount_path doc 30de61d57 Sachin Punadikar FSAL_CEPH: Avoid duplicate refcounting of cmount b96b0b2e6 Frank S. Filz FSAL: Fix up gsh_refstr references for op_context. 24df89347 Frank S. Filz nfs4_Check_Stateid must not return NFS4ERR_STALE_STATEID for 4.1 3e926c9cb Frank S. Filz export_mgr: Dont' leak refstr in get_gsh_export_by_pseudo_locked 2efe050cd Frank S. Filz CEPH: Free EXPORT { FSAL {} } config strings 4b720e15b Assaf Yaari commonlib: Skip redundant aces in mode_to_acl 146d6616b Assaf Yaari commonlib: In fsal_mode_to_acl identify if can reuse aces ddea6c571 Assaf Yaari commonlib: Place OWNER & GROUP mode generated aces at the beginning 1e67914ef Assaf Yaari commonlib: In fsal_mode_gen_acl the group perm should serve as mask 2689c727b Assaf Yaari access_check: In fsal_check_access_acl reset the FSAL_ACE4_REQ_FLAG from missing_access d67ca51b6 Assaf Yaari commonlib: In fsal_mode_to_acl don't dup inherit aces 6f74355c3 Assaf Yaari commonlib: Identify correctly special id's 072ebac56 Assaf Yaari commonlib: Remove GROUP_ID flag from special "who" 1123cc85a Assaf Yaari commonlib: Add default flags for generated aces from mode-bits 9ae4b8c5e Assaf Yaari nfs_proto_tools: In encode_acl convert mode to acl if no acl present f197b9492 Assaf Yaari nfs4_acls: Free the acl object even if not found in the hashtable aedcea44c Assaf Yaari commonlib: In fsal_mode_gen_acl create properly the generates acl object f718c393f Assaf Yaari commonlib: Create properly ACL new entry when converting mode to ACL a3e968728 Assaf Yaari commonlib: Re-write the implementation of fsal_acl_to_mode b54157111 Assaf Yaari fsal_mode_to_acl: Advance the dace pointer only when copying the source ace 400d45d13 Assaf Yaari fsal.h: Don't unset the ATTR_ACL from valid_mask in case it was set by src ed807cdfa Assaf Yaari nfs4_op_putfh: Update logs to info level on case of errors 113530b9b Assaf Yaari nfs4_op_create_session: Upgrade some debug logs to info 7b369850e Assaf Yaari log_functions: Don't call Cleanup from Fatal handling function d688de58f Daniel Gryniewicz GTest - fix tests for new FSAL APIs c628f07d9 Yogendra Charya Tangalapally GPFS: FSAL's state_free function called by free_state doesn't actually free 6376199fb Martin Schwenke doc: Update default for Readdir_Res_Size to 32KB bf6229e47 Sachin Punadikar FSAL_CEPH : Add warnings for export creation cb4fba122 Frank S. Filz fsal_fd fd_work/io_work needs separate condition variables 524ae1ff2 Frank S. Filz Add PTHREAD_COND_broadcast macro 4080d5dd7 Rojin George Free delegation recall context after cleaning expired clients 43134126e Rojin George Decrement Client Ref, if reserve lease or expire fails 46865aea5 Rojin George Handle parallel delegation recalls for the same fsal_objs c4e201cda Rojin George Insert global FDs of reopened files into fd lru 528cd3127 Rojin George Provide FSALs option to remove global FDs from fd_lru upon failures 5c443abb6 Rojin George Expose bump_fd_lru to FSALs to bump up if reusing global FDs 9db158630 Rojin George Insert to fd_lru only if open/reopen succeeds 36f4003d4 Kaleb S. KEITHLEY threads: set appropriate thread stack size for fridge and ntirpc workers 2eaeed5f6 Deepak Arumugam Sankara Subramanian Reduce the default value of Readdir_Res_Size to 32KB a8dd100a1 Deepak Arumugam Sankara Subramanian Use after free in fd_lru 6ab8377b1 Jon Kohler use may_write == false in nfs4_State_Get_Pointer a425b8877 Rojin George Use __GLIBC__ macro to set pthread RW lock kind 310d2dfcb Rojin George Fix log_an_export critical path overhead 3389cf1b7 Frank S. Filz pullup ntirpc v5.8 f11603e6a Frank S. Filz Fix memory leak in DRC 3f8475e77 Frank S. Filz GPFS: Fix unexport of 1st export on filesystem 39dd30960 Frank S. Filz FSAL: add function to get an export ref from a filesystem. 78d589d36 Frank S. Filz Don't leak memory when PSEUDOFS block is not specified. 47c5b4da0 Frank S. Filz Don't leak memory for FSAL_LIST config block af60e7943 Frank S. Filz CONFIG: Add ability to specify a block where memory is not allocated if unused 1e842bce0 sunpxyz bugfix: https://github.com/nfs-ganesha/nfs-ganesha/issues/1044 1c32059c1 Xueqian Hu Enable libunwind in ganesha 2ee8c33ba Rojin George Handle delegation recalls for expired clients 2d2e217b2 Rojin George Grant Read delegations properly d9f49a223 Rojin George Instrumentation changes to debug future lru_try_one crash 6d640a75d Rojin George Clear delegation stats if no active states 8656b8eb4 Rojin George Fix few typos around the fd_lru logics e32917e6c Rojin George Decrement fd_work while reclaiming if FD is busy c3e1f8389 bjfhdhhaha ACL: fix issue of setting posix acl for directory return EINVAL a15e5aaec bjfhdhhaha ACL: fix memory leak cd8a2d06a Prabhu Murugesan Fix crash at release_lock_owner()-> _get_gsh_export_ref() 9d719e666 Jason Woodward FSAL_PSEUDO set change, mtime, ctime on create directory fbb7ff4c9 Kaleb S. KEITHLEY monitoring: fails to build with gcc-14 d34f62ea9 Sachin Punadikar FSAL_CEPH : Add debug statements d0e756adb David Rieber Move the log levels and components to a separate headfer file. 1fdb4face David Rieber NFSv4 READDIR should pass down the attrs from the request. 0b81b57ad Sachin Punadikar Fix compiler warnings for "-Werror" flag ea155b64b Sachin Punadikar LOG : Parse correct name for COMPONENT_MDCACHE_LRU 32fc60b01 Daniel Gryniewicz Allow specifying python version 8315e6df9 Daniel Gryniewicz Fix monitoring build option cef733cb8 Daniel Gryniewicz Ignore clangd cache directory 0a752ade9 Frank S. Filz FSAL: Fix up how reaper_work is used 51d16a390 Frank S. Filz FSAL: in open2 methods, update_share_counters_locked must use *new_obj 849e6547f Frank S. Filz LOG: Fix buffer overflow issue e8293aa47 Kaleb S. KEITHLEY build: use standard (i.e. pep517) build and installer 514763104 Prabhu Murugesan Fix crash at free_client_id()->PTHREAD_MUTEX_destroy() 9b353349f zhitaoli Fix: dec ref_count of confirmed client_id c87fa5f30 David Rieber Fix use-of-uninitialized value error. 848ee3320 David Rieber malloc_trim is unsupported on darwin. 6822ab8c2 David Rieber Initialize export_opt_lock earlier. bb3fe4fbe Trupti Shete Resolving error in ganesha-top utility 247e45d15 Trupti Shete Issue in ganesha_stats json option part_5 ea0ccbb73 Trupti Shete Issue in ganesha_stats json option part_4 d0ace9456 Trupti Shete Issue in ganesha_stats json option part_3 bad643006 Trupti Shete Issue in ganesha_stats json option part_2 7e55566e3 Trupti Shete ganesha_stats json issues part_1 a9a42e780 Frank S. Filz FSAL's state_free function called by free_state doesn't actually free

1 0

lock issue for same clientid on the same node using NFS Ganesha 3.5
by pragash2here＠gmail.com 09 May '24

09 May '24

We are using NFS Ganesha 3.5 and occassionaly we see the following messages on the client host. The applications are no longer able to take locks. They are all taking RDLCKs over the entire file range. fcntl(19, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=1800}) The host has multiple mounts of the SAME exported filesystem. All the mounts are identical except for the mount path. Each mount is sending the the SAME SETCLIENTID to the NFS Ganesha. Any pointers much appreciated. 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_10] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_10] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_14] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_14] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_17] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1310 refcount=7} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}

2 4

Enable POSIX ACL support in VFS FSAL?
by Sean Caron 02 May '24

02 May '24

Hello! I noticed in the release notes that in V4.1 and onward, POSIX ACL support is now available in the VFS FSAL but is disabled by default. I looked through the documentation but I don't see any notes about the parameter that might be used to enable said support. Could someone please let me know how one might enable POSIX ACL support in the VFS FSAL? We're running the latest available stable build from PPA, 5.7GA. Thanks! Sean Sean Caron (she/they) OS Programmer Lead University of Michigan Center for Statistical Genetics Department of Biostatistics, School of Public Health e: scaron(a)umich.edu p: 734-474-5962

3 5