April 2023 - Support - Nfs Ganesha List Archives

ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal

by Robert Toole

Hi, I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date Centos 7 with nfs-ganesha 3.5. After following the Ceph install guide https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha I am able to create a NFS 4.1 Datastore in vmware using the ip address of all three nodes. Everything appears to work OK.. The issue however is that for some reason esxi is creating thick provisioned eager zeroed disks instead of thin provisioned disks on this datastore, whether I am migrating, cloning, or creating new vms. Even running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in a thick eager zeroed vmdk file. This should not be possible on an NFS datastore, because vmware requires a VAAI NAS plugin to accomplish thick provisioning over NFS before it can thick provision disks. Linux clients to the same datastore can create thin qcow2 images, and when looking at the images created by esxi from the linux hosts you can see that the vmdks are indeed thick: ls -lsh total 81G 512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog 40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk 40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk 512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk 512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk 0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd 2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx but the qcow2 files from the linux hosts are thin as one would expect: qemu-img create -f qcow2 big_disk_2.img 500G ls -lsh total 401K 200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img 200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img 512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm These ls -lsh results are the same from esx, linux nfs clients and from cephfs kernel client. What is happening here? Are there undocumented VAAI features in nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I want thin provisioned disks. ceph nfs export ls dev-nfs-cluster --detailed [ { "export_id": 1, "path": "/Development-Datastore", "cluster_id": "dev-nfs-cluster", "pseudo": "/Development-Datastore", "access_type": "RW", "squash": "no_root_squash", "security_label": true, "protocols": [ 4 ], "transports": [ "TCP" ], "fsal": { "name": "CEPH", "user_id": "dev-nfs-cluster1", "fs_name": "dev_cephfs_vol", "sec_label_xattr": "" }, "clients": [] } ] rpm -qa | grep ganesha nfs-ganesha-ceph-3.5-1.el7.x86_64 nfs-ganesha-rados-grace-3.5-1.el7.x86_64 nfs-ganesha-rados-urls-3.5-1.el7.x86_64 nfs-ganesha-3.5-1.el7.x86_64 centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch rpm -qa | grep ceph python3-cephfs-15.2.7-0.el7.x86_64 nfs-ganesha-ceph-3.5-1.el7.x86_64 python3-ceph-argparse-15.2.7-0.el7.x86_64 python3-ceph-common-15.2.7-0.el7.x86_64 cephadm-15.2.7-0.el7.x86_64 libcephfs2-15.2.7-0.el7.x86_64 ceph-common-15.2.7-0.el7.x86_64 ceph -v ceph version 15.2.7 (<ceph_uuid>) octopus (stable) The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm disks. -- Robert Toole rtoole(a)tooleweb.ca 403 368 5680

5 months, 2 weeks

4
5
0 / 0

Building NFS-Ganesha v5

by Kling Petersen, Torben

Hi all As there seem to be no RPMs for version 5 in available repositories, I tried to build the RPMs myself but get stuck at 79% (note I’m not an experienced developer so this may be a simple question). [ 79%] Linking C shared library libganesha_nfsd.so ../os/CMakeFiles/gos.dir/linux/acl.c.o: In function `acl_get_fd_np': /root/git/nfs-ganesha/src/os/linux/acl.c:44: undefined reference to `acl_get_fd' /root/git/nfs-ganesha/src/os/linux/acl.c:57: undefined reference to `acl_get_file' ../os/CMakeFiles/gos.dir/linux/acl.c.o: In function `acl_set_fd_np': /root/git/nfs-ganesha/src/os/linux/acl.c:79: undefined reference to `acl_set_fd' /root/git/nfs-ganesha/src/os/linux/acl.c:92: undefined reference to `acl_set_file' collect2: error: ld returned 1 exit status make[2]: *** [MainNFSD/CMakeFiles/ganesha_nfsd.dir/build.make:754: MainNFSD/libganesha_nfsd.so.5.1] Error 1 make[1]: *** [CMakeFiles/Makefile2:1922: MainNFSD/CMakeFiles/ganesha_nfsd.dir/all] Error 2 make: *** [Makefile:156: all] Error 2 Googling the problem essentially returns Q&As from 2007 with little to no useful info ...…. Any suggestions (or pointers to already built binaries) ?? Regards Torben ______________________ Dr. Torben Kling Petersen Distinguished Technologist +46-704 393 240 [image001.png]

2 years, 2 months

3
4
0 / 0

Re: intermittent problem with NFS

by Frank Filz

Hello, You may be running into a common problem with NFS. A common problem is the client doesn’t get any indication it should invalidate its cache of directory entries. Unless the client does something in the directory so it receives new attributes from the server, it will delay updating its cache. Depending on your workload you might have luck with reducing or eliminating the attribute cache delay. You can eliminate it entirely with the noac mount option, or just reduce or eliminate it for directories with acdirmin and acdirmax (check “man 5 nfs” for details). At least with a single Ganesha instance, you aren’t dealing with basically the same problem between Ganesha instances… Frank Filz From: Renaud Fortier [mailto:renaud.fortier.1@ulaval.ca] Sent: Thursday, April 20, 2023 5:05 AM To: support(a)lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] intermittent problem with NFS Good morning, I have an intermittent problem that I can not solve and today I ask for your help because I have no more idea. I have two Apache/PHP web servers (the clients) that use GlusterFS volumes via NFS (NFS-Ganesha) as data disks. The problem is that sometimes files saved to the server via PHP applications are not visible. Sometimes it's one of the two clients who doesn't see them, other times it's both clients. It's random because in general, everything works well. When I realize that a file is in trouble, I go to the directory “cd /data/folder” and I do an “ls -l”. I have no warnings or errors in the log files (clients and servers) Do you have any ideas or ways to help me solve this problem? Here's some informations. Let me know if you need anything else to help me. Thank you so much Versions: Linux: Debian 10 (clients and servers) NFS-Ganesha: 4.0 GlusterFS: 10.4 Mount Clients: 192.168.11.90:/prod_web on /data type nfs4 (rw,noatime,nodiratime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.11.150,local_lock=none,addr=192.168.11.90) Mount volume GlusterFS: /dev/mapper/vg3-prod_web on /data/glusterfs/prod_web/brick1 type xfs (rw,noatime,nouuid,attr2,inode64,noquota) Ganesha.conf: NFS_CORE_PARAM { NFS_Protocols = 4; # Pour éliminer l'erreur: Cannot bind RQUOTA udp6 socket, error (Address already in use) Enable_RQUOTA = false; RQUOTA_Port = 0; } NFS_KRB5 { Active_krb5 = false; } # You don't want grace to be less than lease, if it is, clients may not detect # server crash in time to reclaim lost locks during grace period. NFSV4 { Lease_Lifetime = 15; Grace_Period = 15; } LOG { ## Default log level for all components Default_Log_Level = WARN; } EXPORT { Export_Id = 3; Path = "/prod_web"; Pseudo = "/prod_web"; Access_Type = RW; Squash = No_root_squash; Disable_ACL = true; Protocols = "4"; Transports = "UDP","TCP"; SecType = "sys"; FSAL { Name = "GLUSTER"; Hostname = localhost; Volume = "prod_web"; } } Gluster Volume Info: Volume Name: prod_web Type: Replicate Volume ID: e918bd26-3318-48b3-8902-1a3b1de4f0f3 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster1.fsaa.local:/data/glusterfs/prod_web/brick1/brick Brick2: gluster2.fsaa.local:/data/glusterfs/prod_web/brick1/brick Brick3: gluster3.fsaa.local:/data/glusterfs/prod_web/brick1/brick Options Reconfigured: storage.build-pgfid: on diagnostics.brick-log-level: WARNING performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.cache-size: 1GB performance.parallel-readdir: off performance.read-ahead: off cluster.readdir-optimize: on client.event-threads: 4 server.event-threads: 4 features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.cache-invalidation: on performance.md-cache-timeout: 600 network.inode-lru-limit: 150000 auth.allow: 192.168.11.150,192.168.11.151,192.168.11.40 performance.nl-cache: on performance.nl-cache-timeout: 600 diagnostics.client-log-level: WARNING cluster.server-quorum-type: none cluster.enable-shared-storage: enable Renaud Fortier Administrateur RI Services communs des ressources informatiques, pédagogiques et technologiques (SCRIPT, FSAA, FFGG, SSE) Université Laval T  418 656-2131, poste 404789 <https://teams.microsoft.com/l/chat/0/0?users=REFOR103@ulaval.ca> Clavardons sur Microsoft Teams Pavillon Paul-Comtois, local 3100-C Québec (Québec) G1V 0A6 <https://www.ulaval.ca/notre-universite/avis-de-confidentialite.html> Avis relatif à la confidentialité

2 years, 2 months

2
1
0 / 0

V5.0 released

by Frank Filz

See release notes: https://github.com/nfs-ganesha/nfs-ganesha/wiki/ReleaseNotes_5

2 years, 2 months

1
0
0 / 0

intermittent problem with NFS

by Renaud Fortier

Good morning, I have an intermittent problem that I can not solve and today I ask for your help because I have no more idea. I have two Apache/PHP web servers (the clients) that use GlusterFS volumes via NFS (NFS-Ganesha) as data disks. The problem is that sometimes files saved to the server via PHP applications are not visible. Sometimes it's one of the two clients who doesn't see them, other times it's both clients. It's random because in general, everything works well. When I realize that a file is in trouble, I go to the directory “cd /data/folder” and I do an “ls -l”. I have no warnings or errors in the log files (clients and servers) Do you have any ideas or ways to help me solve this problem? Here's some informations. Let me know if you need anything else to help me. Thank you so much Versions: Linux: Debian 10 (clients and servers) NFS-Ganesha: 4.0 GlusterFS: 10.4 Mount Clients: 192.168.11.90:/prod_web on /data type nfs4 (rw,noatime,nodiratime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.11.150,local_lock=none,addr=192.168.11.90) Mount volume GlusterFS: /dev/mapper/vg3-prod_web on /data/glusterfs/prod_web/brick1 type xfs (rw,noatime,nouuid,attr2,inode64,noquota) Ganesha.conf: NFS_CORE_PARAM { NFS_Protocols = 4; # Pour éliminer l'erreur: Cannot bind RQUOTA udp6 socket, error (Address already in use) Enable_RQUOTA = false; RQUOTA_Port = 0; } NFS_KRB5 { Active_krb5 = false; } # You don't want grace to be less than lease, if it is, clients may not detect # server crash in time to reclaim lost locks during grace period. NFSV4 { Lease_Lifetime = 15; Grace_Period = 15; } LOG { ## Default log level for all components Default_Log_Level = WARN; } EXPORT { Export_Id = 3; Path = "/prod_web"; Pseudo = "/prod_web"; Access_Type = RW; Squash = No_root_squash; Disable_ACL = true; Protocols = "4"; Transports = "UDP","TCP"; SecType = "sys"; FSAL { Name = "GLUSTER"; Hostname = localhost; Volume = "prod_web"; } } Gluster Volume Info: Volume Name: prod_web Type: Replicate Volume ID: e918bd26-3318-48b3-8902-1a3b1de4f0f3 Status: Started Snapshot Count: 0 Number of Bricks: 1 x 3 = 3 Transport-type: tcp Bricks: Brick1: gluster1.fsaa.local:/data/glusterfs/prod_web/brick1/brick Brick2: gluster2.fsaa.local:/data/glusterfs/prod_web/brick1/brick Brick3: gluster3.fsaa.local:/data/glusterfs/prod_web/brick1/brick Options Reconfigured: storage.build-pgfid: on diagnostics.brick-log-level: WARNING performance.client-io-threads: on nfs.disable: on transport.address-family: inet performance.cache-size: 1GB performance.parallel-readdir: off performance.read-ahead: off cluster.readdir-optimize: on client.event-threads: 4 server.event-threads: 4 features.cache-invalidation: on features.cache-invalidation-timeout: 600 performance.cache-invalidation: on performance.md-cache-timeout: 600 network.inode-lru-limit: 150000 auth.allow: 192.168.11.150,192.168.11.151,192.168.11.40 performance.nl-cache: on performance.nl-cache-timeout: 600 diagnostics.client-log-level: WARNING cluster.server-quorum-type: none cluster.enable-shared-storage: enable Renaud Fortier Administrateur RI Services communs des ressources informatiques, pédagogiques et technologiques (SCRIPT, FSAA, FFGG, SSE) Université Laval T  418 656-2131, poste 404789 Clavardons sur Microsoft Teams<https://teams.microsoft.com/l/chat/0/0?users=REFOR103@ulaval.ca> Pavillon Paul-Comtois, local 3100-C Québec (Québec) G1V 0A6 Avis relatif à la confidentialité<https://www.ulaval.ca/notre-universite/avis-de-confidentialite.html>

2 years, 2 months

1
0
0 / 0

NFS-Ganesha V5.0 comming next week - please verify fixes and review patches

by Frank Filz

I will be tagging Ganesha V5.0 next week. I would appreciate if github issues that are marked for verification could be verified Please mark them verified, no need to close if not merged yet. When I tag V5.0 I will close all verified issues. I will also close all issues marked "try on supported version" that have not been responded to within 30 days. If there are any github issues you are passionate about being included in V5.0, please update the issue. Thanks Frank Filz

2 years, 2 months

1
0
0 / 0

2025

2024

2023

2022

2021

2020

2019

2018

Support April 2023