I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date
Centos 7 with nfs-ganesha 3.5.
After following the Ceph install guide
I am able to create a NFS 4.1 Datastore in vmware using the ip address
of all three nodes. Everything appears to work OK..
The issue however is that for some reason esxi is creating thick
provisioned eager zeroed disks instead of thin provisioned disks on this
datastore, whether I am migrating, cloning, or creating new vms. Even
running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in
a thick eager zeroed vmdk file.
This should not be possible on an NFS datastore, because vmware requires
a VAAI NAS plugin to accomplish thick provisioning over NFS before it
can thick provision disks.
Linux clients to the same datastore can create thin qcow2 images, and
when looking at the images created by esxi from the linux hosts you can
see that the vmdks are indeed thick:
512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog
40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk
40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk
512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk
512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk
0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd
2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx
but the qcow2 files from the linux hosts are thin as one would expect:
qemu-img create -f qcow2 big_disk_2.img 500G
200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img
200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img
512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm
These ls -lsh results are the same from esx, linux nfs clients and from
cephfs kernel client.
What is happening here? Are there undocumented VAAI features in
nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I
want thin provisioned disks.
ceph nfs export ls dev-nfs-cluster --detailed
rpm -qa | grep ganesha
rpm -qa | grep ceph
ceph version 15.2.7 (<ceph_uuid>) octopus (stable)
The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm
403 368 5680
We’ve been running Ganesha 3.4 successfully over several months with several PROXY exports. Today for some reason, during Ganesha startup I see the following in the log:
19/03/2021 18:57:26 : epoch 6054f416 : ganesha : ganesha.nfsd-88505[main] export_commit_common :CONFIG :INFO :Export 1002 created at pseudo (/ecco) with path (/srv/rex/exports/ECCO_PD) and tag ((null)) perms (options=033030a2/000001e7 root_squash , R-r-, , , , , , , )
19/03/2021 18:57:27 : epoch 6054f416 : ganesha : ganesha.nfsd-88505[main] mdcache_exp_release :FSAL :INFO :Releasing PROXY export 1002 for /ecco
We’ve been successfully exporting /ecco until today. What would cause Ganesha to release an export? All my other exports are fine. The only thing that’s special about /ecco is that the server directory (/srv/rex/exports/ECCO_PD) contains about ten bind mounts which are scattered across three different Lustre file systems. That hasn’t been a problem until today.
Please advise. Thanks.
Hello new Genasha user here, using it under Ubuntu for serving NFS4.
If I configure ganesha to bind to ipv4 address it serves NFS4 alright:
Protocols = 4;
Bind_addr = 192.168.0.47;
However if I bind it to the IPv6 address on the same Ethernet
interface, the dæmon fails to start:
Protocols = 4;
Bind_addr = fe80::39a5:1385:634b:cb87;
The error is:
Cannot bind NFS udp6 socket, error 22 (Invalid argument)
Full error log at the end of the mail.
Attempts to solve:
Since the error indicates a problem in binding NFS, I thought to try
9P just to see if the problem is NFS-specific. So I changed the
Protocols = 9P;
Bind_addr = fe80::39a5:1385:634b:cb87;
And the dæmon starts successfully without error, however, it was bound to
Result is that other hosts can connect to port 564 using IPv4 address,
not what I thought Bind_addr does. I suspect there is a bug in this
Full Error Log (with NFS4)
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5836[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version 3.0.3
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] nfs_set_param_from_conf :NFS STARTUP :EVENT :Configuration file successfully parsed
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] init_server_pkgs :NFS STARTUP :EVENT :Initializing ID Mapper.
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] init_server_pkgs :NFS STARTUP :EVENT :ID Mapper successfully initialized.
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] nfs_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 90
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: =ep cap_sys_resource-ep
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] Bind_sockets_V6 :DISP :WARN :Cannot bind NFS udp6 socket, error 22 (Invalid argument)
16/03/2021 12:21:10 : epoch 60500806 : vnaeli-ga990fxaud5 : ganesha.nfsd-5842[main] Bind_sockets :DISP :FATAL :Error binding to V6 interface. Cannot continue.
We have been hit with a recent Fedora Linux kernel bug rendering the built-in NFS server unstable. After some web search I found this userspace replacement. Unfortunately, I am unable to get the server to do anything useful, like export a filesystem. I've created the following test config file:
Export_Id = 10;
Path = /srv/nfs/home;
Pseudo = /home;
Access_Type = RW;
Squash = no_root_squash;
Default_Log_Level = DEBUG;
name = FILE;
destination = "/var/log/ganesha/ganesha.log";
enable = active;
The service (called nfs-ganesha.service on Fedora) starts just fine and I see no errors of any kind in the log file. But /srv/nfs/home is not exported. I've checked with "ganesha_mgr show exports", which produces the following output:
Id, path, nfsv3, mnt, nlm4, rquota,nfsv40, nfsv41, nfsv42, 9p, last
0, /, 0, 0, 0, 0, 0, 0, 0, 0, Sun Feb 21 12:51:38 2021, 642993774 nsecs
On a client, I cannot mount "servername:/home" (no such directory), only "servername:/" (which is empty). Can anyone provide some guidance?