ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole
Hi,
I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date
Centos 7 with nfs-ganesha 3.5.
After following the Ceph install guide
https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha
I am able to create a NFS 4.1 Datastore in vmware using the ip address
of all three nodes. Everything appears to work OK..
The issue however is that for some reason esxi is creating thick
provisioned eager zeroed disks instead of thin provisioned disks on this
datastore, whether I am migrating, cloning, or creating new vms. Even
running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in
a thick eager zeroed vmdk file.
This should not be possible on an NFS datastore, because vmware requires
a VAAI NAS plugin to accomplish thick provisioning over NFS before it
can thick provision disks.
Linux clients to the same datastore can create thin qcow2 images, and
when looking at the images created by esxi from the linux hosts you can
see that the vmdks are indeed thick:
ls -lsh
total 81G
512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog
40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk
40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk
512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk
512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk
0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd
2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx
but the qcow2 files from the linux hosts are thin as one would expect:
qemu-img create -f qcow2 big_disk_2.img 500G
ls -lsh
total 401K
200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img
200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img
512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm
These ls -lsh results are the same from esx, linux nfs clients and from
cephfs kernel client.
What is happening here? Are there undocumented VAAI features in
nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I
want thin provisioned disks.
ceph nfs export ls dev-nfs-cluster --detailed
[
{
"export_id": 1,
"path": "/Development-Datastore",
"cluster_id": "dev-nfs-cluster",
"pseudo": "/Development-Datastore",
"access_type": "RW",
"squash": "no_root_squash",
"security_label": true,
"protocols": [
4
],
"transports": [
"TCP"
],
"fsal": {
"name": "CEPH",
"user_id": "dev-nfs-cluster1",
"fs_name": "dev_cephfs_vol",
"sec_label_xattr": ""
},
"clients": []
}
]
rpm -qa | grep ganesha
nfs-ganesha-ceph-3.5-1.el7.x86_64
nfs-ganesha-rados-grace-3.5-1.el7.x86_64
nfs-ganesha-rados-urls-3.5-1.el7.x86_64
nfs-ganesha-3.5-1.el7.x86_64
centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch
rpm -qa | grep ceph
python3-cephfs-15.2.7-0.el7.x86_64
nfs-ganesha-ceph-3.5-1.el7.x86_64
python3-ceph-argparse-15.2.7-0.el7.x86_64
python3-ceph-common-15.2.7-0.el7.x86_64
cephadm-15.2.7-0.el7.x86_64
libcephfs2-15.2.7-0.el7.x86_64
ceph-common-15.2.7-0.el7.x86_64
ceph -v
ceph version 15.2.7 (<ceph_uuid>) octopus (stable)
The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm
disks.
--
Robert Toole
rtoole(a)tooleweb.ca
403 368 5680
4 weeks
Announce Push of V4.0
by Frank Filz
Branch next
Tag:V4.0
Release Notes can be found on the github project wiki:
https://github.com/nfs-ganesha/nfs-ganesha/wiki/ReleaseNotes_4
Merge Highlights
* Protect from null dereference in export_to_dbus()
* ganesha_mgr:Added input validation for display export
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
6156de38f Frank S. Filz V4.0
9e4726788 Prabhu Murugesan ganesha_mgr:Added input validation for display
export
775272db4 Frank S. Filz Protect from null dereference in export_to_dbus()
3 years, 1 month
With V4-rc6 out and the year almost over we are almost there!
by Frank Filz
I just pushed V4-rc6. Next week is the last full week before the end of year
holiday time and I expect to be the last merge before I tag 4.0 (which may
happen separately on Monday the 20th). With the final push, I encourage
anyone who has a github issue that has been marked "Needs Verification" to
verify the issue with V4-rc6. Any additional testing folks are able to do
would be much appreciated. Any patches that have been submitted to gerrithub
but not yet merged will need to be wrapped up next week.
Following the V4.0 tag, I will follow our standard practice where I do not
fork off a v4-stable branch for a few weeks which gives the opportunity for
a V4.1 update to include some bug fixes before I start merging new feature
work for the V5 development stream.
Thanks everyone. This has been a long release, but we have made some
significant code improvements and fixed a lot of bugs and we do have some
features of note:
More information on the wiki:
https://github.com/nfs-ganesha/nfs-ganesha/wiki/ReleaseNotes_4
POSIX ACL support for FSAL_CEPH
Enhancements to Export Update (DBUS) and Reload (SIGHUP)
protocol update (add/remove v3/v4 to export)
Pseudo Path can be changed when reloading exports
Refresh file system table (allows adding exports for new file
systems)
FSAL_PROXY_V3
FSAL_CEPH enhancement to release inodes quicker
NFSACL POSIX ACL side-band protocol for NFSv3
Option to increase aggressiveness of MDCACHE releasing entries
Can actually disable RQUOTA and NFSv3 when building
Improvements to building on MacOS and other compilers
FSAL_KVSFS
Match xattr support up with RFC 8276
Add xattr to FSAL_CEPH
Support BTRFS subvolumes as if they are separate file systems
Said page still needs some updates so if anyone sees anything that needs
updating, let us know.
Thanks
Frank Filz
3 years, 1 month
Announce Push of V4-rc6
by Frank Filz
Branch next
Tag:V4-rc6
NOTE: This merge includes an ntirpc pull up. Please update your submodule
Ganesha V4.0 is in sight!
Merge Highlights
* On resume_op_context call SetClientIP
* DRC: Better handle duplicate requests while a request is in progress
* selinux: non-fatal yum install error of nfs-ganesha
* Remove 3rd parameter from xdrproc_t calls
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
872aab911 Frank S. Filz V4-rc6
ea695cc84 Frank S. Filz Remove 3rd parameter from xdrproc_t calls
dae2da63d Kaleb S. KEITHLEY selinux: non-fatal yum install error of
nfs-ganesha
9316124dd Frank S. Filz DRC: Better handle duplicate requests while a
request is in progress
885db6e52 Frank S. Filz On resume_op_context call SetClientIP
3 years, 1 month