ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole
Hi,
I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date
Centos 7 with nfs-ganesha 3.5.
After following the Ceph install guide
https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha
I am able to create a NFS 4.1 Datastore in vmware using the ip address
of all three nodes. Everything appears to work OK..
The issue however is that for some reason esxi is creating thick
provisioned eager zeroed disks instead of thin provisioned disks on this
datastore, whether I am migrating, cloning, or creating new vms. Even
running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in
a thick eager zeroed vmdk file.
This should not be possible on an NFS datastore, because vmware requires
a VAAI NAS plugin to accomplish thick provisioning over NFS before it
can thick provision disks.
Linux clients to the same datastore can create thin qcow2 images, and
when looking at the images created by esxi from the linux hosts you can
see that the vmdks are indeed thick:
ls -lsh
total 81G
512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog
40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk
40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk
512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk
512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk
0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd
2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx
but the qcow2 files from the linux hosts are thin as one would expect:
qemu-img create -f qcow2 big_disk_2.img 500G
ls -lsh
total 401K
200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img
200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img
512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm
These ls -lsh results are the same from esx, linux nfs clients and from
cephfs kernel client.
What is happening here? Are there undocumented VAAI features in
nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I
want thin provisioned disks.
ceph nfs export ls dev-nfs-cluster --detailed
[
{
"export_id": 1,
"path": "/Development-Datastore",
"cluster_id": "dev-nfs-cluster",
"pseudo": "/Development-Datastore",
"access_type": "RW",
"squash": "no_root_squash",
"security_label": true,
"protocols": [
4
],
"transports": [
"TCP"
],
"fsal": {
"name": "CEPH",
"user_id": "dev-nfs-cluster1",
"fs_name": "dev_cephfs_vol",
"sec_label_xattr": ""
},
"clients": []
}
]
rpm -qa | grep ganesha
nfs-ganesha-ceph-3.5-1.el7.x86_64
nfs-ganesha-rados-grace-3.5-1.el7.x86_64
nfs-ganesha-rados-urls-3.5-1.el7.x86_64
nfs-ganesha-3.5-1.el7.x86_64
centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch
rpm -qa | grep ceph
python3-cephfs-15.2.7-0.el7.x86_64
nfs-ganesha-ceph-3.5-1.el7.x86_64
python3-ceph-argparse-15.2.7-0.el7.x86_64
python3-ceph-common-15.2.7-0.el7.x86_64
cephadm-15.2.7-0.el7.x86_64
libcephfs2-15.2.7-0.el7.x86_64
ceph-common-15.2.7-0.el7.x86_64
ceph -v
ceph version 15.2.7 (<ceph_uuid>) octopus (stable)
The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm
disks.
--
Robert Toole
rtoole(a)tooleweb.ca
403 368 5680
4 weeks, 1 day
Announce Push of V6.2
by Frank Filz
Branch next
Tag:V6.2
Merge Highlights
* config: coverity fixes related to interval
* 2 PNFS fixes
* Initialize read_arg->info properly for read_plus
* Sync lru_run with release_export
* fsal_destoryer.c: Didn't call unload for static fsals
* connection_manager: Add enum for stuck drains, and remove uneeded shutdown
in conenction manager
* src/SAL/nfs4_owner.c: Don't access owner if it's NULL.
* src/MainNFSD/nfs_main.c: Truncate should be after taking the lock.
* Avoid double release of read buffers
* Prevent nfs3 read from generating bad reply
* Avoid deref of uninitialized pointer
* FSAL_CEPH: Fix null pointer reference
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
c9ff03bb1 Frank S. Filz V6.2
59c6edd45 bjfhdhhaha FSAL_CEPH: Fix null pointer reference
e01052490 Steve Lord Avoid deref of uninitialized pointer
2cfbb48b0 Steve Lord Prevent nfs3 read from generating bad reply
98cb6ee3c Steve Lord Avoid double release of read buffers
25d59b8d7 Yoni Couriel src/MainNFSD/nfs_main.c: Truncate should be after
taking the lock.
046ea620e Yoni Couriel src/SAL/nfs4_owner.c: Don't access owner if it's
NULL.
b92252f97 Ofir Vainshtein connection_manager: Add enum for stuck drains, and
remove uneeded shutdown in conenction manager
43c4ae22f Ofir Vainshtein fsal_destoryer.c: Didn't call unload for static
fsals
df5b5e53d Xueqian Hu Sync lru_run with release_export
24da5c304 Deepak Arumugam Sankara Subramanian Initialize read_arg->info
properly for read_plus
1b010ad29 Marc Eshel pNFS need an extra hold for DS
d7d88f420 Marc Eshel GPFS: Add multiple pNFS FH option
bf4ca410e Sachin Punadikar config: coverity fixes related to interval
3 months, 2 weeks