ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole
Hi,
I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date
Centos 7 with nfs-ganesha 3.5.
After following the Ceph install guide
https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha
I am able to create a NFS 4.1 Datastore in vmware using the ip address
of all three nodes. Everything appears to work OK..
The issue however is that for some reason esxi is creating thick
provisioned eager zeroed disks instead of thin provisioned disks on this
datastore, whether I am migrating, cloning, or creating new vms. Even
running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in
a thick eager zeroed vmdk file.
This should not be possible on an NFS datastore, because vmware requires
a VAAI NAS plugin to accomplish thick provisioning over NFS before it
can thick provision disks.
Linux clients to the same datastore can create thin qcow2 images, and
when looking at the images created by esxi from the linux hosts you can
see that the vmdks are indeed thick:
ls -lsh
total 81G
512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog
40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk
40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk
512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk
512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk
0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd
2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx
but the qcow2 files from the linux hosts are thin as one would expect:
qemu-img create -f qcow2 big_disk_2.img 500G
ls -lsh
total 401K
200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img
200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img
512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm
These ls -lsh results are the same from esx, linux nfs clients and from
cephfs kernel client.
What is happening here? Are there undocumented VAAI features in
nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I
want thin provisioned disks.
ceph nfs export ls dev-nfs-cluster --detailed
[
{
"export_id": 1,
"path": "/Development-Datastore",
"cluster_id": "dev-nfs-cluster",
"pseudo": "/Development-Datastore",
"access_type": "RW",
"squash": "no_root_squash",
"security_label": true,
"protocols": [
4
],
"transports": [
"TCP"
],
"fsal": {
"name": "CEPH",
"user_id": "dev-nfs-cluster1",
"fs_name": "dev_cephfs_vol",
"sec_label_xattr": ""
},
"clients": []
}
]
rpm -qa | grep ganesha
nfs-ganesha-ceph-3.5-1.el7.x86_64
nfs-ganesha-rados-grace-3.5-1.el7.x86_64
nfs-ganesha-rados-urls-3.5-1.el7.x86_64
nfs-ganesha-3.5-1.el7.x86_64
centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch
rpm -qa | grep ceph
python3-cephfs-15.2.7-0.el7.x86_64
nfs-ganesha-ceph-3.5-1.el7.x86_64
python3-ceph-argparse-15.2.7-0.el7.x86_64
python3-ceph-common-15.2.7-0.el7.x86_64
cephadm-15.2.7-0.el7.x86_64
libcephfs2-15.2.7-0.el7.x86_64
ceph-common-15.2.7-0.el7.x86_64
ceph -v
ceph version 15.2.7 (<ceph_uuid>) octopus (stable)
The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm
disks.
--
Robert Toole
rtoole(a)tooleweb.ca
403 368 5680
4 weeks, 1 day
Announce Push of V5.5.3
by Frank Filz
Branch next
We're getting closer to V5.6, but this isn't quite it yet.
Tag:V5.5.3
Merge Highlights
* FSAL: in wait_to_start_io only increment wamt_read and want_write once
* FSAL: close_fsal_fd should only do fd lru stuff for global fd
* Get ref of gsh_export to prevent incorrect release and remove unnecessary
dec in lru_reclaim
* modify the incorrect value in comments
* FSAL: add the get_fsal_obj_handle for FSALs
* FSAL_VFS: Don't attempt to get ACLs from FDs opened with O_PATH
* Fix accounting of number of operations, per protocol, for server stats.
* Decrement fds_num_write_opens, if client don't close the file
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
2a57b6d53 Frank S. Filz V5.5.3
885598ce9 Rojin George Decrement fds_num_write_opens, if client don't close
the file
25cc51d00 Bjorn Leffler Fix accounting of number of operations, per
protocol, for server stats.
a8bbf2486 Martin Schwenke FSAL_VFS: Don't attempt to get ACLs from FDs
opened with O_PATH
34470a613 bjfhdhhaha FSAL: add the get_fsal_obj_handle for FSALs
1a0274fb9 bjfhdhhaha modify the incorrect value in comments
73e0d7b36 bjfhdhhaha Get ref of gsh_export to prevent incorrect release and
remove unnecessary dec in lru_reclaim
43f6878f1 Frank S. Filz FSAL: close_fsal_fd should only do fd lru stuff for
global fd
55aad71f4 Frank S. Filz FSAL: in wait_to_start_io only increment wamt_read
and want_write once
1 year, 4 months
Announce Push of V5.5.2
by Frank Filz
Branch next
Tag:V5.5.2
Merge Highlights
* A variety of minor fixes for compile under various platforms or
conditionals
* Handle Expired IP to Name mapping in cache
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
386c95c4a Frank S. Filz V5.5.2
d3b02e317 David Rieber It appears <sys/statfs.h> was included by mistake in
https://github.com/nfs-ganesha/nfs-ganesha/commit/43c28da6c88cbd842036599da0
4ddcfd24d959ea, the right #include should be <sys/stat.h>.
752f9bd8d David Rieber malloc_trim is unsupported on macOS.
f0cde4ae1 David Rieber Moved enum recovery_backend and associated #define to
solve forward declaration issue.
3d3fb7784 David Rieber Added extern declaration of reread_config to
nfs_lib.h
de2798974 David Rieber In vfs_getattr2, initialize variables status and
out_fd to fix uninitialized use.
d495eaa9b David Rieber fs_lock is only defined when GSH_CAN_HOST_LOCAL_FS is
#defined.
ca70ef529 Rojin George Handle Expired IP to Name mapping in cache
7b4e16826 bjfhdhhaha LOG: modify the incorrect function name in the log
1 year, 5 months