ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole
Hi,
I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date
Centos 7 with nfs-ganesha 3.5.
After following the Ceph install guide
https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha
I am able to create a NFS 4.1 Datastore in vmware using the ip address
of all three nodes. Everything appears to work OK..
The issue however is that for some reason esxi is creating thick
provisioned eager zeroed disks instead of thin provisioned disks on this
datastore, whether I am migrating, cloning, or creating new vms. Even
running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in
a thick eager zeroed vmdk file.
This should not be possible on an NFS datastore, because vmware requires
a VAAI NAS plugin to accomplish thick provisioning over NFS before it
can thick provision disks.
Linux clients to the same datastore can create thin qcow2 images, and
when looking at the images created by esxi from the linux hosts you can
see that the vmdks are indeed thick:
ls -lsh
total 81G
512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog
40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk
40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk
512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk
512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk
0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd
2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx
but the qcow2 files from the linux hosts are thin as one would expect:
qemu-img create -f qcow2 big_disk_2.img 500G
ls -lsh
total 401K
200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img
200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img
512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm
These ls -lsh results are the same from esx, linux nfs clients and from
cephfs kernel client.
What is happening here? Are there undocumented VAAI features in
nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I
want thin provisioned disks.
ceph nfs export ls dev-nfs-cluster --detailed
[
{
"export_id": 1,
"path": "/Development-Datastore",
"cluster_id": "dev-nfs-cluster",
"pseudo": "/Development-Datastore",
"access_type": "RW",
"squash": "no_root_squash",
"security_label": true,
"protocols": [
4
],
"transports": [
"TCP"
],
"fsal": {
"name": "CEPH",
"user_id": "dev-nfs-cluster1",
"fs_name": "dev_cephfs_vol",
"sec_label_xattr": ""
},
"clients": []
}
]
rpm -qa | grep ganesha
nfs-ganesha-ceph-3.5-1.el7.x86_64
nfs-ganesha-rados-grace-3.5-1.el7.x86_64
nfs-ganesha-rados-urls-3.5-1.el7.x86_64
nfs-ganesha-3.5-1.el7.x86_64
centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch
rpm -qa | grep ceph
python3-cephfs-15.2.7-0.el7.x86_64
nfs-ganesha-ceph-3.5-1.el7.x86_64
python3-ceph-argparse-15.2.7-0.el7.x86_64
python3-ceph-common-15.2.7-0.el7.x86_64
cephadm-15.2.7-0.el7.x86_64
libcephfs2-15.2.7-0.el7.x86_64
ceph-common-15.2.7-0.el7.x86_64
ceph -v
ceph version 15.2.7 (<ceph_uuid>) octopus (stable)
The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm
disks.
--
Robert Toole
rtoole(a)tooleweb.ca
403 368 5680
2 months
Announce Push of V6.1
by Frank Filz
Branch next
Tag:V6.1
Merge Highlights
* Coverity fixes
* dramatic config parsing speedup
* imroved LTTNG tracing
* initialize connection manager for metrics
* FSAL_CEPH - Add inode invalidation callback
* fix update prune unmount on pseudo change
* mdcache: Add config param Close_Fast
* Fix vfs_sub_getattrs if called for referral handling
* log_functions: Call gsh_backtrace on Fatal
* nfs_convert.c: add missing NFS4_OP_LAYOUTSTATS
* Setattr access check should succeed if client needs no permission
* nfs4_op_readdir: Fix xdr memory allocation
* Fix error code on exclusive open when file exists
* Add a config option to allow ignoring EPERM when setting PR_SET_IO_FLUSHER
* and more...
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
e13bc5dae Frank S. Filz V6.1
f936de8b3 Sachin Punadikar idmapper : Fix Coverity CIDs 502063, 502122,
502116, 502230
aa5e0f70d Martin Schwenke mdcache: Add config param Close_Fast
1168b3872 Martin Schwenke fsal: Make fsal_start_global_io() easier to
understand
b37df0af0 Martin Schwenke doc: Add missing word "default"
734f2c935 Roopkamal Tyagi Fixing ganesha monitoring C++ code compilation
errors
53b0f7140 Sachin Punadikar FSAL_CEPH - Add inode invalidation callback
7c65985ed Rojin George Unify data type for fsal fd counters
0b2c0ceac Rojin George Fail NFSoRDMA mount for GSS Flavour krb5p
b9c0c26ef Jason Woodward src/support/exports.c: fix update prune unmount on
pseudo change
f7b0c18ca Christoph Martin Fix vfs_sub_getattrs if called for referral
handling
1adeee8e6 izxl007 Fix the incorrect comment about the default value of
LRU_Run_Interval
5e922f399 Assaf Yaari log_functions: Call gsh_backtrace on Fatal
f654ad8d2 Shivam Singh Config Parser: Optimized the parsing of config file
43e01026a Prabhu Murugesan set NULL to sle_block_data after free
7eb1d2c5f Marc Eshel GPFS: Use DS for commit.
698ce0fcb Marc Eshel Add GPFS verifier for write/commit.
0766143fb Jason Woodward nfs_init.c: initialize connection manager for
metrics
d32f821cc Jason Woodward nfs_convert.c: add missing NFS4_OP_LAYOUTSTATS
5a3246609 Shahar Hochma Setattr access check should succeed if client needs
no permission
9145c831b Shahar Hochma nfs4_op_readdir: Fix xdr memory allocation
c4c09e94c Shahar Hochma Fix error code on exclusive open when file exists
767cd2737 Shahar Hochma Add function to LTTNG trace lines
1eb82f72f Shahar Hochma Add a config option to allow ignoring EPERM when
setting PR_SET_IO_FLUSHER
b7da7e281 Shahar Hochma Add LTTNG traces across the system
71392c9f8 Shahar Hochma MDCACHE: Bypass unnecessary entry locking when
attribute cache is disabled
1d5cd4a1f Shahar Hochma Add an option to not use potentially invalid cached
owner in mdcache_test_access
8885d61aa Shahar Hochma Small spelling mistake fix
3bd063837 Frank S. Filz MDCACHE: Fix Coverity CID 509132 - init
active_refcnt
fb713644e Frank S. Filz MDCACHE: Fix Coverity CID 502123 - fix lock
inversion
8aa0a686a Frank S. Filz clientid: fix Coverity CID 502248 - mutex ordering
2fb814592 Frank S. Filz Fix some clang-format issues
0374e3618 Frank S. Filz Sessions: Fix Coverity CID 502092 - check return of
nfs41_Session_Del
bffffd86f Frank S. Filz Connection Manager: Fix Coverity CID 502085 - check
return of sprint_sockip
433800701 Frank S. Filz PSEUDO: Fix Coverity CID 502071 - log
junction_export with lock held
f70f16412 Frank S. Filz reaper: Fox Coverity CID 502101 - use unsigned long
for values
2ea749516 Frank S. Filz FSAL_VFS: Fix Coverity CID 502222 Remove extraneous
retval assignmemt
f7ed09305 Frank S. Filz FSAL_MEM: Fix Coverity CID 502144 and 502168 - use
after free
5 months, 2 weeks
UID and GID Mapping with NFS Ganesha 3.5
by TomK
Hey Folks!
Hope everyone is doing great!
Question. Tried a few sources to solve the UID and GID NFS mapping
ability I would like to implement. Mounting NFS Ganesha using *sec=sys*
works but as soon as I use krb5, krb5i, krb5p etc I get 'permission
denied' on the UID that IS the owner of that folder.
Tried idmapd, as well as this suggestion:
UID_GID_Mapping = NFSv4_ID_Mapping
in FSAL which obviously didn't work. So wondering, what is the proper
way to get UID and GID mapping working with NFS Ganesha and GlusterFS
using *sec=krb5** ? Would love to look at a working example with
sample configurations if possible.
My config:
[root@nfs01 glusterfs]# cat /etc/ganesha/ganesha.conf
/etc/ganesha/export.conf|grep -v "^#"
/etc/ganesha/ganesha.conf
LOG {
Components {
ALL = EVENT;
}
Facility {
name = FILE;
destination = "/var/log/ganesha/ganesha-rgw.log";
enable = active;
}
}
NFSv4 {
Lease_Lifetime = 20 ;
IdmapConf = "/etc/idmapd.conf" ;
DomainName = "nix.mds.xyz" ;
}
NFS_KRB5 {
PrincipalName = "nfs/nfs01.nix.mds.xyz(a)NIX.MDS.XYZ" ;
KeytabPath = /etc/krb5.keytab ;
Active_krb5 = YES ;
}
NFS_Core_Param {
Bind_addr=192.168.0.100;
NFS_Port=2049;
MNT_Port=20048;
NLM_Port=38468;
Rquota_Port=4501;
}
%include "/etc/ganesha/export.conf"
/etc/ganesha/export.conf
EXPORT {
Export_Id = 1 ;
Path = "/n";
FSAL {
name = GLUSTER;
hostname = "nfs01.nix.mds.xyz";
volume = "gv01";
# UID_GID_Mapping = NFSv4_ID_Mapping;
}
Access_type = RW;
Squash = No_root_squash;
Disable_ACL = FALSE;
Pseudo = "/n";
Protocols = "3", "4";
Transports = "UDP", "TCP" ;
SecType = "sys","krb5","krb5i","krb5p"; #
"sys","krb5","krb5i","krb5p";
}
[root@nfs01 glusterfs]#
--
Thx,
TK.
5 months, 4 weeks