ESXI 6.7 client creating Thick Eager zeroed vmdk files using ceph fsal
by Robert Toole
Hi,
I have a 3 node Ceph octopus 15.2.7 cluster running on fully up to date
Centos 7 with nfs-ganesha 3.5.
After following the Ceph install guide
https://docs.ceph.com/en/octopus/cephadm/install/#deploying-nfs-ganesha
I am able to create a NFS 4.1 Datastore in vmware using the ip address
of all three nodes. Everything appears to work OK..
The issue however is that for some reason esxi is creating thick
provisioned eager zeroed disks instead of thin provisioned disks on this
datastore, whether I am migrating, cloning, or creating new vms. Even
running vmkfstools -i disk.vmdk -d thin thin_disk.vmdk still results in
a thick eager zeroed vmdk file.
This should not be possible on an NFS datastore, because vmware requires
a VAAI NAS plugin to accomplish thick provisioning over NFS before it
can thick provision disks.
Linux clients to the same datastore can create thin qcow2 images, and
when looking at the images created by esxi from the linux hosts you can
see that the vmdks are indeed thick:
ls -lsh
total 81G
512 -rw-r--r--. 1 root root 230 Mar 25 15:17 test_vm-2221e939.hlog
40G -rw-------. 1 root root 40G Mar 25 15:17 test_vm-flat.vmdk
40G -rw-------. 1 root root 40G Mar 25 15:56 test_vm_thin-flat.vmdk
512 -rw-------. 1 root root 501 Mar 25 15:57 test_vm_thin.vmdk
512 -rw-------. 1 root root 473 Mar 25 15:17 test_vm.vmdk
0 -rw-r--r--. 1 root root 0 Jan 6 1970 test_vm.vmsd
2.0K -rwxr-xr-x. 1 root root 2.0K Mar 25 15:17 test_vm.vmx
but the qcow2 files from the linux hosts are thin as one would expect:
qemu-img create -f qcow2 big_disk_2.img 500G
ls -lsh
total 401K
200K -rw-r--r--. 1 root root 200K Mar 25 15:47 big_disk_2.img
200K -rw-r--r--. 1 root root 200K Mar 25 15:44 big_disk.img
512 drwxr-xr-x. 2 root root 81G Mar 25 15:57 test_vm
These ls -lsh results are the same from esx, linux nfs clients and from
cephfs kernel client.
What is happening here? Are there undocumented VAAI features in
nfs-ganesha with the cephfs fsal ? If so, how do I turn them off ? I
want thin provisioned disks.
ceph nfs export ls dev-nfs-cluster --detailed
[
{
"export_id": 1,
"path": "/Development-Datastore",
"cluster_id": "dev-nfs-cluster",
"pseudo": "/Development-Datastore",
"access_type": "RW",
"squash": "no_root_squash",
"security_label": true,
"protocols": [
4
],
"transports": [
"TCP"
],
"fsal": {
"name": "CEPH",
"user_id": "dev-nfs-cluster1",
"fs_name": "dev_cephfs_vol",
"sec_label_xattr": ""
},
"clients": []
}
]
rpm -qa | grep ganesha
nfs-ganesha-ceph-3.5-1.el7.x86_64
nfs-ganesha-rados-grace-3.5-1.el7.x86_64
nfs-ganesha-rados-urls-3.5-1.el7.x86_64
nfs-ganesha-3.5-1.el7.x86_64
centos-release-nfs-ganesha30-1.0-2.el7.centos.noarch
rpm -qa | grep ceph
python3-cephfs-15.2.7-0.el7.x86_64
nfs-ganesha-ceph-3.5-1.el7.x86_64
python3-ceph-argparse-15.2.7-0.el7.x86_64
python3-ceph-common-15.2.7-0.el7.x86_64
cephadm-15.2.7-0.el7.x86_64
libcephfs2-15.2.7-0.el7.x86_64
ceph-common-15.2.7-0.el7.x86_64
ceph -v
ceph version 15.2.7 (<ceph_uuid>) octopus (stable)
The ceph cluster is healthy using bluestore on raw 3.84TB sata 7200 rpm
disks.
--
Robert Toole
rtoole(a)tooleweb.ca
403 368 5680
2 months
Announce Push of V5.9
by Frank Filz
Branch next
Tag:V5.9
Sorry, this patch should not have been back ported.
Merge Highlights
* Revert "Require librgw 1.2.1"
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
7b6f5b543 Frank S. Filz V5.9
8aae04dca Frank S. Filz Revert "Require librgw 1.2.1"
9 months, 3 weeks
Announce Push of V5.8
by Frank Filz
Branch: V%-stable
Tag:V5.8
NOTE: This release includes an ntirpc pullup, please update your submodule.
Collected bug fixes from V5.7 through V6-dev.13
Merge Highlights
* Fixes to POSIX ACL handling
* Introduction of SaunaFS FSAL
* Memory leak fixes
* Delegation fixes
* ntirpc V5.8 fixes
* fd_lru and other fsal_fd related fixes
* FSAL_PSEUDO set change, mtime, ctime on create directory
* Some logging changes
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
96adc44d9 Frank S. Filz V5.8
7bb7e8ef7 JAGNYA DATTA TRIPATHY rtpref and wtpref values should not exceed
rtmax and wtmax in fsinfo
e8a5a2bd0 Deepak Arumugam Sankara Subramanian Release refcount on entry
before restarting readdir
505e0ab8a Rojin George Handle parallel delegation recalls for the same
fsal_objs
f305617fe Rubén Alcolea Núñez Add SaunaFS FSAL
d93544d8a Daniel Gryniewicz Require librgw 1.2.1
4cfd08de6 Dhairya Parmar doc: fix cmount_path doc
30de61d57 Sachin Punadikar FSAL_CEPH: Avoid duplicate refcounting of cmount
b96b0b2e6 Frank S. Filz FSAL: Fix up gsh_refstr references for op_context.
24df89347 Frank S. Filz nfs4_Check_Stateid must not return
NFS4ERR_STALE_STATEID for 4.1
3e926c9cb Frank S. Filz export_mgr: Dont' leak refstr in
get_gsh_export_by_pseudo_locked
2efe050cd Frank S. Filz CEPH: Free EXPORT { FSAL {} } config strings
4b720e15b Assaf Yaari commonlib: Skip redundant aces in mode_to_acl
146d6616b Assaf Yaari commonlib: In fsal_mode_to_acl identify if can reuse
aces
ddea6c571 Assaf Yaari commonlib: Place OWNER & GROUP mode generated aces at
the beginning
1e67914ef Assaf Yaari commonlib: In fsal_mode_gen_acl the group perm should
serve as mask
2689c727b Assaf Yaari access_check: In fsal_check_access_acl reset the
FSAL_ACE4_REQ_FLAG from missing_access
d67ca51b6 Assaf Yaari commonlib: In fsal_mode_to_acl don't dup inherit aces
6f74355c3 Assaf Yaari commonlib: Identify correctly special id's
072ebac56 Assaf Yaari commonlib: Remove GROUP_ID flag from special "who"
1123cc85a Assaf Yaari commonlib: Add default flags for generated aces from
mode-bits
9ae4b8c5e Assaf Yaari nfs_proto_tools: In encode_acl convert mode to acl if
no acl present
f197b9492 Assaf Yaari nfs4_acls: Free the acl object even if not found in
the hashtable
aedcea44c Assaf Yaari commonlib: In fsal_mode_gen_acl create properly the
generates acl object
f718c393f Assaf Yaari commonlib: Create properly ACL new entry when
converting mode to ACL
a3e968728 Assaf Yaari commonlib: Re-write the implementation of
fsal_acl_to_mode
b54157111 Assaf Yaari fsal_mode_to_acl: Advance the dace pointer only when
copying the source ace
400d45d13 Assaf Yaari fsal.h: Don't unset the ATTR_ACL from valid_mask in
case it was set by src
ed807cdfa Assaf Yaari nfs4_op_putfh: Update logs to info level on case of
errors
113530b9b Assaf Yaari nfs4_op_create_session: Upgrade some debug logs to
info
7b369850e Assaf Yaari log_functions: Don't call Cleanup from Fatal handling
function
d688de58f Daniel Gryniewicz GTest - fix tests for new FSAL APIs
c628f07d9 Yogendra Charya Tangalapally GPFS: FSAL's state_free function
called by free_state doesn't actually free
6376199fb Martin Schwenke doc: Update default for Readdir_Res_Size to 32KB
bf6229e47 Sachin Punadikar FSAL_CEPH : Add warnings for export creation
cb4fba122 Frank S. Filz fsal_fd fd_work/io_work needs separate condition
variables
524ae1ff2 Frank S. Filz Add PTHREAD_COND_broadcast macro
4080d5dd7 Rojin George Free delegation recall context after cleaning expired
clients
43134126e Rojin George Decrement Client Ref, if reserve lease or expire
fails
46865aea5 Rojin George Handle parallel delegation recalls for the same
fsal_objs
c4e201cda Rojin George Insert global FDs of reopened files into fd lru
528cd3127 Rojin George Provide FSALs option to remove global FDs from fd_lru
upon failures
5c443abb6 Rojin George Expose bump_fd_lru to FSALs to bump up if reusing
global FDs
9db158630 Rojin George Insert to fd_lru only if open/reopen succeeds
36f4003d4 Kaleb S. KEITHLEY threads: set appropriate thread stack size for
fridge and ntirpc workers
2eaeed5f6 Deepak Arumugam Sankara Subramanian Reduce the default value of
Readdir_Res_Size to 32KB
a8dd100a1 Deepak Arumugam Sankara Subramanian Use after free in fd_lru
6ab8377b1 Jon Kohler use may_write == false in nfs4_State_Get_Pointer
a425b8877 Rojin George Use __GLIBC__ macro to set pthread RW lock kind
310d2dfcb Rojin George Fix log_an_export critical path overhead
3389cf1b7 Frank S. Filz pullup ntirpc v5.8
f11603e6a Frank S. Filz Fix memory leak in DRC
3f8475e77 Frank S. Filz GPFS: Fix unexport of 1st export on filesystem
39dd30960 Frank S. Filz FSAL: add function to get an export ref from a
filesystem.
78d589d36 Frank S. Filz Don't leak memory when PSEUDOFS block is not
specified.
47c5b4da0 Frank S. Filz Don't leak memory for FSAL_LIST config block
af60e7943 Frank S. Filz CONFIG: Add ability to specify a block where memory
is not allocated if unused
1e842bce0 sunpxyz bugfix:
https://github.com/nfs-ganesha/nfs-ganesha/issues/1044
1c32059c1 Xueqian Hu Enable libunwind in ganesha
2ee8c33ba Rojin George Handle delegation recalls for expired clients
2d2e217b2 Rojin George Grant Read delegations properly
d9f49a223 Rojin George Instrumentation changes to debug future lru_try_one
crash
6d640a75d Rojin George Clear delegation stats if no active states
8656b8eb4 Rojin George Fix few typos around the fd_lru logics
e32917e6c Rojin George Decrement fd_work while reclaiming if FD is busy
c3e1f8389 bjfhdhhaha ACL: fix issue of setting posix acl for directory
return EINVAL
a15e5aaec bjfhdhhaha ACL: fix memory leak
cd8a2d06a Prabhu Murugesan Fix crash at release_lock_owner()->
_get_gsh_export_ref()
9d719e666 Jason Woodward FSAL_PSEUDO set change, mtime, ctime on create
directory
fbb7ff4c9 Kaleb S. KEITHLEY monitoring: fails to build with gcc-14
d34f62ea9 Sachin Punadikar FSAL_CEPH : Add debug statements
d0e756adb David Rieber Move the log levels and components to a separate
headfer file.
1fdb4face David Rieber NFSv4 READDIR should pass down the attrs from the
request.
0b81b57ad Sachin Punadikar Fix compiler warnings for "-Werror" flag
ea155b64b Sachin Punadikar LOG : Parse correct name for
COMPONENT_MDCACHE_LRU
32fc60b01 Daniel Gryniewicz Allow specifying python version
8315e6df9 Daniel Gryniewicz Fix monitoring build option
cef733cb8 Daniel Gryniewicz Ignore clangd cache directory
0a752ade9 Frank S. Filz FSAL: Fix up how reaper_work is used
51d16a390 Frank S. Filz FSAL: in open2 methods, update_share_counters_locked
must use *new_obj
849e6547f Frank S. Filz LOG: Fix buffer overflow issue
e8293aa47 Kaleb S. KEITHLEY build: use standard (i.e. pep517) build and
installer
514763104 Prabhu Murugesan Fix crash at
free_client_id()->PTHREAD_MUTEX_destroy()
9b353349f zhitaoli Fix: dec ref_count of confirmed client_id
c87fa5f30 David Rieber Fix use-of-uninitialized value error.
848ee3320 David Rieber malloc_trim is unsupported on darwin.
6822ab8c2 David Rieber Initialize export_opt_lock earlier.
bb3fe4fbe Trupti Shete Resolving error in ganesha-top utility
247e45d15 Trupti Shete Issue in ganesha_stats json option part_5
ea0ccbb73 Trupti Shete Issue in ganesha_stats json option part_4
d0ace9456 Trupti Shete Issue in ganesha_stats json option part_3
bad643006 Trupti Shete Issue in ganesha_stats json option part_2
7e55566e3 Trupti Shete ganesha_stats json issues part_1
a9a42e780 Frank S. Filz FSAL's state_free function called by free_state
doesn't actually free
9 months, 4 weeks
lock issue for same clientid on the same node using NFS Ganesha 3.5
by pragash2here＠gmail.com
We are using NFS Ganesha 3.5 and occassionaly we see the following messages on the client host. The applications are no longer able to take locks.
They are all taking RDLCKs over the entire file range.
fcntl(19, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=1800})
The host has multiple mounts of the SAME exported filesystem.
All the mounts are identical except for the mount path.
Each mount is sending the the SAME SETCLIENTID to the NFS Ganesha.
Any pointers much appreciated.
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_10] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000
002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_10] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6}
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_14] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000
002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_14] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6}
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_17] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1310 refcount=7} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000
002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}
10 months
Enable POSIX ACL support in VFS FSAL?
by Sean Caron
Hello!
I noticed in the release notes that in V4.1 and onward, POSIX ACL
support is now available in the VFS FSAL but is disabled by default.
I looked through the documentation but I don't see any notes about the
parameter that might be used to enable said support.
Could someone please let me know how one might enable POSIX ACL
support in the VFS FSAL?
We're running the latest available stable build from PPA, 5.7GA.
Thanks!
Sean
Sean Caron (she/they)
OS Programmer Lead
University of Michigan Center for Statistical Genetics
Department of Biostatistics, School of Public Health
e: scaron(a)umich.edu p: 734-474-5962
10 months, 1 week
