Re: [Nfs-ganesha-devel] NFS-Ganesha CEPH_FSAL ceph.quota.max_bytes not enforced
On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote:
This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. On Fri, 2019-03-01 at 15:49 +0000, David C wrote:
This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. Hi All
Exporting cephfs with the CEPH_FSAL
I set the following on a dir:
setfattr -n ceph.quota.max_bytes -v 100000000 /dir setfattr -n ceph.quota.max_files -v 10 /dir
From an NFSv4 client, the quota.max_bytes appears to be completely ignored, I can go GBs over the quota in the dir. The quota.max_files DOES work however, if I try and create more than 10 files, I'll get "Error opening file 'dir/new file': Disk quota exceeded" as expected.
From a fuse-mount on the same server that is running nfs-ganesha, I've confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to copy more than 100MB into the dir.
According to [1] and [2] this should work.
Cluster is Luminous 12.2.10
Package versions on nfs-ganesha server:
nfs-ganesha-rados-grace-2.7.1-0.1.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64 libcephfs2-13.2.2-0.el7.x86_64 ceph-fuse-12.2.10-0.el7.x86_64
My Ganesha export:
EXPORT { Export_ID=100; Protocols = 4; Transports = TCP; Path = /; Pseudo = /ceph/; Access_Type = RW; Attr_Expiration_Time = 0; #Manage_Gids = TRUE; Filesystem_Id = 100.1; FSAL { Name = CEPH; } }
My ceph.conf client section:
[client] mon host = 10.10.10.210:6789, 10.10.10.211:6789, 10.10.10.212:6789 client_oc_size = 8388608000 #fuse_default_permission=0 client_acl_type=posix_acl client_quota = true client_quota_df = true
Related links:
[1] http://tracker.ceph.com/issues/16526 [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
Thanks David
It looks like you're having ganesha do the mount as "client.admin", and I suspect that that may allow you to bypass quotas? You may want to try creating a cephx user with less privileges, have ganesha connect as that user and see if it changes things?
Actually, this may be wrong info. How are you testing being able to write to the file past quota? Are you using O_DIRECT I/O? If not, then it may just be that you're seeing the effect of the NFS client caching writes. -- Jeff Layton <jlayton@redhat.com>
Looks like you're right, Jeff. Just tried to write into the dir and am now getting the quota warning. So I guess it was the libcephfs cache as you say. That's fine for me, I don't need the quotas to be too strict, just a failsafe really. Interestingly, if I create a new dir, set the same 100MB quota, I can write multiple files with "dd if=/dev/zero of=1G bs=1M count=1024 oflag=direct". Wouldn't that bypass the cache? I have the following in my ganesha.conf which I believe effectively disables Ganesha's caching: CACHEINODE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; } Thanks, On Mon, Mar 4, 2019 at 2:50 PM Jeff Layton <jlayton@redhat.com> wrote:
This list has been deprecated. Please subscribe to the new devel list at
On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote: lists.nfs-ganesha.org.
On Fri, 2019-03-01 at 15:49 +0000, David C wrote:
This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. Hi All
Exporting cephfs with the CEPH_FSAL
I set the following on a dir:
setfattr -n ceph.quota.max_bytes -v 100000000 /dir setfattr -n ceph.quota.max_files -v 10 /dir
From an NFSv4 client, the quota.max_bytes appears to be completely ignored, I can go GBs over the quota in the dir. The quota.max_files DOES work however, if I try and create more than 10 files, I'll get "Error opening file 'dir/new file': Disk quota exceeded" as expected.
From a fuse-mount on the same server that is running nfs-ganesha, I've confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to copy more than 100MB into the dir.
According to [1] and [2] this should work.
Cluster is Luminous 12.2.10
Package versions on nfs-ganesha server:
nfs-ganesha-rados-grace-2.7.1-0.1.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs-ganesha-vfs-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64 libcephfs2-13.2.2-0.el7.x86_64 ceph-fuse-12.2.10-0.el7.x86_64
My Ganesha export:
EXPORT { Export_ID=100; Protocols = 4; Transports = TCP; Path = /; Pseudo = /ceph/; Access_Type = RW; Attr_Expiration_Time = 0; #Manage_Gids = TRUE; Filesystem_Id = 100.1; FSAL { Name = CEPH; } }
My ceph.conf client section:
[client] mon host = 10.10.10.210:6789, 10.10.10.211:6789, 10.10.10.212:6789 client_oc_size = 8388608000 #fuse_default_permission=0 client_acl_type=posix_acl client_quota = true client_quota_df = true
Related links:
[1] http://tracker.ceph.com/issues/16526 [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
Thanks David
It looks like you're having ganesha do the mount as "client.admin", and I suspect that that may allow you to bypass quotas? You may want to try creating a cephx user with less privileges, have ganesha connect as that user and see if it changes things?
Actually, this may be wrong info.
How are you testing being able to write to the file past quota? Are you using O_DIRECT I/O? If not, then it may just be that you're seeing the effect of the NFS client caching writes. -- Jeff Layton <jlayton@redhat.com>
On Mon, 2019-03-04 at 17:26 +0000, David C wrote:
Looks like you're right, Jeff. Just tried to write into the dir and am now getting the quota warning. So I guess it was the libcephfs cache as you say. That's fine for me, I don't need the quotas to be too strict, just a failsafe really.
Actually, I said it was likely the NFS client cache. The Linux kernel is allowed to aggressively cache writes if you're doing buffered I/O. The NFS client has no concept of the quota here, so you'd only see enforcement once those writes start getting flushed back to the server.
Interestingly, if I create a new dir, set the same 100MB quota, I can write multiple files with "dd if=/dev/zero of=1G bs=1M count=1024 oflag=direct". Wouldn't that bypass the cache? I have the following in my ganesha.conf which I believe effectively disables Ganesha's caching:
CACHEINODE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; }
Using direct I/O like that should take the NFS client cache out of the picture. That said, cephfs quota enforcement is pretty "lazy". According to http://docs.ceph.com/docs/mimic/cephfs/quota/ : "Quotas are imprecise. Processes that are writing to the file system will be stopped a short time after the quota limit is reached. They will inevitably be allowed to write some amount of data over the configured limit. How far over the quota they are able to go depends primarily on the amount of time, not the amount of data. Generally speaking writers will be stopped within 10s of seconds of crossing the configured limit." You can write quite a bit of data in 10s of seconds (multiple GBs is not unreasonable here).
On Mon, Mar 4, 2019 at 2:50 PM Jeff Layton <jlayton@redhat.com> wrote:
On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote: This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. On Fri, 2019-03-01 at 15:49 +0000, David C wrote: This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. Hi All
Exporting cephfs with the CEPH_FSAL
I set the following on a dir:
setfattr -n ceph.quota.max_bytes -v 100000000 /dir setfattr -n ceph.quota.max_files -v 10 /dir
From an NFSv4 client, the quota.max_bytes appears to be completely ignored, I can go GBs over the quota in the dir. The quota.max_files DOES work however, if I try and create more than 10 files, I'll get "Error opening file 'dir/new file': Disk quota exceeded" as expected.
From a fuse-mount on the same server that is running nfs-ganesha, I've confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to copy more than 100MB into the dir.
According to [1] and [2] this should work.
Cluster is Luminous 12.2.10
Package versions on nfs-ganesha server:
nfs-ganesha-rados-grace- 2.7.1-0.1.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs- ganesha-vfs-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1- 0.1.el7.x86_64 libcephfs2-13.2.2-0.el7.x86_64 ceph-fuse- 12.2.10-0.el7.x86_64
My Ganesha export:
EXPORT {
Export_ID=100;
Protocols = 4; Transports = TCP;
Path = /;
Pseudo = /ceph/; Access_Type = RW;
Attr_Expiration_Time = 0; #Manage_Gids = TRUE;
Filesystem_Id = 100.1; FSAL { Name = CEPH;
} }
My ceph.conf client section:
[client]
mon host = 10.10.10.210:6789, 10.10.10.211:6789, 10.10.10.212:6789 client_oc_size = 8388608000
#fuse_default_permission=0 client_acl_type=posix_acl
client_quota = true client_quota_df = true
Related links:
[1] http://tracker.ceph.com/issues/16526 [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
Thanks David
It looks like you're having ganesha do the mount as "client.admin", and I suspect that that may allow you to bypass quotas? You may want to try creating a cephx user with less privileges, have ganesha connect as that user and see if it changes things?
Actually, this may be wrong info. How are you testing being able to write to the file past quota? Are you using O_DIRECT I/O? If not, then it may just be that you're seeing the effect of the NFS client caching writes. -- Jeff Layton <jlayton@redhat.com>
On Mon, Mar 4, 2019 at 5:53 PM Jeff Layton <jlayton@redhat.com> wrote:
On Mon, 2019-03-04 at 17:26 +0000, David C wrote:
Looks like you're right, Jeff. Just tried to write into the dir and am now getting the quota warning. So I guess it was the libcephfs cache as you say. That's fine for me, I don't need the quotas to be too strict, just a failsafe really.
Actually, I said it was likely the NFS client cache. The Linux kernel is allowed to aggressively cache writes if you're doing buffered I/O. The NFS client has no concept of the quota here, so you'd only see enforcement once those writes start getting flushed back to the server.
Ah sorry, that makes a lot of sense!
Interestingly, if I create a new dir, set the same 100MB quota, I can write multiple files with "dd if=/dev/zero of=1G bs=1M count=1024 oflag=direct". Wouldn't that bypass the cache? I have the following in my ganesha.conf which I believe effectively disables Ganesha's caching:
CACHEINODE { Dir_Chunk = 0; NParts = 1; Cache_Size = 1; }
Using direct I/O like that should take the NFS client cache out of the picture. That said, cephfs quota enforcement is pretty "lazy". According to http://docs.ceph.com/docs/mimic/cephfs/quota/ :
"Quotas are imprecise. Processes that are writing to the file system will be stopped a short time after the quota limit is reached. They will inevitably be allowed to write some amount of data over the configured limit. How far over the quota they are able to go depends primarily on the amount of time, not the amount of data. Generally speaking writers will be stopped within 10s of seconds of crossing the configured limit."
You can write quite a bit of data in 10s of seconds (multiple GBs is not unreasonable here).
On Mon, Mar 4, 2019 at 2:50 PM Jeff Layton <jlayton@redhat.com> wrote:
On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote: This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. On Fri, 2019-03-01 at 15:49 +0000, David C wrote: This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. Hi All
Exporting cephfs with the CEPH_FSAL
I set the following on a dir:
setfattr -n ceph.quota.max_bytes -v 100000000 /dir setfattr -n ceph.quota.max_files -v 10 /dir
From an NFSv4 client, the quota.max_bytes appears to be completely ignored, I can go GBs over the quota in the dir. The quota.max_files DOES work however, if I try and create more than 10 files, I'll get "Error opening file 'dir/new file': Disk quota exceeded" as expected.
From a fuse-mount on the same server that is running nfs-ganesha, I've confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to copy more than 100MB into the dir.
According to [1] and [2] this should work.
Cluster is Luminous 12.2.10
Package versions on nfs-ganesha server:
nfs-ganesha-rados-grace- 2.7.1-0.1.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs- ganesha-vfs-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1- 0.1.el7.x86_64 libcephfs2-13.2.2-0.el7.x86_64 ceph-fuse- 12.2.10-0.el7.x86_64
My Ganesha export:
EXPORT {
Export_ID=100;
Protocols = 4; Transports = TCP;
Path = /;
Pseudo = /ceph/; Access_Type = RW;
Attr_Expiration_Time = 0; #Manage_Gids = TRUE;
Filesystem_Id = 100.1; FSAL { Name = CEPH;
} }
My ceph.conf client section:
[client]
mon host = 10.10.10.210:6789, 10.10.10.211:6789, 10.10.10.212:6789 client_oc_size = 8388608000
#fuse_default_permission=0 client_acl_type=posix_acl
client_quota = true client_quota_df = true
Related links:
[1] http://tracker.ceph.com/issues/16526 [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
Thanks David
It looks like you're having ganesha do the mount as "client.admin", and I suspect that that may allow you to bypass quotas? You may want to try creating a cephx user with less privileges, have ganesha connect as that user and see if it changes things?
Actually, this may be wrong info.
How are you testing being able to write to the file past quota? Are you using O_DIRECT I/O? If not, then it may just be that you're seeing the effect of the NFS client caching writes.
-- Jeff Layton <jlayton@redhat.com>
participants (2)
-
David C -
Jeff Layton