On Mon, 2019-03-04 at 17:26 +0000, David C wrote:
> Looks like you're right, Jeff. Just tried to write into the dir and am
> now getting the quota warning. So I guess it was the libcephfs cache
> as you say. That's fine for me, I don't need the quotas to be too
> strict, just a failsafe really.
>
Actually, I said it was likely the NFS client cache. The Linux kernel is
allowed to aggressively cache writes if you're doing buffered I/O. The
NFS client has no concept of the quota here, so you'd only see
enforcement once those writes start getting flushed back to the server.
Ah sorry, that makes a lot of sense!
> Interestingly, if I create a new dir, set the same 100MB quota, I can
> write multiple files with "dd if=/dev/zero of=1G bs=1M count=1024
> oflag=direct". Wouldn't that bypass the cache? I have the following in
> my ganesha.conf which I believe effectively disables Ganesha's
> caching:
>
> CACHEINODE {
> Dir_Chunk = 0;
> NParts = 1;
> Cache_Size = 1;
> }
>
Using direct I/O like that should take the NFS client cache out of the
picture. That said, cephfs quota enforcement is pretty "lazy". According
to http://docs.ceph.com/docs/mimic/cephfs/quota/ :
"Quotas are imprecise. Processes that are writing to the file system
will be stopped a short time after the quota limit is reached. They will
inevitably be allowed to write some amount of data over the configured
limit. How far over the quota they are able to go depends primarily on
the amount of time, not the amount of data. Generally speaking writers
will be stopped within 10s of seconds of crossing the configured limit."
You can write quite a bit of data in 10s of seconds (multiple GBs is not
unreasonable here).
> On Mon, Mar 4, 2019 at 2:50 PM Jeff Layton <jlayton@redhat.com> wrote:
> > > > On Mon, 2019-03-04 at 09:11 -0500, Jeff Layton wrote:
> This list has
> > been deprecated. Please subscribe to the new devel list at
> > lists.nfs-ganesha.org.
> On Fri, 2019-03-01 at 15:49 +0000, David C
> > wrote:
> > This list has been deprecated. Please subscribe to the new
> > devel list at lists.nfs-ganesha.org.
> > Hi All
> >
> > Exporting
> > cephfs with the CEPH_FSAL
> >
> > I set the following on a dir:
> >
>
> > > setfattr -n ceph.quota.max_bytes -v 100000000 /dir
> > setfattr -n
> > ceph.quota.max_files -v 10 /dir
> >
> > From an NFSv4 client, the
> > quota.max_bytes appears to be completely ignored, I can go GBs over
> > the quota in the dir. The quota.max_files DOES work however, if I
> > try and create more than 10 files, I'll get "Error opening file
> > 'dir/new file': Disk quota exceeded" as expected.
> >
> > From a
> > fuse-mount on the same server that is running nfs-ganesha, I've
> > confirmed ceph.quota.max_bytes is enforcing the quota, I'm unable to
> > copy more than 100MB into the dir.
> >
> > According to [1] and [2]
> > this should work.
> >
> > Cluster is Luminous 12.2.10
> >
> > Package
> > versions on nfs-ganesha server:
> >
> > nfs-ganesha-rados-grace-
> > 2.7.1-0.1.el7.x86_64
> > nfs-ganesha-2.7.1-0.1.el7.x86_64
> > nfs-
> > ganesha-vfs-2.7.1-0.1.el7.x86_64
> > nfs-ganesha-ceph-2.7.1-
> > 0.1.el7.x86_64
> > libcephfs2-13.2.2-0.el7.x86_64
> > ceph-fuse-
> > 12.2.10-0.el7.x86_64
> >
> > My Ganesha export:
> >
> > EXPORT
> > {
>
> > > Export_ID=100;
> > Protocols = 4;
> > Transports = TCP;
>
> > > Path = /;
> > Pseudo = /ceph/;
> > Access_Type = RW;
> >
> > Attr_Expiration_Time = 0;
> > #Manage_Gids = TRUE;
> >
> > Filesystem_Id = 100.1;
> > FSAL {
> > Name = CEPH;
> >
> > }
> > }
> >
> > My ceph.conf client section:
> >
> > [client]
> >
> > mon host = 10.10.10.210:6789, 10.10.10.211:6789,
> > 10.10.10.212:6789
> > client_oc_size = 8388608000
> >
> > #fuse_default_permission=0
> > client_acl_type=posix_acl
> >
> > client_quota = true
> > client_quota_df = true
> >
> >
> > Related links:
> >
> > [1] http://tracker.ceph.com/issues/16526
> > [2] https://github.com/nfs-ganesha/nfs-ganesha/issues/100
> >
> > Thanks
> > David
> >
>
> It looks like you're having ganesha do the mount as "client.admin", and
> I suspect that that may allow you to bypass quotas? You may want to try
> creating a cephx user with less privileges, have ganesha connect as that
> user and see if it changes things?
>
Actually, this may be wrong info.
How are you testing being able to write to the file past quota? Are you
using O_DIRECT I/O? If not, then it may just be that you're seeing the
effect of the NFS client caching writes.
--
Jeff Layton <jlayton@redhat.com>