11:36 p.m.
Hi,
We'e setting up ganesha using FSAL_VFS, and NFSv3 only. We have a cluster of them
running with floating IPs being failed over between them when a host dies.
The question we have is regarding writes which are UNSTABLE - meaning the client writes
out from its cache for example with UNSTABLE, and then sends a COMMIT. In the event of a
ganesha host crashing after receiving the writeouts, but before the COMMIT - is it assured
that when the floating IP will failover to another host the client will resend the
unstable writes again as well?
I guess this has several parts to it:
* Client side - assuming it's a linux kernel nfs client and it uses the pagecache -
does it mark the pages clean right after successfully sending their writes as unstable, or
does it mark them as clean only after it successfully sends a COMMIT on them as well? Do
you happen to know?
* Server side - Does nfs-ganesha have some data cache? Or the writes being unstable just
means they can land in the pagecache of the server side, and stay there until the COMMIT
does a sync? in that case do we need to set `NFS_Commit = true`?
What are the guidelines on such a setup to ensure data doesn't get lost and
doesn't corrupt?
Thank you,
Alex
7:36 a.m.
The biggest guideline to make sure you don't get corruption is to not
use unstable writes. I realize this is a performance hit, but it's the
only way to guarantee that every ack'd write is committed to stable
storage. (Obviously, if a crash occurs mid-write, all bets are off).
Ganesha does not currently have any kind if data cache. However,
FSAL_VFS uses the local pagecache. A stable write and a commit will
each cause an fsync(), so the result will be committed to disk. An
unstable write, however, will just end up in the pagecache, and so will
likely not be stable (depending, of course, on how busy the server is).
Looking at the kernel client, it looks like it sends unstable writes if
there are more writes queued to the file, or if a commit is already
scheduled. Otherwise, it sends stable writes. Beyond that, we're
outside of my knowledge of the client. I assume it will commit or
perform a stable write periodically, even if writes are constant?
Note that all this is unrelated to the sync/async mount flags; those
determine how writes affect the client pagecache, not the server at all.
Daniel
On 3/12/20 2:36 AM, alxg(a)weka.io wrote:
Hi,
We'e setting up ganesha using FSAL_VFS, and NFSv3 only. We have a cluster of them
running with floating IPs being failed over between them when a host dies.
The question we have is regarding writes which are UNSTABLE - meaning the client writes
out from its cache for example with UNSTABLE, and then sends a COMMIT. In the event of a
ganesha host crashing after receiving the writeouts, but before the COMMIT - is it assured
that when the floating IP will failover to another host the client will resend the
unstable writes again as well?
I guess this has several parts to it:
* Client side - assuming it's a linux kernel nfs client and it uses the pagecache -
does it mark the pages clean right after successfully sending their writes as unstable, or
does it mark them as clean only after it successfully sends a COMMIT on them as well? Do
you happen to know?
* Server side - Does nfs-ganesha have some data cache? Or the writes being unstable just
means they can land in the pagecache of the server side, and stay there until the COMMIT
does a sync? in that case do we need to set `NFS_Commit = true`?
What are the guidelines on such a setup to ensure data doesn't get lost and
doesn't corrupt?
Thank you,
Alex
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
6:14 a.m.
New subject: nfs-ganesha in a cluster with IP failover
Hey Daniel,
So you're saying that clients which are working against a cluster of hosts with
floating IPs should not send unstable writes? In theory it should be possible to make them
work with unstable writes no?
Do all the FSALs of clustered FSs always sync the writes regardless of them being
requested as unstable?
And as long as we make sure that all writes to pagecache on the server side are synced
(i.e. the underlying mount for the export for our weka fs is with "-o sync") -
everything should be coherent?
So what is the `NFS_Commit` flag is for? in 2.8.3 I think it actually is unused - is it
not?
Alex
7:01 a.m.
New subject: nfs-ganesha in a cluster with IP failover
On 3/18/20 9:14 AM, alxg(a)weka.io wrote:
Hey Daniel,
So you're saying that clients which are working against a cluster of hosts with
floating IPs should not send unstable writes? In theory it should be possible to make them
work with unstable writes no?
If everything operates correctly, it should be okay to use unstable
writes with commits, even in a cluster, assuming the cluster has proper
Grace handling.
Do all the FSALs of clustered FSs always sync the writes regardless
of them being requested as unstable?
All FSALs, as far as I know, honor the write request flags.
And as long as we make sure that all writes to pagecache on the
server side are synced (i.e. the underlying mount for the export for our weka fs is with
"-o sync") - everything should be coherent?
Yes, for FSALs that use the pagecache.
So what is the `NFS_Commit` flag is for? in 2.8.3 I think it actually is unused - is it
not?
This was added for VMWare clients, to work around an odd performance
issue they had. It forced a sync on every write. It was lost in 2.7
with the transition to async read/write.
Daniel
9:25 a.m.
New subject: nfs-ganesha in a cluster with IP failover
If everything operates correctly, it should be okay to use unstable
writes with commits, even in a cluster, assuming the cluster has proper
Grace handling.
So what you mentioned before about using only stable writes to avoid corruption - did you
mean this only in the sense of persistency? But IMHO the user should anyways be aware of
using `fsync` to make sure data gets to the disk. I was mainly concerned about the
interoperability of unstable writes and floating IPs failovers - but you're saying
that most FSALs do allow unstable writes to some kind of RAM only cache (e.g. pagecache or
some userspace lib) on the ganesha host, and syncing them to the actual disks only on
`commit`, and that works with floating IPs, and the client knows to correctly mark
pagecache on its side as not dirty anymore only after a commit - right?
10:38 a.m.
New subject: nfs-ganesha in a cluster with IP failover
On 3/18/20 12:25 PM, alxg(a)weka.io wrote:
> If everything operates correctly, it should be okay to use
unstable
> writes with commits, even in a cluster, assuming the cluster has proper
> Grace handling.
So what you mentioned before about using only stable writes to avoid corruption - did you
mean this only in the sense of persistency?
I was actually thinking of client side (sync vs. async mounts).
But IMHO the user should anyways be aware of using `fsync` to make
sure data gets to the disk. I was mainly concerned about the interoperability of unstable
writes and floating IPs failovers - but you're saying that most FSALs do allow
unstable writes to some kind of RAM only cache (e.g. pagecache or some userspace lib) on
the ganesha host, and syncing them to the actual disks only on `commit`, and that works
with floating IPs, and the client knows to correctly mark pagecache on its side as not
dirty anymore only after a commit - right?
You misunderstand, or I'm not clear. Ganesha does not have any data
caching, in RAM or otherwise. Various FSALs can/do have various forms
of caching built into their filesystems. One example is the kernel
pagecache used by FSAL_VFS. libcephfs, used by FSAL_CEPH, also has data
caching, both on the client and on the server. Unstable writes use
these caches, whether they be on the Ganesha server, or elsewhere in the
cluster, whether they be in RAM or on some other form of storage. All
Ganesha requires is that the FSAL provide the commit2() op, that ensures
that all previous writes to that handle are permanently committed to
durable storage.
Daniel
11:13 a.m.
NFSv3 doesn't have any IP failover concept but it does have NFS server or
NFS client reboots or crashes. Think of IP failover as NFS server crashing
and restarting (albeit on another node). Yes, NFS client is supposed to
re-write all UNSTABLE writes. Write verifier is meant to detect this. We
found some bugs in NFS clients where they didn't actually resend UNSTABLE
writes but Linux is not one of them!
If you set NFS_Commit = TRUE, yes Ganesha will write the data to persistent
storage (aka disk), but you will have heavy performance hit. As Dan
indicated this was really meant for VMWare client where it was sending
write followed by commit for any random write (Linux client just sends one
FILE_SYNC write and avoids unnecessary commit)
On Wed, Mar 18, 2020 at 11:08 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
On 3/18/20 12:25 PM, alxg(a)weka.io wrote:
>> If everything operates correctly, it should be okay to use unstable
>> writes with commits, even in a cluster, assuming the cluster has proper
>> Grace handling.
>
> So what you mentioned before about using only stable writes to avoid
corruption - did you mean this only in the sense of persistency?
I was actually thinking of client side (sync vs. async mounts).
> But IMHO the user should anyways be aware of using `fsync` to make sure
data gets to the disk. I was mainly concerned about the interoperability of
unstable writes and floating IPs failovers - but you're saying that most
FSALs do allow unstable writes to some kind of RAM only cache (e.g.
pagecache or some userspace lib) on the ganesha host, and syncing them to
the actual disks only on `commit`, and that works with floating IPs, and
the client knows to correctly mark pagecache on its side as not dirty
anymore only after a commit - right?
You misunderstand, or I'm not clear. Ganesha does not have any data
caching, in RAM or otherwise. Various FSALs can/do have various forms
of caching built into their filesystems. One example is the kernel
pagecache used by FSAL_VFS. libcephfs, used by FSAL_CEPH, also has data
caching, both on the client and on the server. Unstable writes use
these caches, whether they be on the Ganesha server, or elsewhere in the
cluster, whether they be in RAM or on some other form of storage. All
Ganesha requires is that the FSAL provide the commit2() op, that ensures
that all previous writes to that handle are permanently committed to
durable storage.
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
5:25 a.m.
New subject: nfs-ganesha in a cluster with IP failover
Thanks Malahal, the client UNSTABLE resend behaviour was what I was trying to verify.
Thanks Daniel for your previous explanations before.
Malhal, note regarding NFS_Commit as I've noted it's not used, and Daniel wrote
> It was lost in 2.7 with the transition to async read/write.
1483
days inactive
1507
days old
7 comments
3 participants
participants (3)
-
alxg＠weka.io -
Daniel Gryniewicz -
Malahal Naineni