nfs-ganesha in a cluster with IP failover
Hi, We'e setting up ganesha using FSAL_VFS, and NFSv3 only. We have a cluster of them running with floating IPs being failed over between them when a host dies. The question we have is regarding writes which are UNSTABLE - meaning the client writes out from its cache for example with UNSTABLE, and then sends a COMMIT. In the event of a ganesha host crashing after receiving the writeouts, but before the COMMIT - is it assured that when the floating IP will failover to another host the client will resend the unstable writes again as well? I guess this has several parts to it: * Client side - assuming it's a linux kernel nfs client and it uses the pagecache - does it mark the pages clean right after successfully sending their writes as unstable, or does it mark them as clean only after it successfully sends a COMMIT on them as well? Do you happen to know? * Server side - Does nfs-ganesha have some data cache? Or the writes being unstable just means they can land in the pagecache of the server side, and stay there until the COMMIT does a sync? in that case do we need to set `NFS_Commit = true`? What are the guidelines on such a setup to ensure data doesn't get lost and doesn't corrupt? Thank you, Alex
The biggest guideline to make sure you don't get corruption is to not use unstable writes. I realize this is a performance hit, but it's the only way to guarantee that every ack'd write is committed to stable storage. (Obviously, if a crash occurs mid-write, all bets are off). Ganesha does not currently have any kind if data cache. However, FSAL_VFS uses the local pagecache. A stable write and a commit will each cause an fsync(), so the result will be committed to disk. An unstable write, however, will just end up in the pagecache, and so will likely not be stable (depending, of course, on how busy the server is). Looking at the kernel client, it looks like it sends unstable writes if there are more writes queued to the file, or if a commit is already scheduled. Otherwise, it sends stable writes. Beyond that, we're outside of my knowledge of the client. I assume it will commit or perform a stable write periodically, even if writes are constant? Note that all this is unrelated to the sync/async mount flags; those determine how writes affect the client pagecache, not the server at all. Daniel On 3/12/20 2:36 AM, alxg@weka.io wrote:
Hi, We'e setting up ganesha using FSAL_VFS, and NFSv3 only. We have a cluster of them running with floating IPs being failed over between them when a host dies. The question we have is regarding writes which are UNSTABLE - meaning the client writes out from its cache for example with UNSTABLE, and then sends a COMMIT. In the event of a ganesha host crashing after receiving the writeouts, but before the COMMIT - is it assured that when the floating IP will failover to another host the client will resend the unstable writes again as well? I guess this has several parts to it: * Client side - assuming it's a linux kernel nfs client and it uses the pagecache - does it mark the pages clean right after successfully sending their writes as unstable, or does it mark them as clean only after it successfully sends a COMMIT on them as well? Do you happen to know? * Server side - Does nfs-ganesha have some data cache? Or the writes being unstable just means they can land in the pagecache of the server side, and stay there until the COMMIT does a sync? in that case do we need to set `NFS_Commit = true`?
What are the guidelines on such a setup to ensure data doesn't get lost and doesn't corrupt?
Thank you, Alex _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Hey Daniel, So you're saying that clients which are working against a cluster of hosts with floating IPs should not send unstable writes? In theory it should be possible to make them work with unstable writes no? Do all the FSALs of clustered FSs always sync the writes regardless of them being requested as unstable? And as long as we make sure that all writes to pagecache on the server side are synced (i.e. the underlying mount for the export for our weka fs is with "-o sync") - everything should be coherent? So what is the `NFS_Commit` flag is for? in 2.8.3 I think it actually is unused - is it not? Alex
On 3/18/20 9:14 AM, alxg@weka.io wrote:
Hey Daniel,
So you're saying that clients which are working against a cluster of hosts with floating IPs should not send unstable writes? In theory it should be possible to make them work with unstable writes no?
If everything operates correctly, it should be okay to use unstable writes with commits, even in a cluster, assuming the cluster has proper Grace handling.
Do all the FSALs of clustered FSs always sync the writes regardless of them being requested as unstable?
All FSALs, as far as I know, honor the write request flags.
And as long as we make sure that all writes to pagecache on the server side are synced (i.e. the underlying mount for the export for our weka fs is with "-o sync") - everything should be coherent?
Yes, for FSALs that use the pagecache.
So what is the `NFS_Commit` flag is for? in 2.8.3 I think it actually is unused - is it not?
This was added for VMWare clients, to work around an odd performance issue they had. It forced a sync on every write. It was lost in 2.7 with the transition to async read/write. Daniel
If everything operates correctly, it should be okay to use unstable writes with commits, even in a cluster, assuming the cluster has proper Grace handling.
So what you mentioned before about using only stable writes to avoid corruption - did you mean this only in the sense of persistency? But IMHO the user should anyways be aware of using `fsync` to make sure data gets to the disk. I was mainly concerned about the interoperability of unstable writes and floating IPs failovers - but you're saying that most FSALs do allow unstable writes to some kind of RAM only cache (e.g. pagecache or some userspace lib) on the ganesha host, and syncing them to the actual disks only on `commit`, and that works with floating IPs, and the client knows to correctly mark pagecache on its side as not dirty anymore only after a commit - right?
On 3/18/20 12:25 PM, alxg@weka.io wrote:
If everything operates correctly, it should be okay to use unstable writes with commits, even in a cluster, assuming the cluster has proper Grace handling.
So what you mentioned before about using only stable writes to avoid corruption - did you mean this only in the sense of persistency?
I was actually thinking of client side (sync vs. async mounts).
But IMHO the user should anyways be aware of using `fsync` to make sure data gets to the disk. I was mainly concerned about the interoperability of unstable writes and floating IPs failovers - but you're saying that most FSALs do allow unstable writes to some kind of RAM only cache (e.g. pagecache or some userspace lib) on the ganesha host, and syncing them to the actual disks only on `commit`, and that works with floating IPs, and the client knows to correctly mark pagecache on its side as not dirty anymore only after a commit - right?
You misunderstand, or I'm not clear. Ganesha does not have any data caching, in RAM or otherwise. Various FSALs can/do have various forms of caching built into their filesystems. One example is the kernel pagecache used by FSAL_VFS. libcephfs, used by FSAL_CEPH, also has data caching, both on the client and on the server. Unstable writes use these caches, whether they be on the Ganesha server, or elsewhere in the cluster, whether they be in RAM or on some other form of storage. All Ganesha requires is that the FSAL provide the commit2() op, that ensures that all previous writes to that handle are permanently committed to durable storage. Daniel
NFSv3 doesn't have any IP failover concept but it does have NFS server or NFS client reboots or crashes. Think of IP failover as NFS server crashing and restarting (albeit on another node). Yes, NFS client is supposed to re-write all UNSTABLE writes. Write verifier is meant to detect this. We found some bugs in NFS clients where they didn't actually resend UNSTABLE writes but Linux is not one of them! If you set NFS_Commit = TRUE, yes Ganesha will write the data to persistent storage (aka disk), but you will have heavy performance hit. As Dan indicated this was really meant for VMWare client where it was sending write followed by commit for any random write (Linux client just sends one FILE_SYNC write and avoids unnecessary commit) On Wed, Mar 18, 2020 at 11:08 PM Daniel Gryniewicz <dang@redhat.com> wrote:
On 3/18/20 12:25 PM, alxg@weka.io wrote:
If everything operates correctly, it should be okay to use unstable writes with commits, even in a cluster, assuming the cluster has proper Grace handling.
So what you mentioned before about using only stable writes to avoid corruption - did you mean this only in the sense of persistency?
I was actually thinking of client side (sync vs. async mounts).
But IMHO the user should anyways be aware of using `fsync` to make sure data gets to the disk. I was mainly concerned about the interoperability of unstable writes and floating IPs failovers - but you're saying that most FSALs do allow unstable writes to some kind of RAM only cache (e.g. pagecache or some userspace lib) on the ganesha host, and syncing them to the actual disks only on `commit`, and that works with floating IPs, and the client knows to correctly mark pagecache on its side as not dirty anymore only after a commit - right?
You misunderstand, or I'm not clear. Ganesha does not have any data caching, in RAM or otherwise. Various FSALs can/do have various forms of caching built into their filesystems. One example is the kernel pagecache used by FSAL_VFS. libcephfs, used by FSAL_CEPH, also has data caching, both on the client and on the server. Unstable writes use these caches, whether they be on the Ganesha server, or elsewhere in the cluster, whether they be in RAM or on some other form of storage. All Ganesha requires is that the FSAL provide the commit2() op, that ensures that all previous writes to that handle are permanently committed to durable storage.
Daniel _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Thanks Malahal, the client UNSTABLE resend behaviour was what I was trying to verify. Thanks Daniel for your previous explanations before. Malhal, note regarding NFS_Commit as I've noted it's not used, and Daniel wrote
It was lost in 2.7 with the transition to async read/write.
participants (3)
-
alxg＠weka.io -
Daniel Gryniewicz -
Malahal Naineni