Question about recovery on Ceph FSAL
Hi Layton ,Dan and other experts, When a client connect Server A, and Sever A crush. So the client try to connect Server B. If Server A reboot very slowly so that Server B don't know it should enter cluster grace period .If so, this client have no opportunity to reclaim previous resource. Is it right? Marvin
On 4/8/19 4:37 AM, fanzi2009@hotmail.com wrote:
Hi Layton ,Dan and other experts, When a client connect Server A, and Sever A crush. So the client try to connect Server B. If Server A reboot very slowly so that Server B don't know it should enter cluster grace period .If so, this client have no opportunity to reclaim previous resource. Is it right?
So, a couple of points here. First, the client can't just connect to server B. That's not part of the NFS protocol. Something has to cause the client to connect to B (such as Pacemaker moving the virtual IP for A over to B, or a load balancer deciding A is dead and sending it's connections to B). This means that whatever is controlling your NFS cluster must know that A is dead before an of it's clients can be sent to B. At this point, that controller must signal all other NFS servers (B in this case) to enter grace *before* it sends any of A's clients to any other server. Once the cluster is in grace, then it can send the client to B. Basically, the cluster goes into grace when A crashes, not when it reboots, and this has to happen before any clients fail over. This doesn't automatically happen; something (pacemaker/corosync, ctdb, kuberneties, etc.) must be controlling the cluster, and sending signals to all the NFS servers to send them into grace. Daniel
Hi Daniel, I agree with your thought, but the implemation of FSAL_CEPH seems to be different with yours. Here is the design of it. https://www.mankier.com/8/ganesha-rados-cluster-design I also checked the code. When one of servers enter a grace period, it will call rados_cluster_read_clids() to read culster wide variable C and R. Only if R==0, the whole culster enter grace period. If so, the cluster goes grace period when reboot, not when crash. It also seems no interface for pacemaker/corosync to get information and put the cluster into grace period. Thanks, Marvin
On 4/10/19 5:27 AM, fanzi2009@hotmail.com wrote:
Hi Daniel, I agree with your thought, but the implemation of FSAL_CEPH seems to be different with yours. Here is the design of it. https://www.mankier.com/8/ganesha-rados-cluster-design I also checked the code. When one of servers enter a grace period, it will call rados_cluster_read_clids() to read culster wide variable C and R. Only if R==0, the whole culster enter grace period. If so, the cluster goes grace period when reboot, not when crash. It also seems no interface for pacemaker/corosync to get information and put the cluster into grace period.
The FSAL_CEPH design is different. It's based on kubernetes and containerized ceph, so that if a ganesha instance crashes, kubernetes immediately starts a new instance of that ganesha, with the same IP address and everything. This means that we *can* go into grace on "reboot", since the reboot is a matter if a very few seconds, rather than a full reboot. If this was running on normal hardware, especially server hardware with it's many minute reboot time, this would not be possible. This is why traditional HA systems based on pacemaker/corosync or ctdb need to put the cluster into grace on crash, rather than on reboot. Daniel
Hi Daniel, I have another idea for reclaim lock in nfs-ganesha cluster. Just remove all the logic judgement on lock request in nfs-ganesha no matter it is in grace period or not. nfs-ganesha just redirect the lock command to Ceph cluster. Ceph cluster will get clientid and open owner, so Ceph have enough information to grant the lock or not. If the client has been granted for the lock before, so the lock reclaim will also be successful. If the lock request is coming from other client, it will failed. If so, we don't need the grace period at all and don't need to put other server into grace periold too. Thanks, Marvin
On Tue, 2019-04-23 at 15:24 +0000, fanzi2009@hotmail.com wrote:
Hi Daniel,
I have another idea for reclaim lock in nfs-ganesha cluster. Just remove all the logic judgement on lock request in nfs-ganesha no matter it is in grace period or not. nfs-ganesha just redirect the lock command to Ceph cluster. Ceph cluster will get clientid and open owner, so Ceph have enough information to grant the lock or not. If the client has been granted for the lock before, so the lock reclaim will also be successful. If the lock request is coming from other client, it will failed. If so, we don't need the grace period at all and don't need to put other server into grace periold too.
In general, that logic exists to handle corner cases relating to network partitions and multiple server reboots. I suggest you work out how that will handle those situations if you plan to pursue this. See: https://tools.ietf.org/html/rfc5661#section-8.4.3 -- Jeff Layton <jlayton@poochiereds.net>
participants (3)
-
Daniel Gryniewicz -
fanzi2009＠hotmail.com -
Jeff Layton