3:37 a.m.
Hi Layton ,Dan and other experts,
When a client connect Server A, and Sever A crush. So the client try to connect Server B.
If Server A reboot very slowly so that Server B don't know it should enter cluster
grace period .If so, this client have no opportunity to reclaim previous resource. Is it
right?
Marvin
7:21 a.m.
On 4/8/19 4:37 AM, fanzi2009(a)hotmail.com wrote:
Hi Layton ,Dan and other experts,
When a client connect Server A, and Sever A crush. So the client try to connect Server B.
If Server A reboot very slowly so that Server B don't know it should enter cluster
grace period .If so, this client have no opportunity to reclaim previous resource. Is it
right?
So, a couple of points here. First, the client can't just connect to
server B. That's not part of the NFS protocol. Something has to cause
the client to connect to B (such as Pacemaker moving the virtual IP for
A over to B, or a load balancer deciding A is dead and sending it's
connections to B). This means that whatever is controlling your NFS
cluster must know that A is dead before an of it's clients can be sent to B.
At this point, that controller must signal all other NFS servers (B in
this case) to enter grace *before* it sends any of A's clients to any
other server. Once the cluster is in grace, then it can send the client
to B.
Basically, the cluster goes into grace when A crashes, not when it
reboots, and this has to happen before any clients fail over. This
doesn't automatically happen; something (pacemaker/corosync, ctdb,
kuberneties, etc.) must be controlling the cluster, and sending signals
to all the NFS servers to send them into grace.
Daniel
4:27 a.m.
Hi Daniel,
I agree with your thought, but the implemation of FSAL_CEPH seems to be different with
yours. Here is the design of it. https://www.mankier.com/8/ganesha-rados-cluster-design
I also checked the code. When one of servers enter a grace period, it will call
rados_cluster_read_clids() to read culster wide variable C and R. Only if R==0, the whole
culster enter grace period. If so, the cluster goes grace period when reboot, not when
crash. It also seems no interface for pacemaker/corosync to get information and put the
cluster into grace period.
Thanks,
Marvin
7:07 a.m.
On 4/10/19 5:27 AM, fanzi2009(a)hotmail.com wrote:
Hi Daniel,
I agree with your thought, but the implemation of FSAL_CEPH seems to be different with
yours. Here is the design of it. https://www.mankier.com/8/ganesha-rados-cluster-design
I also checked the code. When one of servers enter a grace period, it will call
rados_cluster_read_clids() to read culster wide variable C and R. Only if R==0, the whole
culster enter grace period. If so, the cluster goes grace period when reboot, not when
crash. It also seems no interface for pacemaker/corosync to get information and put the
cluster into grace period.
The FSAL_CEPH design is different. It's based on kubernetes and
containerized ceph, so that if a ganesha instance crashes, kubernetes
immediately starts a new instance of that ganesha, with the same IP
address and everything. This means that we *can* go into grace on
"reboot", since the reboot is a matter if a very few seconds, rather
than a full reboot. If this was running on normal hardware, especially
server hardware with it's many minute reboot time, this would not be
possible. This is why traditional HA systems based on
pacemaker/corosync or ctdb need to put the cluster into grace on crash,
rather than on reboot.
Daniel
10:24 a.m.
Hi Daniel,
I have another idea for reclaim lock in nfs-ganesha cluster. Just remove all the logic
judgement on lock request in nfs-ganesha no matter it is in grace period or not.
nfs-ganesha just redirect the lock command to Ceph cluster. Ceph cluster will get clientid
and open owner, so Ceph have enough information to grant the lock or not. If the client
has been granted for the lock before, so the lock reclaim will also be successful. If the
lock request is coming from other client, it will failed. If so, we don't need the
grace period at all and don't need to put other server into grace periold too.
Thanks,
Marvin
10:56 a.m.
On Tue, 2019-04-23 at 15:24 +0000, fanzi2009(a)hotmail.com wrote:
Hi Daniel,
I have another idea for reclaim lock in nfs-ganesha cluster. Just remove all the logic
judgement on lock request in nfs-ganesha no matter it is in grace period or not.
nfs-ganesha just redirect the lock command to Ceph cluster. Ceph cluster will get clientid
and open owner, so Ceph have enough information to grant the lock or not. If the client
has been granted for the lock before, so the lock reclaim will also be successful. If the
lock request is coming from other client, it will failed. If so, we don't need the
grace period at all and don't need to put other server into grace periold too.
In general, that logic exists to handle corner cases relating to network
partitions and multiple server reboots. I suggest you work out how that
will handle those situations if you plan to pursue this.
See: https://tools.ietf.org/html/rfc5661#section-8.4.3
--
Jeff Layton <jlayton(a)poochiereds.net>
2101
days inactive
2116
days old
5 comments
3 participants
participants (3)
-
Daniel Gryniewicz -
fanzi2009＠hotmail.com -
Jeff Layton