2:38 a.m.
Hi ,
Suppose I have mulitple nfs-ganesha servers behind a load balancer. All nfs clients
connect the load balancer, then balancer choose a nfs-ganesha server and redirect the
request to it. When a nfs-ganesha (Ganesha-A)which has connection with a client restart in
a time, the server enter grace period and read previous clients from backend. However, as
client connection is broken and load balancer choose another nfs-ganesha sever(Ganesha-B).
Even the client want to reclaim open and clock, Ganesha-B is not in grace period. In this
scenario, how could client relaim the resource?
Thanks,
Marvin
5:16 a.m.
Hi Marvin,
On 4/2/19 3:08 PM, fanzi2009(a)hotmail.com wrote:
Hi ,
Suppose I have mulitple nfs-ganesha servers behind a load balancer. All nfs clients
connect the load balancer, then balancer choose a nfs-ganesha server and redirect the
request to it. When a nfs-ganesha (Ganesha-A)which has connection with a client restart in
a time, the server enter grace period and read previous clients from backend. However, as
client connection is broken and load balancer choose another nfs-ganesha sever(Ganesha-B).
Even the client want to reclaim open and clock, Ganesha-B is not in grace period. In this
scenario, how could client relaim the resource?
Thats right. A simple load-balancer will not do the trick. You shall
need special handling to put entire cluster (all ganesha servers) in
grace even if one server goes down or restarts. There is dbus command
available to put a ganesha server in grace anytime while it is active.
And you would need to configure your cluster manager to invoke that
command for all the servers when needed (for eg., for pcs/pacemaker
using pacemaker resource agents; for CTDB using event scripts)
Thanks,
Soumya
Thanks,
> Marvin
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org
> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>
5:26 a.m.
On 4/2/19 5:38 AM, fanzi2009(a)hotmail.com wrote:
Hi ,
Suppose I have mulitple nfs-ganesha servers behind a load balancer. All nfs clients
connect the load balancer, then balancer choose a nfs-ganesha server and redirect the
request to it. When a nfs-ganesha (Ganesha-A)which has connection with a client restart in
a time, the server enter grace period and read previous clients from backend. However, as
client connection is broken and load balancer choose another nfs-ganesha sever(Ganesha-B).
Even the client want to reclaim open and clock, Ganesha-B is not in grace period. In this
scenario, how could client relaim the resource?
Thanks,
Marvin
If you're going to have multiple Ganesha's serving the same clients,
whether load balanced or with failover, you need to have a system to put
the entire set of Ganeshas into Grace when any of them fails. In this
case, I suspect the load balancer needs to know when a Ganesha fails,
and block any new client connections and not move any clients between
Ganeshas until all the Ganeshas have entered Grace.
Basically, the load balancer moving a client from one Ganesha to
another, for any reason, looks like a failover to NFS, and all the
Ganeshas need to be in Grace before it can happen.
NFS is not well designed for a clustered server setup, so it's a bit
awkward to get it working well.
Daniel
6:07 a.m.
New subject: Question about recovery in cluster envirnment
Thanks for Daniel and Soumya!
Putting whole cluster into grace period, will it cause performance issue?
Is it OK that we save all the session informations into backend. When ServerA crashed and
client connect ServerB, ServerB can resume session from backend including open and locking
resource.
6:21 a.m.
New subject: Question about recovery in cluster envirnment
On 4/2/19 9:07 AM, fanzi2009(a)hotmail.com wrote:
Thanks for Daniel and Soumya!
Putting whole cluster into grace period, will it cause performance issue?
Is it OK that we save all the session informations into backend. When ServerA crashed and
client connect ServerB, ServerB can resume session from backend including open and locking
resource.
You have to do that, and you also have to put the cluster into Grace.
When the client connects to the new server, it will re-acquire any state
it had before, and so the new server needs to know what clients can do this.
As I said, NFS is not a great clustered protocol. If you have multiple
servers serving the same data, and any of them goes down, you need to
put them all into Grace. This is to keep any *other* clients from
getting locks that should belong to the clients that were connected to
the down server. Yes, this will have performance issues, but it's the
only way to avoid data corruption.
Daniel
7:40 a.m.
On Tue, Apr 2, 2019 at 9:21 AM Daniel Gryniewicz <dang(a)redhat.com> wrote:
On 4/2/19 9:07 AM, fanzi2009(a)hotmail.com wrote:
> Thanks for Daniel and Soumya!
> Putting whole cluster into grace period, will it cause performance issue?
>
> Is it OK that we save all the session informations into backend. When ServerA
crashed and client connect ServerB, ServerB can resume session from backend including open
and locking resource.
You have to do that, and you also have to put the cluster into Grace.
When the client connects to the new server, it will re-acquire any state
it had before, and so the new server needs to know what clients can do this.
As I said, NFS is not a great clustered protocol. If you have multiple
servers serving the same data, and any of them goes down, you need to
put them all into Grace. This is to keep any *other* clients from
getting locks that should belong to the clients that were connected to
the down server. Yes, this will have performance issues, but it's the
only way to avoid data corruption.
I mostly agree with everything Dan said here. Clustered environments
require extra care, and we don't currently have a way to migrate
clients between different ganesha heads that are exporting the same
clustered fs. I've done some experimentation with v4 migration here,
but there's nothing in stock ganesha today for this.
You can, however, avoid putting all of the other cluster nodes into
the grace period if you know that none of the other cluster nodes can
acquire state that will need to be reclaimed by a clients of the node
that is restarting. We have some tentative plans to implement this for
FSAL_CEPH + the RADOS recovery backends someday, but it requires
support in the Ceph MDS and userland client libraries that does not
yet exist.
--
Jeff Layton <jlayton(a)poochiereds.net>
7:46 p.m.
New subject: Question about recovery in cluster envirnment
How about I persist all the session information into backend(rados for example)? If so,
even though the client connect to another server, the server can reconstruct session from
backend. Client don't need to re-create session and don't need to reclaim.
> On Tue, Apr 2, 2019 at 9:21 AM Daniel Gryniewicz <dang(a)redhat.com>
wrote:
>
> I mostly agree with everything Dan said here. Clustered environments
> require extra care, and we don't currently have a way to migrate
> clients between different ganesha heads that are exporting the same
> clustered fs. I've done some experimentation with v4 migration here,
> but there's nothing in stock ganesha today for this.
>
> You can, however, avoid putting all of the other cluster nodes into
> the grace period if you know that none of the other cluster nodes can
> acquire state that will need to be reclaimed by a clients of the node
> that is restarting. We have some tentative plans to implement this for
> FSAL_CEPH + the RADOS recovery backends someday, but it requires
> support in the Ceph MDS and userland client libraries that does not
> yet exist.
7:27 a.m.
New subject: Question about recovery in cluster envirnment
This might work, but keep in mind that *any* change to the client (or
it's associated state) needs to be persisted to synchronously rados,
which will introduce huge latency to most ops. This isn't even memory
-> disk latency (100x - 1000x) but memory to network latency (10000x or
higher).
Daniel
On 4/2/19 10:46 PM, fanzi2009(a)hotmail.com wrote:
How about I persist all the session information into backend(rados
for example)? If so, even though the client connect to another server, the server can
reconstruct session from backend. Client don't need to re-create session and don't
need to reclaim.
> On Tue, Apr 2, 2019 at 9:21 AM Daniel Gryniewicz <dang(a)redhat.com>
wrote:
>
> I mostly agree with everything Dan said here. Clustered environments
> require extra care, and we don't currently have a way to migrate
> clients between different ganesha heads that are exporting the same
> clustered fs. I've done some experimentation with v4 migration here,
> but there's nothing in stock ganesha today for this.
>
> You can, however, avoid putting all of the other cluster nodes into
> the grace period if you know that none of the other cluster nodes can
> acquire state that will need to be reclaimed by a clients of the node
> that is restarting. We have some tentative plans to implement this for
> FSAL_CEPH + the RADOS recovery backends someday, but it requires
> support in the Ceph MDS and userland client libraries that does not
> yet exist.
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
6:20 a.m.
Easier said than done. Bear in mind that all of the recovery backend
stuff is entirely for the purpose of dealing with server restarts, so
you really do have to be careful not to leave gaps in particular
scenarios.
Let's say you do decide to synchronously store open and lock records
in a central RADOS-based database and both Server A and Server B are
using it.
Server A crashes, and the client decides to reconnect to Server B
using the recorded clientid/session. Server B says "Oh, this session
was previously held by Server A." Now what happens?
You need a mechanism to transfer the CephFS state (opens, locks, caps,
etc.) from Server A to Server B. Nothing like that exists today, but
we do have some tentative plans to allow cephfs clients to reclaim
state they previously held. In principle, that could be extended to
allow "takeover" in some fashion.
But wait...it gets worse!
Suppose we have a 3 node ganesha cluster and some of Server A's
clients decide to go to Server C instead? Now a simple takeover is not
enough -- you need a way to split that state granularly.
Couple all of this with the basic truism that failures in these sorts
of architectures are often cascading. You need to deal with the
possibility that any node could just die at any time, and decide how
you're going to deal with that. A lot of the original ganesha recovery
backend work had gaping holes in the "takeover" mechanisms where
failures at an inopportune time could make it so no clients could
recover anything.
This is very much a non-trivial problem in my experience, but don't
let me dissuade you if you have considered these scenarios and have
thoughts on how to address it.
--
Jeff Layton <jlayton(a)poochiereds.net>
On Tue, Apr 2, 2019 at 10:46 PM <fanzi2009(a)hotmail.com> wrote:
>
> How about I persist all the session information into backend(rados for example)? If
so, even though the client connect to another server, the server can reconstruct session
from backend. Client don't need to re-create session and don't need to reclaim.
>
> > On Tue, Apr 2, 2019 at 9:21 AM Daniel Gryniewicz <dang(a)redhat.com>
wrote:
> >
> > I mostly agree with everything Dan said here. Clustered environments
> > require extra care, and we don't currently have a way to migrate
> > clients between different ganesha heads that are exporting the same
> > clustered fs. I've done some experimentation with v4 migration here,
> > but there's nothing in stock ganesha today for this.
> >
> > You can, however, avoid putting all of the other cluster nodes into
> > the grace period if you know that none of the other cluster nodes can
> > acquire state that will need to be reclaimed by a clients of the node
> > that is restarting. We have some tentative plans to implement this for
> > FSAL_CEPH + the RADOS recovery backends someday, but it requires
> > support in the Ceph MDS and userland client libraries that does not
> > yet exist.
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org
> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
8 a.m.
New subject: Question about recovery in cluster envirnment
One more thing to keep in mind: Getting this wrong is a data corruption
bug, which is the worst possible bug you can have in a storage system,
so you should be pretty sure you've gotten it right.
Daniel
On 4/5/19 9:20 AM, Jeff Layton wrote:
Easier said than done. Bear in mind that all of the recovery backend
stuff is entirely for the purpose of dealing with server restarts, so
you really do have to be careful not to leave gaps in particular
scenarios.
Let's say you do decide to synchronously store open and lock records
in a central RADOS-based database and both Server A and Server B are
using it.
Server A crashes, and the client decides to reconnect to Server B
using the recorded clientid/session. Server B says "Oh, this session
was previously held by Server A." Now what happens?
You need a mechanism to transfer the CephFS state (opens, locks, caps,
etc.) from Server A to Server B. Nothing like that exists today, but
we do have some tentative plans to allow cephfs clients to reclaim
state they previously held. In principle, that could be extended to
allow "takeover" in some fashion.
But wait...it gets worse!
Suppose we have a 3 node ganesha cluster and some of Server A's
clients decide to go to Server C instead? Now a simple takeover is not
enough -- you need a way to split that state granularly.
Couple all of this with the basic truism that failures in these sorts
of architectures are often cascading. You need to deal with the
possibility that any node could just die at any time, and decide how
you're going to deal with that. A lot of the original ganesha recovery
backend work had gaping holes in the "takeover" mechanisms where
failures at an inopportune time could make it so no clients could
recover anything.
This is very much a non-trivial problem in my experience, but don't
let me dissuade you if you have considered these scenarios and have
thoughts on how to address it.
2120
days inactive
2123
days old
9 comments
4 participants
participants (4)
-
Daniel Gryniewicz -
fanzi2009＠hotmail.com -
Jeff Layton -
Soumya Koduri