[RFC PATCH] rados_cluster: add a "design" manpage
From: Jeff Layton <jlayton@redhat.com> Bruce asked for better design documentation, so this is my attempt at it. Let me know what you think. I'll probably end up squashing this into one of the code patches but for now I'm sending this separately to see if it helps clarify things. Suggestions and feedback are welcome. Change-Id: I53cc77f66b2407c2083638e5760666639ba1fd57 Signed-off-by: Jeff Layton <jlayton@redhat.com> --- src/doc/man/ganesha-rados-cluster.rst | 227 ++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 src/doc/man/ganesha-rados-cluster.rst diff --git a/src/doc/man/ganesha-rados-cluster.rst b/src/doc/man/ganesha-rados-cluster.rst new file mode 100644 index 000000000000..1ba2d3c29093 --- /dev/null +++ b/src/doc/man/ganesha-rados-cluster.rst @@ -0,0 +1,227 @@ +============================================================================== +ganesha-rados-cluster-design -- Clustered RADOS Recovery Backend Design +============================================================================== + +.. program:: ganesha-rados-cluster-design + +This document aims to explain the theory and design behind the +rados_cluster recovery backend, which coordinates grace period +enforcement among multiple, independent NFS servers. + +In order to understand the clustered recovery backend, it's first necessary +to understand how recovery works with a single server: + +Singleton Server Recovery +------------------------- +NFSv4 is a lease-based protocol. Clients set up a relationship to the +server and must periodically renew their lease in order to maintain +their ephemeral state (open files, locks, delegations or layouts). + +When a singleton NFS server is restarted, any ephemeral state is lost. When +the server comes comes back online, NFS clients detect that the server has +been restarted and will reclaim the ephemeral state that they held at the +time of their last contact with the server. + +Singleton Grace Period +---------------------- + +In order to ensure that we don't end up with conflicts, clients are +barred from acquiring any new state while in the Recovery phase. Only +reclaim operations are allowed. + +This period of time is called the **grace period**. Most NFS servers +have a grace period that lasts around two lease periods, however +nfs-ganesha can and will lift the grace period early if it determines +that no more clients will be allowed to recover. + +Once the grace period ends, the server will move into its Normal +operation state. During this period, no more recovery is allowed and new +state can be acquired by NFS clients. + +Reboot Epochs +------------- +The lifecycle of a singleton NFS server can be considered to be a series +of transitions from the Recovery period to Normal operation and back. In the +remainder of this document we'll consider such a period to be an +**epoch**, and assign each a number beginning with 1. + +Visually, we can represent it like this, such that each +Normal -> Recovery transition is marked by a change in the epoch value: + + +---------------------------------------+ + | R | N | R | N | R | R | R | N | R | N | <=== Operational State + +---------------------------------------+ + | 1 | 2 | 3 | 4 | <=== Epoch + +---------------------------------------+ + +Note that it is possible to restart during the grace period (as shown +above during epoch 3). That just serves to extend the recovery period +and the epoch for longer. A new epoch is only declared during a +Recovery -> Normal transition. + +Client Recovery Database +------------------------ +There are some potential edge cases that can occur involving network +partitions and multiple reboots. In order to prevent those, the server +must maintain a list of clients that hold state on the server at any +given time. This list must be maintained on stable storage. If a client +sends a request to reclaim some state, then the server must check to +make sure it's on that list before allowing the request. + +Thus when the server allows reclaim requests it must always gate it +against the recovery database from the previous epoch. As clients come +in to reclaim, we establish records for them in a new database +associated with the current epoch. + +The transition from recovery to normal operation should perform an +atomic switch of recovery databases. A recovery database only becomes +legitimate on a recovery to normal transition. Until that point, the +recovery database from the previous epoch is the canonical one. + +Exporting a Clustered Filesystem +-------------------------------- +Let's consider a set of independent NFS servers, all serving out the same +content from a clustered backend filesystem of any flavor. Each NFS +server in this case can itself be considered a clustered FS client. This +means that the NFS server is really just a proxy for state on the +clustered filesystem (XXX: diagram here?) + +The filesystem must make some guarantees to the NFS server. First filesystem +guarantee: + +1. The filesystem ensures that the NFS servers (aka the FS clients) + cannot obtain state that conflicts with that of another NFS server. + +This is somewhat obvious and is what we expect from any clustered filesystem +outside of any requirements of NFS. If the clustered filesystem can +provide this, then we know that conflicting state during normal +operations cannot be granted. + +The recovery period has a different set of rules. If an NFS server +crashes and is restarted, then we have a window of time when that NFS +server does not know what state was held by its clients. + +If the state held by the crashed NFS server is immediately released +after the crash, another NFS server could hand out conflicting state +before the original NFS client has a chance to recover it. + +This *must* be prevented. Second filesystem guarantee: + +2. The filesystem must not release state held by a server during the + previous epoch until all servers in the cluster are enforcing the + grace period. + +In practical terms, we want the filesystem to provide a way for an NFS +server to tell it when it's safe to release state held by a previous +instance of itself. The server should do this once it knows that all of +its siblings are enforcing the grace period. + +Note that we do not require that all servers restart and allow reclaim +at that point. It's sufficient for them to simply begin grace period +enforcement as soon as possible once one server needs it. + +Clustered Grace Period Database +------------------------------- +At this point the cluster siblings are no longer completely independent, +and the grace period has become a cluster-wide property. This means that +we must track the current epoch on some sort of shared storage that the +servers can all access. + +Additionally we must also keep track of whether a cluster-wide grace period +is in effect. Any running nodes should all be informed when either of this +info changes, so they can take appropriate steps when it occurs. + +In the rados_cluster backend, we track these using two epoch values: + +- **C**: is the current epoch. This represents the current epoch value + of the cluster + +- **R**: is the recovery epoch. This represents the epoch from which + clients are allowed to recover. A non-zero value here means + that a cluster-wide grace period is in effect. Setting this to + 0 ends that grace period. + +In order to decide when to make grace period transitions, we must also +have each server to advertise its state to the other nodes. Specifically, +we need to allow servers to determine these two things about each of +its siblings: + +1. Does this server have clients from the previous epoch that will require + recovery? (NEED) + +2. Is this server allowing clients to acquire new state? (ENFORCING) + +We do this with a pair of flags per sibling (NEED and ENFORCING). Each +server typically manages its own flags. + +The rados_cluster backend stores all of this information in a single +RADOS object that is modified using read/modify/write cycles. Typically +we'll read the whole object, modify it, and then attept to write it +back. If something changes between the read and write, we redo the read +and try it again. + +Clustered Client Recovery Databases +----------------------------------- +In rados_cluster the client recovery databases are stored as RADOS +objects. Each NFS server has its own set of them and they are given +names that have the current epoch (C) embedded in it. This ensures +that recovery databases are specific to a particular epoch. + +In general, it's safe to delete any recovery database that precedes R +when R is non-zero, and safe to remove any recovery database except for +the current one (the one with C in the name) when the grace period is +not in effect (R==0). + +Establishing a New Grace Period +------------------------------- +When a server restarts and wants to allow clients to reclaim their +state, it must establish a new epoch by incrementing the current epoch +to declare a new grace period (R=C; C=C+1). + +The exception to this rule is when the cluster is already in a grace +period. Servers can just join an in-progress grace period instead of +establishing a new one if one is already active. + +In either case, the server should also set its NEED and ENFORCING flags +at the same time. + +The other surviving cluster siblings should take steps to begin grace +period enforcement as soon as possible. This entails "draining off" any +in-progress state morphing operations and then blocking the acquisition +of any new state (usually with a return of NFS4ERR_GRACE to clients that +attempt it). Again, there is no need for the survivors from the previous +epoch to allow recovery here. + +The surviving servers must however establish a new client recovery +database at this point to ensure that their clients can do recovery in +the event of a crash afterward. + +Once all of the siblings are enforcing the grace period, the recovering +server can then request that the filesystem release the old state, and +allow clients to begin reclaiming their state. In the rados_cluster +backend driver, we do this by stalling server startup until all hosts +in the cluster are enforcing the grace period. + +Lifting the Grace Period +------------------------ +Transitioning from recovery to normal operation really consists of two +different steps: + +1. the server decides that it no longer requires a grace period, either + due to it timing out or there not being any clients that would be + allowed to reclaim. + +2. the server stops enforcing the grace period and transitions to normal + operation + +These concepts are often conflated in a singleton servers, but in +a cluster we must consider them independently. + +When a server is finished with its own local recovery period, it should +clear its NEED flag. That server should continue enforcing the grace +period however until the grace period is fully lifted. + +If the servers' own NEED flag is the last one set, then it can lift the +grace period (by setting R=0). At that point, all servers in the cluster +can end grace period enforcement, and communicate that fact to the +others by clearing their ENFORCING flags. -- 2.17.0
Looks good. A few nits. On Wed, May 23, 2018 at 8:21 AM, Jeff Layton <jlayton@kernel.org> wrote:
From: Jeff Layton <jlayton@redhat.com>
Bruce asked for better design documentation, so this is my attempt at it. Let me know what you think. I'll probably end up squashing this into one of the code patches but for now I'm sending this separately to see if it helps clarify things.
Suggestions and feedback are welcome.
Change-Id: I53cc77f66b2407c2083638e5760666639ba1fd57 Signed-off-by: Jeff Layton <jlayton@redhat.com> --- src/doc/man/ganesha-rados-cluster.rst | 227 ++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 src/doc/man/ganesha-rados-cluster.rst
diff --git a/src/doc/man/ganesha-rados-cluster.rst b/src/doc/man/ganesha-rados-cluster.rst new file mode 100644 index 000000000000..1ba2d3c29093 --- /dev/null +++ b/src/doc/man/ganesha-rados-cluster.rst @@ -0,0 +1,227 @@ +============================================================================== +ganesha-rados-cluster-design -- Clustered RADOS Recovery Backend Design +============================================================================== + +.. program:: ganesha-rados-cluster-design + +This document aims to explain the theory and design behind the +rados_cluster recovery backend, which coordinates grace period +enforcement among multiple, independent NFS servers. + +In order to understand the clustered recovery backend, it's first necessary +to understand how recovery works with a single server: + +Singleton Server Recovery +------------------------- +NFSv4 is a lease-based protocol. Clients set up a relationship to the +server and must periodically renew their lease in order to maintain +their ephemeral state (open files, locks, delegations or layouts). + +When a singleton NFS server is restarted, any ephemeral state is lost. When +the server comes comes back online, NFS clients detect that the server has +been restarted and will reclaim the ephemeral state that they held at the +time of their last contact with the server. + +Singleton Grace Period +---------------------- + +In order to ensure that we don't end up with conflicts, clients are +barred from acquiring any new state while in the Recovery phase. Only +reclaim operations are allowed. + +This period of time is called the **grace period**. Most NFS servers +have a grace period that lasts around two lease periods, however +nfs-ganesha can and will lift the grace period early if it determines +that no more clients will be allowed to recover. + +Once the grace period ends, the server will move into its Normal +operation state. During this period, no more recovery is allowed and new +state can be acquired by NFS clients. + +Reboot Epochs +------------- +The lifecycle of a singleton NFS server can be considered to be a series +of transitions from the Recovery period to Normal operation and back. In the +remainder of this document we'll consider such a period to be an +**epoch**, and assign each a number beginning with 1. + +Visually, we can represent it like this, such that each +Normal -> Recovery transition is marked by a change in the epoch value: + + +---------------------------------------+ + | R | N | R | N | R | R | R | N | R | N | <=== Operational State + +---------------------------------------+ + | 1 | 2 | 3 | 4 | <=== Epoch + +---------------------------------------+ + +Note that it is possible to restart during the grace period (as shown +above during epoch 3). That just serves to extend the recovery period +and the epoch for longer. A new epoch is only declared during a +Recovery -> Normal transition. + +Client Recovery Database +------------------------ +There are some potential edge cases that can occur involving network +partitions and multiple reboots. In order to prevent those, the server +must maintain a list of clients that hold state on the server at any +given time. This list must be maintained on stable storage. If a client +sends a request to reclaim some state, then the server must check to +make sure it's on that list before allowing the request. + +Thus when the server allows reclaim requests it must always gate it +against the recovery database from the previous epoch. As clients come +in to reclaim, we establish records for them in a new database +associated with the current epoch. + +The transition from recovery to normal operation should perform an +atomic switch of recovery databases. A recovery database only becomes +legitimate on a recovery to normal transition. Until that point, the +recovery database from the previous epoch is the canonical one. + +Exporting a Clustered Filesystem +-------------------------------- +Let's consider a set of independent NFS servers, all serving out the same +content from a clustered backend filesystem of any flavor. Each NFS +server in this case can itself be considered a clustered FS client. This +means that the NFS server is really just a proxy for state on the +clustered filesystem (XXX: diagram here?) + +The filesystem must make some guarantees to the NFS server. First filesystem +guarantee: + +1. The filesystem ensures that the NFS servers (aka the FS clients) + cannot obtain state that conflicts with that of another NFS server. + +This is somewhat obvious and is what we expect from any clustered filesystem +outside of any requirements of NFS. If the clustered filesystem can +provide this, then we know that conflicting state during normal +operations cannot be granted. + +The recovery period has a different set of rules. If an NFS server +crashes and is restarted, then we have a window of time when that NFS +server does not know what state was held by its clients. + +If the state held by the crashed NFS server is immediately released +after the crash, another NFS server could hand out conflicting state +before the original NFS client has a chance to recover it. + +This *must* be prevented. Second filesystem guarantee: + +2. The filesystem must not release state held by a server during the + previous epoch until all servers in the cluster are enforcing the + grace period. + +In practical terms, we want the filesystem to provide a way for an NFS +server to tell it when it's safe to release state held by a previous +instance of itself. The server should do this once it knows that all of +its siblings are enforcing the grace period. + +Note that we do not require that all servers restart and allow reclaim +at that point. It's sufficient for them to simply begin grace period +enforcement as soon as possible once one server needs it. + +Clustered Grace Period Database +------------------------------- +At this point the cluster siblings are no longer completely independent, +and the grace period has become a cluster-wide property. This means that +we must track the current epoch on some sort of shared storage that the +servers can all access. + +Additionally we must also keep track of whether a cluster-wide grace period +is in effect. Any running nodes should all be informed when either of this +info changes, so they can take appropriate steps when it occurs. + +In the rados_cluster backend, we track these using two epoch values: + +- **C**: is the current epoch. This represents the current epoch value + of the cluster + +- **R**: is the recovery epoch. This represents the epoch from which + clients are allowed to recover. A non-zero value here means + that a cluster-wide grace period is in effect. Setting this to + 0 ends that grace period. + +In order to decide when to make grace period transitions, we must also +have each server to advertise its state to the other nodes. Specifically, +we need to allow servers to determine these two things about each of +its siblings: + +1. Does this server have clients from the previous epoch that will require + recovery? (NEED) + +2. Is this server allowing clients to acquire new state? (ENFORCING)
It's unclear here which state a true value for ENFORCING represents. Is it allowing new state or not? This becomes clear later, but is unclear here.
+ +We do this with a pair of flags per sibling (NEED and ENFORCING). Each +server typically manages its own flags. + +The rados_cluster backend stores all of this information in a single +RADOS object that is modified using read/modify/write cycles. Typically +we'll read the whole object, modify it, and then attept to write it +back. If something changes between the read and write, we redo the read +and try it again. + +Clustered Client Recovery Databases +----------------------------------- +In rados_cluster the client recovery databases are stored as RADOS +objects. Each NFS server has its own set of them and they are given +names that have the current epoch (C) embedded in it. This ensures +that recovery databases are specific to a particular epoch. + +In general, it's safe to delete any recovery database that precedes R +when R is non-zero, and safe to remove any recovery database except for +the current one (the one with C in the name) when the grace period is +not in effect (R==0). + +Establishing a New Grace Period +------------------------------- +When a server restarts and wants to allow clients to reclaim their +state, it must establish a new epoch by incrementing the current epoch +to declare a new grace period (R=C; C=C+1). + +The exception to this rule is when the cluster is already in a grace +period. Servers can just join an in-progress grace period instead of +establishing a new one if one is already active. + +In either case, the server should also set its NEED and ENFORCING flags +at the same time. + +The other surviving cluster siblings should take steps to begin grace +period enforcement as soon as possible. This entails "draining off" any +in-progress state morphing operations and then blocking the acquisition +of any new state (usually with a return of NFS4ERR_GRACE to clients that +attempt it). Again, there is no need for the survivors from the previous +epoch to allow recovery here. + +The surviving servers must however establish a new client recovery +database at this point to ensure that their clients can do recovery in +the event of a crash afterward. + +Once all of the siblings are enforcing the grace period, the recovering +server can then request that the filesystem release the old state, and +allow clients to begin reclaiming their state. In the rados_cluster +backend driver, we do this by stalling server startup until all hosts +in the cluster are enforcing the grace period. + +Lifting the Grace Period +------------------------ +Transitioning from recovery to normal operation really consists of two +different steps: + +1. the server decides that it no longer requires a grace period, either + due to it timing out or there not being any clients that would be + allowed to reclaim. + +2. the server stops enforcing the grace period and transitions to normal + operation + +These concepts are often conflated in a singleton servers, but in
singleton server
+a cluster we must consider them independently. + +When a server is finished with its own local recovery period, it should +clear its NEED flag. That server should continue enforcing the grace +period however until the grace period is fully lifted. + +If the servers' own NEED flag is the last one set, then it can lift the +grace period (by setting R=0). At that point, all servers in the cluster +can end grace period enforcement, and communicate that fact to the +others by clearing their ENFORCING flags. -- 2.17.0 _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
From: Jeff Layton <jlayton@redhat.com>
Bruce asked for better design documentation, so this is my attempt at it. Let me know what you think. I'll probably end up squashing this into one of the code patches but for now I'm sending this separately to see if it helps clarify things.
Suggestions and feedback are welcome.
Change-Id: I53cc77f66b2407c2083638e5760666639ba1fd57 Signed-off-by: Jeff Layton <jlayton@redhat.com> --- src/doc/man/ganesha-rados-cluster.rst | 227 ++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 src/doc/man/ganesha-rados-cluster.rst
diff --git a/src/doc/man/ganesha-rados-cluster.rst b/src/doc/man/ganesha-rados-cluster.rst new file mode 100644 index 000000000000..1ba2d3c29093 --- /dev/null +++ b/src/doc/man/ganesha-rados-cluster.rst @@ -0,0 +1,227 @@ +============================================================================== +ganesha-rados-cluster-design -- Clustered RADOS Recovery Backend Design +============================================================================== + +.. program:: ganesha-rados-cluster-design + +This document aims to explain the theory and design behind the +rados_cluster recovery backend, which coordinates grace period +enforcement among multiple, independent NFS servers. + +In order to understand the clustered recovery backend, it's first necessary +to understand how recovery works with a single server: + +Singleton Server Recovery +------------------------- +NFSv4 is a lease-based protocol. Clients set up a relationship to the +server and must periodically renew their lease in order to maintain +their ephemeral state (open files, locks, delegations or layouts). + +When a singleton NFS server is restarted, any ephemeral state is lost. When +the server comes comes back online, NFS clients detect that the server has +been restarted and will reclaim the ephemeral state that they held at the +time of their last contact with the server. + +Singleton Grace Period +---------------------- + +In order to ensure that we don't end up with conflicts, clients are +barred from acquiring any new state while in the Recovery phase. Only +reclaim operations are allowed. + +This period of time is called the **grace period**. Most NFS servers +have a grace period that lasts around two lease periods, however
knfsd's is one lease period, who does two? (Still catching up on the rest. Looks good.) --b.
On Thu, 2018-05-31 at 17:37 -0400, J. Bruce Fields wrote:
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
From: Jeff Layton <jlayton@redhat.com>
Bruce asked for better design documentation, so this is my attempt at it. Let me know what you think. I'll probably end up squashing this into one of the code patches but for now I'm sending this separately to see if it helps clarify things.
Suggestions and feedback are welcome.
Change-Id: I53cc77f66b2407c2083638e5760666639ba1fd57 Signed-off-by: Jeff Layton <jlayton@redhat.com> --- src/doc/man/ganesha-rados-cluster.rst | 227 ++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 src/doc/man/ganesha-rados-cluster.rst
diff --git a/src/doc/man/ganesha-rados-cluster.rst b/src/doc/man/ganesha-rados-cluster.rst new file mode 100644 index 000000000000..1ba2d3c29093 --- /dev/null +++ b/src/doc/man/ganesha-rados-cluster.rst @@ -0,0 +1,227 @@ +============================================================================== +ganesha-rados-cluster-design -- Clustered RADOS Recovery Backend Design +============================================================================== + +.. program:: ganesha-rados-cluster-design + +This document aims to explain the theory and design behind the +rados_cluster recovery backend, which coordinates grace period +enforcement among multiple, independent NFS servers. + +In order to understand the clustered recovery backend, it's first necessary +to understand how recovery works with a single server: + +Singleton Server Recovery +------------------------- +NFSv4 is a lease-based protocol. Clients set up a relationship to the +server and must periodically renew their lease in order to maintain +their ephemeral state (open files, locks, delegations or layouts). + +When a singleton NFS server is restarted, any ephemeral state is lost. When +the server comes comes back online, NFS clients detect that the server has +been restarted and will reclaim the ephemeral state that they held at the +time of their last contact with the server. + +Singleton Grace Period +---------------------- + +In order to ensure that we don't end up with conflicts, clients are +barred from acquiring any new state while in the Recovery phase. Only +reclaim operations are allowed. + +This period of time is called the **grace period**. Most NFS servers +have a grace period that lasts around two lease periods, however
knfsd's is one lease period, who does two?
(Still catching up on the rest. Looks good.)
--b.
(cc'ing linux-nfs) Thanks for having a look. Hmm...you're right. nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90; nit: We should probably add a #define'd constant for that at some point...but, might this be problematic? In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim. ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though. Even worse: $ cat /proc/sys/fs/lease-break-time 45 Maybe we should be basing the v4 lease time on the lease-break-time value? It seems like we ought to revoke delegations after two lease periods rather than after half of one. -- Jeff Layton <jlayton@kernel.org>
On Fri, Jun 01, 2018 at 06:42:37AM -0400, Jeff Layton wrote:
Thanks for having a look. Hmm...you're right.
nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90;
nit: We should probably add a #define'd constant for that at some point...but, might this be problematic?
In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim.
ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though.
OK, I'd be interested in any of that. If we decide we need a larger grace/lease ratio, we may want to adjust default lease down at same time to avoid increasing grace period, as that's always an annoyance.
Even worse:
$ cat /proc/sys/fs/lease-break-time 45
Maybe we should be basing the v4 lease time on the lease-break-time value? It seems like we ought to revoke delegations after two lease periods rather than after half of one.
No, we ignore lease-break-time on delegations. See the comment in fs/nfsd/nfs4state.c:nfsd_break_deleg_cb(): /* * We don't want the locks code to timeout the lease for us; * we'll remove it ourself if a delegation isn't returned * in time: */ fl->fl_break_time = 0; --b.
On Fri, 2018-06-01 at 10:17 -0400, J. Bruce Fields wrote:
On Fri, Jun 01, 2018 at 06:42:37AM -0400, Jeff Layton wrote:
Thanks for having a look. Hmm...you're right.
nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90;
nit: We should probably add a #define'd constant for that at some point...but, might this be problematic?
In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim.
ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though.
OK, I'd be interested in any of that.
If we decide we need a larger grace/lease ratio, we may want to adjust default lease down at same time to avoid increasing grace period, as that's always an annoyance.
I think we just figured that we ought to give the client a full lease period to reclaim everything, in case it holds a lot of state. You also need to give it a full lease period to figure out that the server has rebooted, and then I think we added a few seconds to account for "other factors" (transport delays and whatnot).
Even worse:
$ cat /proc/sys/fs/lease-break-time 45
Maybe we should be basing the v4 lease time on the lease-break-time value? It seems like we ought to revoke delegations after two lease periods rather than after half of one.
No, we ignore lease-break-time on delegations. See the comment in fs/nfsd/nfs4state.c:nfsd_break_deleg_cb():
/* * We don't want the locks code to timeout the lease for us; * we'll remove it ourself if a delegation isn't returned * in time: */ fl->fl_break_time = 0;
Ahh thanks, I had forgotten that. Whew! -- Jeff Layton <jlayton@kernel.org>
On Fri, Jun 01, 2018 at 10:33:14AM -0400, Jeff Layton wrote:
On Fri, 2018-06-01 at 10:17 -0400, J. Bruce Fields wrote:
On Fri, Jun 01, 2018 at 06:42:37AM -0400, Jeff Layton wrote:
Thanks for having a look. Hmm...you're right.
nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90;
nit: We should probably add a #define'd constant for that at some point...but, might this be problematic?
In the pessimal case, you might renew your lease just before the server crashes. It then comes back up quickly and starts the grace period. By the time the client contacts the server again the grace period is almost over and you may have very little time to actually do any reclaim.
ISTR that when we were working on the server at PD we had determined that we needed around 2 grace periods + a small fudge factor. I don't recall the details of how we determined it though.
OK, I'd be interested in any of that.
If we decide we need a larger grace/lease ratio, we may want to adjust default lease down at same time to avoid increasing grace period, as that's always an annoyance.
I think we just figured that we ought to give the client a full lease period to reclaim everything, in case it holds a lot of state. You also need to give it a full lease period to figure out that the server has rebooted, and then I think we added a few seconds to account for "other factors" (transport delays and whatnot).
I think it would also be easy to extend it on demand. So, for example: end the grace period when: (one lease period has passed *and* we've received no reclaim request in the last second) *or* two lease periods have passed Maybe think about the exact choice of numbers a little. --b.
On Fri, Jun 01, 2018 at 10:46:55AM -0400, J. Bruce Fields wrote:
I think it would also be easy to extend it on demand.
So, for example: end the grace period when:
(one lease period has passed *and* we've received no reclaim request in the last second) *or* two lease periods have passed
Maybe think about the exact choice of numbers a little.
Something like this. (Totally untested.) I also wonder if 90 seconds is overkill as our default lease time. What does anyone else do? Maybe I'll just half it to 45s at the same time. --b. commit 90c471ab0150 Author: J. Bruce Fields <bfields@redhat.com> Date: Fri Jun 8 12:28:47 2018 -0400 nfsd4: extend reclaim period for reclaiming clients If the client is only renewing state a little sooner than once a lease period, then it might not discover the server has restarted till close to the end of the grace period, and might run out of time to do the actual reclaim. Extend the grace period by a second each time we notice there are clients still trying to reclaim, up to a limit of another whole lease period. Signed-off-by: J. Bruce Fields <bfields@redhat.com> diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 36358d435cb0..426f55005697 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -102,6 +102,7 @@ struct nfsd_net { time_t nfsd4_lease; time_t nfsd4_grace; + bool somebody_reclaimed; bool nfsd_net_up; bool lockd_up; diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c index 5d99e8810b85..1929f85b8269 100644 --- a/fs/nfsd/nfs4proc.c +++ b/fs/nfsd/nfs4proc.c @@ -354,6 +354,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, struct svc_fh *resfh = NULL; struct net *net = SVC_NET(rqstp); struct nfsd_net *nn = net_generic(net, nfsd_net_id); + bool reclaim = false; dprintk("NFSD: nfsd4_open filename %.*s op_openowner %p\n", (int)open->op_fname.len, open->op_fname.data, @@ -424,6 +425,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if (status) goto out; open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED; + reclaim = true; case NFS4_OPEN_CLAIM_FH: case NFS4_OPEN_CLAIM_DELEG_CUR_FH: status = do_open_fhandle(rqstp, cstate, open); @@ -452,6 +454,8 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, WARN(status && open->op_created, "nfsd4_process_open2 failed to open newly-created file! status=%u\n", be32_to_cpu(status)); + if (reclaim && !status) + nn->somebody_reclaimed = true; out: if (resfh && resfh != &cstate->current_fh) { fh_dup2(&cstate->current_fh, resfh); diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 59ae65d3eec3..ffe816fe6e89 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -4689,6 +4689,28 @@ nfsd4_end_grace(struct nfsd_net *nn) */ } +/* + * If we've waited a lease period but there are still clients trying to + * reclaim, wait a little longer to give them a chance to finish. + */ +static bool clients_still_reclaiming(struct nfsd_net *nn) +{ + unsigned long now = get_seconds(); + unsigned long double_grace_period_end = nn->boot_time + + 2 * nn->nfsd4_lease; + + if (!nn->somebody_reclaimed) + return false; + nn->somebody_reclaimed = false; + /* + * If we've given them *two* lease times to reclaim, and they're + * still not done, give up: + */ + if (time_after(now, double_grace_period_end)) + return false; + return true; +} + static time_t nfs4_laundromat(struct nfsd_net *nn) { @@ -4702,6 +4724,11 @@ nfs4_laundromat(struct nfsd_net *nn) time_t t, new_timeo = nn->nfsd4_lease; dprintk("NFSD: laundromat service - starting\n"); + + if (clients_still_reclaiming(nn)) { + new_timeo = 0; + goto out; + } nfsd4_end_grace(nn); INIT_LIST_HEAD(&reaplist); spin_lock(&nn->client_lock); @@ -4799,7 +4826,7 @@ nfs4_laundromat(struct nfsd_net *nn) posix_unblock_lock(&nbl->nbl_lock); free_blocked_lock(nbl); } - +out: new_timeo = max_t(time_t, new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT); return new_timeo; } @@ -6049,6 +6076,8 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, case 0: /* success! */ nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid); status = 0; + if (lock->lk_reclaim) + nn->somebody_reclaimed = true; break; case FILE_LOCK_DEFERRED: nbl = NULL; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index d107b4426f7e..5f22476cf371 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1239,6 +1239,7 @@ static __net_init int nfsd_init_net(struct net *net) goto out_idmap_error; nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90; + nn->somebody_reclaimed = false; nn->clverifier_counter = prandom_u32(); nn->clientid_counter = prandom_u32();
On Fri, 2018-06-08 at 12:33 -0400, J. Bruce Fields wrote:
On Fri, Jun 01, 2018 at 10:46:55AM -0400, J. Bruce Fields wrote:
I think it would also be easy to extend it on demand.
So, for example: end the grace period when:
(one lease period has passed *and* we've received no reclaim request in the last second) *or* two lease periods have passed
Maybe think about the exact choice of numbers a little.
Something like this. (Totally untested.)
I also wonder if 90 seconds is overkill as our default lease time. What does anyone else do? Maybe I'll just half it to 45s at the same time.
Yeah, that might not be a bad idea. Worth experimenting with anyway.
--b.
commit 90c471ab0150 Author: J. Bruce Fields <bfields@redhat.com> Date: Fri Jun 8 12:28:47 2018 -0400
nfsd4: extend reclaim period for reclaiming clients
If the client is only renewing state a little sooner than once a lease period, then it might not discover the server has restarted till close to the end of the grace period, and might run out of time to do the actual reclaim.
Extend the grace period by a second each time we notice there are clients still trying to reclaim, up to a limit of another whole lease period.
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
Seems like a reasonable thing to do. Ack from my standpoint.
diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 36358d435cb0..426f55005697 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -102,6 +102,7 @@ struct nfsd_net {
time_t nfsd4_lease; time_t nfsd4_grace; + bool somebody_reclaimed;
bool nfsd_net_up; bool lockd_up; diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c index 5d99e8810b85..1929f85b8269 100644 --- a/fs/nfsd/nfs4proc.c +++ b/fs/nfsd/nfs4proc.c @@ -354,6 +354,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, struct svc_fh *resfh = NULL; struct net *net = SVC_NET(rqstp); struct nfsd_net *nn = net_generic(net, nfsd_net_id); + bool reclaim = false;
I know this is a "best effort" sort of thing, but should this be done with atomic loads and stores (READ_ONCE/WRITE_ONCE)?
dprintk("NFSD: nfsd4_open filename %.*s op_openowner %p\n", (int)open->op_fname.len, open->op_fname.data, @@ -424,6 +425,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if (status) goto out; open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED; + reclaim = true; case NFS4_OPEN_CLAIM_FH: case NFS4_OPEN_CLAIM_DELEG_CUR_FH: status = do_open_fhandle(rqstp, cstate, open); @@ -452,6 +454,8 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, WARN(status && open->op_created, "nfsd4_process_open2 failed to open newly-created file! status=%u\n", be32_to_cpu(status)); + if (reclaim && !status) + nn->somebody_reclaimed = true; out: if (resfh && resfh != &cstate->current_fh) { fh_dup2(&cstate->current_fh, resfh); diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 59ae65d3eec3..ffe816fe6e89 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -4689,6 +4689,28 @@ nfsd4_end_grace(struct nfsd_net *nn) */ }
+/* + * If we've waited a lease period but there are still clients trying to + * reclaim, wait a little longer to give them a chance to finish. + */ +static bool clients_still_reclaiming(struct nfsd_net *nn) +{ + unsigned long now = get_seconds(); + unsigned long double_grace_period_end = nn->boot_time + + 2 * nn->nfsd4_lease; + + if (!nn->somebody_reclaimed) + return false; + nn->somebody_reclaimed = false; + /* + * If we've given them *two* lease times to reclaim, and they're + * still not done, give up: + */ + if (time_after(now, double_grace_period_end)) + return false; + return true; +} + static time_t nfs4_laundromat(struct nfsd_net *nn) { @@ -4702,6 +4724,11 @@ nfs4_laundromat(struct nfsd_net *nn) time_t t, new_timeo = nn->nfsd4_lease;
dprintk("NFSD: laundromat service - starting\n"); + + if (clients_still_reclaiming(nn)) { + new_timeo = 0; + goto out; + } nfsd4_end_grace(nn); INIT_LIST_HEAD(&reaplist); spin_lock(&nn->client_lock); @@ -4799,7 +4826,7 @@ nfs4_laundromat(struct nfsd_net *nn) posix_unblock_lock(&nbl->nbl_lock); free_blocked_lock(nbl); } - +out: new_timeo = max_t(time_t, new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT); return new_timeo; } @@ -6049,6 +6076,8 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, case 0: /* success! */ nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid); status = 0; + if (lock->lk_reclaim) + nn->somebody_reclaimed = true; break; case FILE_LOCK_DEFERRED: nbl = NULL; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index d107b4426f7e..5f22476cf371 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1239,6 +1239,7 @@ static __net_init int nfsd_init_net(struct net *net) goto out_idmap_error; nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90; + nn->somebody_reclaimed = false; nn->clverifier_counter = prandom_u32(); nn->clientid_counter = prandom_u32();
-- Jeff Layton <jlayton@kernel.org>
On Thu, Jun 14, 2018 at 06:01:27AM -0400, Jeff Layton wrote:
I know this is a "best effort" sort of thing, but should this be done with atomic loads and stores (READ_ONCE/WRITE_ONCE)?
We're just setting it to 1 when a reclaim comes in, and then once a second checking the result and clearing it. So yes that's a slightly sloppy way of checking whether something happens once a second. I *think* the worst that could happen is something like: read 0 from somebody_reclaimed reclaim writes 1 to somebody_reclaimed write 0 to somebody_reclaimed and then a client that's reclaiming only very close to the 1-second mark could get overlooked. I'm assuming in any reasonable situation that a client can manage at least 10-100 reclaims a second, and that the race window is probably several orders of magnitude less than a second (even after taking into account weird memory ordering). Would READ/WRITE_ONCE do it right? I'm not sure it's worth it. --b.
dprintk("NFSD: nfsd4_open filename %.*s op_openowner %p\n", (int)open->op_fname.len, open->op_fname.data, @@ -424,6 +425,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if (status) goto out; open->op_openowner->oo_flags |= NFS4_OO_CONFIRMED; + reclaim = true; case NFS4_OPEN_CLAIM_FH: case NFS4_OPEN_CLAIM_DELEG_CUR_FH: status = do_open_fhandle(rqstp, cstate, open); @@ -452,6 +454,8 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, WARN(status && open->op_created, "nfsd4_process_open2 failed to open newly-created file! status=%u\n", be32_to_cpu(status)); + if (reclaim && !status) + nn->somebody_reclaimed = true; out: if (resfh && resfh != &cstate->current_fh) { fh_dup2(&cstate->current_fh, resfh); diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 59ae65d3eec3..ffe816fe6e89 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -4689,6 +4689,28 @@ nfsd4_end_grace(struct nfsd_net *nn) */ }
+/* + * If we've waited a lease period but there are still clients trying to + * reclaim, wait a little longer to give them a chance to finish. + */ +static bool clients_still_reclaiming(struct nfsd_net *nn) +{ + unsigned long now = get_seconds(); + unsigned long double_grace_period_end = nn->boot_time + + 2 * nn->nfsd4_lease; + + if (!nn->somebody_reclaimed) + return false; + nn->somebody_reclaimed = false; + /* + * If we've given them *two* lease times to reclaim, and they're + * still not done, give up: + */ + if (time_after(now, double_grace_period_end)) + return false; + return true; +} + static time_t nfs4_laundromat(struct nfsd_net *nn) { @@ -4702,6 +4724,11 @@ nfs4_laundromat(struct nfsd_net *nn) time_t t, new_timeo = nn->nfsd4_lease;
dprintk("NFSD: laundromat service - starting\n"); + + if (clients_still_reclaiming(nn)) { + new_timeo = 0; + goto out; + } nfsd4_end_grace(nn); INIT_LIST_HEAD(&reaplist); spin_lock(&nn->client_lock); @@ -4799,7 +4826,7 @@ nfs4_laundromat(struct nfsd_net *nn) posix_unblock_lock(&nbl->nbl_lock); free_blocked_lock(nbl); } - +out: new_timeo = max_t(time_t, new_timeo, NFSD_LAUNDROMAT_MINTIMEOUT); return new_timeo; } @@ -6049,6 +6076,8 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, case 0: /* success! */ nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid); status = 0; + if (lock->lk_reclaim) + nn->somebody_reclaimed = true; break; case FILE_LOCK_DEFERRED: nbl = NULL; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index d107b4426f7e..5f22476cf371 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1239,6 +1239,7 @@ static __net_init int nfsd_init_net(struct net *net) goto out_idmap_error; nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90; + nn->somebody_reclaimed = false; nn->clverifier_counter = prandom_u32(); nn->clientid_counter = prandom_u32();
-- Jeff Layton <jlayton@kernel.org>
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+In order to decide when to make grace period transitions, we must also +have each server to advertise its state to the other nodes. Specifically,
"We must also have each server to advertise"/"each server must also advertise"/?
+The surviving servers must however establish a new client recovery +database at this point to ensure that their clients can do recovery in +the event of a crash afterward.
This seems clumsy to me--why should they have to make identical copies of their databases? Is there any reason not to make the epoch numbers per server? --b.
On Thu, 2018-06-07 at 20:33 -0400, J. Bruce Fields wrote:
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+In order to decide when to make grace period transitions, we must also +have each server to advertise its state to the other nodes. Specifically,
"We must also have each server to advertise"/"each server must also advertise"/?
Thanks, will fix.
+The surviving servers must however establish a new client recovery +database at this point to ensure that their clients can do recovery in +the event of a crash afterward.
This seems clumsy to me--why should they have to make identical copies of their databases?
Is there any reason not to make the epoch numbers per server?
I had been thinking about the epoch being a cluster-wide property, but the value itself doesn't matter as much. In principle, I suppose we could make it a per-server property. I'd have to think about how we'd implement it. I suppose we could just keep the epoch values in the omap, and update them all en-masse when there is a change. That's also a bit clumsy though. In practice, it's not as clumsy as you might think. We just batch up the create and OMAP setting as a single operation and it's done atomically. If the host crashes in the middle of that, no problem -- we still have the database from the previous epoch to recovery from. -- Jeff Layton <jlayton@kernel.org>
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+In order to decide when to make grace period transitions, we must also +have each server to advertise its state to the other nodes. Specifically, +we need to allow servers to determine these two things about each of +its siblings: + +1. Does this server have clients from the previous epoch that will require + recovery? (NEED) + +2. Is this server allowing clients to acquire new state? (ENFORCING)
It took me a while to figure that one out: ENFORCING is set when the server does *not* allow new state (only reclaims), right? I might say: 2. Is this server enforcing the grace period by refusing non-reclaim locks? (ENFORCING) --b.
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+Lifting the Grace Period +------------------------ +Transitioning from recovery to normal operation really consists of two +different steps: + +1. the server decides that it no longer requires a grace period, either + due to it timing out or there not being any clients that would be + allowed to reclaim. + +2. the server stops enforcing the grace period and transitions to normal + operation + +These concepts are often conflated in a singleton servers, but in +a cluster we must consider them independently. + +When a server is finished with its own local recovery period, it should +clear its NEED flag. That server should continue enforcing the grace +period however until the grace period is fully lifted.
I'd also state explicitly that a server must not permit reclaims after clearing its NEED flag. (I guess it's implied by #1 above, but there might be a temptation to say "oh I'm still in the grace period, I can be nice and let in this late reclaim". But NEED is your only guarantee that nobody else is granting non-reclaim locks, so once you've cleared it it's no longer safe to accept reclaims.) --b
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+Lifting the Grace Period +------------------------ +Transitioning from recovery to normal operation really consists of two +different steps: + +1. the server decides that it no longer requires a grace period, either + due to it timing out or there not being any clients that would be + allowed to reclaim. + +2. the server stops enforcing the grace period and transitions to normal + operation + +These concepts are often conflated in a singleton servers, but in +a cluster we must consider them independently. + +When a server is finished with its own local recovery period, it should +clear its NEED flag. That server should continue enforcing the grace +period however until the grace period is fully lifted. + +If the servers' own NEED flag is the last one set, then it can lift the +grace period (by setting R=0). At that point, all servers in the cluster +can end grace period enforcement, and communicate that fact to the +others by clearing their ENFORCING flags.
I think this also needs to describe the ordering of the recovery database switch and the epoch increment in the clustered case. For "surviving" servers it doesn't matter since their recovery database isn't changing. For restarting servers, there's a window between clearing NEED and clearing ENFORCING when their recovery database can't change. The epoch musn't change till everybody's created a new recovery database. It must change before anyone grants a new non-reclaim lock, because at that point it's no longer safe to use the older recovery databases. (That could result in allowing a reclaim from a client which conflicts with the new lock.) So I think servers should 1) stop allowing reclaims, 2) create the new recovery database, 3) atomically: clear NEED, check whether they're the last to clear NEED, and bump the epoch, and 4) clear ENFORCING. ?? Do we have a race like this, in a 2-node cluster?: - server 1 clears NEED - server 2 restarts, sets NEED and ENFORCING - server 2 sees that 1 still has ENFORCING set, starts accepting reclaims - server 1 clears ENFORCING, starts accepting non-reclaims. --b.
On Fri, 2018-06-08 at 10:22 -0400, J. Bruce Fields wrote:
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+Lifting the Grace Period +------------------------ +Transitioning from recovery to normal operation really consists of two +different steps: + +1. the server decides that it no longer requires a grace period, either + due to it timing out or there not being any clients that would be + allowed to reclaim. + +2. the server stops enforcing the grace period and transitions to normal + operation + +These concepts are often conflated in a singleton servers, but in +a cluster we must consider them independently. + +When a server is finished with its own local recovery period, it should +clear its NEED flag. That server should continue enforcing the grace +period however until the grace period is fully lifted. + +If the servers' own NEED flag is the last one set, then it can lift the +grace period (by setting R=0). At that point, all servers in the cluster +can end grace period enforcement, and communicate that fact to the +others by clearing their ENFORCING flags.
I think this also needs to describe the ordering of the recovery database switch and the epoch increment in the clustered case.
For "surviving" servers it doesn't matter since their recovery database isn't changing.
For restarting servers, there's a window between clearing NEED and clearing ENFORCING when their recovery database can't change.
The epoch musn't change till everybody's created a new recovery database.
It must change before anyone grants a new non-reclaim lock, because at that point it's no longer safe to use the older recovery databases. (That could result in allowing a reclaim from a client which conflicts with the new lock.)
So I think servers should 1) stop allowing reclaims, 2) create the new recovery database, 3) atomically: clear NEED, check whether they're the last to clear NEED, and bump the epoch, and 4) clear ENFORCING. ??
First, we create the new database as reclaim requests come in, so we don't need to do that at any particular time. Basically, we allow reclaims from the server's recovery DB for R (when R != 0), and we create new records in the DB for C when those requests come in. If the server already has a DB for C when it restarts, then that DB is truncated and recreated from the reclaim requests coming in. A surviving server will need to create a new recovery DB for the new epoch. That's done after we start enforcing the grace period but before marking ourselves as enforcing.
Do we have a race like this, in a 2-node cluster?:
- server 1 clears NEED - server 2 restarts, sets NEED and ENFORCING - server 2 sees that 1 still has ENFORCING set, starts accepting reclaims - server 1 clears ENFORCING, starts accepting non-reclaims.
No. The transactions are done atomically in RADOS. Basically we do: 1) read operation that fetches the object version 2) modify in memory 3) write operation that asserts on the object version If the assertion fails, then we restart the whole cycle again with a new read op. The update of the epochs is also atomic wrt to the flag updates because they're part of the same operation, and RADOS is atomic at the operation level. The first NEED flag being set and last need flag being cleared are done atomically with changes to the epochs. In practice, in the above example, server1 would not clear its E flag, because when it went to do so, it would see that server 2 had set N in the meantime, and would need to maintain keep enforcing its grace period. We always treat the E flag conservatively too. We only set it after we're actually enforcing the grace period locally, and only stop enforcing grace after we've cleared the flag (and thus ensured that no one needs it). -- Jeff Layton <jlayton@kernel.org>
On Fri, 2018-06-08 at 10:22 -0400, J. Bruce Fields wrote:
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
+Lifting the Grace Period +------------------------ +Transitioning from recovery to normal operation really consists of two +different steps: + +1. the server decides that it no longer requires a grace period, either + due to it timing out or there not being any clients that would be + allowed to reclaim. + +2. the server stops enforcing the grace period and transitions to normal + operation + +These concepts are often conflated in a singleton servers, but in +a cluster we must consider them independently. + +When a server is finished with its own local recovery period, it should +clear its NEED flag. That server should continue enforcing the grace +period however until the grace period is fully lifted. + +If the servers' own NEED flag is the last one set, then it can lift the +grace period (by setting R=0). At that point, all servers in the cluster +can end grace period enforcement, and communicate that fact to the +others by clearing their ENFORCING flags.
I think this also needs to describe the ordering of the recovery database switch and the epoch increment in the clustered case.
For "surviving" servers it doesn't matter since their recovery database isn't changing.
For restarting servers, there's a window between clearing NEED and clearing ENFORCING when their recovery database can't change.
The epoch musn't change till everybody's created a new recovery database.
It must change before anyone grants a new non-reclaim lock, because at that point it's no longer safe to use the older recovery databases. (That could result in allowing a reclaim from a client which conflicts with the new lock.)
So I think servers should 1) stop allowing reclaims, 2) create the new recovery database, 3) atomically: clear NEED, check whether they're the last to clear NEED, and bump the epoch, and 4) clear ENFORCING. ??
Do we have a race like this, in a 2-node cluster?:
- server 1 clears NEED - server 2 restarts, sets NEED and ENFORCING - server 2 sees that 1 still has ENFORCING set, starts accepting reclaims - server 1 clears ENFORCING, starts accepting non-reclaims.
After looking over this a bit more, I think there is a potential problem. We're currently starting a new (local) grace period (and setting our own enforcing flag) and then generating the recovery DBs afterward. If we crash between those two events then another node could lift the grace period before this node comes back up and marks its NEED flag. The easy fix is to just create a new recovery DB for the new epoch prior to starting the grace period locally. Lightly tested patch here: https://review.gerrithub.io/#/c/ffilz/nfs-ganesha/+/415232 -- Jeff Layton <jlayton@kernel.org>
participants (3)
-
Daniel Gryniewicz -
J. Bruce Fields -
Jeff Layton