2:45 p.m.
On Fri, 2018-06-08 at 10:22 -0400, J. Bruce Fields wrote:
On Wed, May 23, 2018 at 08:21:40AM -0400, Jeff Layton wrote:
> +Lifting the Grace Period
> +------------------------
> +Transitioning from recovery to normal operation really consists of two
> +different steps:
> +
> +1. the server decides that it no longer requires a grace period, either
> + due to it timing out or there not being any clients that would be
> + allowed to reclaim.
> +
> +2. the server stops enforcing the grace period and transitions to normal
> + operation
> +
> +These concepts are often conflated in a singleton servers, but in
> +a cluster we must consider them independently.
> +
> +When a server is finished with its own local recovery period, it should
> +clear its NEED flag. That server should continue enforcing the grace
> +period however until the grace period is fully lifted.
> +
> +If the servers' own NEED flag is the last one set, then it can lift the
> +grace period (by setting R=0). At that point, all servers in the cluster
> +can end grace period enforcement, and communicate that fact to the
> +others by clearing their ENFORCING flags.
I think this also needs to describe the ordering of the recovery
database switch and the epoch increment in the clustered case.
For "surviving" servers it doesn't matter since their recovery database
isn't changing.
For restarting servers, there's a window between clearing NEED and
clearing ENFORCING when their recovery database can't change.
The epoch musn't change till everybody's created a new recovery
database.
It must change before anyone grants a new non-reclaim lock, because at
that point it's no longer safe to use the older recovery databases.
(That could result in allowing a reclaim from a client which conflicts
with the new lock.)
So I think servers should 1) stop allowing reclaims, 2) create the new
recovery database, 3) atomically: clear NEED, check whether they're the
last to clear NEED, and bump the epoch, and 4) clear ENFORCING. ??
First, we create the new database as reclaim requests come in, so we
don't need to do that at any particular time. Basically, we allow
reclaims from the server's recovery DB for R (when R != 0), and we
create new records in the DB for C when those requests come in.
If the server already has a DB for C when it restarts, then that DB is
truncated and recreated from the reclaim requests coming in.
A surviving server will need to create a new recovery DB for the new
epoch. That's done after we start enforcing the grace period but before
marking ourselves as enforcing.
Do we have a race like this, in a 2-node cluster?:
- server 1 clears NEED
- server 2 restarts, sets NEED and ENFORCING
- server 2 sees that 1 still has ENFORCING set, starts accepting
reclaims
- server 1 clears ENFORCING, starts accepting non-reclaims.
No. The transactions are done atomically in RADOS. Basically we do:
1) read operation that fetches the object version
2) modify in memory
3) write operation that asserts on the object version
If the assertion fails, then we restart the whole cycle again with a new
read op. The update of the epochs is also atomic wrt to the flag updates
because they're part of the same operation, and RADOS is atomic at the
operation level.
The first NEED flag being set and last need flag being cleared are done
atomically with changes to the epochs.
In practice, in the above example, server1 would not clear its E flag,
because when it went to do so, it would see that server 2 had set N in
the meantime, and would need to maintain keep enforcing its grace
period.
We always treat the E flag conservatively too. We only set it after
we're actually enforcing the grace period locally, and only stop
enforcing grace after we've cleared the flag (and thus ensured that no
one needs it).
--
Jeff Layton <jlayton(a)kernel.org>