Callbacks supported with 4.1
Hi all, Recently while doing some testing we observed that CB_NOTIFY_LOCK callback does not seem to supported by Ganesha. Is that understanding correct? Also, any insights on the 4.1 callbacks that we do support, would be good to know. The RFC mentions quite a few of them. Thanks and regards, Trishali. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
It appears the implemented callbacks are layout recall, device notify, delegation recall, and getattr. You can check the callpoints of nfs_rpc_cb_single() to see which callbacks are sent. Daniel On 7/1/20 9:13 AM, Trishali Nayar wrote:
Hi all,
Recently while doing some testing we observed that CB_NOTIFY_LOCK callback does not seem to supported by Ganesha. Is that understanding correct?
Also, any insights on the 4.1 callbacks that we do support, would be good to know. The RFC mentions quite a few of them.
Thanks and regards, Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal, that the client will send some kind of cancellation if it gives up waiting for a blocked lock, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime. This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance). Frank
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: Wednesday, July 1, 2020 7:21 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Callbacks supported with 4.1
It appears the implemented callbacks are layout recall, device notify, delegation recall, and getattr. You can check the callpoints of nfs_rpc_cb_single() to see which callbacks are sent.
Daniel
On 7/1/20 9:13 AM, Trishali Nayar wrote:
Hi all,
Recently while doing some testing we observed that CB_NOTIFY_LOCK callback does not seem to supported by Ganesha. Is that understanding correct?
Also, any insights on the 4.1 callbacks that we do support, would be good to know. The RFC mentions quite a few of them.
Thanks and regards, Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Since we don't implement callbacks or fairness, we could just drop locks of type STATE_NFSV4_BLOCKING? The current code might be there to implement callback in future though, just trying to understand the code. Regards, Malahal. On Thu, Jul 2, 2020 at 12:10 AM Frank Filz <ffilzlnx@mindspring.com> wrote:
Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal, that the client will send some kind of cancellation if it gives up waiting for a blocked lock, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime. This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance).
Frank
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: Wednesday, July 1, 2020 7:21 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Callbacks supported with 4.1
It appears the implemented callbacks are layout recall, device notify, delegation recall, and getattr. You can check the callpoints of nfs_rpc_cb_single() to see which callbacks are sent.
Daniel
On 7/1/20 9:13 AM, Trishali Nayar wrote:
Hi all,
Recently while doing some testing we observed that CB_NOTIFY_LOCK callback does not seem to supported by Ganesha. Is that understanding correct?
Also, any insights on the 4.1 callbacks that we do support, would be good to know. The RFC mentions quite a few of them.
Thanks and regards, Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
I meant to say "no need to put them in a list in a conflict case" On Thu, Jul 2, 2020 at 12:36 AM Malahal Naineni <malahal@gmail.com> wrote:
Since we don't implement callbacks or fairness, we could just drop locks of type STATE_NFSV4_BLOCKING? The current code might be there to implement callback in future though, just trying to understand the code.
Regards, Malahal.
On Thu, Jul 2, 2020 at 12:10 AM Frank Filz <ffilzlnx@mindspring.com> wrote:
Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal, that the client will send some kind of cancellation if it gives up waiting for a blocked lock, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime. This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance).
Frank
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: Wednesday, July 1, 2020 7:21 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: Callbacks supported with 4.1
It appears the implemented callbacks are layout recall, device notify, delegation recall, and getattr. You can check the callpoints of nfs_rpc_cb_single() to see which callbacks are sent.
Daniel
On 7/1/20 9:13 AM, Trishali Nayar wrote:
Hi all,
Recently while doing some testing we observed that CB_NOTIFY_LOCK callback does not seem to supported by Ganesha. Is that understanding correct?
Also, any insights on the 4.1 callbacks that we do support, would be good to know. The RFC mentions quite a few of them.
Thanks and regards, Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
You can’t send a notification without having some memory of the fact that a blocking lock was requested. I think the notification is just a hint that the lock might be available, so you could just remember the clients that were looking for the lock and send out notifications and erase their “wait”. The lock would go to whoever asked for it next but the notification can short cut the polling delay the client otherwise has to use. I suppose you don’t have to put any kind of courtesy hold on the lock and just use the notifications. In any case there is still a resource that must be time managed. And it does require at least the lock being held momentarily in the back end unless the backend can also support lock availability notification (GPFS does have a notify upcall but I don’t know if there’s a way to ask GPFS just for a notify instead of a grant). Frank From: Malahal Naineni [mailto:malahal@gmail.com] Sent: Wednesday, July 1, 2020 12:08 PM To: Frank Filz <ffilzlnx@mindspring.com> Cc: nfs-ganesha <devel@lists.nfs-ganesha.org> Subject: Re: [NFS-Ganesha-Devel] Re: Callbacks supported with 4.1 I meant to say "no need to put them in a list in a conflict case" On Thu, Jul 2, 2020 at 12:36 AM Malahal Naineni <malahal@gmail.com <mailto:malahal@gmail.com> > wrote: Since we don't implement callbacks or fairness, we could just drop locks of type STATE_NFSV4_BLOCKING? The current code might be there to implement callback in future though, just trying to understand the code. Regards, Malahal. On Thu, Jul 2, 2020 at 12:10 AM Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > wrote: Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal, that the client will send some kind of cancellation if it gives up waiting for a blocked lock, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime. This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance). Frank
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com <mailto:dang@redhat.com> ] Sent: Wednesday, July 1, 2020 7:21 AM To: devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: Callbacks supported with 4.1
It appears the implemented callbacks are layout recall, device notify, delegation recall, and getattr. You can check the callpoints of nfs_rpc_cb_single() to see which callbacks are sent.
Daniel
On 7/1/20 9:13 AM, Trishali Nayar wrote:
Hi all,
Recently while doing some testing we observed that CB_NOTIFY_LOCK callback does not seem to supported by Ganesha. Is that understanding correct?
Also, any insights on the 4.1 callbacks that we do support, would be good to know. The RFC mentions quite a few of them.
Thanks and regards, Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Trishali Nayar IBM Systems ETZ, Pune. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org>
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org>
Devel mailing list -- devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org <mailto:devel-leave@lists.nfs-ganesha.org>
On Wed, 2020-07-01 at 11:39 -0700, Frank Filz wrote:
Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal , that the client will send some kind of cancellation if it gives up waiting for a blocked lock
There's nothing guaranteed. The client can always just drop off the net. In knfsd, we just assume that the lock will be revisited at least once each lease period, and just drop the blocked request if they aren't. The Linux kernel client will still re-poll for the lock every 30s if it doesn't get a notification.
, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime.
A v4.1 CB_NOTIFY_LOCK is _completely_ advisory. It just says "this is a good time to retry that lock you wanted". There's never a guarantee that you'll get it and the server is under no obligation to hold it for the client being notified. This is fundamentally different from (and simpler than) NLM.
This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance).
I think it's probably possible to do it with some FSALs. For example, it'd probably be doable in libcephfs and then we could wire that into nfs-ganesha. FSAL_VFS (and similar) might actually be the toughest place to make it work, since the kernel would need new interfaces for it. -- Jeff Layton <jlayton@poochiereds.net>
On Wed, 2020-07-01 at 11:39 -0700, Frank Filz wrote:
Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal , that the client will send some kind of cancellation if it gives up waiting for a blocked lock
There's nothing guaranteed. The client can always just drop off the net. In knfsd, we just assume that the lock will be revisited at least once each lease period, and just drop the blocked request if they aren't. The Linux kernel client will still re-poll for the lock every 30s if it doesn't get a notification.
, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime.
A v4.1 CB_NOTIFY_LOCK is _completely_ advisory. It just says "this is a good time to retry that lock you wanted". There's never a guarantee that you'll get it and the server is under no obligation to hold it for the client being notified. This is fundamentally different from (and simpler than) NLM.
Ok, so the 4.1 blocks would have to be maintained on a list in order of most recently polled. Then the reaper thread would look at the LRU of the list and release any wait for notification entries that have expired until it hits one that hasn't expired.
This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance).
I think it's probably possible to do it with some FSALs. For example, it'd probably be doable in libcephfs and then we could wire that into nfs-ganesha. FSAL_VFS (and similar) might actually be the toughest place to make it work, since the kernel would need new interfaces for it.
Yea, I assume it would be straightforward to implement in Ceph, tying somehow into the caps logic. And yea, FSAL_VFS would either have to modify the kernel, or it could just wait on a blocked lock, and then when the blocked lock is granted, immediately drop it and send a notification. Frank
On Wed, 2020-07-01 at 14:14 -0700, Frank Filz wrote:
On Wed, 2020-07-01 at 11:39 -0700, Frank Filz wrote:
Yea, the notify lock callback is not supported, that would take some effort to implement non-blocking locks for NFS v4.1 which is complicated in that I'm not sure if the callback is a guaranteed deal , that the client will send some kind of cancellation if it gives up waiting for a blocked lock
There's nothing guaranteed. The client can always just drop off the net. In knfsd, we just assume that the lock will be revisited at least once each lease period, and just drop the blocked request if they aren't. The Linux kernel client will still re-poll for the lock every 30s if it doesn't get a notification.
, If not, then we need to time out 4.x blocked locks. We also have to make them provisional until the client actually comes back with a new lock request, which means they COULD be handed out to another client in the meantime.
A v4.1 CB_NOTIFY_LOCK is _completely_ advisory. It just says "this is a good time to retry that lock you wanted". There's never a guarantee that you'll get it and the server is under no obligation to hold it for the client being notified. This is fundamentally different from (and simpler than) NLM.
Ok, so the 4.1 blocks would have to be maintained on a list in order of most recently polled. Then the reaper thread would look at the LRU of the list and release any wait for notification entries that have expired until it hits one that hasn't expired.
Yep -- that's more or less what we do in knfsd.
This provisional (or courtesy) behavior may be hard to assure in a cluster or even a single node where there are other processes using locking on the same file system (since the courtesy lock would be a normal held lock on the backend - Ganesha can only give a conflicting lock out and cancel the courtesy lock for locks that come through that Ganesha instance).
I think it's probably possible to do it with some FSALs. For example, it'd probably be doable in libcephfs and then we could wire that into nfs-ganesha. FSAL_VFS (and similar) might actually be the toughest place to make it work, since the kernel would need new interfaces for it.
Yea, I assume it would be straightforward to implement in Ceph, tying somehow into the caps logic. And yea, FSAL_VFS would either have to modify the kernel, or it could just wait on a blocked lock, and then when the blocked lock is granted, immediately drop it and send a notification.
Technically, file locking is handled outside the caps subsystem, but it should be possible to do something that we could plug ganesha into. Not saying it'll be _simple_ but that's a lot simpler to deal with that than new kernel interfaces. :) -- Jeff Layton <jlayton@poochiereds.net>
participants (5)
-
Daniel Gryniewicz -
Frank Filz -
Jeff Layton -
Malahal Naineni -
Trishali Nayar