Re: Better interop for NFS/SMB file share mode/reservation
On Tue, Mar 05, 2019 at 04:47:48PM -0500, J. Bruce Fields wrote:
On Thu, Feb 14, 2019 at 04:06:52PM -0500, J. Bruce Fields wrote:
After this:
https://marc.info/?l=linux-nfs&m=154966239918297&w=2
delegations would no longer conflict with opens from the same tgid. So if your threads all run in the same process and you're willing to manage conflicts among your own clients, that should still allow you to do multiple opens of the same file without giving up your lease/delegation.
I'd be curious to know whether that works with Samba's design.
Any idea whether that would work?
(Easy? Impossible? Possible, but realistically the changes required to Samba would be painful enough that it'd be unlikely to get done?)
Volker reminds me off-list that he'd like to see Ganesha and Samba work out an API in userspace first before commiting to a user<->kernel API. Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed? In any case, my immediate goal is just to get knfsd fixed, which doesn't really commit us to anything--knfsd only needs kernel internal interfaces. But it'd be nice to have at least some idea if we're on the right track, to save having to redo that work later. --b.
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
On Tue, Mar 05, 2019 at 04:47:48PM -0500, J. Bruce Fields wrote:
On Thu, Feb 14, 2019 at 04:06:52PM -0500, J. Bruce Fields wrote:
After this:
https://marc.info/?l=linux-nfs&m=154966239918297&w=2
delegations would no longer conflict with opens from the same tgid. So if your threads all run in the same process and you're willing to manage conflicts among your own clients, that should still allow you to do multiple opens of the same file without giving up your lease/delegation.
I'd be curious to know whether that works with Samba's design.
Any idea whether that would work?
(Easy? Impossible? Possible, but realistically the changes required to Samba would be painful enough that it'd be unlikely to get done?)
Volker reminds me off-list that he'd like to see Ganesha and Samba work out an API in userspace first before commiting to a user<->kernel API.
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff: https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s... It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
In any case, my immediate goal is just to get knfsd fixed, which doesn't really commit us to anything--knfsd only needs kernel internal interfaces. But it'd be nice to have at least some idea if we're on the right track, to save having to redo that work later.
-- Jeff Layton <jlayton@kernel.org>
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
On Tue, Mar 05, 2019 at 04:47:48PM -0500, J. Bruce Fields wrote:
On Thu, Feb 14, 2019 at 04:06:52PM -0500, J. Bruce Fields wrote:
After this:
https://marc.info/?l=linux-nfs&m=154966239918297&w=2
delegations would no longer conflict with opens from the same tgid. So if your threads all run in the same process and you're willing to manage conflicts among your own clients, that should still allow you to do multiple opens of the same file without giving up your lease/delegation.
I'd be curious to know whether that works with Samba's design.
Any idea whether that would work?
(Easy? Impossible? Possible, but realistically the changes required to Samba would be painful enough that it'd be unlikely to get done?)
Volker reminds me off-list that he'd like to see Ganesha and Samba work out an API in userspace first before commiting to a user<->kernel API.
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases. Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease. Here are the semantics: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943... I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
Jeremy Allison wrote:
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases.
Just to be precise: SMB2.1+ has R, RH, RW and RWH leases.
Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.
That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request
Here are the semantics:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943...
I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
Yes, that would be ideal. -slow
Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical:
Jeremy Allison wrote:
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases.
Just to be precise: SMB2.1+ has R, RH, RW and RWH leases.
Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.
That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request
Here are the semantics:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943...
I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
Yes, that would be ideal.
If we want to design an useful API, we also need to think about all features: - file oplock/leases - directory leases - share modes - disconnected handles (for durable and persistent handles), which exists within the kernel for a while and can be reattached to process, using some kind of cookie and the same euid - the API needs ways to use epoll in order to do async opens and lease breaks. For opens the model of async socket connects could be used. Leases could have a signalfd-style api. We may not need everything at once, but we should have the full picture in mind. And we need working code in kernel and userspace that passes all tests (we may need to add additional test). Otherwise the kernel creates new syscalls, which wouldn't be used by Samba in the end. metze
On Thu, 2019-03-07 at 12:03 +0100, Stefan Metzmacher via samba- technical wrote:
Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical:
Jeremy Allison wrote:
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases.
Just to be precise: SMB2.1+ has R, RH, RW and RWH leases.
Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.
That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request
Here are the semantics:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943...
I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
Yes, that would be ideal.
If we want to design an useful API, we also need to think about all features: - file oplock/leases - directory leases - share modes - disconnected handles (for durable and persistent handles), which exists within the kernel for a while and can be reattached to process, using some kind of cookie and the same euid - the API needs ways to use epoll in order to do async opens and lease breaks. For opens the model of async socket connects could be used. Leases could have a signalfd-style api.
We may not need everything at once, but we should have the full picture in mind. And we need working code in kernel and userspace that passes all tests (we may need to add additional test). Otherwise the kernel creates new syscalls, which wouldn't be used by Samba in the end.
Just a thought, but you should probably classify these facilities in two lists, one for items that can only reasonably be done via a kernel API and one for items that can be satisfactorily be handled via a coordinating userspace component (daemon/database/convention/other). Getting all that stuff in kernel may prove overly hard and contentious so being able to negotiate on the critical items only may be important. Simo.
On Thu, Mar 7, 2019 at 1:04 PM Stefan Metzmacher <metze@samba.org> wrote:
Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical:
Jeremy Allison wrote:
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases.
Just to be precise: SMB2.1+ has R, RH, RW and RWH leases.
Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.
That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request
Here are the semantics:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943...
I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
Yes, that would be ideal.
Getting back to this. Thanks all for the valuable inputs. Next week is LSF/MM and I was assigned a 30 minute slot on filesystems track to discuss "NFS/SMB file share". So let me try to echo what I read on this thread and how I understand what APIs samba needs from the kernel.
If we want to design an useful API, we also need to think about all features: - file oplock/leases
Kernel can have a flavor of leases which are not broken by opens from threads of the process holding the lease. Bruce has some patches along those lines for knfsd and SMB R/RW leases could use this flavor if it was exported to userspace? For SMB RH/RWH leases and Ganesha delegations, server could keep track of its own handles/clients and break leases within the same process without involving the kernel. Am I wrong?
- directory leases
I have WIP on fsnotify directory pre modification hooks. There is opposition from fsnotify maintainer to add new userspace APIs that can create kernel->user->kernel deadlocks, like the deadlocks currently reported with fanotify permission events. Need to see if we can find a middle ground between "post modification notifications" and "pre modification permission" API, somewhere along the lines of regular file lease breaking API.
- share modes
Volker told me he thinks samba can enforce share modes by a single daemon policing all opens in the system with fanotify. I think he is right. If anyone thinks differently please speak up.
- disconnected handles (for durable and persistent handles), which exists within the kernel for a while and can be reattached to process, using some kind of cookie and the same euid
So this interface exists in the kernel. Nothing more required from the kernel API. Right?
- the API needs ways to use epoll in order to do async opens and lease breaks. For opens the model of async socket connects could be used. Leases could have a signalfd-style api.
I should hope that the new AIO API (http://kernel.dk/io_uring.pdf) would solve those problems as well as other issues that samba has w.r.t dispatching AIO.
We may not need everything at once, but we should have the full picture in mind. And we need working code in kernel and userspace that passes all tests (we may need to add additional test). Otherwise the kernel creates new syscalls, which wouldn't be used by Samba in the end.
Tested interfaces - good idea ;-) If anyone has any comments about my view of required new interfaces, or important things that I missed, please say so before Tuesday! Thanks, Amir.
[dropping linux-fsdevel] On Thu, Apr 25, 2019 at 9:11 PM Amir Goldstein <amir73il@gmail.com> wrote:
On Thu, Mar 7, 2019 at 1:04 PM Stefan Metzmacher <metze@samba.org> wrote:
Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical:
Jeremy Allison wrote:
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases.
Just to be precise: SMB2.1+ has R, RH, RW and RWH leases.
Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.
That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request
Here are the semantics:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943...
I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
Yes, that would be ideal.
Getting back to this. Thanks all for the valuable inputs.
Next week is LSF/MM and I was assigned a 30 minute slot on filesystems track to discuss "NFS/SMB file share".
So let me try to echo what I read on this thread and how I understand what APIs samba needs from the kernel.
If we want to design an useful API, we also need to think about all features: - file oplock/leases
Kernel can have a flavor of leases which are not broken by opens from threads of the process holding the lease. Bruce has some patches along those lines for knfsd and SMB R/RW leases could use this flavor if it was exported to userspace?
For SMB RH/RWH leases and Ganesha delegations, server could keep track of its own handles/clients and break leases within the same process without involving the kernel. Am I wrong?
- directory leases
I have WIP on fsnotify directory pre modification hooks. There is opposition from fsnotify maintainer to add new userspace APIs that can create kernel->user->kernel deadlocks, like the deadlocks currently reported with fanotify permission events.
Need to see if we can find a middle ground between "post modification notifications" and "pre modification permission" API, somewhere along the lines of regular file lease breaking API.
- share modes
Volker told me he thinks samba can enforce share modes by a single daemon policing all opens in the system with fanotify. I think he is right. If anyone thinks differently please speak up.
- disconnected handles (for durable and persistent handles), which exists within the kernel for a while and can be reattached to process, using some kind of cookie and the same euid
So this interface exists in the kernel. Nothing more required from the kernel API. Right?
- the API needs ways to use epoll in order to do async opens and lease breaks. For opens the model of async socket connects could be used. Leases could have a signalfd-style api.
I should hope that the new AIO API (http://kernel.dk/io_uring.pdf) would solve those problems as well as other issues that samba has w.r.t dispatching AIO.
We may not need everything at once, but we should have the full picture in mind. And we need working code in kernel and userspace that passes all tests (we may need to add additional test). Otherwise the kernel creates new syscalls, which wouldn't be used by Samba in the end.
Tested interfaces - good idea ;-)
If anyone has any comments about my view of required new interfaces, or important things that I missed, please say so before Tuesday!
Hello Samba-team, Some of you may have already seen the reports from my session at LSF/MM on Samba/NFS interop: https://lwn.net/Articles/788335/ It should not be a surprise to anyone here to know that I have had interesting and productive conversations with NFS folks about improving samba interop. It should not be a surprise to anyone here to know that the rest of the audience was, generally speaking, uninterested in the problem. Which provides a re-enforcement to the point I was trying to make in session - The path of least resistance for NFS-Samba interop is the communicate with each other (both human and software wise) and try to leave VFS out of the discussion for as much as possible (hence dropping linux-fsdevel from this thread). An idea that has already been thrown around is to use some samba daemon as an arbitrator for opening files and locks. Of course, this would be an opt-in feature for NFS servers. For example, can we use fanotify permission hooks to delegate access control checks from knfsd to a daemon? Right now, the information in permission events is rather minimal, but as an fanotify developer, I can assure you, that we can enrich the information passed by knfsd on open permission events if that is deemed useful. I will be attending SambaXP, so if any of the Samba guys would like to, we could find a slot in the Hallway track or at a local bar to discuss those options. Thanks, Amir.
On Fri, May 24, 2019 at 10:12:10AM +0300, Amir Goldstein wrote:
I will be attending SambaXP, so if any of the Samba guys would like to, we could find a slot in the Hallway track or at a local bar to discuss those options.
awesome! I'll join as well. Looking forward to see you at SambaXP! -slow -- Ralph Boehme, Samba Team https://samba.org/ Samba Developer, SerNet GmbH https://sernet.de/en/samba/ GPG-Fingerprint FAE2C6088A24252051C559E4AA1E9B7126399E46
On Fri, May 24, 2019 at 10:12:10AM +0300, Amir Goldstein wrote:
Some of you may have already seen the reports from my session at LSF/MM on Samba/NFS interop: https://lwn.net/Articles/788335/
It should not be a surprise to anyone here to know that I have had interesting and productive conversations with NFS folks about improving samba interop. It should not be a surprise to anyone here to know that the rest of the audience was, generally speaking, uninterested in the problem.
Eh, especially after a couple days of highly technical talks and people have trouble focusing on stuff outside their area. I wouldn't take that as opposition, if that's what you mean. I think the only place where there's any entrenched opposition is (alas) ACLs. Lease/lock stuff, for example, should be no problem. It's mainly just a matter of people finding time.
Which provides a re-enforcement to the point I was trying to make in session - The path of least resistance for NFS-Samba interop is the communicate with each other (both human and software wise) and try to leave VFS out of the discussion for as much as possible (hence dropping linux-fsdevel from this thread).
I've got a strong preference for doing stuff in the VFS. Maybe the approaches aren't incompatible--if we can do something without new kernel interfaces for now, it doesn't rule out later moving some of the logic into the kernel if that helps. That said, I'm not comfortable depending on an assumption that knfsd and SMB are the only users of a filesystem. If we're going to introduce some new kind of lock, for example, I'd like it enforced against everyone. In knfsd, we broke that rule for open deny modes and I think it was a mistake. --b.
An idea that has already been thrown around is to use some samba daemon as an arbitrator for opening files and locks. Of course, this would be an opt-in feature for NFS servers.
For example, can we use fanotify permission hooks to delegate access control checks from knfsd to a daemon? Right now, the information in permission events is rather minimal, but as an fanotify developer, I can assure you, that we can enrich the information passed by knfsd on open permission events if that is deemed useful.
I will be attending SambaXP, so if any of the Samba guys would like to, we could find a slot in the Hallway track or at a local bar to discuss those options.
On Wed, 2019-03-06 at 13:07 -0800, Jeremy Allison wrote:
On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote:
On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote:
On Tue, Mar 05, 2019 at 04:47:48PM -0500, J. Bruce Fields wrote:
On Thu, Feb 14, 2019 at 04:06:52PM -0500, J. Bruce Fields wrote:
After this:
https://marc.info/?l=linux-nfs&m=154966239918297&w=2
delegations would no longer conflict with opens from the same tgid. So if your threads all run in the same process and you're willing to manage conflicts among your own clients, that should still allow you to do multiple opens of the same file without giving up your lease/delegation.
I'd be curious to know whether that works with Samba's design.
Any idea whether that would work?
(Easy? Impossible? Possible, but realistically the changes required to Samba would be painful enough that it'd be unlikely to get done?)
Volker reminds me off-list that he'd like to see Ganesha and Samba work out an API in userspace first before commiting to a user<->kernel API.
Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?
Here's the C headers for that stuff:
https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287bd/s...
It's simple enough and works for us in ganesha, and I think we can probably adapt it to samba without too much difficulty. The callback doesn't seem like it'll do for a kernel API though -- you'd almost certainly need to do something different there (signals? inotify?).
SMB3 leases have R/RW and Handle-based leases.
Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.
Here are the semantics:
https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df943...
I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?
The libcephfs API takes an opaque void * that could hold such a token. It gets passed back to the callback function when it's called to handle a delegation break. I could envision a delegation storing something similar (maybe an unsigned long or uint64_t) and pass it back in a delegation break. With that you could set multiple delegations on a fd, and use that to store a key for each one. Would something like that work for samba, or am I misunderstanding what it needs? -- Jeff Layton <jlayton@kernel.org>
participants (8)
-
Amir Goldstein -
J. Bruce Fields -
Jeff Layton -
Jeremy Allison -
Ralph Boehme -
Ralph Böhme
-
Simo -
Stefan Metzmacher