On Tue, Mar 05, 2019 at 04:47:48PM -0500, J. Bruce Fields wrote: > On Thu, Feb 14, 2019 at 04:06:52PM -0500, J. Bruce Fields wrote: > > After this: > > > > https://marc.info/?l=linux-nfs&m=154966239918297&w=2 > > > > delegations would no longer conflict with opens from the same tgid. So > > if your threads all run in the same process and you're willing to manage > > conflicts among your own clients, that should still allow you to do > > multiple opens of the same file without giving up your lease/delegation. > > > > I'd be curious to know whether that works with Samba's design. > > Any idea whether that would work? > > (Easy? Impossible? Possible, but realistically the changes required to > Samba would be painful enough that it'd be unlikely to get done?) Volker reminds me off-list that he'd like to see Ganesha and Samba work out an API in userspace first before commiting to a user<->kernel API. Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation API? Is that close to what's needed?

In any case, my immediate goal is just to get knfsd fixed, which doesn't really commit us to anything--knfsd only needs kernel internal interfaces. But it'd be nice to have at least some idea if we're on the right track, to save having to redo that work later.

On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote: > On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote: >> >> Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation >> API? Is that close to what's needed? >> > > Here's the C headers for that stuff: > > https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287b... > > It's simple enough and works for us in ganesha, and I think we can > probably adapt it to samba without too much difficulty. The callback > doesn't seem like it'll do for a kernel API though -- you'd almost > certainly need to do something different there (signals? inotify?). SMB3 leases have R/RW and Handle-based leases.

Handle leases allow multiple opens of the same pathname that get different handles to share the lease, allowing a client redirector to delay opens or closes locally so long as it has a handle lease.

Here are the semantics: https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df... I'm not sure a simple file-descriptor based API is enough for us. Can he have a uuid or token based API instead where the server can chose what fd's to cover with a token ?

Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical: > Jeremy Allison wrote: > > On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote: > > > On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote: > > > > Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation > > > > API? Is that close to what's needed? > > > > > > > > > > Here's the C headers for that stuff: > > > > > > https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287b... > > > > > > It's simple enough and works for us in ganesha, and I think we can > > > probably adapt it to samba without too much difficulty. The callback > > > doesn't seem like it'll do for a kernel API though -- you'd almost > > > certainly need to do something different there (signals? inotify?). > > > > SMB3 leases have R/RW and Handle-based leases. > > Just to be precise: SMB2.1+ has R, RH, RW and RWH leases. > > > Handle leases allow multiple opens of the same pathname > > that get different handles to share the lease, allowing > > a client redirector to delay opens or closes locally > > so long as it has a handle lease. > > That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request > > > Here are the semantics: > > > > https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df... > > > > I'm not sure a simple file-descriptor based API is > > enough for us. Can he have a uuid or token based > > API instead where the server can chose what fd's > > to cover with a token ? > > Yes, that would be ideal. If we want to design an useful API, we also need to think about all features: - file oplock/leases - directory leases - share modes - disconnected handles (for durable and persistent handles), which exists within the kernel for a while and can be reattached to process, using some kind of cookie and the same euid - the API needs ways to use epoll in order to do async opens and lease breaks. For opens the model of async socket connects could be used. Leases could have a signalfd-style api. We may not need everything at once, but we should have the full picture in mind. And we need working code in kernel and userspace that passes all tests (we may need to add additional test). Otherwise the kernel creates new syscalls, which wouldn't be used by Samba in the end.

On Thu, Mar 7, 2019 at 1:04 PM Stefan Metzmacher <metze(a)samba.org&gt; wrote: > > Am 06.03.19 um 22:25 schrieb Ralph Böhme via samba-technical: > > > > Jeremy Allison wrote: > >> On Wed, Mar 06, 2019 at 03:31:08PM -0500, Jeff Layton wrote: > >>> On Wed, 2019-03-06 at 10:11 -0500, J. Bruce Fields wrote: > >>>> > >>>> Jeff, wasn't there some work (on Ceph maybe?) on a userspace delegation > >>>> API? Is that close to what's needed? > >>>> > >>> > >>> Here's the C headers for that stuff: > >>> > >>> https://github.com/ceph/ceph/blob/7ba6bece4187eda5d05a9b84211fe6ba8dd287b... > >>> > >>> It's simple enough and works for us in ganesha, and I think we can > >>> probably adapt it to samba without too much difficulty. The callback > >>> doesn't seem like it'll do for a kernel API though -- you'd almost > >>> certainly need to do something different there (signals? inotify?). > >> > >> SMB3 leases have R/RW and Handle-based leases. > > > > Just to be precise: SMB2.1+ has R, RH, RW and RWH leases. > > > >> Handle leases allow multiple opens of the same pathname > >> that get different handles to share the lease, allowing > >> a client redirector to delay opens or closes locally > >> so long as it has a handle lease. > > > > That'a a propertly of leases in general, not just H-leases. The client provides a lease key which is a GUID with each lease request > > > >> > >> Here are the semantics: > >> > >> https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-smb2/d8df... > >> > >> I'm not sure a simple file-descriptor based API is > >> enough for us. Can he have a uuid or token based > >> API instead where the server can chose what fd's > >> to cover with a token ? > > > > Yes, that would be ideal. Getting back to this. Thanks all for the valuable inputs. Next week is LSF/MM and I was assigned a 30 minute slot on filesystems track to discuss "NFS/SMB file share". So let me try to echo what I read on this thread and how I understand what APIs samba needs from the kernel. > > If we want to design an useful API, we also need to think about > all features: > - file oplock/leases Kernel can have a flavor of leases which are not broken by opens from threads of the process holding the lease. Bruce has some patches along those lines for knfsd and SMB R/RW leases could use this flavor if it was exported to userspace? For SMB RH/RWH leases and Ganesha delegations, server could keep track of its own handles/clients and break leases within the same process without involving the kernel. Am I wrong? > - directory leases I have WIP on fsnotify directory pre modification hooks. There is opposition from fsnotify maintainer to add new userspace APIs that can create kernel->user->kernel deadlocks, like the deadlocks currently reported with fanotify permission events. Need to see if we can find a middle ground between "post modification notifications" and "pre modification permission" API, somewhere along the lines of regular file lease breaking API. > - share modes Volker told me he thinks samba can enforce share modes by a single daemon policing all opens in the system with fanotify. I think he is right. If anyone thinks differently please speak up. > - disconnected handles (for durable and persistent handles), > which exists within the kernel for a while and can be reattached > to process, using some kind of cookie and the same euid So this interface exists in the kernel. Nothing more required from the kernel API. Right? > - the API needs ways to use epoll in order to do async opens > and lease breaks. For opens the model of async socket connects > could be used. Leases could have a signalfd-style api. I should hope that the new AIO API (http://kernel.dk/io_uring.pdf) would solve those problems as well as other issues that samba has w.r.t dispatching AIO. > > We may not need everything at once, but we should have the full picture > in mind. And we need working code in kernel and userspace that passes > all tests (we may need to add additional test). Otherwise the kernel > creates new syscalls, which wouldn't be used by Samba in the end. > Tested interfaces - good idea ;-) If anyone has any comments about my view of required new interfaces, or important things that I missed, please say so before Tuesday!

Some of you may have already seen the reports from my session at LSF/MM on Samba/NFS interop: https://lwn.net/Articles/788335/ It should not be a surprise to anyone here to know that I have had interesting and productive conversations with NFS folks about improving samba interop. It should not be a surprise to anyone here to know that the rest of the audience was, generally speaking, uninterested in the problem.

Which provides a re-enforcement to the point I was trying to make in session - The path of least resistance for NFS-Samba interop is the communicate with each other (both human and software wise) and try to leave VFS out of the discussion for as much as possible (hence dropping linux-fsdevel from this thread).