I have a cephfs filesystem and I want to export a subdirectory of my cephfs tree. Is this possible or do I have to export the entire tree from the root?

If it is possible, then is it also possible to define exports for multiple subdirectories of my cephfs FS such as /cephfs/exports/foo and /cephfs/exports/bar as 2 separate exports?

> On Thu, Apr 4, 2019 at 11:41 AM Wyllys Ingersoll > <wyllys.ingersoll(a)keepertech.com&gt; wrote: > > > > I have a cephfs filesystem and I want to export a subdirectory of my cephfs > tree. Is this possible or do I have to export the entire tree from the root? > > > > You can export a subdirectory, but there is no subtree checking. It's possible for > someone to guess filehandles and access files outside the exported directory. > See the "no_subtree_check" section in the linux > exports(5) manpage. It is generally best to export along filesystem boundaries > for that reason.

> > > If it is possible, then is it also possible to define exports for multiple > subdirectories of my cephfs FS such as /cephfs/exports/foo and > /cephfs/exports/bar as 2 separate exports? > > Yes. They will each get their own cephfs client though, so in the event that two > clients end up accessing the same inodes via two different exports, performance > may suffer. The clients will end up competing with one another for caps.

Hmm, if this was something we actually wanted to generally support (I'm not convinced it would be a good idea), we could use the fsal_filesystem infrastructure to share the cephfs client.

On Fri, Apr 5, 2019 at 11:00 AM Wyllys Ingersoll <wyllys.ingersoll(a)keepertech.com&gt; wrote: > > > > On Fri, Apr 5, 2019 at 10:49 AM Frank Filz <ffilzlnx(a)mindspring.com&gt; wrote: >> >> > On Thu, Apr 4, 2019 at 11:41 AM Wyllys Ingersoll >> > <wyllys.ingersoll(a)keepertech.com&gt; wrote: >> > > >> > > I have a cephfs filesystem and I want to export a subdirectory of my cephfs >> > tree. Is this possible or do I have to export the entire tree from the root? >> > > >> > >> > You can export a subdirectory, but there is no subtree checking. It's possible for >> > someone to guess filehandles and access files outside the exported directory. >> > See the "no_subtree_check" section in the linux >> > exports(5) manpage. It is generally best to export along filesystem boundaries >> > for that reason. > > > > Most cephfs filesystems I've encountered are vast and wide and I have seen many cases where one wants to export multiple subdirectories with different parameters and restrictions. Only supporting the top of the tree seems like a really basic starting point. If thats a real concern, it should be spelled out explicitly in the documentation. It's not at all clear from what I've read. > That'd be nice. Care to suggest some text, or a patch to the docs in the tree? This is more of a general problem with NFS servers, fwiw.

> >>>> > >> > > If it is possible, then is it also possible to define exports for multiple >> > subdirectories of my cephfs FS such as /cephfs/exports/foo and >> > /cephfs/exports/bar as 2 separate exports? >> > >> > Yes. They will each get their own cephfs client though, so in the event that two >> > clients end up accessing the same inodes via two different exports, performance >> > may suffer. The clients will end up competing with one another for caps. > > > > I guess thats possible with hard links, but I think that's an edge case at best. Doesn't that problem also exist if the subdirectories are exported using the basic fsal_vfs ? > Yes, I don't think it does subtree checking either. This is quite a difficult problem for a userland server, fwiw. > >> >> >> Hmm, if this was something we actually wanted to generally support (I'm not convinced it would be a good idea), we could use the fsal_filesystem infrastructure to share the cephfs client. > > > Are you saying that ganesha does NOT officially support exporting subdirectories using fsal ceph? > > Im just trying to understand the limitations and future direction of the project in this area. Consider an implementation where a large cephfs filesystem is used as part of a NAS server and multiple subdirectories of it are exported using NFS and SMB to a variety of clients. > No, he's just pointing out that this is something that could be improved. In principle, we ought to be able to use the same client for two exports that refer to the same cephfs. I agree that cross-export hardlinks are pretty rare, so this turns out not to be a big problem in most cases.