On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton <jlayton(a)redhat.com&gt; wrote: > (I sent this to a smaller list of folks yesterday, but I think it > probably warrants wider discussion). > > Recently Varsha added the necessary infrastructure to bring up > nfs-ganesha via vstart.sh. The current implementation requires that > ganesha already be installed on the box (usually via distro packaging), > but that poses a bit of a problem. > > A distro ganesha package will have likely been built vs. a completely > different version of libcephfs and librados. Even if you build right off > of ceph master branch, you won't get the benefit of any recent client > bugfixes when you want to test ganesha. You'd have to build new ganesha > packages, install them, etc. Is it not using stable APIs from those two shared libraries? i.e. QEMU is compiled and linked against librbd/librados, but I can set "LD_LIBRARY_PATH" to pull in development versions if needed.

(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion). Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem. A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc. I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something). It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled. OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart. I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.

On 6/18/20 1:59 AM, Willem Jan Withagen wrote: > On 17-6-2020 16:09, Jeff Layton wrote: >> (I sent this to a smaller list of folks yesterday, but I think it >> probably warrants wider discussion). >> >> Recently Varsha added the necessary infrastructure to bring up >> nfs-ganesha via vstart.sh. The current implementation requires that >> ganesha already be installed on the box (usually via distro packaging), >> but that poses a bit of a problem. >> >> A distro ganesha package will have likely been built vs. a completely >> different version of libcephfs and librados. Even if you build right >> off >> of ceph master branch, you won't get the benefit of any recent client >> bugfixes when you want to test ganesha. You'd have to build new ganesha >> packages, install them, etc. >> >> I think we ought to consider making a nfs-ganesha build an optional >> part >> of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or >> something). >> >> It doesn't take very long to build it (typically only a minute or >> two on >> my box), and we could disable the parts that ceph doesn't care about >> (other FSALs primarily). We could also have vstart just error out when >> you run it with NFS=X on a build that didn't have ganesha enabled. >> >> OTOH, the potential downside here is that it'll likely add other build- >> time dependencies, and would require some extra cmake or scripting >> wizardry. Nothing insurmountable, but it might represent a maintenance >> burden going forward, particularly for something that's basically only >> going to be used for vstart. >> >> I'm also not sure how we'd do this in practice. I don't think you >> can do >> optional submodules, so we might have to look at other methods of >> pulling in the ganesha tree, or just live with it as a submodule that >> only gets used when ganesha is enabled. > > To my (big)  surprise is there a FreeBSD package for nfs-ganesha so > it would be perhaps > a nice addon there as well. > Have never looked at it, so I don't know if it plays nice with the > native NFS implementation > that FreeBSD brings with its kernel. And it surely will require some > specific tinkering but > that is a nice challenge. > > CMake wizardry is just a one-time obstacle, usually a not too big > hurdle. > I would be certainly helped by a CMake switch that turns this off, > including all tests > that are added to test the nfs-ganesha implementation. I'm not sure what the current state of nfs-ganesha on FreeBSD is, but it has had attention from various folks in the past. It would be good to see more active maintenance of nfs-ganesha on FreeBSD.

On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton <jlayton(a)redhat.com&gt; wrote: > (I sent this to a smaller list of folks yesterday, but I think it > probably warrants wider discussion). > > Recently Varsha added the necessary infrastructure to bring up > nfs-ganesha via vstart.sh. The current implementation requires that > ganesha already be installed on the box (usually via distro packaging), > but that poses a bit of a problem. > > A distro ganesha package will have likely been built vs. a completely > different version of libcephfs and librados. Even if you build right off > of ceph master branch, you won't get the benefit of any recent client > bugfixes when you want to test ganesha. You'd have to build new ganesha > packages, install them, etc. > > I think we ought to consider making a nfs-ganesha build an optional part > of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or > something). we tried to shorten the time running the "make check". that's why we have qa/suites/rados/standalone tests. personally, i don't want to add tests which cannot be categorized into unit test to "make check" even if they only take less than 2 minutes to build. probably we could add a task either performed by jenkins or by teuthology for building and testing nfs-ganesha? > It doesn't take very long to build it (typically only a minute or two on > my box), and we could disable the parts that ceph doesn't care about > (other FSALs primarily). We could also have vstart just error out when > you run it with NFS=X on a build that didn't have ganesha enabled. > > OTOH, the potential downside here is that it'll likely add other build- > time dependencies, and would require some extra cmake or scripting > wizardry. Nothing insurmountable, but it might represent a maintenance > burden going forward, particularly for something that's basically only > going to be used for vstart. > > I'm also not sure how we'd do this in practice. I don't think you can do > optional submodules, so we might have to look at other methods of > pulling in the ganesha tree, or just live with it as a submodule that > only gets used when ganesha is enabled. adding ganesha as a subtree or a submodule does not make sense to me. i see ganesha as a consumer of libcephfs and librados. in the long run, if we go this way, the ceph repo will be bloated like a balloon. not to mention, it's already very big now..

The main reason I wanted it was for local testing (a'la vstart). For teuthology, we usually have rather up-to-date ganesha packages. That's not always the case with distro packaging (consider Centos/RHEL).