should we make nfs-ganesha an optional part of a ceph build?
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion). Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem. A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc. I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something). It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled. OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart. I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled. Thoughts? -- Jeff Layton <jlayton@redhat.com>
On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton <jlayton@redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
Is it not using stable APIs from those two shared libraries? i.e. QEMU is compiled and linked against librbd/librados, but I can set "LD_LIBRARY_PATH" to pull in development versions if needed.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
Thoughts? -- Jeff Layton <jlayton@redhat.com> _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Jason
On Wed, 2020-06-17 at 10:16 -0400, Jason Dillaman wrote:
On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton <jlayton@redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
Is it not using stable APIs from those two shared libraries? i.e. QEMU is compiled and linked against librbd/librados, but I can set "LD_LIBRARY_PATH" to pull in development versions if needed.
It is using stable APIs, and yeah that would be a simpler solution. We do sometimes change how ganesha uses those APIs, or add new interfaces that ganesha might call (and only conditionally build in), but that's more rare and can be handled in other ways. Let me look into just setting LD_LIBRARY_PATH. Thanks, -- Jeff Layton <jlayton@redhat.com>
On Wed, Jun 17, 2020 at 10:09 AM Jeff Layton <jlayton@redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
libcephfs and librgw have semi-stable APIs by now. (And IIRC libcephfs has versioned symbols.) So there's some risk that if a versioned symbol changes that an old build of ganesha would continue to use an old API. But one of the whole points of using versioned symbols — IMO — is to make people think really hard about changing the API. How often does it happen in reality that someone runs a ganesha built against mimic/13 or nautilus/14 with octopus/15? I can't imagine it really happens that often. And if it does, I think there's a different problem; one that should be fixed. I'm not really a git submodule and cmake expert, so I don't really know how much work it would be to add ganesha and the ntirpc submodule into the ceph build. And build them with all the right options for whatever the particular environment is. I'm sure it's doable though. It doesn't really affect ganesha. Ganesha packages will still get built independent of whatever ceph is doing. I'd kinda hate to see there be YAPTDGF (yet another place to download ganesha from.) Personally I'd rather try to figure out proper solutions to the concern you raise -- Kaleb
On 17-6-2020 16:09, Jeff Layton wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
To my (big) surprise is there a FreeBSD package for nfs-ganesha so it would be perhaps a nice addon there as well. Have never looked at it, so I don't know if it plays nice with the native NFS implementation that FreeBSD brings with its kernel. And it surely will require some specific tinkering but that is a nice challenge. CMake wizardry is just a one-time obstacle, usually a not too big hurdle. I would be certainly helped by a CMake switch that turns this off, including all tests that are added to test the nfs-ganesha implementation. But like you say: Things are getting bigger and bigger. --WjW To my
On 6/18/20 1:59 AM, Willem Jan Withagen wrote:
On 17-6-2020 16:09, Jeff Layton wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
To my (big)Â surprise is there a FreeBSD package for nfs-ganesha so it would be perhaps a nice addon there as well. Have never looked at it, so I don't know if it plays nice with the native NFS implementation that FreeBSD brings with its kernel. And it surely will require some specific tinkering but that is a nice challenge.
CMake wizardry is just a one-time obstacle, usually a not too big hurdle. I would be certainly helped by a CMake switch that turns this off, including all tests that are added to test the nfs-ganesha implementation.
I'm not sure what the current state of nfs-ganesha on FreeBSD is, but it has had attention from various folks in the past. It would be good to see more active maintenance of nfs-ganesha on FreeBSD. Frank
On 18-6-2020 19:57, Frank Filz wrote:
On 6/18/20 1:59 AM, Willem Jan Withagen wrote:
On 17-6-2020 16:09, Jeff Layton wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
To my (big)Â surprise is there a FreeBSD package for nfs-ganesha so it would be perhaps a nice addon there as well. Have never looked at it, so I don't know if it plays nice with the native NFS implementation that FreeBSD brings with its kernel. And it surely will require some specific tinkering but that is a nice challenge.
CMake wizardry is just a one-time obstacle, usually a not too big hurdle. I would be certainly helped by a CMake switch that turns this off, including all tests that are added to test the nfs-ganesha implementation.
I'm not sure what the current state of nfs-ganesha on FreeBSD is, but it has had attention from various folks in the past. It would be good to see more active maintenance of nfs-ganesha on FreeBSD.
As far as I can tell is the last work in nfs-ganesha done by the Gandi people. And the latest release they have in Ports is 2.7, with an matching kernel module Port for not supported syscalls. Problem there is that the port is based on Python 2.7, which is deprecated on all ends. That 2.7 ganesha version is already EOL according the data on GitHub. Not sure if ceph-dashboard expects a more recent version? --WjW
On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton <jlayton@redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
we tried to shorten the time running the "make check". that's why we have qa/suites/rados/standalone tests. personally, i don't want to add tests which cannot be categorized into unit test to "make check" even if they only take less than 2 minutes to build. probably we could add a task either performed by jenkins or by teuthology for building and testing nfs-ganesha?
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
adding ganesha as a subtree or a submodule does not make sense to me. i see ganesha as a consumer of libcephfs and librados. in the long run, if we go this way, the ceph repo will be bloated like a balloon. not to mention, it's already very big now..
Thoughts? -- Jeff Layton <jlayton@redhat.com> _______________________________________________ Dev mailing list -- dev@ceph.io To unsubscribe send an email to dev-leave@ceph.io
-- Regards Kefu Chai
On 19-6-2020 09:15, kefu chai wrote:
On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton <jlayton@redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something). we tried to shorten the time running the "make check". that's why we have qa/suites/rados/standalone tests. personally, i don't want to add tests which cannot be categorized into unit test to "make check" even if they only take less than 2 minutes to build.
probably we could add a task either performed by jenkins or by teuthology for building and testing nfs-ganesha?
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled. adding ganesha as a subtree or a submodule does not make sense to me. i see ganesha as a consumer of libcephfs and librados. in the long run, if we go this way, the ceph repo will be bloated like a balloon. not to mention, it's already very big now..
Yup, that would be one of the worrying consequences. I previously complained about the large volume of stuff we incorporate and compile for the project. What sort of tipped the balance in favor, was the "challenge" to keep the libs in sync with the ganesha part. And as I read it, forcing the user to recompile ganesha to get a match with the used versions. Perhaps that {is, would be} fixed by versioning the libraries? --WjW
On Fri, 2020-06-19 at 15:15 +0800, kefu chai wrote:
On Wed, Jun 17, 2020 at 10:09 PM Jeff Layton <jlayton@redhat.com> wrote:
(I sent this to a smaller list of folks yesterday, but I think it probably warrants wider discussion).
Recently Varsha added the necessary infrastructure to bring up nfs-ganesha via vstart.sh. The current implementation requires that ganesha already be installed on the box (usually via distro packaging), but that poses a bit of a problem.
A distro ganesha package will have likely been built vs. a completely different version of libcephfs and librados. Even if you build right off of ceph master branch, you won't get the benefit of any recent client bugfixes when you want to test ganesha. You'd have to build new ganesha packages, install them, etc.
I think we ought to consider making a nfs-ganesha build an optional part of a ceph build (maybe enable it with cmake -DNFS_GANESHA=ON or something).
we tried to shorten the time running the "make check". that's why we have qa/suites/rados/standalone tests. personally, i don't want to add tests which cannot be categorized into unit test to "make check" even if they only take less than 2 minutes to build.
probably we could add a task either performed by jenkins or by teuthology for building and testing nfs-ganesha?
The main reason I wanted it was for local testing (a'la vstart). For teuthology, we usually have rather up-to-date ganesha packages. That's not always the case with distro packaging (consider Centos/RHEL).
It doesn't take very long to build it (typically only a minute or two on my box), and we could disable the parts that ceph doesn't care about (other FSALs primarily). We could also have vstart just error out when you run it with NFS=X on a build that didn't have ganesha enabled.
OTOH, the potential downside here is that it'll likely add other build- time dependencies, and would require some extra cmake or scripting wizardry. Nothing insurmountable, but it might represent a maintenance burden going forward, particularly for something that's basically only going to be used for vstart.
I'm also not sure how we'd do this in practice. I don't think you can do optional submodules, so we might have to look at other methods of pulling in the ganesha tree, or just live with it as a submodule that only gets used when ganesha is enabled.
adding ganesha as a subtree or a submodule does not make sense to me. i see ganesha as a consumer of libcephfs and librados. in the long run, if we go this way, the ceph repo will be bloated like a balloon. not to mention, it's already very big now..
Agreed. I decided to drop this idea for now. The better solution is to just allow vstart to run ganesha from an alternate location (as Jason sort of suggested). It turns out that we already do set LD_LIBRARY_PATH in vstart, so we do use the libraries from the ceph build. -- Jeff Layton <jlayton@redhat.com>
On Fri, Jun 19, 2020 at 8:33 AM Jeff Layton <jlayton@redhat.com> wrote:
The main reason I wanted it was for local testing (a'la vstart). For teuthology, we usually have rather up-to-date ganesha packages. That's not always the case with distro packaging (consider Centos/RHEL).
huh? The CentOS Storage SIG has the latest releases of ganesha. I make a point of making sure that new releases are updated promptly there. -- Kaleb
participants (6)
-
Frank Filz -
Jason Dillaman -
Jeff Layton -
Kaleb Keithley -
kefu chai -
Willem Jan Withagen