libcephfs / librados namespace support for nfs-ganesha
Dear devs (ceph-devel / nfs-ganesha-devel), Matt Benjamin asked me in this github issue [1] to send an email to the dev mailing lists to discuss a feature [1]. As this feature needs support from both "(lib)cephfs", librados as well as nfs-ganesha I'm sending it to both lists. TL;DR: Add support for nfs-ganesha to make use of 1) librados namespace, 2) (lib)cephfs namespace in order to avoid seperate pools for each nfs-ganesha instance. Motivation: As far as I'm aware of libcephfs / librados support "namespaces". Note: namespaces in this context are not "multiple filesystems" but rather an "object prefix" to separate different objects in cephfs / RADOS. One of the benefits of "namespaces" is that you can have multiple instances of ganesha using the same pool / filesystem without objects clashing. Currently you would need a separate pool for "RADOS_KV" and "RADOS_URLS" per nfs-ganesha instance. With namespace support in libcephfs you would be able to logically seperate the objects per namespace: one for each nfs-ganesha instance ... and restrict access to only this namespace instead of the whole pool: caps: [mds] allow rw path=/path/on/cephfs caps: [mon] allow r caps: [osd] allow rw pool=cephfs_data namespace=your-name-space-here As a (Ceph) storage admin you would be able to provide tenants the (cephx) capabilities to use nfs-ganesha, without the need to give them rw access to your cephfs_data pool(s). As it turns out the "librados" support is already present in nfs-ganesha 2.7 [2]. However, only when you are using the CEPH FSAL in "active/active" mode. It would be nice to "backport" / reuse this code to be able to use it with the "rados_ng" RADOS_KV recovery backend and / or RADOS_URLS. This would be useful for those that (still) want to use a "active/passive" setup (CTDB, pacemaker / corosync). The "libcephfs" namespace support is not yet implemented in nfs-ganesha AFAIK. This would provide the benefits as discussed in motivation. With adding support for "namespaces" it opens the way for nfs-ganesha multi-tenancy. Does this makes sense? Do you agree on the use case? Would this indeed need support from both ceph as well as nfs-ganesha developers? Thanks, Stefan Kooman [1]: https://github.com/nfs-ganesha/nfs-ganesha/issues/353 [2]: https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/doc/man/ganesha-rad... -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
On Fri, 2018-10-05 at 22:31 +0200, Stefan Kooman wrote:
Dear devs (ceph-devel / nfs-ganesha-devel),
Matt Benjamin asked me in this github issue [1] to send an email to the dev mailing lists to discuss a feature [1]. As this feature needs support from both "(lib)cephfs", librados as well as nfs-ganesha I'm sending it to both lists.
TL;DR:
Add support for nfs-ganesha to make use of 1) librados namespace, 2) (lib)cephfs namespace in order to avoid seperate pools for each nfs-ganesha instance.
Motivation:
As far as I'm aware of libcephfs / librados support "namespaces". Note: namespaces in this context are not "multiple filesystems" but rather an "object prefix" to separate different objects in cephfs / RADOS. One of the benefits of "namespaces" is that you can have multiple instances of ganesha using the same pool / filesystem without objects clashing.
Currently you would need a separate pool for "RADOS_KV" and "RADOS_URLS" per nfs-ganesha instance. With namespace support in libcephfs you would be able to logically seperate the objects per namespace: one for each nfs-ganesha instance ... and restrict access to only this namespace instead of the whole pool:
caps: [mds] allow rw path=/path/on/cephfs caps: [mon] allow r caps: [osd] allow rw pool=cephfs_data namespace=your-name-space-here
As a (Ceph) storage admin you would be able to provide tenants the (cephx) capabilities to use nfs-ganesha, without the need to give them rw access to your cephfs_data pool(s).
As it turns out the "librados" support is already present in nfs-ganesha 2.7 [2]. However, only when you are using the CEPH FSAL in "active/active" mode. It would be nice to "backport" / reuse this code to be able to use it with the "rados_ng" RADOS_KV recovery backend and / or RADOS_URLS. This would be useful for those that (still) want to use a "active/passive" setup (CTDB, pacemaker / corosync).
The "libcephfs" namespace support is not yet implemented in nfs-ganesha AFAIK. This would provide the benefits as discussed in motivation.
With adding support for "namespaces" it opens the way for nfs-ganesha multi-tenancy.
Does this makes sense? Do you agree on the use case? Would this indeed need support from both ceph as well as nfs-ganesha developers?
Thanks,
Stefan Kooman
[1]: https://github.com/nfs-ganesha/nfs-ganesha/issues/353 [2]: https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/doc/man/ganesha-rad...
This is already implemented via a "namespace" parameter to RADOS_KV in nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't be too hard to do. -- Jeff Layton <jlayton@redhat.com>
Quoting Jeff Layton (jlayton@redhat.com):
This is already implemented via a "namespace" parameter to RADOS_KV in nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't be too hard to do.
OK. I'll close this thread here in ceph-devel. Sorry for the noise. Gr. Stefan -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Quoting Stefan Kooman (stefan@bit.nl):
Quoting Jeff Layton (jlayton@redhat.com):
This is already implemented via a "namespace" parameter to RADOS_KV in nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't be too hard to do.
It's nice to have your (cluster) config in a centralized location so you know for sure all daemons are using the same config. So yeah, would be nice to have this support for RADOS_URLS too. If I understand you correctly, you have the possibility to use one namespace, which is then used for *both* librados as well as libcephfs, right? Is it possible to use the "namespace" parameter in the ceph.conf for use with rados_ng? Would this work? ... RADOS_KV { # Path to a ceph.conf file for this cluster. # Ceph_Conf = /etc/ceph/ceph.conf; # The recoverybackend has its own ceph client. The default is to # let libcephfs autogenerate the userid. Note that RADOS_KV block does # not have a setting for Secret_Access_Key. A cephx keyring file must # be used for authenticated access. # UserId = "ganesharecov"; # Pool ID of the ceph storage pool that contains the recovery objects. # The default is "nfs-ganesha". # pool = "nfs-ganesha"; # Set the RADOS namespace to use within the pool (default is NULL) namespace = your-name-space-here # Set the object id of the grace database RADOS object (default is "grace") oid = grace } Thanks, Stefan P.s. It would be a waste if nobody is using these useful features because they don't know of their existence / know how to use it. -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
On Sat, 2018-10-06 at 09:11 +0200, Stefan Kooman wrote:
Quoting Stefan Kooman (stefan@bit.nl):
Quoting Jeff Layton (jlayton@redhat.com):
This is already implemented via a "namespace" parameter to RADOS_KV in nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't be too hard to do.
It's nice to have your (cluster) config in a centralized location so you know for sure all daemons are using the same config. So yeah, would be nice to have this support for RADOS_URLS too.
If I understand you correctly, you have the possibility to use one namespace, which is then used for *both* librados as well as libcephfs, right?
Is it possible to use the "namespace" parameter in the ceph.conf for use with rados_ng? Would this work?
... RADOS_KV { # Path to a ceph.conf file for this cluster. # Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to # let libcephfs autogenerate the userid. Note that RADOS_KV block does # not have a setting for Secret_Access_Key. A cephx keyring file must # be used for authenticated access. # UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects. # The default is "nfs-ganesha". # pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is "grace") oid = grace }
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features because they don't know of their existence / know how to use it.
Yes, that should work. The code that handles establishing the session is common between all of the RADOS recovery backends, so if "namespace" works in one, then it will work in all of them. RADOS_URLS does its own thing there though, so we'll need a separate patch to plumb in support there. -- Jeff Layton <jlayton@redhat.com>
On Sat, 2018-10-06 at 09:11 +0200, Stefan Kooman wrote:
Quoting Stefan Kooman (stefan@bit.nl):
Quoting Jeff Layton (jlayton@redhat.com):
This is already implemented via a "namespace" parameter to RADOS_KV in nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't be too hard to do.
It's nice to have your (cluster) config in a centralized location so you know for sure all daemons are using the same config. So yeah, would be nice to have this support for RADOS_URLS too.
If I understand you correctly, you have the possibility to use one namespace, which is then used for *both* librados as well as libcephfs, right?
Is it possible to use the "namespace" parameter in the ceph.conf for use with rados_ng? Would this work?
... RADOS_KV { # Path to a ceph.conf file for this cluster. # Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to # let libcephfs autogenerate the userid. Note that RADOS_KV block does # not have a setting for Secret_Access_Key. A cephx keyring file must # be used for authenticated access. # UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects. # The default is "nfs-ganesha". # pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is "grace") oid = grace }
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features because they don't know of their existence / know how to use it.
Draft patch to add this to RADOS_URLS follows. We'll still need to update the docs but if you want to help test this, then that would be good: ------------------------------8<--------------------------------- [PATCH] RADOS_URLS: add a namespace parameter to the config block ...and set it on the ioctx. Change-Id: I076f43ec3ced206b787273455fd2e4e566bc80a2 Signed-off-by: Jeff Layton <jlayton@redhat.com> --- src/config_parsing/conf_url_rados.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/config_parsing/conf_url_rados.c b/src/config_parsing/conf_url_rados.c index 692e4cc348ec..2baad0a4ed17 100644 --- a/src/config_parsing/conf_url_rados.c +++ b/src/config_parsing/conf_url_rados.c @@ -38,6 +38,8 @@ static struct rados_url_parameter { char *ceph_conf; /** Userid (?) */ char *userid; + /** Namespace */ + char *namespace; } rados_url_param; static struct config_item rados_url_params[] = { @@ -45,6 +47,8 @@ static struct config_item rados_url_params[] = { rados_url_parameter, ceph_conf), CONF_ITEM_STR("userid", 1, MAXPATHLEN, NULL, rados_url_parameter, userid), + CONF_ITEM_STR("namespace", 1, NAME_MAX, NULL, + rados_url_parameter, namespace), CONFIG_EOL }; @@ -262,6 +266,8 @@ static int cu_rados_url_fetch(const char *url, FILE **f, char **fbuf) cu_rados_url_shutdown(); goto out; } + rados_ioctx_set_namespace(io_ctx, rados_url_param.namespace); + do { int nread, wrt, nwrt; -- 2.17.1
Quoting Jeff Layton (jlayton@redhat.com):
Draft patch to add this to RADOS_URLS follows. We'll still need to update the docs but if you want to help test this, then that would be good:
I just rebuild packages with your patch in. Let's see how I build an active/active cluster ... I'll let you know how it goes. I was thinking ... wouldn't it be nice to be able to pass the "rados_url" as an argument to ganesha.nfsd. No need for the stub config file ... probably easier for container setups as well. Gr. Stefan -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Quoting Jeff Layton (jlayton@redhat.com):
Is it possible to use the "namespace" parameter in the ceph.conf for use with rados_ng? Would this work?
... RADOS_KV { # Path to a ceph.conf file for this cluster. # Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to # let libcephfs autogenerate the userid. Note that RADOS_KV block does # not have a setting for Secret_Access_Key. A cephx keyring file must # be used for authenticated access. # UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects. # The default is "nfs-ganesha". # pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is "grace") oid = grace }
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features because they don't know of their existence / know how to use it.
Draft patch to add this to RADOS_URLS follows. We'll still need to update the docs but if you want to help test this, then that would be good:
The config parses correctly when adding the namespace parameter to the RADOS_URLS block. However it's unclear to me where the "url" config option has to enabled, and how it should look like. If I put below config option ganesha complains about syntax errors %url rados://nfs-ganesha-27-fcal-ceph/ganesha.conf; The config also parses correctly when the namespace parameter is added to the CEPH_KV block. And the objects are added in the right namespace if one is specified. However, when the namespace parameter is included in the "CEPH" BLOCK a warning is issued: Unknown parameter (namespace). This is true for the "FSAL" block as well. So I guess there needs to be done some more work to get the CEPH FSAL recognize the namespace. Gr. Stefan -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
On Sat, 2018-10-06 at 23:54 +0200, Stefan Kooman wrote:
Quoting Jeff Layton (jlayton@redhat.com):
Is it possible to use the "namespace" parameter in the ceph.conf for use with rados_ng? Would this work?
... RADOS_KV { # Path to a ceph.conf file for this cluster. # Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to # let libcephfs autogenerate the userid. Note that RADOS_KV block does # not have a setting for Secret_Access_Key. A cephx keyring file must # be used for authenticated access. # UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects. # The default is "nfs-ganesha". # pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is "grace") oid = grace }
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features because they don't know of their existence / know how to use it.
Draft patch to add this to RADOS_URLS follows. We'll still need to update the docs but if you want to help test this, then that would be good:
The config parses correctly when adding the namespace parameter to the RADOS_URLS block. However it's unclear to me where the "url" config option has to enabled, and how it should look like.
If I put below config option ganesha complains about syntax errors
%url rados://nfs-ganesha-27-fcal-ceph/ganesha.conf;
The config also parses correctly when the namespace parameter is added to the CEPH_KV block. And the objects are added in the right namespace if one is specified.
However, when the namespace parameter is included in the "CEPH" BLOCK a warning is issued: Unknown parameter (namespace). This is true for the "FSAL" block as well.
So I guess there needs to be done some more work to get the CEPH FSAL recognize the namespace.
Gr. Stefan
Yes, there's no namespace support in FSAL_CEPH config block as of yet, as libcephfs doesn't have that yet either. -- Jeff Layton <jlayton@redhat.com>
Quoting Jeff Layton (jlayton@redhat.com):
Yes, there's no namespace support in FSAL_CEPH config block as of yet, as libcephfs doesn't have that yet either.
Check. Then I'll add a feature request for namespace support in libcephfs (and as soon as it's there, ask in ganesha-devel to support it). Gr. Stefan -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
Quoting Stefan Kooman (stefan@bit.nl):
Quoting Jeff Layton (jlayton@redhat.com):
This is already implemented via a "namespace" parameter to RADOS_KV in nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't be too hard to do.
Turns out that libcephfs does not have support for namespaces yet, so nfs-ganesha can't make use of them. I created an issue for this [1]. Gr. Stefan [1]: http://tracker.ceph.com/issues/36338 -- | BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351 | GPG: 0xD14839C6 +31 318 648 688 / info@bit.nl
participants (2)
-
Jeff Layton -
Stefan Kooman