3:31 p.m.
Dear devs (ceph-devel / nfs-ganesha-devel),
Matt Benjamin asked me in this github issue [1] to send an email to the
dev mailing lists to discuss a feature [1]. As this feature needs
support from both "(lib)cephfs", librados as well as nfs-ganesha I'm
sending it to both lists.
TL;DR:
Add support for nfs-ganesha to make use of 1) librados namespace, 2)
(lib)cephfs namespace in order to avoid seperate pools for each
nfs-ganesha instance.
Motivation:
As far as I'm aware of libcephfs / librados support "namespaces". Note:
namespaces in this context are not "multiple filesystems" but rather an
"object prefix" to separate different objects in cephfs / RADOS. One of
the benefits of "namespaces" is that you can have multiple instances of
ganesha using the same pool / filesystem without objects clashing.
Currently you would need a separate pool for "RADOS_KV" and
"RADOS_URLS"
per nfs-ganesha instance. With namespace support in libcephfs you
would be able to logically seperate the objects per namespace: one for
each nfs-ganesha instance ... and restrict access to only this namespace
instead of the whole pool:
caps: [mds] allow rw path=/path/on/cephfs
caps: [mon] allow r
caps: [osd] allow rw pool=cephfs_data namespace=your-name-space-here
As a (Ceph) storage admin you would be able to provide tenants the
(cephx) capabilities to use nfs-ganesha, without the need to give them
rw access to your cephfs_data pool(s).
As it turns out the "librados" support is already present in nfs-ganesha
2.7 [2]. However, only when you are using the CEPH FSAL in
"active/active" mode. It would be nice to "backport" / reuse this
code
to be able to use it with the "rados_ng" RADOS_KV recovery backend and /
or RADOS_URLS. This would be useful for those that (still) want to use a
"active/passive" setup (CTDB, pacemaker / corosync).
The "libcephfs" namespace support is not yet implemented in nfs-ganesha
AFAIK. This would provide the benefits as discussed in motivation.
With adding support for "namespaces" it opens the way for nfs-ganesha
multi-tenancy.
Does this makes sense? Do you agree on the use case? Would this indeed
need support from both ceph as well as nfs-ganesha developers?
Thanks,
Stefan Kooman
[1]: https://github.com/nfs-ganesha/nfs-ganesha/issues/353
[2]:
https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/doc/man/ganesha-...
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
4:20 p.m.
On Fri, 2018-10-05 at 22:31 +0200, Stefan Kooman wrote:
Dear devs (ceph-devel / nfs-ganesha-devel),
Matt Benjamin asked me in this github issue [1] to send an email to the
dev mailing lists to discuss a feature [1]. As this feature needs
support from both "(lib)cephfs", librados as well as nfs-ganesha I'm
sending it to both lists.
TL;DR:
Add support for nfs-ganesha to make use of 1) librados namespace, 2)
(lib)cephfs namespace in order to avoid seperate pools for each
nfs-ganesha instance.
Motivation:
As far as I'm aware of libcephfs / librados support "namespaces". Note:
namespaces in this context are not "multiple filesystems" but rather an
"object prefix" to separate different objects in cephfs / RADOS. One of
the benefits of "namespaces" is that you can have multiple instances of
ganesha using the same pool / filesystem without objects clashing.
Currently you would need a separate pool for "RADOS_KV" and
"RADOS_URLS"
per nfs-ganesha instance. With namespace support in libcephfs you
would be able to logically seperate the objects per namespace: one for
each nfs-ganesha instance ... and restrict access to only this namespace
instead of the whole pool:
caps: [mds] allow rw path=/path/on/cephfs
caps: [mon] allow r
caps: [osd] allow rw pool=cephfs_data namespace=your-name-space-here
As a (Ceph) storage admin you would be able to provide tenants the
(cephx) capabilities to use nfs-ganesha, without the need to give them
rw access to your cephfs_data pool(s).
As it turns out the "librados" support is already present in nfs-ganesha
2.7 [2]. However, only when you are using the CEPH FSAL in
"active/active" mode. It would be nice to "backport" / reuse this
code
to be able to use it with the "rados_ng" RADOS_KV recovery backend and /
or RADOS_URLS. This would be useful for those that (still) want to use a
"active/passive" setup (CTDB, pacemaker / corosync).
The "libcephfs" namespace support is not yet implemented in nfs-ganesha
AFAIK. This would provide the benefits as discussed in motivation.
With adding support for "namespaces" it opens the way for nfs-ganesha
multi-tenancy.
Does this makes sense? Do you agree on the use case? Would this indeed
need support from both ceph as well as nfs-ganesha developers?
Thanks,
Stefan Kooman
[1]: https://github.com/nfs-ganesha/nfs-ganesha/issues/353
[2]:
https://github.com/nfs-ganesha/nfs-ganesha/blob/next/src/doc/man/ganesha-...
This is already implemented via a "namespace" parameter to RADOS_KV in
nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't
be too hard to do.
--
Jeff Layton <jlayton(a)redhat.com>
1:58 a.m.
Quoting Jeff Layton (jlayton(a)redhat.com):
This is already implemented via a "namespace" parameter to
RADOS_KV in
nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't
be too hard to do.
OK. I'll close this thread here in ceph-devel.
Sorry for the noise.
Gr. Stefan
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
2:11 a.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
Quoting Stefan Kooman (stefan(a)bit.nl):
Quoting Jeff Layton (jlayton(a)redhat.com):
> This is already implemented via a "namespace" parameter to RADOS_KV in
> nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't
> be too hard to do.
It's nice to have your (cluster) config in a centralized location so you
know for sure all daemons are using the same config. So yeah, would be
nice to have this support for RADOS_URLS too.
If I understand you correctly, you have the possibility to use one
namespace, which is then used for *both* librados as well as libcephfs,
right?
Is it possible to use the "namespace" parameter in the ceph.conf for
use with rados_ng? Would this work?
...
RADOS_KV
{
# Path to a ceph.conf file for this cluster.
# Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to
# let libcephfs autogenerate the userid. Note that RADOS_KV block does
# not have a setting for Secret_Access_Key. A cephx keyring file must
# be used for authenticated access.
# UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects.
# The default is "nfs-ganesha".
# pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is
"grace")
oid = grace
}
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features
because they don't know of their existence / know how to use it.
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
5:58 a.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
On Sat, 2018-10-06 at 09:11 +0200, Stefan Kooman wrote:
Quoting Stefan Kooman (stefan(a)bit.nl):
> Quoting Jeff Layton (jlayton(a)redhat.com):
> > This is already implemented via a "namespace" parameter to RADOS_KV
in
> > nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it
wouldn't
> > be too hard to do.
It's nice to have your (cluster) config in a centralized location so you
know for sure all daemons are using the same config. So yeah, would be
nice to have this support for RADOS_URLS too.
If I understand you correctly, you have the possibility to use one
namespace, which is then used for *both* librados as well as libcephfs,
right?
Is it possible to use the "namespace" parameter in the ceph.conf for
use with rados_ng? Would this work?
...
RADOS_KV
{
# Path to a ceph.conf file for this cluster.
# Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to
# let libcephfs autogenerate the userid. Note that RADOS_KV block does
# not have a setting for Secret_Access_Key. A cephx keyring file must
# be used for authenticated access.
# UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects.
# The default is "nfs-ganesha".
# pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is
"grace")
oid = grace
}
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features
because they don't know of their existence / know how to use it.
Yes, that should work. The code that handles establishing the session is
common between all of the RADOS recovery backends, so if "namespace"
works in one, then it will work in all of them.
RADOS_URLS does its own thing there though, so we'll need a separate
patch to plumb in support there.
--
Jeff Layton <jlayton(a)redhat.com>
6:42 a.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
On Sat, 2018-10-06 at 09:11 +0200, Stefan Kooman wrote:
Quoting Stefan Kooman (stefan(a)bit.nl):
> Quoting Jeff Layton (jlayton(a)redhat.com):
> > This is already implemented via a "namespace" parameter to RADOS_KV
in
> > nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it
wouldn't
> > be too hard to do.
It's nice to have your (cluster) config in a centralized location so you
know for sure all daemons are using the same config. So yeah, would be
nice to have this support for RADOS_URLS too.
If I understand you correctly, you have the possibility to use one
namespace, which is then used for *both* librados as well as libcephfs,
right?
Is it possible to use the "namespace" parameter in the ceph.conf for
use with rados_ng? Would this work?
...
RADOS_KV
{
# Path to a ceph.conf file for this cluster.
# Ceph_Conf = /etc/ceph/ceph.conf;
# The recoverybackend has its own ceph client. The default is to
# let libcephfs autogenerate the userid. Note that RADOS_KV block does
# not have a setting for Secret_Access_Key. A cephx keyring file must
# be used for authenticated access.
# UserId = "ganesharecov";
# Pool ID of the ceph storage pool that contains the recovery objects.
# The default is "nfs-ganesha".
# pool = "nfs-ganesha";
# Set the RADOS namespace to use within the pool (default is NULL)
namespace = your-name-space-here
# Set the object id of the grace database RADOS object (default is
"grace")
oid = grace
}
Thanks,
Stefan
P.s. It would be a waste if nobody is using these useful features
because they don't know of their existence / know how to use it.
Draft patch to add this to RADOS_URLS follows. We'll still need to
update the docs but if you want to help test this, then that would be
good:
------------------------------8<---------------------------------
[PATCH] RADOS_URLS: add a namespace parameter to the config block
...and set it on the ioctx.
Change-Id: I076f43ec3ced206b787273455fd2e4e566bc80a2
Signed-off-by: Jeff Layton <jlayton(a)redhat.com>
---
src/config_parsing/conf_url_rados.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/src/config_parsing/conf_url_rados.c b/src/config_parsing/conf_url_rados.c
index 692e4cc348ec..2baad0a4ed17 100644
--- a/src/config_parsing/conf_url_rados.c
+++ b/src/config_parsing/conf_url_rados.c
@@ -38,6 +38,8 @@ static struct rados_url_parameter {
char *ceph_conf;
/** Userid (?) */
char *userid;
+ /** Namespace */
+ char *namespace;
} rados_url_param;
static struct config_item rados_url_params[] = {
@@ -45,6 +47,8 @@ static struct config_item rados_url_params[] = {
rados_url_parameter, ceph_conf),
CONF_ITEM_STR("userid", 1, MAXPATHLEN, NULL,
rados_url_parameter, userid),
+ CONF_ITEM_STR("namespace", 1, NAME_MAX, NULL,
+ rados_url_parameter, namespace),
CONFIG_EOL
};
@@ -262,6 +266,8 @@ static int cu_rados_url_fetch(const char *url, FILE **f, char **fbuf)
cu_rados_url_shutdown();
goto out;
}
+ rados_ioctx_set_namespace(io_ctx, rados_url_param.namespace);
+
do {
int nread, wrt, nwrt;
--
2.17.1
1:58 p.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
Quoting Jeff Layton (jlayton(a)redhat.com):
Draft patch to add this to RADOS_URLS follows. We'll still need
to
update the docs but if you want to help test this, then that would be
good:
I just rebuild packages with your patch in. Let's see how I build an
active/active cluster ... I'll let you know how it goes.
I was thinking ... wouldn't it be nice to be able to pass the
"rados_url" as an argument to ganesha.nfsd. No need for the stub config
file ... probably easier for container setups as well.
Gr. Stefan
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
4:54 p.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
Quoting Jeff Layton (jlayton(a)redhat.com):
>
> Is it possible to use the "namespace" parameter in the ceph.conf for
> use with rados_ng? Would this work?
>
> ...
> RADOS_KV
> {
> # Path to a ceph.conf file for this cluster.
> # Ceph_Conf = /etc/ceph/ceph.conf;
>
> # The recoverybackend has its own ceph client. The default is to
> # let libcephfs autogenerate the userid. Note that RADOS_KV block does
> # not have a setting for Secret_Access_Key. A cephx keyring file must
> # be used for authenticated access.
> # UserId = "ganesharecov";
>
> # Pool ID of the ceph storage pool that contains the recovery objects.
> # The default is "nfs-ganesha".
> # pool = "nfs-ganesha";
>
>
> # Set the RADOS namespace to use within the pool (default is NULL)
>
> namespace = your-name-space-here
>
> # Set the object id of the grace database RADOS object (default is
"grace")
> oid = grace
> }
>
> Thanks,
>
> Stefan
>
> P.s. It would be a waste if nobody is using these useful features
> because they don't know of their existence / know how to use it.
>
Draft patch to add this to RADOS_URLS follows. We'll still need to
update the docs but if you want to help test this, then that would be
good:
The config parses correctly when adding the namespace parameter to the
RADOS_URLS block. However it's unclear to me where the "url" config
option has to enabled, and how it should look like.
If I put below config option ganesha complains about syntax errors
%url rados://nfs-ganesha-27-fcal-ceph/ganesha.conf;
The config also parses correctly when the namespace parameter is added
to the CEPH_KV block. And the objects are added in the right namespace if one
is specified.
However, when the namespace parameter is included in the "CEPH" BLOCK
a warning is issued: Unknown parameter (namespace). This is true for the
"FSAL" block as well.
So I guess there needs to be done some more work to get the CEPH FSAL
recognize the namespace.
Gr. Stefan
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
9:21 a.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
On Sat, 2018-10-06 at 23:54 +0200, Stefan Kooman wrote:
Quoting Jeff Layton (jlayton(a)redhat.com):
> >
> > Is it possible to use the "namespace" parameter in the ceph.conf for
> > use with rados_ng? Would this work?
> >
> > ...
> > RADOS_KV
> > {
> > # Path to a ceph.conf file for this cluster.
> > # Ceph_Conf = /etc/ceph/ceph.conf;
> >
> > # The recoverybackend has its own ceph client. The default is to
> > # let libcephfs autogenerate the userid. Note that RADOS_KV block does
> > # not have a setting for Secret_Access_Key. A cephx keyring file must
> > # be used for authenticated access.
> > # UserId = "ganesharecov";
> >
> > # Pool ID of the ceph storage pool that contains the recovery objects.
> > # The default is "nfs-ganesha".
> > # pool = "nfs-ganesha";
> >
> >
> > # Set the RADOS namespace to use within the pool (default is NULL)
> >
> > namespace = your-name-space-here
> >
> > # Set the object id of the grace database RADOS object (default is
"grace")
> > oid = grace
> > }
> >
> > Thanks,
> >
> > Stefan
> >
> > P.s. It would be a waste if nobody is using these useful features
> > because they don't know of their existence / know how to use it.
> >
>
> Draft patch to add this to RADOS_URLS follows. We'll still need to
> update the docs but if you want to help test this, then that would be
> good:
The config parses correctly when adding the namespace parameter to the
RADOS_URLS block. However it's unclear to me where the "url" config
option has to enabled, and how it should look like.
If I put below config option ganesha complains about syntax errors
%url rados://nfs-ganesha-27-fcal-ceph/ganesha.conf;
The config also parses correctly when the namespace parameter is added
to the CEPH_KV block. And the objects are added in the right namespace if one
is specified.
However, when the namespace parameter is included in the "CEPH" BLOCK
a warning is issued: Unknown parameter (namespace). This is true for the
"FSAL" block as well.
So I guess there needs to be done some more work to get the CEPH FSAL
recognize the namespace.
Gr. Stefan
Yes, there's no namespace support in FSAL_CEPH config block as of yet,
as libcephfs doesn't have that yet either.
--
Jeff Layton <jlayton(a)redhat.com>
10:18 a.m.
New subject: libcephfs / librados namespace support for nfs-ganesha
Quoting Jeff Layton (jlayton(a)redhat.com):
Yes, there's no namespace support in FSAL_CEPH config block as of
yet,
as libcephfs doesn't have that yet either.
Check. Then I'll add a feature request for namespace support in
libcephfs (and as soon as it's there, ask in ganesha-devel to support
it).
Gr. Stefan
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
1:17 p.m.
Quoting Stefan Kooman (stefan(a)bit.nl):
Quoting Jeff Layton (jlayton(a)redhat.com):
> This is already implemented via a "namespace" parameter to RADOS_KV in
> nfs-ganesha 2.7. It's not implemented in RADOS_URLS yet, but it wouldn't
> be too hard to do.
Turns out that libcephfs does not have support for namespaces yet, so
nfs-ganesha can't make use of them. I created an issue for this [1].
Gr. Stefan
[1]: http://tracker.ceph.com/issues/36338
--
| BIT BV http://www.bit.nl/ Kamer van Koophandel 09090351
| GPG: 0xD14839C6 +31 318 648 688 / info(a)bit.nl
2048
days inactive
2050
days old
10 comments
2 participants
participants (2)
-
Jeff Layton -
Stefan Kooman