FSAL :CRIT :This libcephfs version doesn't support named filesystems
Hi, I am running a Debian 10 with root@nfsshares-a:~# dpkg -l | grep ganesha ii nfs-ganesha 3.3-4 amd64 nfs-ganesha is a NFS server in User Space ii nfs-ganesha-ceph:amd64 3.3-4 amd64 nfs-ganesha fsal ceph libraries ii nfs-ganesha-rados-grace:amd64 3.3-4 amd64 nfs-ganesha ganesha-rados-grace program and root@nfsshares-a:~# dpkg -l | grep ceph ii ceph-common 14.2.16-pve1 amd64 common utilities to mount and interact with a ceph storage cluster ii libcephfs2 14.2.16-pve1 amd64 Ceph distributed file system client library ii nfs-ganesha-ceph:amd64 3.3-4 amd64 nfs-ganesha fsal ceph libraries ii python-ceph-argparse 14.2.16-pve1 all Python 2 utility libraries for Ceph CLI ii python-cephfs 14.2.16-pve1 amd64 Python 2 libraries for the Ceph libcephfs library The config looks like this: root@nfsshares-a:~# grep '^[[:blank:]]*[^[:blank:]#;]' /etc/ganesha/ganesha.conf NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 3,4; mount_path_pseudo = true; } NFS_KRB5 { Active_krb5 = false; } NFSv4 { RecoveryBackend = rados_ng; Minor_Versions = 1,2; } MDCACHE { Dir_Chunk = 0; } EXPORT_DEFAULTS { SecType = "sys"; Squash = No_Root_Squash; Attr_Expiration_Time = 0; } CEPH { Ceph_Conf = /etc/ceph/ceph.conf; } RADOS_KV { Ceph_Conf = /etc/ceph/ceph.conf; UserId = "ganesharecov"; nodeid = "nfsshares-a"; } RADOS_URLS { Ceph_Conf = /etc/ceph/ceph.conf; UserId = "ganeshaurls"; watch_url = "rados://nfs-ganesha/ganesha-namespace/conf-nfsshares"; } %url rados://nfs-ganesha/ganesha-namespace/conf-nfsshares The Export configuration is created on the ceph dashboard and looks like this: root@proxmox07:~# cat /tmp/conf-nfsshares %url "rados://nfs-ganesha/ganesha-namespace/export-1" root@proxmox07:~# cat /tmp/export-1 EXPORT { FSAL { secret_access_key = "AQDm7eFfJWvhIBAAuI++d0SNkdylNkMx5t/N8w=="; user_id = "ganesha"; name = "CEPH"; filesystem = "cephfs"; } CLIENT { clients = 10.20.56.254; squash = "no_root_squash"; access_type = "RW"; } pseudo = "/"; squash = "no_root_squash"; access_type = "NONE"; path = "/"; export_id = 1; transports = "UDP", "TCP"; protocols = 4; } I get the following error message: .... 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] register_callbacks :FSAL :WARN :This libcephfs does not support registering callbacks. Ganesha will be unable to respond to MDS cache pressure. 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] select_filesystem :FSAL :CRIT :This libcephfs version doesn't support named filesystems. 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/) to (/) 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":2): 1 validation errors in block FSAL 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":2): Errors processing block (FSAL) 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":1): 1 validation errors in block EXPORT 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":1): Errors processing block (EXPORT) .... When I remove filesystem = "cephfs"; in export-1 it works. I believe my dashboard does not work together with its version or configuration of the libcephfs2 (both same source and version). Or did I misconfigure something here? Best regards and a healthy new year! Rainer
So, what matters is the version of Ceph that Ganesha was built against, not the version you're running against. 14.2 should be new enough to have ceph_select_filesystem() (which is the API in question. Where did you get the Ganesha packages, who built them, and which version of libcephfs was used when they were built? Daniel On 1/15/21 6:50 AM, rainer.stumbaum@gmail.com wrote:
Hi,
I am running a Debian 10 with root@nfsshares-a:~# dpkg -l | grep ganesha ii nfs-ganesha 3.3-4 amd64 nfs-ganesha is a NFS server in User Space ii nfs-ganesha-ceph:amd64 3.3-4 amd64 nfs-ganesha fsal ceph libraries ii nfs-ganesha-rados-grace:amd64 3.3-4 amd64 nfs-ganesha ganesha-rados-grace program
and
root@nfsshares-a:~# dpkg -l | grep ceph ii ceph-common 14.2.16-pve1 amd64 common utilities to mount and interact with a ceph storage cluster ii libcephfs2 14.2.16-pve1 amd64 Ceph distributed file system client library ii nfs-ganesha-ceph:amd64 3.3-4 amd64 nfs-ganesha fsal ceph libraries ii python-ceph-argparse 14.2.16-pve1 all Python 2 utility libraries for Ceph CLI ii python-cephfs 14.2.16-pve1 amd64 Python 2 libraries for the Ceph libcephfs library
The config looks like this: root@nfsshares-a:~# grep '^[[:blank:]]*[^[:blank:]#;]' /etc/ganesha/ganesha.conf NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 3,4; mount_path_pseudo = true; } NFS_KRB5 { Active_krb5 = false; } NFSv4 { RecoveryBackend = rados_ng; Minor_Versions = 1,2; } MDCACHE { Dir_Chunk = 0; } EXPORT_DEFAULTS { SecType = "sys"; Squash = No_Root_Squash; Attr_Expiration_Time = 0; } CEPH { Ceph_Conf = /etc/ceph/ceph.conf; } RADOS_KV { Ceph_Conf = /etc/ceph/ceph.conf; UserId = "ganesharecov"; nodeid = "nfsshares-a"; } RADOS_URLS { Ceph_Conf = /etc/ceph/ceph.conf; UserId = "ganeshaurls"; watch_url = "rados://nfs-ganesha/ganesha-namespace/conf-nfsshares"; } %url rados://nfs-ganesha/ganesha-namespace/conf-nfsshares
The Export configuration is created on the ceph dashboard and looks like this: root@proxmox07:~# cat /tmp/conf-nfsshares %url "rados://nfs-ganesha/ganesha-namespace/export-1"
root@proxmox07:~# cat /tmp/export-1 EXPORT { FSAL { secret_access_key = "AQDm7eFfJWvhIBAAuI++d0SNkdylNkMx5t/N8w=="; user_id = "ganesha"; name = "CEPH"; filesystem = "cephfs"; }
CLIENT { clients = 10.20.56.254; squash = "no_root_squash"; access_type = "RW"; }
pseudo = "/"; squash = "no_root_squash"; access_type = "NONE"; path = "/"; export_id = 1; transports = "UDP", "TCP"; protocols = 4; }
I get the following error message: .... 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] register_callbacks :FSAL :WARN :This libcephfs does not support registering callbacks. Ganesha will be unable to respond to MDS cache pressure. 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] select_filesystem :FSAL :CRIT :This libcephfs version doesn't support named filesystems. 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/) to (/) 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":2): 1 validation errors in block FSAL 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":2): Errors processing block (FSAL) 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":1): 1 validation errors in block EXPORT 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":1): Errors processing block (EXPORT) ....
When I remove filesystem = "cephfs"; in export-1 it works.
I believe my dashboard does not work together with its version or configuration of the libcephfs2 (both same source and version).
Or did I misconfigure something here?
Best regards and a healthy new year! Rainer _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Looks suspiciously like the ganesha packages from download.nfs-ganesha.org. Which were built with the ceph packages that were available from the mirrors.edge.kernel.org/debian repos at the time, (early October 2020) And that was ceph-12.2.11+dfsg1-2.1+b1. As far as "official" Debian ceph packages go that seems to be the latest AFAICT. On Fri, Jan 15, 2021 at 10:38 AM Daniel Gryniewicz <dang@redhat.com> wrote:
So, what matters is the version of Ceph that Ganesha was built against, not the version you're running against. 14.2 should be new enough to have ceph_select_filesystem() (which is the API in question. Where did you get the Ganesha packages, who built them, and which version of libcephfs was used when they were built?
Daniel
Hi,
I am running a Debian 10 with root@nfsshares-a:~# dpkg -l | grep ganesha ii nfs-ganesha 3.3-4 amd64 nfs-ganesha is a NFS server in User Space ii nfs-ganesha-ceph:amd64 3.3-4 amd64 nfs-ganesha fsal ceph libraries ii nfs-ganesha-rados-grace:amd64 3.3-4 amd64 nfs-ganesha ganesha-rados-grace program
and
root@nfsshares-a:~# dpkg -l | grep ceph ii ceph-common 14.2.16-pve1 amd64 common utilities to mount and interact with a ceph storage cluster ii libcephfs2 14.2.16-pve1 amd64 Ceph distributed file system client library ii nfs-ganesha-ceph:amd64 3.3-4 amd64 nfs-ganesha fsal ceph libraries ii python-ceph-argparse 14.2.16-pve1 all Python 2 utility libraries for Ceph CLI ii python-cephfs 14.2.16-pve1 amd64 Python 2 libraries for the Ceph libcephfs library
The config looks like this: root@nfsshares-a:~# grep '^[[:blank:]]*[^[:blank:]#;]' /etc/ganesha/ganesha.conf NFS_CORE_PARAM { Enable_NLM = false; Enable_RQUOTA = false; Protocols = 3,4; mount_path_pseudo = true; } NFS_KRB5 { Active_krb5 = false; } NFSv4 { RecoveryBackend = rados_ng; Minor_Versions = 1,2; } MDCACHE { Dir_Chunk = 0; } EXPORT_DEFAULTS { SecType = "sys"; Squash = No_Root_Squash; Attr_Expiration_Time = 0; } CEPH { Ceph_Conf = /etc/ceph/ceph.conf; } RADOS_KV { Ceph_Conf = /etc/ceph/ceph.conf; UserId = "ganesharecov"; nodeid = "nfsshares-a"; } RADOS_URLS { Ceph_Conf = /etc/ceph/ceph.conf; UserId = "ganeshaurls"; watch_url = "rados://nfs-ganesha/ganesha-namespace/conf-nfsshares"; } %url rados://nfs-ganesha/ganesha-namespace/conf-nfsshares
The Export configuration is created on the ceph dashboard and looks like
On 1/15/21 6:50 AM, rainer.stumbaum@gmail.com wrote: this:
root@proxmox07:~# cat /tmp/conf-nfsshares %url "rados://nfs-ganesha/ganesha-namespace/export-1"
root@proxmox07:~# cat /tmp/export-1 EXPORT { FSAL { secret_access_key = "AQDm7eFfJWvhIBAAuI++d0SNkdylNkMx5t/N8w=="; user_id = "ganesha"; name = "CEPH"; filesystem = "cephfs"; }
CLIENT { clients = 10.20.56.254; squash = "no_root_squash"; access_type = "RW"; }
pseudo = "/"; squash = "no_root_squash"; access_type = "NONE"; path = "/"; export_id = 1; transports = "UDP", "TCP"; protocols = 4; }
I get the following error message: .... 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] register_callbacks :FSAL :WARN :This libcephfs does not support registering callbacks. Ganesha will be unable to respond to MDS cache pressure. 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] select_filesystem :FSAL :CRIT :This libcephfs version doesn't support named filesystems. 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] mdcache_fsal_create_export :FSAL :MAJ :Failed to call create_export on underlying FSAL Ceph 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] fsal_cfg_commit :CONFIG :CRIT :Could not create export for (/) to (/) 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] main :NFS STARTUP :WARN :No export entries found in configuration file !!! 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":2): 1 validation errors in block FSAL 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":2): Errors processing block (FSAL) 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":1): 1 validation errors in block EXPORT 15/01/2021 12:47:30 : epoch 600180d2 : nfsshares-a : ganesha.nfsd-21185[main] config_errs_to_log :CONFIG :CRIT :Config File ("rados://nfs-ganesha/ganesha-namespace/export-1":1): Errors processing block (EXPORT) ....
When I remove filesystem = "cephfs"; in export-1 it works.
I believe my dashboard does not work together with its version or configuration of the libcephfs2 (both same source and version).
Or did I misconfigure something here?
Best regards and a healthy new year! Rainer _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Oh, sorry about that confusion! The Ceph is coming with Proxmox (https://pve.proxmox.com/wiki/Main_Page ) and I use the same version for the NFS Ganesha host. The NFS Ganesha is from download.nfs-ganesha.org. root@nfsshares-a:~# cat /etc/apt/sources.list.d/*.list deb http://download.proxmox.com/debian/ceph-nautilus buster main deb [arch=amd64] https://download.nfs-ganesha.org/3/LATEST/Debian/buster/amd64/apt buster main
Okay, so the builds on download.nfs-ganesha.org are build against ceph 12, which is too old to have those APIs. If Ceph 14 is available on the official debian mirrors, we can rebuild it, but if not, you'll not be able so use the "filesystem =" parameter on those builds. Daniel On 1/15/21 12:10 PM, rainer.stumbaum@gmail.com wrote:
Oh, sorry about that confusion!
The Ceph is coming with Proxmox (https://pve.proxmox.com/wiki/Main_Page ) and I use the same version for the NFS Ganesha host. The NFS Ganesha is from download.nfs-ganesha.org.
root@nfsshares-a:~# cat /etc/apt/sources.list.d/*.list deb http://download.proxmox.com/debian/ceph-nautilus buster main deb [arch=amd64] https://download.nfs-ganesha.org/3/LATEST/Debian/buster/amd64/apt buster main _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Actually I only have one CephFS file system. But the Ceph dashboard forces that parameter upon me. And NFS Ganesha apparently knows that the parameter "filesystem" with the compile time libcephfs is not usable and instead of ignoring this parameter it ignores the entire export. So I have two options: - Ask the Proxmox people to kindly build NFS Ganesha with their version of Ceph - Build NFS Ganesha on my own Or are there any other options? Rainer
It's not that hard to build ganesha. The .deb packaging bits are at https://github.com/nfs-ganesha/nfs-ganesha-debian Another option might be to convince the powers that be at Debian to update ceph in Buster/10. Buster is going to be around for a long time and ceph-12 is really just too old for people to reasonably use. On Fri, Jan 15, 2021 at 5:26 PM <rainer.stumbaum@gmail.com> wrote:
Actually I only have one CephFS file system. But the Ceph dashboard forces that parameter upon me.
And NFS Ganesha apparently knows that the parameter "filesystem" with the compile time libcephfs is not usable and instead of ignoring this parameter it ignores the entire export.
So I have two options: - Ask the Proxmox people to kindly build NFS Ganesha with their version of Ceph - Build NFS Ganesha on my own
Or are there any other options?
Rainer _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
So I tried to build it by myself and failed numerous times. Finally I just went the debian way and to have it clean and reproducible I did it then again within a docker container. Here is the Dockerfile to build the latest nfs-ganesha on top of Debian with the Ceph version provided by Proxmox. ---- rstumbaum@controlnode01.dc1:~/docker-nfs-ganesha-build$ cat Dockerfile ARG DEBIAN_RELEASE="buster" ARG CEPH_RELEASE_PVE="nautilus" FROM debian:${DEBIAN_RELEASE} AS build-env ARG DEBIAN_RELEASE ARG CEPH_RELEASE_PVE ADD http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg RUN chmod 644 /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg &&\ echo "deb http://download.proxmox.com/debian/ceph-${CEPH_RELEASE_PVE} ${DEBIAN_RELEASE} main" >/etc/apt/sources.list.d/ceph.list &&\ echo "deb http://deb.debian.org/debian ${DEBIAN_RELEASE}-backports main" >/etc/apt/sources.list.d/backports.list &&\ echo "deb-src http://ftp.de.debian.org/debian/ sid main contrib non-free" >//etc/apt/sources.list.d/sid-src.list RUN apt-get update RUN apt-get install -y build-essential git-buildpackage cmake bison flex doxygen lsb-release pkgconf \ nfs-common ceph-common RUN apt-get install -y -t ${DEBIAN_RELEASE}-backports libglusterfs-dev WORKDIR /build RUN apt-get build-dep -y libntirpc-dev &&\ apt-get source -y --compile libntirpc-dev &&\ dpkg -i *.deb RUN apt-get build-dep -y nfs-ganesha &&\ apt-get source -y --compile nfs-ganesha VOLUME /export rstumbaum@controlnode01.dc1:~/docker-nfs-ganesha-build$ ---- To build I just run docker build . -t nfs-ganesha-debs Then to get the build result I just run docker run -v $HOME/docker-nfs-ganesha-build/export:/export -it nfs-ganesha-debs bash and the within the container cp -a *.deb /export/ exit The Dockerfile uses the Debian sid build rules to currently build version 3.4 . The backports is used to provide only the libglusterfs-dev package. After building I just docker image rm the 1.6GByte build container and keep only the 16 Mbyte of .deb files... It would be easier for newbies like me to get started if the project could provide a Dockerfile to build and run nfs-ganesha as a multilevel container with a default config inside. Best regards Rainer
The whole point of that parameter is to be used in situations when no parameter would fail (since no parameter mounts /, not any particular /path). Thus, we can't ignore it, sorry. Daniel On 1/15/21 5:25 PM, rainer.stumbaum@gmail.com wrote:
Actually I only have one CephFS file system. But the Ceph dashboard forces that parameter upon me.
And NFS Ganesha apparently knows that the parameter "filesystem" with the compile time libcephfs is not usable and instead of ignoring this parameter it ignores the entire export.
So I have two options: - Ask the Proxmox people to kindly build NFS Ganesha with their version of Ceph - Build NFS Ganesha on my own
Or are there any other options?
Rainer _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
participants (3)
-
Daniel Gryniewicz -
Kaleb Keithley -
rainer.stumbaum＠gmail.com