Okay, now I have some real errors: 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd-1006530[svc_24] fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral, handle: 0x7f90bda3cc00, valid_mask: 0, request_mask: 82, supported: 0, error: Forbidden action 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd-1006530[svc_24] fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral, handle: 0x7f90bda3cc00, valid_mask: 0, request_mask: 82, supported: 0, error: Forbidden action 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd-1006530[svc_5] fs_rm_revoked_handles :CLIENT ID :EVENT :opendir /var/lib/nfs/ganesha/v4recov/node0/::ffff:127.0.0.1-(22:Linux NFSv4.2 tom-7590) failed errno: Permission denied (13) 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd-1006530[svc_5] fs_rm_clid_impl :CLIENT ID :EVENT :Failed to remove client recovery dir (/var/lib/nfs/ganesha/v4recov/node0/::ffff:127.0.0.1-(22:Linux NFSv4.2 tom-7590)), errno: Permission denied (13) For the second two errors, it seems I need to change where the client recovery dir is kept to something user-accessible. I can't seem to find an option for this though. Not sure what to do about "Failed to get attrs for referral"?

So, we have tested Ganesha run as a non-root user, and it works, but with quite a few caveats, most of which you've found already. First, you cannot use nfsv3 at all, as that uses portmap/rpcbind, which cannot be used unless you're root. You can turn it off at runtime, as you've discovered, and nfsv4 will work then. Or, if you're compiling yourself, you can disable it at compile time with USE_NFS3=OFF USE_RQUOTA=OFF when you run cmake. Second, as Frank mentioned, not all FSALs work as non-root. Specifically, FSAL_VFS doesn't work, since it depends on *_by_handle APIs that are restricted to root. Third is that Ganesha uses a number of files for various things. The conf file is only read, so it's usually okay that it lives in /etc. The log file is written, so you need write access to the directory it lives in; log file is configurable, so that's okay. The PID file is written in /var/run by default, but is settable on the command line with -p. And the last is the recovery directory. This is read/write, so you need access to that. As Frank said, this is configurable at runtime, but only on next. Everything is changeable at build time by building Ganesha to run in a prefix. This is how I do my testing, so I know it works. Basically, you pix a prefix, and set it using CMAKE_INSTALL_PREFIX=/prefix Ganesha will then be installed in and use: /prefix/bin /prefix/etc /prefix/var /prefix/run /prefix/lib and so on. You can make the entire prefix owned by the user, and then directory permissions don't matter. Hope this helps, and feel free to ask questions. Daniel On 6/16/21 11:28 PM, Frank Filz wrote:

If you use the next branch the recovery directory is now configurable.

What FSAL are you using? FSAL_VFS depends on name_to_handle_at and open_by_handle both of which require a CAP. There may be a few other CAPs necessary. Also, you will have to squash all users to the user running Ganesha since Ganesha won't be able to seteuid/setegid to act as other users. These problems are less of an issue for other FSALs.

Frank

...

-----Original Message----- From: Tom McLaughlin [mailto:pyro777@gmail.com] Sent: Wednesday, June 16, 2021 6:04 PM To: support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Re: Possible to run NFSv4 server as non-root user?

Okay, now I have some real errors:

16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd- 1006530[svc_24] fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral, handle: 0x7f90bda3cc00, valid_mask: 0, request_mask: 82, supported: 0, error: Forbidden action 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd- 1006530[svc_24] fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral, handle: 0x7f90bda3cc00, valid_mask: 0, request_mask: 82, supported: 0, error: Forbidden action 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd- 1006530[svc_5] fs_rm_revoked_handles :CLIENT ID :EVENT :opendir /var/lib/nfs/ganesha/v4recov/node0/::ffff:127.0.0.1-(22:Linux NFSv4.2 tom- 7590) failed errno: Permission denied (13) 16/06/2021 17:58:47 : epoch 60ca9d4a : tom-7590 : ganesha.nfsd- 1006530[svc_5] fs_rm_clid_impl :CLIENT ID :EVENT :Failed to remove client recovery dir (/var/lib/nfs/ganesha/v4recov/node0/::ffff:127.0.0.1-(22:Linux NFSv4.2 tom-7590)), errno: Permission denied (13)

For the second two errors, it seems I need to change where the client recovery dir is kept to something user-accessible. I can't seem to find an option for this though.

Not sure what to do about "Failed to get attrs for referral"? _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org

---

Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org

This is an unfortunate side effect of the difference between NFS semantics and POSIX semantics. The API provided by the kernel to support NFS semantics requires special privileges. I believe it's possible to write a library that maps open FDs to NFS handles. This would allow you to open a file when you give out a handle, and keep it open, mapping the handle back to the FD. The downside of this is that you either need to keep FDs open forever (using up your max FDs pretty quickly), or time them out, at which point you have to send an error for that handle. This would more-or-less work for some circumstances, but definitely not for all. Another possibility is to run Ganesha in a user namespace. This maps UIDs inside the namespace to non-privileged UIDs outside the namespace. I don't know if this allows the necessary CAPs inside the namespace or not, but it might. If it does, then you could have access via Ganesha to the subtree inside the namespace. Again, not a 100% solution, but maybe better than nothing. Daniel On 6/27/21 10:04 PM, Tom McLaughlin wrote:

Thank you both for the help!

The capabilities needed to run on FSAL_VFS are a bit of a show-stopper for me. But for posterity let me write down my experience trying to get it to work.

Here's the crazy capsh command I came up with to get the process running with the necessary ~5 or so capabilities (I found these listed in the source):

sudo capsh --caps="cap_dac_read_search+eip cap_dac_override=+eip cap_fowner=+eip cap_setgid=+eip cap_setuid=+eip cap_setpcap,cap_setuid,cap_setgid+ep" --keep=1 --user=tom --addamb=cap_dac_read_search,cap_dac_override,cap_fowner,cap_setgid,cap_setuid -- -c "/home/tom/tools/ganesha/result/bin/ganesha.nfsd -F -f ganesha.conf -L ./logs.txt -p ./ganesha.pid"

and here's my config:

NFS_KRB5 { Active_krb5 = false; }

NFS_CORE_PARAM { Protocols = 4; NFS_Port = 7777; Rquota_Port = 7778; Enable_RQUOTA = false; }

NFSv4 { RecoveryRoot = "/tmp/recovery_root"; }

EXPORT { Export_Id = 2;

Path = /tmp/exported; Pseudo = /tmp/exported;

Access_Type = RW; Squash = No_Root_Squash;

Transports = "TCP"; Protocols = 4; # SecType = none; SecType = "sys";

FSAL { Name = VFS; } }

Note that the custom recovery root and the export dirs need to exist, or it immediately segfaults.

With this setup, the server *sort of* works. I'm able to run a `mount` command on the same machine. If I put a file in the exported dir, I can read it in the mount. Writing to the mount is more problematic -- if I echo some data into a file, it tends to hang for several minutes and then sometimes succeed (such that I can read it from the exported dir).

My dream would be to be able to run Ganesha against FSAL_VFS as a normal user inside a Docker container, in order to share some folder that the user controls. But I can see that at a minimum this will require running the container with special capabilities. _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org

On Mon, 2021-06-28 at 09:07 -0700, Frank Filz wrote:

There used to be a FSAL_POSIX that kept a handle database to map handles to files by path. This was removed once we had FSAL_VFS with the open_by_handle and name_to_handle_at facilities. Unfortunately, due to the things they can do, it was deemed they needed to be privileged system calls requiring a CAP. Additionally, if nfs-ganesha is going to manage multiple users, it needs to be able to seteuid/setegid. We DO have a new capability to build without that need (it doesn't work on MacOS) and if you only needed to export files owned by a single user, it would be easy to setup Ganesha to do so, and if you needed to be able to export to non-owner users, we could probably make that work, especially if non-owners would be read-only. There are some complexities about writing to files, but I think they can be ignored in this use case (there may be some quota implications, but if an owner is allowing non-owners to expand files, the quota definitely should be assigned to the owner anyway, so doing the writes as the owner would be fine).

But the only way to work around the CAP for open_by_handle is to add some kind of handle mapping.

Note that as long as the file is in mdcache, even if not open, we don't need to do open_by_handle. Then the question becomes if name_to_handle_at also requires the CAP, if so, we're sunk because we have no way of creating handles (it doesn't look like name_to_handle_at requires CAP_DAC_READ_SEARCH).

Otherwise, we might actually be OK for your use case if the clients don't need access to files that have been evicted from cache (and of course a Ganesha restart evicts everything from cache...).

I'm not sure why we need cap_dac_override and cap_fowner - Jeff you did some looking at CAPs, do you know why?

Not right offhand. DAC_OVERRIDE allows you to override any permission bit checks, and FOWNER allows allows you to override checks for matching owner (see capabilities(7)). It's conceivable that FSAL_VFS would need those, but I'm not familiar with what it's doing specifically. Long term, it might be nice to be able to drop even more capabilities when using FSALs that don't require it. For instance, FSAL_CEPH doesn't need any of them since it's a completely synthetic backend. It can generally run with ganesha as a non-privileged user, but most deployments still run it as root.

Frank

...

-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: Monday, June 28, 2021 6:38 AM To: support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Re: Possible to run NFSv4 server as non-root user?

This is an unfortunate side effect of the difference between NFS semantics and POSIX semantics. The API provided by the kernel to support NFS semantics requires special privileges.

I believe it's possible to write a library that maps open FDs to NFS handles. This would allow you to open a file when you give out a handle, and keep it open, mapping the handle back to the FD. The downside of this is that you either need to keep FDs open forever (using up your max FDs pretty quickly), or time them out, at which point you have to send an error for that handle. This would more- or-less work for some circumstances, but definitely not for all.

Another possibility is to run Ganesha in a user namespace. This maps UIDs inside the namespace to non-privileged UIDs outside the namespace. I don't know if this allows the necessary CAPs inside the namespace or not, but it might. If it does, then you could have access via Ganesha to the subtree inside the namespace. Again, not a 100% solution, but maybe better than nothing.

Daniel

...

Thank you both for the help!

The capabilities needed to run on FSAL_VFS are a bit of a show-stopper for me. But for posterity let me write down my experience trying to get it to work.

Here's the crazy capsh command I came up with to get the process running with the necessary ~5 or so capabilities (I found these listed in the source):

sudo capsh --caps="cap_dac_read_search+eip cap_dac_override=+eip cap_fowner=+eip cap_setgid=+eip cap_setuid=+eip cap_setpcap,cap_setuid,cap_setgid+ep" --keep=1 --user=tom -- addamb=cap_dac_read_search,cap_dac_override,cap_fowner,cap_setgid,cap_s etuid -- -c "/home/tom/tools/ganesha/result/bin/ganesha.nfsd -F -f ganesha.conf -L ./logs.txt -p ./ganesha.pid"

and here's my config:

NFS_KRB5 { Active_krb5 = false; }

NFS_CORE_PARAM { Protocols = 4; NFS_Port = 7777; Rquota_Port = 7778; Enable_RQUOTA = false; }

NFSv4 { RecoveryRoot = "/tmp/recovery_root"; }

EXPORT { Export_Id = 2;

Path = /tmp/exported; Pseudo = /tmp/exported;

Access_Type = RW; Squash = No_Root_Squash;

Transports = "TCP"; Protocols = 4; # SecType = none; SecType = "sys";

FSAL { Name = VFS; } }

Note that the custom recovery root and the export dirs need to exist, or it immediately segfaults.

With this setup, the server *sort of* works. I'm able to run a `mount` command on the same machine. If I put a file in the exported dir, I can read it in

On 6/27/21 10:04 PM, Tom McLaughlin wrote: the mount. Writing to the mount is more problematic -- if I echo some data into a file, it tends to hang for several minutes and then sometimes succeed (such that I can read it from the exported dir).

...

My dream would be to be able to run Ganesha against FSAL_VFS as a normal

user inside a Docker container, in order to share some folder that the user controls. But I can see that at a minimum this will require running the container with special capabilities.

...

_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org

_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org

-- Jeff Layton <jlayton@redhat.com>

Hello, I was trying to run the ganesha.nfsd as nonroot user and got this error: 21/07/2021 08:53:34 : epoch 60f7e018 : nfs-ganesha-nfs-server-provisioner-0 : nfs-ganesha-20[svc_9] fsal_common_is_referral :FSAL :EVENT :Failed to get attrs for referral, handle: 0x7f9cdaf5b780, valid_mask: 0, request_mask: 82, supported: 0, error: Forbidden action My vfs.conf file is below: EXPORT { # Export Id (mandatory, each EXPORT must have a unique Export_Id) Export_Id = 0; # Exported path (mandatory) Path = /nonexistent; # Pseudo Path (required for NFS v4) Pseudo = /nonexistent; # Required for access (default is None) # Could use CLIENT blocks instead Access_Type = RW; # Exporting FSAL FSAL { Name = VFS; } } NFS_Core_Param { MNT_Port = 20048; NLM_Port = 32803; fsid_device = true; Protocols = 4; enable_RQUOTA = false; } NFSV4 { Grace_Period = 90; } EXPORT { Export_Id = 1; Path = /export/pvc-7de28a2a-75da-4136-b381-fe5ebc686caa; Pseudo = /export/pvc-7de28a2a-75da-4136-b381-fe5ebc686caa; Access_Type = RW; Squash = no_root_squash; SecType = sys; Filesystem_id = 1.1; Protocols = 4; Transports = TCP; FSAL { Name = VFS; } } Is it the same issue being faced in this thread? Am I missing capabilities?