4:40 a.m.
We have issues with NFSv4 ACLs. When we want to change the ACL on the mountpoint on the
client as a domain (freeipa) user, we get bizarre behavior. It works if we set ACLs as
root user. We are using a brick formatted as xfs which should support extended attributes
but somehow it's not being recognized. Maybe somebody in here could bring us on the
right track. At the moment we don't know how to solve this.
We are running a gluster environment (V3.12.14) on Centos 7.5.1804 with a separate
Ganesha-NFS (V2.5.5) server in a FreeIPA-joined environment.
Assuming working ganesha-configuration is in place this can be reproduced as follows:
1) Mount share as root on a client (NFS version doesn't seem to matter, tested with
4.0,4.1,4.2):
mount -vvv -t nfs4 -o sec=krb5,rw,acl,timeo=10 myhost.domain.com:/srv/shares /mnt/
2) Create a file as domain user
touch /mnt/test
nfs4_getfacl /mnt/test # getfacl returns nothing for a couple of seconds after that
nfs4_getfacl /mnt/test
A::OWNER@:rwatTcCy
A::GROUP@:rwatcy
A::EVERYONE@:tcy
3) After this we try to change the ACL
nfs4_setfacl -a "A:g:admins@mydomain.com:rwaDxtcy" "/mnt/test"
Failed setxattr operation: Invalid argument
This fails and we can see in the ganesha.log on the server
20/06/2018 11:49:37 : epoch 5b2a225d : myhost.mydomain.com : ganesha.nfsd-14820[work-38]
glusterfs_set_acl :FSAL :MAJ :failed to set access type posix acl 20/06/2018 11:49:37 :
epoch 5b2a225d : myhost.mydomain.com : ganesha.nfsd-14820[work-38] glusterfs_setattr2
:FSAL :CRIT :setattrs failed with error Success
and on the gluster server
[2018-06-20 09:49:37.673738] I [MSGID: 139001]
[posix-acl.c:269:posix_acl_log_permit_denied] 0-myglustervolume-access-control: client:
vm-0026.service.int.rabe.ch-14820-2018/06/20-09:46:05:292146-myglustervolume-client-1-0-0,
gfid: d0d93931-8915-4a2f-b6be-f53ac154eacf, req(uid:1101,gid:1101,perm:2,ngrps:3),
ctx(uid:1101,gid:1101,in-groups:1,perm:000,updated-fop:SETATTR, acl:-) [Permission denied]
[2018-06-20 09:49:37.674074] I [MSGID: 115060]
[server-rpc-fops.c:899:_gf_server_log_setxattr_failure] 0-myglustervolume-server: 43:
SETXATTR /test (d0d93931-8915-4a2f-b6be-f53ac154eacf) ==> glusterfs.posix.acl, client:
myhost.mydomain.com-14820-2018/06/20-09:46:05:292146-myglustervolume-client-1-0-0,
error-xlator: myglustervolume-access-control
[2018-06-20 09:49:37.674090] I [MSGID: 115060] [server-rpc-fops.c:929:server_setxattr_cbk]
0-myglustervolume-server: client:
myhost.mydomain.com-14820-2018/06/20-09:46:05:292146-myglustervolume-client-1-0-0,
error-xlator: myglustervolume-access-control [Permission denied]
Today I have noticed something that I can't understand. xfs should support extended
attributes by default but on the gluster-server I can see this error:
[2018-12-05 16:30:35.492623] W [posix.c:4929:posix_getxattr] 0-myglustervolume-posix:
Extended attributes not supported (try remounting brick with 'user_xattr' flag)
[2018-12-05 16:30:35.492649] E [MSGID: 113001] [posix.c:4940:posix_getxattr]
0-myglustervolume-posix: getxattr failed on
/srv/gluster/myglustervolume/gb-01/brick/.glusterfs/b8/26/b8264d88-43a9-4a5a-8381-aaa041cd8c9f:
system.nfs4_acl [Operation not supported]
4) after a couple of seconds then the permissions are completely removed
nfs4_getfacl /mnt/test
A::OWNER@:tcy
A::GROUP@:tcy
A::EVERYONE@:tcy
All this works perfectly fine if do it without nfs-ganesha. Mount gluster volume on the
server and serve it via nfs. There is a good chance we are doing something fundamentally
wrong. We are not looking to get it entirely resolved. But please share your thoughts
what's going on here or where we have to look for an issue.
Thanks
Simon