(Sorry for the duplicate email to ganesha lists, but I wanted to widen
it to include the ceph lists)
In response to some cries for help over IRC, I wrote up this blog post
the other day, which discusses how to set up parallel serving over
CephFS:
https://jtlayton.wordpress.com/2018/12/10/deploying-an-active-active-nfs-cl…
Feel free to comment if you have questions. We may be want to eventually
turn this into a document in the ganesha or ceph trees as well.
Cheers!
--
Jeff Layton <jlayton(a)redhat.com>
In response to some cries for help over IRC, I wrote up this blog post
the other day, which discusses how to set up parallel serving over
CephFS:
https://jtlayton.wordpress.com/2018/12/10/deploying-an-active-active-nfs-cl…
Feel free to comment if you have questions. We may be want to eventually
turn this into a document in the ganesha tree as well.
Cheers!
--
Jeff Layton <jlayton(a)redhat.com>
Dear Tom
We have it running here and this is our ganesha configuration:
NFS_KRB5
{
PrincipalName = nfs(a)myhost.mydomain.com ;
KeytabPath = /etc/krb5.keytab ;
Active_krb5 = YES ;
}
NFS_CORE_PARAM {
NSM_Use_Caller_Name = true;
Clustered = false;
Rquota_Port = 875;
}
EXPORT
{
Export_Id = 1;
Path = "/srv/shares";
Pseudo = "/srv/shares";
Protocols = "4";
Access_Type = RW;
Squash = no_root_squash;
# Squash = root_squash;
Sectype = krb5,krb5i,krb5p;
Disable_ACL = FALSE;
FSAL
{
name = GLUSTER;
hostname = "myglusterserver.mydomain.com";
volume = "myglustervolume";
}
}
But we have issues with NFSv4 ACLs. When we want to change the ACL on the mountpoint as a domain user, we get bizarre behavior. We are using a brick formatted as xfs which should support extended attributes but somehow it's not bein recognized. Maybe somebody in here could bring us on the right track. At the moment we don't know how to solve this.
This can be reproduced as follows with the configuration above :
1) Mount share as root on a client:
mount -vvv -t nfs4 -o sec=krb5,rw,acl,timeo=10 myhost.domain.com:/srv/shares /mnt/
2) Create a file as domain user
touch /mnt/test
nfs4_getfacl /mnt/test # getfacl returns nothing for a couple of seconds after that
nfs4_getfacl /mnt/test
A::OWNER@:rwatTcCy
A::GROUP@:rwatcy
A::EVERYONE@:tcy
3) After this we try to change the ACL
nfs4_setfacl -a "A:g:admins@mydomain.com:rwaDxtcy" "/mnt/test"
Failed setxattr operation: Invalid argument
This fails and we can see in the ganesha.log on the server
20/06/2018 11:49:37 : epoch 5b2a225d : myhost.mydomain.com : ganesha.nfsd-14820[work-38] glusterfs_set_acl :FSAL :MAJ :failed to set access type posix acl 20/06/2018 11:49:37 : epoch 5b2a225d : myhost.mydomain.com : ganesha.nfsd-14820[work-38] glusterfs_setattr2 :FSAL :CRIT :setattrs failed with error Success
and on the gluster server
[2018-06-20 09:49:37.673738] I [MSGID: 139001] [posix-acl.c:269:posix_acl_log_permit_denied] 0-myglustervolume-access-control: client: vm-0026.service.int.rabe.ch-14820-2018/06/20-09:46:05:292146-myglustervolume-client-1-0-0, gfid: d0d93931-8915-4a2f-b6be-f53ac154eacf, req(uid:1101,gid:1101,perm:2,ngrps:3), ctx(uid:1101,gid:1101,in-groups:1,perm:000,updated-fop:SETATTR, acl:-) [Permission denied]
[2018-06-20 09:49:37.674074] I [MSGID: 115060] [server-rpc-fops.c:899:_gf_server_log_setxattr_failure] 0-myglustervolume-server: 43: SETXATTR /test (d0d93931-8915-4a2f-b6be-f53ac154eacf) ==> glusterfs.posix.acl, client: myhost.mydomain.com-14820-2018/06/20-09:46:05:292146-myglustervolume-client-1-0-0, error-xlator: myglustervolume-access-control
[2018-06-20 09:49:37.674090] I [MSGID: 115060] [server-rpc-fops.c:929:server_setxattr_cbk] 0-myglustervolume-server: client: myhost.mydomain.com-14820-2018/06/20-09:46:05:292146-myglustervolume-client-1-0-0, error-xlator: myglustervolume-access-control [Permission denied]
Today I have noticed something that I can't understand. xfs should support extended attributes by default but on the gluster-server I can see this error:
[2018-12-05 16:30:35.492623] W [posix.c:4929:posix_getxattr] 0-myglustervolume-posix: Extended attributes not supported (try remounting brick with 'user_xattr' flag)
[2018-12-05 16:30:35.492649] E [MSGID: 113001] [posix.c:4940:posix_getxattr] 0-myglustervolume-posix: getxattr failed on /srv/gluster/myglustervolume/gb-01/brick/.glusterfs/b8/26/b8264d88-43a9-4a5a-8381-aaa041cd8c9f: system.nfs4_acl [Operation not supported]
Anybody a hint, what the problem is?
Thanks
Simon
