Thank you for the links. For the su case, since it works with gluster NFS and works with kernel NFS... I do not believe the problem to be an issue with the image itself such as permissions as /bin/su. In the case of gluster nfs, it's the exact same volume exported and gnfs doesn't trigger the issue. I'm trying to be careful for the nfs3 case not to say where the fault is as I have no idea but ganesha nfs clearly is involved in the difference in some way (potentially a victim). For the case of 'noacl' for the overlay... It appears this is not yet in rhel77. I can investigate building my own kernel and trying it out. Thanks for the ideas and feel free to pass any others along. I think I may have lost my chance to test this on the arm supercomputer but I may be able squeeze in if I figure out a work around in the next few days. I started to look if I can hack the ganesha source to send an empty ACL list instead of "not supported" for the case of NFS v3 as we think that's the only difference. Ganesha NFS v4 seems to be unusable for this workload at this time. Overlay w/ro NFS is something I'd like to make available for other large clusters we release because having writable compute nodes when there are no disks is very helpful for "stunt mode" where you may wish to parallel-copy rpms or files without rebooting thousands of nodes. Typically, as the system becomes a more stable workhorse, read-only NFS and/or a means of having per-host becomes preferred. I think you're on to something with the 'noacl' thing and I can build my own test kernel to try it out. I wouldn't be able to run that kernel on customer systems but it would at least be a really good data point.
Okay. Could you try steps mentioned below -
https://www.thelinuxfaq.com/101-linux-su-cannot-set-groups-operation-not-per...
Also found that a new option has been added to overlayfs to honour disabling ACLs -
https://lkml.org/lkml/2019/5/2/7
Not sure if the kernel version you are using has this change. Maybe you can give a try adding that mount option and use NFSv3.
Thanks, Soumya
On 8/15/19 2:46 AM, Erik Jacobson wrote:
Thank you - responses below:
Checked the tcpdump in more detail. "Execute" permission seem to have got denied for following paths even for root user -
/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel/usr/share/zoneinfo/America/Chicago
Whereas in case of gNFS pkt trace, root user had Execute access.
I am not sure why there is a difference here anf if this caused issue. Could you cross check the perms on both mount point and at the backend bricks path?
leader1 (ganesha in question) has that gluster mounted under /opt/clmgr/shared_storage,so:
[root@leader1 ~]# ls -l /opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group /opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel//usr/share/zoneinfo/America/Chicago -rw-r--r-- 1 root root 718 Aug 11 12:19 /opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group -rw-r--r-- 2 root root 3559 May 7 2018 /opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel//usr/share/zoneinfo/America/Chicago
Here is the path on the 9 leaders (gluster 3x3): These two files are on subvolume 1, so they appear on three of the 9 bricks and leaders (see end).
[root@indeed ~]# pdsh -w leader[1-9] ls -l /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago|dshbak leader8: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or directory leader5: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or directory leader5: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago: No such file or directory leader8: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago: No such file or directory pdsh@indeed: leader5: ssh exited with exit code 2 pdsh@indeed: leader8: ssh exited with exit code 2 leader6: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or directory leader6: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago: No such file or directory pdsh@indeed: leader6: ssh exited with exit code 2 leader4: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or directory leader4: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago: No such file or directory pdsh@indeed: leader4: ssh exited with exit code 2 leader9: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or directory leader9: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago: No such file or directory leader7: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or directory leader7: ls: cannot access /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago: No such file or directory pdsh@indeed: leader9: ssh exited with exit code 2 pdsh@indeed: leader7: ssh exited with exit code 2 ---------------- leader1 ---------------- -rw-r--r-- 2 root root 718 Aug 11 12:19 /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group -rw-r--r-- 3 root root 3559 May 7 2018 /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago ---------------- leader2 ---------------- -rw-r--r-- 2 root root 718 Aug 11 12:19 /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group -rw-r--r-- 3 root root 3559 May 7 2018 /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago ---------------- leader3 ---------------- -rw-r--r-- 2 root root 718 Aug 11 12:19 /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group -rw-r--r-- 3 root root 3559 May 7 2018 /data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org