6:26 a.m.
Thank you for the links. For the su case, since it works with gluster
NFS and works with kernel NFS... I do not believe the problem to be an issue
with the image itself such as permissions as /bin/su. In the case of gluster
nfs, it's the exact same volume exported and gnfs doesn't trigger the
issue.
I'm trying to be careful for the nfs3 case not to say where the fault is
as I have no idea but ganesha nfs clearly is involved in the difference
in some way (potentially a victim).
For the case of 'noacl' for the overlay... It appears this is not yet in
rhel77. I can investigate building my own kernel and trying it out.
Thanks for the ideas and feel free to pass any others along.
I think I may have lost my chance to test this on the arm supercomputer but
I may be able squeeze in if I figure out a work around in the next few
days. I started to look if I can hack the ganesha source to send an
empty ACL list instead of "not supported" for the case of NFS v3 as we
think that's the only difference.
Ganesha NFS v4 seems to be unusable for this workload at this time.
Overlay w/ro NFS is something I'd like to make available for other large
clusters we release because having writable compute nodes when there are no
disks is very helpful for "stunt mode" where you may wish to parallel-copy
rpms or files without rebooting thousands of nodes. Typically, as the system
becomes a more stable workhorse, read-only NFS and/or a means of
having per-host becomes preferred.
I think you're on to something with the 'noacl' thing and I can build my
own test kernel to try it out. I wouldn't be able to run that kernel on
customer systems but it would at least be a really good data point.
> Okay. Could you try steps mentioned below -
>
> https://www.thelinuxfaq.com/101-linux-su-cannot-set-groups-operation-not-...
>
> Also found that a new option has been added to overlayfs to honour disabling
> ACLs -
>
> https://lkml.org/lkml/2019/5/2/7
>
> Not sure if the kernel version you are using has this change. Maybe you can
> give a try adding that mount option and use NFSv3.
>
> Thanks,
> Soumya
>
> On 8/15/19 2:46 AM, Erik Jacobson wrote:
> > Thank you - responses below:
> >
> > > Checked the tcpdump in more detail. "Execute" permission seem to
have got
> > > denied for following paths even for root user -
> > >
> > > /cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
> > >
> > >
/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel/usr/share/zoneinfo/America/Chicago
> > >
> > > Whereas in case of gNFS pkt trace, root user had Execute access.
> > >
> > > I am not sure why there is a difference here anf if this caused issue.
Could
> > > you cross check the perms on both mount point and at the backend bricks
> > > path?
> >
> >
> > leader1 (ganesha in question) has that gluster mounted under
> > /opt/clmgr/shared_storage,so:
> >
> >
> > [root@leader1 ~]# ls -l
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel//usr/share/zoneinfo/America/Chicago
> > -rw-r--r-- 1 root root 718 Aug 11 12:19
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
> > -rw-r--r-- 2 root root 3559 May 7 2018
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel//usr/share/zoneinfo/America/Chicago
> >
> >
> >
> > Here is the path on the 9 leaders (gluster 3x3):
> > These two files are on subvolume 1, so they appear on three of the 9
> > bricks and leaders (see end).
> >
> >
> >
> > [root@indeed ~]# pdsh -w leader[1-9] ls -l
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago|dshbak
> > leader8: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
> > leader5: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
> > leader5: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
> > leader8: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
> > pdsh@indeed: leader5: ssh exited with exit code 2
> > pdsh@indeed: leader8: ssh exited with exit code 2
> > leader6: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
> > leader6: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
> > pdsh@indeed: leader6: ssh exited with exit code 2
> > leader4: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
> > leader4: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
> > pdsh@indeed: leader4: ssh exited with exit code 2
> > leader9: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
> > leader9: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
> > leader7: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
> > leader7: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
> > pdsh@indeed: leader9: ssh exited with exit code 2
> > pdsh@indeed: leader7: ssh exited with exit code 2
> > ----------------
> > leader1
> > ----------------
> > -rw-r--r-- 2 root root 718 Aug 11 12:19
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
> > -rw-r--r-- 3 root root 3559 May 7 2018
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago
> > ----------------
> > leader2
> > ----------------
> > -rw-r--r-- 2 root root 718 Aug 11 12:19
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
> > -rw-r--r-- 3 root root 3559 May 7 2018
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago
> > ----------------
> > leader3
> > ----------------
> > -rw-r--r-- 2 root root 718 Aug 11 12:19
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
> > -rw-r--r-- 3 root root 3559 May 7 2018
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago
> > _______________________________________________
> > Devel mailing list -- devel(a)lists.nfs-ganesha.org
> > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
> >