2:16 p.m.
Thank you - responses below:
Checked the tcpdump in more detail. "Execute" permission
seem to have got
denied for following paths even for root user -
/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel/usr/share/zoneinfo/America/Chicago
Whereas in case of gNFS pkt trace, root user had Execute access.
I am not sure why there is a difference here anf if this caused issue. Could
you cross check the perms on both mount point and at the backend bricks
path?
leader1 (ganesha in question) has that gluster mounted under
/opt/clmgr/shared_storage,so:
[root@leader1 ~]# ls -l
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel//usr/share/zoneinfo/America/Chicago
-rw-r--r-- 1 root root 718 Aug 11 12:19
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel/etc/group
-rw-r--r-- 2 root root 3559 May 7 2018
/opt/clmgr/shared_storage/image/images_ro_nfs/rhel76-aarch64-newkernel//usr/share/zoneinfo/America/Chicago
Here is the path on the 9 leaders (gluster 3x3):
These two files are on subvolume 1, so they appear on three of the 9
bricks and leaders (see end).
[root@indeed ~]# pdsh -w leader[1-9] ls -l
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago|dshbak
leader8: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
leader5: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
leader5: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
leader8: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
pdsh@indeed: leader5: ssh exited with exit code 2
pdsh@indeed: leader8: ssh exited with exit code 2
leader6: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
leader6: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
pdsh@indeed: leader6: ssh exited with exit code 2
leader4: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
leader4: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
pdsh@indeed: leader4: ssh exited with exit code 2
leader9: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
leader9: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
leader7: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group: No such file or
directory
leader7: ls: cannot access
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago:
No such file or directory
pdsh@indeed: leader9: ssh exited with exit code 2
pdsh@indeed: leader7: ssh exited with exit code 2
----------------
leader1
----------------
-rw-r--r-- 2 root root 718 Aug 11 12:19
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
-rw-r--r-- 3 root root 3559 May 7 2018
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago
----------------
leader2
----------------
-rw-r--r-- 2 root root 718 Aug 11 12:19
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
-rw-r--r-- 3 root root 3559 May 7 2018
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago
----------------
leader3
----------------
-rw-r--r-- 2 root root 718 Aug 11 12:19
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64/etc/group
-rw-r--r-- 3 root root 3559 May 7 2018
/data/brick_cm_shared/image/images_ro_nfs/rhel76-aarch64//usr/share/zoneinfo/America/Chicago