I did more debgging and research. It seems to me the problem (only on
aarch64) only happens when a process is no longer root. So like when
polkit changes to the polkitd user, then it is unable to access files.
This is supported by me making a local user in root image. I cannot
su to that user no matter what. ("Operation not supported").
However, if I get rid of the overlay and just use the read-only mount
vers3 nfs point, I can chroot in to it and su to the user just fine.
So this problem only happens with a combination of:
- Ganesha NFS (kernel nfs, gluster NFS ok)
- rhel76 aarch64 client (x86_64 rhel76 client ok)
- overlay kernel module doing a union
The other combinations work.
Note that I haven't tried other Linux distros in any part of the
solution yet.
I seem to be stuck in the inconvenient place of being in a corner case
without the experience to get myself out :( There is some interaction of
x86_64 + overlayfs + ganesha that fails related to file permissions for
non-root users. :(
I'm not sure how much further I can dive in to this on my own but I'll
try to think of some things to look at.
Erik