Thank you for logging as well at the logs! A couple replies below.
/home permissions returned are 0755 and '/' has 0555
That's interesting. bash-4.2# chroot /a ls -ld / /home drwxr-xr-x 1 root root 60 Aug 13 08:13 / drwxr-xr-x 3 root root 4096 Aug 11 12:19 /home Here are the permissions of the pieces outside of the chroot: bash-4.2# ls -ld /a /rootfs.rw /root_ro_nfs drwxr-xr-x 1 root root 80 Aug 13 13:13 /a dr-xr-xr-x 17 root root 4096 Aug 12 13:08 /root_ro_nfs drwxrwxrwt 4 root root 80 Aug 13 13:11 /rootfs.rw
Maybe there is something else blocking the user. I am not much familiar with aarch64 or overlayfs. Could you check if there are any errors (maybe from selinux or similar) in system logs.
/red face We disable selinux (confirmed selinux=0 in /proc/cmdline just now) At this point, the test case is very simplified - I stop in our miniroot (fat initrd) right after the target root is constructed at /a. So I don't have system logging going. I can switch back to a normal boot where I started. You'll find tons of services unhappy about polkit being unhappy. logins won't work, sshd won't start, etc. That's where I learned the issue is when things were running as non-root and simplified the test case to going to miniroot (fat initrd) shell right before the switchroot to poke at it with a stick. 'dmesg' doesn't show anything interesting.
I found some old thread with similar issue reported [1]. Could you check permissions of 'ls -la /'
Thanks, Soumya
This thread would explain the behavior, it could be a clue, but I can't find the wrong permissions. Still, you mentioned a permissions difference in the logs. If I understnad right, that's because the images are stored that way on the nfs servers/gluster servers. I notice that all images for x86_64 and aarch64 as stored on the gluster servers have that permission though: [root@leader1 images]# ls -ld /opt/clmgr/image/images/{rhel7.6,rhel76-aarch64,su-rhel7.6} dr-xr-xr-x 17 root root 4096 Apr 17 10:15 /opt/clmgr/image/images/rhel7.6 dr-xr-xr-x 17 root root 4096 May 24 17:27 /opt/clmgr/image/images/rhel76-aarch64 dr-xr-xr-x 18 root root 4096 Apr 17 10:52 /opt/clmgr/image/images/su-rhel7.6 As an experiment, I added write (dir) permisison to the /opt/clmgr/image/images/rhel76-aarch64 and re-tested. Even though it still had search permission, etc, I just thought I'd try it. No change. I feel like there is one ity bity tiny thing preventing this from working.... I will start seeing if I can get more out of the overlay mount. It seems that's what I need to make progress.