Thank you for logging as well at the logs!
A couple replies below.
/home permissions returned are 0755
and '/' has 0555
That's interesting.
bash-4.2# chroot /a ls -ld / /home
drwxr-xr-x 1 root root 60 Aug 13 08:13 /
drwxr-xr-x 3 root root 4096 Aug 11 12:19 /home
Here are the permissions of the pieces outside of the chroot:
bash-4.2# ls -ld /a /rootfs.rw /root_ro_nfs
drwxr-xr-x 1 root root 80 Aug 13 13:13 /a
dr-xr-xr-x 17 root root 4096 Aug 12 13:08 /root_ro_nfs
drwxrwxrwt 4 root root 80 Aug 13 13:11 /rootfs.rw
Maybe there is something else blocking the user. I am not much
familiar with
aarch64 or overlayfs. Could you check if there are any errors (maybe from
selinux or similar) in system logs.
/red face We disable selinux (confirmed selinux=0 in /proc/cmdline just
now)
At this point, the test case is very simplified - I stop in our miniroot
(fat initrd) right after the target root is constructed at /a. So I
don't have system logging going. I can switch back to a normal boot
where I started. You'll find tons of services unhappy about polkit being
unhappy. logins won't work, sshd won't start, etc. That's where I
learned the issue is when things were running as non-root and simplified
the test case to going to miniroot (fat initrd) shell right before the
switchroot to poke at it with a stick. 'dmesg' doesn't show anything
interesting.
I found some old thread with similar issue reported [1]. Could you
check
permissions of 'ls -la /'
Thanks,
Soumya
[1]
https://bbs.archlinux.org/viewtopic.php?id=154628&p=2
This thread would explain the behavior, it could be a clue, but I can't
find the wrong permissions. Still, you mentioned a permissions
difference in the logs. If I understnad right, that's because the images
are stored that way on the nfs servers/gluster servers.
I notice that all images for x86_64 and aarch64 as stored on the gluster
servers have that permission though:
[root@leader1 images]# ls -ld /opt/clmgr/image/images/{rhel7.6,rhel76-aarch64,su-rhel7.6}
dr-xr-xr-x 17 root root 4096 Apr 17 10:15 /opt/clmgr/image/images/rhel7.6
dr-xr-xr-x 17 root root 4096 May 24 17:27 /opt/clmgr/image/images/rhel76-aarch64
dr-xr-xr-x 18 root root 4096 Apr 17 10:52 /opt/clmgr/image/images/su-rhel7.6
As an experiment, I added write (dir) permisison to the
/opt/clmgr/image/images/rhel76-aarch64 and re-tested. Even though it
still had search permission, etc, I just thought I'd try it. No change.
I feel like there is one ity bity tiny thing preventing this from
working....
I will start seeing if I can get more out of the overlay mount. It seems
that's what I need to make progress.