4:57 p.m.
- I don't think it's a squash issue - from the miniroot in
rescue mode,
I can poke around the RO NFS + TMPFS overlay and see the contents of
root-only readable files.
- If I chroot from rescue mode in to the root, sshd stats by hand, so I think
the issue is related to systemd.
- Not being able to log in after switchroot makes it harder but I'll
start researching a way to get in, perhaps starting systemd by hand.
I did a bit more debugging today before calling it quits.
from the miniroot environment, I went in to the rescue mode. The root is
mounted at /a. I was able to chroot in to /a and start an sshd on a
speical port so that I could ssh in (even though the real console can't
be used and sshd would normally not start due to this problem).
Then I exec'd init without switchroot.
The same problems happened but I was able to poke at things more.
When polkit starts, on the x86_64 machine, it does this inotify3 that
does not fail (from strace):
26247 inotify_add_watch(3, "/usr/share/polkit-1/actions",
IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR
<unfinished ...>
On the aarch64 node using the same methods to get its root that I
described earlier, there is a bunch of these retrying over and over:
[pid 13195] inotify_add_watch(3, "/usr/share/polkit-1/actions",
IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR)
= -1 EOPNOTSUPP (Operation not supported)
[pid 13195] ppoll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}], 2, {3, 996000000}, NULL,
0) = 0 (Timeout)
[pid 13195] inotify_add_watch(3, "/usr/share/polkit-1/actions",
IN_MODIFY|IN_ATTRIB|IN_CLOSE_WRITE|IN_MOVED_FROM|IN_MOVED_TO|IN_CREATE|IN_DELETE|IN_DELETE_SELF|IN_MOVE_SELF|IN_UNMOUNT|IN_ONLYDIR)
= -1 EOPNOTSUPP (Operation not supported)
[pid 13195] ppoll([{fd=3, events=POLLIN}, {fd=4, events=POLLIN}], 2, {3, 996000000}, NULL,
0) = 0 (Timeout)
...
For fun, I ran pyinotify on /usr/share/polkit-1/actions on the same list
and it worked. I know that isn't apples to apples but was a quick test.
I'm not sure if this is related or not. I'm sort of in over my head at
this point.
I will stop for today.
Tomorrow, I may make a test C program that tries to isolate this possible
inotify difference.
By the way, I did try rebuilding ganesha to disable readdirplus but it
did not help. So I put it back.
Best wishes and thank you for any advice,
Erik