New subject: improperly booting aarch64 node (ganesha nfs client) x86 OK

Saturday, 10 August 2019

First, just a quick thanks for all you do. I'm new to Ganesha.

I am using Gluster and Gluster NFS as an NFS-boot solution. I've done a
lot of testing with 2,500 aarch64 nodes spread across 9 x86_64 NFS servers for
booting (we get abut 10 minute boot times with NFS roots on Gluster with
Gluster NFS). We're using CTDB for a level of "HA" and it works great.
We've lost nodes and haven't even noticed until we checked
alerting/logs.

We didn't start with Ganesha because I was on a deadline and it was
crashing on us under heavy boot load (daemons would disappear over
time). I didn't have time to do proper debugging and reporting and the
customer had to move on too at that time.  It wouldn't have been fair to
you guys to post at that time.

Since then, I've learned a lot more about Gluster and system tuning,
etc.

Our customer likes the boot times but their job launch times are taking
too long. Compared to kernel NFS (and full TMPFS no-network-FS) roots,
Gluster NFS is taking 2 more minutes to launch. They have a large
LD_LIBRARY_PATH and tons of libraries being loaded by all nodes at the
same time. Some gluster profiling is showing caching seems to be
working, we are guessing the issue is metadata load and related latency.

Because Gluster NFS is being depreciated anyway and because we want to
improve the job launch time, I have come back to trying Ganesha. I
noticed it has 'MDCACHE' and we wanted to try that out. Maybe we can
get our 2 minutes back.

The NFS solution is unique in that it combines a read-only NFS root with
TMPFS using overlay FS on each node. This gives the client nodes a true
"writable" feel even though they are based on read-only NFS.

Kernel NFS exports from standard ext4, and Gluster NFS exports are
working ok in this configuration for both architectures.

Ganesha is working OK for x86_64 nodes - normal bootup, no problem.

However, Ganesha for aarch64 client nodes is causing us trouble. A typical
RHEL7.6 bootup has many errors. Among services that never start is
"sshd" and "login". Sort of a bummer!  :)

[FAILED] Failed to start Login Service.
over and over.

At this time, I'm using our systems in the lab - a tiny version of the
big one - to debug the above. If I can solve the above, I will be able
to test on the ARM supercomputer a week from today. It will be my last
chance to run on the system personally.

My goal of dedicated time next week, if I can resolve the above, is to
improve job launch times. If there are stability problems, I want to
know the right method to report them to you guys before we revert to
Gluster NFS if needed. To get to that point, I need to solve the above
issue.

Gluster info:

gluster-4.1.6
  * My own build of it
Volume type: 3x3 (3 replicas for 3 total subvolumes, 9 total x86_64 servers)
Gluster server OS: RHEL 7.6
Server architecture: x86_64

Ganesha info:
Ganesha 2.8.2
  * I did a personal build of this for testing, with gluster 4.1.6
Ganesha clients running on the Gluster Servers themselves.

NFS Client info:
RHEL 7.6 aarch64 (fails)
RHEL 7.6 x86_64 (works)

NFS Client environment:
- UEFI pxe to retrieve grub2
  grub2 loads kernel and initrd with tftp
  initrd loads a special "miniroot" with a vendor script
  miniroot is an "expanded initrd" that has our toolchain
    * I can enable a rescue mode where I shell out just before
      SWITCHROOT if that helps

NFS / Overlay mount commands done by client:

mount -n -o ro,noatime,nocto,actimeo=3600,lookupcache=all,nolock,tcp,vers=3
172.23.255.249:/cm_shared/image/images_ro_nfs/rhel7.6
mount -t tmpfs -o mpol=interleave tmpfs /rootfs.rw
mount -t overlay overlay -o
lowerdir=/root_ro_nfs,upperdir=/rootfs.rw/upperdir,workdir=/rootfs.rw/work /a

(then later the magic to make the above work with SWITCHROOT).

- I don't think it's a squash issue - from the miniroot in rescue mode,
  I can poke around the RO NFS + TMPFS overlay and see the contents of
  root-only readable files.
- If I chroot from rescue mode in to the root, sshd stats by hand, so I think
  the issue is related to systemd.
- Not being able to log in after switchroot makes it harder but I'll
  start researching a way to get in, perhaps starting systemd by hand.

Any advice appreciated. Sorry this got so long.

PS: Ganesha configuration:

Disable_ACL = TRUE;  # To enable/disable ACL

NFS_CORE_PARAM {
	Clustered = TRUE;
}

EXPORT
{
   # Export Id (mandatory, each EXPORT must have a unique Export_Id)
   Export_Id = 10;

   # Exported path (mandatory)
   Path = "/cm_shared"; # assuming 'testvol' is the Gluster volume name

   # Pseudo Path (required for NFS v4)
   Pseudo = "/cm_shared";

   # Required for access (default is None)
   # Could use CLIENT blocks instead
   Access_Type = RW;

   # Allow root access
   Squash = none;
   #Squash = No_Root_Squash;

   # Security flavor supported
   SecType = "sys";

   # Exporting FSAL
   FSAL {
     Name = "GLUSTER";
     Hostname = "127.0.0.1"; # IP of one of the nodes in the trusted pool
     Volume = "cm_shared";
   }
}

2025

2024

2023

2022

2021

2020

2019

2018

improperly booting aarch64 node (ganesha nfs client) x86 OK