12:59 p.m.
Good morning. Since the problem didn't happen for me any more in
the lab,
Avoiding Overlayfs... using NFS RO root with certain TMPFS directories
mounted on top for things like /etc, /var, /root, etc.
Closing the loop here.... With nfs4, I had times when mounts blasted from
all nodes caused every Ganesha to exit (all 9 servers).
(2592, so 288 requests around the same time per server).
I had other times where the nfs mount requests just hung on the clients
but later trying the mount from an unrelated node worked. I couldn't
figure that out.
When I switched to NFS v3, the mount hangs disappeared completely.
However, for this workload, it was far too slow. We also got some stale
NFS file handle errors even though nothing changed on the NFS server
side. I say that knowing I may be doing something wrong.
I tried adjusting up the threads but it didn't seem to help (NB_Worker).
I can't get you what you need at this moment because the customer
doesn't allow my copy stuff off the system. Now that there are a few
different issues in play, it is getting confusing. I do plan to request
some logs but I didn't see much in them -- especially the NIV_FULL_DEBUG
didn't seem to have much in it.
My dedicated time ended and I switched them back to the way it was
before.
What I will do for us is, when we get a big system in the factory
that I can access, I'll be able to have full control of it and get any
log or trace, try experiments, etc.
I really want to see Ganesha work with these workloads so we can move
off of Gluster NFS.
Thank you so much for the help so far, and I'll get back to you when I
have a bigger system or when I get at least some logs cleared to send.
Erik