Re: nfs-ganesha-2.8 service restart times
Over the past several days I've migrated our server from 2.8 to "next" branch. I'm now running 4-dev.14 and most of my problems have disappeared. I'm not likely going back to 2.8 or 3.2. The only change required in my ganesha.conf was renaming PROXY to PROXY_V4. In my nfs-ganesha systemd unit, I have an override of TimeoutStopUSec=5s so if the DBUS message doesn't cause ganesha to stop within 5 seconds then systemd will kill the process. So far, even with 4-14.dev, it appears that the timer is always expiring so I may be able to still debug this issue. However, if you're using SIGTERM then I'm inclined to just change my systemd unit to use SIGTERM and not bother with DBUS issue - although it does leave me wondering why the DBUS bit is in the CentOS packages. Thanks, Todd On Mon, 27 Apr 2020, Daniel Gryniewicz wrote:
That's a good question. I don't use DBUS to shutdown, I use SIGTERM, which takes a few seconds depending on what debugging I have enabled. However, the dbus command just calls the same function (admin_halt()) that the signal handler calls, so it should behave the same.
You can turn on debug logging, and look at the logs from the time that the "NFS EXIT: stopping NFS service" message appears until it's killed. All the log messages have timestamps, so that should help.
Daniel
On 4/24/20 7:46 PM, Todd Pfaff wrote:
Aha, I just learned about this:
systemctl show node.service -p TimeoutStopUSec TimeoutStopUSec=1min 30s
That explains where the 90 seconds is coming from.
The next question is: why isn't the dbus-send causing my nfs-ganesha process to shutdown quickly, long before the systemd TimeoutStopUSec expires?
Thanks, Todd
On Fri, 24 Apr 2020, Todd Pfaff wrote:
I'm looking at this again since my situation hasn't changed - a simple nfs-ganesha shutdown with the command:
systemctl stop nfs-ganesha
takes 90 seconds to return. I'm not sure how best to debug this but looking at the systemctl unit I see this:
ExecStop=/bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown
If I run that dbus-send manually, it returns immediately. If I then wait 90 seconds, nothing changes, the nfs-ganesha process is still running. A kill command with default TERM signal does nothing to the process. A kill -9 is required.
If I instead do 'time systemctl stop nfs-ganesha', after 90 seconds the nfs-ganesha process ends and the time command returns.
It appears to me as if the dbus command is having no impact, and systemd waits 90 seconds and then stops the nfs-ganesha process with KILL signal.
Anyone have any ideas about why I'm seeing this behaviour?
Any ideas about how to debug this further?
Thanks, Todd
On Thu, 16 Apr 2020, Daniel Gryniewicz wrote:
90 seconds is definitely wrong. Shutdown for me takes ~2-5 seconds, tops. This should be fixed.
Daniel
On 4/15/20 12:54 PM, Todd Pfaff wrote:
My environment:
# nfs-ganesha-2.8.3-4.el7.x86_64 # CentOS Linux release 7.7.1908 (Core)
In light of the stability problems that I've reported here recently, and given that I really do want to continue to use nfs-ganesha, I'm considering mitigating problems by either periodic or on-demand restarts of the nfs-ganesha service.
A complete nfs-ganesha restart, until an nfs client is successfully accessing the server again, consistently takes:
~ 90 seconds for the nfs-ganesha stop ~ 2 seconds for the nfs-ganesha start ~ 60 seconds for an nfs client to resume
for a total of about 2.5 minutes.
Do these times indicate a problem or is this normal behaviour for nfs-ganesha?
If these times are normal and not the result of some problem, is there any way to safely decrease this restart time?
Thanks, Todd _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Like I said, it shouldn't make a difference if it's DBUS or SIGTERM, as it's the same codepaths, unless it's hanging somewhere in libdbus code. Daniel On 4/27/20 9:12 AM, Todd Pfaff wrote:
Over the past several days I've migrated our server from 2.8 to "next" branch. I'm now running 4-dev.14 and most of my problems have disappeared. I'm not likely going back to 2.8 or 3.2. The only change required in my ganesha.conf was renaming PROXY to PROXY_V4.
In my nfs-ganesha systemd unit, I have an override of TimeoutStopUSec=5s so if the DBUS message doesn't cause ganesha to stop within 5 seconds then systemd will kill the process. So far, even with 4-14.dev, it appears that the timer is always expiring so I may be able to still debug this issue. However, if you're using SIGTERM then I'm inclined to just change my systemd unit to use SIGTERM and not bother with DBUS issue - although it does leave me wondering why the DBUS bit is in the CentOS packages.
Thanks, Todd
On Mon, 27 Apr 2020, Daniel Gryniewicz wrote:
That's a good question. I don't use DBUS to shutdown, I use SIGTERM, which takes a few seconds depending on what debugging I have enabled. However, the dbus command just calls the same function (admin_halt()) that the signal handler calls, so it should behave the same.
You can turn on debug logging, and look at the logs from the time that the "NFS EXIT: stopping NFS service" message appears until it's killed. All the log messages have timestamps, so that should help.
Daniel
On 4/24/20 7:46 PM, Todd Pfaff wrote:
Aha, I just learned about this:
systemctl show node.service -p TimeoutStopUSec TimeoutStopUSec=1min 30s
That explains where the 90 seconds is coming from.
The next question is: why isn't the dbus-send causing my nfs-ganesha process to shutdown quickly, long before the systemd TimeoutStopUSec expires?
Thanks, Todd
On Fri, 24 Apr 2020, Todd Pfaff wrote:
I'm looking at this again since my situation hasn't changed - a simple nfs-ganesha shutdown with the command:
systemctl stop nfs-ganesha
takes 90 seconds to return. I'm not sure how best to debug this but looking at the systemctl unit I see this:
ExecStop=/bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown
If I run that dbus-send manually, it returns immediately. If I then wait 90 seconds, nothing changes, the nfs-ganesha process is still running. A kill command with default TERM signal does nothing to the process. A kill -9 is required.
If I instead do 'time systemctl stop nfs-ganesha', after 90 seconds the nfs-ganesha process ends and the time command returns.
It appears to me as if the dbus command is having no impact, and systemd waits 90 seconds and then stops the nfs-ganesha process with KILL signal.
Anyone have any ideas about why I'm seeing this behaviour?
Any ideas about how to debug this further?
Thanks, Todd
On Thu, 16 Apr 2020, Daniel Gryniewicz wrote:
90 seconds is definitely wrong. Shutdown for me takes ~2-5 seconds, tops. This should be fixed.
Daniel
On 4/15/20 12:54 PM, Todd Pfaff wrote:
My environment:
# nfs-ganesha-2.8.3-4.el7.x86_64 # CentOS Linux release 7.7.1908 (Core)
In light of the stability problems that I've reported here recently, and given that I really do want to continue to use nfs-ganesha, I'm considering mitigating problems by either periodic or on-demand restarts of the nfs-ganesha service.
A complete nfs-ganesha restart, until an nfs client is successfully accessing the server again, consistently takes:
~ 90 seconds for the nfs-ganesha stop ~ 2 seconds for the nfs-ganesha start ~ 60 seconds for an nfs client to resume
for a total of about 2.5 minutes.
Do these times indicate a problem or is this normal behaviour for nfs-ganesha?
If these times are normal and not the result of some problem, is there any way to safely decrease this restart time?
Thanks, Todd _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
participants (2)
-
Daniel Gryniewicz -
Todd Pfaff