Over the past several days I've migrated our server from 2.8 to "next" branch. I'm now running 4-dev.14 and most of my problems have disappeared. I'm not likely going back to 2.8 or 3.2. The only change required in my ganesha.conf was renaming PROXY to PROXY_V4. In my nfs-ganesha systemd unit, I have an override of TimeoutStopUSec=5s so if the DBUS message doesn't cause ganesha to stop within 5 seconds then systemd will kill the process. So far, even with 4-14.dev, it appears that the timer is always expiring so I may be able to still debug this issue. However, if you're using SIGTERM then I'm inclined to just change my systemd unit to use SIGTERM and not bother with DBUS issue - although it does leave me wondering why the DBUS bit is in the CentOS packages. Thanks, Todd On Mon, 27 Apr 2020, Daniel Gryniewicz wrote: > That's a good question. I don't use DBUS to shutdown, I use SIGTERM, > which takes a few seconds depending on what debugging I have enabled. > However, the dbus command just calls the same function (admin_halt()) > that the signal handler calls, so it should behave the same. > > You can turn on debug logging, and look at the logs from the time that > the "NFS EXIT: stopping NFS service" message appears until it's killed. > All the log messages have timestamps, so that should help. > > Daniel > > On 4/24/20 7:46 PM, Todd Pfaff wrote: >> Aha, I just learned about this: >> >>   systemctl show node.service -p TimeoutStopUSec >>   TimeoutStopUSec=1min 30s >> >> That explains where the 90 seconds is coming from. >> >> The next question is: why isn't the dbus-send causing my nfs-ganesha >> process to shutdown quickly, long before the systemd TimeoutStopUSec >> expires? >> >> Thanks, >> Todd >> >> >> On Fri, 24 Apr 2020, Todd Pfaff wrote: >> >>> I'm looking at this again since my situation hasn't changed - a simple >>> nfs-ganesha shutdown with the command: >>> >>>  systemctl stop nfs-ganesha >>> >>> takes 90 seconds to return.  I'm not sure how best to debug this but >>> looking at the systemctl unit I see this: >>> >>> ExecStop=/bin/dbus-send --system --dest=org.ganesha.nfsd >>> --type=method_call /org/ganesha/nfsd/admin >>> org.ganesha.nfsd.admin.shutdown >>> >>> If I run that dbus-send manually, it returns immediately.  If I then >>> wait 90 seconds, nothing changes, the nfs-ganesha process is still >>> running.  A kill command with default TERM signal does nothing to the >>> process.  A kill >>> -9 is required. >>> >>> If I instead do 'time systemctl stop nfs-ganesha', after 90 seconds >>> the nfs-ganesha process ends and the time command returns. >>> >>> It appears to me as if the dbus command is having no impact, and >>> systemd waits 90 seconds and then stops the nfs-ganesha process with >>> KILL signal. >>> >>> Anyone have any ideas about why I'm seeing this behaviour? >>> >>> Any ideas about how to debug this further? >>> >>> Thanks, >>> Todd >>> >>> >>> On Thu, 16 Apr 2020, Daniel Gryniewicz wrote: >>> >>>>  90 seconds is definitely wrong.  Shutdown for me takes ~2-5 seconds, >>>>  tops.  This should be fixed. >>>> >>>>  Daniel >>>> >>>>  On 4/15/20 12:54 PM, Todd Pfaff wrote: >>>>>  My environment: >>>>> >>>>> #  nfs-ganesha-2.8.3-4.el7.x86_64 >>>>> #  CentOS Linux release 7.7.1908 (Core) >>>>> >>>>> >>>>>  In light of the stability problems that I've reported here >>>>> recently, and >>>>>  given that I really do want to continue to use nfs-ganesha, I'm >>>>>  considering mitigating problems by either periodic or on-demand >>>>> restarts >>>>>  of the nfs-ganesha service. >>>>> >>>>>  A complete nfs-ganesha restart, until an nfs client is successfully >>>>>  accessing the server again, consistently takes: >>>>> >>>>>     ~ 90 seconds for the nfs-ganesha stop >>>>>     ~  2 seconds for the nfs-ganesha start >>>>>     ~ 60 seconds for an nfs client to resume >>>>> >>>>>  for a total of about 2.5 minutes. >>>>> >>>>>  Do these times indicate a problem or is this normal behaviour for >>>>>  nfs-ganesha? >>>>> >>>>>  If these times are normal and not the result of some problem, is there >>>>>  any way to safely decrease this restart time? >>>>> >>>>>  Thanks, >>>>>  Todd >>>>>  _______________________________________________ >>>>>  Support mailing list -- support(a)lists.nfs-ganesha.org >>>>>  To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>>  _______________________________________________ >>>>  Support mailing list -- support(a)lists.nfs-ganesha.org >>>>  To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>> >>> >>> >>> >> > >