April 2020 - Support - NFS Ganesha List Archives

Re: nfs-ganesha-2.8 service restart times
by Todd Pfaff 27 Apr '20

27 Apr '20

Over the past several days I've migrated our server from 2.8 to "next" branch. I'm now running 4-dev.14 and most of my problems have disappeared. I'm not likely going back to 2.8 or 3.2. The only change required in my ganesha.conf was renaming PROXY to PROXY_V4. In my nfs-ganesha systemd unit, I have an override of TimeoutStopUSec=5s so if the DBUS message doesn't cause ganesha to stop within 5 seconds then systemd will kill the process. So far, even with 4-14.dev, it appears that the timer is always expiring so I may be able to still debug this issue. However, if you're using SIGTERM then I'm inclined to just change my systemd unit to use SIGTERM and not bother with DBUS issue - although it does leave me wondering why the DBUS bit is in the CentOS packages. Thanks, Todd On Mon, 27 Apr 2020, Daniel Gryniewicz wrote: > That's a good question. I don't use DBUS to shutdown, I use SIGTERM, > which takes a few seconds depending on what debugging I have enabled. > However, the dbus command just calls the same function (admin_halt()) > that the signal handler calls, so it should behave the same. > > You can turn on debug logging, and look at the logs from the time that > the "NFS EXIT: stopping NFS service" message appears until it's killed. > All the log messages have timestamps, so that should help. > > Daniel > > On 4/24/20 7:46 PM, Todd Pfaff wrote: >> Aha, I just learned about this: >> >> systemctl show node.service -p TimeoutStopUSec >> TimeoutStopUSec=1min 30s >> >> That explains where the 90 seconds is coming from. >> >> The next question is: why isn't the dbus-send causing my nfs-ganesha >> process to shutdown quickly, long before the systemd TimeoutStopUSec >> expires? >> >> Thanks, >> Todd >> >> >> On Fri, 24 Apr 2020, Todd Pfaff wrote: >> >>> I'm looking at this again since my situation hasn't changed - a simple >>> nfs-ganesha shutdown with the command: >>> >>> systemctl stop nfs-ganesha >>> >>> takes 90 seconds to return. I'm not sure how best to debug this but >>> looking at the systemctl unit I see this: >>> >>> ExecStop=/bin/dbus-send --system --dest=org.ganesha.nfsd >>> --type=method_call /org/ganesha/nfsd/admin >>> org.ganesha.nfsd.admin.shutdown >>> >>> If I run that dbus-send manually, it returns immediately. If I then >>> wait 90 seconds, nothing changes, the nfs-ganesha process is still >>> running. A kill command with default TERM signal does nothing to the >>> process. A kill >>> -9 is required. >>> >>> If I instead do 'time systemctl stop nfs-ganesha', after 90 seconds >>> the nfs-ganesha process ends and the time command returns. >>> >>> It appears to me as if the dbus command is having no impact, and >>> systemd waits 90 seconds and then stops the nfs-ganesha process with >>> KILL signal. >>> >>> Anyone have any ideas about why I'm seeing this behaviour? >>> >>> Any ideas about how to debug this further? >>> >>> Thanks, >>> Todd >>> >>> >>> On Thu, 16 Apr 2020, Daniel Gryniewicz wrote: >>> >>>> 90 seconds is definitely wrong. Shutdown for me takes ~2-5 seconds, >>>> tops. This should be fixed. >>>> >>>> Daniel >>>> >>>> On 4/15/20 12:54 PM, Todd Pfaff wrote: >>>>> My environment: >>>>> >>>>> # nfs-ganesha-2.8.3-4.el7.x86_64 >>>>> # CentOS Linux release 7.7.1908 (Core) >>>>> >>>>> >>>>> In light of the stability problems that I've reported here >>>>> recently, and >>>>> given that I really do want to continue to use nfs-ganesha, I'm >>>>> considering mitigating problems by either periodic or on-demand >>>>> restarts >>>>> of the nfs-ganesha service. >>>>> >>>>> A complete nfs-ganesha restart, until an nfs client is successfully >>>>> accessing the server again, consistently takes: >>>>> >>>>> ~ 90 seconds for the nfs-ganesha stop >>>>> ~ 2 seconds for the nfs-ganesha start >>>>> ~ 60 seconds for an nfs client to resume >>>>> >>>>> for a total of about 2.5 minutes. >>>>> >>>>> Do these times indicate a problem or is this normal behaviour for >>>>> nfs-ganesha? >>>>> >>>>> If these times are normal and not the result of some problem, is there >>>>> any way to safely decrease this restart time? >>>>> >>>>> Thanks, >>>>> Todd >>>>> _______________________________________________ >>>>> Support mailing list -- support(a)lists.nfs-ganesha.org >>>>> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>> _______________________________________________ >>>> Support mailing list -- support(a)lists.nfs-ganesha.org >>>> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>> >>> >>> >>> >> > >

2 1

Re: nfs-ganesha-2.8 service restart times
by Todd Pfaff 27 Apr '20

27 Apr '20

I'm looking at this again since my situation hasn't changed - a simple nfs-ganesha shutdown with the command: systemctl stop nfs-ganesha takes 90 seconds to return. I'm not sure how best to debug this but looking at the systemctl unit I see this: ExecStop=/bin/dbus-send --system --dest=org.ganesha.nfsd --type=method_call /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.shutdown If I run that dbus-send manually, it returns immediately. If I then wait 90 seconds, nothing changes, the nfs-ganesha process is still running. A kill command with default TERM signal does nothing to the process. A kill -9 is required. If I instead do 'time systemctl stop nfs-ganesha', after 90 seconds the nfs-ganesha process ends and the time command returns. It appears to me as if the dbus command is having no impact, and systemd waits 90 seconds and then stops the nfs-ganesha process with KILL signal. Anyone have any ideas about why I'm seeing this behaviour? Any ideas about how to debug this further? Thanks, Todd On Thu, 16 Apr 2020, Daniel Gryniewicz wrote: > 90 seconds is definitely wrong. Shutdown for me takes ~2-5 seconds, > tops. This should be fixed. > > Daniel > > On 4/15/20 12:54 PM, Todd Pfaff wrote: >> My environment: >> >> # nfs-ganesha-2.8.3-4.el7.x86_64 >> # CentOS Linux release 7.7.1908 (Core) >> >> >> In light of the stability problems that I've reported here recently, and >> given that I really do want to continue to use nfs-ganesha, I'm >> considering mitigating problems by either periodic or on-demand restarts >> of the nfs-ganesha service. >> >> A complete nfs-ganesha restart, until an nfs client is successfully >> accessing the server again, consistently takes: >> >> ~ 90 seconds for the nfs-ganesha stop >> ~ 2 seconds for the nfs-ganesha start >> ~ 60 seconds for an nfs client to resume >> >> for a total of about 2.5 minutes. >> >> Do these times indicate a problem or is this normal behaviour for >> nfs-ganesha? >> >> If these times are normal and not the result of some problem, is there >> any way to safely decrease this restart time? >> >> Thanks, >> Todd >> _______________________________________________ >> Support mailing list -- support(a)lists.nfs-ganesha.org >> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org > _______________________________________________ > Support mailing list -- support(a)lists.nfs-ganesha.org > To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >

2 2

Re: questions about stability of nfs-ganesha versions and PROXY FSAL
by Todd Pfaff 21 Apr '20

21 Apr '20

done : https://github.com/nfs-ganesha/nfs-ganesha/issues/571. On Tue, 21 Apr 2020, Daniel Gryniewicz wrote: > Open an issue on github with backtrace and so on. > > Daniel > > On 4/20/20 2:38 PM, Todd Pfaff wrote: >> Daniel, >> >> I have an abrt crash dump now. What would you like me to do with this? >> >> # abrt-cli list >> id 50e331b3cd0ec038df60b9346727abdfdc55f807 >> reason: ganesha.nfsd killed by SIGSEGV >> time: Fri 17 Apr 2020 11:50:22 PM EDT >> cmdline: /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f >> /etc/ganesha/ganesha.conf -N NIV_CRIT >> package: nfs-ganesha-3.2-6.el7 >> uid: 0 (root) >> count: 1 >> Directory: /var/spool/abrt/ccpp-2020-04-17-23:50:22-1648 >> >> The Autoreporting feature is disabled. Please consider enabling it by >> issuing 'abrt-auto-reporting enabled' as a user with root privileges >> >> >> Thanks, >> Todd >> >> >> On Fri, 17 Apr 2020, Daniel Gryniewicz wrote: >> >>> An upgrade should never trigger a segfault. Do you have backtraces >>> saved somewhere? >>> >>> Daniel >>> >>> On 4/15/20 12:34 PM, Todd Pfaff wrote: >>>> Thanks for the feedback, Daniel. I quickly tried upgrading to from 2.8 >>>> to 3.2 on our CentOS 7 nfs-ganesha server. The 'yum upgrade' process >>>> appeared to go just fine - it was simply a matter of: >>>> >>>> systemctl stop nfs-ganesha >>>> yum install centos-release-nfs-ganesha30 >>>> yum update >>>> systemctl start nfs-ganesha >>>> >>>> to go from nfs-ganesha-2.8.3-4.el7.x86_64 to >>>> nfs-ganesha-3.2-6.el7.x86_64. >>>> Unfortunately that was the extent of my joy. Upon startup, >>>> ganesha.nfsd >>>> crashed with these log entries: >>>> >>>> Apr 13 22:48:17 ganesha systemd: Started NFS-Ganesha file server. >>>> Apr 13 22:48:17 ganesha rpc.statd[30746]: Received SM_UNMON_ALL >>>> request >>>> from ganesha while not monitoring any hosts >>>> Apr 13 22:48:52 ganesha kernel: ganesha.nfsd[30820]: segfault at >>>> 7f561b35d26f ip 00007f561b35d26f sp 00007f561b35bc18 error 15 >>>> Apr 13 22:48:52 ganesha systemd: nfs-ganesha.service: main process >>>> exited, code=killed, status=11/SEGV >>>> >>>> I quickly downgraded since I wasn't in a position to try to debug this >>>> at that time. >>>> >>>> I admit that I did not dig for documentation that describes potential >>>> issues when upgrading from 2.8 to 3.2. Are there any that I should be >>>> aware of before trying this again? For example, are there any >>>> significant ganesha.conf changes required for this upgrade? >>>> >>>> Thanks, >>>> Todd >>>> >>>> >>>> On Thu, 9 Apr 2020, Daniel Gryniewicz wrote: >>>> >>>>> 3.2 is newer, and therefore has more bug fixes, than 2.8.3. It >>>>> also has >>>>> much more development, so it has more potential bugs. I would >>>>> consider >>>>> the newest stable version (3.2 in this case) to be the most stable >>>>> version. >>>>> >>>>> That said, we are about to release new versions of both 2.8.x and >>>>> 3.x in >>>>> the next week or two, so you may want to wait for that. >>>>> >>>>> Daniel >>>>> >>>>> On 4/8/20 10:40 PM, Todd Pfaff wrote: >>>>>> I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on >>>>>> CentOS 7. >>>>>> >>>>>> I haven't yet tried nfs-ganesha 3.0 but I could probably quickly and >>>>>> easily move to that if there is an advantage in doing so. >>>>>> >>>>>> Before I spend time switching from 2.8 to 3.0, can anyone tell me >>>>>> whether either of these versions of nfs-ganesha is considered to >>>>>> be more >>>>>> stable than the other? >>>>>> >>>>>> centos-release-nfs-ganesha28.noarch : NFS-Ganesha 2.8 packages >>>>>> from the >>>>>> CentOS >>>>>> : Storage SIG repository >>>>>> centos-release-nfs-ganesha30.noarch : NFS-Ganesha 3.0 packages >>>>>> from the >>>>>> CentOS >>>>>> : Storage SIG repository >>>>>> >>>>>> What about PROXY FSAL in either of these versions - is one or the >>>>>> other >>>>>> known to be more stable? >>>>>> >>>>>> Is there a more robust version of nfs-ganesha and PROXY FSAL that I >>>>>> should consider trying instead? >>>>>> >>>>>> Thanks, >>>>>> Todd >>>>>> _______________________________________________ >>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org >>>>>> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>>> >>>>> >>>> >>> >>> >> _______________________________________________ >> Support mailing list -- support(a)lists.nfs-ganesha.org >> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org > _______________________________________________ > Support mailing list -- support(a)lists.nfs-ganesha.org > To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >

1 0

Re: questions about stability of nfs-ganesha versions and PROXY FSAL
by Todd Pfaff 21 Apr '20

21 Apr '20

Daniel, I have an abrt crash dump now. What would you like me to do with this? # abrt-cli list id 50e331b3cd0ec038df60b9346727abdfdc55f807 reason: ganesha.nfsd killed by SIGSEGV time: Fri 17 Apr 2020 11:50:22 PM EDT cmdline: /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_CRIT package: nfs-ganesha-3.2-6.el7 uid: 0 (root) count: 1 Directory: /var/spool/abrt/ccpp-2020-04-17-23:50:22-1648 The Autoreporting feature is disabled. Please consider enabling it by issuing 'abrt-auto-reporting enabled' as a user with root privileges Thanks, Todd On Fri, 17 Apr 2020, Daniel Gryniewicz wrote: > An upgrade should never trigger a segfault. Do you have backtraces > saved somewhere? > > Daniel > > On 4/15/20 12:34 PM, Todd Pfaff wrote: >> Thanks for the feedback, Daniel. I quickly tried upgrading to from 2.8 >> to 3.2 on our CentOS 7 nfs-ganesha server. The 'yum upgrade' process >> appeared to go just fine - it was simply a matter of: >> >> systemctl stop nfs-ganesha >> yum install centos-release-nfs-ganesha30 >> yum update >> systemctl start nfs-ganesha >> >> to go from nfs-ganesha-2.8.3-4.el7.x86_64 to nfs-ganesha-3.2-6.el7.x86_64. >> Unfortunately that was the extent of my joy. Upon startup, ganesha.nfsd >> crashed with these log entries: >> >> Apr 13 22:48:17 ganesha systemd: Started NFS-Ganesha file server. >> Apr 13 22:48:17 ganesha rpc.statd[30746]: Received SM_UNMON_ALL request >> from ganesha while not monitoring any hosts >> Apr 13 22:48:52 ganesha kernel: ganesha.nfsd[30820]: segfault at >> 7f561b35d26f ip 00007f561b35d26f sp 00007f561b35bc18 error 15 >> Apr 13 22:48:52 ganesha systemd: nfs-ganesha.service: main process >> exited, code=killed, status=11/SEGV >> >> I quickly downgraded since I wasn't in a position to try to debug this >> at that time. >> >> I admit that I did not dig for documentation that describes potential >> issues when upgrading from 2.8 to 3.2. Are there any that I should be >> aware of before trying this again? For example, are there any >> significant ganesha.conf changes required for this upgrade? >> >> Thanks, >> Todd >> >> >> On Thu, 9 Apr 2020, Daniel Gryniewicz wrote: >> >>> 3.2 is newer, and therefore has more bug fixes, than 2.8.3. It also has >>> much more development, so it has more potential bugs. I would consider >>> the newest stable version (3.2 in this case) to be the most stable >>> version. >>> >>> That said, we are about to release new versions of both 2.8.x and 3.x in >>> the next week or two, so you may want to wait for that. >>> >>> Daniel >>> >>> On 4/8/20 10:40 PM, Todd Pfaff wrote: >>>> I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on >>>> CentOS 7. >>>> >>>> I haven't yet tried nfs-ganesha 3.0 but I could probably quickly and >>>> easily move to that if there is an advantage in doing so. >>>> >>>> Before I spend time switching from 2.8 to 3.0, can anyone tell me >>>> whether either of these versions of nfs-ganesha is considered to be more >>>> stable than the other? >>>> >>>> centos-release-nfs-ganesha28.noarch : NFS-Ganesha 2.8 packages >>>> from the >>>> CentOS >>>> : Storage SIG repository >>>> centos-release-nfs-ganesha30.noarch : NFS-Ganesha 3.0 packages >>>> from the >>>> CentOS >>>> : Storage SIG repository >>>> >>>> What about PROXY FSAL in either of these versions - is one or the other >>>> known to be more stable? >>>> >>>> Is there a more robust version of nfs-ganesha and PROXY FSAL that I >>>> should consider trying instead? >>>> >>>> Thanks, >>>> Todd >>>> _______________________________________________ >>>> Support mailing list -- support(a)lists.nfs-ganesha.org >>>> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>> >>> >> > >

2 1

Troubleshooting nfs ganesha
by randysr＠kamradtfamily.net 20 Apr '20

20 Apr '20

I'm doing an article on creating a box for storage using GlusterFS. I wanted to include NFS, so I installed NFS-Ganesha. But I'm struggling to get it to work correctly. It appears to mount, but the mount point doesn't have any files, and I can't create any files. Most suspicious is that if I cat /proc/mounts, I see an entry: artful:/ /home/rkamradt/gluster nfs4 rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.225,local_lock=none,addr=192.168.0.104 0 0 but if I run showmount artful -e I get an empty list. What approach should I take to troubleshoot this issue? I haven't dealt with NFS in a while, and most of the information I can glean from the internet is focused on kernal nfs. Thanks, Randy

2 2

Re: questions about stability of nfs-ganesha versions and PROXY FSAL
by Todd Pfaff 17 Apr '20

17 Apr '20

Daniel, I don't have this available from my previous attempt but I'm going through this now: https://wiki.centos.org/TipsAndTricks/ABRT to get this particular server set up for crash handling and reporting, and then I'll test again. If there's anything else that I should do to improve what I can provide to you folks, please let me know, or point me at what I should read, and I'll do what I can. Thanks, Todd On Fri, 17 Apr 2020, Daniel Gryniewicz wrote: > An upgrade should never trigger a segfault. Do you have backtraces > saved somewhere? > > Daniel > > On 4/15/20 12:34 PM, Todd Pfaff wrote: >> Thanks for the feedback, Daniel. I quickly tried upgrading to from 2.8 >> to 3.2 on our CentOS 7 nfs-ganesha server. The 'yum upgrade' process >> appeared to go just fine - it was simply a matter of: >> >> systemctl stop nfs-ganesha >> yum install centos-release-nfs-ganesha30 >> yum update >> systemctl start nfs-ganesha >> >> to go from nfs-ganesha-2.8.3-4.el7.x86_64 to nfs-ganesha-3.2-6.el7.x86_64. >> Unfortunately that was the extent of my joy. Upon startup, ganesha.nfsd >> crashed with these log entries: >> >> Apr 13 22:48:17 ganesha systemd: Started NFS-Ganesha file server. >> Apr 13 22:48:17 ganesha rpc.statd[30746]: Received SM_UNMON_ALL request >> from ganesha while not monitoring any hosts >> Apr 13 22:48:52 ganesha kernel: ganesha.nfsd[30820]: segfault at >> 7f561b35d26f ip 00007f561b35d26f sp 00007f561b35bc18 error 15 >> Apr 13 22:48:52 ganesha systemd: nfs-ganesha.service: main process >> exited, code=killed, status=11/SEGV >> >> I quickly downgraded since I wasn't in a position to try to debug this >> at that time. >> >> I admit that I did not dig for documentation that describes potential >> issues when upgrading from 2.8 to 3.2. Are there any that I should be >> aware of before trying this again? For example, are there any >> significant ganesha.conf changes required for this upgrade? >> >> Thanks, >> Todd >> >> >> On Thu, 9 Apr 2020, Daniel Gryniewicz wrote: >> >>> 3.2 is newer, and therefore has more bug fixes, than 2.8.3. It also has >>> much more development, so it has more potential bugs. I would consider >>> the newest stable version (3.2 in this case) to be the most stable >>> version. >>> >>> That said, we are about to release new versions of both 2.8.x and 3.x in >>> the next week or two, so you may want to wait for that. >>> >>> Daniel >>> >>> On 4/8/20 10:40 PM, Todd Pfaff wrote: >>>> I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on >>>> CentOS 7. >>>> >>>> I haven't yet tried nfs-ganesha 3.0 but I could probably quickly and >>>> easily move to that if there is an advantage in doing so. >>>> >>>> Before I spend time switching from 2.8 to 3.0, can anyone tell me >>>> whether either of these versions of nfs-ganesha is considered to be more >>>> stable than the other? >>>> >>>> centos-release-nfs-ganesha28.noarch : NFS-Ganesha 2.8 packages >>>> from the >>>> CentOS >>>> : Storage SIG repository >>>> centos-release-nfs-ganesha30.noarch : NFS-Ganesha 3.0 packages >>>> from the >>>> CentOS >>>> : Storage SIG repository >>>> >>>> What about PROXY FSAL in either of these versions - is one or the other >>>> known to be more stable? >>>> >>>> Is there a more robust version of nfs-ganesha and PROXY FSAL that I >>>> should consider trying instead? >>>> >>>> Thanks, >>>> Todd >>>> _______________________________________________ >>>> Support mailing list -- support(a)lists.nfs-ganesha.org >>>> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>> >>> >> > >

1 0

Re: questions about stability of nfs-ganesha versions and PROXY FSAL
by Todd Pfaff 17 Apr '20

17 Apr '20

Thanks for the feedback, Daniel. I quickly tried upgrading to from 2.8 to 3.2 on our CentOS 7 nfs-ganesha server. The 'yum upgrade' process appeared to go just fine - it was simply a matter of: systemctl stop nfs-ganesha yum install centos-release-nfs-ganesha30 yum update systemctl start nfs-ganesha to go from nfs-ganesha-2.8.3-4.el7.x86_64 to nfs-ganesha-3.2-6.el7.x86_64. Unfortunately that was the extent of my joy. Upon startup, ganesha.nfsd crashed with these log entries: Apr 13 22:48:17 ganesha systemd: Started NFS-Ganesha file server. Apr 13 22:48:17 ganesha rpc.statd[30746]: Received SM_UNMON_ALL request from ganesha while not monitoring any hosts Apr 13 22:48:52 ganesha kernel: ganesha.nfsd[30820]: segfault at 7f561b35d26f ip 00007f561b35d26f sp 00007f561b35bc18 error 15 Apr 13 22:48:52 ganesha systemd: nfs-ganesha.service: main process exited, code=killed, status=11/SEGV I quickly downgraded since I wasn't in a position to try to debug this at that time. I admit that I did not dig for documentation that describes potential issues when upgrading from 2.8 to 3.2. Are there any that I should be aware of before trying this again? For example, are there any significant ganesha.conf changes required for this upgrade? Thanks, Todd On Thu, 9 Apr 2020, Daniel Gryniewicz wrote: > 3.2 is newer, and therefore has more bug fixes, than 2.8.3. It also has > much more development, so it has more potential bugs. I would consider > the newest stable version (3.2 in this case) to be the most stable version. > > That said, we are about to release new versions of both 2.8.x and 3.x in > the next week or two, so you may want to wait for that. > > Daniel > > On 4/8/20 10:40 PM, Todd Pfaff wrote: >> I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on >> CentOS 7. >> >> I haven't yet tried nfs-ganesha 3.0 but I could probably quickly and >> easily move to that if there is an advantage in doing so. >> >> Before I spend time switching from 2.8 to 3.0, can anyone tell me >> whether either of these versions of nfs-ganesha is considered to be more >> stable than the other? >> >> centos-release-nfs-ganesha28.noarch : NFS-Ganesha 2.8 packages from the >> CentOS >> : Storage SIG repository >> centos-release-nfs-ganesha30.noarch : NFS-Ganesha 3.0 packages from the >> CentOS >> : Storage SIG repository >> >> What about PROXY FSAL in either of these versions - is one or the other >> known to be more stable? >> >> Is there a more robust version of nfs-ganesha and PROXY FSAL that I >> should consider trying instead? >> >> Thanks, >> Todd >> _______________________________________________ >> Support mailing list -- support(a)lists.nfs-ganesha.org >> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org > >

2 1

nfs-ganesha-2.8 service restart times
by Todd Pfaff 16 Apr '20

16 Apr '20

My environment: # nfs-ganesha-2.8.3-4.el7.x86_64 # CentOS Linux release 7.7.1908 (Core) In light of the stability problems that I've reported here recently, and given that I really do want to continue to use nfs-ganesha, I'm considering mitigating problems by either periodic or on-demand restarts of the nfs-ganesha service. A complete nfs-ganesha restart, until an nfs client is successfully accessing the server again, consistently takes: ~ 90 seconds for the nfs-ganesha stop ~ 2 seconds for the nfs-ganesha start ~ 60 seconds for an nfs client to resume for a total of about 2.5 minutes. Do these times indicate a problem or is this normal behaviour for nfs-ganesha? If these times are normal and not the result of some problem, is there any way to safely decrease this restart time? Thanks, Todd

4 3

Re: inconsistent nfs-ganesha error at startup (v2.8, CentOS 7)
by Todd Pfaff 16 Apr '20

16 Apr '20

Any guesses as to what can lead to such an improper tear-down, or how to diagnose this further? Is there anything that I should look for in either nfs-ganesha server or nfs client logs that would give any indication of such an improper tear-down. I can tell you that, almost invariably, I am restarting nfs-ganesha via either: systemctl restart nfs-ganesha or systemctl stop nfs-ganesha systemctl start nfs-ganesha so I don't know what else I could do to improve the tear-down. Thanks, Todd On Thu, 9 Apr 2020, Daniel Gryniewicz wrote: > This error means that Ganesha got NFS4ERR_SEQ_MISORDERED from the remote > server. I'm not sure what would cause this in this case, but a guess is > that maybe Ganesha is being restarted, and the previous session was not > properly torn down? > > Daniel > > On 4/9/20 9:34 AM, Todd Pfaff wrote: >> I'm using nfs-ganesha 2.8 on CentOS 7. >> >> Inconsistently, I will get errors like this in ganesha.log at >> nfs-ganesha startup: >> >> 08/04/2020 22:25:26 : epoch 5e8e8796 : ganesha : ganesha.nfsd-1647[main] >> init_export_root :EXPORT :CRIT :Lookup failed on path, ExportId=20040 >> Path=/pool/export/a FSAL_ERROR=(Invalid object type,10063) >> >> This is using PROXY FSAL for a valid, functioning nfs export from >> another host X. I have other PROXY exports with different export IDs >> and different paths for this same host X that succeed during the same >> startup for which the error above is generated. I will see different >> exports IDs and paths in these errors for different startups, so it's >> not consistently always failing on one particular export ID. >> >> From one startup to another: >> >> - I may see this error about this same export, >> - I may see this error about some other export, >> - I may see no errors. >> >> Thoughts? >> >> Thanks, >> Todd >> _______________________________________________ >> Support mailing list -- support(a)lists.nfs-ganesha.org >> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org > >

2 1

Installing NFS-Ganesha Offline
by David Hansen 09 Apr '20

09 Apr '20

Howdy, Is there any documentation, resources or tools for installing NFS-Ganesha offline? The environment I hope to install this on does not have access to the outside world and trying to install it currently has only gotten me into dependency hell. I would appreciate any pointers or tips or anything really. Thanks!

2 2

inconsistent nfs-ganesha error at startup (v2.8, CentOS 7)
by Todd Pfaff 09 Apr '20

09 Apr '20

I'm using nfs-ganesha 2.8 on CentOS 7. Inconsistently, I will get errors like this in ganesha.log at nfs-ganesha startup: 08/04/2020 22:25:26 : epoch 5e8e8796 : ganesha : ganesha.nfsd-1647[main] init_export_root :EXPORT :CRIT :Lookup failed on path, ExportId=20040 Path=/pool/export/a FSAL_ERROR=(Invalid object type,10063) This is using PROXY FSAL for a valid, functioning nfs export from another host X. I have other PROXY exports with different export IDs and different paths for this same host X that succeed during the same startup for which the error above is generated. I will see different exports IDs and paths in these errors for different startups, so it's not consistently always failing on one particular export ID. From one startup to another: - I may see this error about this same export, - I may see this error about some other export, - I may see no errors. Thoughts? Thanks, Todd

2 1

questions about stability of nfs-ganesha versions and PROXY FSAL
by Todd Pfaff 09 Apr '20

09 Apr '20

I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on CentOS 7. I haven't yet tried nfs-ganesha 3.0 but I could probably quickly and easily move to that if there is an advantage in doing so. Before I spend time switching from 2.8 to 3.0, can anyone tell me whether either of these versions of nfs-ganesha is considered to be more stable than the other? centos-release-nfs-ganesha28.noarch : NFS-Ganesha 2.8 packages from the CentOS : Storage SIG repository centos-release-nfs-ganesha30.noarch : NFS-Ganesha 3.0 packages from the CentOS : Storage SIG repository What about PROXY FSAL in either of these versions - is one or the other known to be more stable? Is there a more robust version of nfs-ganesha and PROXY FSAL that I should consider trying instead? Thanks, Todd

2 1

ganesha nfs recovery on shared storage
by Grant Chesy 08 Apr '20

08 Apr '20

Hello, I want to setup an active-passive two node nfsv4 only cluster. Is it safe to have /var/lib/nfs/ganesha on shared storage with both instances of ganesha running but only one owning the sole VIP, so only one node is answering client requests / updating the recovery files? I.e., by safe, I mean the recovery files will not get corrupted / deleted by the other node? I see paths "node0" that suggest that a shared storage use case was considered. /var/lib/nfs/ganesha/v4old/node0 /var/lib/nfs/ganesha/v4recov/node0 But, I don't see anywhere in the man pages to indicate that an instance is e.g., node1. And, if there was, I would need a way to tell node1 to use node0's recovery files when node1 becomes master. If safe, my thought was to have a keepalived notify script run something like this when a node enters MASTER state. But, if there were separate node0 and node1 recovery file paths, I don't see how to tell nodeX to use nodeY's recovery files on failover initiated recovery: dbus-send --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/admin org.ganesha.nfsd.admin.grace string:5:${VIP} If the shared storage recovery bit is OK, is there anything I need to add to the nfs-ganesha config to make this work? And, does that dbus command look OK? I'm running nfs-ganesha version 2.7.1 packaged in Debian Buster. I have 2X working nfs-ganesha single node setups using gluster FSAL that I hope to turn into an nfs cluster. Regards

2 1

nfs-ganesha in a cluster with IP failover
by alxg＠weka.io 05 Apr '20

05 Apr '20

Hi, We'e setting up ganesha using FSAL_VFS, and NFSv3 only. We have a cluster of them running with floating IPs being failed over between them when a host dies. The question we have is regarding writes which are UNSTABLE - meaning the client writes out from its cache for example with UNSTABLE, and then sends a COMMIT. In the event of a ganesha host crashing after receiving the writeouts, but before the COMMIT - is it assured that when the floating IP will failover to another host the client will resend the unstable writes again as well? I guess this has several parts to it: * Client side - assuming it's a linux kernel nfs client and it uses the pagecache - does it mark the pages clean right after successfully sending their writes as unstable, or does it mark them as clean only after it successfully sends a COMMIT on them as well? Do you happen to know? * Server side - Does nfs-ganesha have some data cache? Or the writes being unstable just means they can land in the pagecache of the server side, and stay there until the COMMIT does a sync? in that case do we need to set `NFS_Commit = true`? What are the guidelines on such a setup to ensure data doesn't get lost and doesn't corrupt? Thank you, Alex

3 7

method to use client_mgr.c and server_stats.c in NFS Ganesha
by Suresh Kosuru 03 Apr '20

03 Apr '20

Hi, Can you point me help? how can I use client_mgr.c and server_stats.c in nfs ganesha and where will this info logging? Thanks Suresh

3 2

Unable to delete last ACE entry of an ACL on FSAL_VFS with ENABLE_VFS_DEBUG_ACL as ON
by subhash.arya＠seagate.com 01 Apr '20

01 Apr '20

Hello all, We are trying to use ACL functionality for FSAL_VFS and seeing the following issue while deleting the ACE entries. 1. Get/List file ACL -bash-4.2$ nfs4_getfacl abc # file: abc A::dev1:rxtTncy A::sub:rxtTncy 2. Delete specified ACE entry from the ACL and check -bash-4.2$ nfs4_setfacl -x A::dev1:RXT abc -bash-4.2$ nfs4_getfacl abc # file: abc A::sub:rxtTncy 3. Delete the last ACE bash-4.2$ nfs4_setfacl -x A::sub:RXT abc 4. Check if the last ACE is deleted and it's not. -bash-4.2$ nfs4_getfacl abc # file: abc A::sub:rxtTncy Inspite of deleting the last ACE, the nfs4_getfacl still shows the deleted ACE. Could anyone help and let us know if we are missing something here? Thanks, Subhash

3 2