Re: [NFS-Ganesha-Support] Re: questions about stability of nfs-ganesha versions and PROXY FSAL
Back in April I wrote:
On Thu, 9 Apr 2020, Pfaff, Todd wrote:
I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on CentOS 7.
and this has been ongoing for me since then. Today I finally stumbled onto something that may have solved at least some of the problems that I've been having with nfs-ganesha. I also posted the following as a follow up to the nfs- ganesha github issue: frequent failures with PROXY FSAL (#580).
...
I've discovered something that may be relevant to this issue.
- My nfs-ganesha server is serving several EXPORTs.
- Two of these EXPORTs refer to identically named PATHs on two different back-end servers.
- I've now changed these so that the back-end server paths are different.
- I've tested this multiple times with multiple VMs and it appears that this change may have eliminated all traces of the problems that I've been describing here.
Does this make sense? Is it known that nfs-ganesha can not have multiple exports with the same value for the Path?
I'm not that familiar with FSAL_PROXY, and I'm not sure how it handles multiple exports. That is actually an area that needs more testing and more resolution. Do you use NFS_CORE_PARAM { mount_path_pseudo = true; } You may be running into issues if path is the same with NFS v3 clients. If that isn't the issue, we do need to make the proxy support more robust because certainly we should allow multiple back end servers exporting the same path. Frank
On Fri, Aug 28, 2020, 16:24 Frank Filz <ffilzlnx@mindspring.com> wrote:
Back in April I wrote:
On Thu, 9 Apr 2020, Pfaff, Todd wrote:
I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on CentOS 7.
and this has been ongoing for me since then. Today I finally stumbled onto something that may have solved at least some of the problems that I've been having with nfs-ganesha. I also posted the following as a follow up to the nfs- ganesha github issue: frequent failures with PROXY FSAL (#580).
...
I've discovered something that may be relevant to this issue.
- My nfs-ganesha server is serving several EXPORTs.
- Two of these EXPORTs refer to identically named PATHs on two different back-end servers.
Are the exports different though? Meaning both are exporting /mnt or whatever, but one is doing so as /foo and the other as /bar? Can you share the example here?
- I've now changed these so that the back-end server paths are different.
- I've tested this multiple times with multiple VMs and it appears that this change may have eliminated all traces of the problems that I've been describing here.
Does this make sense? Is it known that nfs-ganesha can not have multiple exports with the same value for the Path?
I'm not that familiar with FSAL_PROXY, and I'm not sure how it handles multiple exports. That is actually an area that needs more testing and more resolution.
Do you use
NFS_CORE_PARAM { mount_path_pseudo = true; }
You may be running into issues if path is the same with NFS v3 clients.
If that isn't the issue, we do need to make the proxy support more robust because certainly we should allow multiple back end servers exporting the same path.
Do the different backend servers also have different FSINFO / parameters? (I now forget how clearly or not the two proxy codepaths ensure that they correctly have per-export state/config versus per FSAL config for things like write size, attributes, and so on). The main things that show up as being path dependent are LOOKUP by path (e.g. for initial mounting) and then READDIR caching in the MDCACHE (IIRC). Other things are FH dependent, which I'd be curious if knfsd generates FH3/4 values as consistent with a path (so if both have /mnt/dir as a path on two separate servers, would that produce the same FH3/FH4 handle in response? All of the handle to key stuff assumes handles would be unique).
Frank _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
I will share my ganesha.conf in another message to clarify the "same path" that I was referring to. Sadly, I may have spoken too soon earlier today since something happened later that leads me to believe the issue is not yet resolved. Of course, I could be dealing with multiple issues related to different workloads that result in similar failures. I am certain that something significant did change though simply by my changing the path as I described earlier today. Before that, and going back months, I had been unable to ever successfully boot a linux vm that had its root disk image on this nfs-ganesha proxy fsal storage. After making this path change, not only was I able to boot that test vm, I was able to boot several other linux guests without a hint of any i/o errors on /dev/vda. However, sometime later today I decided to try booting a Windows 10 guest on this same proxy path. Well, that was the end of my joy. The Windows guest hung during boot, and all of the linux guests that were using this nfs-ganesha proxy path and had been working fine up until this point had their root filesystems go read only. I had to restart the nfs-ganesha service and reboot all of the linux guests to recover. I'm next going to repeat some tests with this and other Windows guests but using a different ganesha server with access to the same back-end storage path, to avoid impacting the happily running linux guests. Todd
Here are the relevant parts of my ganesha.conf related to the two EXPORTs with same Path. This is the state before, when booting guests was consistently failing. NFS_CORE_PARAM { mount_path_pseudo = true; } EXPORT { Export_Id = 10101; Path = /pool/vm-images; Pseudo = /export/ss1/vm-images; #Tag = ss1; SecType = sys; Access_Type = none; Squash = All_Squash; FSAL { Name = PROXY_V3; Srv_Addr=10.168.168.1; } CLIENT { Clients = @vh; Access_Type = RW; Squash = No_Root_Squash; } } EXPORT { Export_Id = 10102; Path = /pool/vm-images; Pseudo = /export/ss2/vm-images; #Tag = ss2; SecType = sys; Access_Type = none; Squash = All_Squash; FSAL { Name = PROXY_V3; Srv_Addr=10.168.168.2; } CLIENT { Clients = @vh; Access_Type = RW; Squash = No_Root_Squash; } } This is the the state after I changed the second EXPORT Path to differ from the first. NFS_CORE_PARAM { mount_path_pseudo = true; } EXPORT { Export_Id = 10101; Path = /pool/vm-images; Pseudo = /export/ss1/vm-images; #Tag = ss1; SecType = sys; Access_Type = none; Squash = All_Squash; FSAL { Name = PROXY_V3; Srv_Addr=10.168.168.1; } CLIENT { Clients = @vh; Access_Type = RW; Squash = No_Root_Squash; } } EXPORT { Export_Id = 10102; Path = /pool/vm-images2; Pseudo = /export/ss2/vm-images; #Tag = ss2; SecType = sys; Access_Type = none; Squash = All_Squash; FSAL { Name = PROXY_V3; Srv_Addr=10.168.168.2; } CLIENT { Clients = @vh; Access_Type = RW; Squash = No_Root_Squash; } }
participants (3)
-
Frank Filz -
Solomon Boulos -
Todd Pfaff