noon
Back in April I wrote:
On Thu, 9 Apr 2020, Pfaff, Todd wrote:
I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL
on
CentOS 7.
and this has been ongoing for me since then. Today I finally stumbled
onto something that may have solved at least some of the problems that
I've been having with nfs-ganesha. I also posted the following as a
follow up to the nfs-ganesha github issue: frequent failures with PROXY
FSAL (#580).
...
I've discovered something that may be relevant to this issue.
- My nfs-ganesha server is serving several EXPORTs.
- Two of these EXPORTs refer to identically named PATHs on two different
back-end servers.
- I've now changed these so that the back-end server paths are different.
- I've tested this multiple times with multiple VMs and it appears that
this change may have eliminated all traces of the problems that I've
been describing here.
Does this make sense? Is it known that nfs-ganesha can not have multiple
exports with the same value for the Path?
4:23 p.m.
New subject: questions about stability of nfs-ganesha versions and PROXY FSAL
Back in April I wrote:
On Thu, 9 Apr 2020, Pfaff, Todd wrote:
> I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on
> CentOS 7.
and this has been ongoing for me since then. Today I finally stumbled onto
something that may have solved at least some of the problems that I've been
having with nfs-ganesha. I also posted the following as a follow up to the nfs-
ganesha github issue: frequent failures with PROXY FSAL (#580).
...
I've discovered something that may be relevant to this issue.
- My nfs-ganesha server is serving several EXPORTs.
- Two of these EXPORTs refer to identically named PATHs on two different
back-end servers.
- I've now changed these so that the back-end server paths are different.
- I've tested this multiple times with multiple VMs and it appears that
this change may have eliminated all traces of the problems that I've
been describing here.
Does this make sense? Is it known that nfs-ganesha can not have multiple
exports with the same value for the Path?
I'm not that familiar with FSAL_PROXY, and I'm not sure how it handles multiple
exports. That is actually an area that needs more testing and more resolution.
Do you use
NFS_CORE_PARAM { mount_path_pseudo = true; }
You may be running into issues if path is the same with NFS v3 clients.
If that isn't the issue, we do need to make the proxy support more robust because
certainly we should allow multiple back end servers exporting the same path.
Frank
5:09 p.m.
New subject: [NFS-Ganesha-Devel] Re: Re: questions about stability of nfs-ganesha versions and PROXY FSAL
On Fri, Aug 28, 2020, 16:24 Frank Filz <ffilzlnx(a)mindspring.com> wrote:
> Back in April I wrote:
>
> On Thu, 9 Apr 2020, Pfaff, Todd wrote:
>
> > I'm having stability problems with nfs-ganesha 2.8 and PROXY FSAL on
> > CentOS 7.
>
> and this has been ongoing for me since then. Today I finally stumbled
onto
> something that may have solved at least some of the problems that I've
been
> having with nfs-ganesha. I also posted the following as a follow up to
the nfs-
> ganesha github issue: frequent failures with PROXY FSAL (#580).
>
> ...
>
> I've discovered something that may be relevant to this issue.
>
> - My nfs-ganesha server is serving several EXPORTs.
>
> - Two of these EXPORTs refer to identically named PATHs on two different
> back-end servers.
Are the exports different though? Meaning both are exporting /mnt or
whatever, but one is doing so as /foo and the other as /bar?
Can you share the example here?
> - I've now changed these so that the back-end server paths are different.
>
> - I've tested this multiple times with multiple VMs and it appears that
> this change may have eliminated all traces of the problems that I've
> been describing here.
>
>
> Does this make sense? Is it known that nfs-ganesha can not have multiple
> exports with the same value for the Path?
I'm not that familiar with FSAL_PROXY, and I'm not sure how it handles
multiple exports. That is actually an area that needs more testing and more
resolution.
Do you use
NFS_CORE_PARAM { mount_path_pseudo = true; }
You may be running into issues if path is the same with NFS v3 clients.
If that isn't the issue, we do need to make the proxy support more robust
because certainly we should allow multiple back end servers exporting the
same path.
Do the different backend servers also have different FSINFO / parameters?
(I now forget how clearly or not the two proxy codepaths ensure that they
correctly have per-export state/config versus per FSAL config for things
like write size, attributes, and so on).
The main things that show up as being path dependent are LOOKUP by path
(e.g. for initial mounting) and then READDIR caching in the MDCACHE (IIRC).
Other things are FH dependent, which I'd be curious if knfsd generates
FH3/4 values as consistent with a path (so if both have /mnt/dir as a path
on two separate servers, would that produce the same FH3/FH4 handle in
response? All of the handle to key stuff assumes handles would be unique).
Frank
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
5:41 p.m.
New subject: [NFS-Ganesha-Devel] Re: Re: questions about stability of nfs-ganesha versions and PROXY FSAL
I will share my ganesha.conf in another message to clarify the "same path"
that I was referring to.
Sadly, I may have spoken too soon earlier today since something happened
later that leads me to believe the issue is not yet resolved. Of course,
I could be dealing with multiple issues related to different workloads
that result in similar failures.
I am certain that something significant did change though simply by my
changing the path as I described earlier today. Before that, and going
back months, I had been unable to ever successfully boot a linux vm that
had its root disk image on this nfs-ganesha proxy fsal storage. After
making this path change, not only was I able to boot that test vm, I was
able to boot several other linux guests without a hint of any i/o errors
on /dev/vda.
However, sometime later today I decided to try booting a Windows 10 guest
on this same proxy path. Well, that was the end of my joy. The Windows
guest hung during boot, and all of the linux guests that were using
this nfs-ganesha proxy path and had been working fine up until this point
had their root filesystems go read only. I had to restart the nfs-ganesha
service and reboot all of the linux guests to recover.
I'm next going to repeat some tests with this and other Windows guests but
using a different ganesha server with access to the same back-end storage
path, to avoid impacting the happily running linux guests.
Todd
6:04 p.m.
New subject: [NFS-Ganesha-Devel] Re: Re: questions about stability of nfs-ganesha versions and PROXY FSAL
Here are the relevant parts of my ganesha.conf related to the two EXPORTs with same Path.
This is the state before, when booting guests was consistently failing.
NFS_CORE_PARAM {
mount_path_pseudo = true;
}
EXPORT {
Export_Id = 10101;
Path = /pool/vm-images;
Pseudo = /export/ss1/vm-images;
#Tag = ss1;
SecType = sys;
Access_Type = none;
Squash = All_Squash;
FSAL {
Name = PROXY_V3;
Srv_Addr=10.168.168.1;
}
CLIENT {
Clients = @vh;
Access_Type = RW;
Squash = No_Root_Squash;
}
}
EXPORT {
Export_Id = 10102;
Path = /pool/vm-images;
Pseudo = /export/ss2/vm-images;
#Tag = ss2;
SecType = sys;
Access_Type = none;
Squash = All_Squash;
FSAL {
Name = PROXY_V3;
Srv_Addr=10.168.168.2;
}
CLIENT {
Clients = @vh;
Access_Type = RW;
Squash = No_Root_Squash;
}
}
This is the the state after I changed the second EXPORT Path to differ from the first.
NFS_CORE_PARAM {
mount_path_pseudo = true;
}
EXPORT {
Export_Id = 10101;
Path = /pool/vm-images;
Pseudo = /export/ss1/vm-images;
#Tag = ss1;
SecType = sys;
Access_Type = none;
Squash = All_Squash;
FSAL {
Name = PROXY_V3;
Srv_Addr=10.168.168.1;
}
CLIENT {
Clients = @vh;
Access_Type = RW;
Squash = No_Root_Squash;
}
}
EXPORT {
Export_Id = 10102;
Path = /pool/vm-images2;
Pseudo = /export/ss2/vm-images;
#Tag = ss2;
SecType = sys;
Access_Type = none;
Squash = All_Squash;
FSAL {
Name = PROXY_V3;
Srv_Addr=10.168.168.2;
}
CLIENT {
Clients = @vh;
Access_Type = RW;
Squash = No_Root_Squash;
}
}
1330
days inactive
1331
days old
4 comments
3 participants
participants (3)
-
Frank Filz -
Solomon Boulos -
Todd Pfaff