4:37 p.m.
I'm using Ganesha NFS with the Proxy FSAL. It seems to work fine as long as
I'm going between NFS clients and servers of the same version (NFSv4.1
client -> NFSv4.1 server). However, I have a situation where I'm trying to
proxy a NFSv4.1 service (Azure File Share NFS) for clients that do not
support NFSv4. When I do this, the "mount" command on the NFSv3 clients
works fine, but as soon as I try to do any data operations, I get I/O
errors:
# mount -t nfs -o vers=3 10.11.12.13:/azure-nfs-file /mnt/azure
# ls -l /mnt/azure
ls: reading directory '/mnt/azure': Remote I/O error
total 0
I've posted my (sanitized) configuration below. My questions are:
1) Is it possible to use Ganesha to proxy this way - allowing clients to
access a NFS server that differs in version, or is this not supported?
2) Am I missing something within my Ganesha config to enable this?
I can also provide the debug logging, if that's helpful.
Thanks - Nick
==ganesha.conf==
NFS_CORE_PARAM {
mount_path_pseudo = true;
Protocols = 3,4;
MNT_Port = 20048;
}
LOG {
Default_Log_Level = INFO;
Components {
FSAL = FULL_DEBUG;
}
}
EXPORT_DEFAULTS {
Access_Type = RW;
}
## Azure NFS Share
EXPORT
{
Export_id = 902;
Path = "/stgaccount/azure-file";
Pseudo = "/azure-file";
Access_Type = RW;
Squash = no_root_squash;
Sectype = sys;
FSAL {
Name = proxy;
Srv_Addr = 10.1.2.3;
Enable_Handle_Mapping = TRUE;
HandleMap_DB_Dir = "/var/ganesha/handledb/902";
HandleMap_Tmp_Dir = "/run/ganesha/tmp/902";
HandleMap_DB_Count = 8;
}
}
4:49 p.m.
I haven’t done enough with Proxy… There is a handle mapping option that you may need to
use if the proxied handles are too big for NFSv3.
Frank
From: Nick Couchman [mailto:nick.e.couchman@gmail.com]
Sent: Friday, June 25, 2021 4:38 PM
To: support(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Support] Proxy FSAL - NFS4 -> 3
I'm using Ganesha NFS with the Proxy FSAL. It seems to work fine as long as I'm
going between NFS clients and servers of the same version (NFSv4.1 client -> NFSv4.1
server). However, I have a situation where I'm trying to proxy a NFSv4.1 service
(Azure File Share NFS) for clients that do not support NFSv4. When I do this, the
"mount" command on the NFSv3 clients works fine, but as soon as I try to do any
data operations, I get I/O errors:
# mount -t nfs -o vers=3 10.11.12.13:/azure-nfs-file /mnt/azure
# ls -l /mnt/azure
ls: reading directory '/mnt/azure': Remote I/O error
total 0
I've posted my (sanitized) configuration below. My questions are:
1) Is it possible to use Ganesha to proxy this way - allowing clients to access a NFS
server that differs in version, or is this not supported?
2) Am I missing something within my Ganesha config to enable this?
I can also provide the debug logging, if that's helpful.
Thanks - Nick
==ganesha.conf==
NFS_CORE_PARAM {
mount_path_pseudo = true;
Protocols = 3,4;
MNT_Port = 20048;
}
LOG {
Default_Log_Level = INFO;
Components {
FSAL = FULL_DEBUG;
}
}
EXPORT_DEFAULTS {
Access_Type = RW;
}
## Azure NFS Share
EXPORT
{
Export_id = 902;
Path = "/stgaccount/azure-file";
Pseudo = "/azure-file";
Access_Type = RW;
Squash = no_root_squash;
Sectype = sys;
FSAL {
Name = proxy;
Srv_Addr = 10.1.2.3;
Enable_Handle_Mapping = TRUE;
HandleMap_DB_Dir = "/var/ganesha/handledb/902";
HandleMap_Tmp_Dir = "/run/ganesha/tmp/902";
HandleMap_DB_Count = 8;
}
}
4:52 p.m.
It should be!
The proxy code just acts like either a V4 client (ProxyV4 in the current
tree) or V3 client (ProxyV3). Ganesha itself handles the clients and
whatever they want to speak (v3, v4, 9p even I think, etc.).
Your use of Name = proxy, implies you're on an older branch perhaps, and I
feel like I've had some bugfixes in the Proxy V4 code (in particular, I
know for my ProxyV3 I had to fixup the handling of path vs pseudo... I
don't recall if ProxyV4 had the same problem or not). I don't have any
experience with the handle mapping code (though I've certainly looked at
it). Is it strictly required for your test?
Either way, please provide some debug logs. You may want to add debug
logging to NFS_PROTO or whatever the name is as well. (And if I finally get
around to fixing my IRC setup, you can find folks in #ganesha, too for more
live debugging).
On Fri, Jun 25, 2021 at 4:38 PM Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
I'm using Ganesha NFS with the Proxy FSAL. It seems to work fine
as long
as I'm going between NFS clients and servers of the same version (NFSv4.1
client -> NFSv4.1 server). However, I have a situation where I'm trying to
proxy a NFSv4.1 service (Azure File Share NFS) for clients that do not
support NFSv4. When I do this, the "mount" command on the NFSv3 clients
works fine, but as soon as I try to do any data operations, I get I/O
errors:
# mount -t nfs -o vers=3 10.11.12.13:/azure-nfs-file /mnt/azure
# ls -l /mnt/azure
ls: reading directory '/mnt/azure': Remote I/O error
total 0
I've posted my (sanitized) configuration below. My questions are:
1) Is it possible to use Ganesha to proxy this way - allowing clients to
access a NFS server that differs in version, or is this not supported?
2) Am I missing something within my Ganesha config to enable this?
I can also provide the debug logging, if that's helpful.
Thanks - Nick
==ganesha.conf==
NFS_CORE_PARAM {
mount_path_pseudo = true;
Protocols = 3,4;
MNT_Port = 20048;
}
LOG {
Default_Log_Level = INFO;
Components {
FSAL = FULL_DEBUG;
}
}
EXPORT_DEFAULTS {
Access_Type = RW;
}
## Azure NFS Share
EXPORT
{
Export_id = 902;
Path = "/stgaccount/azure-file";
Pseudo = "/azure-file";
Access_Type = RW;
Squash = no_root_squash;
Sectype = sys;
FSAL {
Name = proxy;
Srv_Addr = 10.1.2.3;
Enable_Handle_Mapping = TRUE;
HandleMap_DB_Dir = "/var/ganesha/handledb/902";
HandleMap_Tmp_Dir = "/run/ganesha/tmp/902";
HandleMap_DB_Count = 8;
}
}
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
6 p.m.
On Fri, Jun 25, 2021 at 7:53 PM Solomon Boulos <boulos(a)waymo.com> wrote:
It should be!
The proxy code just acts like either a V4 client (ProxyV4 in the current
tree) or V3 client (ProxyV3). Ganesha itself handles the clients and
whatever they want to speak (v3, v4, 9p even I think, etc.).
That's what I thought, so that's good news.
Your use of Name = proxy, implies you're on an older branch
perhaps, and I
feel like I've had some bugfixes in the Proxy V4 code (in particular, I
know for my ProxyV3 I had to fixup the handling of path vs pseudo... I
don't recall if ProxyV4 had the same problem or not). I don't have any
experience with the handle mapping code (though I've certainly looked at
it). Is it strictly required for your test?
Regarding the "Name = proxy" configuration, do you have examples of what
I
should use for the Proxy V4 code? I had a hard time finding concrete
examples for the Proxy code in particular.
I'm running Ganesha 3.5 on CentOS 8, out of the SCL Ganesha repo. I've
installed the nfs-ganesha and nfs-ganesha-proxy packages (along with vfs
and utils). If there's a different version that I should be using, please
let me know - I'm not opposed to building from source, either.
Regarding the handle mapping, I have tried it both enabled and disabled and
the experience is identical. It's enabled in the config that I sent over
because that's the last thing I tried. I can disabled it, again, just to
make sure it isn't adding any rabbit holes to the debugging process.
Either way, please provide some debug logs. You may want to add
debug
logging to NFS_PROTO or whatever the name is as well. (And if I finally get
around to fixing my IRC setup, you can find folks in #ganesha, too for more
live debugging).
Yeah, no problem - here are the logs in a Pastbin:
https://pastebin.com/yxVsYfxw
I tried adding "NFS_PROTO = FULL_DEBUG;" to the components section of the
configuration, but I'm not sure I got that correct. I'll go back and look
at the available components and try a few other things - maybe NFS_V4 or
something like that?
-Nick
6:46 p.m.
On Fri, Jun 25, 2021 at 9:00 PM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Fri, Jun 25, 2021 at 7:53 PM Solomon Boulos
<boulos(a)waymo.com> wrote:
> It should be!
> The proxy code just acts like either a V4 client (ProxyV4
in the current
> tree) or V3 client (ProxyV3). Ganesha itself handles the clients and
> whatever they want to speak (v3, v4, 9p even I think, etc.).
That's what I thought, so
that's good news.
> Your use of Name = proxy, implies you're on an older branch perhaps, and
> I feel like I've had some bugfixes in the Proxy V4 code (in particular, I
> know for my ProxyV3 I had to fixup the handling of path vs pseudo... I
> don't recall if ProxyV4 had the same problem or not). I don't have any
> experience with the handle mapping code (though I've certainly looked at
> it). Is it strictly required for your test?
Regarding the "Name =
proxy" configuration, do you have examples of what I
should use for the Proxy V4 code? I had a hard time finding concrete
examples for the Proxy code in particular.
I'm running Ganesha 3.5 on CentOS 8, out of the SCL Ganesha repo. I've
installed the nfs-ganesha and nfs-ganesha-proxy packages (along with vfs
and utils). If there's a different version that I should be using, please
let me know - I'm not opposed to building from source, either.
Regarding the handle mapping, I have tried it both enabled and disabled
and the experience is identical. It's enabled in the config that I sent
over because that's the last thing I tried. I can disabled it, again, just
to make sure it isn't adding any rabbit holes to the debugging process.
> Either way, please provide some debug logs. You may want to add debug
> logging to NFS_PROTO or whatever the name is as well. (And if I finally get
> around to fixing my IRC setup, you can find folks in #ganesha, too for more
> live debugging).
Yeah, no problem - here are the logs in a Pastbin:
https://pastebin.com/yxVsYfxw
I tried adding "NFS_PROTO = FULL_DEBUG;" to the components section of the
configuration, but I'm not sure I got that correct. I'll go back and look
at the available components and try a few other things - maybe NFS_V4 or
something like that?
Okay, managed to get NFSPROTO and NFS_V4 debugging enabled - new logs, here:
https://pastebin.com/1dcKR9HR
-Nick
9:34 a.m.
Nick,
I've been using the Proxy FSAL for the past two years and have had much
better results since moving to the V4.dev branch about a year ago,
particularly after Solomon Boulos' improvements to the Proxy FSAL. I
don't claim to be using it exactly as you've described but I believe it
may be worth your while to try the V4.dev branch.
cheers,
Todd
11:04 a.m.
On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff(a)rhpcs.mcmaster.ca> wrote:
Nick,
I've been using the Proxy FSAL for the past two years and have had much
better results since moving to the V4.dev branch about a year ago,
particularly after Solomon Boulos' improvements to the Proxy FSAL. I
don't claim to be using it exactly as you've described but I believe it
may be worth your while to try the V4.dev branch.
Todd,
Thanks for the hints - I'll give it a shot - either see if packages are
available or just compile it myself.
-Nick
7:14 p.m.
On Sat, Jun 26, 2021 at 2:04 PM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff
<pfaff(a)rhpcs.mcmaster.ca wrote:
> Nick,
> I've been using the Proxy FSAL for the past two years
and have had much
> better results since moving to the V4.dev branch about a year ago,
> particularly after Solomon Boulos' improvements to the Proxy FSAL. I
> don't claim to be using it exactly as you've described but I believe it
> may be worth your while to try the V4.dev branch.
Todd,
Thanks for the hints - I'll give it a shot - either see if packages are
available or just compile it myself.
No joy after switching to the V4 Dev branch - behavior seems identical.
There's definitely something in the V3 -> V4 proxy that just doesn't work,
as mounting the proxied NFS share as NFSv4 from a client works fine, but
NFSv3 results in the "Remote I/O Error" behavior.
Here are the things I've tried so far:
* Both versions 3.5 (Current Stable) and V4 dev (latest release).
* Various combinations of Privileged Port enabled/disabled on both the
Export and the Proxy FSAL.
* mount_path_pseudo both enabled and disabled to see if the path
translation was causing issues for the V3 client.
* Both enabling and disabling Handle Mapping.
I'm open to any other suggestions that folks have for me, and am happy to
provide any further debug output.
-Nick
9:02 p.m.
On Sat, Jun 26, 2021 at 10:14 PM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Sat, Jun 26, 2021 at 2:04 PM Nick Couchman
<nick.e.couchman(a)gmail.com wrote:
> On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff(a)rhpcs.mcmaster.ca > wrote:
>> Nick,
> >> I've been using the Proxy FSAL for the past two
years and have had much
>> better results since moving to the V4.dev branch about a year ago,
>> particularly after Solomon Boulos' improvements to the Proxy FSAL. I
>> don't claim to be using it exactly as you've described but I believe it
>> may be worth your while to try the V4.dev branch.
> > > Todd,
> Thanks for the hints - I'll give it a shot - either see if packages are
> available or just compile it myself.
No joy after switching to the V4
Dev branch - behavior seems identical.
There's definitely something in the V3 -> V4 proxy that just doesn't work,
as mounting the proxied NFS share as NFSv4 from a client works fine, but
NFSv3 results in the "Remote I/O Error" behavior.
Here are the things I've tried so far:
* Both versions 3.5 (Current Stable) and V4 dev (latest release).
* Various combinations of Privileged Port enabled/disabled on both the
Export and the Proxy FSAL.
* mount_path_pseudo both enabled and disabled to see if the path
translation was causing issues for the V3 client.
* Both enabling and disabling Handle Mapping.
I'm open to any other suggestions that folks have for me, and am happy to
provide any further debug output.
Okay, I actually made a little bit of progress in figuring out what's going
on, here, with the help of tcpdump. There are two issues:
* Looks like the Ganesha NFSv3 server may not support the "readdirplus"
command? Output from "tcpdump" includes the following:
22:24:49.212153 IP 10.1.2.3.netrcs > 10.1.2.4.nfs: Flags [P.], seq 648:832,
ack 441, win 337, options [nop,nop,TS val 861164029 ecr 2104677449], length
184: NFS request xid 3221395829 180 readdirplus fh
Unknown/430003863737072F240000010000000000000001000000000000000380E6C8CC
2048 bytes @ 0
22:24:49.261230 IP 10.1.2.4.nfs > 10.1.2.3.netrcs: Flags [P.], seq 441:561,
ack 832, win 428, options [nop,nop,TS val 2104677499 ecr 861164029], length
120: NFS reply xid 3221395829 reply ok 116 readdirplus ERROR: Unspecified
error on server
I can get around this issue by adding the "nordirplus" NFS mount option. I
then start to get some part of the directory listing, but with lots of
errors.
* After getting past the readdirplus issue, I still get errors, and this
appears to be due to file handle issues:
22:48:54.686280 IP 10.1.2.4.nfs > 10.1.2.3.phonebook: Flags [P.], seq
4565:4601, ack 5925, win 520, options [nop,nop,TS val 2106122935 ecr
862609458], length 36: NFS reply xid 1689362215 reply ok 32 lookup ERROR:
Illegal NFS file handle
I figured out that the file handle issues are because none of the versions
of Ganesha I've been using are actually compiled with the file handle
mapping enabled. This includes the version in the CentOS Ganesha SIG (3.5,
with the older "Proxy" FSAL) and the version I was building from source via
the RPM SPEC file, which was not enabling the support for file handle
mapping in Proxy V4.
* All that said, when I finally do get a version of V4 Dev built that
enables Handle Mapping, it just core dumps:
Jun 26 23:54:41 nfs-proxy.example.com systemd-coredump[350511]: Process
350496 (ganesha.nfsd) of user 0 dumped core.
Stack
trace of thread 350496:
#0
0x00007f1f7caa6704 digest_alloc (libfsalproxy_v4.so)
#1
0x00007f1f7caa70a2 handle_mapping_hash_add (libfsalproxy_v4.so)
#2
0x00007f1f7caa75b2 HandleMap_SetFH (libfsalproxy_v4.so)
#3
0x00007f1f7caa51df proxyv4_alloc_handle (libfsalproxy_v4.so)
#4
0x00007f1f7caa0883 proxyv4_make_object (libfsalproxy_v4.so)
#5
0x00007f1f7caa0f01 proxyv4_lookup_impl (libfsalproxy_v4.so)
#6
0x00007f1f7caa5465 proxyv4_lookup_path (libfsalproxy_v4.so)
#7
0x00007f1f8968334d mdcache_lookup_path (libganesha_nfsd.so.4)
#8
0x00007f1f895fe1d7 init_export_root (libganesha_nfsd.so.4)
#9
0x00007f1f895fd8e6 init_export_cb (libganesha_nfsd.so.4)
#10
0x00007f1f896164c3 foreach_gsh_export (libganesha_nfsd.so.4)
#11
0x00007f1f895fd93c exports_pkginit (libganesha_nfsd.so.4)
#12
0x00007f1f89582c62 nfs_Init (libganesha_nfsd.so.4)
#13
0x00007f1f89583b2d nfs_start (libganesha_nfsd.so.4)
#14
0x00000000004029b9 main (ganesha.nfsd)
#15
0x00007f1f86e4a493 __libc_start_main (libc.so.6)
#16
0x00000000004018fe _start (ganesha.nfsd)
-Nick
9:55 p.m.
Good progress! I’ll try to take a look on Monday. Does the proxy FSAL
misrepresent it’s readdirplus support? (Seems possible, it should honor
whatever the backend says)
Just to confirm, does the backend not also speak V3? (Since you have v3
clients, you could use my simpler v3 proxy in a modern client and not do 3
<=> 4 via Ganesha).
On Sat, Jun 26, 2021 at 21:05 Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
On Sat, Jun 26, 2021 at 10:14 PM Nick Couchman
<nick.e.couchman(a)gmail.com>
wrote:
> On Sat, Jun 26, 2021 at 2:04 PM Nick Couchman <nick.e.couchman(a)gmail.com>
> wrote:
>
>> On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff(a)rhpcs.mcmaster.ca>
>> wrote:
>>
>>> Nick,
>>>
>>> I've been using the Proxy FSAL for the past two years and have had much
>>> better results since moving to the V4.dev branch about a year ago,
>>> particularly after Solomon Boulos' improvements to the Proxy FSAL. I
>>> don't claim to be using it exactly as you've described but I believe
it
>>> may be worth your while to try the V4.dev branch.
>>>
>>>
>> Todd,
>> Thanks for the hints - I'll give it a shot - either see if packages are
>> available or just compile it myself.
>>
>>
> No joy after switching to the V4 Dev branch - behavior seems identical.
> There's definitely something in the V3 -> V4 proxy that just doesn't
work,
> as mounting the proxied NFS share as NFSv4 from a client works fine, but
> NFSv3 results in the "Remote I/O Error" behavior.
>
> Here are the things I've tried so far:
> * Both versions 3.5 (Current Stable) and V4 dev (latest release).
> * Various combinations of Privileged Port enabled/disabled on both the
> Export and the Proxy FSAL.
> * mount_path_pseudo both enabled and disabled to see if the path
> translation was causing issues for the V3 client.
> * Both enabling and disabling Handle Mapping.
>
> I'm open to any other suggestions that folks have for me, and am happy to
> provide any further debug output.
>
Okay, I actually made a little bit of progress in figuring out what's
going on, here, with the help of tcpdump. There are two issues:
* Looks like the Ganesha NFSv3 server may not support the "readdirplus"
command? Output from "tcpdump" includes the following:
22:24:49.212153 IP 10.1.2.3.netrcs > 10.1.2.4.nfs: Flags [P.], seq
648:832, ack 441, win 337, options [nop,nop,TS val 861164029 ecr
2104677449], length 184: NFS request xid 3221395829 180 readdirplus fh
Unknown/430003863737072F240000010000000000000001000000000000000380E6C8CC
2048 bytes @ 0
22:24:49.261230 IP 10.1.2.4.nfs > 10.1.2.3.netrcs: Flags [P.], seq
441:561, ack 832, win 428, options [nop,nop,TS val 2104677499 ecr
861164029], length 120: NFS reply xid 3221395829 reply ok 116 readdirplus
ERROR: Unspecified error on server
I can get around this issue by adding the "nordirplus" NFS mount option. I
then start to get some part of the directory listing, but with lots of
errors.
* After getting past the readdirplus issue, I still get errors, and this
appears to be due to file handle issues:
22:48:54.686280 IP 10.1.2.4.nfs > 10.1.2.3.phonebook: Flags [P.], seq
4565:4601, ack 5925, win 520, options [nop,nop,TS val 2106122935 ecr
862609458], length 36: NFS reply xid 1689362215 reply ok 32 lookup ERROR:
Illegal NFS file handle
I figured out that the file handle issues are because none of the versions
of Ganesha I've been using are actually compiled with the file handle
mapping enabled. This includes the version in the CentOS Ganesha SIG (3.5,
with the older "Proxy" FSAL) and the version I was building from source via
the RPM SPEC file, which was not enabling the support for file handle
mapping in Proxy V4.
* All that said, when I finally do get a version of V4 Dev built that
enables Handle Mapping, it just core dumps:
Jun 26 23:54:41 nfs-proxy.example.com systemd-coredump[350511]: Process
350496 (ganesha.nfsd) of user 0 dumped core.
Stack
trace of thread 350496:
#0
0x00007f1f7caa6704 digest_alloc (libfsalproxy_v4.so)
#1
0x00007f1f7caa70a2 handle_mapping_hash_add (libfsalproxy_v4.so)
#2
0x00007f1f7caa75b2 HandleMap_SetFH (libfsalproxy_v4.so)
#3
0x00007f1f7caa51df proxyv4_alloc_handle (libfsalproxy_v4.so)
#4
0x00007f1f7caa0883 proxyv4_make_object (libfsalproxy_v4.so)
#5
0x00007f1f7caa0f01 proxyv4_lookup_impl (libfsalproxy_v4.so)
#6
0x00007f1f7caa5465 proxyv4_lookup_path (libfsalproxy_v4.so)
#7
0x00007f1f8968334d mdcache_lookup_path (libganesha_nfsd.so.4)
#8
0x00007f1f895fe1d7 init_export_root (libganesha_nfsd.so.4)
#9
0x00007f1f895fd8e6 init_export_cb (libganesha_nfsd.so.4)
#10
0x00007f1f896164c3 foreach_gsh_export (libganesha_nfsd.so.4)
#11
0x00007f1f895fd93c exports_pkginit (libganesha_nfsd.so.4)
#12
0x00007f1f89582c62 nfs_Init (libganesha_nfsd.so.4)
#13
0x00007f1f89583b2d nfs_start (libganesha_nfsd.so.4)
#14
0x00000000004029b9 main (ganesha.nfsd)
#15
0x00007f1f86e4a493 __libc_start_main (libc.so.6)
#16
0x00000000004018fe _start (ganesha.nfsd)
-Nick
> _______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
7:46 a.m.
On Sun, Jun 27, 2021 at 12:55 AM Solomon Boulos <boulos(a)google.com> wrote:
Good progress! I’ll try to take a look on Monday. Does the proxy
FSAL
misrepresent it’s readdirplus support? (Seems possible, it should honor
whatever the backend says)
That would be great, I'd definitely appreciate any insight. Not sure about
the FSAL module and its representation of readdirplus support, but I can
dig through the code and take a look. That one I can work around - the Core
Dump when the Proxy v4 module has Handle Mapping enabled is a bit more
problematic. The function it's referencing at the top - digest_alloc - is
simple enough, not really much I can see that would cause a core dump,
there.
Just to confirm, does the backend not also speak V3? (Since you have
v3
clients, you could use my simpler v3 proxy in a modern client and not do 3
<=> 4 via Ganesha).
Correct - it's most of the reason why I'm trying to use Ganesha to proxy at
this point, because the back-end (Azure Storage Account with NFS share)
only speaks NFS v4.1, and the systems we need to mount it to (primarily AIX
7.2) only support NFS v3 clients. So, my entire goal here is to use Ganesha
to translate between the two. A side benefit is that I can centralize a
bunch of mounts on a single server.
-Nick
7:45 a.m.
On Sun, Jun 27, 2021 at 10:46 AM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Sun, Jun 27, 2021 at 12:55 AM Solomon Boulos
<boulos(a)google.com> wrote:
> Good progress! I’ll try to take a look on Monday. Does the proxy FSAL
> misrepresent it’s readdirplus support? (Seems possible, it should honor
> whatever the backend says)
That would be great, I'd
definitely appreciate any insight. Not sure about
the FSAL module and its representation of readdirplus support, but I can
dig through the code and take a look. That one I can work around - the Core
Dump when the Proxy v4 module has Handle Mapping enabled is a bit more
problematic. The function it's referencing at the top - digest_alloc - is
simple enough, not really much I can see that would cause a core dump,
there.
Regarding the segfault/core dump, I've got the back trace, here:
(gdb) bt
#0 0x00007f4abbea6704 in digest_alloc ()
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:68
#1 0x00007f4abbea70a2 in handle_mapping_hash_add (p_hash=0x0, object_id=0,
handle_hash=2762487084, data=0x7f4abe858432, datalen=29)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:193
#2 0x00007f4abbea75b2 in HandleMap_SetFH
(p_in_nfs23_digest=0x7f4abe858420, data=0x7f4abe858432, len=29)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:373
#3 0x00007f4abbea51df in proxyv4_alloc_handle (exp=0x7f4abe9fc300,
fh=0x7ffd7e7a0a40, obj_attributes=0x7ffd7e7a0ae0,
attrs_out=0x7ffd7e7a10b0) at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:2997
#4 0x00007f4abbea0883 in proxyv4_make_object (export=0x7f4abe9fc300,
obj_attributes=0x7ffd7e7a0ae0, fh=0x7ffd7e7a0a40,
handle=0x7ffd7e7a1038, attrs_out=0x7ffd7e7a10b0)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1399
#5 0x00007f4abbea0f01 in proxyv4_lookup_impl (parent=0x0,
export=0x7f4abe9fc300, cred=0x7ffd7e7a1258,
path=0x7f4abe813171 "DataBoxDisk1", handle=0x7ffd7e7a1038,
attrs_out=0x7ffd7e7a10b0)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1545
#6 0x00007f4abbea5465 in proxyv4_lookup_path (exp_hdl=0x7f4abe9fc300,
path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a11b0,
attrs_out=0x7ffd7e7a10b0) at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:3068
#7 0x00007f4ac8a0b34d in mdcache_lookup_path (exp_hdl=0x7f4abe82b400,
path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a1338,
attrs_out=0x0) at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1564
#8 0x00007f4ac89861d7 in init_export_root (export=0x7f4abe877cc8)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2619
#9 0x00007f4ac89858e6 in init_export_cb (exp=0x7f4abe877cc8,
state=0x7ffd7e7a1420)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2458
#10 0x00007f4ac899e4c3 in foreach_gsh_export (cb=0x7f4ac89858c2
<init_export_cb>, wrlock=true, state=0x7ffd7e7a1420)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/export_mgr.c:801
#11 0x00007f4ac898593c in exports_pkginit () at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2477
#12 0x00007f4ac890ac62 in nfs_Init (p_start_info=0x6041a8
<my_nfs_start_info>)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:642
#13 0x00007f4ac890bb2d in nfs_start (p_start_info=0x6041a8
<my_nfs_start_info>)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:923
#14 0x00000000004029b9 in main (argc=7, argv=0x7ffd7e7a1838)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_main.c:520
This is quite puzzling, because the line that seems to be causing the
segfault is the pool_alloc line below:
==
pool_t *digest_pool;
static pthread_mutex_t digest_pool_mutex = PTHREAD_MUTEX_INITIALIZER;
pool_t *handle_pool;
static pthread_mutex_t handle_pool_mutex = PTHREAD_MUTEX_INITIALIZER;
/* helpers for pool allocation */
static digest_pool_entry_t *digest_alloc()
{
digest_pool_entry_t *p_new;
PTHREAD_MUTEX_lock(&digest_pool_mutex);
p_new = pool_alloc(digest_pool);
PTHREAD_MUTEX_unlock(&digest_pool_mutex);
return p_new;
}
==
I don't see how this call could be causing a segfault, though, as this
function is supposed to be initializing the pool. I also checked other
instances of pool_alloc() throughout the code and don't really see
substantial differences??
-Nick
10:40 a.m.
On Mon, Jun 28, 2021 at 10:45 AM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Sun, Jun 27, 2021 at 10:46 AM Nick Couchman
<nick.e.couchman(a)gmail.com wrote:
> On Sun, Jun 27, 2021 at 12:55 AM Solomon Boulos <boulos(a)google.com > wrote:
>> Good progress! I’ll try to take a look on Monday. Does
the proxy FSAL
>> misrepresent it’s readdirplus support? (Seems possible, it should honor
>> whatever the backend says)
> > > That would be great, I'd
definitely appreciate any insight. Not sure
> about the FSAL module and its representation of readdirplus support, but I
> can dig through the code and take a look. That one I can work around - the
> Core Dump when the Proxy v4 module has Handle Mapping enabled is a bit more
> problematic. The function it's referencing at the top - digest_alloc - is
> simple enough, not really much I can see that would cause a core dump,
> there.
Regarding the segfault/core dump, I've got the back trace, here:
(gdb) bt
#0 0x00007f4abbea6704 in digest_alloc ()
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:68
#1 0x00007f4abbea70a2 in handle_mapping_hash_add (p_hash=0x0,
object_id=0, handle_hash=2762487084, data=0x7f4abe858432, datalen=29)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:193
#2 0x00007f4abbea75b2 in HandleMap_SetFH
(p_in_nfs23_digest=0x7f4abe858420, data=0x7f4abe858432, len=29)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:373
#3 0x00007f4abbea51df in proxyv4_alloc_handle (exp=0x7f4abe9fc300,
fh=0x7ffd7e7a0a40, obj_attributes=0x7ffd7e7a0ae0,
attrs_out=0x7ffd7e7a10b0) at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:2997
#4 0x00007f4abbea0883 in proxyv4_make_object (export=0x7f4abe9fc300,
obj_attributes=0x7ffd7e7a0ae0, fh=0x7ffd7e7a0a40,
handle=0x7ffd7e7a1038, attrs_out=0x7ffd7e7a10b0)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1399
#5 0x00007f4abbea0f01 in proxyv4_lookup_impl (parent=0x0,
export=0x7f4abe9fc300, cred=0x7ffd7e7a1258,
path=0x7f4abe813171 "DataBoxDisk1", handle=0x7ffd7e7a1038,
attrs_out=0x7ffd7e7a10b0)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1545
#6 0x00007f4abbea5465 in proxyv4_lookup_path (exp_hdl=0x7f4abe9fc300,
path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a11b0,
attrs_out=0x7ffd7e7a10b0) at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:3068
#7 0x00007f4ac8a0b34d in mdcache_lookup_path (exp_hdl=0x7f4abe82b400,
path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a1338,
attrs_out=0x0) at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1564
#8 0x00007f4ac89861d7 in init_export_root (export=0x7f4abe877cc8)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2619
#9 0x00007f4ac89858e6 in init_export_cb (exp=0x7f4abe877cc8,
state=0x7ffd7e7a1420)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2458
#10 0x00007f4ac899e4c3 in foreach_gsh_export (cb=0x7f4ac89858c2
<init_export_cb>, wrlock=true, state=0x7ffd7e7a1420)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/export_mgr.c:801
#11 0x00007f4ac898593c in exports_pkginit () at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2477
#12 0x00007f4ac890ac62 in nfs_Init (p_start_info=0x6041a8
<my_nfs_start_info>)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:642
#13 0x00007f4ac890bb2d in nfs_start (p_start_info=0x6041a8
<my_nfs_start_info>)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:923
#14 0x00000000004029b9 in main (argc=7, argv=0x7ffd7e7a1838)
at
/usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_main.c:520
This is quite puzzling, because the line that seems to be causing the
segfault is the pool_alloc line below:
==
pool_t *digest_pool;
static pthread_mutex_t digest_pool_mutex = PTHREAD_MUTEX_INITIALIZER;
pool_t *handle_pool;
static pthread_mutex_t handle_pool_mutex = PTHREAD_MUTEX_INITIALIZER;
/* helpers for pool allocation */
static digest_pool_entry_t *digest_alloc()
{
digest_pool_entry_t *p_new;
PTHREAD_MUTEX_lock(&digest_pool_mutex);
p_new = pool_alloc(digest_pool);
PTHREAD_MUTEX_unlock(&digest_pool_mutex);
return p_new;
}
==
I don't see how this call could be causing a segfault, though, as this
function is supposed to be initializing the pool. I also checked other
instances of pool_alloc() throughout the code and don't really see
substantial differences??
I'm sure I'm missing something simple, here, but still not sure why this is
causing a segfault, unless the presence of the mutex lock/unlock does
something to mess with it?
Regarding the readdirplus support, I tried manually disabling it in the
proxy_v4 FSAL module by adding ".readdir_plus = false," to the info
initialization in main.c. This doesn't seem to have a made a difference,
though, so it almost seems as if the NFSv3 server support for it in Ganesha
overrides or ignores whatever the V4 proxy module sets, so that it is
expecting readdirplus to work.
I also don't know why readdirplus would not work - maybe something just
needs to be implemented in the V4 proxy module to pass the call through??
-Nick
12:03 p.m.
I'm sure I'm missing something simple, here, but still not sure why this
is causing a segfault, unless the presence of the mutex lock/unlock does
something to mess with it?
Well, I'm feeling a bit silly about this right now, and have resolved the
segfault/core dump issue. The issue was that the directory for handle
mapping didn't exist, so the HandleMap_Init function call would exit out
without actually allocating the structures for digest and handle pools, but
the proxy module would continue to try loading and try to use the handle
map. Apparently I didn't catch this in the log file :-/.
That said, I'm on to the next issue, which is that, with handle mapping
enabled, finally, and the nordirplus mount option (to get around the
readdirplus issue), I now get a "Stale File Handle" error when trying to
mount:
# mount -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure
mount.nfs: Stale file handle
Current log file, during the mount attempt, is here:
https://pastebin.com/wKhLENFk
-Nick
12:10 p.m.
Hmm, we do need to make that SEGFAULT not happen… Directories will be missing in life…
Get a tcpdump and look at where the handle that is causing the stale handle error is
coming from. Do we handle map the handles shared in READDIR?
We also need to figure out why READDIRPLUS isn’t working…
Frank
From: Nick Couchman [mailto:nick.e.couchman@gmail.com]
Sent: Monday, June 28, 2021 12:04 PM
To: Solomon Boulos <boulos(a)google.com>
Cc: Todd Pfaff <pfaff(a)rhpcs.mcmaster.ca>; support(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3
I'm sure I'm missing something simple, here, but still not sure why this is
causing a segfault, unless the presence of the mutex lock/unlock does something to mess
with it?
Well, I'm feeling a bit silly about this right now, and have resolved the
segfault/core dump issue. The issue was that the directory for handle mapping didn't
exist, so the HandleMap_Init function call would exit out without actually allocating the
structures for digest and handle pools, but the proxy module would continue to try loading
and try to use the handle map. Apparently I didn't catch this in the log file :-/.
That said, I'm on to the next issue, which is that, with handle mapping enabled,
finally, and the nordirplus mount option (to get around the readdirplus issue), I now get
a "Stale File Handle" error when trying to mount:
# mount -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure
mount.nfs: Stale file handle
Current log file, during the mount attempt, is here:
https://pastebin.com/wKhLENFk
-Nick
1:28 p.m.
On Mon, Jun 28, 2021 at 4:21 PM Frank Filz <ffilzlnx(a)mindspring.com> wrote:
Hmm, we do need to make that SEGFAULT not happen… Directories will
be
missing in life…
Yes, I'd agree this is not ideal behavior - I would think if Handle Mapping
is enabled and the directory does not exist, the process of configuring
that export ought to be completely stopped.
Get a tcpdump and look at where the handle that is causing the stale
handle error is coming from. Do we handle map the handles shared in READDIR?
I put the tshark output in a pastebin and replied with that to the previous
e-mail. If I can provide any more detail let me know. I think it's the root
handle that is causing it, since it happens during mount?? But this level
of NFS detail is very new to me :-).
We also need to figure out why READDIRPLUS isn’t working…
Yes, this would be nice, too. Looking through the Proxy_V4 code, I do not
see any references to readdir_plus or readdirplus at all, so I'm wondering
if the Proxy module doesn't contain the necessary methods to push the
readdirplus command through?? Again, I'm a novice at this level of NFS
detail as well as the Ganesha code, so I could be way off.
-Nick
2:40 p.m.
On Mon, Jun 28, 2021 at 4:28 PM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Mon, Jun 28, 2021 at 4:21 PM Frank Filz
<ffilzlnx(a)mindspring.com wrote:
> Hmm, we do need to make that SEGFAULT not happen… Directories will be
> missing in life…
Yes, I'd agree this is not ideal behavior - I would think if Handle
Mapping is enabled and the directory does not exist, the process of
configuring that export ought to be completely stopped.
> Get a tcpdump and look at where
the handle that is causing the stale
> handle error is coming from. Do we handle map the handles shared in READDIR?
I put the tshark output in a pastebin and replied with that to the
previous e-mail. If I can provide any more detail let me know. I think it's
the root handle that is causing it, since it happens during mount?? But
this level of NFS detail is very new to me :-).
Actually I'm not sure this is in readdir, but actually during FSINFO
processing, as the NFS3ERR_STALE response comes directly after the client
does a FSINFO request:
Frame 152: 190 bytes on wire (1520 bits), 190 bytes captured (1520 bits) on
interface 0
Ethernet II, Src: Cisco_9d:5a:42 (8c:60:4f:9d:5a:42), Dst:
b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58)
Internet Protocol Version 4, Src: 10.1.2.5, Dst: 10.1.2.4
Transmission Control Protocol, Src Port: 889, Dst Port: 2049, Seq: 169,
Ack: 65, Len: 124
Remote Procedure Call, Type:Call XID:0x76f80e97
Fragment header: Last fragment, 120 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 0111 1000 = Fragment Length: 120
XID: 0x76f80e97 (1995968151)
Message Type: Call (0)
RPC Version: 2
Program: NFS (100003)
Program Version: 3
Procedure: FSINFO (19)
Credentials
Flavor: AUTH_UNIX (1)
Length: 52
Stamp: 0x00000000
Machine Name: nfs-client.example.com
length: 25
contents: nfs-client.example.com
fill bytes: opaque data
UID: 0
GID: 0
Auxiliary GIDs (1) [0]
GID: 0
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Network File System, FSINFO Call DH: 0xb2a31f30
[Program Version: 3]
[V3 Procedure: FSINFO (19)]
object
length: 24
[hash (CRC-32): 0xb2a31f30]
FileHandle: 4300038610102300006eb32fce0000000000000000000000
Frame 153: 102 bytes on wire (816 bits), 102 bytes captured (816 bits) on
interface 0
Ethernet II, Src: b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58), Dst:
Cisco_9f:f0:6a (00:00:0c:9f:f0:6a)
Internet Protocol Version 4, Src: 10.1.2.4, Dst: 10.1.2.5
Transmission Control Protocol, Src Port: 2049, Dst Port: 889, Seq: 65, Ack:
293, Len: 36
Remote Procedure Call, Type:Reply XID:0x76f80e97
Fragment header: Last fragment, 32 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 0010 0000 = Fragment Length: 32
XID: 0x76f80e97 (1995968151)
Message Type: Reply (1)
[Program: NFS (100003)]
[Program Version: 3]
[Procedure: FSINFO (19)]
Reply State: accepted (0)
[This is a reply to a request in frame 152]
[Time from request: 0.000301532 seconds]
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Accept State: RPC executed successfully (0)
Network File System, FSINFO Reply Error: NFS3ERR_STALE
[Program Version: 3]
[V3 Procedure: FSINFO (19)]
Status: NFS3ERR_STALE (70)
obj_attributes
attributes_follow: no value (0)
-Nick
2:51 p.m.
Could you put the binary wire capture file somewhere? I find it a lot easier to read wire
traces in wireshark UI…
Frank
From: Nick Couchman [mailto:nick.e.couchman@gmail.com]
Sent: Monday, June 28, 2021 2:40 PM
To: Frank Filz <ffilzlnx(a)mindspring.com>
Cc: Solomon Boulos <boulos(a)google.com>; Todd Pfaff <pfaff(a)rhpcs.mcmaster.ca>;
support(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3
On Mon, Jun 28, 2021 at 4:28 PM Nick Couchman <nick.e.couchman(a)gmail.com
<mailto:nick.e.couchman@gmail.com> > wrote:
On Mon, Jun 28, 2021 at 4:21 PM Frank Filz <ffilzlnx(a)mindspring.com
<mailto:ffilzlnx@mindspring.com> > wrote:
Hmm, we do need to make that SEGFAULT not happen… Directories will be missing in life…
Yes, I'd agree this is not ideal behavior - I would think if Handle Mapping is enabled
and the directory does not exist, the process of configuring that export ought to be
completely stopped.
Get a tcpdump and look at where the handle that is causing the stale handle error is
coming from. Do we handle map the handles shared in READDIR?
I put the tshark output in a pastebin and replied with that to the previous e-mail. If I
can provide any more detail let me know. I think it's the root handle that is causing
it, since it happens during mount?? But this level of NFS detail is very new to me :-).
Actually I'm not sure this is in readdir, but actually during FSINFO processing, as
the NFS3ERR_STALE response comes directly after the client does a FSINFO request:
Frame 152: 190 bytes on wire (1520 bits), 190 bytes captured (1520 bits) on interface 0
Ethernet II, Src: Cisco_9d:5a:42 (8c:60:4f:9d:5a:42), Dst: b2:5a:dd:d6:98:58
(b2:5a:dd:d6:98:58)
Internet Protocol Version 4, Src: 10.1.2.5, Dst: 10.1.2.4
Transmission Control Protocol, Src Port: 889, Dst Port: 2049, Seq: 169, Ack: 65, Len: 124
Remote Procedure Call, Type:Call XID:0x76f80e97
Fragment header: Last fragment, 120 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 0111 1000 = Fragment Length: 120
XID: 0x76f80e97 (1995968151)
Message Type: Call (0)
RPC Version: 2
Program: NFS (100003)
Program Version: 3
Procedure: FSINFO (19)
Credentials
Flavor: AUTH_UNIX (1)
Length: 52
Stamp: 0x00000000
Machine Name: nfs-client.example.com <http://nfs-client.example.com>
length: 25
contents: nfs-client.example.com <http://nfs-client.example.com>
fill bytes: opaque data
UID: 0
GID: 0
Auxiliary GIDs (1) [0]
GID: 0
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Network File System, FSINFO Call DH: 0xb2a31f30
[Program Version: 3]
[V3 Procedure: FSINFO (19)]
object
length: 24
[hash (CRC-32): 0xb2a31f30]
FileHandle: 4300038610102300006eb32fce0000000000000000000000
Frame 153: 102 bytes on wire (816 bits), 102 bytes captured (816 bits) on interface 0
Ethernet II, Src: b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58), Dst: Cisco_9f:f0:6a
(00:00:0c:9f:f0:6a)
Internet Protocol Version 4, Src: 10.1.2.4, Dst: 10.1.2.5
Transmission Control Protocol, Src Port: 2049, Dst Port: 889, Seq: 65, Ack: 293, Len: 36
Remote Procedure Call, Type:Reply XID:0x76f80e97
Fragment header: Last fragment, 32 bytes
1... .... .... .... .... .... .... .... = Last Fragment: Yes
.000 0000 0000 0000 0000 0000 0010 0000 = Fragment Length: 32
XID: 0x76f80e97 (1995968151)
Message Type: Reply (1)
[Program: NFS (100003)]
[Program Version: 3]
[Procedure: FSINFO (19)]
Reply State: accepted (0)
[This is a reply to a request in frame 152]
[Time from request: 0.000301532 seconds]
Verifier
Flavor: AUTH_NULL (0)
Length: 0
Accept State: RPC executed successfully (0)
Network File System, FSINFO Reply Error: NFS3ERR_STALE
[Program Version: 3]
[V3 Procedure: FSINFO (19)]
Status: NFS3ERR_STALE (70)
obj_attributes
attributes_follow: no value (0)
-Nick
3:35 p.m.
On Mon, Jun 28, 2021 at 5:51 PM Frank Filz <ffilzlnx(a)mindspring.com> wrote:
Could you put the binary wire capture file somewhere? I find it a
lot
easier to read wire traces in wireshark UI…
Sure, here you go - I've done some editing to sanitize it a bit, but it
should still show the overall flow:
https://drive.google.com/file/d/1PI22xjraWQ2b8HXJsySpoIkmrOe-ZIOe/view?us...
-Nick
8:10 p.m.
On Mon, Jun 28, 2021 at 6:35 PM Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
On Mon, Jun 28, 2021 at 5:51 PM Frank Filz
<ffilzlnx(a)mindspring.com>
wrote:
> Could you put the binary wire capture file somewhere? I find it a lot
> easier to read wire traces in wireshark UI…
>
>
>
Sure, here you go - I've done some editing to sanitize it a bit, but it
should still show the overall flow:
https://drive.google.com/file/d/1PI22xjraWQ2b8HXJsySpoIkmrOe-ZIOe/view?us...
-Nick
Okay, added some additional debugging information and here's what I'm
seeing...
* The FSINFO request over NFS triggers the nfs3_fsinfo call.
* This FSINFO request triggers a call to nfs3_FhandleToCache
* nfs3_FhandleToCache calls proxyv4_wire_to_host via
the export->export.exp_ops.wire_to_host callback
* proxyv4_wire_to_host calls HandleMap_GetFH in the handle_mapping.c file.
* HandleMap_GetFH does the hashtable lookup and initially succeeds, but
then fails in this section:
rc = hashtable_getlatch(handle_map_hash, &buffkey, &buffval, 0,
&hl);
if (rc == HASHTABLE_SUCCESS) {
handle_pool_entry_t *h = (handle_pool_entry_t *)
buffval.addr;
if (h->fh_len < fsal_handle->len) {
fsal_handle->len = h->fh_len;
memcpy(fsal_handle->addr, h->fh_data, h->fh_len);
rc = HANDLEMAP_SUCCESS;
} else {
rc = HANDLEMAP_INTERNAL_ERROR;
}
hashtable_releaselatched(handle_map_hash, &hl);
return rc;
}
The statement "if (h->fh_len < fsal_handle->len)" apparently is not
true
(so h->fh_len is either greater than or equal to fsal_handle->len), and it
skips down to the next section, assigning rc the HANDLEMAP_INTERNAL_ERROR
value and then bails out.
-Nick
8:41 p.m.
Huh. Can you LogDebug those values? It seems to me like that check should
be <= not strict less than.
On Mon, Jun 28, 2021 at 20:10 Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
On Mon, Jun 28, 2021 at 6:35 PM Nick Couchman
<nick.e.couchman(a)gmail.com>
wrote:
> On Mon, Jun 28, 2021 at 5:51 PM Frank Filz <ffilzlnx(a)mindspring.com>
> wrote:
>
>> Could you put the binary wire capture file somewhere? I find it a lot
>> easier to read wire traces in wireshark UI…
>>
>>
>>
>
> Sure, here you go - I've done some editing to sanitize it a bit, but it
> should still show the overall flow:
>
>
> https://drive.google.com/file/d/1PI22xjraWQ2b8HXJsySpoIkmrOe-ZIOe/view?us...
>
> -Nick
>
Okay, added some additional debugging information and here's what I'm
seeing...
* The FSINFO request over NFS triggers the nfs3_fsinfo call.
* This FSINFO request triggers a call to nfs3_FhandleToCache
* nfs3_FhandleToCache calls proxyv4_wire_to_host via
the export->export.exp_ops.wire_to_host callback
* proxyv4_wire_to_host calls HandleMap_GetFH in the handle_mapping.c file.
* HandleMap_GetFH does the hashtable lookup and initially succeeds, but
then fails in this section:
rc = hashtable_getlatch(handle_map_hash, &buffkey, &buffval, 0,
&hl);
if (rc == HASHTABLE_SUCCESS) {
handle_pool_entry_t *h = (handle_pool_entry_t *)
buffval.addr;
if (h->fh_len < fsal_handle->len) {
fsal_handle->len = h->fh_len;
memcpy(fsal_handle->addr, h->fh_data, h->fh_len);
rc = HANDLEMAP_SUCCESS;
} else {
rc = HANDLEMAP_INTERNAL_ERROR;
}
hashtable_releaselatched(handle_map_hash, &hl);
return rc;
}
The statement "if (h->fh_len < fsal_handle->len)" apparently is not
true
(so h->fh_len is either greater than or equal to fsal_handle->len), and it
skips down to the next section, assigning rc the HANDLEMAP_INTERNAL_ERROR
value and then bails out.
-Nick
8:45 p.m.
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos(a)waymo.com> wrote:
Huh. Can you LogDebug those values? It seems to me like that check
should
be <= not strict less than.
As a matter of fact, yes:
28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com :
ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55,
fsal_handle->len: 16
So, changing a strict < to <= won't help much...
-Nick
8:54 p.m.
Thanks! I’ll have to re-read this code not on my phone tomorrow :). There’s
an amusing comment above the GetFH call in wire_to_host about reusing the
same buffer for input and output (so maybe the len check got added
afterwards?). But I’d rather re-learn what this code is trying to do anyway.
On Mon, Jun 28, 2021 at 20:45 Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos
<boulos(a)waymo.com> wrote:
> Huh. Can you LogDebug those values? It seems to me like that check should
> be <= not strict less than.
>
As a matter of fact, yes:
28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com :
ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55,
fsal_handle->len: 16
So, changing a strict < to <= won't help much...
-Nick
>
8:57 p.m.
On Mon, Jun 28, 2021 at 11:54 PM Solomon Boulos <boulos(a)google.com> wrote:
Thanks! I’ll have to re-read this code not on my phone tomorrow :).
There’s an amusing comment above the GetFH call in wire_to_host about
reusing the same buffer for input and output (so maybe the len check got
added afterwards?). But I’d rather re-learn what this code is trying to do
anyway.
Sounds good. If I can provide any more information let me know.
-Nick
10:13 p.m.
Looks like the handle mapping code isn’t kicking in or isn’t doing what we think it should
do.
Though we should also have space for a 55 byte wire handle…
Maybe that 16 byte space is wrong.
Frank
From: Nick Couchman [mailto:nick.e.couchman@gmail.com]
Sent: Monday, June 28, 2021 8:45 PM
To: Solomon Boulos <boulos(a)waymo.com>
Cc: Frank Filz <ffilzlnx(a)mindspring.com>; Todd Pfaff
<pfaff(a)rhpcs.mcmaster.ca>; support(a)lists.nfs-ganesha.org
Subject: [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos(a)waymo.com
<mailto:boulos@waymo.com> > wrote:
Huh. Can you LogDebug those values? It seems to me like that check should be <= not
strict less than.
As a matter of fact, yes:
28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com
<http://nfs-proxy.example.com> : ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL
:DEBUG :h->fh_len: 55, fsal_handle->len: 16
So, changing a strict < to <= won't help much...
-Nick
8:52 p.m.
Alright, looking through some git blame -w, I think this broke and then was
restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the
code compile, but crucially the buffer code no longer had a “max len” just
a length.
I think this broke the “clever” reuse of the input buffdesc as the argument
to GetFH (which has been there since 2012 from 878f45493e).
The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and
GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the
code when written had a FH4_SIZE as it’s max length, rather than the
fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there
was a length separate from max length at the time or something).
IIRC, all the handle mapping code *actually* passes around a buffer that is
large enough for the largest FH4 plus some padding. So the “resizing” done
in handle_to_wire that needs to be undone in wire_to_host can actually
happen in-place but the len field doesn’t know that.
Ganesha experts, does that sound right, historically?
On Mon, Jun 28, 2021 at 22:14 Frank Filz <ffilzlnx(a)mindspring.com> wrote:
Looks like the handle mapping code isn’t kicking in or isn’t doing
what we
think it should do.
Though we should also have space for a 55 byte wire handle…
Maybe that 16 byte space is wrong.
Frank
*From:* Nick Couchman [mailto:nick.e.couchman@gmail.com]
*Sent:* Monday, June 28, 2021 8:45 PM
*To:* Solomon Boulos <boulos(a)waymo.com>
*Cc:* Frank Filz <ffilzlnx(a)mindspring.com>; Todd Pfaff <
pfaff(a)rhpcs.mcmaster.ca>; support(a)lists.nfs-ganesha.org
*Subject:* [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos(a)waymo.com> wrote:
Huh. Can you LogDebug those values? It seems to me like that check should
be <= not strict less than.
As a matter of fact, yes:
28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com :
ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55,
fsal_handle->len: 16
So, changing a strict < to <= won't help much...
-Nick
6:13 a.m.
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos(a)google.com> wrote:
Alright, looking through some git blame -w, I think this broke and
then
was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the
code compile, but crucially the buffer code no longer had a “max len” just
a length.
I think this broke the “clever” reuse of the input buffdesc as the
argument to GetFH (which has been there since 2012 from 878f45493e).
I'll try to take a look at each of these commits and see if I can figure
out how to re-integrate the support back into the current code, though I
don't know how successful I will be. If anyone has suggestions for what
needs to be done, please let me know - I'm not sure right now if it's just
some changes in the handle_mapping.c file, or if there are changes
elsewhere that would need to be made (like in the
wire_to_host/handle_to_wire functions, which I think are elsewhere.
-Nick
7:08 p.m.
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos(a)google.com> wrote:
Alright, looking through some git blame -w, I think this broke and
then
was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the
code compile, but crucially the buffer code no longer had a “max len” just
a length.
I think this broke the “clever” reuse of the input buffdesc as the
argument to GetFH (which has been there since 2012 from 878f45493e).
The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and
GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the
code when written had a FH4_SIZE as it’s max length, rather than the
fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there
was a length separate from max length at the time or something).
Digging into this a little more, I think the following commit actually
originally broke things:
3016c4cee45635afa9c25b21c2f5b014d80d1b6d
In that commit, the "netbuf" data type used as an argument
for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far
as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the
following
code in that function would not have compiled after that:
==
...
if(rc == HASHTABLE_SUCCESS)
{
handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata;
if(h->fh_len < p_out_fsal_handle->maxlen)
{
p_out_fsal_handle->len = h->fh_len;
memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len);
memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len);
rc = HANDLEMAP_SUCCESS;
}
...
==
After that change, there are various iterations of renaming the
"p_out_fsal_handle" parameter to other things, up until the commit that you
mentioned that fixed the compile issues
(823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code
having "p_out_fsal_handle" renamed to the value actually being passed in,
and the "maxlen" member replaced with "len".
IIRC, all the handle mapping code *actually* passes around a buffer
that
is large enough for the largest FH4 plus some padding. So the “resizing”
done in handle_to_wire that needs to be undone in wire_to_host can actually
happen in-place but the len field doesn’t know that.
So, any suggestions on the most reliable way to determine "maxlen", again?
Should there be a #define constant somewhere that sets it, or is it
something that should be more dynamically computed based on the data passed
in - maybe the function that does the padding? Or should the "if ()"
statement be removed entirely, and just assume that the length is always
small enough (that seems like a bad option)?
Seems like there probably aren't a lot of users of the handle mapping code
:-D.
-Nick
7:04 a.m.
On Sat, Jul 3, 2021 at 10:08 PM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
NFS4_FHSIZE))
ReturnCode(ERR_FSAL_INVAL, 0);
fh4.nfs_fh4_val = hdl_desc->start;
fh4.nfs_fh4_len = hdl_desc->len;
#ifdef _HANDLE_MAPPING
if(hdl_desc->len == sizeof(nfs23_map_handle_t)) {
nfs23_map_handle_t *h23 =
(nfs23_map_handle_t*)hdl_desc->start;
if(h23->type == PXY_HANDLE_MAPPED) {
struct netbuf fh = {
.maxlen = sizeof(fh_data),
.buf = fh_data
};
struct pxy_handle_blob *blob;
==
From this I gather that the size of maxlen for that netbuf is the size of
the data going into the buffer; however, that member is allocated as
NFS4_FHSIZE + 2.
So, maybe a constant, something like:
#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
Or maybe there's a better way?
-Nick
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos
<boulos(a)google.com> wrote:
> Alright, looking through some git blame -w, I think this broke and then
> was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the
> code compile, but crucially the buffer code no longer had a “max len” just
> a length.
> I think this broke the “clever” reuse of the input
buffdesc as the
> argument to GetFH (which has been there since 2012 from 878f45493e).
> The input when mapped is a 16-byte thingy (sizeof
nfs23_map_handle_t),
> and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume
> the code when written had a FH4_SIZE as it’s max length, rather than the
> fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there
> was a length separate from max length at the time or something).
Digging into this a little more, I
think the following commit actually
originally broke things:
3016c4cee45635afa9c25b21c2f5b014d80d1b6d
In that commit, the "netbuf" data type used as an argument
for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far
as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the
following
code in that function would not have compiled after that:
==
...
if(rc == HASHTABLE_SUCCESS)
{
handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata;
if(h->fh_len < p_out_fsal_handle->maxlen)
{
p_out_fsal_handle->len = h->fh_len;
memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len);
memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len);
rc = HANDLEMAP_SUCCESS;
}
...
==
After that change, there are various iterations of renaming the
"p_out_fsal_handle" parameter to other things, up until the commit that you
mentioned that fixed the compile issues
(823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code
having "p_out_fsal_handle" renamed to the value actually being passed in,
and the "maxlen" member replaced with "len".
And, just another note on this - in the commit prior to 3016c4cee4, I
believe the netbuf that gets passed around is allocated in this block of
code:
==
fsal_status_t
pxy_create_handle(struct fsal_export *exp_hdl,
struct fsal_handle_desc *hdl_desc,
struct fsal_obj_handle **handle)
{
fsal_status_t st;
nfs_fh4 fh4;
fsal_attrib_list_t attr;
struct pxy_obj_handle *ph;
#ifdef _HANDLE_MAPPING
char fh_data[NFS4_FHSIZE+2];
#endif
if(!exp_hdl || !hdl_desc || !handle || (hdl_desc->len
7:30 a.m.
On Sun, Jul 4, 2021 at 10:04 AM Nick Couchman <nick.e.couchman(a)gmail.com wrote:
On Sat, Jul 3, 2021 at 10:08 PM Nick Couchman
<nick.e.couchman(a)gmail.com wrote:
> On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos(a)google.com > wrote:
>> Alright, looking through some git blame -w, I think
this broke and then
>> was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the
>> code compile, but crucially the buffer code no longer had a “max len” just
>> a length.
> >> I think this broke the “clever” reuse of the input
buffdesc as the
>> argument to GetFH (which has been there since 2012 from 878f45493e).
> >> The input when mapped is a 16-byte thingy (sizeof
nfs23_map_handle_t),
>> and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume
>> the code when written had a FH4_SIZE as it’s max length, rather than the
>> fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there
>> was a length separate from max length at the time or something).
> > > Digging into this a little
more, I think the following commit actually
> originally broke things:
> 3016c4cee45635afa9c25b21c2f5b014d80d1b6d
> In that commit, the "netbuf" data type used as
an argument
> for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far
> as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the
following
> code in that function would not have compiled after that:
> ==
> ...
> if(rc == HASHTABLE_SUCCESS)
> {
> handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata;
> if(h->fh_len < p_out_fsal_handle->maxlen)
> {
> p_out_fsal_handle->len = h->fh_len;
> memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len);
> memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len);
> rc = HANDLEMAP_SUCCESS;
> }
> ...
> ==
> After that change, there are various iterations of
renaming the
> "p_out_fsal_handle" parameter to other things, up until the commit that
you
> mentioned that fixed the compile issues
> (823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code
> having "p_out_fsal_handle" renamed to the value actually being passed in,
> and the "maxlen" member replaced with "len".
And, just another note on this - in the commit prior to 3016c4cee4, I
believe the netbuf that gets passed around is allocated in this block of
code:
==
fsal_status_t
pxy_create_handle(struct fsal_export *exp_hdl,
struct fsal_handle_desc *hdl_desc,
struct fsal_obj_handle **handle)
{
fsal_status_t st;
nfs_fh4 fh4;
fsal_attrib_list_t attr;
struct pxy_obj_handle *ph;
#ifdef _HANDLE_MAPPING
char fh_data[NFS4_FHSIZE+2];
#endif
if(!exp_hdl || !hdl_desc || !handle || (hdl_desc->len NFS4_FHSIZE))
ReturnCode(ERR_FSAL_INVAL, 0);
fh4.nfs_fh4_val = hdl_desc->start;
fh4.nfs_fh4_len = hdl_desc->len;
#ifdef _HANDLE_MAPPING
if(hdl_desc->len == sizeof(nfs23_map_handle_t)) {
nfs23_map_handle_t *h23 =
(nfs23_map_handle_t*)hdl_desc->start;
if(h23->type == PXY_HANDLE_MAPPED) {
struct netbuf fh = {
.maxlen = sizeof(fh_data),
.buf = fh_data
};
struct pxy_handle_blob *blob;
==
From this I gather that the size of maxlen for that netbuf is the size of
the data going into the buffer; however, that member is allocated as
NFS4_FHSIZE + 2.
So, maybe a constant, something like:
#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
The following patch seems to fix things:
==
diff -Naur
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
---
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
2021-07-02 18:12:40.000000000 -0400
+++
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
2021-07-04 08:25:04.398912776 -0400
@@ -344,7 +344,7 @@
if (rc == HASHTABLE_SUCCESS) {
handle_pool_entry_t *h = (handle_pool_entry_t *) buffval.addr;
- if (h->fh_len < fsal_handle->len) {
+ if (h->fh_len < PROXYV4_HANDLE_MAXLEN) {
fsal_handle->len = h->fh_len;
memcpy(fsal_handle->addr, h->fh_data, h->fh_len);
rc = HANDLEMAP_SUCCESS;
diff -Naur
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
---
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
2021-07-02 18:12:40.000000000 -0400
+++
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
2021-07-04 08:24:44.496077148 -0400
@@ -81,6 +81,9 @@
#define HANDLEMAP_HASHTABLE_ERROR 7
#define HANDLEMAP_EXISTS 8
+/* Maximum file handle length */
+#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
+
int HandleMap_Init(const handle_map_param_t *p_param);
int HandleMap_GetFH(const nfs23_map_handle_t *, struct gsh_buffdesc *);
==
No extensive testing done, yet, but I'm happy to submit a pull request, if
this makes sense?
Or maybe there's a better way?
Or maybe there's still a better way??
-Nick
11:10 a.m.
I had meant to add lots of comments to the FSAL API header about how all
this handle-to-wire / wire-to-host / handle-to-key stuff works…
(particularly the memory lifetimes and ownership) because I struggled with
it when implementing the v3 proxy.
The giant comment in fsal_api.h is not bad, but the real action is in
nfs_filehandle_mgmt.c (there are other callers of wire_to_host, including
the mdcache just passing it through to the underlying FSAL, and an NFSv4
digest variant, but HandleMap_GetFH is being called on the "get me the v3
handle" path):
https://github.com/nfs-ganesha/nfs-ganesha/blob/2ce0989828a1d3fc1f5dc6164...
which shows that the buffer passed into wire_to_host is NFS3_FHSIZE long
(which makes sense, the function is trying to compute an fh3 as output).
The comment there incorrectly says "no larger than NFS4_FHSIZE" but it
should say NFS3_FHSIZE. (How'd you get NFS4_FHSIZE+2?)
We should still feel bad about that FIXME in FhandleToCache and the reuse
of the buffer as input and output (the size of the buffer doesn't have much
to do with the actual fs len).
On Sun, Jul 4, 2021 at 07:30 Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
On Sun, Jul 4, 2021 at 10:04 AM Nick Couchman
<nick.e.couchman(a)gmail.com>
wrote:
> On Sat, Jul 3, 2021 at 10:08 PM Nick Couchman <nick.e.couchman(a)gmail.com>
> wrote:
>
>> On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos(a)google.com>
>> wrote:
>>
>>> Alright, looking through some git blame -w, I think this broke and then
>>> was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the
>>> code compile, but crucially the buffer code no longer had a “max len” just
>>> a length.
>>>
>>> I think this broke the “clever” reuse of the input buffdesc as the
>>> argument to GetFH (which has been there since 2012 from 878f45493e).
>>>
>>> The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t),
>>> and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume
>>> the code when written had a FH4_SIZE as it’s max length, rather than the
>>> fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there
>>> was a length separate from max length at the time or something).
>>>
>>>
>> Digging into this a little more, I think the following commit actually
>> originally broke things:
>>
>> 3016c4cee45635afa9c25b21c2f5b014d80d1b6d
>>
>> In that commit, the "netbuf" data type used as an argument
>> for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As
far
>> as I can tell, "gsh_buffdesc" never had a "maxlen" member, so
the following
>> code in that function would not have compiled after that:
>>
>> ==
>> ...
>> if(rc == HASHTABLE_SUCCESS)
>> {
>> handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata;
>> if(h->fh_len < p_out_fsal_handle->maxlen)
>> {
>> p_out_fsal_handle->len = h->fh_len;
>> memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len);
>> memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len);
>> rc = HANDLEMAP_SUCCESS;
>> }
>> ...
>> ==
>>
>> After that change, there are various iterations of renaming the
>> "p_out_fsal_handle" parameter to other things, up until the commit that
you
>> mentioned that fixed the compile issues
>> (823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code
>> having "p_out_fsal_handle" renamed to the value actually being passed
in,
>> and the "maxlen" member replaced with "len".
>>
>>
>
> And, just another note on this - in the commit prior to 3016c4cee4, I
> believe the netbuf that gets passed around is allocated in this block of
> code:
>
> ==
> fsal_status_t
> pxy_create_handle(struct fsal_export *exp_hdl,
> struct fsal_handle_desc *hdl_desc,
> struct fsal_obj_handle **handle)
> {
> fsal_status_t st;
> nfs_fh4 fh4;
> fsal_attrib_list_t attr;
> struct pxy_obj_handle *ph;
> #ifdef _HANDLE_MAPPING
> char fh_data[NFS4_FHSIZE+2];
> #endif
>
> if(!exp_hdl || !hdl_desc || !handle || (hdl_desc->len >
> NFS4_FHSIZE))
> ReturnCode(ERR_FSAL_INVAL, 0);
>
> fh4.nfs_fh4_val = hdl_desc->start;
> fh4.nfs_fh4_len = hdl_desc->len;
>
> #ifdef _HANDLE_MAPPING
> if(hdl_desc->len == sizeof(nfs23_map_handle_t)) {
> nfs23_map_handle_t *h23 =
> (nfs23_map_handle_t*)hdl_desc->start;
> if(h23->type == PXY_HANDLE_MAPPED) {
> struct netbuf fh = {
> .maxlen = sizeof(fh_data),
> .buf = fh_data
> };
> struct pxy_handle_blob *blob;
> ==
>
> From this I gather that the size of maxlen for that netbuf is the size of
> the data going into the buffer; however, that member is allocated as
> NFS4_FHSIZE + 2.
>
> So, maybe a constant, something like:
>
> #define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
>
The following patch seems to fix things:
==
diff -Naur
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
---
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
2021-07-02 18:12:40.000000000 -0400
+++
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c
2021-07-04 08:25:04.398912776 -0400
@@ -344,7 +344,7 @@
if (rc == HASHTABLE_SUCCESS) {
handle_pool_entry_t *h = (handle_pool_entry_t *) buffval.addr;
- if (h->fh_len < fsal_handle->len) {
+ if (h->fh_len < PROXYV4_HANDLE_MAXLEN) {
fsal_handle->len = h->fh_len;
memcpy(fsal_handle->addr, h->fh_data, h->fh_len);
rc = HANDLEMAP_SUCCESS;
diff -Naur
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
---
nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
2021-07-02 18:12:40.000000000 -0400
+++
nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h
2021-07-04 08:24:44.496077148 -0400
@@ -81,6 +81,9 @@
#define HANDLEMAP_HASHTABLE_ERROR 7
#define HANDLEMAP_EXISTS 8
+/* Maximum file handle length */
+#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
+
int HandleMap_Init(const handle_map_param_t *p_param);
int HandleMap_GetFH(const nfs23_map_handle_t *, struct gsh_buffdesc *);
==
No extensive testing done, yet, but I'm happy to submit a pull request, if
this makes sense?
>
> Or maybe there's a better way?
>
Or maybe there's still a better way??
-Nick
>
11:46 a.m.
On Fri, Jul 9, 2021 at 2:09 PM Solomon Boulos <boulos(a)waymo.com> wrote:
I had meant to add lots of comments to the FSAL API header about how
all
this handle-to-wire / wire-to-host / handle-to-key stuff works…
(particularly the memory lifetimes and ownership) because I struggled with
it when implementing the v3 proxy.
The giant comment in fsal_api.h is not bad, but the real action is in
nfs_filehandle_mgmt.c (there are other callers of wire_to_host, including
the mdcache just passing it through to the underlying FSAL, and an NFSv4
digest variant, but HandleMap_GetFH is being called on the "get me the v3
handle" path):
https://github.com/nfs-ganesha/nfs-ganesha/blob/2ce0989828a1d3fc1f5dc6164...
which shows that the buffer passed into wire_to_host is NFS3_FHSIZE long
(which makes sense, the function is trying to compute an fh3 as output).
The comment there incorrectly says "no larger than NFS4_FHSIZE" but it
should say NFS3_FHSIZE. (How'd you get NFS4_FHSIZE+2?)
Prior to commit 3016c4cee4, when the netbuf was being used, the fh_data
buffer was allocated with that size, and then the netbuf's "maxlen"
property was set to sizeof(fh_data), effectively setting the max length to
NFS4_FHSIZE + 2. I don't really know the rationale for that, either then or
now, I'm basically just restoring the lengths that were there before. I'm
certainly not saying this is the correct way to do it, nor that it is safe
- just that it appears to work.
I think Frank is correct, though - it's worth trying to figure out what
values we should be using and make sure things are sane - just going with
it because it appears to work isn't necessarily a good idea.
-Nick
12:24 p.m.
Can you do mount -vvv to get some more info on the client? The ganesha side
doesn't *seem* to be sad.
It's getting and handling NULL, MNT_NULL, MNT_MNT, lookup of /azure-file
(lookup of the root filehandle), and then responding to an FSINFO request.
If the mount -vvv isn't revealing, we might need a wireshark dump of the
NFS traffic. For v3 I usually do:
sudo tshark -J "mount,rpc,nfs,nlm,portmap" -O
"mount,rpc,nfs,nlm,portmap"
host <IP>
On Mon, Jun 28, 2021 at 12:03 PM Nick Couchman <nick.e.couchman(a)gmail.com>
wrote:
> I'm sure I'm missing something simple, here, but still not sure why this
> is causing a segfault, unless the presence of the mutex lock/unlock does
> something to mess with it?
>
>
Well, I'm feeling a bit silly about this right now, and have resolved the
segfault/core dump issue. The issue was that the directory for handle
mapping didn't exist, so the HandleMap_Init function call would exit out
without actually allocating the structures for digest and handle pools, but
the proxy module would continue to try loading and try to use the handle
map. Apparently I didn't catch this in the log file :-/.
That said, I'm on to the next issue, which is that, with handle mapping
enabled, finally, and the nordirplus mount option (to get around the
readdirplus issue), I now get a "Stale File Handle" error when trying to
mount:
# mount -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure
mount.nfs: Stale file handle
Current log file, during the mount attempt, is here:
https://pastebin.com/wKhLENFk
-Nick
>
12:42 p.m.
On Mon, Jun 28, 2021 at 3:25 PM Solomon Boulos <boulos(a)waymo.com> wrote:
Can you do mount -vvv to get some more info on the client? The
ganesha
side doesn't *seem* to be sad.
Sure, here you go:
# mount -vvv -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure
mount.nfs: timeout set for Mon Jun 28 15:28:50 2021
mount.nfs: trying text-based options 'vers=3,nordirplus,addr=10.1.2.4'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.1.2.4 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.1.2.4 prog 100005 vers 3 prot TCP port 20048
mount.nfs: mount(2): Stale file handle
mount.nfs: trying text-based options 'vers=3,nordirplus,addr=10.1.2.4'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.1.2.4 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.1.2.4 prog 100005 vers 3 prot TCP port 20048
mount.nfs: mount(2): Stale file handle
mount.nfs: trying text-based options 'vers=3,nordirplus,addr=10.1.2.4'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: trying 10.1.2.4 prog 100003 vers 3 prot TCP port 2049
mount.nfs: prog 100005, trying vers=3, prot=17
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100005, trying vers=3, prot=6
mount.nfs: trying 10.1.2.4 prog 100005 vers 3 prot TCP port 20048
mount.nfs: mount(2): Stale file handle
mount.nfs: Stale file handle
It's getting and handling NULL, MNT_NULL, MNT_MNT, lookup of /azure-file
(lookup of the root filehandle), and then responding to an FSINFO request.
If the mount -vvv isn't revealing, we might need a wireshark dump of the
NFS traffic. For v3 I usually do:
sudo tshark -J "mount,rpc,nfs,nlm,portmap" -O
"mount,rpc,nfs,nlm,portmap"
host <IP
Here's the pastebin for that:
https://pastebin.com/Hp4qgZdk
-Nick
1021
days inactive
1035
days old
33 comments
5 participants
participants (5)
-
Frank Filz -
Nick Couchman -
Solomon Boulos -
Solomon Boulos -
Todd Pfaff