I'm using Ganesha NFS with the Proxy FSAL. It seems to work fine as long as I'm going between NFS clients and servers of the same version (NFSv4.1 client -> NFSv4.1 server). However, I have a situation where I'm trying to proxy a NFSv4.1 service (Azure File Share NFS) for clients that do not support NFSv4. When I do this, the "mount" command on the NFSv3 clients works fine, but as soon as I try to do any data operations, I get I/O errors: # mount -t nfs -o vers=3 10.11.12.13:/azure-nfs-file /mnt/azure # ls -l /mnt/azure ls: reading directory '/mnt/azure': Remote I/O error total 0 I've posted my (sanitized) configuration below. My questions are: 1) Is it possible to use Ganesha to proxy this way - allowing clients to access a NFS server that differs in version, or is this not supported? 2) Am I missing something within my Ganesha config to enable this? I can also provide the debug logging, if that's helpful. Thanks - Nick ==ganesha.conf== NFS_CORE_PARAM { mount_path_pseudo = true; Protocols = 3,4; MNT_Port = 20048; } LOG { Default_Log_Level = INFO; Components { FSAL = FULL_DEBUG; } } EXPORT_DEFAULTS { Access_Type = RW; } ## Azure NFS Share EXPORT { Export_id = 902; Path = "/stgaccount/azure-file"; Pseudo = "/azure-file"; Access_Type = RW; Squash = no_root_squash; Sectype = sys; FSAL { Name = proxy; Srv_Addr = 10.1.2.3; Enable_Handle_Mapping = TRUE; HandleMap_DB_Dir = "/var/ganesha/handledb/902"; HandleMap_Tmp_Dir = "/run/ganesha/tmp/902"; HandleMap_DB_Count = 8; } }
I haven’t done enough with Proxy… There is a handle mapping option that you may need to use if the proxied handles are too big for NFSv3. Frank From: Nick Couchman [mailto:nick.e.couchman@gmail.com] Sent: Friday, June 25, 2021 4:38 PM To: support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Proxy FSAL - NFS4 -> 3 I'm using Ganesha NFS with the Proxy FSAL. It seems to work fine as long as I'm going between NFS clients and servers of the same version (NFSv4.1 client -> NFSv4.1 server). However, I have a situation where I'm trying to proxy a NFSv4.1 service (Azure File Share NFS) for clients that do not support NFSv4. When I do this, the "mount" command on the NFSv3 clients works fine, but as soon as I try to do any data operations, I get I/O errors: # mount -t nfs -o vers=3 10.11.12.13:/azure-nfs-file /mnt/azure # ls -l /mnt/azure ls: reading directory '/mnt/azure': Remote I/O error total 0 I've posted my (sanitized) configuration below. My questions are: 1) Is it possible to use Ganesha to proxy this way - allowing clients to access a NFS server that differs in version, or is this not supported? 2) Am I missing something within my Ganesha config to enable this? I can also provide the debug logging, if that's helpful. Thanks - Nick ==ganesha.conf== NFS_CORE_PARAM { mount_path_pseudo = true; Protocols = 3,4; MNT_Port = 20048; } LOG { Default_Log_Level = INFO; Components { FSAL = FULL_DEBUG; } } EXPORT_DEFAULTS { Access_Type = RW; } ## Azure NFS Share EXPORT { Export_id = 902; Path = "/stgaccount/azure-file"; Pseudo = "/azure-file"; Access_Type = RW; Squash = no_root_squash; Sectype = sys; FSAL { Name = proxy; Srv_Addr = 10.1.2.3; Enable_Handle_Mapping = TRUE; HandleMap_DB_Dir = "/var/ganesha/handledb/902"; HandleMap_Tmp_Dir = "/run/ganesha/tmp/902"; HandleMap_DB_Count = 8; } }
It should be! The proxy code just acts like either a V4 client (ProxyV4 in the current tree) or V3 client (ProxyV3). Ganesha itself handles the clients and whatever they want to speak (v3, v4, 9p even I think, etc.). Your use of Name = proxy, implies you're on an older branch perhaps, and I feel like I've had some bugfixes in the Proxy V4 code (in particular, I know for my ProxyV3 I had to fixup the handling of path vs pseudo... I don't recall if ProxyV4 had the same problem or not). I don't have any experience with the handle mapping code (though I've certainly looked at it). Is it strictly required for your test? Either way, please provide some debug logs. You may want to add debug logging to NFS_PROTO or whatever the name is as well. (And if I finally get around to fixing my IRC setup, you can find folks in #ganesha, too for more live debugging). On Fri, Jun 25, 2021 at 4:38 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
I'm using Ganesha NFS with the Proxy FSAL. It seems to work fine as long as I'm going between NFS clients and servers of the same version (NFSv4.1 client -> NFSv4.1 server). However, I have a situation where I'm trying to proxy a NFSv4.1 service (Azure File Share NFS) for clients that do not support NFSv4. When I do this, the "mount" command on the NFSv3 clients works fine, but as soon as I try to do any data operations, I get I/O errors:
# mount -t nfs -o vers=3 10.11.12.13:/azure-nfs-file /mnt/azure # ls -l /mnt/azure ls: reading directory '/mnt/azure': Remote I/O error total 0
I've posted my (sanitized) configuration below. My questions are: 1) Is it possible to use Ganesha to proxy this way - allowing clients to access a NFS server that differs in version, or is this not supported? 2) Am I missing something within my Ganesha config to enable this?
I can also provide the debug logging, if that's helpful.
Thanks - Nick
==ganesha.conf== NFS_CORE_PARAM { mount_path_pseudo = true; Protocols = 3,4; MNT_Port = 20048; }
LOG { Default_Log_Level = INFO; Components { FSAL = FULL_DEBUG; } }
EXPORT_DEFAULTS { Access_Type = RW; }
## Azure NFS Share EXPORT { Export_id = 902; Path = "/stgaccount/azure-file"; Pseudo = "/azure-file"; Access_Type = RW; Squash = no_root_squash; Sectype = sys; FSAL { Name = proxy; Srv_Addr = 10.1.2.3; Enable_Handle_Mapping = TRUE; HandleMap_DB_Dir = "/var/ganesha/handledb/902"; HandleMap_Tmp_Dir = "/run/ganesha/tmp/902"; HandleMap_DB_Count = 8; } } _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
On Fri, Jun 25, 2021 at 7:53 PM Solomon Boulos <boulos@waymo.com> wrote:
It should be!
The proxy code just acts like either a V4 client (ProxyV4 in the current tree) or V3 client (ProxyV3). Ganesha itself handles the clients and whatever they want to speak (v3, v4, 9p even I think, etc.).
That's what I thought, so that's good news.
Your use of Name = proxy, implies you're on an older branch perhaps, and I feel like I've had some bugfixes in the Proxy V4 code (in particular, I know for my ProxyV3 I had to fixup the handling of path vs pseudo... I don't recall if ProxyV4 had the same problem or not). I don't have any experience with the handle mapping code (though I've certainly looked at it). Is it strictly required for your test?
Regarding the "Name = proxy" configuration, do you have examples of what I should use for the Proxy V4 code? I had a hard time finding concrete examples for the Proxy code in particular. I'm running Ganesha 3.5 on CentOS 8, out of the SCL Ganesha repo. I've installed the nfs-ganesha and nfs-ganesha-proxy packages (along with vfs and utils). If there's a different version that I should be using, please let me know - I'm not opposed to building from source, either. Regarding the handle mapping, I have tried it both enabled and disabled and the experience is identical. It's enabled in the config that I sent over because that's the last thing I tried. I can disabled it, again, just to make sure it isn't adding any rabbit holes to the debugging process.
Either way, please provide some debug logs. You may want to add debug logging to NFS_PROTO or whatever the name is as well. (And if I finally get around to fixing my IRC setup, you can find folks in #ganesha, too for more live debugging).
Yeah, no problem - here are the logs in a Pastbin: https://pastebin.com/yxVsYfxw I tried adding "NFS_PROTO = FULL_DEBUG;" to the components section of the configuration, but I'm not sure I got that correct. I'll go back and look at the available components and try a few other things - maybe NFS_V4 or something like that? -Nick
On Fri, Jun 25, 2021 at 9:00 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Fri, Jun 25, 2021 at 7:53 PM Solomon Boulos <boulos@waymo.com> wrote:
It should be!
The proxy code just acts like either a V4 client (ProxyV4 in the current tree) or V3 client (ProxyV3). Ganesha itself handles the clients and whatever they want to speak (v3, v4, 9p even I think, etc.).
That's what I thought, so that's good news.
Your use of Name = proxy, implies you're on an older branch perhaps, and I feel like I've had some bugfixes in the Proxy V4 code (in particular, I know for my ProxyV3 I had to fixup the handling of path vs pseudo... I don't recall if ProxyV4 had the same problem or not). I don't have any experience with the handle mapping code (though I've certainly looked at it). Is it strictly required for your test?
Regarding the "Name = proxy" configuration, do you have examples of what I should use for the Proxy V4 code? I had a hard time finding concrete examples for the Proxy code in particular.
I'm running Ganesha 3.5 on CentOS 8, out of the SCL Ganesha repo. I've installed the nfs-ganesha and nfs-ganesha-proxy packages (along with vfs and utils). If there's a different version that I should be using, please let me know - I'm not opposed to building from source, either.
Regarding the handle mapping, I have tried it both enabled and disabled and the experience is identical. It's enabled in the config that I sent over because that's the last thing I tried. I can disabled it, again, just to make sure it isn't adding any rabbit holes to the debugging process.
Either way, please provide some debug logs. You may want to add debug logging to NFS_PROTO or whatever the name is as well. (And if I finally get around to fixing my IRC setup, you can find folks in #ganesha, too for more live debugging).
Yeah, no problem - here are the logs in a Pastbin:
I tried adding "NFS_PROTO = FULL_DEBUG;" to the components section of the configuration, but I'm not sure I got that correct. I'll go back and look at the available components and try a few other things - maybe NFS_V4 or something like that?
Okay, managed to get NFSPROTO and NFS_V4 debugging enabled - new logs, here: https://pastebin.com/1dcKR9HR -Nick
Nick, I've been using the Proxy FSAL for the past two years and have had much better results since moving to the V4.dev branch about a year ago, particularly after Solomon Boulos' improvements to the Proxy FSAL. I don't claim to be using it exactly as you've described but I believe it may be worth your while to try the V4.dev branch. cheers, Todd
On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff@rhpcs.mcmaster.ca> wrote:
Nick,
I've been using the Proxy FSAL for the past two years and have had much better results since moving to the V4.dev branch about a year ago, particularly after Solomon Boulos' improvements to the Proxy FSAL. I don't claim to be using it exactly as you've described but I believe it may be worth your while to try the V4.dev branch.
Todd, Thanks for the hints - I'll give it a shot - either see if packages are available or just compile it myself. -Nick
On Sat, Jun 26, 2021 at 2:04 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff@rhpcs.mcmaster.ca> wrote:
Nick,
I've been using the Proxy FSAL for the past two years and have had much better results since moving to the V4.dev branch about a year ago, particularly after Solomon Boulos' improvements to the Proxy FSAL. I don't claim to be using it exactly as you've described but I believe it may be worth your while to try the V4.dev branch.
Todd, Thanks for the hints - I'll give it a shot - either see if packages are available or just compile it myself.
No joy after switching to the V4 Dev branch - behavior seems identical. There's definitely something in the V3 -> V4 proxy that just doesn't work, as mounting the proxied NFS share as NFSv4 from a client works fine, but NFSv3 results in the "Remote I/O Error" behavior. Here are the things I've tried so far: * Both versions 3.5 (Current Stable) and V4 dev (latest release). * Various combinations of Privileged Port enabled/disabled on both the Export and the Proxy FSAL. * mount_path_pseudo both enabled and disabled to see if the path translation was causing issues for the V3 client. * Both enabling and disabling Handle Mapping. I'm open to any other suggestions that folks have for me, and am happy to provide any further debug output. -Nick
On Sat, Jun 26, 2021 at 10:14 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jun 26, 2021 at 2:04 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff@rhpcs.mcmaster.ca> wrote:
Nick,
I've been using the Proxy FSAL for the past two years and have had much better results since moving to the V4.dev branch about a year ago, particularly after Solomon Boulos' improvements to the Proxy FSAL. I don't claim to be using it exactly as you've described but I believe it may be worth your while to try the V4.dev branch.
Todd, Thanks for the hints - I'll give it a shot - either see if packages are available or just compile it myself.
No joy after switching to the V4 Dev branch - behavior seems identical. There's definitely something in the V3 -> V4 proxy that just doesn't work, as mounting the proxied NFS share as NFSv4 from a client works fine, but NFSv3 results in the "Remote I/O Error" behavior.
Here are the things I've tried so far: * Both versions 3.5 (Current Stable) and V4 dev (latest release). * Various combinations of Privileged Port enabled/disabled on both the Export and the Proxy FSAL. * mount_path_pseudo both enabled and disabled to see if the path translation was causing issues for the V3 client. * Both enabling and disabling Handle Mapping.
I'm open to any other suggestions that folks have for me, and am happy to provide any further debug output.
Okay, I actually made a little bit of progress in figuring out what's going on, here, with the help of tcpdump. There are two issues: * Looks like the Ganesha NFSv3 server may not support the "readdirplus" command? Output from "tcpdump" includes the following: 22:24:49.212153 IP 10.1.2.3.netrcs > 10.1.2.4.nfs: Flags [P.], seq 648:832, ack 441, win 337, options [nop,nop,TS val 861164029 ecr 2104677449], length 184: NFS request xid 3221395829 180 readdirplus fh Unknown/430003863737072F240000010000000000000001000000000000000380E6C8CC 2048 bytes @ 0 22:24:49.261230 IP 10.1.2.4.nfs > 10.1.2.3.netrcs: Flags [P.], seq 441:561, ack 832, win 428, options [nop,nop,TS val 2104677499 ecr 861164029], length 120: NFS reply xid 3221395829 reply ok 116 readdirplus ERROR: Unspecified error on server I can get around this issue by adding the "nordirplus" NFS mount option. I then start to get some part of the directory listing, but with lots of errors. * After getting past the readdirplus issue, I still get errors, and this appears to be due to file handle issues: 22:48:54.686280 IP 10.1.2.4.nfs > 10.1.2.3.phonebook: Flags [P.], seq 4565:4601, ack 5925, win 520, options [nop,nop,TS val 2106122935 ecr 862609458], length 36: NFS reply xid 1689362215 reply ok 32 lookup ERROR: Illegal NFS file handle I figured out that the file handle issues are because none of the versions of Ganesha I've been using are actually compiled with the file handle mapping enabled. This includes the version in the CentOS Ganesha SIG (3.5, with the older "Proxy" FSAL) and the version I was building from source via the RPM SPEC file, which was not enabling the support for file handle mapping in Proxy V4. * All that said, when I finally do get a version of V4 Dev built that enables Handle Mapping, it just core dumps: Jun 26 23:54:41 nfs-proxy.example.com systemd-coredump[350511]: Process 350496 (ganesha.nfsd) of user 0 dumped core. Stack trace of thread 350496: #0 0x00007f1f7caa6704 digest_alloc (libfsalproxy_v4.so) #1 0x00007f1f7caa70a2 handle_mapping_hash_add (libfsalproxy_v4.so) #2 0x00007f1f7caa75b2 HandleMap_SetFH (libfsalproxy_v4.so) #3 0x00007f1f7caa51df proxyv4_alloc_handle (libfsalproxy_v4.so) #4 0x00007f1f7caa0883 proxyv4_make_object (libfsalproxy_v4.so) #5 0x00007f1f7caa0f01 proxyv4_lookup_impl (libfsalproxy_v4.so) #6 0x00007f1f7caa5465 proxyv4_lookup_path (libfsalproxy_v4.so) #7 0x00007f1f8968334d mdcache_lookup_path (libganesha_nfsd.so.4) #8 0x00007f1f895fe1d7 init_export_root (libganesha_nfsd.so.4) #9 0x00007f1f895fd8e6 init_export_cb (libganesha_nfsd.so.4) #10 0x00007f1f896164c3 foreach_gsh_export (libganesha_nfsd.so.4) #11 0x00007f1f895fd93c exports_pkginit (libganesha_nfsd.so.4) #12 0x00007f1f89582c62 nfs_Init (libganesha_nfsd.so.4) #13 0x00007f1f89583b2d nfs_start (libganesha_nfsd.so.4) #14 0x00000000004029b9 main (ganesha.nfsd) #15 0x00007f1f86e4a493 __libc_start_main (libc.so.6) #16 0x00000000004018fe _start (ganesha.nfsd) -Nick
Good progress! I’ll try to take a look on Monday. Does the proxy FSAL misrepresent it’s readdirplus support? (Seems possible, it should honor whatever the backend says) Just to confirm, does the backend not also speak V3? (Since you have v3 clients, you could use my simpler v3 proxy in a modern client and not do 3 <=> 4 via Ganesha). On Sat, Jun 26, 2021 at 21:05 Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jun 26, 2021 at 10:14 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jun 26, 2021 at 2:04 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jun 26, 2021 at 12:34 PM Todd Pfaff <pfaff@rhpcs.mcmaster.ca> wrote:
Nick,
I've been using the Proxy FSAL for the past two years and have had much better results since moving to the V4.dev branch about a year ago, particularly after Solomon Boulos' improvements to the Proxy FSAL. I don't claim to be using it exactly as you've described but I believe it may be worth your while to try the V4.dev branch.
Todd, Thanks for the hints - I'll give it a shot - either see if packages are available or just compile it myself.
No joy after switching to the V4 Dev branch - behavior seems identical. There's definitely something in the V3 -> V4 proxy that just doesn't work, as mounting the proxied NFS share as NFSv4 from a client works fine, but NFSv3 results in the "Remote I/O Error" behavior.
Here are the things I've tried so far: * Both versions 3.5 (Current Stable) and V4 dev (latest release). * Various combinations of Privileged Port enabled/disabled on both the Export and the Proxy FSAL. * mount_path_pseudo both enabled and disabled to see if the path translation was causing issues for the V3 client. * Both enabling and disabling Handle Mapping.
I'm open to any other suggestions that folks have for me, and am happy to provide any further debug output.
Okay, I actually made a little bit of progress in figuring out what's going on, here, with the help of tcpdump. There are two issues:
* Looks like the Ganesha NFSv3 server may not support the "readdirplus" command? Output from "tcpdump" includes the following: 22:24:49.212153 IP 10.1.2.3.netrcs > 10.1.2.4.nfs: Flags [P.], seq 648:832, ack 441, win 337, options [nop,nop,TS val 861164029 ecr 2104677449], length 184: NFS request xid 3221395829 180 readdirplus fh Unknown/430003863737072F240000010000000000000001000000000000000380E6C8CC 2048 bytes @ 0 22:24:49.261230 IP 10.1.2.4.nfs > 10.1.2.3.netrcs: Flags [P.], seq 441:561, ack 832, win 428, options [nop,nop,TS val 2104677499 ecr 861164029], length 120: NFS reply xid 3221395829 reply ok 116 readdirplus ERROR: Unspecified error on server
I can get around this issue by adding the "nordirplus" NFS mount option. I then start to get some part of the directory listing, but with lots of errors.
* After getting past the readdirplus issue, I still get errors, and this appears to be due to file handle issues:
22:48:54.686280 IP 10.1.2.4.nfs > 10.1.2.3.phonebook: Flags [P.], seq 4565:4601, ack 5925, win 520, options [nop,nop,TS val 2106122935 ecr 862609458], length 36: NFS reply xid 1689362215 reply ok 32 lookup ERROR: Illegal NFS file handle
I figured out that the file handle issues are because none of the versions of Ganesha I've been using are actually compiled with the file handle mapping enabled. This includes the version in the CentOS Ganesha SIG (3.5, with the older "Proxy" FSAL) and the version I was building from source via the RPM SPEC file, which was not enabling the support for file handle mapping in Proxy V4.
* All that said, when I finally do get a version of V4 Dev built that enables Handle Mapping, it just core dumps:
Jun 26 23:54:41 nfs-proxy.example.com systemd-coredump[350511]: Process 350496 (ganesha.nfsd) of user 0 dumped core.
Stack trace of thread 350496: #0 0x00007f1f7caa6704 digest_alloc (libfsalproxy_v4.so) #1 0x00007f1f7caa70a2 handle_mapping_hash_add (libfsalproxy_v4.so) #2 0x00007f1f7caa75b2 HandleMap_SetFH (libfsalproxy_v4.so) #3 0x00007f1f7caa51df proxyv4_alloc_handle (libfsalproxy_v4.so) #4 0x00007f1f7caa0883 proxyv4_make_object (libfsalproxy_v4.so) #5 0x00007f1f7caa0f01 proxyv4_lookup_impl (libfsalproxy_v4.so) #6 0x00007f1f7caa5465 proxyv4_lookup_path (libfsalproxy_v4.so) #7 0x00007f1f8968334d mdcache_lookup_path (libganesha_nfsd.so.4) #8 0x00007f1f895fe1d7 init_export_root (libganesha_nfsd.so.4) #9 0x00007f1f895fd8e6 init_export_cb (libganesha_nfsd.so.4) #10 0x00007f1f896164c3 foreach_gsh_export (libganesha_nfsd.so.4) #11 0x00007f1f895fd93c exports_pkginit (libganesha_nfsd.so.4) #12 0x00007f1f89582c62 nfs_Init (libganesha_nfsd.so.4) #13 0x00007f1f89583b2d nfs_start (libganesha_nfsd.so.4) #14 0x00000000004029b9 main (ganesha.nfsd) #15 0x00007f1f86e4a493 __libc_start_main (libc.so.6) #16 0x00000000004018fe _start (ganesha.nfsd)
-Nick
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
On Sun, Jun 27, 2021 at 12:55 AM Solomon Boulos <boulos@google.com> wrote:
Good progress! I’ll try to take a look on Monday. Does the proxy FSAL misrepresent it’s readdirplus support? (Seems possible, it should honor whatever the backend says)
That would be great, I'd definitely appreciate any insight. Not sure about the FSAL module and its representation of readdirplus support, but I can dig through the code and take a look. That one I can work around - the Core Dump when the Proxy v4 module has Handle Mapping enabled is a bit more problematic. The function it's referencing at the top - digest_alloc - is simple enough, not really much I can see that would cause a core dump, there.
Just to confirm, does the backend not also speak V3? (Since you have v3 clients, you could use my simpler v3 proxy in a modern client and not do 3 <=> 4 via Ganesha).
Correct - it's most of the reason why I'm trying to use Ganesha to proxy at this point, because the back-end (Azure Storage Account with NFS share) only speaks NFS v4.1, and the systems we need to mount it to (primarily AIX 7.2) only support NFS v3 clients. So, my entire goal here is to use Ganesha to translate between the two. A side benefit is that I can centralize a bunch of mounts on a single server. -Nick
On Sun, Jun 27, 2021 at 10:46 AM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sun, Jun 27, 2021 at 12:55 AM Solomon Boulos <boulos@google.com> wrote:
Good progress! I’ll try to take a look on Monday. Does the proxy FSAL misrepresent it’s readdirplus support? (Seems possible, it should honor whatever the backend says)
That would be great, I'd definitely appreciate any insight. Not sure about the FSAL module and its representation of readdirplus support, but I can dig through the code and take a look. That one I can work around - the Core Dump when the Proxy v4 module has Handle Mapping enabled is a bit more problematic. The function it's referencing at the top - digest_alloc - is simple enough, not really much I can see that would cause a core dump, there.
Regarding the segfault/core dump, I've got the back trace, here: (gdb) bt #0 0x00007f4abbea6704 in digest_alloc () at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:68 #1 0x00007f4abbea70a2 in handle_mapping_hash_add (p_hash=0x0, object_id=0, handle_hash=2762487084, data=0x7f4abe858432, datalen=29) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:193 #2 0x00007f4abbea75b2 in HandleMap_SetFH (p_in_nfs23_digest=0x7f4abe858420, data=0x7f4abe858432, len=29) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:373 #3 0x00007f4abbea51df in proxyv4_alloc_handle (exp=0x7f4abe9fc300, fh=0x7ffd7e7a0a40, obj_attributes=0x7ffd7e7a0ae0, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:2997 #4 0x00007f4abbea0883 in proxyv4_make_object (export=0x7f4abe9fc300, obj_attributes=0x7ffd7e7a0ae0, fh=0x7ffd7e7a0a40, handle=0x7ffd7e7a1038, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1399 #5 0x00007f4abbea0f01 in proxyv4_lookup_impl (parent=0x0, export=0x7f4abe9fc300, cred=0x7ffd7e7a1258, path=0x7f4abe813171 "DataBoxDisk1", handle=0x7ffd7e7a1038, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1545 #6 0x00007f4abbea5465 in proxyv4_lookup_path (exp_hdl=0x7f4abe9fc300, path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a11b0, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:3068 #7 0x00007f4ac8a0b34d in mdcache_lookup_path (exp_hdl=0x7f4abe82b400, path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a1338, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1564 #8 0x00007f4ac89861d7 in init_export_root (export=0x7f4abe877cc8) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2619 #9 0x00007f4ac89858e6 in init_export_cb (exp=0x7f4abe877cc8, state=0x7ffd7e7a1420) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2458 #10 0x00007f4ac899e4c3 in foreach_gsh_export (cb=0x7f4ac89858c2 <init_export_cb>, wrlock=true, state=0x7ffd7e7a1420) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/export_mgr.c:801 #11 0x00007f4ac898593c in exports_pkginit () at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2477 #12 0x00007f4ac890ac62 in nfs_Init (p_start_info=0x6041a8 <my_nfs_start_info>) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:642 #13 0x00007f4ac890bb2d in nfs_start (p_start_info=0x6041a8 <my_nfs_start_info>) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:923 #14 0x00000000004029b9 in main (argc=7, argv=0x7ffd7e7a1838) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_main.c:520 This is quite puzzling, because the line that seems to be causing the segfault is the pool_alloc line below: == pool_t *digest_pool; static pthread_mutex_t digest_pool_mutex = PTHREAD_MUTEX_INITIALIZER; pool_t *handle_pool; static pthread_mutex_t handle_pool_mutex = PTHREAD_MUTEX_INITIALIZER; /* helpers for pool allocation */ static digest_pool_entry_t *digest_alloc() { digest_pool_entry_t *p_new; PTHREAD_MUTEX_lock(&digest_pool_mutex); p_new = pool_alloc(digest_pool); PTHREAD_MUTEX_unlock(&digest_pool_mutex); return p_new; } == I don't see how this call could be causing a segfault, though, as this function is supposed to be initializing the pool. I also checked other instances of pool_alloc() throughout the code and don't really see substantial differences?? -Nick
On Mon, Jun 28, 2021 at 10:45 AM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sun, Jun 27, 2021 at 10:46 AM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sun, Jun 27, 2021 at 12:55 AM Solomon Boulos <boulos@google.com> wrote:
Good progress! I’ll try to take a look on Monday. Does the proxy FSAL misrepresent it’s readdirplus support? (Seems possible, it should honor whatever the backend says)
That would be great, I'd definitely appreciate any insight. Not sure about the FSAL module and its representation of readdirplus support, but I can dig through the code and take a look. That one I can work around - the Core Dump when the Proxy v4 module has Handle Mapping enabled is a bit more problematic. The function it's referencing at the top - digest_alloc - is simple enough, not really much I can see that would cause a core dump, there.
Regarding the segfault/core dump, I've got the back trace, here:
(gdb) bt #0 0x00007f4abbea6704 in digest_alloc () at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:68 #1 0x00007f4abbea70a2 in handle_mapping_hash_add (p_hash=0x0, object_id=0, handle_hash=2762487084, data=0x7f4abe858432, datalen=29) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:193 #2 0x00007f4abbea75b2 in HandleMap_SetFH (p_in_nfs23_digest=0x7f4abe858420, data=0x7f4abe858432, len=29) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c:373 #3 0x00007f4abbea51df in proxyv4_alloc_handle (exp=0x7f4abe9fc300, fh=0x7ffd7e7a0a40, obj_attributes=0x7ffd7e7a0ae0, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:2997 #4 0x00007f4abbea0883 in proxyv4_make_object (export=0x7f4abe9fc300, obj_attributes=0x7ffd7e7a0ae0, fh=0x7ffd7e7a0a40, handle=0x7ffd7e7a1038, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1399 #5 0x00007f4abbea0f01 in proxyv4_lookup_impl (parent=0x0, export=0x7f4abe9fc300, cred=0x7ffd7e7a1258, path=0x7f4abe813171 "DataBoxDisk1", handle=0x7ffd7e7a1038, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:1545 #6 0x00007f4abbea5465 in proxyv4_lookup_path (exp_hdl=0x7f4abe9fc300, path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a11b0, attrs_out=0x7ffd7e7a10b0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/FSAL_PROXY_V4/handle.c:3068 #7 0x00007f4ac8a0b34d in mdcache_lookup_path (exp_hdl=0x7f4abe82b400, path=0x7f4abe897628 "/DataBoxDisk1", handle=0x7ffd7e7a1338, attrs_out=0x0) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1564 #8 0x00007f4ac89861d7 in init_export_root (export=0x7f4abe877cc8) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2619 #9 0x00007f4ac89858e6 in init_export_cb (exp=0x7f4abe877cc8, state=0x7ffd7e7a1420) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2458 #10 0x00007f4ac899e4c3 in foreach_gsh_export (cb=0x7f4ac89858c2 <init_export_cb>, wrlock=true, state=0x7ffd7e7a1420) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/export_mgr.c:801 #11 0x00007f4ac898593c in exports_pkginit () at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/support/exports.c:2477 #12 0x00007f4ac890ac62 in nfs_Init (p_start_info=0x6041a8 <my_nfs_start_info>) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:642 #13 0x00007f4ac890bb2d in nfs_start (p_start_info=0x6041a8 <my_nfs_start_info>) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_init.c:923 #14 0x00000000004029b9 in main (argc=7, argv=0x7ffd7e7a1838) at /usr/src/debug/nfs-ganesha-4-dev.65.2.el8.x86_64/src/MainNFSD/nfs_main.c:520
This is quite puzzling, because the line that seems to be causing the segfault is the pool_alloc line below:
== pool_t *digest_pool; static pthread_mutex_t digest_pool_mutex = PTHREAD_MUTEX_INITIALIZER;
pool_t *handle_pool; static pthread_mutex_t handle_pool_mutex = PTHREAD_MUTEX_INITIALIZER;
/* helpers for pool allocation */
static digest_pool_entry_t *digest_alloc() { digest_pool_entry_t *p_new;
PTHREAD_MUTEX_lock(&digest_pool_mutex); p_new = pool_alloc(digest_pool); PTHREAD_MUTEX_unlock(&digest_pool_mutex);
return p_new; } ==
I don't see how this call could be causing a segfault, though, as this function is supposed to be initializing the pool. I also checked other instances of pool_alloc() throughout the code and don't really see substantial differences??
I'm sure I'm missing something simple, here, but still not sure why this is causing a segfault, unless the presence of the mutex lock/unlock does something to mess with it? Regarding the readdirplus support, I tried manually disabling it in the proxy_v4 FSAL module by adding ".readdir_plus = false," to the info initialization in main.c. This doesn't seem to have a made a difference, though, so it almost seems as if the NFSv3 server support for it in Ganesha overrides or ignores whatever the V4 proxy module sets, so that it is expecting readdirplus to work. I also don't know why readdirplus would not work - maybe something just needs to be implemented in the V4 proxy module to pass the call through?? -Nick
I'm sure I'm missing something simple, here, but still not sure why this is causing a segfault, unless the presence of the mutex lock/unlock does something to mess with it?
Well, I'm feeling a bit silly about this right now, and have resolved the segfault/core dump issue. The issue was that the directory for handle mapping didn't exist, so the HandleMap_Init function call would exit out without actually allocating the structures for digest and handle pools, but the proxy module would continue to try loading and try to use the handle map. Apparently I didn't catch this in the log file :-/. That said, I'm on to the next issue, which is that, with handle mapping enabled, finally, and the nordirplus mount option (to get around the readdirplus issue), I now get a "Stale File Handle" error when trying to mount: # mount -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure mount.nfs: Stale file handle Current log file, during the mount attempt, is here: https://pastebin.com/wKhLENFk -Nick
Hmm, we do need to make that SEGFAULT not happen… Directories will be missing in life… Get a tcpdump and look at where the handle that is causing the stale handle error is coming from. Do we handle map the handles shared in READDIR? We also need to figure out why READDIRPLUS isn’t working… Frank From: Nick Couchman [mailto:nick.e.couchman@gmail.com] Sent: Monday, June 28, 2021 12:04 PM To: Solomon Boulos <boulos@google.com> Cc: Todd Pfaff <pfaff@rhpcs.mcmaster.ca>; support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3 I'm sure I'm missing something simple, here, but still not sure why this is causing a segfault, unless the presence of the mutex lock/unlock does something to mess with it? Well, I'm feeling a bit silly about this right now, and have resolved the segfault/core dump issue. The issue was that the directory for handle mapping didn't exist, so the HandleMap_Init function call would exit out without actually allocating the structures for digest and handle pools, but the proxy module would continue to try loading and try to use the handle map. Apparently I didn't catch this in the log file :-/. That said, I'm on to the next issue, which is that, with handle mapping enabled, finally, and the nordirplus mount option (to get around the readdirplus issue), I now get a "Stale File Handle" error when trying to mount: # mount -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure mount.nfs: Stale file handle Current log file, during the mount attempt, is here: https://pastebin.com/wKhLENFk -Nick
On Mon, Jun 28, 2021 at 4:21 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Hmm, we do need to make that SEGFAULT not happen… Directories will be missing in life…
Yes, I'd agree this is not ideal behavior - I would think if Handle Mapping is enabled and the directory does not exist, the process of configuring that export ought to be completely stopped.
Get a tcpdump and look at where the handle that is causing the stale handle error is coming from. Do we handle map the handles shared in READDIR?
I put the tshark output in a pastebin and replied with that to the previous e-mail. If I can provide any more detail let me know. I think it's the root handle that is causing it, since it happens during mount?? But this level of NFS detail is very new to me :-).
We also need to figure out why READDIRPLUS isn’t working…
Yes, this would be nice, too. Looking through the Proxy_V4 code, I do not see any references to readdir_plus or readdirplus at all, so I'm wondering if the Proxy module doesn't contain the necessary methods to push the readdirplus command through?? Again, I'm a novice at this level of NFS detail as well as the Ganesha code, so I could be way off. -Nick
On Mon, Jun 28, 2021 at 4:28 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Mon, Jun 28, 2021 at 4:21 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Hmm, we do need to make that SEGFAULT not happen… Directories will be missing in life…
Yes, I'd agree this is not ideal behavior - I would think if Handle Mapping is enabled and the directory does not exist, the process of configuring that export ought to be completely stopped.
Get a tcpdump and look at where the handle that is causing the stale handle error is coming from. Do we handle map the handles shared in READDIR?
I put the tshark output in a pastebin and replied with that to the previous e-mail. If I can provide any more detail let me know. I think it's the root handle that is causing it, since it happens during mount?? But this level of NFS detail is very new to me :-).
Actually I'm not sure this is in readdir, but actually during FSINFO processing, as the NFS3ERR_STALE response comes directly after the client does a FSINFO request: Frame 152: 190 bytes on wire (1520 bits), 190 bytes captured (1520 bits) on interface 0 Ethernet II, Src: Cisco_9d:5a:42 (8c:60:4f:9d:5a:42), Dst: b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58) Internet Protocol Version 4, Src: 10.1.2.5, Dst: 10.1.2.4 Transmission Control Protocol, Src Port: 889, Dst Port: 2049, Seq: 169, Ack: 65, Len: 124 Remote Procedure Call, Type:Call XID:0x76f80e97 Fragment header: Last fragment, 120 bytes 1... .... .... .... .... .... .... .... = Last Fragment: Yes .000 0000 0000 0000 0000 0000 0111 1000 = Fragment Length: 120 XID: 0x76f80e97 (1995968151) Message Type: Call (0) RPC Version: 2 Program: NFS (100003) Program Version: 3 Procedure: FSINFO (19) Credentials Flavor: AUTH_UNIX (1) Length: 52 Stamp: 0x00000000 Machine Name: nfs-client.example.com length: 25 contents: nfs-client.example.com fill bytes: opaque data UID: 0 GID: 0 Auxiliary GIDs (1) [0] GID: 0 Verifier Flavor: AUTH_NULL (0) Length: 0 Network File System, FSINFO Call DH: 0xb2a31f30 [Program Version: 3] [V3 Procedure: FSINFO (19)] object length: 24 [hash (CRC-32): 0xb2a31f30] FileHandle: 4300038610102300006eb32fce0000000000000000000000 Frame 153: 102 bytes on wire (816 bits), 102 bytes captured (816 bits) on interface 0 Ethernet II, Src: b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58), Dst: Cisco_9f:f0:6a (00:00:0c:9f:f0:6a) Internet Protocol Version 4, Src: 10.1.2.4, Dst: 10.1.2.5 Transmission Control Protocol, Src Port: 2049, Dst Port: 889, Seq: 65, Ack: 293, Len: 36 Remote Procedure Call, Type:Reply XID:0x76f80e97 Fragment header: Last fragment, 32 bytes 1... .... .... .... .... .... .... .... = Last Fragment: Yes .000 0000 0000 0000 0000 0000 0010 0000 = Fragment Length: 32 XID: 0x76f80e97 (1995968151) Message Type: Reply (1) [Program: NFS (100003)] [Program Version: 3] [Procedure: FSINFO (19)] Reply State: accepted (0) [This is a reply to a request in frame 152] [Time from request: 0.000301532 seconds] Verifier Flavor: AUTH_NULL (0) Length: 0 Accept State: RPC executed successfully (0) Network File System, FSINFO Reply Error: NFS3ERR_STALE [Program Version: 3] [V3 Procedure: FSINFO (19)] Status: NFS3ERR_STALE (70) obj_attributes attributes_follow: no value (0) -Nick
Could you put the binary wire capture file somewhere? I find it a lot easier to read wire traces in wireshark UI… Frank From: Nick Couchman [mailto:nick.e.couchman@gmail.com] Sent: Monday, June 28, 2021 2:40 PM To: Frank Filz <ffilzlnx@mindspring.com> Cc: Solomon Boulos <boulos@google.com>; Todd Pfaff <pfaff@rhpcs.mcmaster.ca>; support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3 On Mon, Jun 28, 2021 at 4:28 PM Nick Couchman <nick.e.couchman@gmail.com <mailto:nick.e.couchman@gmail.com> > wrote: On Mon, Jun 28, 2021 at 4:21 PM Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > wrote: Hmm, we do need to make that SEGFAULT not happen… Directories will be missing in life… Yes, I'd agree this is not ideal behavior - I would think if Handle Mapping is enabled and the directory does not exist, the process of configuring that export ought to be completely stopped. Get a tcpdump and look at where the handle that is causing the stale handle error is coming from. Do we handle map the handles shared in READDIR? I put the tshark output in a pastebin and replied with that to the previous e-mail. If I can provide any more detail let me know. I think it's the root handle that is causing it, since it happens during mount?? But this level of NFS detail is very new to me :-). Actually I'm not sure this is in readdir, but actually during FSINFO processing, as the NFS3ERR_STALE response comes directly after the client does a FSINFO request: Frame 152: 190 bytes on wire (1520 bits), 190 bytes captured (1520 bits) on interface 0 Ethernet II, Src: Cisco_9d:5a:42 (8c:60:4f:9d:5a:42), Dst: b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58) Internet Protocol Version 4, Src: 10.1.2.5, Dst: 10.1.2.4 Transmission Control Protocol, Src Port: 889, Dst Port: 2049, Seq: 169, Ack: 65, Len: 124 Remote Procedure Call, Type:Call XID:0x76f80e97 Fragment header: Last fragment, 120 bytes 1... .... .... .... .... .... .... .... = Last Fragment: Yes .000 0000 0000 0000 0000 0000 0111 1000 = Fragment Length: 120 XID: 0x76f80e97 (1995968151) Message Type: Call (0) RPC Version: 2 Program: NFS (100003) Program Version: 3 Procedure: FSINFO (19) Credentials Flavor: AUTH_UNIX (1) Length: 52 Stamp: 0x00000000 Machine Name: nfs-client.example.com <http://nfs-client.example.com> length: 25 contents: nfs-client.example.com <http://nfs-client.example.com> fill bytes: opaque data UID: 0 GID: 0 Auxiliary GIDs (1) [0] GID: 0 Verifier Flavor: AUTH_NULL (0) Length: 0 Network File System, FSINFO Call DH: 0xb2a31f30 [Program Version: 3] [V3 Procedure: FSINFO (19)] object length: 24 [hash (CRC-32): 0xb2a31f30] FileHandle: 4300038610102300006eb32fce0000000000000000000000 Frame 153: 102 bytes on wire (816 bits), 102 bytes captured (816 bits) on interface 0 Ethernet II, Src: b2:5a:dd:d6:98:58 (b2:5a:dd:d6:98:58), Dst: Cisco_9f:f0:6a (00:00:0c:9f:f0:6a) Internet Protocol Version 4, Src: 10.1.2.4, Dst: 10.1.2.5 Transmission Control Protocol, Src Port: 2049, Dst Port: 889, Seq: 65, Ack: 293, Len: 36 Remote Procedure Call, Type:Reply XID:0x76f80e97 Fragment header: Last fragment, 32 bytes 1... .... .... .... .... .... .... .... = Last Fragment: Yes .000 0000 0000 0000 0000 0000 0010 0000 = Fragment Length: 32 XID: 0x76f80e97 (1995968151) Message Type: Reply (1) [Program: NFS (100003)] [Program Version: 3] [Procedure: FSINFO (19)] Reply State: accepted (0) [This is a reply to a request in frame 152] [Time from request: 0.000301532 seconds] Verifier Flavor: AUTH_NULL (0) Length: 0 Accept State: RPC executed successfully (0) Network File System, FSINFO Reply Error: NFS3ERR_STALE [Program Version: 3] [V3 Procedure: FSINFO (19)] Status: NFS3ERR_STALE (70) obj_attributes attributes_follow: no value (0) -Nick
On Mon, Jun 28, 2021 at 5:51 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Could you put the binary wire capture file somewhere? I find it a lot easier to read wire traces in wireshark UI…
Sure, here you go - I've done some editing to sanitize it a bit, but it should still show the overall flow: https://drive.google.com/file/d/1PI22xjraWQ2b8HXJsySpoIkmrOe-ZIOe/view?usp=s... -Nick
On Mon, Jun 28, 2021 at 6:35 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Mon, Jun 28, 2021 at 5:51 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Could you put the binary wire capture file somewhere? I find it a lot easier to read wire traces in wireshark UI…
Sure, here you go - I've done some editing to sanitize it a bit, but it should still show the overall flow:
https://drive.google.com/file/d/1PI22xjraWQ2b8HXJsySpoIkmrOe-ZIOe/view?usp=s...
-Nick
Okay, added some additional debugging information and here's what I'm seeing... * The FSINFO request over NFS triggers the nfs3_fsinfo call. * This FSINFO request triggers a call to nfs3_FhandleToCache * nfs3_FhandleToCache calls proxyv4_wire_to_host via the export->export.exp_ops.wire_to_host callback * proxyv4_wire_to_host calls HandleMap_GetFH in the handle_mapping.c file. * HandleMap_GetFH does the hashtable lookup and initially succeeds, but then fails in this section: rc = hashtable_getlatch(handle_map_hash, &buffkey, &buffval, 0, &hl); if (rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *) buffval.addr; if (h->fh_len < fsal_handle->len) { fsal_handle->len = h->fh_len; memcpy(fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; } else { rc = HANDLEMAP_INTERNAL_ERROR; } hashtable_releaselatched(handle_map_hash, &hl); return rc; } The statement "if (h->fh_len < fsal_handle->len)" apparently is not true (so h->fh_len is either greater than or equal to fsal_handle->len), and it skips down to the next section, assigning rc the HANDLEMAP_INTERNAL_ERROR value and then bails out. -Nick
Huh. Can you LogDebug those values? It seems to me like that check should be <= not strict less than. On Mon, Jun 28, 2021 at 20:10 Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Mon, Jun 28, 2021 at 6:35 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Mon, Jun 28, 2021 at 5:51 PM Frank Filz <ffilzlnx@mindspring.com> wrote:
Could you put the binary wire capture file somewhere? I find it a lot easier to read wire traces in wireshark UI…
Sure, here you go - I've done some editing to sanitize it a bit, but it should still show the overall flow:
https://drive.google.com/file/d/1PI22xjraWQ2b8HXJsySpoIkmrOe-ZIOe/view?usp=s...
-Nick
Okay, added some additional debugging information and here's what I'm seeing... * The FSINFO request over NFS triggers the nfs3_fsinfo call. * This FSINFO request triggers a call to nfs3_FhandleToCache * nfs3_FhandleToCache calls proxyv4_wire_to_host via the export->export.exp_ops.wire_to_host callback * proxyv4_wire_to_host calls HandleMap_GetFH in the handle_mapping.c file. * HandleMap_GetFH does the hashtable lookup and initially succeeds, but then fails in this section:
rc = hashtable_getlatch(handle_map_hash, &buffkey, &buffval, 0, &hl);
if (rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *) buffval.addr;
if (h->fh_len < fsal_handle->len) { fsal_handle->len = h->fh_len; memcpy(fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; } else { rc = HANDLEMAP_INTERNAL_ERROR; } hashtable_releaselatched(handle_map_hash, &hl); return rc; }
The statement "if (h->fh_len < fsal_handle->len)" apparently is not true (so h->fh_len is either greater than or equal to fsal_handle->len), and it skips down to the next section, assigning rc the HANDLEMAP_INTERNAL_ERROR value and then bails out.
-Nick
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos@waymo.com> wrote:
Huh. Can you LogDebug those values? It seems to me like that check should be <= not strict less than.
As a matter of fact, yes: 28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com : ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55, fsal_handle->len: 16 So, changing a strict < to <= won't help much... -Nick
Thanks! I’ll have to re-read this code not on my phone tomorrow :). There’s an amusing comment above the GetFH call in wire_to_host about reusing the same buffer for input and output (so maybe the len check got added afterwards?). But I’d rather re-learn what this code is trying to do anyway. On Mon, Jun 28, 2021 at 20:45 Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos@waymo.com> wrote:
Huh. Can you LogDebug those values? It seems to me like that check should be <= not strict less than.
As a matter of fact, yes:
28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com : ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55, fsal_handle->len: 16
So, changing a strict < to <= won't help much...
-Nick
On Mon, Jun 28, 2021 at 11:54 PM Solomon Boulos <boulos@google.com> wrote:
Thanks! I’ll have to re-read this code not on my phone tomorrow :). There’s an amusing comment above the GetFH call in wire_to_host about reusing the same buffer for input and output (so maybe the len check got added afterwards?). But I’d rather re-learn what this code is trying to do anyway.
Sounds good. If I can provide any more information let me know. -Nick
Looks like the handle mapping code isn’t kicking in or isn’t doing what we think it should do. Though we should also have space for a 55 byte wire handle… Maybe that 16 byte space is wrong. Frank From: Nick Couchman [mailto:nick.e.couchman@gmail.com] Sent: Monday, June 28, 2021 8:45 PM To: Solomon Boulos <boulos@waymo.com> Cc: Frank Filz <ffilzlnx@mindspring.com>; Todd Pfaff <pfaff@rhpcs.mcmaster.ca>; support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3 On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos@waymo.com <mailto:boulos@waymo.com> > wrote: Huh. Can you LogDebug those values? It seems to me like that check should be <= not strict less than. As a matter of fact, yes: 28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com <http://nfs-proxy.example.com> : ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55, fsal_handle->len: 16 So, changing a strict < to <= won't help much... -Nick
Alright, looking through some git blame -w, I think this broke and then was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the code compile, but crucially the buffer code no longer had a “max len” just a length. I think this broke the “clever” reuse of the input buffdesc as the argument to GetFH (which has been there since 2012 from 878f45493e). The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the code when written had a FH4_SIZE as it’s max length, rather than the fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there was a length separate from max length at the time or something). IIRC, all the handle mapping code *actually* passes around a buffer that is large enough for the largest FH4 plus some padding. So the “resizing” done in handle_to_wire that needs to be undone in wire_to_host can actually happen in-place but the len field doesn’t know that. Ganesha experts, does that sound right, historically? On Mon, Jun 28, 2021 at 22:14 Frank Filz <ffilzlnx@mindspring.com> wrote:
Looks like the handle mapping code isn’t kicking in or isn’t doing what we think it should do.
Though we should also have space for a 55 byte wire handle…
Maybe that 16 byte space is wrong.
Frank
*From:* Nick Couchman [mailto:nick.e.couchman@gmail.com] *Sent:* Monday, June 28, 2021 8:45 PM *To:* Solomon Boulos <boulos@waymo.com> *Cc:* Frank Filz <ffilzlnx@mindspring.com>; Todd Pfaff < pfaff@rhpcs.mcmaster.ca>; support@lists.nfs-ganesha.org *Subject:* [NFS-Ganesha-Support] Re: Proxy FSAL - NFS4 -> 3
On Mon, Jun 28, 2021 at 11:41 PM Solomon Boulos <boulos@waymo.com> wrote:
Huh. Can you LogDebug those values? It seems to me like that check should be <= not strict less than.
As a matter of fact, yes:
28/06/2021 23:29:33 : epoch 60da938f : nfs-proxy.example.com : ganesha.nfsd-464682[svc_10] HandleMap_GetFH :FSAL :DEBUG :h->fh_len: 55, fsal_handle->len: 16
So, changing a strict < to <= won't help much...
-Nick
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos@google.com> wrote:
Alright, looking through some git blame -w, I think this broke and then was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the code compile, but crucially the buffer code no longer had a “max len” just a length.
I think this broke the “clever” reuse of the input buffdesc as the argument to GetFH (which has been there since 2012 from 878f45493e).
I'll try to take a look at each of these commits and see if I can figure out how to re-integrate the support back into the current code, though I don't know how successful I will be. If anyone has suggestions for what needs to be done, please let me know - I'm not sure right now if it's just some changes in the handle_mapping.c file, or if there are changes elsewhere that would need to be made (like in the wire_to_host/handle_to_wire functions, which I think are elsewhere. -Nick
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos@google.com> wrote:
Alright, looking through some git blame -w, I think this broke and then was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the code compile, but crucially the buffer code no longer had a “max len” just a length.
I think this broke the “clever” reuse of the input buffdesc as the argument to GetFH (which has been there since 2012 from 878f45493e).
The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the code when written had a FH4_SIZE as it’s max length, rather than the fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there was a length separate from max length at the time or something).
Digging into this a little more, I think the following commit actually originally broke things: 3016c4cee45635afa9c25b21c2f5b014d80d1b6d In that commit, the "netbuf" data type used as an argument for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the following code in that function would not have compiled after that: == ... if(rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata; if(h->fh_len < p_out_fsal_handle->maxlen) { p_out_fsal_handle->len = h->fh_len; memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len); memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; } ... == After that change, there are various iterations of renaming the "p_out_fsal_handle" parameter to other things, up until the commit that you mentioned that fixed the compile issues (823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code having "p_out_fsal_handle" renamed to the value actually being passed in, and the "maxlen" member replaced with "len".
IIRC, all the handle mapping code *actually* passes around a buffer that is large enough for the largest FH4 plus some padding. So the “resizing” done in handle_to_wire that needs to be undone in wire_to_host can actually happen in-place but the len field doesn’t know that.
So, any suggestions on the most reliable way to determine "maxlen", again? Should there be a #define constant somewhere that sets it, or is it something that should be more dynamically computed based on the data passed in - maybe the function that does the padding? Or should the "if ()" statement be removed entirely, and just assume that the length is always small enough (that seems like a bad option)? Seems like there probably aren't a lot of users of the handle mapping code :-D. -Nick
On Sat, Jul 3, 2021 at 10:08 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos@google.com> wrote:
Alright, looking through some git blame -w, I think this broke and then was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the code compile, but crucially the buffer code no longer had a “max len” just a length.
I think this broke the “clever” reuse of the input buffdesc as the argument to GetFH (which has been there since 2012 from 878f45493e).
The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the code when written had a FH4_SIZE as it’s max length, rather than the fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there was a length separate from max length at the time or something).
Digging into this a little more, I think the following commit actually originally broke things:
3016c4cee45635afa9c25b21c2f5b014d80d1b6d
In that commit, the "netbuf" data type used as an argument for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the following code in that function would not have compiled after that:
== ... if(rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata; if(h->fh_len < p_out_fsal_handle->maxlen) { p_out_fsal_handle->len = h->fh_len; memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len); memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; } ... ==
After that change, there are various iterations of renaming the "p_out_fsal_handle" parameter to other things, up until the commit that you mentioned that fixed the compile issues (823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code having "p_out_fsal_handle" renamed to the value actually being passed in, and the "maxlen" member replaced with "len".
And, just another note on this - in the commit prior to 3016c4cee4, I believe the netbuf that gets passed around is allocated in this block of code: == fsal_status_t pxy_create_handle(struct fsal_export *exp_hdl, struct fsal_handle_desc *hdl_desc, struct fsal_obj_handle **handle) { fsal_status_t st; nfs_fh4 fh4; fsal_attrib_list_t attr; struct pxy_obj_handle *ph; #ifdef _HANDLE_MAPPING char fh_data[NFS4_FHSIZE+2]; #endif if(!exp_hdl || !hdl_desc || !handle || (hdl_desc->len > NFS4_FHSIZE)) ReturnCode(ERR_FSAL_INVAL, 0); fh4.nfs_fh4_val = hdl_desc->start; fh4.nfs_fh4_len = hdl_desc->len; #ifdef _HANDLE_MAPPING if(hdl_desc->len == sizeof(nfs23_map_handle_t)) { nfs23_map_handle_t *h23 = (nfs23_map_handle_t*)hdl_desc->start; if(h23->type == PXY_HANDLE_MAPPED) { struct netbuf fh = { .maxlen = sizeof(fh_data), .buf = fh_data }; struct pxy_handle_blob *blob; == From this I gather that the size of maxlen for that netbuf is the size of the data going into the buffer; however, that member is allocated as NFS4_FHSIZE + 2. So, maybe a constant, something like: #define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2 Or maybe there's a better way? -Nick
On Sun, Jul 4, 2021 at 10:04 AM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jul 3, 2021 at 10:08 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos@google.com> wrote:
Alright, looking through some git blame -w, I think this broke and then was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the code compile, but crucially the buffer code no longer had a “max len” just a length.
I think this broke the “clever” reuse of the input buffdesc as the argument to GetFH (which has been there since 2012 from 878f45493e).
The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the code when written had a FH4_SIZE as it’s max length, rather than the fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there was a length separate from max length at the time or something).
Digging into this a little more, I think the following commit actually originally broke things:
3016c4cee45635afa9c25b21c2f5b014d80d1b6d
In that commit, the "netbuf" data type used as an argument for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the following code in that function would not have compiled after that:
== ... if(rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata; if(h->fh_len < p_out_fsal_handle->maxlen) { p_out_fsal_handle->len = h->fh_len; memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len); memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; } ... ==
After that change, there are various iterations of renaming the "p_out_fsal_handle" parameter to other things, up until the commit that you mentioned that fixed the compile issues (823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code having "p_out_fsal_handle" renamed to the value actually being passed in, and the "maxlen" member replaced with "len".
And, just another note on this - in the commit prior to 3016c4cee4, I believe the netbuf that gets passed around is allocated in this block of code:
== fsal_status_t pxy_create_handle(struct fsal_export *exp_hdl, struct fsal_handle_desc *hdl_desc, struct fsal_obj_handle **handle) { fsal_status_t st; nfs_fh4 fh4; fsal_attrib_list_t attr; struct pxy_obj_handle *ph; #ifdef _HANDLE_MAPPING char fh_data[NFS4_FHSIZE+2]; #endif
if(!exp_hdl || !hdl_desc || !handle || (hdl_desc->len > NFS4_FHSIZE)) ReturnCode(ERR_FSAL_INVAL, 0);
fh4.nfs_fh4_val = hdl_desc->start; fh4.nfs_fh4_len = hdl_desc->len;
#ifdef _HANDLE_MAPPING if(hdl_desc->len == sizeof(nfs23_map_handle_t)) { nfs23_map_handle_t *h23 = (nfs23_map_handle_t*)hdl_desc->start; if(h23->type == PXY_HANDLE_MAPPED) { struct netbuf fh = { .maxlen = sizeof(fh_data), .buf = fh_data }; struct pxy_handle_blob *blob; ==
From this I gather that the size of maxlen for that netbuf is the size of the data going into the buffer; however, that member is allocated as NFS4_FHSIZE + 2.
So, maybe a constant, something like:
#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
The following patch seems to fix things: == diff -Naur nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c --- nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c 2021-07-02 18:12:40.000000000 -0400 +++ nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c 2021-07-04 08:25:04.398912776 -0400 @@ -344,7 +344,7 @@ if (rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *) buffval.addr; - if (h->fh_len < fsal_handle->len) { + if (h->fh_len < PROXYV4_HANDLE_MAXLEN) { fsal_handle->len = h->fh_len; memcpy(fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; diff -Naur nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h --- nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h 2021-07-02 18:12:40.000000000 -0400 +++ nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h 2021-07-04 08:24:44.496077148 -0400 @@ -81,6 +81,9 @@ #define HANDLEMAP_HASHTABLE_ERROR 7 #define HANDLEMAP_EXISTS 8 +/* Maximum file handle length */ +#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2 + int HandleMap_Init(const handle_map_param_t *p_param); int HandleMap_GetFH(const nfs23_map_handle_t *, struct gsh_buffdesc *); == No extensive testing done, yet, but I'm happy to submit a pull request, if this makes sense?
Or maybe there's a better way?
Or maybe there's still a better way?? -Nick
I had meant to add lots of comments to the FSAL API header about how all this handle-to-wire / wire-to-host / handle-to-key stuff works… (particularly the memory lifetimes and ownership) because I struggled with it when implementing the v3 proxy. The giant comment in fsal_api.h is not bad, but the real action is in nfs_filehandle_mgmt.c (there are other callers of wire_to_host, including the mdcache just passing it through to the underlying FSAL, and an NFSv4 digest variant, but HandleMap_GetFH is being called on the "get me the v3 handle" path): https://github.com/nfs-ganesha/nfs-ganesha/blob/2ce0989828a1d3fc1f5dc6164ae1... which shows that the buffer passed into wire_to_host is NFS3_FHSIZE long (which makes sense, the function is trying to compute an fh3 as output). The comment there incorrectly says "no larger than NFS4_FHSIZE" but it should say NFS3_FHSIZE. (How'd you get NFS4_FHSIZE+2?) We should still feel bad about that FIXME in FhandleToCache and the reuse of the buffer as input and output (the size of the buffer doesn't have much to do with the actual fs len). On Sun, Jul 4, 2021 at 07:30 Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sun, Jul 4, 2021 at 10:04 AM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Sat, Jul 3, 2021 at 10:08 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
On Tue, Jun 29, 2021 at 11:52 PM Solomon Boulos <boulos@google.com> wrote:
Alright, looking through some git blame -w, I think this broke and then was restored in 2912fad98e. Jim had a diff in 2014 (823625c5da) to make the code compile, but crucially the buffer code no longer had a “max len” just a length.
I think this broke the “clever” reuse of the input buffdesc as the argument to GetFH (which has been there since 2012 from 878f45493e).
The input when mapped is a 16-byte thingy (sizeof nfs23_map_handle_t), and GetFH needs to expand the 16-byte mapped handle into an FH4. I assume the code when written had a FH4_SIZE as it’s max length, rather than the fh_desc buffer appearing to be trimmed to 16-bytes (that is, I think there was a length separate from max length at the time or something).
Digging into this a little more, I think the following commit actually originally broke things:
3016c4cee45635afa9c25b21c2f5b014d80d1b6d
In that commit, the "netbuf" data type used as an argument for HandleMap_GetFH is replaced with the "gsh_buffdesc" data type. As far as I can tell, "gsh_buffdesc" never had a "maxlen" member, so the following code in that function would not have compiled after that:
== ... if(rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *)buffval.pdata; if(h->fh_len < p_out_fsal_handle->maxlen) { p_out_fsal_handle->len = h->fh_len; memcpy(p_out_fsal_handle->buf, h->fh_data, h->fh_len); memcpy(p_out_fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; } ... ==
After that change, there are various iterations of renaming the "p_out_fsal_handle" parameter to other things, up until the commit that you mentioned that fixed the compile issues (823625c5da2373606dd4f75dce46a92be143507c), which resulted in the code having "p_out_fsal_handle" renamed to the value actually being passed in, and the "maxlen" member replaced with "len".
And, just another note on this - in the commit prior to 3016c4cee4, I believe the netbuf that gets passed around is allocated in this block of code:
== fsal_status_t pxy_create_handle(struct fsal_export *exp_hdl, struct fsal_handle_desc *hdl_desc, struct fsal_obj_handle **handle) { fsal_status_t st; nfs_fh4 fh4; fsal_attrib_list_t attr; struct pxy_obj_handle *ph; #ifdef _HANDLE_MAPPING char fh_data[NFS4_FHSIZE+2]; #endif
if(!exp_hdl || !hdl_desc || !handle || (hdl_desc->len > NFS4_FHSIZE)) ReturnCode(ERR_FSAL_INVAL, 0);
fh4.nfs_fh4_val = hdl_desc->start; fh4.nfs_fh4_len = hdl_desc->len;
#ifdef _HANDLE_MAPPING if(hdl_desc->len == sizeof(nfs23_map_handle_t)) { nfs23_map_handle_t *h23 = (nfs23_map_handle_t*)hdl_desc->start; if(h23->type == PXY_HANDLE_MAPPED) { struct netbuf fh = { .maxlen = sizeof(fh_data), .buf = fh_data }; struct pxy_handle_blob *blob; ==
From this I gather that the size of maxlen for that netbuf is the size of the data going into the buffer; however, that member is allocated as NFS4_FHSIZE + 2.
So, maybe a constant, something like:
#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2
The following patch seems to fix things:
== diff -Naur nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c --- nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c 2021-07-02 18:12:40.000000000 -0400 +++ nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.c 2021-07-04 08:25:04.398912776 -0400 @@ -344,7 +344,7 @@ if (rc == HASHTABLE_SUCCESS) { handle_pool_entry_t *h = (handle_pool_entry_t *) buffval.addr;
- if (h->fh_len < fsal_handle->len) { + if (h->fh_len < PROXYV4_HANDLE_MAXLEN) { fsal_handle->len = h->fh_len; memcpy(fsal_handle->addr, h->fh_data, h->fh_len); rc = HANDLEMAP_SUCCESS; diff -Naur nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h --- nfs-ganesha-4-dev.66.a/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h 2021-07-02 18:12:40.000000000 -0400 +++ nfs-ganesha-4-dev.66.b/src/FSAL/FSAL_PROXY_V4/handle_mapping/handle_mapping.h 2021-07-04 08:24:44.496077148 -0400 @@ -81,6 +81,9 @@ #define HANDLEMAP_HASHTABLE_ERROR 7 #define HANDLEMAP_EXISTS 8
+/* Maximum file handle length */ +#define PROXYV4_HANDLE_MAXLEN NFS4_FHSIZE+2 + int HandleMap_Init(const handle_map_param_t *p_param);
int HandleMap_GetFH(const nfs23_map_handle_t *, struct gsh_buffdesc *); ==
No extensive testing done, yet, but I'm happy to submit a pull request, if this makes sense?
Or maybe there's a better way?
Or maybe there's still a better way??
-Nick
On Fri, Jul 9, 2021 at 2:09 PM Solomon Boulos <boulos@waymo.com> wrote:
I had meant to add lots of comments to the FSAL API header about how all this handle-to-wire / wire-to-host / handle-to-key stuff works… (particularly the memory lifetimes and ownership) because I struggled with it when implementing the v3 proxy.
The giant comment in fsal_api.h is not bad, but the real action is in nfs_filehandle_mgmt.c (there are other callers of wire_to_host, including the mdcache just passing it through to the underlying FSAL, and an NFSv4 digest variant, but HandleMap_GetFH is being called on the "get me the v3 handle" path):
https://github.com/nfs-ganesha/nfs-ganesha/blob/2ce0989828a1d3fc1f5dc6164ae1...
which shows that the buffer passed into wire_to_host is NFS3_FHSIZE long (which makes sense, the function is trying to compute an fh3 as output). The comment there incorrectly says "no larger than NFS4_FHSIZE" but it should say NFS3_FHSIZE. (How'd you get NFS4_FHSIZE+2?)
Prior to commit 3016c4cee4, when the netbuf was being used, the fh_data buffer was allocated with that size, and then the netbuf's "maxlen" property was set to sizeof(fh_data), effectively setting the max length to NFS4_FHSIZE + 2. I don't really know the rationale for that, either then or now, I'm basically just restoring the lengths that were there before. I'm certainly not saying this is the correct way to do it, nor that it is safe - just that it appears to work. I think Frank is correct, though - it's worth trying to figure out what values we should be using and make sure things are sane - just going with it because it appears to work isn't necessarily a good idea. -Nick
Can you do mount -vvv to get some more info on the client? The ganesha side doesn't *seem* to be sad. It's getting and handling NULL, MNT_NULL, MNT_MNT, lookup of /azure-file (lookup of the root filehandle), and then responding to an FSINFO request. If the mount -vvv isn't revealing, we might need a wireshark dump of the NFS traffic. For v3 I usually do: sudo tshark -J "mount,rpc,nfs,nlm,portmap" -O "mount,rpc,nfs,nlm,portmap" host <IP> On Mon, Jun 28, 2021 at 12:03 PM Nick Couchman <nick.e.couchman@gmail.com> wrote:
I'm sure I'm missing something simple, here, but still not sure why this is causing a segfault, unless the presence of the mutex lock/unlock does something to mess with it?
Well, I'm feeling a bit silly about this right now, and have resolved the segfault/core dump issue. The issue was that the directory for handle mapping didn't exist, so the HandleMap_Init function call would exit out without actually allocating the structures for digest and handle pools, but the proxy module would continue to try loading and try to use the handle map. Apparently I didn't catch this in the log file :-/.
That said, I'm on to the next issue, which is that, with handle mapping enabled, finally, and the nordirplus mount option (to get around the readdirplus issue), I now get a "Stale File Handle" error when trying to mount:
# mount -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure mount.nfs: Stale file handle
Current log file, during the mount attempt, is here: https://pastebin.com/wKhLENFk
-Nick
On Mon, Jun 28, 2021 at 3:25 PM Solomon Boulos <boulos@waymo.com> wrote:
Can you do mount -vvv to get some more info on the client? The ganesha side doesn't *seem* to be sad.
Sure, here you go: # mount -vvv -t nfs -o vers=3,nordirplus 10.1.2.4:/azure-file /mnt/azure mount.nfs: timeout set for Mon Jun 28 15:28:50 2021 mount.nfs: trying text-based options 'vers=3,nordirplus,addr=10.1.2.4' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.1.2.4 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: portmap query retrying: RPC: Program not registered mount.nfs: prog 100005, trying vers=3, prot=6 mount.nfs: trying 10.1.2.4 prog 100005 vers 3 prot TCP port 20048 mount.nfs: mount(2): Stale file handle mount.nfs: trying text-based options 'vers=3,nordirplus,addr=10.1.2.4' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.1.2.4 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: portmap query retrying: RPC: Program not registered mount.nfs: prog 100005, trying vers=3, prot=6 mount.nfs: trying 10.1.2.4 prog 100005 vers 3 prot TCP port 20048 mount.nfs: mount(2): Stale file handle mount.nfs: trying text-based options 'vers=3,nordirplus,addr=10.1.2.4' mount.nfs: prog 100003, trying vers=3, prot=6 mount.nfs: trying 10.1.2.4 prog 100003 vers 3 prot TCP port 2049 mount.nfs: prog 100005, trying vers=3, prot=17 mount.nfs: portmap query retrying: RPC: Program not registered mount.nfs: prog 100005, trying vers=3, prot=6 mount.nfs: trying 10.1.2.4 prog 100005 vers 3 prot TCP port 20048 mount.nfs: mount(2): Stale file handle mount.nfs: Stale file handle
It's getting and handling NULL, MNT_NULL, MNT_MNT, lookup of /azure-file (lookup of the root filehandle), and then responding to an FSINFO request.
If the mount -vvv isn't revealing, we might need a wireshark dump of the NFS traffic. For v3 I usually do:
sudo tshark -J "mount,rpc,nfs,nlm,portmap" -O "mount,rpc,nfs,nlm,portmap" host <IP>
Here's the pastebin for that: https://pastebin.com/Hp4qgZdk -Nick
participants (5)
-
Frank Filz -
Nick Couchman -
Solomon Boulos -
Solomon Boulos -
Todd Pfaff