EIO on NFSv4.1 mounts
Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
Do you have a tcpdump trace? Ganesha has been used with various versions of the Linux 4.1 client so it’s possible you just hit one that had a regression that has since been fixed. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 4:03 AM To: devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] EIO on NFSv4.1 mounts Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
I should also add that there has been somewhat limited 4.1 testing, I know at least some of the major downstream products have not yet done extensive 4.1 testing. Frank From: Frank Filz [mailto:ffilzlnx@mindspring.com] Sent: Friday, August 3, 2018 7:25 AM To: 'Sriram Patil' <sriramp@vmware.com>; devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Do you have a tcpdump trace? Ganesha has been used with various versions of the Linux 4.1 client so it’s possible you just hit one that had a regression that has since been fixed. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 4:03 AM To: devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] EIO on NFSv4.1 mounts Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
Hi Frank, Thanks for the info. Please find the tcpdump attached. It has very few NFS packets. The OPEN reply is the one which fails on the client side. Thanks, Sriram From: Frank Filz <ffilzlnx@mindspring.com> Date: Friday, August 3, 2018 at 7:56 PM To: Sriram Patil <sriramp@vmware.com>, "devel@lists.nfs-ganesha.org" <devel@lists.nfs-ganesha.org> Subject: RE: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts I should also add that there has been somewhat limited 4.1 testing, I know at least some of the major downstream products have not yet done extensive 4.1 testing. Frank From: Frank Filz [mailto:ffilzlnx@mindspring.com] Sent: Friday, August 3, 2018 7:25 AM To: 'Sriram Patil' <sriramp@vmware.com>; devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Do you have a tcpdump trace? Ganesha has been used with various versions of the Linux 4.1 client so it’s possible you just hit one that had a regression that has since been fixed. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 4:03 AM To: devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] EIO on NFSv4.1 mounts Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
Hmm, I see nothing wrong with the OPEN, it succeeded (though no delegation was granted, as expected considering our current state of delegations). Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 10:03 AM To: Frank Filz <ffilzlnx@mindspring.com>; devel@lists.nfs-ganesha.org Subject: Re: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Hi Frank, Thanks for the info. Please find the tcpdump attached. It has very few NFS packets. The OPEN reply is the one which fails on the client side. Thanks, Sriram From: Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > Date: Friday, August 3, 2018 at 7:56 PM To: Sriram Patil <sriramp@vmware.com <mailto:sriramp@vmware.com> >, "devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> " <devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> > Subject: RE: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts I should also add that there has been somewhat limited 4.1 testing, I know at least some of the major downstream products have not yet done extensive 4.1 testing. Frank From: Frank Filz [mailto:ffilzlnx@mindspring.com] Sent: Friday, August 3, 2018 7:25 AM To: 'Sriram Patil' <sriramp@vmware.com <mailto:sriramp@vmware.com> >; devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Do you have a tcpdump trace? Ganesha has been used with various versions of the Linux 4.1 client so it’s possible you just hit one that had a regression that has since been fixed. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 4:03 AM To: devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] EIO on NFSv4.1 mounts Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
Yes, the error is one the client side when decoding the result for some reason. This is also very easily reproducible on 3.13 kernel (Ubuntu14) or 3.10 kernel (CentOS). - Sriram From: Frank Filz <ffilzlnx@mindspring.com> Date: Friday, August 3, 2018 at 11:20 PM To: Sriram Patil <sriramp@vmware.com>, "devel@lists.nfs-ganesha.org" <devel@lists.nfs-ganesha.org> Subject: RE: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Hmm, I see nothing wrong with the OPEN, it succeeded (though no delegation was granted, as expected considering our current state of delegations). Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 10:03 AM To: Frank Filz <ffilzlnx@mindspring.com>; devel@lists.nfs-ganesha.org Subject: Re: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Hi Frank, Thanks for the info. Please find the tcpdump attached. It has very few NFS packets. The OPEN reply is the one which fails on the client side. Thanks, Sriram From: Frank Filz <ffilzlnx@mindspring.com<mailto:ffilzlnx@mindspring.com>> Date: Friday, August 3, 2018 at 7:56 PM To: Sriram Patil <sriramp@vmware.com<mailto:sriramp@vmware.com>>, "devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org>" <devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org>> Subject: RE: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts I should also add that there has been somewhat limited 4.1 testing, I know at least some of the major downstream products have not yet done extensive 4.1 testing. Frank From: Frank Filz [mailto:ffilzlnx@mindspring.com] Sent: Friday, August 3, 2018 7:25 AM To: 'Sriram Patil' <sriramp@vmware.com<mailto:sriramp@vmware.com>>; devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Do you have a tcpdump trace? Ganesha has been used with various versions of the Linux 4.1 client so it’s possible you just hit one that had a regression that has since been fixed. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 4:03 AM To: devel@lists.nfs-ganesha.org<mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] EIO on NFSv4.1 mounts Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
Consensus is that Bruce Fields big rework of RPC code in the client may have fixed it. 3.13 kernel is really old and may be a lost cause… Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 11:21 AM To: Frank Filz <ffilzlnx@mindspring.com>; devel@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Yes, the error is one the client side when decoding the result for some reason. This is also very easily reproducible on 3.13 kernel (Ubuntu14) or 3.10 kernel (CentOS). - Sriram From: Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > Date: Friday, August 3, 2018 at 11:20 PM To: Sriram Patil <sriramp@vmware.com <mailto:sriramp@vmware.com> >, "devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> " <devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> > Subject: RE: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Hmm, I see nothing wrong with the OPEN, it succeeded (though no delegation was granted, as expected considering our current state of delegations). Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 10:03 AM To: Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> >; devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: Re: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Hi Frank, Thanks for the info. Please find the tcpdump attached. It has very few NFS packets. The OPEN reply is the one which fails on the client side. Thanks, Sriram From: Frank Filz <ffilzlnx@mindspring.com <mailto:ffilzlnx@mindspring.com> > Date: Friday, August 3, 2018 at 7:56 PM To: Sriram Patil <sriramp@vmware.com <mailto:sriramp@vmware.com> >, "devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> " <devel@lists.nfs-ganesha.org> Subject: RE: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts I should also add that there has been somewhat limited 4.1 testing, I know at least some of the major downstream products have not yet done extensive 4.1 testing. Frank From: Frank Filz [mailto:ffilzlnx@mindspring.com] Sent: Friday, August 3, 2018 7:25 AM To: 'Sriram Patil' <sriramp@vmware.com <mailto:sriramp@vmware.com> >; devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] Re: EIO on NFSv4.1 mounts Do you have a tcpdump trace? Ganesha has been used with various versions of the Linux 4.1 client so it’s possible you just hit one that had a regression that has since been fixed. Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Friday, August 3, 2018 4:03 AM To: devel@lists.nfs-ganesha.org <mailto:devel@lists.nfs-ganesha.org> Subject: [NFS-Ganesha-Devel] EIO on NFSv4.1 mounts Hi, I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open, reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5 The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1. This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1? Thanks, Sriram
On Fri, 2018-08-03 at 11:03 +0000, Sriram Patil wrote:
Hi,
I was trying to do NFSv4.1 mounts with VFS exports, but I am getting following error on the client side when doing an open,
reading TCP record fragment of length 376 reading XID (4 bytes) reading reply for XID 6b8aed78 reading CALL/REPLY flag (4 bytes) read reply XID 6b8aed78 XID 6b8aed78 read 368 bytes xprt = ffff88022e5af000, tcp_copied = 376, tcp_offset = 376, tcp_reclen = 376 27 xid 6b8aed78 complete (376 bytes received) 27 __rpc_wake_up_task (now 4294955572) 27 disabling timer 27 removed from queue ffff88022e5af258 "xprt_pending" __rpc_wake_up_task done xs_tcp_data_recv done 27 __rpc_execute flags=0x4881 27 call_status (status 376) 27 call_decode (status 376) 27 validating UNIX cred ffff8800ba74a780 27 using AUTH_UNIX cred ffff8800ba74a780 to unwrap rpc data decode_opaque_inline: prematurely hit end of receive buffer. Remaining buffer length is 49 words. 27 call_decode result -5
The linux kernel version I am using is 3.13. NFSv4.0 mounts work fine without any problem. I also observed that the NFSv4.1 mounts work well on kernel 4.4. I could not find any info about the minimum required kernel version for using NFSv4.1.
This is probably an issue on the NFS client side. Does anyone know the required kernel version for using NFSv4.1?
The client failed while trying to decode the reply. It's almost certainly a bug in the client. As far as what version you need at a minimum for working v4.1 support...I'm not sure anyone has worked that out. It depends on several factors, concerning your workload, etc. In general though, newer kernels should work better than older ones. -- Jeff Layton <jlayton@kernel.org>
participants (3)
-
Frank Filz -
Jeff Layton -
Sriram Patil