12:18 a.m.
Hello,
We are planning to enable support for NFS over RDMA in Ganesha.
There is some existing implementation in tirpc layer for RDMA, which was
not in working state and not maintained any more.
We spent some time to fix this code and add missing functionalities as
mentioned below,
Issue that we found and fixed:
1.
Connection management issues connect and disconnect paths. (Fixed)
2.
Static buffer management, can't scale. (Fixed)
3.
Functional issues due not compliance with RPC over RDMA RFC. (Fixed)
4.
Synchronous callbacks. (Not fixed)
5.
Protocol buffer changes for read/readdir. (Fixed)
6.
NFSv4 referral. (Client side issue, fixed in latest kernels)
7.
Callback channels using RDMA clnt. (Not fixed)
8.
ib verbs apis ibv_post_send. (Are there any issues with ib verbs apis?)
9.
Srq changes. (Not fixed)
10.
Single completion queue manager thread for all connections. (Not fixed)
11.
Per connection buffer management (Nit fixed)
We are planning to discuss it in a Ganesha community call (Tuesday, April
23rd 2024, 7.30 PM - 8.30 PM IST)
Looking for any suggestions/feedback if anyone has tried it or has
expertise in this area.
Regards,
Gaurav Gangalwar
11:20 p.m.
Hello Ganesha Team,
Friendly reminder :-)
Regards,
Rojin George
On Fri, 19 Apr, 2024, 10:49 am gaurav gangalwar, <gaurav.gangalwar(a)gmail.com>
wrote:
Hello,
We are planning to enable support for NFS over RDMA in Ganesha.
There is some existing implementation in tirpc layer for RDMA, which was
not in working state and not maintained any more.
We spent some time to fix this code and add missing functionalities as
mentioned below,
Issue that we found and fixed:
1.
Connection management issues connect and disconnect paths. (Fixed)
2.
Static buffer management, can't scale. (Fixed)
3.
Functional issues due not compliance with RPC over RDMA RFC. (Fixed)
4.
Synchronous callbacks. (Not fixed)
5.
Protocol buffer changes for read/readdir. (Fixed)
6.
NFSv4 referral. (Client side issue, fixed in latest kernels)
7.
Callback channels using RDMA clnt. (Not fixed)
8.
ib verbs apis ibv_post_send. (Are there any issues with ib verbs apis?)
9.
Srq changes. (Not fixed)
10.
Single completion queue manager thread for all connections. (Not fixed)
11.
Per connection buffer management (Nit fixed)
We are planning to discuss it in a Ganesha community call (Tuesday, April
23rd 2024, 7.30 PM - 8.30 PM IST)
Looking for any suggestions/feedback if anyone has tried it or has
expertise in this area.
Regards,
Gaurav Gangalwar
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
11:08 a.m.
Hello,
This is the tirpc pull request to enable support for NFS over RDMA
https://github.com/nfs-ganesha/ntirpc/pull/296
We have a supporting patch in Ganesha to use rdma registered buffers for
read and readdir responses.
With this patch and supporting Ganesha patch, we should be able to use
NFSv3 and NFSv4.0 over RDMA.
Delegation has to be disabled for NFSv4.0, since the callback channel is
not yet supported over RDMA.
We tested NFSoRDMA with both hardware and SoftROCE
Hardware:
Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual
Function Rocky Linux release 8.9 (Green Obsidian)
Linux kernel version 5.10
SoftROCE:
Rocky Linux release 8.9 (Green Obsidian)
Linux kernel version 5.4.275-1.el8.elrepo.x86_64
Tested with fsal=MEM
Regards,
Gaurav Gangalwar
On Tue, Apr 23, 2024 at 9:51 AM Rojin George <itsmerojin(a)gmail.com> wrote:
Hello Ganesha Team,
Friendly reminder :-)
Regards,
Rojin George
On Fri, 19 Apr, 2024, 10:49 am gaurav gangalwar, <
gaurav.gangalwar(a)gmail.com> wrote:
> Hello,
> We are planning to enable support for NFS over RDMA in Ganesha.
> There is some existing implementation in tirpc layer for RDMA, which was
> not in working state and not maintained any more.
> We spent some time to fix this code and add missing functionalities as
> mentioned below,
>
> Issue that we found and fixed:
>
> 1.
>
> Connection management issues connect and disconnect paths. (Fixed)
> 2.
>
> Static buffer management, can't scale. (Fixed)
> 3.
>
> Functional issues due not compliance with RPC over RDMA RFC. (Fixed)
> 4.
>
> Synchronous callbacks. (Not fixed)
> 5.
>
> Protocol buffer changes for read/readdir. (Fixed)
> 6.
>
> NFSv4 referral. (Client side issue, fixed in latest kernels)
> 7.
>
> Callback channels using RDMA clnt. (Not fixed)
> 8.
>
> ib verbs apis ibv_post_send. (Are there any issues with ib verbs
> apis?)
> 9.
>
> Srq changes. (Not fixed)
> 10.
>
> Single completion queue manager thread for all connections. (Not
> fixed)
> 11.
>
> Per connection buffer management (Nit fixed)
>
>
> We are planning to discuss it in a Ganesha community call (Tuesday,
> April 23rd 2024, 7.30 PM - 8.30 PM IST)
> Looking for any suggestions/feedback if anyone has tried it or has
> expertise in this area.
>
> Regards,
> Gaurav Gangalwar
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org
> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>
7
days inactive
27
days old
2 comments
2 participants
participants (2)
-
gaurav gangalwar -
Rojin George