Hello,
This is the tirpc pull request to enable support for NFS over RDMA
https://github.com/nfs-ganesha/ntirpc/pull/296

We have a supporting patch in Ganesha to use rdma registered buffers for read and readdir responses.
With this patch and supporting Ganesha patch, we should be able to use NFSv3 and NFSv4.0 over RDMA.
Delegation has to be disabled for NFSv4.0, since the callback channel is not yet supported over RDMA.

We tested NFSoRDMA with both hardware and SoftROCE

Hardware:
Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function Rocky Linux release 8.9 (Green Obsidian)
Linux kernel version 5.10

SoftROCE:
Rocky Linux release 8.9 (Green Obsidian)
Linux kernel version 5.4.275-1.el8.elrepo.x86_64
Tested with fsal=MEM



Regards,
Gaurav Gangalwar

On Tue, Apr 23, 2024 at 9:51 AM Rojin George <itsmerojin@gmail.com> wrote:
Hello Ganesha Team,

    Friendly reminder :-)

Regards,
Rojin George


On Fri, 19 Apr, 2024, 10:49 am gaurav gangalwar, <gaurav.gangalwar@gmail.com> wrote:
Hello,
We are planning to enable support for NFS over RDMA in Ganesha.
There is some existing implementation in tirpc layer for RDMA, which was not in working state and not maintained any more.
We spent some time to fix this code and add missing functionalities as mentioned below,

Issue that we found and fixed:

  1. Connection management issues connect and disconnect paths. (Fixed)

  2. Static buffer management, can't scale. (Fixed)

  3. Functional issues due not compliance with RPC over RDMA RFC. (Fixed)

  4. Synchronous callbacks. (Not fixed)

  5. Protocol buffer changes for read/readdir. (Fixed)

  6. NFSv4 referral. (Client side issue, fixed in latest kernels)

  7. Callback channels using RDMA clnt. (Not fixed)

  8. ib verbs apis ibv_post_send. (Are there any issues with ib verbs apis?)

  9. Srq changes. (Not fixed)

  10. Single completion queue manager thread for all connections. (Not fixed)

  11. Per connection buffer management (Nit fixed)


We are planning to discuss it in a Ganesha community call (Tuesday, April 23rd 2024, 7.30 PM - 8.30 PM IST)
Looking for any suggestions/feedback if anyone has tried it or has expertise in this area.

Regards,
Gaurav Gangalwar
_______________________________________________
Devel mailing list -- devel@lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org