[NFS-Ganesha-Devel] Performance scalability issues - linear drops in IOPs when adding clients

Hello, We are currently testing an all-flash storage with FIO. Our infrastructure is as follows: A two node all-flash storage which is connected to two GPFS servers (version 5.1.4-1) via point-to-point 16Gb FC connection (4 paths per server). Three ESXi hosts (v 7.0 U3), each running four NFS clients (NFS v4.0), which are connected to the GPFS servers using 100GbE connection via a high-end Ethernet switch with support of jumbo frames. The GPFS servers are using 2x 100 GbE LACP to the Ethernet switch, ESX servers are using 1x 100GbE connection. Connection throughput was tested with iperf and works well. The GPFS servers are exporting the following share: /gpfs/gpfs22/nfs1 *(rw,no_root_squash,async,fsid=402) The NFS clients are mounting the share with the following options: 192.168.5.202:/gpfs/gpfs22/nfs1 on /mnt/nfs1 type nfs4 (rw,noatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,nconnect=8,timeo=600,retrans=2,sec=sys,clientaddr=192.168.5.54,local_lock=none,addr=192.168.5.202) We are experiencing issues with scaling - performance does not scale with added clients We always hit a ceiling of around 165k IOPs regardless of the number of NFS clients. It seems that the performance is directly impacted by the number of NFS servers. We've run tests utilizing 12, 6, 4 and 2 clients respectively. When running the tests with only two clients, we are reaching 82,5k (8k, 70/30 r/w) IOPs per client. However, when running the tests with 4 clients, we are only reaching roughly a half. The performance drops linearly with added clients. We've looked around the archives and found a topic which deals with scalability issues. The following statement seems close to what we are experiencing. That would explain the observation that a single thread is blocked in writev() whenever I look. It also explains why when clients are bandwidth-limited, the aggregate throughput does not scale much above that consumed by a single client. Could it be that we are running into a similar issue? Is there a tunable that could help us with scaling? The thread is quite old. We are running a newer version of NFS-Ganesha: # ganesha_mgr show version NFS-Ganesha Release = V3.5-ibm071.16 ganesha compiled on Jul 11 2022 at 15:28:41 Release comment = GANESHA file server is 64 bits compliant and supports NFS v3,4.0,4.1 (pNFS) and 9P Git HEAD = NOT-GIT Git Describe = /home/ppsbld/workspace/SS_5.1.4/Build_Prod/Ganesha_5.1.4_ProdBuild/label/BLDLNX125_X86_64_REDHAT_8.0.0/ganesha-rpmdir/rpmbuild/BUILD/nfs-ganesha-3.5-ibm071.16 When tested locally, the performance of the storage was around 400k IOPs. We are at our wits end and would really appreciate any insight. Thank you. Regards, Jakub