From Dominique Martinet <asmadeus@codewreck.org>: Dominique Martinet has uploaded this change for review. ( https://review.gerrithub.io/421314 Change subject: RFC: 9p/tcp: add support for KCM (kernel connection multiplexor) ...................................................................... RFC: 9p/tcp: add support for KCM (kernel connection multiplexor) KCM is a framework that lets the application specify how messages are constructed in a BPF program, so that the kernel can split messages at proper length for us. The main advantage of this is that it saves us one user-space <-> kernel round-trip of read(hdr) then read(rest-of-the-size), as a single recvmsg() will always give us a proper message Ultimately, KCM also has other advantages as we could attach multiple tcp sockets on it and have worker threads read from kcm "clones" that do load-balancing instead of our silly one-thread-per-client system that has been bothering me forever, but just this has advantages. A quick benchmark with ior on a different client with very small messages to stress the packet processing shows: $ mpirun -n 32 ./ior -F -o /mnt/ior_out -t 32 -b 128k -i 10 ... with KCM enabled: write 1.47 1.30 1.35 0.05 2.96137 0 32 32 10 1 0 1 0 0 1 131072 32 4194304 POSIX 0 read 1.43 1.28 1.37 0.04 2.92899 0 32 32 10 1 0 1 0 0 1 131072 32 4194304 POSIX 0 -> taking the best, 4k round-trips in 2.93s = 1.4k req/s with KCM disabled: write 1.32 1.23 1.27 0.03 3.15889 0 32 32 10 1 0 1 0 0 1 131072 32 4194304 POSIX 0 read 1.28 1.17 1.22 0.04 3.27371 0 32 32 10 1 0 1 0 0 1 131072 32 4194304 POSIX 0 -> taking the best, 4k round-trips in 3.15s = 1.3k req/s (There is a lot of noise and my test is too short/only 10 iterations, but that's about 7% difference!) That'd be very interesting to try in ntirpc for NFS A few notes: - KCM is currently broken with small messages that get aggregated by TCP, I have a patch here under construction: https://lkml.kernel.org/m/20180805233921.GA5773@nautica - KCM is disabled by default in most distros anyway... - Even with this patch big messages seem to cause a problem, I need to look into that. - This patch does not check for libbcc properly in cmake Change-Id: Idf040f311d6ca1a30c34bc6e9442073f493b0a52 Signed-off-by: Dominique Martinet <dominique.martinet@cea.fr> --- M src/CMakeLists.txt M src/MainNFSD/9p_dispatcher.c M src/MainNFSD/CMakeLists.txt M src/include/config-h.in.cmake 4 files changed, 82 insertions(+), 0 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/14/421314/1 -- To view, visit https://review.gerrithub.io/421314 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-MessageType: newchange Gerrit-Change-Id: Idf040f311d6ca1a30c34bc6e9442073f493b0a52 Gerrit-Change-Number: 421314 Gerrit-PatchSet: 1 Gerrit-Owner: Dominique Martinet <asmadeus@codewreck.org>