6:53 p.m.
From Dominique Martinet <asmadeus(a)codewreck.org>:
Dominique Martinet has uploaded this change for review. (
https://review.gerrithub.io/421314
Change subject: RFC: 9p/tcp: add support for KCM (kernel connection multiplexor)
......................................................................
RFC: 9p/tcp: add support for KCM (kernel connection multiplexor)
KCM is a framework that lets the application specify how messages are
constructed in a BPF program, so that the kernel can split messages at
proper length for us.
The main advantage of this is that it saves us one user-space <-> kernel
round-trip of read(hdr) then read(rest-of-the-size), as a single
recvmsg() will always give us a proper message
Ultimately, KCM also has other advantages as we could attach multiple
tcp sockets on it and have worker threads read from kcm "clones" that do
load-balancing instead of our silly one-thread-per-client system that
has been bothering me forever, but just this has advantages.
A quick benchmark with ior on a different client with very small
messages to stress the packet processing shows:
$ mpirun -n 32 ./ior -F -o /mnt/ior_out -t 32 -b 128k -i 10
...
with KCM enabled:
write 1.47 1.30 1.35 0.05 2.96137 0 32 32 10 1 0 1 0 0 1
131072 32 4194304 POSIX 0
read 1.43 1.28 1.37 0.04 2.92899 0 32 32 10 1 0 1 0 0 1
131072 32 4194304 POSIX 0
-> taking the best, 4k round-trips in 2.93s = 1.4k req/s
with KCM disabled:
write 1.32 1.23 1.27 0.03 3.15889 0 32 32 10 1 0 1 0 0 1
131072 32 4194304 POSIX 0
read 1.28 1.17 1.22 0.04 3.27371 0 32 32 10 1 0 1 0 0 1
131072 32 4194304 POSIX 0
-> taking the best, 4k round-trips in 3.15s = 1.3k req/s
(There is a lot of noise and my test is too short/only 10 iterations,
but that's about 7% difference!)
That'd be very interesting to try in ntirpc for NFS
A few notes:
- KCM is currently broken with small messages that get aggregated by
TCP, I have a patch here under construction:
https://lkml.kernel.org/m/20180805233921.GA5773@nautica
- KCM is disabled by default in most distros anyway...
- Even with this patch big messages seem to cause a problem, I need to
look into that.
- This patch does not check for libbcc properly in cmake
Change-Id: Idf040f311d6ca1a30c34bc6e9442073f493b0a52
Signed-off-by: Dominique Martinet <dominique.martinet(a)cea.fr>
---
M src/CMakeLists.txt
M src/MainNFSD/9p_dispatcher.c
M src/MainNFSD/CMakeLists.txt
M src/include/config-h.in.cmake
4 files changed, 82 insertions(+), 0 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/14/421314/1
--
To view, visit https://review.gerrithub.io/421314
To unsubscribe, or for help writing mail filters, visit
https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idf040f311d6ca1a30c34bc6e9442073f493b0a52
Gerrit-Change-Number: 421314
Gerrit-PatchSet: 1
Gerrit-Owner: Dominique Martinet <asmadeus(a)codewreck.org>