deeraj.patil@ibm.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1253882?usp=email ) Change subject: io_uring infra + dependencies ...................................................................... io_uring infra + dependencies Uring engine interface core. Clean FSAL biffurcations, fallback for non-implemented uring nfsv4_ops by FSAL's Batch opartion support. Mdcache handled. Protocol ops coalesec. Internal buffer pool, to resue for read 4k-1M (buffpool configurable size) FSAL ops specific personalities setting per sge Same engine can be entended for future netwrok uring. What all planned : 1. This is about reducing the systemcall, by which reducing the contextswitches. 2. FSAL's which supports all the OP_* via thier own uring interface, can get max out of the uring because of full compound coalesec. my_config : NFS_CORE_PARAM { Enable_Uring_Compound = true; Uring_Inline_Poll = 1; } VFS_URING { io_uring_queue_depth = 64; # SQ entries per ring (range 8–4096) io_uring = true; # without this it is byte-for-byte FSAL_VFS io_uring_rings = 2; #io_uring_per_cpu_rings = true; io_uring_buffer_min = 4096; # smallest size class in bytes (default 16 KiB) io_uring_buffer_max = 1048576; # largest size class in bytes (default 1 MiB) io_uring_pool_mb = 256; # total pinned budget in MiB (range 1–65536) } FSAL { #Name = VFS; Name = VFS_URING; } Without Uring dppvm2:/home/dpp/workspace# sleep 10 && fio --name=32K --filename=/mnt2/testfile --ioengine=libaio --iodepth=64 --direct=1 --bs=32K --readwrite=randread --runtime=60 --time_based --group_reporting --size=4GiB32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64 fio-3.35 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=705MiB/s][r=22.5k IOPS][eta 00m:00s] 32K: (groupid=0, jobs=1): err= 0: pid=1161340: Fri Oct 9 09:18:10 2026 read: IOPS=20.1k, BW=628MiB/s (659MB/s)(36.8GiB/60003msec) slat (usec): min=2, max=727, avg= 7.26, stdev= 2.66 clat (usec): min=1513, max=11036, avg=3175.45, stdev=498.76 lat (usec): min=1521, max=11045, avg=3182.71, stdev=498.79 clat percentiles (usec): | 1.00th=[ 2147], 5.00th=[ 2343], 10.00th=[ 2474], 20.00th=[ 2704], | 30.00th=[ 2900], 40.00th=[ 3064], 50.00th=[ 3228], 60.00th=[ 3359], | 70.00th=[ 3458], 80.00th=[ 3621], 90.00th=[ 3785], 95.00th=[ 3916], | 99.00th=[ 4228], 99.50th=[ 4359], 99.90th=[ 4948], 99.95th=[ 5407], | 99.99th=[ 8094] bw ( KiB/s): min=525248, max=894656, per=99.86%, avg=642452.97, stdev=89899.40, samples=119 iops : min=16414, max=27958, avg=20076.66, stdev=2809.36, samples=119 lat (msec) : 2=0.23%, 4=96.46%, 10=3.30%, 20=0.01% cpu : usr=4.27%, sys=17.82%, ctx=1166898, majf=0, minf=521 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1206287,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=628MiB/s (659MB/s), 628MiB/s-628MiB/s (659MB/s-659MB/s), io=36.8GiB (39.5GB), run=60003-60003msec with URING and depended changes: dppvm2:/home/dpp/workspace# sleep 10 && fio --name=32K --filename=/mnt2/testfile --ioengine=libaio --iodepth=64 --direct=1 --bs=32K --readwrite=randread --runtime=60 --time_based --group_reporting --size=4GiB32K: (g=0): rw=randread, bs=(R) 32.0KiB-32.0KiB, (W) 32.0KiB-32.0KiB, (T) 32.0KiB-32.0KiB, ioengine=libaio, iodepth=64 fio-3.35 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=947MiB/s][r=30.3k IOPS][eta 00m:00s] 32K: (groupid=0, jobs=1): err= 0: pid=1163173: Fri Oct 9 09:20:06 2026 read: IOPS=30.5k, BW=952MiB/s (998MB/s)(55.8GiB/60003msec) slat (usec): min=2, max=132, avg= 7.24, stdev= 2.30 clat (usec): min=171, max=23312, avg=2093.64, stdev=377.61 lat (usec): min=181, max=23329, avg=2100.87, stdev=377.69 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1844], 10.00th=[ 1893], 20.00th=[ 1958], | 30.00th=[ 2008], 40.00th=[ 2040], 50.00th=[ 2073], 60.00th=[ 2114], | 70.00th=[ 2147], 80.00th=[ 2180], 90.00th=[ 2245], 95.00th=[ 2343], | 99.00th=[ 2835], 99.50th=[ 3884], 99.90th=[ 6849], 99.95th=[ 9110], | 99.99th=[14877] bw ( KiB/s): min=417472, max=1009088, per=100.00%, avg=974890.49, stdev=55488.51, samples=119 iops : min=13046, max=31534, avg=30465.33, stdev=1734.02, samples=119 lat (usec) : 250=0.01%, 500=0.02%, 750=0.03%, 1000=0.03% lat (msec) : 2=29.55%, 4=69.89%, 10=0.44%, 20=0.04%, 50=0.01% cpu : usr=6.20%, sys=26.24%, ctx=1776303, majf=0, minf=520 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=1827220,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=952MiB/s (998MB/s), 952MiB/s-952MiB/s (998MB/s-998MB/s), io=55.8GiB (59.9GB), run=60003-60003msec Note : Needs to gothrough write path, no perf gains yet now. Change-Id: I7f81c972a54d10d0ec3586e8630ff8d150ec58b7 Signed-off-by: dpp <deeraj.patil@ibm.com> --- M src/CMakeLists.txt M src/FSAL/CMakeLists.txt A src/FSAL/FSAL_URING/CMakeLists.txt A src/FSAL/FSAL_URING/fsal_uring_init.c A src/FSAL/FSAL_URING/uring_pers.c A src/FSAL/FSAL_URING/uring_pers.h M src/FSAL/FSAL_VFS/file.c M src/FSAL/FSAL_VFS/vfs/main-c.in.cmake M src/FSAL/FSAL_VFS/vfs_methods.h A src/FSAL/FSAL_VFS_URING/CMakeLists.txt A src/FSAL/FSAL_VFS_URING/main.c A src/FSAL/FSAL_VFS_URING/vfs_uring_fd.c A src/FSAL/FSAL_VFS_URING/vfs_uring_fops.c A src/FSAL/FSAL_VFS_URING/vfs_uring_io.c A src/FSAL/FSAL_VFS_URING/vfs_uring_meta.c A src/FSAL/FSAL_VFS_URING/vfs_uring_methods.h A src/FSAL/FSAL_VFS_URING/vfs_uring_rw.c M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/CMakeLists.txt M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_int.h A src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_uring.c M src/FSAL/fsal_config.c A src/FSAL/fsal_uring.c M src/MainNFSD/CMakeLists.txt M src/MainNFSD/libganesha_nfsd.ver M src/MainNFSD/nfs_admin_thread.c M src/MainNFSD/nfs_init.c M src/Protocols/CMakeLists.txt M src/Protocols/NFS/nfs4_Compound.c A src/Protocols/NFS_URING/CMakeLists.txt A src/Protocols/NFS_URING/nfs4_Compound_uring.c A src/Protocols/NFS_URING/nfs4_op_access_uring.c A src/Protocols/NFS_URING/nfs4_op_allocate_uring.c A src/Protocols/NFS_URING/nfs4_op_commit_uring.c A src/Protocols/NFS_URING/nfs4_op_getattr_uring.c A src/Protocols/NFS_URING/nfs4_op_read_uring.c A src/Protocols/NFS_URING/nfs4_op_write_uring.c A src/Protocols/NFS_URING/nfs4_uring_internal.h A src/Protocols/NFS_URING/nfs4_uring_optab.c A src/Protocols/NFS_URING/nfs4_uring_resume_pool.c M src/doc/man/ganesha-log-config.rst A src/include/FSAL/fsal_uring.h A src/include/FSAL/fsal_uring_fops.h A src/include/FSAL/fsal_uring_meta.h M src/include/config-h.in.cmake M src/include/fsal_api.h M src/include/fsal_types.h M src/include/gsh_config.h A src/include/io_uring/uring_engine.h M src/include/log_common.h M src/include/nfs_proto_data.h M src/include/nfs_proto_functions.h A src/io_uring/CMakeLists.txt A src/io_uring/core/CMakeLists.txt A src/io_uring/core/uring_engine.c A src/io_uring/core/uring_engine.h M src/log/log_functions.c M src/support/nfs_creds.c M src/support/nfs_read_conf.c 58 files changed, 15,567 insertions(+), 14 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/82/1253882/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/1253882?usp=email To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings?usp=email Gerrit-MessageType: newchange Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I7f81c972a54d10d0ec3586e8630ff8d150ec58b7 Gerrit-Change-Number: 1253882 Gerrit-PatchSet: 1 Gerrit-Owner: deeraj.patil@ibm.com