Hello everyone,

Frequent memory crashs have been occurring for few weeks in the nfs-ganesha CEA FSAL-PROXY continuous integration test. I finally make time for looking at these problems today by running the nfs-ganesha server under Address Sanitizer.

I got the following stack wih a double-free error. Could anyone explain this error ? Someone who well understand the dup-req cache ? Or someone who already works with the code of the nfs4_op_test_stateid operation ?

The nfs4_op_test_stateid was introduce this summer by gerrit patch 418826 from
fatih-acar, 07/22/2018.

The dup-req cache stack seems to be involved in this error.

Regards,

Patrice

==7037==ERROR: AddressSanitizer: attempting double-free on 0x60200001ced0 in thread T7:
    #0 0x480c09 in __interceptor_free (/usr/bin/ganesha.nfsd+0x480c09)
    #1 0x897125 in gsh_free /opt/nfs-ganesha/src/include/abstract_mem.h:299
    #2 0x896f88 in nfs4_op_test_stateid_Free /opt/nfs-ganesha/src/Protocols/NFS/nfs4_op_test_stateid.c:121
    #3 0x703702 in nfs4_Compound_FreeOne /opt/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:1081
    #4 0x7042c4 in nfs4_Compound_Free /opt/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:1119
    #5 0x865c4a in nfs4_op_sequence /opt/nfs-ganesha/src/Protocols/NFS/nfs4_op_sequence.c:185
    #6 0x6fd80f in nfs4_Compound /opt/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:903
    #7 0x67167c in nfs_rpc_process_request /opt/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1329
    #8 0x663040 in nfs_rpc_valid_NFS /opt/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1539
    #9 0x7ffff7bb94a1 in svc_vc_decode /opt/nfs-ganesha/src/libntirpc/src/svc_vc.c:824
    #10 0x6542ce in nfs_rpc_decode_request /opt/nfs-ganesha/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1341
    #11 0x7ffff7bb934c in svc_vc_recv /opt/nfs-ganesha/src/libntirpc/src/svc_vc.c:797
    #12 0x7ffff7bb47be in svc_rqst_xprt_task /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:767
    #13 0x7ffff7bb51af in svc_rqst_epoll_events /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:939
    #14 0x7ffff7bb4e94 in svc_rqst_epoll_loop /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:1012:8
    #15 0x7ffff7bb38bf in svc_rqst_run_task /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:1048:14
    #16 0x7ffff7bc077c in work_pool_thread /opt/nfs-ganesha/src/libntirpc/src/work_pool.c:181
    #17 0x7ffff6367e24 in start_thread (/lib64/libpthread.so.0+0x7e24)
    #18 0x7ffff575c34c in __clone (/lib64/libc.so.6+0xf834c)

0x60200001ced0 is located 0 bytes inside of 4-byte region [0x60200001ced0,0x60200001ced4)
freed by thread T10 here:
    #0 0x480c09 in __interceptor_free (/usr/bin/ganesha.nfsd+0x480c09)
    #1 0x897125 in gsh_free /opt/nfs-ganesha/src/include/abstract_mem.h:299
    #2 0x896f88 in nfs4_op_test_stateid_Free /opt/nfs-ganesha/src/Protocols/NFS/nfs4_op_test_stateid.c:121
    #3 0x703702 in nfs4_Compound_FreeOne /opt/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:1081
    #4 0x7042c4 in nfs4_Compound_Free /opt/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:1119
    #5 0xcec2a4 in nfs_dupreq_rele /opt/nfs-ganesha/src/RPCAL/nfs_dupreq.c:1315
    #6 0x673196 in nfs_rpc_process_request /opt/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1442
    #7 0x663040 in nfs_rpc_valid_NFS /opt/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1539
    #8 0x7ffff7bb94a1 in svc_vc_decode /opt/nfs-ganesha/src/libntirpc/src/svc_vc.c:824
    #9 0x6542ce in nfs_rpc_decode_request /opt/nfs-ganesha/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1341
    #10 0x7ffff7bb934c in svc_vc_recv /opt/nfs-ganesha/src/libntirpc/src/svc_vc.c:797
    #11 0x7ffff7bb47be in svc_rqst_xprt_task /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:767
    #12 0x7ffff7bb51af in svc_rqst_epoll_events /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:939
    #13 0x7ffff7bb4e94 in svc_rqst_epoll_loop /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:1012:8
    #14 0x7ffff7bb38bf in svc_rqst_run_task /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:1048:14
    #15 0x7ffff7bc077c in work_pool_thread /opt/nfs-ganesha/src/libntirpc/src/work_pool.c:181
    #16 0x7ffff6367e24 in start_thread (/lib64/libpthread.so.0+0x7e24)

previously allocated by thread T10 here:
    #0 0x480e59 in calloc (/usr/bin/ganesha.nfsd+0x480e59)
    #1 0x89689a in gsh_calloc__ /opt/nfs-ganesha/src/include/abstract_mem.h:145
    #2 0x895c4e in nfs4_op_test_stateid /opt/nfs-ganesha/src/Protocols/NFS/nfs4_op_test_stateid.c:88:3
    #3 0x6fd80f in nfs4_Compound /opt/nfs-ganesha/src/Protocols/NFS/nfs4_Compound.c:903
    #4 0x67167c in nfs_rpc_process_request /opt/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1329
    #5 0x663040 in nfs_rpc_valid_NFS /opt/nfs-ganesha/src/MainNFSD/nfs_worker_thread.c:1539
    #6 0x7ffff7bb94a1 in svc_vc_decode /opt/nfs-ganesha/src/libntirpc/src/svc_vc.c:824
    #7 0x6542ce in nfs_rpc_decode_request /opt/nfs-ganesha/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1341
    #8 0x7ffff7bb934c in svc_vc_recv /opt/nfs-ganesha/src/libntirpc/src/svc_vc.c:797
    #9 0x7ffff7bb47be in svc_rqst_xprt_task /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:767
    #10 0x7ffff7bb51af in svc_rqst_epoll_events /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:939
    #11 0x7ffff7bb4e94 in svc_rqst_epoll_loop /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:1012:8
    #12 0x7ffff7bb38bf in svc_rqst_run_task /opt/nfs-ganesha/src/libntirpc/src/svc_rqst.c:1048:14
    #13 0x7ffff7bc077c in work_pool_thread /opt/nfs-ganesha/src/libntirpc/src/work_pool.c:181
    #14 0x7ffff6367e24 in start_thread (/lib64/libpthread.so.0+0x7e24)