12:27 p.m.
We don't have any current known double-free issues. Do you have a
reproducer? Can you run with ASAN, which will tell you where the memory
was allocated and freed?
Ultimately, these kinds of issues can be tricky to debug; I personally
use the ref/deref LTTng tracepoints to help me track them down.
Daniel
On 07/16/2018 09:55 AM, Fatih Acar wrote:
Hello,
We hit some crashes while using nfs-ganesha.
From my understanding, it looks like there are some races where an entry
is somehow free'd twice.
It mostly happen when MDCACHE is in extremis mode.
Is it a known issue that would be fixed in next? I couldn't find
anything about that.
Thanks.
You can find backtraces below:
nfs-ganesha 2.6.2
#0 rwlock_wrlock_common (rwlock=0xb0, abstime=0x0) at
/usr/local/poudriere/jails/filer_stable_11/usr/src/lib/libthr/thread/thr_rwlock.c:58
58
/usr/local/poudriere/jails/filer_stable_11/usr/src/lib/libthr/thread/thr_rwlock.c:
No such file or directory.
in
/usr/local/poudriere/jails/filer_stable_11/usr/src/lib/libthr/thread/thr_rwlock.c
(gdb) bt
#0 rwlock_wrlock_common (rwlock=0xb0, abstime=0x0) at
/usr/local/poudriere/jails/filer_stable_11/usr/src/lib/libthr/thread/thr_rwlock.c:58
#1 0x000000000041b75e in fsal_obj_handle_fini (obj=0xb02534408) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/commonlib.c:192
#2 0x00000000004d1b13 in mdcache_lru_clean (entry=0xb02534400) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:589
#3 0x00000000004d2467 in _mdcache_lru_unref (entry=0xb02534400,
flags=<value optimized out>, func=<value optimized out>, line=<value
optimized out>) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1915
#4 0x00000000004d31e4 in lru_reap_impl (qid=LRU_ENTRY_L1) at
mdcache_hash.h:238
#5 0x00000000004d166c in mdcache_lru_get () at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:717
#6 0x00000000004dc66e in mdcache_new_entry (export=0x802b19b80,
sub_handle=0x9a4af9100, attrs_in=0x7fffdbdc43f8, attrs_out=0x0,
new_directory=false, entry=0x7fffdbdc4550, state=0x0)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:143
#7 0x00000000004dde9f in mdcache_locate_host (fh_desc=0x7fffdbdc45b0,
export=<value optimized out>, entry=0x7fffdbdc4550, attrs_out=0x0) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1000
#8 0x00000000004d9219 in mdcache_create_handle (exp_hdl=0x802b19b80,
fh_desc=<value optimized out>, handle=0x7fffdbdc4590, attrs_out=0x0) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1757
#9 0x000000000045ba9c in nfs4_op_putfh (op=<value optimized out>,
data=0x7fffdbdc4a50, resp=0x8b3075de0) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/Protocols/NFS/nfs4_op_putfh.c:211
#10 0x000000000044b5e4 in nfs4_Compound (arg=<value optimized out>,
req=<value optimized out>, res=<value optimized out>) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/Protocols/NFS/nfs4_Compound.c:752
#11 0x000000000043db17 in nfs_rpc_process_request (reqdata=<value
optimized out>) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/MainNFSD/nfs_worker_thread.c:1329
#12 0x000000000043c98b in nfs_rpc_valid_NFS (req=0x822426000) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/MainNFSD/nfs_worker_thread.c:1554
#13 0x0000000800974a66 in svc_vc_decode (req=0x822426000) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/libntirpc/src/svc_vc.c:815
#14 0x000000000043c418 in nfs_rpc_decode_request (xprt=0x80b919180,
xdrs=0xa1e7b86c0) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1317
#15 0x0000000800973f54 in svc_vc_recv (xprt=<value optimized out>) at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/libntirpc/src/svc_vc.c:788
#16 0x0000000800972051 in svc_rqst_xprt_task (wpe=<value optimized out>)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/libntirpc/src/svc_rqst.c:750
#17 0x000000080097a2c2 in work_pool_thread (arg=<value optimized out>)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.2.0/src/libntirpc/src/work_pool.c:176
#18 0x000000080159bc06 in thread_start (curthread=0x81ee4ad00) at
/usr/local/poudriere/jails/filer_stable_11/usr/src/lib/libthr/thread/thr_create.c:289
#19 0x0000000000000000 in ?? ()
nfs-ganesha 2.6.1
0 rwlock_wrlock_common (rwlock=0xb0, abstime=0x0)
at
/usr/local/poudriere/jails/filer_stable_11/usr/src/lib/libthr/thread/thr_rwlock.c:58
1 0x000000000041b75e in fsal_obj_handle_fini (obj=0xa6fb74808)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.1.2/src/FSAL/commonlib.c:192
2 0x00000000004d0f5f in mdcache_lru_clean (entry=0xa6fb74800)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.1.2/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:589
3 0x00000000004d1907 in _mdcache_lru_unref (entry=0xa6fb74800,
flags=<value optimized out>, func=<value optimized out>,
line=<value optimized out>)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.1.2/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:1907
4 0x00000000004dbfbc in mdcache_new_entry (export=<value optimized
out>, sub_handle=0xa65d7a500, attrs_in=0x7fffdcbd1388,
attrs_out=0x0, new_directory=false, entry=0x7fffdcbd14e0,
state=0x0) at mdcache_lru.h:190
5 0x00000000004dd23f in mdcache_locate_host
(fh_desc=0x7fffdcbd1540, export=<value optimized out>, entry=0x7fffdcbd14e0,
attrs_out=0x0)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.1.2/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:999
6 0x00000000004d8599 in mdcache_create_handle (exp_hdl=0x802902080,
fh_desc=<value optimized out>, handle=0x7fffdcbd1520,
attrs_out=0x0)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.1.2/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1757
7 0x000000000045b40c in nfs4_op_putfh (op=<value optimized out>,
data=0x7fffdcbd19e0, resp=0xa8abb94e0)
at
/wrkdirs/usr/ports/gandi/ganesha/work/ganesha-2.6.1.2/src/Protocols/NFS/nfs4_op_putfh.c:211