Hi All
I'm running 2.7.3 using the CEPH FSAL to export CephFS (Luminous), it ran
well for a few days and crashed. I have a coredump, could someone assist me
in debugging this please?
(gdb) bt
#0 0x00007f04dcab6207 in raise () from /lib64/libc.so.6
#1 0x00007f04dcab78f8 in abort () from /lib64/libc.so.6
#2 0x00007f04d2a9d6c5 in ceph::__ceph_assert_fail(char const*, char
const*, int, char const*) () from /usr/lib64/ceph/libceph-common.so.0
#3 0x00007f04d2a9d844 in ceph::__ceph_assert_fail(ceph::assert_data
const&) () from /usr/lib64/ceph/libceph-common.so.0
#4 0x00007f04cc807f04 in Client::_lookup_name(Inode*, Inode*, UserPerm
const&) () from /lib64/libcephfs.so.2
#5 0x00007f04cc81c41f in Client::ll_lookup_inode(inodeno_t, UserPerm
const&, Inode**) () from /lib64/libcephfs.so.2
#6 0x00007f04ccadbf0e in create_handle (export_pub=0x1baff10,
desc=<optimized out>, pub_handle=0x7f0470fd4718, attrs_out=0x7f0470fd4740)
at /usr/src/debug/nfs-ganesha-2.7.3/FSAL/FSAL_CEPH/export.c:256
#7 0x0000000000523895 in mdcache_locate_host (fh_desc=0x7f0470fd4920,
export=export@entry=0x1bafbf0, entry=entry@entry=0x7f0470fd48b8,
attrs_out=attrs_out@entry=0x0)
at
/usr/src/debug/nfs-ganesha-2.7.3/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1011
#8 0x000000000051d278 in mdcache_create_handle (exp_hdl=0x1bafbf0,
fh_desc=<optimized out>, handle=0x7f0470fd4900, attrs_out=0x0) at
/usr/src/debug/nfs-ganesha-2.7.3/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:1578
#9 0x000000000046d404 in nfs4_mds_putfh (data=data@entry=0x7f0470fd4ea0)
at /usr/src/debug/nfs-ganesha-2.7.3/Protocols/NFS/nfs4_op_putfh.c:211
#10 0x000000000046d8e8 in nfs4_op_putfh (op=0x7f03effaf1d0,
data=0x7f0470fd4ea0, resp=0x7f03ec1de1f0) at
/usr/src/debug/nfs-ganesha-2.7.3/Protocols/NFS/nfs4_op_putfh.c:281
#11 0x000000000045d120 in nfs4_Compound (arg=<optimized out>,
req=<optimized out>, res=0x7f03ec1de9d0) at
/usr/src/debug/nfs-ganesha-2.7.3/Protocols/NFS/nfs4_Compound.c:942
#12 0x00000000004512cd in nfs_rpc_process_request (reqdata=0x7f03ee5ed4b0)
at /usr/src/debug/nfs-ganesha-2.7.3/MainNFSD/nfs_worker_thread.c:1328
#13 0x0000000000450766 in nfs_rpc_decode_request (xprt=0x7f02180c2320,
xdrs=0x7f03ec568ab0) at
/usr/src/debug/nfs-ganesha-2.7.3/MainNFSD/nfs_rpc_dispatcher_thread.c:1345
#14 0x00007f04df45d07d in svc_rqst_xprt_task (wpe=0x7f02180c2538) at
/usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:769
#15 0x00007f04df45d59a in svc_rqst_epoll_events (n_events=<optimized out>,
sr_rec=0x4bb53e0) at
/usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:941
#16 svc_rqst_epoll_loop (sr_rec=<optimized out>) at
/usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:1014
#17 svc_rqst_run_task (wpe=0x4bb53e0) at
/usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/svc_rqst.c:1050
#18 0x00007f04df465123 in work_pool_thread (arg=0x7f044c0008c0) at
/usr/src/debug/nfs-ganesha-2.7.3/libntirpc/src/work_pool.c:181
#19 0x00007f04dda05dd5 in start_thread () from /lib64/libpthread.so.0
#20 0x00007f04dcb7dead in clone () from /lib64/libc.so.6
Package versions:
nfs-ganesha-2.7.3-0.1.el7.x86_64
nfs-ganesha-ceph-2.7.3-0.1.el7.x86_64
libcephfs2-14.2.1-0.el7.x86_64
librados2-14.2.1-0.el7.x86_64
I notice in my Ceph log I have a bunch of slow requests around the time it
went down, I'm not sure if it's a symptom of Ganesha segfaulting or if it
was a contributing factor.
Thanks,
David
