The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
system version: centos8 nfs-ganesha version: 3.5-3 log: Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4. Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648 bytes. Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0 dumped core. Stack trace of thread 1066334: #0 0x00007fe484c6e37f n/a (n/a) #1 0x000000003789c20b n/a (n/a)
Not enough here to go on. You need to: 1) configure abrt to not truncate the core file at 2G 2) install the debuginfo rpms 3) check /var/log/messages to see if it is being oom killed. On Mon, Aug 30, 2021 at 4:05 AM Desert <chinafangxq@gmail.com> wrote:
system version: centos8 nfs-ganesha version: 3.5-3
log: Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4. Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648 bytes. Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0 dumped core.
Stack trace of thread 1066334: #0 0x00007fe484c6e37f n/a (n/a) #1 0x000000003789c20b n/a (n/a) _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
-- Kaleb
Thank you very much! 1) Core file has no limit on size; 2) There is no trip OOM in the system log, GDB and log are below ============================================================================== GDB: #0 0x00007F92b8F5237f In raise () from /lib64/libc.so.6 #1 0x00007F92b8f3CDB5 in abort () from /lib64/libc.so.6 #2 0x00007f92a6cc4b75 in ceph::__ceph_assert_fail(char const, char const, int, char const*) () from /usr/lib64/ceph/libceph-common.so.2 #3 0x00007f92a6CC4d3e in ceph::__ceph_assert_fail(ceph::assert_data const&) () from /usr/lib64/ceph/ libceph-common.so2 # 4 0 x00007f92afa34627 operator< in; < (std::ostream&, Inode const&) () from /lib64/libcephfs.so.2 #5 0x00007f92AF9f85ae in Client::ll_sync_inode(Inode*, bool) () from /lib64/ libcephf.so # 6 0 x00007f92afd42a15 ceph_fsal_setattr2 in () the from/usr/lib64 / ganesha libfsalceph. So #7 0x00007f92bb495B9f in mdcache_setattr2 () from /lib64/libganesha_nfsd.so.3.5 In fsal_setattr () from /lib64/libganesha_nfsd.so.3.5 #9 0x00007f92bb469d17 in nfs4_op_setattr () from /lib64/ libganesha_nfsd.so3.5 #10 0x00007F92bb44d78f in process_one_op () from /lib64/libganesha_nfsd.so.3.5 #11 0x00007F92bb44e927 in nfs4_Compound () from /lib64/libganesha_nfsd.so.3.5 #12 0x00007f92bb3Cec46 in nfs_rpc_process_Request () from /lib64/libganesha_nfsd.so.3.5 In svc_Request () from /lib64/libntirpc.so.3.5 #14 0x00007f92bb159bf9 in svc_rqst_xprt_task_recv () from /lib64/ libntirpc.so3.5 #15 0x00007f92bb15a5d8 in svc_rqst_epoll_loop () from /lib64/libntirpc.so.3.5 #16 0x00007f92bb16565d in work_pool_thread () from /lib64/libntirpc.so.3.5 #17 0x00007f92b96f814a in start_thread () from /lib64/libpthread.so.0 #18 0x00007f92b9017dc3 in clone () from /lib64/libc.so.6 ============================================================================== System logs: Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4. Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648 bytes. Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0 dumped core. Stack trace of thread 1066334: #0 0x00007fe484c6e37f n/a (n/a) #1 0x000000003789c20b n/a (n/a)
I think something is wrong with your setup, either the libraries or the debuginfo. ceph_fsal_setattr2() doesn't call ceph_ll_sync_inode(), it calls ceph_ll_setattr(). My guess is that the debuginfo version doesn't match the library version for libceph. Daniel On 8/30/21 10:14 PM, Desert wrote:
Thank you very much! 1) Core file has no limit on size; 2) There is no trip OOM in the system log, GDB and log are below
============================================================================== GDB: #0 0x00007F92b8F5237f In raise () from /lib64/libc.so.6 #1 0x00007F92b8f3CDB5 in abort () from /lib64/libc.so.6 #2 0x00007f92a6cc4b75 in ceph::__ceph_assert_fail(char const, char const, int, char const*) () from /usr/lib64/ceph/libceph-common.so.2 #3 0x00007f92a6CC4d3e in ceph::__ceph_assert_fail(ceph::assert_data const&) () from /usr/lib64/ceph/ libceph-common.so2 # 4 0 x00007f92afa34627 operator< in; < (std::ostream&, Inode const&) () from /lib64/libcephfs.so.2 #5 0x00007f92AF9f85ae in Client::ll_sync_inode(Inode*, bool) () from /lib64/ libcephf.so # 6 0 x00007f92afd42a15 ceph_fsal_setattr2 in () the from/usr/lib64 / ganesha libfsalceph. So #7 0x00007f92bb495B9f in mdcache_setattr2 () from /lib64/libganesha_nfsd.so.3.5 In fsal_setattr () from /lib64/libganesha_nfsd.so.3.5 #9 0x00007f92bb469d17 in nfs4_op_setattr () from /lib64/ libganesha_nfsd.so3.5 #10 0x00007F92bb44d78f in process_one_op () from /lib64/libganesha_nfsd.so.3.5 #11 0x00007F92bb44e927 in nfs4_Compound () from /lib64/libganesha_nfsd.so.3.5 #12 0x00007f92bb3Cec46 in nfs_rpc_process_Request () from /lib64/libganesha_nfsd.so.3.5 In svc_Request () from /lib64/libntirpc.so.3.5 #14 0x00007f92bb159bf9 in svc_rqst_xprt_task_recv () from /lib64/ libntirpc.so3.5 #15 0x00007f92bb15a5d8 in svc_rqst_epoll_loop () from /lib64/libntirpc.so.3.5 #16 0x00007f92bb16565d in work_pool_thread () from /lib64/libntirpc.so.3.5 #17 0x00007f92b96f814a in start_thread () from /lib64/libpthread.so.0 #18 0x00007f92b9017dc3 in clone () from /lib64/libc.so.6
============================================================================== System logs: Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4. Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648 bytes. Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0 dumped core.
Stack trace of thread 1066334: #0 0x00007fe484c6e37f n/a (n/a) #1 0x000000003789c20b n/a (n/a) _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
participants (3)
-
Daniel Gryniewicz -
Desert -
Kaleb Keithley