The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
1:03 a.m.
system version: centos8
nfs-ganesha version: 3.5-3
log:
Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump
core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4.
Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648
bytes.
Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0
dumped core.
Stack trace of thread 1066334:
#0 0x00007fe484c6e37f n/a
(n/a)
#1 0x000000003789c20b n/a
(n/a)
4:38 a.m.
New subject: The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
Not enough here to go on. You need to:
1) configure abrt to not truncate the core file at 2G
2) install the debuginfo rpms
3) check /var/log/messages to see if it is being oom killed.
On Mon, Aug 30, 2021 at 4:05 AM Desert <chinafangxq(a)gmail.com> wrote:
system version: centos8
nfs-ganesha version: 3.5-3
log:
Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump
core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4.
Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated
to 2147483648 bytes.
Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997
(ganesha.nfsd) of user 0 dumped core.
Stack trace of
thread 1066334:
#0
0x00007fe484c6e37f n/a (n/a)
#1
0x000000003789c20b n/a (n/a)
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
--
Kaleb
7:14 p.m.
New subject: The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
Thank you very much!
1) Core file has no limit on size;
2) There is no trip OOM in the system log, GDB and log are below
==============================================================================
GDB:
#0 0x00007F92b8F5237f In raise () from /lib64/libc.so.6
#1 0x00007F92b8f3CDB5 in abort () from /lib64/libc.so.6
#2 0x00007f92a6cc4b75 in ceph::__ceph_assert_fail(char const, char const, int, char
const*) () from /usr/lib64/ceph/libceph-common.so.2
#3 0x00007f92a6CC4d3e in ceph::__ceph_assert_fail(ceph::assert_data const&) () from
/usr/lib64/ceph/ libceph-common.so2
# 4 0 x00007f92afa34627 operator< in; < (std::ostream&, Inode const&)
() from /lib64/libcephfs.so.2
#5 0x00007f92AF9f85ae in Client::ll_sync_inode(Inode*, bool) () from /lib64/ libcephf.so
# 6 0 x00007f92afd42a15 ceph_fsal_setattr2 in () the from/usr/lib64 / ganesha libfsalceph.
So
#7 0x00007f92bb495B9f in mdcache_setattr2 () from /lib64/libganesha_nfsd.so.3.5
In fsal_setattr () from /lib64/libganesha_nfsd.so.3.5
#9 0x00007f92bb469d17 in nfs4_op_setattr () from /lib64/ libganesha_nfsd.so3.5
#10 0x00007F92bb44d78f in process_one_op () from /lib64/libganesha_nfsd.so.3.5
#11 0x00007F92bb44e927 in nfs4_Compound () from /lib64/libganesha_nfsd.so.3.5
#12 0x00007f92bb3Cec46 in nfs_rpc_process_Request () from /lib64/libganesha_nfsd.so.3.5
In svc_Request () from /lib64/libntirpc.so.3.5
#14 0x00007f92bb159bf9 in svc_rqst_xprt_task_recv () from /lib64/ libntirpc.so3.5
#15 0x00007f92bb15a5d8 in svc_rqst_epoll_loop () from /lib64/libntirpc.so.3.5
#16 0x00007f92bb16565d in work_pool_thread () from /lib64/libntirpc.so.3.5
#17 0x00007f92b96f814a in start_thread () from /lib64/libpthread.so.0
#18 0x00007f92b9017dc3 in clone () from /lib64/libc.so.6
==============================================================================
System logs:
Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump
core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4.
Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648
bytes.
Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0
dumped core.
Stack trace of thread 1066334:
#0 0x00007fe484c6e37f n/a
(n/a)
#1 0x000000003789c20b n/a
(n/a)
4:58 a.m.
New subject: The NFS-Ganesha service is frequently killed. Main process exited code= KILLED Status =6/ abRT is displayed
I think something is wrong with your setup, either the libraries or the
debuginfo. ceph_fsal_setattr2() doesn't call ceph_ll_sync_inode(), it
calls ceph_ll_setattr(). My guess is that the debuginfo version doesn't
match the library version for libceph.
Daniel
On 8/30/21 10:14 PM, Desert wrote:
Thank you very much!
1) Core file has no limit on size;
2) There is no trip OOM in the system log, GDB and log are below
==============================================================================
GDB:
#0 0x00007F92b8F5237f In raise () from /lib64/libc.so.6
#1 0x00007F92b8f3CDB5 in abort () from /lib64/libc.so.6
#2 0x00007f92a6cc4b75 in ceph::__ceph_assert_fail(char const, char const, int, char
const*) () from /usr/lib64/ceph/libceph-common.so.2
#3 0x00007f92a6CC4d3e in ceph::__ceph_assert_fail(ceph::assert_data const&) () from
/usr/lib64/ceph/ libceph-common.so2
# 4 0 x00007f92afa34627 operator< in; < (std::ostream&, Inode
const&) () from /lib64/libcephfs.so.2
#5 0x00007f92AF9f85ae in Client::ll_sync_inode(Inode*, bool) () from /lib64/ libcephf.so
# 6 0 x00007f92afd42a15 ceph_fsal_setattr2 in () the from/usr/lib64 / ganesha
libfsalceph. So
#7 0x00007f92bb495B9f in mdcache_setattr2 () from /lib64/libganesha_nfsd.so.3.5
In fsal_setattr () from /lib64/libganesha_nfsd.so.3.5
#9 0x00007f92bb469d17 in nfs4_op_setattr () from /lib64/ libganesha_nfsd.so3.5
#10 0x00007F92bb44d78f in process_one_op () from /lib64/libganesha_nfsd.so.3.5
#11 0x00007F92bb44e927 in nfs4_Compound () from /lib64/libganesha_nfsd.so.3.5
#12 0x00007f92bb3Cec46 in nfs_rpc_process_Request () from /lib64/libganesha_nfsd.so.3.5
In svc_Request () from /lib64/libntirpc.so.3.5
#14 0x00007f92bb159bf9 in svc_rqst_xprt_task_recv () from /lib64/ libntirpc.so3.5
#15 0x00007f92bb15a5d8 in svc_rqst_epoll_loop () from /lib64/libntirpc.so.3.5
#16 0x00007f92bb16565d in work_pool_thread () from /lib64/libntirpc.so.3.5
#17 0x00007f92b96f814a in start_thread () from /lib64/libpthread.so.0
#18 0x00007f92b9017dc3 in clone () from /lib64/libc.so.6
==============================================================================
System logs:
Aug 30 09:40:36 ceph1 systemd-coredump[1077029]: Removed old coredump
core.ganesha\x2enfsd.0.9cb285f8135e4823b3fa8ef863a9ca32.965973.1630269785000000.lz4.
Aug 30 09:40:38 ceph1 systemd-coredump[1077029]: Core file was truncated to 2147483648
bytes.
Aug 30 09:40:43 ceph1 systemd-coredump[1077029]: Process 1065997 (ganesha.nfsd) of user 0
dumped core.
Stack trace of thread
1066334:
#0 0x00007fe484c6e37f n/a
(n/a)
#1 0x000000003789c20b n/a
(n/a)
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
963
days inactive
964
days old
3 comments
3 participants
participants (3)
-
Daniel Gryniewicz -
Desert -
Kaleb Keithley