No core file... Easy to run test again... I will turn Grace back on and compare (I turned it off to make tests run faster) I won't be able to do it for a couple hours. Thank you!! Get Outlook for Android<https://aka.ms/ghei36> ________________________________ From: Soumya Koduri <skoduri@redhat.com> Sent: Monday, August 19, 2019 7:09:09 AM To: Jacobson, Erik <erik.jacobson@hpe.com>; devel@lists.nfs-ganesha.org <devel@lists.nfs-ganesha.org> Cc: Frank Filz <ffilz@redhat.com>; Daniel Gryniewicz <dang@redhat.com> Subject: Re: [NFS-Ganesha-Devel] nfs-ganesha clients freeze, nfs ganesha daemon dies Was there any core generated when the ganesha process crashed. If yes, could you provide backtrace? looking at the first failure observed in the logs, 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Session sessionid=(16:0x010000001e82585d0100000000000000) Not Found 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] nfs4_op_sequence :SESSIONS :DEBUG :SESSIONS: DEBUG: SEQUENCE returning status NFS4ERR_BADSESSION 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] LogCompoundFH :FH :F_DBG :Current FH File Handle V4: Len=0 <null> 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] LogCompoundFH :FH :F_DBG :Saved FH File Handle V4: Len=0 <null> 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] complete_op :NFS4 :DEBUG :Status of OP_SEQUENCE in position 0 = NFS4ERR_BADSESSION, op response size is 4 total response size is 40 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] complete_nfs4_compound :NFS4 :DEBUG :End status = NFS4ERR_BADSESSION lastindex = 1 This session seem to have got deleted as part of clientid expire in CREATE_SESSION 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] nfs4_op_create_session :SESSIONS :DEBUG :Expiring 0x7f03c8001cd0 ClientID={Epoch=0x5d58821e Counter=0x00000001} CONFIRMED Client={0x7f03c8001bf0 name=(20:Linux NFSv4.1 (none)) refcount=3} t_delta=0 reservations=0 refcount=10 0x7ffff22b2da4, WSTOPPED, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)_id_expire :RW LOCK :F_DBG :Acquired mutex 0x7f03c8001d50 (&clientid->cid_mutewait4(1469, 0x7ffff22b2da4, WSTOPPED, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)5d58821e : leader1 : ganesha.nfsd-30466[svc_22] nfs_client_id_expire :CLIENT Iwait4(1469, piring {0x7f03c8001cd0 ClientID={Epoch=0x5d58821e Counter=0x00000001} CONFIRMED Client={0x7f03c8001bf0 name=(20:Linux NFSv4.1 (none)) refcount=3} t_delta=0 reservations=0 refcount=10} ... 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] hashtable_getlatch :SESSIONS :F_DBG :Get (null) returning Value=0x7f03c0001840 {session 0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)}} 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] hashtable_deletelatched :SESSIONS :F_DBG :Delete (null) Key=0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)} Value=0x7f03c0001840 {session 0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)}} index=1 rbt_hash=1 was removed I see a potential bug in nfs4_op_create_session -- 156 rc = nfs_client_id_get_confirmed(clientid, &conf); 172 client_record = conf->cid_client_record; 173 found = conf; 366 /* add to head of session list (encapsulate?) */ 367 PTHREAD_MUTEX_lock(&found->cid_mutex); 368 glist_add(&found->cid_cb.v41.cb_session_list, 369 &nfs41_session->session_link); 370 PTHREAD_MUTEX_unlock(&found->cid_mutex); 371 427 if (conf != NULL && conf->cid_clientid != clientid) { 428 /* Old confirmed record - need to expire it */ 429 if (isDebug(component)) { 430 char str[LOG_BUFF_LEN] = "\0"; 431 struct display_buffer dspbuf = {sizeof(str), str, str}; 432 433 display_client_id_rec(&dspbuf, conf); 434 LogDebug(component, "Expiring %s", str); 435 } Old clientid is expired after adding the session created to its cb_session_list. Thus when this old clientid is getting cleaned, it might have deleted even the newly created valid session. This seem to have caused ERR_BADSESSION and probably ERR_EXPIRED,ERR_STALE_CLIENTID errors as well. Also since graceless is set to TRUE, clients were not able to claim their previous state. I am not sure if this can cause any application error but ideally grace period should be as long as lease period set to allow clients to recover their lost state. Request Frank/Dan to comment. Thanks, Soumya On 8/18/19 4:18 AM, Erik Jacobson wrote:

Hello. I'm statring a new thread for this problem.

I have a 3x3 Gluster Volume and I'm trying to use Ganesha for NFS services.

One of the 9 server nodes, I have enabled Ganesha NFS server on one.

The volume is being used being host clients with NFS roots.

A long separate thread shows how I got to this point but what works on the client side is:

RHEL 7.6 aarch64 4.14.0-115.el7a.aarch64 RHEL 7.6 x86_64 3.10.0-957.el7.x86_64

OverlayFS - NFS v4 underdir with TMPFS overlay.

The Ganesha server has Allow_Numeric_Owners = True; Only_Numeric_Owners = True; Disable_ACL = TRUE;

Disable_ACL is required for the aarch64 overlay to properly read non-root files. (However, Disable_ACL must be false for aarch64 if you are using NFS v3 strangely).

The x86_64 node fully boots through full init/systemd startup to the login prompt.

When I startup the aarch64 node, it gets various degrees of done... then both NFS clients freeze up 100%.

Restarting nfs-ganesha gets them going for a moment, then they freeze again. It turned out in some cases the nfs-ganesha daemon was present during the freeze but no longer serving the nodes. However, a more common case (and the captured one) is nfs-ganesha is gone.

I will attach a tarball with a bunch of information on the problem including the config file I used, debugging logs, and some traces.

Ganesha 2.8.2 - Ganesha, Gluster servers x86_64

Since the aarch64 node causes Ganesha to crash early, and the debug log can get to 2GB quickly, I set up a test case as follows:

Tracing starts... - x86_64 fully nfs-root-booted, it comes up fine. * Actively using nfs for root during tests below - aarch64 node - boot to the miniroot env (a "fat" initrd that has more tools and from which we do the NFS mount) - It stops before switching control to the init start to run the tests like below. - cp'd /dev/null to ganesha log here - started the tcpdump to the problem node - Ran the following. Ganesha died at 'wc -l', also notice the Input/Output error on the first attempt:

bash-4.2# bash reset4.sh + umount /a umount: /a: not mounted + umount /root_ro_nfs umount: /root_ro_nfs: not mounted + umount /rootfs.rw + mount -o ro,nolock 172.23.255.249:/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel /root_ro_nfs + mount -t tmpfs -o mpol=interleave tmpfs /rootfs.rw + mkdir /rootfs.rw/upperdir + mkdir /rootfs.rw/work + mount -t overlay overlay -o lowerdir=/root_ro_nfs,upperdir=/rootfs.rw/upperdir,workdir=/rootfs.rw/work /a bash-4.2# chroot /a chroot: failed to run command '/bin/sh': Input/output error bash-4.2# chroot /a sh: no job control in this shell sh-4.2# ls /usr/bin|wc -l

- When the above froze and ganesha died, I stopped tcpdump and collected the pieces in to a tarball.

See attached.

Erik

_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org

On 8/19/19 8:50 PM, Erik Jacobson wrote:

I turned off 'Graceless'.

I ran the daemon by hand instead of systemd, still no core dump.

/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F

To repeat, I just chroot in to the image and do 'su - erikj' a few times in a row. The daemon then exits. No message in -F mode, no core dump.

[root@leader1 ~]# ulimit -c unlimited [root@leader1 ~]# /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F [root@leader1 ~]#

Would you like a new collection with graceless off? Let me know. Or anything else I can run to help, let me know. It's easy to reproduce the problem so no big deal to gather more data.

Okay. Lets wait for inputs from Frank or Dan. Maybe if the issue in nfs4_op_create_session (in ganesha code) which I pointed earlier is indeed a bug, fixing it might help. Thanks, Soumya

...

Was there any core generated when the ganesha process crashed. If yes, could you provide backtrace?

looking at the first failure observed in the logs,

17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Session sessionid=(16:0x010000001e82585d0100000000000000) Not Found 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] nfs4_op_sequence :SESSIONS :DEBUG :SESSIONS: DEBUG: SEQUENCE returning status NFS4ERR_BADSESSION 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] LogCompoundFH :FH :F_DBG :Current FH File Handle V4: Len=0 <null> 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] LogCompoundFH :FH :F_DBG :Saved FH File Handle V4: Len=0 <null> 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] complete_op :NFS4 :DEBUG :Status of OP_SEQUENCE in position 0 = NFS4ERR_BADSESSION, op response size is 4 total response size is 40 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] complete_nfs4_compound :NFS4 :DEBUG :End status = NFS4ERR_BADSESSION lastindex = 1

This session seem to have got deleted as part of clientid expire in CREATE_SESSION

17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] nfs4_op_create_session :SESSIONS :DEBUG :Expiring 0x7f03c8001cd0 ClientID={Epoch=0x5d58821e Counter=0x00000001} CONFIRMED Client={0x7f03c8001bf0 name=(20:Linux NFSv4.1 (none)) refcount=3} t_delta=0 reservations=0 refcount=10 0x7ffff22b2da4, WSTOPPED, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)_id_expire :RW LOCK :F_DBG :Acquired mutex 0x7f03c8001d50 (&clientid->cid_mutewait4(1469, 0x7ffff22b2da4, WSTOPPED, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)5d58821e : leader1 : ganesha.nfsd-30466[svc_22] nfs_client_id_expire :CLIENT Iwait4(1469, piring {0x7f03c8001cd0 ClientID={Epoch=0x5d58821e Counter=0x00000001} CONFIRMED Client={0x7f03c8001bf0 name=(20:Linux NFSv4.1 (none)) refcount=3} t_delta=0 reservations=0 refcount=10} ... 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] hashtable_getlatch :SESSIONS :F_DBG :Get (null) returning Value=0x7f03c0001840 {session 0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)}} 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] hashtable_deletelatched :SESSIONS :F_DBG :Delete (null) Key=0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)} Value=0x7f03c0001840 {session 0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)}} index=1 rbt_hash=1 was removed

I see a potential bug in nfs4_op_create_session --

156 rc = nfs_client_id_get_confirmed(clientid, &conf); 172 client_record = conf->cid_client_record; 173 found = conf;

366 /* add to head of session list (encapsulate?) */ 367 PTHREAD_MUTEX_lock(&found->cid_mutex); 368 glist_add(&found->cid_cb.v41.cb_session_list, 369 &nfs41_session->session_link); 370 PTHREAD_MUTEX_unlock(&found->cid_mutex); 371

427 if (conf != NULL && conf->cid_clientid != clientid) { 428 /* Old confirmed record - need to expire it */ 429 if (isDebug(component)) { 430 char str[LOG_BUFF_LEN] = "\0"; 431 struct display_buffer dspbuf = {sizeof(str), str, str}; 432 433 display_client_id_rec(&dspbuf, conf); 434 LogDebug(component, "Expiring %s", str); 435 }

Old clientid is expired after adding the session created to its cb_session_list. Thus when this old clientid is getting cleaned, it might have deleted even the newly created valid session. This seem to have caused ERR_BADSESSION and probably ERR_EXPIRED,ERR_STALE_CLIENTID errors as well.

Also since graceless is set to TRUE, clients were not able to claim their previous state. I am not sure if this can cause any application error but ideally grace period should be as long as lease period set to allow clients to recover their lost state.

Request Frank/Dan to comment.

Thanks, Soumya

On 8/18/19 4:18 AM, Erik Jacobson wrote:

...

Hello. I'm statring a new thread for this problem.

I have a 3x3 Gluster Volume and I'm trying to use Ganesha for NFS services.

One of the 9 server nodes, I have enabled Ganesha NFS server on one.

The volume is being used being host clients with NFS roots.

A long separate thread shows how I got to this point but what works on the client side is:

RHEL 7.6 aarch64 4.14.0-115.el7a.aarch64 RHEL 7.6 x86_64 3.10.0-957.el7.x86_64

OverlayFS - NFS v4 underdir with TMPFS overlay.

The Ganesha server has Allow_Numeric_Owners = True; Only_Numeric_Owners = True; Disable_ACL = TRUE;

Disable_ACL is required for the aarch64 overlay to properly read non-root files. (However, Disable_ACL must be false for aarch64 if you are using NFS v3 strangely).

The x86_64 node fully boots through full init/systemd startup to the login prompt.

When I startup the aarch64 node, it gets various degrees of done... then both NFS clients freeze up 100%.

Restarting nfs-ganesha gets them going for a moment, then they freeze again. It turned out in some cases the nfs-ganesha daemon was present during the freeze but no longer serving the nodes. However, a more common case (and the captured one) is nfs-ganesha is gone.

I will attach a tarball with a bunch of information on the problem including the config file I used, debugging logs, and some traces.

Ganesha 2.8.2 - Ganesha, Gluster servers x86_64

Since the aarch64 node causes Ganesha to crash early, and the debug log can get to 2GB quickly, I set up a test case as follows:

Tracing starts... - x86_64 fully nfs-root-booted, it comes up fine. * Actively using nfs for root during tests below - aarch64 node - boot to the miniroot env (a "fat" initrd that has more tools and from which we do the NFS mount) - It stops before switching control to the init start to run the tests like below. - cp'd /dev/null to ganesha log here - started the tcpdump to the problem node - Ran the following. Ganesha died at 'wc -l', also notice the Input/Output error on the first attempt:

bash-4.2# bash reset4.sh + umount /a umount: /a: not mounted + umount /root_ro_nfs umount: /root_ro_nfs: not mounted + umount /rootfs.rw + mount -o ro,nolock 172.23.255.249:/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel /root_ro_nfs + mount -t tmpfs -o mpol=interleave tmpfs /rootfs.rw + mkdir /rootfs.rw/upperdir + mkdir /rootfs.rw/work + mount -t overlay overlay -o lowerdir=/root_ro_nfs,upperdir=/rootfs.rw/upperdir,workdir=/rootfs.rw/work /a bash-4.2# chroot /a chroot: failed to run command '/bin/sh': Input/output error bash-4.2# chroot /a sh: no job control in this shell sh-4.2# ls /usr/bin|wc -l

- When the above froze and ganesha died, I stopped tcpdump and collected the pieces in to a tarball.

See attached.

Erik

_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org

Erik Jacobson Software Engineer

erik.jacobson@hpe.com +1 612 851 0550 Office

Eagan, MN hpe.com _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org

- I got the source rpm for libntirpc (I had trouble locating a debuginfo for it) * centos 7.6 'storage' SRPMS - Built it, installed the 3 packages including debuginfo. [root@leader1 x86_64]# rpm -qa | grep libntirpc libntirpc-debuginfo-1.8.0-1.el7erikj.x86_64 libntirpc-devel-1.8.0-1.el7erikj.x86_64 libntirpc-1.8.0-1.el7erikj.x86_64 - The Ganesha I'm using right now was built from souce/cmake/make and installed with 'make install' * I had trouble getting the right places in place with debug mode 'debug' and 'relwithdeb' so I just built 'rpm' as a target and started using RPMs including the debuginfo one - I an it like you said. Then I exercised the problem case. Here is the output. No back trace, looks ike it says a process exited with code 02. Here is the full output. I can try to trck down other debuginfo's but probably not needed right? [root@leader1 ~]# gdb /usr/bin/ganesha.nfsd GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/bin/ganesha.nfsd...Reading symbols from /usr/lib/debug/usr/bin/ganesha.nfsd.debug...done. done. (gdb) run -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F Starting program: /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". [New Thread 0x7fffe829d700 (LWP 46187)] [New Thread 0x7fffd5da4700 (LWP 46188)] [New Thread 0x7fffc386b700 (LWP 46189)] [New Thread 0x7fffb05aa700 (LWP 46190)] [New Thread 0x7fff9d451700 (LWP 46191)] [New Thread 0x7fff8af58700 (LWP 46192)] [New Thread 0x7fff77dff700 (LWP 46193)] [New Thread 0x7fff65906700 (LWP 46194)] [New Thread 0x7fff4cee3700 (LWP 46195)] [New Thread 0x7fff3a7e7700 (LWP 46196)] [New Thread 0x7fff23fff700 (LWP 46197)] [New Thread 0x7fff2809a700 (LWP 46198)] [New Thread 0x7fff03fff700 (LWP 46199)] [New Thread 0x7ffeebfff700 (LWP 46200)] [New Thread 0x7ffed3fff700 (LWP 46201)] [New Thread 0x7ffebbfff700 (LWP 46202)] [New Thread 0x7ffea3fff700 (LWP 46203)] [New Thread 0x7ffe8bfff700 (LWP 46204)] [New Thread 0x7ffe73fff700 (LWP 46205)] [New Thread 0x7ffe5bfff700 (LWP 46206)] [New Thread 0x7ffe43fff700 (LWP 46207)] [New Thread 0x7ffe2bfff700 (LWP 46208)] [New Thread 0x7ffe13fff700 (LWP 46209)] [New Thread 0x7ffdfbfff700 (LWP 46210)] [New Thread 0x7ffde3fff700 (LWP 46211)] [New Thread 0x7ffdcbfff700 (LWP 46212)] [New Thread 0x7ffdb3fff700 (LWP 46213)] [New Thread 0x7ffd9bfff700 (LWP 46214)] [New Thread 0x7ffd83fff700 (LWP 46215)] [New Thread 0x7ffd6bfff700 (LWP 46216)] [New Thread 0x7ffd53fff700 (LWP 46217)] [New Thread 0x7ffd3bfff700 (LWP 46218)] [New Thread 0x7ffd23fff700 (LWP 46219)] [New Thread 0x7ffd0bfff700 (LWP 46220)] [New Thread 0x7ffcf3fff700 (LWP 46221)] [New Thread 0x7ffcdbfff700 (LWP 46222)] [New Thread 0x7ffcc3fff700 (LWP 46223)] [New Thread 0x7ffcabfff700 (LWP 46224)] [New Thread 0x7ffc93fff700 (LWP 46225)] [New Thread 0x7ffc7bfff700 (LWP 46226)] [New Thread 0x7ffc63fff700 (LWP 46227)] [New Thread 0x7ffc4bfff700 (LWP 46228)] [New Thread 0x7ffc33fff700 (LWP 46229)] [New Thread 0x7fff0af48700 (LWP 46230)] [New Thread 0x7fff0ac0f700 (LWP 46231)] [New Thread 0x7fff0ab0e700 (LWP 46232)] [New Thread 0x7fff0aa0d700 (LWP 46233)] [New Thread 0x7fff0a90c700 (LWP 46234)] [New Thread 0x7fff0a80b700 (LWP 46235)] [New Thread 0x7fff0a70a700 (LWP 46236)] [New Thread 0x7fff0a609700 (LWP 46237)] [New Thread 0x7fff0a508700 (LWP 46238)] [New Thread 0x7fff0a407700 (LWP 46239)] [New Thread 0x7fff0a306700 (LWP 46240)] [New Thread 0x7ffbebfff700 (LWP 46241)] [New Thread 0x7ffbd9b06700 (LWP 46242)] [New Thread 0x7ffbc760d700 (LWP 46243)] [New Thread 0x7ffbb5114700 (LWP 46244)] [New Thread 0x7ffba2c1b700 (LWP 46245)] [New Thread 0x7ffb90722700 (LWP 46246)] [New Thread 0x7ffb7e229700 (LWP 46247)] [New Thread 0x7ffb6bd30700 (LWP 46248)] [New Thread 0x7ffb59837700 (LWP 46249)] [New Thread 0x7ffb4733e700 (LWP 46250)] [New Thread 0x7ffb34e45700 (LWP 46251)] [New Thread 0x7ffb2294c700 (LWP 46252)] [New Thread 0x7ffb10453700 (LWP 46253)] [New Thread 0x7ffafdf5a700 (LWP 46254)] [New Thread 0x7ffaeba61700 (LWP 46255)] [New Thread 0x7ffad9568700 (LWP 46256)] [New Thread 0x7ffac706f700 (LWP 46257)] [New Thread 0x7ffab4b76700 (LWP 46258)] [New Thread 0x7ffaa267d700 (LWP 46259)] [New Thread 0x7ffa90184700 (LWP 46260)] [New Thread 0x7ffa7dc8b700 (LWP 46261)] [New Thread 0x7ffa6b792700 (LWP 46262)] [New Thread 0x7ffa59299700 (LWP 46263)] [New Thread 0x7ffa46da0700 (LWP 46264)] [New Thread 0x7ffa348a7700 (LWP 46265)] [New Thread 0x7ffa223ae700 (LWP 46266)] [New Thread 0x7ffa0feb5700 (LWP 46267)] [New Thread 0x7ff9fd9bc700 (LWP 46268)] [New Thread 0x7ff9eb4c3700 (LWP 46269)] [New Thread 0x7ff9d8fca700 (LWP 46270)] [New Thread 0x7ff9c6ad1700 (LWP 46271)] [New Thread 0x7ff9b45d8700 (LWP 46272)] [New Thread 0x7ff9a20df700 (LWP 46273)] [New Thread 0x7ff98fbe6700 (LWP 46274)] [New Thread 0x7ff97d6ed700 (LWP 46275)] [New Thread 0x7ff96b1f4700 (LWP 46276)] [New Thread 0x7ff958cfb700 (LWP 46277)] [New Thread 0x7ff946802700 (LWP 46278)] [New Thread 0x7ff934309700 (LWP 46279)] [New Thread 0x7ff921e10700 (LWP 46280)] [New Thread 0x7ff90f917700 (LWP 46281)] [New Thread 0x7ff8fd41e700 (LWP 46282)] [New Thread 0x7ff8eaf25700 (LWP 46283)] [New Thread 0x7ff8d8a2c700 (LWP 46284)] [New Thread 0x7ff8c6533700 (LWP 46285)] [New Thread 0x7ff8b403a700 (LWP 46286)] [New Thread 0x7ff8a1b41700 (LWP 46287)] [New Thread 0x7ff88f648700 (LWP 46288)] [New Thread 0x7ff87d14f700 (LWP 46289)] [New Thread 0x7ff86ac56700 (LWP 46290)] [New Thread 0x7ff85875d700 (LWP 46291)] [New Thread 0x7ff846264700 (LWP 46292)] [New Thread 0x7ff833d6b700 (LWP 46293)] [New Thread 0x7ff821872700 (LWP 46294)] [New Thread 0x7ff80f379700 (LWP 46295)] [New Thread 0x7ff7fce80700 (LWP 46296)] [New Thread 0x7ff7ea987700 (LWP 46297)] [New Thread 0x7ff7d848e700 (LWP 46298)] [New Thread 0x7ff7c5f95700 (LWP 46299)] [New Thread 0x7ff7b3a9c700 (LWP 46300)] [New Thread 0x7ff7a15a3700 (LWP 46301)] [New Thread 0x7ff78f0aa700 (LWP 46302)] [New Thread 0x7ff77cbb1700 (LWP 46303)] [New Thread 0x7ff76a6b8700 (LWP 46304)] [New Thread 0x7ff7581bf700 (LWP 46305)] [New Thread 0x7ff745cc6700 (LWP 46306)] [New Thread 0x7ff7337cd700 (LWP 46307)] [New Thread 0x7ff7212d4700 (LWP 46308)] [New Thread 0x7ff70eddb700 (LWP 46309)] [New Thread 0x7ff6fc8e2700 (LWP 46310)] [New Thread 0x7ff6ea3e9700 (LWP 46311)] [New Thread 0x7ff6d7ef0700 (LWP 46312)] [New Thread 0x7ff6c59f7700 (LWP 46313)] [New Thread 0x7ff6b34fe700 (LWP 46314)] [New Thread 0x7ff6a1005700 (LWP 46315)] [New Thread 0x7ff68eb0c700 (LWP 46316)] [New Thread 0x7ff67c613700 (LWP 46317)] [New Thread 0x7ff66a11a700 (LWP 46318)] [New Thread 0x7ff657c21700 (LWP 46319)] [New Thread 0x7ff645728700 (LWP 46320)] [New Thread 0x7ff63322f700 (LWP 46321)] [New Thread 0x7ff620d36700 (LWP 46322)] [New Thread 0x7ff60e83d700 (LWP 46323)] [New Thread 0x7ff5fc344700 (LWP 46324)] [New Thread 0x7ff5e9e4b700 (LWP 46325)] [New Thread 0x7ff5d7952700 (LWP 46326)] [New Thread 0x7ff5c5459700 (LWP 46327)] [New Thread 0x7ff5b2f60700 (LWP 46328)] [New Thread 0x7ff5a0a67700 (LWP 46329)] [New Thread 0x7ff58e56e700 (LWP 46330)] [New Thread 0x7ff57c075700 (LWP 46331)] [New Thread 0x7ff569b7c700 (LWP 46332)] [New Thread 0x7ff557683700 (LWP 46333)] [New Thread 0x7ff54518a700 (LWP 46334)] [New Thread 0x7ff532c91700 (LWP 46335)] [New Thread 0x7ff520798700 (LWP 46336)] [New Thread 0x7ff50e29f700 (LWP 46337)] [New Thread 0x7ff4fbda6700 (LWP 46338)] [New Thread 0x7ff4e98ad700 (LWP 46339)] [New Thread 0x7ff4d73b4700 (LWP 46340)] [New Thread 0x7ff4c4ebb700 (LWP 46341)] [New Thread 0x7ff4b29c2700 (LWP 46342)] [New Thread 0x7ff4a04c9700 (LWP 46343)] [New Thread 0x7ff48dfd0700 (LWP 46344)] [New Thread 0x7ff47bad7700 (LWP 46345)] [New Thread 0x7ff4695de700 (LWP 46346)] [New Thread 0x7ff4570e5700 (LWP 46347)] [New Thread 0x7ff444bec700 (LWP 46348)] [New Thread 0x7ff4326f3700 (LWP 46349)] [New Thread 0x7ff4201fa700 (LWP 46350)] [New Thread 0x7ff40dd01700 (LWP 46351)] [New Thread 0x7ff3fb808700 (LWP 46352)] [New Thread 0x7ff3e930f700 (LWP 46353)] [New Thread 0x7ff3d6e16700 (LWP 46354)] [New Thread 0x7ff3c491d700 (LWP 46355)] [New Thread 0x7ff3b2424700 (LWP 46356)] [New Thread 0x7ff39ff2b700 (LWP 46357)] [New Thread 0x7ff38da32700 (LWP 46358)] [New Thread 0x7ff37b539700 (LWP 46359)] [New Thread 0x7ff369040700 (LWP 46360)] [New Thread 0x7ff356b47700 (LWP 46361)] [New Thread 0x7ff34464e700 (LWP 46362)] [New Thread 0x7ff332155700 (LWP 46363)] [New Thread 0x7ff31fc5c700 (LWP 46364)] [New Thread 0x7ff30d763700 (LWP 46365)] [New Thread 0x7ff2fb26a700 (LWP 46366)] [New Thread 0x7ff2e8d71700 (LWP 46367)] [New Thread 0x7ff2d6878700 (LWP 46368)] [New Thread 0x7ff2c437f700 (LWP 46369)] [New Thread 0x7ff2b1e86700 (LWP 46370)] [New Thread 0x7ff29f98d700 (LWP 46371)] [New Thread 0x7ff28d494700 (LWP 46372)] [New Thread 0x7ff27af9b700 (LWP 46373)] [New Thread 0x7ff268aa2700 (LWP 46374)] [New Thread 0x7ff2565a9700 (LWP 46375)] [New Thread 0x7ff2440b0700 (LWP 46376)] [New Thread 0x7ff231bb7700 (LWP 46377)] [New Thread 0x7ff21f6be700 (LWP 46378)] [New Thread 0x7ff20d1c5700 (LWP 46379)] [New Thread 0x7ff1faccc700 (LWP 46380)] [New Thread 0x7ff1e87d3700 (LWP 46381)] [New Thread 0x7ff1d62da700 (LWP 46382)] [New Thread 0x7ff1c3de1700 (LWP 46383)] [New Thread 0x7ff1b18e8700 (LWP 46384)] [New Thread 0x7ff19f3ef700 (LWP 46385)] [New Thread 0x7ff18cef6700 (LWP 46386)] [New Thread 0x7ff17a9fd700 (LWP 46387)] [New Thread 0x7ff168504700 (LWP 46388)] [New Thread 0x7ff15600b700 (LWP 46389)] [New Thread 0x7ff143b12700 (LWP 46390)] [New Thread 0x7ff131619700 (LWP 46391)] [New Thread 0x7ff11f120700 (LWP 46392)] [New Thread 0x7ff10cc27700 (LWP 46393)] [New Thread 0x7ff0fa72e700 (LWP 46394)] [New Thread 0x7ff0e8235700 (LWP 46395)] [New Thread 0x7ff0d5d3c700 (LWP 46396)] [New Thread 0x7ff0c3843700 (LWP 46397)] [New Thread 0x7ff0b134a700 (LWP 46398)] [New Thread 0x7ff09ee51700 (LWP 46399)] [New Thread 0x7ff08c958700 (LWP 46400)] [New Thread 0x7ff07a45f700 (LWP 46401)] [New Thread 0x7ff067f66700 (LWP 46402)] [New Thread 0x7ff055a6d700 (LWP 46403)] [New Thread 0x7ff043574700 (LWP 46404)] [New Thread 0x7ff03107b700 (LWP 46405)] [New Thread 0x7ff01eb82700 (LWP 46406)] [New Thread 0x7ff00c689700 (LWP 46408)] [New Thread 0x7feffa190700 (LWP 46409)] [New Thread 0x7fefe7c97700 (LWP 46410)] [New Thread 0x7fefd579e700 (LWP 46411)] [New Thread 0x7fefc32a5700 (LWP 46412)] [New Thread 0x7fefb0dac700 (LWP 46413)] [New Thread 0x7fef9e8b3700 (LWP 46414)] [New Thread 0x7fef8c3ba700 (LWP 46415)] [New Thread 0x7fef79ec1700 (LWP 46416)] [New Thread 0x7fef679c8700 (LWP 46417)] [New Thread 0x7fef554cf700 (LWP 46418)] [New Thread 0x7fef42fd6700 (LWP 46419)] [New Thread 0x7fef30add700 (LWP 46420)] [New Thread 0x7fef1e5e4700 (LWP 46421)] [New Thread 0x7fef0c0eb700 (LWP 46422)] [New Thread 0x7feef9bf2700 (LWP 46423)] [New Thread 0x7feee76f9700 (LWP 46424)] [New Thread 0x7feed5200700 (LWP 46425)] [New Thread 0x7feec2d07700 (LWP 46426)] [New Thread 0x7feeb080e700 (LWP 46427)] [New Thread 0x7fee9e315700 (LWP 46428)] [New Thread 0x7fee8be1c700 (LWP 46429)] [New Thread 0x7fee79923700 (LWP 46430)] [New Thread 0x7fee6742a700 (LWP 46431)] [New Thread 0x7fee54f31700 (LWP 46432)] [New Thread 0x7fee42a38700 (LWP 46433)] [New Thread 0x7fee3053f700 (LWP 46434)] [New Thread 0x7fee1e046700 (LWP 46435)] [New Thread 0x7fee0bb4d700 (LWP 46436)] [New Thread 0x7fedf9654700 (LWP 46437)] [New Thread 0x7fede715b700 (LWP 46438)] [New Thread 0x7fedd4c62700 (LWP 46439)] [New Thread 0x7fedc2769700 (LWP 46440)] [New Thread 0x7fedb0270700 (LWP 46441)] [New Thread 0x7fed9dd77700 (LWP 46442)] [New Thread 0x7fed8b87e700 (LWP 46443)] [New Thread 0x7fed79385700 (LWP 46444)] [New Thread 0x7fed66e8c700 (LWP 46445)] [New Thread 0x7fed54993700 (LWP 46446)] [New Thread 0x7fed4249a700 (LWP 46447)] [New Thread 0x7fed2ffa1700 (LWP 46448)] [New Thread 0x7fed1daa8700 (LWP 46449)] [New Thread 0x7fed0b5af700 (LWP 46450)] [New Thread 0x7fecf90b6700 (LWP 46451)] [New Thread 0x7fece6bbd700 (LWP 46452)] [New Thread 0x7fecd46c4700 (LWP 46453)] [New Thread 0x7fecc21cb700 (LWP 46454)] [New Thread 0x7fecafcd2700 (LWP 46455)] [New Thread 0x7fec9d7d9700 (LWP 46456)] [New Thread 0x7fec8b2e0700 (LWP 46457)] [New Thread 0x7fec78de7700 (LWP 46458)] [New Thread 0x7fec668ee700 (LWP 46459)] [New Thread 0x7fec543f5700 (LWP 46460)] [New Thread 0x7fec41efc700 (LWP 46461)] [New Thread 0x7fec2fa03700 (LWP 46462)] [New Thread 0x7fec1d50a700 (LWP 46463)] [New Thread 0x7fec0b011700 (LWP 46464)] [New Thread 0x7febf8b18700 (LWP 46465)] [New Thread 0x7febe661f700 (LWP 46466)] [New Thread 0x7febd4126700 (LWP 46467)] [New Thread 0x7febc1c2d700 (LWP 46468)] [New Thread 0x7febaf734700 (LWP 46469)] [New Thread 0x7feb9d23b700 (LWP 46470)] [New Thread 0x7feb8ad42700 (LWP 46471)] [New Thread 0x7feb78849700 (LWP 46472)] [New Thread 0x7feb66350700 (LWP 46473)] [New Thread 0x7feb53e57700 (LWP 46474)] [New Thread 0x7feb4195e700 (LWP 46475)] [New Thread 0x7feb2f465700 (LWP 46476)] [New Thread 0x7feb1cf6c700 (LWP 46477)] [New Thread 0x7feb0aa73700 (LWP 46478)] [New Thread 0x7feaf857a700 (LWP 46479)] [New Thread 0x7feae6081700 (LWP 46480)] [New Thread 0x7fead3b88700 (LWP 46481)] [New Thread 0x7feac168f700 (LWP 46482)] [New Thread 0x7feaaf196700 (LWP 46483)] [New Thread 0x7fea9cc9d700 (LWP 46484)] [New Thread 0x7fea8a7a4700 (LWP 46485)] [New Thread 0x7fea782ab700 (LWP 46486)] [New Thread 0x7fea65db2700 (LWP 46487)] [New Thread 0x7fea538b9700 (LWP 46488)] [New Thread 0x7fea413c0700 (LWP 46489)] [New Thread 0x7fea2eec7700 (LWP 46490)] [New Thread 0x7fea1c9ce700 (LWP 46491)] [New Thread 0x7fea0a4d5700 (LWP 46492)] [New Thread 0x7fe9f7fdc700 (LWP 46493)] [New Thread 0x7fe9e5ae3700 (LWP 46494)] [New Thread 0x7fe9d35ea700 (LWP 46495)] [New Thread 0x7fe9c10f1700 (LWP 46496)] [New Thread 0x7fe9aebf8700 (LWP 46497)] [New Thread 0x7fe99c6ff700 (LWP 46498)] [New Thread 0x7fe98a206700 (LWP 46499)] [New Thread 0x7fe977d0d700 (LWP 46500)] [New Thread 0x7fe965814700 (LWP 46501)] [New Thread 0x7fe95331b700 (LWP 46502)] [New Thread 0x7fe93bfff700 (LWP 46503)] [New Thread 0x7fff09dc8700 (LWP 46505)] [New Thread 0x7fff09cc7700 (LWP 46506)] [New Thread 0x7fff09bc6700 (LWP 46507)] [New Thread 0x7fff09ac5700 (LWP 46508)] [New Thread 0x7fff099c4700 (LWP 46509)] [New Thread 0x7fff098c3700 (LWP 46510)] [New Thread 0x7fff0ae47700 (LWP 46511)] [New Thread 0x7fff097c2700 (LWP 46512)] [New Thread 0x7fff096c1700 (LWP 46513)] [New Thread 0x7fe90bfff700 (LWP 46514)] [New Thread 0x7fe8f9b06700 (LWP 46515)] [New Thread 0x7fe8e760d700 (LWP 46516)] [New Thread 0x7fe8d5114700 (LWP 46517)] [Thread 0x7fe8d5114700 (LWP 46517) exited] [Thread 0x7fe8e760d700 (LWP 46516) exited] [Thread 0x7fe8f9b06700 (LWP 46515) exited] [Thread 0x7fe90bfff700 (LWP 46514) exited] [Thread 0x7fff096c1700 (LWP 46513) exited] [Thread 0x7fff097c2700 (LWP 46512) exited] [Thread 0x7fff0ae47700 (LWP 46511) exited] [Thread 0x7fff098c3700 (LWP 46510) exited] [Thread 0x7fff099c4700 (LWP 46509) exited] [Thread 0x7fff09ac5700 (LWP 46508) exited] [Thread 0x7fff09bc6700 (LWP 46507) exited] [Thread 0x7fff09cc7700 (LWP 46506) exited] [Thread 0x7fff09dc8700 (LWP 46505) exited] [Thread 0x7fe93bfff700 (LWP 46503) exited] [Thread 0x7fe95331b700 (LWP 46502) exited] [Thread 0x7fe965814700 (LWP 46501) exited] [Thread 0x7fe977d0d700 (LWP 46500) exited] [Thread 0x7fe98a206700 (LWP 46499) exited] [Thread 0x7fe99c6ff700 (LWP 46498) exited] [Thread 0x7fe9aebf8700 (LWP 46497) exited] [Thread 0x7fe9c10f1700 (LWP 46496) exited] [Thread 0x7fe9d35ea700 (LWP 46495) exited] [Thread 0x7fe9e5ae3700 (LWP 46494) exited] [Thread 0x7fe9f7fdc700 (LWP 46493) exited] [Thread 0x7fea0a4d5700 (LWP 46492) exited] [Thread 0x7fea1c9ce700 (LWP 46491) exited] [Thread 0x7fea2eec7700 (LWP 46490) exited] [Thread 0x7fea413c0700 (LWP 46489) exited] [Thread 0x7fea538b9700 (LWP 46488) exited] [Thread 0x7fea65db2700 (LWP 46487) exited] [Thread 0x7fea782ab700 (LWP 46486) exited] [Thread 0x7fea8a7a4700 (LWP 46485) exited] [Thread 0x7fea9cc9d700 (LWP 46484) exited] [Thread 0x7feaaf196700 (LWP 46483) exited] [Thread 0x7feac168f700 (LWP 46482) exited] [Thread 0x7fead3b88700 (LWP 46481) exited] [Thread 0x7feae6081700 (LWP 46480) exited] [Thread 0x7feaf857a700 (LWP 46479) exited] [Thread 0x7feb0aa73700 (LWP 46478) exited] [Thread 0x7feb1cf6c700 (LWP 46477) exited] [Thread 0x7feb2f465700 (LWP 46476) exited] [Thread 0x7feb4195e700 (LWP 46475) exited] [Thread 0x7feb53e57700 (LWP 46474) exited] [Thread 0x7feb66350700 (LWP 46473) exited] [Thread 0x7feb78849700 (LWP 46472) exited] [Thread 0x7feb8ad42700 (LWP 46471) exited] [Thread 0x7feb9d23b700 (LWP 46470) exited] [Thread 0x7febaf734700 (LWP 46469) exited] [Thread 0x7febc1c2d700 (LWP 46468) exited] [Thread 0x7febd4126700 (LWP 46467) exited] [Thread 0x7febe661f700 (LWP 46466) exited] [Thread 0x7febf8b18700 (LWP 46465) exited] [Thread 0x7fec0b011700 (LWP 46464) exited] [Thread 0x7fec1d50a700 (LWP 46463) exited] [Thread 0x7fec2fa03700 (LWP 46462) exited] [Thread 0x7fec41efc700 (LWP 46461) exited] [Thread 0x7fec543f5700 (LWP 46460) exited] [Thread 0x7fec668ee700 (LWP 46459) exited] [Thread 0x7fec78de7700 (LWP 46458) exited] [Thread 0x7fec8b2e0700 (LWP 46457) exited] [Thread 0x7fec9d7d9700 (LWP 46456) exited] [Thread 0x7fecafcd2700 (LWP 46455) exited] [Thread 0x7fecc21cb700 (LWP 46454) exited] [Thread 0x7fecd46c4700 (LWP 46453) exited] [Thread 0x7fece6bbd700 (LWP 46452) exited] [Thread 0x7fecf90b6700 (LWP 46451) exited] [Thread 0x7fed0b5af700 (LWP 46450) exited] [Thread 0x7fed1daa8700 (LWP 46449) exited] [Thread 0x7fed2ffa1700 (LWP 46448) exited] [Thread 0x7fed4249a700 (LWP 46447) exited] [Thread 0x7fed54993700 (LWP 46446) exited] [Thread 0x7fed66e8c700 (LWP 46445) exited] [Thread 0x7fed79385700 (LWP 46444) exited] [Thread 0x7fed8b87e700 (LWP 46443) exited] [Thread 0x7fed9dd77700 (LWP 46442) exited] [Thread 0x7fedb0270700 (LWP 46441) exited] [Thread 0x7fedc2769700 (LWP 46440) exited] [Thread 0x7fedd4c62700 (LWP 46439) exited] [Thread 0x7fede715b700 (LWP 46438) exited] [Thread 0x7fedf9654700 (LWP 46437) exited] [Thread 0x7fee0bb4d700 (LWP 46436) exited] [Thread 0x7fee1e046700 (LWP 46435) exited] [Thread 0x7fee3053f700 (LWP 46434) exited] [Thread 0x7fee42a38700 (LWP 46433) exited] [Thread 0x7fee54f31700 (LWP 46432) exited] [Thread 0x7fee6742a700 (LWP 46431) exited] [Thread 0x7fee79923700 (LWP 46430) exited] [Thread 0x7fee8be1c700 (LWP 46429) exited] [Thread 0x7fee9e315700 (LWP 46428) exited] [Thread 0x7feeb080e700 (LWP 46427) exited] [Thread 0x7feec2d07700 (LWP 46426) exited] [Thread 0x7feed5200700 (LWP 46425) exited] [Thread 0x7feee76f9700 (LWP 46424) exited] [Thread 0x7feef9bf2700 (LWP 46423) exited] [Thread 0x7fef0c0eb700 (LWP 46422) exited] [Thread 0x7fef1e5e4700 (LWP 46421) exited] [Thread 0x7fef30add700 (LWP 46420) exited] [Thread 0x7fef42fd6700 (LWP 46419) exited] [Thread 0x7fef554cf700 (LWP 46418) exited] [Thread 0x7fef679c8700 (LWP 46417) exited] [Thread 0x7fef79ec1700 (LWP 46416) exited] [Thread 0x7fef8c3ba700 (LWP 46415) exited] [Thread 0x7fef9e8b3700 (LWP 46414) exited] [Thread 0x7fefb0dac700 (LWP 46413) exited] [Thread 0x7fefc32a5700 (LWP 46412) exited] [Thread 0x7fefd579e700 (LWP 46411) exited] [Thread 0x7fefe7c97700 (LWP 46410) exited] [Thread 0x7feffa190700 (LWP 46409) exited] [Thread 0x7ff00c689700 (LWP 46408) exited] [Thread 0x7ff01eb82700 (LWP 46406) exited] [Thread 0x7ff03107b700 (LWP 46405) exited] [Thread 0x7ff043574700 (LWP 46404) exited] [Thread 0x7ff055a6d700 (LWP 46403) exited] [Thread 0x7ff067f66700 (LWP 46402) exited] [Thread 0x7ff07a45f700 (LWP 46401) exited] [Thread 0x7ff08c958700 (LWP 46400) exited] [Thread 0x7ff09ee51700 (LWP 46399) exited] [Thread 0x7ff0b134a700 (LWP 46398) exited] [Thread 0x7ff0c3843700 (LWP 46397) exited] [Thread 0x7ff0d5d3c700 (LWP 46396) exited] [Thread 0x7ff0e8235700 (LWP 46395) exited] [Thread 0x7ff0fa72e700 (LWP 46394) exited] [Thread 0x7ff10cc27700 (LWP 46393) exited] [Thread 0x7ff11f120700 (LWP 46392) exited] [Thread 0x7ff131619700 (LWP 46391) exited] [Thread 0x7ff143b12700 (LWP 46390) exited] [Thread 0x7ff15600b700 (LWP 46389) exited] [Thread 0x7ff168504700 (LWP 46388) exited] [Thread 0x7ff17a9fd700 (LWP 46387) exited] [Thread 0x7ff18cef6700 (LWP 46386) exited] [Thread 0x7ff19f3ef700 (LWP 46385) exited] [Thread 0x7ff1b18e8700 (LWP 46384) exited] [Thread 0x7ff1c3de1700 (LWP 46383) exited] [Thread 0x7ff1d62da700 (LWP 46382) exited] [Thread 0x7ff1e87d3700 (LWP 46381) exited] [Thread 0x7ff1faccc700 (LWP 46380) exited] [Thread 0x7ff20d1c5700 (LWP 46379) exited] [Thread 0x7ff21f6be700 (LWP 46378) exited] [Thread 0x7ff231bb7700 (LWP 46377) exited] [Thread 0x7ff2440b0700 (LWP 46376) exited] [Thread 0x7ff2565a9700 (LWP 46375) exited] [Thread 0x7ff268aa2700 (LWP 46374) exited] [Thread 0x7ff27af9b700 (LWP 46373) exited] [Thread 0x7ff28d494700 (LWP 46372) exited] [Thread 0x7ff29f98d700 (LWP 46371) exited] [Thread 0x7ff2b1e86700 (LWP 46370) exited] [Thread 0x7ff2c437f700 (LWP 46369) exited] [Thread 0x7ff2d6878700 (LWP 46368) exited] [Thread 0x7ff2e8d71700 (LWP 46367) exited] [Thread 0x7ff2fb26a700 (LWP 46366) exited] [Thread 0x7ff30d763700 (LWP 46365) exited] [Thread 0x7ff31fc5c700 (LWP 46364) exited] [Thread 0x7ff332155700 (LWP 46363) exited] [Thread 0x7ff34464e700 (LWP 46362) exited] [Thread 0x7ff356b47700 (LWP 46361) exited] [Thread 0x7ff369040700 (LWP 46360) exited] [Thread 0x7ff37b539700 (LWP 46359) exited] [Thread 0x7ff38da32700 (LWP 46358) exited] [Thread 0x7ff39ff2b700 (LWP 46357) exited] [Thread 0x7ff3b2424700 (LWP 46356) exited] [Thread 0x7ff3c491d700 (LWP 46355) exited] [Thread 0x7ff3d6e16700 (LWP 46354) exited] [Thread 0x7ff3e930f700 (LWP 46353) exited] [Thread 0x7ff3fb808700 (LWP 46352) exited] [Thread 0x7ff40dd01700 (LWP 46351) exited] [Thread 0x7ff4201fa700 (LWP 46350) exited] [Thread 0x7ff4326f3700 (LWP 46349) exited] [Thread 0x7ff444bec700 (LWP 46348) exited] [Thread 0x7ff4570e5700 (LWP 46347) exited] [Thread 0x7ff4695de700 (LWP 46346) exited] [Thread 0x7ff47bad7700 (LWP 46345) exited] [Thread 0x7ff48dfd0700 (LWP 46344) exited] [Thread 0x7ff4a04c9700 (LWP 46343) exited] [Thread 0x7ff4b29c2700 (LWP 46342) exited] [Thread 0x7ff4c4ebb700 (LWP 46341) exited] [Thread 0x7ff4d73b4700 (LWP 46340) exited] [Thread 0x7ff4e98ad700 (LWP 46339) exited] [Thread 0x7ff4fbda6700 (LWP 46338) exited] [Thread 0x7ff50e29f700 (LWP 46337) exited] [Thread 0x7ff520798700 (LWP 46336) exited] [Thread 0x7ff532c91700 (LWP 46335) exited] [Thread 0x7ff54518a700 (LWP 46334) exited] [Thread 0x7ff557683700 (LWP 46333) exited] [Thread 0x7ff569b7c700 (LWP 46332) exited] [Thread 0x7ff57c075700 (LWP 46331) exited] [Thread 0x7ff58e56e700 (LWP 46330) exited] [Thread 0x7ff5a0a67700 (LWP 46329) exited] [Thread 0x7ff5b2f60700 (LWP 46328) exited] [Thread 0x7ff5c5459700 (LWP 46327) exited] [Thread 0x7ff5d7952700 (LWP 46326) exited] [Thread 0x7ff5e9e4b700 (LWP 46325) exited] [Thread 0x7ff5fc344700 (LWP 46324) exited] [Thread 0x7ff60e83d700 (LWP 46323) exited] [Thread 0x7ff620d36700 (LWP 46322) exited] [Thread 0x7ff63322f700 (LWP 46321) exited] [Thread 0x7ff645728700 (LWP 46320) exited] [Thread 0x7ff657c21700 (LWP 46319) exited] [Thread 0x7ff66a11a700 (LWP 46318) exited] [Thread 0x7ff67c613700 (LWP 46317) exited] [Thread 0x7ff68eb0c700 (LWP 46316) exited] [Thread 0x7ff6a1005700 (LWP 46315) exited] [Thread 0x7ff6b34fe700 (LWP 46314) exited] [Thread 0x7ff6c59f7700 (LWP 46313) exited] [Thread 0x7ff6d7ef0700 (LWP 46312) exited] [Thread 0x7ff6ea3e9700 (LWP 46311) exited] [Thread 0x7ff6fc8e2700 (LWP 46310) exited] [Thread 0x7ff70eddb700 (LWP 46309) exited] [Thread 0x7ff7212d4700 (LWP 46308) exited] [Thread 0x7ff7337cd700 (LWP 46307) exited] [Thread 0x7ff745cc6700 (LWP 46306) exited] [Thread 0x7ff7581bf700 (LWP 46305) exited] [Thread 0x7ff76a6b8700 (LWP 46304) exited] [Thread 0x7ff77cbb1700 (LWP 46303) exited] [Thread 0x7ff78f0aa700 (LWP 46302) exited] [Thread 0x7ff7a15a3700 (LWP 46301) exited] [Thread 0x7ff7b3a9c700 (LWP 46300) exited] [Thread 0x7ff7c5f95700 (LWP 46299) exited] [Thread 0x7ff7d848e700 (LWP 46298) exited] [Thread 0x7ff7ea987700 (LWP 46297) exited] [Thread 0x7ff7fce80700 (LWP 46296) exited] [Thread 0x7ff80f379700 (LWP 46295) exited] [Thread 0x7ff821872700 (LWP 46294) exited] [Thread 0x7ff833d6b700 (LWP 46293) exited] [Thread 0x7ff846264700 (LWP 46292) exited] [Thread 0x7ff85875d700 (LWP 46291) exited] [Thread 0x7ff86ac56700 (LWP 46290) exited] [Thread 0x7ff87d14f700 (LWP 46289) exited] [Thread 0x7ff88f648700 (LWP 46288) exited] [Thread 0x7ff8a1b41700 (LWP 46287) exited] [Thread 0x7ff8b403a700 (LWP 46286) exited] [Thread 0x7ff8c6533700 (LWP 46285) exited] [Thread 0x7ff8d8a2c700 (LWP 46284) exited] [Thread 0x7ff8eaf25700 (LWP 46283) exited] [Thread 0x7ff8fd41e700 (LWP 46282) exited] [Thread 0x7ff90f917700 (LWP 46281) exited] [Thread 0x7ff921e10700 (LWP 46280) exited] [Thread 0x7ff934309700 (LWP 46279) exited] [Thread 0x7ff946802700 (LWP 46278) exited] [Thread 0x7ff958cfb700 (LWP 46277) exited] [Thread 0x7ff96b1f4700 (LWP 46276) exited] [Thread 0x7ff97d6ed700 (LWP 46275) exited] [Thread 0x7ff98fbe6700 (LWP 46274) exited] [Thread 0x7ff9a20df700 (LWP 46273) exited] [Thread 0x7ff9b45d8700 (LWP 46272) exited] [Thread 0x7ff9c6ad1700 (LWP 46271) exited] [Thread 0x7ff9d8fca700 (LWP 46270) exited] [Thread 0x7ff9eb4c3700 (LWP 46269) exited] [Thread 0x7ff9fd9bc700 (LWP 46268) exited] [Thread 0x7ffa0feb5700 (LWP 46267) exited] [Thread 0x7ffa223ae700 (LWP 46266) exited] [Thread 0x7ffa348a7700 (LWP 46265) exited] [Thread 0x7ffa46da0700 (LWP 46264) exited] [Thread 0x7ffa59299700 (LWP 46263) exited] [Thread 0x7ffa6b792700 (LWP 46262) exited] [Thread 0x7ffa7dc8b700 (LWP 46261) exited] [Thread 0x7ffa90184700 (LWP 46260) exited] [Thread 0x7ffaa267d700 (LWP 46259) exited] [Thread 0x7ffab4b76700 (LWP 46258) exited] [Thread 0x7ffac706f700 (LWP 46257) exited] [Thread 0x7ffad9568700 (LWP 46256) exited] [Thread 0x7ffaeba61700 (LWP 46255) exited] [Thread 0x7ffafdf5a700 (LWP 46254) exited] [Thread 0x7ffb10453700 (LWP 46253) exited] [Thread 0x7ffb2294c700 (LWP 46252) exited] [Thread 0x7ffb34e45700 (LWP 46251) exited] [Thread 0x7ffb4733e700 (LWP 46250) exited] [Thread 0x7ffb59837700 (LWP 46249) exited] [Thread 0x7ffb6bd30700 (LWP 46248) exited] [Thread 0x7ffb7e229700 (LWP 46247) exited] [Thread 0x7ffb90722700 (LWP 46246) exited] [Thread 0x7ffba2c1b700 (LWP 46245) exited] [Thread 0x7ffbb5114700 (LWP 46244) exited] [Thread 0x7ffbc760d700 (LWP 46243) exited] [Thread 0x7ffbd9b06700 (LWP 46242) exited] [Thread 0x7ffbebfff700 (LWP 46241) exited] [Thread 0x7fff0a306700 (LWP 46240) exited] [Thread 0x7fff0a407700 (LWP 46239) exited] [Thread 0x7fff0a508700 (LWP 46238) exited] [Thread 0x7fff0a609700 (LWP 46237) exited] [Thread 0x7fff0a70a700 (LWP 46236) exited] [Thread 0x7fff0a80b700 (LWP 46235) exited] [Thread 0x7fff0a90c700 (LWP 46234) exited] [Thread 0x7fff0aa0d700 (LWP 46233) exited] [Thread 0x7fff0ab0e700 (LWP 46232) exited] [Thread 0x7fff0ac0f700 (LWP 46231) exited] [Thread 0x7fff0af48700 (LWP 46230) exited] [Thread 0x7ffc33fff700 (LWP 46229) exited] [Thread 0x7ffc4bfff700 (LWP 46228) exited] [Thread 0x7ffc63fff700 (LWP 46227) exited] [Thread 0x7ffc7bfff700 (LWP 46226) exited] [Thread 0x7ffc93fff700 (LWP 46225) exited] [Thread 0x7ffcabfff700 (LWP 46224) exited] [Thread 0x7ffcc3fff700 (LWP 46223) exited] [Thread 0x7ffcdbfff700 (LWP 46222) exited] [Thread 0x7ffcf3fff700 (LWP 46221) exited] [Thread 0x7ffd0bfff700 (LWP 46220) exited] [Thread 0x7ffd23fff700 (LWP 46219) exited] [Thread 0x7ffd3bfff700 (LWP 46218) exited] [Thread 0x7ffd53fff700 (LWP 46217) exited] [Thread 0x7ffd6bfff700 (LWP 46216) exited] [Thread 0x7ffd83fff700 (LWP 46215) exited] [Thread 0x7ffd9bfff700 (LWP 46214) exited] [Thread 0x7ffdb3fff700 (LWP 46213) exited] [Thread 0x7ffdcbfff700 (LWP 46212) exited] [Thread 0x7ffde3fff700 (LWP 46211) exited] [Thread 0x7ffdfbfff700 (LWP 46210) exited] [Thread 0x7ffe13fff700 (LWP 46209) exited] [Thread 0x7ffe2bfff700 (LWP 46208) exited] [Thread 0x7ffe43fff700 (LWP 46207) exited] [Thread 0x7ffe5bfff700 (LWP 46206) exited] [Thread 0x7ffe73fff700 (LWP 46205) exited] [Thread 0x7ffe8bfff700 (LWP 46204) exited] [Thread 0x7ffea3fff700 (LWP 46203) exited] [Thread 0x7ffebbfff700 (LWP 46202) exited] [Thread 0x7ffed3fff700 (LWP 46201) exited] [Thread 0x7ffeebfff700 (LWP 46200) exited] [Thread 0x7fff03fff700 (LWP 46199) exited] [Thread 0x7fff2809a700 (LWP 46198) exited] [Thread 0x7fff23fff700 (LWP 46197) exited] [Thread 0x7fff3a7e7700 (LWP 46196) exited] [Thread 0x7fff4cee3700 (LWP 46195) exited] [Thread 0x7fff65906700 (LWP 46194) exited] [Thread 0x7fff77dff700 (LWP 46193) exited] [Thread 0x7fff8af58700 (LWP 46192) exited] [Thread 0x7fffb05aa700 (LWP 46190) exited] [Thread 0x7fffc386b700 (LWP 46189) exited] [Thread 0x7fffd5da4700 (LWP 46188) exited] [Thread 0x7fffe829d700 (LWP 46187) exited] [Thread 0x7fffedae4180 (LWP 46183) exited] [Inferior 1 (process 46183) exited with code 02] Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86_64 dbus-libs-1.10.24-12.el7.x86_64 elfutils-libelf-0.172-2.el7.x86_64 elfutils-libs-0.172-2.el7.x86_64 glibc-2.17-260.el7.x86_64 glusterfs-4.1.6-1.el7721a30.rhel76hpe.x86_64 glusterfs-api-4.1.6-1.el7721a30.rhel76hpe.x86_64 glusterfs-client-xlators-4.1.6-1.el7721a30.rhel76hpe.x86_64 glusterfs-libs-4.1.6-1.el7721a30.rhel76hpe.x86_64 gssproxy-0.7.0-21.el7.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.15.1-34.el7.x86_64 libacl-2.2.51-14.el7.x86_64 libattr-2.4.46-13.el7.x86_64 libblkid-2.23.2-59.el7.x86_64 libcap-2.22-9.el7.x86_64 libcom_err-1.42.9-13.el7.x86_64 libgcc-4.8.5-36.el7.x86_64 libgcrypt-1.5.3-14.el7.x86_64 libgpg-error-1.12-3.el7.x86_64 libnfsidmap-0.25-19.el7.x86_64 libselinux-2.5-14.1.el7.x86_64 libuuid-2.23.2-59.el7.x86_64 lz4-1.7.5-2.el7.x86_64 openssl-libs-1.0.2k-16.el7.x86_64 pcre-8.32-17.el7.x86_64 systemd-libs-219-62.el7.x86_64 userspace-rcu-0.10.0-3.el7.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-18.el7.x86_64 (gdb) On Mon, Aug 19, 2019 at 11:45:20AM -0400, Daniel Gryniewicz wrote:

You can easily run Ganesha in gdb, which should give you a backtrace even if coredumps are enabled. Just run like this:

gdb /usr/bin/ganesha.nfsd (gdb) run -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F

Then, when it exits, it will tell you when if it crashed or exited normally. If it crached, you can get a backtrace. (You'll need the debug packages for Ganesha and ntirpc to get valid symbols)

Daniel

On 8/19/19 11:20 AM, Erik Jacobson wrote:

...

I turned off 'Graceless'.

I ran the daemon by hand instead of systemd, still no core dump.

/usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F

To repeat, I just chroot in to the image and do 'su - erikj' a few times in a row. The daemon then exits. No message in -F mode, no core dump.

[root@leader1 ~]# ulimit -c unlimited [root@leader1 ~]# /usr/bin/ganesha.nfsd -L /var/log/ganesha/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -F [root@leader1 ~]#

Would you like a new collection with graceless off? Let me know. Or anything else I can run to help, let me know. It's easy to reproduce the problem so no big deal to gather more data.

...

Was there any core generated when the ganesha process crashed. If yes, could you provide backtrace?

looking at the first failure observed in the logs,

17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] nfs41_Session_Get_Pointer :SESSIONS :F_DBG :Session sessionid=(16:0x010000001e82585d0100000000000000) Not Found 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] nfs4_op_sequence :SESSIONS :DEBUG :SESSIONS: DEBUG: SEQUENCE returning status NFS4ERR_BADSESSION 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] LogCompoundFH :FH :F_DBG :Current FH File Handle V4: Len=0 <null> 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] LogCompoundFH :FH :F_DBG :Saved FH File Handle V4: Len=0 <null> 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] complete_op :NFS4 :DEBUG :Status of OP_SEQUENCE in position 0 = NFS4ERR_BADSESSION, op response size is 4 total response size is 40 17/08/2019 17:40:07 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_25] complete_nfs4_compound :NFS4 :DEBUG :End status = NFS4ERR_BADSESSION lastindex = 1

This session seem to have got deleted as part of clientid expire in CREATE_SESSION

17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] nfs4_op_create_session :SESSIONS :DEBUG :Expiring 0x7f03c8001cd0 ClientID={Epoch=0x5d58821e Counter=0x00000001} CONFIRMED Client={0x7f03c8001bf0 name=(20:Linux NFSv4.1 (none)) refcount=3} t_delta=0 reservations=0 refcount=10 0x7ffff22b2da4, WSTOPPED, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)_id_expire :RW LOCK :F_DBG :Acquired mutex 0x7f03c8001d50 (&clientid->cid_mutewait4(1469, 0x7ffff22b2da4, WSTOPPED, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)5d58821e : leader1 : ganesha.nfsd-30466[svc_22] nfs_client_id_expire :CLIENT Iwait4(1469, piring {0x7f03c8001cd0 ClientID={Epoch=0x5d58821e Counter=0x00000001} CONFIRMED Client={0x7f03c8001bf0 name=(20:Linux NFSv4.1 (none)) refcount=3} t_delta=0 reservations=0 refcount=10} ... 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] hashtable_getlatch :SESSIONS :F_DBG :Get (null) returning Value=0x7f03c0001840 {session 0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)}} 17/08/2019 17:40:05 : epoch 5d58821e : leader1 : ganesha.nfsd-30466[svc_22] hashtable_deletelatched :SESSIONS :F_DBG :Delete (null) Key=0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)} Value=0x7f03c0001840 {session 0x7f03c0001840 {sessionid=(16:0x010000001e82585d0100000000000000)}} index=1 rbt_hash=1 was removed

I see a potential bug in nfs4_op_create_session --

156 rc = nfs_client_id_get_confirmed(clientid, &conf); 172 client_record = conf->cid_client_record; 173 found = conf;

366 /* add to head of session list (encapsulate?) */ 367 PTHREAD_MUTEX_lock(&found->cid_mutex); 368 glist_add(&found->cid_cb.v41.cb_session_list, 369 &nfs41_session->session_link); 370 PTHREAD_MUTEX_unlock(&found->cid_mutex); 371

427 if (conf != NULL && conf->cid_clientid != clientid) { 428 /* Old confirmed record - need to expire it */ 429 if (isDebug(component)) { 430 char str[LOG_BUFF_LEN] = "\0"; 431 struct display_buffer dspbuf = {sizeof(str), str, str}; 432 433 display_client_id_rec(&dspbuf, conf); 434 LogDebug(component, "Expiring %s", str); 435 }

Old clientid is expired after adding the session created to its cb_session_list. Thus when this old clientid is getting cleaned, it might have deleted even the newly created valid session. This seem to have caused ERR_BADSESSION and probably ERR_EXPIRED,ERR_STALE_CLIENTID errors as well.

Also since graceless is set to TRUE, clients were not able to claim their previous state. I am not sure if this can cause any application error but ideally grace period should be as long as lease period set to allow clients to recover their lost state.

Request Frank/Dan to comment.

Thanks, Soumya

On 8/18/19 4:18 AM, Erik Jacobson wrote:

...

Hello. I'm statring a new thread for this problem.

I have a 3x3 Gluster Volume and I'm trying to use Ganesha for NFS services.

One of the 9 server nodes, I have enabled Ganesha NFS server on one.

The volume is being used being host clients with NFS roots.

A long separate thread shows how I got to this point but what works on the client side is:

RHEL 7.6 aarch64 4.14.0-115.el7a.aarch64 RHEL 7.6 x86_64 3.10.0-957.el7.x86_64

OverlayFS - NFS v4 underdir with TMPFS overlay.

The Ganesha server has Allow_Numeric_Owners = True; Only_Numeric_Owners = True; Disable_ACL = TRUE;

Disable_ACL is required for the aarch64 overlay to properly read non-root files. (However, Disable_ACL must be false for aarch64 if you are using NFS v3 strangely).

The x86_64 node fully boots through full init/systemd startup to the login prompt.

When I startup the aarch64 node, it gets various degrees of done... then both NFS clients freeze up 100%.

Restarting nfs-ganesha gets them going for a moment, then they freeze again. It turned out in some cases the nfs-ganesha daemon was present during the freeze but no longer serving the nodes. However, a more common case (and the captured one) is nfs-ganesha is gone.

I will attach a tarball with a bunch of information on the problem including the config file I used, debugging logs, and some traces.

Ganesha 2.8.2 - Ganesha, Gluster servers x86_64

Since the aarch64 node causes Ganesha to crash early, and the debug log can get to 2GB quickly, I set up a test case as follows:

Tracing starts... - x86_64 fully nfs-root-booted, it comes up fine. * Actively using nfs for root during tests below - aarch64 node - boot to the miniroot env (a "fat" initrd that has more tools and from which we do the NFS mount) - It stops before switching control to the init start to run the tests like below. - cp'd /dev/null to ganesha log here - started the tcpdump to the problem node - Ran the following. Ganesha died at 'wc -l', also notice the Input/Output error on the first attempt:

bash-4.2# bash reset4.sh + umount /a umount: /a: not mounted + umount /root_ro_nfs umount: /root_ro_nfs: not mounted + umount /rootfs.rw + mount -o ro,nolock 172.23.255.249:/cm_shared/image/images_ro_nfs/rhel76-aarch64-newkernel /root_ro_nfs + mount -t tmpfs -o mpol=interleave tmpfs /rootfs.rw + mkdir /rootfs.rw/upperdir + mkdir /rootfs.rw/work + mount -t overlay overlay -o lowerdir=/root_ro_nfs,upperdir=/rootfs.rw/upperdir,workdir=/rootfs.rw/work /a bash-4.2# chroot /a chroot: failed to run command '/bin/sh': Input/output error bash-4.2# chroot /a sh: no job control in this shell sh-4.2# ls /usr/bin|wc -l

- When the above froze and ganesha died, I stopped tcpdump and collected the pieces in to a tarball.

See attached.

Erik

_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org

Erik Jacobson Software Engineer

erik.jacobson@hpe.com +1 612 851 0550 Office

Eagan, MN hpe.com

_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org

Erik Jacobson Software Engineer erik.jacobson@hpe.com +1 612 851 0550 Office Eagan, MN hpe.com

On 8/20/19 10:55 AM, Soumya Koduri wrote:

Ganesha spawns a child process. Can you try

#gdb /usr/bin/ganesha.nfsd (gdb)  set follow-fork-mode child (gdb) run -f /etc/ganesha/ganesha.conf ......

-Soumya

On 8/19/19 10:16 PM, Erik Jacobson wrote:

...

[Inferior 1 (process 46183) exited with code 02]

Or: (gdb) run -F will keep it in the foregroud. Daniel

Looking closer, -F is already there, to Ganesha's not forking. This means it's really exiting. Can you run with NIV_FULL_DEBUG instead, and post the log?

It's very interesting. I returned to this after doing a lot of work unrelated to overlay. It was going well. I put the test case back and it won't fail. It was 100% fail before. The full boots are working fine too, the system comes up just fine. I've never seen it. So I rebooted the leader (ganesha server, 1 of 9 gluster servers) to get a fresh start. It still worked afterwards. The image was only minimally changed on the gluster storage - adding a few top level directories. I returned to an older image (chrootable directory tree on the server), no problems there either. I'm frustrated that something that was failing for over a week is working now since it was important to me to help you fix this. Since I know Ganesha was crashing last year with overlay and aarch64 but I wasn't in a position to report it, I feel bad that I'm unable to make it happen on my tiny system any more. This issue is what made me switch to Gluster NFS back then. No idea what happened or how to make it go back to failing again :( I will be trying this week to start over with the gluster storage and Ganesha installation to see if I can get it to happen again. In the mean time I feel bad that I can't give you more information at this time. Erik

Good morning. Since the problem didn't happen for me any more in the lab, I tried it on the ARM supercomputer during what may be my last slot before it moves to a place I can't get to any more network-wise. I am not allowed to send logs directly without them clearing them for release. However, when using overlay with nfs v4 and the ganesha config file below... While booting 2592 nodes spread across 9 Gluster/Ganesha servers, the ganesha daemons all died on all 9 servers. This run was full debug with the config below. I'm going to archive these logs so that I can get them blessed for release later. I'll mention that the logs seem to have this at the end: <snip> :NFS4 :F_DBG :Maximum allowed attr index = 76 I am going back in the thread to find the other log setup you wanted for one more run. Then I will switch away from overlayfs and use TMPFS overmounts for the writable areas instead, which seemed to be more reliable in the lab. If you happen to be reading this now and need me to collect anything in a different way, let me know. After my time slot I may never get access again to this specific system. The next large systems I'll have access to months from now will be x86_64 based clusters. NFS_CORE_PARAM { Clustered = TRUE; RPC_Max_Connections = 2048; } NFSv4 { Graceless = True; Allow_Numeric_Owners = True; Only_Numeric_Owners = True; } EXPORT_DEFAULTS { Squash = none; } Disable_ACL = TRUE; EXPORT { # Export Id (mandatory, each EXPORT must have a unique Export_Id) Export_Id = 10; Disable_ACL = TRUE; # Exported path (mandatory) Path = "/cm_shared"; # assuming 'testvol' is the Gluster volume name # Pseudo Path (required for NFS v4) Pseudo = "/cm_shared"; # Required for access (default is None) # Could use CLIENT blocks instead Access_Type = RW; Squash = none; # Security flavor supported SecType = "sys"; # Exporting FSAL FSAL { Name = "GLUSTER"; Hostname = "127.0.0.1"; # IP of one of the nodes in the trusted pool Volume = "cm_shared"; } } LOG { Components { ALL = FULL_DEBUG; } }

Good morning. Since the problem didn't happen for me any more in the lab, I tried it on the ARM supercomputer during what may be my last slot before it moves to a place I can't get to any more network-wise.

I am not allowed to send logs directly without them clearing them for release. However, when using overlay with nfs v4 and the ganesha config file below... While booting 2592 nodes spread across 9 Gluster/Ganesha servers, the ganesha daemons all died on all 9 servers.

This run was full debug with the config below.

I'm going to archive these logs so that I can get them blessed for release later.

I'll mention that the logs seem to have this at the end:

<snip> :NFS4 :F_DBG :Maximum allowed attr index = 76

That's the max for NFS v4.1, so nothing special there, however, it does indicate a GETATTR was the last thing processed.

I am going back in the thread to find the other log setup you wanted for one more run. Then I will switch away from overlayfs and use TMPFS overmounts for the writable areas instead, which seemed to be more reliable in the lab.

If you happen to be reading this now and need me to collect anything in a different way, let me know. After my time slot I may never get access again to this specific system. The next large systems I'll have access to months from now will be x86_64 based clusters.

NFS_CORE_PARAM { Clustered = TRUE; RPC_Max_Connections = 2048; }

NFSv4 { Graceless = True; Allow_Numeric_Owners = True; Only_Numeric_Owners = True; }

EXPORT_DEFAULTS { Squash = none; }

Disable_ACL = TRUE;

EXPORT { # Export Id (mandatory, each EXPORT must have a unique Export_Id) Export_Id = 10; Disable_ACL = TRUE;

# Exported path (mandatory) Path = "/cm_shared"; # assuming 'testvol' is the Gluster volume name

# Pseudo Path (required for NFS v4) Pseudo = "/cm_shared";

# Required for access (default is None) # Could use CLIENT blocks instead Access_Type = RW;

Squash = none;

# Security flavor supported SecType = "sys";

# Exporting FSAL FSAL { Name = "GLUSTER"; Hostname = "127.0.0.1"; # IP of one of the nodes in the trusted pool Volume = "cm_shared"; }

}

LOG { Components { ALL = FULL_DEBUG; } } _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org