That's normal for a worker thread. They wait on the condition variable, until work is available, then wake up and do the work. op_ctx is safe, it's stored in thread-local-storage, so it's completely thread safe. Usually, on a hung Ganesha, you'll find at least one thread with a long backtrace, waiting on something at the bottom, or hard looping, or something like that. Along that call path it's holding a lock, that other threads are waiting on. One possibility is that the async is messed up somehow, and sockets aren't being released back to epoll, so we stop listening on them. You can turn on ntirpc tracing (set TIRPC = FULL_DEBUG in Components, and RPC_Debug_Flags = 0xffffffff in NFS_CORE_PARAM). It should then log xprts being added and removed from epoll, and you can look at that. Daniel On 12/18/20 4:47 AM, Chakra Divi wrote:
Hi Daniel,
Thanks for responding. I have attached the hung ganesha and captured the backtraces for all threads - all worker threads are waiting on the conditional variable. I could see no locks as such in any of the threads. Is there anything i need to take care while updating op_ctx ?
backtrace of one of the worker thread ************************************************************************* (gdb) bt #0 pthread_cond_timedwait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_timedwait.S:225 #1 0x00007f98a96a3d7b in work_pool_thread (arg=0x7f9899373020) at /root/chakra/nfs-ganesha_V3-stable/nfs-ganesha/src/libntirpc/src/work_pool.c:215 #2 0x00007f98a9e9b6ba in start_thread (arg=0x7f987b2fe700) at pthread_create.c:333 #3 0x00007f98a99c941d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 *************************************************************************
fio command im using - see weird output...at the end ************************************************************************* root@testnode:~# fio --direct=1 --name=testfile --directory=/nfs4/testnode/test --thread --runtime=120 --time_based --refill_buffers --overwrite=0 --ioengine=libaio --rw=read --bs=128k --numjobs=4 --filesize=4g --fallocate=none --group_reporting --iodepth=512 testfile: (g=0): rw=read, bs=128K-128K/128K-128K/128K-128K, ioengine=libaio, iodepth=512 ... fio-2.2.10 Starting 4 threads testfile: Laying out IO file(s) (1 file(s) / 4096MB) testfile: Laying out IO file(s) (1 file(s) / 4096MB) testfile: Laying out IO file(s) (1 file(s) / 4096MB) testfile: Laying out IO file(s) (1 file(s) / 4096MB) Jobs: 4 (f=4): [R(4)] [inf% done] [0KB/0KB/0KB /s] [0/0/0 iops] [eta 1158050441d:06h:43m:07s] *************************************************************************
Regards, Chakra _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org