12:28 p.m.
Maybe this piece of the log maybe more useful:
0x39c62a80 is freed, but mdcache_readdir_chunked still has a pointer to it?
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/Protocols/NFS/nfs3_readdirplus.c:151 :nfs3_readdirplus :NFS READDIR :DEBUG :NFS
READDIR: DEBUG: REQUEST PROCESSING: Calling NFS3_READDIRPLUS handle: File Handle V3:
Len=16 430000010a5c00000000000000010000
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/Protocols/NFS/nfs3_readdirplus.c:183 :nfs3_readdirplus :NFS READDIR :DEBUG
:NFS3_READDIRPLUS: dircount=16384 begin_cookie=0 estimated_num_entries=50, mem_left=14625
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892
:mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=cb4fc9bfcd8aca61
fsal=0x7fa690762d20 key=0x5c000000000000000100
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:930
:mdcache_find_keyed_reason :INODE :F_DBG :Found entry 0x80cb4d0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE
:F_DBG :attrs obj attributes Request Mask=0001dfce Valid Mask=0015dfce Supported
Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe
atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/Protocols/NFS/nfs3_readdirplus.c:436 :nfs3_readdirplus_callback :NFS READDIR
:DEBUG :Callback for . cookie 1
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1140 :mdc_lookup :NFS READDIR
:F_DBG :NFS READDIR: FULLDEBUG: Lookup ..
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1150 :mdc_lookup :NFS READDIR
:F_DBG :NFS READDIR: FULLDEBUG: Lookup parent (..) of 0x80cb4d0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892
:mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=bdca7caa2ce92921
fsal=0x7fa690762d20 key=0x02000000000000000100
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:930
:mdcache_find_keyed_reason :INODE :F_DBG :Found entry 0xab4200
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE
:F_DBG :attrs obj attributes Request Mask=0001dfce Valid Mask=0015dfce Supported
Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe
atime=Tue Feb 05 09:56:32 2019 UTC mtime=Tue Feb 05 09:56:32 2019 UTC
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/Protocols/NFS/nfs3_readdirplus.c:436 :nfs3_readdirplus_callback :NFS READDIR
:DEBUG :Callback for .. cookie 2
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/Protocols/NFS/nfs3_readdirplus.c:307 :nfs3_readdirplus :NFS READDIR :DEBUG
:Readdirplus3 -> Call to fsal_readdir, cookie=0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:853 :mdcache_refresh_attrs
:INODE :F_DBG :attrs obj attributes Request Mask=0015dfee Valid Mask=0015dfce Supported
Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe
atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE
:F_DBG :attrs obj attributes Request Mask=000006a0 Valid Mask=0015dfce Supported
Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe
atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE
:F_DBG :attrs obj attributes Request Mask=000006a0 Valid Mask=0015dfce Supported
Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe
atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:557 :mdcache_readdir :NFS
READDIR :DEBUG :NFS READDIR: DEBUG: Calling mdcache_readdir_chunked whence=0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2778 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Starting chunked READDIR for 0x80cb4d0,
MDCACHE_TRUST_CONTENT true, MDCACHE_TRUST_DIR_CHUNKS true
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2812 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Readdir chunked next_ck=0x0 look_ck=1bec77
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2969 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: found dirent in cached chunk 0x6f70300 dirent
0x39c62a80 key-file-5m-0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2977 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: About to read directory=0x80cb4d0 cookie=0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892
:mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=39db57ea4d9d4b74
fsal=0x7fa690762d20 key=0x74ec1b00000000000100
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3007 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Lookup by key for key-file-5m-0 failed,
lookup by name now
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2812 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Readdir chunked next_ck=0x0 look_ck=1bec77
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2969 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: found dirent in cached chunk 0x6f70300 dirent
0x39c62a80 key-file-5m-0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2977 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: About to read directory=0x80cb4d0 cookie=0
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892
:mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=39db57ea4d9d4b74
fsal=0x7fa690762d20 key=0x74ec1b00000000000100
Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3007 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Lookup by key for key-file-5m-0 failed,
lookup by name now
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892
:mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=3d259480a5e679c3
fsal=0x7fa690762d20 key=0x076fbf00000000000100
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:558 :mdcache_lru_clean :INODE
:F_DBG :Trusting op_ctx export id 1
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:662 :mdcache_new_entry :INODE
:DEBUG :Adding a REGULAR_FILE, entry=0x1d6b4a10
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:757 :mdcache_new_entry :INODE
:F_DBG :New entry 0x1d6b4a10 added with fh_hk.key hk=3d259480a5e679c3 fsal=0x7fa690762d20
key=0x076fbf00000000000100
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:112
:mdcache_alloc_and_check_handle :INODE :F_DBG :lookup Created entry 0x1d6b4a10 FSAL FOO
for key-file-5m-0
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1446 :mdcache_dirent_add
:INODE :F_DBG :Add dir entry key-file-5m-0
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:323 :mdcache_avl_insert :NFS
READDIR :F_DBG :NFS READDIR: FULLDEBUG: Insert dir entry 0x39cec330 key-file-5m-0
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:381 :mdcache_avl_insert :NFS
READDIR :DEBUG :NFS READDIR: DEBUG: Already existent when inserting new dirent on
entry=0x80cb4d0 name=key-file-5m-0
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:402 :mdcache_avl_insert :NFS
READDIR :F_DBG :NFS READDIR: FULLDEBUG: Keys for key-file-5m-0 don't match
v=hk=3d259480a5e679c3 fsal=0x7fa690762d20 key=0x076fbf00000000000100
v2=hk=39db57ea4d9d4b74 fsal=0x7fa690762d20 key=0x74ec1b00000000000100
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:167 :unchunk_dirent :NFS READDIR
:F_DBG :NFS READDIR: FULLDEBUG: Unchunking 0x39c62a80 key-file-5m-0
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:245 :mdcache_avl_remove :NFS
READDIR :F_DBG :NFS READDIR: FULLDEBUG: Just freed dirent 0x39c62a80 from chunk 0x6f70300
parent 0x80cb4d0
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:369 :mdcache_avl_insert :NFS
READDIR :F_DBG :NFS READDIR: FULLDEBUG: Inserted dirent key-file-5m-0 with ckey
hk=3d259480a5e679c3 fsal=0x7fa690762d20 key=0x076fbf00000000000100
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1702 :place_new_dirent :NFS
READDIR :F_DBG :NFS READDIR: FULLDEBUG: Could not add key-file-5m-0 to chunk for directory
0x80cb4d0, compute_readdir_cookie failed
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2049 :place_new_dirent :NFS
READDIR :F_DBG :NFS READDIR: FULLDEBUG: Entry 0x80cb4d0 clearing MDCACHE_DIR_POPULATED,
MDCACHE_TRUST_DIR_CHUNKS
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3112 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Setting next_ck=1bec77
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE
:F_DBG :attrs obj attributes Request Mask=0001dfce Valid Mask=0015dfce Supported
Mask=0015dfee REGULAR_FILE numlinks=0x1 size=0x400 mode=0666 owner=0xfffe group=0xfffe
atime=Tue Feb 05 23:20:22 2019 UTC mtime=Tue Feb 05 23:20:22 2019 UTC
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/Protocols/NFS/nfs3_readdirplus.c:436 :nfs3_readdirplus_callback :NFS READDIR
:DEBUG :Callback for key-file-5m-0 cookie 1830007
Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56]
/src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3151 :mdcache_readdir_chunked
:NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: dirent = 0x39c62a80 key-file-5m-0, cb_result
= DIR_CONTINUE, eod = false