[NFS-Ganesha-Devel] Re: Crash in mdc_lookup_uncached with a bad name pointer

Maybe this piece of the log maybe more useful: 0x39c62a80 is freed, but mdcache_readdir_chunked still has a pointer to it? Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/Protocols/NFS/nfs3_readdirplus.c:151 :nfs3_readdirplus :NFS READDIR :DEBUG :NFS READDIR: DEBUG: REQUEST PROCESSING: Calling NFS3_READDIRPLUS handle: File Handle V3: Len=16 430000010a5c00000000000000010000 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/Protocols/NFS/nfs3_readdirplus.c:183 :nfs3_readdirplus :NFS READDIR :DEBUG :NFS3_READDIRPLUS: dircount=16384 begin_cookie=0 estimated_num_entries=50, mem_left=14625 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892 :mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=cb4fc9bfcd8aca61 fsal=0x7fa690762d20 key=0x5c000000000000000100 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:930 :mdcache_find_keyed_reason :INODE :F_DBG :Found entry 0x80cb4d0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=0001dfce Valid Mask=0015dfce Supported Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/Protocols/NFS/nfs3_readdirplus.c:436 :nfs3_readdirplus_callback :NFS READDIR :DEBUG :Callback for . cookie 1 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1140 :mdc_lookup :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Lookup .. Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1150 :mdc_lookup :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Lookup parent (..) of 0x80cb4d0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892 :mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=bdca7caa2ce92921 fsal=0x7fa690762d20 key=0x02000000000000000100 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:930 :mdcache_find_keyed_reason :INODE :F_DBG :Found entry 0xab4200 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=0001dfce Valid Mask=0015dfce Supported Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe atime=Tue Feb 05 09:56:32 2019 UTC mtime=Tue Feb 05 09:56:32 2019 UTC Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/Protocols/NFS/nfs3_readdirplus.c:436 :nfs3_readdirplus_callback :NFS READDIR :DEBUG :Callback for .. cookie 2 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/Protocols/NFS/nfs3_readdirplus.c:307 :nfs3_readdirplus :NFS READDIR :DEBUG :Readdirplus3 -> Call to fsal_readdir, cookie=0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:853 :mdcache_refresh_attrs :INODE :F_DBG :attrs obj attributes Request Mask=0015dfee Valid Mask=0015dfce Supported Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=000006a0 Valid Mask=0015dfce Supported Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=000006a0 Valid Mask=0015dfce Supported Mask=0015dfee DIRECTORY numlinks=0x1 size=0x0 mode=0777 owner=0xfffe group=0xfffe atime=Tue Feb 05 09:58:19 2019 UTC mtime=Tue Feb 05 09:58:19 2019 UTC Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:557 :mdcache_readdir :NFS READDIR :DEBUG :NFS READDIR: DEBUG: Calling mdcache_readdir_chunked whence=0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2778 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Starting chunked READDIR for 0x80cb4d0, MDCACHE_TRUST_CONTENT true, MDCACHE_TRUST_DIR_CHUNKS true Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2812 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Readdir chunked next_ck=0x0 look_ck=1bec77 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2969 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: found dirent in cached chunk 0x6f70300 dirent 0x39c62a80 key-file-5m-0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2977 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: About to read directory=0x80cb4d0 cookie=0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892 :mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=39db57ea4d9d4b74 fsal=0x7fa690762d20 key=0x74ec1b00000000000100 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3007 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Lookup by key for key-file-5m-0 failed, lookup by name now Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2812 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Readdir chunked next_ck=0x0 look_ck=1bec77 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2969 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: found dirent in cached chunk 0x6f70300 dirent 0x39c62a80 key-file-5m-0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2977 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: About to read directory=0x80cb4d0 cookie=0 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892 :mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=39db57ea4d9d4b74 fsal=0x7fa690762d20 key=0x74ec1b00000000000100 Tue Feb 05 23:20:17 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3007 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Lookup by key for key-file-5m-0 failed, lookup by name now Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:892 :mdcache_find_keyed_reason :INODE :F_DBG :Looking for hk=3d259480a5e679c3 fsal=0x7fa690762d20 key=0x076fbf00000000000100 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:558 :mdcache_lru_clean :INODE :F_DBG :Trusting op_ctx export id 1 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:662 :mdcache_new_entry :INODE :DEBUG :Adding a REGULAR_FILE, entry=0x1d6b4a10 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:757 :mdcache_new_entry :INODE :F_DBG :New entry 0x1d6b4a10 added with fh_hk.key hk=3d259480a5e679c3 fsal=0x7fa690762d20 key=0x076fbf00000000000100 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:112 :mdcache_alloc_and_check_handle :INODE :F_DBG :lookup Created entry 0x1d6b4a10 FSAL FOO for key-file-5m-0 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1446 :mdcache_dirent_add :INODE :F_DBG :Add dir entry key-file-5m-0 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:323 :mdcache_avl_insert :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Insert dir entry 0x39cec330 key-file-5m-0 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:381 :mdcache_avl_insert :NFS READDIR :DEBUG :NFS READDIR: DEBUG: Already existent when inserting new dirent on entry=0x80cb4d0 name=key-file-5m-0 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:402 :mdcache_avl_insert :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Keys for key-file-5m-0 don't match v=hk=3d259480a5e679c3 fsal=0x7fa690762d20 key=0x076fbf00000000000100 v2=hk=39db57ea4d9d4b74 fsal=0x7fa690762d20 key=0x74ec1b00000000000100 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:167 :unchunk_dirent :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Unchunking 0x39c62a80 key-file-5m-0 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:245 :mdcache_avl_remove :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Just freed dirent 0x39c62a80 from chunk 0x6f70300 parent 0x80cb4d0 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_avl.c:369 :mdcache_avl_insert :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Inserted dirent key-file-5m-0 with ckey hk=3d259480a5e679c3 fsal=0x7fa690762d20 key=0x076fbf00000000000100 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1702 :place_new_dirent :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Could not add key-file-5m-0 to chunk for directory 0x80cb4d0, compute_readdir_cookie failed Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2049 :place_new_dirent :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Entry 0x80cb4d0 clearing MDCACHE_DIR_POPULATED, MDCACHE_TRUST_DIR_CHUNKS Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3112 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: Setting next_ck=1bec77 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:927 :mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=0001dfce Valid Mask=0015dfce Supported Mask=0015dfee REGULAR_FILE numlinks=0x1 size=0x400 mode=0666 owner=0xfffe group=0xfffe atime=Tue Feb 05 23:20:22 2019 UTC mtime=Tue Feb 05 23:20:22 2019 UTC Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/Protocols/NFS/nfs3_readdirplus.c:436 :nfs3_readdirplus_callback :NFS READDIR :DEBUG :Callback for key-file-5m-0 cookie 1830007 Tue Feb 05 23:20:24 2019 UTC 24320[::ffff:10.0.136.88] [svc_56] /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3151 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS READDIR: FULLDEBUG: dirent = 0x39c62a80 key-file-5m-0, cb_result = DIR_CONTINUE, eod = false