[NFS-Ganesha-Devel] Re: V2.7.2 Crash in mdcache_readdir_chunked

If the key can change on eviction, then you cannot support NFSv4. The sequence is this: Client sends NFS4_OP_GETFH for object o ... time passes ... Object o is evicted Client sends NFS4_OP_PUTFH for object o The key must find the same underlying filesystem object as was referenced when the GETFH was used. This is a requirement of the protocol. If the object can have a different key each time it's instantiated, then this will fail. In addition, it's an assumption throughout MDCACHE that a key is a valid way to find/match an object, even across invalidation. This is the entire basis of dirents in MDCACHE. So, I don't think this is a fixable problem. Either the object has to be re-instantiated with the same key, or you can't evict the object. Are you sure there's noting you can use as a stable key for the objects? FSAL_RGW, for example, uses a hash of the bucket and object names. Daniel On 3/4/19 12:03 PM, Rungta, Vandana wrote: > It is a custom FSAL for a caching filesystem. It is a characteristic of the FSAL that the key for the object can change if the object was evicted and then re- fetched. > > While changing the key for the same object is a corner case, a dedicated stress tests such as we are running can trigger it. We have upcall support on our roadmap so we will eventually be able to invalidate mdcache on evictions which I think would avoid this whole area. In the meantime, I'd like to find a way to prevent the crash. Since the code in mdcache_avl_insert handles this case correctly, it seems that the only change needed is for mdcache_readdir_chunked to do so as well.

> > What do you think of the idea of re-fetching the dirent when mdc_lookup_uncached returns success? > > Thanks, > Vandana > > On 3/4/19, 7:39 AM, "Daniel Gryniewicz" <dang(a)redhat.com&gt; wrote: > > This seems like a FSAL bug, I think. The new object found by lookup() > should have the same key as the old object did. This is required so > that the NFS handles match. If the keys do match, then > mdcache_avl_insert() deletes the new dirent, and keeps the old dirent. > What FSAL is this? > > Daniel > > On 3/3/19 12:37 PM, Rungta, Vandana wrote: > > Testing with version 2.7.2 of Ganesha with the following patch applied. > > > > https://github.com/nfs-ganesha/nfs- ganesha/commit/23c05a5a3e37a8bd960073ef591ace718fca050a > > > > Evidence in the log files point to the crash because mdc_lookup_uncached > > can free the dirent that continues to be used by > > mdcache_readdir_chunked. In mdcache_avl_insert it finds an existing > > dirent, so removes and frees the old dirent and inserts the new one. > > mdcache_readdir_chunked still has a pointer to the dirent that is freed. > > > > Suggest a change in mdcache_readdir_chunked (circa line 3046) – if the > > call to mdc_lookup_uncached returns success, to jump back to again to > > get the dirent. ( I am not sure which states need to be restored before > > jumping back to again) > > > > status = mdc_lookup_uncached(directory, > > dirent->name, > > > > &entry, NULL); > > > > if(FSAL_IS_ERROR(status)) { > > > > . . . > > > > returnstatus; > > > > } > > > > + mdcache_put(entry); > > > > + first_pass = true; > > > > + chunk = NULL; > > > > + gotoagain; > > > > } > > > > Relevant lines from the log file: > > > > mdcache_handle.c:557 :mdcache_readdir :NFS READDIR :DEBUG :NFS READDIR: > > DEBUG: Calling mdcache_readdir_chunked whence=0 > > > > mdcache_helpers.c:2934 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS > > READDIR: FULLDEBUG: found dirent in cached chunk 0x51e375e0 *dirent > > 0x4a2eadf0* created-on-rd-1 > > > > mdcache_helpers.c:2976 :mdcache_readdir_chunked :NFS READDIR :F_DBG :NFS > > READDIR: FULLDEBUG: Lookup by key for created-on-rd-1 failed, lookup by > > name now > > > > (gets the write lock and repeats) > > > > mdcache_helpers.c:663 :mdcache_new_entry :INODE :DEBUG :Adding a > > REGULAR_FILE, entry=0x17ab9150 > > > > mdcache_helpers.c:758 :mdcache_new_entry :INODE :F_DBG :New entry > > 0x17ab9150 added with fh_hk.key hk=d68582e4df05b9f5 fsal=0x7f4e48b0ed20 > > key=0xfb956a03000000000100 > > > > mdcache_handle.c:112 :mdcache_alloc_and_check_handle :INODE :F_DBG > > :lookup Created entry 0x17ab9150 FSAL FOO for created-on-rd-1 > > > > mdcache_helpers.c:1447 :mdcache_dirent_add :INODE :F_DBG :Add dir entry > > created-on-rd-1 > > > > mdcache_avl.c:327 :mdcache_avl_insert :NFS READDIR :F_DBG :NFS READDIR: > > FULLDEBUG: Insert dir entry 0x51731610 created-on-rd-1 > > > > mdcache_avl.c:385 :mdcache_avl_insert :NFS READDIR :DEBUG :NFS READDIR: > > DEBUG: Already existent when inserting new dirent on entry=0x329efe50 > > name=created-on-rd-1 > > > > mdcache_avl.c:406 :mdcache_avl_insert :NFS READDIR :F_DBG :NFS READDIR: > > FULLDEBUG: Keys for created-on-rd-1 don't match v=hk=d68582e4df05b9f5 > > fsal=0x7f4e48b0ed20 key=0xfb956a03000000000100 v2=hk=41227e96857f4696 > > fsal=0x7f4e48b0ed20 key=0xe08a5903000000000100 > > > > mdcache_avl.c:171 :unchunk_dirent :NFS READDIR :F_DBG :NFS READDIR: > > FULLDEBUG: Unchunking 0x4a2eadf0 created-on-rd-1 > > > > mdcache_avl.c:249 :mdcache_avl_remove :NFS READDIR :F_DBG :NFS READDIR: > > FULLDEBUG: Just *freed dirent 0x4a2eadf0* from chunk 0x51e375e0 parent > > 0x329efe50 > > > > mdcache_avl.c:373 :mdcache_avl_insert :NFS READDIR :F_DBG :NFS READDIR: > > FULLDEBUG: Inserted dirent created-on-rd-1 with ckey hk=d68582e4df05b9f5 > > fsal=0x7f4e48b0ed20 key=0xfb956a03000000000100 > > > > Long running tests with a windows client doing a readdir over 5 million > > files while other threads do IO. > > > > Reproduced with fsal, nfs_readdir, and inode_cache debug turned on. > > > > I am happy to provide any additional debug info from the logs that will > > help. > > > > Using host libthread_db library "/lib64/libthread_db.so.1". > > > > Core was generated by `bin/ganesha.nfsd -f etc/ganesha/ganesha.conf -p > > var/run/ganesha.pid -F'. > > > > Program terminated with signal 11, Segmentation fault. > > > > #00x00000000005146e8 in display_opaque_bytes (dspbuf=0x7f4e416a35b0, > > value=0x1f72a01d349d6820, > > > > len=1219554592) at /src/src/log/display.c:364 > > > > 364/src/src/log/display.c: No such file or directory. > > > > Missing separate debuginfos, use: debuginfo-install > > sgw-nfs-ganesha-2.0.32.0-1.x86_64 > > > > (gdb) bt > > > > #00x00000000005146e8 in display_opaque_bytes (dspbuf=0x7f4e416a35b0, > > value=0x1f72a01d349d6820, > > > > len=1219554592) at /src/src/log/display.c:364 > > > > #10x000000000053a8be in display_mdcache_key (dspbuf=0x7f4e416a35b0, > > key=0x6096508) > > > > at /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:858 > > > > #20x000000000053a9cd in mdcache_find_keyed_reason (key=0x6096508, > > entry=0x7f4e416a3738, > > > > reason=MDC_REASON_SCAN) at > > /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:890 > > > > #30x0000000000542603 in mdcache_readdir_chunked (directory=0x329efe50, > > whence=0, dir_state=0x7f4e416a3900, > > > > cb=0x43225c <populate_dirent>, attrmask=122830, eod_met=0x7f4e416a3ffb) > > > > at /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:2968 > > > > #40x0000000000530387 in mdcache_readdir (dir_hdl=0x329efe88, > > whence=0x7f4e416a38e0, > > > > dir_state=0x7f4e416a3900, cb=0x43225c <populate_dirent>, > > attrmask=122830, eod_met=0x7f4e416a3ffb) > > > > at /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:559 > > > > #50x0000000000432b83 in fsal_readdir (directory=0x329efe88, cookie=0, > > nbfound=0x7f4e416a3ffc, > > > > eod_met=0x7f4e416a3ffb, attrmask=122830, cb=0x493020 > > <nfs3_readdirplus_callback>, opaque=0x7f4e416a3fb0) > > > > at /src/src/FSAL/fsal_helper.c:1164 > > > > #60x0000000000492e79 in nfs3_readdirplus (arg=0x458a3d48, > > req=0x458a3640, res=0xee64330) > > > > at /src/src/Protocols/NFS/nfs3_readdirplus.c:310 > > > > #70x0000000000457d0e in nfs_rpc_process_request (reqdata=0x458a3640) > > > > at /src/src/MainNFSD/nfs_worker_thread.c:1328 > > > > (gdb) select-frame 3 > > > > (gdb) print *dirent > > > > $1 = {chunk_list = {next = 0x0, prev = 0xc1}, chunk = 0x41e3a6a0, > > node_name = {left = 0x4a2eade0, right = 0x0, > > > > parent = 0}, node_ck = {left = 0x0, right = 0x2, parent = 0}, > > node_sorted = {left = 0x0, right = 0x0, > > > > parent = 0}, ck = 0, eod = false, namehash = 0, ckey = {hk = 0, fsal = > > 0xc9e3c5e3b0667c56, kv = { > > > > addr = 0x1f72a01d349d6820, len = 139974203731232}}, flags = 0, name = > > 0x0, name_buffer = 0x6096538 ""} > > > > > > _______________________________________________ > > Devel mailing list -- devel(a)lists.nfs-ganesha.org > > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > > > _______________________________________________ > Devel mailing list -- devel(a)lists.nfs-ganesha.org > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org > > _______________________________________________ Devel mailing list -- devel(a)lists.nfs-ganesha.org To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org