January 2019 - Devel - NFS Ganesha List Archives

lseek gets bad offset from nfs client with ganesha/gluster which supports SEEK
by Kinglong Mee 14 Sep '20

14 Sep '20

The latest ganesha/gluster supports seek according to, https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-41#section-15.11 From the given sa_offset, find the next data_content4 of type sa_what in the file. If the server can not find a corresponding sa_what, then the status will still be NFS4_OK, but sr_eof would be TRUE. If the server can find the sa_what, then the sr_offset is the start of that content. If the sa_offset is beyond the end of the file, then SEEK MUST return NFS4ERR_NXIO. For a file's filemap as, Part 1: HOLE 0x0000000000000000 ---> 0x0000000000600000 Part 2: DATA 0x0000000000600000 ---> 0x0000000000700000 Part 3: HOLE 0x0000000000700000 ---> 0x0000000001000000 SEEK(0x700000, SEEK_DATA) gets result (sr_eof:1, sr_offset:0x70000) from ganesha/gluster; SEEK(0x700000, SEEK_HOLE) gets result (sr_eof:0, sr_offset:0x70000) from ganesha/gluster. If an application depends the lseek result for data searching, it may enter infinite loop. while (1) { next_pos = lseek(fd, cur_pos, seek_type); if (seek_type == SEEK_DATA) { seek_type = SEEK_HOLE; } else { seek_type = SEEK_DATA; } if (next_pos == -1) { return ; cur_pos = next_pos; } The lseek syscall always gets 0x70000 from nfs client for those two cases, but, if underlying filesystem is ext4/f2fs, or the nfs server is knfsd, the lseek(0x700000, SEEK_DATA) gets ENXIO. I wanna to know, should I fix the ganesha/gluster as knfsd return ENXIO for the first case? or should I fix the nfs client to return ENXIO for the first case? thanks, Kinglong Mee

4 8

Crash in mdc_lookup_uncached with a bad name pointer
by Rungta, Vandana 06 Feb '19

06 Feb '19

Hello, We are seeing the following crash with NFS Ganesha 2.7.1 – we crash further in our fsal module’s lookup method trying to use the “name” which has an invalid pointer. This has the following 3 patches applied to NFS Ganesha 2.7.1 https://github.com/nfs-ganesha/nfs-ganesha/commit/654dd706d22663c6ae6029e0c… https://github.com/nfs-ganesha/nfs-ganesha/commit/5dc6a70ed42275a4f6772b980… The most recent patch to not return dead hash entries: https://github.com/nfs-ganesha/nfs-ganesha/commit/25320e6544f6c5a045f20c514… The workload: Concurrent access from multiple threads. 1 thread continuously (in a loop) running python os.walk (i.e., readdir) of the entire filesystem, roughly ~5M files total. 5 more threads are writing a few thousand files each. When the writes complete, a single thread verifies written content, then deletes it. Then the writes repeat again. This is the same workload that causes our OOM issue. https://lists.nfs-ganesha.org/archives/list/devel@lists.nfs-ganesha.org/thr… #5 0x00007f6acd6a1dca in foo_lookup (parent=0x6fe2e420, name=0xa0 <Address 0xa0 out of bounds>, handle=0x7f6abf3255a8, attrs_out=0x7f6abf3254a0) at /opt/src/src/handle.c:364 #6 0x000000000053a7a6 in mdc_lookup_uncached (mdc_parent=0x291a4ba0, name=0xa0 <Address 0xa0 out of bounds>, new_entry=0x7f6abf325728, attrs_out=0x0) at /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:1293 #7 0x0000000000541344 in mdcache_readdir_chunked (directory=0x291a4ba0, whence=0, dir_state=0x7f6abf325900, cb=0x43217c <populate_dirent>, attrmask=122830, eod_met=0x7f6abf325ffb) at /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_helpers.c:3065 #8 0x000000000052e8c3 in mdcache_readdir (dir_hdl=0x291a4bd8, whence=0x7f6abf3258e0, dir_state=0x7f6abf325900, cb=0x43217c <populate_dirent>, attrmask=122830, eod_met=0x7f6abf325ffb) at /src/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:559 #9 0x0000000000432a76 in fsal_readdir (directory=0x291a4bd8, cookie=0, nbfound=0x7f6abf325ffc, eod_met=0x7f6abf325ffb, attrmask=122830, cb=0x492018 <nfs3_readdirplus_callback>, opaque=0x7f6abf325fb0) at /src/src/FSAL/fsal_helper.c:1158 #10 0x0000000000491e71 in nfs3_readdirplus (arg=0x10704818, req=0x10704110, res=0x3103f090) at /src/src/Protocols/NFS/nfs3_readdirplus.c:310 #11 0x00000000004574d1 in nfs_rpc_process_request (reqdata=0x10704110) at /src/src/MainNFSD/nfs_worker_thread.c:1329 #12 0x0000000000457c90 in nfs_rpc_valid_NFS (req=0x10704110) at /src/src/MainNFSD/nfs_worker_thread.c:1549 #13 0x00007f6ad115ce75 in svc_vc_decode (req=0x10704110) at /src/src/libntirpc/src/svc_vc.c:825 ---Type <return> to continue, or q <return> to quit--- #14 0x000000000044a688 in nfs_rpc_decode_request (xprt=0x18390200, xdrs=0x174943c0) at /src/src/MainNFSD/nfs_rpc_dispatcher_thread.c:1341 #15 0x00007f6ad115cd86 in svc_vc_recv (xprt=0x18390200) at /src/src/libntirpc/src/svc_vc.c:798 #16 0x00007f6ad11594d3 in svc_rqst_xprt_task (wpe=0x18390418) at /src/src/libntirpc/src/svc_rqst.c:767 #17 0x00007f6ad115994d in svc_rqst_epoll_events (sr_rec=0x2779260, n_events=1) at /src/src/libntirpc/src/svc_rqst.c:939 #18 0x00007f6ad1159be2 in svc_rqst_epoll_loop (sr_rec=0x2779260) at /src/src/libntirpc/src/svc_rqst.c:1012 #19 0x00007f6ad1159c95 in svc_rqst_run_task (wpe=0x2779260) at /src/src/libntirpc/src/svc_rqst.c:1048 #20 0x00007f6ad11625f6 in work_pool_thread (arg=0x3b5d170) at /src/src/libntirpc/src/work_pool.c:181 #21 0x00007f6ad0169de5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007f6acfa71bad in clone () from /lib64/libc.so.6 (gdb) select-frame 7 (gdb) info locals status = {major = ERR_FSAL_INVAL, minor = 0} cb_result = DIR_CONTINUE entry = 0x0 attrs = {request_mask = 122830, valid_mask = 1433550, supported = 1433582, type = REGULAR_FILE, filesize = 1024, fsid = {major = 0, minor = 0}, acl = 0x0, fileid = 47680710, mode = 438, numlinks = 1, owner = 65534, group = 65534, rawdev = {major = 0, minor = 0}, atime = {tv_sec = 1548784955, tv_nsec = 582000000}, creation = { tv_sec = 0, tv_nsec = 0}, ctime = {tv_sec = 1548784955, tv_nsec = 582000000}, mtime = {tv_sec = 1548784955, tv_nsec = 582000000}, chgtime = {tv_sec = 1548784955, tv_nsec = 582000000}, spaceused = 1024, change = 1548784955582, generation = 0, expire_time_attr = 60, fs_locations = 0x0} dirent = 0x5296230 has_write = true set_first_ck = false next_ck = 2419507 look_ck = 2419507 chunk = 0x156d7240 first_pass = true eod = false reload_chunk = false __func__ = "mdcache_readdir_chunked" __PRETTY_FUNCTION__ = "mdcache_readdir_chunked" (gdb) status = {major = ERR_FSAL_INVAL, minor = 0} cb_result = DIR_CONTINUE entry = 0x0 attrs = {request_mask = 122830, valid_mask = 1433550, supported = 1433582, type = REGULAR_FILE, filesize = 1024, fsid = {major = 0, minor = 0}, acl = 0x0, fileid = 47680710, mode = 438, numlinks = 1, owner = 65534, group = 65534, rawdev = {major = 0, minor = 0}, atime = {tv_sec = 1548784955, tv_nsec = 582000000}, creation = { tv_sec = 0, tv_nsec = 0}, ctime = {tv_sec = 1548784955, tv_nsec = 582000000}, mtime = {tv_sec = 1548784955, tv_nsec = 582000000}, chgtime = {tv_sec = 1548784955, tv_nsec = 582000000}, spaceused = 1024, change = 1548784955582, generation = 0, expire_time_attr = 60, fs_locations = 0x0} dirent = 0x5296230 has_write = true set_first_ck = false next_ck = 2419507 look_ck = 2419507 chunk = 0x156d7240 first_pass = true eod = false reload_chunk = false __func__ = "mdcache_readdir_chunked" __PRETTY_FUNCTION__ = "mdcache_readdir_chunked" (gdb) print *dirent $1 = {chunk_list = {next = 0x0, prev = 0xa1}, chunk = 0x3018770, node_name = { left = 0x7f6acfd39848 <main_arena+232>, right = 0x0, parent = 0}, node_ck = {left = 0x5296258, right = 0x0, parent = 0}, node_sorted = {left = 0xffffffff, right = 0x0, parent = 0}, ck = 0, eod = false, namehash = 8192, ckey = {hk = 1056768, fsal = 0x0, kv = {addr = 0x0, len = 134650068}}, flags = 0, name = 0xa0 <Address 0xa0 out of bounds>, name_buffer = 0x52962d8 " "} (gdb) print dirent.name $3 = 0xa0 <Address 0xa0 out of bounds> (gdb) print *0x52962d8 $4 = 32 (gdb) print dirent.name_buffer $5 = 0x52962d8 " " (gdb) print *dirent.name_buffer $6 = 32 ' ' (gdb) print *chunk $7 = {chunks = {next = 0x291a4e28, prev = 0x291a4e28}, dirents = {next = 0x2ed8c060, prev = 0x48ff2870}, parent = 0x291a4ba0, chunk_lru = {q = {next = 0x7e1c00 <CHUNK_LRU+1792>, prev = 0x4d6f5c88}, qid = LRU_ENTRY_L1, refcnt = 0, flags = 0, lane = 8, cf = 0}, reload_ck = 0, next_ck = 0, num_entries = 2500}

3 6

Re: [Nfs-ganesha-devel] Memory growth in mdcache with entries_used >> entries_hiwat
by Daniel Gryniewicz 05 Feb '19

05 Feb '19

Moving to the new list... I'm not seeing how that patch can have caused this, as an entry should have already been removed from the LRU before hitting the changes in the patch. I'll run some tests, though, and see what I can work out. Daniel On 1/28/19 1:49 PM, Kropelin, Adam via Nfs-ganesha-devel wrote: > This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org. > > > After applying https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/441566 > our long-running test is showing continually increasing memory usage. > Eventually ganesha.nfsd consumes all memory in the box and we OOM. > Looking at a core, it appears that the mdcache lru contains far more > entries than the high water mark would normally allow. > > We have… > > CacheInode { > > Dir_Chunk = 500000; > > Entries_HWMark = 500000; > > } > > …and after we run for a while, we observe from a core (obtained at > runtime using gcore)... > > (gdb) print lru_state > > $8 = {entries_hiwat = 500000, entries_used = 2437134, chunks_hiwat = > 100000, chunks_used = 2002, fds_system_imposed = 400000, fds_hard_limit > = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, > per_lane_work = 50, biggest_window = 160000, prev_fd_count = 160, > prev_time = 1548692973, fd_state = 0} > > So we have 2.4M entries with a high water mark of 500K. The difference > appears to account for the unexpected memory usage. > > This seems to be new behavior after applying the above patch, although > it’s hard to be certain because earlier we hit the core with entries > being freed twice before running into the high memory usage. > > Some more info from the core: > > (gdb) print LRU > > $2 = {{L1 = {q = {next = 0xe4788210, prev = 0x23a701f0}, id = > LRU_ENTRY_L1, size = 147689}, L2 = {q = { > > next = 0x23a8b7b0, prev = 0xe478ef80}, id = LRU_ENTRY_L2, size > = 150}, cleanup = {q = {next = 0xcca1ff70, > > prev = 0x139e2bf0}, id = LRU_ENTRY_CLEANUP, size = 748}, mtx = > {__data = {__lock = 0, __count = 0, > > __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = > {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1dd98360, > prev = 0x1100cc30}, id = LRU_ENTRY_L1, > > size = 143935}, L2 = {q = {next = 0xfa204e0, prev = 0x1e26a0a0}, > id = LRU_ENTRY_L2, size = 145}, cleanup = { > > q = {next = 0xb26f8720, prev = 0x18f1810}, id = > LRU_ENTRY_CLEANUP, size = 811}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x4903dc0, > prev = 0x96cb4320}, id = LRU_ENTRY_L1, > > size = 141362}, L2 = {q = {next = 0x2eea11f0, prev = 0x48fd050}, > id = LRU_ENTRY_L2, size = 147}, cleanup = { > > q = {next = 0xdc74e5b0, prev = 0xca6f990}, id = > LRU_ENTRY_CLEANUP, size = 817}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x79dcf850, > prev = 0x7d9445d0}, id = LRU_ENTRY_L1, > > size = 122171}, L2 = {q = {next = 0x7e823b30, prev = 0x79dcfc90}, > id = LRU_ENTRY_L2, size = 146}, cleanup = { > > q = {next = 0x3245ddb0, prev = 0x1121e510}, id = > LRU_ENTRY_CLEANUP, size = 761}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1a94af70, > prev = 0xec32de0}, id = LRU_ENTRY_L1, > > size = 142951}, L2 = {q = {next = 0xec25300, prev = 0x1a944200}, > id = LRU_ENTRY_L2, size = 150}, cleanup = { > > q = {next = 0xcf8eada0, prev = 0xba9afd0}, id = > LRU_ENTRY_CLEANUP, size = 759}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x49b54c90, > prev = 0x1896b90}, id = LRU_ENTRY_L1, > > size = 133691}, L2 = {q = {next = 0x413dbca0, prev = 0x66d15ce0}, > id = LRU_ENTRY_L2, size = 142}, cleanup = { > > q = {next = 0xe249dcb0, prev = 0x135e9ae0}, id = > LRU_ENTRY_CLEANUP, size = 804}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x25eba620, > prev = 0x96783780}, id = LRU_ENTRY_L1, > > ---Type <return> to continue, or q <return> to quit--- > > size = 146051}, L2 = {q = {next = 0x1d179420, prev = 0x25eb38b0}, > id = LRU_ENTRY_L2, size = 150}, cleanup = { > > q = {next = 0xab572d30, prev = 0x11a0d550}, id = > LRU_ENTRY_CLEANUP, size = 765}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x15bc6d50, > prev = 0x3c292070}, id = LRU_ENTRY_L1, > > size = 135767}, L2 = {q = {next = 0x5d1fad10, prev = 0x15bbffe0}, > id = LRU_ENTRY_L2, size = 148}, cleanup = { > > q = {next = 0x5b5c1490, prev = 0x88016f0}, id = > LRU_ENTRY_CLEANUP, size = 720}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x67772f0, > prev = 0x964eb1b0}, id = LRU_ENTRY_L1, > > size = 140729}, L2 = {q = {next = 0x30d68c80, prev = 0x6770580}, > id = LRU_ENTRY_L2, size = 147}, cleanup = { > > q = {next = 0xc4710dd0, prev = 0x874dc10}, id = > LRU_ENTRY_CLEANUP, size = 791}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x5d1c210, > prev = 0x302682d0}, id = LRU_ENTRY_L1, > > size = 140670}, L2 = {q = {next = 0x3026f040, prev = 0x5d154a0}, > id = LRU_ENTRY_L2, size = 147}, cleanup = { > > q = {next = 0xcb5487f0, prev = 0xc679490}, id = > LRU_ENTRY_CLEANUP, size = 794}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe2791600, > prev = 0x96252be0}, id = LRU_ENTRY_L1, > > size = 148189}, L2 = {q = {next = 0xec06a870, prev = 0xe2798370}, > id = LRU_ENTRY_L2, size = 145}, cleanup = { > > q = {next = 0xa4dec380, prev = 0xb29dc50}, id = > LRU_ENTRY_CLEANUP, size = 746}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe5dff590, > prev = 0x23d43b90}, id = LRU_ENTRY_L1, > > size = 147297}, L2 = {q = {next = 0x23de7e10, prev = 0xe5e06300}, > id = LRU_ENTRY_L2, size = 141}, cleanup = { > > q = {next = 0xa6b8d390, prev = 0x10229cb0}, id = > LRU_ENTRY_CLEANUP, size = 744}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1cc790a0, > prev = 0x4ba049f0}, id = LRU_ENTRY_L1, > > size = 143750}, L2 = {q = {next = 0xf39d9d0, prev = 0x1cc72330}, > id = LRU_ENTRY_L2, size = 147}, cleanup = { > > ---Type <return> to continue, or q <return> to quit--- > > q = {next = 0xd37ff620, prev = 0x13b47010}, id = > LRU_ENTRY_CLEANUP, size = 780}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xeaf4fd00, > prev = 0x25ab91d0}, id = LRU_ENTRY_L1, > > size = 146505}, L2 = {q = {next = 0x25aa4980, prev = 0xeaf56a70}, > id = LRU_ENTRY_L2, size = 150}, cleanup = { > > q = {next = 0x23c6c110, prev = 0x11e02670}, id = > LRU_ENTRY_CLEANUP, size = 788}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe3dea010, > prev = 0x2831da50}, id = LRU_ENTRY_L1, > > size = 148717}, L2 = {q = {next = 0x283247c0, prev = 0xe3df0d80}, > id = LRU_ENTRY_L2, size = 149}, cleanup = { > > q = {next = 0xb84809e0, prev = 0x87cee00}, id = > LRU_ENTRY_CLEANUP, size = 850}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xce3ced10, > prev = 0xd5f2db20}, id = LRU_ENTRY_L1, > > size = 153934}, L2 = {q = {next = 0xd5f12560, prev = 0xce3d5a80}, > id = LRU_ENTRY_L2, size = 149}, cleanup = { > > q = {next = 0xca840ac0, prev = 0xda6b170}, id = > LRU_ENTRY_CLEANUP, size = 779}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x2d292f90, > prev = 0x17851300}, id = LRU_ENTRY_L1, > > size = 138008}, L2 = {q = {next = 0x1786c8c0, prev = 0x2d2854b0}, > id = LRU_ENTRY_L2, size = 147}, cleanup = { > > q = {next = 0xc7c1e980, prev = 0xb2a3820}, id = > LRU_ENTRY_CLEANUP, size = 761}, mtx = {__data = {__lock = 0, > > __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = > 0, __list = {__prev = 0x0, __next = 0x0}}, > > __size = '\000' <repeats 39 times>, __align = 0}, iter = {active > = false, glist = 0x0, glistn = 0x0}, > > __pad0 = '\000' <repeats 63 times>}} > > (gdb) > > (gdb) print *(struct mdcache_lru__ *)0xe4788210 > > $3 = {q = {next = 0xe47814a0, prev = 0x7e0620 <LRU>}, qid = > LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xe47814a0 > > $5 = {q = {next = 0xe477a730, prev = 0xe4788210}, qid = LRU_ENTRY_L1, > refcnt = 1, flags = 0, lane = 0, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xe477a730 > > $6 = {q = {next = 0xe47739c0, prev = 0xe47814a0}, qid = LRU_ENTRY_L1, > refcnt = 1, flags = 0, lane = 0, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xe47739c0 > > $7 = {q = {next = 0xe476cc50, prev = 0xe477a730}, qid = LRU_ENTRY_L1, > refcnt = 1, flags = 0, lane = 0, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xb84809e0 > > $8 = {q = {next = 0xaa559720, prev = 0x7e12a0 <LRU+3200>}, qid = > LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, > > lane = 14, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xaa559720 > > $9 = {q = {next = 0xd9754930, prev = 0xb84809e0}, qid = > LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xd9754930 > > $10 = {q = {next = 0xa94e7f40, prev = 0xaa559720}, qid = > LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xa94e7f40 > > $11 = {q = {next = 0x185aa7d0, prev = 0xd9754930}, qid = > LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0x185aa7d0 > > $12 = {q = {next = 0xcb912a30, prev = 0xa94e7f40}, qid = > LRU_ENTRY_CLEANUP, refcnt = 1, flags = 3, lane = 14, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xcb912a30 > > $13 = {q = {next = 0xc7727370, prev = 0x185aa7d0}, qid = > LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0} > > (gdb) > > $14 = {q = {next = 0xc7727370, prev = 0x185aa7d0}, qid = > LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xf39d9d0 > > $15 = {q = {next = 0xf3a4740, prev = 0x7e10c0 <LRU+2720>}, qid = > LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12, > > cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xf3a4740 > > $16 = {q = {next = 0xf3ab4b0, prev = 0xf39d9d0}, qid = LRU_ENTRY_L2, > refcnt = 1, flags = 0, lane = 12, cf = 0} > > (gdb) print *(struct mdcache_lru__ *)0xf3ab4b0 > > $17 = {q = {next = 0xf3b2220, prev = 0xf3a4740}, qid = LRU_ENTRY_L2, > refcnt = 1, flags = 0, lane = 12, cf = 0} > > > > _______________________________________________ > Nfs-ganesha-devel mailing list > Nfs-ganesha-devel(a)lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel >

3 13

V2.8 release planned date
by Denis Kondratenko 05 Feb '19

05 Feb '19

Hi guys, what is the planned date for the V2.8 release? Thanks, -- Denis Kondratenko Engineering Manager SUSE Linux Enterprise Storage SUSE LINUX GmbH Maxfeldstraße 5 90409 Nürnberg Germany GF: SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Dilip Upmanyu, Graham Norton, HRB 21284 (AG Nürnberg)

2 1

Proposed 2.7.2
by Daniel Gryniewicz 04 Feb '19

04 Feb '19

Here's my proposed 2.7.2 patchset. I've only included bug fixes, except for the requested export cleanup. If you have other changes you want in, please propose them in this thread. 2f6d8b7d6 (HEAD -> V2.7-stable) Remove -Wabi from maintainer mode 949de68f8 Fix READDIR duplicate entries 1249a2723 Fix attribute comparison in NFS4_OP_VERIFY 20ed57717 Fix GTest build ba756d3ce nfs4_op_open: NULL pointer bug - replace goto out3 with return ebcde738d Don't call nfs_req_creds if we don't have export 7b5c6c833 Handle race while adding host to IP-name cache 9edc1fbab exports: prune off old exports after reloading config 8ea299930 exports: copy the config tree generation to the export when adding or updating 914469434 exports: add generation counter to config_root and helpers to fetch it d0d4d0201 exports: don't allow dbus remove to leave subexports disconnected in pseudoroot 5d20939bf ganesha_status.py - fix missing parens 9fa5bb085 rpm/selinux: fix %pre install and %postun uninstall of selinux policy 1d5e66792 Fix to set ATTR_RDATTR_ERR correctly as valid_mask in open2. 1d8f0d242 CEPH: do a getattr after creating a dir and applying extra attributes b62c60b49 Read/Write - Don't leak owners/states 7e7c01f23 SAL: fix multiple state reference leaks 3be24ba32 Set op_ctx for lock_avail and lock_grant ad2b9a30a FSAL_VFS, FreeBSD: upstream support for d_off c4119a3c5 Add checks for client access in NFS user access checks. a1b0888f3 FSAL_UP: add missing locks in async delegation recall 0eb7faf67 Handle NLM share FREE_ALL for windows clients 6cdc3fb22 NLM share reservation access check with owner_skip 770db8de3 Fixed dereferencing null ctx_export on nfs_read 789b56d35 fix the bug of import config form rados url 01f6e20d3 Remove idmap entry only if it is present in uid_tree 218d7f8ec MDCACHE: fix invalid assert in mdcache_avl_lookup_ck() 0fc6e3a2a Restore op_ctx->ctx_export and fsal_export in nfs4_op_readdir adce80154 Fix incorrect parent file handle update on rename. 7a22a86fc Fix NFSv3 EOF handling. 738550a12 FSAL_GLUSTER/FSAL_GPFS: Do not error out if setattr ATTR_ACL with empty acl 6fce461ad FSAL_GLUSTER: fix memory leak of acl_t 1b5ba9978 [GPFS] Handle failures with NFS readdir operation ae73a8f32 FSAL_GLUSTER: Copy user creds and lease_id after reopening fd 3ea561a75 Acquire state's fdlock in close2 9383863eb Remove unexported export on DBUS unexport. 40c7b7d54 rpc_callback : check return value for nfs_rpc_create_chan_v40() properly c86969ef0 FSAL_PROXY : module options e23eaf9c9 SAL: Fix dead lock in revoke_owner_delegs() Daniel

8 16

Metadata sync for create/update operations.
by Pradeep 31 Jan '19

31 Jan '19

Hello, I noticed that ganesha does not do metadata sync after operations like create, remove, setattr etc., (with FSAL_VFS). According to NFSv3 RFC: "The following data modifying procedures are synchronous: WRITE (with stable flag set to FILE_SYNC), CREATE, MKDIR, SYMLINK, MKNOD, REMOVE, RMDIR, RENAME, LINK, and COMMIT. Kernel NFS calls commit_metadata() to sync - see create procedure below: https://elixir.bootlin.com/linux/latest/source/fs/nfsd/vfs.c#L1460 Is this to improve performance or a bug? Thanks, Pradeep

2 2

Change in ...nfs-ganesha[next]: Do not refresh attrs if the file is delegated
by Soumya (GerritHub) 31 Jan '19

31 Jan '19

Soumya has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/442905 Change subject: Do not refresh attrs if the file is delegated ...................................................................... Do not refresh attrs if the file is delegated If a file is read delegated, then no other client can modify the file data and hence we can trust the cached attrs. And incase any client holds write delegation, we either update cached attributes via CB_GETATTR or recall the delegation before responding to the other clients. Hence it seems safe to avoid refreshing regular attributes from the backend FSAL if there is delegation granted to a file. However this should be the case only if the md-cache is enabled (i.e, expire_time_attr is not set to '0'). Change-Id: Iffbe76b6c089b6435826a7f72240f9a9804e5ddd Signed-off-by: Soumya Koduri <skoduri(a)redhat.com> --- M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c 1 file changed, 12 insertions(+), 3 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/05/442905/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/442905 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: Iffbe76b6c089b6435826a7f72240f9a9804e5ddd Gerrit-Change-Number: 442905 Gerrit-PatchSet: 1 Gerrit-Owner: Soumya <skoduri(a)redhat.com> Gerrit-MessageType: newchange

1 0

Change in ...nfs-ganesha[next]: test: do not merge, -lrados_recov whiskey tango foxtrot
by Name of user not set (GerritHub) 30 Jan '19

30 Jan '19

kaleb(a)redhat.com has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/442746 Change subject: test: do not merge, -lrados_recov whiskey tango foxtrot ...................................................................... test: do not merge, -lrados_recov whiskey tango foxtrot Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com> Change-Id: I7ab8123e5484372e236a08977ad4a47577029a31 --- M src/CMakeLists.txt 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/46/442746/1 -- To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/442746 To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings Gerrit-Project: ffilz/nfs-ganesha Gerrit-Branch: next Gerrit-Change-Id: I7ab8123e5484372e236a08977ad4a47577029a31 Gerrit-Change-Number: 442746 Gerrit-PatchSet: 1 Gerrit-Owner: kaleb(a)redhat.com Gerrit-MessageType: newchange

1 0

backport the patches to fix up SIGHUP for v2.7?
by Jeff Layton 29 Jan '19

29 Jan '19

I was at devconf this week and a couple of folks working with ganesha in production asked specifically about fixing up SIGHUP to remove defunct exports in addition to just adding them. I mentioned that it'll be fixed in v2.8, but they asked about backporting that to v2.7. Should we? In particular, I think we'd need to pull in: f6896176ef87053c5de3351be80416e949ad4bdb eedcd2cf1b4b1b34c24d762a2fb6bc1b3df33aab c328125b59581d95e5593cae2877bc9f5cfba846 46bb5e561ceb2901daa7ebdd87a944c4eb9d01c6 The only potential problem I can see with doing it is that f6896176ef87053c5de3351be80416e949ad4bdb is potentially a behavioral change (it disallows dbus from removing an export that would leave another disconnected from the pseudoroot). It seems unlikely that anyone would be relying on being able to do that, but it is possible. Thoughts? -- Jeff Layton <jlayton(a)redhat.com>

3 2

Memory growth in mdcache with entries_used >> entries_hiwat
by Kropelin, Adam 29 Jan '19

29 Jan '19

(Apologies, used the old list address last time) After applying https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/441566 our long-running test is showing continually increasing memory usage. Eventually ganesha.nfsd consumes all memory in the box and we OOM. Looking at a core, it appears that the mdcache lru contains far more entries than the high water mark would normally allow. We have... CacheInode { Dir_Chunk = 500000; Entries_HWMark = 500000; } ...and after we run for a while, we observe from a core (obtained at runtime using gcore)... (gdb) print lru_state $8 = {entries_hiwat = 500000, entries_used = 2437134, chunks_hiwat = 100000, chunks_used = 2002, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 160, prev_time = 1548692973, fd_state = 0} So we have 2.4M entries with a high water mark of 500K. The difference appears to account for the unexpected memory usage. This seems to be new behavior after applying the above patch, although it's hard to be certain because earlier we hit the core with entries being freed twice before running into the high memory usage. Some more info from the core: (gdb) print LRU $2 = {{L1 = {q = {next = 0xe4788210, prev = 0x23a701f0}, id = LRU_ENTRY_L1, size = 147689}, L2 = {q = { next = 0x23a8b7b0, prev = 0xe478ef80}, id = LRU_ENTRY_L2, size = 150}, cleanup = {q = {next = 0xcca1ff70, prev = 0x139e2bf0}, id = LRU_ENTRY_CLEANUP, size = 748}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1dd98360, prev = 0x1100cc30}, id = LRU_ENTRY_L1, size = 143935}, L2 = {q = {next = 0xfa204e0, prev = 0x1e26a0a0}, id = LRU_ENTRY_L2, size = 145}, cleanup = { q = {next = 0xb26f8720, prev = 0x18f1810}, id = LRU_ENTRY_CLEANUP, size = 811}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x4903dc0, prev = 0x96cb4320}, id = LRU_ENTRY_L1, size = 141362}, L2 = {q = {next = 0x2eea11f0, prev = 0x48fd050}, id = LRU_ENTRY_L2, size = 147}, cleanup = { q = {next = 0xdc74e5b0, prev = 0xca6f990}, id = LRU_ENTRY_CLEANUP, size = 817}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x79dcf850, prev = 0x7d9445d0}, id = LRU_ENTRY_L1, size = 122171}, L2 = {q = {next = 0x7e823b30, prev = 0x79dcfc90}, id = LRU_ENTRY_L2, size = 146}, cleanup = { q = {next = 0x3245ddb0, prev = 0x1121e510}, id = LRU_ENTRY_CLEANUP, size = 761}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1a94af70, prev = 0xec32de0}, id = LRU_ENTRY_L1, size = 142951}, L2 = {q = {next = 0xec25300, prev = 0x1a944200}, id = LRU_ENTRY_L2, size = 150}, cleanup = { q = {next = 0xcf8eada0, prev = 0xba9afd0}, id = LRU_ENTRY_CLEANUP, size = 759}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x49b54c90, prev = 0x1896b90}, id = LRU_ENTRY_L1, size = 133691}, L2 = {q = {next = 0x413dbca0, prev = 0x66d15ce0}, id = LRU_ENTRY_L2, size = 142}, cleanup = { q = {next = 0xe249dcb0, prev = 0x135e9ae0}, id = LRU_ENTRY_CLEANUP, size = 804}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x25eba620, prev = 0x96783780}, id = LRU_ENTRY_L1, ---Type <return> to continue, or q <return> to quit--- size = 146051}, L2 = {q = {next = 0x1d179420, prev = 0x25eb38b0}, id = LRU_ENTRY_L2, size = 150}, cleanup = { q = {next = 0xab572d30, prev = 0x11a0d550}, id = LRU_ENTRY_CLEANUP, size = 765}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x15bc6d50, prev = 0x3c292070}, id = LRU_ENTRY_L1, size = 135767}, L2 = {q = {next = 0x5d1fad10, prev = 0x15bbffe0}, id = LRU_ENTRY_L2, size = 148}, cleanup = { q = {next = 0x5b5c1490, prev = 0x88016f0}, id = LRU_ENTRY_CLEANUP, size = 720}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x67772f0, prev = 0x964eb1b0}, id = LRU_ENTRY_L1, size = 140729}, L2 = {q = {next = 0x30d68c80, prev = 0x6770580}, id = LRU_ENTRY_L2, size = 147}, cleanup = { q = {next = 0xc4710dd0, prev = 0x874dc10}, id = LRU_ENTRY_CLEANUP, size = 791}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x5d1c210, prev = 0x302682d0}, id = LRU_ENTRY_L1, size = 140670}, L2 = {q = {next = 0x3026f040, prev = 0x5d154a0}, id = LRU_ENTRY_L2, size = 147}, cleanup = { q = {next = 0xcb5487f0, prev = 0xc679490}, id = LRU_ENTRY_CLEANUP, size = 794}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe2791600, prev = 0x96252be0}, id = LRU_ENTRY_L1, size = 148189}, L2 = {q = {next = 0xec06a870, prev = 0xe2798370}, id = LRU_ENTRY_L2, size = 145}, cleanup = { q = {next = 0xa4dec380, prev = 0xb29dc50}, id = LRU_ENTRY_CLEANUP, size = 746}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe5dff590, prev = 0x23d43b90}, id = LRU_ENTRY_L1, size = 147297}, L2 = {q = {next = 0x23de7e10, prev = 0xe5e06300}, id = LRU_ENTRY_L2, size = 141}, cleanup = { q = {next = 0xa6b8d390, prev = 0x10229cb0}, id = LRU_ENTRY_CLEANUP, size = 744}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1cc790a0, prev = 0x4ba049f0}, id = LRU_ENTRY_L1, size = 143750}, L2 = {q = {next = 0xf39d9d0, prev = 0x1cc72330}, id = LRU_ENTRY_L2, size = 147}, cleanup = { ---Type <return> to continue, or q <return> to quit--- q = {next = 0xd37ff620, prev = 0x13b47010}, id = LRU_ENTRY_CLEANUP, size = 780}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xeaf4fd00, prev = 0x25ab91d0}, id = LRU_ENTRY_L1, size = 146505}, L2 = {q = {next = 0x25aa4980, prev = 0xeaf56a70}, id = LRU_ENTRY_L2, size = 150}, cleanup = { q = {next = 0x23c6c110, prev = 0x11e02670}, id = LRU_ENTRY_CLEANUP, size = 788}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe3dea010, prev = 0x2831da50}, id = LRU_ENTRY_L1, size = 148717}, L2 = {q = {next = 0x283247c0, prev = 0xe3df0d80}, id = LRU_ENTRY_L2, size = 149}, cleanup = { q = {next = 0xb84809e0, prev = 0x87cee00}, id = LRU_ENTRY_CLEANUP, size = 850}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xce3ced10, prev = 0xd5f2db20}, id = LRU_ENTRY_L1, size = 153934}, L2 = {q = {next = 0xd5f12560, prev = 0xce3d5a80}, id = LRU_ENTRY_L2, size = 149}, cleanup = { q = {next = 0xca840ac0, prev = 0xda6b170}, id = LRU_ENTRY_CLEANUP, size = 779}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x2d292f90, prev = 0x17851300}, id = LRU_ENTRY_L1, size = 138008}, L2 = {q = {next = 0x1786c8c0, prev = 0x2d2854b0}, id = LRU_ENTRY_L2, size = 147}, cleanup = { q = {next = 0xc7c1e980, prev = 0xb2a3820}, id = LRU_ENTRY_CLEANUP, size = 761}, mtx = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0}, __pad0 = '\000' <repeats 63 times>}} (gdb) (gdb) print *(struct mdcache_lru__ *)0xe4788210 $3 = {q = {next = 0xe47814a0, prev = 0x7e0620 <LRU>}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xe47814a0 $5 = {q = {next = 0xe477a730, prev = 0xe4788210}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xe477a730 $6 = {q = {next = 0xe47739c0, prev = 0xe47814a0}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xe47739c0 $7 = {q = {next = 0xe476cc50, prev = 0xe477a730}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xb84809e0 $8 = {q = {next = 0xaa559720, prev = 0x7e12a0 <LRU+3200>}, qid = LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xaa559720 $9 = {q = {next = 0xd9754930, prev = 0xb84809e0}, qid = LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xd9754930 $10 = {q = {next = 0xa94e7f40, prev = 0xaa559720}, qid = LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xa94e7f40 $11 = {q = {next = 0x185aa7d0, prev = 0xd9754930}, qid = LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0} (gdb) print *(struct mdcache_lru__ *)0x185aa7d0 $12 = {q = {next = 0xcb912a30, prev = 0xa94e7f40}, qid = LRU_ENTRY_CLEANUP, refcnt = 1, flags = 3, lane = 14, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xcb912a30 $13 = {q = {next = 0xc7727370, prev = 0x185aa7d0}, qid = LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0} (gdb) $14 = {q = {next = 0xc7727370, prev = 0x185aa7d0}, qid = LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xf39d9d0 $15 = {q = {next = 0xf3a4740, prev = 0x7e10c0 <LRU+2720>}, qid = LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xf3a4740 $16 = {q = {next = 0xf3ab4b0, prev = 0xf39d9d0}, qid = LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12, cf = 0} (gdb) print *(struct mdcache_lru__ *)0xf3ab4b0 $17 = {q = {next = 0xf3b2220, prev = 0xf3a4740}, qid = LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12, cf = 0}

2 1