Re: [Nfs-ganesha-devel] Memory growth in mdcache with entries_used >> entries_hiwat
Moving to the new list... I'm not seeing how that patch can have caused this, as an entry should have already been removed from the LRU before hitting the changes in the patch. I'll run some tests, though, and see what I can work out. Daniel On 1/28/19 1:49 PM, Kropelin, Adam via Nfs-ganesha-devel wrote:
This list has been deprecated. Please subscribe to the new devel list at lists.nfs-ganesha.org.
After applying https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/441566 our long-running test is showing continually increasing memory usage. Eventually ganesha.nfsd consumes all memory in the box and we OOM. Looking at a core, it appears that the mdcache lru contains far more entries than the high water mark would normally allow.
We have…
CacheInode {
Dir_Chunk = 500000;
Entries_HWMark = 500000;
}
…and after we run for a while, we observe from a core (obtained at runtime using gcore)...
(gdb) print lru_state
$8 = {entries_hiwat = 500000, entries_used = 2437134, chunks_hiwat = 100000, chunks_used = 2002, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 160, prev_time = 1548692973, fd_state = 0}
So we have 2.4M entries with a high water mark of 500K. The difference appears to account for the unexpected memory usage.
This seems to be new behavior after applying the above patch, although it’s hard to be certain because earlier we hit the core with entries being freed twice before running into the high memory usage.
Some more info from the core:
(gdb) print LRU
$2 = {{L1 = {q = {next = 0xe4788210, prev = 0x23a701f0}, id = LRU_ENTRY_L1, size = 147689}, L2 = {q = {
next = 0x23a8b7b0, prev = 0xe478ef80}, id = LRU_ENTRY_L2, size = 150}, cleanup = {q = {next = 0xcca1ff70,
prev = 0x139e2bf0}, id = LRU_ENTRY_CLEANUP, size = 748}, mtx = {__data = {__lock = 0, __count = 0,
__owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1dd98360, prev = 0x1100cc30}, id = LRU_ENTRY_L1,
size = 143935}, L2 = {q = {next = 0xfa204e0, prev = 0x1e26a0a0}, id = LRU_ENTRY_L2, size = 145}, cleanup = {
q = {next = 0xb26f8720, prev = 0x18f1810}, id = LRU_ENTRY_CLEANUP, size = 811}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x4903dc0, prev = 0x96cb4320}, id = LRU_ENTRY_L1,
size = 141362}, L2 = {q = {next = 0x2eea11f0, prev = 0x48fd050}, id = LRU_ENTRY_L2, size = 147}, cleanup = {
q = {next = 0xdc74e5b0, prev = 0xca6f990}, id = LRU_ENTRY_CLEANUP, size = 817}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x79dcf850, prev = 0x7d9445d0}, id = LRU_ENTRY_L1,
size = 122171}, L2 = {q = {next = 0x7e823b30, prev = 0x79dcfc90}, id = LRU_ENTRY_L2, size = 146}, cleanup = {
q = {next = 0x3245ddb0, prev = 0x1121e510}, id = LRU_ENTRY_CLEANUP, size = 761}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1a94af70, prev = 0xec32de0}, id = LRU_ENTRY_L1,
size = 142951}, L2 = {q = {next = 0xec25300, prev = 0x1a944200}, id = LRU_ENTRY_L2, size = 150}, cleanup = {
q = {next = 0xcf8eada0, prev = 0xba9afd0}, id = LRU_ENTRY_CLEANUP, size = 759}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x49b54c90, prev = 0x1896b90}, id = LRU_ENTRY_L1,
size = 133691}, L2 = {q = {next = 0x413dbca0, prev = 0x66d15ce0}, id = LRU_ENTRY_L2, size = 142}, cleanup = {
q = {next = 0xe249dcb0, prev = 0x135e9ae0}, id = LRU_ENTRY_CLEANUP, size = 804}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x25eba620, prev = 0x96783780}, id = LRU_ENTRY_L1,
---Type <return> to continue, or q <return> to quit---
size = 146051}, L2 = {q = {next = 0x1d179420, prev = 0x25eb38b0}, id = LRU_ENTRY_L2, size = 150}, cleanup = {
q = {next = 0xab572d30, prev = 0x11a0d550}, id = LRU_ENTRY_CLEANUP, size = 765}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x15bc6d50, prev = 0x3c292070}, id = LRU_ENTRY_L1,
size = 135767}, L2 = {q = {next = 0x5d1fad10, prev = 0x15bbffe0}, id = LRU_ENTRY_L2, size = 148}, cleanup = {
q = {next = 0x5b5c1490, prev = 0x88016f0}, id = LRU_ENTRY_CLEANUP, size = 720}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x67772f0, prev = 0x964eb1b0}, id = LRU_ENTRY_L1,
size = 140729}, L2 = {q = {next = 0x30d68c80, prev = 0x6770580}, id = LRU_ENTRY_L2, size = 147}, cleanup = {
q = {next = 0xc4710dd0, prev = 0x874dc10}, id = LRU_ENTRY_CLEANUP, size = 791}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x5d1c210, prev = 0x302682d0}, id = LRU_ENTRY_L1,
size = 140670}, L2 = {q = {next = 0x3026f040, prev = 0x5d154a0}, id = LRU_ENTRY_L2, size = 147}, cleanup = {
q = {next = 0xcb5487f0, prev = 0xc679490}, id = LRU_ENTRY_CLEANUP, size = 794}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe2791600, prev = 0x96252be0}, id = LRU_ENTRY_L1,
size = 148189}, L2 = {q = {next = 0xec06a870, prev = 0xe2798370}, id = LRU_ENTRY_L2, size = 145}, cleanup = {
q = {next = 0xa4dec380, prev = 0xb29dc50}, id = LRU_ENTRY_CLEANUP, size = 746}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe5dff590, prev = 0x23d43b90}, id = LRU_ENTRY_L1,
size = 147297}, L2 = {q = {next = 0x23de7e10, prev = 0xe5e06300}, id = LRU_ENTRY_L2, size = 141}, cleanup = {
q = {next = 0xa6b8d390, prev = 0x10229cb0}, id = LRU_ENTRY_CLEANUP, size = 744}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x1cc790a0, prev = 0x4ba049f0}, id = LRU_ENTRY_L1,
size = 143750}, L2 = {q = {next = 0xf39d9d0, prev = 0x1cc72330}, id = LRU_ENTRY_L2, size = 147}, cleanup = {
---Type <return> to continue, or q <return> to quit---
q = {next = 0xd37ff620, prev = 0x13b47010}, id = LRU_ENTRY_CLEANUP, size = 780}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xeaf4fd00, prev = 0x25ab91d0}, id = LRU_ENTRY_L1,
size = 146505}, L2 = {q = {next = 0x25aa4980, prev = 0xeaf56a70}, id = LRU_ENTRY_L2, size = 150}, cleanup = {
q = {next = 0x23c6c110, prev = 0x11e02670}, id = LRU_ENTRY_CLEANUP, size = 788}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xe3dea010, prev = 0x2831da50}, id = LRU_ENTRY_L1,
size = 148717}, L2 = {q = {next = 0x283247c0, prev = 0xe3df0d80}, id = LRU_ENTRY_L2, size = 149}, cleanup = {
q = {next = 0xb84809e0, prev = 0x87cee00}, id = LRU_ENTRY_CLEANUP, size = 850}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0xce3ced10, prev = 0xd5f2db20}, id = LRU_ENTRY_L1,
size = 153934}, L2 = {q = {next = 0xd5f12560, prev = 0xce3d5a80}, id = LRU_ENTRY_L2, size = 149}, cleanup = {
q = {next = 0xca840ac0, prev = 0xda6b170}, id = LRU_ENTRY_CLEANUP, size = 779}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}, {L1 = {q = {next = 0x2d292f90, prev = 0x17851300}, id = LRU_ENTRY_L1,
size = 138008}, L2 = {q = {next = 0x1786c8c0, prev = 0x2d2854b0}, id = LRU_ENTRY_L2, size = 147}, cleanup = {
q = {next = 0xc7c1e980, prev = 0xb2a3820}, id = LRU_ENTRY_CLEANUP, size = 761}, mtx = {__data = {__lock = 0,
__count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, iter = {active = false, glist = 0x0, glistn = 0x0},
__pad0 = '\000' <repeats 63 times>}}
(gdb)
(gdb) print *(struct mdcache_lru__ *)0xe4788210
$3 = {q = {next = 0xe47814a0, prev = 0x7e0620 <LRU>}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xe47814a0
$5 = {q = {next = 0xe477a730, prev = 0xe4788210}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xe477a730
$6 = {q = {next = 0xe47739c0, prev = 0xe47814a0}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xe47739c0
$7 = {q = {next = 0xe476cc50, prev = 0xe477a730}, qid = LRU_ENTRY_L1, refcnt = 1, flags = 0, lane = 0, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xb84809e0
$8 = {q = {next = 0xaa559720, prev = 0x7e12a0 <LRU+3200>}, qid = LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3,
lane = 14, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xaa559720
$9 = {q = {next = 0xd9754930, prev = 0xb84809e0}, qid = LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xd9754930
$10 = {q = {next = 0xa94e7f40, prev = 0xaa559720}, qid = LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xa94e7f40
$11 = {q = {next = 0x185aa7d0, prev = 0xd9754930}, qid = LRU_ENTRY_CLEANUP, refcnt = 3, flags = 3, lane = 14, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0x185aa7d0
$12 = {q = {next = 0xcb912a30, prev = 0xa94e7f40}, qid = LRU_ENTRY_CLEANUP, refcnt = 1, flags = 3, lane = 14, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xcb912a30
$13 = {q = {next = 0xc7727370, prev = 0x185aa7d0}, qid = LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0}
(gdb)
$14 = {q = {next = 0xc7727370, prev = 0x185aa7d0}, qid = LRU_ENTRY_CLEANUP, refcnt = 2, flags = 3, lane = 14, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xf39d9d0
$15 = {q = {next = 0xf3a4740, prev = 0x7e10c0 <LRU+2720>}, qid = LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12,
cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xf3a4740
$16 = {q = {next = 0xf3ab4b0, prev = 0xf39d9d0}, qid = LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12, cf = 0}
(gdb) print *(struct mdcache_lru__ *)0xf3ab4b0
$17 = {q = {next = 0xf3b2220, prev = 0xf3a4740}, qid = LRU_ENTRY_L2, refcnt = 1, flags = 0, lane = 12, cf = 0}
_______________________________________________ Nfs-ganesha-devel mailing list Nfs-ganesha-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
The issue is very reproducible. Just takes a few hours. If there is any debug info you would like turned on, I'd be glad to enable it and re-run the test. The test workload that causes this: Concurrent access from multiple threads. 1 thread continuously (in a loop) running python os.walk (i.e., readdir) of the entire filesystem, roughly ~5M files total. 5 more threads are writing a few thousand files each. When the writes complete, a single thread verifies written content, then deletes it. Then the writes repeat again. I did see about 97,000 of the following messages in a few minutes of enabling debug: LogDebugAlt(COMPONENT_NFS_READDIR, COMPONENT_CACHE_INODE, "Already existent when inserting new dirent on entry=%p name=%s", entry, v->name); LogFullDebugAlt(COMPONENT_NFS_READDIR, COMPONENT_CACHE_INODE, "Duplicate insert of %s v->chunk=%p v2->chunk=%p", v->name, v->chunk, v2->chunk);
So, this is interesting, since it means that entries are being reaped. There are dirents for entries that don't exist, so re-adding the entry is colliding with the existing dirent. If you stop the load and wait a while, does the number of entries go down? On Tue, Jan 29, 2019 at 4:13 PM <vrungta@amazon.com> wrote:
The issue is very reproducible. Just takes a few hours. If there is any debug info you would like turned on, I'd be glad to enable it and re-run the test.
The test workload that causes this: Concurrent access from multiple threads. 1 thread continuously (in a loop) running python os.walk (i.e., readdir) of the entire filesystem, roughly ~5M files total. 5 more threads are writing a few thousand files each. When the writes complete, a single thread verifies written content, then deletes it. Then the writes repeat again.
I did see about 97,000 of the following messages in a few minutes of enabling debug: LogDebugAlt(COMPONENT_NFS_READDIR, COMPONENT_CACHE_INODE, "Already existent when inserting new dirent on entry=%p name=%s", entry, v->name); LogFullDebugAlt(COMPONENT_NFS_READDIR, COMPONENT_CACHE_INODE, "Duplicate insert of %s v->chunk=%p v2->chunk=%p", v->name, v->chunk, v2->chunk); _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
I stopped the tests and waited 90 minutes. The number of entries did not go down at all. Few minutes before stopping the tests: (gdb) print lru_state $1 = {entries_hiwat = 500000, entries_used = 954943, chunks_hiwat = 100000, chunks_used = 1742, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 0, prev_time = 1548862024, fd_state = 0} 30 minutes after stopping the tests: (gdb) print lru_state $1 = {entries_hiwat = 500000, entries_used = 954944, chunks_hiwat = 100000, chunks_used = 1742, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 0, prev_time = 1548863824, fd_state = 0} 90 minutes after stopping the tests: (gdb) print lru_state $1 = {entries_hiwat = 500000, entries_used = 954944, chunks_hiwat = 100000, chunks_used = 1742, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 0, prev_time = 1548867350, fd_state = 0}
Okay, we're leaking a ref somewhere. I'll see if I can track it down. On Wed, Jan 30, 2019 at 12:03 PM <vrungta@amazon.com> wrote:
I stopped the tests and waited 90 minutes. The number of entries did not go down at all.
Few minutes before stopping the tests:
(gdb) print lru_state $1 = {entries_hiwat = 500000, entries_used = 954943, chunks_hiwat = 100000, chunks_used = 1742, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 0, prev_time = 1548862024, fd_state = 0}
30 minutes after stopping the tests: (gdb) print lru_state $1 = {entries_hiwat = 500000, entries_used = 954944, chunks_hiwat = 100000, chunks_used = 1742, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 0, prev_time = 1548863824, fd_state = 0}
90 minutes after stopping the tests: (gdb) print lru_state $1 = {entries_hiwat = 500000, entries_used = 954944, chunks_hiwat = 100000, chunks_used = 1742, fds_system_imposed = 400000, fds_hard_limit = 396000, fds_hiwat = 360000, fds_lowat = 200000, futility = 0, per_lane_work = 50, biggest_window = 160000, prev_fd_count = 0, prev_time = 1548867350, fd_state = 0} _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Hello, I noticed that ganesha does not do metadata sync after operations like create, remove, setattr etc., (with FSAL_VFS). According to NFSv3 RFC: "The following data modifying procedures are synchronous: WRITE (with stable flag set to FILE_SYNC), CREATE, MKDIR, SYMLINK, MKNOD, REMOVE, RMDIR, RENAME, LINK, and COMMIT. Kernel NFS calls commit_metadata() to sync - see create procedure below: https://elixir.bootlin.com/linux/latest/source/fs/nfsd/vfs.c#L1460 Is this to improve performance or a bug? Thanks, Pradeep
In case it helps -- As an experiment I set Dir_Chunk = 0 and ran the same test load. I was not able to reproduce the issue. LRU entries stayed at the HW mark.
So, I've been unable so far to reproduce this with the workloads I'm using. I'll have to try to replicate your workload more closely. I don't suppose you can share the test program you're running? On Thu, Jan 31, 2019 at 11:18 AM <vrungta@amazon.com> wrote:
In case it helps -- As an experiment I set Dir_Chunk = 0 and ran the same test load. I was not able to reproduce the issue. LRU entries stayed at the HW mark. _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Our test programs are very tightly coupled to our test environment. We'll try to see if we can eliminate the dependencies and get something useful Meanwhile here's complete details on our workload. Pre condition: Create a subdirectory with 2000 subdirectories and 2500 1K files in each subdir for a total of 5 million files. One thread doing a python (os.walk) readdir of the entire share (pre-populated subdir as well as the 2nd subdir where files are being created, deleted, recreated). This thread continues for the duration of test. One test run is 6 hours. N files/dir, M dirs, S file size Files to create are submitted to the thread pool in the following order: Dir1/file1, Dir2/file1, DirM/file1, Dir1/file2, Dir2/file2.. 5 threads will pull work from the thread pool and create files. Once the writes are complete, one thread will read the files, verify MD5 and then delete them. The process repeats. N,M,S stay constant for the test run. The names of the files in the 2nd iteration are the same as the first iteration. After 6 hours the test run is repeated with a different set of N,M,S and the readdir thread is restarted. N is one of 10, 50, 100, 500, 1000 M is from 1 to 10 S is one if 1K, 5K, 10K, 50K, 100K, 500K, 1M, 5M, 10M, 50M One other thought: The new code in cih_get_by_key_latch will now return not found even though it is still in the hash with a refcount of 0. mdcache_new_entry will go through oentry=NULL and will add the new entry to the hash (avltree?). Can this result in collisions between the oentry that still exists and the new entry being added?
Thanks. I'll see what I can whip up quickly. Daniel On 1/31/19 7:47 PM, vrungta@amazon.com wrote:
Our test programs are very tightly coupled to our test environment. We'll try to see if we can eliminate the dependencies and get something useful
Meanwhile here's complete details on our workload. Pre condition: Create a subdirectory with 2000 subdirectories and 2500 1K files in each subdir for a total of 5 million files.
One thread doing a python (os.walk) readdir of the entire share (pre-populated subdir as well as the 2nd subdir where files are being created, deleted, recreated). This thread continues for the duration of test.
One test run is 6 hours. N files/dir, M dirs, S file size Files to create are submitted to the thread pool in the following order: Dir1/file1, Dir2/file1, DirM/file1, Dir1/file2, Dir2/file2.. 5 threads will pull work from the thread pool and create files. Once the writes are complete, one thread will read the files, verify MD5 and then delete them. The process repeats. N,M,S stay constant for the test run. The names of the files in the 2nd iteration are the same as the first iteration.
After 6 hours the test run is repeated with a different set of N,M,S and the readdir thread is restarted.
N is one of 10, 50, 100, 500, 1000 M is from 1 to 10 S is one if 1K, 5K, 10K, 50K, 100K, 500K, 1M, 5M, 10M, 50M
One other thought: The new code in cih_get_by_key_latch will now return not found even though it is still in the hash with a refcount of 0. mdcache_new_entry will go through oentry=NULL and will add the new entry to the hash (avltree?). Can this result in collisions between the oentry that still exists and the new entry being added? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Okay, I'm able to reproduce this. But, more importantly, I'm able to reproduce is on stock 2.7.1, so it's not caused by the recent changes. I haven't yet checked next, I'll do that, well, next. Daniel On 2/1/19 9:33 AM, Daniel Gryniewicz wrote:
Thanks. I'll see what I can whip up quickly.
Daniel
On 1/31/19 7:47 PM, vrungta@amazon.com wrote:
Our test programs are very tightly coupled to our test environment. We'll try to see if we can eliminate the dependencies and get something useful Meanwhile here's complete details on our workload. Pre condition: Create a subdirectory with 2000 subdirectories and 2500 1K files in each subdir for a total of 5 million files. One thread doing a python (os.walk) readdir of the entire share (pre-populated subdir as well as the 2nd subdir where files are being created, deleted, recreated). This thread continues for the duration of test. One test run is 6 hours. N files/dir, M dirs, S file size Files to create are submitted to the thread pool in the following order: Dir1/file1, Dir2/file1, DirM/file1, Dir1/file2, Dir2/file2.. 5 threads will pull work from the thread pool and create files. Once the writes are complete, one thread will read the files, verify MD5 and then delete them. The process repeats. N,M,S stay constant for the test run. The names of the files in the 2nd iteration are the same as the first iteration. After 6 hours the test run is repeated with a different set of N,M,S and the readdir thread is restarted. N is one of 10, 50, 100, 500, 1000 M is from 1 to 10 S is one if 1K, 5K, 10K, 50K, 100K, 500K, 1M, 5M, 10M, 50M One other thought: The new code in cih_get_by_key_latch will now return not found even though it is still in the hash with a refcount of 0. mdcache_new_entry will go through oentry=NULL and will add the new entry to the hash (avltree?). Can this result in collisions between the oentry that still exists and the new entry being added? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
This fixes it for me: https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/443260 I'm delaying 2.7.2 long enough to get this in, so it will likely come out next Monday. I'll also include 25320e6544f6c5a045f20c51446f57c9dc036412 MDCACHE - Don't return dead entries from hashtable Daniel On 2/1/19 11:48 AM, Daniel Gryniewicz wrote:
Okay, I'm able to reproduce this. But, more importantly, I'm able to reproduce is on stock 2.7.1, so it's not caused by the recent changes. I haven't yet checked next, I'll do that, well, next.
Daniel
On 2/1/19 9:33 AM, Daniel Gryniewicz wrote:
Thanks. I'll see what I can whip up quickly.
Daniel
On 1/31/19 7:47 PM, vrungta@amazon.com wrote:
Our test programs are very tightly coupled to our test environment. We'll try to see if we can eliminate the dependencies and get something useful Meanwhile here's complete details on our workload. Pre condition: Create a subdirectory with 2000 subdirectories and 2500 1K files in each subdir for a total of 5 million files. One thread doing a python (os.walk) readdir of the entire share (pre-populated subdir as well as the 2nd subdir where files are being created, deleted, recreated). This thread continues for the duration of test. One test run is 6 hours. N files/dir, M dirs, S file size Files to create are submitted to the thread pool in the following order: Dir1/file1, Dir2/file1, DirM/file1, Dir1/file2, Dir2/file2.. 5 threads will pull work from the thread pool and create files. Once the writes are complete, one thread will read the files, verify MD5 and then delete them. The process repeats. N,M,S stay constant for the test run. The names of the files in the 2nd iteration are the same as the first iteration. After 6 hours the test run is repeated with a different set of N,M,S and the readdir thread is restarted. N is one of 10, 50, 100, 500, 1000 M is from 1 to 10 S is one if 1K, 5K, 10K, 50K, 100K, 500K, 1M, 5M, 10M, 50M One other thought: The new code in cih_get_by_key_latch will now return not found even though it is still in the hash with a refcount of 0. mdcache_new_entry will go through oentry=NULL and will add the new entry to the hash (avltree?). Can this result in collisions between the oentry that still exists and the new entry being added? _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
Thanks Daniel. I will test with the fix and report back. Vandana
The memory fix looks good. Thank you. I continue to hit the crash I reported earlier in https://lists.nfs-ganesha.org/archives/list/devel@lists.nfs-ganesha.org/thre... I will post additional details in that thread.
participants (3)
-
Daniel Gryniewicz -
Pradeep -
vrungta＠amazon.com