Re: segfault in nfs_rpc_destroy_chan while releasing xprt
by Deepthi Shivaramu
Soumya,
I see there was discussion in github about the exact same segfault and you were debugging this issue :
https://github.com/nfs-ganesha/ntirpc/issues/153
There were multiple fixes discussed in there but ultimately I see this fix was checked in :
https://github.com/nfs-ganesha/ntirpc/pull/155/commits/ca74cde10ef02a322b...
But the strange part is I have that fix already in my source and still hitting this same segfault.
Also one correction from my previous mail, actually we are using libntirpc1.7.0 with ganesha2.7.2.
@Soumya, do you know any other fix which was related to this problem?
Regards,
Deepthi
On 14/05/20, 5:09 PM, "Deepthi Shivaramu" <des(a)vmware.com> wrote:
I see this segfault is in nfs_rpc_destroy_chan() and not specific to setclientid_confirm.
We are not seeing it with NFSv4.1 but seeing it frequently with NFSv4.0 tests.
I saw one more core today with bt:
(gdb) bt
#0 0x00007ff7b1dde71a in svc_release_it (xprt=0x7ff780001740, flags=0, tag=0x7ff7b1e05fd0 "clnt_vc_destroy", line=462)
at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/svc.h:433
#1 0x00007ff7b1ddf4fb in clnt_vc_destroy (clnt=0x7ff780001620) at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/src/clnt_vc.c:462
#2 0x000000000043b4e1 in clnt_release_it (clnt=0x7ff780001620, flags=0, tag=0x55e550 <__func__.21824> "_nfs_rpc_destroy_chan", line=628)
at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/clnt.h:319
#3 0x000000000043b577 in clnt_destroy_it (clnt=0x7ff780001620, tag=0x55e550 <__func__.21824> "_nfs_rpc_destroy_chan", line=628)
at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/clnt.h:341
#4 0x000000000043eb97 in _nfs_rpc_destroy_chan (chan=0x7ff7940023a8) at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_rpc_callback.c:628
#5 0x000000000043f800 in nfs_rpc_destroy_chan (chan=0x7ff7940023a8) at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_rpc_callback.c:864
#6 0x00000000004bde35 in nfs_client_id_expire (clientid=0x7ff794002300, make_stale=false)
at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/SAL/nfs4_clientid.c:1099
#7 0x00000000004442bf in reap_hash_table (ht_reap=0xf35f40) at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_reaper_thread.c:109
#8 0x0000000000444a62 in reaper_run (ctx=0xf66ca0) at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_reaper_thread.c:232
#9 0x00000000004fdc38 in fridgethr_start_routine (arg=0xf66ca0) at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/support/fridgethr.c:550
#10 0x00007ff7b09aa3d4 in start_thread (arg=0x7ff791ffb700) at pthread_create.c:334
#11 0x00007ff7b02c9ebd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) f 0
#0 0x00007ff7b1dde71a in svc_release_it (xprt=0x7ff780001740, flags=0, tag=0x7ff7b1e05fd0 "clnt_vc_destroy", line=462)
at /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/svc.h:433
433 in /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/svc.h
(gdb) p xprt
$1 = (SVCXPRT *) 0x7ff780001740
(gdb) p *$
$2 = {xp_ops = 0x0, xp_dispatch = {process_cb = 0x0, rendezvous_cb = 0x0}, xp_parent = 0x7ff770004730, xp_tp = 0x6d00000001 <error: Cannot access memory at address 0x6d00000001>,
xp_netid = 0x7ff79c00a160 "", xp_p1 = 0x7ff770004750, xp_p2 = 0x0, xp_p3 = 0x0, xp_u1 = 0x3, xp_u2 = 0x0, xp_local = {nb = {maxlen = 0, len = 0, buf = 0x7ff7940018a0}, ss = {
ss_family = 0, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}}, xp_remote = {nb = {maxlen = 4280583506, len = 0, buf = 0x0}, ss = {ss_family = 34467,
__ss_align = 1,
__ss_padding = "_:P\346ju\200\223\001\000\000\000\001\000\000\000`,\000\200\367\177\000\000\341\376\266^", '\000' <repeats 12 times>, "\061\000\000\000\000\000\000\000\000\061\000\200\367\177\000\000\220]\000\200\367\177\000\000c3-edbe-2fea12000\000\000\000\000\000\000\000\064\001", '\000' <repeats 21 times>}}, xp_lock = {__data = {__lock = -1946148624,
__count = 32759, __owner = 0, __nusers = 37, __kind = -1946148624, __spins = 32759, __list = {__prev = 0x7ff77c001530, __next = 0x0}},
__size = "\360 \000\214\367\177\000\000\000\000\000\000%\000\000\000\360 \000\214\367\177\000\000\060\025\000|\367\177\000\000\000\000\000\000\000\000\000",
__align = 140701182468336}, xp_fd = 0, xp_ifindex = 0, xp_si_type = 3, xp_type = 0, xp_refcnt = -1, xp_flags = 64}
(gdb) p xprt->xp_ops
$3 = (struct xp_ops *) 0x0
(gdb)
Regards,
Deepthi
On 14/05/20, 12:17 PM, "Deepthi Shivaramu" <des(a)vmware.com> wrote:
Daniel,
I am seeing this segfault in the libntirpc1.8.0 with ganesha2.8.2 in setclientid_confirm code path.
Can you please check and let me know if you have seen this issue before and if the fix is already available in latest versions?
(gdb) bt
#0 0x0000000000000000 in ?? ()
#1 0x00007fd66badf72e in svc_release_it (xprt=0x7fd658002e90, flags=0,
tag=0x7fd66bb06fd0 "clnt_vc_destroy", line=462)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/svc.h:433
#2 0x00007fd66bae04fb in clnt_vc_destroy (clnt=0x7fd658002ba0)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/src/clnt_vc.c:462
#3 0x000000000043b4e1 in clnt_release_it (clnt=0x7fd658002ba0, flags=0,
tag=0x55e550 <__func__.21824> "_nfs_rpc_destroy_chan", line=628)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/clnt.h:319
#4 0x000000000043b577 in clnt_destroy_it (clnt=0x7fd658002ba0,
tag=0x55e550 <__func__.21824> "_nfs_rpc_destroy_chan", line=628)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/clnt.h:341
#5 0x000000000043eb97 in _nfs_rpc_destroy_chan (chan=0x7fd64c002648)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_rpc_callback.c:628
#6 0x000000000043f800 in nfs_rpc_destroy_chan (chan=0x7fd64c002648)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_rpc_callback.c:864
#7 0x000000000048011c in nfs4_op_setclientid_confirm (op=0x7fd62c001d90,
---Type <return> to continue, or q <return> to quit---
data=0x7fd6607dff70, resp=0x7fd62c002070)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/Protocols/NFS/nfs4_op_setclientid_confirm.c:382
#8 0x000000000045b4b1 in nfs4_Compound (arg=0x7fd62c0011a8,
req=0x7fd62c000aa0, res=0x7fd62c001f60)
at
....
.......
#20 0x00007fd669fcaebd in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) f 1
#1 0x00007fd66badf72e in svc_release_it (xprt=0x7fd658002e90, flags=0,
tag=0x7fd66bb06fd0 "clnt_vc_destroy", line=462)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/svc.h:433
433 /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/libntirpc/ntirpc/rpc/svc.h:
No such file or directory.
(gdb) p clnt
No symbol "clnt" in current context.
(gdb) p xprt
$10 = (SVCXPRT *) 0x7fd658002e90
(gdb) p *$
$11 = {xp_ops = 0x7fd658000e20, xp_dispatch = {process_cb = 0x7fd658000078,
rendezvous_cb = 0x7fd658000078}, xp_parent = 0x0, xp_tp = 0x0,
xp_netid = 0x0, xp_p1 = 0x0, xp_p2 = 0x0, xp_p3 = 0x0, xp_u1 = 0x0,
xp_u2 = 0x0, xp_local = {nb = {maxlen = 483619223, len = 1, buf = 0x2},
ss = {ss_family = 0, __ss_align = 0,
__ss_padding =
"\313)\260k\326\177\000\000\020\320\236b\326\177\000\000\006\000\000\000\034\000\000\000\004\004\005\377\377\377\377\377\000\000\000\000\020\373\364\310\333c\335\363\245\332\362b\324.M\332",
'\000' <repeats 59 times>}}, xp_remote = {nb = {maxlen = 0, len = 0, buf =
0x0}, ss = {ss_family = 0,
__ss_align = 0, __ss_padding = '\000' <repeats 111 times>}}, xp_lock =
{
__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0,
__spins = 0, __list = {__prev = 0x0, __next = 0x0}},
__size = '\000' <repeats 39 times>, __align = 0}, xp_fd = 0,
xp_ifindex = 0, xp_si_type = 0, xp_type = 0, xp_refcnt = -1, xp_flags = 64}
(gdb) f 6
#6 0x000000000043f800 in nfs_rpc_destroy_chan (chan=0x7fd64c002648)
at
/build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_rpc_callback.c:864
864 /build/mts/release/bora-16138726/cayman_nfs-ganesha/nfs-ganesha/src/src/MainNFSD/nfs_rpc_callback.c:
No such file or directory.
(gdb) p chan
$12 = (rpc_call_channel_t *) 0x7fd64c002648
(gdb) p *$
$13 = {type = RPC_CHAN_V40, mtx = {__data = {__lock = 1, __count = 0,
__owner = 163, __nusers = 1, __kind = 0, __spins = 0, __list = {
__prev = 0x0, __next = 0x0}},
__size = "\001\000\000\000\000\000\000\000\243\000\000\000\001",
'\000' <repeats 26 times>, __align = 1}, states = 0, source = {clientid =
0x7fd64c0025a0,
session = 0x7fd64c0025a0}, last_called = 0, clnt = 0x7fd658002ba0,
auth = 0x0, gss_sec = {mech = 0x0, qop = 0, svc = RPCSEC_GSS_SVC_INTEGRITY,
cred = 0x0, req_flags = 0}}
(gdb) p chan->client
There is no member named client.
(gdb) p chan->clnt
$14 = (CLIENT *) 0x7fd658002ba0
(gdb) p *$
$15 = {cl_ops = 0x7fd66bd192e0, cl_netid = 0x0, cl_tp = 0x0, cl_u1 = 0x0,
cl_u2 = 0x0, cl_lock = {__data = {__lock = 0, __count = 0, __owner = 0,
__nusers = 0, __kind = 3, __spins = 0, __list = {__prev = 0x0,
__next = 0x0}},
__size = '\000' <repeats 16 times>, "\003", '\000'
<repeats 22 times>,
__align = 0}, cl_error = {ru = {RE_errno = 0, RE_why = AUTH_OK, RE_vers = {
low = 0, high = 0}, RE_lb = {s1 = 0, s2 = 0}},
re_status = RPC_SUCCESS}, cl_refcnt = 0, cl_flags = 96}
(gdb)
On 06/05/20, 10:00 PM, "Daniel Gryniewicz" <dang(a)redhat.com> wrote:
I'm happy to announce the latest stable versions of NTIRPC and Ganesha
in the 2.8 series. These are NTIRPC 1.8.1 and Ganesha 2.8.4. There are
>40 bug fixes in these releases.
Daniel
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
6 years
Announce Push of V4-dev.20
by Frank Filz
Branch next
Tag:V4-dev.20
NOTE: This release includes an ntirpc pullup, please update your submodules.
Merge Highlights
* Add NFSACL protocol support for NFS v3 ACLs
* Add ganesha_stats iomon.
* Add ACL_DEBUG and RFC_ACL symbols to export list
* NTIRPC Pullup
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
77ca7df Frank S. Filz V4-dev.20
9e9835a Daniel Gryniewicz NTIRPC Pullup
7882be5 Daniel Gryniewicz Add ACL_DEBUG and RFC_ACL symbols to export list
50795cb Xi Jinyu Add ganesha_stats iomon.
11d0261 Luo Muyao Add support for NFSv3 ACLs.
a159db8 Luo Muyao Add NFSACL data structure and XDR functions
6 years
Coredump when exiting
by Sriram Patil
Hi,
I am seeing following core dump which I think happens when shutting down ganesha because it is happening in lib fini functions,
#0 0x00007f387cc547a8 in remove_error_table () from /lib/libcom_err.so.3
#1 0x00007f387ca2d400 in gss_krb5int_lib_fini () from /lib/libgssapi_krb5.so.2
#2 0x00007f387ca14aaa in gssint_mechglue_fini () from /lib/libgssapi_krb5.so.2
#3 0x00007f387de64997 in _dl_fini () at dl-fini.c:252
#4 0x00007f387c05ecb8 in __run_exit_handlers (status=0, listp=0x7f387c3c55f8 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true) at exit.c:82
#5 0x00007f387c05ed05 in __GI_exit (status=<optimized out>) at exit.c:104
#6 0x00007f387c049577 in __libc_start_main (main=0x41dc40 <main>, argc=9, argv=0x7ffe529c5dc8, init=<optimized out>, fini=<optimized out>,
rtld_fini=<optimized out>, stack_end=0x7ffe529c5db8) at libc-start.c:323
#7 0x000000000041e709 in _start () at ../sysdeps/x86_64/elf/start.S:113
Is this a known issue?
Thanks,
Sriram
6 years
Change in ...nfs-ganesha[next]: NTIRPC Pullup
by Daniel Gryniewicz (GerritHub)
Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/494931 )
Change subject: NTIRPC Pullup
......................................................................
NTIRPC Pullup
Change-Id: I7dfdea91f1866957015653a1109ba583cf6ca2c5
Signed-off-by: Daniel Gryniewicz <dang(a)redhat.com>
---
M src/libntirpc
1 file changed, 1 insertion(+), 1 deletion(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/31/494931/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/494931
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I7dfdea91f1866957015653a1109ba583cf6ca2c5
Gerrit-Change-Number: 494931
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com>
Gerrit-MessageType: newchange
6 years
Change in ...nfs-ganesha[next]: Add ganesha_stats iomon.
by xijinyu (GerritHub)
xijinyu has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/494913 )
Change subject: Add ganesha_stats iomon.
......................................................................
Add ganesha_stats iomon.
Monitoring the real-time bandwidth of each export instance.
Signed-off-by: Xi Jinyu <xijinyu(a)cmss.chinamobile.com>
Change-Id: I9e6da1f5e872979a6198d0e68006815732c21608
---
M src/include/server_stats_private.h
M src/scripts/ganeshactl/Ganesha/glib_dbus_stats.py
M src/scripts/ganeshactl/ganesha_stats.py
M src/support/client_mgr.c
M src/support/export_mgr.c
M src/support/server_stats.c
6 files changed, 350 insertions(+), 4 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/13/494913/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/494913
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I9e6da1f5e872979a6198d0e68006815732c21608
Gerrit-Change-Number: 494913
Gerrit-PatchSet: 1
Gerrit-Owner: xijinyu <xijinyu(a)cmss.chinamobile.com>
Gerrit-MessageType: newchange
6 years
Change in ...nfs-ganesha[next]: Add ganesha_stats iomon.
by xijinyu (GerritHub)
xijinyu has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/494891 )
Change subject: Add ganesha_stats iomon.
......................................................................
Add ganesha_stats iomon.
Monitoring the real-time bandwidth of each export instance.
Signed-off-by: Xi Jinyu <xijinyu(a)cmss.chinamobile.com>
Change-Id: I4cabb73eaaee4830ca7178767ad8091f54344cb9
---
M src/include/server_stats_private.h
M src/scripts/ganeshactl/Ganesha/glib_dbus_stats.py
M src/scripts/ganeshactl/ganesha_stats.py
M src/support/client_mgr.c
M src/support/export_mgr.c
M src/support/server_stats.c
6 files changed, 283 insertions(+), 4 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/91/494891/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/494891
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I4cabb73eaaee4830ca7178767ad8091f54344cb9
Gerrit-Change-Number: 494891
Gerrit-PatchSet: 1
Gerrit-Owner: xijinyu <xijinyu(a)cmss.chinamobile.com>
Gerrit-MessageType: newchange
6 years
pynfs tests on 4.1
by Trishali Nayar
Hi all,
While running pynfs test suite with 4.1, we observed that few tests
failed:-
1> SEQ6 st_sequence.testRequestTooBig :
FAILURE
OP_LOOKUP should return NFS4ERR_REQ_TOO_BIG, instead
got NFS4ERR_NAMETOOLONG
2> CSESS21 st_create_session.testCallbackVersion :
FAILURE
No CB_NULL sent
3> CSESS20 st_create_session.testCallbackProgram :
FAILURE
No CB_NULL sent
4> EID50 st_exchange_id.testSSV
: FAILURE
AttributeError: 'ClientStateProtection' object has no
attribute 'context'
Are these failures expected i.e. we know about them and its not actually a
problem OR are these issues that need to be fixed?
Pls. let us know your inputs on same.
Thanks and regards,
Trishali.
6 years
IO hang in non blocking send path.
by gaurav gangalwar
IO hang on next EWOULDBLOCK after adding send fd to epoll.
Problem is when we are rearming send fd after send is blocked, we are
setting data->ptr which is overwriting data->fd since its union.
I am not sure why we are setting data->ptr, don't see any use of it,
all we need is data fd.
Fixed it with this patch
https://github.com/nfs-ganesha/ntirpc/pull/214
-Gaurav
6 years
