Monitoring in Ganesha?
by Bjorn Leffler
Apart from the counters that you can access through dbus, is there any
other monitoring built into Ganesha?
I'm thinking of adding it with this higher level plan:
- Exporting metrics from Ganesha to Prometheus.
- Aggregate data in Prometheus.
- Display monitoring consoles and graphs with Grafana.
- Package up Prometheus, Grafana and the preconfigured rules/dashboards as
a Docker image.
- This makes it straightforward to deploy monitoring alongside a Gansha
binary.
My rough coding plan for the code is to:
- Add a USE_MONITORING directive to the CMakeLists.txt files.
- Add a build dependency to the Prometheus C client
<https://github.com/digitalocean/prometheus-client-c>.
- Create a src/monitoring directory for the new source files and templates.
- Increment counters and timers throughout the code.
- Use histograms to compute latency percentiles, heatmaps, etc.
Is this a good idea? Any objections or suggestions?
Thanks,
Bjorn
1 year, 6 months
connection fd in CLOSE_WAIT state
by gaurav gangalwar
Hi,
I am using nfs4.1 mount with nconnect=16 from linux client (kernel
5.8.6-2.el7.elrepo.x86_64)
I am facing mount hung if client is idle for sometime after running
read/write IO.
From netstat I could see, one of client connection in CLOSE_WAIT on
server and FIN_WAIT2 on client.
On debugging further I found out that if there is back channel client
created on same client xprt then we won’t be able to close socket fd
on connection reset from client, since there will be ref from rpc
clnt, xprt will be released when reaper thread does client id expiry
and do clnt destroy.
I can see in logs client is keep on updating the lease and reaper is
always getting valid lease for client and not expiring it.
I am not sure if its due to multiple connections using nconnect due to
some bug in client code, I have not seen this issue without nconnect.
In nfs_rpc_create_chan_v41 we will create rpc_client on same xprt fd
to create back channel on same connection, this will take ref on xprt
for the rpc client.
We cannot release this ref until rpc client is destroyed so fd will be
open even if client has closed the connection.
Client is updating the lease and reaper is always getting valid lease.
Update lease from client:
25/03/2021 08:55:40 : epoch 605c1d45 : rbt-el7-2 :
ganesha.nfsd-109993[svc_1630] nfs4_op_sequence :CLIENT ID :F_DBG
:Don’t use sesson slot 1=0x7fa0c3c9ee38 for DRC
25/03/2021 08:55:40 : epoch 605c1d45 : rbt-el7-2 :
ganesha.nfsd-109993[svc_1630] update_lease :CLIENT ID :F_DBG :Update
Lease 0x7fa0bf813d00 ClientID={Epoch=0x605c1d45 Counter=0x0000002a}
CONFIRMED Client={0x7fa0bf834630 name=(23:Linux NFSv4.1 rbt-el7-1)
refcount=1} t_delta=0 reservations=0 refcount=2
Reaper check:
25/03/2021 08:56:22 : epoch 605c1d45 : rbt-el7-2 :
ganesha.nfsd-109993[reaper] reaper_run :CLIENT ID :DEBUG :Now checking
NFS4 clients for expiration
25/03/2021 08:56:22 : epoch 605c1d45 : rbt-el7-2 :
ganesha.nfsd-109993[reaper] valid_lease :CLIENT ID :F_DBG :Check Lease
0x7fa0bf813d00 ClientID={Epoch=0x605c1d45 Counter=0x0000002a}
CONFIRMED Client={0x7fa0bf834630 name=(23:Linux NFSv4.1 rbt-el7-1)
refcount=1} t_delta=1 reservations=0 refcount=2 (Valid=YES 59 seconds
left)
Can we destroy the rpc client while destroying svc xprt? these back
channel clients won't required since connection is closed by client.
Regards,
Gaurav
2 years, 2 months
Announce Push of V4-dev.55
by Frank Filz
Branch next
Tag:V4-dev.55
Merge Highlights
* Patches to bring xattr up to speed with RFC 8276
* CEPH: implement and enable xattr
* build: fixup perms on (/var)/run/ganesha
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
21f0535 Frank S. Filz V4-dev.55
0455e7e Kaleb S. KEITHLEY build: fixup perms on (/var)/run/ganesha
e941efb Jeff Layton FSAL_CEPH: enable XATTR support
c312f65 Jeff Layton FSAL_CEPH: plumb in xattr handling
91aad68 Jeff Layton FSAL: add a new listxattr helper routine
1765780 Jeff Layton NFS: fix up handling of ATTR4_XATTR_SUPPORTED
5326059 Jeff Layton NFS: convert LISTXATTRS to RFC 8276
deffaac Jeff Layton NFS: convert REMOVEXATTR to RFC 8276
d59be22 Jeff Layton NFS: convert SETXATTR to RFC 8276
e2b33b8 Jeff Layton NFS: update GETXATTR to RFC 8276
e61965a Jeff Layton NFS: add support for xattr ACCESS bits
cb87135 Jeff Layton NFS: rename xattrname4 to xattrkey4
316eec6 Jeff Layton NFS: add new error codes for xattr support
7b8653d Jeff Layton FSAL: disable ATTR4_XATTR in FSAL_GLUSTER and FSAL_GPFS
2 years, 2 months
TIRPC SVCAUTH_WRAP error in removexattr code
by Jeff Layton
I recently submitted some patches to clean up the xattr handling support
for v4.2+, and discovered that there are xattr tests in pynfs!
I tested against them and XATT8 failed, with the server timing out on
the reply. Here's an excerpt from NIV_FULL_DEBUG run (sans some too-
chatty debug messages that I disabled in the dbus code):
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] process_one_op :NFS4 :DEBUG :Request 2: opcode 75 is OP_REMOVEXATTR
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] check_resp_room :NFS4 :F_DBG :Status of OP_REMOVEXATTR in position 2 is ok so far, op response size = 32 total response size would be = 128 out of max 8192/8192
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] nfs4_op_removexattr :NFS4 :DEBUG :RemoveXattr len 10 name: user.attr1
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] nfs4_Is_Fh_Invalid :FH :F_DBG :NFS4 Handle (29:0x43000064187e06000000010000feffffffffffffff0200000000000000)
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Got read lock on 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:
937
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Unlocked 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:945
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Got write lock on 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c
:946
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_refresh_attrs :INODE :F_DBG :attrs obj attributes Request Mask=0111dfce Valid Mask=0011ffce Supported Mask=0311dfee REGULAR_FILE numlinks=0x1 size=0x0 mode=0644
owner=0x115f group=0x115f atime=15/03/2021 15:17:32 mtime=15/03/2021 15:17:32
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Unlocked 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:976
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=00100000 Valid Mask=0011ffce Supported Mask=0311dfee REGULAR_FILE numlinks=0x1 size=0x0 mode=0644 owner
=0x115f group=0x115f atime=15/03/2021 15:17:32 mtime=15/03/2021 15:17:32
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Got read lock on 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:
937
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Unlocked 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:945
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Got write lock on 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c
:946
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_refresh_attrs :INODE :F_DBG :attrs obj attributes Request Mask=0111dfce Valid Mask=0011ffce Supported Mask=0311dfee REGULAR_FILE numlinks=0x1 size=0x0 mode=0644
owner=0x115f group=0x115f atime=15/03/2021 15:17:32 mtime=15/03/2021 15:17:32
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :RW LOCK :F_DBG :Unlocked 0x7f9120014730 (&entry->attr_lock) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_handle.c:976
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] mdcache_getattrs :INODE :F_DBG :attrs obj attributes Request Mask=00100000 Valid Mask=0011ffce Supported Mask=0311dfee REGULAR_FILE numlinks=0x1 size=0x0 mode=0644 owner
=0x115f group=0x115f atime=15/03/2021 15:17:32 mtime=15/03/2021 15:17:32
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_op :NFS4 :F_DBG :Current FH Len=29 0x70cd004c917f0000000000000000000000000000000000000000000000
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_op :NFS4 :F_DBG :Saved FH Len=0 (EMPTY)
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_op :NFS4 :DEBUG :Status of OP_REMOVEXATTR in position 2 = NFS4_OK, op response size is 32 total response size is 120
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_nfs4_compound :RW LOCK :F_DBG :Acquired mutex 0x7f914c0166e0 (&data->preserved_clientid->cid_mutex) at /home/jlayton/git/ganesha/src/Protocols/NFS/nfs4_Compound.
c:1044
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] update_lease :CLIENT ID :F_DBG :Update Lease 0x7f914c016660 ClientID={Epoch=0x604fb2c8 Counter=0x00000002} CONFIRMED Client={0x7f914c00f2e0 name=(16:XATT8_1615835852) ref
count=1} t_delta=0 reservations=0 refcount=3
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_nfs4_compound :RW LOCK :F_DBG :Released mutex 0x7f914c0166e0 (&data->preserved_clientid->cid_mutex) at /home/jlayton/git/ganesha/src/Protocols/NFS/nfs4_Compound.
c:1048
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] _mdcache_lru_unref :RW LOCK :F_DBG :Acquired mutex 0x7f919ced6fe0 (&(qlane)->mtx) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:2036
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] _mdcache_lru_unref :RW LOCK :F_DBG :Released mutex 0x7f919ced6fe0 (&(qlane)->mtx) at /home/jlayton/git/ganesha/src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_lru.c:2043
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] compound_data_Free :RW LOCK :F_DBG :Released mutex 0x7f9120011fe8 (&slot->lock) at /home/jlayton/git/ganesha/src/Protocols/NFS/nfs4_Compound.c:1453
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] clear_op_context_export_impl :EXPORT :F_DBG :put export ref for id 100 /, refcount = 1
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_request :DISP :F_DBG :Before svc_sendreply on socket 35
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :xdr_ioq_uv_create() uv 0x7f914c00cd70 size 8192
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :xdr_ioq_reset() xioq 0x7f914c0174d0 head 0x7f914c017650 wh_pos 0
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :xdr_reply_encode:93 ACCEPTED INLINE
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :xdr_reply_encode:109 SUCCESS
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :svc_vc_reply: 0x7f9148000e20 fd 35 SVCAUTH_WRAP failed (will set dead)
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] complete_request :DISP :DEBUG :NFS DISPATCHER: FAILURE: Error while calling svc_sendreply on a new request. rpcxid=185455221 socket=35 function:NFS4_COMP client:::ffff:192.168.1.3 program:100003 nfs version:4 proc:1 errno: 0
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :svc_destroy_it() 0x7f9148000e20 fd 35 fd_send -1 xp_refcnt 4 af 10 port 42712 @complete_request:801
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :svc_rqst_xprt_unregister:1146 locking
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :svc_rqst_xprt_unregister:1151 unlocking @svc_rqst_xprt_unregister:1146
15/03/2021 15:17:32 : epoch 604fb2c8 : tleilax.poochiereds.net : ganesha.nfsd-309231[svc_3] rpc :TIRPC :F_DBG :svc_rqst_unhook_events: xprt 0x7f9148000e20 fd 35 ev_flags ADDED_RECV INITIALIZED CLOSE DESTROYING
It looks like it all works, until we hit the SVCAUTH_WRAP
failure...which is odd because this run is using AUTH_SYS, so you'd
think that would be a no-op?
In any case, I could use some help deciphering this log. What happened?
TIA!
--
Jeff Layton <jlayton(a)poochiereds.net>
2 years, 2 months
running out of file descriptors
by Alok Sinha
Version : nfs-ganesha-2.8.3
File descriptor limit : 65536
FSAL used : VFS
Server is running out of file descriptor. /proc/<pid>/fd shows that.
I see few .so ( shared objects ) and executables are open more than
once . Few of them are open about 26 times. I have not debugged LRU yet.
I think all open FDs are by MDCACHE.
Question: Is it expected that MDCACHE would open the same file many times
( multiple FDs )?
-alok
2 years, 2 months
Announce Push of V4-dev.54
by Frank Filz
Branch next
Tag:V4-dev.54
Merge Highlights
* PRIu64 fixups
* GSSAPI: eliminate printerr macro and extraneous newlines
* selinux: (re)start fails to read /var/lib/nfs when it's a symlink
Signed-off-by: Frank S. Filz <ffilzlnx(a)mindspring.com>
Contents:
3a184de Frank S. Filz V4-dev.54
3c08ee2 Kaleb S. KEITHLEY selinux: (re)start fails to read /var/lib/nfs when
it's a symlink
b5e8b1c Jeff Layton GSSAPI: eliminate printerr macro and extraneous newlines
e5d5d00 Frank S. Filz PRIu64 fixups
2 years, 2 months
Change in ...nfs-ganesha[next]: build: fixup perms on (/var)/run/ganesha
by Kaleb KEITHLEY (GerritHub)
Kaleb KEITHLEY has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/512322 )
Change subject: build: fixup perms on (/var)/run/ganesha
......................................................................
build: fixup perms on (/var)/run/ganesha
fix found by downstream testing of package metadata.
Also use %{_rundir} instead of hard-coded path
Change-Id: Icdc4d6d0cd0d253788fb73172cdfa3a03bd08bbd
Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com>
---
M src/nfs-ganesha.spec-in.cmake
1 file changed, 3 insertions(+), 3 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/22/512322/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/512322
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: Icdc4d6d0cd0d253788fb73172cdfa3a03bd08bbd
Gerrit-Change-Number: 512322
Gerrit-PatchSet: 1
Gerrit-Owner: Kaleb KEITHLEY <kaleb(a)redhat.com>
Gerrit-MessageType: newchange
2 years, 2 months
Change in ...nfs-ganesha[next]: selinux: (re)start fails to read /var/lib/nfs when it's a symlink
by Kaleb KEITHLEY (GerritHub)
Kaleb KEITHLEY has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/512320 )
Change subject: selinux: (re)start fails to read /var/lib/nfs when it's a symlink
......................................................................
selinux: (re)start fails to read /var/lib/nfs when it's a symlink
In Red Hat Gluster Storage, the HA configuration changes /var/lib/nfs to
be a symlink to a shared state directory. New selinux policy is prohibiting
the reading of that symlink and ganesha is prevented from starting when it
tries to read it.
Also fixes another AVC (cluster_t:dbus send_msg) seen at the same time.
Change-Id: I567fb165b0e8077f4071b0dc75fa331d8dc42ee0
Signed-off-by: Kaleb S. KEITHLEY <kkeithle(a)redhat.com>
---
M src/selinux/ganesha.te
1 file changed, 7 insertions(+), 0 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/20/512320/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/512320
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I567fb165b0e8077f4071b0dc75fa331d8dc42ee0
Gerrit-Change-Number: 512320
Gerrit-PatchSet: 1
Gerrit-Owner: Kaleb KEITHLEY <kaleb(a)redhat.com>
Gerrit-MessageType: newchange
2 years, 2 months
