Monitoring in Ganesha?
by Bjorn Leffler
Apart from the counters that you can access through dbus, is there any
other monitoring built into Ganesha?
I'm thinking of adding it with this higher level plan:
- Exporting metrics from Ganesha to Prometheus.
- Aggregate data in Prometheus.
- Display monitoring consoles and graphs with Grafana.
- Package up Prometheus, Grafana and the preconfigured rules/dashboards as
a Docker image.
- This makes it straightforward to deploy monitoring alongside a Gansha
binary.
My rough coding plan for the code is to:
- Add a USE_MONITORING directive to the CMakeLists.txt files.
- Add a build dependency to the Prometheus C client
<https://github.com/digitalocean/prometheus-client-c>.
- Create a src/monitoring directory for the new source files and templates.
- Increment counters and timers throughout the code.
- Use histograms to compute latency percentiles, heatmaps, etc.
Is this a good idea? Any objections or suggestions?
Thanks,
Bjorn
1 year, 4 months
Change in ...nfs-ganesha[next]: MDCACHE - Add MDCACHE {} config block
by Daniel Gryniewicz (GerritHub)
Daniel Gryniewicz has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/454929
Change subject: MDCACHE - Add MDCACHE {} config block
......................................................................
MDCACHE - Add MDCACHE {} config block
Add a config block name MDCACHE that is a copy of CACHEINODE. Both can
be configured, but MDCACHE will override CACHEINODE. This allows us to
deprecate CACHEINODE.
Change-Id: I49012723132ae6105b904a60d1a96bb2bf78d51b
Signed-off-by: Daniel Gryniewicz <dang(a)fprintf.net>
---
M src/FSAL/Stackable_FSALs/FSAL_MDCACHE/mdcache_read_conf.c
M src/config_samples/ceph.conf
M src/config_samples/config.txt
M src/config_samples/ganesha.conf.example
M src/doc/man/ganesha-cache-config.rst
M src/doc/man/ganesha-config.rst
6 files changed, 31 insertions(+), 7 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/29/454929/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/454929
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I49012723132ae6105b904a60d1a96bb2bf78d51b
Gerrit-Change-Number: 454929
Gerrit-PatchSet: 1
Gerrit-Owner: Daniel Gryniewicz <dang(a)redhat.com>
Gerrit-MessageType: newchange
2 years, 4 months
Issues seen with krb5i and krb5p mounts
by Trishali Nayar
Hi all,
There are a few clients eg- Ubuntu 18.04.3 (4.15.0-55-generic) and RH7.8
(3.10.0-1127.el7.x86_64) for which we have observed... simple command like
'dd' either hangs or returns EIO. This is happening only on krb5i and krb5p
mounts. It seems to happen for file sizes eg- 100MB and larger mostly. But
sometimes even a 30 MB file sees failures.
A client eg- RH7.6 (3.10.0-957.el7.x86_64) does not seem to hit this
issue...so might be with more recent kernels?
We fixed the issue with check-in
https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/490802 The idea was to
let clients know that Ganesha denied the request VS just dropping the
request.
This fix did seem to help and hangs/errors stopped completely... but for
larger file sizes eg- 1000MB we started seeing "Permission Denied" errors.
This was different than the EIO errors seen earlier. Reason could be we are
now sending an "AUTH DENIED" error so clients translate it to this new
error.
We tried to add more logging into Ganesha and observe that these particular
clients seem to send a lot of requests together. When we process same, the
sequence no. is pretty much out or order and we drop the requests outside
the sequence window, as per the RFC 2203 Section 7.2.1. The sequence window
that we have is 32.
Testing these clients with kNFS does not hit the issue...The kNFS sequence
window seems to be larger and is 128.
So, tried to increase the sequence window as well to 128 for ganesha. That
does not seem to help fix the issue.
We also have below additional 'seqmask' check and many of the requests went
into that category as well and were dropped.
"libntirpc/src/svc_auth_gss.c":
} else if (offset >= gd->win || (gd->seqmask & (1 << offset))) {
*no_dispatch = true;
goto gd_free;
}
Also observed that now these clients sent many requests above the 128
window...which we would again drop.
Wondering what is the proper way to fix this and any idea on what these
clients are doing different.
Thanks and regards,
Trishali.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Trishali Nayar
IBM Systems
ETZ, Pune.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2 years, 5 months
Seeking assistance with Ganesha PROXY FSAL / libntirpc issues
by Jo Seaton
Hi all,
I've been working on re-adding GSS/Kerberos authentication support to
the PROXY (V4) FSAL, with a mind to eventually also adding Kerberos
delegation support for completeness. I've been having some issues I
was hoping to get some feedback on.
The first issue I've been having is to do with the GSS code in libntirpc.
I'm calling authgss_ncreate_default with a valid CLIENT * and service
name, and hopefully a reasonable struct rpc_gss_sec. This fails,
because the call to gss_verify_mic inside authgss_refresh fails. This
appears to be because authgss_verify fills a relevant buffer
(gd->gc_wire_verf) with <empty>, which originally comes from cc_verf
in the clnt code (via the start of authgss_validate). Specifically I'm
looking at clnt_req's cc_verf which gets used for AUTH_VALIDATE in
clnt_generic.c, and always seems to have the same "_null_auth" value -
which seems surprising to me! If anyone can give me some insight into
what exactly cc_verf is supposed to contain that might help me fix it.
Working around it by ignoring the result of gss_verify_mic does seem
to work OK.
The second issue is to do with the structure of the PROXY FSAL.
It appears that it largely handles requests "manually", calling the
relevant xdr_* functions, and reading/writing to sockets itself. The
GSS auth code on the other hand, seems to require use of CLIENT *,
which in my understanding means handing responsibility for the socket
to that CLIENT *. These two approaches appear incompatible to me. I've
made some reasonable progress rewriting PROXY with clnt_req_*
functions, similar to nfs_rpc_callback.c, but if anyone has any
feedback on a) why the original approach (ffilz suggests those
functions didn't used to be threadsafe?) b) the most sensible thing to
do now, it would be very much appreciated.
Anyway any feedback is very welcome, I'm very new to both Ganesha and
GSS/Kerberos.
Many thanks,
Jo
2 years, 5 months
Readdirplus entries without attributes?
by Bjorn Leffler
How does Ganesha (especially MDCACHE) handle readdir responses when some or
all files have no attributes? Is this supported and supposed to work?
I'm testing proxy configurations against a NetApp filer. NetApp filers
don't return attributes in the readdir response for .snapshot directories.
Thanks,
Bjorn
2 years, 5 months
Change in ...nfs-ganesha[next]: grace: avoid add duplicate client entries from backend
by sepia-liu (GerritHub)
sepia-liu has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/503292 )
Change subject: grace: avoid add duplicate client entries from backend
......................................................................
grace: avoid add duplicate client entries from backend
If ganesha is already in grace period, at the point,
it enter grace by admin_dbus_grace() again, it is likely that
add duplicate client entries from same backend object.
The result is that clid_count never equal to reclaim_completes.
Change-Id: I565f807fcd5e806717544a2a8d32f0249463afa1
Signed-off-by: sepia-liu <liuwei_coder(a)163.com>
---
M src/SAL/nfs4_recovery.c
1 file changed, 20 insertions(+), 1 deletion(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/92/503292/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/503292
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I565f807fcd5e806717544a2a8d32f0249463afa1
Gerrit-Change-Number: 503292
Gerrit-PatchSet: 1
Gerrit-Owner: sepia-liu <liuwei_coder(a)163.com>
Gerrit-MessageType: newchange
2 years, 5 months
Change in ...nfs-ganesha[next]: FSAL_CEPH: fix create handle return EINVAL
by freeze (GerritHub)
freeze has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/503258 )
Change subject: FSAL_CEPH: fix create handle return EINVAL
......................................................................
FSAL_CEPH: fix create handle return EINVAL
- correct the compare with key and len on the right data type.
Change-Id: I2a1d8c0e5b1d74a5eb51cd894a7825ba59ce573c
Signed-off-by: Vicente Cheng <vicente_cheng(a)bigtera.com>
---
M src/FSAL/FSAL_CEPH/export.c
1 file changed, 5 insertions(+), 4 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/58/503258/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/503258
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I2a1d8c0e5b1d74a5eb51cd894a7825ba59ce573c
Gerrit-Change-Number: 503258
Gerrit-PatchSet: 1
Gerrit-Owner: freeze <freeze.bilsted(a)gmail.com>
Gerrit-MessageType: newchange
2 years, 5 months
Re: SELinux and GlusterFS
by jiffin tony Thottan
Hey,
The selinux feature supported in glusterfs, but for fuse clients there is
some patch missing in kernel side AFAIR. The feature works with nfs4 via
nfs-ganesha. Currently I am not actively working on those projects. Adding
Arjun in loop who worked for the support in nfs-ganesha. Also ccing dev
list of both projects.
--
Jiffin
On Fri, 25 Sep 2020 at 3:07 AM, Karl MacMillan <karl(a)bigbadwolfsecurity.com>
wrote:
> Hey,
>
> I saw your Fosdem and other presentations about GlusterFS and SELinux -
> last I saw was about missing 3.10 merge. I'm looking for a distributed
> filesystem with SELinux support and was wondering what the current status
> is. Sorry to bother you with email - I just couldn't find anything
> definitive about where things are at the moment.
>
> If GlusterFS isn't ready, are you aware of another option? I guess maybe
> NFSv4? But that's not exactly the same kind of thing.
>
> Thanks - Karl
>
>
> --
Jiffin Tony Thottan
Ph:+918129412217
2 years, 5 months
Change in ...nfs-ganesha[next]: grace: optimize takeip process to end grace early
by sepia-liu (GerritHub)
sepia-liu has uploaded this change for review. ( https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/503054 )
Change subject: grace: optimize takeip process to end grace early
......................................................................
grace: optimize takeip process to end grace early
For NFS4.1+, the agreement allows end grace in advence,
clid_count and reclaim_completes are only effective during
grace period, They should not be cumulative value, in this way,
the last reclaim result will affact the next takeip process.
If reclaim_completes not equal clid_count in one time(some clients nerver
come back), it will lead to the following grace not being able to end early.
Change-Id: I9bc5f95c3bf7a889f62978b7a4f879f11d6b43ed
Signed-off-by: sepia-liu <liuwei_coder(a)163.com>
---
M src/SAL/nfs4_recovery.c
1 file changed, 1 insertion(+), 0 deletions(-)
git pull ssh://review.gerrithub.io:29418/ffilz/nfs-ganesha refs/changes/54/503054/1
--
To view, visit https://review.gerrithub.io/c/ffilz/nfs-ganesha/+/503054
To unsubscribe, or for help writing mail filters, visit https://review.gerrithub.io/settings
Gerrit-Project: ffilz/nfs-ganesha
Gerrit-Branch: next
Gerrit-Change-Id: I9bc5f95c3bf7a889f62978b7a4f879f11d6b43ed
Gerrit-Change-Number: 503054
Gerrit-PatchSet: 1
Gerrit-Owner: sepia-liu <liuwei_coder(a)163.com>
Gerrit-MessageType: newchange
2 years, 5 months
