10:40 a.m.
Hi All
I recently put an nfs-ganesha CEPH_FSAL deployment into production, so far
so good but I'm seeing some errors in the logs I didn't see when testing
and was hoping someone could shed some light on what they mean. I haven't
had any adverse behaviour reported from the clients (apart from a potential
issue with slow 'ls' operations which I'm investigating).
Versions:
libcephfs2-13.2.2-0.el7.x86_64
nfs-ganesha-2.7.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
Ceph cluster is 12.2.10
Log errors:
"posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY"
I'm seeing this one frequently although seems to spam the log with 20 or so
occurrences in a second.
"15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver :
ganesha.nfsd-1990[svc_1653] posix2fsal_error :FSAL :INFO :Mapping 5
to
ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576
15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver :
ganesha.nfsd-1990[svc_1653] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error
in nfs4_mds_putfh converted to NFS4ERR_IO but was set non-retryable"
I've only seen a few occurrences of this one
17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver :
ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO
:nfs4_gid_to_name failed with code -2.
17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver :
ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup
for 1664 failed, using numeric group
This one doesn't seem too serious, my guess is there are accounts on the
clients with gids/uids that the server can't look up. The server is using
SSSD to bind to AD if that helps.
Export:
{
Export_ID=100;
Protocols = 4;
Transports = TCP;
Path = /;
Pseudo = /ceph/;
Access_Type = RW;
Squash = No_root_squash;
Attr_Expiration_Time = 0;
Disable_ACL = FALSE;
Manage_Gids = TRUE;
Filesystem_Id = 100.1;
FSAL {
Name = CEPH;
}
}
Thanks,
David
10:55 a.m.
Hi, David.
Answers inline.
On Fri, May 17, 2019 at 11:42 AM David C <dcsysengineer(a)gmail.com> wrote:
Hi All
I recently put an nfs-ganesha CEPH_FSAL deployment into production, so far so good but
I'm seeing some errors in the logs I didn't see when testing and was hoping
someone could shed some light on what they mean. I haven't had any adverse behaviour
reported from the clients (apart from a potential issue with slow 'ls' operations
which I'm investigating).
Versions:
libcephfs2-13.2.2-0.el7.x86_64
nfs-ganesha-2.7.1-0.1.el7.x86_64
nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
Ceph cluster is 12.2.10
Log errors:
> "posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY"
This is an INFO, so it's not an error. You might not want to run a
production deployment at INFO, because it will be a bit chatty. The
default is to run at EVENT.
I'm seeing this one frequently although seems to spam the log with 20 or so
occurrences in a second.
> "15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : ganesha.nfsd-1990[svc_1653]
posix2fsal_error :FSAL :INFO :Mapping 5 to ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576
> 15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : ganesha.nfsd-1990[svc_1653]
nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error in nfs4_mds_putfh converted to NFS4ERR_IO
but was set non-retryable"
The IO error log is there because NFS4ERR_IO is a catchall error,
that's used for lots and lots of situations. When a client sees
NFS4ERR_IO, it can be really hard to know what caused the error. This
one has traditionally been a CRIT message. I think that's high, I
would put it at WARN, but that's the way it is. Something *did* go
wrong, but not in Ganesha, there was some error in the underlying FS,
and it was returned to the client.
I've only seen a few occurrences of this one
> 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : ganesha.nfsd-4696[svc_258]
xdr_encode_nfs4_princ :ID MAPPER :INFO :nfs4_gid_to_name failed with code -2.
> 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : ganesha.nfsd-4696[svc_258]
xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup for 1664 failed, using numeric group
These, again, are INFO, and so don't indicate a true failure anywhere.
This one doesn't seem too serious, my guess is there are accounts on the clients with
gids/uids that the server can't look up. The server is using SSSD to bind to AD if
that helps.
Correct.
Daniel
7:31 p.m.
Many thanks for the quick response Daniel!
On Fri, May 17, 2019 at 4:55 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
Hi, David.
Answers inline.
On Fri, May 17, 2019 at 11:42 AM David C <dcsysengineer(a)gmail.com> wrote:
>
> Hi All
>
> I recently put an nfs-ganesha CEPH_FSAL deployment into production, so
far so good but I'm seeing some errors in the logs I didn't see when
testing and was hoping someone could shed some light on what they mean. I
haven't had any adverse behaviour reported from the clients (apart from a
potential issue with slow 'ls' operations which I'm investigating).
>
> Versions:
>
> libcephfs2-13.2.2-0.el7.x86_64
> nfs-ganesha-2.7.1-0.1.el7.x86_64
> nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64
> Ceph cluster is 12.2.10
>
> Log errors:
>
>> "posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY"
This is an INFO, so it's not an error. You might not want to run a
production deployment at INFO, because it will be a bit chatty. The
default is to run at EVENT.
>
>
> I'm seeing this one frequently although seems to spam the log with 20 or
so occurrences in a second.
>
>> "15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver :
ganesha.nfsd-1990[svc_1653] posix2fsal_error :FSAL :INFO :Mapping 5 to
ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576
>> 15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver :
ganesha.nfsd-1990[svc_1653] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error
in nfs4_mds_putfh converted to NFS4ERR_IO but was set non-retryable"
The IO error log is there because NFS4ERR_IO is a catchall error,
that's used for lots and lots of situations. When a client sees
NFS4ERR_IO, it can be really hard to know what caused the error. This
one has traditionally been a CRIT message. I think that's high, I
would put it at WARN, but that's the way it is. Something *did* go
wrong, but not in Ganesha, there was some error in the underlying FS,
and it was returned to the client.
>
>
> I've only seen a few occurrences of this one
>
>> 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver :
ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO
:nfs4_gid_to_name failed with code -2.
>> 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver :
ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup
for 1664 failed, using numeric group
These, again, are INFO, and so don't indicate a true failure anywhere.
>
>
> This one doesn't seem too serious, my guess is there are accounts on the
clients with gids/uids that the server can't look up. The server is using
SSSD to bind to AD if that helps.
Correct.
Daniel
2210
days inactive
2211
days old
2 comments
2 participants
participants (2)
-
Daniel Gryniewicz -
David C