Hi All I recently put an nfs-ganesha CEPH_FSAL deployment into production, so far so good but I'm seeing some errors in the logs I didn't see when testing and was hoping someone could shed some light on what they mean. I haven't had any adverse behaviour reported from the clients (apart from a potential issue with slow 'ls' operations which I'm investigating). Versions: libcephfs2-13.2.2-0.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64 Ceph cluster is 12.2.10 Log errors: "posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY" I'm seeing this one frequently although seems to spam the log with 20 or so occurrences in a second. "15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver :
ganesha.nfsd-1990[svc_1653] posix2fsal_error :FSAL :INFO :Mapping 5 to ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576 15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : ganesha.nfsd-1990[svc_1653] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error in nfs4_mds_putfh converted to NFS4ERR_IO but was set non-retryable"
I've only seen a few occurrences of this one 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver :
ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :nfs4_gid_to_name failed with code -2. 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup for 1664 failed, using numeric group
This one doesn't seem too serious, my guess is there are accounts on the clients with gids/uids that the server can't look up. The server is using SSSD to bind to AD if that helps. Export: {
Export_ID=100; Protocols = 4; Transports = TCP; Path = /; Pseudo = /ceph/; Access_Type = RW; Squash = No_root_squash; Attr_Expiration_Time = 0; Disable_ACL = FALSE; Manage_Gids = TRUE; Filesystem_Id = 100.1; FSAL { Name = CEPH; } }
Thanks, David
Hi, David. Answers inline. On Fri, May 17, 2019 at 11:42 AM David C <dcsysengineer@gmail.com> wrote:
Hi All
I recently put an nfs-ganesha CEPH_FSAL deployment into production, so far so good but I'm seeing some errors in the logs I didn't see when testing and was hoping someone could shed some light on what they mean. I haven't had any adverse behaviour reported from the clients (apart from a potential issue with slow 'ls' operations which I'm investigating).
Versions:
libcephfs2-13.2.2-0.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64 Ceph cluster is 12.2.10
Log errors:
"posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY"
This is an INFO, so it's not an error. You might not want to run a production deployment at INFO, because it will be a bit chatty. The default is to run at EVENT.
I'm seeing this one frequently although seems to spam the log with 20 or so occurrences in a second.
"15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : ganesha.nfsd-1990[svc_1653] posix2fsal_error :FSAL :INFO :Mapping 5 to ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576 15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : ganesha.nfsd-1990[svc_1653] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error in nfs4_mds_putfh converted to NFS4ERR_IO but was set non-retryable"
The IO error log is there because NFS4ERR_IO is a catchall error, that's used for lots and lots of situations. When a client sees NFS4ERR_IO, it can be really hard to know what caused the error. This one has traditionally been a CRIT message. I think that's high, I would put it at WARN, but that's the way it is. Something *did* go wrong, but not in Ganesha, there was some error in the underlying FS, and it was returned to the client.
I've only seen a few occurrences of this one
17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :nfs4_gid_to_name failed with code -2. 17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup for 1664 failed, using numeric group
These, again, are INFO, and so don't indicate a true failure anywhere.
This one doesn't seem too serious, my guess is there are accounts on the clients with gids/uids that the server can't look up. The server is using SSSD to bind to AD if that helps.
Correct. Daniel
Many thanks for the quick response Daniel! On Fri, May 17, 2019 at 4:55 PM Daniel Gryniewicz <dang@redhat.com> wrote:
Hi, David.
Answers inline.
On Fri, May 17, 2019 at 11:42 AM David C <dcsysengineer@gmail.com> wrote:
Hi All
I recently put an nfs-ganesha CEPH_FSAL deployment into production, so
far so good but I'm seeing some errors in the logs I didn't see when testing and was hoping someone could shed some light on what they mean. I haven't had any adverse behaviour reported from the clients (apart from a potential issue with slow 'ls' operations which I'm investigating).
Versions:
libcephfs2-13.2.2-0.el7.x86_64 nfs-ganesha-2.7.1-0.1.el7.x86_64 nfs-ganesha-ceph-2.7.1-0.1.el7.x86_64 Ceph cluster is 12.2.10
Log errors:
"posix2fsal_error :FSAL :INFO :Mapping 11 to ERR_FSAL_DELAY"
This is an INFO, so it's not an error. You might not want to run a production deployment at INFO, because it will be a bit chatty. The default is to run at EVENT.
I'm seeing this one frequently although seems to spam the log with 20 or
so occurrences in a second.
"15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver :
ganesha.nfsd-1990[svc_1653] posix2fsal_error :FSAL :INFO :Mapping 5 to ERR_FSAL_IO, rlim_cur=1048576 rlim_max=1048576
15/05/2019 18:27:01 : epoch 5cd99ef1 : nfsserver : ganesha.nfsd-1990[svc_1653] nfs4_Errno_verbose :NFS4 :CRIT :Error I/O error in nfs4_mds_putfh converted to NFS4ERR_IO but was set non-retryable"
The IO error log is there because NFS4ERR_IO is a catchall error, that's used for lots and lots of situations. When a client sees NFS4ERR_IO, it can be really hard to know what caused the error. This one has traditionally been a CRIT message. I think that's high, I would put it at WARN, but that's the way it is. Something *did* go wrong, but not in Ganesha, there was some error in the underlying FS, and it was returned to the client.
I've only seen a few occurrences of this one
17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver :
ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :nfs4_gid_to_name failed with code -2.
17/05/2019 15:34:24 : epoch 5cdd9df8 : nfsserver : ganesha.nfsd-4696[svc_258] xdr_encode_nfs4_princ :ID MAPPER :INFO :Lookup for 1664 failed, using numeric group
These, again, are INFO, and so don't indicate a true failure anywhere.
This one doesn't seem too serious, my guess is there are accounts on the
clients with gids/uids that the server can't look up. The server is using SSSD to bind to AD if that helps.
Correct.
Daniel
participants (2)
-
Daniel Gryniewicz -
David C