lock issue for same clientid on the same node using NFS Ganesha 3.5
We are using NFS Ganesha 3.5 and occassionaly we see the following messages on the client host. The applications are no longer able to take locks. They are all taking RDLCKs over the entire file range. fcntl(19, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=1800}) The host has multiple mounts of the SAME exported filesystem. All the mounts are identical except for the mount path. Each mount is sending the the SAME SETCLIENTID to the NFS Ganesha. Any pointers much appreciated. 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_10] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_10] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_14] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_14] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd-22[svc_17] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1310 refcount=7} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}
I'm not sure what the client is sending, but there is a discrepancy in the open owners from the first log message: owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) vs. owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) Maybe a tcpdump would help, I'm not sure. Frank
-----Original Message----- From: pragash2here@gmail.com [mailto:pragash2here@gmail.com] Sent: Wednesday, May 8, 2024 7:08 PM To: support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] lock issue for same clientid on the same node using NFS Ganesha 3.5
We are using NFS Ganesha 3.5 and occassionaly we see the following messages on the client host. The applications are no longer able to take locks.
They are all taking RDLCKs over the entire file range. fcntl(19, F_SETLK, {l_type=F_RDLCK, l_whence=SEEK_SET, l_start=0, l_len=1800})
The host has multiple mounts of the SAME exported filesystem. All the mounts are identical except for the mount path.
Each mount is sending the the SAME SETCLIENTID to the NFS Ganesha.
Any pointers much appreciated.
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd- 22[svc_10] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd- 22[svc_10] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1294 refcount=6} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd- 22[svc_14] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3}
27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd- 22[svc_14] nfs4_op_lock :NFS4 LOCK :EVENT :LOCK failed to create new lock owner Lock: obj=0x7f40f001fbc8, fileid=50331779, type=READ , start=0x0, end=0xffffffffffffffff, owner={STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1302 refcount=6} 27/03/2024 20:28:16 : epoch 6601203a : tiger-nfs-shared-444 : ganesha.nfsd- 22[svc_17] create_nfs4_owner :NFS4 LOCK :CRIT :Related {STATE_OPEN_OWNER_NFSV4 0x7f40f4041ad0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000b0ddcf9b36316) confirmed=1 seqid=1310 refcount=7} doesn't match for {STATE_LOCK_OWNER_NFSV4 0x7f40b40087f0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(20:0x6c6f636b2069643a000002e70000000000000 002) confirmed=1 seqid=2 related_owner={STATE_OPEN_OWNER_NFSV4 0x7f40c0031ed0: clientid={0x7f40f4004680 ClientID={Epoch=0x6601203a Counter=0x00000002} CONFIRMED Client={0x7f40f4001f40 name=(77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d) refcount=1} t_delta=0 reservations=1 refcount=7 cb_prog=1073741824 r_addr=240b:c0e0:105:5500:b463:2:3:0.135.233 r_netid=tcp6} owner=(24:0x6f70656e2069643a000002e700000000000a61fe2b043999) confirmed=1 seqid=1288 refcount=3} refcount=3} _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Thanks Frank. It is very hard to reproduce. But will keep trying. Do you recommend a log level to turn on ? This is a production environment. We do notice that on the Ganesha side, there are multiple SETCLIENTIDs in a short period of time with the same opaque string over the same TCP connection. Wondering if this is fine. The opaque string is 77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d
The clientid seemed to be the same in the messages, so I don't think that's directly the issue. Unless that refreshing the clientid causes the client to change its lock owner strings. Frank
-----Original Message----- From: pragash2here@gmail.com [mailto:pragash2here@gmail.com] Sent: Thursday, May 9, 2024 12:29 PM To: support@lists.nfs-ganesha.org Subject: [NFS-Ganesha-Support] Re: lock issue for same clientid on the same node using NFS Ganesha 3.5
Thanks Frank. It is very hard to reproduce. But will keep trying. Do you recommend a log level to turn on ? This is a production environment.
We do notice that on the Ganesha side, there are multiple SETCLIENTIDs in a short period of time with the same opaque string over the same TCP connection. Wondering if this is fine.
The opaque string is
77:Linux NFSv4.0 worker007.acmelab.local/240b:c0e0:205:5500:b464:2:0:b57d _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Final question Frank. From the Ganesha logs: The original open owner had a SeqID of 1288. The Request comes in with SeqID 1294 which fails. The Request is sent again with SeqID 1310. The log is filled with messages with increasing SeqIDs that fail. Is this normal. Is the SeqID something that increases for every message ?
participants (2)
-
Frank Filz -
pragash2here＠gmail.com