Client owner collisions
Hi, The client owner generation logic on knfs client is very simple and tries to differentiate on the basis of NFS version and hostname. Here’s a snippet, scnprintf(str, len, "Linux NFSv%u.%u %s", clp->rpc_ops->version, clp->cl_minorversion, clp->cl_rpcclient->cl_nodename); clp->cl_owner_id = str; This causes a problem in ganesha in case there are multiple clients connecting to ganesha with same hostname using same NFS version. When comparing client owners, ganesha only takes the client owner from the NFS client into consideration. But if we consider client owner and client IP address to decide if it is the same client before expiring the state, it will treat them as different clients and avoid unnecessary expiry. Any thoughts? Thanks, Sriram
What is the scenario that has multiple clients with the same hostname? There is a mount option to set a non-uniform client string: migration / nomigration Selects whether the client uses an identification string that is compatible with NFSv4 Transparent State Migration (TSM). If the mounted server supports NFSv4 migration with TSM, specify the migration option. Some server features misbehave in the face of a migration-compatible identification string. The nomigration option retains the use of a traditional client indentification string which is compatible with legacy NFS servers. This is also the behavior if neither option is specified. A client's open and lock state cannot be migrated transparently when it identifies itself via a traditional identification string. This mount option has no effect with NFSv4 minor versions newer than zero, which always use TSM- compatible client identification strings. This option makes the client use one of the first two string generators below: fs/nfs/nfs4proc.c nfs4_init_nonuniform_client_string 5653 scnprintf(str, len, "Linux NFSv4.0 %s/%s/%s", fs/nfs/nfs4proc.c:5653: scnprintf(str, len, "Linux NFSv4.0 %s/%s/%s", clp->cl_rpcclient->cl_nodename, nfs4_client_id_uniquifier, rpc_peeraddr2str(clp->cl_rpcclient, RPC_DISPLAY_ADDR)); fs/nfs/nfs4proc.c nfs4_init_nonuniform_client_string 5659 scnprintf(str, len, "Linux NFSv4.0 %s/%s", fs/nfs/nfs4proc.c:5659: scnprintf(str, len, "Linux NFSv4.0 %s/%s", clp->cl_rpcclient->cl_nodename, rpc_peeraddr2str(clp->cl_rpcclient, RPC_DISPLAY_ADDR)); fs/nfs/nfs4proc.c nfs4_init_uniquifier_client_string 5691 scnprintf(str, len, "Linux NFSv%u.%u %s/%s", fs/nfs/nfs4proc.c:5691: scnprintf(str, len, "Linux NFSv%u.%u %s/%s", clp->rpc_ops->version, clp->cl_minorversion, nfs4_client_id_uniquifier, clp->cl_rpcclient->cl_nodename); THIS ONE: fs/nfs/nfs4proc.c nfs4_init_uniform_client_string 5726 scnprintf(str, len, "Linux NFSv%u.%u %s", fs/nfs/nfs4proc.c:5726: scnprintf(str, len, "Linux NFSv%u.%u %s", clp->rpc_ops->version, clp->cl_minorversion, clp->cl_rpcclient->cl_nodename); I don’t know if those will help at all. I would be reluctant to force use of the client IP address as part of the client identifier on Ganesha. If that’s really necessary, maybe we can make a config option to enable that? Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Wednesday, March 6, 2019 2:38 PM To: devel@lists.nfs-ganesha.org Cc: Frank Filz <ffilzlnx@mindspring.com> Subject: Client owner collisions Hi, The client owner generation logic on knfs client is very simple and tries to differentiate on the basis of NFS version and hostname. Here’s a snippet, scnprintf(str, len, "Linux NFSv%u.%u %s", clp->rpc_ops->version, clp->cl_minorversion, clp->cl_rpcclient->cl_nodename); clp->cl_owner_id = str; This causes a problem in ganesha in case there are multiple clients connecting to ganesha with same hostname using same NFS version. When comparing client owners, ganesha only takes the client owner from the NFS client into consideration. But if we consider client owner and client IP address to decide if it is the same client before expiring the state, it will treat them as different clients and avoid unnecessary expiry. Any thoughts? Thanks, Sriram
Thanks for the info. I have been using NFSv4.1 mounts, so according to the migration option description, it should be enabled by default. But I am still seeing client owner collisions when clients have same host name. I am using linux kernel > 4.0 (ubuntu) for NFS clients. If the NFS client guarantees unique owners/identifiers we could not have seen this issue. Also, the recent race you fixed by adding refcount for the reply so that it is not freed before response is sent, that was also happening because of the client owner collisions but I don’t see them if I set unique hostnames for all the clients. - Sriram From: Frank Filz <ffilzlnx@mindspring.com> Date: Wednesday, March 6, 2019 at 5:17 PM To: Sriram Patil <sriramp@vmware.com>, "devel@lists.nfs-ganesha.org" <devel@lists.nfs-ganesha.org> Subject: RE: Client owner collisions What is the scenario that has multiple clients with the same hostname? There is a mount option to set a non-uniform client string: migration / nomigration Selects whether the client uses an identification string that is compatible with NFSv4 Transparent State Migration (TSM). If the mounted server supports NFSv4 migration with TSM, specify the migration option. Some server features misbehave in the face of a migration-compatible identification string. The nomigration option retains the use of a traditional client indentification string which is compatible with legacy NFS servers. This is also the behavior if neither option is specified. A client's open and lock state cannot be migrated transparently when it identifies itself via a traditional identification string. This mount option has no effect with NFSv4 minor versions newer than zero, which always use TSM- compatible client identification strings. This option makes the client use one of the first two string generators below: fs/nfs/nfs4proc.c nfs4_init_nonuniform_client_string 5653 scnprintf(str, len, "Linux NFSv4.0 %s/%s/%s", fs/nfs/nfs4proc.c:5653: scnprintf(str, len, "Linux NFSv4.0 %s/%s/%s", clp->cl_rpcclient->cl_nodename, nfs4_client_id_uniquifier, rpc_peeraddr2str(clp->cl_rpcclient, RPC_DISPLAY_ADDR)); fs/nfs/nfs4proc.c nfs4_init_nonuniform_client_string 5659 scnprintf(str, len, "Linux NFSv4.0 %s/%s", fs/nfs/nfs4proc.c:5659: scnprintf(str, len, "Linux NFSv4.0 %s/%s", clp->cl_rpcclient->cl_nodename, rpc_peeraddr2str(clp->cl_rpcclient, RPC_DISPLAY_ADDR)); fs/nfs/nfs4proc.c nfs4_init_uniquifier_client_string 5691 scnprintf(str, len, "Linux NFSv%u.%u %s/%s", fs/nfs/nfs4proc.c:5691: scnprintf(str, len, "Linux NFSv%u.%u %s/%s", clp->rpc_ops->version, clp->cl_minorversion, nfs4_client_id_uniquifier, clp->cl_rpcclient->cl_nodename); THIS ONE: fs/nfs/nfs4proc.c nfs4_init_uniform_client_string 5726 scnprintf(str, len, "Linux NFSv%u.%u %s", fs/nfs/nfs4proc.c:5726: scnprintf(str, len, "Linux NFSv%u.%u %s", clp->rpc_ops->version, clp->cl_minorversion, clp->cl_rpcclient->cl_nodename); I don’t know if those will help at all. I would be reluctant to force use of the client IP address as part of the client identifier on Ganesha. If that’s really necessary, maybe we can make a config option to enable that? Frank From: Sriram Patil [mailto:sriramp@vmware.com] Sent: Wednesday, March 6, 2019 2:38 PM To: devel@lists.nfs-ganesha.org Cc: Frank Filz <ffilzlnx@mindspring.com> Subject: Client owner collisions Hi, The client owner generation logic on knfs client is very simple and tries to differentiate on the basis of NFS version and hostname. Here’s a snippet, scnprintf(str, len, "Linux NFSv%u.%u %s", clp->rpc_ops->version, clp->cl_minorversion, clp->cl_rpcclient->cl_nodename); clp->cl_owner_id = str; This causes a problem in ganesha in case there are multiple clients connecting to ganesha with same hostname using same NFS version. When comparing client owners, ganesha only takes the client owner from the NFS client into consideration. But if we consider client owner and client IP address to decide if it is the same client before expiring the state, it will treat them as different clients and avoid unnecessary expiry. Any thoughts? Thanks, Sriram
Hi, I need some help on understanding if there is a stable/sustainable method of setting up HA with ganesha. Has anyone in this list done this before? We are looking into NFS HA server setup. there seems to some stuff in the web, but it is not exactly clear which method is widely tested and used. thanks.
There are several HA setups that are or have been used with Ganesha. Pacemaker/corosync has been used by a number of projects, including Gluster. Gluster currently uses CTDB via the storhaug project (https://github.com/gluster/storhaug). Ceph uses a container/kubernetes based setup where a crashed/stopped Ganesha has it's container restarted and picks up where it left off. Active/active, of course, requires a clustered filesystem, such as Gluster or CephFS, behind Ganesha. Daniel On 3/28/19 12:15 AM, Imam Toufique wrote:
Hi,
I need some help on understanding if there is a stable/sustainable method of setting up HA with ganesha.
Has anyone in this list done this before? We are looking into NFS HA server setup. there seems to some stuff in the web, but it is not exactly clear which method is widely tested and used.
thanks.
_______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
participants (4)
-
Daniel Gryniewicz -
Frank Filz -
Imam Toufique -
Sriram Patil