10:10 p.m.
Hello,
In our environment of Ceph Cluster(version 15.2.7) we are trying to use NFS HA Mode.Facing
certain issues in the same as below:
"Active/Passive HA NFS Cluster"
When we are using Active/Passive HA Config for NFS Server using Corosync/Pacemekar:
1. configuration is done and we are able to perform fail-over, but when an active
node is tested with power-off two scenarios are observed:
1.1 : I/O operations gets stuck until the node is powered on although the
handover from active to other standby node happens immediately once the node is
powered-off. All the existing requests are stuck.
1.2 : from other client if we try to check for the heartbeat of mount-point,
it is also stuck for the same duration.
1.3 from the new client creating a new mount to the same subvolume works
fine.
Issues/Concern:
I/O operations should resume just after the Failover happens.We are not able to achive
this state, Can anyone please help in supporting any known
configuration/solution/work-around that can be done done at NFS-Ganesha level to achieve
healthy NFS HA Mode.
Just a Note:
mount points using Ceph's native FS driver works fine in the same shutdown/poweroff
scenarios.
Ceph version: 15.2.7
NFS Ganesha : 3.3
Ganesha Conf:
- NFS Node 1:
[ansible@cephnode2 ~]$ cat /etc/ganesha/ganesha.conf
# Please do not change this file directly since it is managed by Ansible and will be
overwritten
NFS_Core_Param
{
Enable_NLM = false;
Enable_RQUOTA = false;
Protocols = 3,4;
}
EXPORT_DEFAULTS {
Attr_Expiration_Time = 0;
}
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
RADOS_URLS {
ceph_conf = '/etc/ceph/ceph.conf';
userid = "admin";
watch_url = "rados://nfs_ganesha/ganesha-export/conf-cephnode2";
}
NFSv4 {
RecoveryBackend = 'rados_cluster';
Lease_Lifetime = 10;
Grace_Period = 20;
}
RADOS_KV {
ceph_conf = '/etc/ceph/ceph.conf';
userid = "admin";
pool = "nfs_ganesha";
namespace = "ganesha-grace";
nodeid = "cephnode2";
}
%url rados://nfs_ganesha/ganesha-export/conf-cephnode2
LOG {
Facility {
name = FILE;
destination = "/var/log/ganesha/ganesha.log";
enable = active;
}
}
EXPORT
{
Export_id=20235;
Path = "/volumes/hns/conf/bb21b7c7-c663-40e9-ad11-a61441e6f77f";
Pseudo = /conf;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
EXPORT
{
Export_id=20236;
Path = "/volumes/hns/opr/138304ca-a70d-4962-9754-b572bce196b6";
Pseudo = /opr;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
# NFS Node 2:
[ansible@cephnode3 ~]$ cat /etc/ganesha/ganesha.conf
# Please do not change this file directly since it is managed by Ansible and will be
overwritten
NFS_Core_Param
{
Enable_NLM = false;
Enable_RQUOTA = false;
Protocols = 3,4;
}
EXPORT_DEFAULTS {
Attr_Expiration_Time = 0;
}
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
RADOS_URLS {
ceph_conf = '/etc/ceph/ceph.conf';
userid = "admin";
watch_url = "rados://nfs_ganesha/ganesha-export/conf-cephnode3";
}
NFSv4 {
RecoveryBackend = 'rados_cluster';
Lease_Lifetime = 10;
Grace_Period = 20;
}
RADOS_KV {
ceph_conf = '/etc/ceph/ceph.conf';
userid = "admin";
pool = "nfs_ganesha";
namespace = "ganesha-grace";
nodeid = "cephnode3";
}
%url rados://nfs_ganesha/ganesha-export/conf-cephnode3
LOG {
Facility {
name = FILE;
destination = "/var/log/ganesha/ganesha.log";
enable = active;
}
}
EXPORT
{
Export_id=20235;
Path = "/volumes/hns/conf/bb21b7c7-c663-40e9-ad11-a61441e6f77f";
Pseudo = /conf;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
EXPORT
{
Export_id=20236;
Path = "/volumes/hns/opr/138304ca-a70d-4962-9754-b572bce196b6";
Pseudo = /opr;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
## Mount Commands at client side:
sudo mount -t nfs -o nfsvers=4.1,proto=tcp 10.0.4.14:/conf /mnt/nfsconf
where 10.0.4.14 is the floating IP.
11:20 p.m.
Hello,
In our environment of Ceph Cluster(version 15.2.7) we are trying to use NFS HA
Mode.Facing
certain issues in the same as below:
"Active/Passive HA NFS Cluster"
When we are using Active/Passive HA Config for NFS Server using Corosync/Pacemekar:
1. configuration is done and we are able to perform fail-over, but when an
active
node is tested with power-off two scenarios are observed:
1.1 : I/O operations gets stuck until the node is powered on although the
handover from active to other standby node happens immediately once the node is
powered-off. All the existing requests are stuck.
1.2 : from other client if we try to check for the heartbeat of
mount-point,
it is also stuck for the same duration.
1.3 from the new client creating a new mount to the same subvolume works
fine.
Issues/Concern:
I/O operations should resume just after the Failover happens.We are not able to achive
this state, Can anyone please help in supporting any known
configuration/solution/work-around that can be done done at NFS-Ganesha level to achieve
healthy NFS HA Mode.
Just a Note:
mount points using Ceph's native FS driver works fine in the same shutdown/poweroff
scenarios.
Ceph version: 15.2.7
NFS Ganesha : 3.3
Ganesha Conf:
- NFS Node 1:
[ansible@cephnode2 ~]$ cat /etc/ganesha/ganesha.conf
# Please do not change this file directly since it is managed by Ansible and will be
overwritten
NFS_Core_Param
{
Enable_NLM = false;
Enable_RQUOTA = false;
Protocols = 3,4;
}
EXPORT_DEFAULTS {
Attr_Expiration_Time = 0;
}
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
RADOS_URLS {
ceph_conf = '/etc/ceph/ceph.conf';
userid = "admin";
watch_url = "rados://nfs_ganesha/ganesha-export/conf-cephnode2";
}
NFSv4 {
RecoveryBackend = 'rados_cluster'; #active/active
RecoveryBackend = 'rados_ng'; #active/passive
> Lease_Lifetime = 10;
> Grace_Period = 20;
>
> }
>
>
> RADOS_KV {
> ceph_conf = '/etc/ceph/ceph.conf';
> userid = "admin";
> pool = "nfs_ganesha";
> namespace = "ganesha-grace";
> nodeid = "cephnode2";
> }
>
> %url rados://nfs_ganesha/ganesha-export/conf-cephnode2
>
> LOG {
> Facility {
> name = FILE;
> destination = "/var/log/ganesha/ganesha.log";
> enable = active;
> }
>
>
> }
> EXPORT
> {
> Export_id=20235;
> Path = "/volumes/hns/conf/bb21b7c7-c663-40e9-ad11-a61441e6f77f";
> Pseudo = /conf;
> Access_Type = RW;
> Protocols = 3,4;
> Transports = TCP;
> SecType = sys,krb5,krb5i,krb5p;
> Squash = No_Root_Squash;
> Attr_Expiration_Time = 0;
> FSAL {
> Name = CEPH;
> User_Id = "admin";
> }
> }
> EXPORT
> {
> Export_id=20236;
> Path = "/volumes/hns/opr/138304ca-a70d-4962-9754-b572bce196b6";
> Pseudo = /opr;
> Access_Type = RW;
> Protocols = 3,4;
> Transports = TCP;
> SecType = sys,krb5,krb5i,krb5p;
> Squash = No_Root_Squash;
> Attr_Expiration_Time = 0;
> FSAL {
> Name = CEPH;
> User_Id = "admin";
> }
> }
>
>
> # NFS Node 2:
> [ansible@cephnode3 ~]$ cat /etc/ganesha/ganesha.conf
> # Please do not change this file directly since it is managed by Ansible and will be
> overwritten
>
>
> NFS_Core_Param
> {
> Enable_NLM = false;
> Enable_RQUOTA = false;
> Protocols = 3,4;
> }
>
> EXPORT_DEFAULTS {
> Attr_Expiration_Time = 0;
> }
>
> CACHEINODE {
> Dir_Chunk = 0;
> NParts = 1;
> Cache_Size = 1;
> }
>
> RADOS_URLS {
> ceph_conf = '/etc/ceph/ceph.conf';
> userid = "admin";
> watch_url = "rados://nfs_ganesha/ganesha-export/conf-cephnode3";
> }
> NFSv4 {
> RecoveryBackend = 'rados_cluster'; # for active/active
> RecoveryBackend = 'rados_ng'; #active/passive
> Lease_Lifetime = 10;
> Grace_Period = 20;
>
> }
>
>
> RADOS_KV {
> ceph_conf = '/etc/ceph/ceph.conf';
> userid = "admin";
> pool = "nfs_ganesha";
> namespace = "ganesha-grace";
> nodeid = "cephnode3";
> }
>
> %url rados://nfs_ganesha/ganesha-export/conf-cephnode3
>
> LOG {
> Facility {
> name = FILE;
> destination = "/var/log/ganesha/ganesha.log";
> enable = active;
> }
>
>
> }
> EXPORT
> {
> Export_id=20235;
> Path = "/volumes/hns/conf/bb21b7c7-c663-40e9-ad11-a61441e6f77f";
> Pseudo = /conf;
> Access_Type = RW;
> Protocols = 3,4;
> Transports = TCP;
> SecType = sys,krb5,krb5i,krb5p;
> Squash = No_Root_Squash;
> Attr_Expiration_Time = 0;
> FSAL {
> Name = CEPH;
> User_Id = "admin";
> }
> }
> EXPORT
> {
> Export_id=20236;
> Path = "/volumes/hns/opr/138304ca-a70d-4962-9754-b572bce196b6";
> Pseudo = /opr;
> Access_Type = RW;
> Protocols = 3,4;
> Transports = TCP;
> SecType = sys,krb5,krb5i,krb5p;
> Squash = No_Root_Squash;
> Attr_Expiration_Time = 0;
> FSAL {
> Name = CEPH;
> User_Id = "admin";
> }
> }
>
> ## Mount Commands at client side:
> sudo mount -t nfs -o nfsvers=4.1,proto=tcp 10.0.4.14:/conf /mnt/nfsconf
> where 10.0.4.14 is the floating IP.
4:46 a.m.
On Wed, Apr 14, 2021 at 1:13 AM <lokendrarathour(a)gmail.com> wrote:
Hello,
In our environment of Ceph Cluster(version 15.2.7) we are trying to use
NFS HA Mode.Facing certain issues in the same as below:
"Active/Passive HA NFS Cluster"
When we are using Active/Passive HA Config for NFS Server using
Corosync/Pacemekar:
1. configuration is done and we are able to perform fail-over, but
when an active node is tested with power-off two scenarios are observed:
1.1 : I/O operations gets stuck until the node is powered on
although the handover from active to other standby node happens immediately
once the node is powered-off. All the existing requests are stuck.
And you're sure they're not just stuck for the duration of NFS-GRACE?
(Default = 90sec)
Do you see the entries in the ganesha logs for entering and leaving
NFS-GRACE?
Are you using the appropriate fencing agent(s) for your hardware?
--
Kaleb
12:03 p.m.
New subject: NFS Ganesha Active/Passive HA Failover Issue
Hi Kaleb,
Setting up of the grace-period is not working in case of active/active setup. we have
tried setting up Grace_Period = 20; but it does not impact the problem.
Setup:
3 Node Ceph Cluster,
Node1: ceph(mon/mds/rgw/osd)
Node2: ceph(mon/mds/rgw/osd)+NFS(Active)
Node3: Ceph(only Mon)+NFS(Active)
Issues is :
I/O operations is not resuming after NFS Node is powered off.
if Node 2(refer above) is powered off, it remains stuck unless powered-on.
if Node3is powered off,it takes around 5 min to resume the I/O.
---
Yes i see the ganesha.log for entering and leaving NFS-GRACE?
14/04/2021 17:50:17 : epoch 6076ddcc : cephnode2 : ganesha.nfsd-1769[main] nfs_start_grace
:STATE :EVENT :NFS Server Now IN GRACE, duration 20
14/04/2021 17:50:23 : epoch 6076ddcc : cephnode2 : ganesha.nfsd-1769[main]
rados_cluster_grace_enforcing :CLIENT ID :EVENT :rados_cluster_grace_enforcing: ret=0
14/04/2021 17:50:23 : epoch 6076ddcc : cephnode2 : ganesha.nfsd-1769[main] nfs_start :NFS
STARTUP :EVENT :-------------------------------------------------
14/04/2021 17:50:34 : epoch 6076ddcc : cephnode2 : ganesha.nfsd-1769[reaper]
nfs_lift_grace_locked :STATE :EVENT :NFS Server Now NOT IN GRACE
in current state we are only using pcs cluster to failover IP resource, but in case of
active/passive - pcs property set stonith-enabled=false
any standard configuration for this ?
11:21 p.m.
New subject: NFS Ganesha Active/Passive HA Failover Issue
**correction: we are trying active/passive mode. Although we have tried active/active
configuration with standard change in configuration. but observation is similar.
4:28 a.m.
On Wed, Apr 14, 2021 at 3:04 PM <lokendrarathour(a)gmail.com> wrote:
Hi Kaleb,
Setting up of the grace-period is not working in case of active/active
setup. we have tried setting up Grace_Period = 20; but it does not impact
the problem.
Setup:
3 Node Ceph Cluster,
Node1: ceph(mon/mds/rgw/osd)
Node2: ceph(mon/mds/rgw/osd)+NFS(Active)
Node3: Ceph(only Mon)+NFS(Active)
Issues is :
I/O operations is not resuming after NFS Node is powered off.
if Node 2(refer above) is powered off, it remains stuck unless powered-on.
if Node3is powered off,it takes around 5 min to resume the I/O.
---
Yes i see the ganesha.log for entering and leaving NFS-GRACE?
Next thing to check is whether your switch gear is allowing gratuitous ARP
forwarding. It's usually on by default but some admins turn it off if they
don't know what it's for.
--
Kaleb
3:05 a.m.
New subject: NFS Ganesha Active/Passive HA Failover Issue
Hi Kaleb,
As checked we do not see any issue in the ARP Forwarding.
what other reason u feel for this 5 min to resume I/O Stuck situation ?
is it some constant in the nfs-ganesha which is by force holding the MDS/Request for
constant period(5 min) before it could transfer it to other node ?
to achieve HA, the failover-handover time should be very less.
thanks once again for your support.
-Lokendra
4:26 a.m.
On Fri, Apr 16, 2021 at 6:05 AM <lokendrarathour(a)gmail.com> wrote:
As checked we do not see any issue in the ARP Forwarding.
what other reason u feel for this 5 min to resume I/O Stuck situation ?
is it some constant in the nfs-ganesha which is by force holding the
MDS/Request for constant period(5 min) before it could transfer it to other
node ?
There isn't anything that holds requests.
to achieve HA, the failover-handover time should be very less.
Indeed. And that's exactly what we see in the glusterfs+ganesha HA solution
— it's instantaneous.
The next thing is to collect wireshark/tcpdumps on each of the servers and
on the client that is experiencing the five minute delay. Start the
tcpdumps, then trigger a failover and let the tcpdumps run for a few
minutes before stopping them
Check for every NFS request sent by the client and make sure it arrives on
the correct server. And that every server reply that is sent is received on
the client.
Post the tcpdump files somewhere and one of us will try to make some time
to look at them if you don't find anything.
--
Kaleb
11:14 a.m.
New subject: NFS Ganesha Active/Passive HA Failover Issue
Hi Kaleb, thank you once again for you support.
we have taken the tcpdump and have analysed and are in process of understanding the same
more.
For your reference kindly find the same as from the link :
https://drive.google.com/drive/folders/1bKqlT7wO0gf9mVHyfsrAeZQudtVWnR_B?...
it has "tar.gz" directory which will have tcpdump of both the nodes nfsnode1 and
ndsnode2.
Also for the reference I have added one text file in the same link which will show
some-time stamps and ARP during the fail-over.
we will also update in case we get any input from the wireshark log.
hopeful to hear from you as well.
11:23 a.m.
On Fri, Apr 16, 2021 at 2:15 PM <lokendrarathour(a)gmail.com> wrote:
Hi Kaleb, thank you once again for you support.
we have taken the tcpdump and have analysed and are in process of
understanding the same more.
For your reference kindly find the same as from the link :
https://drive.google.com/drive/folders/1bKqlT7wO0gf9mVHyfsrAeZQudtVWnR_B?...
it has "tar.gz" directory which will have tcpdump of both the nodes
nfsnode1 and ndsnode2.
Are those the servers? What about the client? I need the tcpdump on the
client too.
--
Kaleb
4 a.m.
New subject: NFS Ganesha Active/Passive HA Failover Issue
Hi Kaleb,
Please find the link for the test-case having dump of client and nfsnode.
https://drive.google.com/drive/folders/13j7ynwasASqTAx5Cpdw5IrFdfoVIbqlJ?...
8:39 a.m.
I don't see anything resembling tcpdumps, or links to tcpdumps, in those
files.
I need tcpdumps on all three machines: both servers and the client. They
need to start before the failover, and run a good five minutes after the
failover.
On Sat, Apr 17, 2021 at 7:00 AM <lokendrarathour(a)gmail.com> wrote:
Hi Kaleb,
Please find the link for the test-case having dump of client and nfsnode.
https://drive.google.com/drive/folders/13j7ynwasASqTAx5Cpdw5IrFdfoVIbqlJ?...
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
9:18 a.m.
Hi Kaleb,
This folder have one tar.gz file.
It has all the pcap file from all three nodes.
2 nfs nodes and 1 client machine.
Please check and let me know.
-Lokendra
On Mon, 19 Apr 2021, 21:09 Kaleb Keithley, <kkeithle(a)redhat.com> wrote:
I don't see anything resembling tcpdumps, or links to tcpdumps,
in those
files.
I need tcpdumps on all three machines: both servers and the client. They
need to start before the failover, and run a good five minutes after the
failover.
On Sat, Apr 17, 2021 at 7:00 AM <lokendrarathour(a)gmail.com> wrote:
> Hi Kaleb,
> Please find the link for the test-case having dump of client and nfsnode.
>
> https://drive.google.com/drive/folders/13j7ynwasASqTAx5Cpdw5IrFdfoVIbqlJ?...
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org
> To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
>
1101
days inactive
1106
days old
12 comments
3 participants
participants (3)
-
Kaleb Keithley -
Lokendra Rathour -
lokendrarathour＠gmail.com