[NFS-Ganesha-Support] nfs-ganesha freeze after a few hours

25 Oct 2019

      Hi, 

I’m running a nfs-ganesha on a virtual server with SR-IOV(Intel XP520) which job is basically recording 167 channels of TV feed for a time shift function. The set-up would run for hours without any issues than the VM would just freeze with no apparent reason.

My Ceph cluster is running fine and are barely using the available storage the load on the server is minimal. 

There is no error on the Ceph side the nfs-ganesha virtual machine just freeze.

The nfs-ganesha server is set-up with BGP to the host using frrouting 7 into leaf/spine typologies  which is working fine.

Here is my mount for the NFS 

10.70.0.67:/cephfs/timeshift1 /mnt/cephfs nfs4 noatime,soft,nfsvers=4.1,async,proto=tcp 0 0

Here is my config for Ganesha

NFS_Core_Param
{
}

EXPORT_DEFAULTS {
	Attr_Expiration_Time = 0;
}

CACHEINODE {
	Dir_Chunk = 0;

	NParts = 1;
	Cache_Size = 1;
}

EXPORT
{
	Export_id=20133;

	Path = "/";

	Pseudo = /cephfs;

	Access_Type = RW;

	Protocols = 3,4;

	Transports = TCP;

	SecType = sys,krb5,krb5i,krb5p;

	Squash = No_Root_Squash;

	Attr_Expiration_Time = 0;

	FSAL {
		Name = CEPH;
		User_Id = "admin";
	}

}
EXPORT
{
	Export_id=20134;

	Path = "/";

	Pseudo = /cephobject;

	Access_Type = RW;

	Protocols = 3,4;

	Transports = TCP;

	SecType = sys,krb5,krb5i,krb5p;

	Squash = Root_Squash;

	FSAL {
		Name = RGW;
		User_Id = "cephnfs";
		Access_Key_Id ="5XC5JJPHT1TVF7COSH23";
		Secret_Access_Key = "6347daPBi79srlE3Kw6l4zDA8SMMkJQZjA1ug7LK";
	}

}

RGW {
        ceph_conf = "/etc/ceph/ceph.conf";
        cluster = "ceph";
        name = "client.rgw.cephctl1";

}

LOG {
        Facility {
                name = FILE;
                destination = "/var/log/ganesha/ganesha.log";
                enable = active;
        }

}

Here is my Ceph status at the freeze time. 

  data:
    pools:   7 pools, 172 pgs
    objects: 1.43M objects, 3.7 TiB
    usage:   11 TiB used, 192 TiB / 204 TiB avail
    pgs:     172 active+clean

Ceph version 
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)

Host Version Info 
CentOS 7.2 Kernel 5.3.2-1.el7

Also, tried those kernels also with the same issue.

CentOS Linux (5.3.2-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1062.4.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1062.1.2.el7.x86_64) 7 (Core)

Ganesha Version info : 

nfs-ganesha-ceph 2.8.2 
nfs-ganesha-rgw 2.8.2
libcephfs2 14.2.4

Any guidance on how to resolve this issue would be appreciated

Andre Roberge
andre.roberge@maskicom.net

[NFS-Ganesha-Support] nfs-ganesha freeze after a few hours

andre.roberge＠maskicom.net