1:41 p.m.
Hi,
I’m running a nfs-ganesha on a virtual server with SR-IOV(Intel XP520) which job is
basically recording 167 channels of TV feed for a time shift function. The set-up would
run for hours without any issues than the VM would just freeze with no apparent reason.
My Ceph cluster is running fine and are barely using the available storage the load on the
server is minimal.
There is no error on the Ceph side the nfs-ganesha virtual machine just freeze.
The nfs-ganesha server is set-up with BGP to the host using frrouting 7 into leaf/spine
typologies which is working fine.
Here is my mount for the NFS
10.70.0.67:/cephfs/timeshift1 /mnt/cephfs nfs4 noatime,soft,nfsvers=4.1,async,proto=tcp 0
0
Here is my config for Ganesha
NFS_Core_Param
{
}
EXPORT_DEFAULTS {
Attr_Expiration_Time = 0;
}
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
EXPORT
{
Export_id=20133;
Path = "/";
Pseudo = /cephfs;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
EXPORT
{
Export_id=20134;
Path = "/";
Pseudo = /cephobject;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = Root_Squash;
FSAL {
Name = RGW;
User_Id = "cephnfs";
Access_Key_Id ="5XC5JJPHT1TVF7COSH23";
Secret_Access_Key = "6347daPBi79srlE3Kw6l4zDA8SMMkJQZjA1ug7LK";
}
}
RGW {
ceph_conf = "/etc/ceph/ceph.conf";
cluster = "ceph";
name = "client.rgw.cephctl1";
}
LOG {
Facility {
name = FILE;
destination = "/var/log/ganesha/ganesha.log";
enable = active;
}
}
Here is my Ceph status at the freeze time.
data:
pools: 7 pools, 172 pgs
objects: 1.43M objects, 3.7 TiB
usage: 11 TiB used, 192 TiB / 204 TiB avail
pgs: 172 active+clean
Ceph version
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
Host Version Info
CentOS 7.2 Kernel 5.3.2-1.el7
Also, tried those kernels also with the same issue.
CentOS Linux (5.3.2-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1062.4.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1062.1.2.el7.x86_64) 7 (Core)
Ganesha Version info :
nfs-ganesha-ceph 2.8.2
nfs-ganesha-rgw 2.8.2
libcephfs2 14.2.4
Any guidance on how to resolve this issue would be appreciated
Andre Roberge
andre.roberge(a)maskicom.net
8:23 a.m.
My first question is: Does the whole VM freeze? Or just Ganesha?
Ganesha's a userspace app, so it can't really freeze the VM; if the VM
is frozen, then it's a problem either with the host or the guest OS.
If it's just Ganesha freezing, then you should turn on some logging.
I'd start with NFS4 at DEBUG to start, maybe add FSAL at DEBUG after
that, if nothing useful is in the log. Then, when it freezes, grab the
log and look at the end to see what's going on.
Daniel
On 10/25/19 2:41 PM, andre.roberge(a)maskicom.net wrote:
Hi,
I’m running a nfs-ganesha on a virtual server with SR-IOV(Intel XP520) which job is
basically recording 167 channels of TV feed for a time shift function. The set-up would
run for hours without any issues than the VM would just freeze with no apparent reason.
My Ceph cluster is running fine and are barely using the available storage the load on
the server is minimal.
There is no error on the Ceph side the nfs-ganesha virtual machine just freeze.
The nfs-ganesha server is set-up with BGP to the host using frrouting 7 into leaf/spine
typologies which is working fine.
Here is my mount for the NFS
10.70.0.67:/cephfs/timeshift1 /mnt/cephfs nfs4 noatime,soft,nfsvers=4.1,async,proto=tcp 0
0
Here is my config for Ganesha
NFS_Core_Param
{
}
EXPORT_DEFAULTS {
Attr_Expiration_Time = 0;
}
CACHEINODE {
Dir_Chunk = 0;
NParts = 1;
Cache_Size = 1;
}
EXPORT
{
Export_id=20133;
Path = "/";
Pseudo = /cephfs;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = No_Root_Squash;
Attr_Expiration_Time = 0;
FSAL {
Name = CEPH;
User_Id = "admin";
}
}
EXPORT
{
Export_id=20134;
Path = "/";
Pseudo = /cephobject;
Access_Type = RW;
Protocols = 3,4;
Transports = TCP;
SecType = sys,krb5,krb5i,krb5p;
Squash = Root_Squash;
FSAL {
Name = RGW;
User_Id = "cephnfs";
Access_Key_Id ="5XC5JJPHT1TVF7COSH23";
Secret_Access_Key = "6347daPBi79srlE3Kw6l4zDA8SMMkJQZjA1ug7LK";
}
}
RGW {
ceph_conf = "/etc/ceph/ceph.conf";
cluster = "ceph";
name = "client.rgw.cephctl1";
}
LOG {
Facility {
name = FILE;
destination = "/var/log/ganesha/ganesha.log";
enable = active;
}
}
Here is my Ceph status at the freeze time.
data:
pools: 7 pools, 172 pgs
objects: 1.43M objects, 3.7 TiB
usage: 11 TiB used, 192 TiB / 204 TiB avail
pgs: 172 active+clean
Ceph version
ceph version 14.2.4 (75f4de193b3ea58512f204623e6c5a16e6c1e1ba) nautilus (stable)
Host Version Info
CentOS 7.2 Kernel 5.3.2-1.el7
Also, tried those kernels also with the same issue.
CentOS Linux (5.3.2-1.el7.elrepo.x86_64) 7 (Core)
CentOS Linux (3.10.0-1062.4.1.el7.x86_64) 7 (Core)
CentOS Linux (3.10.0-1062.1.2.el7.x86_64) 7 (Core)
Ganesha Version info :
nfs-ganesha-ceph 2.8.2
nfs-ganesha-rgw 2.8.2
libcephfs2 14.2.4
Any guidance on how to resolve this issue would be appreciated
Andre Roberge
andre.roberge(a)maskicom.net
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
10:42 a.m.
I have installed kdump with all the requirement to be able to get the details of the
crash.
[ 6474.803105] BUG: kernel NULL pointer dereference, address: 0000000000000014
[ 6474.803174] #PF: supervisor read access in kernel mode
[ 6474.803215] #PF: error_code(0x0000) - not-present page
[ 6474.803256] PGD 0 P4D 0
[ 6474.803284] Oops: 0000 [#1] SMP PTI
[ 6474.803317] CPU: 13 PID: 17818 Comm: kworker/13:0 Kdump: loaded Tainted: G W
5.3.2-1.el7.elrepo.x86_64 $
[ 6474.803396] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 6474.803449] Workqueue: events key_garbage_collector
[ 6474.803494] RIP: 0010:keyring_gc_check_iterator+0x33/0x40
[ 6474.803557] Code: 55 48 8b 0e f6 87 80 00 00 00 21 b8 01 00 00 00 48 89 e5 75 19 48 8b
57 58 48 85 d2 7e 05 48 39 d1 7$
[ 6474.803692] RSP: 0018:ffffc9000277fd90 EFLAGS: 00010246
[ 6474.803734] RAX: 0000000000000000 RBX: ffff888424da26d0 RCX: 000000005db6ff0d
[ 6474.803788] RDX: 0000000000000000 RSI: ffffc9000277fdf8 RDI: ffff8883bb51a800
[ 6474.803841] RBP: ffffc9000277fd90 R08: 000073746e657665 R09: 8080808080808080
[ 6474.803894] R10: ffff88842d8000f8 R11: 0000000000000000 R12: ffff888424da2750
[ 6474.803953] R13: ffffffff813791c0 R14: ffff888424da26c0 R15: ffff8883bb51a800
[ 6474.804008] FS: 0000000000000000(0000) GS:ffff88842fb40000(0000)
knlGS:0000000000000000
[ 6474.804069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6474.804113] CR2: 0000000000000014 CR3: 000000000220a003 CR4: 00000000003606e0
[ 6474.804174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6474.804228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6474.804281] Call Trace:
[ 6474.804313] assoc_array_subtree_iterate+0x68/0xe0
[ 6474.804353] assoc_array_iterate+0x1b/0x20
[ 6474.804388] keyring_gc+0x43/0x80
[ 6474.804420] key_garbage_collector+0x161/0x400
[ 6474.804461] process_one_work+0x179/0x390
[ 6474.804497] worker_thread+0x4f/0x3e0
[ 6474.804530] kthread+0x105/0x140
[ 6474.804578] ? max_active_store+0x80/0x80
[ 6474.804612] ? kthread_bind+0x20/0x20
[ 6474.804647] ret_from_fork+0x35/0x40
[ 6474.804678] Modules linked in: binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4
dns_resolver nfs lockd grace fscache nfit$
[ 6474.805200] CR2: 0000000000000014
It is an issue with key_garbage_collector but I have no clue on how to fix it !
André
12:52 p.m.
I don't know anything about the kernel keyring code, sorry. And I
suspect that no one on this list can help either, since Ganesha is
userspace. You might have better luck posting on an appropriate kernel
list, or filing a bug with your distro.
Daniel
On 10/28/19 11:42 AM, andre.roberge(a)maskicom.net wrote:
I have installed kdump with all the requirement to be able to get the
details of the crash.
[ 6474.803105] BUG: kernel NULL pointer dereference, address: 0000000000000014
[ 6474.803174] #PF: supervisor read access in kernel mode
[ 6474.803215] #PF: error_code(0x0000) - not-present page
[ 6474.803256] PGD 0 P4D 0
[ 6474.803284] Oops: 0000 [#1] SMP PTI
[ 6474.803317] CPU: 13 PID: 17818 Comm: kworker/13:0 Kdump: loaded Tainted: G W
5.3.2-1.el7.elrepo.x86_64 $
[ 6474.803396] Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
[ 6474.803449] Workqueue: events key_garbage_collector
[ 6474.803494] RIP: 0010:keyring_gc_check_iterator+0x33/0x40
[ 6474.803557] Code: 55 48 8b 0e f6 87 80 00 00 00 21 b8 01 00 00 00 48 89 e5 75 19 48 8b
57 58 48 85 d2 7e 05 48 39 d1 7$
[ 6474.803692] RSP: 0018:ffffc9000277fd90 EFLAGS: 00010246
[ 6474.803734] RAX: 0000000000000000 RBX: ffff888424da26d0 RCX: 000000005db6ff0d
[ 6474.803788] RDX: 0000000000000000 RSI: ffffc9000277fdf8 RDI: ffff8883bb51a800
[ 6474.803841] RBP: ffffc9000277fd90 R08: 000073746e657665 R09: 8080808080808080
[ 6474.803894] R10: ffff88842d8000f8 R11: 0000000000000000 R12: ffff888424da2750
[ 6474.803953] R13: ffffffff813791c0 R14: ffff888424da26c0 R15: ffff8883bb51a800
[ 6474.804008] FS: 0000000000000000(0000) GS:ffff88842fb40000(0000)
knlGS:0000000000000000
[ 6474.804069] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 6474.804113] CR2: 0000000000000014 CR3: 000000000220a003 CR4: 00000000003606e0
[ 6474.804174] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 6474.804228] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 6474.804281] Call Trace:
[ 6474.804313] assoc_array_subtree_iterate+0x68/0xe0
[ 6474.804353] assoc_array_iterate+0x1b/0x20
[ 6474.804388] keyring_gc+0x43/0x80
[ 6474.804420] key_garbage_collector+0x161/0x400
[ 6474.804461] process_one_work+0x179/0x390
[ 6474.804497] worker_thread+0x4f/0x3e0
[ 6474.804530] kthread+0x105/0x140
[ 6474.804578] ? max_active_store+0x80/0x80
[ 6474.804612] ? kthread_bind+0x20/0x20
[ 6474.804647] ret_from_fork+0x35/0x40
[ 6474.804678] Modules linked in: binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4
dns_resolver nfs lockd grace fscache nfit$
[ 6474.805200] CR2: 0000000000000014
It is an issue with key_garbage_collector but I have no clue on how to fix it !
André
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
2199
days inactive
2202
days old
3 comments
2 participants
participants (2)
-
andre.roberge＠maskicom.net -
Daniel Gryniewicz