Re: consistently reproduceable problem with kvm guest booting root filesystem via nfs-ganesha 2.8 proxy fsal
On Tue, 10 Mar 2020, Daniel Gryniewicz wrote:
Maybe turn on logging on Ganesha and look for anything interesting?
I've tried that by changing "-N NIV_CRIT" to "-N NIV_DEBUG" in /etc/sysconfig/ganesha - is there a better way? - but this did not reveal anything interesting to my eyes. I'm quite willing to accept that my eyes may have missed something. Something that I neglected to mention previously is that sometimes (maybe always, not sure yet) when the error condition occurs in the kvm guest, the kvm host's nfs-ganesha proxy mount has gone into a faulty state and requires an unmount and remount to recover.
Also, packet dumps would be useful.
I'll consider this. I may need to first need to set up a proper test environment, unless I can figure out (or someone can tell me :) how best to focus on the useful packets and not be overwhelmed by extraneous data.
I haven't personally stored KVM images on NFS, let alone via Proxy, so I don't have any personal experience in this.
All of our many kvm images are on NFS and have been for years so I do have plenty of personal experience with this, but having them on nfs via ganesha is a new experiment for us. We do have many terabytes of data filesystems served via nfs-ganesha proxy fsal though, with generally good results.
What's the OS/version of the original NFS server that Ganesha's proxying?
Also CentOS 7. Everything in our mix is CO6 or CO7. Thanks, Todd
Daniel
On 3/9/20 5:46 PM, Todd Pfaff wrote:
I have a reproduceable problem when trying to boot a kvm guest whose root filesystem is on an nfs path that is accessed from the kvm virt host via nfs-ganesha-2.8 and the proxy fsal.
All hosts involved: kvm host, kvm guest, and nfs-ganesha server, run CentOS 7.
The guest begins to boot but consistently fails just after the switch root with ext4 errors like this:
[ OK ] Started Plymouth switch root service. Starting Switch Root... [ 15.154967] systemd-journald[145]: Received SIGTERM from PID 1 (systemd). [ 15.168447] EXT4-fs error (device vda1): __ext4_get_inode_loc:4247: inode #131073: block 524320: comm systemd: unable to read itable block [ 15.269495] EXT4-fs warning (device vda1): __ext4_read_dirblock:676: error reading directory block (ino 524296, block 0) [ 15.278651] systemd[1]: Failed to execute /sbin/init, giving up: Input/output error
I'm using nfs-ganesha from the CentOS 7 Storage SIG.
I've tested with both the latest stable 2.8 version from repo [centos-nfs-ganesha28]:
nfs-ganesha-2.8.3-3.el7.x86_64
and the latest test version from [centos-nfs-ganesha28-test]:
nfs-ganesha-2.8.3-4.el7.x86_64
with similar results. I've also tried with nfs v3 and nfs v4.1 mounts from the kvm host.
The kvm guest disk is defined as:
<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/mnt/ganesha/kvm/guest.qcow2'/> <target dev='vdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk>
The kvm guest works fine if the kvm host accesses this qcow2 image via a direct nfs mount instead of via nfs-ganesha proxy.
Thoughts?
Thanks, Todd _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
On 3/10/20 10:29 AM, Todd Pfaff wrote:
On Tue, 10 Mar 2020, Daniel Gryniewicz wrote:
Maybe turn on logging on Ganesha and look for anything interesting?
I've tried that by changing "-N NIV_CRIT" to "-N NIV_DEBUG" in /etc/sysconfig/ganesha - is there a better way? - but this did not reveal anything interesting to my eyes. I'm quite willing to accept that my eyes may have missed something.
You can set per-component log levels in the LOG section of the config file. Something like this: LOG { Components { ALL = FULL_DEBUG; } } where ALL can be replaced with any of the logging categories. See the ganesha-log-config man page for more. If you enable FULL_DEBUG for a component, it will log just about everything related to that component, but it's a lot of output. Likely interesting components for you are the NFS ones (NFS4, NFS3, NFS4_LOCK, NFS_READDIR).
Something that I neglected to mention previously is that sometimes (maybe always, not sure yet) when the error condition occurs in the kvm guest, the kvm host's nfs-ganesha proxy mount has gone into a faulty state and requires an unmount and remount to recover.
Also, packet dumps would be useful.
I'll consider this. I may need to first need to set up a proper test environment, unless I can figure out (or someone can tell me :) how best to focus on the useful packets and not be overwhelmed by extraneous data.
Unfortunately, capturing all NFS traffic is likely to be the best you can do without a lot more information about what's erroring out.
I haven't personally stored KVM images on NFS, let alone via Proxy, so I don't have any personal experience in this.
All of our many kvm images are on NFS and have been for years so I do have plenty of personal experience with this, but having them on nfs via ganesha is a new experiment for us. We do have many terabytes of data filesystems served via nfs-ganesha proxy fsal though, with generally good results.
This is good to know, thanks. Block devices frequently have different access patterns than normal files, so they're not as well tested. Daniel
What's the OS/version of the original NFS server that Ganesha's proxying?
Also CentOS 7. Everything in our mix is CO6 or CO7.
Thanks, Todd
Daniel
On 3/9/20 5:46 PM, Todd Pfaff wrote:
I have a reproduceable problem when trying to boot a kvm guest whose root filesystem is on an nfs path that is accessed from the kvm virt host via nfs-ganesha-2.8 and the proxy fsal.
All hosts involved: kvm host, kvm guest, and nfs-ganesha server, run CentOS 7.
The guest begins to boot but consistently fails just after the switch root with ext4 errors like this:
[ OK ] Started Plymouth switch root service. Starting Switch Root... [ 15.154967] systemd-journald[145]: Received SIGTERM from PID 1 (systemd). [ 15.168447] EXT4-fs error (device vda1): __ext4_get_inode_loc:4247: inode #131073: block 524320: comm systemd: unable to read itable block [ 15.269495] EXT4-fs warning (device vda1): __ext4_read_dirblock:676: error reading directory block (ino 524296, block 0) [ 15.278651] systemd[1]: Failed to execute /sbin/init, giving up: Input/output error
I'm using nfs-ganesha from the CentOS 7 Storage SIG.
I've tested with both the latest stable 2.8 version from repo [centos-nfs-ganesha28]:
nfs-ganesha-2.8.3-3.el7.x86_64
and the latest test version from [centos-nfs-ganesha28-test]:
nfs-ganesha-2.8.3-4.el7.x86_64
with similar results. I've also tried with nfs v3 and nfs v4.1 mounts from the kvm host.
The kvm guest disk is defined as:
<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/mnt/ganesha/kvm/guest.qcow2'/> <target dev='vdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk>
The kvm guest works fine if the kvm host accesses this qcow2 image via a direct nfs mount instead of via nfs-ganesha proxy.
Thoughts?
Thanks, Todd _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
participants (2)
-
Daniel Gryniewicz -
Todd Pfaff