On Tue, 10 Mar 2020, Daniel Gryniewicz wrote:
Maybe turn on logging on Ganesha and look for anything interesting?
I've tried that by changing "-N NIV_CRIT" to "-N NIV_DEBUG" in /etc/sysconfig/ganesha - is there a better way? - but this did not reveal anything interesting to my eyes. I'm quite willing to accept that my eyes may have missed something. Something that I neglected to mention previously is that sometimes (maybe always, not sure yet) when the error condition occurs in the kvm guest, the kvm host's nfs-ganesha proxy mount has gone into a faulty state and requires an unmount and remount to recover.
Also, packet dumps would be useful.
I'll consider this. I may need to first need to set up a proper test environment, unless I can figure out (or someone can tell me :) how best to focus on the useful packets and not be overwhelmed by extraneous data.
I haven't personally stored KVM images on NFS, let alone via Proxy, so I don't have any personal experience in this.
All of our many kvm images are on NFS and have been for years so I do have plenty of personal experience with this, but having them on nfs via ganesha is a new experiment for us. We do have many terabytes of data filesystems served via nfs-ganesha proxy fsal though, with generally good results.
What's the OS/version of the original NFS server that Ganesha's proxying?
Also CentOS 7. Everything in our mix is CO6 or CO7. Thanks, Todd
Daniel
On 3/9/20 5:46 PM, Todd Pfaff wrote:
I have a reproduceable problem when trying to boot a kvm guest whose root filesystem is on an nfs path that is accessed from the kvm virt host via nfs-ganesha-2.8 and the proxy fsal.
All hosts involved: kvm host, kvm guest, and nfs-ganesha server, run CentOS 7.
The guest begins to boot but consistently fails just after the switch root with ext4 errors like this:
[ OK ] Started Plymouth switch root service. Starting Switch Root... [ 15.154967] systemd-journald[145]: Received SIGTERM from PID 1 (systemd). [ 15.168447] EXT4-fs error (device vda1): __ext4_get_inode_loc:4247: inode #131073: block 524320: comm systemd: unable to read itable block [ 15.269495] EXT4-fs warning (device vda1): __ext4_read_dirblock:676: error reading directory block (ino 524296, block 0) [ 15.278651] systemd[1]: Failed to execute /sbin/init, giving up: Input/output error
I'm using nfs-ganesha from the CentOS 7 Storage SIG.
I've tested with both the latest stable 2.8 version from repo [centos-nfs-ganesha28]:
nfs-ganesha-2.8.3-3.el7.x86_64
and the latest test version from [centos-nfs-ganesha28-test]:
nfs-ganesha-2.8.3-4.el7.x86_64
with similar results. I've also tried with nfs v3 and nfs v4.1 mounts from the kvm host.
The kvm guest disk is defined as:
<disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/mnt/ganesha/kvm/guest.qcow2'/> <target dev='vdc' bus='virtio'/> <boot order='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk>
The kvm guest works fine if the kvm host accesses this qcow2 image via a direct nfs mount instead of via nfs-ganesha proxy.
Thoughts?
Thanks, Todd _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org