April 2018 - Support - NFS Ganesha List Archives

[Support] ERR 20: Auth Rejected Credentials (client should begin new session)
by TomK 05 Aug '18

05 Aug '18

Hey All, I have an external NFS cluster serviced by a VIP. The clients run autofs configured via IPA to provide NFS home directories to client. However, running into an issue on one of the clients and wondering if anyone seen this message from a tcpdump of a simple mount session that's preventing the mount: psql02: mount nfs-c01:/n /m Yields this message ERR 20: Auth Rejected Credentials (client should begin new session) and the mount attempt never exits and never mounts /m . nfs-c01 is a VIP that's serviced by HAproxy / keepalived. nfs-c01 however has a record in IPA Server, both forward and a reverse one. Using one of the underlying hosts that services nfs-c01 works and mounts succeeds for them. All VM hosts are clones of the same template. I have autofs running as part of this IPA client setup and applied the following fix as well: https://access.redhat.com/solutions/3261981 /m is a test mount folder I'm using on this client to troubleshoot the autofs mounting issue. So autofs is also running on the same hosts where I'm trying this mount from. Trying to trace the exact source of this error and not quite sure where to look further. idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively) psql01/02 are the problem VM's. (192.168.0.108/124 ) nfs01/02 are the NFS hosts. (192.168.0.131/119 ) nfs-c01 192.168.0.80 This works fine on the other two VM hosts without any issue but I just can't find any difference comparing all the configs and so looking for suggestions to bounce off of. -- Cheers, Tom K. ------------------------------------------------------------------------------------- Living on earth is expensive, but it includes a free trip around the sun. Apr 15 23:29:54 psql02 kernel: INFO: task mount.nfs:1443 blocked for more than 120 seconds. Apr 15 23:29:54 psql02 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 15 23:29:54 psql02 kernel: mount.nfs D ffff880135ed8000 0 1443 1442 0x00000080 Apr 15 23:29:54 psql02 kernel: Call Trace: Apr 15 23:29:54 psql02 kernel: [<ffffffff816ac7c9>] schedule_preempt_disabled+0x29/0x70 Apr 15 23:29:54 psql02 kernel: [<ffffffff816aa5f7>] __mutex_lock_slowpath+0xc7/0x1d0 Apr 15 23:29:54 psql02 kernel: [<ffffffff816a9a0f>] mutex_lock+0x1f/0x2f Apr 15 23:29:54 psql02 kernel: [<ffffffffc05ddd58>] nfs4_discover_server_trunking+0x48/0x2e0 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffffc05e6906>] nfs4_init_client+0x126/0x300 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffff811e17d3>] ? kmem_cache_alloc+0x193/0x1e0 Apr 15 23:29:54 psql02 kernel: [<ffffffffc0562526>] ? __fscache_acquire_cookie+0x66/0x180 [fscache] Apr 15 23:29:54 psql02 kernel: [<ffffffffc0562526>] ? __fscache_acquire_cookie+0x66/0x180 [fscache] Apr 15 23:29:54 psql02 kernel: [<ffffffffc0341b61>] ? __rpc_init_priority_wait_queue+0x81/0xc0 [sunrpc] Apr 15 23:29:54 psql02 kernel: [<ffffffffc057c6d6>] nfs_get_client+0x2c6/0x3e0 [nfs] Apr 15 23:29:54 psql02 kernel: [<ffffffffc05e5de8>] nfs4_set_client+0x98/0x130 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffffc05e75de>] nfs4_create_server+0x13e/0x3b0 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffffc05de7ce>] nfs4_remote_mount+0x2e/0x60 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffff81207549>] mount_fs+0x39/0x1b0 Apr 15 23:29:54 psql02 kernel: [<ffffffff811a7f25>] ? __alloc_percpu+0x15/0x20 Apr 15 23:29:54 psql02 kernel: [<ffffffff81224177>] vfs_kern_mount+0x67/0x110 Apr 15 23:29:54 psql02 kernel: [<ffffffffc05de6f6>] nfs_do_root_mount+0x86/0xc0 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffffc05deaf4>] nfs4_try_mount+0x44/0xc0 [nfsv4] Apr 15 23:29:54 psql02 kernel: [<ffffffffc057d627>] ? get_nfs_version+0x27/0x90 [nfs] Apr 15 23:29:54 psql02 kernel: [<ffffffffc0589a95>] nfs_fs_mount+0x4c5/0xd90 [nfs] Apr 15 23:29:54 psql02 kernel: [<ffffffffc058a9c0>] ? nfs_clone_super+0x140/0x140 [nfs] Apr 15 23:29:54 psql02 kernel: [<ffffffffc05888c0>] ? param_set_portnr+0x70/0x70 [nfs] Apr 15 23:29:54 psql02 kernel: [<ffffffff81207549>] mount_fs+0x39/0x1b0 Apr 15 23:29:54 psql02 kernel: [<ffffffff811a7f25>] ? __alloc_percpu+0x15/0x20 Apr 15 23:29:54 psql02 kernel: [<ffffffff81224177>] vfs_kern_mount+0x67/0x110 Apr 15 23:29:54 psql02 kernel: [<ffffffff81226683>] do_mount+0x233/0xaf0 Apr 15 23:29:54 psql02 kernel: [<ffffffff812272c6>] SyS_mount+0x96/0xf0 Apr 15 23:29:54 psql02 kernel: [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b Apr 15 23:29:54 psql02 kernel: [<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214 Message from syslogd@psql01 at Apr 17 03:08:31 ... kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mount.nfs:1606] Linux psql02.nix.my.dom 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux [root@nfs02 log]# tcpdump -i eth0|grep -v "192.168.0.76"|grep -v NLB|grep -v nfs01|grep -v netbios|grep -v NBT tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes 01:23:42.183158 IP nfs02.my.dom.xyz.60807 > idmipa01.my.dom.xyz.domain: 23358+ PTR? 76.0.168.192.in-addr.arpa. (43) 01:23:42.183884 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.60807: 23358 NXDomain* 0/1/0 (110) 01:23:42.184090 IP nfs02.my.dom.xyz.59911 > idmipa01.my.dom.xyz.domain: 29059+ PTR? 119.0.168.192.in-addr.arpa. (44) 01:23:42.184601 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.59911: 29059* 1/2/2 PTR nfs02.my.dom.xyz. (153) 01:23:42.184827 IP nfs02.my.dom.xyz.49329 > idmipa01.my.dom.xyz.domain: 50753+ PTR? 44.0.168.192.in-addr.arpa. (43) 01:23:42.185122 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.49329: 50753* 1/2/2 PTR idmipa01.my.dom.xyz. (146) 01:23:42.250263 IP nfs02.my.dom.xyz.49035 > idmipa01.my.dom.xyz.domain: 17264+ PTR? 255.0.168.192.in-addr.arpa. (44) 01:23:42.250983 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.49035: 17264 NXDomain* 0/1/0 (111) 01:23:42.257700 IP nfs02.my.dom.xyz.51938 > idmipa01.my.dom.xyz.domain: 51451+ PTR? 131.0.168.192.in-addr.arpa. (44) 01:23:42.360669 IP nfs02.my.dom.xyz.46447 > idmipa01.my.dom.xyz.domain: 12552+ PTR? 224.0.168.192.in-addr.arpa. (44) 01:23:42.361247 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.46447: 12552 NXDomain* 0/1/0 (111) 01:23:42.361434 IP nfs02.my.dom.xyz.37305 > idmipa01.my.dom.xyz.domain: 34850+ PTR? 223.0.168.192.in-addr.arpa. (44) 01:23:42.361766 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.37305: 34850 NXDomain* 0/1/0 (111) 01:23:42.420742 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:42.421026 IP nfs02.my.dom.xyz.58510 > idmipa01.my.dom.xyz.domain: 7249+ PTR? 18.0.0.224.in-addr.arpa. (41) 01:23:42.421745 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.58510: 7249 1/13/0 PTR vrrp.mcast.net. (277) 01:23:42.751583 IP nfs02.my.dom.xyz.43409 > idmipa01.my.dom.xyz.domain: 29327+ PTR? 222.0.168.192.in-addr.arpa. (44) 01:23:42.752250 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.43409: 29327 NXDomain* 0/1/0 (111) 01:23:43.421723 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:44.422648 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:45.423492 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:46.424492 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:47.188951 ARP, Request who-has nfs02.my.dom.xyz tell idmipa01.my.dom.xyz, length 46 01:23:47.188966 ARP, Reply nfs02.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:23:47.248948 ARP, Request who-has 192.168.0.103 tell 192.168.0.222, length 46 01:23:47.249297 IP nfs02.my.dom.xyz.50518 > idmipa01.my.dom.xyz.domain: 46693+ PTR? 103.0.168.192.in-addr.arpa. (44) 01:23:47.250150 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.50518: 46693 NXDomain* 0/1/0 (111) 01:23:47.425440 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:48.426303 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:49.427153 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:50.428133 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:51.428574 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags [S], seq 1812770089, win 29200, options [mss 1460,sackOK,TS val 167449689 ecr 0,nop,wscale 7], length 0 01:23:51.428634 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags [S.], seq 2612074554, ack 1812770090, win 28960, options [mss 1460,sackOK,TS val 172963836 ecr 167449689,nop,wscale 7], length 0 01:23:51.428787 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 1, win 229, options [nop,nop,TS val 167449689 ecr 172963836], length 0 01:23:51.428838 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags [P.], seq 1:45, ack 1, win 229, options [nop,nop,TS val 167449689 ecr 172963836], length 44: NFS request xid 2544096005 40 null 01:23:51.428859 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags [.], ack 45, win 227, options [nop,nop,TS val 172963836 ecr 167449689], length 0 01:23:51.429003 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:51.429079 IP nfs02.my.dom.xyz.52213 > idmipa01.my.dom.xyz.domain: 17880+ PTR? 80.0.168.192.in-addr.arpa. (43) 01:23:51.429514 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.52213: 17880* 1/2/2 PTR nfs-c01.my.dom.xyz. (154) 01:23:51.429748 IP nfs02.my.dom.xyz.58966 > idmipa01.my.dom.xyz.domain: 50455+ PTR? 124.0.168.192.in-addr.arpa. (44) 01:23:51.430092 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.58966: 50455* 1/2/2 PTR psql02.my.dom.xyz. (154) 01:23:51.430129 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags [P.], seq 1:29, ack 45, win 227, options [nop,nop,TS val 172963838 ecr 167449689], length 28: NFS reply xid 2544096005 reply ok 24 null 01:23:51.430247 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 29, win 229, options [nop,nop,TS val 167449690 ecr 172963838], length 0 01:23:51.433079 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs: Flags [S], seq 3882292069, win 29200, options [mss 1460,sackOK,TS val 167449693 ecr 0,nop,wscale 7], length 0 01:23:51.433124 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999: Flags [S.], seq 876416044, ack 3882292070, win 28960, options [mss 1460,sackOK,TS val 172963841 ecr 167449693,nop,wscale 7], length 0 01:23:51.433214 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 1, win 229, options [nop,nop,TS val 167449693 ecr 172963841], length 0 01:23:51.435147 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs: Flags [P.], seq 1:693, ack 1, win 229, options [nop,nop,TS val 167449695 ecr 172963841], length 692: NFS request xid 3844890745 688 null 01:23:51.435184 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999: Flags [.], ack 693, win 238, options [nop,nop,TS val 172963843 ecr 167449695], length 0 01:23:51.436257 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999: Flags [P.], seq 1:25, ack 693, win 238, options [nop,nop,TS val 172963844 ecr 167449695], length 24: NFS reply xid 3844890745 reply ERR 20: Auth Rejected Credentials (client should begin new session) 01:23:51.436374 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 25, win 229, options [nop,nop,TS val 167449697 ecr 172963844], length 0 01:23:51.483369 IP nfs02.my.dom.xyz.53527 > idmipa01.my.dom.xyz.domain: 62714+ PTR? 105.0.168.192.in-addr.arpa. (44) 01:23:51.484027 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.53527: 62714 NXDomain* 0/1/0 (111) 01:23:51.487612 IP nfs02.my.dom.xyz.41147 > idmipa01.my.dom.xyz.domain: 4106+ PTR? 100.0.168.192.in-addr.arpa. (44) 01:23:51.487992 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.41147: 4106 NXDomain* 0/1/0 (111) 01:23:52.429933 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:53.430801 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:54.432246 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:55.433173 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:55.453824 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs: Flags [P.], seq 2034285336:2034285468, ack 3501594280, win 614, options [nop,nop,TS val 272589184 ecr 172927798], length 132: NFS request xid 2543449196 128 getattr fh 0,1/53 01:23:55.454235 IP nfs02.my.dom.xyz.34495 > idmipa01.my.dom.xyz.domain: 10353+ PTR? 145.0.168.192.in-addr.arpa. (44) 01:23:55.454456 IP nfs-c01.my.dom.xyz.nfs > ovirt01.my.dom.xyz.843: Flags [P.], seq 1:85, ack 132, win 788, options [nop,nop,TS val 172967862 ecr 272589184], length 84: NFS reply xid 2543449196 reply ok 80 getattr NON 1 ids 0/50331648 sz 1518458202 01:23:55.454669 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 85, win 614, options [nop,nop,TS val 272589185 ecr 172967862], length 0 01:23:55.455038 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.34495: 10353* 1/2/2 PTR ovirt01.my.dom.xyz. (155) 01:23:56.434163 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:56.443495 ARP, Request who-has nfs-c01.my.dom.xyz tell psql02.my.dom.xyz, length 46 01:23:56.443577 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:23:56.541140 IP nfs02.my.dom.xyz.51079 > idmipa01.my.dom.xyz.domain: 58928+ PTR? 14.0.168.192.in-addr.arpa. (43) 01:23:56.541904 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.51079: 58928 NXDomain* 0/1/0 (110) 01:23:57.435087 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:58.436052 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:23:59.437014 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:00.437885 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:00.456019 ARP, Request who-has ovirt01.my.dom.xyz tell nfs02.my.dom.xyz, length 28 01:24:00.456439 ARP, Reply ovirt01.my.dom.xyz is-at 00:50:56:86:f7:7e (oui Unknown), length 46 01:24:00.461740 ARP, Request who-has nfs-c01.my.dom.xyz tell ovirt01.my.dom.xyz, length 46 01:24:00.461754 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:24:01.233709 ARP, Request who-has 192.168.0.222 tell 192.168.0.221, length 46 01:24:01.234102 IP nfs02.my.dom.xyz.56105 > idmipa01.my.dom.xyz.domain: 38431+ PTR? 221.0.168.192.in-addr.arpa. (44) 01:24:01.234620 ARP, Request who-has 192.168.0.221 tell 192.168.0.222, length 46 01:24:01.234749 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.56105: 38431 NXDomain* 0/1/0 (111) 01:24:01.438897 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:02.439939 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:03.440864 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:03.552006 ARP, Request who-has 192.168.0.1 tell 192.168.0.105, length 46 01:24:03.552336 IP nfs02.my.dom.xyz.46470 > idmipa01.my.dom.xyz.domain: 51908+ PTR? 1.0.168.192.in-addr.arpa. (42) 01:24:03.552821 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.46470: 51908 NXDomain* 0/1/0 (109) 01:24:03.563789 ARP, Request who-has 192.168.0.1 tell 192.168.0.100, length 46 01:24:04.441825 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:04.947786 ARP, Request who-has idmipa01.my.dom.xyz tell 192.168.0.220, length 46 01:24:04.948254 IP nfs02.my.dom.xyz.57360 > idmipa01.my.dom.xyz.domain: 43921+ PTR? 220.0.168.192.in-addr.arpa. (44) 01:24:04.949368 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.57360: 43921 NXDomain* 0/1/0 (111) 01:24:05.442850 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:05.545549 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs: Flags [P.], seq 3632955218:3632955354, ack 762815828, win 229, options [nop,nop,TS val 797511168 ecr 172937890], length 136: NFS request xid 351180291 132 getattr fh 0,1/53 01:24:05.545893 IP nfs02.my.dom.xyz.44899 > idmipa01.my.dom.xyz.domain: 42739+ PTR? 236.0.168.192.in-addr.arpa. (44) 01:24:05.546508 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.44899: 42739 NXDomain* 0/1/0 (111) 01:24:05.546662 IP nfs-c01.my.dom.xyz.nfs > 192.168.0.236.connendp: Flags [P.], seq 1:85, ack 136, win 361, options [nop,nop,TS val 172977954 ecr 797511168], length 84: NFS reply xid 351180291 reply ok 80 getattr NON 1 ids 0/83886080 sz 260232538 01:24:05.546890 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 85, win 229, options [nop,nop,TS val 797511169 ecr 172977954], length 0 01:24:06.443781 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:07.444765 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:08.445813 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:08.568058 ARP, Request who-has idmipa01.my.dom.xyz tell nfs02.my.dom.xyz, length 28 01:24:08.568511 ARP, Reply idmipa01.my.dom.xyz is-at 00:50:56:86:0d:fa (oui Unknown), length 46 01:24:08.898274 ARP, Request who-has 192.168.0.14 tell 192.168.0.2, length 46 01:24:08.898657 IP nfs02.my.dom.xyz.54390 > idmipa01.my.dom.xyz.domain: 42138+ PTR? 2.0.168.192.in-addr.arpa. (42) 01:24:08.899280 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.54390: 42138 NXDomain* 0/1/0 (109) 01:24:08.899574 ARP, Request who-has 192.168.0.14 tell 192.168.0.222, length 46 01:24:09.446886 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:10.447830 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:10.552020 ARP, Request who-has 192.168.0.236 tell nfs02.my.dom.xyz, length 28 01:24:10.552560 ARP, Reply 192.168.0.236 is-at 00:50:56:86:d7:4c (oui Unknown), length 46 01:24:10.553359 ARP, Request who-has nfs-c01.my.dom.xyz tell 192.168.0.236, length 46 01:24:10.553369 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:24:11.448844 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:12.449409 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:12.832432 ARP, Request who-has idmipa02.my.dom.xyz tell 192.168.0.1, length 46 01:24:12.832806 IP nfs02.my.dom.xyz.33863 > idmipa01.my.dom.xyz.domain: 37466+ PTR? 45.0.168.192.in-addr.arpa. (43) 01:24:12.833413 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.33863: 37466* 1/2/2 PTR idmipa02.my.dom.xyz. (146) 01:24:13.450394 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:14.040800 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.802: Flags [F.], seq 2201716585, ack 1255679435, win 227, options [nop,nop,TS val 172986448 ecr 167412300], length 0 01:24:14.041146 IP psql02.my.dom.xyz.802 > nfs-c01.my.dom.xyz.nfs: Flags [F.], seq 1, ack 1, win 229, options [nop,nop,TS val 167472301 ecr 172986448], length 0 01:24:14.041250 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.802: Flags [.], ack 2, win 227, options [nop,nop,TS val 172986449 ecr 167472301], length 0 01:24:14.046310 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.58177: Flags [F.], seq 4209302181, ack 3983186871, win 238, options [nop,nop,TS val 172986454 ecr 167412306], length 0 01:24:14.086690 IP psql02.my.dom.xyz.58177 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 1, win 229, options [nop,nop,TS val 167472347 ecr 172986454], length 0 01:24:14.451465 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:15.452474 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:16.453415 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:17.454417 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:18.455350 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:19.456342 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:20.457423 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:21.079896 ARP, Request who-has 192.168.0.223 tell 192.168.0.220, length 46 01:24:21.080332 ARP, Request who-has 192.168.0.220 tell 192.168.0.223, length 46 01:24:21.458358 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:22.459358 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:22.628588 ARP, Reply nfs02.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:24:23.460347 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:24.461359 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:25.462168 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:26.463188 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:27.464196 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:28.465215 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:29.466269 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:30.467227 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:31.468185 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:32.469246 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:33.470258 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:34.471274 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:35.472280 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:35.517800 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs: Flags [P.], seq 132:264, ack 85, win 614, options [nop,nop,TS val 272629248 ecr 172967862], length 132: NFS request xid 2560226412 128 getattr fh 0,1/53 01:24:35.518462 IP nfs-c01.my.dom.xyz.nfs > ovirt01.my.dom.xyz.843: Flags [P.], seq 85:169, ack 264, win 796, options [nop,nop,TS val 173007926 ecr 272629248], length 84: NFS reply xid 2560226412 reply ok 80 getattr NON 1 ids 0/50331648 sz 1518458202 01:24:35.518691 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 169, win 614, options [nop,nop,TS val 272629249 ecr 173007926], length 0 01:24:36.473289 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:37.474260 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:38.475265 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:39.476199 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:39.786771 ARP, Request who-has idmipa02.my.dom.xyz tell 192.168.0.222, length 46 01:24:40.477213 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:40.520055 ARP, Request who-has ovirt01.my.dom.xyz tell nfs02.my.dom.xyz, length 28 01:24:40.520485 ARP, Reply ovirt01.my.dom.xyz is-at 00:50:56:86:f7:7e (oui Unknown), length 46 01:24:40.525642 ARP, Request who-has nfs-c01.my.dom.xyz tell ovirt01.my.dom.xyz, length 46 01:24:40.525653 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:24:41.478228 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:42.478738 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:43.479744 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:44.480774 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:45.481793 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:45.609470 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs: Flags [P.], seq 136:272, ack 85, win 229, options [nop,nop,TS val 797551232 ecr 172977954], length 136: NFS request xid 367957507 132 getattr fh 0,1/53 01:24:45.610283 IP nfs-c01.my.dom.xyz.nfs > 192.168.0.236.connendp: Flags [P.], seq 85:169, ack 272, win 369, options [nop,nop,TS val 173018018 ecr 797551232], length 84: NFS reply xid 367957507 reply ok 80 getattr NON 1 ids 0/83886080 sz 260232538 01:24:45.610540 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 169, win 229, options [nop,nop,TS val 797551233 ecr 173018018], length 0 01:24:46.482737 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:47.483693 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:48.484672 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:49.485644 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:50.486707 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:50.615938 ARP, Request who-has 192.168.0.236 tell nfs02.my.dom.xyz, length 28 01:24:50.616486 ARP, Reply 192.168.0.236 is-at 00:50:56:86:d7:4c (oui Unknown), length 46 01:24:50.617258 ARP, Request who-has nfs-c01.my.dom.xyz tell 192.168.0.236, length 46 01:24:50.617273 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21 (oui Unknown), length 28 01:24:51.432203 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags [F.], seq 29, ack 45, win 227, options [nop,nop,TS val 173023840 ecr 167449690], length 0 01:24:51.432585 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags [F.], seq 45, ack 30, win 229, options [nop,nop,TS val 167509693 ecr 173023840], length 0 01:24:51.432617 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags [.], ack 46, win 227, options [nop,nop,TS val 173023840 ecr 167509693], length 0 01:24:51.437650 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999: Flags [F.], seq 25, ack 693, win 238, options [nop,nop,TS val 173023845 ecr 167449697], length 0 01:24:51.477546 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs: Flags [.], ack 26, win 229, options [nop,nop,TS val 167509738 ecr 173023845], length 0 01:24:51.487729 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:52.488796 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:53.489834 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:54.490898 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 01:24:55.492024 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2, Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20 ^C1161 packets captured 1162 packets received by filter 0 packets dropped by kernel [root@nfs02 log]#

5 34

Re: Nfs-ganesha rgw config for multi tenancy rgw users
by Matt Benjamin 30 Apr '18

30 Apr '18

Hi Marc, Yes, you're correct that librgw--the library interface consumed by nfs-ganesha's RGW fsal--is connecting natively to the cluster as an embedded radosgw instance, communicating with the cluster using rados. It's not using http/proxying. Matt On Mon, Apr 30, 2018 at 11:29 AM, Marc Roos <M.Roos(a)f1-outsourcing.eu> wrote: > > I stopped the rgw3 instance that is running on the same server (I am not > sure anymore if I can run radosgw and nfs-ganesha with the same > client.rgw3), started ganesha with this config. I am correct to > understand that the ganesha-rgw is not using the http(s) interface and > directly connects to ceph with librados (because that is why we have the > RGW config section)? > > RGW { > cluster = "ceph"; > name = "client.rgw3"; > ceph_conf = "/etc/ceph/ceph.conf"; > } > > EXPORT { > Export_ID=301; > Path = "test:test3"; > Pseudo = "/rgwtester"; > > FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id > ="xxxxxx"; Secret_Access_Key = "xxxxxx"; } > Disable_ACL = TRUE; > CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } > } > > > Logs of ganesha related to test3 bucket > > [@]# tail -1000 /var/log/messages | grep ganesh | grep test3 > Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release > :FSAL :INFO :Releasing RGW export for test:test3 > Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release > :FSAL :INFO :Releasing RGW export for test:test3 > Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common > :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path > (test:test3) and tag ((null)) perms (options=03303002 , > , , , , , , > ) > Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common > :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path > (test:test3) and tag ((null)) perms (options=03303002 , > , , , , , , > ) > > > [@]# showmount -e nfs > Export list for nfs: > /ram 192.168.x.0 > /test 192.168.x.0 > /rgwtester 192.168.x.0 > > [@ mnt]# mount nfs:/rgwtester /mnt/nfstest > > [@ mnt]# mount > proc on /proc type proc (rw) > sysfs on /sys type sysfs (rw) > devpts on /dev/pts type devpts (rw,gid=5,mode=620) > tmpfs on /dev/shm type tmpfs (rw) > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) > ... > nfs:/test on /home/users/robbert/nfs-test type nfs4 > (rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,naml > en=255,hard,addr= > nfs:/rgwtester on /mnt/nfstest type nfs > (rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x) > > [@ mnt]# ls -l /mnt/nfstest/ > ls: reading directory /mnt/nfstest/: Remote I/O error > total 0 > > > > -----Original Message----- > From: Daniel Gryniewicz [mailto:dang@redhat.com] > Sent: maandag 30 april 2018 20:06 > To: Marc Roos; support > Cc: Matt Benjamin > Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi > tenancy rgw users > > Okay, so on consultation, we think that maybe you should be mounting > test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the > mail thread if you tried that; can you confirm? > > (I have no personal experience with multi-tenant, so I didn't recognize > it as such). > > In the mean time, we'll try to reproduce here and see what we can see. > > Daniel > > On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote: >> All of that looks fine, I think. I'm out of my depth here, so I think > >> Matt needs to chime in here. >> >> Matt? >> >> On 04/30/2018 04:06 AM, Marc Roos wrote: >>> Could it not be, because of a configuration issue in ganesha.conf or >>> the version I am using? I can hardly imagine that I am the only one >>> running a multitenant rgw config with ganesha-rgw. >>> >>> nfs-ganesha-xfs-2.5.5-.el7.x86_64 >>> nfs-ganesha-2.5.5-.el7.x86_64 >>> nfs-ganesha-rgw-2.5.5-.el7.x86_64 >>> nfs-ganesha-ceph-2.5.5-.el7.x86_64 >>> nfs-ganesha-vfs-2.5.5-.el7.x86_64 >>> nfs-ganesha-mem-2.5.5-.el7.x86_64 >>> >>> >>> >>> >>> -----Original Message----- >>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>> Sent: maandag 30 april 2018 15:15 >>> To: support(a)lists.nfs-ganesha.org >>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi >>> tenancy rgw users >>> >>> And test$tester1 is the user ID you created, that is this gives > output: >>> >>> radosgw-admin user info --uid="test$tester1" >>> >>> (I'm pounding on this because it really seems that the UID doesn't >>> have permissions. If it does, I think there's some really strange >>> bug here that's going to be hard to track down.) >>> >>> Daniel >>> >>> >>> On 04/30/2018 03:25 AM, Marc Roos wrote: >>>> Yes same access key and secret access key ( no uid is used ) >>>> >>>> -----Original Message----- >>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>> Sent: maandag 30 april 2018 14:40 >>>> To: Marc Roos; support >>>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>>> multi tenancy rgw users >>>> >>>> I mean does the s3cmd (or cyberduck) use the exact same credentials >>>> as >>> >>>> are specified in the ganesha.conf? >>>> >>>> Daniel >>>> >>>> On 04/26/2018 10:09 AM, Marc Roos wrote: >>>>> >>>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are >>>>> you refering to something else? >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>>> Sent: donderdag 26 april 2018 15:38 >>>>> To: support(a)lists.nfs-ganesha.org >>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi > >>>>> tenancy rgw users >>>>> >>>>> Does test$tester1 have permissions to access RGW? Mounting / >>>>> should see all the buckets, so it's likely a permission issue. >>>>> >>>>> Daniel >>>>> >>>>> On 04/26/2018 03:56 AM, Marc Roos wrote: >>>>>> >>>>>> I have the same as with / path, I can do the mount, but it is just > >>>>>> empty >>>>>> >>>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/ >>>>>> ls: reading directory nfstest/: Remote I/O error total 0 >>>>>> >>>>>> >>>>>> >>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>>> export_commit_common >>>> >>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path >>>>>> (test3) and tag ((null)) perms (options=03303002 , >>> , >>>>>> , , , , , >>>>>> ) >>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>>> export_commit_common >>>> >>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path >>>>>> (test3) and tag ((null)) perms (options=03303002 , >>> , >>>>>> , , , , , >>>>>> ) >>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>>> export_commit_common >>>> >>>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 >>>>>> c01 >>>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO >>>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01 >>>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO >>>>>> :Export >>>> >>>>>> 0 (/) successfully created Apr 26 14:56:15 c01 > nfs-ganesha[3981596]: >>>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully > >>>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] >>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 >>>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] >>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 >>>>>> ngroups=0 >>>>>> >>>>>> -----Original Message----- >>>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>>>> Sent: donderdag 26 april 2018 14:47 >>>>>> To: support(a)lists.nfs-ganesha.org >>>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>>>>> multi tenancy rgw users >>>>>> >>>>>> I think it should be: >>>>>> >>>>>> Path="test3" >>>>>> >>>>>> so without the initial '/'. Buckets aren't paths, we're just >>>>>> abusing >>>> >>>>>> the existing Ganesha config to select them. >>>>>> >>>>>> Other than that, the client entry should match the equivalent >>>>>> entry in >>>>> >>>>>> your ceph.conf. >>>>>> >>>>>> Daniel >>>>>> >>>>>> On 04/26/2018 01:52 AM, Marc Roos wrote: >>>>>>> >>>>>>> >>>>>>> I have problems exporting a bucket that really does exist. I have > >>>>>>> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha >>>>>>> fails >>>>> >>>>>>> to start with message ExportId=301 Path=/test:test3 >>>>>>> FSAL_ERROR=(Invalid object >>>>>>> type,0) >>>>>>> >>>>>>> If I use path=/ I can mount something but that is a empty export, > >>>>>>> but >>>>> >>>>>>> cannot put files there. >>>>>>> >>>>>>> I have this in ganesha (client.rwg3 works with civetweb), should >>>>>>> be >>> >>>>>>> sufficient not? >>>>>>> >>>>>>> RGW { >>>>>>> cluster = "ceph"; >>>>>>> name = "client.rgw3"; >>>>>>> ceph_conf = "/etc/ceph/ceph.conf"; >>>>>>> # for vstart cluster, name = "client.admin" >>>>>>> #init_args = "-d --debug-rgw=16"; } >>>>>>> >>>>>>> EXPORT { >>>>>>> Export_ID=301; >>>>>>> Path = "/test3"; >>>>>>> Pseudo = "/rgwtester"; >>>>>>> >>>>>>> FSAL { Name = RGW; User_Id = "test$tester1"; >>>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key = "sameass3"; } >>>>>>> Disable_ACL = FALSE; >>>>>>> CLIENT { Clients = 192.168.x.0/24; } } >>>>>>> >>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls >>>>>>> 2018-01-31 21:48 s3://test >>>>>>> 2018-02-01 11:44 s3://test2 >>>>>>> 2018-02-02 17:10 s3://test3 >>>>>>> >>>>>>> >>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3 >>>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt >>>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg >>>>>>> >>>>>>> [@ ganesha]# rpm -qa | grep ganesh >>>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64 >>>>>>> >>>>>>> On CentOS7 >>>>>>> _______________________________________________ >>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>>> unsubscribe >>>> >>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>>> >>>>>> _______________________________________________ >>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>> unsubscribe >>> >>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>> >>>>>> _______________________________________________ >>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>> unsubscribe >>> >>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>> >>>>> _______________________________________________ >>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe > >>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>> >>> _______________________________________________ >>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe >>> send an email to support-leave(a)lists.nfs-ganesha.org >>> >>> >> > > > -- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309

1 0

Gluster + NFS-Ganesha Failover
by Philip Fuchs 23 Apr '18

23 Apr '18

Hello All, I am trying to setup a three way replicated Gluster Storage which is exported by NFS Ganesha. This 3 node Ganesha cluster is managed by pacemaker and corosync. I want to use this cluster as a backend for several different web-based applications as well as storage for mailboxes. The cluster is working well but after triggering the failover by stopping the ganesha service on one node, the ganesha services on the other two nodes are also stopping after a couple of minutes, bringing down the whole cluster. Setup: 3 CentOS Gluster Servers with Ganesha 2 CentOS Clients Packages: glusterfs-libs-3.10.11-1.el7.x86_64 glusterfs-3.10.11-1.el7.x86_64 glusterfs-fuse-3.10.11-1.el7.x86_64 centos-release-gluster310-1.0-1.el7.centos.noarch glusterfs-api-3.10.11-1.el7.x86_64 python2-glusterfs-api-1.1-1.el7.noarch glusterfs-client-xlators-3.10.11-1.el7.x86_64 glusterfs-cli-3.10.11-1.el7.x86_64 glusterfs-server-3.10.11-1.el7.x86_64 python2-gluster-3.10.11-1.el7.x86_64 glusterfs-ganesha-3.10.11-1.el7.x86_64 nfs-ganesha-gluster-2.5.2-1.el7.x86_64 nfs-ganesha-2.5.2-1.el7.x86_64 Log messages: ==> ganesha/ganesha-gfapi.log <== [2018-04-16 16:37:52.777997] I [MSGID: 109066] [dht-rename.c:1610:dht_rename] 0-mail-vol-dht: renaming /tmp/1523896461.M716652P30764.rz.uni-augsburg.de (hash=mail-vol-replicate-0/cache=mail-vol-replicate-0) => /cur/1523896461.M716652P30764.rz.uni-augsburg.de,S=1441,W=1478:2,S (hash=mail-vol-replicate-0/cache=<nul>) [2018-04-16 16:37:52.788361] W [inode.c:1341:inode_parent] (-->/lib64/libgfapi.so.0(glfs_resolve_at+0x278) [0x7f900105b0b8] -->/lib64/libglusterfs.so.0(glusterfs_normalize_dentry+0x8e) [0x7f9000d84aee] -->/lib64/libglusterfs.so.0(inode_parent+0xda) [0x7f9000d8270a] ) 0-gfapi: inode not found [2018-04-16 16:37:52.788549] E [inode.c:2567:inode_parent_null_check] (-->/lib64/libgfapi.so.0(glfs_resolve_at+0x278) [0x7f900105b0b8] -->/lib64/libglusterfs.so.0(glusterfs_normalize_dentry+0xa0) [0x7f9000d84b00] -->/lib64/libglusterfs.so.0(+0x398c4) [0x7f9000d818c4] ) 0-inode: invalid argument: inode [Das Argument ist ungültig] ==> messages <== Apr 16 18:37:52 nfsc02 kernel: ganesha.nfsd[29880]: segfault at 0 ip 00007f9000d84b00 sp 00007f8f7a7d1650 error 4 in libglusterfs.so.0.0.1[7f9000d48000+f1000] Apr 16 18:37:52 nfsc02 systemd: nfs-ganesha.service: main process exited, code=killed, status=11/SEGV Apr 16 18:37:52 nfsc02 systemd: Unit nfs-ganesha.service entered failed state. Apr 16 18:37:52 nfsc02 systemd: nfs-ganesha.service failed. Backtrace with gdb: Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fafd6fcd700 (LWP 5171)] 0x00007fb0558eeb00 in glusterfs_normalize_dentry () from /lib64/libglusterfs.so.0 (gdb) bt #0 0x00007fb0558eeb00 in glusterfs_normalize_dentry () from /lib64/libglusterfs.so.0 #1 0x00007fb055bc50b8 in glfs_resolve_at () from /lib64/libgfapi.so.0 #2 0x00007fb055bc6bb4 in glfs_h_lookupat () from /lib64/libgfapi.so.0 #3 0x00007fb055fe375f in lookup () from /usr/lib64/ganesha/libfsalgluster.so #4 0x000055e36bb2362f in mdc_get_parent () #5 0x000055e36bb202a5 in mdcache_create_handle () #6 0x000055e36ba81422 in nfs4_mds_putfh () #7 0x000055e36ba81998 in nfs4_op_putfh () #8 0x000055e36ba7108f in nfs4_Compound () #9 0x000055e36ba604fc in nfs_rpc_execute () #10 0x000055e36ba61dad in worker_run () #11 0x000055e36baf72c9 in fridgethr_start_routine () #12 0x00007fb05914de25 in start_thread (arg=0x7fafd6fcd700) at pthread_create.c:308 #13 0x00007fb05881b34d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Anybody got an idea how to solve this problem? Thanks, Philip

2 1

[Support] Testing
by Frank Filz 16 Apr '18

16 Apr '18

Testing to see who has subscribed so far. Frank

1 0