2:33 a.m.
Hey All,
I have an external NFS cluster serviced by a VIP. The clients run
autofs configured via IPA to provide NFS home directories to client.
However, running into an issue on one of the clients and wondering if
anyone seen this message from a tcpdump of a simple mount session that's
preventing the mount:
psql02: mount nfs-c01:/n /m
Yields this message
ERR 20: Auth Rejected Credentials (client should begin new session)
and the mount attempt never exits and never mounts /m . nfs-c01 is a
VIP that's serviced by HAproxy / keepalived. nfs-c01 however has a
record in IPA Server, both forward and a reverse one. Using one of the
underlying hosts that services nfs-c01 works and mounts succeeds for
them. All VM hosts are clones of the same template.
I have autofs running as part of this IPA client setup and applied the
following fix as well:
https://access.redhat.com/solutions/3261981
/m is a test mount folder I'm using on this client to troubleshoot the
autofs mounting issue. So autofs is also running on the same hosts
where I'm trying this mount from.
Trying to trace the exact source of this error and not quite sure where
to look further.
idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively)
psql01/02 are the problem VM's. (192.168.0.108/124 )
nfs01/02 are the NFS hosts. (192.168.0.131/119 )
nfs-c01 192.168.0.80
This works fine on the other two VM hosts without any issue but I just
can't find any difference comparing all the configs and so looking for
suggestions to bounce off of.
--
Cheers,
Tom K.
-------------------------------------------------------------------------------------
Living on earth is expensive, but it includes a free trip around the sun.
Apr 15 23:29:54 psql02 kernel: INFO: task mount.nfs:1443 blocked for
more than 120 seconds.
Apr 15 23:29:54 psql02 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 15 23:29:54 psql02 kernel: mount.nfs D ffff880135ed8000 0
1443 1442 0x00000080
Apr 15 23:29:54 psql02 kernel: Call Trace:
Apr 15 23:29:54 psql02 kernel: [<ffffffff816ac7c9>]
schedule_preempt_disabled+0x29/0x70
Apr 15 23:29:54 psql02 kernel: [<ffffffff816aa5f7>]
__mutex_lock_slowpath+0xc7/0x1d0
Apr 15 23:29:54 psql02 kernel: [<ffffffff816a9a0f>] mutex_lock+0x1f/0x2f
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05ddd58>]
nfs4_discover_server_trunking+0x48/0x2e0 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05e6906>]
nfs4_init_client+0x126/0x300 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffff811e17d3>] ?
kmem_cache_alloc+0x193/0x1e0
Apr 15 23:29:54 psql02 kernel: [<ffffffffc0562526>] ?
__fscache_acquire_cookie+0x66/0x180 [fscache]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc0562526>] ?
__fscache_acquire_cookie+0x66/0x180 [fscache]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc0341b61>] ?
__rpc_init_priority_wait_queue+0x81/0xc0 [sunrpc]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc057c6d6>]
nfs_get_client+0x2c6/0x3e0 [nfs]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05e5de8>]
nfs4_set_client+0x98/0x130 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05e75de>]
nfs4_create_server+0x13e/0x3b0 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05de7ce>]
nfs4_remote_mount+0x2e/0x60 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffff81207549>] mount_fs+0x39/0x1b0
Apr 15 23:29:54 psql02 kernel: [<ffffffff811a7f25>] ?
__alloc_percpu+0x15/0x20
Apr 15 23:29:54 psql02 kernel: [<ffffffff81224177>]
vfs_kern_mount+0x67/0x110
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05de6f6>]
nfs_do_root_mount+0x86/0xc0 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05deaf4>]
nfs4_try_mount+0x44/0xc0 [nfsv4]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc057d627>] ?
get_nfs_version+0x27/0x90 [nfs]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc0589a95>]
nfs_fs_mount+0x4c5/0xd90 [nfs]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc058a9c0>] ?
nfs_clone_super+0x140/0x140 [nfs]
Apr 15 23:29:54 psql02 kernel: [<ffffffffc05888c0>] ?
param_set_portnr+0x70/0x70 [nfs]
Apr 15 23:29:54 psql02 kernel: [<ffffffff81207549>] mount_fs+0x39/0x1b0
Apr 15 23:29:54 psql02 kernel: [<ffffffff811a7f25>] ?
__alloc_percpu+0x15/0x20
Apr 15 23:29:54 psql02 kernel: [<ffffffff81224177>]
vfs_kern_mount+0x67/0x110
Apr 15 23:29:54 psql02 kernel: [<ffffffff81226683>] do_mount+0x233/0xaf0
Apr 15 23:29:54 psql02 kernel: [<ffffffff812272c6>] SyS_mount+0x96/0xf0
Apr 15 23:29:54 psql02 kernel: [<ffffffff816b89fd>]
system_call_fastpath+0x16/0x1b
Apr 15 23:29:54 psql02 kernel: [<ffffffff816b889d>] ?
system_call_after_swapgs+0xca/0x214
Message from syslogd@psql01 at Apr 17 03:08:31 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s!
[mount.nfs:1606]
Linux psql02.nix.my.dom 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7
19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@nfs02 log]# tcpdump -i eth0|grep -v "192.168.0.76"|grep -v
NLB|grep -v nfs01|grep -v netbios|grep -v NBT
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:23:42.183158 IP nfs02.my.dom.xyz.60807 > idmipa01.my.dom.xyz.domain:
23358+ PTR? 76.0.168.192.in-addr.arpa. (43)
01:23:42.183884 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.60807:
23358 NXDomain* 0/1/0 (110)
01:23:42.184090 IP nfs02.my.dom.xyz.59911 > idmipa01.my.dom.xyz.domain:
29059+ PTR? 119.0.168.192.in-addr.arpa. (44)
01:23:42.184601 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.59911:
29059* 1/2/2 PTR nfs02.my.dom.xyz. (153)
01:23:42.184827 IP nfs02.my.dom.xyz.49329 > idmipa01.my.dom.xyz.domain:
50753+ PTR? 44.0.168.192.in-addr.arpa. (43)
01:23:42.185122 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.49329:
50753* 1/2/2 PTR idmipa01.my.dom.xyz. (146)
01:23:42.250263 IP nfs02.my.dom.xyz.49035 > idmipa01.my.dom.xyz.domain:
17264+ PTR? 255.0.168.192.in-addr.arpa. (44)
01:23:42.250983 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.49035:
17264 NXDomain* 0/1/0 (111)
01:23:42.257700 IP nfs02.my.dom.xyz.51938 > idmipa01.my.dom.xyz.domain:
51451+ PTR? 131.0.168.192.in-addr.arpa. (44)
01:23:42.360669 IP nfs02.my.dom.xyz.46447 > idmipa01.my.dom.xyz.domain:
12552+ PTR? 224.0.168.192.in-addr.arpa. (44)
01:23:42.361247 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.46447:
12552 NXDomain* 0/1/0 (111)
01:23:42.361434 IP nfs02.my.dom.xyz.37305 > idmipa01.my.dom.xyz.domain:
34850+ PTR? 223.0.168.192.in-addr.arpa. (44)
01:23:42.361766 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.37305:
34850 NXDomain* 0/1/0 (111)
01:23:42.420742 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:42.421026 IP nfs02.my.dom.xyz.58510 > idmipa01.my.dom.xyz.domain:
7249+ PTR? 18.0.0.224.in-addr.arpa. (41)
01:23:42.421745 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.58510:
7249 1/13/0 PTR vrrp.mcast.net. (277)
01:23:42.751583 IP nfs02.my.dom.xyz.43409 > idmipa01.my.dom.xyz.domain:
29327+ PTR? 222.0.168.192.in-addr.arpa. (44)
01:23:42.752250 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.43409:
29327 NXDomain* 0/1/0 (111)
01:23:43.421723 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:44.422648 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:45.423492 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:46.424492 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:47.188951 ARP, Request who-has nfs02.my.dom.xyz tell
idmipa01.my.dom.xyz, length 46
01:23:47.188966 ARP, Reply nfs02.my.dom.xyz is-at 00:50:56:86:2d:21 (oui
Unknown), length 28
01:23:47.248948 ARP, Request who-has 192.168.0.103 tell 192.168.0.222,
length 46
01:23:47.249297 IP nfs02.my.dom.xyz.50518 > idmipa01.my.dom.xyz.domain:
46693+ PTR? 103.0.168.192.in-addr.arpa. (44)
01:23:47.250150 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.50518:
46693 NXDomain* 0/1/0 (111)
01:23:47.425440 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:48.426303 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:49.427153 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:50.428133 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:51.428574 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags
[S], seq 1812770089, win 29200, options [mss 1460,sackOK,TS val
167449689 ecr 0,nop,wscale 7], length 0
01:23:51.428634 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags
[S.], seq 2612074554, ack 1812770090, win 28960, options [mss
1460,sackOK,TS val 172963836 ecr 167449689,nop,wscale 7], length 0
01:23:51.428787 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags
[.], ack 1, win 229, options [nop,nop,TS val 167449689 ecr 172963836],
length 0
01:23:51.428838 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags
[P.], seq 1:45, ack 1, win 229, options [nop,nop,TS val 167449689 ecr
172963836], length 44: NFS request xid 2544096005 40 null
01:23:51.428859 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags
[.], ack 45, win 227, options [nop,nop,TS val 172963836 ecr 167449689],
length 0
01:23:51.429003 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:51.429079 IP nfs02.my.dom.xyz.52213 > idmipa01.my.dom.xyz.domain:
17880+ PTR? 80.0.168.192.in-addr.arpa. (43)
01:23:51.429514 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.52213:
17880* 1/2/2 PTR nfs-c01.my.dom.xyz. (154)
01:23:51.429748 IP nfs02.my.dom.xyz.58966 > idmipa01.my.dom.xyz.domain:
50455+ PTR? 124.0.168.192.in-addr.arpa. (44)
01:23:51.430092 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.58966:
50455* 1/2/2 PTR psql02.my.dom.xyz. (154)
01:23:51.430129 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags
[P.], seq 1:29, ack 45, win 227, options [nop,nop,TS val 172963838 ecr
167449689], length 28: NFS reply xid 2544096005 reply ok 24 null
01:23:51.430247 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags
[.], ack 29, win 229, options [nop,nop,TS val 167449690 ecr 172963838],
length 0
01:23:51.433079 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs:
Flags [S], seq 3882292069, win 29200, options [mss 1460,sackOK,TS val
167449693 ecr 0,nop,wscale 7], length 0
01:23:51.433124 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999:
Flags [S.], seq 876416044, ack 3882292070, win 28960, options [mss
1460,sackOK,TS val 172963841 ecr 167449693,nop,wscale 7], length 0
01:23:51.433214 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 1, win 229, options [nop,nop,TS val 167449693 ecr
172963841], length 0
01:23:51.435147 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs:
Flags [P.], seq 1:693, ack 1, win 229, options [nop,nop,TS val 167449695
ecr 172963841], length 692: NFS request xid 3844890745 688 null
01:23:51.435184 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999:
Flags [.], ack 693, win 238, options [nop,nop,TS val 172963843 ecr
167449695], length 0
01:23:51.436257 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999:
Flags [P.], seq 1:25, ack 693, win 238, options [nop,nop,TS val
172963844 ecr 167449695], length 24: NFS reply xid 3844890745 reply ERR
20: Auth Rejected Credentials (client should begin new session)
01:23:51.436374 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 25, win 229, options [nop,nop,TS val 167449697 ecr
172963844], length 0
01:23:51.483369 IP nfs02.my.dom.xyz.53527 > idmipa01.my.dom.xyz.domain:
62714+ PTR? 105.0.168.192.in-addr.arpa. (44)
01:23:51.484027 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.53527:
62714 NXDomain* 0/1/0 (111)
01:23:51.487612 IP nfs02.my.dom.xyz.41147 > idmipa01.my.dom.xyz.domain:
4106+ PTR? 100.0.168.192.in-addr.arpa. (44)
01:23:51.487992 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.41147:
4106 NXDomain* 0/1/0 (111)
01:23:52.429933 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:53.430801 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:54.432246 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:55.433173 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:55.453824 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs:
Flags [P.], seq 2034285336:2034285468, ack 3501594280, win 614, options
[nop,nop,TS val 272589184 ecr 172927798], length 132: NFS request xid
2543449196 128 getattr fh 0,1/53
01:23:55.454235 IP nfs02.my.dom.xyz.34495 > idmipa01.my.dom.xyz.domain:
10353+ PTR? 145.0.168.192.in-addr.arpa. (44)
01:23:55.454456 IP nfs-c01.my.dom.xyz.nfs > ovirt01.my.dom.xyz.843:
Flags [P.], seq 1:85, ack 132, win 788, options [nop,nop,TS val
172967862 ecr 272589184], length 84: NFS reply xid 2543449196 reply ok
80 getattr NON 1 ids 0/50331648 sz 1518458202
01:23:55.454669 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 85, win 614, options [nop,nop,TS val 272589185 ecr
172967862], length 0
01:23:55.455038 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.34495:
10353* 1/2/2 PTR ovirt01.my.dom.xyz. (155)
01:23:56.434163 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:56.443495 ARP, Request who-has nfs-c01.my.dom.xyz tell
psql02.my.dom.xyz, length 46
01:23:56.443577 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21
(oui Unknown), length 28
01:23:56.541140 IP nfs02.my.dom.xyz.51079 > idmipa01.my.dom.xyz.domain:
58928+ PTR? 14.0.168.192.in-addr.arpa. (43)
01:23:56.541904 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.51079:
58928 NXDomain* 0/1/0 (110)
01:23:57.435087 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:58.436052 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:23:59.437014 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:00.437885 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:00.456019 ARP, Request who-has ovirt01.my.dom.xyz tell
nfs02.my.dom.xyz, length 28
01:24:00.456439 ARP, Reply ovirt01.my.dom.xyz is-at 00:50:56:86:f7:7e
(oui Unknown), length 46
01:24:00.461740 ARP, Request who-has nfs-c01.my.dom.xyz tell
ovirt01.my.dom.xyz, length 46
01:24:00.461754 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21
(oui Unknown), length 28
01:24:01.233709 ARP, Request who-has 192.168.0.222 tell 192.168.0.221,
length 46
01:24:01.234102 IP nfs02.my.dom.xyz.56105 > idmipa01.my.dom.xyz.domain:
38431+ PTR? 221.0.168.192.in-addr.arpa. (44)
01:24:01.234620 ARP, Request who-has 192.168.0.221 tell 192.168.0.222,
length 46
01:24:01.234749 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.56105:
38431 NXDomain* 0/1/0 (111)
01:24:01.438897 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:02.439939 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:03.440864 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:03.552006 ARP, Request who-has 192.168.0.1 tell 192.168.0.105,
length 46
01:24:03.552336 IP nfs02.my.dom.xyz.46470 > idmipa01.my.dom.xyz.domain:
51908+ PTR? 1.0.168.192.in-addr.arpa. (42)
01:24:03.552821 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.46470:
51908 NXDomain* 0/1/0 (109)
01:24:03.563789 ARP, Request who-has 192.168.0.1 tell 192.168.0.100,
length 46
01:24:04.441825 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:04.947786 ARP, Request who-has idmipa01.my.dom.xyz tell
192.168.0.220, length 46
01:24:04.948254 IP nfs02.my.dom.xyz.57360 > idmipa01.my.dom.xyz.domain:
43921+ PTR? 220.0.168.192.in-addr.arpa. (44)
01:24:04.949368 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.57360:
43921 NXDomain* 0/1/0 (111)
01:24:05.442850 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:05.545549 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs:
Flags [P.], seq 3632955218:3632955354, ack 762815828, win 229, options
[nop,nop,TS val 797511168 ecr 172937890], length 136: NFS request xid
351180291 132 getattr fh 0,1/53
01:24:05.545893 IP nfs02.my.dom.xyz.44899 > idmipa01.my.dom.xyz.domain:
42739+ PTR? 236.0.168.192.in-addr.arpa. (44)
01:24:05.546508 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.44899:
42739 NXDomain* 0/1/0 (111)
01:24:05.546662 IP nfs-c01.my.dom.xyz.nfs > 192.168.0.236.connendp:
Flags [P.], seq 1:85, ack 136, win 361, options [nop,nop,TS val
172977954 ecr 797511168], length 84: NFS reply xid 351180291 reply ok 80
getattr NON 1 ids 0/83886080 sz 260232538
01:24:05.546890 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 85, win 229, options [nop,nop,TS val 797511169 ecr
172977954], length 0
01:24:06.443781 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:07.444765 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:08.445813 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:08.568058 ARP, Request who-has idmipa01.my.dom.xyz tell
nfs02.my.dom.xyz, length 28
01:24:08.568511 ARP, Reply idmipa01.my.dom.xyz is-at 00:50:56:86:0d:fa
(oui Unknown), length 46
01:24:08.898274 ARP, Request who-has 192.168.0.14 tell 192.168.0.2,
length 46
01:24:08.898657 IP nfs02.my.dom.xyz.54390 > idmipa01.my.dom.xyz.domain:
42138+ PTR? 2.0.168.192.in-addr.arpa. (42)
01:24:08.899280 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.54390:
42138 NXDomain* 0/1/0 (109)
01:24:08.899574 ARP, Request who-has 192.168.0.14 tell 192.168.0.222,
length 46
01:24:09.446886 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:10.447830 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:10.552020 ARP, Request who-has 192.168.0.236 tell
nfs02.my.dom.xyz, length 28
01:24:10.552560 ARP, Reply 192.168.0.236 is-at 00:50:56:86:d7:4c (oui
Unknown), length 46
01:24:10.553359 ARP, Request who-has nfs-c01.my.dom.xyz tell
192.168.0.236, length 46
01:24:10.553369 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21
(oui Unknown), length 28
01:24:11.448844 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:12.449409 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:12.832432 ARP, Request who-has idmipa02.my.dom.xyz tell
192.168.0.1, length 46
01:24:12.832806 IP nfs02.my.dom.xyz.33863 > idmipa01.my.dom.xyz.domain:
37466+ PTR? 45.0.168.192.in-addr.arpa. (43)
01:24:12.833413 IP idmipa01.my.dom.xyz.domain > nfs02.my.dom.xyz.33863:
37466* 1/2/2 PTR idmipa02.my.dom.xyz. (146)
01:24:13.450394 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:14.040800 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.802: Flags
[F.], seq 2201716585, ack 1255679435, win 227, options [nop,nop,TS val
172986448 ecr 167412300], length 0
01:24:14.041146 IP psql02.my.dom.xyz.802 > nfs-c01.my.dom.xyz.nfs: Flags
[F.], seq 1, ack 1, win 229, options [nop,nop,TS val 167472301 ecr
172986448], length 0
01:24:14.041250 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.802: Flags
[.], ack 2, win 227, options [nop,nop,TS val 172986449 ecr 167472301],
length 0
01:24:14.046310 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.58177:
Flags [F.], seq 4209302181, ack 3983186871, win 238, options [nop,nop,TS
val 172986454 ecr 167412306], length 0
01:24:14.086690 IP psql02.my.dom.xyz.58177 > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 1, win 229, options [nop,nop,TS val 167472347 ecr
172986454], length 0
01:24:14.451465 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:15.452474 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:16.453415 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:17.454417 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:18.455350 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:19.456342 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:20.457423 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:21.079896 ARP, Request who-has 192.168.0.223 tell 192.168.0.220,
length 46
01:24:21.080332 ARP, Request who-has 192.168.0.220 tell 192.168.0.223,
length 46
01:24:21.458358 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:22.459358 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:22.628588 ARP, Reply nfs02.my.dom.xyz is-at 00:50:56:86:2d:21 (oui
Unknown), length 28
01:24:23.460347 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:24.461359 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:25.462168 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:26.463188 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:27.464196 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:28.465215 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:29.466269 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:30.467227 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:31.468185 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:32.469246 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:33.470258 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:34.471274 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:35.472280 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:35.517800 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs:
Flags [P.], seq 132:264, ack 85, win 614, options [nop,nop,TS val
272629248 ecr 172967862], length 132: NFS request xid 2560226412 128
getattr fh 0,1/53
01:24:35.518462 IP nfs-c01.my.dom.xyz.nfs > ovirt01.my.dom.xyz.843:
Flags [P.], seq 85:169, ack 264, win 796, options [nop,nop,TS val
173007926 ecr 272629248], length 84: NFS reply xid 2560226412 reply ok
80 getattr NON 1 ids 0/50331648 sz 1518458202
01:24:35.518691 IP ovirt01.my.dom.xyz.843 > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 169, win 614, options [nop,nop,TS val 272629249 ecr
173007926], length 0
01:24:36.473289 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:37.474260 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:38.475265 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:39.476199 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:39.786771 ARP, Request who-has idmipa02.my.dom.xyz tell
192.168.0.222, length 46
01:24:40.477213 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:40.520055 ARP, Request who-has ovirt01.my.dom.xyz tell
nfs02.my.dom.xyz, length 28
01:24:40.520485 ARP, Reply ovirt01.my.dom.xyz is-at 00:50:56:86:f7:7e
(oui Unknown), length 46
01:24:40.525642 ARP, Request who-has nfs-c01.my.dom.xyz tell
ovirt01.my.dom.xyz, length 46
01:24:40.525653 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21
(oui Unknown), length 28
01:24:41.478228 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:42.478738 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:43.479744 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:44.480774 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:45.481793 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:45.609470 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs:
Flags [P.], seq 136:272, ack 85, win 229, options [nop,nop,TS val
797551232 ecr 172977954], length 136: NFS request xid 367957507 132
getattr fh 0,1/53
01:24:45.610283 IP nfs-c01.my.dom.xyz.nfs > 192.168.0.236.connendp:
Flags [P.], seq 85:169, ack 272, win 369, options [nop,nop,TS val
173018018 ecr 797551232], length 84: NFS reply xid 367957507 reply ok 80
getattr NON 1 ids 0/83886080 sz 260232538
01:24:45.610540 IP 192.168.0.236.connendp > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 169, win 229, options [nop,nop,TS val 797551233 ecr
173018018], length 0
01:24:46.482737 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:47.483693 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:48.484672 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:49.485644 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:50.486707 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:50.615938 ARP, Request who-has 192.168.0.236 tell
nfs02.my.dom.xyz, length 28
01:24:50.616486 ARP, Reply 192.168.0.236 is-at 00:50:56:86:d7:4c (oui
Unknown), length 46
01:24:50.617258 ARP, Request who-has nfs-c01.my.dom.xyz tell
192.168.0.236, length 46
01:24:50.617273 ARP, Reply nfs-c01.my.dom.xyz is-at 00:50:56:86:2d:21
(oui Unknown), length 28
01:24:51.432203 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags
[F.], seq 29, ack 45, win 227, options [nop,nop,TS val 173023840 ecr
167449690], length 0
01:24:51.432585 IP psql02.my.dom.xyz.885 > nfs-c01.my.dom.xyz.nfs: Flags
[F.], seq 45, ack 30, win 229, options [nop,nop,TS val 167509693 ecr
173023840], length 0
01:24:51.432617 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.885: Flags
[.], ack 46, win 227, options [nop,nop,TS val 173023840 ecr 167509693],
length 0
01:24:51.437650 IP nfs-c01.my.dom.xyz.nfs > psql02.my.dom.xyz.40999:
Flags [F.], seq 25, ack 693, win 238, options [nop,nop,TS val 173023845
ecr 167449697], length 0
01:24:51.477546 IP psql02.my.dom.xyz.40999 > nfs-c01.my.dom.xyz.nfs:
Flags [.], ack 26, win 229, options [nop,nop,TS val 167509738 ecr
173023845], length 0
01:24:51.487729 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:52.488796 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:53.489834 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:54.490898 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
01:24:55.492024 IP nfs02.my.dom.xyz > vrrp.mcast.net: VRRPv2,
Advertisement, vrid 51, prio 104, authtype none, intvl 1s, length 20
^C1161 packets captured
1162 packets received by filter
0 packets dropped by kernel
[root@nfs02 log]#
1:31 p.m.
New subject: [Nfs-ganesha-support] Re: [Support] ERR 20: Auth Rejected Credentials (client should begin new session)
Hi, Tom.
Sorry I missed this when it came in; the new list was not being filtered
properly.
Rejected Creds is only returned in a few places. It can mean that the
requested Auth type is completely unknown. This seems unlikely, as
Ganesha should understand all auth types used by the linux kernel client.
It can mean that AUTH_SHORT was used (which Ganesha doesn't support...).
It can mean that AUTH_GSS was used, but control procedure is unknown.
Again, unlikely.
It can mean that AUTH_GSS was used with GSS_INIT, but the security
context was invalid or incorrect.
Unfortunately, tcpdump didn't print enough for us to know. Could you
get a raw packet capture that we could load into wireshark? Or load it
yourself, and get a detailed trace of the failed packet?
Thanks,
Daniel
On 04/19/2018 03:33 AM, TomK wrote:
Hey All,
I have an external NFS cluster serviced by a VIP. The clients run
autofs configured via IPA to provide NFS home directories to client.
However, running into an issue on one of the clients and wondering if
anyone seen this message from a tcpdump of a simple mount session that's
preventing the mount:
psql02: mount nfs-c01:/n /m
Yields this message
ERR 20: Auth Rejected Credentials (client should begin new session)
and the mount attempt never exits and never mounts /m . nfs-c01 is a
VIP that's serviced by HAproxy / keepalived. nfs-c01 however has a
record in IPA Server, both forward and a reverse one. Using one of the
underlying hosts that services nfs-c01 works and mounts succeeds for
them. All VM hosts are clones of the same template.
I have autofs running as part of this IPA client setup and applied the
following fix as well:
https://access.redhat.com/solutions/3261981
/m is a test mount folder I'm using on this client to troubleshoot the
autofs mounting issue. So autofs is also running on the same hosts
where I'm trying this mount from.
Trying to trace the exact source of this error and not quite sure where
to look further.
idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively)
psql01/02 are the problem VM's. (192.168.0.108/124 )
nfs01/02 are the NFS hosts. (192.168.0.131/119 )
nfs-c01 192.168.0.80
This works fine on the other two VM hosts without any issue but I just
can't find any difference comparing all the configs and so looking for
suggestions to bounce off of.
11:40 p.m.
New subject: [Nfs-ganesha-support] Re: [Support] ERR 20: Auth Rejected Credentials (client should begin new session)
On 4/20/2018 2:31 PM, Daniel Gryniewicz wrote:
Thanks very much Daniel.
Attaching the tcpdump.
What I used:
1) tcpdump -w nfs-trace.dat -s 0
2) tcpdump -r nfs-trace.dat -nnvvveXXS
Cheers,
Tom
Hi, Tom.
Sorry I missed this when it came in; the new list was not being filtered
properly.
Rejected Creds is only returned in a few places. It can mean that the
requested Auth type is completely unknown. This seems unlikely, as
Ganesha should understand all auth types used by the linux kernel client.
It can mean that AUTH_SHORT was used (which Ganesha doesn't support...).
It can mean that AUTH_GSS was used, but control procedure is unknown.
Again, unlikely.
It can mean that AUTH_GSS was used with GSS_INIT, but the security
context was invalid or incorrect.
Unfortunately, tcpdump didn't print enough for us to know. Could you
get a raw packet capture that we could load into wireshark? Or load it
yourself, and get a detailed trace of the failed packet?
Thanks,
Daniel
On 04/19/2018 03:33 AM, TomK wrote:
> Hey All,
>
> I have an external NFS cluster serviced by a VIP. The clients run
> autofs configured via IPA to provide NFS home directories to client.
>
> However, running into an issue on one of the clients and wondering if
> anyone seen this message from a tcpdump of a simple mount session
> that's preventing the mount:
>
> psql02: mount nfs-c01:/n /m
>
> Yields this message
>
> ERR 20: Auth Rejected Credentials (client should begin new session)
>
> and the mount attempt never exits and never mounts /m . nfs-c01 is a
> VIP that's serviced by HAproxy / keepalived. nfs-c01 however has a
> record in IPA Server, both forward and a reverse one. Using one of
> the underlying hosts that services nfs-c01 works and mounts succeeds
> for them. All VM hosts are clones of the same template.
>
> I have autofs running as part of this IPA client setup and applied the
> following fix as well:
>
> https://access.redhat.com/solutions/3261981
>
> /m is a test mount folder I'm using on this client to troubleshoot the
> autofs mounting issue. So autofs is also running on the same hosts
> where I'm trying this mount from.
>
> Trying to trace the exact source of this error and not quite sure
> where to look further.
>
> idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively)
> psql01/02 are the problem VM's. (192.168.0.108/124 )
> nfs01/02 are the NFS hosts. (192.168.0.131/119 )
> nfs-c01 192.168.0.80
>
> This works fine on the other two VM hosts without any issue but I just
> can't find any difference comparing all the configs and so looking for
> suggestions to bounce off of.
>
--
Cheers,
Tom K.
-------------------------------------------------------------------------------------
Living on earth is expensive, but it includes a free trip around the sun.
8:04 a.m.
New subject: [Nfs-ganesha-support] Re: [Support] ERR 20: Auth Rejected Credentials (client should begin new session)
Thanks. I'll try to dig further into this, but for some strange reason,
a compound with <SEQUENCE, PUTROOTFH, GETFH, GETATTRS> is responded to
with <SEQUENCE, PUTROOTFH, SECINFO_NO_NAME>, which causes the session to
be destroyed. A quick look doesn't show me any way that can happen (nor
does a quick read of the spec indicate it's a legal response).
Frank, any ideas?
Daniel
On 04/22/2018 12:39 AM, TomK wrote:
On 4/20/2018 2:31 PM, Daniel Gryniewicz wrote:
Thanks very much Daniel.
Attaching the tcpdump.
What I used:
1) tcpdump -w nfs-trace.dat -s 0
2) tcpdump -r nfs-trace.dat -nnvvveXXS
Cheers,
Tom
> Hi, Tom.
>
> Sorry I missed this when it came in; the new list was not being
> filtered properly.
>
> Rejected Creds is only returned in a few places. It can mean that the
> requested Auth type is completely unknown. This seems unlikely, as
> Ganesha should understand all auth types used by the linux kernel client.
>
> It can mean that AUTH_SHORT was used (which Ganesha doesn't support...).
>
> It can mean that AUTH_GSS was used, but control procedure is unknown.
> Again, unlikely.
>
> It can mean that AUTH_GSS was used with GSS_INIT, but the security
> context was invalid or incorrect.
>
> Unfortunately, tcpdump didn't print enough for us to know. Could you
> get a raw packet capture that we could load into wireshark? Or load
> it yourself, and get a detailed trace of the failed packet?
>
> Thanks,
> Daniel
>
> On 04/19/2018 03:33 AM, TomK wrote:
>> Hey All,
>>
>> I have an external NFS cluster serviced by a VIP. The clients run
>> autofs configured via IPA to provide NFS home directories to client.
>>
>> However, running into an issue on one of the clients and wondering if
>> anyone seen this message from a tcpdump of a simple mount session
>> that's preventing the mount:
>>
>> psql02: mount nfs-c01:/n /m
>>
>> Yields this message
>>
>> ERR 20: Auth Rejected Credentials (client should begin new session)
>>
>> and the mount attempt never exits and never mounts /m . nfs-c01 is a
>> VIP that's serviced by HAproxy / keepalived. nfs-c01 however has a
>> record in IPA Server, both forward and a reverse one. Using one of
>> the underlying hosts that services nfs-c01 works and mounts succeeds
>> for them. All VM hosts are clones of the same template.
>>
>> I have autofs running as part of this IPA client setup and applied
>> the following fix as well:
>>
>> https://access.redhat.com/solutions/3261981
>>
>> /m is a test mount folder I'm using on this client to troubleshoot
>> the autofs mounting issue. So autofs is also running on the same
>> hosts where I'm trying this mount from.
>>
>> Trying to trace the exact source of this error and not quite sure
>> where to look further.
>>
>> idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively)
>> psql01/02 are the problem VM's. (192.168.0.108/124 )
>> nfs01/02 are the NFS hosts. (192.168.0.131/119 )
>> nfs-c01 192.168.0.80
>>
>> This works fine on the other two VM hosts without any issue but I
>> just can't find any difference comparing all the configs and so
>> looking for suggestions to bounce off of.
>>
>
8:57 a.m.
New subject: [Nfs-ganesha-support] Re: [Support] ERR 20: Auth Rejected Credentials (client should begin new session)
Thanks. I'll try to dig further into this, but for some strange
reason, a compound
with <SEQUENCE, PUTROOTFH, GETFH, GETATTRS> is responded to with
<SEQUENCE, PUTROOTFH, SECINFO_NO_NAME>, which causes the session to
be destroyed. A quick look doesn't show me any way that can happen (nor does
a quick read of the spec indicate it's a legal response).
If you filter on the session ID and slot 0, you will see 4 request response pairs:
196 24.035045 192.168.0.108 192.168.0.80 NFS 270 V4 Call
(Reply In 197) RECLAIM_COMPLETE
GSS Data, Ops(2): SEQUENCE, RECLAIM_COMPLETE
Operations (count: 2): SEQUENCE, RECLAIM_COMPLETE
Opcode: SEQUENCE (53)
seqid: 0x00000001
Opcode: RECLAIM_COMPLETE (58)
197 24.035695 192.168.0.80 192.168.0.108 NFS 226 V4 Reply
(Call In 196) RECLAIM_COMPLETE
GSS Data, Ops(2): SEQUENCE RECLAIM_COMPLETE
Operations (count: 2)
Opcode: SEQUENCE (53)
seqid: 0x00000001
Opcode: RECLAIM_COMPLETE (58)
198 24.035842 192.168.0.108 192.168.0.80 NFS 274 V4 Call
(Reply In 199) SECINFO_NO_NAME
GSS Data, Ops(3): SEQUENCE, PUTROOTFH, SECINFO_NO_NAME
Operations (count: 3): SEQUENCE, PUTROOTFH, SECINFO_NO_NAME
Opcode: SEQUENCE (53)
seqid: 0x00000002
Opcode: PUTROOTFH (24)
Opcode: SECINFO_NO_NAME (52)
199 24.041264 192.168.0.80 192.168.0.108 NFS 330 V4 Reply
(Call In 198) SECINFO_NO_NAME
GSS Data, Ops(3): SEQUENCE PUTROOTFH SECINFO_NO_NAME
Operations (count: 3)
Opcode: SEQUENCE (53)
seqid: 0x00000002
Opcode: PUTROOTFH (24)
Opcode: SECINFO_NO_NAME (52)
536 94.470108 192.168.0.108 192.168.0.80 NFS 226 V4 Call
(Reply In 550) PUTROOTFH | GETATTR
Network File System, Ops(4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Operations (count: 4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Opcode: SEQUENCE (53)
seqid: 0x00000003
Opcode: PUTROOTFH (24)
Opcode: GETFH (10)
Opcode: GETATTR (9)
537 94.470249 192.168.0.108 192.168.0.80 NFS 182 V4 Call
(Reply In 542) PUTROOTFH | GETATTR
Network File System, Ops(4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Operations (count: 4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Opcode: SEQUENCE (53)
seqid: 0x00000003
Opcode: PUTROOTFH (24)
Opcode: GETFH (10)
Opcode: GETATTR (9)
542 94.471061 192.168.0.80 192.168.0.108 NFS 262 V4 Reply
(Call In 537) SECINFO_NO_NAME
Network File System, Ops(3): SEQUENCE PUTROOTFH SECINFO_NO_NAME
Operations (count: 3)
Opcode: SEQUENCE (53)
seqid: 0x00000002
Opcode: PUTROOTFH (24)
Opcode: SECINFO_NO_NAME (52)
550 94.471464 192.168.0.80 192.168.0.108 NFS 474 V4 Reply
(Call In 536) PUTROOTFH | GETATTR
Network File System, Ops(4): SEQUENCE PUTROOTFH GETFH GETATTR
Operations (count: 4)
Opcode: SEQUENCE (53)
seqid: 0x00000003
Opcode: PUTROOTFH (24)
Opcode: GETFH (10)
Opcode: GETATTR (9)
Notice we get two copies of SEQUENCE, PUTROOTFH, GETFH, GETATTR at seqid 3 in fast
succession. And then two responses, one of which looks correct, one of which is a replay
of the response to SEQUENCE PUTROOTFH SECINFO_NO_NAME at seqid 2.
I think what is happening is Ganesha is starting to process both seqid requests in
parallel. One of them is identified as a replay and responded to as such. Unfortunately,
the one being processed as an original request has not actually finished, so what is in
the cache slot for the replay is the seqid 2 response...
Frank
Frank, any ideas?
Daniel
On 04/22/2018 12:39 AM, TomK wrote:
> On 4/20/2018 2:31 PM, Daniel Gryniewicz wrote:
> Thanks very much Daniel.
>
> Attaching the tcpdump.
>
> What I used:
>
> 1) tcpdump -w nfs-trace.dat -s 0
> 2) tcpdump -r nfs-trace.dat -nnvvveXXS
>
> Cheers,
> Tom
>
>> Hi, Tom.
>>
>> Sorry I missed this when it came in; the new list was not being
>> filtered properly.
>>
>> Rejected Creds is only returned in a few places. It can mean that
>> the requested Auth type is completely unknown. This seems unlikely,
>> as Ganesha should understand all auth types used by the linux kernel client.
>>
>> It can mean that AUTH_SHORT was used (which Ganesha doesn't support...).
>>
>> It can mean that AUTH_GSS was used, but control procedure is unknown.
>> Again, unlikely.
>>
>> It can mean that AUTH_GSS was used with GSS_INIT, but the security
>> context was invalid or incorrect.
>>
>> Unfortunately, tcpdump didn't print enough for us to know. Could you
>> get a raw packet capture that we could load into wireshark? Or load
>> it yourself, and get a detailed trace of the failed packet?
>>
>> Thanks,
>> Daniel
>>
>> On 04/19/2018 03:33 AM, TomK wrote:
>>> Hey All,
>>>
>>> I have an external NFS cluster serviced by a VIP. The clients run
>>> autofs configured via IPA to provide NFS home directories to client.
>>>
>>> However, running into an issue on one of the clients and wondering
>>> if anyone seen this message from a tcpdump of a simple mount session
>>> that's preventing the mount:
>>>
>>> psql02: mount nfs-c01:/n /m
>>>
>>> Yields this message
>>>
>>> ERR 20: Auth Rejected Credentials (client should begin new session)
>>>
>>> and the mount attempt never exits and never mounts /m . nfs-c01 is
>>> a VIP that's serviced by HAproxy / keepalived. nfs-c01 however has
>>> a record in IPA Server, both forward and a reverse one. Using one
>>> of the underlying hosts that services nfs-c01 works and mounts
>>> succeeds for them. All VM hosts are clones of the same template.
>>>
>>> I have autofs running as part of this IPA client setup and applied
>>> the following fix as well:
>>>
>>> https://access.redhat.com/solutions/3261981
>>>
>>> /m is a test mount folder I'm using on this client to troubleshoot
>>> the autofs mounting issue. So autofs is also running on the same
>>> hosts where I'm trying this mount from.
>>>
>>> Trying to trace the exact source of this error and not quite sure
>>> where to look further.
>>>
>>> idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively)
>>> psql01/02 are the problem VM's. (192.168.0.108/124 )
>>> nfs01/02 are the NFS hosts. (192.168.0.131/119 )
>>> nfs-c01 192.168.0.80
>>>
>>> This works fine on the other two VM hosts without any issue but I
>>> just can't find any difference comparing all the configs and so
>>> looking for suggestions to bounce off of.
>>>
>>
>
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe send an
email to support-leave(a)lists.nfs-ganesha.org
4:37 p.m.
New subject: [Nfs-ganesha-support] Re: [Support] ERR 20: Auth Rejected Credentials (client should begin new session)
Ok, I've got a patch that serializes use of a slot so a quickly replayed request will
block until the first instance of the request is completed and there is a response to
replay:
https://review.gerrithub.io/#/c/ffilz/nfs-ganesha/+/408756/
Thanks
Frank Filz
-----Original Message-----
From: Frank Filz [mailto:ffilzlnx@mindspring.com]
Sent: Monday, April 23, 2018 6:58 AM
To: dang(a)redhat.com; tomkcpr(a)mdevsys.com; support(a)lists.nfs-ganesha.org
Cc: nfs-ganesha-support(a)lists.sourceforge.net
Subject: [Nfs-ganesha-support] Re: [Support] ERR 20: Auth Rejected Credentials
(client should begin new session)
> Thanks. I'll try to dig further into this, but for some strange
> reason, a compound with <SEQUENCE, PUTROOTFH, GETFH, GETATTRS> is
> responded to with <SEQUENCE, PUTROOTFH, SECINFO_NO_NAME>, which
causes
> the session to be destroyed. A quick look doesn't show me any way
> that can happen (nor does a quick read of the spec indicate it's a legal
response).
If you filter on the session ID and slot 0, you will see 4 request response pairs:
196 24.035045 192.168.0.108 192.168.0.80 NFS 270 V4 Call
(Reply In 197) RECLAIM_COMPLETE
GSS Data, Ops(2): SEQUENCE, RECLAIM_COMPLETE
Operations (count: 2): SEQUENCE, RECLAIM_COMPLETE
Opcode: SEQUENCE (53)
seqid: 0x00000001
Opcode: RECLAIM_COMPLETE (58)
197 24.035695 192.168.0.80 192.168.0.108 NFS 226 V4 Reply
(Call In 196) RECLAIM_COMPLETE
GSS Data, Ops(2): SEQUENCE RECLAIM_COMPLETE
Operations (count: 2)
Opcode: SEQUENCE (53)
seqid: 0x00000001
Opcode: RECLAIM_COMPLETE (58)
198 24.035842 192.168.0.108 192.168.0.80 NFS 274 V4 Call
(Reply In 199) SECINFO_NO_NAME
GSS Data, Ops(3): SEQUENCE, PUTROOTFH, SECINFO_NO_NAME
Operations (count: 3): SEQUENCE, PUTROOTFH, SECINFO_NO_NAME
Opcode: SEQUENCE (53)
seqid: 0x00000002
Opcode: PUTROOTFH (24)
Opcode: SECINFO_NO_NAME (52)
199 24.041264 192.168.0.80 192.168.0.108 NFS 330 V4 Reply
(Call In 198) SECINFO_NO_NAME
GSS Data, Ops(3): SEQUENCE PUTROOTFH SECINFO_NO_NAME
Operations (count: 3)
Opcode: SEQUENCE (53)
seqid: 0x00000002
Opcode: PUTROOTFH (24)
Opcode: SECINFO_NO_NAME (52)
536 94.470108 192.168.0.108 192.168.0.80 NFS 226 V4 Call
(Reply In 550) PUTROOTFH | GETATTR
Network File System, Ops(4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Operations (count: 4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Opcode: SEQUENCE (53)
seqid: 0x00000003
Opcode: PUTROOTFH (24)
Opcode: GETFH (10)
Opcode: GETATTR (9)
537 94.470249 192.168.0.108 192.168.0.80 NFS 182 V4 Call
(Reply In 542) PUTROOTFH | GETATTR
Network File System, Ops(4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Operations (count: 4): SEQUENCE, PUTROOTFH, GETFH, GETATTR
Opcode: SEQUENCE (53)
seqid: 0x00000003
Opcode: PUTROOTFH (24)
Opcode: GETFH (10)
Opcode: GETATTR (9)
542 94.471061 192.168.0.80 192.168.0.108 NFS 262 V4 Reply
(Call In 537) SECINFO_NO_NAME
Network File System, Ops(3): SEQUENCE PUTROOTFH SECINFO_NO_NAME
Operations (count: 3)
Opcode: SEQUENCE (53)
seqid: 0x00000002
Opcode: PUTROOTFH (24)
Opcode: SECINFO_NO_NAME (52)
550 94.471464 192.168.0.80 192.168.0.108 NFS 474 V4 Reply
(Call In 536) PUTROOTFH | GETATTR
Network File System, Ops(4): SEQUENCE PUTROOTFH GETFH GETATTR
Operations (count: 4)
Opcode: SEQUENCE (53)
seqid: 0x00000003
Opcode: PUTROOTFH (24)
Opcode: GETFH (10)
Opcode: GETATTR (9)
Notice we get two copies of SEQUENCE, PUTROOTFH, GETFH, GETATTR at seqid
3 in fast succession. And then two responses, one of which looks correct, one of
which is a replay of the response to SEQUENCE PUTROOTFH
SECINFO_NO_NAME at seqid 2.
I think what is happening is Ganesha is starting to process both seqid requests in
parallel. One of them is identified as a replay and responded to as such.
Unfortunately, the one being processed as an original request has not actually
finished, so what is in the cache slot for the replay is the seqid 2 response...
Frank
> Frank, any ideas?
>
> Daniel
>
> On 04/22/2018 12:39 AM, TomK wrote:
> > On 4/20/2018 2:31 PM, Daniel Gryniewicz wrote:
> > Thanks very much Daniel.
> >
> > Attaching the tcpdump.
> >
> > What I used:
> >
> > 1) tcpdump -w nfs-trace.dat -s 0
> > 2) tcpdump -r nfs-trace.dat -nnvvveXXS
> >
> > Cheers,
> > Tom
> >
> >> Hi, Tom.
> >>
> >> Sorry I missed this when it came in; the new list was not being
> >> filtered properly.
> >>
> >> Rejected Creds is only returned in a few places. It can mean that
> >> the requested Auth type is completely unknown. This seems
> >> unlikely, as Ganesha should understand all auth types used by the linux
kernel client.
> >>
> >> It can mean that AUTH_SHORT was used (which Ganesha doesn't
support...).
> >>
> >> It can mean that AUTH_GSS was used, but control procedure is unknown.
> >> Again, unlikely.
> >>
> >> It can mean that AUTH_GSS was used with GSS_INIT, but the security
> >> context was invalid or incorrect.
> >>
> >> Unfortunately, tcpdump didn't print enough for us to know. Could
> >> you get a raw packet capture that we could load into wireshark? Or
> >> load it yourself, and get a detailed trace of the failed packet?
> >>
> >> Thanks,
> >> Daniel
> >>
> >> On 04/19/2018 03:33 AM, TomK wrote:
> >>> Hey All,
> >>>
> >>> I have an external NFS cluster serviced by a VIP. The clients run
> >>> autofs configured via IPA to provide NFS home directories to client.
> >>>
> >>> However, running into an issue on one of the clients and wondering
> >>> if anyone seen this message from a tcpdump of a simple mount
> >>> session that's preventing the mount:
> >>>
> >>> psql02: mount nfs-c01:/n /m
> >>>
> >>> Yields this message
> >>>
> >>> ERR 20: Auth Rejected Credentials (client should begin new
> >>> session)
> >>>
> >>> and the mount attempt never exits and never mounts /m . nfs-c01
> >>> is a VIP that's serviced by HAproxy / keepalived. nfs-c01 however
> >>> has a record in IPA Server, both forward and a reverse one. Using
> >>> one of the underlying hosts that services nfs-c01 works and mounts
> >>> succeeds for them. All VM hosts are clones of the same template.
> >>>
> >>> I have autofs running as part of this IPA client setup and applied
> >>> the following fix as well:
> >>>
> >>> https://access.redhat.com/solutions/3261981
> >>>
> >>> /m is a test mount folder I'm using on this client to troubleshoot
> >>> the autofs mounting issue. So autofs is also running on the same
> >>> hosts where I'm trying this mount from.
> >>>
> >>> Trying to trace the exact source of this error and not quite sure
> >>> where to look further.
> >>>
> >>> idmipa01/02 are the IPA servers. (192.168.0.44/45 respectively)
> >>> psql01/02 are the problem VM's. (192.168.0.108/124 )
> >>> nfs01/02 are the NFS hosts. (192.168.0.131/119 )
> >>> nfs-c01 192.168.0.80
> >>>
> >>> This works fine on the other two VM hosts without any issue but I
> >>> just can't find any difference comparing all the configs and so
> >>> looking for suggestions to bounce off of.
> >>>
> >>
> >
> >
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe send an
email to support-leave(a)lists.nfs-ganesha.org
12:52 a.m.
New subject: [Nfs-ganesha-support] Nfs-ganesha rgw config for multi tenancy rgw users
I have problems exporting a bucket that really does exist. I have tried
Path = "/test:test3"; Path = "/test3"; Results in ganesha fails to
start
with message ExportId=301 Path=/test:test3 FSAL_ERROR=(Invalid object
type,0)
If I use path=/ I can mount something but that is a empty export, but
cannot put files there.
I have this in ganesha (client.rwg3 works with civetweb), should be
sufficient not?
RGW {
cluster = "ceph";
name = "client.rgw3";
ceph_conf = "/etc/ceph/ceph.conf";
# for vstart cluster, name = "client.admin"
#init_args = "-d --debug-rgw=16"; }
EXPORT {
Export_ID=301;
Path = "/test3";
Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
="sameass3"; Secret_Access_Key = "sameass3"; }
Disable_ACL = FALSE;
CLIENT { Clients = 192.168.x.0/24; } }
[@~]$ s3cmd -c .s3cfg.tester1 ls
2018-01-31 21:48 s3://test
2018-02-01 11:44 s3://test2
2018-02-02 17:10 s3://test3
[@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
2018-02-02 17:13 10485760 s3://test3/10MB.txt
2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
[@ ganesha]# rpm -qa | grep ganesh
nfs-ganesha-xfs-2.5.5-.el7.x86_64
nfs-ganesha-2.5.5-.el7.x86_64
nfs-ganesha-rgw-2.5.5-.el7.x86_64
nfs-ganesha-ceph-2.5.5-.el7.x86_64
nfs-ganesha-vfs-2.5.5-.el7.x86_64
nfs-ganesha-mem-2.5.5-.el7.x86_64
On CentOS7
7:47 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I think it should be:
Path="test3"
so without the initial '/'. Buckets aren't paths, we're just abusing
the existing Ganesha config to select them.
Other than that, the client entry should match the equivalent entry in
your ceph.conf.
Daniel
On 04/26/2018 01:52 AM, Marc Roos wrote:
I have problems exporting a bucket that really does exist. I have tried
Path = "/test:test3"; Path = "/test3"; Results in ganesha fails to
start
with message ExportId=301 Path=/test:test3 FSAL_ERROR=(Invalid object
type,0)
If I use path=/ I can mount something but that is a empty export, but
cannot put files there.
I have this in ganesha (client.rwg3 works with civetweb), should be
sufficient not?
RGW {
cluster = "ceph";
name = "client.rgw3";
ceph_conf = "/etc/ceph/ceph.conf";
# for vstart cluster, name = "client.admin"
#init_args = "-d --debug-rgw=16"; }
EXPORT {
Export_ID=301;
Path = "/test3";
Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
="sameass3"; Secret_Access_Key = "sameass3"; }
Disable_ACL = FALSE;
CLIENT { Clients = 192.168.x.0/24; } }
[@~]$ s3cmd -c .s3cfg.tester1 ls
2018-01-31 21:48 s3://test
2018-02-01 11:44 s3://test2
2018-02-02 17:10 s3://test3
[@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
2018-02-02 17:13 10485760 s3://test3/10MB.txt
2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
[@ ganesha]# rpm -qa | grep ganesh
nfs-ganesha-xfs-2.5.5-.el7.x86_64
nfs-ganesha-2.5.5-.el7.x86_64
nfs-ganesha-rgw-2.5.5-.el7.x86_64
nfs-ganesha-ceph-2.5.5-.el7.x86_64
nfs-ganesha-vfs-2.5.5-.el7.x86_64
nfs-ganesha-mem-2.5.5-.el7.x86_64
On CentOS7
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
2:56 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I have the same as with / path, I can do the mount, but it is just empty
[@mnt]# mount nfs:/rgwtester nfstest/
[@mnt]# ls -l nfstest/
ls: reading directory nfstest/: Remote I/O error
total 0
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test3) and tag ((null)) perms (options=03303002 , ,
, , , , ,
)
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test3) and tag ((null)) perms (options=03303002 , ,
, , , , ,
)
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root
:CONFIG :INFO :Export 0 (/) successfully created
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root
:CONFIG :INFO :Export 0 (/) successfully created
Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0
Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 26 april 2018 14:47
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
I think it should be:
Path="test3"
so without the initial '/'. Buckets aren't paths, we're just abusing
the existing Ganesha config to select them.
Other than that, the client entry should match the equivalent entry in
your ceph.conf.
Daniel
On 04/26/2018 01:52 AM, Marc Roos wrote:
I have problems exporting a bucket that really does exist. I have
tried Path = "/test:test3"; Path = "/test3"; Results in ganesha fails
to start with message ExportId=301 Path=/test:test3
FSAL_ERROR=(Invalid object
type,0)
If I use path=/ I can mount something but that is a empty export, but
cannot put files there.
I have this in ganesha (client.rwg3 works with civetweb), should be
sufficient not?
RGW {
cluster = "ceph";
name = "client.rgw3";
ceph_conf = "/etc/ceph/ceph.conf";
# for vstart cluster, name = "client.admin"
#init_args = "-d --debug-rgw=16"; }
EXPORT {
Export_ID=301;
Path = "/test3";
Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
="sameass3"; Secret_Access_Key = "sameass3"; }
Disable_ACL = FALSE;
CLIENT { Clients = 192.168.x.0/24; } }
[@~]$ s3cmd -c .s3cfg.tester1 ls
2018-01-31 21:48 s3://test
2018-02-01 11:44 s3://test2
2018-02-02 17:10 s3://test3
[@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
2018-02-02 17:13 10485760 s3://test3/10MB.txt
2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
[@ ganesha]# rpm -qa | grep ganesh
nfs-ganesha-xfs-2.5.5-.el7.x86_64
nfs-ganesha-2.5.5-.el7.x86_64
nfs-ganesha-rgw-2.5.5-.el7.x86_64
nfs-ganesha-ceph-2.5.5-.el7.x86_64
nfs-ganesha-vfs-2.5.5-.el7.x86_64
nfs-ganesha-mem-2.5.5-.el7.x86_64
On CentOS7
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
8:37 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Does test$tester1 have permissions to access RGW? Mounting / should see
all the buckets, so it's likely a permission issue.
Daniel
On 04/26/2018 03:56 AM, Marc Roos wrote:
I have the same as with / path, I can do the mount, but it is just empty
[@mnt]# mount nfs:/rgwtester nfstest/
[@mnt]# ls -l nfstest/
ls: reading directory nfstest/: Remote I/O error
total 0
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test3) and tag ((null)) perms (options=03303002 , ,
, , , , ,
)
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test3) and tag ((null)) perms (options=03303002 , ,
, , , , ,
)
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root
:CONFIG :INFO :Export 0 (/) successfully created
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root
:CONFIG :INFO :Export 0 (/) successfully created
Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0
Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 26 april 2018 14:47
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
I think it should be:
Path="test3"
so without the initial '/'. Buckets aren't paths, we're just abusing
the existing Ganesha config to select them.
Other than that, the client entry should match the equivalent entry in
your ceph.conf.
Daniel
On 04/26/2018 01:52 AM, Marc Roos wrote:
>
>
>
> I have problems exporting a bucket that really does exist. I have
> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha
fails
> to start with message ExportId=301 Path=/test:test3
> FSAL_ERROR=(Invalid object
> type,0)
>
> If I use path=/ I can mount something but that is a empty export, but
> cannot put files there.
>
> I have this in ganesha (client.rwg3 works with civetweb), should be
> sufficient not?
>
> RGW {
> cluster = "ceph";
> name = "client.rgw3";
> ceph_conf = "/etc/ceph/ceph.conf";
> # for vstart cluster, name = "client.admin"
> #init_args = "-d --debug-rgw=16"; }
>
> EXPORT {
> Export_ID=301;
> Path = "/test3";
> Pseudo = "/rgwtester";
>
> FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
> ="sameass3"; Secret_Access_Key = "sameass3"; }
> Disable_ACL = FALSE;
> CLIENT { Clients = 192.168.x.0/24; } }
>
> [@~]$ s3cmd -c .s3cfg.tester1 ls
> 2018-01-31 21:48 s3://test
> 2018-02-01 11:44 s3://test2
> 2018-02-02 17:10 s3://test3
>
>
> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>
> [@ ganesha]# rpm -qa | grep ganesh
> nfs-ganesha-xfs-2.5.5-.el7.x86_64
> nfs-ganesha-2.5.5-.el7.x86_64
> nfs-ganesha-rgw-2.5.5-.el7.x86_64
> nfs-ganesha-ceph-2.5.5-.el7.x86_64
> nfs-ganesha-vfs-2.5.5-.el7.x86_64
> nfs-ganesha-mem-2.5.5-.el7.x86_64
>
> On CentOS7
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
9:09 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Yes, I can access the bucket via the s3cmd and cyberduck, or are you
refering to something else?
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 26 april 2018 15:38
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Does test$tester1 have permissions to access RGW? Mounting / should see
all the buckets, so it's likely a permission issue.
Daniel
On 04/26/2018 03:56 AM, Marc Roos wrote:
I have the same as with / path, I can do the mount, but it is just
empty
[@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
ls: reading directory nfstest/: Remote I/O error total 0
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test3) and tag ((null)) perms (options=03303002 , ,
, , , , ,
)
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test3) and tag ((null)) perms (options=03303002 , ,
, , , , ,
)
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
:Export 301 has 1 defined clients Apr 26 14:56:15 c01
nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export
0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
[main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
ngroups=0
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 26 april 2018 14:47
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
I think it should be:
Path="test3"
so without the initial '/'. Buckets aren't paths, we're just abusing
the existing Ganesha config to select them.
Other than that, the client entry should match the equivalent entry in
your ceph.conf.
Daniel
On 04/26/2018 01:52 AM, Marc Roos wrote:
>
>
>
> I have problems exporting a bucket that really does exist. I have
> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha
fails
> to start with message ExportId=301 Path=/test:test3
> FSAL_ERROR=(Invalid object
> type,0)
>
> If I use path=/ I can mount something but that is a empty export, but
> cannot put files there.
>
> I have this in ganesha (client.rwg3 works with civetweb), should be
> sufficient not?
>
> RGW {
> cluster = "ceph";
> name = "client.rgw3";
> ceph_conf = "/etc/ceph/ceph.conf";
> # for vstart cluster, name = "client.admin"
> #init_args = "-d --debug-rgw=16"; }
>
> EXPORT {
> Export_ID=301;
> Path = "/test3";
> Pseudo = "/rgwtester";
>
> FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
> ="sameass3"; Secret_Access_Key = "sameass3"; }
> Disable_ACL = FALSE;
> CLIENT { Clients = 192.168.x.0/24; } }
>
> [@~]$ s3cmd -c .s3cfg.tester1 ls
> 2018-01-31 21:48 s3://test
> 2018-02-01 11:44 s3://test2
> 2018-02-02 17:10 s3://test3
>
>
> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>
> [@ ganesha]# rpm -qa | grep ganesh
> nfs-ganesha-xfs-2.5.5-.el7.x86_64
> nfs-ganesha-2.5.5-.el7.x86_64
> nfs-ganesha-rgw-2.5.5-.el7.x86_64
> nfs-ganesha-ceph-2.5.5-.el7.x86_64
> nfs-ganesha-vfs-2.5.5-.el7.x86_64
> nfs-ganesha-mem-2.5.5-.el7.x86_64
>
> On CentOS7
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
7:40 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
Yes, I can access the bucket via the s3cmd and cyberduck, or are you
refering to something else?
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 26 april 2018 15:38
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Does test$tester1 have permissions to access RGW? Mounting / should see
all the buckets, so it's likely a permission issue.
Daniel
On 04/26/2018 03:56 AM, Marc Roos wrote:
>
>
> I have the same as with / path, I can do the mount, but it is just
> empty
>
> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
> ls: reading directory nfstest/: Remote I/O error total 0
>
>
>
> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
> (test3) and tag ((null)) perms (options=03303002 , ,
> , , , , ,
> )
> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
> (test3) and tag ((null)) perms (options=03303002 , ,
> , , , , ,
> )
> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export
> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
> ngroups=0
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: donderdag 26 april 2018 14:47
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> I think it should be:
>
> Path="test3"
>
> so without the initial '/'. Buckets aren't paths, we're just
abusing
> the existing Ganesha config to select them.
>
> Other than that, the client entry should match the equivalent entry in
> your ceph.conf.
>
> Daniel
>
> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>
>>
>>
>> I have problems exporting a bucket that really does exist. I have
>> tried Path = "/test:test3"; Path = "/test3"; Results in
ganesha fails
>> to start with message ExportId=301 Path=/test:test3
>> FSAL_ERROR=(Invalid object
>> type,0)
>>
>> If I use path=/ I can mount something but that is a empty export, but
>> cannot put files there.
>>
>> I have this in ganesha (client.rwg3 works with civetweb), should be
>> sufficient not?
>>
>> RGW {
>> cluster = "ceph";
>> name = "client.rgw3";
>> ceph_conf = "/etc/ceph/ceph.conf";
>> # for vstart cluster, name = "client.admin"
>> #init_args = "-d --debug-rgw=16"; }
>>
>> EXPORT {
>> Export_ID=301;
>> Path = "/test3";
>> Pseudo = "/rgwtester";
>>
>> FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
>> ="sameass3"; Secret_Access_Key = "sameass3"; }
>> Disable_ACL = FALSE;
>> CLIENT { Clients = 192.168.x.0/24; } }
>>
>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>> 2018-01-31 21:48 s3://test
>> 2018-02-01 11:44 s3://test2
>> 2018-02-02 17:10 s3://test3
>>
>>
>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>
>> [@ ganesha]# rpm -qa | grep ganesh
>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>> nfs-ganesha-2.5.5-.el7.x86_64
>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>
>> On CentOS7
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
2:25 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Yes same access key and secret access key ( no uid is used )
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 14:40
To: Marc Roos; support
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
Yes, I can access the bucket via the s3cmd and cyberduck, or are you
refering to something else?
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 26 april 2018 15:38
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Does test$tester1 have permissions to access RGW? Mounting / should
see all the buckets, so it's likely a permission issue.
Daniel
On 04/26/2018 03:56 AM, Marc Roos wrote:
>
>
> I have the same as with / path, I can do the mount, but it is just
> empty
>
> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
> ls: reading directory nfstest/: Remote I/O error total 0
>
>
>
> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with
path
> (test3) and tag ((null)) perms (options=03303002 , ,
> , , , , ,
> )
> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with
path
> (test3) and tag ((null)) perms (options=03303002 , ,
> , , , , ,
> )
> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15
c01
> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export
> 0 (/) successfully created Apr 26 14:56:15 c01
nfs-ganesha[3981596]:
> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
> ngroups=0
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: donderdag 26 april 2018 14:47
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> I think it should be:
>
> Path="test3"
>
> so without the initial '/'. Buckets aren't paths, we're just abusing
> the existing Ganesha config to select them.
>
> Other than that, the client entry should match the equivalent entry
> in
> your ceph.conf.
>
> Daniel
>
> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>
>>
>>
>> I have problems exporting a bucket that really does exist. I have
>> tried Path = "/test:test3"; Path = "/test3"; Results in
ganesha
>> fails
>> to start with message ExportId=301 Path=/test:test3
>> FSAL_ERROR=(Invalid object
>> type,0)
>>
>> If I use path=/ I can mount something but that is a empty export,
>> but
>> cannot put files there.
>>
>> I have this in ganesha (client.rwg3 works with civetweb), should be
>> sufficient not?
>>
>> RGW {
>> cluster = "ceph";
>> name = "client.rgw3";
>> ceph_conf = "/etc/ceph/ceph.conf";
>> # for vstart cluster, name = "client.admin"
>> #init_args = "-d --debug-rgw=16"; }
>>
>> EXPORT {
>> Export_ID=301;
>> Path = "/test3";
>> Pseudo = "/rgwtester";
>>
>> FSAL { Name = RGW; User_Id = "test$tester1";
>> Access_Key_Id ="sameass3"; Secret_Access_Key = "sameass3"; }
>> Disable_ACL = FALSE;
>> CLIENT { Clients = 192.168.x.0/24; } }
>>
>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>> 2018-01-31 21:48 s3://test
>> 2018-02-01 11:44 s3://test2
>> 2018-02-02 17:10 s3://test3
>>
>>
>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>
>> [@ ganesha]# rpm -qa | grep ganesh
>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>> nfs-ganesha-2.5.5-.el7.x86_64
>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>
>> On CentOS7
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
8:14 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
And test$tester1 is the user ID you created, that is this gives output:
radosgw-admin user info --uid="test$tester1"
(I'm pounding on this because it really seems that the UID doesn't have
permissions. If it does, I think there's some really strange bug here
that's going to be hard to track down.)
Daniel
On 04/30/2018 03:25 AM, Marc Roos wrote:
Yes same access key and secret access key ( no uid is used )
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 14:40
To: Marc Roos; support
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
>
>
> Yes, I can access the bucket via the s3cmd and cyberduck, or are you
> refering to something else?
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: donderdag 26 april 2018 15:38
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> Does test$tester1 have permissions to access RGW? Mounting / should
> see all the buckets, so it's likely a permission issue.
>
> Daniel
>
> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>
>>
>> I have the same as with / path, I can do the mount, but it is just
>> empty
>>
>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>> ls: reading directory nfstest/: Remote I/O error total 0
>>
>>
>>
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>> (test3) and tag ((null)) perms (options=03303002 , ,
>> , , , , ,
>> )
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>> (test3) and tag ((null)) perms (options=03303002 , ,
>> , , , , ,
>> )
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export
>> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>> ngroups=0
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: donderdag 26 april 2018 14:47
>> To: support(a)lists.nfs-ganesha.org
>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>> tenancy rgw users
>>
>> I think it should be:
>>
>> Path="test3"
>>
>> so without the initial '/'. Buckets aren't paths, we're just
abusing
>> the existing Ganesha config to select them.
>>
>> Other than that, the client entry should match the equivalent entry
>> in
>
>> your ceph.conf.
>>
>> Daniel
>>
>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>
>>>
>>>
>>> I have problems exporting a bucket that really does exist. I have
>>> tried Path = "/test:test3"; Path = "/test3"; Results in
ganesha
>>> fails
>
>>> to start with message ExportId=301 Path=/test:test3
>>> FSAL_ERROR=(Invalid object
>>> type,0)
>>>
>>> If I use path=/ I can mount something but that is a empty export,
>>> but
>
>>> cannot put files there.
>>>
>>> I have this in ganesha (client.rwg3 works with civetweb), should be
>>> sufficient not?
>>>
>>> RGW {
>>> cluster = "ceph";
>>> name = "client.rgw3";
>>> ceph_conf = "/etc/ceph/ceph.conf";
>>> # for vstart cluster, name = "client.admin"
>>> #init_args = "-d --debug-rgw=16"; }
>>>
>>> EXPORT {
>>> Export_ID=301;
>>> Path = "/test3";
>>> Pseudo = "/rgwtester";
>>>
>>> FSAL { Name = RGW; User_Id = "test$tester1";
>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>> Disable_ACL = FALSE;
>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>
>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>> 2018-01-31 21:48 s3://test
>>> 2018-02-01 11:44 s3://test2
>>> 2018-02-02 17:10 s3://test3
>>>
>>>
>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>
>>> [@ ganesha]# rpm -qa | grep ganesh
>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-2.5.5-.el7.x86_64
>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>
>>> On CentOS7
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
3:01 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
[@ ganesha]# radosgw-admin user list
[
"test2$tester3",
"test$tester1",
"test$tester2"
]
[@ ganesha]# radosgw-admin user info --uid='test$tester1'
{
"user_id": "test$tester1",
"display_name": "Test1 User",
"email": "",
"suspended": 0,
"max_buckets": 1000,
"auid": 0,
"subusers": [],
"keys": [
{
"user": "test$tester1",
"access_key": "xxxxxxx",
"secret_key": "xxxxxxx"
},
{
"user": "test$tester1",
"access_key": "xxxxxxx",
"secret_key": "xxxxxxx"
}
],
"swift_keys": [],
"caps": [],
"op_mask": "read, write, delete",
"default_placement": "",
"placement_tags": [],
"bucket_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"user_quota": {
"enabled": false,
"check_on_raw": false,
"max_size": -1,
"max_size_kb": 0,
"max_objects": -1
},
"temp_url_keys": [],
"type": "rgw"
}
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 15:15
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
And test$tester1 is the user ID you created, that is this gives output:
radosgw-admin user info --uid="test$tester1"
(I'm pounding on this because it really seems that the UID doesn't have
permissions. If it does, I think there's some really strange bug here
that's going to be hard to track down.)
Daniel
On 04/30/2018 03:25 AM, Marc Roos wrote:
Yes same access key and secret access key ( no uid is used )
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 14:40
To: Marc Roos; support
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
>
>
> Yes, I can access the bucket via the s3cmd and cyberduck, or are you
> refering to something else?
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: donderdag 26 april 2018 15:38
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> Does test$tester1 have permissions to access RGW? Mounting / should
> see all the buckets, so it's likely a permission issue.
>
> Daniel
>
> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>
>>
>> I have the same as with / path, I can do the mount, but it is just
>> empty
>>
>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>> ls: reading directory nfstest/: Remote I/O error total 0
>>
>>
>>
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>> export_commit_common
>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>> (test3) and tag ((null)) perms (options=03303002 ,
,
>> , , , , ,
>> )
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>> export_commit_common
>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>> (test3) and tag ((null)) perms (options=03303002 ,
,
>> , , , , ,
>> )
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>> export_commit_common
>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>> :Export
>> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>> ngroups=0
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: donderdag 26 april 2018 14:47
>> To: support(a)lists.nfs-ganesha.org
>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>> tenancy rgw users
>>
>> I think it should be:
>>
>> Path="test3"
>>
>> so without the initial '/'. Buckets aren't paths, we're just
>> abusing
>> the existing Ganesha config to select them.
>>
>> Other than that, the client entry should match the equivalent entry
>> in
>
>> your ceph.conf.
>>
>> Daniel
>>
>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>
>>>
>>>
>>> I have problems exporting a bucket that really does exist. I have
>>> tried Path = "/test:test3"; Path = "/test3"; Results in
ganesha
>>> fails
>
>>> to start with message ExportId=301 Path=/test:test3
>>> FSAL_ERROR=(Invalid object
>>> type,0)
>>>
>>> If I use path=/ I can mount something but that is a empty export,
>>> but
>
>>> cannot put files there.
>>>
>>> I have this in ganesha (client.rwg3 works with civetweb), should be
>>> sufficient not?
>>>
>>> RGW {
>>> cluster = "ceph";
>>> name = "client.rgw3";
>>> ceph_conf = "/etc/ceph/ceph.conf";
>>> # for vstart cluster, name = "client.admin"
>>> #init_args = "-d --debug-rgw=16"; }
>>>
>>> EXPORT {
>>> Export_ID=301;
>>> Path = "/test3";
>>> Pseudo = "/rgwtester";
>>>
>>> FSAL { Name = RGW; User_Id = "test$tester1";
>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>> Disable_ACL = FALSE;
>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>
>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>> 2018-01-31 21:48 s3://test
>>> 2018-02-01 11:44 s3://test2
>>> 2018-02-02 17:10 s3://test3
>>>
>>>
>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>
>>> [@ ganesha]# rpm -qa | grep ganesh
>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-2.5.5-.el7.x86_64
>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>
>>> On CentOS7
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>> unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
3:06 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Could it not be, because of a configuration issue in ganesha.conf or the
version I am using? I can hardly imagine that I am the only one running
a multitenant rgw config with ganesha-rgw.
nfs-ganesha-xfs-2.5.5-.el7.x86_64
nfs-ganesha-2.5.5-.el7.x86_64
nfs-ganesha-rgw-2.5.5-.el7.x86_64
nfs-ganesha-ceph-2.5.5-.el7.x86_64
nfs-ganesha-vfs-2.5.5-.el7.x86_64
nfs-ganesha-mem-2.5.5-.el7.x86_64
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 15:15
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
And test$tester1 is the user ID you created, that is this gives output:
radosgw-admin user info --uid="test$tester1"
(I'm pounding on this because it really seems that the UID doesn't have
permissions. If it does, I think there's some really strange bug here
that's going to be hard to track down.)
Daniel
On 04/30/2018 03:25 AM, Marc Roos wrote:
Yes same access key and secret access key ( no uid is used )
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 14:40
To: Marc Roos; support
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
>
>
> Yes, I can access the bucket via the s3cmd and cyberduck, or are you
> refering to something else?
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: donderdag 26 april 2018 15:38
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> Does test$tester1 have permissions to access RGW? Mounting / should
> see all the buckets, so it's likely a permission issue.
>
> Daniel
>
> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>
>>
>> I have the same as with / path, I can do the mount, but it is just
>> empty
>>
>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>> ls: reading directory nfstest/: Remote I/O error total 0
>>
>>
>>
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>> export_commit_common
>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>> (test3) and tag ((null)) perms (options=03303002 ,
,
>> , , , , ,
>> )
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>> export_commit_common
>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>> (test3) and tag ((null)) perms (options=03303002 ,
,
>> , , , , ,
>> )
>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>> export_commit_common
>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>> :Export
>> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>> ngroups=0
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: donderdag 26 april 2018 14:47
>> To: support(a)lists.nfs-ganesha.org
>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>> tenancy rgw users
>>
>> I think it should be:
>>
>> Path="test3"
>>
>> so without the initial '/'. Buckets aren't paths, we're just
>> abusing
>> the existing Ganesha config to select them.
>>
>> Other than that, the client entry should match the equivalent entry
>> in
>
>> your ceph.conf.
>>
>> Daniel
>>
>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>
>>>
>>>
>>> I have problems exporting a bucket that really does exist. I have
>>> tried Path = "/test:test3"; Path = "/test3"; Results in
ganesha
>>> fails
>
>>> to start with message ExportId=301 Path=/test:test3
>>> FSAL_ERROR=(Invalid object
>>> type,0)
>>>
>>> If I use path=/ I can mount something but that is a empty export,
>>> but
>
>>> cannot put files there.
>>>
>>> I have this in ganesha (client.rwg3 works with civetweb), should be
>>> sufficient not?
>>>
>>> RGW {
>>> cluster = "ceph";
>>> name = "client.rgw3";
>>> ceph_conf = "/etc/ceph/ceph.conf";
>>> # for vstart cluster, name = "client.admin"
>>> #init_args = "-d --debug-rgw=16"; }
>>>
>>> EXPORT {
>>> Export_ID=301;
>>> Path = "/test3";
>>> Pseudo = "/rgwtester";
>>>
>>> FSAL { Name = RGW; User_Id = "test$tester1";
>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>> Disable_ACL = FALSE;
>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>
>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>> 2018-01-31 21:48 s3://test
>>> 2018-02-01 11:44 s3://test2
>>> 2018-02-02 17:10 s3://test3
>>>
>>>
>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>
>>> [@ ganesha]# rpm -qa | grep ganesh
>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-2.5.5-.el7.x86_64
>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>
>>> On CentOS7
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>> unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
8:28 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
All of that looks fine, I think. I'm out of my depth here, so I think
Matt needs to chime in here.
Matt?
On 04/30/2018 04:06 AM, Marc Roos wrote:
Could it not be, because of a configuration issue in ganesha.conf or the
version I am using? I can hardly imagine that I am the only one running
a multitenant rgw config with ganesha-rgw.
nfs-ganesha-xfs-2.5.5-.el7.x86_64
nfs-ganesha-2.5.5-.el7.x86_64
nfs-ganesha-rgw-2.5.5-.el7.x86_64
nfs-ganesha-ceph-2.5.5-.el7.x86_64
nfs-ganesha-vfs-2.5.5-.el7.x86_64
nfs-ganesha-mem-2.5.5-.el7.x86_64
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 15:15
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
And test$tester1 is the user ID you created, that is this gives output:
radosgw-admin user info --uid="test$tester1"
(I'm pounding on this because it really seems that the UID doesn't have
permissions. If it does, I think there's some really strange bug here
that's going to be hard to track down.)
Daniel
On 04/30/2018 03:25 AM, Marc Roos wrote:
>
> Yes same access key and secret access key ( no uid is used )
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: maandag 30 april 2018 14:40
> To: Marc Roos; support
> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
> multi tenancy rgw users
>
> I mean does the s3cmd (or cyberduck) use the exact same credentials as
> are specified in the ganesha.conf?
>
> Daniel
>
> On 04/26/2018 10:09 AM, Marc Roos wrote:
>>
>>
>> Yes, I can access the bucket via the s3cmd and cyberduck, or are you
>> refering to something else?
>>
>>
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: donderdag 26 april 2018 15:38
>> To: support(a)lists.nfs-ganesha.org
>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>> tenancy rgw users
>>
>> Does test$tester1 have permissions to access RGW? Mounting / should
>> see all the buckets, so it's likely a permission issue.
>>
>> Daniel
>>
>> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>>
>>>
>>> I have the same as with / path, I can do the mount, but it is just
>>> empty
>>>
>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>>> ls: reading directory nfstest/: Remote I/O error total 0
>>>
>>>
>>>
>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>> export_commit_common
>
>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>> (test3) and tag ((null)) perms (options=03303002 ,
,
>>> , , , , ,
>>> )
>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>> export_commit_common
>
>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>> (test3) and tag ((null)) perms (options=03303002 ,
,
>>> , , , , ,
>>> )
>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>> export_commit_common
>
>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>>> :Export
>
>>> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
>>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>> ngroups=0
>>>
>>> -----Original Message-----
>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>> Sent: donderdag 26 april 2018 14:47
>>> To: support(a)lists.nfs-ganesha.org
>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>>> tenancy rgw users
>>>
>>> I think it should be:
>>>
>>> Path="test3"
>>>
>>> so without the initial '/'. Buckets aren't paths, we're
just
>>> abusing
>
>>> the existing Ganesha config to select them.
>>>
>>> Other than that, the client entry should match the equivalent entry
>>> in
>>
>>> your ceph.conf.
>>>
>>> Daniel
>>>
>>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>>
>>>>
>>>>
>>>> I have problems exporting a bucket that really does exist. I have
>>>> tried Path = "/test:test3"; Path = "/test3"; Results
in ganesha
>>>> fails
>>
>>>> to start with message ExportId=301 Path=/test:test3
>>>> FSAL_ERROR=(Invalid object
>>>> type,0)
>>>>
>>>> If I use path=/ I can mount something but that is a empty export,
>>>> but
>>
>>>> cannot put files there.
>>>>
>>>> I have this in ganesha (client.rwg3 works with civetweb), should be
>>>> sufficient not?
>>>>
>>>> RGW {
>>>> cluster = "ceph";
>>>> name = "client.rgw3";
>>>> ceph_conf = "/etc/ceph/ceph.conf";
>>>> # for vstart cluster, name = "client.admin"
>>>> #init_args = "-d --debug-rgw=16"; }
>>>>
>>>> EXPORT {
>>>> Export_ID=301;
>>>> Path = "/test3";
>>>> Pseudo = "/rgwtester";
>>>>
>>>> FSAL { Name = RGW; User_Id = "test$tester1";
>>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>>> Disable_ACL = FALSE;
>>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>>
>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>>> 2018-01-31 21:48 s3://test
>>>> 2018-02-01 11:44 s3://test2
>>>> 2018-02-02 17:10 s3://test3
>>>>
>>>>
>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>>
>>>> [@ ganesha]# rpm -qa | grep ganesh
>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>>> nfs-ganesha-2.5.5-.el7.x86_64
>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>>
>>>> On CentOS7
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>> unsubscribe
>
>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>>
>
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
1:05 p.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Okay, so on consultation, we think that maybe you should be mounting
test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the
mail thread if you tried that; can you confirm?
(I have no personal experience with multi-tenant, so I didn't recognize
it as such).
In the mean time, we'll try to reproduce here and see what we can see.
Daniel
On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
All of that looks fine, I think. I'm out of my depth here, so I
think
Matt needs to chime in here.
Matt?
On 04/30/2018 04:06 AM, Marc Roos wrote:
> Could it not be, because of a configuration issue in ganesha.conf or the
> version I am using? I can hardly imagine that I am the only one running
> a multitenant rgw config with ganesha-rgw.
>
> nfs-ganesha-xfs-2.5.5-.el7.x86_64
> nfs-ganesha-2.5.5-.el7.x86_64
> nfs-ganesha-rgw-2.5.5-.el7.x86_64
> nfs-ganesha-ceph-2.5.5-.el7.x86_64
> nfs-ganesha-vfs-2.5.5-.el7.x86_64
> nfs-ganesha-mem-2.5.5-.el7.x86_64
>
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: maandag 30 april 2018 15:15
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> And test$tester1 is the user ID you created, that is this gives output:
>
> radosgw-admin user info --uid="test$tester1"
>
> (I'm pounding on this because it really seems that the UID doesn't have
> permissions. If it does, I think there's some really strange bug here
> that's going to be hard to track down.)
>
> Daniel
>
>
> On 04/30/2018 03:25 AM, Marc Roos wrote:
>> Yes same access key and secret access key ( no uid is used )
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: maandag 30 april 2018 14:40
>> To: Marc Roos; support
>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>> multi tenancy rgw users
>>
>> I mean does the s3cmd (or cyberduck) use the exact same credentials as
>
>> are specified in the ganesha.conf?
>>
>> Daniel
>>
>> On 04/26/2018 10:09 AM, Marc Roos wrote:
>>>
>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are you
>>> refering to something else?
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>> Sent: donderdag 26 april 2018 15:38
>>> To: support(a)lists.nfs-ganesha.org
>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>>> tenancy rgw users
>>>
>>> Does test$tester1 have permissions to access RGW? Mounting / should
>>> see all the buckets, so it's likely a permission issue.
>>>
>>> Daniel
>>>
>>> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>>>
>>>> I have the same as with / path, I can do the mount, but it is just
>>>> empty
>>>>
>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>>>> ls: reading directory nfstest/: Remote I/O error total 0
>>>>
>>>>
>>>>
>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>> export_commit_common
>>
>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>>> (test3) and tag ((null)) perms (options=03303002 ,
> ,
>>>> , , , , ,
>>>> )
>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>> export_commit_common
>>
>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>>> (test3) and tag ((null)) perms (options=03303002 ,
> ,
>>>> , , , , ,
>>>> )
>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>> export_commit_common
>>
>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>>>> :Export
>>
>>>> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]:
>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>> ngroups=0
>>>>
>>>> -----Original Message-----
>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>> Sent: donderdag 26 april 2018 14:47
>>>> To: support(a)lists.nfs-ganesha.org
>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>>>> tenancy rgw users
>>>>
>>>> I think it should be:
>>>>
>>>> Path="test3"
>>>>
>>>> so without the initial '/'. Buckets aren't paths, we're
just
>>>> abusing
>>
>>>> the existing Ganesha config to select them.
>>>>
>>>> Other than that, the client entry should match the equivalent entry
>>>> in
>>>
>>>> your ceph.conf.
>>>>
>>>> Daniel
>>>>
>>>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>>>
>>>>>
>>>>> I have problems exporting a bucket that really does exist. I have
>>>>> tried Path = "/test:test3"; Path = "/test3";
Results in ganesha
>>>>> fails
>>>
>>>>> to start with message ExportId=301 Path=/test:test3
>>>>> FSAL_ERROR=(Invalid object
>>>>> type,0)
>>>>>
>>>>> If I use path=/ I can mount something but that is a empty export,
>>>>> but
>>>
>>>>> cannot put files there.
>>>>>
>>>>> I have this in ganesha (client.rwg3 works with civetweb), should be
>
>>>>> sufficient not?
>>>>>
>>>>> RGW {
>>>>> cluster = "ceph";
>>>>> name = "client.rgw3";
>>>>> ceph_conf = "/etc/ceph/ceph.conf";
>>>>> # for vstart cluster, name = "client.admin"
>>>>> #init_args = "-d --debug-rgw=16"; }
>>>>>
>>>>> EXPORT {
>>>>> Export_ID=301;
>>>>> Path = "/test3";
>>>>> Pseudo = "/rgwtester";
>>>>>
>>>>> FSAL { Name = RGW; User_Id = "test$tester1";
>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>>>> Disable_ACL = FALSE;
>>>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>>>
>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>>>> 2018-01-31 21:48 s3://test
>>>>> 2018-02-01 11:44 s3://test2
>>>>> 2018-02-02 17:10 s3://test3
>>>>>
>>>>>
>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>>>
>>>>> [@ ganesha]# rpm -qa | grep ganesh
>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>>>
>>>>> On CentOS7
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe
>>
>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>
>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>
>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>>>
>>
>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
>
10:29 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I stopped the rgw3 instance that is running on the same server (I am not
sure anymore if I can run radosgw and nfs-ganesha with the same
client.rgw3), started ganesha with this config. I am correct to
understand that the ganesha-rgw is not using the http(s) interface and
directly connects to ceph with librados (because that is why we have the
RGW config section)?
RGW {
cluster = "ceph";
name = "client.rgw3";
ceph_conf = "/etc/ceph/ceph.conf";
}
EXPORT {
Export_ID=301;
Path = "test:test3";
Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
="xxxxxx"; Secret_Access_Key = "xxxxxx"; }
Disable_ACL = TRUE;
CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; }
}
Logs of ganesha related to test3 bucket
[@]# tail -1000 /var/log/messages | grep ganesh | grep test3
Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release
:FSAL :INFO :Releasing RGW export for test:test3
Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release
:FSAL :INFO :Releasing RGW export for test:test3
Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test:test3) and tag ((null)) perms (options=03303002 ,
, , , , , ,
)
Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test:test3) and tag ((null)) perms (options=03303002 ,
, , , , , ,
)
[@]# showmount -e nfs
Export list for nfs:
/ram 192.168.x.0
/test 192.168.x.0
/rgwtester 192.168.x.0
[@ mnt]# mount nfs:/rgwtester /mnt/nfstest
[@ mnt]# mount
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
...
nfs:/test on /home/users/robbert/nfs-test type nfs4
(rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,naml
en=255,hard,addr=
nfs:/rgwtester on /mnt/nfstest type nfs
(rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x)
[@ mnt]# ls -l /mnt/nfstest/
ls: reading directory /mnt/nfstest/: Remote I/O error
total 0
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 20:06
To: Marc Roos; support
Cc: Matt Benjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so on consultation, we think that maybe you should be mounting
test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the
mail thread if you tried that; can you confirm?
(I have no personal experience with multi-tenant, so I didn't recognize
it as such).
In the mean time, we'll try to reproduce here and see what we can see.
Daniel
On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
All of that looks fine, I think. I'm out of my depth here, so I
think
Matt needs to chime in here.
Matt?
On 04/30/2018 04:06 AM, Marc Roos wrote:
> Could it not be, because of a configuration issue in ganesha.conf or
> the version I am using? I can hardly imagine that I am the only one
> running a multitenant rgw config with ganesha-rgw.
>
> nfs-ganesha-xfs-2.5.5-.el7.x86_64
> nfs-ganesha-2.5.5-.el7.x86_64
> nfs-ganesha-rgw-2.5.5-.el7.x86_64
> nfs-ganesha-ceph-2.5.5-.el7.x86_64
> nfs-ganesha-vfs-2.5.5-.el7.x86_64
> nfs-ganesha-mem-2.5.5-.el7.x86_64
>
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: maandag 30 april 2018 15:15
> To: support(a)lists.nfs-ganesha.org
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
> And test$tester1 is the user ID you created, that is this gives
output:
>
> radosgw-admin user info --uid="test$tester1"
>
> (I'm pounding on this because it really seems that the UID doesn't
> have permissions. If it does, I think there's some really strange
> bug here that's going to be hard to track down.)
>
> Daniel
>
>
> On 04/30/2018 03:25 AM, Marc Roos wrote:
>> Yes same access key and secret access key ( no uid is used )
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: maandag 30 april 2018 14:40
>> To: Marc Roos; support
>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>> multi tenancy rgw users
>>
>> I mean does the s3cmd (or cyberduck) use the exact same credentials
>> as
>
>> are specified in the ganesha.conf?
>>
>> Daniel
>>
>> On 04/26/2018 10:09 AM, Marc Roos wrote:
>>>
>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are
>>> you refering to something else?
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>> Sent: donderdag 26 april 2018 15:38
>>> To: support(a)lists.nfs-ganesha.org
>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>>> tenancy rgw users
>>>
>>> Does test$tester1 have permissions to access RGW? Mounting /
>>> should see all the buckets, so it's likely a permission issue.
>>>
>>> Daniel
>>>
>>> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>>>
>>>> I have the same as with / path, I can do the mount, but it is just
>>>> empty
>>>>
>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>>>> ls: reading directory nfstest/: Remote I/O error total 0
>>>>
>>>>
>>>>
>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>> export_commit_common
>>
>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>>> (test3) and tag ((null)) perms (options=03303002 ,
> ,
>>>> , , , , ,
>>>> )
>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>> export_commit_common
>>
>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>>> (test3) and tag ((null)) perms (options=03303002 ,
> ,
>>>> , , , , ,
>>>> )
>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>> export_commit_common
>>
>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15
>>>> c01
>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>>>> :Export
>>
>>>> 0 (/) successfully created Apr 26 14:56:15 c01
nfs-ganesha[3981596]:
>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/)
successfully
>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]:
[main]
>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>> ngroups=0
>>>>
>>>> -----Original Message-----
>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>> Sent: donderdag 26 april 2018 14:47
>>>> To: support(a)lists.nfs-ganesha.org
>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>> multi tenancy rgw users
>>>>
>>>> I think it should be:
>>>>
>>>> Path="test3"
>>>>
>>>> so without the initial '/'. Buckets aren't paths, we're
just
>>>> abusing
>>
>>>> the existing Ganesha config to select them.
>>>>
>>>> Other than that, the client entry should match the equivalent
>>>> entry in
>>>
>>>> your ceph.conf.
>>>>
>>>> Daniel
>>>>
>>>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>>>
>>>>>
>>>>> I have problems exporting a bucket that really does exist. I have
>>>>> tried Path = "/test:test3"; Path =
"/test3"; Results in ganesha
>>>>> fails
>>>
>>>>> to start with message ExportId=301 Path=/test:test3
>>>>> FSAL_ERROR=(Invalid object
>>>>> type,0)
>>>>>
>>>>> If I use path=/ I can mount something but that is a empty export,
>>>>> but
>>>
>>>>> cannot put files there.
>>>>>
>>>>> I have this in ganesha (client.rwg3 works with civetweb), should
>>>>> be
>
>>>>> sufficient not?
>>>>>
>>>>> RGW {
>>>>> cluster = "ceph";
>>>>> name = "client.rgw3";
>>>>> ceph_conf = "/etc/ceph/ceph.conf";
>>>>> # for vstart cluster, name = "client.admin"
>>>>> #init_args = "-d --debug-rgw=16"; }
>>>>>
>>>>> EXPORT {
>>>>> Export_ID=301;
>>>>> Path = "/test3";
>>>>> Pseudo = "/rgwtester";
>>>>>
>>>>> FSAL { Name = RGW; User_Id = "test$tester1";
>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>>>> Disable_ACL = FALSE;
>>>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>>>
>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>>>> 2018-01-31 21:48 s3://test
>>>>> 2018-02-01 11:44 s3://test2
>>>>> 2018-02-02 17:10 s3://test3
>>>>>
>>>>>
>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>>>
>>>>> [@ ganesha]# rpm -qa | grep ganesh
>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>>>
>>>>> On CentOS7
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe
>>
>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>> unsubscribe
>
>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>> unsubscribe
>
>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>>>
>>
>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
>
7:11 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Okay, so that looks like Ganesha thinks the export is working, but the
actual I/O on the bucket is failing. Can you turn on DEBUG for the NFS4
and FSAL components, perform this same simple test (mount + ls + umount)
and post the ganesha log? This can be done like this:
LOG {
Components {
NFS4 = DEBUG;
FSAL = DEBUG;
}
}
Daniel
On 04/30/2018 11:29 AM, Marc Roos wrote:
I stopped the rgw3 instance that is running on the same server (I am not
sure anymore if I can run radosgw and nfs-ganesha with the same
client.rgw3), started ganesha with this config. I am correct to
understand that the ganesha-rgw is not using the http(s) interface and
directly connects to ceph with librados (because that is why we have the
RGW config section)?
RGW {
cluster = "ceph";
name = "client.rgw3";
ceph_conf = "/etc/ceph/ceph.conf";
}
EXPORT {
Export_ID=301;
Path = "test:test3";
Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
="xxxxxx"; Secret_Access_Key = "xxxxxx"; }
Disable_ACL = TRUE;
CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; }
}
Logs of ganesha related to test3 bucket
[@]# tail -1000 /var/log/messages | grep ganesh | grep test3
Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release
:FSAL :INFO :Releasing RGW export for test:test3
Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release
:FSAL :INFO :Releasing RGW export for test:test3
Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test:test3) and tag ((null)) perms (options=03303002 ,
, , , , , ,
)
Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test:test3) and tag ((null)) perms (options=03303002 ,
, , , , , ,
)
[@]# showmount -e nfs
Export list for nfs:
/ram 192.168.x.0
/test 192.168.x.0
/rgwtester 192.168.x.0
[@ mnt]# mount nfs:/rgwtester /mnt/nfstest
[@ mnt]# mount
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
...
nfs:/test on /home/users/robbert/nfs-test type nfs4
(rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,naml
en=255,hard,addr=
nfs:/rgwtester on /mnt/nfstest type nfs
(rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x)
[@ mnt]# ls -l /mnt/nfstest/
ls: reading directory /mnt/nfstest/: Remote I/O error
total 0
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 20:06
To: Marc Roos; support
Cc: Matt Benjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so on consultation, we think that maybe you should be mounting
test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the
mail thread if you tried that; can you confirm?
(I have no personal experience with multi-tenant, so I didn't recognize
it as such).
In the mean time, we'll try to reproduce here and see what we can see.
Daniel
On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
> All of that looks fine, I think. I'm out of my depth here, so I think
> Matt needs to chime in here.
>
> Matt?
>
> On 04/30/2018 04:06 AM, Marc Roos wrote:
>> Could it not be, because of a configuration issue in ganesha.conf or
>> the version I am using? I can hardly imagine that I am the only one
>> running a multitenant rgw config with ganesha-rgw.
>>
>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>> nfs-ganesha-2.5.5-.el7.x86_64
>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>
>>
>>
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: maandag 30 april 2018 15:15
>> To: support(a)lists.nfs-ganesha.org
>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>> tenancy rgw users
>>
>> And test$tester1 is the user ID you created, that is this gives
output:
>>
>> radosgw-admin user info --uid="test$tester1"
>>
>> (I'm pounding on this because it really seems that the UID doesn't
>> have permissions. If it does, I think there's some really strange
>> bug here that's going to be hard to track down.)
>>
>> Daniel
>>
>>
>> On 04/30/2018 03:25 AM, Marc Roos wrote:
>>> Yes same access key and secret access key ( no uid is used )
>>>
>>> -----Original Message-----
>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>> Sent: maandag 30 april 2018 14:40
>>> To: Marc Roos; support
>>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>> multi tenancy rgw users
>>>
>>> I mean does the s3cmd (or cyberduck) use the exact same credentials
>>> as
>>
>>> are specified in the ganesha.conf?
>>>
>>> Daniel
>>>
>>> On 04/26/2018 10:09 AM, Marc Roos wrote:
>>>>
>>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are
>>>> you refering to something else?
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>> Sent: donderdag 26 april 2018 15:38
>>>> To: support(a)lists.nfs-ganesha.org
>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>>>> tenancy rgw users
>>>>
>>>> Does test$tester1 have permissions to access RGW? Mounting /
>>>> should see all the buckets, so it's likely a permission issue.
>>>>
>>>> Daniel
>>>>
>>>> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>>>>
>>>>> I have the same as with / path, I can do the mount, but it is just
>>>>> empty
>>>>>
>>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>>>>> ls: reading directory nfstest/: Remote I/O error total 0
>>>>>
>>>>>
>>>>>
>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>> export_commit_common
>>>
>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>>>> (test3) and tag ((null)) perms (options=03303002 ,
>> ,
>>>>> , , , , ,
>>>>> )
>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>> export_commit_common
>>>
>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
>>>>> (test3) and tag ((null)) perms (options=03303002 ,
>> ,
>>>>> , , , , ,
>>>>> )
>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>> export_commit_common
>>>
>>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15
>>>>> c01
>>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>>>>> :Export
>>>
>>>>> 0 (/) successfully created Apr 26 14:56:15 c01
nfs-ganesha[3981596]:
>>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
>>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>>> ngroups=0
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>>> Sent: donderdag 26 april 2018 14:47
>>>>> To: support(a)lists.nfs-ganesha.org
>>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>>> multi tenancy rgw users
>>>>>
>>>>> I think it should be:
>>>>>
>>>>> Path="test3"
>>>>>
>>>>> so without the initial '/'. Buckets aren't paths,
we're just
>>>>> abusing
>>>
>>>>> the existing Ganesha config to select them.
>>>>>
>>>>> Other than that, the client entry should match the equivalent
>>>>> entry in
>>>>
>>>>> your ceph.conf.
>>>>>
>>>>> Daniel
>>>>>
>>>>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>>>>
>>>>>>
>>>>>> I have problems exporting a bucket that really does exist. I
have
>>>>>> tried Path = "/test:test3"; Path = "/test3";
Results in ganesha
>>>>>> fails
>>>>
>>>>>> to start with message ExportId=301 Path=/test:test3
>>>>>> FSAL_ERROR=(Invalid object
>>>>>> type,0)
>>>>>>
>>>>>> If I use path=/ I can mount something but that is a empty
export,
>>>>>> but
>>>>
>>>>>> cannot put files there.
>>>>>>
>>>>>> I have this in ganesha (client.rwg3 works with civetweb), should
>>>>>> be
>>
>>>>>> sufficient not?
>>>>>>
>>>>>> RGW {
>>>>>> cluster = "ceph";
>>>>>> name = "client.rgw3";
>>>>>> ceph_conf = "/etc/ceph/ceph.conf";
>>>>>> # for vstart cluster, name =
"client.admin"
>>>>>> #init_args = "-d --debug-rgw=16"; }
>>>>>>
>>>>>> EXPORT {
>>>>>> Export_ID=301;
>>>>>> Path = "/test3";
>>>>>> Pseudo = "/rgwtester";
>>>>>>
>>>>>> FSAL { Name = RGW; User_Id =
"test$tester1";
>>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>>>>> Disable_ACL = FALSE;
>>>>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>>>>
>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>>>>> 2018-01-31 21:48 s3://test
>>>>>> 2018-02-01 11:44 s3://test2
>>>>>> 2018-02-02 17:10 s3://test3
>>>>>>
>>>>>>
>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>>>>
>>>>>> [@ ganesha]# rpm -qa | grep ganesh
>>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>>>>
>>>>>> On CentOS7
>>>>>> _______________________________________________
>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>>> unsubscribe
>>>
>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>>
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe
>>
>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe
>>
>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>>
>
8:20 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Attached the logs mount.log, ls.log and unmount.log
With this config:
## configured from the command line
LOG {
## Default log level for all components
# NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG,
M_DBG, FULL_DEBUG, F_DBG], default EVENT
default_log_level = FATAL;
## Configure per-component log levels.
# ALL,
LOG,LOG_EMERG,MEMLEAKS,FSAL,NFSPROTO(NFS3),NFS_V4(NFSV4),EXPORT,FILEHAND
LE,DISPATCH,CACHE_INODE,
# CACHE_INODE_LRU,HASHTABLE,HASHTABLE_CACHE,DUPREQ,INIT,MAIN,
IDMAPPER,NFS_READDIR,NFS_V4_LOCK,CONFIG,CLIENTID,
#
SESSIONS,PNFS,RW_LOCK,NLM,RPC,NFS_CB,THREAD,NFS_V4_ACL,STATE,9P,9P_DISPA
TCH,FSAL_UP,DBUS
Components {
#ALL = FATAL;
FSAL = DEBUG;
NFS4 = DEBUG;
EXPORT = F_DBG;
CONFIG = F_DBG;
}
## Where to log
Facility {
name = FILE;
destination = "/var/log/ganesha.log";
enable = default;
}
}
PS. The DEBUG level is not working with syslog. I have to enable the
local logging.
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: dinsdag 1 mei 2018 14:11
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so that looks like Ganesha thinks the export is working, but the
actual I/O on the bucket is failing. Can you turn on DEBUG for the NFS4
and FSAL components, perform this same simple test (mount + ls + umount)
and post the ganesha log? This can be done like this:
LOG {
Components {
NFS4 = DEBUG;
FSAL = DEBUG;
}
}
Daniel
On 04/30/2018 11:29 AM, Marc Roos wrote:
I stopped the rgw3 instance that is running on the same server (I am
not sure anymore if I can run radosgw and nfs-ganesha with the same
client.rgw3), started ganesha with this config. I am correct to
understand that the ganesha-rgw is not using the http(s) interface and
directly connects to ceph with librados (because that is why we have
the RGW config section)?
RGW {
cluster = "ceph";
name = "client.rgw3";
ceph_conf = "/etc/ceph/ceph.conf"; }
EXPORT {
Export_ID=301;
Path = "test:test3";
Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
="xxxxxx"; Secret_Access_Key = "xxxxxx"; }
Disable_ACL = TRUE;
CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } }
Logs of ganesha related to test3 bucket
[@]# tail -1000 /var/log/messages | grep ganesh | grep test3 Apr 30
22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL
:INFO :Releasing RGW export for test:test3 Apr 30 22:24:55 c01
nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO
:Releasing RGW export for test:test3 Apr 30 22:27:07 c01
nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export
301 created at pseudo (/rgwtester) with path
(test:test3) and tag ((null)) perms (options=03303002 ,
, , , , , ,
)
Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
(test:test3) and tag ((null)) perms (options=03303002 ,
, , , , , ,
)
[@]# showmount -e nfs
Export list for nfs:
/ram 192.168.x.0
/test 192.168.x.0
/rgwtester 192.168.x.0
[@ mnt]# mount nfs:/rgwtester /mnt/nfstest
[@ mnt]# mount
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm
type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ...
nfs:/test on /home/users/robbert/nfs-test type nfs4
(rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,na
ml
en=255,hard,addr=
nfs:/rgwtester on /mnt/nfstest type nfs
(rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x)
[@ mnt]# ls -l /mnt/nfstest/
ls: reading directory /mnt/nfstest/: Remote I/O error total 0
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 30 april 2018 20:06
To: Marc Roos; support
Cc: Matt Benjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
Okay, so on consultation, we think that maybe you should be mounting
test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the
mail thread if you tried that; can you confirm?
(I have no personal experience with multi-tenant, so I didn't
recognize it as such).
In the mean time, we'll try to reproduce here and see what we can see.
Daniel
On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
> All of that looks fine, I think. I'm out of my depth here, so I
> think
> Matt needs to chime in here.
>
> Matt?
>
> On 04/30/2018 04:06 AM, Marc Roos wrote:
>> Could it not be, because of a configuration issue in ganesha.conf or
>> the version I am using? I can hardly imagine that I am the
only one
>> running a multitenant rgw config with ganesha-rgw.
>>
>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>> nfs-ganesha-2.5.5-.el7.x86_64
>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>
>>
>>
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: maandag 30 april 2018 15:15
>> To: support(a)lists.nfs-ganesha.org
>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>> tenancy rgw users
>>
>> And test$tester1 is the user ID you created, that is this gives
output:
>>
>> radosgw-admin user info --uid="test$tester1"
>>
>> (I'm pounding on this because it really seems that the UID doesn't
>> have permissions. If it does, I think there's some really strange
>> bug here that's going to be hard to track down.)
>>
>> Daniel
>>
>>
>> On 04/30/2018 03:25 AM, Marc Roos wrote:
>>> Yes same access key and secret access key ( no uid is used )
>>>
>>> -----Original Message-----
>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>> Sent: maandag 30 april 2018 14:40
>>> To: Marc Roos; support
>>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>> multi tenancy rgw users
>>>
>>> I mean does the s3cmd (or cyberduck) use the exact same credentials
>>> as
>>
>>> are specified in the ganesha.conf?
>>>
>>> Daniel
>>>
>>> On 04/26/2018 10:09 AM, Marc Roos wrote:
>>>>
>>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are
>>>> you refering to something else?
>>>>
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>> Sent: donderdag 26 april 2018 15:38
>>>> To: support(a)lists.nfs-ganesha.org
>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>> multi
>>>> tenancy rgw users
>>>>
>>>> Does test$tester1 have permissions to access RGW? Mounting /
>>>> should see all the buckets, so it's likely a permission issue.
>>>>
>>>> Daniel
>>>>
>>>> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>>>>
>>>>> I have the same as with / path, I can do the mount, but it is
>>>>> just
>>>>> empty
>>>>>
>>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>>>>> ls: reading directory nfstest/: Remote I/O error total 0
>>>>>
>>>>>
>>>>>
>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>> export_commit_common
>>>
>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with
>>>>> path
>>>>> (test3) and tag ((null)) perms (options=03303002 ,
>> ,
>>>>> , , , , ,
>>>>> )
>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>> export_commit_common
>>>
>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with
>>>>> path
>>>>> (test3) and tag ((null)) perms (options=03303002 ,
>> ,
>>>>> , , , , ,
>>>>> )
>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>> export_commit_common
>>>
>>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15
>>>>> c01
>>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>>>>> :Export
>>>
>>>>> 0 (/) successfully created Apr 26 14:56:15 c01
nfs-ganesha[3981596]:
>>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/)
>>>>> successfully
>>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>>> ngroups=0
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>>> Sent: donderdag 26 april 2018 14:47
>>>>> To: support(a)lists.nfs-ganesha.org
>>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>>> multi tenancy rgw users
>>>>>
>>>>> I think it should be:
>>>>>
>>>>> Path="test3"
>>>>>
>>>>> so without the initial '/'. Buckets aren't paths,
we're just
>>>>> abusing
>>>
>>>>> the existing Ganesha config to select them.
>>>>>
>>>>> Other than that, the client entry should match the equivalent
>>>>> entry in
>>>>
>>>>> your ceph.conf.
>>>>>
>>>>> Daniel
>>>>>
>>>>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>>>>
>>>>>>
>>>>>> I have problems exporting a bucket that really does exist. I
>>>>>> have
>>>>>> tried Path = "/test:test3"; Path = "/test3";
Results in ganesha
>>>>>> fails
>>>>
>>>>>> to start with message ExportId=301 Path=/test:test3
>>>>>> FSAL_ERROR=(Invalid object
>>>>>> type,0)
>>>>>>
>>>>>> If I use path=/ I can mount something but that is a empty
>>>>>> export,
>>>>>> but
>>>>
>>>>>> cannot put files there.
>>>>>>
>>>>>> I have this in ganesha (client.rwg3 works with civetweb), should
>>>>>> be
>>
>>>>>> sufficient not?
>>>>>>
>>>>>> RGW {
>>>>>> cluster = "ceph";
>>>>>> name = "client.rgw3";
>>>>>> ceph_conf = "/etc/ceph/ceph.conf";
>>>>>> # for vstart cluster, name =
"client.admin"
>>>>>> #init_args = "-d --debug-rgw=16"; }
>>>>>>
>>>>>> EXPORT {
>>>>>> Export_ID=301;
>>>>>> Path = "/test3";
>>>>>> Pseudo = "/rgwtester";
>>>>>>
>>>>>> FSAL { Name = RGW; User_Id =
"test$tester1";
>>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>>>>> Disable_ACL = FALSE;
>>>>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>>>>
>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>>>>> 2018-01-31 21:48 s3://test
>>>>>> 2018-02-01 11:44 s3://test2
>>>>>> 2018-02-02 17:10 s3://test3
>>>>>>
>>>>>>
>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>>>>
>>>>>> [@ ganesha]# rpm -qa | grep ganesh
>>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>>>>
>>>>>> On CentOS7
>>>>>> _______________________________________________
>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>>> unsubscribe
>>>
>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>>
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe
>>
>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe
>>
>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>> unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>>
>
11:47 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many ways
this can happen; unfortunately, the logging in 2.5 is a bit out of date,
so I can't tell which issue hit. Can you run the ls with FULL_DEBUG on
for everything? That should give me enough output to figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
On 05/01/2018 09:20 AM, Marc Roos wrote:
Attached the logs mount.log, ls.log and unmount.log
With this config:
## configured from the command line
LOG {
## Default log level for all components
# NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG,
M_DBG, FULL_DEBUG, F_DBG], default EVENT
default_log_level = FATAL;
## Configure per-component log levels.
# ALL,
LOG,LOG_EMERG,MEMLEAKS,FSAL,NFSPROTO(NFS3),NFS_V4(NFSV4),EXPORT,FILEHAND
LE,DISPATCH,CACHE_INODE,
# CACHE_INODE_LRU,HASHTABLE,HASHTABLE_CACHE,DUPREQ,INIT,MAIN,
IDMAPPER,NFS_READDIR,NFS_V4_LOCK,CONFIG,CLIENTID,
#
SESSIONS,PNFS,RW_LOCK,NLM,RPC,NFS_CB,THREAD,NFS_V4_ACL,STATE,9P,9P_DISPA
TCH,FSAL_UP,DBUS
Components {
#ALL = FATAL;
FSAL = DEBUG;
NFS4 = DEBUG;
EXPORT = F_DBG;
CONFIG = F_DBG;
}
## Where to log
Facility {
name = FILE;
destination = "/var/log/ganesha.log";
enable = default;
}
}
PS. The DEBUG level is not working with syslog. I have to enable the
local logging.
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: dinsdag 1 mei 2018 14:11
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so that looks like Ganesha thinks the export is working, but the
actual I/O on the bucket is failing. Can you turn on DEBUG for the NFS4
and FSAL components, perform this same simple test (mount + ls + umount)
and post the ganesha log? This can be done like this:
LOG {
Components {
NFS4 = DEBUG;
FSAL = DEBUG;
}
}
Daniel
On 04/30/2018 11:29 AM, Marc Roos wrote:
>
> I stopped the rgw3 instance that is running on the same server (I am
> not sure anymore if I can run radosgw and nfs-ganesha with the same
> client.rgw3), started ganesha with this config. I am correct to
> understand that the ganesha-rgw is not using the http(s) interface and
> directly connects to ceph with librados (because that is why we have
> the RGW config section)?
>
> RGW {
> cluster = "ceph";
> name = "client.rgw3";
> ceph_conf = "/etc/ceph/ceph.conf"; }
>
> EXPORT {
> Export_ID=301;
> Path = "test:test3";
> Pseudo = "/rgwtester";
>
> FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id
> ="xxxxxx"; Secret_Access_Key = "xxxxxx"; }
> Disable_ACL = TRUE;
> CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } }
>
>
> Logs of ganesha related to test3 bucket
>
> [@]# tail -1000 /var/log/messages | grep ganesh | grep test3 Apr 30
> 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL
> :INFO :Releasing RGW export for test:test3 Apr 30 22:24:55 c01
> nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO
> :Releasing RGW export for test:test3 Apr 30 22:27:07 c01
> nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export
> 301 created at pseudo (/rgwtester) with path
> (test:test3) and tag ((null)) perms (options=03303002 ,
> , , , , , ,
> )
> Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common
> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path
> (test:test3) and tag ((null)) perms (options=03303002 ,
> , , , , , ,
> )
>
>
> [@]# showmount -e nfs
> Export list for nfs:
> /ram 192.168.x.0
> /test 192.168.x.0
> /rgwtester 192.168.x.0
>
> [@ mnt]# mount nfs:/rgwtester /mnt/nfstest
>
> [@ mnt]# mount
> proc on /proc type proc (rw)
> sysfs on /sys type sysfs (rw)
> devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm
> type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ...
> nfs:/test on /home/users/robbert/nfs-test type nfs4
> (rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,na
> ml
> en=255,hard,addr=
> nfs:/rgwtester on /mnt/nfstest type nfs
> (rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x)
>
> [@ mnt]# ls -l /mnt/nfstest/
> ls: reading directory /mnt/nfstest/: Remote I/O error total 0
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: maandag 30 april 2018 20:06
> To: Marc Roos; support
> Cc: Matt Benjamin
> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
> multi tenancy rgw users
>
> Okay, so on consultation, we think that maybe you should be mounting
> test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from
the
> mail thread if you tried that; can you confirm?
>
> (I have no personal experience with multi-tenant, so I didn't
> recognize it as such).
>
> In the mean time, we'll try to reproduce here and see what we can see.
>
> Daniel
>
> On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
>> All of that looks fine, I think. I'm out of my depth here, so I
>> think
>
>> Matt needs to chime in here.
>>
>> Matt?
>>
>> On 04/30/2018 04:06 AM, Marc Roos wrote:
>>> Could it not be, because of a configuration issue in ganesha.conf or
>>> the version I am using? I can hardly imagine that I am the only one
>>> running a multitenant rgw config with ganesha-rgw.
>>>
>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-2.5.5-.el7.x86_64
>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>
>>>
>>>
>>>
>>> -----Original Message-----
>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>> Sent: maandag 30 april 2018 15:15
>>> To: support(a)lists.nfs-ganesha.org
>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
>>> tenancy rgw users
>>>
>>> And test$tester1 is the user ID you created, that is this gives
> output:
>>>
>>> radosgw-admin user info --uid="test$tester1"
>>>
>>> (I'm pounding on this because it really seems that the UID doesn't
>>> have permissions. If it does, I think there's some really strange
>>> bug here that's going to be hard to track down.)
>>>
>>> Daniel
>>>
>>>
>>> On 04/30/2018 03:25 AM, Marc Roos wrote:
>>>> Yes same access key and secret access key ( no uid is used )
>>>>
>>>> -----Original Message-----
>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>> Sent: maandag 30 april 2018 14:40
>>>> To: Marc Roos; support
>>>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>> multi tenancy rgw users
>>>>
>>>> I mean does the s3cmd (or cyberduck) use the exact same credentials
>>>> as
>>>
>>>> are specified in the ganesha.conf?
>>>>
>>>> Daniel
>>>>
>>>> On 04/26/2018 10:09 AM, Marc Roos wrote:
>>>>>
>>>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are
>>>>> you refering to something else?
>>>>>
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>>> Sent: donderdag 26 april 2018 15:38
>>>>> To: support(a)lists.nfs-ganesha.org
>>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>>> multi
>
>>>>> tenancy rgw users
>>>>>
>>>>> Does test$tester1 have permissions to access RGW? Mounting /
>>>>> should see all the buckets, so it's likely a permission issue.
>>>>>
>>>>> Daniel
>>>>>
>>>>> On 04/26/2018 03:56 AM, Marc Roos wrote:
>>>>>>
>>>>>> I have the same as with / path, I can do the mount, but it is
>>>>>> just
>
>>>>>> empty
>>>>>>
>>>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/
>>>>>> ls: reading directory nfstest/: Remote I/O error total 0
>>>>>>
>>>>>>
>>>>>>
>>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>>> export_commit_common
>>>>
>>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with
>>>>>> path
>>>>>> (test3) and tag ((null)) perms (options=03303002 ,
>>> ,
>>>>>> , , , , ,
>>>>>> )
>>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>>> export_commit_common
>>>>
>>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with
>>>>>> path
>>>>>> (test3) and tag ((null)) perms (options=03303002 ,
>>> ,
>>>>>> , , , , ,
>>>>>> )
>>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main]
>>>>>> export_commit_common
>>>>
>>>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15
>>>>>> c01
>>>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO
>>>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01
>>>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO
>>>>>> :Export
>>>>
>>>>>> 0 (/) successfully created Apr 26 14:56:15 c01
> nfs-ganesha[3981596]:
>>>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/)
>>>>>> successfully
>
>>>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main]
>>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0
>>>>>> ngroups=0
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>>>>>> Sent: donderdag 26 april 2018 14:47
>>>>>> To: support(a)lists.nfs-ganesha.org
>>>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>>>>>> multi tenancy rgw users
>>>>>>
>>>>>> I think it should be:
>>>>>>
>>>>>> Path="test3"
>>>>>>
>>>>>> so without the initial '/'. Buckets aren't paths,
we're just
>>>>>> abusing
>>>>
>>>>>> the existing Ganesha config to select them.
>>>>>>
>>>>>> Other than that, the client entry should match the equivalent
>>>>>> entry in
>>>>>
>>>>>> your ceph.conf.
>>>>>>
>>>>>> Daniel
>>>>>>
>>>>>> On 04/26/2018 01:52 AM, Marc Roos wrote:
>>>>>>>
>>>>>>>
>>>>>>> I have problems exporting a bucket that really does exist. I
>>>>>>> have
>
>>>>>>> tried Path = "/test:test3"; Path =
"/test3"; Results in ganesha
>>>>>>> fails
>>>>>
>>>>>>> to start with message ExportId=301 Path=/test:test3
>>>>>>> FSAL_ERROR=(Invalid object
>>>>>>> type,0)
>>>>>>>
>>>>>>> If I use path=/ I can mount something but that is a empty
>>>>>>> export,
>
>>>>>>> but
>>>>>
>>>>>>> cannot put files there.
>>>>>>>
>>>>>>> I have this in ganesha (client.rwg3 works with civetweb),
should
>>>>>>> be
>>>
>>>>>>> sufficient not?
>>>>>>>
>>>>>>> RGW {
>>>>>>> cluster = "ceph";
>>>>>>> name = "client.rgw3";
>>>>>>> ceph_conf = "/etc/ceph/ceph.conf";
>>>>>>> # for vstart cluster, name =
"client.admin"
>>>>>>> #init_args = "-d --debug-rgw=16"; }
>>>>>>>
>>>>>>> EXPORT {
>>>>>>> Export_ID=301;
>>>>>>> Path = "/test3";
>>>>>>> Pseudo = "/rgwtester";
>>>>>>>
>>>>>>> FSAL { Name = RGW; User_Id =
"test$tester1";
>>>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key =
"sameass3"; }
>>>>>>> Disable_ACL = FALSE;
>>>>>>> CLIENT { Clients = 192.168.x.0/24; } }
>>>>>>>
>>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls
>>>>>>> 2018-01-31 21:48 s3://test
>>>>>>> 2018-02-01 11:44 s3://test2
>>>>>>> 2018-02-02 17:10 s3://test3
>>>>>>>
>>>>>>>
>>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3
>>>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt
>>>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg
>>>>>>>
>>>>>>> [@ ganesha]# rpm -qa | grep ganesh
>>>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64
>>>>>>> nfs-ganesha-2.5.5-.el7.x86_64
>>>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64
>>>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64
>>>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64
>>>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64
>>>>>>>
>>>>>>> On CentOS7
>>>>>>> _______________________________________________
>>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>>>> unsubscribe
>>>>
>>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>>> unsubscribe
>>>
>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>>
>>>>>> _______________________________________________
>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>>> unsubscribe
>>>
>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>>
>>>>> _______________________________________________
>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To
>>>> unsubscribe
>
>>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>>
>>> _______________________________________________
>>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>>> send an email to support-leave(a)lists.nfs-ganesha.org
>>>
>>>
>>
>
>
>
9:47 p.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many ways
this can happen; unfortunately, the logging in 2.5 is a bit out of date,
so I can't tell which issue hit. Can you run the ls with FULL_DEBUG on
for everything? That should give me enough output to figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
11:38 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many ways
this can happen; unfortunately, the logging in 2.5 is a bit out of date,
so I can't tell which issue hit. Can you run the ls with FULL_DEBUG on
for everything? That should give me enough output to figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
12:22 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Sure I will give it a try, this builds correctly.
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init --recursive
cmake src
make
Although I am missing the configure, I thought I would build statically
linked, so it is easier to copy to the ceph server. (compiling on a
different node)
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many
ways this can happen; unfortunately, the logging in 2.5 is a bit out
of date, so I can't tell which issue hit. Can you run the ls with
FULL_DEBUG on for everything? That should give me enough output to
figure it
out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
2:26 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Maybe some pointers on compiling rgw could be helpful
cmake -DUSE_FSAL_RGW=ON src/
-- Found rgw libraries: /usr/lib64/librgw.so
CMake Error at cmake/modules/FindPackageHandleStandardArgs.cmake:109
(message):
Could NOT find RGW: Found unsuitable version "0.0.0", but required is
at
least "1.1.6" (found )
Call Stack (most recent call first):
cmake/modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindRGW.cmake:95 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:699 (find_package)
[@svr nfs-ganesha]$ rpm -qa | egrep '(rgw|rados|ceph)'
libcephfs-devel-12.2.5-0.el7.x86_64
librados-devel-12.2.5-0.el7.x86_64
librgw-devel-12.2.5-0.el7.x86_64
librgw2-12.2.5-0.el7.x86_64
libcephfs2-12.2.5-0.el7.x86_64
librados2-12.2.5-0.el7.x86_64
-----Original Message-----
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Sure I will give it a try, this builds correctly.
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init --recursive
cmake src
make
Although I am missing the configure, I thought I would build statically
linked, so it is easier to copy to the ceph server. (compiling on a
different node)
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many
ways this can happen; unfortunately, the logging in 2.5 is a bit out
of date, so I can't tell which issue hit. Can you run the ls with
FULL_DEBUG on for everything? That should give me enough output to
figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
8:52 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
It seems like path issue. FindRGW is basically looking for rados/rgw_file.h and reading
the version number from there.
There should be a message "FindRGW: using PATH HINT:". You may need to set the
path to correct location where
librgw-devel is installing rados/rgw_flie.h. In my environment it is at
/usr/include/rados. Also, you should ensure that
you have compiling with right version of librgw.
------
Supriti Singh SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
>> "Marc Roos" <M.Roos(a)f1-outsourcing.eu>
05/04/18 2:47 PM >>>
Maybe some pointers on compiling rgw could be helpful
cmake -DUSE_FSAL_RGW=ON src/
-- Found rgw libraries: /usr/lib64/librgw.so
CMake Error at cmake/modules/FindPackageHandleStandardArgs.cmake:109
(message):
Could NOT find RGW: Found unsuitable version "0.0.0", but required is
at
least "1.1.6" (found )
Call Stack (most recent call first):
cmake/modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindRGW.cmake:95 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:699 (find_package)
[@svr nfs-ganesha]$ rpm -qa | egrep '(rgw|rados|ceph)'
libcephfs-devel-12.2.5-0.el7.x86_64
librados-devel-12.2.5-0.el7.x86_64
librgw-devel-12.2.5-0.el7.x86_64
librgw2-12.2.5-0.el7.x86_64
libcephfs2-12.2.5-0.el7.x86_64
librados2-12.2.5-0.el7.x86_64
-----Original Message-----
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Sure I will give it a try, this builds correctly.
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init --recursive
cmake src
make
Although I am missing the configure, I thought I would build statically
linked, so it is easier to copy to the ceph server. (compiling on a
different node)
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many
ways this can happen; unfortunately, the logging in 2.5 is a bit out
of date, so I can't tell which issue hit. Can you run the ls with
FULL_DEBUG on for everything? That should give me enough output to
figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
12:53 p.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I just wanted write I don’t know what I am doing, never used cmake,
can't find a simple command line to add an include or library path, I
added in src/CMakeLists.txt
include_directories("/usr/include/rados")
But I get now other errors about NLM and I just disabled it. Now it
looks like it is building. I hope this is enough to apply the patch.
-----Original Message-----
From: Supriti Singh [mailto:Supriti.Singh@suse.com]
Sent: vrijdag 4 mei 2018 15:52
To: Marc Roos; support(a)lists.nfs-ganesha.org; dang(a)redhat.com
Cc: mbenjamin(a)redhat.com
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
It seems like path issue. FindRGW is basically looking for
rados/rgw_file.h and reading the version number from there. There
should be a message "FindRGW: using PATH HINT:". You may need to set the
path to correct location where librgw-devel is installing
rados/rgw_flie.h. In my environment it is at /usr/include/rados. Also,
you should ensure that you have compiling with right version of librgw.
------
Supriti Singh
SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton,
HRB 21284 (AG Nürnberg)
>> "Marc Roos" <M.Roos(a)f1-outsourcing.eu>
05/04/18 2:47 PM >>>
Maybe some pointers on compiling rgw could be helpful
cmake -DUSE_FSAL_RGW=ON src/
-- Found rgw libraries: /usr/lib64/librgw.so CMake Error at
cmake/modules/FindPackageHandleStandardArgs.cmake:109
(message):
Could NOT find RGW: Found unsuitable version "0.0.0", but required is at
least "1.1.6" (found ) Call Stack (most recent call first):
cmake/modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindRGW.cmake:95 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:699 (find_package)
[@svr nfs-ganesha]$ rpm -qa | egrep '(rgw|rados|ceph)'
libcephfs-devel-12.2.5-0.el7.x86_64
librados-devel-12.2.5-0.el7.x86_64
librgw-devel-12.2.5-0.el7.x86_64
librgw2-12.2.5-0.el7.x86_64
libcephfs2-12.2.5-0.el7.x86_64
librados2-12.2.5-0.el7.x86_64
-----Original Message-----
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Sure I will give it a try, this builds correctly.
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init --recursive
cmake src
make
Although I am missing the configure, I thought I would build statically
linked, so it is easier to copy to the ceph server. (compiling on a
different node)
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many
ways this can happen; unfortunately, the logging in 2.5 is a bit out
of date, so I can't tell which issue hit. Can you run the ls with
FULL_DEBUG on for everything? That should give me enough output to
figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
2:48 a.m.
New subject: [Nfs-ganesha-support] Nfs-ganesha rgw move/rename files and folders
I was wondering since ganesha rgw is using rados directly, it would be
'easier' and more 'efficient' in renaming files and folders?
I don’t get why files that are moved are copied and then deleted, I
take it the file name is just some meta data stored somewhere that can
be easily changed.
7:25 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
So, it works for me on Fedora with 12.2.4. All that should be necessary
is librgw-devel and it's dependencies. You shouldn't need any paths,
since librgw is installed in the system paths, which are searched by
default. All I did was this:
cmake -D USE_FSAL_RGW=ON ../nfs-ganesha/src
What it's looking for is LIBRGW_VER_* in /usr/include/rados/librgw.h.
Mine looks like this:
#define LIBRGW_VER_MAJOR 1
#define LIBRGW_VER_MINOR 1
#define LIBRGW_VER_EXTRA 0
Do you have similar defines in that file? Maybe your packaging is
messed up?
Daniel
On 05/04/2018 03:26 AM, Marc Roos wrote:
Maybe some pointers on compiling rgw could be helpful
cmake -DUSE_FSAL_RGW=ON src/
-- Found rgw libraries: /usr/lib64/librgw.so
CMake Error at cmake/modules/FindPackageHandleStandardArgs.cmake:109
(message):
Could NOT find RGW: Found unsuitable version "0.0.0", but required is
at
least "1.1.6" (found )
Call Stack (most recent call first):
cmake/modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindRGW.cmake:95 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:699 (find_package)
[@svr nfs-ganesha]$ rpm -qa | egrep '(rgw|rados|ceph)'
libcephfs-devel-12.2.5-0.el7.x86_64
librados-devel-12.2.5-0.el7.x86_64
librgw-devel-12.2.5-0.el7.x86_64
librgw2-12.2.5-0.el7.x86_64
libcephfs2-12.2.5-0.el7.x86_64
librados2-12.2.5-0.el7.x86_64
-----Original Message-----
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Sure I will give it a try, this builds correctly.
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init --recursive
cmake src
make
Although I am missing the configure, I thought I would build statically
linked, so it is easier to copy to the ceph server. (compiling on a
different node)
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
>
> Still on luminous 12.2.4, attached the log when the ls was done. I
> forgot to turn of the radosgw that also uses the client.rgw3, that
> should not be a problem not?
>
> client.rgw3
> caps: [mon] allow rwx
> caps: [osd] allow rwx
>
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: woensdag 2 mei 2018 18:48
> To: Marc Roos; support
> Cc: mbenjamin
> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
> multi tenancy rgw users
>
> So, the READDIR op got a SERVERFAULT error. There aren't very many
> ways this can happen; unfortunately, the logging in 2.5 is a bit out
> of date, so I can't tell which issue hit. Can you run the ls with
> FULL_DEBUG on for everything? That should give me enough output to
figure it out.
>
> Also, which version of ceph are you using on the Ganesha box?
>
> Thanks,
> Daniel
>
>
>
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org
To unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org
10:18 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Yes I already managed to get it to compile. I saw this
./src/cmake/build_configurations/rpmbuild.cmake. Is this to create the
rpms? How do I use it?
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 7 mei 2018 14:25
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
So, it works for me on Fedora with 12.2.4. All that should be necessary
is librgw-devel and it's dependencies. You shouldn't need any paths,
since librgw is installed in the system paths, which are searched by
default. All I did was this:
cmake -D USE_FSAL_RGW=ON ../nfs-ganesha/src
What it's looking for is LIBRGW_VER_* in /usr/include/rados/librgw.h.
Mine looks like this:
#define LIBRGW_VER_MAJOR 1
#define LIBRGW_VER_MINOR 1
#define LIBRGW_VER_EXTRA 0
Do you have similar defines in that file? Maybe your packaging is
messed up?
Daniel
On 05/04/2018 03:26 AM, Marc Roos wrote:
Maybe some pointers on compiling rgw could be helpful
cmake -DUSE_FSAL_RGW=ON src/
-- Found rgw libraries: /usr/lib64/librgw.so CMake Error at
cmake/modules/FindPackageHandleStandardArgs.cmake:109
(message):
Could NOT find RGW: Found unsuitable version "0.0.0", but required
is at
least "1.1.6" (found )
Call Stack (most recent call first):
cmake/modules/FindPackageHandleStandardArgs.cmake:315
(_FPHSA_FAILURE_MESSAGE)
cmake/modules/FindRGW.cmake:95 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
CMakeLists.txt:699 (find_package)
[@svr nfs-ganesha]$ rpm -qa | egrep '(rgw|rados|ceph)'
libcephfs-devel-12.2.5-0.el7.x86_64
librados-devel-12.2.5-0.el7.x86_64
librgw-devel-12.2.5-0.el7.x86_64
librgw2-12.2.5-0.el7.x86_64
libcephfs2-12.2.5-0.el7.x86_64
librados2-12.2.5-0.el7.x86_64
-----Original Message-----
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Sure I will give it a try, this builds correctly.
git clone https://github.com/nfs-ganesha/nfs-ganesha.git
cd nfs-ganesha
git submodule update --init --recursive cmake src make
Although I am missing the configure, I thought I would build
statically linked, so it is easier to copy to the ceph server.
(compiling on a different node)
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a
quick
look.
Can you compile Ganesha from source? If so, I can send you a patch
that will print the actual error returned from ceph. This might help
us track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
>
> Still on luminous 12.2.4, attached the log when the ls was done. I
> forgot to turn of the radosgw that also uses the client.rgw3, that
> should not be a problem not?
>
> client.rgw3
> caps: [mon] allow rwx
> caps: [osd] allow rwx
>
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: woensdag 2 mei 2018 18:48
> To: Marc Roos; support
> Cc: mbenjamin
> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
> multi tenancy rgw users
>
> So, the READDIR op got a SERVERFAULT error. There aren't very many
> ways this can happen; unfortunately, the logging in 2.5 is a bit out
> of date, so I can't tell which issue hit. Can you run the ls with
> FULL_DEBUG on for everything? That should give me enough output to
figure it out.
>
> Also, which version of ceph are you using on the Ganesha box?
>
> Thanks,
> Daniel
>
>
>
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
8:50 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
To create RPMs, just run cmake with the options you want, and then:
make rpm
The result will be in your default rpmbuild root (~/rpmbuild by default)
Daniel
On 05/07/2018 11:18 AM, Marc Roos wrote:
Yes I already managed to get it to compile. I saw this
./src/cmake/build_configurations/rpmbuild.cmake. Is this to create the
rpms? How do I use it?
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: maandag 7 mei 2018 14:25
To: support(a)lists.nfs-ganesha.org
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
So, it works for me on Fedora with 12.2.4. All that should be necessary
is librgw-devel and it's dependencies. You shouldn't need any paths,
since librgw is installed in the system paths, which are searched by
default. All I did was this:
cmake -D USE_FSAL_RGW=ON ../nfs-ganesha/src
What it's looking for is LIBRGW_VER_* in /usr/include/rados/librgw.h.
Mine looks like this:
#define LIBRGW_VER_MAJOR 1
#define LIBRGW_VER_MINOR 1
#define LIBRGW_VER_EXTRA 0
Do you have similar defines in that file? Maybe your packaging is
messed up?
Daniel
On 05/04/2018 03:26 AM, Marc Roos wrote:
>
> Maybe some pointers on compiling rgw could be helpful
>
> cmake -DUSE_FSAL_RGW=ON src/
>
> -- Found rgw libraries: /usr/lib64/librgw.so CMake Error at
> cmake/modules/FindPackageHandleStandardArgs.cmake:109
> (message):
> Could NOT find RGW: Found unsuitable version "0.0.0", but required
> is at
> least "1.1.6" (found )
> Call Stack (most recent call first):
> cmake/modules/FindPackageHandleStandardArgs.cmake:315
> (_FPHSA_FAILURE_MESSAGE)
> cmake/modules/FindRGW.cmake:95 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
> CMakeLists.txt:699 (find_package)
>
> [@svr nfs-ganesha]$ rpm -qa | egrep '(rgw|rados|ceph)'
> libcephfs-devel-12.2.5-0.el7.x86_64
> librados-devel-12.2.5-0.el7.x86_64
> librgw-devel-12.2.5-0.el7.x86_64
> librgw2-12.2.5-0.el7.x86_64
> libcephfs2-12.2.5-0.el7.x86_64
> librados2-12.2.5-0.el7.x86_64
>
> -----Original Message-----
> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
> tenancy rgw users
>
>
>
> Sure I will give it a try, this builds correctly.
>
> git clone https://github.com/nfs-ganesha/nfs-ganesha.git
> cd nfs-ganesha
> git submodule update --init --recursive cmake src make
>
> Although I am missing the configure, I thought I would build
> statically linked, so it is easier to copy to the ceph server.
> (compiling on a different node)
>
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: donderdag 3 mei 2018 18:38
> To: Marc Roos; support
> Cc: mbenjamin
> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
> multi tenancy rgw users
>
> Okay, so I've looked through this, and librgw is returning a readdir
> error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
> large, and there's at least several things it could be on just a quick
> look.
>
> Can you compile Ganesha from source? If so, I can send you a patch
> that will print the actual error returned from ceph. This might help
> us track it down.
>
> Daniel
>
> On 05/02/2018 10:47 PM, Marc Roos wrote:
>>
>> Still on luminous 12.2.4, attached the log when the ls was done. I
>> forgot to turn of the radosgw that also uses the client.rgw3, that
>> should not be a problem not?
>>
>> client.rgw3
>> caps: [mon] allow rwx
>> caps: [osd] allow rwx
>>
>>
>>
>>
>> -----Original Message-----
>> From: Daniel Gryniewicz [mailto:dang@redhat.com]
>> Sent: woensdag 2 mei 2018 18:48
>> To: Marc Roos; support
>> Cc: mbenjamin
>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
>> multi tenancy rgw users
>>
>> So, the READDIR op got a SERVERFAULT error. There aren't very many
>> ways this can happen; unfortunately, the logging in 2.5 is a bit out
>> of date, so I can't tell which issue hit. Can you run the ls with
>> FULL_DEBUG on for everything? That should give me enough output to
> figure it out.
>>
>> Also, which version of ceph are you using on the Ganesha box?
>>
>> Thanks,
>> Daniel
>>
>>
>>
>>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
>> send an email to support-leave(a)lists.nfs-ganesha.org
>>
>
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
11:34 p.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Daniel,
Do you still want me to test with the patch. I just upgraded to 2.6. and
have luminous on 12.2.5
Thanks,
Marc
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many
ways this can happen; unfortunately, the logging in 2.5 is a bit out
of date, so I can't tell which issue hit. Can you run the ls with
FULL_DEBUG on for everything? That should give me enough output to
figure it
out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
7:29 a.m.
New subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Here's a simple patch to trace the actual error. It will trace at EVENT
level, so no need to run it a FULL_DEBUG
Daniel
On 05/11/2018 12:34 AM, Marc Roos wrote:
Daniel,
Do you still want me to test with the patch. I just upgraded to 2.6. and
have luminous on 12.2.5
Thanks,
Marc
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
>
> Still on luminous 12.2.4, attached the log when the ls was done. I
> forgot to turn of the radosgw that also uses the client.rgw3, that
> should not be a problem not?
>
> client.rgw3
> caps: [mon] allow rwx
> caps: [osd] allow rwx
>
>
>
>
> -----Original Message-----
> From: Daniel Gryniewicz [mailto:dang@redhat.com]
> Sent: woensdag 2 mei 2018 18:48
> To: Marc Roos; support
> Cc: mbenjamin
> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
> multi tenancy rgw users
>
> So, the READDIR op got a SERVERFAULT error. There aren't very many
> ways this can happen; unfortunately, the logging in 2.5 is a bit out
> of date, so I can't tell which issue hit. Can you run the ls with
> FULL_DEBUG on for everything? That should give me enough output to
figure it out.
>
> Also, which version of ceph are you using on the Ganesha box?
>
> Thanks,
> Daniel
>
>
>
>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
> send an email to support-leave(a)lists.nfs-ganesha.org
>
10:50 a.m.
New subject: [Nfs-Ganesha-Support] Nfs-ganesha rgw multi user
Is anyone using nfs-ganesha in a rgw multi user / tenant environment?
I recently upgraded to nfs-ganesha 2.6 / luminous 12.2.7
-----Original Message-----
From: Marc Roos
Sent: vrijdag 11 mei 2018 6:34
To: dang; support
Cc: mbenjamin
Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Daniel,
Do you still want me to test with the patch. I just upgraded to 2.6. and
have luminous on 12.2.5
Thanks,
Marc
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: donderdag 3 mei 2018 18:38
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Okay, so I've looked through this, and librgw is returning a readdir
error that Ganesha doesn't recognize. Unfortunately, the Ceph code is
large, and there's at least several things it could be on just a quick
look.
Can you compile Ganesha from source? If so, I can send you a patch that
will print the actual error returned from ceph. This might help us
track it down.
Daniel
On 05/02/2018 10:47 PM, Marc Roos wrote:
Still on luminous 12.2.4, attached the log when the ls was done. I
forgot to turn of the radosgw that also uses the client.rgw3, that
should not be a problem not?
client.rgw3
caps: [mon] allow rwx
caps: [osd] allow rwx
-----Original Message-----
From: Daniel Gryniewicz [mailto:dang@redhat.com]
Sent: woensdag 2 mei 2018 18:48
To: Marc Roos; support
Cc: mbenjamin
Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for
multi tenancy rgw users
So, the READDIR op got a SERVERFAULT error. There aren't very many
ways this can happen; unfortunately, the logging in 2.5 is a bit out
of date, so I can't tell which issue hit. Can you run the ls with
FULL_DEBUG on for everything? That should give me enough output to
figure it out.
Also, which version of ceph are you using on the Ganesha box?
Thanks,
Daniel
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
_______________________________________________
Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe
send an email to support-leave(a)lists.nfs-ganesha.org
2330
days inactive
2438
days old
34 comments
5 participants
participants (5)
-
Daniel Gryniewicz -
Frank Filz -
Marc Roos -
Supriti Singh -
TomK