I stopped the rgw3 instance that is running on the same server (I am not sure anymore if I can run radosgw and nfs-ganesha with the same client.rgw3), started ganesha with this config. I am correct to understand that the ganesha-rgw is not using the http(s) interface and directly connects to ceph with librados (because that is why we have the RGW config section)? RGW { cluster = "ceph"; name = "client.rgw3"; ceph_conf = "/etc/ceph/ceph.conf"; } EXPORT { Export_ID=301; Path = "test:test3"; Pseudo = "/rgwtester"; FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id ="xxxxxx"; Secret_Access_Key = "xxxxxx"; } Disable_ACL = TRUE; CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } } Logs of ganesha related to test3 bucket [@]# tail -1000 /var/log/messages | grep ganesh | grep test3 Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO :Releasing RGW export for test:test3 Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO :Releasing RGW export for test:test3 Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test:test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test:test3) and tag ((null)) perms (options=03303002 , , , , , , , ) [@]# showmount -e nfs Export list for nfs: /ram 192.168.x.0 /test 192.168.x.0 /rgwtester 192.168.x.0 [@ mnt]# mount nfs:/rgwtester /mnt/nfstest [@ mnt]# mount proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ... nfs:/test on /home/users/robbert/nfs-test type nfs4 (rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,naml en=255,hard,addr= nfs:/rgwtester on /mnt/nfstest type nfs (rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x) [@ mnt]# ls -l /mnt/nfstest/ ls: reading directory /mnt/nfstest/: Remote I/O error total 0 -----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 20:06 To: Marc Roos; support Cc: Matt Benjamin Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users Okay, so on consultation, we think that maybe you should be mounting test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the mail thread if you tried that; can you confirm? (I have no personal experience with multi-tenant, so I didn't recognize it as such). In the mean time, we'll try to reproduce here and see what we can see. Daniel On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote: > All of that looks fine, I think. I'm out of my depth here, so I think > Matt needs to chime in here. > > Matt? > > On 04/30/2018 04:06 AM, Marc Roos wrote: >> Could it not be, because of a configuration issue in ganesha.conf or >> the version I am using? I can hardly imagine that I am the only one >> running a multitenant rgw config with ganesha-rgw. >> >> nfs-ganesha-xfs-2.5.5-.el7.x86_64 >> nfs-ganesha-2.5.5-.el7.x86_64 >> nfs-ganesha-rgw-2.5.5-.el7.x86_64 >> nfs-ganesha-ceph-2.5.5-.el7.x86_64 >> nfs-ganesha-vfs-2.5.5-.el7.x86_64 >> nfs-ganesha-mem-2.5.5-.el7.x86_64 >> >> >> >> >> -----Original Message----- >> From: Daniel Gryniewicz [mailto:dang@redhat.com] >> Sent: maandag 30 april 2018 15:15 >> To: support(a)lists.nfs-ganesha.org >> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi >> tenancy rgw users >> >> And test$tester1 is the user ID you created, that is this gives output: >> >> radosgw-admin user info --uid="test$tester1" >> >> (I'm pounding on this because it really seems that the UID doesn't >> have permissions. If it does, I think there's some really strange >> bug here that's going to be hard to track down.) >> >> Daniel >> >> >> On 04/30/2018 03:25 AM, Marc Roos wrote: >>> Yes same access key and secret access key ( no uid is used ) >>> >>> -----Original Message----- >>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>> Sent: maandag 30 april 2018 14:40 >>> To: Marc Roos; support >>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>> multi tenancy rgw users >>> >>> I mean does the s3cmd (or cyberduck) use the exact same credentials >>> as >> >>> are specified in the ganesha.conf? >>> >>> Daniel >>> >>> On 04/26/2018 10:09 AM, Marc Roos wrote: >>>> >>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are >>>> you refering to something else? >>>> >>>> >>>> >>>> -----Original Message----- >>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>> Sent: donderdag 26 april 2018 15:38 >>>> To: support(a)lists.nfs-ganesha.org >>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi >>>> tenancy rgw users >>>> >>>> Does test$tester1 have permissions to access RGW? Mounting / >>>> should see all the buckets, so it's likely a permission issue. >>>> >>>> Daniel >>>> >>>> On 04/26/2018 03:56 AM, Marc Roos wrote: >>>>> >>>>> I have the same as with / path, I can do the mount, but it is just >>>>> empty >>>>> >>>>> [@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/ >>>>> ls: reading directory nfstest/: Remote I/O error total 0 >>>>> >>>>> >>>>> >>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>> export_commit_common >>> >>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path >>>>> (test3) and tag ((null)) perms (options=03303002 , >> , >>>>> , , , , , >>>>> ) >>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>> export_commit_common >>> >>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path >>>>> (test3) and tag ((null)) perms (options=03303002 , >> , >>>>> , , , , , >>>>> ) >>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>> export_commit_common >>> >>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 >>>>> c01 >>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO >>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01 >>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO >>>>> :Export >>> >>>>> 0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]: >>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully >>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] >>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 >>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] >>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 >>>>> ngroups=0 >>>>> >>>>> -----Original Message----- >>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>>> Sent: donderdag 26 april 2018 14:47 >>>>> To: support(a)lists.nfs-ganesha.org >>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>>>> multi tenancy rgw users >>>>> >>>>> I think it should be: >>>>> >>>>> Path="test3" >>>>> >>>>> so without the initial '/'. Buckets aren't paths, we're just >>>>> abusing >>> >>>>> the existing Ganesha config to select them. >>>>> >>>>> Other than that, the client entry should match the equivalent >>>>> entry in >>>> >>>>> your ceph.conf. >>>>> >>>>> Daniel >>>>> >>>>> On 04/26/2018 01:52 AM, Marc Roos wrote: >>>>>> >>>>>> >>>>>> I have problems exporting a bucket that really does exist. I have >>>>>> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha >>>>>> fails >>>> >>>>>> to start with message ExportId=301 Path=/test:test3 >>>>>> FSAL_ERROR=(Invalid object >>>>>> type,0) >>>>>> >>>>>> If I use path=/ I can mount something but that is a empty export, >>>>>> but >>>> >>>>>> cannot put files there. >>>>>> >>>>>> I have this in ganesha (client.rwg3 works with civetweb), should >>>>>> be >> >>>>>> sufficient not? >>>>>> >>>>>> RGW { >>>>>> cluster = "ceph"; >>>>>> name = "client.rgw3"; >>>>>> ceph_conf = "/etc/ceph/ceph.conf"; >>>>>> # for vstart cluster, name = "client.admin" >>>>>> #init_args = "-d --debug-rgw=16"; } >>>>>> >>>>>> EXPORT { >>>>>> Export_ID=301; >>>>>> Path = "/test3"; >>>>>> Pseudo = "/rgwtester"; >>>>>> >>>>>> FSAL { Name = RGW; User_Id = "test$tester1"; >>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key = "sameass3"; } >>>>>> Disable_ACL = FALSE; >>>>>> CLIENT { Clients = 192.168.x.0/24; } } >>>>>> >>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls >>>>>> 2018-01-31 21:48 s3://test >>>>>> 2018-02-01 11:44 s3://test2 >>>>>> 2018-02-02 17:10 s3://test3 >>>>>> >>>>>> >>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3 >>>>>> 2018-02-02 17:13 10485760 s3://test3/10MB.txt >>>>>> 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg >>>>>> >>>>>> [@ ganesha]# rpm -qa | grep ganesh >>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64 >>>>>> nfs-ganesha-2.5.5-.el7.x86_64 >>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64 >>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64 >>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64 >>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64 >>>>>> >>>>>> On CentOS7 >>>>>> _______________________________________________ >>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>> unsubscribe >>> >>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>> >>>>> _______________________________________________ >>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>> unsubscribe >> >>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>> >>>>> _______________________________________________ >>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>> unsubscribe >> >>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>> >>>> _______________________________________________ >>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>> >>>> >>> >>> >>> _______________________________________________ >>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe >>> send an email to support-leave(a)lists.nfs-ganesha.org >>> >> _______________________________________________ >> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe >> send an email to support-leave(a)lists.nfs-ganesha.org >> >> >