[Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users

Attached the logs mount.log, ls.log and unmount.log With this config: ## configured from the command line LOG { ## Default log level for all components # NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG, M_DBG, FULL_DEBUG, F_DBG], default EVENT default_log_level = FATAL; ## Configure per-component log levels. # ALL, LOG,LOG_EMERG,MEMLEAKS,FSAL,NFSPROTO(NFS3),NFS_V4(NFSV4),EXPORT,FILEHAND LE,DISPATCH,CACHE_INODE, # CACHE_INODE_LRU,HASHTABLE,HASHTABLE_CACHE,DUPREQ,INIT,MAIN, IDMAPPER,NFS_READDIR,NFS_V4_LOCK,CONFIG,CLIENTID, # SESSIONS,PNFS,RW_LOCK,NLM,RPC,NFS_CB,THREAD,NFS_V4_ACL,STATE,9P,9P_DISPA TCH,FSAL_UP,DBUS Components { #ALL = FATAL; FSAL = DEBUG; NFS4 = DEBUG; EXPORT = F_DBG; CONFIG = F_DBG; } ## Where to log Facility { name = FILE; destination = "/var/log/ganesha.log"; enable = default; } } PS. The DEBUG level is not working with syslog. I have to enable the local logging. -----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: dinsdag 1 mei 2018 14:11 To: Marc Roos; support Cc: mbenjamin Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users Okay, so that looks like Ganesha thinks the export is working, but the actual I/O on the bucket is failing. Can you turn on DEBUG for the NFS4 and FSAL components, perform this same simple test (mount + ls + umount) and post the ganesha log? This can be done like this: LOG { Components { NFS4 = DEBUG; FSAL = DEBUG; } } Daniel On 04/30/2018 11:29 AM, Marc Roos wrote: > > I stopped the rgw3 instance that is running on the same server (I am > not sure anymore if I can run radosgw and nfs-ganesha with the same > client.rgw3), started ganesha with this config. I am correct to > understand that the ganesha-rgw is not using the http(s) interface and > directly connects to ceph with librados (because that is why we have > the RGW config section)? > > RGW { > cluster = "ceph"; > name = "client.rgw3"; > ceph_conf = "/etc/ceph/ceph.conf"; } > > EXPORT { > Export_ID=301; > Path = "test:test3"; > Pseudo = "/rgwtester"; > > FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id > ="xxxxxx"; Secret_Access_Key = "xxxxxx"; } > Disable_ACL = TRUE; > CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } } > > > Logs of ganesha related to test3 bucket > > [@]# tail -1000 /var/log/messages | grep ganesh | grep test3 Apr 30 > 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL > :INFO :Releasing RGW export for test:test3 Apr 30 22:24:55 c01 > nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO > :Releasing RGW export for test:test3 Apr 30 22:27:07 c01 > nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export > 301 created at pseudo (/rgwtester) with path > (test:test3) and tag ((null)) perms (options=03303002 , > , , , , , , > ) > Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common > :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path > (test:test3) and tag ((null)) perms (options=03303002 , > , , , , , , > ) > > > [@]# showmount -e nfs > Export list for nfs: > /ram 192.168.x.0 > /test 192.168.x.0 > /rgwtester 192.168.x.0 > > [@ mnt]# mount nfs:/rgwtester /mnt/nfstest > > [@ mnt]# mount > proc on /proc type proc (rw) > sysfs on /sys type sysfs (rw) > devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm > type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ... > nfs:/test on /home/users/robbert/nfs-test type nfs4 > (rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,na > ml > en=255,hard,addr= > nfs:/rgwtester on /mnt/nfstest type nfs > (rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x) > > [@ mnt]# ls -l /mnt/nfstest/ > ls: reading directory /mnt/nfstest/: Remote I/O error total 0 > > > > -----Original Message----- > From: Daniel Gryniewicz [mailto:dang@redhat.com] > Sent: maandag 30 april 2018 20:06 > To: Marc Roos; support > Cc: Matt Benjamin > Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for > multi tenancy rgw users > > Okay, so on consultation, we think that maybe you should be mounting > test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the > mail thread if you tried that; can you confirm? > > (I have no personal experience with multi-tenant, so I didn't > recognize it as such). > > In the mean time, we'll try to reproduce here and see what we can see. > > Daniel > > On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote: >> All of that looks fine, I think.  I'm out of my depth here, so I >> think > >> Matt needs to chime in here. >> >> Matt? >> >> On 04/30/2018 04:06 AM, Marc Roos wrote: >>> Could it not be, because of a configuration issue in ganesha.conf or >>> the version I am using? I can hardly imagine that I am the only one >>> running a multitenant rgw config with ganesha-rgw. >>> >>> nfs-ganesha-xfs-2.5.5-.el7.x86_64 >>> nfs-ganesha-2.5.5-.el7.x86_64 >>> nfs-ganesha-rgw-2.5.5-.el7.x86_64 >>> nfs-ganesha-ceph-2.5.5-.el7.x86_64 >>> nfs-ganesha-vfs-2.5.5-.el7.x86_64 >>> nfs-ganesha-mem-2.5.5-.el7.x86_64 >>> >>> >>> >>> >>> -----Original Message----- >>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>> Sent: maandag 30 april 2018 15:15 >>> To: support(a)lists.nfs-ganesha.org >>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi >>> tenancy rgw users >>> >>> And test$tester1 is the user ID you created, that is this gives > output: >>> >>> radosgw-admin user info --uid="test$tester1" >>> >>> (I'm pounding on this because it really seems that the UID doesn't >>> have permissions.  If it does, I think there's some really strange >>> bug here that's going to be hard to track down.) >>> >>> Daniel >>> >>> >>> On 04/30/2018 03:25 AM, Marc Roos wrote: >>>> Yes same access key and secret access key ( no uid is used ) >>>> >>>> -----Original Message----- >>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>> Sent: maandag 30 april 2018 14:40 >>>> To: Marc Roos; support >>>> Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>>> multi tenancy rgw users >>>> >>>> I mean does the s3cmd (or cyberduck) use the exact same credentials >>>> as >>> >>>> are specified in the ganesha.conf? >>>> >>>> Daniel >>>> >>>> On 04/26/2018 10:09 AM, Marc Roos wrote: >>>>> >>>>> Yes, I can access the bucket via the s3cmd and cyberduck, or are >>>>> you refering to something else? >>>>> >>>>> >>>>> >>>>> -----Original Message----- >>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>>> Sent: donderdag 26 april 2018 15:38 >>>>> To: support(a)lists.nfs-ganesha.org >>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>>>> multi > >>>>> tenancy rgw users >>>>> >>>>> Does test$tester1 have permissions to access RGW?  Mounting / >>>>> should see all the buckets, so it's likely a permission issue. >>>>> >>>>> Daniel >>>>> >>>>> On 04/26/2018 03:56 AM, Marc Roos wrote: >>>>>> >>>>>> I have the same as with / path, I can do the mount, but it is >>>>>> just > >>>>>> empty >>>>>> >>>>>> [@mnt]#  mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/ >>>>>> ls: reading directory nfstest/: Remote I/O error total 0 >>>>>> >>>>>> >>>>>> >>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>>> export_commit_common >>>> >>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with >>>>>> path >>>>>> (test3) and tag ((null)) perms (options=03303002              , >>> , >>>>>> ,               ,               ,         ,                , >>>>>>        ) >>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>>> export_commit_common >>>> >>>>>> :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with >>>>>> path >>>>>> (test3) and tag ((null)) perms (options=03303002              , >>> , >>>>>> ,               ,               ,         ,                , >>>>>>        ) >>>>>> Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] >>>>>> export_commit_common >>>> >>>>>> :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 >>>>>> c01 >>>>>> nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO >>>>>> :Export 301 has 1 defined clients Apr 26 14:56:15 c01 >>>>>> nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO >>>>>> :Export >>>> >>>>>> 0 (/) successfully created Apr 26 14:56:15 c01 > nfs-ganesha[3981596]: >>>>>> [main] build_default_root :CONFIG :INFO :Export 0 (/) >>>>>> successfully > >>>>>> created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] >>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 >>>>>> ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] >>>>>> fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 >>>>>> ngroups=0 >>>>>> >>>>>> -----Original Message----- >>>>>> From: Daniel Gryniewicz [mailto:dang@redhat.com] >>>>>> Sent: donderdag 26 april 2018 14:47 >>>>>> To: support(a)lists.nfs-ganesha.org >>>>>> Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for >>>>>> multi tenancy rgw users >>>>>> >>>>>> I think it should be: >>>>>> >>>>>> Path="test3" >>>>>> >>>>>> so without the initial '/'.  Buckets aren't paths, we're just >>>>>> abusing >>>> >>>>>> the existing Ganesha config to select them. >>>>>> >>>>>> Other than that, the client entry should match the equivalent >>>>>> entry in >>>>> >>>>>> your ceph.conf. >>>>>> >>>>>> Daniel >>>>>> >>>>>> On 04/26/2018 01:52 AM, Marc Roos wrote: >>>>>>> >>>>>>> >>>>>>> I have problems exporting a bucket that really does exist. I >>>>>>> have > >>>>>>> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha >>>>>>> fails >>>>> >>>>>>> to start with message ExportId=301 Path=/test:test3 >>>>>>> FSAL_ERROR=(Invalid object >>>>>>> type,0) >>>>>>> >>>>>>> If I use path=/ I can mount something but that is a empty >>>>>>> export, > >>>>>>> but >>>>> >>>>>>> cannot put files there. >>>>>>> >>>>>>> I have this in ganesha (client.rwg3 works with civetweb), should >>>>>>> be >>> >>>>>>> sufficient not? >>>>>>> >>>>>>> RGW { >>>>>>>              cluster = "ceph"; >>>>>>>              name = "client.rgw3"; >>>>>>>              ceph_conf = "/etc/ceph/ceph.conf"; >>>>>>>              # for vstart cluster, name = "client.admin" >>>>>>>              #init_args = "-d --debug-rgw=16"; } >>>>>>> >>>>>>> EXPORT { >>>>>>>              Export_ID=301; >>>>>>>              Path = "/test3"; >>>>>>>              Pseudo = "/rgwtester"; >>>>>>> >>>>>>>              FSAL { Name = RGW; User_Id = "test$tester1"; >>>>>>> Access_Key_Id ="sameass3"; Secret_Access_Key = "sameass3"; } >>>>>>>              Disable_ACL = FALSE; >>>>>>>              CLIENT { Clients = 192.168.x.0/24; } } >>>>>>> >>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls >>>>>>> 2018-01-31 21:48  s3://test >>>>>>> 2018-02-01 11:44  s3://test2 >>>>>>> 2018-02-02 17:10  s3://test3 >>>>>>> >>>>>>> >>>>>>> [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3 >>>>>>> 2018-02-02 17:13  10485760   s3://test3/10MB.txt >>>>>>> 2018-02-05 12:57     26784   s3://test3/aB8q5BA_460s.jpg >>>>>>> >>>>>>> [@ ganesha]# rpm -qa | grep ganesh >>>>>>> nfs-ganesha-xfs-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-rgw-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-ceph-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-vfs-2.5.5-.el7.x86_64 >>>>>>> nfs-ganesha-mem-2.5.5-.el7.x86_64 >>>>>>> >>>>>>> On CentOS7 >>>>>>> _______________________________________________ >>>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>>> unsubscribe >>>> >>>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>>> >>>>>> _______________________________________________ >>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>> unsubscribe >>> >>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>> >>>>>> _______________________________________________ >>>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>>> unsubscribe >>> >>>>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>>>> >>>>> _______________________________________________ >>>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>>> unsubscribe send an email to support-leave(a)lists.nfs-ganesha.org >>>>> >>>>> >>>> >>>> >>>> _______________________________________________ >>>> Support mailing list -- support(a)lists.nfs-ganesha.org To >>>> unsubscribe > >>>> send an email to support-leave(a)lists.nfs-ganesha.org >>>> >>> _______________________________________________ >>> Support mailing list -- support(a)lists.nfs-ganesha.org To unsubscribe >>> send an email to support-leave(a)lists.nfs-ganesha.org >>> >>> >> > > >