Okay, so that looks like Ganesha thinks the export is working, but the actual I/O on the bucket is failing. Can you turn on DEBUG for the NFS4 and FSAL components, perform this same simple test (mount + ls + umount) and post the ganesha log? This can be done like this: LOG { Components { NFS4 = DEBUG; FSAL = DEBUG; } } Daniel On 04/30/2018 11:29 AM, Marc Roos wrote:
I stopped the rgw3 instance that is running on the same server (I am not sure anymore if I can run radosgw and nfs-ganesha with the same client.rgw3), started ganesha with this config. I am correct to understand that the ganesha-rgw is not using the http(s) interface and directly connects to ceph with librados (because that is why we have the RGW config section)?
RGW { cluster = "ceph"; name = "client.rgw3"; ceph_conf = "/etc/ceph/ceph.conf"; }
EXPORT { Export_ID=301; Path = "test:test3"; Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id ="xxxxxx"; Secret_Access_Key = "xxxxxx"; } Disable_ACL = TRUE; CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } }
Logs of ganesha related to test3 bucket
[@]# tail -1000 /var/log/messages | grep ganesh | grep test3 Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO :Releasing RGW export for test:test3 Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO :Releasing RGW export for test:test3 Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test:test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test:test3) and tag ((null)) perms (options=03303002 , , , , , , , )
[@]# showmount -e nfs Export list for nfs: /ram 192.168.x.0 /test 192.168.x.0 /rgwtester 192.168.x.0
[@ mnt]# mount nfs:/rgwtester /mnt/nfstest
[@ mnt]# mount proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw) sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ... nfs:/test on /home/users/robbert/nfs-test type nfs4 (rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,naml en=255,hard,addr= nfs:/rgwtester on /mnt/nfstest type nfs (rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x)
[@ mnt]# ls -l /mnt/nfstest/ ls: reading directory /mnt/nfstest/: Remote I/O error total 0
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 20:06 To: Marc Roos; support Cc: Matt Benjamin Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Okay, so on consultation, we think that maybe you should be mounting test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the mail thread if you tried that; can you confirm?
(I have no personal experience with multi-tenant, so I didn't recognize it as such).
In the mean time, we'll try to reproduce here and see what we can see.
Daniel
On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
All of that looks fine, I think. I'm out of my depth here, so I think
Matt needs to chime in here.
Matt?
On 04/30/2018 04:06 AM, Marc Roos wrote:
Could it not be, because of a configuration issue in ganesha.conf or the version I am using? I can hardly imagine that I am the only one running a multitenant rgw config with ganesha-rgw.
nfs-ganesha-xfs-2.5.5-.el7.x86_64 nfs-ganesha-2.5.5-.el7.x86_64 nfs-ganesha-rgw-2.5.5-.el7.x86_64 nfs-ganesha-ceph-2.5.5-.el7.x86_64 nfs-ganesha-vfs-2.5.5-.el7.x86_64 nfs-ganesha-mem-2.5.5-.el7.x86_64
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 15:15 To: support@lists.nfs-ganesha.org Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
And test$tester1 is the user ID you created, that is this gives output:
radosgw-admin user info --uid="test$tester1"
(I'm pounding on this because it really seems that the UID doesn't have permissions. If it does, I think there's some really strange bug here that's going to be hard to track down.)
Daniel
On 04/30/2018 03:25 AM, Marc Roos wrote:
Yes same access key and secret access key ( no uid is used )
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 14:40 To: Marc Roos; support Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
Yes, I can access the bucket via the s3cmd and cyberduck, or are you refering to something else?
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: donderdag 26 april 2018 15:38 To: support@lists.nfs-ganesha.org Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Does test$tester1 have permissions to access RGW? Mounting / should see all the buckets, so it's likely a permission issue.
Daniel
On 04/26/2018 03:56 AM, Marc Roos wrote:
I have the same as with / path, I can do the mount, but it is just
empty
[@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/ ls: reading directory nfstest/: Remote I/O error total 0
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export
0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: donderdag 26 april 2018 14:47 To: support@lists.nfs-ganesha.org Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I think it should be:
Path="test3"
so without the initial '/'. Buckets aren't paths, we're just abusing
the existing Ganesha config to select them.
Other than that, the client entry should match the equivalent entry in
your ceph.conf.
Daniel
On 04/26/2018 01:52 AM, Marc Roos wrote: > > > I have problems exporting a bucket that really does exist. I have
> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha > fails
> to start with message ExportId=301 Path=/test:test3 > FSAL_ERROR=(Invalid object > type,0) > > If I use path=/ I can mount something but that is a empty export,
> but
> cannot put files there. > > I have this in ganesha (client.rwg3 works with civetweb), should > be
> sufficient not? > > RGW { > cluster = "ceph"; > name = "client.rgw3"; > ceph_conf = "/etc/ceph/ceph.conf"; > # for vstart cluster, name = "client.admin" > #init_args = "-d --debug-rgw=16"; } > > EXPORT { > Export_ID=301; > Path = "/test3"; > Pseudo = "/rgwtester"; > > FSAL { Name = RGW; User_Id = "test$tester1"; > Access_Key_Id ="sameass3"; Secret_Access_Key = "sameass3"; } > Disable_ACL = FALSE; > CLIENT { Clients = 192.168.x.0/24; } } > > [@~]$ s3cmd -c .s3cfg.tester1 ls > 2018-01-31 21:48 s3://test > 2018-02-01 11:44 s3://test2 > 2018-02-02 17:10 s3://test3 > > > [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3 > 2018-02-02 17:13 10485760 s3://test3/10MB.txt > 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg > > [@ ganesha]# rpm -qa | grep ganesh > nfs-ganesha-xfs-2.5.5-.el7.x86_64 > nfs-ganesha-2.5.5-.el7.x86_64 > nfs-ganesha-rgw-2.5.5-.el7.x86_64 > nfs-ganesha-ceph-2.5.5-.el7.x86_64 > nfs-ganesha-vfs-2.5.5-.el7.x86_64 > nfs-ganesha-mem-2.5.5-.el7.x86_64 > > On CentOS7 > _______________________________________________ > Support mailing list -- support@lists.nfs-ganesha.org To > unsubscribe
> send an email to support-leave@lists.nfs-ganesha.org > _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org