Attached the logs mount.log, ls.log and unmount.log With this config: ## configured from the command line LOG { ## Default log level for all components # NULL, FATAL, MAJ, CRIT, WARN, EVENT, INFO, DEBUG, MID_DEBUG, M_DBG, FULL_DEBUG, F_DBG], default EVENT default_log_level = FATAL; ## Configure per-component log levels. # ALL, LOG,LOG_EMERG,MEMLEAKS,FSAL,NFSPROTO(NFS3),NFS_V4(NFSV4),EXPORT,FILEHAND LE,DISPATCH,CACHE_INODE, # CACHE_INODE_LRU,HASHTABLE,HASHTABLE_CACHE,DUPREQ,INIT,MAIN, IDMAPPER,NFS_READDIR,NFS_V4_LOCK,CONFIG,CLIENTID, # SESSIONS,PNFS,RW_LOCK,NLM,RPC,NFS_CB,THREAD,NFS_V4_ACL,STATE,9P,9P_DISPA TCH,FSAL_UP,DBUS Components { #ALL = FATAL; FSAL = DEBUG; NFS4 = DEBUG; EXPORT = F_DBG; CONFIG = F_DBG; } ## Where to log Facility { name = FILE; destination = "/var/log/ganesha.log"; enable = default; } } PS. The DEBUG level is not working with syslog. I have to enable the local logging. -----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: dinsdag 1 mei 2018 14:11 To: Marc Roos; support Cc: mbenjamin Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users Okay, so that looks like Ganesha thinks the export is working, but the actual I/O on the bucket is failing. Can you turn on DEBUG for the NFS4 and FSAL components, perform this same simple test (mount + ls + umount) and post the ganesha log? This can be done like this: LOG { Components { NFS4 = DEBUG; FSAL = DEBUG; } } Daniel On 04/30/2018 11:29 AM, Marc Roos wrote:
I stopped the rgw3 instance that is running on the same server (I am not sure anymore if I can run radosgw and nfs-ganesha with the same client.rgw3), started ganesha with this config. I am correct to understand that the ganesha-rgw is not using the http(s) interface and
directly connects to ceph with librados (because that is why we have the RGW config section)?
RGW { cluster = "ceph"; name = "client.rgw3"; ceph_conf = "/etc/ceph/ceph.conf"; }
EXPORT { Export_ID=301; Path = "test:test3"; Pseudo = "/rgwtester";
FSAL { Name = RGW; User_Id = "test$tester1"; Access_Key_Id ="xxxxxx"; Secret_Access_Key = "xxxxxx"; } Disable_ACL = TRUE; CLIENT { Clients = 192.168.10.0/24; access_type = "RO"; } }
Logs of ganesha related to test3 bucket
[@]# tail -1000 /var/log/messages | grep ganesh | grep test3 Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO :Releasing RGW export for test:test3 Apr 30 22:24:55 c01 nfs-ganesha[604956]: [Admin] mdcache_exp_release :FSAL :INFO :Releasing RGW export for test:test3 Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export
301 created at pseudo (/rgwtester) with path (test:test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 30 22:27:07 c01 nfs-ganesha[644097]: [main] export_commit_common :CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test:test3) and tag ((null)) perms (options=03303002 , , , , , , , )
[@]# showmount -e nfs Export list for nfs: /ram 192.168.x.0 /test 192.168.x.0 /rgwtester 192.168.x.0
[@ mnt]# mount nfs:/rgwtester /mnt/nfstest
[@ mnt]# mount proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) ... nfs:/test on /home/users/robbert/nfs-test type nfs4 (rw,nodev,relatime,vers=4,intr,timeo=600,nolock,retrans=2,proto=tcp,na ml en=255,hard,addr= nfs:/rgwtester on /mnt/nfstest type nfs (rw,vers=4,addr=192.168.x.x,clientaddr=192.168.x.x)
[@ mnt]# ls -l /mnt/nfstest/ ls: reading directory /mnt/nfstest/: Remote I/O error total 0
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 20:06 To: Marc Roos; support Cc: Matt Benjamin Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
Okay, so on consultation, we think that maybe you should be mounting test:test3 (ie, <tenant:bucket>) with no '/'. I can't tell from the mail thread if you tried that; can you confirm?
(I have no personal experience with multi-tenant, so I didn't recognize it as such).
In the mean time, we'll try to reproduce here and see what we can see.
Daniel
On 04/30/2018 09:28 AM, Daniel Gryniewicz wrote:
All of that looks fine, I think. I'm out of my depth here, so I think
Matt needs to chime in here.
Matt?
On 04/30/2018 04:06 AM, Marc Roos wrote:
Could it not be, because of a configuration issue in ganesha.conf or
the version I am using? I can hardly imagine that I am the only one running a multitenant rgw config with ganesha-rgw.
nfs-ganesha-xfs-2.5.5-.el7.x86_64 nfs-ganesha-2.5.5-.el7.x86_64 nfs-ganesha-rgw-2.5.5-.el7.x86_64 nfs-ganesha-ceph-2.5.5-.el7.x86_64 nfs-ganesha-vfs-2.5.5-.el7.x86_64 nfs-ganesha-mem-2.5.5-.el7.x86_64
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 15:15 To: support@lists.nfs-ganesha.org Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
And test$tester1 is the user ID you created, that is this gives output:
radosgw-admin user info --uid="test$tester1"
(I'm pounding on this because it really seems that the UID doesn't have permissions. If it does, I think there's some really strange bug here that's going to be hard to track down.)
Daniel
On 04/30/2018 03:25 AM, Marc Roos wrote:
Yes same access key and secret access key ( no uid is used )
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: maandag 30 april 2018 14:40 To: Marc Roos; support Subject: Re: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I mean does the s3cmd (or cyberduck) use the exact same credentials
as
are specified in the ganesha.conf?
Daniel
On 04/26/2018 10:09 AM, Marc Roos wrote:
Yes, I can access the bucket via the s3cmd and cyberduck, or are you refering to something else?
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: donderdag 26 april 2018 15:38 To: support@lists.nfs-ganesha.org Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi
tenancy rgw users
Does test$tester1 have permissions to access RGW? Mounting / should see all the buckets, so it's likely a permission issue.
Daniel
On 04/26/2018 03:56 AM, Marc Roos wrote:
I have the same as with / path, I can do the mount, but it is just
empty
[@mnt]# mount nfs:/rgwtester nfstest/ [@mnt]# ls -l nfstest/ ls: reading directory nfstest/: Remote I/O error total 0
Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 created at pseudo (/rgwtester) with path (test3) and tag ((null)) perms (options=03303002 , , , , , , , ) Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common
:CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] export_commit_common :CONFIG :INFO :Export 301 has 1 defined clients Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export
0 (/) successfully created Apr 26 14:56:15 c01 nfs-ganesha[3981596]: [main] build_default_root :CONFIG :INFO :Export 0 (/) successfully
created Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0 Apr 26 14:56:16 c01 nfs-ganesha[3981596]: [main] fsal_save_ganesha_credentials :FSAL :INFO :Ganesha uid=0 gid=0 ngroups=0
-----Original Message----- From: Daniel Gryniewicz [mailto:dang@redhat.com] Sent: donderdag 26 april 2018 14:47 To: support@lists.nfs-ganesha.org Subject: [Nfs-ganesha-support] Re: Nfs-ganesha rgw config for multi tenancy rgw users
I think it should be:
Path="test3"
so without the initial '/'. Buckets aren't paths, we're just abusing
the existing Ganesha config to select them.
Other than that, the client entry should match the equivalent entry in
your ceph.conf.
Daniel
On 04/26/2018 01:52 AM, Marc Roos wrote: > > > I have problems exporting a bucket that really does exist. I > have
> tried Path = "/test:test3"; Path = "/test3"; Results in ganesha > fails
> to start with message ExportId=301 Path=/test:test3 > FSAL_ERROR=(Invalid object > type,0) > > If I use path=/ I can mount something but that is a empty > export,
> but
> cannot put files there. > > I have this in ganesha (client.rwg3 works with civetweb), should
> be
> sufficient not? > > RGW { > cluster = "ceph"; > name = "client.rgw3"; > ceph_conf = "/etc/ceph/ceph.conf"; > # for vstart cluster, name = "client.admin" > #init_args = "-d --debug-rgw=16"; } > > EXPORT { > Export_ID=301; > Path = "/test3"; > Pseudo = "/rgwtester"; > > FSAL { Name = RGW; User_Id = "test$tester1"; > Access_Key_Id ="sameass3"; Secret_Access_Key = "sameass3"; } > Disable_ACL = FALSE; > CLIENT { Clients = 192.168.x.0/24; } } > > [@~]$ s3cmd -c .s3cfg.tester1 ls > 2018-01-31 21:48 s3://test > 2018-02-01 11:44 s3://test2 > 2018-02-02 17:10 s3://test3 > > > [@~]$ s3cmd -c .s3cfg.tester1 ls s3://test3 > 2018-02-02 17:13 10485760 s3://test3/10MB.txt > 2018-02-05 12:57 26784 s3://test3/aB8q5BA_460s.jpg > > [@ ganesha]# rpm -qa | grep ganesh > nfs-ganesha-xfs-2.5.5-.el7.x86_64 > nfs-ganesha-2.5.5-.el7.x86_64 > nfs-ganesha-rgw-2.5.5-.el7.x86_64 > nfs-ganesha-ceph-2.5.5-.el7.x86_64 > nfs-ganesha-vfs-2.5.5-.el7.x86_64 > nfs-ganesha-mem-2.5.5-.el7.x86_64 > > On CentOS7 > _______________________________________________ > Support mailing list -- support@lists.nfs-ganesha.org To > unsubscribe
> send an email to support-leave@lists.nfs-ganesha.org > _______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org
_______________________________________________ Support mailing list -- support@lists.nfs-ganesha.org To unsubscribe
send an email to support-leave@lists.nfs-ganesha.org