Hi Guys, I tried to write files through FSAL_RGW and it's successful. But after a little while, I found all the file size become 0. I just follow the writing logic for FSAL_RGW. I found the `RGW_OPEN_FLAG_V3` is only set when state (which is a parameter rgw_fsal_open2() in ) is NULL. However, it seems that the `state` is always NOT NULL. When I changed the code to transfer RGW_OPEN_FLAG_V3 into rgw_open, the issue is gone. Is there any other guy meet a similar issue? ``` rc = rgw_open(export->rgw_fs, handle->rgw_fh, posix_flags, (!state) ? RGW_OPEN_FLAG_V3 : RGW_OPEN_FLAG_NONE); ``` Marvin
On 11/8/19 5:06 AM, fanzi2009@hotmail.com wrote:
Hi Guys, I tried to write files through FSAL_RGW and it's successful. But after a little while, I found all the file size become 0. I just follow the writing logic for FSAL_RGW. I found the `RGW_OPEN_FLAG_V3` is only set when state (which is a parameter rgw_fsal_open2() in ) is NULL. However, it seems that the `state` is always NOT NULL. When I changed the code to transfer RGW_OPEN_FLAG_V3 into rgw_open, the issue is gone. Is there any other guy meet a similar issue?
``` rc = rgw_open(export->rgw_fs, handle->rgw_fh, posix_flags, (!state) ? RGW_OPEN_FLAG_V3 : RGW_OPEN_FLAG_NONE); ```
RGW_OPEN_FLAG_V3 indicates to RGW that this is an anonymous I/O, as is done by NFSv3. It should only be passed when state is NULL, so the code is correct as currently written. I'm CC'ing Matt, who wrote this code. Daniel
I'm guessing the problem is with the underlying librgw version. When the mount is NFSv3, the file will be written out (by default) 10s after the last append to the file via NFS. That write-out may be failing. Have you tried using NFSv4.1/2 as the NFS protocol version? That may be easier to debug. Matt On Fri, Nov 8, 2019 at 9:46 AM Daniel Gryniewicz <dang@redhat.com> wrote:
On 11/8/19 5:06 AM, fanzi2009@hotmail.com wrote:
Hi Guys, I tried to write files through FSAL_RGW and it's successful. But after a little while, I found all the file size become 0. I just follow the writing logic for FSAL_RGW. I found the `RGW_OPEN_FLAG_V3` is only set when state (which is a parameter rgw_fsal_open2() in ) is NULL. However, it seems that the `state` is always NOT NULL. When I changed the code to transfer RGW_OPEN_FLAG_V3 into rgw_open, the issue is gone. Is there any other guy meet a similar issue?
``` rc = rgw_open(export->rgw_fs, handle->rgw_fh, posix_flags, (!state) ? RGW_OPEN_FLAG_V3 : RGW_OPEN_FLAG_NONE); ```
RGW_OPEN_FLAG_V3 indicates to RGW that this is an anonymous I/O, as is done by NFSv3. It should only be passed when state is NULL, so the code is correct as currently written.
I'm CC'ing Matt, who wrote this code.
Daniel
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
In any case, though: the issue is pretty much certain to be part of the librgw write path--which means that the problem depends on the Ceph version. I can't promise to find a solution on any possible version of Ceph, but, for recent versions (e.g., Nautilus or master) look for a commit which fixes tracker issue 41173. IIRC, that issue resulted in write failures such as you're experiencing. It existed (on master) for a couple of days, and may have persisted a bit longer on Nautilus. (Looks like Nautilus took the fix on 8/21/2019.) Other issues could look similar, though (but I don't know of any that is open). Matt On Fri, Nov 8, 2019 at 10:00 AM Matt Benjamin <mbenjami@redhat.com> wrote:
I'm guessing the problem is with the underlying librgw version. When the mount is NFSv3, the file will be written out (by default) 10s after the last append to the file via NFS. That write-out may be failing.
Have you tried using NFSv4.1/2 as the NFS protocol version? That may be easier to debug.
Matt
On Fri, Nov 8, 2019 at 9:46 AM Daniel Gryniewicz <dang@redhat.com> wrote:
On 11/8/19 5:06 AM, fanzi2009@hotmail.com wrote:
Hi Guys, I tried to write files through FSAL_RGW and it's successful. But after a little while, I found all the file size become 0. I just follow the writing logic for FSAL_RGW. I found the `RGW_OPEN_FLAG_V3` is only set when state (which is a parameter rgw_fsal_open2() in ) is NULL. However, it seems that the `state` is always NOT NULL. When I changed the code to transfer RGW_OPEN_FLAG_V3 into rgw_open, the issue is gone. Is there any other guy meet a similar issue?
``` rc = rgw_open(export->rgw_fs, handle->rgw_fh, posix_flags, (!state) ? RGW_OPEN_FLAG_V3 : RGW_OPEN_FLAG_NONE); ```
RGW_OPEN_FLAG_V3 indicates to RGW that this is an anonymous I/O, as is done by NFSv3. It should only be passed when state is NULL, so the code is correct as currently written.
I'm CC'ing Matt, who wrote this code.
Daniel
--
Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103
http://www.redhat.com/en/technologies/storage
tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Hi Matt, As you said, file will be written out 10s after the last append. Here is the latest code from master branch. It will do it only if it's stateless_open. And stateless_open is true only if RGW_OPEN_FLAG_V3 is set. Please let me know if I'm wrong. Thanks, ``` //code in RGWFileHandle::write if (stateless_open()) { ... RGWLibFS::write_timer.add_event(...) ... } ``` Marvin
That's correct reading of the code intent, yes. Matt On Mon, Nov 11, 2019 at 2:00 AM <fanzi2009@hotmail.com> wrote:
Hi Matt, As you said, file will be written out 10s after the last append. Here is the latest code from master branch. It will do it only if it's stateless_open. And stateless_open is true only if RGW_OPEN_FLAG_V3 is set. Please let me know if I'm wrong. Thanks, ``` //code in RGWFileHandle::write
if (stateless_open()) { ... RGWLibFS::write_timer.add_event(...) ... } ``` Marvin _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
Hi Matt, Ganesha call rgw_open() with RGW_OPEN_FLAG_NONE parameter. However, only stateless_open can trigger real writing in ceph rgw side. In this way, Ganesha will never write. Is it a bug? Marvin
Hi, But that's not the case. When the mount is NFSv4+, then we write on close() or commit(). Matt On Wed, Nov 13, 2019 at 1:25 AM <fanzi2009@hotmail.com> wrote:
Hi Matt, Ganesha call rgw_open() with RGW_OPEN_FLAG_NONE parameter. However, only stateless_open can trigger real writing in ceph rgw side. In this way, Ganesha will never write. Is it a bug? Marvin _______________________________________________ Devel mailing list -- devel@lists.nfs-ganesha.org To unsubscribe send an email to devel-leave@lists.nfs-ganesha.org
-- Matt Benjamin Red Hat, Inc. 315 West Huron Street, Suite 140A Ann Arbor, Michigan 48103 http://www.redhat.com/en/technologies/storage tel. 734-821-5101 fax. 734-769-8938 cel. 734-216-5309
participants (3)
-
Daniel Gryniewicz -
fanzi2009＠hotmail.com -
Matt Benjamin