7:47 p.m.
Hi All,
I am using nfs-ganesha 2.6.3. I am dumping Netezza to a Linux machine running nfs-ganesha.
Netezza writes data in 1MB chunks. After each write it checks size of file. After dumping
some data it failed saying that expected size is 106MB. But the file size is 104MB.
Nfs-Ganesha debug logs indicated that one of the workers processed a getattr RPC (supposed
to have been sent after writing 104MB) after receiving write RPC for 106MB but before it
was written to disk.
Is this valid behavior from nfs-ganesha? Are there any settings (on nfs-client or on
nfs-ganesha) to make sure operations on a given file are processed in the order they are
received?
I have seen another weird problem. I have not got a chance to check logs for it yet. My
application creates a temp file, writes small data (10K) to it, closes the file and then
renames it. This rename failed with ENOEXIST. I see that temp file is indeed present.
Based on my observation with Netezza I am suspecting that that rename was processed before
the temp file create RPC was processed. Could that be possible?
--Satish
--Satish
8:47 a.m.
A couple of questions to begin with. What NFS version are you mounting?
I assume this is a Linux client?
NFS is an asynchronous protocol. That is, no two ops that are not in
the same compound have any guaranteed ordering with respect to each
other. This is one of the reasons that NFSv4 introduced compounds: to
guarantee such ordering. The client can get correct ordering by waiting
for the reply to the first op before sending the second op, but if the
second op arrives while the first op is being processed, it's perfectly
fine for the second op to return to the client before the first op.
My guess is that you have the client mounted with the async flag. This
causes the kernel to return success on write to the application before
it's actually sent the data to the server. It sends the write op in the
background, but this means that, even if the app waits for the write to
"complete", there's no guarantee that the write has been sent to the
server yet. If this is the case, mount your filesystem sync, and this
should fix it.
Daniel
On 10/4/19 8:47 PM, Satish Chandra Kilaru wrote:
Hi All,
I am using nfs-ganesha 2.6.3. I am dumping Netezza to a Linux machine running
nfs-ganesha. Netezza writes data in 1MB chunks. After each write it checks size of file.
After dumping some data it failed saying that expected size is 106MB. But the file size is
104MB.
Nfs-Ganesha debug logs indicated that one of the workers processed a getattr RPC
(supposed to have been sent after writing 104MB) after receiving write RPC for 106MB but
before it was written to disk.
Is this valid behavior from nfs-ganesha? Are there any settings (on nfs-client or on
nfs-ganesha) to make sure operations on a given file are processed in the order they are
received?
I have seen another weird problem. I have not got a chance to check logs for it yet. My
application creates a temp file, writes small data (10K) to it, closes the file and then
renames it. This rename failed with ENOEXIST. I see that temp file is indeed present.
Based on my observation with Netezza I am suspecting that that rename was processed before
the temp file create RPC was processed. Could that be possible?
--Satish
--Satish
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
3:15 a.m.
The application doing "create a temp file, write some data, close the the
file, then rename" should be doing in that order. In other words, it has to
wait at least for the temp file creation before sending rename.
Linux NFS client does cache attributes by default. It should manipulate and
do the right things, if the writes are cached. Can you simulate what
Netezza is doing in a small program or script and recreate it? Mount option
"-onoac" or "-osync" may help but would like to understand this issue
further as file size should be consistent if viewed from the same NFS
client.
On Mon, Oct 7, 2019 at 7:18 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
A couple of questions to begin with. What NFS version are you
mounting?
I assume this is a Linux client?
NFS is an asynchronous protocol. That is, no two ops that are not in
the same compound have any guaranteed ordering with respect to each
other. This is one of the reasons that NFSv4 introduced compounds: to
guarantee such ordering. The client can get correct ordering by waiting
for the reply to the first op before sending the second op, but if the
second op arrives while the first op is being processed, it's perfectly
fine for the second op to return to the client before the first op.
My guess is that you have the client mounted with the async flag. This
causes the kernel to return success on write to the application before
it's actually sent the data to the server. It sends the write op in the
background, but this means that, even if the app waits for the write to
"complete", there's no guarantee that the write has been sent to the
server yet. If this is the case, mount your filesystem sync, and this
should fix it.
Daniel
On 10/4/19 8:47 PM, Satish Chandra Kilaru wrote:
> Hi All,
>
> I am using nfs-ganesha 2.6.3. I am dumping Netezza to a Linux machine
running nfs-ganesha. Netezza writes data in 1MB chunks. After each write it
checks size of file. After dumping some data it failed saying that expected
size is 106MB. But the file size is 104MB.
> Nfs-Ganesha debug logs indicated that one of the workers processed a
getattr RPC (supposed to have been sent after writing 104MB) after
receiving write RPC for 106MB but before it was written to disk.
>
> Is this valid behavior from nfs-ganesha? Are there any settings (on
nfs-client or on nfs-ganesha) to make sure operations on a given file are
processed in the order they are received?
>
> I have seen another weird problem. I have not got a chance to check logs
for it yet. My application creates a temp file, writes small data (10K) to
it, closes the file and then renames it. This rename failed with ENOEXIST.
I see that temp file is indeed present. Based on my observation with
Netezza I am suspecting that that rename was processed before the temp file
create RPC was processed. Could that be possible?
>
> --Satish
> --Satish
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org
> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
7:54 a.m.
I tried to reproduce the problem using a test application. But could not.
The application doing "create a temp file, write some data, close the the
file, then rename" is Oracle DNFS. It works fine if the file is large. This
rename problem is seen once in a while with tiny incremental dumps.
--Satish
On Thu, Oct 10, 2019 at 4:16 AM Malahal Naineni <malahal(a)gmail.com> wrote:
The application doing "create a temp file, write some data,
close the the
file, then rename" should be doing in that order. In other words, it has to
wait at least for the temp file creation before sending rename.
Linux NFS client does cache attributes by default. It should manipulate
and do the right things, if the writes are cached. Can you simulate what
Netezza is doing in a small program or script and recreate it? Mount option
"-onoac" or "-osync" may help but would like to understand this
issue
further as file size should be consistent if viewed from the same NFS
client.
On Mon, Oct 7, 2019 at 7:18 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
> A couple of questions to begin with. What NFS version are you mounting?
> I assume this is a Linux client?
>
> NFS is an asynchronous protocol. That is, no two ops that are not in
> the same compound have any guaranteed ordering with respect to each
> other. This is one of the reasons that NFSv4 introduced compounds: to
> guarantee such ordering. The client can get correct ordering by waiting
> for the reply to the first op before sending the second op, but if the
> second op arrives while the first op is being processed, it's perfectly
> fine for the second op to return to the client before the first op.
>
> My guess is that you have the client mounted with the async flag. This
> causes the kernel to return success on write to the application before
> it's actually sent the data to the server. It sends the write op in the
> background, but this means that, even if the app waits for the write to
> "complete", there's no guarantee that the write has been sent to the
> server yet. If this is the case, mount your filesystem sync, and this
> should fix it.
>
> Daniel
>
> On 10/4/19 8:47 PM, Satish Chandra Kilaru wrote:
> > Hi All,
> >
> > I am using nfs-ganesha 2.6.3. I am dumping Netezza to a Linux machine
> running nfs-ganesha. Netezza writes data in 1MB chunks. After each write it
> checks size of file. After dumping some data it failed saying that expected
> size is 106MB. But the file size is 104MB.
> > Nfs-Ganesha debug logs indicated that one of the workers processed a
> getattr RPC (supposed to have been sent after writing 104MB) after
> receiving write RPC for 106MB but before it was written to disk.
> >
> > Is this valid behavior from nfs-ganesha? Are there any settings (on
> nfs-client or on nfs-ganesha) to make sure operations on a given file are
> processed in the order they are received?
> >
> > I have seen another weird problem. I have not got a chance to check
> logs for it yet. My application creates a temp file, writes small data
> (10K) to it, closes the file and then renames it. This rename failed with
> ENOEXIST. I see that temp file is indeed present. Based on my observation
> with Netezza I am suspecting that that rename was processed before the temp
> file create RPC was processed. Could that be possible?
> >
> > --Satish
> > --Satish
> > _______________________________________________
> > Devel mailing list -- devel(a)lists.nfs-ganesha.org
> > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
> >
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org
> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>
_______________________________________________
Devel mailing list -- devel(a)lists.nfs-ganesha.org
To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
--
Please Donate to www.wikipedia.org
8:28 a.m.
tcpdump is your best bet to find if this is something to do with NFS client
(DNFS) or Ganesha. What FSAL are you using with Ganesha?
On Thu, Oct 10, 2019 at 6:24 PM Satish Chandra Kilaru <iam.kilaru(a)gmail.com>
wrote:
I tried to reproduce the problem using a test application. But could
not.
The application doing "create a temp file, write some data, close the the
file, then rename" is Oracle DNFS. It works fine if the file is large. This
rename problem is seen once in a while with tiny incremental dumps.
--Satish
On Thu, Oct 10, 2019 at 4:16 AM Malahal Naineni <malahal(a)gmail.com> wrote:
> The application doing "create a temp file, write some data, close the the
> file, then rename" should be doing in that order. In other words, it has to
> wait at least for the temp file creation before sending rename.
>
> Linux NFS client does cache attributes by default. It should manipulate
> and do the right things, if the writes are cached. Can you simulate what
> Netezza is doing in a small program or script and recreate it? Mount option
> "-onoac" or "-osync" may help but would like to understand this
issue
> further as file size should be consistent if viewed from the same NFS
> client.
>
> On Mon, Oct 7, 2019 at 7:18 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
>
>> A couple of questions to begin with. What NFS version are you mounting?
>> I assume this is a Linux client?
>>
>> NFS is an asynchronous protocol. That is, no two ops that are not in
>> the same compound have any guaranteed ordering with respect to each
>> other. This is one of the reasons that NFSv4 introduced compounds: to
>> guarantee such ordering. The client can get correct ordering by waiting
>> for the reply to the first op before sending the second op, but if the
>> second op arrives while the first op is being processed, it's perfectly
>> fine for the second op to return to the client before the first op.
>>
>> My guess is that you have the client mounted with the async flag. This
>> causes the kernel to return success on write to the application before
>> it's actually sent the data to the server. It sends the write op in the
>> background, but this means that, even if the app waits for the write to
>> "complete", there's no guarantee that the write has been sent to
the
>> server yet. If this is the case, mount your filesystem sync, and this
>> should fix it.
>>
>> Daniel
>>
>> On 10/4/19 8:47 PM, Satish Chandra Kilaru wrote:
>> > Hi All,
>> >
>> > I am using nfs-ganesha 2.6.3. I am dumping Netezza to a Linux machine
>> running nfs-ganesha. Netezza writes data in 1MB chunks. After each write it
>> checks size of file. After dumping some data it failed saying that expected
>> size is 106MB. But the file size is 104MB.
>> > Nfs-Ganesha debug logs indicated that one of the workers processed a
>> getattr RPC (supposed to have been sent after writing 104MB) after
>> receiving write RPC for 106MB but before it was written to disk.
>> >
>> > Is this valid behavior from nfs-ganesha? Are there any settings (on
>> nfs-client or on nfs-ganesha) to make sure operations on a given file are
>> processed in the order they are received?
>> >
>> > I have seen another weird problem. I have not got a chance to check
>> logs for it yet. My application creates a temp file, writes small data
>> (10K) to it, closes the file and then renames it. This rename failed with
>> ENOEXIST. I see that temp file is indeed present. Based on my observation
>> with Netezza I am suspecting that that rename was processed before the temp
>> file create RPC was processed. Could that be possible?
>> >
>> > --Satish
>> > --Satish
>> > _______________________________________________
>> > Devel mailing list -- devel(a)lists.nfs-ganesha.org
>> > To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>> >
>> _______________________________________________
>> Devel mailing list -- devel(a)lists.nfs-ganesha.org
>> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>>
> _______________________________________________
> Devel mailing list -- devel(a)lists.nfs-ganesha.org
> To unsubscribe send an email to devel-leave(a)lists.nfs-ganesha.org
>
--
Please Donate to www.wikipedia.org
1669
days inactive
1674
days old
4 comments
3 participants
participants (3)
-
Daniel Gryniewicz -
Malahal Naineni -
Satish Chandra Kilaru