7 a.m.
We've had problems with Gluster and Ceph, where the clean shutdown
closes all the files and releases the locks. Then, another client can
get those locks before the failover is complete, and they can't be
recovered. For Ceph, we added a special mode where it doesn't release
any state.
Daniel
On 11/22/19 9:57 AM, Daniel Gryniewicz wrote:
We've definitely had problems with graceful shutdown on Gluster.
It
closed all the files, dropping all the locks, and another client was
able to take those locks before failover occurred. For Ceph, we added a
special mode where it doesn't release locks or state.
Daniel
On 11/22/19 9:33 AM, Malahal Naineni wrote:
> Even a graceful shutdown should work fine unless there are some bugs
> in the shutdown path. The client should NOT use "soft" mount option
> though. We have used graceful shutdown (with GPFS fsal), and didn't
> encounter any issue yet.
>
> On Fri, Nov 22, 2019 at 1:06 AM Daniel Gryniewicz <dgryniew(a)redhat.com
> <mailto:dgryniew@redhat.com>> wrote:
>
> Expanding on that, the information we save for recovery hasn't
> changed
> in a while, so upgrades should be fine. However, make sure you do an
> unclean shutdown of the Ganesha to be upgraded (e.g., kill -9 rather
> than kill) since a clean shutdown will clean up all the
> state/locks/open
> files and so on, making recovery by active clients impossible.
>
> Daniel
>
> On 11/21/19 2:17 PM, Kaleb Keithley wrote:
> > Hi,
> >
> > To the best of my knowledge there's no reason why a rolling upgrade
> > wouldn't work seamlessly.
> >
> > There's nothing version specific about the recovery and no
> reason I'm
> > aware of that differing versions of ganesha wouldn't work side
> by side
> > in an HA cluster over the course of an upgrade.
> >
> > I suggest you play it safe and try it in a non-production
> environment
> > before doing it on a production system.
> >
> > HTH.
> >
> >
> >
> >
> > On Thu, Nov 21, 2019 at 2:07 PM Charles Hedrick
> <hedrick(a)rutgers.edu <mailto:hedrick@rutgers.edu>
> > <mailto:hedrick@rutgers.edu <mailto:hedrick@rutgers.edu>>>
wrote:
> >
> > I understand that Genesha supports HA configurations. Software
> > upgrades are a lot more frequent than hardware failures. The
> > ability to do upgrades is actually more of a motivation for HA
> > than handling failures. Do you support rolling upgrades, i.e.
> > upgrading one node while the other is active? That would
> require
> > that the two nodes would end up with different versions for at
> > least some period of time.
> > _______________________________________________
> > Support mailing list -- support(a)lists.nfs-ganesha.org
> <mailto:support@lists.nfs-ganesha.org>
> > <mailto:support@lists.nfs-ganesha.org
> <mailto:support@lists.nfs-ganesha.org>>
> > To unsubscribe send an email to
> > support-leave(a)lists.nfs-ganesha.org
> <mailto:support-leave@lists.nfs-ganesha.org>
> > <mailto:support-leave@lists.nfs-ganesha.org
> <mailto:support-leave@lists.nfs-ganesha.org>>
> >
> >
> > _______________________________________________
> > Support mailing list -- support(a)lists.nfs-ganesha.org
> <mailto:support@lists.nfs-ganesha.org>
> > To unsubscribe send an email to
> support-leave(a)lists.nfs-ganesha.org
> <mailto:support-leave@lists.nfs-ganesha.org>
> _______________________________________________
> Support mailing list -- support(a)lists.nfs-ganesha.org
> <mailto:support@lists.nfs-ganesha.org>
> To unsubscribe send an email to
> support-leave(a)lists.nfs-ganesha.org
> <mailto:support-leave@lists.nfs-ganesha.org>
>
7:58 a.m.
New subject: upgrades in HA configuration
Oh ok, Ganesha being a user level program, it loses all POSIX locks! GPFS
fsal has some kernel part (not needed for graceful shutdown but abrupt
shutdown needs kernel part) where it notifies some monitoring tool which
puts the entire cluster of Ganesha nodes in grace period to process only
reclaims! I assume neither Gluster nor Cephs use POSIX locks for NFS locks,
correct? Didn't realize that graceful shutdown is worse than abrupt
shutdown with some FSALs.
On Fri, Nov 22, 2019 at 8:30 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
We've had problems with Gluster and Ceph, where the clean
shutdown
closes all the files and releases the locks. Then, another client can
get those locks before the failover is complete, and they can't be
recovered. For Ceph, we added a special mode where it doesn't release
any state.
Daniel
On 11/22/19 9:57 AM, Daniel Gryniewicz wrote:
> We've definitely had problems with graceful shutdown on Gluster. It
> closed all the files, dropping all the locks, and another client was
> able to take those locks before failover occurred. For Ceph, we added a
> special mode where it doesn't release locks or state.
>
> Daniel
>
> On 11/22/19 9:33 AM, Malahal Naineni wrote:
>> Even a graceful shutdown should work fine unless there are some bugs
>> in the shutdown path. The client should NOT use "soft" mount option
>> though. We have used graceful shutdown (with GPFS fsal), and didn't
>> encounter any issue yet.
>>
>> On Fri, Nov 22, 2019 at 1:06 AM Daniel Gryniewicz <dgryniew(a)redhat.com
>> <mailto:dgryniew@redhat.com>> wrote:
>>
>> Expanding on that, the information we save for recovery hasn't
>> changed
>> in a while, so upgrades should be fine. However, make sure you do
an
>> unclean shutdown of the Ganesha to be upgraded (e.g., kill -9 rather
>> than kill) since a clean shutdown will clean up all the
>> state/locks/open
>> files and so on, making recovery by active clients impossible.
>>
>> Daniel
>>
>> On 11/21/19 2:17 PM, Kaleb Keithley wrote:
>> > Hi,
>> >
>> > To the best of my knowledge there's no reason why a rolling
upgrade
>> > wouldn't work seamlessly.
>> >
>> > There's nothing version specific about the recovery and no
>> reason I'm
>> > aware of that differing versions of ganesha wouldn't work side
>> by side
>> > in an HA cluster over the course of an upgrade.
>> >
>> > I suggest you play it safe and try it in a non-production
>> environment
>> > before doing it on a production system.
>> >
>> > HTH.
>> >
>> >
>> >
>> >
>> > On Thu, Nov 21, 2019 at 2:07 PM Charles Hedrick
>> <hedrick(a)rutgers.edu <mailto:hedrick@rutgers.edu>
>> > <mailto:hedrick@rutgers.edu
<mailto:hedrick@rutgers.edu>>> wrote:
>> >
>> > I understand that Genesha supports HA configurations. Software
>> > upgrades are a lot more frequent than hardware failures. The
>> > ability to do upgrades is actually more of a motivation for HA
>> > than handling failures. Do you support rolling upgrades, i.e.
>> > upgrading one node while the other is active? That would
>> require
>> > that the two nodes would end up with different versions for at
>> > least some period of time.
>> > _______________________________________________
>> > Support mailing list -- support(a)lists.nfs-ganesha.org
>> <mailto:support@lists.nfs-ganesha.org>
>> > <mailto:support@lists.nfs-ganesha.org
>> <mailto:support@lists.nfs-ganesha.org>>
>> > To unsubscribe send an email to
>> > support-leave(a)lists.nfs-ganesha.org
>> <mailto:support-leave@lists.nfs-ganesha.org>
>> > <mailto:support-leave@lists.nfs-ganesha.org
>> <mailto:support-leave@lists.nfs-ganesha.org>>
>> >
>> >
>> > _______________________________________________
>> > Support mailing list -- support(a)lists.nfs-ganesha.org
>> <mailto:support@lists.nfs-ganesha.org>
>> > To unsubscribe send an email to
>> support-leave(a)lists.nfs-ganesha.org
>> <mailto:support-leave@lists.nfs-ganesha.org>
>> _______________________________________________
>> Support mailing list -- support(a)lists.nfs-ganesha.org
>> <mailto:support@lists.nfs-ganesha.org>
>> To unsubscribe send an email to
>> support-leave(a)lists.nfs-ganesha.org
>> <mailto:support-leave@lists.nfs-ganesha.org>
>>
10:19 a.m.
New subject: upgrades in HA configuration
FSAL_GLUSTER passes locks down through the gluster stack where eventually
gluster uses POSIX locking in the backing file system.
Hmmm. Apropos of nothing, I thought I'd been told that Samba keeps all its
lock state in CTDB, but the Gluster VFS in Samba 4.10 clearly uses gluster
locking. Interesting.
On Fri, Nov 22, 2019 at 10:59 AM Malahal Naineni <malahal(a)gmail.com> wrote:
Oh ok, Ganesha being a user level program, it loses all POSIX locks!
GPFS
fsal has some kernel part (not needed for graceful shutdown but abrupt
shutdown needs kernel part) where it notifies some monitoring tool which
puts the entire cluster of Ganesha nodes in grace period to process only
reclaims! I assume neither Gluster nor Cephs use POSIX locks for NFS locks,
correct? Didn't realize that graceful shutdown is worse than abrupt
shutdown with some FSALs.
On Fri, Nov 22, 2019 at 8:30 PM Daniel Gryniewicz <dang(a)redhat.com> wrote:
> We've had problems with Gluster and Ceph, where the clean shutdown
> closes all the files and releases the locks. Then, another client can
> get those locks before the failover is complete, and they can't be
> recovered. For Ceph, we added a special mode where it doesn't release
> any state.
>
> Daniel
>
> On 11/22/19 9:57 AM, Daniel Gryniewicz wrote:
> > We've definitely had problems with graceful shutdown on Gluster. It
> > closed all the files, dropping all the locks, and another client was
> > able to take those locks before failover occurred. For Ceph, we added
> a
> > special mode where it doesn't release locks or state.
> >
> > Daniel
> >
> > On 11/22/19 9:33 AM, Malahal Naineni wrote:
> >> Even a graceful shutdown should work fine unless there are some bugs
> >> in the shutdown path. The client should NOT use "soft" mount
option
> >> though. We have used graceful shutdown (with GPFS fsal), and didn't
> >> encounter any issue yet.
> >>
> >> On Fri, Nov 22, 2019 at 1:06 AM Daniel Gryniewicz <dgryniew(a)redhat.com
> >> <mailto:dgryniew@redhat.com>> wrote:
> >>
> >> Expanding on that, the information we save for recovery hasn't
> >> changed
> >> in a while, so upgrades should be fine. However, make sure you do
> an
> >> unclean shutdown of the Ganesha to be upgraded (e.g., kill -9
> rather
> >> than kill) since a clean shutdown will clean up all the
> >> state/locks/open
> >> files and so on, making recovery by active clients impossible.
> >>
> >> Daniel
> >>
> >> On 11/21/19 2:17 PM, Kaleb Keithley wrote:
> >> > Hi,
> >> >
> >> > To the best of my knowledge there's no reason why a rolling
> upgrade
> >> > wouldn't work seamlessly.
> >> >
> >> > There's nothing version specific about the recovery and no
> >> reason I'm
> >> > aware of that differing versions of ganesha wouldn't work side
> >> by side
> >> > in an HA cluster over the course of an upgrade.
> >> >
> >> > I suggest you play it safe and try it in a non-production
> >> environment
> >> > before doing it on a production system.
> >> >
> >> > HTH.
> >> >
> >> >
> >> >
> >> >
> >> > On Thu, Nov 21, 2019 at 2:07 PM Charles Hedrick
> >> <hedrick(a)rutgers.edu <mailto:hedrick@rutgers.edu>
> >> > <mailto:hedrick@rutgers.edu
<mailto:hedrick@rutgers.edu>>>
> wrote:
> >> >
> >> > I understand that Genesha supports HA configurations.
> Software
> >> > upgrades are a lot more frequent than hardware failures. The
> >> > ability to do upgrades is actually more of a motivation for
> HA
> >> > than handling failures. Do you support rolling upgrades, i.e.
> >> > upgrading one node while the other is active? That would
> >> require
> >> > that the two nodes would end up with different versions for
> at
> >> > least some period of time.
> >> > _______________________________________________
> >> > Support mailing list -- support(a)lists.nfs-ganesha.org
> >> <mailto:support@lists.nfs-ganesha.org>
> >> > <mailto:support@lists.nfs-ganesha.org
> >> <mailto:support@lists.nfs-ganesha.org>>
> >> > To unsubscribe send an email to
> >> > support-leave(a)lists.nfs-ganesha.org
> >> <mailto:support-leave@lists.nfs-ganesha.org>
> >> > <mailto:support-leave@lists.nfs-ganesha.org
> >> <mailto:support-leave@lists.nfs-ganesha.org>>
> >> >
> >> >
> >> > _______________________________________________
> >> > Support mailing list -- support(a)lists.nfs-ganesha.org
> >> <mailto:support@lists.nfs-ganesha.org>
> >> > To unsubscribe send an email to
> >> support-leave(a)lists.nfs-ganesha.org
> >> <mailto:support-leave@lists.nfs-ganesha.org>
> >> _______________________________________________
> >> Support mailing list -- support(a)lists.nfs-ganesha.org
> >> <mailto:support@lists.nfs-ganesha.org>
> >> To unsubscribe send an email to
> >> support-leave(a)lists.nfs-ganesha.org
> >> <mailto:support-leave@lists.nfs-ganesha.org>
> >>
>
>
2507
days inactive
2507
days old
2 comments
3 participants
participants (3)
-
Daniel Gryniewicz -
Kaleb Keithley -
Malahal Naineni