►
From YouTube: 2020-04-01: High availability Gitaly demo
Description
A
B
C
C
All
right,
I'm
gonna
go
ahead
and
hi
James
welcome
aboard
I'm
gonna
get
started
here.
Today's
demo
is
going
to
involve
multiple
prefix
sized,
something
I,
don't
think
anybody
else
has
done
yet.
So
if
you
look
at
this
picture,
we've
we
claimed
to
have
multiple
prefix,
but
we
actually
haven't
done
this
before
so
this
will
be
the
new
wrinkle
and
then
I
think
we've
also
added
replication
queue
to
this
whole
system.
So
we'll
be
demoing
some
new
stuff
here.
So
now,
I've
gone
ahead
and
I've
done
most
of
the
basic
prep
works.
C
All
this
stuff,
getting
Postgres
up
and
running
is
done.
Getting
prefect.
The
like
I
said:
the
difference
was
that
I
actually
had
now
on
this.
You
can
see
my
bottom.
These
are
three
prefect
notes
running
on
you
know
then
cloud
and
then
top
three
are
particularly
notes,
but
I
did
want
to
leave
some
things
to
kind
of
configure,
so
we
could
all
sort
of
get
on
the
same
page.
So
this
is
all
done
on
two
on
three
of
the
prefect
notes.
C
That's
all
the
same
and
giddily
notes
and
likewise
I
configure
all
these
notes
to
be
up
with
these
configuration
settings
didn't
run
into
anything
strange
there.
The
only
thing
I
left
was
just
configuring
get
lab
itself
here.
So
here
we
say
configure
add
the
prefect
cluster
and
this
is
a
little
confusing
because
it's
called
prefect
and
then
I've
got
like
multiple
prefix.
But
anyway,
that's
I,
don't
know
it's
it's
intentional.
C
C
Something
like
making
it
obvious
that
it's
a
virtual
storage
or
something
like
that,
because
I
have
I
was
almost
tempted
to
put
in
like
giddily
stuff
I'm
prefect
stuff
in
here,
but
then
they
saw
that
anyway.
So
if
we
look
at
this
config
data,
editors,
I
went
ahead,
just
we're
just
using
the
same
external
token,
but
this
I
guess
this
confuse
me
a
little,
because
this
is
supposed
to
be
prefect
on
only
one.
A
prefect
like
we
assuming
there's
gonna,
be
a
load
balancer
in
front
of
this
prefect
IP.
Now.
C
B
I
think
it
would
like
it
would
be
a
load
balancer,
that's
my
guess
yeah,
but
the
other.
The
other
question
is
like
what
load
balancer
do
we
recommend
like?
Is
there
like,
there's,
probably
like
s3
and
I,
know
sorry,
AWS
and
Google
load
balancer
products
that
you
could
use,
or
maybe
you
could
run
your
own
load
balancer
like
what
load
balancer
would
that
be,
and
it
does
omnibus
need
to
package
a
load
balancer
that
can
be
used
for
this
purpose.
Yeah.
C
These
are
all
good
questions,
I
think
for
Gila
back
on
the
simplest
thing
we
would
probably
do
already
using
Google
load
balancer.
We
would
just
basically
plug
in
that
IP
address,
but
then
it
does
bring.
The
question
is
like
well:
how
do
we
desire
to
prefect
notice
unhealthy
like?
Is
it
just
a
TCP
check
so
anyway,
all
these
questions
will
have
to
answer
I'm
not
going
to
answer
them
today,
but
I'm
just
raising
them
because,
as
I
was
going
through
this
documentation,
that
was
a
little
confused
okay.
C
C
C
C
And
while
that's
going,
I'm
just
gonna
go
ahead
and
well,
let's
see
okay,
so
everything's
good
there
I'm
gonna
go
to
the
UI
and
I
think
there
was
a
bug
before
that.
You
guys
were
worried
about.
If
you
click,
this
button
doesn't
actually
take
effect
right
away.
It
should
down
so
I'm
gonna
toggle.
That
off
is
that
what
we
tell
people
in
the
documentation.
C
C
C
C
I'm
trying
to
see
and
make
sure
there's
all
so
Promethea
should
be
bringing
in
the
latest
script.
Okay,
oh
I
know
I.
Okay,
so
I
wanted
to
show
now
that
we've
got
three
prefix.
They
are
each
kind
of
paying
all
the
different
giddily
nodes
and
I
think
before
they
were
just
doing
this
in
memory,
I
think
John,
your
invitation
was
just
you
know,
one
of
the
prefect
and
see
if
you
see
here.
You
know
we
have
dear
friends
it's
kind
of
hard
to
see
here,
but
you
can
see
there's
two
right
now.
C
There's
two
prefix
notes
all
saying
guilty:
one
is
the
primary,
and
so
how
do
they
know?
How
do
they
agree
on
it
all
right?
Well,
they
use
this
database
now.
So
if
you
look
at
the
database
now,
there's
these
two
new
tables
prefect
note
status
and
prefix
shard
primaries.
So
the
note
status
is
drill.
It's
just
really
a
table
of.
C
It's
just
showing
I've
got
a
bunch
of
giddily
nodes,
and
so
you
see
if
we
took
one
two
and
three
on
the
left
side
and
then
you
see
like
which
nodes
get
only
one
two
two
and
three
and
then
each
row
represents,
like
hey
I
last
tried
to
talk
to
this
person.
This
is
when
I
last
saw
this
act.
Person
act.
There's
this
note
active.
C
So
you
can
see
we're
missing
one
here,
and
that
means
it
wasn't
able
to
talk
to
get
early
3,
so
good,
early,
3
well
actually,
and
the
reason
why
it's
giddily,
3
isn't
being
figure
yet
I
purposely
left
it
off.
If
you
actually
look
at
the
prefix,
this
is
prefect.
If
you
look
at
the
prefect
logs
you'll
see
that
the
health
checks
are
failing
for
that
nosy.
These
failures
here
so
I'm
gonna
go
ahead,
and
just
so,
if
you
look
at
the
gitlab
RB
here,
it's
pretty
much
empty.
C
It's
basically
the
default
config,
so
I'm
gonna
I
put
a
working
version
of
it
here,
all
the
right.
You
know
configuration
get
alene
able,
true
internal
IP
set
so
I'm
just
gonna
head
and
reconfigure
that
and
then,
when
we
do
that,
we
should
see
this
table
being
updated
once
it's
up
with
no
more
caps.
There
I
will
point
out
in
the
prefix
there's
new
configuration
that
this
isn't.
This
is
hasn't
been
merged,
that
isn't
an
omnibus
yet
but
for
the
purposes
stem
I'm
showing
you
which
configurations
are
being
used
now.
C
So
this
is
going
back
to
the
whole
sequel
election
process.
Here
now
I
mentioned
there
was
two
tables
on
this
database.
The
other
tables
show
is
the
elected
primary.
So
if
I
look
at
what's
there,
it's
a
single
row
right
now,
because
it's
saying
hey
the
shard,
what
we
called
prefect
Amanda
current
primary
is
giddily
one,
and
this
just
tells
you
say
this
Gilly
one,
just
the
prefect
one
decided.
This
is
going
to
be
the
the
leader
and
then
everybody
else
is
going
to
just
look
at
this
database.
C
E
C
Can
see
just
periodically
it's
there's
two
different
guy
queries.
This
one
is
just
like
telling
me
hey
who's
the
primary
right.
Obviously
we
did
that
select
statement
and
all
the
nodes
are
saying
hey.
It
was
a
primary
I'll
update
my
journal
state
to
reflect
that
periodically
and
then
the
other
one
is
I'm.
Just
gonna
update
the
state
of
what
I
see
when
I'm.
All
these
briefing
nodes
are
doing,
pings
based
health
checks
with
all
the
different
early
nodes,
and
then
it's
just
updating
its
database.
C
B
C
C
E
C
B
Mean
I
would
only
impacts
like
I
guess
like
a
transient
failure,
where,
like
some
other
node,
has
decided
to
elect
a
different
giddily
I
it
selected
Gilly,
but
then
immediately
after
Gilly
is
elected,
could
leave
one
becomes
accessible,
and
so
a
prefinished
that
hasn't
known
as
caliche
was
elected
continues
right.
Give
me
one,
and
you
know
it's
like
a
split
situation.
C
Yeah,
so
that's
a
split
brain!
That's
why
I'm
thinking
that
I'm
right
right
now,
the
current
merge
request,
does
just
update
this
internal
state
but
I
think
I'm
thinking
just
removing
that
and
just
passing
it
directly
to
the
database.
Every
time
we
need
to
know
who
the
actual
primary
is.
But
yeah
that's
a
question.
C
C
A
C
B
C
C
C
B
Sorry
go
ahead,
I
was
like
I
guess.
My
question
is:
do
you
think
we
should
just
sort
of
like
to
start
with
what
we
think
is
a
good
default
which
gives
us
more
flexibility
to
completely
change
and
rewrite
that
whole
mechanics
like
this
is
offering
some
sort
of
like
variable
configuration
which
allows
us
to
make
it
more
or
less
aggressive,
yeah
right
now.
C
But
yeah
the
thresholds
and
all
that
what
is
a
failure
right
now?
So
it's
so
this
table
is
being
used
as
kind
of
like
an
election
process,
because
it
will
only
consider
healthy
notes.
Is
that
this
be
that
have
two
or
more
prefix
saying
it's
up
right?
So,
if
you'll
actually
look
at
the
query
that
it's
running
this
query
here
is
basically
saying
give
me
the
list
of
active
nodes
that
to
prefect
nodes,
said
we're
up
right.
This
is
kind
of
like
your
quorum
right.
So
if.
E
C
B
Cool
I
guess
we
could
also
start
on
like
a
relatively
conservative
approach
anyway,
like
having
an
automatic
failover
after
a
node
being
inaccessible
by
majority
of
nodes
for
like
30
seconds
and
then
automatically
recovery
like
in
most
situations.
That
is
like
ideal,
like
it's
a
good
outcome
like
compared
to
like
a
complete
outage.
So
if
we're
comparing
the
current
situation
being
conservative
so
that
we
avoid,
like
node,
bouncing
like
every
couple
of
seconds
that
that's,
maybe
you
better
to
start
there
and
then
dial
it
down,
this
would
become
more
confident
methodology
right.
C
A
C
I
think
having
in
Prometheus
is
gonna,
be
like
the
number
one
priority
having
in
the
UI
is
kind
of
nice,
but
I
know
from
like
geo
experience.
The
UI
was
handy,
definitely
for
getting
kind
of
high-level
summaries,
but
knowing
when
fail
overs
happen,
these
grabs
are
much
more
important
right
having
like
timelines
and
knowing
which
nodes
are
aware,
especially
if
you
got
like
if
we
have
like
40
shards
right,
that's
going
to
be
a
lot
of
data
to
kind
of
have
to
show
on
our
admin
panel.
A
A
C
C
I,
don't
really
have
a
demo
to
bring
down
prefect
right
now,
cuz,
that's
a
dude!
That's
the
next
step
of
Foucault,
and
that's
that
you
know
we're
gonna
have
to
figure
that
out
as
well
right
like
if
we
use
like
a
load,
balancer
and
somehow
prefect
encounters
an
error.
How
do
we
bring
down
that
prefect
gracefully
so
that
it
doesn't
like
screw
everybody
else?
Oh
yeah.
B
C
B
A
B
Over
to
use
the
sequel
so
just
to
clarify
stand,
the
current
this
configuration
you've
got
is
actually
pointing
the
get
live
application,
just
a
prefect
one.
So
if
prefect
one
goes
down
right
exactly,
and
so
that
also
means
that
the
replication
queue
would
be
entirely
stored
in
memory
on
prefect
one.
That's.
B
C
C
E
E
C
D
E
D
D
I
was
just
wondering
the
the
consensus
I
haven't
visited
this
in
a
while,
since
I
think
the
original
discussions
came
up
on
using
consoles
and
all
that.
The
the
main
thing
that
we're
concerned
about
here
is
just
the
split
brain
where
we're
we're
sending
queries
to
a
primary
and
it's
not
getting
replicated,
because
it's
like
some
kind
of
network
isolation
right
where
we've
got
some
kind
of
we've
got
like
a
network
partition.
That's
that's
the
primary
issue
that
we're
concerned
about
when
we're
looking
through
the
demos
and
evaluating
did
the
leader
election
first
shard.
C
There's
a
couple
things
right:
first
of
all,
it's
you've
got
to
nominate
a
leader
right,
and
so
we
we
need
a
chain
prefect.
We
need
eh-eh
in
the
giddily
notes
right.
So
in
order
to
have
let's
say
you
know,
we
have
to
multiple
prefix
means
they
all
have
to
agree
on.
Who
is
going
to
be
the
primary
for
the
giddily
node
right,
so
we're
kind
of
tackling
a
bunch
of
problems
here,
we're
tackling
okay,
they
all
have
to
agree.
C
So
you
can't
have
it
in
memory
because
they
all
have
to
share
this
kind
of
the
consensus
that
a
believe
one
is
the
primary
right
and
then
the
second
thing
is
okay.
How
do
we
get
them
to
agree
on
which
one
is
up
right
so
that
my
first
iteration
was,
let's
just
you
know.
Basically
any
given
whatever
giddily
note
is
healthy
will
try
to
elect
that
as
like
the
first
I
was
like
the
primary
and
just
that
will
be,
that
will
be
the
primary
and
that
that
works.
C
That
works
okay
right
because
you
know
if
it's
just
like
a
greedy
thing,
where
everybody
just
tries
to
nominate
somebody
and
whoever
you
know,
whoever
happens
to
insert
it
into
the
table,
wins
like
that.
That
can
work,
but
then
what
happens
if
you
get
to
a
state
where
to
only
one
of
these
prefect
nodes,
can
actually
talk
to
that
giddily
node
right
now
you
have
a
problem
where
suddenly
like
that
node
is
clearly
broken,
but
to
the
two
out
of
the
three
noes
are
saying:
hey
somebody
else
should
be
the
leader
here
now.
C
What
do
you
do
right?
So
this
is
why
we
had.
Is
this
a
little
bit
more
complex
where
I
have
the
separate
table
and
then
I
I
do
a
query
based
on
activity
of
who
reports
in
here
to
actually
do
the
election
right?
So
everybody
is
doing
this
similar
query
I
mean
if
you
look
at
this
log,
everyone
is
basically
running
the
same
query
as
like
these
are
the
healthy
know.
This
is
a
healthy
node
query,
so
they
should,
in
theory,
get
consistent
results.
So
something
like
this-
you
know
the
actually.
This
is
not
yeah.
C
This
is
the
active
one.
So
assuming
that
you
know
the
there
aren't
any
there
isn't
too
much
craziness
with
no
it's
a
ping
times
like
everyone
should
be
reporting
in
within
10
seconds,
so
they
should
all
come
a
consistent
list
of
who's,
healthy
right
and
then
based
on
who's
healthy.
They
should
nominate
almost
the
same
node
each
time
right.
So
even
if
they
right
now,
the
friend
the
thing
is,
if
this
is
healthy,
if
this
entry
is
healthy,
it
doesn't
do
anything
right.
Okay,
I
agree
that
this
node
is
fine.
C
I'm,
not
gonna,
touch
it
right.
As
soon
as
we
get
to
the
point
where
this
node
ends
up
being
deemed
unhealthy,
then
somebody's
gonna
say
hey
that
guy
is
no
longer
good.
We
need
to
do
something
and
then
each
each
prefect
node
basically
do
the
same
query
and
they
should
come
up
with
the
same
result
and
say
you
know:
I'm
q2
goes
down
guilty.
Three
is
now
the
new
guy.
C
You
know
the
neg,
that's
the
next
step,
but
this
is
really
our.
You
know
our
bridge
gap
because
you
know
consoles
gonna
be
another
component.
We're
gonna
need
to
get
that
up
and
running.
Gonna
have
to
be
knows
of
that,
and
so
this
is
sort
of
the
interim
step
of
getting
helping
us
get
the
kill
eh-eh
without
having
to
introduce
a
whole
new
component.
C
B
It
also
provides
a
bridge
towards
a
cloud
native
approach,
because
it's
it
would
be
quite
possible
to
run
the
prefect
nodes
in
kubernetes
in
some
kind
of
configuration,
and
we
probably
don't
want
to
be
running
console
in
communities
when
kubernetes
provides
like
alternative
primitives.
That
may
be
able
to
solve
some
stuff.
So
yeah.
A
C
Anyway,
the
note
that
the
code,
you
merge
the
code
refactoring
that
allow
us
to
kind
of
plug
in
different
strategies.
So
hopefully
we
can.
You
know
eventually
settle
on
like
the
right
strategy
that
we
think
is
going
to
be
sort
of
the
majority
like
what
everybody
wants
to
do.
So
this
is
kind
of
a
stopgap
I'd.
D
C
That's
actually
that's
a
good
question.
I
mean
this
I
mean
this
is
what
this
table
is
for.
So
if
we
want
to
use
this
as
a
health
check
for
secondaries,
we
can
do
that
too.
Right
like
we
can.
We
can
start
to
mark
unhealthy
secondaries
as
well,
but
that's
like
yeah,
that's
the
main
focus
is
get
the
primary
right
and
then
obviously
we
need
to
figure
out
like
okay.
What
happens
when
a
secondary
does
go
down?
It's
probably
less
of
an
issue,
because
the
primary
right
is
the
most
important
thing
right
now
right.
D
Cool,
thank
you.
So
this
is
kind
of
a
step
back
I'm
wondering
do
we
actually
need
to
do.
We
actually
have
to
worry
about
split
brain
if
we
are
introducing
transactions
where
we
have
like
a
majority
quorum
that
is
getting
written
to
if
we
know
that
we
always
have
a
quorum
every
time
that
we're
you
know,
making
a
change
and
then
seeing
it
propagate
to
the
secondaries.
Do
we
still
need
to
worry
about
split
brain
I.
B
Think
so,
because,
like
the
quorum
doesn't
mean
every
node
agrees,
so
we
should
consider
the
situation
where
you've
got
like
some
repository
that
isn't
in
sync,
because
it
might
not
be
a
whole
node
level
failure
and
you
could
end
up
with
failing
over
to
a
different
get
early
node.
But
for
some
reason
the
most
recent
transaction
failed
on
that
node
for
one
of
the
repos.
B
D
B
What
a
replication
you
can
help
with,
because
that
can
essentially
become
a
repair
q,
so
the
replication
queue
essentially
looks
at
nodes
that
are
considered
good,
and
so,
if
we're
tracking,
with
fine
grained
resolution
in
tracking
database,
which
I
think
Pablo's
work
would
lead
like
we
can
know
like.
Oh
this
repo
is
on
this.
Shard
is
stale,
so
like
basically,
not
only
should
it
be
and
like
recovered,
but
also
excluded
from
transactions
for
the
time
being,
because
we
shouldn't
be
considering
bad
nodes
in
future
transactions.
Probably.
D
Yeah
good
point:
I'll
create
an
issue,
so
we
can
chat
more
about
that.
Async.
C
Yeah,
that's
interesting
question,
though
I
think
it'd
be
nice,
designed
the
system
where
like
if
we
did
have
a
split
brain
them
its.
It
may
be
mitigated
by
that
whole
consensus
of
like
three
phase
commit
right
like
okay,
I,
maybe
I
got
the
wrong.
Let's
say:
I
have
a
B
and
C
and
a
somebody
thinks
a
is
a
master,
but
P
be
somebody
else,
things
B!
Well.
If
they
happen
to
go
to
those
nodes,
they
all
say:
okay,
well,
you
know
agree
on
something
and
then
eventually
it
gets
persisted
anyway.
B
Yeah
I
was
gonna
say
like,
depending
on
the
situation
like
it
might
be
possible
to
like
try
and
recover
in
in
the
middle
of
the
transaction,
like
just
that.
A
force
might
decide
like
we've
reached
quorum
and
then
like
force,
not
agreeing
nodes
to
like
come
into
sync
in
the
transaction.
Because
then
you
can
read
like
you're
reducing
the
situations
where
like
and
if,
for
some
reason
that
no
refuses
and
the
stubborn
like
doesn't
come
into
consistency.
You
just
mark
the
whole
node
as
like
combat
because,
like
that
would
be
treated
as
like.
B
C
I
have
to
look
at
what
other
people
are
doing
about
deep
balancing
right
because,
like
you
know,
this
is
a
concern
that
we
have
in
this
case.
You
know
the
balancing
would
happen
if,
like
let's
say
you
know,
different
nodes
went
up
and
down
like
let's
say
like
the
network,
and
maybe
you
know
so
in
this
case,
like
two
out
of
the
two
out
of
the
three
nodes
have
to
say:
hey
that
guy
is
not
healthy
anymore
right
so
again
that
helps
mitigate
that,
because
it
was
just
one
prefect
node
saying
that
something
is
wrong.
C
Then
at
least
there's
a
little
more
stability
there
there.
You
know,
you
could
add
debouncing
by
you
know
backing
and
backing
off
your
fellow
thresholds
a
bit
like
if
it
happens
in
the
last
you
know
minute,
or
so
you
start
to
increase
the
threshold
at
which
you
actually
do
fill
up
her,
but
I
don't
know
like
week.
I
think
we're
gonna
have
to
look
at
what
other
products
are
doing
because
I
don't
think
but
I'm
guessing
Patroni
doesn't
do
anything
that
sophisticated
either
like
I
think
you
can
have.
C
Yeah
I'm
wondering
if
I'm
wondering
if
it's
you
know
we
could
get
to
the
point
where
ballot
balancing
isn't
as
big
of
a
deal
as
we
think
it
is
because,
let's
say
we
just
keep
rotating
primaries,
you
know
that
may
be
okay,
hopefully
isn't
such
a
such
a
pain
to
recover.
But
again,
like
I,
said
I,
don't
I
haven't
really
thought
through
all
the
kinds
of
things
that
we're
worried
about
here.
B
Yeah
I
mean
it's
a
good
point
like.
Maybe
it's
not
a
problem
to
be
solved
urgently.
If
we're
not
observing
you,
and
maybe
all
we
need
is
like
some
kill
switch
so
that
it's
like
really
easy
for
admin
to
be
like
just
stop
doing
automatic
fail
looks
like
we
were,
observe
we're
in
a
situation
where
some
really
bouncing
is
happening.
Just
stop,
maybe
there's
an
easy
way
to
turn
failover
off
a
crack
off
across
the
whole
fleet
or
like
a
specific
shard.
Then
we
got
em
to
run
reconfigure
on
all
three
nodes.
C
I'm
not
sure
we
want
to
configuration,
cuz
I
could
change
from
time
to
time,
but
maybe
what
we
need
to
do
is
keep
track
of
how
many
fail.
Overs
have
happened
in
the
last
like
a
minute
or
so,
and
that's
a
narrow
threshold
right
like
if,
if
we
exceed
that
threshold
by
some
amount,
we
Eve
there
stop
doing
a
failover
or
you'll
be
sent
on
an
alert,
and
we
do
something.
C
And
then
the
second
question
is:
do
you
raise
your
failure
threshold
see
like
like
right
now?
It's
a
notice
deem
inactive
if
it
has
not
reported
in
once
in
the
last
10
seconds
right.
That's,
you
know,
I
think
John's.
Your
memory,
one
does
a
little
bit
more
like
hey.
If
it's,
if
it
hasn't
passed,
three
health
checks
in
a
row,
then
it's
down
right,
like
we
can't
we
in
our
threshold.
C
We
can
either
like
you
know,
we
start
to
see
it
flapping,
so
we
start
either
increasing
the
threshold
by
which
you
decide
a
notice
up
or
down
right.
You
can
tweak
those
knobs
to
to
say,
hey,
look,
we're
not
gonna
fail
over,
because
you
know
somebody's
got
a
you
know
a
copy
network
and
silver.
You
know
hold
off
and
not
worry
about
that
until
they
really
are
getting
higher
rates.
So.
A
C
D
C
C
B
I'm,
just
thinking
about
like
get
lab
calm,
essentially
what
like
kill
switches
do
we
need
to
be
more
like
I,
think
we've
got
the
play
books
where
we
can
just
turn
it
off
by
like
running
reconfigure,
a
little
slow
though
right
so
like
what
would
that
mean?
I
also
know
100
the
other
day
I
was
like.
Oh,
how
do
you
feel
about
like
running
automatic
failover
in
production
once
we
have
that
running
in
the
next
like
2
weeks,
it
was
like
oh
yeah
feeling
pretty
good
about
it.
Like
I
was
like
okay,
that's
surprising.
B
But
also
glad
you
have
such
confidence
in
the
team
because
they
are
all
smart
people
good
on
my
resume.
Welcome
to
as
a
product
manager
like
if
there
was
like
some
sort
of
really
fast
kill
switch
where
we
could
like
see
a
problem
and
just
be
like
slack,
stop
and
then
like
prevent
that
being
a
problem
and.
C
B
The
kill
switch
is
not
a
bad
idea
like
just
having
some
sort
of
thing
like
it's
a
breach
of
some
level
of
flapping,
then
just
like
disable
automatic
failover
before
so
I'm
like
enormous
amount
of
time.
It's
like
just
a
database
row
that
basically
says
like
favor
is
enabled
for
like
24
hours,
was
it
disabled
for
24
hours
or
like
6
hours
until
someone
could
like
reevaluate
the
situation.
B
D
C
D
That's
what
I'm
thinking
is
I
mean,
maybe
the
bounce
may
be
bouncing
around-
is
really
what
we
want,
because
if
we
can't
communicate
with
something,
why
would
we
stay
elected
to
it?
I
I
would
think.
If
we
saw
a
lot
of
bouncing,
we
would
just
want
to
kind
of
increase
the
amount
of
time
it
takes
to
make
a
decision
before
the
next
bounce.
D
C
D
D
C
Right
now
it
just
so,
it's
actually
like
the
way
this
thing
works
right
now
is
it
will
sort
by
the
name
right
now
it
has
sending
so
it'll
sort
by
the
number
of
prefect
nodes.
That
say
that
thing
is
up
and
then
also
type
breaks
are
made
by
just
a
name
right
now,
it's
you
know.
Obviously
we
can
make
that
random
if
we
care,
if
we
wanted
to.
C
C
C
B
C
B
I'm
thinking
more
about
like
in
that
admin
settings
area,
it
would
be
interesting
to
have
like
some
failure.
The
configuration
settings,
like
if
they's
alike,
can
I
level
on
and
off
like
turn,
because
then
you
can
expose
it
through
like
the
standard
D
lab
IEP
is,
and
then
you
don't
have
to
have
like
some
special
tooling.
That's
like
kind
of
unique
is
more
what
I'm
thinking
right.