►
From YouTube: Databases on Kubernetes Using a Custom Operator: Day 1, Day 2, and Beyond - Johannes Unterstein
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Databases on Kubernetes Using a Custom Operator: Day 1, Day 2, and Beyond - Johannes Unterstein, Neo4j
We started the journey of building a managed cloud version of the graph database Neo4j. A bit later we started developing an operator to manage multiple database clusters in k8s. Handling persistence and Neo4j's own distributed consensus algorithm within k8s gave us a challenge. In this session we want to share the lessons we learned writing this operator and using it in production. We will start with how to get started using the k8s controller tooling to create an operator to manage a CRD. We go beyond the "day 1" tasks of creating and deleting databases and discuss how we meet "day 2" concerns such as: - Unit testing our operator using k8s fakes. - Continuously deploying an operator into a GKE cluster. - Automatic rolling updates of Neo4j databases with zero downtime and fault tolerance. - Database administration (backup, restore, password resets etc.) via an operator.
https://sched.co/MPYU
A
Yeah,
thank
you
for
the
introduction,
the
next
30
minutes.
We
talk
about
databases
on
kubernetes,
using
a
custom
operator
on
day,
one
two
and
a
year
later,
and
actually
we
do
have
databases
running
in
communities
for
a
year.
Now,
though,
before
we
begin,
this
is
me.
My
name
is
Johannes
I'm
working
for
a
database
company.
The
data
this
company
is
called
neo4j.
It's
a
graph
database.
A
We
store
things
a
bit
differently
than
you
would
expect
from
a
relational
database,
but
actually
I'm,
not
a
database
guy
and
more
like
a
container
guy,
which
is
good
because
we
keep
conrad
and
we
do
have
a
team
at
near
forge
a
building
near
future
cloud.
It's
kind
of
a
managed
neo4j
as
a
service
database
as
a
service,
and
we
want
to
talk
about
this
today
about
our
story.
Building
this
offering,
but
maybe
before
we
begin
give
me
a
short
hands
up.
Do
you
building
operators
on
your
own?
Do
you
have
experience
with
that?
A
All
right,
so
they're
quite
some
hands,
and
this
is
amazing,
see
if
I'm
telling
something
that
didn't
work
out
for
you
at
all.
Please
start
the
conversation
with
me.
I
would
love
to
hear
your
story,
what
worked
for
you
and
what
worked?
Not
for
you,
because
I
can
only
tell
you
what
worked
for
us
and
what
didn't
work
for
us
and
I
would
super
interested
in
what
worked
for.
You
is
one
okay,
so
new
for
what
you
maybe
seen
this
logo
before.
A
Let
me
shortly
introduce
Naver
J
because
it
helps
the
story,
so
you
may
have
been
heard
about
the
Panama
papers,
so
this
was
about
taxing
the
fraud
system,
so
people
try
to
hide
the
money
from
the
text
system
and
they
usually
do
this
by
opening
a
bank
account
in
the
Bahamas,
and
this
case
on
Panama
and
a
few
years
ago
there
was
a
league
of
a
law
company
in
Panama.
They
leaked
at
11
and
a
half
million
document.
A
So
they
recognized
that
they
didn't
work
out
that
well
so
because
they
tried
to
find
patterns
of
people
trying
to
rob
system,
and
in
that
case
they
try
to
find
the
pattern
of
persons
that
has
a
bank
account
and
the
bank
account
is
in
a
bank
in
Panama,
so
they
didn't
found
annual
any
of
those
people
trying
to
throw
the
Texas
and
because
people
who
want
to
hide
loads
of
Texas
are
smarter
than
that.
So
they
search
deeper
and
find
another
pattern.
A
So
there
was
the
pattern
of
an
address
where
the
people
lived,
and
there
was
another
one
living
at
the
same
address
and
the
other
one
has
was
officer.
The
company
and
the
company
has
a
registered
bank
account
in
Panama
and
whoever
tried
to
do
join
over
70
tables
in
relation
database
probably
know
that
this
is
can
be
potentially
quite
painful,
but
this
is
the
innate
a
model
of
nature
for
days.
So
we
do
have
notes
like
a
person's
and
notes
could
have
classes
it's
optional,
but
there
can
have.
Relations
are
really
easy
to
traverse
relationships.
A
But
the
great
thing
is
once
you
have
found
the
pattern
of
those
seven
nodes.
You
can
query
all
of
your
data
set
or
the
eleven
and
a
half
million
documents,
and
you
get
all
the
people
who
try
to
probe
the
text
system.
So
all
the
Formula
One
drivers
who
brought
the
text
system,
it
didn't
pay
taxes
in
the
home
country
or
the
football
players
or
the
Russian
nobleman,
so
they
were
found
through
the
journalism's
and
they
use
neo4j
for
that
which
is
really
cool
and
they're.
A
Also,
doctors
trying
to
find
the
cure
for
cancer
and
they're,
putting
our
patient
data
in
there
for
Jay
and
their
treatments
and
how
this
worked
out
and
tried
to
find
a
pattern
to
help
cure
cancer
under
NASA
is
putting
all
the
engineering
data
like
decades
of
engineering
data
in
in
a
graph
to
build
up
a
knowledge
graph
to
help
to
bring
people
to
the
mass.
So
this
is
really
really
exciting,
but
we
also
have
more
enterprise.
Use
cases,
for
example,
for
protection
is
highly
it's
really
good,
doable
by
pattern
matching
or
recommendations.
A
So
these
are
more
enterprise-e
use
cases,
but
what
all
of
those
users
have
in
common
that
they
put
their
mission-critical
data
in
a
database?
But
it's
not
their
core
core
business
to
operate
a
database
to
make
it
away
with
a
twenty
4:7.
They
needed
24/7,
but
it's
not
their
core
business,
and
this
is
why
people
tend
to
like,
as
a
service
product,
a
data
basis
as
a
services
monitoring
as
a
service
metric
system
like
all
the
products
there
in
the
exhibition
hall.
A
So
we
decided
to
build
and
manage
database
as
a
service
offering
and
we
started
over
two
years
ago
and
we
started
with
a
tiny
web
application
and
in
the
web
application
the
user
was
had
a
button
to
start
an
FHA
instance.
So
we
provisioned
an
ec2
instance
started
a
single
instance
neo4j
database
on
it
and
then
report
at
the
end
point
back
to
the
user.
A
This
was
a
first
iteration
of
the
product
and
where
did
invite-only
beta
we're
roughly
50
customers
using
this,
and
there
was
no
too
bad,
so
they,
this
was
okayish,
but
it
had
some
downsides
to
it.
The
first
down
said:
is
we
forced
the
customer
to
give
us
a
downtown
and
maintenance
window,
while
the
customer
allowed
us
to
take
the
database
down
to
do,
updates
and
fixes
and
whatever
and
I,
don't
think
this
is
the
way
to
offer
a
database
a
service
offering
in
218
or
to
19.
A
Furthermore,
we
relied
on
the
availability
on
ec2,
which
wasn't
that
bad
either,
except
we
had
some
problems
in
August
last
year.
We
had
really
tough
time,
but
I
don't
want
to
talk
about
that
today
and
we
relied
on
for
tolerance
on
EBS.
So
we
use
block
storage
to
set
store
data
and
again
sometimes
we
had
hiccups
in
EBS,
where
it
took
up
to
30
minutes
to
unknown
the
EBS
from
a
failing
instance
to
remount
it
to
a
new
one.
So
this
was
also
not
really.
Let's
say
that
way.
A
Our
Christmas
were
not
really
happy
about
that.
So
we
wanted
to
go
away
from
single
instance
to
clusters,
because
clusters
promised
to
have
zero
downtime,
because
we
can
do
reroll
in
updates
on
the
fly
and
don't
expose
down
times
to
our
customers
and
again,
downtime
was
really
bad
and
availability
is
probably
not
the
best
thing
in
single
instance,
and
while
we
were
operating
all
the
single
instances
in
easy,
we
were
building
all
our
or
management
services,
the
UI
services,
the
backend
or
the
workers.
We
deployed
that
on
kubernetes
who
were
operating
this
on
kubernetes.
A
We
opted
in
for
manage
given
IDs
on
Google,
so
we
use
GK
for
that
because
at
the
time
we
started
it
was
a
really
good
and
easy
way
for
us
to
get
started.
So
we
gained
experience
over
one
year
running
stateless
services
in
communities
and
at
that
time,
unfortunately,
there
was
no
graph
database
as
a
service
offering,
so
we
also
bought
in
relational
database
as
a
service,
because
at
that
point
it
was
not
our
business
to
operate
a
relational
database.
A
So
we
just
bought
it
in
at
that
time
and
currently
we
are
evaluating
to
change
it
to
our
own
product,
which
is
super
super
funny
anyways.
So
we
had
experience
in
communities
running
statistics
services
and
then
we
thought
we
want
to
do
clusters,
but
we
don't
want
to
do
crust
this
on
ec2
and
doing
all
the
scheduling
on
our
own.
So
we
thought
we're
gonna
leverage,
our
communities,
the
scheduling,
our
tools
in
kubernetes
and
then
Kelsey
Hightower
gave
presentation
most.
B
A
Yeah
I'm
still
happily
employed,
but
this
was
too
thick
16.
So
Kelsey
is
right.
So
there's
more
in
the
talk,
if
you,
if
you
want
check
it
out
us
from
hash
econ
4
to
16,
but
then
this
just
the
backup
in
case
the
video
didn't
work.
But
last
year
Kelsey
said
companies
made
huge
improvements
to
run
stateful
workouts,
including
databases
and
managed
cue.
A
Our
message
queues,
but
I
still
prefer
not
to
do
it
fair
enough
and
on
the
same
day,
in
the
further
threat,
Kelsey
said,
I
think
it's
important
to
remember
that
kubernetes
only
solves
half
of
a
problem.
The
other
parts
must
be
solved
by
stay
for
services
and
operational
experience
and
I.
Think
he's
very
true,
and
the
great
thing
is
we
had
loads
of
experience,
operational
experience
in
running
databases,
and
we
were
confident
that
we
want
to
put
it
in
communities
and
we
wanted
to
run
clusters.
A
So
let's
have
a
short
look
about
clusters
in
there
for
J
Nia
J
is
a
fully
replicated
database.
So
all
the
nodes
have
all
the
data.
Therefore,
J
implements
the
Ralphs
protocol.
It's
basically
a
distributed
algorithm
to
do
distributed.
Consensus
so
commits
in
a
distributed
system,
and
it's
basically
like
praxis
just
a
bit
weaker.
So
we
have
a
leader
follower
concept.
A
We
are
running
all
over
near
4-day
instances
in
a
dedicated
step
for
said,
so
we
do
have
three
stateful
sets
for
cluster
of
three
notes:
we're
using
persistent
volumes
offered
by
GK
to
store
the
data
directories
and
we
do
backup
we
would
be
using
backup
workers
and
the
end
goal
is
were
not
quite
there
yet
in
the
fermentation.
But
the
end
goal
is
that
we
have
the
backup
workers
and
the
backup
worker
is
doing
backups
constantly,
but
he's
not
doing
a
full
backup
he's
doing
incremental
backups.
A
A
So
when
we
have
that
that
we're
running
backups
constantly
all
the
time,
we
maybe
want
to
get
rid
of
the
PBS
of
the
main
cluster
to
gain
more
performance,
because
then
we
actually
don't
need
it.
When
we
have
the
backup
working
constantly
doing
the
backups
and
the
cluster
is
faltering
by
itself,
but
we're
not
quite
there
yet
and
we're
not
decided
if
we
want
to
go
that
step
or
not.
So,
let's
see
and
wait
for
the
future
okay,
but
this
cluster
technology.
A
So
let's
dive
in
communities
and
the
operator
we
started
the
operator
building
operator
about
one
year
ago,
a
bit
before
Kubek
on
Europe
last
year,
and
we
decided
to
use
the
go
language
to
do
it
as
a
neo4j
is
written
in
Java.
Most
of
the
other
stuff
is
written
in
Python
or
bash
and
other
languages.
So
we
didn't
have
go
experience
at
all.
A
If
you
know
it's
just
so
thin
that
we
didn't
fell
that
we
want
to
do
it
in
the
last
year,
so
we
sticked
with
a
vanilla
coaching
generated
coins
from
the
kubernetes
code
and
in
the
data
model
of
kubernetes,
and
you
probably
all
know
that
you
have
like
a
loose
set
of
resources.
So
you
have
deployments
replica,
set
spots,
stateful
services
and
so
on,
and
the
great
thing
about
all
those
resources
is
that
you
have
a
controller
attached
on
every
abstraction
layer.
A
So
you
will
have
a
stateful
set
controller
who's,
doing
a
constant
task,
reconciliation
and
reconciliation.
Just
a
fancy
word
of
comparing
the
current
state
to
the
target
state.
So
he
would
ask
for
the
CDs
and
then
he
would
dis
calculate
the
target
state
if
this
is
not
matching.
What
is
currently
running
in
kubernetes,
the
stateful
set
controller
would
decide
to
start
a
pot
and
he
does
that,
but
just
creating
a
new
pot
resource.
A
So
then
the
pot
controller
would
kick
in
and
see
that
there's
a
new
port
in
the
targets
de
that
is
not
in
the
current
state.
So
then
the
port
controller
will
try
to
find
a
Koopa
to
stop
the
pod
and
so
on
and
so
on.
And
the
great
thing
here
is
that
we
can
extend
the
data
model,
so
we
can
just
add
a
new
type
and
call
it
near
for
a
database,
and
we
wrote
this
controller
logic
so
that
a
client
could
do
cube.
A
Cut'
will
apply
whatever
thing
you
want
supply
in
forge
a
database
and
underneath
for
Jo
a
controller
will
kick
in
and
see.
Oh
there's
something
in
my
target
state
that
it's
not
in
my
running
current
state,
so
I
need
to
start
a
new
stateful
set,
for
example,
and
then
the
state
for
such
controller,
which
again
and
do
its
thing
and
then
a
pod
controller,
and
so
on
answer-
and
this
really
creates
we
re
using
loads
of
logic
from
kubernetes
itself,
which
is
super
helpful
for
building
this
product.
A
But
as
we
were
developing
the
operator,
the
this
reconciliation
logic
got
bigger
and
bigger
and
bigger,
and
we
thought
that
we
want
to
change
that.
So
we
split
down
the
whole
reconciliation
logic
in
things
and,
let's
call
it
mini
reconcile
for
now.
So
we're
splitting
out
logic
for
dedicated
things
in
me
mini
reconcile
us.
So
we
have
a
dedicated
mini
reconciler
for
backups
or
mini
reconciler
for
load.
So
when
the
user
requests
the
loader
database
or
restore
a
database,
we
have
a
dedicated
mini
reconcile
of
config
Maps
offer
certificates
or
the
communication
is
encrypted.
A
So
we
haven't
reconcile
that
to
create
the
certificates
and
put
it
in
secrets
or
for
DNS
entries
so
that
the
end
user
connects
the
database
from
the
outside
and
for
the
instances
and
for
eviction
and
pdb.
So
we
just
added
last
week
a
new
reconciler
for
eviction
and
we'll
talk
about
a
bit
later.
So
we
want
to
talk
to
the
communities
the
GK
autoscaler
I
want
to
give
him
hints
when
it's
safe
to
clean
pots
and
and
whatnot,
and
we
talk
late
a
bit
about
it.
So
we
separated
this.
A
So
our
main
reconciliation
said
the
main
reconciler
basically
just
builds
up
the
current
state
and
the
target
state,
and
he
does
that
by
asking
the
mini
reconcile
s
for
their
view
on
the
world.
So
the
main
reconciler
would
ask
the
config
mate
reconcile
for
example.
What
is
your
current
state
and
what
is
your
target
state?
So
he
would
gather
all
those
many
information
from
the
mini
reconcile.
A
I
would
aggregated
and
declare
this
the
whole
current
state
and
the
whole
target
state,
and
then
he
would
loop
over
all
the
mini
reconcile
s,
so
they
are
ordered
and
when
everything
finished
completely
the
reconciler,
a
reconciliation
for
that
database
would
be
marked
as
good
ago.
No,
nothing
needs
to
be
changed
or
maybe
there's
an
error
on
the
way.
So
one
of
the
reconciler
marks
an
error.
A
He
can't
talk
to
communities
or
whatever,
so
reconciliation
would
stop
at
that
point,
and
the
reconciler
would
fall
to
sleep
and
then
maybe
picks
that
database
up
in
the
next
reconciliation
loop
or
maybe
one
of
the
mini,
reconciles
requesters
changes.
So
we
separate
the
mini
reconcile
us
are
talking
to
cabana
DS,
but
they're
reporting
back
to
the
main
reconciler
that
they've
changed
something
so
that
the
main
reconsider
would
also
step
stopped.
A
At
that
point,
when
one
of
the
mini
reconciler
requested
a
change
and
will
pick
it
up
in
the
next
reconciliation
who
began
to
give
some
time
for
the
current
action
to
take
into
place
and
we're
pretty
happy
about
the
current
situation.
Maybe
it's
not
the
most
performant
one.
So
maybe,
as
some
one
of
you
have
some
suggestions,
how
we
maybe
improve
that,
but
so
far
it
was
pretty
reliable
for
us
and
let's
dive
into
some
examples.
So
this
box
is
a
CID.
A
So
someone
creates
a
new
co
DS
saying
I
want
to
have
an
earful
j
database,
so
the
reconciler
would
pick
it
up
and
decide
that
he
wants
to
start
three
databases
in
the
target
state.
So
he
would
start
the
first
instance
white
until
is
healthy
and
then
the
second
one
and
then
the
third
one,
and
then
he
would
mark
the
entire
our
CID
as
healthy,
so
we're
using
conditions
for
that.
We
come
back
to
it
later,
but
let's
do
the
next
example
sewer
heal.
A
So
we
have
three
running
healthy
instances,
so
everything
is
fine,
but
all
of
a
sudden
one
of
the
pots
has
a
problem.
So
it
goes
back
to
two
running
instances
that
are
healthy
and
we
want
to
start
a
third
one
to
go
back
to
the
desired
state.
We
have
three
running
instances,
so
then
the
operator
would
just
start
a
new
port
wait
under
it's
healthy,
undone,
dijo
of
theory
would
be
marked
as
healthy,
and
the
great
thing
here
about
the
this
reconciliation
pattern
is
that
this
is
the
exact
same
logic
for
create
at
healing.
A
So
we
don't
have
dedicated
failure
handling
here
because
we're
doing
target
and
his
target
state
calculation.
So
this
are
the
exact
same
steps
here.
So
this
is
really
amazing
for
us
and
if
you
think
about
how
you
would
do
this
in
the
shell
script,
for
example,
this
potentially
could
pretty
messy,
but
in
the
undiscovered
Eddie's
way
of
engineering.
This
is
really
clean
and
straightforward.
Okay,
let's
look
how
we
update
a
database,
how
we
do
a
zero
downtime
rolling
update
to
a
database.
A
So
when
we
request
a
change
to
the
CID-
and
this
could
have
multiple
reasons,
for
example
this
in
Union
for
JV
version
or
we
have
changed
resources,
so
the
customer
wants
a
bigger
database.
You
wants
more
CPU
or
more
memory
or
whatever,
or
he
wants
to
read
that
the
password
so
neo4j
itself
has
user
management,
but
maybe
the
customer
loss
are
the
master
password,
so
we
can
reserve
that
for
the
customer,
so
we
want
to
change
it
and
that's
a
simple
example.
A
The
lowest
note
is
the
leader,
and
the
idea
here
is
to
print
leader
elections.
So
the
leader,
the
one
who
can
talk
to
for
write,
requests
can
change,
but
when
you
do
an
update,
you
want
to
not
do
this.
You
don't
want
to
do
more
confusing
in
the
cluster,
so
we
want
to
preserve
the
leader
as
long
as
possible.
A
Our
strategy
here
is
to,
firstly,
start
a
new
pot
so
that
we
go
up
to
four
as
you
notice,
the
majority
changed
now
so
in
a
class
out
of
three,
you
need
the
majority
of
two
nodes
to
acknowledge
and
commit.
But
when
you
have
four
nodes
in
the
cluster,
the
majority
all
of
a
sudden
is
three
so
near
itself
would
cope
with
that,
so
that
you
don't
run
into
sprain
scenarios
or
something.
So
now
you
have
a
higher
quorum
sauce
and
if
that
node
becomes
healthy,
the
operator
would
start
a
new
one.
A
So
then
we
have
five,
this
core
mister
three-year
and
then
he
would
start
to
stop
old
notes,
so
he
would
start
a
follower.
So
we
again
we
want
to
keep
the
leader
being
a
leader,
so
he
would
start
a
follower
and
then
we
would
go
up
to
five
again
wait
under
there
notice
how
see
and
then
we
have
here
we
have
the
target
state.
So
this
is
the
Custer.
A
We
want
to
have
three
healthy
new
nodes
in
the
cluster
and
then
the
operator
would
stop
one
of
the
followers
and
then
he
would
trigger
a
leader
application.
That
means
he
asked
that
that
know
to
step
down
as
a
leader,
so
that
someone
else
can
pick
up
the
leadership
and
after
that
happened
we
stopped
that
node
and
then
our
update
is
completed
and
we
can
mark
this
year,
Diaz
healthy
again
great.
A
The
next
logical
step
is
to
update
the
operator
and
our
first
iteration
was.
We
just
deployed
the
new
operator
code.
The
problem
here
was,
assuming
you
have
I,
don't
know.
50
100
running
databases,
so
the
operator,
maybe
you
added
a
new
feature-
detected
a
change
in
the
current
state
and
the
target
state,
so
he
would
loop
over
all
hundred
databases
and
detect
that
there's
something
to
be
done,
so
he
would
do
one
baby
step
start
a
new
pot
for
every
of
those
hundred
databases.
A
So
all
of
a
sudden
you
have
hundred
new
stateful
sets
to
be
started.
Probably
don't
have
enough
capacity
in
you
come
and
I
discussed,
sir.
So
this
would
kick
in
the
GK
autoscaler,
so
you
will
get
loads
of
new
couplets
to
complete
your
deployment
and
at
the
end
you
have
loads
of
key
loads
of
new
couplets
and
you
have
a
low
utilization
and
this
will
cost
lots
of
money.
You
probably
don't
want
to
do
this,
maybe
you
maybe
you
want
we.
We
didn't
want
it
to
spend
that
much
money,
so
we
introduced
a
number.
A
So
the
resources
are
aware
of
the
operator
number
that
is
currently
responsible
for
them
and
the
operator
itself
is
aware
of
its
current
number.
So
the
operator
with
the
number
42,
for
example,
would
only
pick
up
databases
that
are
assigned
to
that
operator
and
when
we
deploy
new,
your
praetor
operator
42.
The
first
thing
he
does
is
nothing
because
you
don't
have
any
resources
attached
and
now
we
ordered
the
data.
A
So
if
we
then
change,
maybe
20%
of
our
databases
of
the
low
priority
to
operate
a
42,
he
would
pick
it
up.
He
would
see
those
resources
in
the
reconciliation
loop
and
he
would
roll
in
like
the
first
row
of
clusters,
he
would
do
the
rolling
update
we
just
described
and
then
we
would
do
it
for
me.
The
improper
tea,
and
maybe
lastly,
for
the
high
priority
and
that
everything
went
good
and
no
databases
are
left
for
the
old
operator.
A
We
can
then
safely
shut
down
the
old
operator
version
because
it's
not
needed
anymore,
and
this
is
how
we
do
like
rolling
deployments
off
the
operator,
but
there's
so
much
more
to
talk
about
so
we're
in
23
minutes.
So
we
have
some
time
left,
there's
so
much
more
to
talk
about
so
we're
using
heavily
conditions
to
reflect
if
a
Custer
is
fourth
ruined
or
available.
So
in
every
conciliation
you
we
ask
the
cluster
for
higher
level
metrics.
A
So
if
it's
the
cluster
available,
so
do
we
have
enough
nodes
inside
the
raft
group
to
perform
right
requests
or
if
it's
for
pterence,
we
have
quorum
size
plus
one
and
we
store
it
in
conditions
in
the
CD
and
after
reports
were
using
the
conditions
in
metrics,
so
we're
building
dashboards
out
of
it
if
the
databases
are
photo
ruined
or
if
they
are
available,
so
that
our
SRE
team
can
kick
in
when
a
database
is
only
for.
Turrent
is
only
available
for
let's
say
five
minutes,
but
not
for
turrent.
A
They
can
actively
go
into
and
fix
the
problem
to
bring
the
database
back
to
four
pterence
before
it
loses
the
available
another
tea
that
he
was
to
do
this
for
eviction
and
put
disruption,
budgets
so
put
disruption.
Budget
basically
says
for
the
autoscaler
when
you
want
in
to
roll
in
a
new
notes,
note
pool
through
your
cluster
that
cluster
out
of
three
nodes
is
allowed,
that
you
can
take
one
note
down
at
a
time
and
to
like
to
keep
the
cluster
alive,
but
this
is
only
for
the
happy
case.
A
So
what
if
one
of
the
nodes
is
currently
catching
up,
so
the
cluster
can't
sacrifice
any
of
the
nodes
you
can't
yeah.
You
could
do
it
with
the
pot
disruption
budget,
but
there's
an
annotation.
You
can
set
to
a
pot
to
tell
the
kubernetes
autoscaler
to
not
touch
it
when
rolling
in
new
a
new
node
pool-
and
this
is
the
safe
to
evict
annotation-
so
have
a
reconciler
doing
this.
A
Every
reconciliation
you
checking
if
the
database
is
for
torrent
and
if
the,
if
the
cluster
is
not
for
through
and
setting
the
annotation
to
stop
the
autoscaler
from
killing
node
killing
pots
and
bringing
the
cluster
in
a
not
really
good
situation
and
we
need
to
talk
about
site
casts.
So
we
try
to
stick
to
the
official
neo4j
docker
images
as
close
as
possible
and
we're
using
the
official
name
of
which
a
docker
images
but
there's
loads
of
stuff
inside
guys.
Going
on
that,
we
needed
to
add
to
make
that
happen.
A
So
we
have
cycles
for
logging
for
metrics,
for
initialization,
for
debug
ability
and
all
that
kind
of
stuff,
because
we
wanted
to
be
compatible
to
the
official
release.
So
when
there's
the
new
release,
we
can
get
it
right
away
without
going
through
an
insane
build
pipeline
on
our
own.
So
we're
using
the
official
images
for
that
and
we
added
a
thing
for
debug
ability.
A
So
this
is
a
really
cool,
because
when
our
operator
was
in
a
state
where
it
was
not
as
good
as
it
is
today,
so
when
we
needed
to
execute
in
the
cluster
quite
often
to
debug
why
something
is
not
working
properly.
We
have
the
problem
that
we
were
in
a
race
condition
with
the
operator
trying
to
hear
the
custom.
So
we
have
a
lot
of
time
that
we
executed
in
the
inner
pod
to
Ted
why
it's
not
working,
and
then
the
operator
killed
the
instance
underneath
our
shells.
A
So
we
added
a
simple
annotation
and
we
called
it.
We
did,
we
didn't
call
it
doing
a
touch
mode,
but
it
is
a
dinner
touch
mode.
If
the
annotation
is
present,
the
operator
won't
touch
any
of
the
pots
or
any
of
the
CDs,
so
it
will
leave
it
alone
so
that
we
can
go
into
and
repair
what.
We
think
we
need
to
do
to
bring
it
back
to
life,
and
then
we
remove
the
annotation
the
operator
kicks
in
again
for
that
cluster
and
then
take
takes
over
the
the
operation
for
the
database.
A
Cool
is
everything
happy,
then
kind
of
so
we
lost
loads
of
pots
over
the
time
loads
of
containers.
But
we
didn't
didn't
lost
data
because
we
we
had
the
Peavey's
in
place
and
we
had
the
backups
in
place.
So
this
was
really
good.
So,
as
I
said,
we
had
some
tough
time
when
we
had
problems
like
we're.
Doing
a
rolling
deployment
or
operator
code
is
not
as
good
as
today
and
then
the
autoscaler
kicks
in
all
of
a
sudden
and
kills
some
pots,
and
sometimes
he
killed
the
wrong
one.
A
So
the
cluster
was
stuck
so
then
we
needed
to
go
in
and
recover
the
cluster
manually
until
the
cluster
okay
shut
down
those
two
ports
at
the
Doudna
touch:
annotation
first
shut
down
those
two
ports
then
mark
that
node
as
you're
the
survivor
remove
the
annotation,
so
the
operator
kicks
in
and
then
deuced
the
thing
we
have
one
healthy
instance
and
we
want
to
have
two
more
and
then
he's
doing
basically
the
create
a
government
again.
So
we
had
loads
of
this
and.
B
A
A
Okay,
so
we
have
some
time
left
before
the
question
so
would
shortly
go
over
testing
because
we're
you
really
feel
strongly
about
testing
our
operator
code
and
I
last
year
in
Cuba,
and
there
was
loads
of
discussions
about
testing
so
one
to
give
you
a
short
overview.
How
we
do
it,
so
we
do
testing
on
different
levels.
So
the
first
thing
is
unit
testing,
so
we
have
loads
of
table
tests.
So
please
be
patient
I'm
coming
from
a
Java
backgrounds,
we
don't
have
table
tests
and
super
excited
about
table
tests.
A
So
what
we
do
here
is
we
describe
a
fixture,
so
the
current
state,
and
then
we
describe
our
expectation
and
then
we
have
a
tiny
function
like
if
the
cluster
is
available.
So
the
calculation,
if
I,
have
at
least
quorum
size
of
available
notes-
and
this
is
looping,
although
all
of
those
tests
in
the
table
test.
So
we
have
hundreds
of
those
and
the
next
step
up
would
be
into
what
we
call
integration
tests.
So
we
mock
the
entire
cabinet
API
and
in
go
language,
there
is
a
library
to
also
make
those
calls
Bibles.
A
So
you
can
assert
that
something
was
called
in
that
case
we're
testing
a
conflict
map.
The
actual
state
is
no
conflict.
Map
is
present
the
desired
status
that
we
want
to
have
a
conflict
map,
so
we're
mocking
all
the
stuff.
Then
we
ask
the
mini
reconcile
of
config
Maps.
Please
do
the
reconciliation
and
we
expect
that
this
was
true.
So
the
the
reconciliation
returned
true
and
we
assert
that
create
config
map
from
the
communities.
Api
was
called.
A
Ok,
so
the
next
logical
step
up
would
be
what
we
call
system
integration
tests
where
you
have
here,
Yama
files-
and
you
do
Q
cuttle-
apply
in
Yama
file,
and
then
you
expect
a
running
cluster.
Maybe
five
minutes
later
two
minutes
later.
Whatever
timespan
later,
we
don't
have
that.
In
that
case,
what
we
have
we
have
loads
of
automated
and
turn
tests.
So
we
heard
under
an
test
from
our
previous
systems
and
those
covered
all
the
user
stories
we
defined
and
those
automated
and
turn
tests.
A
This
is
also
really
cool
and
then,
as
I
tease
that
maybe
before
we
are
running
Canaries,
so
in
every
environment,
staging
production
and
personal
environments,
we've
run
the
canary
databases,
so
data
with
no
customer
touched
it,
but
other
processes
running
constant
load
against
it
and
we
run
in
chaos
monkeys
against
those
canary
databases
in
staging
in
in
production,
and
this
gave
us
so
much
valuable
insights
about
the
cost
of
technology.
How
to
handle
failure.
How
clients
would
react
to
failure,
so
we
learn
loads
of
those
using
chaos,
monkeys
and
also
in
production.
A
C
Hey
thanks
for
your
talk.
You
talked
a
little
bit
about
leadership.
Election,
yes
aware,
operation
of
you
know
things.
Could
you
talk
a
little
more
about
the
data
aspect
of
operating
databases
so,
for
instance,
when
you
kill
nodes
and
spawn
new
notes,
you
probably
want
to
make
sure
they
spawn
in
a
location
where
the
data
is
available
to
limit
copying
stuff
around.
How
do
you
deal
with
things
like
corruptions
at
the
data
level?
C
A
A
I
hope
I
got
you
a
question
correct,
so
it
was
quite
noisy
here.
So
if
I
go
to
correct
the
question
was
how
to
deal
with
data
corruption
in
case
of
failure,
so
in
the
in
the
raft
protocol
and
make
sure
that
you,
but
you
only
can
talk
to
the
leader,
and
if
you
don't
have
leadership,
you
can't
perform
right,
recurse.
A
So
beside
the
transaction
IDs
of
the
actual
database,
we
do
have
transaction
IDs
for
the
raft
messages
herself.
So
when
the
leader
loses
leadership
so
when
he
is
not
able
to
ping
the
other
nodes,
if
he
would
detect
that
he's
in
a
cluster
and
don't
get
the
Kuril
correlated
health
checks
from
the
other
nodes,
he
would
step
down
the
leadership
so
that
he
would
recognize
that
he's
not
allowed
to
write
anymore.
A
What
you
could
what
could
happen
in
theory
when
you
on
the
leader
you
get
a
transaction
through
and
you
get
only
one
commit
back
from
one
of
the
followers,
but
not
from
the
other
one,
because
maybe
is
already
petitioned
away
or
something,
and
then
you
could
both
of
the
nodes
you
that
acknowledge
the
transaction.
This
could
potentially
lead
into
problems
when
you
just
hit
it
in
that
situation,
as
as
I
described,
we're
using
PBS
to
that.
A
So
currently,
it
would
not
be
possible
to
kill
the
entire
stay,
because,
as
long
as
you
don't
kill
the
remote
storage
as
well,
we
wouldn't
lose
the
overall
data
set
from
those
two
nodes.
So
we
would
then
try
to
bring
back
the
nodes
based
on
on
those
PVS.
But
if
the
PB
has
also
gone
like
both
TVs,
which
we
didn't
had
so
far,
then
potentially
we
would
lose
the
transaction
because
we
don't
have
the
data
said,
because
this
was
like
a
failure
of
two
nodes
too
quick.
A
After
only
though
two
nodes
committed
the
transaction
and
the
third
one
didn't
committed
and
the
backup
worker
didn't
got
it
so
in
that
case
you
could
potentially
lose
data.
To
be
honest,
we
didn't
lost
the
PV
since
we're
using
Kevin
alleys,
so
we
really
where
we
were
trusting
Google
to
do
good
things
with
our
pivot,
persistent
disks
and
so
far
one
year
of
operating.
We
didn't
what
so
we
weren't
disappointed.
A
Yet
I
did
like
it
doesn't
it
doesn't
say
that
we
will
never
lose
data,
but
so
far
with
the
persistent
volumes
we
were
pretty
good
so
far,
maybe
if
you're
like,
if
you
want
to
talk
more,
but
maybe
you
can
talk
it
offline,
so
I
would
love
to
her
about
you,
like
your
concerns
and
what
problems
you
probably
experience
to
ask
that
question
in
that
forum.
Okay,
thank
you
very
much.
A
D
A
So
currently,
this
is
a
halfway
automated
process,
so
every
deploys
goes
through
it
release
pipeline,
so
the
operator
would
be
deployed
when
someone
says
that
he
wants
a
particular
version
going
from
staging
to
production.
Well,
then,
currently,
the
process
I
think
it's
partly
manual
where
we
say
we
deploy
lower
priority
databases
first
and
when
everything
is
happily,
we
go
to
the
next
ones.
I
think
this
is
just
like
this.
We
net,
we
call
it
phasing
rollout
of
like
of
database
updates,
is
so
new
that
we
want
to
experience
currently.
A
D
A
We
calculated
so
we
do
in
the
mini
reconciliation
for
for
tolerance
and
availability.
We
calculate
if
a
cluster
is
4,000
anyways,
because
we're
only
doing
certain
things
when
the
database
is
fault-tolerant,
so
we
calculate
it
anyways
and
then
we're
doing
like
the
update
on
the
cid
in
every
reconciliation
loop.
If
we
detect
that
there's
a
change,
so
this
is
something
we
we
do
in
encode.
Basically,.
E
E
E
A
So
we're
only
passing
to
the
next
one
if
the
previous
step
was
was
fine
at
that
point.
So
if,
if
a
mini,
reconciler
and
position,
two
maybe
request
the
changes
and
we
would
not
go
to
the
next
level.
We
equate
to
the
next
entire
reconciliation
loop
to
them
check
in
step
two,
that
everything
is
okay
and
then
go
to
the
next
step.
So
we
only
always
going
through
the
entire
sequence
of
freaking
silos
and
only
we're
going
to
the
next
step.
A
F
I'm
terracing
in
data
two,
and
it's
not
clear
to
me.
If
you
wanna
talk
about
persistent
volume,
you
are
talking
about
file
system
volume
or,
if
you
using
some
overlay,
you
talk
about
the
abs
from
Amazon
I
know,
if
you're
using
it
in
overlay,
what
kind
of
latency
some
problems
in
performances
it
brings.
Yeah.
A
A
I,
don't
know
how
Google
is
managing
this,
but
the
latency
is
not
that
bad
as
you
would
expect
it.
So
this
blog
post
on
on
the
on
the
Google
computing
website
why
they
say
how
much
performance
you
get
in
order,
like
related
to
the
instance
I
like
this
size,
you're
buying.
So
you
get
more.
I
ops,
if
you're,
buying
big
inside
like
pick
this
and
so
on.
There's
a
huge
blog
post
about
that
and
we're
trying
to
get
the
best
performance
for
our
of
our
customers
but
you're.
A
Right
always
when
you
try
when
you're
talking
to
remote
search,
you
have
you,
you
will
lose
performance,
that's
why
we
maybe
want
to
change
that
in
future
for
near
4,
J
itself.
It's
kind
of
easy
in
comparison
to
other
databases,
because
you
don't
have
random
reads
so
often
an
ephah
J.
You
would
have
way
more
reads
than
you
have
rights
and,
most
of
the
time
the
reads
can
surf
by
memory.
So
you
don't
have,
as
you
would
have
in
Cassandra
or
something
while
you
have
random
access
to
whatever
plays
in
your
disk.
A
You
typically
don't
have
that
in
AO.
So
for
the
read:
it's
not
that
important,
because
you're
mostly
surf
it
from
memory,
so
you
don't
have
the
performance
impact
there
and
for
the
right
yeah
there.
They
take
longer
that
you,
if
you
would
have
like
a
single
instance
running
on
an
SSD
and
the
same
in
the
same
box.
Yeah.
That's
true.