►
From YouTube: Elephants on Automatic: HA Clustered PostgreSQL with Helm [I] - Josh Berkus & Oleksii Kliukin
Description
Elephants on Automatic: HA Clustered PostgreSQL with Helm [I] - Josh Berkus, Red Hat & Oleksii Kliukin, Zalando
Kuberntes and Helm are not only the best way to automate high-availability PostgreSQL clusters, they're also the easiest! Join us to find out how you can deploy several different PostgreSQL cluster types using Helm in 30 minutes or less.
Helm is a package manager of choice for the Kubernetes community. PostgreSQL is the most advanced open-source database and is quickly becoming the relational database of choice for numerous developers running their applications in the cloud. In this talk, we are going to show two different types of fully automated PostgreSQL clusters with Helm: Spilo and CitusDB. Spilo provides automated failover and support services for single-master database clusters, and CitusDB provides sharded, big-data PostgreSQL.
We will demonstrate both of these types of clusters (and possibly others), explain how the Helm charts which set them up work, and how you can modify them to support your production environment.
About Josh Berkus
Josh Berkus containerized and automates all the things, using Project Atomic, Kubernetes, Ansible and Python. He's also known for his work on the PostgreSQL database.
About Oleksii Kliukin
Oleksii Kliukin is an engineer for Zalando, where he helps improve and maintain Spilo and Patroni, providing the fashion giant with full-automated, self-deploying database clusters.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
So
since
I
first
tried
out
kubernetes
a
couple
of
years
ago
and
actually
docker
about
a
year
before
that,
I've
been
on
a
quest
to
actually
get
to
managing
stateful
applications
under
on
a
containerized
stack.
This
has
been
a
long
quest,
doesn't
really
start
out
supporting
stateful
applications
in
containers.
We
started
out
with
the
stateless
ones
because
those
were
easier,
and
this
is
our
sort
of
next
stop
in
in
stateful
stuff.
Now,
obviously,
everybody
right
now
is
running
on
kubernetes
and
docker
and
rocket
and
other
things
they're
running
all
your
stateless
applications
on
there.
A
You
know
your
web
servers
and
routers
and
other
things,
but
a
lot
of
people
like
actually
the
one
question
asked
earlier
is
how
many
people
are
running
your
application
code,
andrew
burnett,
'yes,
but
running
your
database
on
a
more
traditional
system,
yeah.
So
there's
a
lot
of
that,
and
I
understand
why
there's
a
lot
of
that,
we
need
to
actually
work
on
that
now.
There
has
been
a
lot
of
work
in
1.5.
A
Things
can
communicate
with
each
other,
but
stateful
set
doesn't
do
everything
that
we
need
in
order
to
run
replicated
or
cluster
databases
under
kubernetes
now
some
things
it
simply
can't
do
because
they
require
too
much
application
knowledge
and
that
those
need
to
be
implemented
in
our
application
frameworks
specific
to
the
application.
So
these
are
things
like
having
an
individual
database.
Node
join
a
cluster
I.
A
You
know
automating
failover
handling
special
nodes
for
things
like
backup
and
administration,
etc.
Rebalancing
data
for
sharted
stuff,
so
there's
still
more.
There's
implementation
that
needs
to
happen
outside
of
guberniya
stateful
sets.
Now,
one
of
the
things
we
look
at
sort
of
immediately
for
this
is
hey.
Let's
have
custom
scheduling
right,
because
guberniya
supports
custom
schedule
is
a
pretty
good
idea.
There
are
some
problems
for
that
as
a
reason
why
it's
a
route
you
know
I
didn't
take,
and
it's
not
a
route
that
I've
seen
anyone
else
take
number
one.
A
The
problem
of
the
custom
scheduling
is
you
now
have
this
thing
that
is
intimately
wedded
to
how
your
database
platform
works.
That
is
nevertheless
deployed
completely
separately
from
it,
because
this
is
being
deployed
in
the
kubernetes
master
nodes,
the
other
things
we
need
deployed
through
a
container.
The
second
thing
is,
frankly,
the
overlap
of
people
who
are
database.
Geeks
and
kubernetes
experts
is
pretty
low.
A
You
know
for
that
and
the
biggest
problem.
The
reason
why
I
didn't
feel
like
it
was
worth
pursuing
is
the
whole
message
lag
sort
of
telephone
problem
which
is
anytime.
You
have
a
central
controller
that
controls
things
like
failover.
You
have
the
possibility
of
not
knowing
what
the
state
of
the
node
you
just
told.
The
failover
is
so
it's
much
better
to
have
the
node
control
itself,
and
for
that
reason,
what
I'm
pretending
is
something
that
I
call
the
bot
pattern.
I
think
some
other
people
also
cause
this.
A
That
Alexis
will
be
talking
about,
are
based
on
the
sort
of
bot
pattern
where
it
will
do
things
like
initialization
and
cluster
joining
and
failing
and
providing
information
to
the
system
about
the
state
of
health
of
the
individual
nodes.
There's
been
a
few
examples
of
this
is
the
original
governor
project,
which
is
being
reincarnated.
There's
pitoni,
which
we
both
work
on.
A
There's
a
lot
of
simple
sort
of
entry
point
scripts
I
was
just
talking
to
the
chef
habitat
folks
and
habitat
is
very
much
around
the
same
sort
of
model,
and
we
will
be
showing
you
a
couple
of
these
now.
The
other
thing
that
we're
actually
adding
in
for
this
cube
con
is
to
talk
a
little
bit
about
helm.
A
So
there
are
now
helm
charts
for
a
couple
of
different
versions
of
deploying
post
squares.
Helm
adds
a
few
things
that
actually
make
it
easier
parameterization.
The
idea
of
having
an
actual
deployment
of
postgrads
with
all
of
its
parts
have
a
name,
so
you
can
do
several
of
them,
which
is
something
where
otherwise
doing
through
editing
the
ml
code
having
a
sort
of
registry
of
those
things
so
that
you
can
I
upload
and
download
that,
and
once
we
actually
add
them
into
the
charts
which
they're
not
support
for
sidecar
containers
like
backup.
B
This
one
thank
you,
so
Patroni
is
a
way
to
run
your
highly
available
phosphorus
in
the
container
environment
and
we
actually
started
developing
in
two
years
ago,
when
we
moved
B
as
Orlando
the
technology
company
that
happens
to
sell
shoes
and
clothing.
When
we
move
to
the
cloud
and
we
what
the
transpose
R
is
in
the
cloud
as
well,
so
there
were
no
easy
way
to
actually
do
the
automatic
failover,
which
is
a
requirement
if
you
want
to
run
hundreds
of
possible
nodes
in
the
cloud
and
therefore
we
started
to
develop
patrolling.
B
B
It
takes
some
time
we'll
clean
up
all
the
resources
that
we
have.
We
are
running
actually
on
the
database
cluster
that
is
running
inside
cylinder,
so
production
one.
So
let's
get
the
latest
running
help
chart
actually
I'm
running
it
currently
locally.
We
have
a
chart
that
we
have
the
official
one
needs
to
be
updated
and
today,
I've
had
a
work
that
will
be
changes
for
the
state
offsets
reversed,
but
a
few
changes
that
the
suit
has
to
be
merged
with
the
public
place.
So
once
it
is
updated,
you
will
be
able
to
get
from
help.
B
Now
you
can
get
it
from
my
repository
and
I'm
running
the
help
locally
and
helm
just
solves
to
do
because
of
your
vocal
as
well.
So
now,
let's
make
sure
the
chart
is
here,
you
can
see
header
row
we'll
have
fun
here
of
Roni.
Let's
take
a
look
at
the
configuration.
This
is
basically
what
you
need
to
configure,
preferring
this
is
the
doctor
image
totally
there,
the
number
of
replicas
the
resources,
the
service
account.
We
that
we
use
if
your
cluster
is
configured
so
that
no
one
not
everyone,
can
actually
write
your
name
spaces.
B
The
credentials
for
the
superuserraven
user
and
the
replication-
and
this
is
very
important-
how
do
you
find
the
HDD
because
they
stay
the
storage,
where
you
give
way
to
keep
the
configuration
some
a
degree,
a
specific
configuration
like
you
can
use
the
s3
bucket
and
all
that
will
put
back
up
and
roll
files
there,
and
basically
we
accept
this
configuration.
So,
let's
deploy
the
new
cluster
with
helm.
This
is
super
easy
help,
install
cube
con.
B
So
now
the
cluster
is
being
deployed
and
you
can
see
the
way
to
connect
to
this
cluster
with
different
users
and
yeah.
Let's
take
a
look
at
the
state.
Whole
set
was
generated,
we
use
stateful
sets.
We
have
three
replicas
and
basically,
we
supply
configuration
via
environment
variables
and
we
use
the
persistent
programmers
in
order
to
store
pinchy
data,
the
data
directory
for
Possible's.
You
know
we
usually
standard
class
and
we
define
the
storage
as
five
gigabytes
level.
So,
let's
check
the
status
of
the
pots.
B
B
So
this
is
what
we
have
on
the
kubernetes.
This
is
what
help
creates
it
creates.
The
stateful
set
is
a
stateful
set
contains
the
pots.
Each
pot
runs
one
docker
container
inside
this
container.
We
have
phosphorus.
Obviously
that
stores
your
data
and
we
have
petroleum
that
acts
that
communicates
between
phosphorus
and
HCD.
That
stores
the
state
of
the
cluster,
so
xev
is
used
in
order
to
make
sure
that
we
run
only
one
little
time.
B
B
We
could
use
labels
in
order
to
get
the
connection
information
to
the
client,
but
instead
we
use
endpoint
because
it
is
only
there
can
be
only
one
address
in
the
endpoint
and
we
want
to
avoid
the
case
where
the
labels
were
signed
incorrectly,
and
you
have
two
addresses
for
the
master,
so
endpoint
make
sure
we
have
only
one
address.
So,
let's
return
to
the
cluster
appointment
now
we
can
see
that
all
the
nodes
are
running,
so
we
can
actually
take
a
look
at
the
locks,
so
it
will
take
some
time
here.
B
We
get
the
rocks
for
the
master
replica,
so
one
of
the
notes
says
I
am
literally
the
work.
This
is
the
one.
This
is
the
master,
the
others
are
actually
say:
I'm
a
secondary
I
am
folding
the
leader.
That
means
that
they
serve
as
a
secondary,
and
here
is
the
contents
of
the
HCD.
These
are
the
keys.
We
have
the
leader
key,
we
have
the
keys
for
each
member,
and
you
know
we
actually
also
store
configuration
of
post-race
in
the
cluster.
So
this
is
basically
if
you
are
familiar
with
the
possible
configuration.
B
And
let's
open
also
connection
to
the
database,
so
now
what
I'm
going
to
show
you?
Because
Patroni
manages
automatic
failover
for
you,
they
are
going
to
kill
one
of
the
nodes
and
make
sure
that
just
the
application
can
still
keep
connection
to
so
make
sure
that
one
of
the
other
poles
we
were
seeing
the
master
all
and
the
connect
application
connections
will
be
kept.
So,
let's
take
a
look.
B
What
we
have
this
is
the
tool
that
is
called
patron
ezekiel
and
it
is
showing
us
that
one
of
the
one
of
the
node
is
later
it's
running,
so
the
others
also
running
very
replicas
and
we
get
the
same
information
from
cube
CTO.
We
also
use
via
science
people.
They
we
assign
the
labels.
So
there
is
a
master
and
two
replicas,
and
here
we
get
the
connections
we
connect
to
the
master
we
can
see
because
peaches
in
recovery
says
false.
B
B
That
means
that
how
the
crossroad
is
working,
so
what
we
are
going
to
do
is
delete
the
master
thought
stay
home
yep,
so
we
say:
keep
contact
Rani
delete
and
here
is
what
happening
the
master
not
stirred,
and
now
the
leader
immediately
moved
to
another
pot,
so
cube
comp
attorney
2
is
currently
the
leader
and
we
can
see
that
all
the
notes.
All
the
containers
are
running.
B
And
we
can
see
that
the
application
connections
are
running
and
the
transaction
IDs
are
incrementing
as
well
being
that
the
application
managed
to
reconnect
to
the
new
leader.
So
now
the
lot
of
things,
let's
use
patron
Ezequiel-
to
make
a
control
failover
because
per
20
supports
also
maintenance
tasks.
If
you
want
to
be
controlled,
failover
or
scheduled
restarts,
you
can
also
do
this.
So
I
run
patron
evening.
Cto
failover,
it
prompts
me
to
choose
the
master
now
I
can
say
enter
it,
clicks
the
default
and
to
choose
the
candidate.
B
B
Yep
I
think
we
switch
the
leader
to
Patroni
1
and
we
can
see
that,
yes,
the
master
switched,
the
replicas
are
being
initialized
and
what
about
connections?
So
connections
are
still
running.
That
means
that
favor
one
went
successfully
and
we
have
some
features
inside
the
training.
So
it's
not
just
favorable.
It
supports
different
replication
modes.
You
can
do
synchronous
replication.
If
you
want
to
make
sure
that
your
you
don't
lose
your
data,
it
supports
cascading,
dynamic,
configure,
cascading
replication.
B
It
supports
dynamically
configuring,
your
Postgres
cluster,
basically
setting
the
Postgres
parameters
on
the
fly,
and
if
these
parameter
requires
advise
you
to
restart
your
Possible's
instance,
then
Petronio
set
a
special
flag
and
then
you
can
say
please
restart
only
if
we
start
is
required
at
3
a.m.
to
the
night
when
no
active
equations,
and
but
we
will
do
this
for
you
and
supports
cluster
maintenance.
B
If
you
need
to
do
something
like
make
something
like
major
upgrades,
you
can
turn
off
automatic
failover
completely
for
a
certain
amount
of
time
to
be
upgrade
and
then
get
back
yeah
and
supports
managing
itself
using
patron
Ezekiel
and
it
supports
race
KPIs.
So
some
parameters
will
be
covered
and
the
demo
is
already
done
so
we
managed
to
get
the
failover
two
times
and
still
have
the
master
so
with
that
I
will
get
to
decided.
Okay.
A
So
one
of
the
other
things,
one
of
the
reasons
why
I
got
really
interested
in
running
PostgreSQL
on
kubernetes
is
that
there's
a
lot
of
new
various
versions.
Experimental
often
experimental
versions
of
clustered
postcards
being
released
these
days
and
kubernetes
provides
an
excellent
test
environment
for
these
sorts
of
clusters,
because
otherwise,
in
order
to
run
a
test,
you
end
up
spending
an
hour
hour
and
a
half
on
actually
doing
a
conventional
deployment,
as
opposed
to
just
being
able
deploy,
bunsen
containers,
including
testing
on
the
same
machine.
A
So
the
way
that
I
actually
have
the
stateful
set
for
this
and
the
home
cart
organized.
Is
you
have
one
query
node,
which
is
no
zero?
You
have,
however,
you
know,
however
many
shards
and
then
your
service
is
going
to
direct
connections
to
the
query
node.
Only
because
that's
what
you
should
run
queries
so
now.
A
A
A
A
So
we
actually
look
at
this.
We've
got
a
much
simpler
sort
of
bot
arrangement
where
it
says:
hey.
We've
got
a
bunch
of
set
up
here
for
adding
labels
and
that
sort
of
thing,
and
then
we
have
a
much
simpler
chart
arrangement
where,
if
we've
spun
up
it's
good
to
the
actual
logic
err,
you
know.
First,
we
see
whether
or
not
the
data
is
initialized
and
we
say
hey.
A
If
the
data
is
not
initialized,
then
I'm
going
to
go
ahead
and
initialize
the
post
with
data
directory
set
up
some
initial
users
based
on
the
credentials
that
have
been
passed
to
us
as
part
of
the
config
map,
and
then
we
go
through
there
and
we
say
hey.
If
this
is
the
query
node,
which
we
determine
by
having
it
be
the
first
node,
then
we're
going
to
go
ahead
and
register
all
the
nodes
that
we
expect
to
have.
A
But
we
also
have
logic
that
says
hey
if
this
is
a
shard,
node
I'm
going
to
check
and
make
sure
I'm
registered
with
the
master
and
if
I'm
not
I'm,
going
to
register
myself
and
that's
in
case
you
want
to
add
more
shards
later
right.
So
it
needs
to
actually
work
by
autopilot
there
and
then
update
the
labels.
A
Sorry
not
me,
that's
up,
so
that's
a
it's
a
much
simpler
example
of
the
sort
of
bot
pattern.
If
you
actually
want
to
look
at
doing
that
for
your
own
deployments
and
then
what
he'll
mainly
gives
us
is,
it
gives
a
little
parameterization
where
you
know
we
can
actually
supply
a
name
for
the
application
in
the
sequester
cluster
size,
storage,
allocation
for
the
individual
shards
and
then,
of
course,
credentials.
A
A
A
And
there's
several
bits
to
this
chart,
but
here's
the
main
part,
which
is
the
stateful,
set
sorry
everything
still
named
pet
set
because
I
started
developing
this
one.
It
was
actually
still
named
cadets
at
the
and
you
know,
and
then
we're
actually
pulling
down
all
of
that
information
from
from
the
chart.
But
you
can
see
we're
just
going
through
and
using
that
to
set
a
bunch
of
parameters
which
are
going
to
get
passed
through
to
the
to
the
startup,
the
bootstrap,
the
bot
script,
as
it
were
so.
A
A
A
No,
that
is
not
what
I
wanted.
Well
sorry,
what
I'm,
showing
you
that
this
is
actually
a
query.
That
I've
got
is
an
example
where
it
looks
at
somebody's
logging
information
from
the
edits
whether
or
not
it's
an
IP
address,
and
a
couple
of
other
things
to
make.
It
see
whether
it's
probable
that
it's
a
bot
versus
a
real
person
and
then
counts
up
the
edits
here,
and
actually
things
have
gotten
better
using
that
heuristic,
because
the
first
time
I
ran
this
last
year
it
was
more
half
and
half
as
opposed
to
1/3
and
2/3.
A
So,
but
the
main
you
know
point
is
this:
is
that
actually
now
I
have
to
get
this
set
up
when
we
do
a
new
release
of
siteís?
It's
easier
for
me
to
test
it
by
deploying
it
on
kubernetes
cluster
than
anything
else,
and
by
extending
some
of
the
charts
and
stuff
that
I
have.
You
could
actually
very
quickly
go
to
having
a
production
deployment,
so
we
have
a
number
of
resources
and
other
things
up
on
a
website.
A
There's
the
Patroni
main
main
chart
the
my
various
database
efforts
in
atomic
DB
lb,
keep
adding
more
postcode
cluster
examples
to
this
BDR
and
multi
master
synchronous
and
some
of
the
other
new
clustering
types
that
we
have
proposed
for
us.
You
know,
plus
whoops
I
thought
I
fixed
that
okay.
That
is
the
wrong
link
for
the
presentation,
the
go-to
J
brokers
that
could
have
died
out
and
just
link
to
that,
and
as
well
as
some
of
the
other
things
we
can
look
at
for
resources.
A
C
D
A
Okay,
so
question
was
how
you
make
backups
with
situs
Phoebe
the
there
might
be
something
for
the
enterprise
version,
because
they've
they're
a
start-up,
so
they
have
a
commercial
version
that
has
a
bunch
of
extra
services.
There
might
be
a
backup
tool
for
the
enterprise
version,
otherwise
you
pretty
much
have
to
take
continuous
backups
and
try
to
synchronize
them
yourself.
A
The
that
being
said,
situs
doesn't
actually
care
if
your
shards
are
a
few
seconds
out
of
sync
now
that
might
cause
some
inconsistency
in
your
data,
but
situs
will
still
work
so
the
way
I
would
do
it
if
I
wasn't
using
the
the
Enterprise
version
would
be
to
take
continuous
backups
and
then
try
to
restore
them
to
the
same,
commit
point
more
questions.
Go
back,
get
the
two.
A
Question
was
how
big
of
a
data
set
for
the
noticeable
speed-up
in
charting
the
difference
comes
when
you
can
fit
the
data
in
memory
on
multiple
nodes,
but
you
can't
fit
it
in
memory
on
one
node,
that's
pretty
much.
It
is
or
alternately
I
at
a
much
larger
size
when
you
can
fit
the
working
set
in
memory
on
multiple
nodes,
but
not
in
one
node.
You
know
if
you
can
get
enough
memory
to
cache
your
whole
database
on
one.
You
know
machine
or
Amazon
instance
or
whatever
do
that.
It's
always
going
to
be
easier.
B
Yeah
so
basically
we
use
the
major
minor
version,
but
now
we
are
switching
to
the
major
minor
and
the
patch
said,
basically
semantic
versioning,
because
we
at
some
point
released
version,
I,
think,
1.1
and
found
out
an
issue
immediately
and
that
we
decided
not
to
release
1.2
I
decided
to
raise
1.1.1
so
yeah.
But
we
are
doing
these
pretty
consistently
so
expect
the
next
version
to
be
like
1.3,
the
major
one
and
minor
one.
Probably
like
we
increment
my
number.
B
B
Yeah,
so
the
question
is:
is
it
possible
to
communicate
with
a
city
cluster
securely
yeah?
We
sort
Patroni
supports
using
passwords
and
usernames
in
order
to
communicate
there
and
yeah
you
can
use
tables
as
well.
We
we
actually
don't
use
it
inside
the
lambda,
because
because
we
don't
have
these,
we
don't
communicate
the
ICD
via
Internet
to
the
HCD.
But
if
you
have
any
issues
you
can
just
create
the
issue
only
partially
get
happen.
A
little
take
a
look.
D
B
Yeah,
basically,
if
petronia
has
any
way
to
actually
learn
in
that
here,
so
the
question
is:
if
you
dig
the
pot
with
cubes,
if
your
brain
will
be
graceful,
fail
will
happen
and
I
never
I
haven't
actually
tested
this
use
case
in
case
you
just
kill
the
pot,
the
petroleum.
It
would
take
some
time
for
training
to
learn
that
the
muscle
is
not
there.
Basically,
the
key
in
EDD
has
to
expire,
and
then
the
other
nodes
will
see.
B
There
is
no
key
and
initiate
elections,
but
if
you
do
using,
if
you
suspect,
Ronnie
CTL,
then
you
explicitly
inform
the
master
that
it
has
to
step
down
and
then,
if
you
remove
the
key
and
at
the
same
time,
inform
all
other
nodes
that
the
key
is
not
there
and
they
will
stop
the
election
immediately.
That's
the
difference
and
in
case
of
cube
city
of
brain
I,
actually
don't
know.
So
it
might
be
that
actually
you
know
about
this.
It
receives
a
signal.
The
master
will
be
able
to
receive
a
signal
from
from
the
pot.