►
From YouTube: Kube-native Postgres [I] - Josh Berkus, RedHat
Description
Kube-native Postgres [I] - Josh Berkus, RedHat
Database systems remain the last frontier for Kubernetes, and at the Patroni Project we're working on conquering it. Having fully automated PostgreSQL clusters using Patroni, the project is now working on making Patroni more "Kubernetes native", so that SQL databases can be seen simply as a PostgreSQL resource.
In this talk, we will explain and demonstrate the current projects integrating Patroni PostgreSQL with Kubernetes, including:
* Patroni Operator, using the CoreOS Operator pattern
* Kube-native Patroni, which uses the Kubernetes controller instead of its own management
These works in progress will both acquaint attendees with tools they can use for their own high-availability database architectures, and explore some areas where Kubernetes could improve to support database systems better.
About Josh Berkus
Josh Berkus works on Project Atomic at Red Hat, where he is helping build the immutable infrastructure of the future. He loves Atomic Host, Kubernetes, distributed systems, Ansible, container building, PostgreSQL, Django, Python, community building, and the Oxford Comma. Josh lives in Portland with an overly large cat, where he cooks from his garden and makes pottery.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Well,
welcome
everybody,
I'm
Josh
burkas
I
work
for
Red
Hat
in
the
OSS,
which
is
always
known
as
the
community
department,
where
I
get
to
work
on
kubernetes,
which
is
truly
awesome,
and
if
you
know
me
from
earlier
times
you
look
at
my
CV.
You
know
the
way
that
I
got
into
kubernetes
was
actually
moving
Postgres
to
kubernetes.
So
this
has
been
my
sort
of
personal
project
for
a
while,
of
course,
what
you're
about
to
see
is
not
just
my
personal
project,
even
primarily
my
personal
project,
the
lion's
share.
A
The
work
is
done
by
the
engineering
team
at
zoeland.
Oh
we're,
specifically
the
database
team
Enzo
endo.
We
started
collaborating
two
years
ago
on
the
Patroni
project.
They
nobody
from
that
database
team
could
be
here
at
coop
con,
which
is
why
I'm
on
this
podium
by
myself
but
I'll,
tell
you
all
about
it
and
I'm
sure
they
will
be
at
coop
con
Europe
I'm
talking
about
things.
So
my
goal
and
really
the
goal.
A
It
is
not
something
that
you
have
to
think
about,
as
in
you
deploy
Postgres,
and
then
you
save
your
data
to
Postgres
and
in
Postgres,
as
your
data
has
saved
it's
saved
and
opposed
to
assess
your
data
is
replicated
it's
replicated
and
you
don't
really
have
to
think
about
whether
that's
true
or
not,
or
about
starting
strategies
or
about
all
kinds
of
other
stuff
that
you
have
to
with
some
of
the
more
interesting
scale
of
databases.
Now
it's
a
good
statement:
it's
not
hip
it!
A
A
What's
changed
since,
like
the
last
coop
con
is
a
lot
tighter,
kubernetes
integration
in
order
to
just
rely
more
on
kubernetes
for
all
of
the
infrastructure
things
I'll
show
you
that
the
second
thing
is
that
we
have
an
operator
now
which
is
fairly
alpha,
but
there's
an
operator
out
there
for
anybody.
It's
really
the
operator
pattern
and
again
I
will
show
you
that
and
then.
A
So
some
terminology,
so
post
res
PostgreSQL.
That's
the
database
we're
talking
about
they.
You
know
one
that's
been
around
again,
you
know,
since
the
80s
and
yet
continues
to
add
new
things.
Patroni
is
a
high
availability
demon
service
that
runs
inside
a
container.
Explain
a
little
bit
more
about
that
and
spy
low
is
a
distribution
that
wraps
Patroni
and
PostgreSQL,
and
some
other
common
utilities
in
a
container
to
be
your
container
of
Postgres.
A
Goodness
so,
you'll
see
those
words
again
in
this
presentation
and
that's
what
the
three
different
sort
of
things
are
so
petite
them
to
talk
a
lot
about
a
Patroni
which
is
the
high
availability
demon.
So
this
does
a
few
things
you
run
in
your
container
as
PID
one.
It
controls
whether
or
not
Postgres
is
running
in
that
container
it
automates
failover
and
replication
for
your
cluster
of
whatever
defined
size,
and
it
also
supplies
a
management
API
high
for
the
things
that
you
can't
do
through
the
normal
per
squares
interface.
A
Now,
question
of
you
know:
what's
the
idea?
What's
the
design
behind
this,
what
are
sort
of
specifically?
What
do
we
think
why
we're
doing
Patroni
and
not
using
some
other
systems?
And
you
know
what
are
our
goals
and
there's
three
basic
ones,
so
we're
looking
for
simplicity,
usability
and
availability
and
what
I
mean
by
simplicity?
Is
we're
really
trying
to
the
majority
people
write
us
people
who
are
post
quiz
geeks,
we
think
of
Postgres?
Is
this
huge
constellation
of
services?
You
know
that.
A
Does
all
these
different
things
need
that
for
it,
data
wrappers
and
you've
got
monitoring
tools
know
these.
Are
these
things
for
us,
you
know,
having
push
must
be
being
complex,
is
actually
a
virtue
for
everyone
else.
Postgres
is
one
thing:
it's
an
endpoint
and
they
want
that
to
be
one
endpoint
and
not
sixteen
different
things
in
so.
A
Our
goal,
partly
with
the
patron
Ian's
phyllo,
is
to
make
postcards
one
endpoint
and
not
sixteen
different
things
in
you
know,
rely
on
kubernetes
as
much
as
possible
so
that
the
Patroni,
if
you
know
install,
does
not
have
to
be
complicated
that
instead
we're
simply
relying
on
the
tools
that
ship
with
or
tools
that
we
can
enable,
with
kubernetes
in
to
supply
a
set
of
defaults.
That
are
what
we
believe
just
work
for
most
users.
A
Just
in
case
you
lose
your
entire
cluster,
so
you
still
have
data
and
then
the
other
thing
is
that,
of
course,
unlike
a
lot
of
public
cloud
providers,
we
want
you
to
be
able
to
run
something
that
places
no
limitation
on
what
Postgres
plugins
or
configuration
options
you're
going
to
run.
You
know,
because
maybe
you
want
some
obscure
plugin
for
doing
I,
don't
know
biological
data
in
Postgres
that
you
can't
get
off
of
Amazon
RDS.
A
A
So
now,
let's
talk
about
a
little
bit
about
what's
changed
recently
and
by
change
recently
I
mean
this
is
still
a
feature
branch,
the
so
our
Patroni
design,
I
started
out
actually
before
kubernetes
was
really
usable
as
anything,
but
we
had
at
CD,
and
so
we
designed
it
where
you
spin
up
an
fcd
cluster
to
provide
to
provide
a
single
source
of
truth.
I
think
everybody
here
knows
what
@cd
is
and
what
it
does.
A
So
and
that
works
pretty
well
and
that's
actually
what's
in
production
in
several
places.
However,
Wilkin
at
this
and
we're
saying
you
know
what
that's
still
more
complicated
than
it
needs
to
be.
Why
don't
we
need
this
separate
at
CD
when
I
already
have
a
single
source
of
truth,
which
is
called
the
kubernetes
api?
A
So
what
we've
done
recently
is
get
rid
of
that
separate
act
at
CD
as
an
option.
There's
actually
always
going
to
be
reasons
why
you
want
to
run
a
separate
CD
for
some
clusters,
but
again,
simple
defaults
and
as
a
default,
you
know
instead
of
having
a
separate
DCs.
Everything
goes
through
the
kubernetes
api.
So
you
know
leader.
Elections
go
through
the
kubernetes
api
to
some
other
structures
and
everything
else.
So
this
is
as
simple
as
possible
because
the
idea
is
now
I
can
deploy
a
high
availability,
puskás
cluster.
A
So
now,
because
most
people
in
the
room
have
not
seen
any
Patroni
presentations
before,
let
me
go
over
basically
how
this
works
in
terms
of
high
availability
with
an
animation
and
then
I'll
go
ahead
and
actually
demo
in
action.
So
the
basic
idea
here
is:
we
have
our
container
with
Postgres
inside
we've
got
the
Patroni
daemon
running
in
the
container
is
PID
one
and
then
what
happens
is
we
have
kubernetes?
A
We
tell
it
to
create
a
stateful
set
and
it
distributes
a
bunch
of
these
creates
a
whole
bunch
of
these,
and
then
we've
got
a
bunch
up
and
running
that
are
empty
currently
because
we're
deploying
a
new
cluster.
So
what
happens?
Is
they
all
communicate
with
kubernetes
api?
And
they
say
we
want
to
have
a
leader
election
by
updating,
actually
a
config
map
and
an
endpoint,
and
then
they
have
that
leader
election
and
one
of
the
nodes
wins
that
node
becomes
the
master
at
which
point
replication,
the
other
two
nodes
start
replicating
from
that
master.
A
A
You
know
this
and
then
via
kubernetes
services
and
endpoints.
We
can
make
sure
that
there
is
always
a
connection
to
the
master
available.
So
let
me
actually
go
a
little
bit
more
into
detail
and
that
failover,
because
it
is
a
little
more
complicated
than
just
making
a
call
to
the
API.
So
what
happens?
Is
the
master
van
vanishes
and
it's
gotten
this
timeout.
You
know
we've
got
a
time
stamp
on
it,
and
so
we
know
the
master
hasn't
checked
in
in
a
while.
A
A
A
If
replication
status
is
okay,
it
will
update
the
config
map
indicating
that
it
is
the
master
and
then
the
other
replicas
who
are
not
able
to
grab
the
end
point,
will
remaster
off
of
that
master
and
you
will
have
a
new
cluster.
So
it's
a
little
bit
it's
a
two-stage
failover
which
can
affect
you.
If
you
have
a
really
crappy
network,
because
if
your
network
support
is
really
crappy,
then
it's
gonna
be
a
little
slow,
but
so
let's
actually
do
that.
A
A
So
we've
got
here
a
sample
manifest.
This
is
just
a
slight
modification
of
the
one
given
as
an
example
in
the
kubernetes
branch
of
the
Patroni
repo.
So
you
can
take
a
look
at
this
yourself
later,
but
basically
we
just
have
we
define
a
cluster
name
in
there.
We
have
labeled
applications,
we're
telling
it
to
do
three
replicas.
You
know
for
obviously
for
high
availability
purposes.
I
recommend
always
at
least
three,
because
there's
a
lot
of
things
that
can
cause
two
to
fail.
A
The
and
then
we
have
our
container
here.
I
have
a
slightly
modified
image,
because
this
is
a
feature,
branch
and
I've
been
debugging,
but
it
will
be
merged
before
the
end
of
the
year.
The
end,
then,
we
just
have
some
definition
of
the
container
and
everything
else
and
your
usual
volumes,
and
then
we
define
a
whole
bunch
of
environment
variables
that
are
getting
passed
through
to
the
Patroni
daemon
in
order
to
configure
the
behavior
of
Postgres.
So
the
usual
things
password
ports.
A
If
we
were
going
to
put
in
any
kind
of
performance
modifications
from
the
post,
whose
defaults
like
how
much
memory
uses
etc,
we
would
also
pass
them
here.
The
and
I've
got
this
commented
out
because
I've
got
it
running
in
me
and
cubed
and
there's
no
actual
volumes
the
then
we
have
to
define
a
couple
of
other
things.
A
However,
you
can
create
a
service
without
a
selector,
in
which
case
there
is
no
endpoint
and
you
can
define
your
own
invoice
most
of
the
time.
People
use
this
to
connect
to
services
outside
of
kubernetes,
but
in
our
case
we're
using
this
to
redefine
the
connection
specifically
to
the
master
node,
so
that
we
update
only
the
endpoint
the
and
then
we
create
this
sort
of
headless
service
with
no
selector.
Now
I
also
want
a
separate
service.
A
A
A
A
A
And
we'll
get
the
rep
of
this
talking
about
being
replicas
where's
your
main
logout,
but
we
also
funnel,
of
course,
the
post
course
log
output
to
this
I.
In
the
process
of
this
we
create
a
couple
of
config
maps
and
those
should
also
not
be
there.
Okay,
you
know,
one
of
which
actually
has
the
configuration
of
the
cluster,
which
would
have
any
non
default
items.
I
haven't
I,
don't
have
a
lot
of
non
default
settings
in
here.
A
One
of
the
non
default
settings
I
have
in
there
is
there's
a
tool
called
PG
rewind
that
can
allow
us
to
bring
individual
post
quizzes
into
sync,
if
they've
gotten
a
little
ahead
during
a
failover
circumstance
that
post
goes
our
application
details
the
and
but
the
more
important
thing
is
this
leader.
Config
map
actually
gives
you
the
is
one
of
the
one
of
these
several
ways
we
offer
to
get
information
about
who's,
the
current
leader,
so
there's
a
lot
more
interesting
extra
fishery,
the
failover.
A
And,
as
you
can
see
when,
when
network
latency
is
negligible
as
it
is
when
you're
running
this
on
mini
cube,
it
picks
up
the
fact
that
the
master
is
gone
really
fast,
any
when
it's
slower
than
that.
It's
slower
than
that
specifically
because
of
network
round-trip
time
to
actually
sort
of
detect
that
things
are
gone.
And
obviously,
when
you
do
things
like,
when
you
lose
physical
nodes
from
your
network,
it
can
take
a
little
bit
longer
for
a
network
to
timeout
and
then
kubernetes
will
spin
up
a
new
one
and
we'll
have
a
replica
again.
A
A
Now
one
of
the
things
that
you
can
actually
say
here,
he
is
Wow,
that's
nifty
and
that's
really
super
useful,
but
among
other
things,
who
want
to
actually
do,
among
other
things,
do
all
this
passing
configuration
the
environment
variables
in
kubernetes
manifests
is
really
irritating
and
really
doesn't
help
me
when
I
need
to
actually
manage
dozens
or
hundreds
of
postcodes
clusters.
So
it
would
be
nice
if
there
was
another
way
well,
there
is,
and
that's
called
operators,
so
are
people
how
many
people
here
are
familiar
with
the
operator
pattern.
A
The
for
that,
if
you
have
divided
up
teams,
you
can
actually
pass
a
much
simpler
structure
set
of
information
to
define
the
cluster
so
that
people
don't
have
to
understand
kubernetes,
yeah
mole
structure
and
how
to
define
kubernetes
objects
and
stuff
and
have
the
opportunity
to
screw
up
your
kubernetes
cluster
in
the
process.
And
then
the
other
thing
is
an
operator
is
actually
an
active
thing.
A
Sometimes
you
want
to
run
other
forms
of
maintenance
and
that
sort
of
thing,
so
simply
a
stateful
deployment
deployment
you're
not
done,
and
the
big
thing
is
hey
I'm
effectively
creating
this
a
manifest
anyway.
So
if
I'm
doing
that,
why
not
have
that
run
everything?
Why
don't
just
create
a
manifest
and
have
everything
else
be
automatic,
so
I'm
gonna
show
you
one
Operator.
So,
like
I
said
spy
lo
is
the
sort
of
full
packaging
of
Patroni
plus
the
other
stuff.
A
So
there's
an
operator
for
spy
lo,
which
is
what
zillion
o
currently
uses
in
their
staging
cluster.
It
hasn't
gone
to
the
production
cluster
yet
and
what
you
do
for
that
is
you
install
the
operator?
You
create
a
manifest
for
each
cluster.
If
you
want
to
modify
clusters,
you
do
that
by
modifying
the
manifest.
So
let's
actually
do
that.
Although
it'll
take
me
a
minute
because
apparently
I
have
a
lot
of
remnants
of
the
previous
uninstall,
so.
A
A
A
I
can,
by
simply
defining
a
manifest
in
passing
the
manifest
to
the
operator,
tell
it
to
create
a
cluster.
So
this
is
what
a
manifest
looks
like
for
an
operator.
You
know
it's
still
yellow,
but
it
is
not.
You
don't
have
to
understand
the
kubernetes
infrastructure
to
be
able
to
write
these,
which
is
really
important
if
you're
gonna
have
say
development
teams,
breeding
databases
for
their
own
applications.
Instead,
you
define
some
users,
you
define
some
post
Chris
settings.
I.
A
A
A
There
we
go
yeah
when
one
of
the
outstanding
bugs,
if
you
go
to
the
post
cruise
operator
thing,
you
will
see
that
there's
an
outstanding
issue
for
it,
failing
to
clean
up
when
we
delete
clusters,
so
there
is
actually
an
issue
there,
it's
being
worked
on
the,
but
we
go
ahead
and
install
that,
and
why
are
we
not
seeing
this
pilot
roll?
Oh
there,
we
are
so
again
like
rate
like
the
the
bear
Patroni
one
of
them
get
selected
whoops.
No,
it's
not
what
it
wanted.
A
Now
the
spinal
operator
is
not
the
only
operator
out
there
there's
another
whole
nother
project
for
post-closing
kubernetes,
that
comes
from
crunchy
data
hi.
There
we've
one
of
the
quincy
data
staff
here
that
comes
from
crunchy
data
they
have
their
own
operator,
that's
been
out
for
a
little
bit
longer,
so
it's
a
little
bit
more
mature.
Their
architecture
is
very
different
from
their
architectures
very
different
from
the
petronia
architecture.
A
They
were
very
much
coming
from
the
enterprise
side
of
things
and
so
they're,
really
looking
at
having
a
very
full
service
Postgres
offering
with
a
lot
of
tooling
the,
which
is
what
their
thing
the
crunchy
container
suite
is
what
it's
called.
We
will
get
up
the
and
also
the
design
of
operate
their
operators
a
little
bit
different
design
of
our
operator.
Spile
operators
extremely
manifest
driven,
and
there
really
aren't,
like
other
ways
to
interact
with
it.
A
In
terms
of
how
you
want
to
do
things,
there
are
advantages
both
ways
they
don't
pop
in
sorry,
I
had
your
hole.
I
had
the
whole
architecture,
diagram
of
the
container
suite
again
industry,
just
a
lot
of
things
in
it.
So,
like
I
said
it's
also
a
very
different
set
up
for
high
availability,
Postgres
and
I
am
a
little
bit
too
far
forward,
so
so
very
different
setup
for
high
availability
post
quiz.
A
Then
the
Patroni
setup,
quick
comparison
of
actually
going
through
and
trying
out
both
operators
and
going
through
this
there's,
some
of
it
I
mean
the
biggest
thing
is
that
the
spy
low
operator
is
still
kind
of
alpha
because
it
was
created
recently
yeah
things
move
fast,
we'll
say
it
forever
sort
of
different
ways.
Looking
in
there,
one
of
the
main
reasons
why
it's
still
kind
of
alpha
is:
we
haven't
really
written
any
documentation
for
it,
which
will
change
the
eye.
There's
not
yeah,
there's
not
a
specific
CLI
for
the
Spile
operator.
They're.
A
Very
much
is
one
for
the
crunch
operator.
Very
sophisticated
CLI
depends
on
with
likes.
The
eyes
are
not
crunchy
is
God,
is
openshift
compatible.
Now
Spile
o
is
not
really
both
support
rolling
up
rates
of
post
press,
which
is
another
important
thing
that
the
operator
does
for
you,
because
you
can
do
that
by
specifying
a
lot
of
things
in
your
upgrade
strategy
for
new
containers
in
kubernetes
deployment,
but
you'd
rather
not
have
to
write
that
since,
since
the
operator
already
knows
what
to
do
so,
some
future
work
plans
because
is
active
number.
A
One
is
obviously
merge.
The
kubernetes
native
branch
into
the
main
Patroni
second
thing
is
to
actually
right
now
the
operator
integrate
the
operator
and
the
kubernetes
native
branch,
which
currently
currently
the
operator
uses
the
older
version
of
Patroni
I'm
working
on
OpenShift
compatibility
because
they
work
for
Red,
Hat
and
I
have
to
work
on
openshift
compatibility
and,
besides
there's
lots
of
people
who
use
openshift
and
use
postcards
on
open
chests.
A
The
there
actually
gives
a
command
line
tool
for
Patroni.
That's
used
for
a
few
things,
and
right
now
it's
completely
separate
from
the
operator,
and
maybe
it
shouldn't
be
we're
discussing
that.
Also
I
was
interested
in
Brendan's
meta
particle
locks,
er
D,
so
we're
gonna,
look
at
that
and
potentially
having
something
that's
defined
block
operator
rather
than
overloading
in
to
perform
that
function.
A
Now
there
are
a
few
things
that
we
run
into
in
kubernetes
that
are
still
limitations
that
make
life
hard
for
us.
There
really
aren't
very
many
I
mean
in
mind.
You
I've
been
working
with
the
team
that
does
stateful
sat
for
quite
a
while,
and
so
there
really
aren't
very
many,
because
a
lot
of
the
things
that
I
had
in
my
issue
list
got
done,
but
multi
datacenter,
multi
availability
zone
cluster
is
hard
to
do
with.
Stateful
sets
right
now.
I.
A
Upgrading
stateful
sets
pretty
much
requires
you
do
to
an
operator
right
now,
because
the
stateful
set
has
no
intelligence
about
what
order
it
should
upgrade
things
in,
and
it
could
so
again
up
for
discussion
and
then
it
would
be
really
nice
to
be
able
to
extending
of
coop
control
so
that
we
could
stop
having
all
of
these
separate
Co
eyes.
One
per
operator
that's
completely
separate
from
Google
control.
A
So,
but
if
you're
interested
in
this
is
something
going
to
use,
we
really
would
be
happy
to
have
more
contributors
open
project.
The
Patroni
I
admit
that
we
do
most
of
our
communication
via
github
issues
is
not
a
mailing
list
or
anything.
So
if
you
have
a
question
or
something
like
that
file
an
issue
or
find
one
of
us
on
the
Postgres
IRC
and
let's
make
posters
boring
I'll,
take
questions
in
just
a
second
research
links
and
now
questions
go
ahead.
Do
to
do.
A
A
Yeah,
okay,
so
question
was
the
replication
is
with
replication
lag
yeah
the
default
is
a
synchronous
replication,
which
means.
Indeed
there
is
replication
lag,
and
if
you
have
an
unexpected
master
failure,
then
you
will
lose
some
transactions
if
they're
in
flight.
However,
petronia
is
completely
compatible
with
you.
A
Turning
synchronous,
replication
on
there's
examples
in
the
documentation
of
what
that
would
look
like,
and
so,
if
you
want
to
run
it
with
synchronous
replication,
then
you
can
turn
that
on
and
particularly
if
you're
going
to
do
that,
I'd
recommend
using
post
was
10
and
running
it
with
synchronous,
quorum
replication
so
that
you
don't
get
replicas
failure
blocking
your
your
write
transactions.
So
there's
another
question
up
here:.
A
Okay,
so
Reyes
question
was:
will
you
miss
most?
Was
user
management
and
Korean
users
via
config
maps,
the
operator,
if
you
modified
the
manifest
as
a
section
of
the
manifests
devoted
to
users?
And
if
you
operate
the
man,
if
you
modify
the
manifest
it
should
create
the
users,
if
you're
using
the
operator,
if
you're
using
Patroni
without
the
operator,
then
you
can
either
create
the
users
when
you
create
the
cluster,
because
that
goes
into
Patroni,
config
or
if
you
want
to
create
a
user
later
on.