►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Benchmark Your Cloud Native Database - Josh Berkus, Red Hat
You can run your stateful apps on Kubernetes. You can even run your databases on Kubernetes. But what are you giving up in performance? Is it worth it, or should you stick to the hosting you know? For the past several months, we've been benchmarking various forms of Kubernetes storage, including host storage, network storage, cloud storage and cloud-native storage systems like Rook. Let us share with you the results of running PostgreSQL, CockroachDB and filesystem benchmarks so that you can make the best possible tradeoffs. We'll even show you how to do your own, to test your own platform. You will leave this talk with a much better idea of the quantitative tradeoffs between performance, reliability, data retention, and manageability.
https://sched.co/Nrol
A
Welcome
everyone,
we're
gonna,
go
ahead
and
get
started,
because
we
don't
have
a
lot
of
time.
So
if
you
need
translation
things,
please
put
them
in
now.
My
name
is
Josh
burkas
I
work
for
Red
Hat
these
days.
What
I
work
on
all
the
time
is:
kubernetes
I'm
in
said:
release
the
dribble
experience
spend
a
bunch
of
time
working
around
with
storage
stuff,
but
how
I
got
into
kubernetes
in
the
first
place
was
I
was
looking
for
a
way
to
host
and
automate
databases
for
fully
automated
h.a
application
stacks
and
one
of
the
things
I.
A
Let's
come
up.
There
is
one
of
things
I
got
originally
was:
can
you
run
databases
in
kubernetes
cloud
native
environment
and
there
are
people
who
said
no,
it's
a
fairly
famous
tweet
from
a
conversation
I
had
with
Kelsey
Hightower,
where
he's
like.
No
don't
run
databases
on
kubernetes
for
a
variety
of
reasons,
and
so
we
discussed
some
of
the
set
of
previous
conference
and
talked
about
the
reasons
why
he
would
say
something
like
that,
and
there
are
a
number
of
reasons
number
one.
A
He
is
running
a
database
on
kubernetes,
with
the
exception
of
implementing
a
Chae
aside
from
that
running
in
kubernetes
and
database,
a
database
in
kubernetes
there's
no
easier
than
running
a
database
on
bare
metal
and
therefore
you
need
the
same
skills.
Just
putting
it
on
kubernetes
does
not
make
B
be
a
headaches
go
away.
Second
thing
is
setting
up,
storage
for
kubernetes
can
be
complicated
and
hard
to
figure
out,
and
the
third
reason
potentially
is
performance.
A
This
is
one
of
the
things
people
are
worried
about
right
about
who
my
database
and
kubernetes.
What
is
performance
going
to
be
like
now,
I'm
involved
with
another
project?
That's
dealing
with
the
storage
set
up
complexity
issue.
That's
the
rook
project
in
order
to
be
able
to
set
up
cloud
native
storage
for
kubernetes
and
openshift
and
make
that
easy,
I
don't
have
ancestry
in
the
database
management
thing.
A
So
what
this
talk
is
about,
actually
is
the
third
component,
which
is
performance,
which
is,
if
I,
throw
my
databases
on
kubernetes
on
cloud
native
storage.
How
do
they
perform?
Is
it
good
enough
for
my
applications
and
that's
what
I've
been
spending
a
few
months
on
a
particular
cluster
working
on
now?
A
The
reason
why
performance
is
critical,
there's
a
couple
of
things.
One
is:
if
you
talk
to
any
group
of
database
geeks,
any
group
of
system
infrastructure
admin
gigs
they
care
about
performance.
Write
me
one
of
the
first
questions
they
ask
and
therefore
you
have
to
have
an
answer
for
them.
The
second
thing
is
for
any
platform.
There's
this
trade-off
between
ease
of
use,
ease
of
management
and
speed.
A
The
I,
you
know,
does
an
example,
for
example,
low
level
programming
languages,
things
like
C
little
and
assembler
execute
faster
than
higher-level
languages
that
are
often
easier
to
use
and
easier
to
learn
and
the
same
is
truly
an
infrastructure
platforms
and
then
a
third
is
if
your
application
was
designed
to
expect
a
certain
level
performance
from
the
database.
Then,
if
you
can't
get
that
level
of
performance
on
a
cloud
native
platform
that
is
going
to
be
a
blocker
to
migrating
to
kubernetes
and
cloud
native
technology
in
general,
you
don't
want
it
to
be
a
blocker.
A
So
I'm
not
gonna,
stop
I
actually
find
it
kind
of
fun,
because
I'm
weird
that
way
so
and
the
main
way
that
you
actually
deal
with
database
performance
is
through
benchmarking.
Now
we've
got
a
bunch
of
people
here
from
ping
cap.
They
know
what
database
benchmarking
is
etc,
but
for
everybody
else
to
talk
a
little
bit,
what
Barry
King
is
and
benchmarking
isn't.
A
So,
when
we're
talking
about
benchmarking,
stuff,
we're
actually
talking
about
comparing
right,
we
want
to
compare
two
things
that
are
equal
in
all,
but
one
respect,
so
we
can
find
out
what
effect
that
difference.
Has
you
know
so
benchmarking
between
two
different
types
of
storage,
from
one
release
to
another,
from
one
configuration
change
to
another
or
from
the
requirements
that
we
already
have
on
spec
of
must
return
X
number
of
responses
per
second
right.
This
is
what
we're
looking
on
comparing
and
so
anytime.
You
were
looking
at
benchmarking.
A
You
should
be
saying
what
two
things
are:
three
things
or
ten
things
am
I
going
to
be.
Comparing
now
for
this
talk,
what
I
am
actually
comparing
is
types
of
storage,
because
part
of
the
way
I
get
started
in
this
I
work
at
Red,
Hat
I
work
collaborate
with
the
Ceph
and
rook
teams
that
we
have
at
Red
Hat,
and
one
of
the
things
we
wanted
to
look
at
is,
if
you
have
pure
cloud
native
storage
like
a
rooks.
F
stack
is
performance
on
that
acceptable
for
a
database
workload.
Nobody
had
an
answer.
A
A
So
no
logs,
there's
like
host
path,
Network
storage,
which
is
like
EBS
cloud
provider,
storage
or
whatever,
and
then
cloud
native
distributed
storage.
So
something
like
a
real,
safe,
combo
or
other
advanced
file
systems.
Now
I
already
did
a
run
for
this,
and
you
can
actually
find
my
talk
from
coop
con
Seattle.
A
Now,
beyond
that,
we
need
to
talk
about
types
of
I/o
performance
that
you
get
in
a
database
workload
so
database
words
basically
do
four
things
and
they
often
do
them
concurrently
with
each
other
right.
You
have
random
reads
so
reading
one
small
fact
at
a
time,
random,
writes
writing
one
small
fact
at
a
time
sequential
reads
when
you
read
big
blocks
of
data
and
sequential
writes
when
you
write
big
blocks
of
data,
databases
will
do
all
of
these
things
concurrently
or
consecutively,
and
so
we
care
about
the
performance
of
all
of
them.
A
Now,
generally,
when
we're
doing
our
synthetic
benchmark,
we
can
actually
combine
the
random
reads
and
random,
writes
which
we'll
be
doing
in
these
benchmarks.
That
I
show
you,
because
most
workloads
that
do
random
rates
do
random,
writes,
etc,
and
you
can
do
them
together
without
messing
with
the
numbers,
and
then
we
actually
care
about
two
different
classes
of
metrics.
For
those
right,
we
care
about
latency,
which
is
if
we
make
a
request.
A
How
long
does
it
take
that
request
to
return
affects
application
performance
right
and
we
also
care
about
throughput,
which
is
how
many
requests
per
second
or
how
many
megabytes
of
data
per
second
can
I
say
read
from
or
write
to
the
database
for
throughput,
so
we
care
about
both
of
these,
and
so
what
we
have
here
is
three
different
storage
platforms.
Three.
A
Fortunately,
the
database
industry
for
a
long
time
has
been
obsessed
with
benchmarking
things,
and
so
there
are
tons
of
tools
and
benchmarking
tools
and
measurement
tools
already
designed
to
generate
these
kinds
of
traffic
and
record
this
kinds
of
information
that
you
can
just
use
and
tap
into
and
I'll
be
naming
a
few
of
those
here,
particularly
what
I'm
gonna
be
talking
about,
because
this
was
easy
for
me
to
do.
Since
this
is
not
my
full-time
job.
A
Most
of
my
job
is
managing
kubernetes
community
stuff
is
I,
ran
a
series
of
what
I
know
is
micro
benchmarks.
So
you
have
your
benchmark:
Suites
things
like
transaction
processing,
council
and
spec
and
stuff
that
have
an
enormous
amount
of
battery
of
different
things
that
they
do
and
they're
audited
and
that
sort
of
thing
and
that's
that's
a
benchmark
benchmark
with
a
capital
B,
and
then
you
have
micro
benchmarks,
which
are
smaller,
easy
to
run
workloads,
and
these
are
generally,
if
you're
running
it
personally
for
yourself
and
you're
not
doing
it
for
publication.
A
Unfortunately,
I'm
gonna
show
you
the
cockroach
DB
workloads
and
results
that
I
got,
but
these
are
no
longer
open-source.
So
this
is
the
last
time
that
you
will
see
these
for
me
because
they
change
the
license
for
them
and
they're
not
open
source
anymore.
Unfortunately,
so
talk
about
suspense
suspense
is
a
nice
toolkit.
Like
first
thing
you
should
try
honestly,
is
as
much
because
it's
a
sort
of
omnibus
micro
benchmark
created
by
the
my
sequel
folks
years
ago.
A
A
It's
a
common
problem
in
databases
and
the
two
of
these
within
that
suite
is
I
found.
Bank
was
really
good
for
measuring
throughput
and
TPCC
was
really
good
for
measuring
latency
for
complex
operations
that
need
to
do
transactions.
So
now
let
me
give
you
some
tips,
because
part
of
my
goal
in
this
talk
is
not
necessarily
to
show
you
the
numbers
that
I
have,
because
the
numbers
that
I
have
are
not
your
numbers.
Your
hardware
is
not
my
hardware,
your
application
is
not
my
application
and
your
stack
is
not
my
stack.
A
A
One
is
if
you're
running
micro,
benchmarking,
you
need
to
do
a
bunch
of
runs,
don't
run
it
once
recorded
that
number
and
say
that's
how
it
is
because
there's
a
certain
amount
of
randomness
and
all
of
these
benchmarks,
and
if
you
just
depend
on
a
single
run,
the
randomness
may
be
what
you
get
instead
of
a
real
thing.
You
also
need
to
do
long
runs.
A
A
lot
of
people
would
say:
oh
I
did
a
30
second
run
and
I
got
this
well,
you
have
memory
cache
effects
and
CPU
cache
effects
and
a
lot
of
things
where
you
will
get
artificially
inflated
performance
on
really
short
runs,
and
you
really
need
to
see
how
the
system
is
going
to
behave
under
more
sustained
load.
Ideally,
you
want
to
do
multiple
database
and
file
sizes.
Particular
you
want
to
measure
both
things
that
fit
in
memory
and
things
that
don't
you
wanted
you
multiple.
A
You
want
a
concurrent
workload,
because
your
real
production
workload
is
going
to
be
concurrent,
so
multiple
users
accessing
the
database
or
the
file
storage
at
the
same
time,
and
you
want
to
use
bare
metal
now
a
lot
of
people
like
this.
Well,
why
would
I
want
to
use
bare
metal?
Well,
here's
the
problem.
I've
done
also
a
lot
of
benchmarking
on
cloud
providers
and
the
problem
with
benchmarking
in
cloud
providers
is
that
a
lot
of
your
performance
effects
have
more
to
do
with.
A
Who
else
is
on
the
cloud
with
you
than
they
do
with
any
changes
in
platform
so
like?
If
you
look
at
the
stuff
that
are
presented
in
Seattle,
if
you
look
at
Ben's
run,
runs
in
AWS
I,
consider
a
minimum
number
of
runs
for
a
single
AWS
workload
at
a
specific
size
to
be
like
25
and
for
each
of
those
I
actually
recreate
the
instances,
because
you
never
know
what's
gonna,
be
the
effect
of
a
bad
instance
or
a
noisy
neighbor.
A
A
So
that
said,
let's
look
at
some
numbers,
because
it's
one
of
the
things
that
we
care
about
here
right
so
I
also
want
to
add
some
caution
here,
which
is:
please
do
not
compare
the
numbers
between
different
benchmarks
and
different
databases.
These
numbers
are
not
meant
to
be
comparable
right,
so
the
TPCC
benchmark
for
cockroach
DB
does
not
perform
the
same
activity
as
the
PG
vent
one
for
Postgres
and
those
numbers
are
not
comparable.
A
The
databases
were
minimally
tuned
on
purpose.
Basically,
I
did
the
Proform
or
performance
tuning
that
you
could
do
in
ten
minutes
for
any
of
the
databases,
because
that's
kind
of
what
most
people
are
running
on
on
top
of
which
I
didn't
want
to
make
this
an
exercise
in
performance
tuning
the
database.
My
goal
was
to
test
the
cloud
native
platforms
and
then
again,
like
I,
said
my
software.
My
hardware
is
going
to
be
different
from
yours.
You
need
to
test
yours
and
not
just
read
my
numbers.
So
here
was
our
bare
metal
platform.
A
That
I
was
doing
this
on
I'm
continuing
to
run
tests
on
this
because
I
have
this
cluster,
it's
a
six
blade
cluster.
Each
of
the
blades
is
20
cores,
128
gigabytes
of
RAM,
two
SSDs
with
200
gigabytes
of
storage,
each
and
a
shared
network.
This
is
in
the
open
source
lab
at
Red
Hat,
the
stuff
that
we
have.
We
had
it
for
testing
something
else
in
the
past
and
I
repurposed
it
for
this.
A
Now
there
is
one
caveat
here
that
limited
the
kind
of
tests
I
can
run
now,
for
one
of
the
things
you
want
to
test
is
things
that
fit
in
memory
and
things
that
don't
fit
in
memory.
The
problem
is
that
when
you
have
a
drive,
that's
200
gigabytes
and
you
have
128
gigabytes
of
RAM,
it's
very
hard
to
test
the
out
of
memory
use
case
because
you'll
run
out
of
disk
space.
A
That's
going
to
dominate
our
numbers,
the
other
thing
is
I've
assured
Network
and
that
had
an
effect
on
one
of
the
tests
later
on.
That
I
will
show
you
that
that
eclipses,
the
effect
that
I
wanted
to
get
in
terms
of
actually
comparing
things
so
the
first
set
of
tests
that
I
ran.
Is
you
want
to
start
out
with
a
cavity?
A
You
want
to
start
out
with
a
benchmark
of
this
is
this
is
what
I
would
get
on
a
plane
platform
before
I
do
cloud
native
anything
right,
and
so
we
get
a
host
file
system
where
I've
got
a
host,
install
no
kubernetes
to
get
reference
numbers
just
using
X,
FS
and
l,
VM
and
hosting
databases
and
file
storage
on
that,
so
first
I
want
to
start
out
with
suspense.
That
is
just
going
to
check
the
sort
of
direct
IO
numbers.
A
It
does
a
bunch
of
I/o
operations
and
see
and
record
this
back,
and
so
the
numbers
that
we
got
here
from
the
SSDs
here
right
is
that
random
reads
per
second
10,000,
whatever
high
number,
because
we're
getting
this
out
of
memory
because
again,
I
can't
actually
use
a
file
size,
that's
bigger
than
memory,
so
that
should
be
really
fast
and
it
is,
but
also
random.
Writes,
are
also
really
fast
right.
7,000
random
writes
per
second
nice
fast,
SSD
being
able
to
use
two
of
them
there.
A
I
can
read
22
gigabytes
per
second
of
data,
so
here's
our
throughput
numbers
and
I
can
write
88
megabytes
per
second
of
data.
So
here
you
see,
this
is
gonna,
be
a
typical
pattern
of
SSDs,
where
SSDs
are
much
faster
than
old.
Spinning
drives
for
random,
writes
but
they're,
not
necessarily
all
that
faster
in
throughput,
because
that's
writing
big
blocks
of
data
now
a
little
bit
more
complicated
chart
here.
This
is
the
set
of
database
benchmark.
A
So
we've
got
PG
bench
well,
we
want
to
get
some
throughput
in
terms
of
database
load
time,
simulating
loading
large
amounts
of
data
into
the
database
transactions
per
second
for
another
measure
of
throughput
and
then
average
latency
to
measure
latency
and
then
for
bank.
Similarly,
on
cockroach
DBU
transactions
per
second
95%,
latency
and
TBC
see
when
you're
actually
doing
that
benchmark
competitively.
The
standard
is
new
orders
per
something
new
order,
transaction
per
minute,
I'm
doing
per
second,
and
so
that's
what
I
actually
have
there?
A
Whether
or
not
I'm
meeting
an
arbitrary
target
of
how
many
new
orders
I?
Can
process
in
90%
latency
so
on
the
bare
thing
for
PG
bench,
404
seconds
for
the
bulk
load
so
again,
lower
is
better
here,
440
seconds
for
the
book
load
11,000
transactions
per
second,
so
I'm
actually
doing
slightly
better
than
the
file
system.
So
there
we
see
post,
quizzes,
batch,
writing
kicking
in
and
then
an
average
latency
of
2.8
milliseconds.
A
A
A
A
A
You
remember
I,
mentioned
that
we
were
on
a
shared
network
for
the
set
of
blades.
Well,
that's
actually
what's
going
on
here,
because
so
for
the
bare
metal
test.
I
had
to
run
the
PG
bench
client
on
bare
metal,
which
means
that
to
make
things
a
good
comparison,
I'm
still
running
the
PGP
client
unbearable
and
using
node
port
to
route
it
to
post
was
running
on
kubernetes.
The
problem
with
that
is
that
means
that
we've
actually
added
a
couple
of
extra
Network
hops
for
every
request
and
it'll
work
load
like
PG
bench.
A
A
A
I
didn't
want
to
do
the
standard,
3x
replication,
because
we're
only
talking
about
five
rook
nodes,
the
some
default
tweaks
off
of
the
rook
documentation
for
performance
and
also
by
the
way
whose
important
thing
I
will
be
showing
you
cockroach
TV
things,
there's
two
different
ways:
you
can
do:
cockroach
TV
with
rock
one.
Is
you
the
standard
thing
to
do?
Is
you
run?
Have
rook
manage
cockroach
TV
for
you,
which
is
honestly
the
easiest
way
to
install
cockroach
TV?
A
If
you
still
want
to
install
it
at
this
point,
but
my
goal
was
to
test
set
performance,
so
I
was
actually
installing
cockroach
TV
on
top
of
Ceph,
rather
than
in
so
on
cockroach
to
be
with
rook.
This
is
probably
not
something
you
would
actually
do,
because
cockroach,
TB
and
Seth
provides
similar
levels
of
redundancy,
and
so
I
have
double
redundancy
here,
which
is
probably
not
necessary
for
most
workloads,
but
it
does
help
me
measure
performance.
A
So,
let's
look
at
suspense
here.
So
just
off
of
raw
stuff,
I
was
really
pleasantly
surprised
by
what
we
were
looking
for.
Rafael
IO
right
I
mean
that
is
minority
loss
of
performance
now
mind
you
keep
in
mind.
This
is
files
that
fit
in
memory,
but
even
with
the
right
Brook
is
just
not
imposing
much
of
a
penalty
on
my
right
speed
for
those
I
mean
considering
that
I
basically
haven't
automated
backup
in
this
right.
I'm
those
files
all
exist
in
new
places.
A
That's
actually
really
nice,
the
biggest
thing
that
we're
actually
seeing
there
is,
and
then
the
other
weird
thing
is.
For
my
sequential
reads,
my
large
block
reads:
it's
actually
faster.
So
talk
to
some
of
the
people
about
this
like
why
it
would
be
faster
on
sequential
reads
than
bare
metal
and
it
turns
out
that
Seth,
if
you're
reading
a
bunch
of
contiguous
blocks,
it
will
try
to
pull
them
from
multiple
replicas,
so
you
can
read
them
faster
and
it
turns
out
that
that
has
a
real-life
benefit
in
the
sequential
read
case.
A
Now,
let's
actually
talk
a
little
bit
about
the
database.
Mint
marks
here
right,
so
bulk
load
gets
lower
spec
to
that,
but
only
about
a
third
slower
and
the
biggest
thing
is
my
latency
doubled,
which
is
not
a
big
surprise
again
for
redundant
cloud
native
storage.
What
is
pleasant
is
that,
yet
only
about
doubled,
how
many
people
here
have
worked
with
forms
of
redundant
clustered
file
systems
before
cuz
I've
been
doing
this
for
a
while.
A
We
knew
this
for
a
number
of
years,
and
doubled
is
actually
a
very
good
number
for
some
of
the
older
ones
like
moose,
Orange,
FS
and
stuff.
We
would
be
looking
at
more
like
quadrupling
or
Quinn
tooling,
the
latency
and
so
double
latency
is
actually
pretty
good
because
we
are,
after
all,
writing
everything
twice
so
same
thing
with
the
copper
HD
benchmarks
is
we're
getting
about
double
the
latency
and
half
the
throughput,
which
are
directly
related
to
each
other.
A
Now
TPCC
is
a
target
based
benchmark,
so
that
1290
is
how
many
transactions
did
we
complete
that
will
within
the
target
window,
not
what's
the
maximum
number
of
transaction
that
we
can
complete
and
we
could
still
make
the
threshold,
because
that
double
agency
was
still
below
our
threshold
and
rook.
You
know,
Seth,
didn't
add
any
additional
overhead
now.
Some
things
I
actually
want
to
do
to
tinker
with
this
is
obviously
I
want
to
actually
do
it
on
something
without
a
shared
network.
So
I
can
eliminate
the
shirt
now
effects
from
my
figures.
A
I
want
to
actually
get
bigger
SSDs
so
that
I
can
do
bigger
workload,
not
in
median
cockroach
TV
anymore
I'll
be
looking
at
another
cloud
native
database
to
actually
do
performance
tests
on
and
to
maybe
do
some
additional
tests
tuning.
But
let
me
give
you
some
conclusions
and
walk
away.
It's
from
this.
Yes,
first
of
all,
you
can
benchmark
your
own
hardware
and
your
own
cloud
native
stack
with
simple
database
benchmarks
to
test
the
performance
of
that
stack
and
whether
or
not
you
need
to
change
it.
A
Local
volume
performance
should
be
roughly
equivalent
to
bare
metal
and
rooks
F
has
good
throughput,
but
about
double
the
latency
compared
to
running
on
bare
metal
which,
again,
if
the
redundancy,
if
the
cloud
native
storage,
is
valuable
to
you
double
the
latency
is
good.
If
it's
not
valuable
to
you,
you
should
be
aware
that
that
exists
more
important
things.
Beware
secondary
issues
that
will
click
performance
differences
like
my
shared
network
problems
and
an
important
thing,
and
go
back
to
look
away.
A
My
presentation
from
coupe
con
see
at
all,
and
you
will
see
that
on
public
cloud
cloud,
latency
effects
mask
a
lot
of
performance
differences
that
you
can't
actually
see
a
lot
of
these
performance
differences,
because
the
effect
of
the
cloud
itself
is
so
large.
So
questions
contact.
If
you
have
questions
about
rook,
a
couple
of
the
rook
developers
are
here
at
the
conference
and
you
can
find
them
there's
going
to
be
actually
two
rook
talks
later
today,
one
in
the
next
time
slot
one
at
6:00
p.m.
so
you
can
find
out
about
that.
A
B
A
A
A
The
probably
there's
the
SSDs
right
is
that
ultimately
I'm
writing
to
a
single
SSD,
and
you
know
both
the
bus
and
the
RAM
cache
and
the
SSD.
You
know
with
a
large
right,
because
it's
writing
several
gigabytes
of
data.
The
REM
cache
in
the
SSD
is
going
to
fill
up
pretty
quickly
and
then
at
that
point,
you're
going
to
actually
be
at
its
performance,
which
apparently
isn't
that
great.
A
So
the
I
didn't
actually
care
that
much
about
it,
except
to
get
it
for
reference,
but
it's
not
actually
that
unusual
for
large
writes,
is
all
of
us,
as
these
benefits
are
in
random,
writes
in
small
rights,
etc
for
large,
writes,
you're
operating
in
the
performance
of
whatever
the
slowest
component
in
the
whole
bus
is,
which
apparently
is
not
that
fast.
In
this
case,
these
are
blades
after
all,
and
not
full
servers.
Ok,.