►
From YouTube: How Container Networking Affects Database Performance - Tyler Duzan & Vadim Tkachenko, Percona
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
How Container Networking Affects Database Performance - Tyler Duzan & Vadim Tkachenko, Percona
Through benchmarks, Percona Labs explores the effects of different container networking drivers used in Kubernetes when hosting database workloads. For this talk, we will perform benchmarks using Percona's PXC Operator deploying a 3-member PXC MySQL cluster on top of Kubernetes and use our standard database benchmarking stack with TPCC and Sysbench to analyze query throughput and replication performance as affected by our choice of networking driver. Drivers we'll test will be CNI core plugins, Flannel, Cilium, Calico, Kube-Router, and the new Red Hat SR-IOV driver. This Dual Presentation (35 minutes) will address our benchmark methodology and results, as well as recommendations regarding networking and tuning database performance on Kubernetes with a focus on MySQL. Both speakers are experts on this topic, and Vadim co-authored "High Performance MySQL", now in its 3rd Edition.
https://sched.co/UaZ8
A
Good
afternoon
my
name
is
Jason
de
Torres
and
I'm
your
track
host.
Today,
if
you
can,
and
if
you're
sitting
near
the
edge
of
a
row,
please
move
in
so
that
it's
folks
come
in,
they
have
a
chance
to
sit
down
and
also,
please
remember,
to
rate
this
talk
on
sketch
comm.
At
the
end,
our
next
speakers
are
going
to
tell
us
about
how
container
networking
effects
database
performance.
First,
we
have
co-author
of
high
performance.
My
sequel
he's
also
the
co-founder
and
CTO
of
Percona
Vadim.
B
Cool
so
yeah,
thank
you
for
the
introduction,
so
we're
gonna
be
talking
about
how
can
hit
our
networking
effects
database
performance
in
kubernetes,
but
before
we
get
into
that,
I
want
to
tell
you
who
Prachanda
is
since
we're
relatively
new
to
the
kubernetes
community.
So
percona
is
a
group
of
open-source
database
experts.
We
were
the
unbiased
champions
of
open-source
databases
and
we
do
services
and
software
around
Postgres
and
MySQL,
and
all
of
our
software
is
100%,
free
and
open
source.
B
So
our
commercial
business
is
based
on
support
and
consulting
services,
and
then
I'm
gonna
skip
our
little
intros
since
we
had
a
nice
intro
and
talk
about.
Why
we're
doing
this
talk
so
we
we
started
kind
of
entering
into
the
kubernetes
space
because
we
want
to
be
able
to
make
a
bridge
for
databases
into
the
containerized
world.
You
know.
Historically,
containerized
systems
were
really
focused
on
stateless
applications,
not
stateful,
and
so
we
wanted
to
move
from
a
place
where
it's
like
I.
Don't
do
this
kind
of
recommendation
to
hey.
B
This
is
a
possible
thing
to
do
and
we
should
do
this
in
production.
So
this
talk
is
not
about
blaming
or
finger-pointing.
We
are
doing
benchmarking
and
showing
results,
but
our
focus
is
really
to
start
a
conversation
in
the
community
and
understand
how
our
adventure
started
with
databases
and
then
how
that
caused
us
to
get
into
other
underlying
primitives
like
networking
and
storage
in
communities.
B
In
deploying
this
into
production
workloads
and
in
the
process,
we
noticed
really
significant
performance
differentials
that
we
didn't
expect,
and
so
the
other
thing
that
we've
seen
is
that
performance
could
really
be
a
barrier
of
entry
to
adoption,
because
a
lot
of
people
right
now
are
running
kubernetes
for
their
applications,
but
their
databases
still
live
outside
of
kubernetes
on
bare
metal,
and
so
we
wanted
to
understand.
Basically,
how
does
the
performance
differential
sit?
What
are
some
things
that
are
available
in
the
ecosystem?
B
What
are
the
right
recommendations
we
can
make
to
our
customers
to
make
it
possible
for
them
to
migrate
their
data
into
kubernetes
and
part
of
that
ties
into
storage?
Part
of
it
ties
in
and
networking
and
today
we're
gonna
focus
on
the
networking
piece,
and
so,
with
that
I'll
hand
it
over
to
Vadim
to
talk
about
our
methodology
for
benchmarking.
Yes,.
C
Yeah,
hello,
everybody
so
I
am
a
performance
guy
I
like
to
run
Auto
benchmarks,
and
for
this
specific
case
we
had
to
design
our
specific
benchmark
and
first,
let's
talk
about
our
layout.
So
for
our
benchmark.
We
used
four
nodes:
kubernetes
cluster,
with
one
master
and
three
nodes
and
for
internal
communication
we
used
ten
gigabit
network
cards
and
network
switch
and
for
external
communication
with
the
outside
world.
It
was
one
gigabyte
switch
and
it
seems
this
set
up
already
caused
some
complication
for
several
network
plugins.
C
We
had
issue
like
some
of
them
didn't
know
how
to
route
network
properly
seems
to
they
mixed
up
ten
gigabit
and
when
we
give
it
a
connections,
so
we
had
to
deal
with
that
and
the
some
of
them
I
don't
think
we
resolved
properly
yet
yet
you
will
see
that
in
a
result,
maybe
we
can
improve
our
results.
If
you
can
resolve
network
issue
head
and
therefore
client
workload,
client
generation
were
quote,
we
use
seat
bench
which,
which
also
was
connected
on
separate
boxed
or
ten
gigabit
network.
C
And
let's
talk
a
little
bit
about
database
setup
for
a
database.
We
use
per
connects
to
the
cluster
which,
again,
as
product
may
be
not
very
familiar
for
this
audience.
So
I
wanted
to
took
a
minute
and
talk
about
whatever
connects
to
a
big
cluster
and
basically
he,
if
you're
familiar
with
MySQL,
and
if
you
interested
in
that
database
I'm
sure
he
familiar
is
my
scale.
Reckon
extra
big
cluster
is
basically
multiple
MySQL
nodes
communication
together
using
some
synchronous
protocol
for
this
setup
and
the
minimum
recommended
and
recommended
way
to
use
is
three
nodes.
C
Just
in
case
there's
not
for
failure.
You
still
have
two
nodes
running
in
the
issue:
have
only
two
nodes:
they're
always
split
brain
situation
which
are
not
good
to
have
so
three
nodes,
minimal
recommended
set
up
and
just
a
magical
mode,
Iranian
together
through
synchronous
protocol,
and
he
set
up,
allow
us
to
achieve
two
important
qualities
for
databases.
What
we
were
like
from
databases
like
high
availability
in
the
data
consistency.
The
consistency
is
achieved
by
this
synchronous
interaction
between
north
and
this
synchronous.
C
Interaction
between
nodes
happens
on
the
internal
communication,
which
happens
saloon
network
between
nodes,
so
it
would
by
itself
it
with
a
require
and
demand
the
high
performance
network.
Well,
if
you
would
like
to
achieve
high
performance
result
and
they're
both
characteristic
from
Network
latency
is
important,
because
transaction
latency
in
production,
latency
in
databases,
basically
how
quickly
your
application
I
respond
to
your
query
and
you
run
some
query
refresh
Facebook
page.
C
You
expect
answer
pretty
much
within
second,
so
latency
network
latency
in
this
case
will
contribute
the
transaction
latency
after
user
experience
and
also
important
throughput,
because
not
only
a
one
user,
you
worried
about
you
worried
about
high
performance
for
called
hundreds
thousands
transactions
running
per
second,
and
in
this
case,
transactional
throughput
will
depend
on
your
network
support.
So
this
setup
applies
quite
the
amount
of
network
performance.
C
Another
aspect
in
databases
is
storage
and
especially
if
you
use
shared
storage,
which
Iranian
across
network,
there
is
additional
demand
of
network,
but
in
our
setup
we
will
leave
storage
out
of
equation
just
to
understand
network
networking
aspect
of
database
performance
and
for
database
layout
setup.
It
will
be
very
simple:
three
nodes.
C
Three
nodes
of
Peconic
syllabic
cluster
for
this
we
allocate
each
node
would
run
in
individual
pod
as
used
in
our
operator,
and
then
we
prescribe
anti
affinity
rules
in
our
operators,
recurrently
that
each
node
will
run
on
individual
parameter
server.
So
this
kind
of
mapping
for
kubernetes
port
I
mean
we
guarantee
that
each
ended
up
order
ants
on
a
separate
server
and
we
use
clients
outside
the
fur.
This
network,
outside
of
her
kubernetes,
using
port
mapping
to
access
databases.
C
So,
for
benchmarking,
myself
be
used
on
a
software
side
is
his
bench
with
OTP.
Read/Write
overcoat
is
very
simple
workload,
but
it
allows
to
or
database
pretty
decently
and
read/write
would
allow
us
to
load
network
communication
between
nodes.
Where
network
communication
happens,
when
there
is
change
changes,
there
is
a
right
for
court
or
database
and,
along
with
this,
will
measure
just
roll
network
bandwidth
which
we
able
to
achieve
with
some
or
another
plugin
and
produce
I
used
type
of
three
very
simple
to
capture
network
support.
C
We
also
collected
for
baseline,
we
collected
results
on
bare
metal
on
system
running
without
cobrin
edges,
so
we
understand
what
is
kind
of
ideal
performance
we
would
like
to
achieve
in
the
compare
kubernetes
operation
with
bare
metal
performance
network.
Aside,
like
I
mentioned,
we
use
two
network
cards,
one
ten
gigabit
another
one
gigabit
for
communication
between
nodes
and
operation
system,
a
little
bit
outdated,
but
that's
what
we
had
on
the
system:
ubuntu
1604
with
415
linux
kernel.
C
Then
connects
to
deploy
kubernetes,
I
used
very
simple
cube
admin
script
which
would
allow
us
to
spoon
up
three
nodes
and
one
master,
and
then
we
installed
network
plugins,
probably
Tyler,
will
speak
about
speed
budget
more
and
we
did
our
best
to
configure
is
Network
plugin
to
use
10
gigabit
network
interface
like
I
mentioned
I,
don't
think
we
were
successful
in
all
cases.
It
requires
some
extra
work
for
hardware.
Why
was
there
a
hardware
specification
of
our
servers?
C
I
will
not
list,
it
I
think
that's
pretty
decent,
inter
machineries
enough
memory
and
decent
storage
options
and
on
software
side
we
used
per
connect
3,
because
there
577
are
nothing
special.
Using
our
system
packages
from
our
repose
for
kubernetes
tests.
We
used
our
operator
version
1.2,
which
would
deploy
the
same
version
of
per
connected
cluster
in
kubernetes,
5,
727
and
but
to
achieve
better
database
performance.
C
We
cannot
rely
on
out
of
box
parameters,
yet
we
still
have
to
tune
in
any
configuration
like
my
C
and
F
in
cases
with
MySQL,
and
we
used
local
storage,
local
SSD
storage.
Again,
to
avoid
this
shared
storage
case,
where
which
would
introduce
additional
record
on
a
network,
I
would
like
to
oscillate
storage
problem
from
network
problem
in
this
case.
C
So
there
is
some
basic
parameters
if
you
would
like
to
repeat
in
the
actually
very
first
issue,
I
faced
here,
trying
to
measure
database
performance
is
quality
of
service
in
the
Copernicus
which
by
default,
is
the
best
effort,
but
I
will
show
the
next
slide.
If
you
use
with
default
quality
of
service
and
kubernetes,
your
result,
variance
will
be
first,
results
will
be
suboptimal
and
second
variance
in
results
also
will
be
quite
problematic.
C
So
it
also
took
time
to
figure
out
what's
going
on
here,
then
we
came
to
the
heat
of
fusion,
guaranteed
quality
of
service
and
actually
to
make
sure
that
you
using
guaranteed
quality
of
service
for
kubernetes
pods.
You
have
to
specify
this
kind
of
magic
parameter
as
a
request
and
limits
to
be
exactly
the
same
and
as
high
as
possible
for
your
pods
than
your
pod
and
kubernetes
will
provide
best
optimal
performance
and
still
I
have
had
to
use
this
magic
number
CPU
55.
C
Even
there
is
a
56
course,
including
high
high
persuasion
course.
Kubernetes
would
not
allow
me
to
put
56
EPOC
here,
communities
complaining
or
lack
of
resources,
so
I
had
to
increase
on
one
simple
core
allocated
for
our
database
ports,
which
also
might
contribute
to
some
decrease
of
performance,
but
I
hope
for
the
hope.
Not
that
sentence
significant
and
sorry
and,
as
I
said,
to
compare
guaranteed
quality
of
service,
which
we
used
with
best
effort.
C
C
Period
right
test
these
parameters,
ten
tables-
is
ten
million
rows.
Each
you'll
get
about
thirty
two
gigabytes
of
data.
We
have
connect
gig
of
memory
which
much
bigger
than
32
gig
database.
So
pretty
much
all
our
work
quality
will
be
done
in
memory
with
minimal
impact
on
storage.
So
again,
with
goal
to
Lord
Network
to
see
what
happens,
engagement,
network
performance
is
bounded
by
network
plugins
and
on
networks,
plugins
Tyler
will
talk
about
four
plugins,
which
we
use.
Thank.
B
You
buddy,
and
so,
as
you
can
see,
we're
trying
to
be
really
transparent
about
what
our
methodology
is
here.
We
we
recognize
a
lot
of
times
when
folks
do
benchmarking,
it's
hard
to
replicate
them
or
they're,
not
very
transparent.
So
our
goal
is
to
start
a
conversation,
the
community
and
really
understand
how
we
can
improve
performance
across
the
board
and
kubernetes
for
stateful
applications,
and
so
I'm,
just
gonna
give
a
quick
overview
of
what
the
network
plugins
are
that
we
tested-
and
maybe
some
notes
that
we
took
on
these,
but
no
strong
recommendations
or
conclusions.
B
So
also
I
was
going
to
say.
The
list
of
plugins
we
tested
is
not
exhaustive.
We
chose
the
ones
that
we
heard
about
from
our
customer
base,
so
if
customers
were
using
them
or
had
considered
using
them,
they
got
added
to
the
list.
So
I
already
found
like
three
more
at
the
conference
this
week
that
I
hadn't
even
heard
of
yet
so
we
may
be
testing
more
in
the
future.
B
So
first
up,
we
tested
calico,
which
is
described
on
their
website
as
project
calico
is
designed
to
simplify
scale
and
secure
cloud
networks,
and
one
of
the
interesting
things
is.
It
supports
an
IP
and
IP
feature,
so
we
tested
with
and
without-
and
it's
enabled
by
default,
we
tested
flannel
flannels
described
on
their
website
is
network
fabric
for
containers
designed
for
kubernetes,
and
it
has
support
for
actually
multiple
backends.
B
The
ones
we
tested
was
the
default
UDP
and
we
also
tested
the
excellent
we
attempted
to
test
host
gateway
and
we
saw
some
interesting
behavior
and
then
ultimately
were
able
to
complete
a
test
successfully.
So
we
removed
those
results,
but
we
were
able
to
see
pretty
good
on
average
performance,
VX
land.
But,
interestingly
with
VX,
we
saw
weird
stalls.
B
So
peak
performance
was
good,
but
we
had
not
very
good
consistency
and
we
think
that
might
be
a
kernel
issue
because
of
the
age
of
the
kernel
we
were
running,
we
were
on
for
15,
so
we're
gonna
look
at
that
in
the
future,
but
since
all
tests
were
run
on
the
same
hardware,
it's
still
a
valid
set
of
results.
We
also
tested
cilium,
say
liam's
described
on
their
website
as
api
aware
networking
and
security.
B
B
Similarly,
we've
which
is
described
on
his
website
is
simple:
resilient
multi,
hose
container
networking
and
more
was
actually
probably
the
easiest
to
install,
but
we
still
had
some
concerns
with
trying
to
get
it
to
route
across
10
gig.
This
was
actually
a
very
interesting
one.
Where
the
result
was.
We
thought
that
it
was
routing
across
10
gig,
but
the
the
performance
results
didn't
match
that
and
then
on
further
investigation
P
capping.
B
We
saw
it
was
going
across
the
gigabit
interface,
even
though
the
routing
rules
that
we
can
see
showed
it
should
have
gone
across
a
10
Gator
pace.
So
we
ended
up
doing
really
significant
troubleshooting
on
slack
with
some
folks
and
then
ran
out
of
time.
So
we
ended
up
reporting
the
results
that
we
had
and
we'd
like
to
redress
this
one.
Also
after
the
conference
we
tested
intel's
SRO
V
CNI
driver
alongside
Malta's,
and
this
is
kind
of
interesting.
B
It
Malta's
basically
enables
you
to
have
multiple
CNI
drivers
used
to
attach
multiple
interfaces
to
a
pod.
So
with
this
one,
we
actually
use
cube
router
as
a
default
pod
Network,
and
then
we
added
an
additional
interface
inside
the
pod,
that
was
the
10
gig
interface
provided
by
SRO
V
and
then
finally,
we
tested
keep
router,
and
this
one
was
kind
of
it's
described
as
a
turnkey
solution
for
kubernetes
networking.
We
found
that
relatively
be
true.
It
pretty
much
worked.
B
C
So,
let's
talk
about
our
results.
Firstly,
I
measure
the
database
throughput
on
bare
metal
and
again
to
make
sure
that
network
is
a
limiting
factor
here.
Imagine
that
the
database
performance
through
one
gigabit,
a
channel
and
in
this
case
for
a
signature
called
the
achieved,
2700
reduction
per
second
and
when
we
switched
when
I
switch
to
Tammy
a
gigabit
network,
the
result
improved
to
over
seven
thousand
three
hundred
thousand
per
second.
C
Actually
you
see
the
results,
Lynnette
improve
quite
ten
times
as
a
network
characteristic,
but
we
can
see
that
the
database
performance
is
clearly
limited
by
one
gigabit
network
bandwidth
in
this
case
and
on
this
chart
I
trying
to
summarize
the
results
that
may
be
out
of
data
to
process.
So
let's
go
through
this.
So
this
kind
of
green
line
is
network
bandwidth
which
I
measured
with
RI
perv
and.
C
C
Best
performance
we
were
able
to
achieve
a
cube
router
with
calico
being
a
second.
We
mentioned.
There
is
some
plugins
like
a
cilium
view
and
the
flannel.
It
clearly
flows
by
some
reason:
plugins
we
directed
our
traffic,
so
one
gigabit
network
interface
and
there
is
still
work
to
figure
out
how
to
set
up
them
properly.
In
any
case,
in
the
the
network,
by
default,
even
with
some
efforts
to
set
up
it
properly,
but
we
ready
to
work
is
never
provide
us
to
be
able
to
figure
out
how
we
can
resolve
this
issue.
C
So
in
conclusion
of
a
results,
we
still
see
some
performance
penalty
using
Google
nature's
using
the
division
kubernetes.
Even
the
best
air
is
out
here,
with
a
cube.
Router
produces
about
a
certain
percent
performance
penalty.
If
you
compare
an
identical
set
up
in
bare
metal,
some
network
plugins
have
mentioned,
with
the
require
extra
work,
to
make
them
working
as
as
expected,
and
you
can
see
there's
a
strong
correlation
between
available
Network
bandits
in
database
results.
B
D
B
B
The
so
yeah
to
clarify
all
of
the
traffic
was
directed
to
a
single
node.
So,
typically
in
our
operator,
we
deploy
a
three-member
PXE
cluster
with
three
members
and
a
proxy
SQL
cluster
in
front
of
that
routing
traffic,
and
we
do
that
for
resiliency
because
of
some
deficiencies
that
exist
in
staples
set
for
the
purpose
of
MySQL.
So
this
is
kind
of
a
crutch
we
use
around
it,
but
for
the
purpose
of
the
benchmark,
we
eliminated
proxy
SQL.
B
So
there's
no
proxying
layer
and
we
targeted
all
the
traffic
to
a
particular
instance
of
MySQL
running
in
the
cluster
right.
But
it's
still
replicating
right,
that's
correct,
and
you
did
that
intentionally
because
in
the
in
the
case
of
running
MySQL
for
production
purposes
and
kubernetes,
you
need
synchronous
replication
to
have
some
of
the
guarantees
around
reliability
and
resiliency.
So
it
would
be
a
less
effective
test
in
our
opinion,
for
our
purposes,
if
we
were
to
do
single
incident,
yeah
yeah.
B
We
have
run
similar
benchmarks
in
the
past
with
Aurora,
but
it
wasn't
really
in
the
scope
of
what
we
were
trying
to
discover
here.
So
this
isn't
intended
to
be
comparative
to
other
types
of
database
deployments
in
the
cloud
space.
This
is
really
focused
on
customers
that
are
considering
moving
from
bare
metal
to
urban
Nettie's.
B
We
think,
in
our
opinion,
we
agree
with
what
sugu
said
the
other
day
in
his
keynote
that
to
gain
the
agility
of
kubernetes,
you
need
your
data
and
communities
to
and
part
of
what
we're
doing
at
Percona
is
trying
to
do
this
in
a
really
community
driven
open-source
way,
so
that
everybody
can
get
their
data
into
kubernetes
and
the
production-ready,
but
part
of
that
is
to
address
the
barriers
of
entry
like
performance.
So
we
were
very
specifically
focused
on
bare
metal
to
kubernetes.
B
B
Not
for
this,
so
we
very
specifically
isolated
storage
away.
It
is
true
that
storage
is
also
important
for
database
performance,
but
we
want
to
make
sure
that
we're
making
effective
and
valid
benchmarks
so
for
benchmarking,
networking
we
we
intentionally
set
up
a
workload
in
a
scenario
where
there
would
not
be
any
real
storage
aspect
to
the
performance
that
was
delivered.
E
D
B
F
B
Measured
as
part
of
this
I
mean
I,
perf
does
give
some
results,
but
we
were,
we
were
looking
at
throughput,
primarily
obviously,
latency
has
an
impact
on
throughput
once
you
get
to
database
workloads.
So
it's
part
of
something
we'd
like
to
expand
on
here,
but
the
way
that
we
reported
the
results
right
now
is:
it's
is
its
associated
to
throughput
out
of
iperf
for
pod
to
pod.
G
G
B
So
essentially,
because
the
10
gig
and
one
gig
network
interfaces
on
the
host
were
on
completely
separate
layer,
two
in
environments
and
all
of
the
internal
communication
based
on
the
configuration
of
the
database,
was
going
across
the
interfaces
provided
on
the
10
gig
and
in
most
cases,
the
only
real
case
where
we
had
to
explicitly
do
that
was
with
SRV
because
it
provided
two
interfaces.
All
the
other
network
plugins
basically
provided
d-pod
network
and
the
pod
network
was
on
10
gigs.
B
B
C
Well,
let
me
commander
said
typically
when
I
measure
database
results,
unless
it's
very
specific
environment
I
want
to
see
that
results
very
stable,
that's
very
minimal
or
availability.
The
only
issue
we
had
like
Tyler
mentioned
with
flannel
reacts
net
plugin
that
we
reserved
some
stalls
in
databases
outlet.
It
was
stable
results
and
was
a
drop
to
zero
for
a
couple
of
seconds
and
again
stable,
stable
result,
so
that
only
wish
only
issues
variability.
We
observed
in
this.