►
Description
Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Envoy’s Using 10GB of Memory and It’s All My Fault! - Steve Sloka, VMware
Implementing your own management xDS API server to configure and drive Envoy is a natural step when implementing any Envoy-based infrastructure. Examples exist in various languages that demonstrate how to set up an xDS API server for LDS, EDS, RDS, and CDS. However, most implementers probably wouldn’t realize that it's possible to write an xDS server that appears to work--Envoy serves traffic--but causes Envoy to leak gigabytes of memory.
This talk will discuss how we built an xDS server which ended up with a memory leak in Envoy all because we implemented the API incorrectly. We’ll look at ways we identified the issues through metrics & monitoring, how our users were affected, and also provide an overview of how we caused the issue, then describe some takeaways that should be kept in mind for your implementation. Come learn from our failures so you don’t experience the same!
A
Okay,
good
morning,
everyone
thanks
for
coming
so
today,
I'm
gonna
talk
about
how
we
configured
our
Envoy
to
use
a
whole
bunch
of
memory
and
it
turned
out
to
be
all
of
our
fault,
so
I'm
Steve,
sloka
I
work
at
VMware,
quick
a
little
bit
about
me.
Yeah
I've
got
me
before
I've
been
working
on
kubernetes
for
a
while
now
I
helped
contribute
to
kubernetes
I
work
on
contour
and
gimble
at
VMware.
I've
got
my
certifications
and
I
helped
run
the
the
meetup
in
Pittsburgh.
A
This
was
issued
$4.99,
so
kind
of
early
in
the
contour
days,
and
he
said:
hey
I've
seen
some
memory
issues
while
adding
and
removing
ingress,
and
the
TLDR
of
this
was
that
Envoy
was
using
a
lot
of
memory.
So
as
the
maintainer
of
contour,
you
know,
I
run
it
on
my
machine.
I
run
it
locally,
but
you
don't
always
get
to
see.
You
know
a
real
production
system.
You
know
the
contour
team
is
for
just
an
open-source
project
and
there's
nothing
like
running
a
real
production
system.
A
We
can
run
performance
tests
on
ourselves,
but
those
are
scripted
right.
It's
nothing
like
having
real
use
cases,
real
users
real
real
software
running
through,
so
we
haven't
really
seen
on
by
using
lots
of
memory
before,
but
we
looked
at
some
graphs
and
this
is
what
showed
up
so
on
the
left.
If
you
can't
see,
there's
the
the
y-axis
has
memory
and
it
goes
from
0
to
16
gigs
and
at
the
bottom.
You
can
see
we
have
time
so
it's
8:00
a.m.
to
6
p.m.
A
A
A
This
is
how
it's
architected.
We
obviously
use
envoy
under
the
hood
as
our
other
data
path
component.
So
all
traffic
routes
through
envoy
contour
is
the
ingress
controller
in
kubernetes,
so
its
goal
is
to
go
watch
the
cluster
for
service
is
for
endpoints
for
ingress
object,
secrets,
all
those
sort
of
things.
What
anything
changes
in
that
setup
then
contour
will
then
stream
that
data
down
to
envoy
in
contour
is
the
XDS
server
for
envoy.
So
in
this
in
this
world,
the
G
RPC
connection
is
between
on
valayan
contour.
A
We
give
enough
bootstrap
just
to
go
back,
basically
talk
back
to
contour,
so
this
is
the
general
overview.
This
may
work
also
for
any
kind
of
ingress
controller
you
might
see
out
on
the
market,
so
this
here
is
a
little
more
detailed
view
of
that
of
that
same
implementation.
So
again,
at
the
top
you
can
see
those
are
all
the
things
we
watch
for
in
kubernetes,
so
we
get
services,
secrets
and
points
all
the
sort
of
things
we
put
a
watch
on
the
kubernetes
api
using
client
go
once
something
changes
on
the
top.
A
Once
we
build
that
dag
we
go
ahead
and
walk
that
dag
and
basically
output
on
boy
configuration
again
then,
from
the
information
goes
down
to
our
XD
s,
server
which
contour
is
maintaining,
and
then
we
chop
them
send
the
data
down
to
on
way
under
the
hood,
and
if
you
look
over
to
Cleon
those
columns,
you
can
see
that
we
have
our
correlation
between
services
and
kubernetes
and
CBS
and
envoy
secrets
and
secrets,
and
so
on.
So
you
can
almost
think
of
contour.
As
being
you
know,
a
an
abstraction
layer
between
kubernetes
and
envoy.
A
It's
translating
create
these
objects
into
envoy
objects.
In
a
sense
now,
I
told
you
all
this
just
to
help
you
give
you
background
to
understand
kind
of
how
we
want
to
approach
this
problem.
So
here's
another
picture
of
that
same
memory
graph
and
here
the
memory
is
a
little
higher
we're
using
almost
70
gigs
of
memory,
which
is
kind
of
crazy.
But
what's
interesting
is
that
we
have
this
linear
growth
right.
So
we
have
this
linear
growth
and
it
tapers
off
and
levels
off
at
some
point.
A
What
we
can
see
here
is
that
and
any
good
problem
he'll
go
back
to
hear
any
good
Palamon
computer
science.
You
get
an
issue,
you
always
want
replicated.
First
right,
so
I
want
to
make
sure
it
works
on
my
machine.
So
what
we
found
when
we
looked
at
this
issue
and
how
this
got
built
was
that
there
wasn't
any
traffic
in
the
cluster
at
all,
so
it
was
not
traffic
related.
A
There
was
0
requests
coming
in
and
out
of
the
cluster,
and
we
also
realized
that
this
only
happened
when
we
dealt
with
TLS
connections
right
when
we
did
something
with
plain
HTTP.
We
never
saw
this.
This
experience
happen
and
it
was
also
dealt
with
whenever
we
changed
configuration
right.
So
it's
down
to
the
XDS
server
later.
So
when
we
were
pushing
updates
to
envoy,
we
saw
this
memory
growth
going
up
and
quite
significantly,
if
you're,
using
70
gigs
of
memory
in
your
cluster
for
just
envoy,
that's
a
pretty
big
cost.
A
You
know
in
your
cluster,
so
this
is
our
how
we
reproduced
it.
We
basically
started
out
in
a
cluster
and
had
zero
ingress
resources,
and
then
we
inserted
5,000
total
and
we
did
that
in
blocks
of
a
hundred
after
a
while,
we
saw
that
memory
grow
and
we
let
the
this
cluster
get
stabilized
after
it
got
stabilized
and
they
got
idle.
We
would
work
down
how
much
memory
got
used
and
then
we
would
go
and
restart
the
Envoy
pod
and
we
wanted
to
see
was
on
a
cult
start
with
5,000
objects.
What
was
memories?
A
What
was
on
voice
memory
consumption?
You
know
versus
having
it
getting
configured
as
we
went
so
here's
a
picture
of
that
graph
right
and
hearing
left.
You
can
see
again,
there's
zero
objects,
so
there's
your
memory
getting
used
and
then
we
have
again
that
linear
growth
up
and
hitting
this
in
this
case
it's
30
gigs.
But
that
number
doesn't
matter
as
much
and
then
you
know
everything
stabled
off
and
then
we
restarted
it
and
it
dropped
down.
A
So,
looking
at
the
problem,
what
we
figured
out
was
that
when
we
changed
secrets
the
key
there
was
the
TLS
connections,
so
we
modified
secrets
and
kubernetes
or
SDS.
Basically,
we
caused
and
updates
who
are
LDS
or
listeners
and
envoy
at
a
high
rate
of
change.
We
basically
caused
a
lot
of
listeners
to
get
created
and
ended
up
with
lots
of
old
configurations
sitting
around.
A
So
when
you
create
it
lots
of
different
listeners,
an
envoy
you
end
up
with
a
draining
mode
and
all
of
your
old
listeners
sit
around
until
they
get
drained
out
which
by
default
is
600
seconds.
So
if
I
create
5,000
objects,
I
ended
up
with
5,000
different
listeners,
running
all
consuming
memory
and
all
holding
on
to
it
for
600
seconds
again.
Each
listening
trainer
holds
on
to
memory.
So
this
is
kind
of
interesting
right
to
figure
this
out
again
on
a
small
scale.
This
works,
fine
traffic's,
still
routed
just
fine.
A
It
just
used
up
a
ton
of
memory
I'm.
So
back
to
this
picture
right
before
we
saw
that
linear
growth
well
in
this
picture.
Here
it's
going
up
and
down
because
once
we
realize
what
was
happening.
Our
first
thing
we
did
was
so:
let's
make
that
that
draining
time
smaller,
let's
go
from
6
600
600
seconds
to
maybe
a
minute
and
that's
what
our
team
did,
and
this
is
why
you
see
that
up
and
down
drop,
so
we
consume
a
whole
bunch
of
memory
and
listen
or
timeout
with
hit
it
would
drop.
A
So
the
current
solution,
how
we're
looking
at
solving
this
today
I
put
current,
because
we
haven't
officially
solved
this
100%.
Yes,
I
think
we
still
have
some
ways
to
go,
but
the
first
thing
we
did
to
help
solve
this
listener
problem.
Was
this
sort
our
secrets
before
we
sent
them
off
to
envoy
so
before
we
would
just
get
secrets.
A
Don't
ignore
your
axe
and
axe,
so
we've
been
using
xes
since
probably
since
it
started,
and
we
just
never
implemented
the
axe
and
axe.
So
it's
a
good
idea
to
do
that.
We
went
and
did
that
and
that
helped
us
out
as
well
I'm
also
implementing
new
discovery
requests
so
again
in
your
XDS
server.
Waiting
for
that
discovery.
Request
to
come
through
before
you
respond
was
a
big
help
for
that
and
there's
something
we've
had
for
a
while,
but
batching
up
at
requests.
A
So
one
thing
we
saw
with
with
contour
was
that
kubernetes
will
send
you
lots
of
events
really
quickly.
What
we're
doing
is
we
were
rebuilding
that
that
graph
in
memory
over
and
over
and
over
and
flooding
on
away
with
updates,
so
we
added
a
timeout
window
so
a
way
that
we
can
get
a
whole
bunch
of
requests
that
ones
batch
them
and
then
just
process
them
at
a
smaller
interval.
So
I
think
having
that
that
batched
request
go
through
was
very
helpful
and
beneficial.
A
So
looking
back
at
this,
you
know
this
is
an
interesting
problem
that
again
we
had
to
find
in
production
find
it
was
somebody's
use
case
it's
great
to
have
users
out
in
the
world
to
help
us
out
with
these
things
so
interested
to
see
what
all
you
folks
are
doing
with
your
with
your
clusters.
So
thanks
free
time,
if
you
have
any
time
tomorrow,
I
have
a
talk
again
with
envoy
talking
about
doing
lots
of
crazy
messy
things
without
meshes
so
hope
to
see
you
there
tomorrow.
So
thanks
for
I'm
appreciated.