►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Livin' on the Edge (of your cluster) - Nick Young, VMWare
Envoy is often used as a sidecar proxy, but what if you’re using Envoy as an edge proxy to bring traffic into your cluster? What can bite you?
This talk is based on my experience building Contour. We’ll cover the key considerations for deploying Envoy as an edge proxy, including the settings you need to check. And we’ll inevitably get into war stories on the importance of keepalives, buffer size tuning, and draining connections correctly.
A
Okay,
hi
everyone.
My
name
is
Nick
young
I
work
on
the
contour
team.
At
VMware.
We
make
an
ingress
controller
that
uses
envoy.
I've
got
five
minutes
only
so
I'm
gonna
absolutely
smash
through
this.
Please
come
and
see
me
afterwards.
If
you
have
a
questions.
Okay,
the
name
of
the
talk
is
living
on
the
edge
of
your
cluster,
because
that's
what
contour
is
all
about
is
getting
traffic
from
outside
your
cluster
to
inside.
A
So
what
he's
gone
to
our?
It
is
an
ingress
controller.
Like
I
said:
there's
the
website
at
Trent,
the
the
big
thing
that
it
does
is.
It
translates
between
kubernetes
objects
and
XTS.
It's
basically
a
translation
layer
for
either
ingress,
or
we
have
our
own
Ciardi
called
HTTP
proxy
that
you
can.
That
lets
you
configure
envoy
effectively
using
kubernetes
objects,
that's
and
then,
because
of
this,
it
effectively
works
at
the
edge
of
the
cluster
to
take
traffic
from
outside
your
cluster
and
route
it
inside
your
cluster
using
em
boy.
A
Now
this
is
pretty
different
from
the
the
mesh
case
in
that
the
envoys
that
heaps
longer
lived
in
any
individual
service,
and
so
are
the
XDS
connections
and
the
other
one
is
that
a
larger
number
of
connections
is
shared
like
across
a
single
envoy
to
multiple
services,
and
so
because
of
that
you
know
some
of
the
defaults
that
are
good
for
the
mesh
case
are
not
so
good
for
us.
So
these
are
the
things
that
we
learned
like
I
said.
Some
settings
need
tuning
for
edge
proxy
use.
A
Being
careful
about
connection
draining
is
far
more
important
when
you've
got
long-lived
clot
proxies
that
you
know
that
are
processing
a
lot
of
different
connections
and
yet
leaving
your
TCP
sessions
running
for
a
long
time
can
have
some
big
caveats
that
can
be
really
weird.
Okay,
so
yeah
envoy
he's
got
lots
of
buffers.
A
Yes,
there
are
a
few
80s
hair
metal
references
in
there,
please
spot
them
and
enjoy
so
yeah.
We
started
from.
There
was
a
pull
request
from
a
number
a
thousand
and
one.
Thank
you
very
much.
That
was
a
really
great
starting
point
for
us
to
sort
of
see
some
of
the
things
that
we
needed
to
think
about.
A
You
can
see
them
all
there.
It
did
take
me
a
while
to
sort
of
parse
which
time
I
were
which,
in
a
slightly
different
context
to
the
to
the
service
mesh
context,
but
I
drew
a
diagram
for
myself
and
it
was
very
helpful
and
so
yeah
I'm,
pretty
sure.
We've
come
close
to
reasonable
defaults
and
we've
sort
of
exposed
little
knobs
for
people
to
mess
with
their
contour
configuration
to
be
able
to
italo's
if
they
need
to
the
HMP
protocol,
options
were
important.
A
What
else
we
got
yeah
an
important
one
is
for
our
use
case.
We
can't
use
the
overload
manager
because
there's
not
a
real,
sensible
value
for
the
max-heap
size
and
so
yeah
we're
still
sort
of
figuring
out
how
we
might
do
something
about
that.
Okay,
so
yeah
we
built
off
this
horrifying
bash
command
to
run
in
a
pre,
stop
book
in
our
Envoy
pods
that,
basically,
just
you
know,
post
to
the
health
check
thing
as
a
priest
or
book
and
just
kind
of
hope.
Maybe
that'll
be
enough
to
drain
connections
in
time.
A
It's
not
great,
and
we
are
looking
to
do
something
a
bit
more
advanced
like
this,
where
we
hit
the
endpoint
and
then
block
the
pre
stop
hook.
While
we
wait
for
everything
to
actually
drain
by
checking
the
stats
and
then
shut
down
the
priests
topic
and
let
the
Envoy
terminate
that
should
give
you
people
a
possibility
to
do
a
proper,
rolling
envoy
upgrade
when
you're
doing
this
again.
This
is
this
only
comes
up
because
these
envoys,
a
very
long,
live
compared
to
the
services
behind
them.
A
Yeah,
this
one
was
really
weird.
We
really
did
have
a
failure
to
communicate.
We
had
real
weird
problem
reports
where
envoy
would
just
stop
receiving
updates
from
contour
and
we're
like
pulling
our
hair
up
and
like
what
is
going
on.
We
just
couldn't
find
it
find
the
cause
of
it
and
what
it
turned
out
was
that
they
were
using
say
an
eye
plug-in
that
would
drop
long-running
idle
idle
TCP
connections
in
the
case
of
the
Envoy
to
contour
er
PC
connection.
A
If
there
are
no
config
changes
happening,
then
that
is
an
idle
TCP
connection,
and
so
sometimes
I
see
and
I
would
just
be
like
hey
I,
don't
need
that
anymore
off
you
go
and
what
then
happens
is
the
connection
gets
stuck
in
a
half
closed
State
on
the
Envoy
side,
and
so
envoy
can
never
connect
back
to
the
XDS
to
be
able
to
get
updates
to
fix
itself,
and
so
would
be
stuck
in
a
situation
where
you
had
to
restart
it
turned
out
really
simple
problem.
Add
TCP
keep
lives
then
it's
no
longer
idle.
A
This
problem
just
goes
away,
but
a
really
good
example
of
how,
when
you're,
when
you're
envoys
are
not
coterminous
with
the
services
behind
them,
you
can
get
problems
that
you
just
don't
see
in
the
service
mesh
case.
So
yeah
things
to
remember
our
thirty
seconds
left
awesome
envoy
settings
need
tuning
for
edge
proxy
use.
Please
come
feel
free
to
come
and
see
me.
A
You've
got
to
be
really
careful
that
you
don't
screw
people
by
dropping
their
connections
out
from
under
them,
just
because
you're
rotating
something
on
the
back
end
and
yet,
lastly,
long-running
TCP
isms
have
caveats,
use
keeper
lives,
we
actually
added
GRP
CK
pillars
from
the
contour
side
as
well
to
make
sure
that
we
got
both
sides
covered,
so
you
know
there's
no
way
that
that
situation
can
come
up
anymore.
Okay,
that's
the
content
of
the
talk.
I
am
at
zero
seconds.
Here
is
more
information.
A
Please
come
and
find
me
today
or
anywhere
else
in
the
conference
and
head
to
project.
Contour
IO
find
us
on
slack
all
that
usual
sort
of
stuff
and
also
we're
having
a
community
lunch
tomorrow.
So
if
you
want
to
come
and
talk
to
us
about
any
weird
stuff,
we
were
done
with
envoy,
please
feel
free
will
be
it
over
in
the
over.
In
Q
Khan
thanks
very
much.