►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building an Edge Control Plane with Kubernetes and Envoy - Flynn, Datawire
The Envoy proxy is fast becoming ubiquitous as the universal data plane API for cloud-native networking and communications. However, the power of Envoy comes at the cost of configuration complexity. In this talk, I’ll discuss what we learned from designing and implementing the Ambassador edge control plane for Envoy, built around the Kubernetes API and Envoy’s v2 configuration. I’ll talk about the evolution of Ambassador from a simple Envoy configuration engine built around Jinja2 templates and variable substitution to the more sophisticated, multi-pass, compiler-type architecture that is in use today. I’ll also discuss how engineers today are using Ambassador, the community that has developed around this project, and where we see the requirements and technology evolving.
https://sched.co/MPfA
A
All
right,
hi
everyone,
my
name,
is
Turin
I
work
for
Stara
I'm,
introducing
the
talk.
So
today
we
have
Flynn
and
he's
going
to
talk
to
us
about
building
an
edge
control
plane
for
Envoy
Flynn
comes
from
data
wire.
Where
he's
a
principal
software
engineer
and
I'm
super
excited
about
this
talk.
So
with
that
anyway,
to
Flynn
and
let's
see
what
he
has
to
say,
I
am.
B
Lead
the
ambassador,
open
source
project
at
data
wire
ambassador
is
a
a
edge
proxy
built
to
be
kubernetes
a
native
this
built
on
top
of
envoy.
We
basically
let
envoy
wrangle
all
of
the
business
of
moving
your
data
around
and
we
let
ambassador
worry
about
wrangling
envoy
in
the
two
years,
since
our
first
publicly
release,
we've
actually
picked
up
thousands
of
active
installations
added
nearly
7,000
commits
from
70
contributors,
some
of
whom
are
in
this
room.
Thank
you
very
much.
B
It's
important
to
recognize
that
we
tend
to
understand
things
by
looking
backwards,
even
though
we
have
to
look
forward
while
we
do
them
so
I'm
gonna
try
to
point
out
places
where
what
we
saw
looking
forward
ended
up,
not
being
what
we
see
when
we
look
backward,
but
as
we
look
backward,
we
can
identify
five,
pretty
distinct
phases
in
ambassador's
life.
The
first
one
was
the
experiment.
We
didn't
really
know
what
we
were
doing.
B
We
saw
an
opportunity
that
we
wanted
to
try
to
accomplish
something
with
the
second
one
was
when
our
experiment
started
to
succeed
a
little
bit,
and
we
had
to
actually
turn
it
into
a
real
product
that
people
could
rely
on
once
we
got
it
to
that
point,
then
we
needed
to
go
through
and
pick
up
a
bunch
of
features
so
that
we
could
start
attracting
more
users
so
that
we
could
continue
having
a
product.
We
deliberately
did
that
in
a
way
the
was
expensive
later
on
and
led
us
into
the
phase.
B
We
called
the
grand
refactor
where
we
had
to
go
through
and
do
an
enormous
amount
of
work
to
try
to
save
ourselves
from
what
we'd
written
ourselves
into
beforehand.
After
that.
That
leads
us
into
our
present
balancing
act
where
we're
trying
to
do
the
usual
of
balancing
features
against
technical
debt
against
time
to
market
against
everything
else,
so
that
we
don't
end
up
back
in
the
situation
of
having
to
do
another
grand
refactor.
So
we'll
start
off
with
the
experiment.
B
This
is
at
the
very
beginning,
we're
actually
going
to
spend
a
fair
amount
of
time
talking
about
the
experiment,
because
we
learned
a
bunch
of
really
really
critical
things
during
this
phase.
We
also
learned
them
by
doing
a
lot
of
things
that
technically
turned
out
to
be
not
what
we
really
wanted
to
do
in
the
long
term.
But
there
was
no
way
to
know
about
that
ahead
of
time.
We
had
to
actually
do
them
and
get
feedback
from
our
users
and
iterate
and
come
up
with
something
that
was
workable.
B
That
was
our
initial
basis
for
where
to
start
from,
and
we
formed
a
couple
of
early
hypotheses
about
how
we
might
be
able
to
use
envoy
to
make
it
better
and
what
sorts
of
things
people
would
think
about
this,
a
big
one
was
that
we
figured
developers
didn't
really
want
to
think
in
the
same
terms
that
you
can
figure
on
volume
on
boys
a
pretty
low-level
tool,
it's
powerful!
It's
flexible!
It's
hard
to
use.
We
thought
we
could
do
better
than
that
by
mid-july.
B
B
A
B
My
database
pod
crashed.
What
do
we
do
now?
The
turning
point
of
that
was
realizing.
That
kubernetes
has
already
solved
that
problem,
that
if
we
did
everything
with
kubernetes
resources
or
things
stored
in
kubernetes,
then
we
could
let
that
be
kubernetes
problem
and
get
on
with
the
rest
of
our
lives.
So
we
shifted
away
from
the
REST
API
and
databases
into
storing
the
design
and
config
maps,
and
then
we
figured
out
how
much
of
a
pain
it
is
to
Patrick
and
fig
map.
B
So
we
shifted
over
to
annotations
and
now
to
see
RDS
or
annotations
and
CRTs
both
that
bought
us
a
lot
of
flexibility
without
making
us
reinvent
a
lot
of
low-level
things.
Another
good
one
is
at
the
point
where
Jane
has
a
service
running
that
on
her
laptop
and
she's
ready
to
test
it
with
real
traffic.
B
B
What
we
ended
up
doing
again
was
deciding
that
we
would
just
let
kubernetes
wrangle
the
hard
stuff
for
us.
Don't
worry
about
an
English
controller,
just
deploy
it
as
a
service
or
deploy
it
as
a
daemon
set.
And
then,
although
we
were
initially
concerned
about
this,
it
turns
out
to
enable
some
really
cool
things
where
you
can
use
a
layer
for
load
balancer
in
front
of
Ambassador
to
make
it
really
easy
to
horizontally
scale
ambassador.
B
Also,
since
ambassador
is
an
edge
proxy,
it
lives
on
the
edge,
and
that
means
we
get
to
worry
about
people
who
want
to
do
things
like
offloading
TLS
termination.
That's
actually
a
particularly
nasty
example
of
a
real-world
area.
We're
living
at
the
edge
makes
your
life
really
complicated.
If
you
have
a
load
balancer
ahead
of
you
terminating
TLS,
ambassador
might
still
need
to
make
policy
and
routing
decisions
based
on
where
the
client
was
originally
whether
they
originally
spoke.
B
Http
or
HTTPS,
so
you
have
to
have
a
way
of
getting
all
that
information
from
the
TLS
termination
points
into
ambassador,
and
we
have
a
lot
of
tools
for
this.
We
have
the
proxy
protocol
and
all
the
exported
stuff
and
envoy
knows
how
to
do
these
things,
but
it
turns
out
that
configuring
everything
to
play
nicely
in
the
real
world
can
get
really
very
complicated
very
very
quickly.
B
So
one
of
the
things
that
we
learned
fairly
quickly
through
all
this
was
that
being
in
this
experimental
phase,
where
we
didn't
really
have
any
users
relying
on
us
yet,
but
we
did
have
users
who
were
willing
to
try
things
and
talk
to
us
turned
out
to
be
an
incredible
opportunity.
The
things
that
we
learned
through
that
are
still
the
core
of
the
product
that
we
builds
today,
probably
always
will
be
so.
B
At
one
point,
we
started
to
realize
that
this
experiment
was
actually
succeeding,
so
we
needed
to
run
through
quickly
and
try
to
turn
it
into
an
actual
product
so
that
we
can
answer
questions
like
oh
wait.
What
version
of
envoy
is
in
this
and
what
version
is
coming
next?
What
are
you
planning
to
do
in
the
future,
not
widely
advertised
before
ambassador
0.10.0
deuced,
a
given
ambassador
bill
from
source?
Don't
tell
anybody,
that's
a
secret,
but
we
basically
had
to
go
through
and
retrofit
release
engineering
onto
our
experiment.
B
You
tend
not
to
do
a
lot
of
release,
engineering
and
experiments.
It's
not
worth
the
pain
upfront,
but
you
really
really
need
it
for
products
so
having
some
idea
of
how
you're
going
to
do
it.
When
you
realize
that
you're
succeeding
can
be
important
and
it
was
particularly
interesting
because
you
don't
usually
get
to
predict
that
point
of
becoming
a
product
instead
of
being
an
experiment.
You
can
usually
only
see
that
in
hindsight,
so
there
was
a
lot
of
things
that
we
needed
to
do
very
quickly
there.
B
This
was
also
the
point
that
we
sort
of
got
into
the
first
recognizably
modern
ambassador,
using
annotations
instead
of
using
config
maps
and
having
the
Diagnostics
UI,
and
things
like
that,
so
yeah
again
being
able
to
see
the
experiment
turning
into
a
product,
maybe
something
you
only
see
in
hindsight.
So
that
may
be
something
that
you
need
to
be
prepared
to
react
quickly
to
handle
as
opposed
to
getting
to
planet
and
schedule
it.
B
So
at
that
point
we
had
a
product.
We
were
pretty
confident
that
we
could
reproduce
the
builds.
We
thought
we
had
the
release
engineering
under
control.
That
was
the
point
that
we
decided.
We
wanted
all
the
users,
and
so
we
decided
to
go
through
and
add
a
bunch
of
features
to
make
Ambassador
more
compelling,
and
we
very
deliberately
chose
to
do
this
by
incurring
technical
debt.
B
We
were
financing
development
by
with
technical
debt
in
order
to
get
things
to
market
more
quickly,
and
we
did
that
because
we
realized
this
was
an
area
where
we
got
to
see
something
useful
going
forward.
We
realized
that
if
we
took
the
time
at
that
point
to
fix
all
the
technical
debt
stuff
that
was
already
visible
coming
down
the
road,
we
would
not
have
a
product,
it
would
not
survive.
We
would
never
get
to
point
where
we
were
able
to
fix
these
things,
so
we
did
features
first.
B
B
That
meant
we
had
to
go
and
put
a
lot
of
effort
into
figuring
out
how
we
could
support
external
contributors,
because
it
was
also
very
evident
by
this
point
that
there
was
a
lot
more
work
that
we
could
do
with
just
a
small
team
of
engineers.
That's
part
of
the
reason
you
do
open
source
to
have
the
community
to
let
the
community
contribute.
It
requires
some
effort
to
make
that
work
out.
Well,
we
also
added
a
ton
of
features.
B
There's
I
I
went
through
the
changelog
and
I
couldn't
fit
them
all
on
the
slide,
which
was
kind
of
cool.
We
also
another
area
of
growth
was
the
number
of
people
who
was
actually
were
actually
running.
Ambassador
in
production
really
started
skyrocketing
through
the
summer
of
2018,
as
we
were
doing
this,
and
that
meant
that
we
were
starting
to
get
more
questions
about
performance
and
reliability,
and
things
like
that,
so
there's
a
whole
other
area
of
work
going
in
there
and
driving
us
into
thinking.
How
do
we
sustain
this
stuff
long
term?
B
So
that
brings
us
into
an
area
that
was
kind
of
less
fun
where,
as
we
were
going
through
this
growth
we're
also
starting
to
hit
some
pretty
significant
limits
in
the
architecture
of
the
product
ambassador
at
this
point
was
primarily
a
template
engine
doing
text
processing
internally,
it
had
never
been
architected
for
really
rapid
growth.
We
knew
this
when
we
started
the
features
phase.
B
B
It
was
equally
obvious
that
we
had
to
go
to
envoy
b2.
We
needed
to
switch
to
envoys
ATS
so
that
we
could
stop
having
to
do
hot
restarts
and
just
do
live
updates
immediately.
I
already
mentioned
the
bit
with
small
changes
should
need
small
changes.
We
also
needed
tests
that
could
run
more
quickly,
but
also
could
be
written
more
quickly,
especially
for
external
people.
We
were
able
to
do
these
things,
so
we
shipped
ambassador
zero
about
50
to
zero.
B
That
was
the
result
of
the
ground
refactor
we
shipped
it
in
January,
2019,
five
months
after
we
started,
we
expected
it
to
take
about
three
months.
There
are
a
lot
of
moving
parts
in
there.
The
code
actually
wasn't
the
really
hard
bit
the
really
hard
bit
was
trying
to
make
sure
that
we
didn't
actually
change
behavior
between
ambassadors.
B
They
were
about
forty
to
two
and
zero
to
50
does
zero
and
we
almost
succeeded
in
that,
but
only
because
a
bunch
of
people
in
the
community
helped
us
out
by
running
early
access
releases
of
0.50
and
then
providing
feedback
for
places
that
didn't
work.
There's
no
possible
way.
We
could
have
done
it
without
that.
We
still
have
a
lot
of
pain
around
testing.
B
If
there
were
any
wood
around
here,
I
would
knock
on
it.
V2
plus
the
ATS
is
definitely
nicer
to
work
with.
We've
been
hearing
from
people
that
the
refactor
code
based
is
easier
to
contribute
to
we've
been
seeing
more
external
contributors,
which
is
wonderful
and
also
there's
a
bunch
of
features
in
there
like
sni
and
TCP
mappings.
Those
things
up
there
would
have
been
absolutely
impossible
to
do
in
the
old
code
base.
We
had
to
have
the
new
code
base
to
have
any
chance
of
doing
them
at
all.
B
B
There's
a
lot
of
stuff
going
in
with
release
engineering
and
being
able
to
more
closely
track
envoy
releases.
Things
like
that
and
a
lot
of
it
is
still
you
know,
balancing
features
paying
down
technical
debt,
avoiding
getting
into
more
trouble
the
biggest
things
that
we've
learned
through
all
of
this
stuff.
We've
been
learning
a
lot
about
the
complexity
at
the
edge
versus
service
to
service.
B
We've
learned
that
the
attention
we've
paid
to
trying
to
help
out
the
inner
loop
of
development
within
data
wire
has
helped
somewhat
and
is
seems
to
help
translate
for
external
contributors
as
well,
but
easily
the
biggest
thing
that
we've
learned
through
all
this
one
is
that
being
able
to
test
things
by
talking
to
users
and
listening
to
them
is
a
huge
win
and
very
very
important
crystal
ball
time.
What's
coming
next,
we
know
that
having
a
low
barrier
to
adoption,
making
it
easier
to
try
out
ambassador
is
going
to
stay
critical.
B
We
know
that
there
are
things
about
performance
that
are
going
to
be
important.
There's
a
lot
of
stuff
around
debug
ability
and
transparency
that
I
am
looking
really
forward
into
getting
into
there's.
Also,
a
bunch
of
things
that
are
still
dealing
with
on
void
directly
we've
had
people
asking
for,
but
listeners
which
envoy
can
do
but
ambassador
only
kind
of
does
it
when
you're
talking
about
redirecting
30
X
HTTP.
B
That
sort
of
thing
I
would
like
to
make
G
RPC
a
first
class
thing
so
that
you
don't
have
to
talk
about
mappings
with
random
strange
looking
prefixes
when
you
do
g
RPC,
a
bunch
of
other
care
and
feeding
stuff
with
envoy
and
making
sure
we're
ready
for
envoy
version
3,
which
will
be
coming
at
some
point
and
who
knows
yet?
Maybe
there
will
be
a
next-gen
ingress
that
we
can
worry
about,
but
that
is
it
for
here.
What
kind
of
questions
do
you
guys
have.
C
Alright,
thanks
for
sharing
all
of
that,
I
just
well
you're
talking
a
lot
of
the
complexity
and
difficulties
people
have
with
at
the
edge,
and
particularly
with
just
things
like
Native
ingress
at
end
boy.
We
just
wanted
to
understand
a
little
bit
more.
What
do
you
think's,
causing
that
kind
of
difficulty
in
complexity
like
where
is
most
of
that
coming
from?
Is
there
there
one
or
two
key
points
or
I.
B
We
could
talk
a
lot
about
horizontal
scaling
and
whatever
failover
and
all
that
kind
of
stuff.
But
the
fact
of
the
matter
is
that
doing
those
things
that
layer
4
with
only
clear
text
is
pretty
straightforward
and
as
soon
as
you
throw
TLS
into
the
mix,
your
life
gets
much
much
more
miserable.
Most
of
the
you
know,
things
like
the
proxy
protocol
and
the
X
forwarded
headers
and
all
of
that
stuff.
B
When
you
get
down
to
it,
boiled
down
to
trying
to
have
a
way
to
reintroduce
the
original
envelope,
as
it
were,
the
metadata
about
the
original
connection
to
be
able
to
carry
that
all
the
way
through
HTTP
never
really
had
a
great
way
to
do
that,
and
so
we've
been
retrofitting
stuff
along
as
we
go.
But
to
me
that's
where
most
of
it
lives
great.