►
From YouTube: Building a Dev/Test Loop for a Kubernetes Edge Gateway with Envoy Proxy - Flynn, Datawire
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Building a Dev/Test Loop for a Kubernetes Edge Gateway with Envoy Proxy - Flynn, Datawire
As we worked with the community to build the open source Ambassador API gateway on top of Envoy Proxy we learned a bunch of lessons about our dev/test loop. One of the more unpleasant realities that we’ve had to come to terms with is that writing code is easy. What's hard is making sure it's working, and making sure that it keeps working as changes are made. Over the life of Ambassador we've gone through multiple cycles of adding tests to increase confidence, from simple unit tests to larger integration suites, such as our Kubernetes Acceptance Test (KAT) framework. Several times these tests have become too slow, and then we had to work to speed them up so our velocity doesn't suffer. Join Flynn to learn what we would do again in regard to our dev/test loop if we chose to build another open source tool, and also (more critically), what we would change.
https://sched.co/UaWc
A
C
What
I'd,
really
like
you
to
take
away
from
this
talk,
is
that
over
the
past
couple
of
years,
two-and-a-half
years
or
so
that
ambassador's
been
around
data
wire
has
done
a
lot
of
suffering
when
it
comes
to
testing,
it
would
be
really
nice
if
you
guys,
could
learn
from
that,
so
that
you
don't
have
to
do
it.
For
those
who
are
not
familiar
ambassador
is
an
API
gateway.
Its
purpose
in
life
is
to
sit
between
the
internet
in
your
kubernetes
cluster
and
mediates.
Access
between
the
Internet
and
the
service
is
running
in
your
native
cluster.
C
It
is
open
source.
It
is
self-service.
It's
built
for
kubernetes
on
top
of
the
Envoy
proxy
and
unsurprisingly,
if
we
take
a
look
downstairs
under
the
hood,
it's
a
little
bit
more
complicated
than
that
first
picture.
Well,
we
actually
have
is
envoy
and
ambassador
running
together
in
a
kubernetes
pod
ambassadors
configuration
lives
in
kubernetes,
it
reads
it
from
kubernetes
resources.
C
It
generates
the
envoy
config,
corresponding
to
what
it
sees
in
his
resources,
and
then
envoy
is
the
one
that
does
all
the
heavy
lifting
of
moving
data
around
for
you,
the
resources
that
ambassador
reads
are
fairly
simple.
They
look
kind
of
like
that.
Ambassador
then,
will
take
that
turn
it
into
an
intermediate
representation,
which
is
considerably
less
simple
and
then
from
the
intermediate
representation,
we'll
compile
the
Envoy
configuration,
which
is
also
quite
a
bit
less
simple
than
we
start
off
with
so.
C
This
leaves
us
with
at
least
three
distinct
places
that
we
need
to
tester
all
this
one
of
them
is.
We
need
to
test
fit
ambassadors,
seeing
the
correct
configuration
one
of
them
is
we
need
to
make
sure
that
it's
generating
the
correct
envoy
configuration
the
last
one
is
that
we
need
to
make
sure
that
envoy
is
actually
doing
with
that
configuration.
What
we
think
it's
supposed
to
do
with
that
configuration.
The
combination
of
those
three
things
makes
ambassador
relatively
complex
to
test.
So
with
that
in
mind,
let's
back
up
and
take
a
look
at
development.
C
First,
if
you
want
to
do
something
different
in
this
world,
you're
gonna
start
with
your
typical
developer.
Loop
you're
gonna
have
a
change
that
you
want
to
make
you'll
start
making
it
you'll
do
something
to
give
you
some
confidence
that
you
are
on
the
right
track
with
your
change
and
you'll
use
the
results
of
that
something
as
feedback
to
refine
your
change
and
continue
working
with
it,
and
you
go
through
this
loop
a
few
times.
C
This
is
pretty
much
where
we
started
when
we
first
started
writing
tests
for
ambassador
at
about
the
point
where
we
might.
Oh
crap
people
are
actually
using
this
thing
only.
We
started
not
just
with
one
set
of
tests,
but
with
two
we
had
a
set
of
compiler
tests
and
we
had
a
separate
set
of
end-to-end
tests.
A
compiler
test
in
this
context.
Oh
sorry,
and
both
of
them
had
to
pass
before
we
could
release
anything
when
I
say
a
compiler
test.
C
A
C
We
actually
ended
up
realizing
that
we
had
not
three
loops
but
five
and
that
first
one
we
were
able
to
get
down
to
about
30
seconds-1
passed
through
the
loop,
the
compiler
tests,
maybe
10
minutes,
and
to
end
tests
half
an
hour
or
longer,
if
you
actually
needed
to
work
on
any
other
tests.
That
was
worse
than
just
running
them.
If
you
had
all
those
things
together,
you
end
up
with
a
total
loop
time
in
hours
which,
as
we
all
know,
is
just
shy
of
forever.
C
When
a
basilar
0.50
came
along,
we
ended
up
actually
getting
rid
of
the
compiler
tests
entirely
and
shifting
to
this
world,
where
we
put
a
lot
of
effort
and
the
end-to-end
tests
to
try
to
make
them
run
more
quickly,
and
we
ended
up
with
this
thing
called
cat.
The
kubernetes
acceptance
test
suite,
and
this
was
better
in
some
ways.
We
could
still
do
the
first
chunk
of
development
work
second
loop
time
and
we
started
off
with
about
15
minutes
sensitive.
C
You
know
half
an
hour,
so
that
was
good,
but
cat
was
way
more
complex
than
the
original
and
and
sweet
was
and
so
anytime.
We
wanted
to
do
something
with
cat
that
by
itself
would
take
just
shy
forever
and
we
found
out
also
that
there
was
this
nasty
situation
where,
as
we
added
more
features
to
ambassador
and
we
added
more
ants
and
tests,
things
got
slower
running
through
that
loop.
C
Until
eventually,
we
got
to
a
point
where
they
really
didn't
run
reliably
at
all,
and
that
put
us
in
a
very
unpleasant
situation
where
we
could
make
changes,
but
we
couldn't
actually
test
them,
which
meant
we
couldn't
release
them,
and
since
there
are
only
two
kinds
of
code
in
the
world
there's
code
that
ships
and
code
is
pointless.
If
you
can't
release
you
got
nothing,
so
that
was
a
really
good
opportunity
to
step
back
and
think
hard
about
what
we
actually
wanted
to
do.
C
What
we
really
wanted
was
just
to
make
a
change
and
release
the
change,
or
perhaps
slightly
more
honestly,
we
wanted
to
make
good
changes
and
release
good
changes,
and
when
we
sat
back
and
really
talk
about
that,
we
realized
something
that
we
hadn't
thought
much
about
before,
which
is
that
a
change
is
not
just
code.
A
change
is
code,
plus
the
confidence
that
your
code
is
correct
and
that
it
will
continue
to
be
correct
over
time.
That
was
the
thing
that
I
started
cataloging
as
the
first
of
the
epiphanies
through
here.
C
If
you
look
at
it
that
way,
this
loop,
you
make
your
change
as
you
go
through
and
you
do
something
to
gain
confidence.
You
build
up
your
confidence,
you
keep
doing
that
and
eventually
you
get
to
a
point
where
you're
happy-
and
you
feel
good
about
this,
and
that
led
us
to
another
interesting
thing,
as
we
thought
further,
which
was
the
problem
of
running
off
to
run
the
tests
wasn't
so
much
that
you
had
to
run
tests.
C
C
If
you
start
thinking
harder
about
how
to
avoid
shifting
loops,
how
to
do
it
all
as
one
loop,
it
becomes
clear
that
that
thing
you
do
to
gain
confidence
is
actually
a
test,
and
you
are
in
fact
testing,
while
you're
going
through
this
loop.
I
personally
felt,
like
I,
should
have
realized
that
a
little
bit
earlier
than
I
did
now,
but.
C
C
C
C
C
That
remember
that
I
said
that
we
got
that
first
loop
down
to
30
seconds.
That's
that's
a
little
bit
of
a
lie.
I
could
get
it
down
to
30
seconds,
but
I
mostly
did
that
by
knowing
enough
about
the
internals.
That
I
was
able
to
cheat
a
lot
of
other
people
that
turned
out
just
kind
of
ignored
that
part
of
the
world
and
used
the
compiler
tests
as
their
inner
development
loop.
That
was
not
30
seconds
that
was
10
minutes
worse.
C
A
C
Rid
of
it
when
we
went
to
0.50
so
then
this
people
had
to
use
the
whole
loop
running
cat
tests,
which
again
is
one
of
the
ones.
That's
just
show
you
forever
part
of
the
reason.
That's
just
shy
of
forever
is
that
you
have
to
spin
up
a
kubernetes
cluster
core
running,
and
what
we
found
actually
was
that
a
lot
of
people
found
that
problematic.
So
instead
they
would
just
submit
PRS
and
let
our
CI
rather
their
loop
for
them,
which
added
even
more
time
and
made
everything
even
more
painful.
C
It's
worth
asking
why
we
ditched
the
compiler
chests
as
you're
about
50.
We
fondly
called
the
grande
refactor.
It
was
actually
a
ground-up
rewrite.
We
had
to
refactor
a
bunch
of
code
to
make
it
easier
to
work
with,
but
we
also
had
to
switch
from
on
voice
version.
One
configuration
to
on
voice
version,
two
configuration
and
the
v1
config
in
the
v2
config
are
very,
very
different.
C
We
had
to
go
through
and
laborious
ly,
bootstrap
everything
using
our
end-to-end
tests
with
version
1
then
change
everything
to
a
version.
Two
config
then
run
the
same.
End-To-End
tests
then
figure
out.
If
it
passed
then
figure
out
how
to
tweak
the
version
to
configure
when
it
didn't,
and
eventually
we
could
get
to
a
place
where
we
could
start
saving
some
configurations
and
to
think
about
running
compiler
tests.
Again
that
didn't
happen
for
a
while.
C
So
we
really
didn't
get
rid
of
the
into
the
compiler
test
just
to
mess
people
up
there.
We
didn't
have
a
choice,
but
that
does
leave
us
in
this
place
where
we
know
a
lot
of
people
are
running
this
and
we
want
to
have
a
single
loop
anyway,
and
we
needed
to
figure
out
how
to
arrange
it
so
that
first
this
could
actually
be
fast
and
second,
we
could
do
it
still
supporting
kubernetes
and
CI,
but
not
requiring
the
two
of
them.
C
So
all
of
that
gave
us
a
couple
of
places
that
we
could
focus.
One
of
them
was.
We
should
really
arrange
it
so
that
people
other
than
me
can
do
the
really
fast
loop,
because
that
would
make
sense
right.
The
other
one.
Is
we
needed
to
figure
out
a
way
that
the
end-to-end
tests
could
be
more
functional
as
part
of
your
development
loop,
so
the
first
one
with
the
loop
tools.
C
This
is
basically
just
a
redrawing
of
that
original
one
that
we
were
running
back
in
the
original
compiler
tests.
We
had
a
saved
input
configuration,
we
run
it
through
Ambassador.
We
got
it
generated
envoy
configuration
and
we
had
a
problem
in
the
beginning,
where
you
couldn't
take
a
kubernetes
input,
you
can
take
the
same
input
configuration
that
you
used
for
an
end-to-end
test
running
in
the
cluster
and
just
feed
it
into
this
process.
B
C
I
know,
but
it's
worth
pointing
out,
it
wasn't
just
Docs.
It
was
also
we
could
not
have
done
this
without
the
refactor
for
0.50.
The
original
set
of
compiler
tests
were
basically
hooking
into
a
bunch
of
internal
things
that
nobody
was
really
supposed
to
mess
with,
and
once
we
did
the
grant
refactor
we
actually
had
api's
and
we
had
formalized
data
structures
and
we
had
things
that
we
could
use
to
really
start
looking
at
properly
specifying
how
you
could
separate
the
input
from
the
cluster
and
how
you
could
do
all
that
processing.
C
So
that
turned
out
to
be
a
really
good
thing.
You
can
run
a
single
test
with
mockery
in
about
three
seconds
or
I
can
do
that
on
my
Macintosh.
It
might
actually
be
faster
on
Linux,
because
a
substantial
part
of
that
time
is
syncing
configuration
crap
from
my
Macintosh
into
the
docker
container.
That's
running
mockery,
so
that
seemed
do
a
decent
job
of
addressing
that
or
you
know
at
least
you
get
in
there,
the
next
one
at
the
end-to-end
test.
That
was
still
something
of
a
challenge.
C
This
is
the
picture
from
before
we
can
redraw
it
to
talk
a
little
bit
more
about
the
high-level
functionality
of.
What's
going
on,
we
take
a
configuration,
we
save
it
in
kubernetes
cluster
relate
Ambassador,
read
the
configuration
generate
envoy
start
the
Envoy
running
with
that,
and
then
we
feed
a
bunch
of
requests
through
and
save
the
responses
in
the
world
of
cat
were
actually
not
doing
just
one
of
those
we
have
one
cluster,
but
there's
a
ton
of
ambassadors
running
in
it.
C
This
has
a
nasty
habit
of
making
the
cluster
catch
on
fire
and
die,
and
we
tried
a
bunch
of
things
to
make
that
better.
We
tried
tuning
the
number
things
we
were
running
in
parallel.
We
tried
partitioning
input,
sets
we
tried
switching
arranging
so
that
ambassador
was
only
using
label.
Selectors
to
look
at
the
configure
was
particularly
interested
in
instead
of
having
to
parse
stuff
I
mean
a
bunch
of
that
helps
for
a
little
while.
But
what
seems
to
really
get
us?
C
Some
traction
was
when
we
started
thinking
instead
about
what
do
we
really
need
to
test
here?
In
general,
we
run
on
kubernetes,
but
we
don't
generally
need
to
test
kubernetes
a
new
version
of
Kubb
comes
out.
We
need
to
make
sure
that
ambassador
can
run
with
it,
but
that's
different
from
hey
kubernetes
might
break
tomorrow.
For
our
usage
we
generally
get
to
rely
on
kubernetes
being
mostly
stable.
C
So
we
know
exactly
which
set
of
Onaway
bits
went
in
there
for
every
single
build,
at
least
now
we
do,
and
that
means
that,
while
we
have
to
worry
about
this,
if
we've
just
changed
our
version
of
envoy
way
that
we're
going
to
ship,
if
we
know
the
version
of
envoy
hasn't
changed,
we
don't
need
to
test
on
voice
charity
again,
but
we
always
have
to
test.
The
ambassador
part.
D
C
Itself
to
an
interesting
idea
of
wedging
that
mockery
tool
into
the
end-to-end
test
scenario
and
using
it
when
we
know
that
nothing
has
changed
so
that
we
get
to
make
sure
that
we're
still
generating
a
correct
envoy
configuration.
But
we
don't
have
to
test
to
make
sure
that
envoy
has
not
changed
when
the
bits
haven't
changed.
C
C
We
also
have
it
set
up
so
that
you
can
go
ahead
and
run
everything
on
kubernetes
do
the
full
end-to-end
functional
test
and
then,
instead
of
doing
a
release,
just
capture
the
snapshots
that
you
used
to
feed
back
into
Mochrie.
So
we
also
got
rid
of
a
lot
of
that
toil
that
I
was
talking
about
earlier.
C
There's
also
a
mode
where
you
can
have
it
run
where
it
uses
the
cache
for
most
things
and
then
goes
back
and
actually
runs
some
tricky
stuff
on
kubernetes
itself,
for
example,
the
way
the
tooling
is
put
together.
We
haven't
yet
really
gotten
it
set
up
interoperate
when
we're
also
doing
console.
This
is
an
example.
That's
done
in
about
14
minutes,
it's
not
where
I
want
it
to
be,
but
on
the
other
hand,
it's
14
minutes
and
it's
actually
reliable,
as
opposed
to
it's
half
an
hour
and
a
lot
of
the
time.
It
doesn't
work.
C
One
really
important
note,
as
we
start
to
wrap
up
here,
is
that
it
is
not
free
to
go
through
and
put
in
the
efforts
to
try
to
arrange
it
so
that
you
can
speed
all
of
these
things
up
and
have
a
better
developer
experience
and
not
have
to
suffer
quite
so
much
as
you're
trying
to
get
things
done.
I
still
think
it's
worth
doing.
C
So
we're
going
to
end
up
with
a
fair
amount
of
time
for
questions
which
is
handy,
I'm
gonna
leave
you
both
with
the
list
of
epiphanies
that
we
ran
across
doing
this.
Probably
the
biggest
one
was
simply
acknowledging
that
a
change
is
not
just
code.
The
part
where
you're,
taking
your
code
and
going
through
and
gaining
confidence
that
it's
going
to
do
the
right
thing
turned
out
to
be
really
important
to
acknowledge
the
rest
of
it.
C
C
E
So
this
is
a
really
interesting
subject
to
me
because
it
applies
in
the
micro
where
you're
talking
about
a
single
thing
ambassador
and
then
it
applies
when
you
scale
up
to
multiple
services
and
a
application
built
of
micro
services
and
etc
etc.
Have
you
given
any
thought
to
how
you
might
apply
lessons
learned
here
to
a
distributed
system
of
services
or
systems
of
systems
of
services.
C
C
That
technique
certainly
works
well
on
distributed
systems,
but
yeah,
that's
probably
where
I
would
start
a
lot
of
micro
services
is
about
is
about
being
able
to
release
things
independently
of
the
whole
shootin
match
should
match
so
extending
that
to
thinking
about
micro
services
and
being
able
to
test
independently
of
testing
the
whole
world
as
well.
I
think
can
be
important.
C
You
know,
and
it's
pretty
common
for
us
to
be
working
at
this
point
collaboratively
on
something
where
you
may
have
two
guys
who
get
together
to
work
on
a
feature
and
one
guy
is
gonna,
go
off
and
hammer
on
the
code
and
the
other
guy
thinks
it
would
be
more
interesting
to
figure
out
what
the
new
configuration
will
look
like
and
that
sort
of
thing-
and
so
we
end
up
with
stuff
like
that.
But
it's
probably
more
common
for
small
things.
Anyway.