►
From YouTube: How We Built Contour, and What You Can Learn From Our Experience - Dave Cheney, Heptio
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)
How We Built Contour, and What You Can Learn From Our Experience - Dave Cheney, Heptio (Any Skill Level)
In 2017 Heptio launched Contour, a Kubernetes Ingress controller built using the Envoy proxy project from Lyft. This session will cover: - Why we chose Envoy as the data plane for our ingress controller. - How Contour works as a translator from Kubernetes to Envoy objects. The parts that were a good match, the parts that weren't, and how we dealt with it. - Traps, pitfalls, and ambiguities in the Ingress spec. - Specific patterns we developed to reliably watching many different kinds of API resources at the same time. The presentation will also give real world advice on the traps involved in the components that make up a management server, the k8s Go client, gRPC, and concurrency issues.
About Dave
You can talk to me about Go, Kubernetes, and Contour.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Alright,
thank
you
for
coming.
My
name
is
David.
I
am
a
staff
systems
engineer
at
hep,
tear
and
I'm
here
to
talk
to
you
about
contour,
which
is
a
project
that
I've
been
working
on
since
I
joined
hippy.
Oh,
this
is
a
presentation
it's
in
unequal
parts.
The
first
is
about
the
design
of
contour
and
the
second
is
really
about
the
act
of
developing
a
kubernetes
controller.
So
hopefully
there'll
be
something
something
for
everyone
here
in
this
room.
I
want
to
I
want
to
recognize
that
I'm
actually
in
the
envoy
track.
A
So
let's
get
a
let's
get
an
idea
here
like
who
here
is
here
because
you're
here
for
to
talk
about
envoy
some
people-
ok
good
who's
here,
because
it's
koukaon
you
here
to
talk
about
kubernetes,
other
people,
excellent
and
who's
here.
To
talk
about.
Go
nice
all
right,
ok,
so
I
want
to
give
a
little
with
the
background
we
have
in
this
room.
I
want
to
give
a
little
bit
about
what
an
ingress
controller
does
we're
here
in
Northern
Europe,
so
you'll
recognize
signs
like
this.
A
That's
it,
and
the
next
question
you
might
have
is
probably
along
the
lines
of
well
I'm,
already
running,
enable
us
or
GCE,
or
something
and
I'm,
using
a
service
load,
balancer
like
why
don't
need
to
care
about
this
ingress
thing
like
what
is
it
going
to
do
for
me
and
to
answer
that
I
want
to
go
through
a
little
bit
of
what
I,
perhaps
pretentiously
call
the
philosophy
of
ingress
like
if
you're
running
this
extra
component
in
your
infrastructure,
it
has
a
non
zero
maintenance
cost.
So
what
is
it?
A
What
is
it
going
to
give
you
in
return
for
that
cost,
and
the
simples
work
and
I
can
put
it?
Is
that
a
good
ingress
controller
should
about
90%
of
the
things
that
you
would
have
to
deploy,
Apache
or
nginx
or
squid,
or
some
kind
of
front
proxy
in
your
request
path,
it
should
take
care
of
that.
You
know
that
90%
number
now
now
don't
get
fixated
on
like
Dave
said
90,
it
could
be
80
percent,
it
could
be
82
percent.
The
exact
number
is
not
important.
A
A
If
every
time
you
deploy
a
service,
you
deploy
it
behind
it's
only
lb
or
GK
load
balancer
or
whatever
that's
going
to
have
a
cost.
It's
going
to
have
a
maintenance
cost.
These
things
are
not
free
and
neither
a
public
IPS,
they're
kind
of
getting
rare.
These
days
and
depending
on
the
security
posture
of
your
organization,
there
may
not
be
cool
with
having
so
many
ways
for
traffic
to
come
into
your
network.
A
So
the
second
is
TLS
management.
Its
2018
in
a
couple
of
months,
chrome
is
going
to
start
marking.
Things
which
are
not
HTTPS
is
insecure,
so
we've
got
to
get
on
board
with
that.
We
have
great
projects
like
let's
encrypt
and
cert
manager,
we're
going
to
take
care
of
that.
The
mechanisms
of
getting
you
that
TLS
certificate-
all
you
have
to
do
is
now
present
it
to
people.
Now
you
can
do
this
with
your
cloud
provider.
You
can
go
directly
to
your
EOB
or
alb
and
set
up
the
certificate,
but
that's
not
something.
A
The
third
is
perhaps
an
ocean
of
abstract
configuration
like
if
you've.
If
ten
years
ago,
you
deployed
your
application
behind
Apache,
you
have
an
Apache
config.
Can
you
take
that
and
run
that
straight
on
nginx?
No,
they
speak
different
languages.
So
a
good
ingress
control
I
should
give
you
a
notion
of
abstracting
this
configuration.
So
it's
portable
across
solutions
and
the
last
thing
like
once
you've
got
a
single
set
of
IPs.
You've
got
your
TLS
set
up.
There
might
be
some
other
things
which
might
still.
You
might
still
need
to
have
something
in
your
request.
A
Flow
and
I
call
these
kind
of
table
stakes
path,
based
routing
if
using
a
service
load,
bouncer,
you're,
working
at
layer,
3
all
that
traffic
is
going
to
go
directly
to
your
application.
There's
plenty
of
cases
where
you
want
to
take
your
static
data
and
have
that
served
from
a
different
application.
Maybe
your
ads
comes
from
a
different
service,
so
you
want
to
do
path
based
routing
and
we'd,
like
that
to
be
in
an
abstract
configuration
form,
not
kind
of
handled
in
your
application,
as
I
said
before
TLS.
A
So
it's
very
common
to
need
something
to
do
that.
Http
HTTP
ups
HTTP
upgrade
and
it'll
be
a
real
shame
to
have
to
run
a
web
server
in
front
of
your
application.
Just
to
do
that
fix
up-
and
this
is
the
last
one
request.
Rewriting-
is
something
that
we
don't
currently
do
in
contour,
but
we
certainly
want
to
do
it
is
that
there
are
plenty
of
times
in
my
career
where
a
URL
has
been
published
like
got
tweeted
out
or
it's
in
some
marketing
material.
A
It's
slightly
wrong,
so
you
need
you,
have
you
have
the
choice
in
do
I
add
do
I
change
my
application
to
like
match
the
marketing
material
or
like
how
am
I
going
to
fix
this
up,
and
so
one
of
the
things
I
think
an
ingress
controller
should
do.
Is
some
at
least
basic
rewrite,
rewriting
not
going
crazy,
but
just
some
basic
stuff
and
so
contour
exists
as
an
ingress
controller
to
fulfill
these
requirements
that
I
talked
about.
A
This
is
our
goal,
and
it's
also
important
to
note
that
contour
is
only
concerned
with
what
you
can
put
in
the
ingress
document.
You
might
want
TCP,
forwarding
or
SSL
pass-through,
but
if
you
can't
talk
about
it
in
the
schema
of
the
ingress
object
and
unfortunately,
contour
can't
help
you
do
that.
This
isn't
necessarily
like
contours
fault
or
envoys
fault.
It's
widely
recognized
that
the
ingress
object
in
kubernetes
is
underdeveloped,
but
that's
kind
of
out
of
scope.
For
this
talk,
the
goal
of
contour
is
to
accompany
as
much
of
this
ingress
philosophy
get
to
that.
A
So
the
question
is:
why
did
we
choose
to
build
contour
on
envoy,
and
the
simplest
answer
is
that
envoy
is
a
proxy
design
for
dynamic
configuration?
The
cute
appreciating
why
I
think
that
this
is
important?
Is
that
the
difference
to
every
other
server
I've
used
every
other
Network
appliance
that
I've
used?
That
has
an
API?
Is
that
yeah
you
can?
You
can
connect
to
their
API
and
you
can
change
some
configuration
values
as
a
client
of
that
device,
but
the
the
the
source
of
truth.
A
The
configuration
lives
in
that
device
they
own
it
envoy,
is
completely
different.
It's
an
API
that
connects
to
a
server
which
unimaginative
Lee,
they
call
the
management
server
and
it
asks
for
its
configuration,
and
this
is
the
linchpin.
This
is
what
makes
the
difference
by
default.
Envoy
doesn't
do
anything,
it
just
waits
for
configuration
to
be
streamed
to
it,
and
the
similarity
with
the
couplet
is
really
strong.
Think
about
what
the
Kubler
does.
It
just
runs.
Pods
that
match
a
filter,
it
doesn't
know,
doesn't
know
what
care
how
they
got.
A
A
And
so,
if
this
boils
down
to,
contour
is
just
a
gateway,
a
translator
between
kubernetes
api
and
the
Envoy
api.
The
traffic
flows
from
your
left
to
right,
contour
just
watches
the
kubernetes
api
and
translates
it
into
the
gr
pc
stream.
That
envoy
needs
and
it
turns
out,
there's
some
pretty
good
synergies
between
coop,
the
kubernetes
starter
model
and
the
envoy
data
model.
A
Kubernetes
is
in
grocers
mapped
to
RDS,
which
is
routes
and
LDS,
which
is
listeners
for
TLS
and
SNI
services
in
kubernetes
map,
almost
directly
to
the
cluster
object
in
Envoy,
the
backend
abstract
cluster
endpoints
map
even
more
directly
to
EDS,
which
is
the
endpoint
data
service.
They,
which
contains
cluster
load
assignments.
A
The
elements
of
instances
of
a
service
and
kubernetes
secrets
are
a
little
bit
a
little
bit
harder
to
handle
because,
while
the
ingress
says
I
want
this
URL
to
be
on
tier
on
HTTP,
the
secret
actually
contains
the
certificate
data,
but
we
can
manage
that
with
a
small
state
machine
and,
as
a
brief
digression,
people
people
frequently
ask
me
so
contour
uses
envoy
ride.
So
it's
a
service,
mesh,
right
and
I
think
the
answer
I've.
A
Clearly
shown
is
that
no,
we
are
focused
just
on
HTTP
contour
only
deals
with
ingress,
but
it
has
made
a
really
great
base
to
build
upon.
So,
for
example,
if
you'd
like
to
hear
about
the
things
we're
building
on
top
of
contour,
you
can
drop
by
the
egipto
booth
outside
or
you
can
come
here.
My
colleague
ross
talked
about
gimbal,
which
is
a
project
which
builds
about
contour
and
a
number
of
our
other
open-source
products
for
multicast
or
traffic
management
right.
A
A
A
Also,
a
contour
as
a
moderately
complicated,
kubernetes
controller
10,000
line
feels
about
the
right
size,
certainly
3,000
lines
for
actual
the
code
that
feels
about
the
right
size.
So
if
you're
writing
a
controller
and
you're
getting
into
tens
or
hundreds
of
thousands
of
lines
of
code.
My
suggestion
is
to
stop
and
wonder
if
you're
trying
to
do
too
much,
if
you
need
to
decompose
your
problem
a
bit
more,
this
is
a
quick
overview
of
what
contour
looks
like
in
the
repo.
It's
slightly
edited.
We
have
our
nice
skinny
command,
which
just
sets
up.
A
The
logging
connects
to
the
Kuban
A's
client
sets
up,
sets
up
all
the
little
bits
and
pieces
and
then
lets
them
go
off
and
run.
We
do
very
little
work
actually
in
the
command
internal
contour.
Is
the
translator
the
bit
in
the
middle
that
you
saw,
and
this
is
where
all
the
horror
show
of
converting
kubernetes
objects
to
envoy
objects
happens
each?
We
is
our
integration
tests
and
we'll
come
back
to
those
later
in
the
talk,
and
then
we
just
have
a
bunch
of
helpers.
This
is
mainly
around
constructing
the
bootstrap
config.
A
The
this
is
a
gr
PC
server
which
implements
the
protocol
that
envoy
uses
so
called
XDS.
Some
kubernetes
helpers
and
some
goroutine
helpers,
which
we'll
talk
about
later
on
and
that's
all
there
is
that's
all
there
is
to
it
now.
It's
common
in
projects
that
descend
from
the
kubernetes
family
to
put
all
your
code
into
a
package
subdirectory
and
personally
I
think
this
is
a
little
redundant
like
in
go.
A
We
don't
start
types
that
our
interfaces
with
an
I,
so
putting
packages
in
a
directory
called
package
seems
a
little
bit
redundant,
but
I
do
see
the
the
similarity
with
putting
commands
in
a
directory
called
command.
So
I'd
like
to
offer
an
alternate
suggestion,
which
is
to
place
your
code,
that
you
do
not
intend
other
projects
or
packages
to
use
inside
an
internal
folder
internal
is
a
special
name
that
the
go
tool
recognizes
and
will
effectively
prevent
packages
which
are
not
siblings
of
that
package
from
importing
it.
A
The
next
thing
I
want
to
talk
about
is
go
team
management,
all
all
go
programs.
All
kubernetes
controllers
are
consisting
of
a
number
of
go
routines
working
at
there
very
few
that
are
just
a
single
thread
of
execution.
I
mean,
for
example,
contour.
We
have
to
watch
services,
ingresses,
endpoints
and
secrets,
so
it's
a
goroutine
each
just
for
the
watcher.
A
Then
we
need
to
host
a
gr,
PC
server
and
we
also
host
that
HTTP
server
for
the
debug
prof
endpoint,
so
at
six
or
seven
different
go
routines
right
there
and
they
all
represent
the
running
program.
What
we
need
is
a
way
to
link
them
together,
because
if
one
of
them
stops,
we
want
the
rest
of
them
to
stop
like.
A
If,
if,
if,
if
one
starts
working,
then
the
service
doesn't
work,
so
we
we
want
if
one
stops,
we're
going
to
shut
the
rest
of
them
down
and
what
we
came
up
with
uninventive
Lee
is
called
a
workgroup
or
just
a
group,
and
with
this
you
can
register
a
function
with
the
group.
It
takes
one
parameter
which
is
a
channel
which
will
be
closed
when
you're
asked
to
shut
down.
So
it's
your
your
shutdown
signal
and
then
this
is
the
main
run
method.
A
A
And
at
the
end,
we
just
wait
for
one
of
them
to
stop.
Then
we
ask
all
the
rest
of
them
to
stop,
wait
for
them
to
do
and
then
then
run
returns
now
how
this
works.
In
practice,
this
is
verbatim
from
command
contour.
This
is
how
we
construct
contour
and
run
it.
We
start
with
a
work
group.
We
build
a
kubernetes
client
and
we
register
each
of
the
watches,
passing
in
a
reference
to
that
group.
So
they
register
themselves.
A
A
Of
course
we
check
errors
and
we
don't
want
our
code
to
be
crappy
in
crash,
but
if
there
is
something
that
causes
one
of
those
go
routines
to
exit,
we
just
we
let
contour
exit
and
we
go
through
the
recovery
process
that
kubernetes
provides
to
us.
We
don't
try
to
recover
failures
inside
the
program,
and
a
lot
of
these
ideas
came
from
a
wonderful
presentation
from
peter
berg
on
last
year
at
go
SF.
If
you've
got
time,
you
should
check
that
out
now
in
kubernetes.
Speak
contours.
A
Translator
is
an
instance
of
a
cache
resource
event
handler.
This
is
the
type
that
you
pass
to
the
watcher
to
be
called
with
events,
and
so
what
it
looks
like
is
that
we
have
a
bunch
of
different
watches
running
in
their
own
go
routines,
who
are
all
calling
back
to
contour
and
so
so
to
be
safe
in
this
situation.
We
need
to
do
two
things.
A
To
properly
set
up
LDS
to
set
up
a
TLS
listener,
you
need
the
ingress,
which
has
the
name
and
mentions
a
secret
by
name,
and
you
also
need
the
secret,
which
has
the
the
certificate
data
you
need
both
of
these
pieces
to
make
it
work.
So,
if
we're
watching
for
events,
we
may
see
a
new
ingress
arrive
and
we
look
and
we
say:
oh,
it
mentions
the
secret.
We
haven't
seen
that
secret.
Yet
what
what
we
need
to
do
is
we.
We
have
to
wait
until
we
have
the
secret
before
we
can
proceed
now.
A
If
these
events
could
happen
concurrently,
we
do,
we
would
just
have
to
sit
and
spin
and
wait,
but
what
we
arrange
to
do
is
that
only
one
event
is
in
processed
by
contour
at
a
time.
So
we
can
say
we
don't
have
the
secret.
Now,
that's
fine,
we'll
just
record
that
we
record
what
we
have
and
then
the
secret
will
come
along
later
and
we'll
match
up
with
the
ingress
that
we
were
just
putting
together.
A
A
But
I
was
a
little
bit
concerned
about
blocking
the
API
watches
for
a
long
time,
and
so
this
is
a
good
application
of
channels
and
go
so
we
have
a
small
helper
inside
contour,
which
is
called
a
buffer
which
wraps
a
resource
event
handler
so
create
a
create
a
new
buffer
around
around
the
resource
event
handler
that
we
want
to
buffer
now
buffer
fulfills
the
resource
event
handler
interface.
So
it
can
be
substituted
for
the
thing
that
you
were
wanting
to
buffer
immediately.
A
Each
of
the
sends
will
try
and
put
the
event
that
happened
onto
a
channel
and
in
most
cases
that
will
succeed,
because
the
channel
has
some
free
space.
If
it
doesn't,
we
default
we
go
back
to
blocking,
but
we
also
have
the
opportunity
of
saying
hey.
Now
the
the
channel
is
full
is
you
you
should
be
aware
of
that,
whereas
if
we
were
just
using
a
sink
mutex,
this
blocking
would
be
completely
hidden
to
us
and
then
on
the
other
side
of
the
buffer.
A
It
just
loops
reading
events
from
the
channel
and
dispatching
them
to
the
event
handler
that
we
passed
in,
and
so
that
means
that
we
know
two
things.
Events
now
happen
in
sequence.
We
are
either
reading
from
the
channel
or
down
in
the
on
add
an
update
or
on
delete
event.
We
know
these
things
do
not
happen
simultaneously.
A
A
The
second
one
is
that
we
had
now
some
buffering
so
that
watchers
consider
can
go,
can
send
their
data
on
the
channel
and
then
go
back
to
reading
I
reading
immediately.
I
was.
I
was
initially
concerned
that
this
would
this
would
be
a
problem.
I
didn't
know
what
would
happen
if
we
blocked
the
watches
for
too
long.
A
Okay,
so
at
the
moment
we
use
depth
for
dependency
management.
I'm
sure
the
question
is
going
to
be
what
do
I
think
about
Vigo
and
the
simple
answer
is
I
tried
to
use
Vigo
to
build
a
project
with
client
go
and
it
couldn't
do
it
I?
Consider
that
to
be
the
kind
of
table
stakes
of
any
dependency
management
tool,
I'm
sure
this
will
change
in
the
future,
but
for
the
moment,
depths
working
for
us
now.
A
Client
go
has
been
traditionally
quite
hard
to
manage
with
tools
like
Glide
and
depth,
but
it
has
gotten
sufficient
significantly
better
in
release.
Six
and
upgrading
from
six
to
seven
was
virtually
painless
and,
in
fact,
removed
a
transit
transitive
dependencies
from
our
project,
who
went
from
46
dependencies
down
to
42
down
to
40
down
to
36,
and
so,
if
you
want
to
use
client
go
with
a
go
project
using
depth.
These
are
the
three
lines
that
you
need.
This
is
all
you
need.
A
The
next
bit
is
that
we
don't
check
in
vendor
and
from
a
person
who's
written
a
go
dependency
management
tool
specifically
focused
on
repeatable
builds
I.
Think
I
should
be
quite
clear
about
this.
Not
checking
in
vendor
keeps
our
repository
much
much
smaller,
like
those
nine
and
a
half
thousand
lines
of
code
is
tiny
comparison
to
the
half
a
million
lines
of
code
of
our
dependencies.
So
we
don't
have
to
maintain
that
in
our
repo
we
don't
have
to
have
huge
pull
requests.
A
The
github
can't
even
show
you
so
instead,
when
you're,
when
you
go
to
contribute
to
contour,
just
check
out
the
project
as
usual,
and
then
you
run
this
one-line
DEP
ensure
vendor
only,
and
that
just
says,
replay
the
contents
of
go
package.
Dot
lock
just
check
that
check
out
exactly
those
revisions
and
on
the
whole,
this
has
worked
out
very
well.
For
us,
there
have
been
a
few
very
few
downsides.
A
If
you
change
your
branch,
you
may
occasionally
need
to
rerun
DEP
ensure
because
the
the
set
of
dependencies
has
changed
between
a
branch,
but
we
normally
do
that
kind
of
update
at
the
start
of
each
development
cycle.
I'll
have
a
card
very
early
on
in
the
cycle
to
freshen
our
dependencies
upgrade
to
the
latest
client.
Do
that,
and
so
that's
done,
and
then
we're
good
for
two
months
and
in
the
unlikely
case
that
you
get
a
compile
failure,
you
just
run
DEP
ensure
and
you're
good
to
go
again,
so
it's
working
out
pretty
well.
A
The
second
one
is,
of
course,
this
doesn't
protect
us
from
a
dependency
going
away.
This
is
a
reasonable
concern.
Like
some
environments,
you
do
not
want
to
have
to
depend
on
the
Internet
to
build
your
code
where
we
are
as
an
open
source
project
and
especially
because
we
depend
on
like
the
kubernetes
libraries
is
so
hot,
like
all
of
our
dependencies
coming
because
of
client
go.
So
if
there
was
going
to
be
a
problem
with
one
of
those,
a
lot
of
people
are
going
to
work
very
hard
to
fix
those
straightaway.
A
A
One
of
our
contributors,
is
docker,
ignore
when
you
run
docker
build
it,
takes
everything
in
your
working
directory
and
downwards
from
that
and
sends
it
all
to
the
dock
edemen
just
in
case,
so
it's
taking
the
vendor
directory,
which
we
don't
need,
it's
taking
the
whole
get
subdirectory
which
we
don't
need.
In
our
case,
this
was
getting
up
to
about
120
megabytes
of
data
for
no
no
good
reason.
A
So,
there's
one
little
PR
change
the
amount
of
data
we
sent
to
the
docker
demon
to
under
a
megabyte,
I
think
it
actually
doesn't
go
lower
than
a
megabyte.
I
think
it's!
The
granularity
is
too
is
too
low
there,
but
all
we
do
is
we
just
ignore
the
vendor
directory
and
the
get
directory,
and
so
I
want
to
give
you
a
walkthrough
about
how
I
approach
building
go
projects
with
docker.
So
we
use
the
main
golang
image
that
comes
from
the
docker
hub.
A
The
next
thing
we
do
is
that
we
fetch
DEP
and
we
build
it
inside
the
image.
Then
we
copy
in
our
go
package
lock
and
tunnel
files,
and
then
we
run
DEP
ensure
of
minus
V,
so
we
actually
check
out
the
dependencies.
We
need
to
build
this
code
inside
the
container.
Now.
The
thing
is
that
the
run
step
is
only
going
to
run
if
the
hash
of
the
previous
step
has
changed.
A
So
if
you
have
not
touch
go
package,
dot,
Tamil
and
go
package
dot
lock,
you
won't
need
to
take
the
hit
of
downloading
all
those
dependencies
in
this
container
and
even
better
if
you
switch
between
branches.
As
long
as
you
have
a
previous
build
that
has
the
hash
that
matches
go
package
stock
and
lock.
So
it's
not.
If
you
change
them,
it's
only
if
you
change
them
to
something
new.
Do
you
need
to
do
this
download
step
so
that
works
out
pretty?
A
Well
then,
we
do
the
traditional
thing
of
copying
our
source
in
building
building
our
binary,
and
then
we
use
a
feature,
call
I
believe
it's
multistage
builds
where
we
actually
start
again
with
a
new
Alpine
image.
We
stick
in
the
certificates
and
we
copy
in
the
contour
binary
from
the
previous
step.
So
we
don't
have
this
enormous
enormous
set
of
of
images
built
that
contain
all
the
artifacts
from
building
the
code.
A
How
does
it
show
me
that,
on
my
screen,
so
you
saw
we
didn't
do
any
downloading
of
dependencies.
That
step
we're
skipped
it's
a
little
harder
to
skip
the
actual
full
full
build
of
the
code,
but
I
mean
it's.
It
would
be
possible,
but
it
would
be
quite
fragile,
but,
as
you
see
it's
only
going
to
take
a
few
seconds.
A
Have
it
pull
down
the
image
so
really
try
to
avoid
this
as
much
as
you
can
try
to
avoid
making
docker
build
and
push
part
of
your
edit
compile
in
a
loop,
because,
even
with
like
my
Australian
education,
I
can
tell
you
that
five
minutes
to
do
this
cycle
means
you
can
do
it,
whose
is
it
twelve
times
an
hour
yeah?
It's
you
want
to
avoid
that.
You
want
to
keep
your
velocity
up,
and
so
this
is
what
a
normal
edit
compile
cycle
looks
for
us.
We
take
advantage
of
go
one
tens
built-in
test
occasion.
A
You
can
see
some
of
those
tests
occation,
because
none
of
the
none
of
the
files
they
depend
on
changed.
Therefore,
we
don't
need
to
rerun
the
tests,
and
so
we
have
a
cycle
on
the
order
of
seconds
not
minutes,
and
you
really
really
want
this.
You
want
to
be
able
to
catch
mistakes
and
iterate
in
the
order
of
seconds,
not
in
minutes,
and
so
we
have
unit
tests
that
pass.
We
know,
but
how
do
we
know
that
things
are
actually
working?
A
You
know
like
that.
This
would
be
that
this
would
be
the
step.
You
would
actually
do
the
darker
push
and
then
do
some
testing
and
we
have
a
bunch
of
ways
of
avoiding
this.
So
what
happens
is
that
we
can
run
I
can
run
contour
locally
on
my
laptop
connected
to
my
gke
cluster,
so
it's
running
as
it
would,
but
I
don't
have
to
go
through
the
whole
docker
push
step
and,
on
the
right
hand,
side.
We
have
a
tool
called
the
CLI
which
pretends
to
be
Envoy.
A
It
connects
the
same
the
same
endpoints
that
Envoy
does
and
receives
the
same
data
and
because
envoy
uses
protocol
buffers
which
are
strongly
typed.
You
have
a
fairly
good
chance
that
if
it
actually
can
pass,
if
you
actually
put
the
right
type
values
in
the
right
parts
of
the
data
structure,
your
message
is
syntactically
correct.
So
that
eliminates
a
whole
bunch
of
like
kind
of
experimental.
Try
and
file
type
development,
and
we
now
include
the
CLI
in
the
docker
image,
because
it
turns
out
that
it's
a
really
useful
tool
to
debug
problems
in
production.
A
A
So
in
my
experience,
functional
tests
into
n
tests
that
sometimes
called
are
terrible,
they're,
really
slow
and
because
they're
really
slow.
That
leads
to
engineering
effort
spent
to
make
them
faster,
make
them
paralyzed,
rather
than
start
and
stop
processes
which
are
being
tested
against
every
time.
You
just
kind
of
like
try
and
reset
them
or
scrub
the
data
in
them,
and
that
just
makes
them
unreliable.
A
A
Do
them
I'm,
just
saying
they're
very
expensive,
so
I
put
them
off
as
long
as
I
could
being
a
single
person
on
a
single
team,
but
there
are
some
scenarios
that
unit
tests
cannot
cover,
because
there
is
an
impedance
mismatch
between
kubernetes
and
envoy,
and
what
we
need
to
do
to
functionally
test
contour
is
that
we
need
to
model
not
just
a
single
interaction,
a
single
translation
of
like
here's,
an
ingress
object.
What
did
you?
How
did
you
translate
it?
We
need
to
actually
model
the
sequence
of
events.
A
A
A
A
So
if
we
apply
those
two
rules,
this
is
what
we're
actually
testing.
We
just
need
the
inputs
and
we
check
the
outputs
and
you
can
do
that
pretty
reasonably.
With
just
the
standard
go
testing
framework,
we
have
a
little
setup
helper.
It
makes
a
contour.
It
creates
a
G
RPC
server,
listening
on
a
random
port,
because
G
RPC
refuses
to
not
listen
on
the
network.
A
We
create
a
client
to
dial
that
server
we
just
created,
so
we've
created
a
little
a
single
use,
client
and
server,
and
then
we
hand
them
back
to
the
caller,
along
with
a
shutdown
function.
That's
going
to
clean
up
all
of
this
when
the
test
is
done,
and
so
this
is
an
example
of
how
we
do
an
interim
test
case.
A
We
get
the
resource
handler,
which
is
our
input,
the
left-hand
side.
We
get
the
G
RPC
client,
which
is
our
output.
We
can
every
time
we
change
the
input.
We
can
then
check
the
output,
and
then
we
make
a
kubernetes
service
object.
We
just
construct
it
by
hand.
We
insert
it
into
the
API
like
we're.
Not
we
don't
need
to
have
the
kubernetes
api
server
running,
because,
fundamentally,
all
that's
going
to
do
is
call
a
watcher
to
call
our
resource
event
handler.
A
So
we
just
call
it
directly
and
skip
the
middleman,
and
then
we
query
contour
to
check
the
expected
results
and
we
just
go
on
and
do
that.
We
can
model
it
by
adding
extra
services
changing
them
from
one
to
another.
We
can
model
all
the
all
the
cases
of
data
arriving
in
any
sequence,
pretty
pretty
easily
with
this
so
how's.
A
This
worked
out
for
us,
we'll
start
with
the
bad
news,
as
you
probably
figured
out,
it's
quite
verbose,
even
with
lots
of
helpers-
and
you
know
this
is
a
sad
face,
but
it's
explicit
when
this
event
happens.
I
expect
this
output.
You
will
not
find
a
sleep
or
a
retry
in
our
test
code.
We're
never
waiting
for
we're,
never
kind
of
dealing
with
the
fact
that
we
put
the
input
into
the
kubernetes
api
server
and
it
hasn't
quite
got
back
to
us.
Yet
you
will
not
find
a
sleep
in
our
into
n
tests.
A
They
run
in
seconds,
not
tens
of
minutes.
The
highlights
we
get
from
this
approach,
a
really
high
success
rate
of
reproducing
bugs
that
are
reported
in
the
field.
We
have
people
come
onto
slack
or
raise
issues
in
github,
I
can
model
what
they're
telling
me
like
I
asked
them.
Tell
me
about
the
service.
Show
me.
The
Amaral
document
show
me
what
you
see
in
the
output
of
on
voice.
Debugging
and
I
can
model
that
as
a
test.
A
It's
really
easy:
it's
a
really
easy
pattern
for
contributors
to
add
tests,
like
literally
just
follow
the
follow
the
pattern,
and
so
when
they
submit
features
we
can
have
tests
along
with
that
to
make
sure
they
don't
regress
in
the
future
and
because
it's
easy
to
model
failing
scenarios,
it
enables
us
to
do
good
old
TDD.
We
can
start
with
a
failing
end-to-end
test
work
down
through
the
different
packages
right
down
until
we
find
writing
a
test
every
time
right
down
to
the
point
where
we
find
the
bug,
we
fix
that
bug.
A
All
of
those
new
tests
go
green,
put
it
in
a
PR
job
done
nothing
more
to
do
and
it
avoids
debugging
in
production
involves
that
docker
push
and
then
doing
manual
testing
of
I'm.
Trying
to
recreate
the
scenario
someone
told
me
on
slack
like.we:
it
avoids
entirely
both
the
the
overhead,
the
human
overhead
of
doing
manual,
testing
and
just
the
time
complexity
of
this
docker
pushed
kind
of
workflow.