►
From YouTube: Safely Deploying a 100K line Envoy YAML Configuration to Production - Lisa Lu & Jyoti Mahapatra
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
Safely Deploying a 100K line Envoy YAML Configuration to Production - Lisa Lu & Jyoti Mahapatra, Lyft
Have you ever caused a production incident due to an Envoy misconfiguration? You’re not alone! This talk is about how Lyft has built guardrails to prevent such failures. The presenters will share their experience operating Envoy configurations at scale. They will explore the challenges around handling constantly changing cluster and routing configurations & the tools used to guarantee accuracy and consistency in those changes. These tools empower service owners less familiar with Envoy to make configuration changes independently and quickly without approval barriers. This talk will introduce the audience to various Envoy configuration testing strategies: •Validating behavior for thousands of routes to avoid blackholing traffic •Auditing & safely removing unused routes and clusters •Safely deprecating fields between Envoy versions •Validating Envoy’s static & realtime configurations
https://sched.co/ekEq
A
Hi,
everyone
welcome
to
our
talk
on
how
to
validate
envoy
configs
at
scale
we'll
be
going
over
strategies
for
automating
the
config
validation
and
testing
process,
so
that
your
service
owners
can
iterate
quickly
and
independently.
This
will
also
prevent
your
envoy
operators
from
being
bogged
down
with
code
reviews.
A
A
A
A
We
validate
these
before
sending
them
to
the
control
plane,
because
we
want
to
avoid
the
control
plane
needing
to
do
any
validation
as
it
gets
the
route
and
cluster
information
in
real
time,
and
rather
frequently
as
the
number
of
services
and
routes
that
lift
has
grown
on
some
side
cars.
We
have
configurations
that
are
upwards
of
a
hundred
thousand
lines
of
yaml,
which
makes
maintenance
and
modification
extremely
complex
and
risky
I'll
describe
some
of
the
issues,
in
fact
that
we've
encountered
over
the
last
year.
A
Firstly,
as
a
growing
company,
we
have
new
developers
joining
every
week,
and
business
expansion
also
means
that
new
services
are
being
spun
up
to
cater
to
new
scenarios.
Quite
frequently,
this
means
more
code
review
requests
from
developers
who
aren't
familiar
with
envoy
and
want
to
know
if
their
change
will
do
what
they
want
it
to
do.
While
it's
great
that
we're
having
more
and
more
development,
it's
not
so
great
that
the
on-call
burden
has
increased.
It
means
the
on-call
spends
more
time,
triaging
these
tickets
and
having
to
stare
at
yaml.
A
Another
issue
is
human,
con
human
error
in
configs,
with
such
large
yaml
files,
it
isn't
feasible
for
catching
everything.
For
example,
one
common
source
of
bugs
is
how
envoy
does
first
order
matching
for
its
routes,
which
means
that
in
a
huge
list
of
routes,
it
might
be
easy
for
a
developer
or
a
viewer
to
miss
a
route
that
captures
all
the
traffic
that
is
actually
intended
for
a
new
route.
A
In
response
to
such
errors,
we
actually
mandated
a
code
review
from
the
networking
team
for
all
routes.
At
one
point,
this
quickly
became
the
bottleneck
for
iteration,
because
our
networking
team
is
about
six
people,
and
while
we
were
the
guardians
of
these
route,
changes
for
a
while
and
tried
to
eyeball
every
change
going
in
this
just
wasn't
proving
to
be
sustainable
and
was
slowing,
iteration
and
rollouts
for
the
rest
of
the
company.
A
Next
reason
is
that
changes
are
hard
to
test.
Actually,
this
was
a
main
reason
for
the
first
issue
that
I
highlighted,
which
is
that
service
owners
weren't
sure
how
to
validate
their
changes,
which
is
why
they
wanted
to
reach
out
to
the
networking
team
in
the
first
place
prior
to
automating
our
testing.
It
was
really
difficult
to
validate
changes.
You
had
to
have
an
end-to-end
setup
with
setting
up
an
envoy,
sidecar
and
sending
requests
to
see
if
your
config
changes
worked.
A
A
A
So
in
this
scenario,
ideally
in
the
top
row,
you
would
have
rats
pointing
to
your
service
or
cluster,
but
when
it
comes
to
new
service
launches
or
service
deprecations,
sometimes
developers
only
address
the
route
definitions
and
forget
the
clusters,
or
vice
versa.
So,
for
example,
in
the
second
row,
you
see
that
there's
no
route
definition
pointing
to
the
service,
and
this
could
mean
that
the
developer
didn't
leave
in
any
routes,
which
means
the
cluster
definition
isn't
being
used
or
they
deprecated
the
routes
and
forgot
to
remove
the
cluster.
A
While
this
cluster
is
not
accessible
by
outside
users,
your
control
plane
is
still
creating
unnecessary
resources
here
and
sending
it
to
side
cars.
In
addition,
it
just
creates
tech
debt
and
can
make
reading
through
configs
confusing,
since
the
service
is
presumably
no
longer
being
used
or
isn't
currently
being
used.
A
In
this
scenario,
envoy
will
return
a
503
to
request
going
to
these
routes
because
it
doesn't
see
any
healthy
members
for
this
cluster
and
obviously
you
don't
want
your
users
to
be
hitting
503s
for
no
reason.
So
both
checks
here
can
prevent
human
error
from
derailing
a
service
launch
or
deprecation,
which
usually
happens
when
someone
forgets
one
of
the
configs.
A
Next,
to
validate
the
bootstrap
config,
we
make
use
of
a
couple
of
open
source
tools.
The
first
one
I'll
discuss
is
the
validation
server.
This
refers
to
running
the
envoy,
binary
in
validation
mode,
which
you
can
set
via
the
command
line
flag
mode,
and
what
this
does
is
it
takes
your
binary
and
it
takes
your
bootstrap
config
and
it
tries
to
boot
up
envoy
without
starting
workers,
but
it
will
go
through
the
server
initialization
process
as
far
as
it
can,
and
if
there
are
no
errors
it
will
exit,
it
will
exit
successfully.
A
The
next
tool
we
use
is
the
config
load
check
tool
and
you
can
see
the
docs
on
the
left.
This
is
a
standalone
binary,
which
you
can
run
on
all
of
your
bootstrap
configs
and
it
basically
checks
that
all
the
values
in
the
fields
are
valid,
as
defined
by
the
proto,
and
that
the
schema
is
a
valid
envoy
schema.
A
We
run
both
of
these
checks
on
all
of
our
pull
requests
in
ci
and
that
just
makes
reviewing
these
changes
so
much
easier
because,
as
the
reviewer,
you
already
know
that
the
config
is
a
valid
envoy
config.
And
so
you
only
need
to
ensure
that
the
change
does
what
it
should
I'll
now
hand
it
over
to
jyoti
to
discuss
the
router
check
tool
and
its
various
functionalities.
B
Thank
you
lisa.
Next,
I'm
going
to
talk
about
the
router
check
tool.
The
part
of
configuration
in
most
flux
over
the
last
few
years
have
been
modifications
to
the
routes
routes
work
in
a
very
sensitive
way.
The
routing
engine
runs
an
incoming
request
or
a
set
of
routing
rules
and
the
first
one
to
match
wins.
B
B
B
B
The
tests
always
run
the
input
route
through
the
routing
engine
and
compares
the
assertions
with
the
actual
results.
Let's
follow
an
example.
Imagine
we
have
a
routing
table
assured
on
the
left
picture.
Imagine
thousands
of
sith
routes.
One
can
imagine
the
plight
of
the
developer.
Who
is
making
changes
to
such
a
long
conflict?
B
A
B
B
B
B
All
this
was
exciting,
but
the
tool
did
not
have
a
way
of
enforcing
adding
tests.
Developers
were
able
to
work
around
the
stop
gap.
Phase
of
test
enforcement.
We
decided
we
needed
a
way
to
add
code
coverage
to
the
tool.
There
are
two
ways
of
adding
coverage.
First,
one
is
shallow
and
just
checks
that
there
is
a
test
for
every
routing
rule.
The
second
one
is
more
strict
and
enforces
writing
all
assertions
for
each
test.
B
B
B
System,
our
infra
is
based
on
runtime
configs
for
safe
rollouts
developers
needed
a
way
to
flip
the
runtime
in
config
and
test
how
the
routes
behaved.
We
added
new
fields
in
the
input
to
crank
the
runtime
using
some
random
values
and
test
the
output.
We
had
now
almost
100
question
code
coverage
in
our
system.
B
B
Constantly
updating
the
onward
binary
comes
at
a
cost.
There
are
fields
getting
deprecated
all
the
time
and
having
them
in
the
system
introduces
spectate
and
a
higher
migration.
Time
later
we
added
a
deprecated
field
check
in
the
tool
so
that
it
fails
whenever
it
observes
a
deprecated
field.
We
quickly
resolve
it,
since
usually
it's
one
or
two
fields
at
a
time
and
doesn't
stay
long
enough
to
become
risky.
B
B
A
Great
thanks
so
hopefully,
through
this
presentation,
you've
seen
the
different
ways
you
can
utilize
open
source
tooling
or
writing
your
own
scripts
to
automate
the
testing
and
validation
of
envoy
configs.
And
while
this
is
really
powerful,
I
mean
it
certainly
helped
lift
keep
up
with
our
huge
ammo
configs.
There's
still
a
lot
of
room
for
improvement
and
how
we
build
the
tool
and
to
test.
A
A
Building
off
of
this
one
major
improvement
would
be
utilizing
production
code.
Currently,
the
router
check
tool
uses
the
same
routing
function
as
you
know:
onboarding
production
itself,
but
when
it
comes
to
things
like
header
validation,
it's
more
or
less
a
copy
of
the
code
that
runs
in
production-
and
this
is
also
one
of
the
main
roadblocks
to
implementing
something
like
cores
testing
support,
because,
ideally,
you
want
to
be
able
to
use
the
same
code
as
production.
A
So
that
testing
support
is
consistent
and
doesn't
diverge
and
also
just
makes
adding
test
support
a
lot
less
brutal
because,
as
you
may
know,
envoy,
open
source
envoy
gets
a
lot
of
changes
every
day
and
so
just
copying.
The
code
over
from
production
is
not
very
feasible
and
then,
finally,
I
think
the
ideal
state
of
the
tooling
would
be
having
true
black
box
testing.
A
What
this
would
mean
is
just
having
to
input
a
full
envoy,
config
versus
knowing
specific
route
inputs
for
your
unit
tests
beforehand,
and
this
way
users
could
simulate
request
behavior
without
having
to
come
up
with
the
certain
test
cases.
They
want
and
just
inspect
the
resultant
response.
And
you
know
this
could
be
used
hand-in-hand
with
the
existing
unit
testing
flow.
A
But
I
think
this
particular
approach
also
lends
itself
well
to
a
ui
based
way
of
testing
the
routing
table
in
which
you
can
just
input
some
parameters
and
see
what
the
resultant
behavior
is,
and
that
might
be
something
that's
more
intuitive
for
service
owners.
Instead
of
having
to
fill
out
the
values
for
very
specific
envoy
route,
configuration
fields,
so
yeah
definitely
a
lot
of
room
for
improvement
in
the
router
check
tool
and
we'd
love
to
chat.
If
you
are
interested
in
contributing
with
that.