►
From YouTube: Safely deploying a 100K line Envoy YAML configuration to production - Jyoti Mahapatra, Lisa Lu
Description
Safely deploying a 100K line Envoy YAML configuration to production - Jyoti Mahapatra, Lisa Lu
Have you ever caused a production incident due to an Envoy misconfiguration? You’re not alone! This talk is about how Lyft has built guardrails to prevent such failures. The presenters will share their experience operating Envoy configurations at scale. They will explore the challenges around handling constantly changing cluster and routing configurations and the tools used to guarantee accuracy and consistency in those changes. These tools empower service owners less familiar with Envoy to make configuration changes independently and quickly without approval barriers. This talk will introduce the audience to various Envoy configuration testing strategies:
Validating behavior for thousands of routes to avoid blackholing traffic
Auditing and safely removing unused routes and clusters
Safely deprecating fields between Envoy versions
Validating Envoy’s static and realtime configurations
A
Hi,
everyone
welcome
to
our
talk
on
how
to
validate
envoy
configs
at
scale
we'll
be
going
over
strategies
for
automating
the
config
validation
and
testing
process,
so
that
your
service
owners
can
iterate
quickly
and
independently.
This
will
also
prevent
your
envoy
operators
from
being
bogged
down
with
code
reviews.
A
A
Meanwhile,
the
routes
and
clusters
which
are
stored
in
s3
are
fetched
by
the
control
plane,
which
creates
the
respective
route
configuration
and
cluster
objects
to
send
via
xds
to
the
sidecars.
As
such,
there
are
two
main
points
at
which
we
have
to
validate
the
format
and
content.
The
configs
the
first
such
place
is
in
the
bootstrap
config,
which
ensures
that
the
sidecar
is
able
to
run
properly
and
start
up
without
crashing.
A
We
validate
these
before
sending
them
to
the
control
plane,
because
we
want
to
avoid
the
control
plane
needing
to
do
any
validation
as
it
gets
the
route
and
cluster
information
in
real
time,
and
rather
frequently
as
the
number
of
services
and
routes
that
lift
has
grown
on
some
side
cars.
We
have
configurations
that
are
upwards
of
a
hundred
thousand
lines
of
yaml,
which
makes
maintenance
and
modification
extremely
complex
and
risky
I'll
describe
some
of
the
issues,
in
fact
that
we've
encountered
over
the
last
year.
A
Firstly,
as
a
growing
company,
we
have
new
developers
joining
every
week,
and
business
expansion
also
means
that
new
services
are
being
spun
up
to
cater
to
new
scenarios.
Quite
frequently,
this
means
more
code
review
requests
from
developers
who
aren't
familiar
with
envoy
and
want
to
know
if
their
change
will
do
what
they
want
it
to
do.
While
it's
great
that
we're
having
more
and
more
development,
it's
not
so
great
that
the
on-call
burden
has
increased.
It
means
the
on-call
spends
more
time,
triaging
these
tickets
and
having
to
stare
at
yaml.
A
Another
issue
is
human,
con
human
error
in
configs,
with
such
large
yaml
files,
it
isn't
feasible
for
catching
everything.
For
example,
one
common
source
of
bugs
is
how
envoy
does
first
order
matching
for
its
routes,
which
means
that
in
a
huge
list
of
routes,
it
might
be
easy
for
a
developer
or
a
viewer
to
miss
a
route
that
captures
all
the
traffic
that
is
actually
intended
for
a
new
route.
A
In
response
to
such
errors,
we
actually
mandated
a
code
review
from
the
networking
team
for
all
routes.
At
one
point,
this
quickly
became
the
bottleneck
for
iteration,
because
our
networking
team
is
about
six
people,
and
while
we
were
the
guardians
of
these
route
changes
for
a
while
and
tried
to
eyeball
every
change
going
in
this
just
wasn't
proving
to
be
sustainable
and
was
slowing,
iteration
and
rollouts
for
the
rest
of
the
company.
A
Next
reason
is
that
changes
are
hard
to
test.
Actually,
this
was
a
main
reason
for
the
first
issue
that
I
highlighted,
which
is
that
service
owners
weren't
sure
how
to
validate
their
changes,
which
is
why
they
wanted
to
reach
out
to
the
networking
team
in
the
first
place
prior
to
automating
our
testing.
It
was
really
difficult
to
validate
changes.
You
had
to
have
an
end-to-end
setup
with
setting
up
an
envoy,
sidecar
and
sending
requests
to
see
if
your
config
changes
worked.
A
A
A
So
in
this
scenario,
ideally
in
the
top
row,
you
would
have
rats
pointing
to
your
service
or
cluster,
but
when
it
comes
to
new
service
launches
or
service
deprecations,
sometimes
developers
only
address
the
route
definitions
and
forget
the
clusters,
or
vice
versa.
So,
for
example,
in
the
second
row,
you
see
that
there's
no
route
definition
pointing
to
the
service,
and
this
could
mean
that
the
developer
didn't
leave
any
routes,
which
means
the
cluster
definition
isn't
being.
B
A
While
this
cluster
is
not
accessible
by
outside
users,
your
control
plane
is
still
creating
unnecessary
resources
here
and
sending
them
to
side
cars.
In
addition,
it
just
creates
tech
da
and
can
make
reading
through
configs
confusing,
since
the
service
is
presumably
no
longer
being
used
or
isn't
currently
being
used,
and
then
in
the
third
scenario,
you
have
accessible
routes
that
are
pointing
to
a
cluster
that
isn't
configured
or
doesn't
exist.
A
Next,
to
validate
the
bootstrap
config,
we
make
use
of
a
couple
of
open
source
tools.
The
first
one
I'll
discuss
is
the
validation
server.
This
refers
to
running
the
envoy,
binary
in
validation
mode,
which
you
can
set
via
the
command
line
flag
mode,
and
what
this
does
is
it
takes
your
binary
and
it
takes
your
bootstrap
config
and
it
tries
to
boot
up
envoy
without
starting
workers,
but
it
will
go
through
the
server
initialization
process
as
far
as
it
can,
and
if
there
are
no
errors
it
will
exit,
it
will
exit
successfully.
A
The
next
tool
we
use
is
the
config
load
check
tool
and
you
can
see
the
docs
on
the
left.
This
is
a
standalone
binary,
which
you
can
run
on
all
of
your
bootstrap
configs
and
it
basically
checks
that
all
the
values
in
the
fields
are
valid,
as
defined
by
the
proto,
and
that
the
schema
is
a
valid
envoy
schema.
A
We
run
both
of
these
checks
on
all
of
our
pull
requests
in
ci
and
that
just
makes
reviewing
these
changes
so
much
easier
because,
as
the
reviewer,
you
already
know
that
the
config
is
a
valid
envoy
config.
And
so
you
only
need
to
ensure
that
the
change
does
what
it
should
I'll
now
hand
it
over
to
jyoti
to
discuss
the
router
check
tool
and
its
various
functionalities.
C
Thank
you
lisa.
Next,
I'm
going
to
talk
about
the
router
check
tool.
The
part
of
configuration
in
most
flux
over
the
last
few
years
have
been
modifications
to
the
routes
routes
work
in
a
very
sensitive
way.
The
routing
engine
runs
an
incoming
request
over
a
set
of
routing
rules
and
the
first
one
to
match
wins.
This
becomes
risky
in
a
high
flux,
change
scenario.
C
A
route
mistakenly
added
at
the
top
of
the
list
can
black
hole.
All
traffic
can
cause
incidents
envoy
has
a
router
check
tool
executable.
This
exercises
the
routing
engine
and
runs
it
over
the
routing
configurations.
Given
it
lets
us,
add
unit
tests,
check,
field
applications,
add
code
coverage,
constraints,
test,
complex
routing
configurations
based
on
header,
match,
run
time
and
weight
cluster
configs
we'll
look
at
how
to
write
the
tests
and
in
test
for
code
is
a
well-known
pattern.
C
There's
a
subject
under
test
defined
by
test
name,
some
setup
and
assertions.
Similarly,
routing
tests
have
it
name,
a
set
of
input,
conditions
and
a
set
of
assertions.
The
tests
always
run
the
input
route
through
the
routing
engine
and
compares
the
assertions
with
the
actual
results.
Let's
follow
an
example.
C
C
C
C
C
C
All
this
was
exciting,
but
the
tool
did
not
have
a
way
of
enforcing
adding
tests.
Developers
were
able
to
work
around
the
stop
gap.
Phase
of
test
enforcement.
We
decided
we
needed
a
way
to
add
code
coverage
to
the
tool.
There
are
two
ways
of
adding
coverage.
First,
one
is
shallow
and
just
checks
that
there
is
a
test
for
every
routing
rule.
The
second
one
is
more
strict
and
enforces
writing
all
assertions
for
each
test.
C
C
C
C
C
Constantly
updating
the
on-way
binary
comes
at
a
cost.
There
are
fields
getting
deprecated
all
the
time
and
having
them
in
the
system
introduces
spectate
and
a
higher
migration.
Time
later
we
added
a
deprecated
field
check
in
the
tool
so
that
it
fails
whenever
it
observes
a
deprecated
field.
We
quickly
resolve
it,
since
usually
it's
one
or
two
fields
at
a
time
and
doesn't
stay
long
enough
to
become
risky.
C
C
A
A
The
code
over
from
production
is
not
very
feasible
and
then,
finally,
I
think
the
ideal
state
of
the
tooling
would
be
having
true
black
box
testing.
What
this
would
mean
is
just
having
to
input
a
full
envoy,
config
versus
knowing
specific
route
inputs
for
your
unit
tests
beforehand,
and
this
way
users
could
simulate
request
behavior
without
having
to
come
up
with
the
certain
test
cases.
They
want
and
just
inspect
the
resultant
response.
And
you
know
this
could
be
used
hand-in-hand
with
the
existing
unit
testing
flow.
A
But
I
think
this
particular
approach
also
lends
itself
well
to
a
ui
based
way
of
testing
the
routing
table
in
which
you
can
just
input
some
parameters
and
see
what
the
resultant
behavior
is,
and
that
might
be
something
that's
more
intuitive
for
service
owners
instead
of
having
to
fill
out
the
values
for
very
specific
envoy
route
configuration
fields,
so
yeah,
definitely
a
lot
of
room
for
improvement
in
the
router
check
tool
and
we'd
love
to
chat.
If
you
are
interested
in
contributing
with
that.
C
C
C
C
C
C
C
D
B
B
To
communicate
with
the
other
services
so
as
a
result
like
lots
of
clusters
and
route
information
for
them,
but
as
a
number
of
services
has
grown,
we
just
have
a
lot
of
user-defined
routes
and
a
lot
of
sort
of
default,
config
content
that
gets
added
to
every
services
config.
So
I
think
over
time,
and
it's
just
grown
a
lot
in
size.
C
Yeah
we
struggle
with
v2
and
v3
quite
a
bit,
but
we're
about
the
hump.
I
guess
the
next
question
is
about
how
are
they
stored
in
the
guitar?
We
have
a
different,
a
separate
repo
just
for
storing
the
routes
so
that
people
can
add
and
change
routes.
As
for
their
will,
we
have
different
mechanisms.
We
have
json-based
conflicts
that
we
parse
and
create
xds
requests
out
of.
We
also
have
conflicts
which
replicate
exactly
the
routes
and
we
just
deserialize
them
and
pass
it
on
to
the
side
cars.