►
From YouTube: Making Envoy Sustainable - Cynthia Coan, Datawire
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Making Envoy Sustainable - Cynthia Coan, Datawire
Adopting a new technology is never easy; but making that technology stick is even harder. This is because when adopting a new technology you face both technical AND social/cultural hurdles.
At the gateway and service mesh layers, breaking changes affect huge swaths, if not all, of your users and can consume a lot of development time.
In this talk, Cynthia will explain how you can approach building a maintainable API Gateway using Envoy that is set up to last for numerous years. She will delve into how to work with internal consumers and the Envoy community and how you can create internal filters/extensions for the gateway in a sane way. The end goal is that your gateway will continue to run and have the ability to adapt as we change other parts of our stack, moving from platform to platform and language to language.
A
Work
at
data
wire
I'm,
going
to
be
talking
about
making
envoy
sustainable
I,
got
a
lot
of
feedback
from
colleagues
that
this
title
was
a
bit
hard
to
understand
was
I
talking
about
like
changes
that
envoy
was
making
or
something
else
and
sort
of
clarify
I'd
like
to
re-engineer
my
title
right
here.
If
the
slide
is
going
to
go,
it
may
not
come
on.
A
A
Swap
in
the
backup
slide,
so
we're
going
to
set
a
scene
because
every
great
talk
requires
a
story,
so
let's
say
you're
a
normal
or
not
a
normal
feature,
engineer
working
on
your
service
and
all
of
a
sudden,
you
notice,
a
lot
of
problems
are
happening
at
the
network.
Layer,
right
and
lots
of
feature.
Engineering
teams
are
having
this
problem.
A
So
you
talk
to
your
bosses
and
you
say
you
know:
I
want
to
deploy
envoy
or
ISTE
o
or
whatever
in
order
to
solve
these
problems,
because
it
can't
easily
be
solved
another
way,
and
so
your
bosses
finally
say
to
you
yeah,
we'll.
Let
you
go
build
that
thing
and
since
you've
been
talking
about
it,
we'll
even
let
you
be
on
the
team.
However,
you've
never
been
on
a
platform
team
before
so
one
of
the
questions
that
you'll,
probably
first
ask
yourself,
is
like:
how
are
we
actually
going
to
build
this
service?
A
You
know,
what
are
we
going
to
optimize
for
most?
Is
it
security?
Is
it
availability?
What
are
we
doing?
I
think
one
of
the
core
tenets
that
any
platform
team
will
agree
is
one
of
your
core
things
that
you
want
to
do.
Is
you
want
to
make
it
easier
for
other
teams
at
your
company
or
other
companies,
to
really
build
amazing
services?
Whatever
your
definition
of
amazing
is
right,
you
want
to
abstract
away
this
core
component
and
make
it
easier
so
that
way
they
just
don't
have
to
think
about
it.
A
They
just
tell
you
what
they
want
to
happen
and
it
happens.
They
don't
have
to
know
all
the
internal
details
right.
You
want
to
help
keep
technical
debt
low
with
your
teams,
because,
right
in
this
one
system,
where
you're
deploying
on
voyeurs
to
you
or
whatever
you're,
impacting
every
team
who
uses
that,
so
you
really
want
to
make
sure
your
technical
debt
is
low.
You
want
to
ensure
that
envoy
is
frequently
updated
right.
A
You
want
to
make
it
easy
to
do
the
right
thing
hard
to
do
the
wrong
thing
so
that
way,
people
don't
shoot
themselves
in
the
foot.
In
other
words,
you
want
to
keep
up
with
envoy,
because
these
are
the
same
goals
that
envoy
has
and
even
though
envoy
doesn't
always
do
the
best
it's
constantly
getting
better
than
through
the
amazing
community.
That's
here
and
through
lots
of
other
contributions
from
other
people
right.
You
know.
A
The
new
adaptive
concurrency
stuff
is
a
perfect
example
of,
like
the
last
one
making
it
super
easy
to
do
the
right
thing
without
having
to
turn
as
many
knobs
all
right.
So
you
want
to
keep
up
with
envoy,
but
envoy
moves
very
fast.
How
do
we
actually
do
that?
So
maybe
you
start
writing
software.
You
get
a
team
to
work
with
you
right
and
you
start
rolling
out
software,
and
then
one
day
you
roll
out
of
bad
config,
update
or
a
bad
change.
A
It
happens
to
everyone
and
all
of
a
sudden
when
you
want
to
update
that
component
again,
they
say
we
don't
really
want
to
do
that
right
now,
we've
got
a
large
customer
using
you
know
this.
Is
this
right
now
they're
doing
an
evaluation?
We
don't
want
to
roll
out,
or
maybe
you
start
working
with
a
new
team
and
they
say
you
guys
are
rolling
out
updates
really
fast.
A
We
kind
of
want
you
to
pump
the
brakes,
and
so
why
are
people
not
rolling
out
our
software
we're
building
this
amazing
thing
we're
constantly
developing
new
features
and
new
fixes,
and-
and
you
know
this
is
a
very
critical
piece
of
infrastructure.
So
why
are
people
essentially
hitting
the
remind
me
a
day
later
update?
You
know
for
all
of
our
updates?
They
should
want
to
apply
this
because
it
makes
it
better.
There's
really
two
reasons.
A
The
first
one
is
time
right
and
the
second
one
that
at
least
according
to
me
is
poorly
defined,
breaking
change
policies,
or
just
one
in
some
cases,
and
so
I'd
like
to
break
these
down
and
really
understand
why
they
cause
problems.
So
why
is
time
a
problem
right,
we'd
like
to
think
teams
have
a
lot
of
time,
but
they
actually
have
a
lot
going
on
right.
They
have
their
own
application,
updates,
I
mean
think
about
how
often
a
node
module
updates.
A
For
example,
you
know
I
mean
everyone
likes
to
make
jokes
about
it,
like
these
updates
happen,
all
the
time
sometimes
they're,
pretty
massive,
sometimes
they're,
not
they
have
their
own
infrastructure
to
maintain,
even
in
like
a
serverless
environment,
they're
still
maintaining
you
know
their
logs
they're
monitoring
what
their
alerting
on
you
know.
They're.
It's
not
fully
automated
you're,
never
going
to
have
a
fully
automated
monitoring,
alerting
logging
setup
just
perfect
right,
so
they
are
going
to
have
their
own
infrastructure
to
maintain
they
have
their
own
support
issues
and
they
have
their
own
feature
work.
A
You
know
you
can
sometimes
see
future
work,
pinned
multiple
teams
of
a
product
down
for
months
at
a
time
because
they
just
need
to
roll
out
of
feature.
You
know
this
isn't
always
ideal,
but
it
happens
a
lot
right.
It's
not
because
the
team
is
unwilling
to
actually
apply
your
update.
They
just
don't
physically
have
the
time
to
do
it
because
they
have
all
of
this
other
stuff
going
on
breaking
changes.
I
state
are,
are
really
what
happens
more.
You
know
a
lot
of
people
like
to
think.
A
Oh,
this
is
just
used
semantic
versioning
semantic
versioning
gives
us
a
perfect
way
of
telling
whether
or
not
a
change
is
breaking
and
we've
never
incremented
a
major
version.
So
therefore,
we've
never
had
a
breaking
change.
Hey,
so
is
that
slight
just
going
to
be
used,
some
very
no
I
actually
think
semantic.
Versioning
can
hurt
us
in
some
cases
when
we
try
to
follow
it
too
closely,
and
this
is
because
semantic
versioning
got
one
to
two
things
wrong.
A
In
my
opinion,
it
states
that
you
can
tell
if
it
changes
breaking
based
purely
off
of
your
own
code
and
documentation,
and
it
also
states
that,
in
order
to
release
something
else,
you
have
to
release
a
minor
update
and
someone
actually
has
to
apply
that
update.
If
you
go,
if
you're
going
by
the
specs,
specifically,
which
we've
I've
definitely
seen,
some
people
do
so.
This
is
an
example
code.
It's
slide
code,
it's
absolutely
terrible.
Class
C
and
above
you
can
presume
as
a
library,
Class,
D
and
below,
are
you
know,
potentially
someone
using
your
library.
A
So
you
have
these
two
interfaces:
classes
a
and
B
and
C
provides
a
way
to
filter
based
off
of
A's.
So
this
happens.
Everything
is
going
great.
All
of
a
sudden,
as
the
library
team
decide,
you
know
what
we
should
really
be
able
to
filter
on
B's,
so
I
go
ahead
and
add
this
method
and
all
of
a
sudden,
even
though
the
user
hasn't
changed
their
code.
A
They've
now
they've
now
got
a
build
break
and,
what's
worse,
is
it's
kind
of
hard
to
tell
what
behavior
they
were
getting
before,
because
unless
I'm
documenting
when
I'm
adding
these
features,
you
know
the
user
doesn't
know.
Was
I
filtering
on
A's?
Was
I
filtering
on
B's?
Who
knows
now?
This
is
obviously
ridiculous
site
code,
but
you
can
imagine
how
this
can
happen
in
real-world
code.
So
sometimes
adding
a
method
isn't
safe
and
you
know
in
your
documentation.
You
may
not
have
realized
that
someone
was
using
both
a
and
B
at
the
same
time.
A
So
you
just
add
a
new
function.
Thinking
everything
is
fine,
but
someone
now
has
to
go
change
all
of
their
tests.
You
know
and
make
sure
that
they're
filtering
on
the
correct
thing
this
I've
actually
seen
happen
quite
a
bit.
We
like
to
think
that
changing
lines
is
generally
safe
and
most
of
the
time
it
is
unfortunately,
sometimes
people
right
tests
with
line
numbers
in
them,
either
accidentally
or
on
purpose,
and
as
much
as
I
would
like
to
say.
We
shouldn't
support
this
at
all.
A
This
is
still
something
that's
going
to
happen
from
time
to
time,
so
I
really
I
really
wouldn't
say.
We
would
want
to
support
this,
but
this
is
something
that
happens
that
you're
going
to
need
to
realize
seriously.
If
you
haven't
searched
through
all
of
your
code
bases
for
a
line
number
something,
do
it
and
you'll
probably
be
terrified.
A
My
previous
company,
we
had
live
grep,
and
so
I
was
able
to
do
this.
While
writing
this
talk
and
I
found
at
least
six
different
instances
of
us
matching
on
a
line
number
three
of
them
were
in
a
logging
utility,
but
they
were.
They
were
still
being
used.
What
about
this
case,
where
I
have
I've
replaced
a
with
method,
B
and
I've
marked
it
deprecated
for
a
year.
A
lot
of
the
cases
and
platform
teams
I
can
see
everyone
who's
using
my
library,
my
library,
isn't
necessarily
open-source.
A
So
I
can
see
everyone
who
is
using
my
library,
I
know
that
everyone
has
moved
off
of
method
a
and
onto
method.
B
right,
I
created
the
new
function,
just
in
case
there
was
some
undefined
behavior
that
broke.
That
I
wasn't
aware
of
you
know,
but
now
no
one
is
using
a
do.
I
really
need
to
call
a
major
version
break
even
though
I'm
removing
a
is
this
really
a
major
API
change,
if
no
one
is
using
it.
A
Basically
what
I'm
trying
to
say
is
context
matters
a
whole
heck
of
a
lot
and
everything
has
trade-offs,
so
you
should
never
take
anything
for
granted
and-
and
that
includes
being
able
to
assume
that
you
can
find
everything
based
solely
off
of
your
documentation.
If
you
don't
talk
with
the
people
who
are
using
your
library
and
more
often
proactively,
look
at
what
they're
actually
doing
with
it.
A
You're
going
to
end
up
missing
things
and
I
think
we've
all
experienced
the
pain
of
upgrading,
a
library
somewhere
and
even
though
it
was
a
minor
version
update,
it's
caused
weeks
of
pain
as
we
try
to
upgrade
because
they
accidentally
broke
us.
So
how
do
we
fix
this
as
a
platform
team
like?
What
can
we
actually
do
as
a
platform
team
right,
you
make
your
upgrades
easy,
like
really
easy
and
and
based
off
of
where
you
are
in
the
software
world.
A
But
if
every
single
one
of
your
upgrades
is
bad,
then
then
you
really
have
a
problem.
So
four
code
bases
you
own.
How
can
you
make
it
easy
enough
for
an
intern
to
do
well?
The
first
thing
you
need
to
do
is
like
I
said:
you
need
to
communicate
with
other
teams,
and
you
need
to
start
documenting
more
than
just
a
public
API
because,
as
it
turns
out,
teams
depend
on
more
than
a
public
API
right,
and
so
what
I
like
to
do
is
I
like
to
create
a
series
of
small
agreements
with
teams.
A
So
that
way
we
both
understand
exactly
what
is
being
agreed
upon
and
then
what
we
can
do
is
we
can
also
let
other
people
agree
to
an
agreement
that
has
already
been
made
so
when
I
want
to
make
a
change,
I
know
exactly
who
to
talk
to
right.
So
if
I
have
this
list
of
agreements
and
who
depends
on
them,
then
I
can
very
easily
communicate
with
those
teams,
and
the
only
requirement
for
an
agreement
that
I
have
is
that
it
must
have
a
test.
A
So
if
I
want
to
make
an
agreement
with
someone
that
we
will
support,
20,000
requests
per
second,
that's
great.
All
I
need
to
do
is
write
a
test
and
if
I
can't,
you
know,
afford
the
machinery
to
do
a
perfect
test
for
that.
Well,
then,
I
shouldn't
be
making
that
agreement,
because
I
can't
guarantee
it,
and
you
should
invest
seriously
to
make
sure
your
API
is
followed.
A
I
think
this
is
where
most
people
end
up
following
falling
short
short,
because
I
feel,
like
most
of
us,
probably
communicate
with
other
teams
and
document
more
than
a
public
API.
So
when
was
the
last
time
anyone
wrote
a
script
to
do
an
atomic
code
change
I
saw
Spotify
did
this
this
morning,
and
so
that's
great,
but
you
know
most
of
the
time
we're
dealing
with
yeah
Mille
an
envoy
right.
So
why
can't
we
just
write
a
small
Python
script
to
move
from
UML
a
to
yeahmobi,
most
cases
it's
possible.
A
If
you're
using
you
know
a
custom
control
playing,
you
may
be
able
to
just
do
it
in
your
control
plane
because,
as
it
turns
out
envoy,
does
a
really
good
job
most
of
the
time,
making
sure
that
there's
always
a
compatible
feature,
even
if
it's
defined
a
bit
differently
in
configuration.
So
why
can't
we
just
migrate
someone
for
them,
so
that
way
we
don't
have
to
have
a
break
or
if
we
do
have
to
have
a
break,
we
can
just
provide
them
a
script
that
works
on
well-formed
inputs.
A
So
we
can
say
here
you
know
we
know
you
have
this
Hamill
configuration.
This
is
changing
and
a
new
envoy
version.
So
here's
a
Python
script
that
you
can
just
run
right.
It
takes
a
couple
of
minutes
to
write
when
you're
just
dealing
with
Hamel.
If
you
have
something
more
complex,
obviously
that's
going
to
take
time,
but
it
gets
easier
over
time
and
then
you
can
use
custom
lint
rolls
to
actually
make
sure
that
the
configuration
works
but
is
also
doing
the
correct
thing
we
found.
A
A
So
we
were
in
a
very
highly
distributed
cloud
environment
where
we
weren't
running
multiple
pods
per
hosts
for
a
whole
bunch
of
crazy
reasons,
so
we
made
sure
that
you
weren't
trying
to
set
timeouts
of
one
second,
because
we're
never
going
to
be
able
to
make
a
time
out
of
one
second,
you
know
when
potentially
talking
across
AZ
when
you're
not
talking
locally-
and
we
know
you're
not
talking
locally.
So
really
writing
lint
rolls
to
validate.
Are
you
actually
trying
to
do
what
you're
trying
to
do
for
code?
You
don't
control!
You
know
it's!
A
You
can't
always
guarantee
that
it
will
be
easy
enough
for
an
intern
to
do
because
they
may
not
agree
to
the
same.
Every
upgrade
should
be
easy
or
they
may
just
have
a
different
idea
of
what
easy
is
right.
Easy
is
is
kind
of
relative.
So
what
can
you
do
you?
You
give
the
intern
time
right?
If,
if
something
can't
be
done,
because
it's
a
very
hard
problem,
you
give
them
time
right.
If
you're
testing
a
release
the
day
it's
released
or
the
day
and
RC
is
cut,
it's
probably
too
late.
A
If
there's
a
massive
change
that
you
know,
the
the
library
provider
didn't
think
was
important
enough,
but
you
depended
on
somehow
through
some
crazy
mannerism
right,
like
it's
going
to
take
you
time
to
figure
out
what
changed,
how
you
need
to
change
it
and
how
you're
going
to
expose
that
to
people
right
and-
and
you
don't
just
want
to
run,
build
tests.
You
really
want
to
run
a
whole
series
of
tests,
which
is
where
agreements
can
come
in
and
helpful
too
right
where
you
can
test
all
of
these
small
things.
A
So
you
can
validate
that
a
release
is
totally
fine.
You
don't
necessarily
have
to
fix
these
right
away
right.
You
can
just
track
envoy
master
and
say
you
know
what
there's
going
to
be
a
change
to
the
default
timeout.
So
that's
going
to
be
a
relatively
small
change,
so
I'm
just
going
to
push
that
off
and
let
teams
handle
it
later
because
I
know
it's
going
to
be
small
right,
whereas
if
it's
a
massive
build
break,
you
probably
want
to
fix
that
as
soon
as
possible,
because
then
it
prevents
you
from
even
seeing
new
failures.
A
You
don't
have
to
fix
it
right
away,
but
this
gives
you
the
time
so
that
way,
you
can
properly
notify
people
ahead
of
time.
Obviously
there
are
pros
and
cons
of
this.
The
essential
work,
though,
is
that
you
are
moving
work
from
all
of
the
teams
using
your
product
to
yourself
as
a
platform
team,
so
you
are
you're
giving
yourself
more
work,
but
you
are
lowering
the
amount
of
work
that
everyone
else
has
to
follow.
A
So
this
is
going
to
slow
down
your
velocity
as
a
platform
team
somewhat,
but
I
think
you'll
find
it'll
actually
raise
your
velocity
as
a
business,
because
these
teams
don't
have
to
worry
as
much
about
your
upgrades
you've
just
given
them
a
little
chunk
of
time
back.
So
you
keep
building.
You
think
your
tech
Ted
is
under
control
you're,
using
the
same
process
that
you've
always
used
for
feature
teams.
However,
you're
still
noticing
a
lot
of
tech
debt
issues
pop
up.
Why?
Why
is
this
happening?
A
Is
it
sort
of
falls
back
to
what
I
said
at
the
beginning,
tech
debt
is
affecting
more
than
your
team
in
your
customers.
Right,
you
are
affecting
tons
and
tons
of
people
with
this,
especially
if
Envoy
is
everywhere
envoy
in
a
sense,
is
sort
of
becoming
a
single
point
of
failure.
It's
not
really
a
single
point
of
failure,
but
it's
becoming
something
that
everybody
inherits
right
and
everybody
could
inherit
the
buck
right
and
remember.
A
They
won't
want
to
tell
you
what
you
know
new
agreements
they
want
with
this
new
feature,
or
they
won't
be
able
to
tell
you
yeah.
This
thing
is:
isn't
working
because
they
they
don't
think
anything
is
actually
going
to
change
right.
So
team
image
matters
a
whole
lot,
but
at
the
same
time
realize
that
tech
debt
will
always
happen.
So
I'm
not
saying
cut
out
all
tech
debt,
it's
totally
easy
because
you
can
never
cut
out
all
tech,
tech,
tech
debt
is
going
to
happen.
A
So
what
you
need
to
do
is
you
just
need
to
balance
it
better
I'm,
going
to
talk
specifically
about
things
that
you
can
do
during
planning,
but
if
you're
really
looking
for
things
that
will
work
at
your
business,
I
highly
recommend
talking
to
people
at
your
business
who
are
actually
aware
of
this
program.
Managers
are
usually
a
really
good
place
to
start
and
I
guarantee.
You
they'll
tell
you
about
like
50
different
buttons
in
JIRA
that
you
never
even
knew
existed.
That
will
probably
solve
all
your
problems
when
they're
not
broken
right.
A
So
when,
when
planning
the
the
first
thing
that
I
like
to
tell
everyone
is
keep
it
boring
right,
you
really
really
probably
don't
need
that
cool
thing
right
away.
For
example,
an
instructor
we
didn't
deploy
XDS
for
our
first
five
production,
apps
that
were
behind
the
Envoy.
We
were
able
to
do
it
managing
config
the
exact
same
way
we
always
have
been,
which
is
pushing
keys
into
console
and
then
having
a
local
script,
fetch
those
merge
it
into
one
final
yamo
file
and
then
use
envoy,
Hawtree
starter
right
control.
Planes
are
amazing.
A
They
do
amazing
things,
but
they're
also
incredibly
complex,
as
I'm
sure,
you've
gleamed
from
just
a
little
bit
of
these
talks
that
are
talking
about
just
a
little
bit
of
the
problems.
I
advanced
right
and
remember,
every
bit
of
technology,
you
add,
adds
complexity,
even
if
it's
a
managed
solution
right,
a
managed
solution
isn't
inherently
better.
Neither
is
managing
a
solution
yourself.
Everything
requires
trade-offs
and
the
trade-off
that
you
should
ideally
shoot
for
is
the
thing
that
makes
it
as
boring
as
possible
for
you
and
also
works
the
best
for
your
specific
consideration.
A
If
you
think
those
considerations
are
going
to
change,
then
you
should
probably
set
up
a
time
to
re-review.
You
know
what
you
decided
on,
because
you
may
come
to
a
different
answer
and
you
may
decide
to
start
planning
a
migration
right
and
another
thing
that
I
really
like
to
ask.
Whenever
someone
proposes
that
we
use
this
new
thing
is
they'll,
say
yeah,
we'll
just
you
know
like
throw
in
envoy,
Lua,
filter
or
envoy
wasm,
and
the
question
is:
do
we
really
need
all
the
all
the
functionality
of
that
right?
A
We
we
tend
to
say,
like
oh
yeah,
we
need
we
need
Kafka
to
do
all
this
great
things,
but
but
do
we
really
do
we
really
need
to
send
millions
of
messages
a
second?
Maybe
we
do
and
maybe
that's
the
best
choice
right?
Maybe
we
don't,
and
so
maybe
we
can
write
like
a
custom,
lint
rule
that
that
blocks
certain
functionality
of
features
right.
So
that
way
we
can
say
yeah
we're
not
ready
to
support
that
yet
because
that's
going
to
be
really
really
complex.
A
So
that
way
as
a
platform
team,
we
can
roll
out
features
as
we
plan
right
and
again,
like
I
said
you
could
you
can
lock
some
of
it
down
with
linters
or
whatever,
where
you
can
say:
yeah
I,
just
don't
want
people
using
that
bit
of
functionality
by
planning
features
ahead
of
time.
You
get
a
lot
of
benefits
because
you
are
actually
understanding
what
you're,
building
and
you've
you've
done
competent.
You
know
trade-off
diagrams
where
you
can
say
yes,
this
is
really
the
best
thing
for
us.
A
Planning
is
also
a
really
really
easy
way
to
just
get
stuck
right.
Sometimes
the
best
way
to
do
planning
is
to
work
with
the
team
on
experimental
stuff,
and
just
say
you
know
what
no
one
else
can
use
this,
but
this
one
team
and
we're
just
going
to
work
really
closely
with
you
right.
Don't
let
yourself
get
stuck
just
because
you
you
wanted
to
plan
everything
perfectly
sometimes
you're
going
to
plan
wrong.
That's
okay!
Don't
let
it
don't
let
it
get
you
stuck,
but
don't
always
you
know
go
without
so.
A
As
an
example
I'd
like
to
use
in
structures
code
base
which
I
can't
show
you
any
pictures
of,
it
was
supposed
to
be
open
source
by
this
talk,
but
it
wasn't
if
you
want
to
go
talk
to
the
people
who
are
actually
in
charge
of
it.
They're
sitting
on
the
fourth
row
back
right
there,
their
names
are
Christian
and
John
and
they
have
instructure
on
their
badge.
So
if
you
want
to
go
see
what
the
actual
code
base
looks
like
you
can
go
talk
to
them.
The
first
thing
we
did
was
we
mandated.
A
The
start
was
every
new
feature
required
a
tech
plan
test
plan
and
then
agreements
with
teams
like
I
said
we
did
these
small
agreements
with
teams
and
they
were
really
easy
to
manage
as
long
as
we
kept
up
with
them
right
that
first,
like
getting
everyone
to
understand
this
Agreement
saying
like
that
was
a
huge
hurdle,
but
now
that
we've
been
doing
it,
it's
really
easy
to
keep
it
going.
The
tech
plan
is
where
we
sort
of
decide
on
like
yes,
Redis
really
is
the
best
for
us
over
memcache
Dee.
A
You
know-
and
maybe
that
is
because
we
don't
have
time
to
really
look
into
it
and
on
voice
supports
Redis
right
out
of
the
box,
really
well
right.
It
allows
us
to
do
that
and
then
the
test
plan
is
where
we
document
all
success/failure
fuzz
and
chaos
testing
scenarios.
Those
were
what
we
want
every
feature
that
we
roll
out
to
have.
So
we
do
that
we
have
a
linter
that
powers
automated
code
reviews
and
enforces
best
practices.
So
we,
as
the
platform
team,
don't
have
to
review
anybody's
configuration.
A
We
can
just
run
a
linter
and
as
soon
as
that,
linter
has
given
it
a
plus-one,
we
don't
care
where
they
put
it.
We
don't
care
if
they
put
it
in
production
right
away.
We
don't
care
if
they
put
in
development
and
testing
it
doesn't
matter.
You
roll
it
out.
It's
your
configuration.
This
linter
is
going
to
validate
that
it
is
correct
according
to
us,
so
this
isn't
just
Envoy
mote
validate,
which
is
one
part
of
the
linter.
A
We
also
do
things
like
make
sure
that
you
aren't
doing
a
potentially
bad
reg
X
or
we
make
sure
you're
not
using
Lua,
because
we
don't
want
to
support
Lua
at
this
point
in
time.
We
do
all
of
that.
So
that
way,
we
can
ensure
that
as
long
as
this
linter
is
run,
which
doesn't
require
any
manual
interaction,
this
code
is
good.
A
A
So
that
way
we
could
keep
watching
for
failures
if
it
was
just
a
test
failure
we
just
logged
about,
and
in
slack
so
that
way
we
knew
there
was
work
to
do,
but
again
we
only
had
to
actually
fix
it
if
it
was
a
bill
break
which
didn't
happen
too
often,
so
it
was
really
helpful
there.
We
kept
all
of
our
meeting
heavy
days
too
one
day.
A
lot
of
this
requires
like
reviewing
with
teams
constantly
right.
You
constantly
want
to
review
these.
These
things
that
you're
setting
up
with
teams.
A
You
constantly
want
to
review
the
code
that
they're
pushing
into
production
to
make
sure
you
don't
need
to
add
rules
to
your
linter
right.
All
of
this
takes
time,
and
we
just
put
it
on
one
day
of
the
week.
So
that
way
it
was
really
easy
to
plan
around,
because
the
rest
of
the
days,
you
could
essentially
avoid
context.
Switching
you
didn't
have
to
be
like.
Oh
it's
meeting
mode
now
we
just
had
it
on
one
day
we
tracked
all
envoy
deprecations
in
config
separately
envoy
actually
has
a
flag
for
this.
A
Now
that
makes
it
really
easy.
It
will
terminate
the
process
if
there's
a
deprecated
flag
that
is
is
run
into
so
we
actually
can
just
use
that,
and
the
essential
process
here
is
as
soon
as
the
config
option
was
deprecated
we're
actually
able
to
completely
migrate
away
from
it.
Using
the
whole,
we
would
write
a
script
migrate,
all
of
our
gamma
over
automatically
and
then
just
introduce
a
lint
rule
to
make
sure
no
one
else
added
it
back
in
and
if
someone
did
try,
the
lint
rule
would
explain
why
we
don't
support
that
way
anymore.
A
A
This
could
work
as
a
wiki
doc,
but
we
wanted
to
formally
specify
it
due
to
a
couple
reasons
that
are
sort
of
internal
to
instructure,
but,
like
I
said,
this
was
really
just
trying
to
communicate
to
teams
like
we
want
to
build
more
reliable
services,
but
in
order
to
do
that,
we
need
to
know
how
you're
using
our
platform
more
concretely.
So
this
is
what
we
expect
from
you,
and
this
is
what
you
should
expect
from
us.
That's
really
all
I
have.
A
If
you
have
any
questions
about
it,
you
can
ask
me
on
one
of
those
links
there.
Github
would
probably
be
a
weird
place
to
ask,
but
I
mean
I'm
sure,
there's
a
way
you
can
figure
out
how
to
do
it
and
I
have
four
minutes,
because
I
talked
really
fast.
So
if
anybody
has
any
questions
about
concrete
specifics
or
or
anything
like
that,
I'd
be
more
than
happy
and
without
that,
and
without
that
I
hope.
At
least
I
got
you
to
think
about
some
things
and
sustainability
that
you
didn't
think
about
before
Thanks.