►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Envoy at Square - Michael Puncel + Snow Petteren, Square, Inc.
This talk will cover:
- motivation for Square's move to Envoy
- design considerations that went into our implementation and rollout process
- particulars about our Envoy setup in our infrastructure (unix sockets, no TLS in apps, etc)
- how we supported legacy client behavior in Envoy
- how we streamlined the migration process to minimize interruption to app engineer road maps
A
Sorry
about
that
hello,
everyone,
we're
really
excited
to
talk
to
you
today
about
how
envoy
has
been
sari.
How
Square
has
been
using
envoy
as
the
basis
for
our
service
mesh
I'm
Michael
pencil
I've
been
at
square
for
about
five
years
for
the
first
four
of
those
I
was
working
primarily
on
our
deployment
systems,
the
most
recent
one
being
called
p2
and
then,
for
the
past
year,
I've
been
on
our
traffic
team
working
on
exclusively
this
project
growing
out
our
own
voice
service,
mich,
hi,.
B
B
So
one
of
the
big
things
for
us
was
that
we
really
needed
to
simplify
the
way
a
client
and
server
libraries'
worked
when
it
came
to
service
service
traffic
score
has
three
major
languages,
several
minor
ones,
and
you
get
a
lot
of
different
languages
and
frameworks.
They
have
to
give
up
the
bay
and
they
can
feature
parity
becomes
a
huge
problem.
B
We
also
wanted
to
really
abstract
a
way
in
which
our
show
from
app
developers,
because
previously
they
hadn't,
know
a
lot
about
a
lot
of
important
details.
They
never
do
their
job
well,
so
we
really
want
to
make
sure
that
they
just
keep
doing
what
they're
good
night,
the
Pelican
alert
products.
And,
lastly,
we
really
wanted
to
make.
B
So
Square
runs
primarily
in
our
own
data
centers.
So
we
have.
We
have
a
few
college
expenses
as
well,
but
primarily
our
my
data
centers.
This
is
oldest
bare-metal.
We
run
our
own
kubernetes
light
deployment
system
somewhere
in
the
sense
that
it
has
pods
that
you
can
run
multiple
things
in
the
same
pod.
You
come
here
it's
like,
but
it
does
not
use
containers.
It's
all.
Just
multi
tenant
hosts
the
only
isolation
between
multiple
apps
on
a
single
host
is
user
permissions
and,
like
some
c
groups,
so
there's
no
net
renamed
network.
B
No
spacing,
then
we
MPLS
between
everywhere
for
a
while
now
you're
pretty
strict
security
requirements
as
a
famous
company,
so
this
is
ban
and
requirement
previously
in
continues
to
be
so
with
with
anyone.
And
lastly,
our
we've
been
running
we're
running
on
mostly
Sango
6
we're
slowly
moving
toward
seven,
but
it's
it's
a
long
process.
So
we're
we're
working
on
it.
A
Again
now
we'll
get
to
the
main
part
of
our
talk
where
we'll
go
over
a
few
parts
of
our
design
of
our
service
mesh
that
we
think
are
particularly
interesting
to
talk
about.
So
the
first
thing
that
you'll,
probably
think
about
if
you're
rolling
out
a
service
mesh,
is
what
your
control
plane
will
look
like.
So
we
decided
to
go
with
a
centralized
control
plan
because,
as
snow
mentioned,
that
really
helps
you
make
changes
very
quickly
or
you
can
deploy
a
single
server
and
like
really
change
the
way
that
all
the
envoys
are
wired
together.
A
We
also
went
to
with
the
decision
to
build
our
own
control
plane.
The
main
reason
for
that
is
because
we
already
had
a
fairly
sophisticated
mechanism
for
how
traffic
is
routed
at
square,
and
so
we
felt
like
we
really
needed
control
over
all
the
knobs.
That
envoy
has,
in
order
to
make
things
look
very
similar
to
the
way
they
looked
before,
but
moving
on
to
envoy.
A
So
our
control
plane
looks
a
little
something
like
this.
We
use
the
Java
control
plane,
library,
which
is
the
same
as
the
go
control
plane
one
if
you're
familiar
with
that,
it's
on
the
omelet
proxy
github,
the
bottom
part.
There
is
the
XDS
server
which
was
discussed
in
the
last
talk.
So
that's
basically,
the
actual
G
RPC
server
that
handles
all
the
version,
acknowledgement
and
all
that
kind
of
stuff.
A
Then
there's
a
snapshot
cache,
which
is
a
place
where
you,
after
you
generate
a
given
Envoy
configuration
you
save
it
into
the
cache
where
it
can
be
used
later,
I'll
walk
through
how
exactly
that
works
in
a
second
and
then
the
final
piece
is
what
we
call
the
snapshot
Factory,
and
so
that's
some
code
that
we
wrote
that's
very
specific
to
the
way
squarer
works
and
it
generates
square
specific
envoy
configuration
on
the
right.
We
have
the
two
primary
data
sources
that
are
control
plane,
uses
to
generate
config.
A
We
have
an
Access
Management
registry
that
knows
about
every
micro
service
we
deploy.
It
also
knows
which
micro
services
talk
to
which
others.
So
it
knows
the
dependency
graph.
It
also
knows
the
identities
of
each
cert
that
the
clients
will
be
using,
so
we
can
use
that
for
mt
LS,
and
then
we
have
a
service
discovery
database.
That
of
course
tells
us
where,
on
the
network,
certain
pods
are
available.
It
also
has
a
lot
of
traffic
shaping
information.
That
snow
is
going
to
talk
about
later
in
more
detail.
A
So
when
a
non-void
boots
up,
let's
call
it
foo
envoy,
so
it's
associated
with
the
app
foo
it'll
make
an
XDS
request,
as
described
in
the
last
talk
where
it
doesn't
yet
have
a
version
that
it
can
act.
So
it
sends
an
empty
version.
Are
the
xes
server?
Will
then
look
in
the
snapshot?
Cache
or
do
I
have
a
config,
that's
suitable
for
foo.
It
doesn't
so
it
calls
a
callback
in
our
code.
A
We
then
pull
in
information
from
those
two
databases
generate
a
non-void
configuration
for
foo,
save
it
in
the
snapshot
cache
and
then
the
XDS
server
is
able
to
respond
to
that
envoy,
and
we
think
that
it's,
it's
potentially
interesting,
that
we're
doing
lazy
generation
of
snapshots
here.
That's
really
cool,
because
it
means
you
don't
have
to
pre-compute
every
possible
permutation
of
Envoy
configuration.
You
might
need
up
front,
and
it
also
means
that
we
can
pass
up
a
ton
of
information
in
the
metadata
of
the
XDS
requests.
A
And
so
we
can
do
things
like
toggle
features
on
and
off
in
the
client
where
we,
you
pass
your
envoy
some
configuration
when
it
boots
and
then
that
will
later
be
used
in
the
control
plane
to
decide
which
features
to
turn
on
then,
if
something
changes.
So
the
envoy
re
requests
for
version
one,
because
it's
last
seen
version
one.
If
something
changes
in
our
databases,
our
snapshot
factory
can
go
ahead
and
create
a
new
snapshot.
Configuration
with
a
new
version
and
the
XDS
server
will
respond
with
that.
A
And
then
our
fou
envoy
re
requests
to
show
you
a
little
bit
what
the
control
plane
library
gets
you
for
free.
If
we
have
a
second
foo
envoy,
startup
its
able
to
just
look
in
the
cache
and
respond,
and
so
our
snapshot
factory
didn't
even
get
invoked,
didn't
have
to
do
anything.
So
it's
very
low
overhead.
If
you
have
a
lot
of
the
same
configurations
being
served
over
and
over
and
then
when
we
get
an
envoy
with
a
different
identity
bar,
we
could
kind
of
go
through
that
original
flow.
A
Where
you
generate
a
bar
configuration,
oh
and
then,
when
the
foo
envoys
drop
off
the
network
and
stop
requesting
eventually
a
garbage
collection,
routine
and
the
Java
control
plane
goes
and
clears
out
that
entry
in
the
cache.
So
that
again
helps
you
not
have
to
think
about
what
kinds
of
configs
you
need
at
a
given
moment.
You
can
just
generate
whatever
on-demand
or
whatever
requests
come
in
and
then
eventually
they'll
get
cleaned
up.
If
they
stop
being
used.
B
Alright,
so,
in
addition
to
the
control
thing
itself,
we
felt
that
we
needed
to
do
some
additional
work
to
make
it
safer
for
all
no
changes
to
the
control
plane.
So
previously
we
were
discovering
the
control
plane
itself,
just
using
DNS
and
like
randomly
picking
a
host
of
connect
to
which
meant
that,
when
were
deployed
a
new
version,
in
this
case
the
bad
version.
Probably
no,
these
English
might
only
restored
van
link
boards
connect
to
the
bad
control
plane
and
they
bullets
crash,
like
whatever
happens,
was
there's
no
employee.
B
So
we
thought
this
was
like
either
being
pretty
risky
whenever
a
true
look
changes,
especially
it
changes
and
I
could
not
easily
be
controlled
within
the
app
such
as
like
a
high-level
configuration
changes.
So
we
decided
to
put
a
proxy
in
front
of
it
because
when
it's
all
put
more
process
in
the
system-
and
what
does
it
allows
us
to
do-
is
that
when
you
deploy
you
for
sale
about
prefer
playing
the
proxy
consults
a
feature
service
edge
flight
service?
B
The
use
it
because
it
uses
that,
alongside
the
metadata
and
orestes
--and--,
which
identifies
the
the
my
so
like,
which
which
akka
runs
next
to
you
all
the
soap
and
using
that,
alongside
with
the
role
I
play,
the
role
of
Elias
and
the
future
like
service
I,
can
decide
which
control
planet
connects
to
someone.
It
does.
A
B
See
this
boat,
we
can
flip
it
back
just
on
water
flying
the
future
flight
and
we
can
move
like
the
but
no
deploy
good
version
and
then
slowly
move
over.
So
this
is
really
cool
because
it
gives
us
essentially
granblue
deploys
of
the
control
plane
and
well
we're
still
in
the
process
of
broadens
out.
It
has
given
us
a
lot
more
confidence
in
making
bigger
changes
to
the
control
plane
and
ensuring
that
doesn't
take
down
the
entire
company.
A
Another
piece
of
rolling
out
of
service
mesh
is
actually
migrating
and
this
kind
of
ends
up
being
the
most
time
intensive
part.
It's
it's
hard
of
harder
than
the
engineering
pieces,
so
at
square.
We
have
about
500
different
micro
services
that
are
deployed,
and
we
were
three
ICS
working
on
this.
So
it
was
really
key
that
it's
as
easy
as
it
could,
possibly
as
we
can
possibly
make
it
to
my
greatest
service.
So
the
first
thing
that
developers
do
when
they
want
to
add
on
way
to
their
service
is
they
run
a
command.
A
We
give
them
called
traffic
CLI
ad
envoy,
and
that
does
basically
all
the
infrastructure
things
that
need
to
go
into
it.
It's
going
to
be
different
for
everyone,
but
the
main
thing
it
does
is
actually
deploy
envoy
into
your
pod.
There's
like
ports
that
open
things
get
registered
in
various
databases,
just
everything
that
you
need
to
do
to
get
it
deployed.
A
We
also
provide
client
and
server
libraries
in
all
the
major
languages
that
use
it's
in
use
at
square
and
actually
the
server
libraries
kind
of
just
talk
to
the
deployment
system
and
enable
the
listener
you
get
traffic
in
from
envoy.
When
you
do
your
deploy
so
that
traffic
seal
I
add
on
voice
step
is
all
you
have
to
do
to
then
be
ready
to
receive
traffic
ingress
through
your
envoy,
which
is
pretty
cool.
That's
it
takes
like
five
minutes
for
someone
to
do.
A
Why
we're
using
UNIX
sockets
later
and
you
format
your
hosts
headers
for
your
HTTP
requests
as
production
or
staging
dot,
the
name
of
the
server
app.
You
want
to
talk
to
square,
and
so
that's
basically
exactly
the
information
that
developers
know
when
they're
developing
an
integration.
They
don't
want
to
know
anything
about
load,
balance
or
host
names
or
IP
addresses
or
whether
they're
on
the
cloud
or
on-premises.
A
They
know
what
they're
trying
to
talk
to,
and
another
thing
that
makes
migration
possible
for
us
is
preserving
the
way
that
our
existing
service
discovery
and
traffic
shaping
stuff
works
and
we're
going
to
talk
about
that
in
detail
later.
But
you're
gonna
live
in
a
world
where
you're
halfway
on
Envoy
and
halfway,
not-
and
if
that's,
if
things
are
really
operating
very
differently,
that's
going
to
be
a
challenge,
and
then
we
also
have
a
feature
flag
that
we
call
the
VIP
to
mesh
feature
flag
to
show
you
what
that
does.
A
We
suppose
we
have
a
ring
of
three
micro
services
that
all
the
circular
dependency
on
each
other
Moe
Larry
and
curly.
They
talk
to
each
other
through
a
VIP
which
is
a
virtual
IP,
that's
sitting
on
a
hardware,
a
load
balancer
somewhere
and
let's
say
that
Mo's
team
finally
has
some
time
to
do
their
own
voicemail
Gration.
So
they
do
traffic
CLI
at
envoy
there
they
have
envoy
deployed
in
their
pod,
but
we
haven't
done
anything
yet
to
actually
change
the
flow
of
traffic.
A
So
it's
still
going
from
Moe's
VIP
into
the
app
and
the
app
is
still
talking
to
Curly's
VIP.
So
the
the
naive
thing
to
do
here
would
then
be
to
have
them
flip.
Their
client
configuration
to
start
using
the
mesh,
but
there's
no
envoy
on
currently
side
yet
because
they
haven't
gotten
around
to
it.
So
what
we
ended
up
doing
is
we
have
our
control
plane
by
default.
You'll,
your
envoy
will
send
traffic
to
that
old
VIP.
A
That
we
know
is
working
because
we're
already
using
it
and
then
later
on,
when
Curly's
team
gets
a
to
deploying
their
envoy,
they
can
flip
a
feature
flag.
That
says:
hey
I'm,
ready
to
take
traffic
ingress
through
my
envoy
and
then
our
control
plane
will
tell
all
the
clients
to
go
through
that.
So
this
is
really
nice,
because
it
means
that
Moe's
team
can
finish
their
envoy
migration
where
they
deploy
their
their
envoy.
They
flip
all
their
client
configuration
and
then
they're
done
as
far
as
they're
concerned.
A
They
can
go
back
to
the
to
writing
their
products
and
curlies
team
later
on
sometime
at
some
point,
we'll
flip
a
flag
that
actually
turns
on
the
full
mesh
and
then,
when
you
continue
this,
you
eventually
get
rid
of
all
the
VIPs
and
have
a
full
service
mesh,
which
is
what
we're
trying
to
get
to
one
more
thing:
that's
pretty
important
if
you're
using
envoy
is
and
you're
using
TLS
internally
is
being
able
to
offload
that
into
the
proxy.
So
you
want
to
be
able
to
take
advantage
of
all
the
layer.
A
7
features
that
envoy
has
like
tracing
and
metrics
and
request
level
load
balancing,
and
it's
also
nice
to
be
able
to
enforce
TLS
in
the
proxy
layer.
So
you're
not
depending
on
client
like
app
configuration
to
do
it
and
for
us,
as
Snow
mentioned
earlier,
we're
not
using
containers
with
Network
namespaces.
Yet
so
that's
why
we're
using
UNIX
sockets
for
egress
traffic.
A
So
the
app
is
talking
unencrypted
HTTP
to
the
proxy
into
a
UNIX
socket,
that's
protected
with
file
system
permissions,
and
so
that
lets
us
kind
of
get
the
behavior
of
having
a
private
network
on
loopback
without
actually
being
able
to
do
that.
Yet
and
then
we
are
still
having
Apps
server
apps,
enforce
the
layer,
7
access
control.
So
if
you
want
/foo
2
to
be
allowed
for
a
certain
client
but
not
slash
bar
the
apps
still
have
to
enforce
that.
A
B
Right,
so
you
don't
want
to
talk
about
some
of
the
traffic
shaping
and
routing
features
seven
years,
and
most
of
these
always
have
been
in
order
to
so
I
replicate
the
existing
purpose,
a
framework
that
we
had
again
because,
that's
my
might
say,
we
want
to
make
sure
that
people
can
in
this
transitionary
world
without
having
to
like
know
who's,
calling
you
so
make
everything.
Look
the
same.
So
first
I
want
to
go
quickly
over
like
what
the
service
is
covering.
All
that
is
not
good
square.
B
Where
you
have
the
app
will
just
like
the
routable
target,
and
then
your
channel
contains
multiple
deployments
corresponds
like
a
cluster
of
clothes.
So
when
you
would
target
in
hacked,
you'll
hit
one
of
their
deployments
and
the
Palomas
can
be
the
same
or
they
can
be
select
a
different
different
configuration.
They
were
running
a
different
version,
but
politically
really
good
thing
up.
Additionally,
each
deployment
exists
within
a
data
center
which
describes
their
physical
location
in
the
world,
and
so
was
this.
B
It
was
important
for
us
to
ensure,
like
traffic,
always
prefers
the
most
as
possible
data's
him.
So,
in
the
case
where
you
have
and
we're
running
and
one
datacenter
22
different
deployments,
this
is
one
out
to
the
point
one
of
the
same
thing
center,
one
one
in
in
every
datacenter,
and
you
want
to
prefer
something
traumatic
to
the
first
one
and
then
start
failing
over
to
the
other.
One
only
has
the
second
one
becomes
available.
The
first
one
becomes
unavailable,
something
that
this
we
simply
issues
in
with
priorities.
B
Additionally,
we
love
app
users
to
kind
of
mess
with
this
concept
by
introducing
waiver
for
commercial
data
centers,
which
is
the
ability
for
them
to
put
your
data
centers
into
the
same
data
center
so
that
they
can
say
I'll
treat
these
to
us
that
they're
the
same
thing
Center.
What
essentially
puts
them
gives
them
the
priority
of
like
the
closest
data
center.
This
becomes
really
useful
when
you
use
it
alongside
locality
waiting,
which
is
where
we
use
in
order
to
allow
an
alchemist
to
assign
weights
during
the
vigil
deployments.
B
This
instructs
anyway
to
load
balance
accordingly.
So
this
can
like,
when
I
bake,
it's
more
traffic
than
deployment
a
and
giving
up
owners
the
ability
to
flip
these
changes
as
they
please
becomes
really
powerful
because
they're
they
give
them
a
lot
of
flexibility.
They
can
do
going
blue
deploys.
They
can
do
gradual
failures
between
data
centers
about
utilizing
virtual
data
centers.
Alongside
this,
essentially,
you
have
two
davis
owners.
B
A
B
Each
of
the
deployments
and
used
a
subsidy
balancer
essential
earliness.
Let
me
look
people
to
put
on
the
effluents
whenever
run
time,
thirst
and
they'll
be
updated
in
real
time
which
allows
a
client
sheera
specify
where
they
want
chocolate
ago.
When
it
says
setting
header
this
case,
you're
gonna
target
the
mass
and
we've
also
just
specify
one
reader
will
ensure
that
all
your
traffic
only
hits
the
points
with
the
specified
label.
B
So,
similarly,
now
that
you
can
hear
me
to
label
which
will
now
head
possible
and
being
able
to
move
this,
these
labels
around
at
runtime,
it
makes
them
really
powerful
and
flexible
when
it
comes
to
failing
what
we're
not
just
traffic
but
also
like
stuff
like
which
is
the
master
site.
Like
was
responsible
right
into
the
database
in
an
active
active
system.
So
you
can
use
this
both
sort
of
what
the
clients
calling
the
app
and
internally
you
can
proxy
all
these
use
the
same
labels.
B
So
you
can
you
get
a
lot
of
flexibility
from
this,
and
so
the
last
thing
that
I
want
to
talk
about
is
my
choice,
so
where
every
request,
a
lot
of
a
requests,
are
very
important
because
it's
payments
and
one
because
fails
somebody's,
not
getting
paid
so
a
lot
of
cases.
We
really
wanna
make
sure
that
requests
succeed,
so
even
use
of
won't
only
to
the
end
of
my
career
choice,
but
also
a
bunch
of
smart
retry.
B
So
the
idea
being
the
when
you
retry
request,
do
you
want
to
make
use
of
the
information
from
previous
attempts
to
improve
your
chances?
So
the
interesting
one
most
interesting
one
is
the
cross
latest
entry
choice,
which
is
the
idea,
but
for
cases
where
you
have
an
active,
active
setup
where,
in
multiple
data
centers,
you
have
all
the
resources
somewhat
duplicate.
You
have
two
clusters
of
databases:
a
shorter
database
in
1h
data.
B
It's
the
sugary
regime
would
be
partially
down
and
the
closest
data
center,
so
one
shortest
down
you're
still
a
bit
to
serve
traffic
90%
of
traffic's
is
succeeding.
You
don't
want
to
go
and
healthy,
but
10%
of
junkies
is
consistently
failing
because
the
shortest
in
so
it
happens
in
the
older
data
center,
those
the
entire
resources
available.
Well,
it
was
very
different
when
you
target
just
with
minorities,
you
will
never
go
down
to
the
second
priority
unless
the
first
priority
is
unavailable
or
healthiest
away.
B
A
So
what's
next
for
us,
of
course,
we're
going
to
finish
this
migration
get
rid
of
those
Hardware
load,
balancers,
all
that
good
stuff
in
2019.
We're
also
going
to
look
into
migrating
to
kubernetes,
so
we're
interested
in
chatting
with
people
this
week
to
get
people's
wisdom
on
that.
We
are
excited
to
try
out
spiffy
integration,
particularly
using
envoy
secrets,
discovery
service
API,
which
lets
you
do,
sir
tro
tation
on
a
really
regular
basis
without
having
to
do
any
restarts
of
processes.
A
A
The
question
was:
have
we
are
we
running
envoy
as
a
container
or
as
a
normal
process
right
now
we're
using
it
as
a
normal
process?
For
a
little
bit,
we
had
we're
trying
it
with
docker,
but
for
for
unrelated
reasons,
we
decided
to
roll
that
back.
We
have
not
done
any
horse
racing
or
anything
like
that,
comparing
the
two,
but
it
works
totally
fine
as
a
normal
process.
B
Yeah,
oh
yeah,
of
course,
was
how
high
available
we
share
for
the
control
plane.
So,
first
off
it's
somewhat
point
design.
It's
the
entire
system
is
resilient
to
the
control
plane
going
down.
The
control
band
being
unavailable
simply
means
our
conferences
are
no
longer
being
propagated,
which
is
fine
for
it
just
means.
In
our
case,
it
means
that
if
hosts
are
moved
around,
they
won't
be
updated
immediately.
If
I
deploy
a
planner
sting
goes
out,
it's
not
going
to
get
propagated
up.
So
short,
failure
like
I,
have
like
a
temporary
family
of
the
control.
B
Plane
is
not
like
that.
That's
said,
we
do
run
it
multiple
instances,
so
we
have
maybe
six
or
seven
instances
running
so
the
individual
host
goes
down.
This
is
all
just
again
with
the
proxy
Isabelle
is
aware
of
all
of
the
backend
hosts
and
below
don'ts
between
them,
so
not
going
just
retry
in
two
different
one.
If
one
is
unavailable
and.
A
Then
there's
also
caching
in
the
control
plane
itself
of
like
so.
If
the
the
primary
data
sources
like
service
discovery
or
access
database
go
away,
you'll
you'll
be
able
to
run
on
those
old
configs
for
a
while,
so
I
guess
like
as
long
as
your
service,
topology
isn't
moving
too
much
you'll
be
okay
for
a
little
bit.
A
We
haven't
really
seen
a
problem
with
that
I
mean
it's
things,
kind
of
are
coming
and
going
at
a
slow
rate.
The
question
was:
do
we
see
problems
with
control
planes
not
being
in
sync,
but
in
practice,
processes
are
not
moving
around
so
frequently
that
that's
a
problem.
If
you
do
a
little
time
traveling,
we
haven't
really
seen
a
problem
with
that.
A
The
question
was
about
our
keeping
our
clients
simple.
How
does
that
work?
So,
for
now
we
so
we've
been
using
clients
that
have
very
heavy
amyl
configuration
for
a
long
time
which
is,
and
we
use
mana
repos
pretty
extensively.
So
it
was
pretty
easy
for
us
to
just
like
change
the
config
structure
and
then
people
just
rewrite
their
there,
yeah
mile
config
and
then
now
they're
using
our
Envoy
client.
We
wrote
for
retry
behavior,
we
use
protobufs
almost
everywhere
and
we
have
special
annotations
that
dictate
like
what
the
per
try
timeout
should
be.
A
Clients
decide
the
the
overall
timeout
themselves
and
we're
not
using
circuit
breaking
or
anything
like
that.
Yet
because
we've
not
wanted
to
surprise
anyone
with,
like
all
of
a
sudden
requests,
are
failing
and
its
envoys
fault.
That's
something
that
we
want
to
do
at
some
point.
But
for
now
we
just
set
everything
to
like
10,000
another.