►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Instrumenting and Scaling Cloud-Native Databases with Envoy - Daniel Hochman, Lyft
Every request to a database at Lyft is proxied by Envoy, providing complete visibility into the L3/L4 aspects of database interactions. This allows engineers to easily visualize changes to a database's load profile and pinpoint the root cause if necessary. Lyft has also open-sourced codecs for MongoDB, DynamoDB, and Redis. Protocol codecs in combination with custom filters yield benefits ranging from operation-level observability to horizontal scalability via sharding. Using Envoy for this purpose means that enhancements are implemented once and usable across a polyglot stack. The talk will demonstrate Envoy's utility beyond traditional RPC service interactions in the network. Topics will include protocol analysis, using a proxy for connection pooling, network-level rate limiting, and partitioning with consistent hashing.
To learn more: https://sched.co/Gray
A
Hello,
everyone
thank
you
for
coming.
My
name
is
Daniel
Hawkman
I'm
software
engineer
at
lyft,
I've
been
at
left
for
about
five
years
and
in
that
time
we've
grown
quite
a
bit.
Obviously,
and
when
I
started
we
had,
you
know
a
PHP
application,
monolithic
application
with
a
single
database
behind
it,
and
so,
if
you
fast
forward
to
today,
we
have
you
know
10
clusters,
100
Redis
clusters,
500
dynamo
tables
100.
Can
you
see
streams?
The
list
goes
on
right,
there's
a
lot
of
state
lifts
and
that
all
is
backing
250
plus
micro
services.
A
So
it
turns
out
that
on
boy
was
actually
instrumental
and
allowing
us
to
scale
up
to
the
current
levels
that
we're
at
so
we're
going
to
go
through
a
little
bit
about
instrumenting
envoy
databases
using
envoy
in
the
service
mesh
and
also
how
we
actually
used
envoy
to
keep
the
lights
on,
but
also
to
scale
up
so
I
just
want
to
start
briefly
by
talking
about
a
database
outage
that
we
used
to
commonly
experience-
and
this
happens
in
a
lot
of
databases
is
actually
so.
We
would
have
this
issue
where
our
database
would
slow
down.
A
Briefly,
in
our
case
it
was
disk.
I/O
weight
was
spiking
a
little
bit,
so
the
client
would
open
more
connections.
There
would
be
some
slowdown
because
the
connections
aren't
free
and
in
2
6,
which
we
were
running
at
the
time.
You
know
there's
only
one
request
per
connection,
so
user
requests
are
continuing
to
come
in
we're
continuing
to
open,
more
connections
which
causes
additional
slowdown
to
the
authentication
and
you
just
kind
of
spin
out
of
control
and
cascade
until
things
blow
up
so
yeah.
A
This
was
a
real
problem
for
us
and
we
actually
weren't
running
ourselves.
We
were
actually
relying
on
a
provider
to
do
it
for
us
a
vendor,
and
you
know
we
were
lucky
basically
that
we
even
had
the
disk
I/o
weights
that
available
in
the
console,
because
that's
the
only
thing
we
were
able
to
correlate
to
the
event
actually
happening.
So
I
just
want
to
talk
briefly
for
context
like
what
what
does
it
mean
to
be
running
a
database?
A
You
know
in
a
cloud
native
environment,
well
you're
kind
of
making
a
trade-off
right,
so
you
can
instantly
provision.
You
know
really
high
throughput
infrastructure.
Like
I
said,
we
have
500
dynamo
tables,
it
would
take
a
much
larger
team
to
run
that
level
of
infrastructure
than
we
have
today
currently
on
our
storage
team.
But
the
trade-off
is
that
you
are,
you
know,
can
be
running
on
share
tenancy,
Hardware
and
probably
in
most
cases
not
even
have
access
to
the
underlying
hardware.
A
You're
gonna
have
limited
access
to
telemetry,
so
you
know
if
I
have
a
Linux
box
and
I'm
running
it
myself,
I
can
collect.
You
know
an
infinite
number
of
stats.
Basically,
but
the
vendor
is
not
going
to
be
able
to
do
that
they're
only
going
to
provide
what
they
think
is
valuable.
You
can
have
limited
access
to
a
configuration,
so
you
know
for
security
reasons
or
potentially
for
scale
reasons,
usually
you're
only
going
to
access
to
a
subset
of
features
that
the
database
offers.
A
And
finally,
you
know
a
lot
of
times
you're
dealing
with
a
closed
source
product
like
DynamoDB,
but
even
if
you're
not
and
you're
doing
something
like
RDS,
where
you're
running
my
sequel,
you're
not
gonna,
have
the
ability
to
just
drop
in
a
custom
binary
or
with
some
patch
that
you
think
might
fix
your
problem.
You're
not
going
to
attach
a
debugger
or
do
print
line
debugging.
Anything
like
that.
A
So
what
it
really
comes
down
to
is
that
we're
making
a
trade-off
right,
so
we're
going
cloud
native,
and
that
gives
us
access
to
a
lot
of
great
things.
But
the
flip
side
of
that
is
that
we
don't
actually
have
access
to
anything.
So
when
we
really
want
to
debug
when
we
really
wanted
to
dive
in
and
look
at
hard
problems,
we're
not
gonna
have
any
ability
to
do
that.
A
So
I'd
like
to
take
a
step
back
briefly
and
just
talk
about
how
envoy
is
deployed
before
we
talk
about
the
benefits
that
it's
actually
giving
us
with
relation
with
regards
to
databases.
So
a
service
mesh
is
basically
a
network
topology,
and
it
just
means
that
you
know
loosely
that
the
proxy
is
deployed
at
every
single
hop
of
the
network.
So
in
this
case
we've
got
our
service
mesh,
which
is
you
know,
different
clusters
of
applications
with
different
runtimes.
Every
single
host
is
running
envoy
as
a
sidecar
on
the
Box.
A
Every
single
host
is
registering
into
the
discovery
service
and
when
services
need
to
communicate
to
each
other,
they
do
so
via
that
sidecar
envoy,
which
knows
how
to
address
the
rest
of
the
instances
in
the
in
the
destination
cluster.
So
this
is
what
it
would
look
like
on
an
individual
instance.
The
application
is
going
to
communicate
locally
to
the
sidecar
over.
A
You
know
the
loopback
interface
or
UNIX
domain
socket
and
well
proxy,
all
traffic
out
so,
but
it
turns
out,
we
don't
just
use
this
for
for
internal
services.
We
also
use
this
for
third-party
services
for
cloud
services
and
we'll
see
the
benefit
of
doing
that
as
we
kind
of
look
at
how
Amba
can
help
us
at
different
areas
of
the
the
OSI
model.
So
we're
going
to
start
with
layer,
3
and
layer
4.
So
basically,
layer
1
is
like
electrons
moving
in
wires
and.
B
A
You
go
up
to
layer,
7
you're,
talking
about
the
actual
application
specific
protocol,
so
we're
gonna
start
at
layer,
3
and
4
and
work
our
way
up
to
the
application,
so
at
layer,
3
and
4.
The
most
common
thing
that
envoy
does
is
proxy
TCP,
and
so
you
might
think
well
what?
What
really
is
the
benefit
of
doing
that?
Well,
the
first
is
that
you're
going
to
get
some
nice
stats
about.
A
You
know
what's
happening
like
everyone
here
is
probably
tuned
to
connection
pool
at
some
point
or
try
to
figure
out
what
the
application
is
doing,
how
its
optimizing
for
open
connections
etc.
So,
just
with
the
stats
that
we're
getting
from
Envoy
by
proxying
through
we're
gonna
be
able
to
see
the
number
of
active
connections,
the
number
of
connections
that
are
being
created,
the
number
of
bytes
that
are
flowing
over
those
connections,
so
bytes
can
be
useful.
A
For
example,
let's
say
I
have
a
database
or
or
some
endpoint
and
it's
receiving
a
lot
of
traffic
well,
I
can
actually
break
it
down.
You
know
by
by
which
clusters
are
sending
the
traffic
to
that,
and
if
we
look
at
you
know
DNS,
let's
say
I
want
to
talk
to
a
service
in
Amazon
or
really
any
service.
How
many
people
here
could
answer?
You
know
what
their
application
is
doing
when
it
receives
two
or
more
a
records
in
the
DNS
resolution.
A
Like
probably
know
about
you,
even
if
you
could
for
one
of
your
applications,
you
probably
don't
know
how
it's
going
to
work
in
Python,
how
it's
going
to
work
and
go,
how
it's
going
to
work
in
JavaScript
or
any
of
the
different
runtimes
that
you're,
using
so
with
boy
you're,
gonna,
get
kind
of
a
known
quantity
of
load,
balancing
between
the
records
that
are
returned
when
you're
using
DNS.
So
you
can
tell
it
to
just
use
one
record
or
you
can
have
it
load
balanced
round-robin
at
least
request
random
between
them.
A
So
this
is
just
great
because
we
don't
have
to
think
about
how
the
individual
application
is
optimizing
for
what's
returning
DNS.
Furthermore,
we
can
do
health
checking.
So
if
I'm
boy
sees
that
one
of
the
IPS
returned
in
DNS
is
not
reachable
I'll
be
able
to
stop
sending
traffic
to
that,
and
it
will
send
it
to
the
other.
A
A
You
know
you
need
to
fix
this
specific
IP
and
the
certificates
broken
on
that
node.
So
the
other
nice
benefit
is
that
you
know
envoy
running
as
a
separate
process
on
the
host.
So
if
we
find
some
kind
of
bug
in
the
SSL
library-
and
we
need
to
upgrade
it-
we
don't
have
to
ship
250
applications
that
lyft
we
can
just
ship
envoy
and
with
the
upgrade
and
then
everyone
will
be
patched.
It's
also
going
to
be
more
efficient.
You
know,
then
doing
it
in
Python.
A
Potentially,
you
know
negotiating
the
SSL
up
session,
so
finally,
we're
at
layer
7,
which
is
the
application
layer
and
the
most
common
thing
that
envoy
does
at
layer.
7
is
manage
HTTP,
so
again
we're
getting
a
nice
wealth
of
stats,
including
a
you
know,
nice
histogram
of
the
timings
of
each
request
and
another
thing
that
on
boy
can
do
is
actually
upgrade
to
from
HTTP
1
to
HTTP
2.
A
If
the
backend
service
supports
it,
so
HTTP
1
means
that
you
have
one
request
per
connection
and
you
have
to
wait
for
that
request
to
finish
before
you
can
issue
another
one,
but
with
HTTP
2
you
can
actually
stream
multiple
requests
or
multiplex
multiple
requests
over
a
single
connection.
There's
some
other
benefits
which
give
you
a
really
nice
efficiency
like
header
compression.
But
it's
just
going
to
be
a
lot
more
efficient.
A
So
these
are
all
the
statistics
that
I
kind
of
talked
about
and
if
you
just
look
at
them
for
a
moment,
you
can
kind
of
start
to
appreciate
how
much
data
on
boy
is
going
to
give
you
about
what's
happening
in
your
system,
so
we've
been,
if
we're
not
going
to.
You
know
talk
about
some
of
the
other
database
scaling
techniques
that
we're
using
just
to
pulling
Envoy
as
kind
of
a
proxy
and
of
itself
it's
going
to.
A
A
Similarly,
because
we
have
such
a
nice
set
of
stats
around
the
system,
we
can
easily
create
dashboards
like
this
one.
So
we
can
let
the
user
select
basically
any
two
points
in
the
network
in
any
environment
and
see
the
number
of
connections
that
are
occurring
between
them.
We
can
see
the
you
know
the
number
of
bytes
that
are
going
between
them.
This
is
a
small
sample
of
the
data
that
we
have,
but
because
it's
all
in
the
same
format,
we
can
easily
create
this,
and
similarly
we
can
give
other
people.
A
A
So
kind
of
the
point
of
this
all
is
that
you
know
having
telemetry
data
be
in
the
same
format,
makes
it
really
easy
to
correlate
behavior
in
a
large
system,
so
you
can
see
on
the
right-hand
side
of
the
screen,
there's
a
giant
blob
with
a
bunch
of
connections
between
it.
That's
actually
a
network
graph
of
what's
happening
at
left
at
any
given
time
and
I
believe
only
a
subset
of
it
so
trying
to
debug.
A
So
what
does
it
actually
look
like
to
enable
envoy
in
an
application
once
we've
configured
it
to
proxy
traffic?
It's
usually
as
simple
as
just
a
one-line
configuration
change.
So
in
this
case
you
know
in
the
NGO
library
we're
talking
to
dynamo
DB
and
we
can
just
patch
it
with
localhost,
8080
and
'red
access
log
stats.
We
create
a
standard
set
of
alarms
for
everyone
at
lift
when
they're
talking
to
different
services.
So
they
know
if
the
upstream
is
having
problems,
whether
again
it's
an
internal
service
or
it's
a
vendored
service.
A
A
So
now
I
want
to
get
to
kind
of
the
heart
of
the
talk
right,
which
is
how
we
actually
manage
to
scale
the
databases.
So
you
can
already
imagine
some
of
the
benefits
we
will
get
from
deploying
Envoy
and
tcp
mode
in
front
of
or
an
HTTP
mode
in
front
of
dynamo,
but
we
actually
have
specific
codecs,
which
can
understand
the
wire
format
of
each
of
these
databases
and
do
something
a
little
bit
different
with
it.
A
So
the
first
is
going
to
be
dynamo
DB,
so
dynamo
DB
is
just
JSON
over
HTTP
for
anyone
who's
ever
used
cloud
watch.
It
doesn't
offer
like
a
high
cardinality,
a
wide
range
of
statistics.
The
only
thing
that
you
can
see
about
dynamo,
DB
and
cloud
watch
is
the
minimum
average
and
maximum
latency.
You
can
also
see
you
know
some
other
stats
like
the
capacity
throughput
or
some
permanent
stats,
but
at
lift
where
we're
obsessed
with
you
know
the
P
95
and
the
P
99
behavior
of
our
services
in
any
given
session.
A
For
a
user
who's
taking
a
ride,
you
know
they
may
make
hundreds
of
different
API
requests.
So,
even
if
your
p
99
is
pretty
good,
they
may
actually
experience
or
if
your
p
95
is
good,
they
may
hit
p
99
in
every
single
session
and
that's
going
to
be
a
bad
experience
for
the
user.
So
what
are
the
benefits
again
of
just
deploying
Envoy
in
HTTP
mode,
directly
in
front
of
DynamoDB
we're
going
to
get
a
histogram
of
latency?
So
beyond
just
min
average
of
max?
A
We
can
see
all
the
percentiles
that
we
want
to
capture
p
50
median.
We
can
do
custom
windowing
of
metrics,
so
cloud
watch
only
offers
permanent
stats,
which
sometime
can
be.
You
know
under
sampled,
if
you're
trying
to
debug
a
particularly
hard
problem,
Jose
and
I.
My
colleague
here
from
lift
we
had
the
debug
a
problem
where
there
was
a
two-second
loop
that
was
occurring
in
an
application,
and
so
basically
every
two
seconds
there'd
be
an
enormous
spike
of
requests,
and
in
between
that
there
would
be
no
requests
and
only
having
per
minute
data.
A
You
would
never
be
able
to
really
tell
what's
happening
and
finally,
we
have
per
hosts
per
zone
and
per
cluster
statistics
so
anywhere.
That
envoy
is
running.
It's
also
emitting
stats
that
are
tagged
with
its
its
location.
So
you
know
when
we
look
at
the
the
dynamo
statistics,
let's
say
we're
running
an
online
batch
box
and
a
batch
box
for
our
service.
We're
not
gonna,
be
able
to
break
that
down
in
any
meaningful
way.
So
with
envoy
running,
we
can
actually
see
you
know.
Oh,
is
it
batch?
A
We
can
go
a
step
further
and
with
one
line
of
code
or
configuration
rather,
we
can
enable
the
dynamo,
filter
and
dynamo
filter
is
about
300
lines
of
code
or
so
and
what
it
does
is
it
just
interprets
the
JSON
that
is
being
emitted
by
dynamo
or
the
library
that's
talking
to
dynamo.
So
this
is
a
really
all
that
the
dynamo
DB
codec
is
doing
it's
taking
the
incoming
request.
It's
parsing,
the
body
for
the
table
name
and
it's
parsing,
the
headers
to
get
the
operation.
A
A
But
we
can
do
this
across
any
cluster
at
lift
whether
they're
running
and
go
or
JavaScript
or
Python,
and
so
then
we
can
aggregate
stats
in
any
manner
that
we
want.
So
not
only
can
we
look
at
you
know
a
specific
cluster,
but
we
can
look
across
all
of
our
services
and
see
how
they're
interacting
dynamo,
which
is
useful
when
the
AWS
status
page
is
showing
okay,
but
things
are
not
working
well
at
all.
A
So,
let's
move
on
to
MongoDB
and
remember
the
outages
that
we
talked
about
earlier,
we'll
get
back
to
that
outage.
So
the
protocol
for
MongoDB
is
binary,
JSON
and
again
we're
going
at
the
benefits
of
knowing
exactly
who's
doing
what
network
in
the
cluster
when
we
deploy
it
from
a
TCP
perspective,
is
it
the
batch
box,
creating
more
connections
or
is
it
the
online
cluster,
creating
more
connections?
But
if
we
add
in
the
codec
it
can
do
the
same
thing
that
we
were
doing
in
dynamo
DB
and
we
can
get
per
operation
latency.
A
So
we
can
look
at
whether
it's
a
query
or
whether
it's
a
right.
We
can
count
the
size
and
number
of
documents
that
are
sent
today
or
tomorrow.
Db
and
the
number
that
are
returned
and
we
can
count
scattered,
gets
as
well,
which
is
really
important
when
you're
in
a
sharded
cluster
scattered
gets,
are
extremely
expensive
and
we
had
to
do
a
lot
of
optimization
to
kind
of
pay
them
down.
A
So
developer
can
now
ask
the
question:
how
did
the
number
of
documents
returned
by
queries
change
and
you
know
specific
availability
zone
when
I
deployed
my
service
at
a
certain
time?
And
again
this
is
data
that
was
not
possible
to
really
collect
before
when
you
have.
You
know,
just
a
few
stats
being
kind
of
sent
back
to
you
from
your
vendors
API,
which
you
might
be
scraping.
A
A
You
know
from
users
with
different
IPS
if
they're
trying
to
post
different
data
or
attack
our
API,
we
apply
the
exact
same
rate
limit
or
to
new
connection
creation
against
the
database,
and
this
actually
applies
the
necessary
back
pressure
so
that
the
database
can
recover
when
there's
a
slowdown,
so
we
may
return.
You
know
a
few
errors
to
our
customer.
We
may
have
a
one-second
outage,
but
that's
a
lot
better
than
a
minute
outage,
or
you
know
a
five
minute
outage
which
we
used
to
experience.
A
So
in
practice
we
never
actually
solved
the
disk
Iowa
problem.
We
don't
actually
know
what's
happening.
We
assume
it's
some
noisy
neighbor
problem.
It
could
be.
You
know,
log
archival
by
the
vendor
or
something
we
never
did
get
to
the
bottom
of
it.
But
at
this
point,
once
a
month
or
maybe
even
once
a
week,
we
can
go
look
at
the
graph
and
see
connection
rate
limiting
which
the
Envoy
is
saving
the
service
from
going
down.
A
There's
a
lot
of
other
problems
that
we've
had
with
as
well,
so
we
had
people
will
try,
you
know
to
delete
indexes
and
they'll
be
reading
the
legacy
application.
You
know
it's
a
3,000
line
class,
it's
very
hard
to
tell
what's
actually
happening
so
they
delete
an
index
and
the
database
would
just
fall
over
because
the
the
queries
that
were
being
issued
were
no
longer
covered
by
a
certain
index.
So
the
you
know
they
would
suggest
if
we
try
the
built-in
query
profiler.
A
Well,
that
caused
another
outage,
because
the
query
profiler
is
extremely
expensive
to
actually
use.
So
what
we
do
is
actually
leverage
envoy
in
this
case.
So
again,
envoy
is
seeing
all
the
traffic
that's
coming
through.
It
knows
it
how
to
speak
the
protocol,
so
when
we
enable
the
codec,
we
can
actually
log
every
single
query
to
a
JSON
file
so
that
we
can
audit
logs
to
see
what's
going
on.
A
So
you
know,
if
you
look
at
six
months
and
even
more
outages
later,
you
know,
we've
got
the
rate-limiting
in
the
logging,
but
now
we
can
also
track
queries
that
don't
have
a
specific
time
out
field
set,
and
we
did
this
a
neat
thing
where
we
would
actually
pass
some
metadata,
along
with
the
query
in
a
common
field,
parse
that
with
envoy
and
then
immense.
That's
with
it
and
we'll
see
what
that
looks
like
in
a
minute.
A
But
the
other
thing
to
note
is
that
at
this
point,
we're
no
longer
just
running
a
PHP
application
with
we
have
a
go
application
with.
We
have
a
Python
application
with,
and
all
of
them
are
protected
by
the
work
that
we
did.
You
know
in
in
envoy,
so
this
is
what
our
envoy
log
would
look
like
and
so
yeah
we
in
the
application
itself.
We
insert
that
data
into
the
comment
field
and
then
on
Boyd
can
emit
a
stat.
That
basically
says
you
know
this.
A
How
this
is
how
long
this
exact
function
is
taking
every
time
it
calls.
So
when
a
user
deploys
code
deploys
a
change,
it
makes
it
really
easy
to
go
and
see
exactly
what's
happening
or
if
we
get
a
really
expensive
query,
it's
easy
to
find
in
the
application
exactly
what
admitted
it.
Furthermore,
this
simplifies
development
by
quite
a
bit.
It's
very
hard,
sometimes
to
understand.
A
You
know
the
query
that
the
or
it
might
generate-
and
if
you
coax
it
enough,
usually
you
can
get
it
to
output,
some
kind
of
debugging
log,
but
again
in
every
language
that
we
have
today,
when
you're
in
your
development,
environment
and
you're
issuing
queries
against
MongoDB
from
your
ORM.
You
don't
need
to
try
and
read
a
thousand
lines
of
code
to
figure
out
what's
happening,
you
don't
need
to
enable
some
kind
of
debugging
mode.
You're,
just
gonna
get
a
nice
stat.
A
That's
going
to
tell
you
what's
happening
and
you're
gonna
get
a
log
as
well,
so
that
you
can
pinpoint
in
the
application
exactly
what's
happening
with
the
database,
so
we're
gonna
move
on
to
Redis
now.
So
this
is
gonna.
Be
brief,
because
I've
talked
about
this
before
Redis,
we
were
using
a
deprecated
tool
from
Twitter
called
twin
proxy,
so
it's
very
popular
in
time
and
similar
to
Mac
router.
A
If
you
use
memcache
or
some
of
the
other
partitioning
proxies
that
are
out
there,
but
we
had
a
problem
with
it,
which
is
basically
that
it
required
a
static
config
file.
And
when
you
wanted
to
update
something
in
the
config
file,
we
would
basically
take
an
outage.
So
at
that
point
we
were
like
okay,
we
can
do
some
crazy
dance
where
we
run
two
of
them
and
have
the
applications
switched
between
them
when
we
need
to
restart
one
or
we
can
go
ahead
and
just
recreate
this
thing.
A
So
so
what
really
is
it
at
its
core?
Well,
it
speaks
the
rhibs
protocol,
so
it's
got
a
codec
and
it
can
do
consistent
hashing.
So
it's
got
a
map
of
where
the
instances
live
and
what
data
should
be
aligned
to
them.
So
we
were
able
to
create
this
in
envoy.
We,
you
know,
use
the
katana
algorithm
similar
to
what
a
twin
proxy
does
we
take
the
discovery
data
you
know
envoy
knows
about
wherever
instance
is
deployed
from
the
discovery
service,
and
we
can
just
create
a
logical
sharded
partitioned
horizontally,
scalable
Redis.
A
So
did
the
application.
We
don't
have
to
make
any
modifications.
We
can
continue
doing
the
same
thing
that
we
were
doing
with
Nutcracker,
but
now
we've
got
you
know,
clusters
with
up
to
200,
Redis
nodes
in
them
and
globally
we're
doing
about
35
million
requests
per
second
through
this
flow
to
Redis
and
yeah
I've,
given
talks
at
Redis
kampf
about
this
before
so.
If
you're
interested
in
more
of
the
finer
details
about
how
this
works,
you
can
just
search
left
Redis
on
YouTube
and
you
can
find
those
talks.
A
So
really
what
this
boils
down
to
is
that
there's
there's
two
main
approaches
once
we're
able
to
speak
the
protocol
and
even
before
we're
able
to
speak
the
protocol.
The
first
is
just
bumping.
The
wire
right
I
can
get
some
basic
observer
ability
if
my
application
speaks
TCP,
if
it
speaking
HTTP
to
an
upstream
and
if
I
want
to
go
further.
I
can
use
the
codec
to
extract
features
and
emit
even
more
stats.
We
can
add
some
of
the
basic
capabilities
of
envoy,
like
connection
rate,
limiting
no
problem.
A
So
what
are
some
of
the
future
codecs
that
we're
seeing
development
on
and
that
will
probably
be
coming
soon
and
the
people
have
really
asked
for
quite
a
bit?
Well,
we've
got
my
sequel.
There
was
actually
a
talk
from
a
Sri
I
believe
it
was
on
Tuesday
about
how
they're
gonna
use
this
VMware.
How
this
is
gonna
be
really
important
to
actually
their
kubernetes
integration,
and
so
he's
working
on
that
currently
and
I
believe
it's
close
to
merging
and
it
will
probably
start
as
just
an
observability
codec.
A
You
know
decoding
what's
happening
and
outputting
some
stats,
but
once
you
have
that
it's
not
that
big
of
a
leap
to
kind
of
go
to
the
fully
routing
type
of
proxy
Kafka,
extremely
popular
there's,
a
lot
of
people,
including
lifts
that
aren't
going
to
be
investing
in
this
as
we
migrate
over
to
Kafka
and
then
Postgres
as
well.
I
think
there's
only
an
issue
for
that
one,
but
it
should
be
coming
down
the
pipe
soon.
A
So,
what's
the
road
map
really
look
like
we're?
Gonna
see
more
codecs
as
I
just
talked
about.
We
can
expand
the
capability
of
existing
codecs
to
have
full
l7
capabilities
which
can
provide
some
additional
benefits.
We
get
a
better
integration
of
distributed
tracing
so
right
now,
you
know
when
you
make
a
call
I,
don't
know
if
you've
seen
any
of
the
other
talks
or
use
the
distributed
tracing
tool.
A
Basically,
when
you
have
a
call
from
the
client,
you
can
see
every
hop
of
the
network,
you
know
what
service
is
hit
and
that
will
let
you
determine
you
know
what
is
the
cause
of
latency?
What
exactly
is
happening
there
and
wouldn't
it
be
great
if
you
could
actually
see
down
to
the
database
shard,
you
know
if
that
shard
caused
the
latency
and
debug
it
that
way.
Right
now
we
do
emit
some
stats
from
or
some
traces
from
the
database
side,
but
but
not
enough.
There
could
be
a
lot
more.
A
Another
thing
that
we
use
frequently
at
lift
is
a
fault
injection.
So
today
you
can't
apply
fault
injection
to
and
you
can
apply
fault
injection
to
dynamo,
and
what
this
means
is
that
you
can
basically
tell
envoy
to
air
out
some
percentage
of
traffic
or
to
add
additional
latency
to
traffic,
so
we're
thinking
about
migrating,
for
example,
to
dynamo,
DB,
global
tables
from
just
single
region
tables
at
lift.
This
is
a
new
feature
of
dynamo
that
was
announced
at
reinvent.
A
Well,
we
assumed
that
there
will
be
some
additional
latency
impact
doing
so
right
because
we've
got
our
Traverse,
multiple
regions
or
dynamos
Traverse,
multiple
regions
when
when
doing
the
right,
so
we
can
actually
inject
you
know
100
milliseconds
of
latency
and
seeing
our
application
behaves
before
just
turning
on
some
feature
that
we
know
will
cause
a
performance.
In
fact,
we
can
also
inject
airs
a
frequent
issue
that
we
would
have
at
lyft.
A
A
You
can
do
it
with
dynamo
DB,
but
it's
not
possible
in
Redis
and
I,
don't
believe
it's
possible
at
the
generic
TCP
layer
so
having
that
capability
everywhere
will
let
you
test
your
services
to
make
sure
that
they
get
will
behave
correctly
in
the
event
of
some
slowdown
or
some
errors
and
finally,
role
based
access
control,
and
this
is
what
Shriram
talks
about
extensively
he
works
at
VMware.
You
know
they're
looking
at
deploying
envoy
right
and
proxying,
my
sequel
through
it.
A
Well,
you
know
in
a
kubernetes
environment
in
particular,
you
may
not
have
just
an
envoy
deployed
directly
alongside
the
application
and
you
may
not
have
full
control
over.
You
know
what
configuration
is
being
injected
into
that
if
multiple
services
are
running
in
the
same
pod,
so
what
they
actually
plan
to
do
is
you
know,
identify
the
caller
with
with
mutual
TLS
or
something
else
and
then
decide
whether
or
not
they're
allowed
to
talk
to
a
my
sequel
database
from
that
that
envoy,
so
you
can
get
really
advanced
again
with
the
the
l7
capabilities.
A
A
Yeah
a
good
question,
so
they
asked
if
there
was
any
a
latency
impacts
of
running
envoy
in
this
path.
So
we
haven't
done.
You
know
specific
benchmarks,
for
it
I
mean
we
have.
We
have
benchmarked
it,
but
not
you
know
in
a
scientific
way,
but
we
see
it
as
a
sub-millisecond
impact,
basically,
because
we're
deploying
it
locally
on
the
box
we're
talking
over
a
UNIX
domain
socket.
So
you
know
for
us
most
of
our
quests.
You
know
we're
running
like
matching
algorithms
and
they
take
a
little
bit
longer.
A
A
Yeah,
so
there
are
certain
commands
like
in
Redis,
where
you
can
issue
multiple
keys
and
the
proxy
'its
that
know
how
to
route
those
right.
So
if
you
issue
a
multiple
get
with
multiple
keys
envoy
will
actually,
in
parallel
fan-out
the
requests
to
the
backends
reconstitute
the
response
and
send
it
back
to
the
application.
But
there
are
certain
commands
like
S
Union,
where
Redis
is
actually
computing.
You
know
some
result
for
multiple
keys
on
a
single
instance.
A
Those
commands
are
not
supported
currently,
and
there
is
a
feature
called
hash
tagging
which
will
allow
you
to
designate
a
portion
of
the
key
and
to
make
sure
that
it
will
hash
to
the
same
back-end.
But
we
don't
that's
the
one
feature
actually
from
Nutcracker
that
we
we
don't
have
today,
but
it
wouldn't
be
be
that
hard
to
add
honestly.
A
Right
yeah,
so
the
question
was:
is
on
boy
doing
too
much,
or
will
it
maybe
be
doing
too
much
in
the
future,
and
so
I
mean
the
first
thing
I'll
say
about
envoy.
Is
that
it's
built
on
top
of
an
extension
model?
So
you
know,
maybe
there
are
some
people
out
there
that
are
doing
things
that
really
want
to
make
sense
for
the
core
product,
but
we
don't
let
it
actually
affect.
You
know
the
design
of
the
API
or
the
design
of
the
core
product.
So
we
try
to
keep
on
boys
core.
A
You
know
together
like
as
just
as
high-performance
proxy
that
can
transit
network
traffic.
But
if
you
look
internally
at
lyft
right
now
we're
doing
something
where
we're
actually
abstracting
every
single
database
away
from
the
application
and
using
G
RPC
to
issue
queries
to
issue
rights,
and
you
know,
then
we
can
put
a
Redis
in
front
of
it.
If
we
want
to
do
some,
some
some
like
read
through
a
right
through
caching,
we
can
switch
between
dynamo
and
other
backing
stores.
A
You
know,
depending
on
the
desire
of
the
client
there
and
what
consistency
requirements
they
have
and
I
mean
feasibly
right.
You
could
build
that
into
envoy,
but
but
we
built
a
middleware
tier
right,
which
you
know
we
talked
to
you
RPC
to
this
middle
tier,
and
then
it
translates
it
into
you
know.
Whatever
the
database
specific
protocol
is
and
then
returns
that
so
yeah
I
mean
there's,
there's
going
to
be
a
trade-off
and
it's
just
a
matter
of
complexity
and
whatever
people
decide
to
do
so.
A
You
know,
at
this
conference
we've
seen
people
doing
a
lot
of
things
with
envoy
that
we
didn't
know
about
that.
We
didn't
expect
that
are
really
cool
and
on
a
lot
of
cases
makes
sense
and
in
other
cases
yeah
it's
like
well.
This
is
probably
not
useful
outside
of
a
single
person's
organization,
so
it
really.
It
really
is
a
balance
and
I
think
that
the
like
community
and
organization
are
doing
the
right
thing
in
this
case.
A
Yeah
Emily
can't
so
the
question
was:
is
there
any
advantage
to
you
know
having
languages
beyond
C++
available
for
extensions,
because
I'm
boy
is
written
in
C++,
and
so
we
actually
saw
a
tomboy
Khan.
There
was
a
presenter
from
cilium
and
they're,
actually
allowing
you
to.
Basically,
you
know
use
go
directly
in
your
application,
so,
for
example,
with
Postgres
we
don't
have
to
write
an
entirely
new
codec
to
parse
the
the
database
language.
We
could
just
use
the
existing
ones
that
are
out
there.
A
We
could
use
the
one
that's
most
commonly
used
by
you,
know
the
community
and
so
yeah
that
will
definitely
open
up
some
more.
If
that
becomes
a
common
thing
right
now,
I
would
say
I
mean
most
protocols
are
not
that
complicated
and
I
think
the
development
of
the
codec
I
mean
with
with
dynamo
right.
It's
just
parsing
JSON
Redis
response
protocol
is
pretty
simple
yeah.
If
you
get
to
more
advanced
protocols
and
certainly
a
feature
I'm
sure
we'll
see
you
know,
more
extensions
come
on
that
are
potentially
being
driven
by
by
other
languages,
but
for.
A
Yeah
so
we
run
a
single
binary,
but
the
cool
thing
about
our
envoy,
which
which
I
didn't
mention,
is
that
it
can
be
dynamically
configured
via
API.
So
what
actually
happens
is
that
we
give
on
boy
kind
of
a
bootstrap
configuration
which
says
go
to
this
server
and
and
fetch
your
configuration,
so
we
deploy
the
same
binary
everywhere
and,
yes,
we
do
compile
and
extensions
that
may
not
be
needed
in
every
context,
but
the
overhead
of
doing
that
is
not
that
large
or
not
important
in
our
case.
A
A
The
question
was:
how
does
this
overlap
with
this
Co,
which
is
the
most
common
question,
asked
anyone
who
talked
to
talked
about
envoy
I
think
so
so
it's
do.
We
are
building
a
lot
of
components
in
parallel
with
this,
do
and
actually
working
on
getting
them
into
kora
envoy,
so
the
current
are
back
filter.
A
The
role
based
access
control,
filter
that
you
can
use
for
HTTP,
for
example,
was
a
combination
of
contributions
from
the
sto
project
and
from
envoy
now,
at
the
same
time,
they
are
building
their
own
kind
of
higher
level,
constructs
and
routing
methods
and
authentication.
You
know
for
maybe
doing
identity,
for
example
that
wouldn't
make
it
into
core
envoy,
so
I
would
expect
that
anything.
That's
related
to
you
know,
for
example,
talking
to
one
of
the
core
databases
that
would
be
probably
up
streamed
in
the
core
project.
A
Yeah
question
was:
can
we
compile
to
arm
architecture
I,
believe
it's
being
worked
on?
Is
it
working
right
now,
I'm,
looking
at
Matt
who's,
the
maintainer
Matt's?
Not
in
yes,
so
yeah
they're
there
people,
you
know
we,
we
wrote
envoy
to
run
on.
You
know
a
bunch
of
Linux
and
that's
what
our
CI
environment,
what
it
is
was
in,
but
people
are
doing
CI
now
and
CentOS
people
are
doing
it
for
arm.
We
also
have
it
on
Windows
people
reporting
it
to
Windows.
So
we're
really
suing.
A
You
know
good
uptake
of
olive
oil
and
a
lot
of
people
porting
it
to
a
lot
of
different
platforms
and
I.
Believe
one
of
the
goals
of
next
year
is
going
to
get
envoy
running
actually
on
native
devices
like
mobile
phones,
so
you
can
get
kind
of
the
same
stats
visibility.
You
know
even
directly
from
the
phone
so
in
in
particular.
At
that
point,
you
know,
will
definitely
be
supporting.
Arm
is
a
compilation
target.
A
Yeah
the
question
was
which,
which
end
of
the
TLS
connection
are
we
inspecting?
Are
we
inspecting?
You
know
what
the
client
is
presenting
to
the
server
or
what
the
server
is
presenting
back
to
the
returning
to
the
client
I.
Think
most
of
the
metrics
are
around.
You
know
what
the
the
server
is
emitting,
but
some
of,
like
the
the
one,
that's
checking
well
the
the
days
until
the
cert
expires.
That's
coming
from
the
client
side,
for
example,
when
we're
actually
serving
a
certificate,
so
it
just
buried.
A
A
Right,
so
the
question
is
whether
or
not
we're
running
on
boy.
You
know
on
the
database
side
as
well.
So
if
you
were
to
look
at
something
imagine
go
as
a
database
here,
you
know
you
can
run
on
boy
at
that
point.
We're
not
doing
that
today,
because
there
hasn't
really
been
a
big
benefit
for
us
to
do
that.
But,
for
example,
if
we
wanted
to
add
ssl
on
top
of
Redis
for
compliance
reasons,
you
know
brightest
doesn't
support
SSL.
So
that's
exactly
what
we
even
do.
A
We
would
run
on
boy
alongside
the
individual
Redis
instance,
and
then
the
two
on
boys
would
negotiate
SSL
and
the
applications
would
speak
unencrypted
and
we
would
get
that
out
of
that.
So
they're
definitely
use
cases
we
just
haven't
had
a
need
to
to
do
any
of
them
yet
and
what
time
isn't
I'll
take?
One
more
question
sure.
A
Yeah,
the
question
was:
can
you
just
use
the
standard
database
library?
Even
if
there's
no
codec
written
and
the
answer
is,
is
yes,
so
it
will
just.
It
should
just
work
like
unless
there's
something
in
Postgres,
where
you
know
it's
trying
to
glean
information
from
like
an
ssl
certificate
or
manage
identity
in
that
way,
I
have
it
Ivan
run
Postgres,
so
I
don't
know,
but
you
you
should
be
able
to
just
use
it
actually
and,
and
sometimes
I
mean
there
are
additional
patching.
A
There's
additional
passion,
that's
required
so
like
in
the
Python
library,
for
example,
if
you
put
localhost
in
there,
it
tries
to
sign
the
request
with
localhost
and
then
Amazon
rejects
it.
So
there's
one
additional
thing
that
you
have
to
patch,
where
it
says
you
know:
you're
talking
to
DynamoDB
US
East
one
as
your
host,
even
though
you're
transiting
over
the
local
socket
thanks.
Everyone
for
coming
I'll
be
available
for
questions
after
if
you
want
to
come
up
and
check.