►
From YouTube: Deep Dive: Linkerd - Oliver Gould, Buoyant
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Deep Dive: Linkerd - Oliver Gould, Buoyant
The 2.0 release of Linkerd marks a substantial shift in both technology and purpose for the project. In this talk, we describe Linkerd’s evolution from 1.x’s “service mesh” focus on platform-level problems for platform operators, to a 2.x “end user” focus on giving Kubernetes users the ability to debug, inspect, and manipulate the traffic runtime of their services. We describe Linkerd’s shift from a Finagle, Netty, and JVM-based stack to one built on Rust and Go, how we incorporated lessons learned from the Conduit project, and what the future of Linkerd holds.
To learn more: https://sched.co/H3Ij
A
Should
we
get
started
everybody
all
right?
Okay,
before
we
get
going,
I
tell
me
a
little
about
who
I'm
talking
to
here,
how
many
people
are
using
kubernetes
in
their
day,
job
wow,
it's
so
different
than
last
year
cool.
How
many
of
you
have
like
used
a
service
mesh
as
part
of
that?
That's
a
bunch
of
you
how
many
of
you
Xin
linker
D
as
part
of
that
good,
good
good.
A
How
many
people
have
been
to
Seattle
before
yeah
cool?
We
were
all
s
who's,
the
last
conference.
Let
me
put
it
that
way.
At
the
last
Seattle
coop
con,
nobody
won
one,
two,
okay,
three!
Okay.
We
had
a
couple
but
you're
all
newbies
I
wasn't
here
either,
though
so
it's
okay,
alright,
so
my
name
is
Oliver
Gould.
These
are
my
dogs
dream
on
Dumbo,
my
backgrounds
in
production
operations,
I
out
of
school
spent
some
time
school
know
what
to
Yahoo
Yahoo.
A
For
those
you
remember
what
that
was
was
a
really
large
Internet
directory
and
it
was
really
like
20
different
companies
that
inquired
over
15
years,
all
in
some
crazy
bastardized
infrastructure
across
the
globe.
All
the
systems
were
in
local
time,
which
is
crazy
and
FreeBSD
two
to
FreeBSD
nine,
and
then
we
said
interesting
Linux.
But
that
would
sell
to
say
I
learned
that
the
lit
that
the
internet
is
run
with
duct
tape.
A
That
there's
not
this
like
really
well
oiled
machine
out
there,
that
the
perfect
internet
exists
that
we're
lucky
DNS
works
at
all
and
then
from
there.
I
went
to
Twitter
and
I
was
really
lucky
to
start
when
I
was
kind
of
the
second
programmer
in
ops
and
I
got
a
lot
of
good
projects.
We
replaced
the
ganglion
audio
system
with
a
custom-built
observability.
A
Working
on
this
I
I
have
a
github
account
and
again
these
are
my
very
lovely
dogs.
That's
from
this
morning.
Actually,
so
what
is
this
talk?
What
is
the
deep
dive?
I
have
like
25
minutes
or
so
to
do
a
deep
dive
I
will
go
as
deep
as
I
can
and
that.
But
what
I'm
not
going
to
do
is
explain
to
you
what
a
service
mesh
is
or
why
you
want
to
use
it
or
any
of
that
stuff.
A
So
we
started,
we
launched
linker
D
at
coop
Connie
you
in
2016
that
one
the
left
there
that
then
about
a
year
later,
we
we
gave
it
to
the
CNC
F
and
that
year
was
really
getting
our
first
set
of
production
users,
making
them
successful,
helping
them
be
successful,
and
once
we
had
really
proven
out
that
linker
D
was
a
valuable
thing
that
people
wanted
to
use.
We
got
more
serious
about
the
open
source
community
and
donated
to
CN
CF
as
a
kind
of
investment
to
say
that
this
is
not
a
proprietary.
A
A
You
know
these
are
a
lot
of
big
fancy
logos,
but
we
also
have
a
lot
of
small
little
logos
that
aren't
up
here
that
are
using
this
to
power,
small
ecommerce
shops
and
banks
and
what-have-you,
and
it's
a
pretty
active
project.
We
have
a
lot
of
real-world
users
solving
problems
with
it
for
who
is
at
Kubb
con
in
Austin
last
year,
who
saw
my
like
keynote
demo,
where
I
got
up
and
tried
to
do
a
yeah,
so
we
launched
this
project
called
conduit.
Then,
and
conduit
was
kind
of
a
departure
from
link
reading.
A
A
We
we
gave
it
back
to
the
scene
CF
and
merge
it
in
as
link
ready
to,
and
we
think
this
is
the
future
of
the
project
and
the
real
hypothesis
that
we,
the
shift
going
from
liquidy
one
delinquent
to
is
really
on
these
three
tenants,
one
being
lightweight
and
we've
called
ourselves
the
ultralight
service
mission
and
that's
for
a
reason
and
lightweight
both
in
terms
of
resources.
Their
proxy
takes.
You
know
less
than
10
Meg's,
it's
quite
small
and
but
also
lightweight
in
terms
of
cake
complexity.
A
I
don't
want
you
to
have
to
go,
learn
something
to
get
value
out
of
a
service
mesh
like
well
I've.
An
aside
here
that
service
mesh
is
a
pattern
like
the
actor
model.
You
can
build
all
sorts
of
different
applications
with
the
actor
model.
They
all
have
different,
focuses
and
strengths
and
weaknesses
with
a
service
mesh.
We
don't
think
you
have
to
go,
learn
some
big
policy
or
compliance
system
to
go,
get
started.
A
We
want
something,
that's
really
lightweight
conceptually
that
you
can
reason
about
yourself
at
3m
without
calling
me
and
a
big
part
of
that
is
not
having
magic.
If
there's
a
feature
that
you
don't
understand,
that's
magic
and
if
you're
on
call
for
that,
no
be
great
for
demo
right
in
the
conferences,
we
have
all
these
magic
features
which
everyone
who
is
in
Oz
at,
but
when
you
go
to
ploy
that
and
a
month
later
and
like
you
don't
know
my
phone
number
and
you
wake
up
with
an
incident.
A
How
are
you
going
to
figure
out
what's
going
on?
Well,
if
you're
in
a
cloud
provider,
maybe
you
can
call
your
cloud
s
eries,
but
a
lot
of
the
customers.
We
have
need
us
to
be
able
to
solve
these
problems
themselves
and
with
us.
Hopefully,
although
we
do,
you
know,
we
have
a
bunch
of
support
relationships
and
those
are
great,
but
what
we've
learned
from
that
is
that
we
need
to
give
you
the
ammunition
and
in
a
service
mesh,
because
this
is
the
first
hop
into
a
network.
A
The
service
mesh
is
guilty
until
proven
innocent
and
that's
how
it
should
be
so.
Linker
D
needs
a
ship
with
an
alibi
it
needs
to
like
you
need
to
be
able
to
introspect
and
say:
did
you
mess
up
or
is
this
the
network,
or
is
it
something
else
and
that's
really
informed
our
strategy
in
terms
of
future
development?
A
That
is
the
goal
of
Linkwood
e
that
if
you
run
this
thing,
you
have
to
be
able
to
budget
yourself,
and
that
brings
us
to
the
third
point,
which
is
tactical,
not
strategic,
strategic
things.
You're
things
you
have
to
get
the
whole
company
on
board
with
you,
have
to
get
all
the
architects
in
the
room
around
the
white
board
and
have
a
consortium
to
decide.
What
is
the
one
true
service
mesh
for
you
know
beyond
corp
or
whatever
you're
at.
A
We
don't
think
that
has
to
be
the
case,
our
lincolnís
designed
to
be
tactical
and
there
it
means
incremental.
So
you
can
use
it
to
solve
specific
problems
today.
You
can
install
it
today
and
it
will
help
you
and
it
will
grow
with
you
and
you
can
insult
to
one
service.
You
don't
have
to
install
to
a
whole
cluster,
let
alone
your
whole
company
and
then
you
can
expand
from
there
and
it
doesn't
get
more
complex.
A
A
Now
we
have
a
web
and
CLI
that
uses
an
interface
which
is
in
a
linker,
D
namespace,
or
maybe
your
own
namespace,
depending
we
should
put
a
couple
small
micro
services
there
and
we
also
bundled
Prometheus
and
graph
on
ax
and
that's
not
to
replace
your
Prometheus
or
fauna
and
in
fact
we
don't
want
to
do
any
long-term
storage
of
time
series
here
at
all.
We
want
this
to
be
something
that
lets.
A
You
ask
questions
of
the
proxies
now
people
when
they
go
to
production
10
either
configure
their
own
Prometheus's
to
scrape
scrape
from
errors
or
to
scrape
from
the
proxies
directly.
We
don't
prevent
any
of
that,
but
this
is
meant
again.
You
install
this
bundle
and
it's
immediately
useful.
You
don't
have
to
go,
build
other
things
or
configure
dashboards,
or
do
any
of
that
work
and
all
of
that
modulo
some
of
the
web
stuff
is
written.
Go.
A
We
also
have
react
and
some
other
technologies
in
there,
but
and
the
choice
for
that
is
to
use
client
go
I've
in
Lincoln,
e1
written
a
kubernetes
client
in
Scala
and
almost
well
I,
don't
know
how
what
percentage,
but
a
large
percentages
of
bugs
and
linker
D
with
kubernetes
have
been
related
to
that
API
client
and
changes.
Subtle,
subtle
changes
in
the
versions
between
releases
and
things
like
that
that
are
quite
hard
to
debug,
and
so
our
grand
genius
idea
was
well.
Client
goes
perfect
right.
A
A
Then,
in
the
data
plane
now
we've
elected
to
write
our
own
proxy
and
rust.
Anyone
here
write
rust.
Anyone
here,
wanna
write,
rust,
cool
yep
more
next
year,
it'll
be
way
more
hands.
I
promise
rust
is
anyway
are
not
know
what
rust
is
I
thought
so,
okay,
yeah,
so
rust
is
a
modern
programming
language,
its
memory
safe.
A
A
This
all
makes
sense
so
far
great
so
I
want
to
start
and
go
through
a
quick
walkthrough
of
how
we
get
bootstraps
with
linker
D
I'm,
just
going
to
hit
a
bunch
of
technical
notes
as
quickly
as
I
can
and
then
hopefully
get
to
questions
at
the
end.
If
we
have
time
so,
the
problem
is,
you
have
an
application
and
we
have
to
add
a
proxy
to
your
application.
How
do
we
do
that?
Well,
we
have
this
linker
T
inject
logic.
On
the
left
hand,
here
we
have
a
bunch
of
uninjected
man.
A
We
have
a
Koopa
titties,
manifest
with
a
bunch
of
deploys
and
services
and
demon
sets
whatever,
and
it
just
has
apps
and
nothing
special.
We
then
transform
it
through
the
magic
of
UNIX
pipes
through
a
link
ready
inject
and
that
turns
into
a
new
manifest
with
a
proxy
in
it
container
and
a
proxy
container
in
the
resources
that
deserve
those
things
and,
as
we
heard
in
the
last
talk,
Ivan
just
developed
an
experimental
feature
to
replace
that
link
ready
inject
command-line
tool
with
a
mutating
web
hook.
A
That
will
do
this
in
the
server
in
the
cluster
directly,
and
here
we
can
see
if
there's
a
whole
bunch
of
options
that
you
can
configure
when
you're
doing
an
injection-
and
this
is
really
around.
How
do
we
configure
the
proxy
in
it
contain
in
the
proxy
unit
container,
is
really
there
just
to
configure
iptables
rules.
So
all
the
traffic
going
into
the
pod
goes
to
the
proxy
and
all
the
TCP
traffic
coming
out
of
the
proxy
goes
going
out
of
the
pod.
It
goes
through
the
proxy.
A
This
requires
nedderman
capabilities,
which
is
kind
of
a
deficiency
right
now,
but
we
have
a
whole
bunch
of
interest
on
replacing
that
likely
with
a
scene,
I
plugin.
So
that's
some
upcoming
work
there
again.
So
this
adds
two
pods
or
two
containers
to
each
pod
resource
described
and
manifest
an
innate
container,
which
manages
IP
tables
rules
and
a
proxy
container,
which
is
this
rust.
Application
deals
with
trends
for
transmitting
and
proxying
TCP
connections.
A
So
I
want
to
do
a
quick
dive
through
the
life
of
requests
to
kind
of
walk
or
help.
You
give
some
context
of
how
the
proxy
is
built
and
what
it
does
so
we're
going
to
be
focusing
on
this
data
flow
going
through
the
proxy
and
are
actually
I'm
going
to
talk
mostly
about
the
outbound
side,
so
going
from
the
application
out
of
the
pod
to
another
service
or
an
external
service,
and
there's
just
very
slightly
different
logic
on
the
inbound
side.
A
A
So
I
wanted
that
when
we
accept
a
connection,
we
just
get
something
from
the
kubernetes
network
right
and
that's
coming
in
through
the
IP
tables
rules
that
we
configured
with
the
the
inject
command
in
the
unit
container.
And
what
that
gives
us
is.
We
have
a
TCP
server
that
accepts
connections
and
we
use
the
socket
option
original
desk
to
determine
where
that
traffic
was
initially
headed.
A
So
if
you're
sending
traffic
to
Google,
calm
or
better,
yet
Twitter
comm,
as
you
go
through
through
the
pod,
it's
going
to
get
hit
the
proxy
and
we
now
are
no
longer
sending
it
doesn't
like
the
destinations
of
proxy.
So
we
have
to
ask
the
kernel:
where
was
this
socket
going
before
we
intercepted
it?
And
so
we
maintain
that
information
as
we
make
our
rad
and
decisions
below.
A
So
if
you
have
TLS
connections
that
are
going
to
the
proxy
and
a
proxy
did
not
send
that
it'll
just
forward
it
through
as
normal
TCP
as
if
nothing
ever
happened,
and
so
our
philosophy
there
is
to
kind
of
fail
open.
We
don't
want
to
have
a
big
mandatory
thing,
we're
all
break
existing
applications.
We
want
this
to
layer
in
and
add
value
wherever
it
can
and
well.
A
A
A
So,
with
HTTP
routing
we
take
both.
We,
we
decode
through
either
h1
or
h2,
depending
on
the
protocol
we
detect,
but
everything
else
in
the
proxy.
Then
it
just
gets
merged
in
and
the
only
difference
between
them
in
our
logic,
is
the
HP
version
on
the
request,
metadata
and
you'll
see
why
that
plays
them
later
so
HP
one
comes
in.
A
A
So
if
it's
h2
there's
an
authority
header,
if
it's
an
interview,
one,
we
look
in
either
the
resource
for
the
authority
or
look
at
the
host
header
and
if
you're
in
some
weird
HB
1o
situation,
will
use
that
original
desk
address
from
that
socket
option
as
the
destination
to
route
2.
And
then
we
have
this
Kanak
canonicalization
logic,
and
here
is
what
you
know,
because
the
names
can
go
through
the
proxy
and
be
relative.
You
can
have
foo
and
food
LMS
and
Futaba
mess
that
service
that
clustered
at
local
all
going
through
the
proxy.
A
A
What
are
the
policies
for
this
name,
food
out
service
to
closer
about
local
or
google.com
or
twitter.com,
or
anything
we
can
go
to
the
control
plane
and
say:
do
I
have
policies
for
that
and
right
now,
policies
are
basically,
you
can
think
of
it
like
open,
API,
swagger,
spec
type
things,
and
that's
allows
us
to
hydrate
metrics
here
and
response
classification
response
classification
being.
How
do
I
know
whether
this
request
should
be
considered
a
success
or
failure
for
the
purposes
of
success
rate,
calculation
or
retries?
A
We're
not,
we
don't
have
retries
yet,
but
that
will
be
in
coming
out
towards
the
end
of
January.
Once
we
go
through
that
route.
Second,
there's
many
routes
tax,
so
you
you
download
a
bunch
of
these
from
the
control
plane.
That's
all
what
a
G
RPC
watch
is
they're.
All
of
the
API
is
with
the
control
planner
GRP
see.
A
So
when
you
go
from
atheists,
you
actually
have
all
of
your
kubernetes
resources
in
the
labels
and
dimensions.
So
you
can
ask
questions
like
what
is
the
success
rate
in
this
namespace
and
that
just
Falls
up
the
previous
model
there's
nothing
special.
We
have
to
do
other
than
add
these
labels
from
the
proxy
API.
The
the
proxy
itself
knows
nothing
about
kubernetes
at
this
point,
it's
just
dealing
with
names
and
labels
and
very
generic
things.
A
So
we
get
that
and
we
also
get
a
little
bit
of
extra
metadata
there,
including
is
there
a
proxy
installed
on
the
other
end
of
this
on
the
other
pod?
Here,
if
there's
a
proxy
installed
there?
Well,
we
might
be
able
to
do
TLS
upgrading
and
we
can
also
do
protocol
upgrading
which
I'll
get
to
in
a
second.
If
none
of
that
exists,
though,
and
we
don't
necessarily
go
to
the
proxy
API
for
all
domains,
it's
only
white
listed
for
service
that
cluster,
don't
local
right
now
that
gets
configurable.
A
But
if
that's
not
there,
then
we'll
fall
back
to
Venus
again
to
not
break
anything
and
we'll
do
load.
Balancing
based
on
that
as
well,
which
is
still
better
than
ku
proxy,
will
do,
will
do
requester,
we're
low
daunting,
and
so
that
means
we
measure
the
latencies
for
each
request
the
proxy
sends
to
each
endpoint.
We
measure
the
latency
from
the
time
the
request
is
sent
until
the
first
byte
comes
back:
it's
not
the
headers,
the
first
byte
and
the
payload,
and
that
lets
us
actually
get
really
good
requests
to
our
load.
A
We
have
another
set
of
policies,
and
so
we,
the
end
point,
is
the
IP
whether
we
can
HQ
upgrade,
whether
there's
TLS
and
again
that
set
of
metric
labels
and
within
there
we
have
another
TLS
watching
use
system,
and
that's
if
this
endpoint
supports
the
proxy
TLS
and
there
are
certificate
credentials
local
then
we
will
the
rest
of
the
below.
This
stack
will
be
aware
of
that.
Tls
configuration
within
that
is
a
metric
system,
and
so
we're
recording
you
know,
request,
totals
response,
total,
latency,
etc
for
every
request.
A
This
is
really
limited
to
the
metadata
about
the
routing,
but,
as
Williams
said
before
at
the
point,
we,
after
the
last
release
the
proxy
little
place
for
adding
payloads
and
arbitrary
data.
There
is
not
really
technically
difficult.
It's
mostly
aux
problem
that
I
really
want
to
solve
the
next
few
months
or
I
really
want
you
to
help
me
solve
the
next
few
months,
maybe
okay,
anyway,
from
there,
we
we
also,
we
get
this
information
back
from
the
cluster
from
the
the
communities
API
that
are
from
the
approxi.
A
That
tells
us
whether
there's
a
proxy
running
out
of
the
other
side.
If
there
is
a
proxy
other
side
well,
we
can
do
HTTP
to
upgrading
only
in
that
proxy
only
to
proxy
proxy,
so
your
application
doesn't
have
to
speak
HP.
We
all
have
one
multiplex
connection
between
proxies,
at
which,
in
a
TLS
world,
gives
you
a
lot
of
our
metro
station
of
costs,
so
you're
not
reestablishing
connections
for
every
request.
We
can
just
trunk
this
all
safely
and
efficiently
and
then
from
there
we
were
to
say
based
on
all
these
decisions
we've
made.
A
Is
this
go
out
an
h1
client
or
an
h2
client?
So
we
can
really
do
protocol
upgrading
and
downgrading
with
some
copy
ups
just
to
the
proxy
configuration
okay
and
now
for
the
fun
part
of
the
talk.
I
want
to
talk
a
little
bit
briefly
about
our
use
of
rust
and
how
and
why
and
hopefully
be
able
to
get
some
impressions
at
the
end
so
number
one
strong
typing.
This
might
look
like
a
bunch
of
gobbledygook
and
when
I
got
started
with
rust.
Just
like
really
hurt
my
head,
but
now
I've
learned
to
love
this.
A
This
is
a
beautiful
explanation
of
a
contract
and
this
is
actually
the
metric
service
that
we're
using
and
it
is
totally
type
safe
and
memory
safe
and
all
these
things
and
generic
over
the
types
of
modules
underneath
the
service
is
underneath
it,
and
so,
if
anyone's
written
finagle
before
Georgie.
This
might
look
really
familiar,
though
a
little
bit
different
and
this
this
is
super
powerful.
I
can't
really
explain
how
powerful
this
is.
But
the
point
here
is
we're
able
to
build
region
arrack
modules,
so
the
metrics
model
module.
A
We
can
test
totally
independently
of
the
proxy.
In
fact,
we
use
it
in
multiple
places
in
the
proxy,
so
this
exists
modules
used
in
the
routing
layer
where
we
have
these
per
metrics
and
also
in
the
endpoints
layer,
and
if
we
want
to
add
other
scopes
and
metrics,
we
can
just
copy
well
not
copy
and
paste
the
code
instantiate
new
instances
of
this
module
elsewhere,
and
this
type
of
modularity
is
super
powerful.
When
building
modular,
correct
systems.
A
We
have
a
service,
that's
that's
doing
the
lookup
and
that
actually
goes
and
send
some
requests
to
the
API
and
they'll.
Have
a
stream
open?
That's
watching
the
request
to
the
API,
and
if
this
service
is
to
be
deleted,
say
it
falls
out
of
the
cache
and
we
don't
need
it
anymore
to
idle
times
out.
Well,
all
of
that
can
just
be
garbage
collected
without
having
to
do
a
real
garbage,
collector,
and
so
just
by
dropping
their
references
in
the
listener.
A
Everything
else
can
just
be
dropped
for
free
if
you're
doing
this
correctly,
and
so
here
is
in
the
metric
system
where,
when
the
request
body,
so
when
the
request
is
done,
but
when
the
the
the
reader
of
their
sorry
the
response
when
the
reader
of
the
response
is
done
processing,
it
drops
the
object
and
LLVM
will
say:
okay
when
this
object
drops.
We
have
to
record
latency
here.
So
this
is
kind
of
similar
to
finalize
errs
and
in
other
languages,
but
rusts
primitives
here
and
assurances
and
like
the
the
model
here,
is
really
compelling.
A
A
We've
built
a
lot
of
kind
of
core
libraries
here,
so
everything
from
the
hiv-2
implementation
is
a
native
imitation
rus
that
we
wrote,
as
is
the
g
RPC
implementation,
which
so
now
you
have
contributors
from
google
working
on
that
and
the
load
balancer
we've
written
there
and
the
router
we've
written
there
and
the
circuit
breaking
is
being
added
there.
So
there's
actually
a
big
framework
of
components
and
libraries
that
you
can
go
use
to
build
your
own
proxies.
If
you
want
to
but
I
hope
you
go
build.
A
Libraries
there
to
help
us
make
link,
could
be
better
that'd,
be
even
better,
but
the
point
is:
there's
a
whole
ecosystem
that
underlies
linker
D
that
is
being
used
at
other
notable
companies
like
Amazon
and
Dropbox
they're,
building
on
the
same
stack
that
link
you
to
use
both
on
okay
and
so
coming
up.
We're
really
even
continued
investment
service
profiles.
A
But
with
that,
the
next
step
will
be
setting
retries
and
latency
slows
on
those
routes,
so
the
uploading
a
photo
endpoint
is
gonna,
have
a
different
retry
policy
and
a
different
timeout
policy
than
the
download
a
tweet
endpoint
right,
and
we
have
to
be
able
to
describe
this
thing,
so
the
descriptions
just
formed
into
one
and
the
policies
will
start
to
land
following
that,
including
rate
limiting
and
and
I.
Think
probably
Canaries
will
happen
before
really
winning.
A
But
another
big
initiative
right
now,
as
I
mentioned,
is
dropping
the
Nedim
in
require
so
that
you
don't
have
to
grant
all
of
your
pods
and
editing
capabilities
and
hopefully
just
have
that
as
a
cluster
or
owner
automatic
injection
is
ready.
People
are
using
an
Impreza
know.
Thanks
will
and
Ivan
and
I
expect,
there's
a
few
more
kind
of
UI
tweaks.
We
want
to
do
before.
We
call
it
stable,
but
I'd
expect
that
in
a
January
time
frame,
I
hope,
Ivan,
yeah
really
come
get
involved
and
help
us
make
that
good
again.
A
The
Diagnostics
are
really
core
and
key
to
what
we're
doing
here
and
if
it
has
to
be
double
and
one
of
the
problems
I
think
in
the
proxy
is
the
logs
are
kind
of
a
mess
just
because
we
have
a
whole
bunch
of
libraries.
All
logging
different
things
and
getting
the
full
firehose
of
that
can
be
quite
hard
to
debug,
and
so
we're
actually
building
a
into
the
core
networking
libraries
all
the
way
down
as
far
low
as
we
can
a
tracing
system
to
do
structured
logging.
A
It's
that
allow
us
to
subscribe
to
events
and
actually
have
know
the
structure,
so
I
can
say
for
requests
to
this
pod.
Show
me
all
the
logs
and
I
can
get
really
structured
data
about
just
that
stream
in
that
connection,
without
having
all
the
other
things
and
will
be
able
to
tune
that
turn
that
on
and
off
at
runtime
from
the
control
plane,
so
that
you
can
debug
this
fear
yourself
or
you
can
give
us
really
good
bug
reports.
Another
thing,
I
think
that's
pretty
easy
and
I
would
encourage
a
contributor
get
involved
here.
A
Is
that
since
we're
using
Prometheus-
and
we
have
a
control,
plane
and
Prometheus
alerts
are
just
Prometheus
queries?
What
we're
really
looking
to
do
is
build
a
library
of
alerts
that
we
should
have
on
the
control
plane
itself
or
on
common
aspects
of
the
data
plane,
so
that,
if
you
want
to
point
alert
manager,
our
Prometheus
instance,
you
can
get
alerts
for
free
about
lease.
The
health
of
the
control
plane
itself.
A
If
you
don't
have
that
already,
but
we
think
it
should
be
pretty
easy
to
have
out-of-the-box
things
like
that
of
is
link
ready,
healthy
again,
so
that
we
come
with
our
own
alibi.
So
you
don't
have
to
do
the
murder
mystery
thing
and
one
of
the
things
that
is
a
little
bit
of
a
science
project
that
we're
starting
to
work
on
is
deterministic
resource
consumption.
A
You
can
throw
more
memory
at
the
proxy
and
it'll
be
able
to
way
more
buffering
and
way
more
concurrent
work
and
that's
okay,
but
that
comes
at
the
cost
of
memory
write
a
book
we
can
manage
latency
or
you
can
say
well,
io,
don't
want
this
proxy
to
take
more
than
10
Meg's
and
let
the
latency
be
damned
right.
We
want
to
have
back
pressure
there
in
a
really
good
way,
and
so
we
have
a
bunch
of
the
fundamentals
are
moving
there.
Things
like
having
a
channel
implementation.
A
If
you're
really
running
out
the
details,
we
have
a
multi
producer,
single
consumer
chat,
elephant
ation
right
now
it
allocates
more
than
it
should,
and
so
we're
really
stripping
allocations
out
of
that.
So
we
can
be
very
predictable
in
terms
of
these
things
that
I
think
will
be
get
to
the
point
where
there
is
a
mathematical
of
an
easy
mathematical
formula
to
determine
the
resource,
consumption
of
a
proxy
base,
no
latency
target
and
I
hope
we
can
do
that
this
year.
A
A
I
add
them
up
here.
Told
me:
I
have
to
tell
you
all
that
we're
hiring
but
we're
working
about
you
really
cool
stuff,
we're
working
on
the
rest,
proxy,
we're
hiring
for
the
control
plane
and
the
UI.
We
just
want
people
who
really
want
to
work
on
this
stuff.
So
if
you
really
want
to
work
on
the
stuff
come
to
talk
to
this
guy
up
here
afterwards,
we
have
a
website
with
a
bunch
of
Docs.
A
I
actually
saw
some
of
you
reading
the
docs
during
last
talk,
that's
pretty
cool
and
we
have
a
slack
where
you
can
come
chat
with
us
and
there's
again
a
pretty
active
community.
There
are
people
willing
to
help
who
don't
all
work,
a
boy
and
then
we
a
bunch
of
github
repos
and
so
the
linker
D
org
has
where
Lincoln
you
wanna,
link,
82
and
only
52
proxy
are
housed.
Hyperion
is
the
HTTP
library
or
hyper.
A
Is
the
HTTP
library,
that's
kind
of
the
rust
of
factory
HTTP
library,
the
author
of
that
works
at
buoyant
and
we're
really
tightly
integrating
that
into
linker
D
lincolnís
built
on
hyper
and
tokyo.
Tokyo
is
networking,
asynchronous
networking,
library
and
rust.
You
can
think
of
it
similar
to
Neddie
you
Java
folks
or
I.
Don't
know
what
the
other
equivalents
are:
Lib,
Eevee,
I,
guess
and
Tower
is
our
finagle
replacement
and
rust,
and
so
Tower
is
a
service
abstraction.
This
is
a
bit
of
load
balancer
lives.
This
is
where
the
retrial
was
live.
A
This
is
where
the
rate
limiters
live,
and
so
this
is
a
bunch
of
reusable
components
that
you
can
go
build
web
services
or
proxies
or
whatever
you
want,
and
all
of
this
like
this
is
all
investment.
The
point
has
made
in
the
open-source
community
on
the
core
primitives
under
Liberty
to
that
again
are
used
in
a
way
more
places
than
liquidity
right
now,
like
the
Dropbox,
client,
I
believe
desktop
client.
Finally,
I
have
a
Twitter
account.
It's
pretty
bad,
don't
follow
it
anyway.
A
A
A
A
A
Are
there
any
scaling
considerations
right
now?
The
proxy
is
not
actually
multi-core.
Asha
said
this
right.
We
only
have
one
tour
for
proxying
and
an
admin
core,
and
so
you
will
hit
some
limit
there
and
especially
with
low
memory
in
terms
of
how
much
concurrency
you
can
get
through
the
single
proxy
there's,
some
work
going
on
right
now,
I
think
again,
q1
time
frame
will
be
all
scaled
it
up
and
once
we
can
scale
it
up,
we'll
start
to
tackle
ingress
use
cases,
and
things
like
that
there
than
that.
Please
find
some
bugs
for
me.
A
B
A
A
Api
is
to
innovate,
we're
a
start-up
and
not
IBM,
and
so
we
want
to
be
able
to
iterate
really
quickly,
and
that
means
that
we
can
make
changes
in
the
proxy
and
in
the
control
plane
and
the
same
version
and
just
make
those
together,
and
so
we
like
that,
coupling
that,
but
it's
they
are
decoupled
and
that
there's
just
a
G
RPC
interface
between
the
control,
plane
and
proxy.
So
it's
not
really
that
hard
to
go
write
a
new
control
plane
for
the
proxy
that
maybe
is
an
adapter
to
those
other
control
planes.
A
A
The
Federation
between
namespaces
is
the
same
thing
as
federation
between
clusters
to
some
degree,
and
so
our
plans
there
to
really
nail
the
identity
model
and
make
sure
we
get
the
mutual
tol
story,
bailed
and
then
once
we
have
that
we'll
expand
outside
the
cluster,
but
yeah
definitely
a
2019
thing
and
we
have
an
issue
open
on.
It
definitely
comment
on
it
or
give
it
a
thumbs
up
or
whatever
that
lets
us
know
you
care
about
it
and
things.
A
question
up
here:
Adam.
D
A
Right
now,
link
video
out
of
the
box
says
this
optional
TLS
module,
where
it's
not
doing
mutual
TLS,
but
it's
doing
the
server-side
certificates
and
it's
automatic
evasion.
So
you
don't
have
to
worry
about
it
or
anything
like
that.
There
was
a
demo
in
the
last
talk
where
we
a
step
is
building
an
other
CA
that
can
do
this
as
well,
and
that
will
be
the
mutual
T
loss
story.
D
A
A
really
good
question
people
I,
please
open
an
issue
on
the
link
ready
to
repo
they're.
Like
you
know,
this
is
other
people.
Ask
me
at
this
conference
about
that
yeah
and
we
I
just
want
these
forums
where
other
people
can
go.
Add
to
that.
So
it's
not
there
yet,
but
like
it's.
If
the
community
wants
it
and
people
will
use
it
if
we
build
it,
it'll
get
built
cool.
D
To
do
with
protocols
right
yeah,
are
you
saying
that
you
essentially
have
a
mechanism
that
allows
you
to
kind
of
detect?
What
protocol
were
you
talking?
So
then
you
use
the
appropriate
one
yeah,
but
I
suppose
at
the
moment
is
TCP
and
HTTP
I
think
you
said
yeah
all
right
is
any
intention
or
any
any.
You
have
any
intention
to
add
anything
beyond
that.
D
So
you
mentioned
my
sequel,
for
example,
right
so
I'm
familiar
with
other
proxies
that,
like
pre,
not
even
preemptively,
they
specifically
say
that,
like
a
particular
port
is
going
to
speak,
let's
say
my
sequel
right,
so
you
can
like
you,
can
do
a
lot
of
cool
stuff.
If
you
know
stuff
ahead
of
time
yeah.
Is
there
any
intention
to
kind
of
put
this
into
like
a
D.
A
Yes,
with
the
caveat,
so
you
know
in
the
kubernetes,
like
environment
or
distributed
environment,
port
reservations
are
not
really
a
real
thing,
and
so
you
can't
like
if
someone
was
like
Oh
influx
TV
runs
an
ad
servus
speaks
first,
so,
like
you
should
whitelist
that
and
like
well,
our
proxy
api
around
the
net
port.
So
what
do
you
want
me
to
do?
But
I
definitely
think
that
will
we
do
have
a
set
of
default,
well-known
protocols
that
we
will
just
pass?
A
There
is
TCP
without
configuration
today
and
as
far
as
being
aware
of
those
other
protocols,
it
really
depends
on
how
that
protocol
works.
Some
protocols
really
work
well
with
middlemen,
like
HTTP
their
protocols,
like
my
sequel,
Allah,
so
because
they're
very
stateful
in
terms
of
the
protocol,
and
so
if
there
are
specific
protocols
that
you'd
like
to
get
and
really
what
we
get
there
is
the
stats
and
the
routing
right.
A
D
A
D
So
specifically,
the
reason
why
I'm
asking
about
protocols
is
because
it
kind
of
ties
in
to
what
I
was
asking
in
the
earlier
question
about
authentication
right.
So
you
have
something
like
my
sequel,
which
clearly
has
like
an
authentication
strategy
which,
like
you're,
not
gonna,
do
mutual
TLS.
You,
you
have
to
provide
credentials.
So
then
I
was
wondering
if
there's
like
an
integration
point
in
link
D,
where
you
can
essentially
have
to
say
well,
I
just
want
link
a
deed
to
kind
of
do
the
handshake
for
me
and
then
shuttle
bites
it
off
right,
yeah.
A
E
Hey
so
quick
question,
this
is
more
about
go
versus
Ross
panada,
at
a
technical
level,
more
community
levels.
Yeah
have
you
found
that
you
know
switching
to
rust
for
or
starting
the
proxy
and
rust
in
terms
of
like
open-source
contributors
like
have
you
found
that
they
might
have
had
like
a
harder
time
just
coming
up
with
rust
and
the
ramp
up
time?
For
that.
A
Yeah
and
the
very
fact
that
we've
had
to
build
all
of
these
libraries
and
rust
is
kind
of
a
testament
to
the
fact
that
Yukos
was
not
there
and
I.
Think
for
a
lot
of
people
want
to
go,
build
web
frameworks,
it's
just
really
just
not
there
yet
and
that's
okay.
We
have
had
a
where
we
have
seen.
Community
interaction
in
rust
is
the
community
of
like
serious
production
users
who
want
to
build
things
like
systems,
people
at
Amazon
and
Dropbox,
etc.
Mit
we
have
a
bunch
of
community
there.
A
That's
we're
really
working
on
Tower
in
Tokyo,
etc,
less
than
the
proxy.
So
we
have
not
had
a
lot
of
external
contribution
in
the
proxy
and
that's
actually
been
okay,
because
it's
been
changing
really
rapidly.
I
expected
in
2019
I
really
want
them
to
change
so
yeah.
It's
been
way
easier
for
people
to
get
in
and
contribute
to
the
NGO
side,
but
part
of
our
investment
in
rust
is
to
raise
the
bar,
so
everyone
else
can
get
there
too,
but
we
were
definitely
believe
that,
like
rust,
is
a
good
bet.