►
From YouTube: JustFootball’s Journey to gRPC + Linkerd in Production - Ben Lambert, & Kevin Lingerfelt
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
JustFootball’s Journey to gRPC + Linkerd in Production - Ben Lambert, JustFootball & Kevin Lingerfelt, Buoyant
Ben (Just Football) will talk about their journey with Kubernetes and microservices from the world of HTTP/1.1 to HTTP/2.0 and gRPC. He will talk about the reasons for moving, and the best practices Just Football adopted for using gRPC in production, including monitoring + design decisions and distribution of gRPC proto + clients. Kevin (Buoyant) will cover how Linkerd provides Just Football with observability and load balancing for their gRPC services. He'll also describe how the Linkerd project itself employs multiple gRPC features to facilitate robust communication between its control plane and its data plane.
https://sched.co/MPf4
A
Okay,
so
you
stopped:
oh,
yes,
honey,
okay,
so
this
is
just
footballs
generator,
G,
RPC
unlink
it
in
production.
If
you
skip
the
neck
side,
sorry
Ben
I
used
to
be
the
CTO
of
just
football,
and
when
this
talk
got
approved,
I
still
was
and
then
in
the
distance
between
not
talking
now
I
accept
the
job
as
partly
fire,
so
I
now
work
now
instead,
but
none
of
the
content
in
this
talk
made
me
leave
just
football,
so
you're
still
going
to
learn
a
lot
of
stuff,
it's
still
relevant
yeah
and.
B
I'm
Kevin
Lee,
your
fault,
I
work
for
buoyant
I've,
been
there
for
about
four
years
and
I
am
a
core
contributor
to
linker
D
prior
to
buoyant
I
was
working
at
Twitter
for
five
years
sort
of
took
part
in
ripping
apart
a
giant
monolithic
application
and
converting
it
to
micro
services.
So
I'm
very
familiar
with
just
footballs
story
as
well.
I'm
really
excited
to
be
speaking
with
Ben
and
he's
gonna.
Kick
it
off
with
talking
about
just
footballs
yeah.
A
A
Interesting?
Okay,
linka,
d,
yeah,
cool?
Okay?
So
let's
get
start
by
just
football
I'm
not
going
to
talk
too
much
about
the
product
itself
like
what
it
does,
but
we
were
a
startup
founded
in
2016.
We
were
basically
in
Sweden
and
we
were
basically
designed
to
get
kids
into
football.
So
if
you
skip
to
the
next
slide,
so
this
is
what
you
still
like.
So
it
was
like
a
front-end
game,
I
bet,
okay,
I,
guess,
Android
and
iOS.
We
used
to
do
a
lot
of
challenges.
The
idea
is
like
it's
pokemon
golf
or
football.
A
So
you
go
to
a
pitch
you
check
in
to
the
pitch
with
your
friends.
You
challenge
each
other,
and
then
we
give
you
some
points
as
leaderboards
and
coin
rewards.
So
far.
We
also
do
some
cool
stuff
with
like
a
are
so
we
could
basically
track
a
ball
on
a
football
field.
Kicked
by
your
friends
measure
how
fast
it
went,
we
could
do
like
fake
goalkeepers,
so
you
have
to
be
the
goalkeeper
so
far.
So
there's
a
lot
of
cool
challenges
to
solve,
didn't,
try
and
get
kids
on
a
football
Thanks.
A
So
this
is
what
our
text
art
look
like.
So
for
a
start
off
using
Cuba
90s,
that
was
pretty
fun.
A
lot
of
node
react
native.
We
run
everything
Lindy
breaths,
we
have
bits
of
goal.
We
were
using
graphic
wealth,
quite
a
while,
so
before
I
kind
of
get
into
you
like
why
we
chose
the
RPC
and
it's
gonna
go
through
the
monolith.
The
microscope
is
a
story,
so
in
2016.
This
is
what
our
apologies
for
the
terrible
diagram
but
I
have.
This
is
what
our
stuck
use
look
like.
A
So
we
have
the
rat
net
front
end
just
call
in
our
graph
QL
monolith,
which
means
that
all
the
business
logic
for
the
app
was
in
graph
QL
like
it
was
just
dumped
in
there,
and
then
we
have
on
big
massive
database
at
the
end.
So
this
is
2016.
This
worked
kind
of
fine
for
us
and
then,
by
the
time
we
prove
that
it
all
worked.
A
We
wanted
to
split
things
out
and
make
things
a
little
bit
easy
to
build
on
like
when
you've
got
monolith,
everything
just
kind
of
push
it,
and
then
it
becomes
really
hard
to
maintain
and
build
on
so
by
2018.
This
is
what
we
moved
to
so
we
still
have
cabal
in
there,
but
like
it
became
a
composition.
Layer
like
it
was,
it
would
call
out
to
many
many
services
aggregate
the
responses
together
and
then
send
that
back
to
the
client
and
then
obviously
we
supply
the
databases
they
became
smaller
and
billing
with
the
service.
A
Now
this
was
great
until
things
start
to
break,
so
the
thing
is
like
building
a
lot
of
services
and
putting
them
behind
graph.
Ql
is
great
for
splitting
apart
your
monolith,
but
not
so
great
in
terms
of
performance.
So
what
we
found
is
that
under
load
under
pressure
graph,
QL
jatropha
connections
to
any
service
in
sinus
tach,
so
remember
that
one
graph
QL
call
might
call
eighty
different
services
inside
our
suck.
A
That's
a
lot
of
overhead
of
great
and
CPU
creating
connections
to
other
services
internally,
so
we
had
a
lot,
we
kind
of
had
a
lot
of
thought
about.
How
do
we
fix
this,
and
then
we
settled
on
using
your
PC,
so
geo
PC
for
people
that
haven't
heard
it's
based
on
HTTP
HB
2?
It
has
this
protobuf
kind
of
schema
to
define
inputs
and
outputs
a
function
we
support
multiplex
in
for
us.
A
So
this
is
what
so.
This
was
another
main
reason
that
we
switched
from
HTTP
and
rest
is
because
no
one
could
ever
decide
how
to
build
REST
API
inside
just
a
wall.
So
it's
been
days
arguing
about
whether
we
should
pass
stuff
in
the
URL.
We
should
pass
it
in
the
post
body
and
we
could
never
agree
on
this.
So
if
you
imagine
here
you
can
see.
A
So
this
is
what
G
RPC
makes
us.
Do
it
forces
us
to
define
our
inputs
and
outputs
upfront,
so
you
can
see
that
we
have
this
edit
function,
which
has
which
takes
in
types
of
edit
user
requests,
which
is
an
ID
and
a
user
name,
and
then
it
returns
user
response
which
it
so
it's
all
types
basically
gives
you
typed
type
pins
around
your
inputs
and
outputs.
A
So
how
we
did
GL
PC
just
football,
so
we
didn't
have
a
mono
repo
either
so
we
hand
like
say
30
services.
They
were
all
in
different
repositories
in
github,
so
they
were
all
over
the
place
and
the
protobufs
lived
with
the
service,
so
the
service
defined
its
G
OPC
schema
and
it
lived
in
the
same
repository
I
hear.
A
lot
of
people
like
a
railroad
into
place
is
saying
that
managing
job
PC
schemas
is
super
hard.
A
So
we
think
that
we
can't
come
up
with
a
nice
way
to
do
this,
which
is
to
publish
the
g-o-p,
see
service
definition
of
schema
with
the
language
that
are
using,
so
whether
that
is
NPM,
whether
that's
Ruby
gems,
whether
that's
get
off,
we
go,
whether
that's
whatever
is
to
publish
it
to
the
native
ecosystem,
because
that's
really
good
at
managing
dependencies
right.
So
if
we
want
to
do
like
nested
protobuf
schemas,
so
you
can,
including
everywhere.
Then,
let's,
like
the
package
manage
to
deal
with
that,
like
let's
use
a
packing
only
for
its
for.
A
If
you
wanna
know
more
about
that.
Come
talk
to
me
then,
and
we
can
go
through
a
little
bit
and
then
also
get
we
used
to
write
our
own
tax,
quick
definitions
to
so
tax
crypt
was
kind
of
early
support
for
jealousy
I
think
when
we
were
generating
clients.
So
we
ended
up
handwriting
our
own
types
for
everything
which
was
kind
of
pain,
but
now
there
is
support
for
it.
A
I
think
yeah,
so
G
OPC
was
not
a
silver
bullet
for
us,
like
they
still
cause
a
lot
of
problems,
so
one
was
that
we
still
have
no
idea.
What's
going
on
inside
the
system,
all
like
developers
create
new
services
every
day
like
we're
shipping
features
every
week
like
we
still
want
to
push
new
stuffs
and
kids
and
get
them
to
test
new
stuff,
which
means
building
new
features,
which
means
that
the
system
grows
out
of
hand
pretty
quickly,
and
we
have
no
idea
what
talks
to
what
at
any
point.
A
So
that's
kind
of
hard
to
understand,
and
then
we
also
have
another
problem
with
geography,
because
it's
connection
connection
level
load,
balancing.
We
have
a
real
problem,
then,
when
we
want
to
add
more
pods
or
whatever
to
new
services
to
scale
them
out,
because
they're
never
going
to
get
traffic,
because
your
graphical
instance
in
our
case
is
just
connected
to
that
one
GL
PC
pot-
and
it's
never
going
to
change.
How
do
we
load
balance
this
we'd
get
yeah?
B
Thanks
Ben
guess
so,
let's
just
jump
right
into
it.
I'm
gonna
give
a
quick
overview
of
what
link
it
is
and
then
how
it
can
really
help
when
you're
using
gyro
PC.
Let's
see
here,
we
go
so
yelling
ready
service
mash
to
CN
CF
project.
It
has
sort
of
three
pillars
that
it
really
tries
to
help
your
application
developers
with
one
is
observability,
one
is
reliability
and
the
last
is
security
and
I
will
talk
a
little
bit
about
its
architecture
to
see
how
it
accomplishes
all
three
of
those.
B
What's
the
price
of
writing
all
traffic
separately
in
a
different
namespace
you'll
be
running
a
control
plane
and
the
control
plane
is
sort
of
where
all
the
configuration
and
metrics
collection
is
happening
for
the
whole
cluster
and
all
the
proxies
are
connected
back
to
the
control
plane,
vog
RPC
actually,
and
so
they
are
getting
routing
information
from
the
control
plane
on
a
RPC
stream,
and
then
we
have
a
Prometheus
instance.
That's
scraping
all
the
proxies
and
pulling
in
all
the
request
data.
B
So
by
virtue
of
sending
all
your
traffic
through
the
proxies,
you
get
very
good,
uniform
visibility
across
all
of
your
applications.
Now,
by
default
we
will
TLS
all
of
the
connections
between
proxies
as
well,
and
you
can
apply
client
policy
like
retries
and
timeouts
that
kind
of
thing
yeah.
So
specifically
with
regards
to
G
RPC,
you
get
a
couple
really
great
things
in
the
first
is
load
balancing
like
been
mentioned.
So
since
Jerry
PC
is
built
on
h-e-b
2,
an
issue
Mewtwo
is
multiplexed.
B
B
So
you
need
an
application
aware
load
balancer,
one
that
sort
of
knows
the
traffic
that's
flowing
on
the
connections
in
order
to
fan
out
to
all
the
backends,
and
so
that's
exactly
what
I
think
we
do
does
in
your
d-does
request
level
load
balancing
rather
than
connection
level,
though
balancing-
and
in
this
way
it
can
really
distribute
the
traffic
across
all
of
your
backends
evenly
and
also
the
link.
It
uses
a
pretty
sophisticated
load,
balancing
algorithm
based
on
perceived
latency
to
all
the
backends,
so
you'll
get
very,
even
distribution
of
requests
as
well.
B
Let's
see
and
then
like
I
said
once
all
the
traffic
is
flowing
through
the
linker
D
proxies
you
get
really
good,
observability
and
yeah,
and
so
we
were
talking
about
a
lower-level
proxy.
Like
a
TCP
proxy,
you
might
really
only
get
sort
of
bytes
in
and
bytes
out
a
number
of
connections,
but
since
we're
introspecting
the
traffic
on
the
connections,
we
can
give
you
a
lot
more
detail
like
number
of
requests
and
responses
and
with
G
RPC.
We
can
actually
give
you
more
structured
data
about
the
individual
endpoints.
B
So
if
you
have
a
given
G
RPC
back-end
that
has
10
different
endpoints,
you
can
actually
just
pipe
that
your
protobuf
file
directly
into
linker
D
and
then
link
or
you
can
start
reporting
on
all
of
the
endpoints
for
the
service
separately.
So
you
can
see
the
success
rate
of
like
your
get
endpoints
versus
your
create
endpoints,
and
sometimes
you
know
if
the
endpoints
have
really
disparate
levels
of
traffic.
B
A
failure
of
one
of
the
endpoints
might
get
lost
in
the
noise,
but
breaking
it
out
by
endpoint
is
like
really
a
better
way
to
slice
and
dice
it.
You
know
so
you
can
see
here.
This
is
linker
D
output,
but
you
would
kind
of
be
able
to
see
that
in
a
single
endpoint
is
failing,
whereas
everything
else
looks
really
healthy.
B
So
if
I
I'm
a
service
owner
and
you
if
it's
a
rest
service
or
G
or
PC,
backend
or
anything,
the
sort
of
callers
of
your
service
can
configure
their
clients
and
like
really
weird
and
wacky
ways,
and
that
can
cause
problems
for
your
service.
You
can
like
wake
you
up
in
the
middle
of
night,
like
when
I
was
at
Twitter,
I
was
working
on
the
tweet
service.
B
It
turns
out
lots
of
applications
within
Twitter's
architecture
need
tweets
and
they
would
request
them
in
very
weird
ways,
sometimes
with
like
really
aggressive,
timeouts
and
really
high
retry
budgets,
and
that
would
cause
like
a
cascade
of
requests
when
we
started
to
slow
down,
and
that
was
never
good.
So
with
linger
D
we
have
service
profiles,
so
you
can
basically
create
a
it's
a
one-to-one
mapping
with
kubernetes
services.
B
You
create
a
profile
for
your
community
service
and
in
the
profile
you
can
specify
the
client
policy
that
you
expect
callers
to
use
and
then
the
clients
would
sort
of
for
are
creating
their
policy
to
link
or
T,
and
then
Lincoln
would
use
the
correct
policy.
So
you
can
set
timeout
and
a
retry
budget,
and
that
kind
of
thing
there
and
it
sort
of
mitigates
a
lot
of
the
more
nuanced
problems
that
you
get
with.
B
Like
client
configuration
across
multiple
apps
in
a
distributed
system,
cool,
okay,
so
I
think
we
will
do
a
quick
demo
now
of
linker
D
and
G
RPC,
and
then
we'll
have
a
little
bit
of
time
for
questions
cool.
Ok,
so
this
demo
is
available
on
my
github.
If
you
want
to
see
it,
it's
pretty
simple:
it's
a!
We
have
a
app
that
we
created
it's
single
kubernetes,
manifest
which
I'll
go
through,
but
basically
it
spawns
four
deployments.
B
One
is
a
traffic
generator
which
sends
HTTP
1.1
requests
to
a
gateway
application
and
then
the
Gateway
fans
that
out
to
two
backends
and
one
back-end,
is
rest
and
one
is
G
RPC
and
we
sort
of
leverage
some
phone
open
source
projects
to
build
it.
Ok,
so
let's
see
all
right
so
I
have
a
kubernetes
cluster
running
here
and
it
is
already
running
linker
D.
Let
me
bump
the
font
real,
quick,
yeah.
B
Okay,
let
me
just
make
sure
the
linker
D
control
plane
is
up
and
running
and
it
is
a
little
bit
cool.
Alright.
So,
in
this
structure,
I
have
an
app
file
app
demo
and
it
really.
This
is
sort
of
should
look
like
standard
kubernetes
services
and
deployments
are
not
really
doing
anything
special,
but
so
here's
the
rest
back
in
we
have
a
service
in
front
of
it
and
the
deployment
this
deployment
has
an
annotation
on
it.
B
This
says
link
we
do
such
inject
enabled,
which
means
that
this
deployment
will
pick
up
the
proxy
when
we
apply
it
to
the
kubernetes
cluster
same
that's
the
rest
back
end
same
goes
for
the
G
RPC
back
end
and
then
the
Gateway
application
and
the
traffic
application
are
not
inject,
enabled
so
we're.
When
we
initially
apply
this
configuration,
the
two
backends
will
be
injected
with
linker
D,
the
Gateway
and
the
traffic
service
will
not
so
let's
go
ahead
and
do
that.
B
It'll
create
all
of
our
objects,
have
a
look
in
the
default
clusters,
and
these
are
initializing
and
I
can
actually
run
a
link
or
D
check
proxy,
which
will
wait
for
everything
to
come
up
for
me,
but
basically
the
checks
to
make
sure
all
the
pods
are
up
and
link
ready,
uses
a
an
admission
controller
mutating
webhook
to
actually
do
the
proxy
injection.
So
it's
watching
all
pod
creates
and
when
the
pod
with
annotation
shows
up,
it
will
add
the
proxy
for
you
automatically
and
it
should
be
all
set.
B
C
B
Then
we
can
start
getting
stats
from
the
injected
deployments
right
away,
so
we
can
run,
link,
read
e-stat,
deploy
and
yep,
and
you
can
see
that
so
the
Gateway
and
the
traffic
service
are
unmatched
in
this.
You
actually
can
bump
it
just
a
little
bit
more.
Even
there
we
go
yes,
a
gateway
and
traffic
are
unmatched,
but
G
RPC,
back-end
and
rest
are
meshed
they're,
both
at
100%
success
rate.
B
All
these
stats
are
coming
from
the
linker
D
proxy,
like
the
apps
themselves,
are
not
instrumented
to
export
any
data,
but
the
proxies
are
routing
the
traffic,
and
so
that's
how
we
are
able
to
collect
it.
You
can
see,
and
then
you
can
see
sort
of
golden
metrics
for
all
of
the
the
deployments
you
get
their
requests
per
second
latency
percentiles
and
the
number
of
TCP
connections,
and
you
can
see
that
the
G
RPC
backend
is
much
more
efficient
on
TCP
connections.
B
But
if
we
look
at
the
individual
pods
in
each
of
the
backends
Oh
a
little
smaller
yeah,
so
this
is
this
is
exactly
the
load
balancing
problem
that
we
were
talking
about
right.
So
the
rest
traffic
is
being
sort
of
evenly
distributed
across
all
three
of
the
rest
back
ends,
but
the
G
RPC
back
end
is
only
a
single
replicas
receiving
all
the
traffic
and
basically
when,
when
you
use
the
default
load,
balancing
strategy
in
kubernetes
its
connection
level,
though
balancing
and
you're
gonna
end
up
in
this
situation.
B
So
there's
a
couple
workarounds
actually
there's
another
talk
later
today
about
gr
PC
load,
balancing
too
but
I.
Think
liberty
is
a
really
elegant
solution
for
this,
so
we
just
add
linker
D
to
the
gateway
deployment,
the
client
of
the
jaribg
service
that
should
start
balancing
traffic
right
away.
So
let's
go
ahead
and
do
that
actually
I'll
kook
at
will
get
deployed
gateway,
oh
yeah
mo
and
then
I'll
pipe
that
to
link
reading
inject
in
order
to
add
the
proxy
annotation
and
then
I
will
apply
it
back
to
my
cluster.
B
There
we
go
and
so
now
the
deployment.
If
we
look
at
all
the
pods,
we'll
see
that
there's
a
new
injected
gateway
pod
coming
up
that
has
two
containers
in
it
and
the
old
one
will
be
shut
down
momentarily
and
then
right
away.
I
think
we
should
be
able
to
start
seeing
the
change
in
traffic.
I
mean
just
shorten
the
time
window
in
this
yeah.
There
we
go
so
then
now
the
G
RPC
back-end
is
fully
load
balanced
and
the
traffic
is
is
a
lot
there's.
B
This
will
dump
a
bunch
of
data,
but
you
can
get
this
in
different
formats,
but
basically
you
can
see
the
source
and
destination
IP
the
method,
that's
being
called
the
endpoint
that
kind
of
thing
and
then,
if
so
and
there's
a
lot
of
other
CLI
functionality,
but
I
think
he
also
has
a
dashboard
that
you
can
access
as
well.
So
let's
pull
that
up
and
there
was
a
lightning
talk
about
the
dashboard
on
Monday
I'll
bump
the
phone
up
here,
a
little
bit
yeah
and
basically
the
CLI
is
fully
featured.
B
Has
all
the
data
you
need?
The
dashboard
really
lets
us
tell
like
a
better
narrative
about
like
what's
happening
within
your
services,
so
it's
the
another
useful
view
lately
larger
cool.
So
you
know
this
is
the
sort
of
same
thing
we
were
seeing
on
the
command
line,
but
if
I
click
into
one
of
these,
then
I
can
see
kind
of
the
up
streams
and
downstream
deployments,
and
then
all
the
live
request
data
right
here
you
know
we
all
make.
This
is
a
sample
Jerry
co-op.
B
So
if
I
just
click
on
this
for
final
link,
that
will
bring
up
a
dashboard
and
it
has
sort
of
all
the
stats
again
good
visibility,
one
into
an
appointment
and
then
the
other
thing
to
note
on
this
graph
is
that,
prior
to
injecting
the
gateway
deployment,
the
traffic
between
the
Gateway
and
the
G
RPC
back
end
was
not
TLS
and
that's.
You
can
sort
of
see
it
with
the
there's,
no
lock
symbol
for
the
yellow
line
there.
But
as
soon
as
we
injected
the
Gateway.
B
D
B
Then
I
think
that's
about
it
for
the
front
end
yeah
I'm
gonna
come
back
here
nope
here
we
go
so
just
a
quick
recap.
If
you,
if
you
try
the
demo
out
it
also
there's
also
service
profiles
in
the
demo.
I
know
it's,
it
would
take
a
while
to
to
demo
it,
but
you
can
actually
use
the
protobuf
definition
to
create
a
service
profile
for
the
GRP
city
back
in
and
then
start
playing
around
with
its
client
policy
and
I
recommend
you
try
that
oh
yeah,
so
that's
about
it
for
the
MIDI
portion.
B
E
B
F
B
A
Pc,
yes,
we
did,
but
I
don't
have
the
stats
down,
I,
don't
know,
but
we
found
it
way
quicker,
though
like
gee,
obviously
that
the
overhead,
so
sorry,
the
overhead
of
like
the
connections
for
us,
were
the
biggest
dream
that
we
saw
that
wasn't
just
the
latencies
that
was
slow.
It's
like
the
entire,
even
stuff
that
wasn't
done
over
the
wire
right.
From
the
graphical
point
of
view,
that
hole
came
to
a
halt
to
like
the
amount
of
memory
that
we,
the
entire
pod
was
using
was
ridiculous,
but
then
switches
G,
obviously
was
much
quicker.
A
C
Yep,
thank
you
for
the
talk
it's
not
out
like
Lincoln
is
adding
a
lot,
but
you
said
that
yeah
you
had
this
problem
with
gr
PC
load
balancing,
have
you
tried
different
client
load,
balancing
policies
that
would
even
the
load?
That's
a
second
question.
What
version
of
the
linker
G
are
you
showing?
Is
it
I
guess
right?
Yes,.
C
A
So,
for
the
first
question
was
how
we
tried
clients-
sorry
I'd
load,
balancing
I
was
not
really
aware
of
stuff.
At
that
point,
with
clients,
I
love,
bouncing
I,
know:
I
met
a
few
guys
us
neither
doing
a
talk
today
about
client
side,
log
balance
and
in
Cuba
nineties
I
will
be
going
to
that
by.
We
didn't
look
at
that.
No
yep,
here's
one
over
there.
B
E
B
So
as
part
of
the
injection
process,
we
add
the
proxy
container
into
the
pot
spec.
We
also
add
in
an
NIC
container
and
the
net
container
runs
before
the
pod
initializes
and
it
rewrites
the
IP
tables
rules
of
the
pods
so
that
all
inbound
and
outbound
TCP
traffic
go
through
a
port
on
the
link
or
D
router
running
in
the
pod.
E
H
If
I
have
my
G
RPC
service
and
expose
it
on
an
odd
part,
I
want
to
do
my
own
load,
balancer
I,
don't
know
a
chai
proxy
or
whatever
can
I
target
like
as
a
proper
load
balancer
all
the
nodes
where
this
Jerry
PC
is
running.
So
basically
like.
Does
it
work
through
the
ingress
and
does
it
work
if
I
implement
my
own
load
answer
as
a
certain
node
port
sure.
B
E
B
Yeah,
so
you
would
use
one
of
the
existing
kubernetes
ingress
controllers
to
get
traffic
into
your
cluster.
Nginx
works
just
fine
and
then,
if
you
have
your
own
load
balancer,
it
really
I
think
the
main
takeaways
that
you
need
request
level
load,
balancing
for
GR
PC
if
you're
doing
connection
level
load,
balancing,
there's
just
not
enough.
New
connections
created
to
actually
successfully
balance
across
all
your
backends
whew
Mohit.
A
F
That
and
question
around
the
request
streams.
Have
you
had
any
experience
with
leaving
request
streams
or
response
streams
open
for
very
long
periods
of
time
like
hours
or
even
days,
because
I
know
you
can
just
just
do
a
simple
request
response,
but
then
you
can
also
reply
with
a
stream
can't
you
yeah.
E
B
I
A
B
But
yeah
they're,
really
powerful
and
yeah
I've,
seen
mostly
you
see
I've
seen,
is
for
like
streaming
data
back.
You
know
from
the
client
to
the
server
that
kind
of
thing
in
it.
It's
very
seamless
the
other
thing
I
like
about
GRP,
see
it
since
I
used
it
protobuf.
You
end
up
with
language
specific
bindings.
So
when
you're
working
with
the
GF,
you
see
clients
and
servers,
they
feel
really
natural
in
whatever
language
you're
working
in
so
you
know
the
go
implementation
or
whatever
streaming
is
really
straightforward
to
to
work
on.
D
Thank
you,
two
questions
about
self
abilities,
so
certainly
architecture
diagram
that
you
have
premier
desistance,
which,
as
same
as
the
one,
is
also
telling
to
the
CLI
latencies
that
all
the
other
things
would
you
suggest
to
run
a
separate
from
existence
and
then
pull
metrics
from
that
one
or
doing
another
way.
Second,
questions
you're
pulling
out
some
requests.
As
that
anything
we
do
with
tracing
and
the
sling
caddy
support,
tracing
and
now
yep.
B
Two
questions
so
the
first
one
was
about
running
your
own
observability
stack.
My
recommendation
is
to
scrape
the
link
ready,
Prometheus
you
can
use
Prometheus
Federation
Meereenese
Prometheus
is
tuned
for
a
very
short
live
data,
because
it's
sort
of
for
real-time
Diagnostics,
so
typically
folks
in
it.
It
doesn't
really
won't
take
that
my
name
resources
in
your
cluster
anyway.
B
So
typically
folks
will
just
fetter
a
temper
Mathias
or
set
up
a
script
config
to
scrape
all
the
proxies,
because
the
proxies
are
all
labeled
already
for
that,
and
then
your
second
question
was
about
tracing
yeah,
so
we
have
the
tap
feature
to
dump
live
requests.
We're
also
able
to
draw
service
graphs
using
metrics
data
from
Prometheus,
but
we
don't
have
support
for
tracing
right
now.
It's
definitely
on
the
roadmap.
B
It's
I
think
the
thing
that
has
been
difficulty
tracing
evolution
implemented
is
that
it
requires
participation
from
the
applications
themselves,
so
not
break
the
trace,
and
one
of
the
sort
of
ideals
behind
service
mesh
is
that
the
application
should
kind
of
not
know
that
they
are
running
Mestre
environment
right.
They
should
be
sort
of
kind
of
totally
ignorant
of
the
fact
that
the
oldest
magic
is
happening
around
them
and
I
think
tracing
breaks
that
model
a
little
bit,
so
we
haven't
quite
figured
out
how
to
how
to
do
it.
Yeah.
I
Hello,
a
quick
question
about
having
multiple
services
and
kubernetes
connecting
to
the
same
more
clothes.
So
now
in
your
model,
you
can
have
one
service
going
to
say.
One
deployment
and
I
was
wondering
if
you
ran
into
that
issue,
that's
possible
to
have
multiple
services
pointing
to
the
same
multiple
workloads
and
if
a
link,
Rd
dashboard
or
the
service
graph.
You
were
talking
about
handles
that
in
a
specific
way,
and
is
that
an
issue
even
for
you
and
not
really?
No.
B
I
mean
I
think
it
most
of
the
stats
are
that
you
see
in
the
dashboard
are
reported
from
the
pod.
The
inbound
link
ID
on
the
pod,
that's
receiving
the
traffic,
so
the
backend
pod
would
just
be
reporting
all
the
traffic
source
en,
regardless
of
which
service
is
coming
from,
but
you
can
also
look
at
the
outbound
stats
for
the
routers
that
are
sending
the
traffic
and
then
that
way
you
could
slice
it
up
by
those
services
that
they're
going
to
yeah.
So
I
think
it
all
just
works
in
that
in.
B
A
J
Thank
you
for
the
presentation.
My
question
is
about
the
protocol
buffer,
publish
mechanisms,
kind
of
things.
Have
you
tried
a
variety
flights
like
single
mono
repo
for
all
the
proto
buffers
in
your
relation
and
try
in
this
way
to
share
the
bastions
and
how
to
manage,
diversions
and
kind
of
things?
Can
you
explain
a
little
more
about
dude?
A
J
A
The
probe
of
management
is
kind
of
a
facility
Opik
that
everyone
talks
about
when
think
semana
repos
makes
more
sense,
I
guess
because
everything's
contained
in
one
repo,
but
we
never
did
that.
We
start
off
with
everything
was
in
a
lot
of
different
repositories,
and
that
brings
a
lot
of
problems
with
boiler
play
and
everything
also
like
create
new
repositories.
We
never
played
about
with
the
mana
reaper
thing,
so
I'm,
not
sure,
if
that's
better
or
worse
than
the
solution
that
we
came
up
with
like
I
said
earlier.
A
So
they
were
still
coupled
with
the
same
version,
so
you
still
had
one
version
for
both
both
artifacts,
but
they
were
Sacre,
yeah,
I'm,
yeah,
I'm,
not
sure.
If
that
answers
your
question
or
not,
but
all
right
yeah,
we
didn't
really
look
at
mono
repos,
so
much
yeah,
there's
a
yeah
there's
a
lot
of
oh
yeah
yeah.
It's
a
quite
big
pain
point:
yep
cool!