►
From YouTube: Scale Your Service on What Matters: Autoscaling on Latency - Thomas Rampelberg, Buoyant
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Scale Your Service on What Matters: Autoscaling on Latency - Thomas Rampelberg, Buoyant
Scaling HTTP based workloads is about more than cpu and memory. This talk will show why it is critical to scale based on latency, as well as how to do it for your own service by combining Linkerd, Prometheus, and Kubernetes. We demonstrate how to use Linkerd to instrument your service to collect aggregated service latency, store these metrics in Prometheus, and use them as custom metrics for consumption by Kubernetes’s Horizontal Pod Autoscaler. We demonstrate how latency-based autoscaling outperforms CPU- and memory-based autoscaling under a variety of conditions including live traffic from the attendees of this talk, and suggest ways to safely apply this technique to existing systems.
To Learn More: https://sched.co/GrR6
A
Hey
everybody.
Welcome
to
my
talk.
Let
me
get
started
I'll
be
chatting
a
little
bit
about
it,
scaling
your
service
on
what
matters
which
spoiler
I'm
going
to
claim
as
latency,
but
before
we
get
started
since
I'm
going
to
be
talking
about
auto
scaling.
If
my
slide
moves
forward
here,
I
have
got
a
application
up
that
I'm,
hoping
everyone
is
gonna.
Do
s
me
on
I'll
just
show
it
here,
while
I'm
waiting,
please
vote
on
what
the
best
editor
in
the
world
is.
A
I've
got
it
on
the
slide,
just
give
it
a
second
here
there
we
go,
but
first
before
we
get
started
now
that
I've
introduced
the
app.
Let
me
tell
you
a
little
bit
about
myself.
My
name
is
Thomas
ramble
burg,
I'm,
a
software
engineer
at
buoyant,
and
what
that
means
is
that
I'm
spoiled
rotten
because
I
get
to
work
on
open
source
all
day
every
day
and
contribute
to
linker
D,
which
is
going
to
be
part
of
this
talk.
If
anyone's
remotely
interested
in
hearing
grants
about
pretty
much
anything
in
the
world.
A
Follow
me
on
Twitter,
my
Twitter
handles
up
there,
and
here
we
go
KCl
5di.
Oh,
please
go
get
your
votes
in.
If
anyone
wants
to
try
and
do
s,
this
I'm
very
excited
so
now
that
we've
got
that
introduced,
I'd
like
to
talk
a
little
bit
about
what
auto-scaling
is
in
kubernetes,
because
it's
actually
a
little
complex,
there's
three
major
types
of
auto-scaling
kubernetes
one
is
the
cluster
autoscaler.
That
goes
and
creates
new
nodes
for
you
and
grows
and
shrinks
the
pool
based
off
of
the
request
limits
on
your
pods
and
containers.
A
Basically,
if
you
run
out
of
resources,
kubernetes
can't
schedule
something
your
cloud
provider
will
go,
create
your
new
nodes.
What
this
talk
is
going
to
be
about
is
the
horizontal
pod
autoscaler
that
goes
and
takes
a
metric
by
default,
the
CPU
and
memory
and
adjusts
the
replicas
for
you
to
go
and
increase
the
number
of
replicas.
On
your
deployments,
for
example,
then
we
have
the
vertical
pod
autoscaler,
which
goes
and
does
that
vertically,
instead
of
horizontally
it'll,
go
monitor
the
CPU
and
memory
usage
and
adjust
your
resource
limits
so
that
your
pod
fits
in
it.
A
A
Well,
we'll
just
continue
on
and
I
apologize,
okay,
so
three
types
of
metrics
in
the
kubernetes
api
number
one
is
the
metric
server
or
the
metrics
api
that
is
in
works
off
of
the
metric
server,
which
is
now
integrated
into
the
couplet,
and
that
gets
you
all
of
your
system
metrics
that
you
would
expect
that
CPU
and
memory
free
out
of
the
box.
It's
a
really
great
way
to
start
doing,
monitoring
on
your
cluster.
A
We
then
have
the
custom
metrics
API,
that's
what
I'm
going
to
be
talking
about,
and
that
goes
and
lets
you
do
it
on
any
metric
that
you
want.
Obviously
we're
going
to
be
taking
a
look
at
latency
today
and
then
you've
got
external
metrics,
which
is
something
external
to
a
pod
or
a
namespace.
So
this
could
be
the
ingress
that
you
have
or
since
we're
doing
a
talk
on
auto-scaling.
Imagine
a
cron
job
that
goes
and
sets
a
bit
to
auto
scale.
A
When
you
have
a
flash
sale
about
to
happen
so
now
you
can
go
and
do
scheduled,
auto
scaling,
which
is
pretty
awesome.
So
let
me
get
started
with
why
CPU
is
not
necessarily
what
you
want
to
do.
The
first
obvious
one
is
not.
Every
workload
is
CPU
bound.
You
really
have
to
figure
out
what
your
workload
is.
Sometimes
it's
I/o
bound.
Sometimes
it's
memory
bound.
You
really
have
to
guess.
The
biggest
thing
for
me,
though,
is
to
auto-scale.
A
You
have
to
put
the
threshold
below
100%,
so
you
have
to
say,
ok
all
autoscale,
but
it
will
only
be
at
90%,
which
means
that
you
now
have
10%
unused
capacity
in
your
cluster,
isn't
auto-scaling
all
about
being
cheap
and
getting
the
best
use
out
of
your
resources.
Another
good
reason:
CPUs
are
different.
I'm
sure
all
of
us
have
experienced
it
depending
on
a
region
depending
on
your
instance
type
depending
on
your
cloud
provider.
A
A
CPU
is
kind
of
a
random
guess
as
to
whether
you're
going
to
get
good
performance
or
not,
and
then
probably
the
biggest
thing
for
me
is
that
orchestrated
environments
are
super
complex.
Imagine
having
someone
schedule
a
Big,
Data
job
right
next
to
your
service,
that's
going
to
destroy
the
disk,
it's
gonna
kill
the
CPU,
and
even
though
you're
using
your
resources
correctly,
your
response
times
are
going
to
go
out
the
window,
and
so
you
really
need
to
have
a
way
to
look
at
it.
A
A
A
Let's
talk
about
what
you
should
be
monitoring
for
your
services,
the
Google
sre
handbook
talks
a
lot
about
the
golden
signals,
and
these
four
golden
signals
are
what
they
claim
are
the
only
things
that
you
absolutely
need
to
monitor
on
your
service
sure
you
can
implement
other
ones,
but
these
are
the
four
that
they
say
are
the
most
important
we
have
latency,
which
is
the
time
that
a
customer
requires
to
get
a
response.
There
is
traffic.
The
Quest's
per
second
that
are
going
through.
There
are
errors.
A
This
is
the
number
of
500s,
for
example,
that
you're
returning
or
unknown,
if
you're
in
the
GRP
sea
world,
and
then
there
is
saturation
at
which
point
the
service
degrades
so
heavily
just
because
you've
saturated
it.
You
need
to
track
these
four
things
and
I've
got
a
link
there
to
the
page
actually
in
the
book.
If
anyone
is
interested
so
we're
talking
about
latency,
I've
kind
of
said,
obviously
you
should
be
measuring
latency.
A
Why
don't
we
use
that
as
a
way
to
do
auto
scaling
the
other
thing
to
think
about,
and-
and
this
actually
blew
my
mind
when
I
found
it.
The
link
at
the
bottom
is
to
an
article
called
everything
you
know
about.
Latency
is
wrong.
Tail
latency
is
super-important.
In
fact,
I've
got
a
chart
here
on
the
right.
That
is
the
number
of
requests
for
popular
websites.
There
are
a
hundred
ninety
requests
fired
off
when
you
load
amazon.com.
A
If
you're
looking
at
AP
99,
which
is
the
99th
percentile
latency,
what
that
means
is
we've
thrown
out
the
99
fastest
latency
measurements
and
take
the
one
slowest
you
have
an
85
percent
chance
of
having
your
customer
on
a
single
load
of
amazon.com
hit
your
P
99
latency,
so
they're
gonna
have
a
bad
experience.
85%
of
the
time
you've
got
Google
and
that's
just
the
basic
Google
search
page
you're,
guaranteed
25
percent
of
the
time
to
hit
a
P
99
latency
on
at
least
one
of
those
requests.
A
A
So
here
is
a
little
bit
of
a
background
on
what
I'm
going
to
be
doing
from
a
live
demo
perspective
I'm,
going
to
show
you
measuring
the
latency
of
a
service,
we're
gonna,
expose
the
custom
metrics
and
then
we're
going
to
do
some
auto
scaling
to
talk
about
how
to
measure
the
latency
of
a
service,
though
I'd
like
to
talk
a
little
bit
about
link
rudy,
and
the
reason
for
that
is
how
many
of
you
here
actually
I'm
interested,
have
latency
based
monitoring
on
their
services.
Today.
A
That's
pretty
awesome
is
it
for
all
of
the
applications
in
your
organization
or
just
at
the
ingress
anyways,
it's
kind
of
hard
I've
done
the
instrumentation.
You
have
to
go
figure
out
Prometheus,
you
have
to
go
hook
it
up.
It
takes
a
while,
so
I've
got
the
world's
simplest
Python
application
serving
up
the
leaderboard
that
I
hope,
you're
all
are
still
voting
for
and
linker
D
is
going
to
be
used
to
instrument
it.
So
what
is
link
or
D
link
rudy
is
a
open
source
service
mesh.
A
It's
a
CNC
F
member
project
used
in
production
with
quite
a
few
people,
but
that
doesn't
really
explain
what
it
is
or
why
you
need
it.
So,
at
its
most
basic
link,
Rudy
is
nothing
more
than
a
proxy
that
gets
scheduled
alongside
your
application
because
of
the
magic
of
IP
tables
it
receives
all
of
the
incoming
traffic
and
all
of
the
outgoing
traffic,
and
because
it's
at
layer
7,
we
can
go
and
instrument
all
of
the
requests
and
responses.
This
isn't
TCP
traffic.
A
This
is
actually
what
requests
are
and
how
long
responses
take,
but
that's
only
half
of
the
picture
to
do
a
full
monitoring
setup.
You
actually
need
a
control
plane
to
go
along
with
your
data
plan,
so
we
ship
with
Gravano,
dashboards
and
Prometheus
out
of
the
box.
So,
with
a
single
install
that
I'm
about
to
show
you,
you
get
the
latency
and
all
of
the
four
golden
signals
immediately
for
your
services
really
quite
quickly.
So
let's
go
do
that.
A
So
I've
downloaded
the
linker
de
CLI
onto
my
laptop
already,
but
we
provide
this
thing
called
link
ready
check
which
will
pre-flight
and
post-flight
your
cluster
I
have
gone
and
installed
so
many
kubernetes
packages
on
my
cluster
and
they
work
half
the
time,
sometimes
ish
I'm,
not
usually
sure,
and
so
here
we
go
with
the
tool
that
tells
you.
Ok,
you
have
all
the
permissions
you're
ready
to
go
which,
at
which
point
you
can
type
link
your
d,
install
that
outputs,
the
gamal
that
you
can
then
pipe
through
cuddle.
A
A
And
while
the
internet
gods
smile
on
me,
I
hope
there
we
go.
What
we'll
see
is
we're
just
applying
all
of
the
things
that
I
explained
for
the
control
plane.
This
is
the
Prometheus,
the
Griffin.
We
have
a
web
dashboard
that
ships
and
a
couple
other
things
and
then
once
this
is
done,
we
also
have
a
link
ready
check
command
for
post
flighting,
and
this
is
actually
going
to
hang
up
because
it
hasn't
started
the
containers.
A
Yet
we'll
come
back
to
this,
but
this
is
a
way
to
validate
that
your
install
was
successful
and
is
actually
working.
So
you
don't
need
to
go
debug,
weird
edge
cases.
So
now
that's
the
control
plane.
It's
going
to
be
up
and
running,
I
promise.
Let's
talk
a
little
bit
about
the
data
plane
and
getting
it
into
my
application.
A
So
here
I
am
getting
the
ammo
and
then
I'm
just
going
to
inject,
and
here
you'll
see
if
I
scroll
up
a
little
bit.
We've
got
the
IP
tables
init
container
and
we
have
the
proxy
injected.
We've
also
done
the
pre
and
post-flight
checks
on
the
data
layer,
as
well
as
the
control
plane.
So
you
can
validate
that
your
gamal
is
correctly
formed.
It
doesn't
have
any
weird
configuration
it's
ready
to
go.
So
if
we
pipe
that
on
through.
A
A
A
And
get
a
feel
for
what's
actually
happening
on
our
cluster,
so
here
you'll
see
that
we're
actually
doing
about
500
RPS
I
have
a
load
generator
that
I
added
that's
already
doing.
A
thousand
RPS
and
you'll
see
that
our
p99
latency
is
three
milliseconds.
So
let's
open
up
the
graph
on
a
dashboard
to
have
it
there
to
be
a
reference.
A
They
introduced
this
idea
of
the
API
server
as
an
aggregator,
the
API
server
being
the
single
place
that
you
want
to
talk
to,
to
figure
out
all
of
your
data
and
to
do
that.
What
they
did
is
introduced.
A
API
service
object,
so
I've
got
the
resource
up
here,
and
basically
this
is
just
saying.
I
would
like
to
proxy
everything
to
the
custom
metrics
API
back
to
a
Prometheus
adapter,
which
is
about
what
I'm
going
to
show
you
right
now.
So
here's
the
Prometheus
adapter
it
takes
the
custom,
metrics,
API
and
literally
just
translate.
A
It
translates
it
into
Prometheus
queries,
so
I've
got
the
API
at
the
top
and
I
have
the
Prometheus
query
that
we
are
translating
to
in
the
bottom,
and
so
you
can
almost
think
of
this
as
just
a
simple
translator,
it's
going
to
run
one
pot
on
our
cluster
and
be
ready
to
go
that
way.
So
here
is
kind
of
an
architecture.
It's
fun
to
talk
about
this
and
show
how
all
of
the
kubernetes
components
actually
work
together.
So
here
we've
got
a
cube.
A
A
A
Configure
that
so
here
I
have
pretty
simple
I
tell
it
where
my
Prometheus
is
and
I've
configured
three
queries:
I've
got
a
50th
percentile,
a
95th
percentile
and
a
99th
percentile
out
of
the
box.
They
actually
have
quite
a
few
more
metrics
I've,
stripped
this
down
for
the
demo,
so
that
we
don't
get
overwhelmed
with
the
sheer
number
of
awesome
metrics
that
you
can
get
out
of
this
so
to
get
it
installed.
I'll
do
a
helm,
install
here.
A
That's
going
to
render
some
templates
get
them
applied
to
the
cluster.
If
I
scroll
up
here,
you'll
see
that
there
is
an
API
service,
that's
been
added,
so
that's
going
to
be
what
tells
the
kubernetes
api
to
proxy
it
back
to
the
Prometheus
adapter
and
a
couple
other
things,
including
a
command
so
before
I
run
that
let's
go
take
a
look
at
what
the.
A
A
So
let
me
show
you
now
what
the
custom
metrics
API
looks
like
and
because
I
don't
like
to
remember
this
one
here,
we've
got
a
query:
you'll
see
the
route
right
here
we
go
and
say
tell
us
all
of
your
metrics
and
if
we
look
through
this,
we've
got
the
pods
50th
percentile
latency
I
promised
that
the
99th
there
it
is
right
there
and
then
I've
got
this
other
query
here.
That's
gonna,
go
and
say
tell
me
all
of
your
values
and
you'll
see
that
it's
this
insanely
large
value.
Remember
that
Prometheus.
A
To
avoid
any
kind.
Excuse
me
remember
that
kubernetes,
to
avoid
any
kind
of
floating
point,
math
multiplies
everything
times
a
thousand,
and
so
this
is
basically
what
is
that
20
milliseconds
give
or
take,
and
that's
pretty
much
it
I've
now
exposed
my
metrics
so
we're
now
measuring
the
latency
of
our
service,
just
by
injecting
link
or
D
into
our
data
plane.
No
code
changes.
Nothing
special
happened
there
I'm
exposing
the
custom
metrics
through
the
API
service
through
the
Prometheus
adapter,
and
now,
let's
actually
talk
about
doing
the
auto
scaling
and
to
do
that.
A
Let's
talk
about
what
the
horizontal
pod
autoscaler
does
for
real.
So
what
it
does
is
every
30
seconds
as
a
controller
on
your
kubernetes
cluster.
It
queries
the
API
server
that
that's
it.
It's
so
simple,
it's
awesome!
So
it
queries
the
API
server
every
30
seconds
that
eventually
gets
proxy
back
to
our
applications
that
have
been
fed
up
into
Prometheus
and
if
your
stat
is
out
of
its
range
well,
it'll
auto
scale
it
for
you.
A
The
algorithm
is
pretty
simple:
I
recommend
anybody
go
look
at
the
documentation
if
you're
interested
the
whole
thing
is
just
kind
of
simple:
it's
really
nice.
So
let's
go
do
some
auto
scaling
and
to
do
that.
What
I'm
gonna
do
is
first
show
you
what
the
policy
looks
like.
So
here
we
have
a
HPA
e
resource
and
I'm
saying
I
want
a
min
replicas
of
one
max
of
100
and
target
8
milliseconds
I'm
gonna
apply
this
and
we're
gonna
see
some
auto
scaling
happening.
A
So,
while
that
is
doing
its
thing,
let's
go
take
a
look
at
the
blinker
de
CLI.
Again
take
a
look
at
what
our
Layton
sees
our
whoo.
Oh
boy,
you
guys
are
really
killing
me.
This
is
awesome,
so
we're
doing
it's
pretty
obvious.
This
needs
to
get
auto
scaled
at
this
point
right
away.
There's
no
question,
but
I
bet
you
that
we're
actually
already
doing
that.
So
if
we
go
do
a
coop
CTL
described
on
the
web,
HPA
excuse
me.
A
Will
see
that
it's
already
scaled,
my
pods
up
and
the
response
latency
is
way
out
of
the
range
of
possibility,
and
so,
if
I
go
over
here
and
do
a
watch
we'll
see
that
I've
got
four
pods
that
have
spun
up
it's
taking
them
a
little
time
to
get
the
traffic
spread
out.
But
here
in
just
a
couple
seconds,
thank
goodness
it
didn't
make
a
liar
out
of
me.
You'll
see
that
we
start
to
automatically
load
balanced
traffic
from
you
guys
and
my
load
generator
and
immediately
start
to
see
the
latencies
go
down.
A
This
is
even
more
optimized
than,
for
example,
changing
at
a
hundred
percent,
CPU
utilization
and
because
I
think
it's
an
awesome
demo
I'm
going
to
show
that
off.
Just
really
quick
here,
in
fact,
I
have
the
ability
to
introspect
and
get
a
open,
API,
aka
swagger
out
of
the
leaderboard
application
and
then
I'm
going
to
pipe
that
into
the
link.
Rudy
CLI,
that's
going
to
convert
it
from
the
open,
API
specification
into
a
CRD
that
basically
is
regex
is
to
be
mount
to
be
matches
and
handle
that.
A
We
just
started
generating
statistics
for
this,
so
it
will
take
a
couple
seconds,
but
in
that
time
we
could
go
and
have
crafted
a
query
for
Prometheus
exported
it
through
the
custom,
metrics
API
and
be
able
to
get
all
of
the
metrics
for
what's
going
on
there.
In
fact,
you'll
see
here
that
we're
getting
the
latency
it's
slowly
shifting
over
on
a
per
route
basis.
A
Wow,
that's
awesome
and
then
finally-
and
this
is
the
one
I
think
maybe
I'm-
actually
the
most
excited
about
prometheus-
is
basically
the
most
amazing
thing
on
earth.
As
far
as
I
can
tell-
and
it
comes
with
the
ability
to
do
linear
regression,
so
you
can
actually
do
predictive,
auto
scaling
with
this
setup
on
Prometheus.
Remember
it
takes
30
seconds
60
seconds
for
everything
to
warm
up.
You
can
say:
I
know
that
my
latency
is
going
to
be
past.
A
8
milliseconds,
let's
scale
up
60
seconds
in
advance,
so
that
everything
is
running
and
ready
to
go
right
when
we
get
there
and
that's
actually,
it's
really
simple
to
get
going
with
this.
You
don't
need
to
stick
with
CPU
and
memory
in
the
box,
but
really
quick,
because
I've
thrown
so
much
data
at
everybody.
I've
got
this
final
slide
here.
I've
got
my
slides
up
on
github,
it's
a
PDF,
I'd
love
for
everybody
to
go.
Take
a
look.
I
also
have
the
code
up
to
hopefully
run
this
yourself.
A
We
have
a
getting
started
guide
for
linker
D.
If
anyone's
remotely
interested.
Don't
worry,
you
don't
need
a
kubernetes
cluster,
we're
actually
using
Cod
ACOTA
to
be
able
to
get
be
able
to
run
a
kubernetes
cluster
in
your
browser
run
through
the
whole
getting
started.
And
finally,
we've
got
the
Prometheus
adapter
there
for
everyone
give
it
a
try
and
that's
it.
Thank
you
so
much.
A
So
the
question
was:
what
happens
if
you
have
a
dependency?
In
fact,
my
a
pact
has
a
dependency
on
Redis
and
I
suspect
that
the
reason
why
the
numbers
right
now
are
not
what
I've
tested
in
the
pastor,
because
Redis
is
going
a
little
bit
slow.
You
need
to
understand
auto
scaling,
isn't
a
silver
bullet.
You
have
to
really
think
deeply
about
what
your
architecture
is
and
how
to
scale
it.
A
If
you're
running
a
micro
services,
architectural,
you
want
auto
scaling
all
of
the
different
layers
and
then
you
want
to
go
and
think
how
it
interacts
while
I
I
made
it
seem
super
easy.
It
really
is
not
just
a
silver
bullet.
You
have
to
think
about
your
workloads
and
how
it
all
works
together
for
better
or
worse
any
other
questions.
Yes,.
A
So
you
wouldn't
necessarily
need
to
use,
link
or
D
for
that.
I
would
probably
use
the
custom
metrics
based
off
of
your
cue
length
and
just
Auto
scale
on
that,
but
it
is.
It
gets
to
the
real
power
of
custom,
metrics
or
external
metrics.
Again,
if
you've
got
your
kafka
cluster,
for
example,
hosted
off
kubernetes,
you
can
go
use
the
external
metrics
API
to
do
the
auto
scaling
on
that.
But
that
gets
back
to
your
workload.
Is
your
workload.
You
need
to
kind
of
think
through
how
it
works.
A
A
So
you
definitely
could
do
that.
The
problem
is
that,
then
you
have
to
think
out
think
through
what
your
overhead
is
and
really
understand
what
your
saturation
is,
because
I
don't
actually
know
what
my
service
here,
saturates
at
it's
really
hard
to
say:
ok,
Auto
scale
but
Auto
scale
at
a
thousand
RPS,
because
who
knows
it's
going
to
get
scheduled
to
something
different,
and
so
this
does
again
kind
of
get
back
to
it's
your
own
workload
to
figure
it
out.
But
this
is
kind
of
the
next
level
of
approximation.
A
If
you
know
your
saturation
layer-
and
you
know
what
CPUs
you're
scheduled
on
and
you're
sure
that
you're
getting
all
those
resources,
that
is
actually
a
better
way
to
do
it,
but
the
amount
of
information
required
to
get
there
is
so
much
that
taking
a
step
back.
I
actually
feel
that
this
is
a
better
approximation,
but
not
the
best.
A
So
you
can,
but
it
would
end
up
being
a
external
metric,
and
then
you
need
to
think
through
what
that
means
on
a
pod
by
pod
basis,
because
maybe
one
pod
is
going
away,
slow
and
one
pod
is
going
fast,
so
you
definitely
can
do
it
I
mean
honestly,
technically
you
don't
need
link
Rd
for
this.
If
you
want
to
go
instrument,
your
app
for
Prometheus,
you
can
go
stand
that
up
for
yourself.
A
The
power
is,
is
that
you
don't
have
to
it's
monitoring,
and
my
experience
is
always
the
thing
that
falls
to
the
bottom
of
the
priority
list,
and
so
you
would
normally
have
to
make
this
trade-off.
Are
we
going
to
have
some
better
to
monitoring,
so
people
get
page
less
or
is
a
product
manager
gonna
ask
to
put
more
features
in
and
I'm
sure
you
know
which
way
that
goes,
and
so
you
can
really
think
of
link
or
D
is
just
this
like
easy
way
to
start
out.
A
A
A
So
I'm
so
glad
you
asked
that
one
of
the
coolest
things
about
Lync
rudy
is,
is
that
it's
G
RPC
aware
and
not
just
that,
but
it's
G
RPC
aware
from
a
status
code
perspective,
because
every
gr,
if
you
see
response,
is
a
HTTP,
200,
ok,
and
so
not
only
does
it
each
do
HTTP,
1
and
2.
It
also
is
aware
of
G
RPC.
It
doesn't
do
things
like
Kafka
or
Redis
or
MySQL,
some
of
the
other
binary
protocols.
It
could
be
added,
but
the
instrumentation
isn't
there.
A
A
A
A
So
that's
actually
what
it
does.
You're
teeing
me
up
for
a
very
large
rant
that
I
have
I'm
gonna,
take
a
step
back
but
say
the
latency
is
actually
getting
measured
depending
completely
depending
on
where
you
look
in
the
system,
so
it
is
at
a
pod
level.
It's
at
a
deploy
level,
it's
at
a
service
level,
and
so
you
can
slice
and
dice
it
all
without
passing
headers
on
which
is
kind
of
the
Achilles
heel
of
most
of
the
distributed
tracing.
A
B
B
A
Honestly
think
that
the
end
goal
here
is
to
go
and
create
aggregate
metrics,
exactly
where
you're
going,
where
you
don't
just
Auto
scale
on
latency,
you
actually
measure
a
bunch
of
the
golden
signals
and
MIT
police
CPU
and
feed
that
into
the
horizontal
pot
autoscaler.
So
it's
totally
possible
to
do.
I
just
didn't
happen
to
show
it
off
here.
I
think
we're
out
of
time.
I'm
gonna
hang
around
here
and
be
back
at
the
buoyant
booth
for
all
of
your
questions.
I
would
love
to
answer
them.
Thank
you.
So
much
for
coming.