►
From YouTube: Linkerd 2.0, Now with Extra Prometheus - Andrew Seigner, Buoyant & Frederic Branczyk, CoreOS
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Linkerd 2.0, Now with Extra Prometheus - Andrew Seigner, Buoyant & Frederic Branczyk, CoreOS
In order to provide zero-config observability for service owners running their application on Kubernetes, Linkerd 2.0 needs to scale to any size Kubernetes cluster. In this talk, Andrew Seigner from the Linkerd team and Frederic Branczyk from the Prometheus team will demonstrate Linkerd 2.0 features made possible by Prometheus, including a user-friendly dashboard, a simple yet powerful CLI, and Grafana integration. They’ll outline how Linkerd evolved its Prometheus integration to leverage Prometheus’s ability to handle massive sets of metrics, and demonstrate the scale this integration enables.
To Learn More: https://sched.co/GrXs
A
Hey
folks,
thanks
for
coming,
my
name
is
Andrew
Ziegler
I'm,
a
software
engineer,
Boyett
I'm
here.
To
give
this
talk
with
my
friend
Frederick.
We
kind
of
wanted
to
showcase
linker,
D
and
Prometheus,
and
how
they're
complementary
technologies
and
kind
of
show
up
the
latest
and
greatest
features,
but
also
talk
about
kind
of
the
origin
story
of
of
how
these
tools
evolved.
How
we
ended
up
with
these
features
and
kind
of
how
the
integration
between
the
two
developed
along
the
way.
So
with
that
Frederick
burnsy,
okay,.
B
So
first
we
want
to
have
a
little
quick
run
through
Prometheus,
quick
show
of
hands.
How
many
people
know
about
Prometheus
awesome
I
think
that's
like
90
95
%
of
the
hands.
That's
pretty
good,
but
specifically
I
want
to
talk
about
a
couple
of
features
that
make
it
such
a
great
fit
for
linker,
D
and
Prometheus.
But
first,
let's
talk
about
some
history
just
for
everyone
to
know
where
Prometheus
came
from
and
how
it
developed
over
time.
B
So
Prometheus
was
initially
created
at
SoundCloud
by
a
couple
of
X
coolers
who
knew
some
systems
within
within
Google
and
they
liked
it.
But
there
was
nothing
out
there,
and
so
they
created
they
new
Borg
within
Google,
and
there
was
Berkman,
which
is
the
monitoring
system
and
sadly,
nothing
is
named
in
the
open
source
after
the
fork
or
anything
in
Star
Trek.
Unfortunately,
at
least
I'm
sad
about
that
so
yeah.
So
they
created
this
and
today
fast-forward
like
five
years
since
they
publicly
started
speaking
about
it.
B
So,
let's
start
with
a
really
simple
intro
to
how
Prometheus
works
on
a
really
basic
example.
So
Prometheus
is
a
pull
based
system,
so
in
some
regular
interval
it
will
go
to
your
application
and
grab
the
metrics
that
your
application
exposes,
and
it
just
does
that
via
an
HTTP
call.
So
we
just
have
an
HTTP
endpoint
on
our
application
that
exposes
these
metrics
and
we
instrument
our
application
to
whenever
something
happens.
To
increment,
for
example,
our
our
metrics
and
Prometheus
comes
every
15
seconds,
that's
the
default,
but
it's
configurable
and
whatever
our
application
returns.
B
We
add
the
time
stamp
to
it
and
write
it
into
our
internal
time.
Series
database-
and
in
this
case
our
process
hasn't
actually
received
any
traffic
itself,
so
the
request
count
was
zero,
but
now
we're
actually
routing
production
traffic
to
it
and
over
time
it
increases
its
metrics
internally
right
and
so,
for
example.
Now
we
have
two
requests
that
were
hit
against
this
service
and
Promethea.
B
The
15
seconds
have
elapsed
again,
so
Prometheus
comes
around
and
scrapes
the
metrics
and
inserts
it
into
its
time
series
database
and
furius
just
as
this
forever
and
eternity-
and
this
is
essentially
the
example
of
a
single
time
series
from
one
target
and
we'll
keep
reiterating
on
this
terminology
to
guide
us
through
this
talk.
Basically,
so
what
we
just
saw
was
collection
of
time
series
data,
but
Prometheus
is
a
monitoring
system.
No,
it
does
that.
B
What
does
that
actually
mean
for
something
to
be
a
monitoring
system,
at
least
in
my
point
of
view,
it
means
that
we
need
to
be
able
to
do
something
actionable
with
that
data
that
we're
collecting
and
I
think
the
best.
That's
example
of
that
are
so.
If
there's
some
condition
in
our
system,
then
we
want
to
get
a
page,
for
example,
or
a
slack
notification,
or
something
that
tells
us
that
there's
a
status
that
something's
off
in
our
in
our
infrastructure
and
I
just
wanted
to
show
that
there's
really
nothing.
B
That's
like
black
magic
in
Prometheus,
like
we
saw
that
the
collection
is
super
simple,
it's
just
every
15
seconds.
It
goes
around
and
collect
all
these
this
data.
So
that's
really
simple,
right
and
alerting.
It's
really
the
exact
same
thing,
and
you
see
this
throughout
the
Prometheus
code
base
that
really
everything's
like
super
crisp
and
just
pretty
I
I,
don't
want
to
say
stupid,
because
it's
it's
good,
but
it's
stupid,
as
in
it's
simple
to
understand
so
yeah.
B
This
also
just
happens
every
15
seconds,
Prometheus
just
evaluates
all
the
rules
it's
configured
to
evaluate,
and
then,
if
an
alerting
rule
is
firing,
then
it
will
actually
fire
that,
but
now
we're
at
coop
con.
So
what
does
that
have
to
do
with
linker
d,
&,
kubernetes
and
prometheus
all
the
CNCs
projects,
so
Prometheus
has
built-in
service
discovery
mechanisms
for
kubernetes
and
what
that
means
is
Prometheus.
B
Can
go
to
the
criminales
API
and
ask
for
all
the
pods
in
your
kubernetes
cluster,
for
example,
and
that
way
Prometheus
then
knows:
okay,
I
have
this
part
of
this
pot
and
goes
ahead
and
scrapes
all
of
those
individually,
and
this
is
a
really
subtle,
but
very
important
thing
which
is
Prometheus
is
about
white
box
monitoring,
and
that
actually
does
mean
that
we
do
go
to
every
single
pod
and
want
to
grab
metrics
from
every
single
instance.
We
don't
load-balanced
this
or
something
because
then
we
would
be
getting
metrics
from
different
processes.
B
Every
time
we
scrape
our
metrics,
we
actually
wanted
the
grannie.
We
want
the
granular
targets
to
be
scraped,
and
that
way
we
true
have
true
insight
into
our
applications
and
how
every
single
process
is
doing,
and
then
we
can
do
aggregations
over
our
entire
infrastructure
and
make
sense
of
all
of
this
data.
So
this
is
really
important
when
you
think
about
Prometheus,
we
always
want
to
make
sure
we
get
data
as
close
to
the
origin
of
some
action.
B
So
if
we
are
talking
about
a
network
service,
we
always
want
to
have
the
instrumentation
as
close
as
possible
to
the
code.
That
actually
does
the
network
requests.
So,
let's
visualize
this
a
bit
so
we
saw
previously,
we
had
like
a
table
where
we
inserted
all
of
our
time
stems,
but
it
turns
out
time
series
are
actually
nice
for
graphing
right.
So
we
have
a
couple
of
samples
here
and
we
have
a
metric
called
HTTP
requests.
B
That's
a
really
common
thing
to
have,
but
Prometheus
has
a
model
where
you
can
basically
enrich
all
of
these
metrics
with
additional
meta
information,
which
is
label
which
are
labels,
and
in
this
case
for
example,
we
don't
only
have
one
series
for
HTTP
requests,
but
we
have
lots
of
series
describing
different
aspects
of
our
HTTP
server,
so
we
can
have,
for
example,
metrics
based
on
request
paths
or
on
response
latencies.
All
things
like
that
are
totally
possible
with
Prometheus
and
we'll
see
later
how
that's
great,
with
link
or
D.
B
Now
Prometheus
was
created
at
a
time
where
there
was
no
kubernetes
and
at
the
time
the
the
engineers
looked
at
how
what
is
the
lifetime
of
a
time
series
and
typically
things
were
deployed
on
virtual
machines,
for
example,
and
all
processes,
even
if
they
were
restarted,
they
would
be
restarted
on
the
same
virtual
machine
and
everything
would
basically
stay
the
same.
So
that
means
that
our
time
series
were
always
continuous.
We
didn't
have
lots
of
time
series
that
started
and
stopped
now.
B
This
changed
changes
dramatically
on
kubernetes
or
even
on
cloud
providers
that
sometimes
start
to
evict
your
virtual
machines
or
stuff
like
that.
So
we
will
it's
super
important.
That
Prometheus
is
purposely
built
for
an
environment
like
this,
and
this
is
what
we
did
when
we
created
the
Prometheus
2.0
storage.
We
specifically
had
this
problem
that
there
are
lots
of
pods
coming
and
going
in
kubernetes,
and
that
meant
there's.
There
are
a
lot
of
times
used
that
keep
being
created
and
stopped
and
to
make
that
problem
even
worse.
B
People
like
insight
and
that's
a
good
thing,
but
it
also
means
that
it's
that
much
more
data,
so
the
more
labels
we
have
the
more
time
series
we're
going
to
produce
the
more
load
we
put
on
Prometheus
and
essentially
the
most
costly
operation,
is
creating
a
new
time
series.
You
can
roughly
compare
that
to
creating
a
new
file,
that's
relatively
expensive,
compared
to
appending
to
an
existing
file,
that's
a
pretty
cheap
or
at
least
comparatively
a
very
cheap
operation.
B
B
So
we
completely
rewrote
the
entire
storage
engine
and
the
problem
is
not
just
inserting
this
data,
but
then
people
also
wanted
to
queries.
Also,
it's
all
sorts
of
queries,
and
basically
we
have
an
visualization
here
that
show
we
have
all
sorts
of
queries
where
we
slice
and
dice
data,
and
we
want
all
of
this
to
be
fast
right
and
all
of
that
as
possible
with
Prometheus
to
print
oh
and
another
cool
feature
that
we
I
think
it's
pretty
unique
about
Prometheus
in
the
space,
because
Prometheus
is
pull
based.
B
It
can
immediately
detect
if
a
time
series
goes
away.
So
if
our
slash
metrics
endpoint
had
metrics
for
some
path,
information
and
that
went
away,
then
we
can
immediately
tell
in
Prometheus
on
the
next
scrape
that
this
time
series
has
stopped.
That
is
not
particularly
useful
for
HTTP
requests,
but
if
you
want
to
have,
for
example,
within
the
creeper
Nettie's
case,
we
often
have
that
pod
information
comes
and
goes
all
the
time,
and
we
want
to
have.
We
want
to
be
able
to
tell
that
the
pot
has
disappeared
immediately,
so
stay
on.
B
This
is
really
how
do?
How
can
we
tell
that
a
time
series
is
done
and
previously
in
Prometheus,
one
point
X?
This
was
just
like
a
time
out
based
system
where,
if
a
time
series
hasn't
received
a
new
sample
within
the
last
five
minutes,
its
marked
as
stale,
but
that's
problematic,
because
that
means
that
it
potentially
takes
up
to
five
minutes
for
alerts
to
actually
register
that
this
time
series
is
done,
so
that's
too
late
and
in
Prometheus
2.0.
What
we
did
instead
is.
B
This
looked
totally
fine.
We
have
continuous
lines,
everything
looks
happy,
but
in
2.0
with
correct
or
better
stay
on
this
handling,
we
immediately
noticed
that
there
could
be
a
problem
and,
as
I
said,
this
is
super
important
in
order
to
be
sure
that
the
data
that
we're
looking
at
is
always
fresh
and
even
if
you
go
out
of
the
talk,
not
remembering
any
of
the
things
that
I've
just
said,
then
just
remember.
Basically,
this
slide
prometheus.
B
A
single
Prometheus
server
can
get
you
really
far
and
we've
seen
in
the
in
the
wild
in
actual
production
systems.
That
Prometheus
is
ingesting
over
700,000
samples.
A
second
prometheus
is
capable
of
doing
more.
We've
done
this
synthetically,
but
this
is
like
one
of
the
an
example
in
the
wild
that
we've
seen
that
real
production
in
real
production
usage.
Usually
people
then
start
to
split
up
Prometheus
by
responsibility,
so
you
have
one
class
per
kubernetes
cluster
or
per
namespace,
or
something
like
that
and
700,000
samples
per
second
roughly
translates
to
a
350
node
kubernetes
cluster.
B
That,
of
course,
is
like
a
generalization
because
it
depends
on
which
metrics
you
collect,
but
350
nodes.
Kubernetes
cluster
is
already.
You
can
already
run
quite
a
lot
of
workload
on
that,
and
now
we
see
with
we'll
hear
from
Andrew
how
it
how
all
these
features
and
Prometheus
makes
make
linker
D
that
much
more
awesome.
A
All
right
thanks
a
lot
Frederick
so
yeah
when
I'm
hoping
to
use
kind
of
walk
through
a
similar
history
of
how
Lincoln
he
involved,
there's
actually
a
lot
of
parallels
with
a
link
31
and
then
to
evolved
along
with
Prometheus.
So
to
give
a
little
bit
of
a
background,
so
link
Rd
is
an
open
source
service
mesh
for
cloud
native
applications.
What
what
does
that
mean
so
service
mesh
is
a
term
for
a
layer
of
software
that
provides
security,
reliability
and
visibility
to
your
micro
service
architecture
and
so
there's
typically
two
components
of
it.
A
You
have
a
control
plane
which
is
really
like
the
operational
view
of
what's
going
on.
So
that's
something
that's
a
place
where
Prometheus
would
live.
You
collect
metrics.
It
gives
you
some
kind
of
a
user
interface,
either
through
a
dashboard
or
through
like
a
command-line
interface,
and
it's
it's
talking
to
its
brian
service
discovery
to
your
cluster.
It's
also
gathering
metrics.
The
other
side
is
the
data
plane,
that's
typically
a
proxy
that's
running
as
a
sidecar.
Next
to
each
of
your,
your
pods
or
your
containers
in
your
your
application.
A
So
back
up
a
little
bit
and
kind
of
figure
out
like
how
did
we
get
here?
Lincolnís
origins
go
back
to
Twitter
in
2010.
At
that
time
it
was
probably
the
largest
rails
installation
on
the
planet.
The
summer
of
2010
was
the
World
Cup.
Every
time,
Brazil
scored
a
goal,
everyone
we
treat
go
and
the
site
would
crash,
and
so
we
were
frantically
trying
to
decompose
this
thing
into
micro
services
and
we
found
that
every
service
kind
of
needed,
the
same
thing.
A
It
needed
security,
visibility
and
reliability,
and
so
we
factored
that
logic
out
into
a
library
called
finagle.
It
was
a
scholar,
library
which
is
still
used,
probably
in
every
micro
service
at
Twitter,
take
over
virtually
every
service
and
it's
a
it's
gained.
Pretty
good
acceptance
across
the
community,
I
think
even
SoundCloud
maybe
uses
it,
and
so
it
was
great.
We
found
it
very
useful.
The
drawback
was
since
it
was
a
Scala
library.
A
You
needed
to
be
writing
a
Scala
application
to
take
advantage
of
this
kind
of
thing,
so
that
was
kind
of
the
inspiration
for
building
link
or
D,
but
link
or
D
1.0
did.
Is
it
took
this
Scala
library
and
wrapped
it
into
a
proxy
so
that
any
language
any
framework
you
could
be
running
locally?
You
could
be
on
ms-dos
or
kubernetes
or
whatever.
You
could
take
advantage
of
this
like
service
mesh
concept,
and
it
was
agnostic
to
like
what
technology
you
are
using.
A
So
about
a
few
months
after
we
released
linker
d1,
we
started
getting
requests
for
integration
with
Prometheus,
and
so
we
added
this
endpoint
as
Fredrik
mentioned.
Prometheus
is
a
pull
based
system.
So
all
we
had
to
do
is
expose
this
nice
little
metrics
endpoint
on
linker,
D
and
then
Prometheus
could
go
around
and
collect
metrics
from
all
the
linker
D
proxies
running
on
your
cluster.
A
A
A
Scala
served
us
really
well
for
a
long
time,
but
you
know
when
you're
trying
to
run
a
proxy
on
next
to
every
pod
and
people
are
like
you
know,
writing
little
tiny
services
and
go
or
whatever,
and
they
take
up
10
megabytes
of
RAM
and
we're
asking
you
to
install
like
a
a
JVM
process
that
requires
two
hundred
Meg's
of
RAM
to
run
right
next
to
it.
That's
not
really
acceptable
so
for
the
control
plane
working
in
the
kubernetes
ecosystem
go
seem
like
a
natural
choice.
A
The
the
community
there's
there's
a
lot
going
on
and
it
plugs
really
nicely
into
to
all
the
kubernetes
cooling.
On
the
the
proxy
side,
we
made
a
bit
of
a
bet
and
we
went
with
rust
because
we
wanted
that
native
performance,
but
we
also
wanted
kind
of
the
the
safety
that
we
were
used
to
from
Scala.
As
I
say,
it
was
a
little
bit
about
it,
a
little
bit
of
a
bed
a
couple
of
years
ago,
but
I
think
it's
proved
out
pretty
well.
A
We've
gotten
great
community
support
and
we've
even
hired
some
of
the
members
of
the
community
and
to
work
on
libraries
that
help
support
the
link
or
D
proxy,
so
just
super
high-level
overview
of
how
it
works.
This
is
kind
of
that
same
diagram.
You
saw
a
minute
ago
with
a
control
plane
and
a
data
plane
and
you've
got
your
your
linker
D
controller.
That's
the
go
side
talking
to
kubernetes
very
similar
to
how
Prometheus
works.
It's
getting
all
the
service
discovery
information!
It's
telling
the
proxies!
A
Where
all
of
the
pods
are,
and
then
next
to
every
pod
in
your
in
your
cluster,
you've
got
a
little
linker,
D
proxy
running,
and
that
thing
knows
how
to
intelligently
route
to
different
pods
in
your
cluster.
So
this
is
a
really
similar
pattern.
As
I
said,
Prometheus
is
a
first
class
component
of
linker
d2.
It's
set
up
to
also
listen
to
kubernetes,
while
linker
D
is
doing
routing
information
based
on
what
kerbin
is
telling
you
Prometheus
is
doing
collection
as
Frederick
mentions,
so
it
goes
and
asks
kubernetes.
A
Where
is
every
linker
see
running
and
then
it
goes
out
and
it
collects
data
from
every
link,
your
new
proxy
and
lets
us
roll
up
that
information,
and
because
that
proxy
is
responsible
for
every
request.
In
and
out
of
all
of
your
applications,
you
can
get
everything
from
request,
load,
latency,
success
rate.
You
know,
link
reduce
protocol
aware
so
you
can.
You
can
get
things
that
are
HTTP
or
HTTP
specific
or
G
RPC
or
whatever
you're,
using
a
little
bit
more.
If
you
just
want
to
introspect
the
the
metrics
themselves,
we
also
include
agra
fauna.
A
That
was
another
lesson
like
people
kind
of
wandered
all
of
this
stuff
integrated
together.
So
if
you're
familiar
with
using
ver
fauna
and
Prometheus
like
if
you
install
link
order,
you
will
feel
right
at
home
right
away,
that
stuff
will
work
out
of
the
box
and
then
to
administer
that
control
plane.
We
also
include
a
CLI
and
a
dashboard
and
we'll
kind
of
run
through
some
of
those
tools
in
in
a
moment,
so
I'm
gonna
attempt
a
bit
of
a
demo.
A
We
got
a
really
basic,
app
native
three
micro
services,
its
intended
for
managing
your
bookshelf,
so
you've
got
a
front-end
web
app
and
you've
got
an
book
service
behind
it.
An
author
service
behind
it
also
they're
talking
to
on
my
sequel
database.
Each
of
the
three
micro
services
has
three
replicas
and
we'll
kind
of
see
how
how
they
all
interact.
A
We've
got
a
traffic
server
that
or
a
traffic,
like
instance,
that
just
is
meant
to
you
know,
put
load
on
the
system
just
to
make
the
data
interesting,
we'll
also
look
at
the
books,
app
from
a
browser
and
make
sure
everything's
working
and
then
we'll
also
deploy
like
kind
of
a
chaos.
Monkey
script
that'll
start
tearing
it
all
down
and
kind
of
demonstrate
how
link
ID
and
Prometheus
handle
that
you
know
Frederick
was
was
mentioning
churn
and
you're
in
this
orchestra
environment
things
are
coming
up
and
down.
A
A
Okay,
so
here's
our
books
app
pretty
simple,
you
can
click
on
books.
You
can
click
on
authors.
You
can
edit
things.
You
know
pretty
basic
stuff
shoutout
to
NK
jemisin,
just
season
good
book.
So
that's
that's
kind
of
that
now
we're
gonna
do
a
quick
install
of
the
control
plane,
as
I
said,
like
linker
t1
was
super
complicated
to
set
up
when
I
want
to
install
link
or
d2
I
should
have
mentioned
I'm
using
a
linker
D
to
CLI
that
I've
installed.
A
A
If
you
wanted
to
set
different
ports
or
edit,
the
ammo
yourself,
like
you're,
totally
free
to
do
that.
So
if
we
kind
of
wanted
to
look
at
all
the
components,
everything
that's
installed
into
a
linker
D
namespace,
so
you
can
kind
of
see
everything
that
we
just
put
on.
You
can
see
the
the
four
components
of
the
control
plane.
We've
got
our
controller
ver
fond
of
Prometheus
and
our
front-end
web
dashboard,
which
I'll
show
you
in
a
minute
and
if
I
want
to
make
sure
they're
everything
came
up
correctly.
A
We've
got
a
check
command
that
that
you
know
boots
up
and
checks
that
all
of
the
different
components
of
linker
D
are
up
and
running.
So
if
you
install
anchor
D
and
something's
weird,
this
is
the
first
thing
we
will
tell
you
to
run
and
it
should
give
you
a
nice
little
report
on
what's
going
on
so
we're
good.
A
B
A
De
Deus,
so
for
those
familiar
with
Prometheus,
this
is
you
know
you
should
recognize
this.
The
interesting
part
is
how
Prometheus
is
configured
to
collect
metrics.
The
two
jobs
I
I
want
to
point
out.
Are
the
linker
view
controller?
Those
are
control
playing
like
like
go
components
that
were
collecting
metrics
from,
but
more
importantly,
we're
collecting
from
the
linker
the
proxies
that
are
running
inside
the
control
planes.
So
we
use
linker
D
in
linker
D.
A
A
So
now
I
want
to
show
you
the
the
linker
D
dashboard,
just
to
kind
of
give.
You
can
do
all
of
this
from
the
command
line,
but
it's
also
nice
to
just
kind
of
be
able
to
browse
around
and
see
all
the
information.
So
you
can
see
here
we're
getting
metrics
on
the
control
point
components,
and
this
is
the
same
way
that
you
would
get
metrics
on
your
your
own,
app,
which
you'll
see
in
a
minute
when
I
inject
into
the
book.
A
So
as
I
mentioned
before,
if
you
just
want
to
use,
grow
fauna
and
talk
directly
to
Prometheus
and
click
around
and
see
things
like
success
rate
request
rate
latency,
it's
all
here
and
we'll
dig
into
this
a
little
bit
more.
But
yeah
but
kind
of
the
point
was
like
this.
This
all
comes
up,
like
you
know,
with
without
a
lot
of
setup.
So
the
next
thing
we're
gonna
do
is
we're
going
to
inject
linker
D
proxies
into
the
books
app.
So
let's
just
kind
of
make
sure
that
the
books
app
is
still
here.
A
I
can
still
click
on
it.
It's
still
running
nothing
up
my
sleeve,
so
we're
going
to
take
that
books,
app
llamo
that
we
had
before
and
we're
gonna
pipe
it
into
a
linker,
D,
inject
command
and
all
that's
going
to
do
and
then
we'll
pipe
that
into
kubernetes
and
if
I
kind
of
want
to
look
at
what
just
happened.
It
took
our
application,
llamo
and
injected
several
Rus
proxies
into
it.
The
app.
If
we
did
this
right,
didn't,
should
not
have
gone
down.
A
You'll
see
some
pods
are
spinning
up,
other
pods
are
coming
down,
but
the
app
should
continue
to
work
without
without
issues
so
I
I
can
refresh
it.
I
can
still
click
around
and
everything
continues
to
work.
So,
if
I
go
back
to
my
dashboard,
if
if
if
things
works,
as
expected,
I
should
start
to
see
some
to
lemon
information
of
that
of
the
books,
app
that
I
that
I
just
deployed-
and
you
can
see
we're
already
getting
success
rate
request
rate
latency.
A
All
of
that
stuff
and
I
can
click
into
like
a
graph
on
a
dashboard
again
and
see
all
of
the
same
information
for
this
book
service.
So
if
you
take
away
nothing
else
from
this
talk
in
addition
to
this
700,000
requests
per
second
for
Prometheus,
no
of
it
like
without
changing
your
app
at
all,
you
can
run
a
link
or
D
in
jet
command
and
get
metrics
like
this
for
free.
Without
doing
a
single
thing,
you
don't
have
to
tear
down
we've,
specifically
targeted
link
or
D
to
run
in
in
a
number
of
different
environments.
A
So,
even
if
you're,
not
a
cluster
operator,
you
can't
even
create
namespaces.
You
can
still
install
linker,
D
and
inject
it
into
like
your
own
namespace
that
you
have
control
over
to
dig
into
a
little
more
data.
If
I
wanted
to
look
at
stats
for
all
of
all
the
things
that
are
running,
here's
my
different
deployments
in
the
books,
app
I
can
also
do
it
over
namespaces
again,
if
you've
used
q,
petal
you'll
recognize
all
of
these.
A
If
I
wanted
to
filter
from
one
deployment
to
a
from,
maybe
all
deployments
into-
let's
say
the
author
service
I
can
do
that
too.
So
again,
really
we
try
to
make
these
patterns
very
familiar.
We
also
have
a
tap
command
that
allows
you
to
tap
into
every
request.
That's
running
through
the
system.
This
isn't
like
tailing,
a
log,
this
activates
the
G
RPC
connection
and
starts
streaming
data
from
the
proxies
dynamically.
So
it's
only
doing
work
when
you're
running
the
tap
command.
A
You
can
obviously
filter
this
a
bunch
of
different
ways
to
kind
of
identify
requests
that
are
that
are
flowing
through
your
system.
If
you
want
a
slightly
more
rolled
up
version,
we
have
a
top
command.
That'll.
Give
you
it's
a
lot
like
top
on
your
laptop
and
it's
like
a
top
for
my
per
services,
so
it
shows
all
of
the
requests
that
are
that
are
flowing
through,
but
in
a
rolled
up
way.
Now,
Frederic
mentioned
earlier
the
the
cardinality
issued
from
you
piece.
A
It
can
ingest
a
ton
of
stuff,
but
it's
not
really
appropriate
to
to
abuse
it
as
far
as
putting
a
lot
of
different
time
series
into
it.
You
can
see
here
like
in
this
list,
some
of
the
requests
we're
seeing
of
path,
information
and
a
URL
that
includes,
like
author
slash,
168
JSON,
like
that's,
not
really
appropriate.
You
have
thousands
or
millions
of
objects
in
your
database
and
you
it
would.
A
A
Okay,
so
we
already
see
that
our
metrics
are
starting
to
get
rolled
up
in
a
way
that
is
aware
of
our
API
is
which
you
can
get
here.
You
can
get
if
I
jump
over
to
the
linker
D
dashboard
I
should
be
able
to
drill
into
you
know
the
deployment
of
my
author
service
I
can
kind
of
see.
What's
calling
it,
what
it's
calling
here's
kind
of
the
exploded
view
of
all
the
requests,
but
then
here's
the
nice
rolled-up
view
that's
again
aware
of
like
the
design
of
my
API.
A
A
So
this
is
going
to
I
mentioned.
We
have
three
services
with
three
replicas,
each:
that's
nine
pods!
This
thing
is
going
to
randomly
pick
a
pod,
every
10
seconds
and
terror
and
just
tear
it
down.
So
you'll
start
seeing
the
pods
bounce
up
and
down
and
kind
of
the
point
in
how
linker,
D
and
Prometheus
integrate
with
kubernetes.
Is
that
because
they're
getting
real-time
information
from
from
kubernetes,
your
your
applications
should
continue
to
work?
A
Okay
and
your
metrics
information
will
continue
to
be
like
would
be
continuous,
correct
and
real-time
like,
as
as
Frederic
mentioned,
having
like
a
five-minute
timeout
when
your
your
pods
go
away,
isn't
really
it's
it's
much
better
model
found
no
out
of
the
box.
You
know
when
things
are
bouncing
up
and
down
and
still
get
consistent,
metrics
across
that.
A
Hello,
computer,
ok,
we're
gonna,
look
at
it
like
this,
so
I
ran
a
linker,
the
install
command
to
install
the
control
plane,
I
ran
in
and
Jack
come
and
iddin
change
the
app
at
all.
We
use
stat
and
top
to
kind
of
get
the
information
about
about.
What's
going
on,
we
ran
a
dashboard
command
to
give
kind
of
a
high-level
browsing
over.
We
looked
at
four
Fano
and
Prometheus.
We
ran
a
profile
command
to
take
advantage
of
Swagger
and
we
ran
a
routes
command
that
gave
us
API
aware
statistics
about
our
about
our
application.
A
So
again,
quick
shout-out
to
the
linker
D
community.
It's
been
in
production
for
for
over
two
years
now,
we've
had
over
100
contributors,
there's
2,500
people
in
our
slack
Channel.
Please
come
and
join.
Ask
us
questions.
There
ask
us
questions
and
our
booth,
all
that
stuff.
So
on
behalf
of
Fredrik
and
I,
think,
thank
you
all
so
much.
A
So
good
question,
sorry
yeah.
The
question
was:
can
you
get
things
like
like
OS
and
container
level
information
you
can
do
that
with
with
Prometheus
and
Griffin?
Oh,
that's
part
of
one
career.
You
could
install
it
separately
yourself.
Lincoln
is
really
concerned
with
the
request
level
metrics,
but
we
have
variations
of
link
Rd
that
have
Prometheus
configured
to
gather
that
kind
of
information,
and
that's
totally
something
that
you
could
take
that
link
or
D
install
yeah
Moe
and
edit
yourself.
A
C
A
Yes,
so
the
question
was:
can
Lincoln
II
kind
of
be
your
one-stop
shop
for
TLS?
Yes,
we
have
when
I
ran
that
install
command.
There
are
flags
to
turn
on
TLS,
which
will
use
TLS
between
all
of
the
proxies
we're
working
on
things
like
ingress,
that's
coming
soon
and
working
on,
including
like
like
a
full
CA.
So
it's
kind
of
in
progress
right
now,
but
you
can
try
it
out
if
you
download
the
latest
version
of
liquor,
D
right
now,
yeah
go.
D
A
Great
question:
so
the
question
was:
do
you
get
Zipkin,
so
the
linker
d1
supported
Zipkin
linker
d2?
We
we
decided
to
drop
support,
for
we
felt
that
we
could
get
like
about
90%
of
the
benefits
of
tracing
by
just
having
metrics
about.
What's
calling
what
we
can
kind
of
build
your
call
graph
and
and
give
you
tracing
like
information,
but
if
you
really
need
something
like
what
Zipkin
does
where
you
get
a
request
and
you
need
to
see
that
request,
transit
through
all
your
micro
services,
you'll
need
to
configure
a
Zipkin
separately
to
do
that.
D
A
So
you
can,
you
can
deploy
the
control
plane
with
a
flag
that
says
how
many
replicas
of
each
component
you
want.
So
that
number
like
depends
on
on
all
kinds
of
like
performance,
but
it's
it's
designed
to
run
in
an
H
a
configuration.
So
if
you,
if
you
encounter
performance
problems
with
the
control
plane,
you
can
just
scale
up
that
replica
account
to
accommodate
your
cluster.
A
Okay,
that's
a
good
question,
so
it
the
questions
around.
How
could
I
split
up
my
linker
DS
if
I
have
certain
set
of
concerns
for
a
group
of
applications
over
here
and
another
over
here?
When
you
run
that
link
or
the
install
command
and
the
inject
command
you
can
specify
which
namespace
you
want
to
be
concerned
with,
so
you
can
deploy
n
number
of
completely
independent
linker
DS
in
different
namespaces,
and
when
you
go
to
inject
a
proxy,
you
can
decide
which
link
Rd.
A
A
A
B
So
the
question
was:
is
there
like
an
integration
story
with
the
Prometheus
operator?
So
for
the
most
part?
Yes,
the
specific
configuration
that
link
Rd
ships
out
of
the
box
uses
the
vanilla
current
prometheus
configuration,
but
there's
nothing
stopping
you
from
using
the
Prometheus
operators.
Here
these
two
model
the
same
thing:
it's
just
not
default.
Sorry.
B
A
Yep,
okay,
so
the
question
was
someone
astutely
noticed
that
one
of
the
containers
we
inject
with
the
proxy
requires
in
that
admin
privileges?
What's
going
on
there
is
we
have
two
pods.
One
is
the
proxy
itself.
The
other
is
in
a
container.
Those
knit
containers
are
changing,
IP
tables
to
to
manage
that
we
have
an
open
ticket
right
now
in
the
link
or
D
repo
to
support
inject
without
a
net
admin.
A
So
so
the
question
was:
have
we
looked
at
configurations
of
linker
D
that
run
kind
of
per
node
rather
than
as
a
sidecar?
That's
it's!
It's
interesting
like
linker
D,
one
that
was
kind
of
our
default
pattern,
and
so
it's
something
you
you
could
configure
link
or
D
to
to
do.
We
don't
have
a
lot
of
reference
designs
on
how,
but
it
is
something
that
we're
looking
at
so.
A
Everything
you
saw
online
is
is
available
in
open
source,
so
obviously
like
linker
DS
is
totally
open
source.
You
can
see
all
of
it.
We
have
a
linker,
the
examples,
repo
that
shows
alive.
The
examples
I
was
working
through
and
if
you
just
look
for
like
buoyant
books,
app
you're
gonna
find
that
the
books
demo,
you
could
run
through
everything
we
just
did
yeah
totally.
This
was
all
like
self-contained
crew
manatees
on
the
laptop.
A
A
A
We
reduce
we
do
support
ALS
connections
between
between
the
proxies,
so
you
can
turn
that
on,
but
it's
we
have
a
lot
more
work
to
do
there,
particularly
for
ingress
so
again,
there's
probably
open
tickets
in
the
link
of
the
repo
like
we're
gathering
requirements
for
right
now.
So
please,
please,
take
a
look
in
there.
Yeah.
C
A
A
That's
right:
yeah
they're,
an
we
similar,
similar
issue,
we've
recently
added
a
community
member
added
audio
injection
capability.
So
you
can
install
your
control
plane
like
through
through
helm
and
then
set
everything
to
be
auto
injected
and
then
anytime
you
deploy
an
app.
It
gets
proxies
for
free
yeah.