►
From YouTube: Autoscaling Multi-Cluster Observability with Thanos and Linkerd - Andrew Seigner & Frederic Branczyk
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Autoscaling Multi-Cluster Observability with Thanos and Linkerd - Andrew Seigner, Buoyant & Frederic Branczyk, Red Hat
Uniform observability across polyglot applications can be challenging, even more so across multiple clusters. In this talk, Andrew Seigner from the Linkerd team and Frederic Branczyk from the Prometheus team will demonstrate adding observability to applications across multiple Kubernetes clusters with zero code changes, and then autoscaling Thanos to enable observability across all clusters in one unified view. They'll outline how Linkerd uses Prometheus to provide zero-config observability for applications running in Kubernetes, and how Thanos enables observability to scale across any number of Prometheus instances.
https://sched.co/MPbU
A
All
right,
hey
folks,
my
name
is
Andres
ignor
I
work
on
linker
D,
we're
gonna
go
through
a
little
bit
of
background
on
linker
D,
then
we're
gonna
turn
it
over
to
Fredrik
to
go
through
Santos
and
then
hopefully,
we're
gonna
do
a
demo
that
works
that
integrates
the
two
and
kind
of
shows
kind
of
the
benefits
of
of
these
two
projects
working
together.
So
just
a
little
bit
of
background,
linker
D
is
a
service
mesh.
The
service
mesh
really
does
three
things
for
you.
A
It
gives
you
observability
security
and
reliability
across
all
of
your
applications,
and
you
shouldn't
have
to
change
in
your
application
code.
It's
usually
composed
of
two
major
components
that
control
plane,
which
is
kind
of
the
administrative
side
of
it.
Where
you
do
things
like
observability,
like
you,
roll
up
all
your
metrics,
you
can
set
policy
on
managing
traffic
and
it's
like
your
user
interface
into
it.
So
you
have
a
CLI
or
or
like
a
web
dashboard
or
something
to
interact
with
it
and
then
on
the
other
side,
you
have
the
data
plane.
A
A
Why
I
think
our
DUI
name
ended
up
being
written
in
Scala,
which
worked
great
for
a
while,
but
in
the
world
of
like
everybody's
running
a
little
go
process,
and
you
know
you
you're,
taking
up
five
or
ten
megabytes
of
memory,
trying
to
run
a
JVM
as
a
sidecar
that
may
need
a
hundred
megabytes
or
a
gigabyte
of
memory.
It
didn't
really
fit
so
about
a
year
year
and
a
half
ago,
we
said
about
doing
it
complete
from
the
ground
up
rewrite.
A
We
kind
of
looked
at
like
what
we
had
learned
over
the
last
several
years
with
linker.
Do
you
want
and
and
made
some
like
pretty
specific
design
decisions
with
linker
d2?
The
first
one
was,
it
would
support
kubernetes
only
linker
d1
would
would
run
anywhere
and
we
will
continue
supporting
it
for
a
long,
but
it
made
configuration
pretty
complex.
It
could
use
zookeeper.
It
could
use
Maysles
whatever
by
just
targeting
kubernetes.
A
It's
super
easy
to
set
up
and
you'll
see
that
in
a
demo
in
a
little
bit,
the
other
piece
was
that
prometheus
was
gaining
a
lot
of
traction
and
we
had
most
of
our
users
were
wanting
integrations
with
it.
We
had
some
reference
designs
on
how
those
two
things
work
together
with
link
31,
but
in
linker
t2
we
decided
to
just
make
Prometheus
like
a
first-class
component
of
linker
t2.
So
when
you
install
linker
D,
you
get
Prometheus
with
it.
A
The
other
part
was
looking
at
kind
of
available.
Languages
go
seem
like
a
very
natural
fit
for
the
control
plane
we're
integrating
with
kubernetes.
So
we
end
up
using
a
ton
of
libraries
that
are
in
the
kubernetes,
codebase
and
related
projects.
It's
been.
It's
been
really
helpful
to
like
use
that
code.
Instead
of
writing
our
own
for
the
proxy,
we
knew
we
needed
performance,
and
so
we,
we
kind
of,
took
a
look
at
what
was
out
there
and
rust
seemed
to
fit
the
bill.
A
We've
been
lucky
enough
to
work
with
a
number
of
the
rust
community,
like
library
developers,
the
folks
that
are
responsible
for
things
like
protocol
and
I/o
and
they've
they've
helped
out
a
lot
in
helping
build
up
this
proxy,
so
just
kind
of
a
really
quick
picture
of
the
architecture.
If
you
remember
that
two
components
of
a
service
mesh
you've
got
a
control
plane
in
a
data
plane
I'm
not
expecting
you
all
to
understand
every
box
here,
but
these
slides
are
attached
to
the
schedule.
A
So
you
can
dig
into
it
later
and
come
ask
many
questions,
but
really
I
just
wanted
to
call
out
like
there
is
an
observer,
the
layer
in
here
and
that's
where
we're
really
focused
on
for
this
talk
and
we're
using
Prometheus
like
we
found
it's
worked.
So
much
better
than
trying
to
invent
our
own
and
in
the
spirit
of
reusing
things
like
we
just
plug
a
graph
on
or
write
on
top
of
it,
and
so,
if
you're,
using
linker
D,
all
of
your
observability
tooling
should
work
like
with
like
it.
A
A
So
just
really
quick
I
wanted
to
focus
on
like
kind
of
the
core
piece
of
technology
that
ties
all
of
this
together,
which
is
when
we
inject
that
little
proxy
into
each
of
your
pods.
We're
exporting
some
very
specific
metrics
that
give
us
kind
of
three
golden
things
to
look
at
request
rate
success
rate
and
latency,
and
the
demo
we're
gonna
do
in
a
minute
is
going
to
focus
mostly
on
latency,
but
by
counting
those
things.
It
allows
us
to
give
you
very
useful
metrics
on
how
your
micro
services
are
behaving.
So
with
that
writer.
B
Okay,
so
let's
do
a
little
poll.
How
many
people
here
are
already
using
Prometheus?
That's
wow,
that's
almost
everybody,
so
I'm
gonna
run
through
this
one
a
little
bit
more
quickly.
So,
but
there
were
a
couple
hands
that
didn't
go
up.
So,
let's
make
sure
everybody's
on
the
same
page.
So
Prometheus
is
a
pull
based
monitoring
system.
What
that
means
is
that
Prometheus
in
a
regular
interval
will
go
to
our
applications,
literally
an
HTTP
endpoint
and
grab
all
the
metrics
and
adjust
it
into
its
time
series
database.
B
So
basically
what
the
example
that
we
have
up
here
is:
we
have
our
application
or
user
finishing
application.
That
is
exposing
Prometheus
style,
metrics
and
Prometheus
grabs
those
at
one
point
in
time
and
inserts
it
into
the
time
series
database.
Then
we
actually
get
user
facing
our
users
actually
are
using
our
applications.
We
are
actually
making
money
or
something
and
these
requests
increase
and
over
time
the
interval
has
passed.
B
The
Prometheus
ecosystem
has
main
primarily
revolved
around
in
within
a
single
kubernetes
cluster
and
everybody's
here
to
see
multi-class
or
manner,
obviously,
but
that
part
I
feel
like
we've
figured
out
pretty
well.
There
are
setups
that
give
you
out-of-the-box
cluster
monitoring
I
happen
to
work
on.
One
of
those
they're
Prometheus
has
a
lot
of
guides
out
there.
There
are
lots
of
blog
posts
how
to
do
app
monitoring
with
all
these
things,
so
I
feel
like
we've
covered
this
pretty
well,
but
once
people
have
are
comfortable
with
that,
we
have
pretty
much
the
same,
repeating
questions.
B
So
what
about
long-term
storage?
How
can
I
get
a
global
view
of
all
my
clusters
out
there?
Maybe
even
if
I
want
to
do
some
analysis
over
a
year
worth
of
data,
my
queries
are
getting
really
slow.
So
how
can
we
solve
this
and
just
about
a
year
and
a
half
ago,
maybe
a
year
ago,
a
team?
Actually
two
individuals
at
improbable
came
up
with
a
really
really
awesome
solution
to
this,
and
they
named
it
honest
so
tano's
does
have
all
of
these
features
and
you
can
kind
of
think
of
it
as
a
distributed.
Prometheus.
B
Basically,
Thanos
goes
and
takes
all
of
these
pieces
that
are,
in
the
that's
called
monolithic
application
that
it
makes
up
Prometheus
and
it
pulls
out
the
query
layer.
It
pulls
out
the
storage
kind
of
and
makes
it
a
distributed
prometheus
instead
of
the
amount
of
the
thick
process.
Now
that
doesn't
mean
that
you
won't
need
your
Prometheus
anymore.
We
want
to
monitor
our
distributed
monitoring
system
with
a
non
distributed
monitoring
system,
but
nevertheless
this
is
still
a
really
really
awesome
project.
B
So
I
happen
to
be
now
one
of
the
maintainer
of
tano's
as
well,
even
though
I
do
not
think
I
deserve
that,
but
the
group
was
kind
enough
to
grant
me
maintain
ership.
So,
let's
dive
a
little
bit
into
more
how
Thanos
works.
So,
as
you
see
we
on
the
slide,
we
actually
even
when
we
are
using
tunnels,
we
are
still
using
vanilla
upstream
prometheus,
but
how
does
that
tie
into
the
whole
story?
B
So
the
way
Thomas
works
is
that
it
puts,
gives
us
a
sidecar
that
runs
next
to
our
prometheus
Prometheus
server
and
that
way
basically
puts
a
common
interface
in
front
of
Prometheus.
That
is,
this
tunnel
store.
Api
and
I'll
keep
coming
back
to
this
to
the
story.
Api
theme,
because
I
think
this
is
a
really
cool
aspect
of
tano's.
Basically,
all
the
components
within
the
tunnels
project
implement
this
API
and
it's
kind
of
the
data
exchange
API
for
four
metrics.
B
You
could
say
for
a
time
series
data,
so
what
the
sidecar
then
does
is
Prometheus
builds.
These
immutable
blocks
of
data
Prometheus
always
depends
to
the
right
headlock
we're
very
frequently
that's
when
it
collects
data
and
writes
to
disk
and
every
two
hours
it
takes
all
of
that
and
puts
it
into
an
immutable
block
on
disk.
B
So
it's
never
gonna
be
changed
again,
it's
not
entirely
true,
but
let's
treat
it
as
that
for
now
and
Thomas
goes
ahead
and
sees
that
oh
there's,
a
new
meter,
immutable
block,
I'm
gonna,
upload
that
to
object,
storage
and
then
there's
another
component
called
Thanos
store.
I
know,
store,
API
store,
not
great
naming.
Unfortunately,
it's
hard
to
change
that
naming
now,
but
the
story
API
is
essentially
puts
the
story.
Sorry
I
didn't
mistake:
myself.
B
The
store
component
exposes
the
store
API
as
a
wrapper
around
an
object
storage
like
Amazon
s3,
for
example,
and
that
way
that's
a
nose
query
or
can
read
all
the
live
data,
basically
from
the
Prometheus
that
is
live,
appending
data
and
the
historic
data
from
object,
storage,
and
that
way
we
can.
Essentially.
This
is
how
the
long-term
storage
part
was
solved,
because
we
can
treat
object,
storage
essentially
as
an
endless
bucket
of
storage.
B
Now
Thanos.
Actually-
and
this
is
how
I
got
involved
into
tano's-
harness
is
about
deployment
flexibility,
and
while
the
sidecar
approach
is
a
really
desirable
one
and
it
works
extremely
well,
there
are
lots
of
deployment
scenarios
where
you
may
not
be
able
to
have
direct
access
to
your
edge
Prometheus,
where
your
Santos
courier
cannot
directly
do
these
G
RPC
calls
to
the
Prometheus
to
the
live
Prometheus
instance.
So,
actually,
I
had
just
just
had
a
conversation
about
Prometheus
servers
that
may
be
running
on
cars,
for
example,
or
just
things
that
may
have
interrupted
internet
connections.
B
B
Thanos
is
really
a
toolkit
of
components,
and
you
can
use
all
of
these
components
to
build
basically,
your
own,
very
own
monitoring
system,
so
we're
all
here
for
multi-class
or
monitoring
so
is
tano's.
Equal
does
planas
equal
multi
class
or
monitoring
no,
but
you
can
make
it
your
multi
cluster
monitoring
system
and
again
the
store
API
is
the
I
think
this
is
the
coolest
thing
about
Panos,
because
the
store
API
allows
you
to
build
all
these
wild
combinations
of
of
components
that
ultimately
make
up
the
system
that
you
exactly
need.
B
So
I
think
this
is
really
what
I
want
people
to
go
away
with
here.
Sano's
is
about
flexibility,
and
you
should
architect
your
setup
to
your
need
and
the
store.
Api
is
really
the
thing
that
allows
us
to
do
this
with
Thanos
so
architect
to
your
need.
So
with
that,
so
you
just
gonna
show
us
some
action,
we'll.
A
A
All
of
that
up
into
a
fan,
o
Square
and
we're
gonna
be
able
to
view
these
metrics
in
in
an
aggregated
way
and
then
we'll
use
graph
honor
to
kind
of
visualize
all
that
data
to
dig
into
specifically
how
the
auto
scaling
is
going
to
work.
I'll,
I'll
kind
of
walk
through
this
really
quick.
So
again,
we've
got
our
client
server.
We've
got
our
lucre
D
injected,
because
linker
D
is
working
with
Prometheus
we're
getting
things
like
request
and
latency
information.
A
Then
the
the
custom
part
of
this
demo
is
that
we're
going
to
attach
a
fan
owes
to
the
linker
D
that
comes
with
the
attached
at
Thanos
to
the
prometheus
that
comes
with
linker
D.
When
you
install
a
like
a
vanilla,
linker
D.
Obviously,
you
don't
get
danos
today,
but
the
the
demo
and
the
examples
that
that
I've
got
on
github
kind
of
demonstrate
how
you
could
do
this
and
I
found
this
other
nice
piece
of
software
called
the
case,
Prometheus
adapter.
A
What
that
does
is
it
takes
Prometheus
data
and
in
this
case
it's
the
Thanos,
but
because
they
know
some
Prometheus
look
exactly
the
same
from
an
API
perspective,
I
just
plug
it
into
the
global
Thanos
query,
and
that
allows
us
that
allows
kubernetes
to
see
this
Thanos
data
as
as
kubernetes
metrics
and
from
there
when
a
latency
value
goes
above
what
we
want
it
to.
In
this
case,
10
milliseconds.
A
It's
going
to
scale
up
the
server
replicas
on
all
of
the
clusters,
so
kind
of
the
the
gist
of
this
is
given
running
the
same
system
across
four
different
cloud
providers.
If
there's
a
problem
or
an
observability
event
in
one
of
them,
we
can
react
and
adjust
in
a
bunch
of
other
data
centers
and
to
just
kind
of
call
out
the
the
line
of
I've
drawn
around.
All
of
that's.
What
we're
deploying
for
copies
of
the
global
fan?
A
A
The
first
thing
I'm
gonna
do
is
just
show
you
some
empty
dashboards,
I've
already
deployed
the
Thanos
square.
Cuz
I
didn't
want
to
bother
waiting
for
all
of
us
to
like
stare
at
this
thing
and
deploy.
But
the
important
thing
to
note
here
is
that
we've
got
our
Thanos
query.
It's
configured
to
talk
to
basically
eight
backends,
there's
two
backends
per
per
cloud
provider.
It's
those
those
Thanos,
sidecar
and
Thanos
stores
that
that
Fredrik
was
talking
about
the
sidecar
gives
you
like.
A
A
A
So
let
me
unpack
the
the
command
that
I
just
ran
there.
So,
let's
see
when
you
install
linker
D
like
what
what
I
want
you
to
take
away
from
this
is
how
easy
it
is
to
do
that.
We
have
a
CLI
tool
called
linker
D,
it's
got
if
you
use
coop
cuddle,
it
feels
like
very,
very
familiar.
We
use
a
lot
of
the
same
code
as
coop
cuddle.
If
you
want
to
install
it
all
you
do,
is
you
type
linker,
D
install
and
that
just
dumps
a
bunch
of
yamo?
A
We
tried
to
be
really
explicit
about
what
we
were
doing
so,
if
you're
comfortable
with
that,
you
would
just
pipe
it
into
coop
cuddle,
apply
and
I'm
not
going
to
do
that
now,
because
I
don't
want
to
install
linker
D
on
my
laptop,
but
that's
like
the
entire
installation
procedure.
You
may
have
noticed
that
the
command
I
just
ran
referenced,
an
environment
variable
which
was
a
file
it's
because
I
did
had
to
make
a
slight
modification
to
the
vanilla,
linker
D
to
enable
it
for
thanos.
A
But
again,
all
of
that's
documented
in
the
repo,
so
I'm
just
gonna
run
a
command
really
quick.
A
version
check
just
to
make
sure
that
all
the
clusters
came
up.
It
looks
pretty
good.
I'm
gonna
run
one
more
command,
which
is
kind
of
a
Diagnostics
check,
just
to
make
sure
things
are
looking
good.
Let
me
unpack
this
command
a
little
bit,
so
we
have
I'm.
Just
gonna
run
this
command
against
one
of
the
clusters
arbitrarily
any
time
you
install
linker
D.
You
can
run
this
check
command.
A
A
So
so
far
so
good,
the
next
thing
I'm
going
to
do,
is
to
play
our
sample
application
and
again
I.
Don't
expect
to
be
like
reading
every
line
here.
The
point
is
that,
like
this
is
like
really
simple
and
easy
to
get
going,
so
let
me
unpack
the
command.
I
just
ran
there.
So
what
I
did
was
I
added,
a
very
simple
client-server
application.
There's
a
client
there's
a
server
and
there's
a
horizontal
pod
autoscaler
with
it
that's
configured
to
say
our
target
value
is
10
milliseconds.
A
So
what
that
inject
command
does
it
makes
one
small
change
to
your
e
mo.
It
adds
an
annotation
that
says
when
this
thing
gets
deployed
onto
the
cluster,
please
inject
that
Russ
proxy
into
each
pod.
You
can
set
this
annotation
at
the
pod
spec
level
at
the
name
space
level
by
default.
Obviously
we
don't
want
to
start
injecting
all
of
your
your
applications,
but
so
you
can
kind
of
roll
it
out
gradually.
A
So
if
that
all
worked,
I
should
be
able
to
refresh
this.
Yes
we're
almost
there.
So
we
can
see
that
the
fan
of
the
global
Thanos
query
is
now
connecting
to
four
sidecars
and
for
store
API.
So
that
means
that
all
of
the
linker
DS
have
come
up,
we're
now
collecting
metrics
from
each
of
those
providers.
If
I
flip
over
to
this
dashboard,
I
should
start
to
see
metrics
things
are
looking
good,
so
the
dashboard
we
were
looking
at
here.
This
is
what
you
get
like
with
a
generic
like
vanilla,
link
or
D
installation.
A
The
one
thing
I
did
do
all
of
these
words
you
see
here
are
what
you
would
get
by
default
by
deploying
linker
D,
but
I
added
one
for
this
demo
and
it's
specifically
to
kind
of
get
a
view
on
this
particular
application
and
what's
going
on.
So,
if
you
look
on
the
Left,
we've
got
one
server
instance
like
per
cluster
and
like
the
those
are
the
different
providers.
A
So
I'm
gonna
set
the
replicate
count
on
my
load,
generator
on
just
one
of
the
cloud
providers
on
Google,
just
arbitrarily
I'm
gonna
increase
it
from
1
to
20.
So
that's
where
we're
gonna
get
the
2,000
requests
per
second.
This
is
all
gonna
go
pretty
fast.
I'll
try
to
talk
through
all
the
changes,
but
I
think
you'll
get
the
gist
okay,
so
we
scaled
it
up.
So
the
first
thing
we
should
notice.
Probably
if
this
worked
is
the
the
middle
column
should
start
to
have
a
huge
request:
volume
spike.
A
There
we
go
and
then
all
of
a
sudden
you
see
the
p99
latency
vite
went
from
3
milliseconds
to
60.
So
like
we're
in
trouble.
That
was
a
massive
increase
in
in
p99
latency
starting
to
come
back
down
to
27.
We
should
hopefully
start
to
see
the
horizontal
pod
auto
scalars
kicking
in.
If,
if
we
did
all
of
this
right,
let's
see
there
we
go
so
Google
is
first
to
notice
it.
A
It
is
a
little
bit
bumpy
as
you're
cranking
up
the
servers,
but
you
can
see
that
where
we
are
starting
to
get
these
up,
these
have
a
limit
of
ten,
but
as
those
go
up,
I
would
expect
that
latency
value
to
to
stabilize
a
bit
so
we'll
just
give
it
like
a
few
more
seconds.
It's
a
little
bumpy
we're
at
37
milliseconds,
hopefully
it'll
get
down
to
about
10
milliseconds
within
a
few
moments,
but
you
can
see,
like
all
of
this
happened
within
a
few
seconds
and
kind
of
the
gist
of
this.
A
Is
that,
like
you,
should
not
have
gotten
woken
up
in
the
middle
of
the
night
for
like
a
20x
increase
in
traffic
and
a
massive
latency
spike?
And
it's
important
to
note
that,
like
you
know,
typically
horizontal
pata,
auto
scalars
they
out
of
the
box,
they
they
work
more
often
on
things
like
CPU
and
memory
which
are
important
metrics
to
look
at
and
they
are
valid
things
to
scale
on.
A
But
as
far
as
focusing
on
your
users
experience
whether
it's
somebody
looking
at
a
web
page
or
somebody
calling
your
service,
something
like
a
p99
latency
values
is,
can
be
much
more
valuable
to
ensure
that
you
provide
like
a
consistent
experience
for
the
person
calling
your
service
a
couple
more
things.
I
want
to
call
out.
A
So
not
only
is
are
we
leveraging
fan
O's
to
do
this
Multi
cluster
setup,
but
because
we're
using
those
object,
stores
I've
been
working
on
this
demo
for
a
few
days
and
we
should
be
able
to
go
back
way
beyond
what
Prometheus
like
would
typically
give
you
like
default
I.
Think
it's
like
6
hours
or
24
hours,
or
something
like
that.
It's
limited
right,
but
fanox,
because
it's
plugged
into
these
object
storage
in
each
of
the
providers.
We
can
go
back
like
several
days
and
kind
of
roll
up.
A
All
of
these
aggregated
metrics
over
a
long
period
of
time,
so
really
that,
like
the
takeaways
from
this,
is
that,
like
all
of
this
stuff,
just
like
should
work
like
out
of
the
box,
you
get
p99
latency
values
with
with
very
little
work,
just
by
installing
linker
D
and
by
installing
Thanos.
You
get
really
nice
flexible
monitoring
that
can
work
across
clusters.
A
If
you
want
it
and
yeah
yeah
it
just
kind
of
all
works,
I'm
really
upset
that
it
didn't
get
below
10
milliseconds,
while
we're
standing
here
so
I
might
click
back
after
a
slide
or
two,
but
we'll
see
what
happens.
So.
Let
me
just
just
summarize
really
quick.
Please
check
out
both
of
these
projects
as
far
as
as
link
or
D,
we've
got
a
pretty
active
slack
Channel,
there's
about
3,000
of
us
in
there
a
lot
of
me
and
my
co-workers
sitting
in
there.
So
please
come
say,
hi
and
and
thanks
a
lot.
C
Thank
you
for
the
talk
we
are
in
the
process
of
also
setting
up
Thanos
and
there's.
One
thing
that
we
have
noticed
is
we
are
using
a
GCS
and,
if
we
cure
a
longer
than
so,
if
acute,
a
long
storage
basically
are
for
requests
times
out.
If
you
also
experience
such
such
issues,
when
you
have
implemented
this
and
if
so,
how
to
choose
or
solve
this
I.
B
Can
use
this
mic,
so
you
want
to
make
sure
that
you
set
up
the
compactor,
because
essentially
what
the
compactor
does
is
it's
the
thing
that's
responsible
for
down
sampling
so
I
that
the
blocks
are
immutable
right
and
this
is
kind
of
where
the
it's
not
a
lie.
But
what
the
compactor
does
is
multiple
things.
It
takes
smaller
blocks
and
produces
larger
blocks
out
of
those,
but
at
the
same
time
it
performs
down
something.
So
it
rolls
up
the
values
to
a
five-minute,
roll-up
and
an
hour,
roll-up
and
so
on,
and
then
the
Thanos
courier.
B
When
you
do
a
query
over
a
really
long
time
span,
it
can
automatically
detect
the
optimal
resolution
for
this
particular
query
and
then,
instead
of
querying
resolution
of
15
seconds
when
you're
querying
over
a
year,
you
don't
actually
meet
that
resolution.
Right.
Duda
can't
even
place
the
dots
in
your
graph
on
our
graph
for
that,
so
instead
it
would
use
the
one
hour
resolution,
so
I
highly
recommend
checking
that
out.
If
you
haven't
already
and
if
that
still
doesn't
work,
then
open
an
issue
for
us
and
we'll
look
at
it
into
it.
D
Hi
guys,
thanks
for
the
talk,
I've,
also
a
short
question.
We
have
customers
which
are
collecting
per
day
metrics
and
a
size
of
to
3,
terabytes
and
I
saw
tano's,
is
storing
all
the
metrics
on
objects
or
how
does
it
fit
in
terms
of
performance?
I
mean
our
customers
are
trying
to
push
all
these
metrics
currently
to
Hadoop.
We
spark
or
something
like
that,
to
train
models
to
analyze
all
the
metrics
and
does
it
fit
also
with
castanos
and
this
scale
yeah.
B
So
Thanos
has
multiple
layers
of
caches,
so
not
it
doesn't.
It
doesn't
necessarily
always
go
to
object,
storage
directly.
So,
basically,
when
the
store
API
starts
it
downloads
the
index,
so
that
it
basically,
you
can
think
of
it
as
an
M
map,
but
it
instead
of
against
your
local
disk
against
object,
storage,
and
so
it
will
kind
of
fetch.
Only
the
data
that
it
needs
and
has
an
additional
cash
on
disk.
F
Just
now
about
scaling
of
the
actual
set
up,
apart
from
the
your
test
out,
so
obviously
across
different
cloud
providers,
make
sense
that
you
have
like
the
query,
API
or
the
query
are
part
of
it
they're
available
to
do
the
store
API
if
you're
on
a
single
cloud
provider-
and
you
wanted
to
scale
light
Falmouth's
itself,
does
it
make
sense
putting
that
extra
layer
of
the
query
er
between
queriers
or
would
it
be
scaling
it--
store
instead?
So.
B
I
guess
that's
a
little
bit
my
own
opinion
I
think
you
should
probably
try
out
a
couple
of
things,
but
I
don't
think
you
would
need
the
the
additional
layer
of
of
the
courier.
That's
usually
meant
for
to
like
work
with
your
topology
and
you
don't
actually
have
that
problem.
So
I'd
recommend
just
using
the
sidecars
of
the
receive
to
do
the
hot
querying
and
then
up
upload
to
object,
storage
and
do
the
long
term
data
there.
G
B
E
Thanks
for
the
talk
both
of
you,
we
are
already
using
linker
the
one
for
some
of
the
legacy
infrastructure
we
have.
We
are
looking
forward
for
using
service
mesh
in
our
new
infrastructure.
We
didn't
decide
yet.
What
can
you
give
me?
What
like
right
now,
most
people
talk
about
SEO
invoice
or
link
RT.
Can
you
give
me
what
the
point
that
you
think,
in
your
opinion,
the
best
for
link
Rd
to
go
to
or
what
is
the
point
that
is
missing
from
link
Rd
and
then
go
to
East
you
and
thanks
for
being
honest
sure.
A
Yeah
yeah,
that's
a
question:
we
get
a
lot
yeah,
you
probably
hear
quite
a
bit
about
sto
and
the
linker
D,
and
you
are
kind
of
two
of
the
main
service
meshes
in
the
space.
I'd
say
two
areas
where
we
were
doing
pretty
well
in
one
area
that
we
need
a
little
ketchup
on
I.
Think
we
with
linker
D.
We
really
focus
on
usability
and
being
really
clear
to
the
the
operator
on
what's
going
on
so
you
saw
like
in
this
demo.
A
I
did
like
a
full
installation
of
linker
D
and
then
like
layered
in
an
application
and
that
all
took
like
a
few
seconds.
Two
minutes,
that's
one
of
our
like
primary
design
goals.
We
want
to
be
as
simple
as
possible
and
with
with
more
time,
there's
a
bunch
of
debugging
tools
that
we
build
into
linker
D,
because
we
want
it
to
be
very
transparent.
It's
it's
a
really
critical
part
of
your
your
infrastructure,
and
so
we
to
be
really
clear
on
like
what's
going
on
and
kind
of
what
are
all
the
different.
A
What's
the
state
of
all
the
different
components.
So
that's
why
I'm
like
usability?
The
other
is
performance.
There
are
some
like
sort
of
controversial
blog
posts
right
now.
If
you,
google,
for
SEO
link
or
D
benchmarking,
but
overall
I
think
we've
done
pretty
well.
There
rust
was
a
bit
of
a
bet
a
year
year
and
a
half
ago,
but
I
think
it's
paying
off
for
us.
The
area
that
I
think
we
need
to
play
catch-up.
A
Oh
that's
a
good
question,
so
the
question
was:
do
you
get
tracing
with
linker
D,
so
like
explicit
tracing
like
Zipkin
or
open
tracing,
we
do
not
provide
that.
You
would
have
to
add
it
yourself.
Our
thinking
is
that,
with
with
linker
D,
because
it's
managing
traffic
between
all
of
the
micro
services
in
your
cluster,
we
can
show
you
every
edge
in
your
topology
graph
and
we
can
give
you
success
rate,
latency
and
request
rate
on
every
edge.