►
From YouTube: Shipping Metrics From the Edge - Matthias Loibl, Red Hat
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Shipping Metrics From the Edge - Matthias Loibl, Red Hat
Computing is getting pushed to the edge, it may be your car, TV, washing machine, or your toaster. All these devices have a lot of computing power these days. While extending the cloud to the edge is getting solved with projects like KubeEdge or k3s, in this talk we want to take a closer look at how to run Prometheus on them. We want to configure Prometheus in a way that we can replicate its data to a central collecting point, that is running Thanos on Kubernetes in a replicated setup, and then make use of all the shipped metrics to efficiently query across the entire fleet.
https://sched.co/UaZc
A
Hello,
welcome
to
this
dry
room.
Please
take
a
seat
or
Vadra
start.
My
name
is
bjorn.
I'm
your
track
host
I'm
a
Prometheus
maintainer,
and
this
is
where
I'm
really
really
excited
about
people
who
do
like
weird
things
with
Prometheus
and
all
the
other
projects
in
the
wider
ecosystem.
We
have
here
on
stage
a
person,
his
real
name
as
metal
Matt,
see.
Whatever
you
see
in
the
program
he's
the
maintainer
of
the
Prometheus
meter
in
Berlin,
the
organizer.
A
A
B
You
Baron,
all
right,
I,
don't
say
anything
has
been,
did
all
the
talking
on
this
yeah.
So
let's
get
started.
So
we
have
a
problem
and
we
want
to
solve
that
problem
and
I
actually
brought
a
Raspberry
Pi
with
me,
and
we
want
to
be
able
to
basically
know
what
is
going
on
on
a
Raspberry,
Pi
or
any
other
like
small
device
that
is
running
somewhere
and
yeah.
B
These
devices
might
be
a
gap
devices,
so
we
might
not
be
able
to
just
like
go
onto
them
and
yeah
find
out
what's
going
on,
but
like
we
need
something
that
that
we
can
use
to
to
make
that
happen,
and
there
are
many
different
applications
running
on
that
and
we
want
to
like
have
metrics
on
on
these
applications
themselves.
So
we
want
to
know
if
one
of
the
devices
has
a
problem.
B
So
this
basically
the
set
up
I,
don't
know
like
we
could
imagine
something
where
we
have
a
big
house
and
running
Raspberry
Pi
in
each
room,
and
then
we
want
to
I,
don't
know
like
measure
the
temperature
and
we
are
doing
this
with
a
Raspberry
Pi.
So
what
happens
if
one
of
them
dies?
We
don't
really
know,
and
we
could
like
do
something,
but
as
we
are
cube
con-
and
we
know
what
prometheus
is,
we
want
to
run
a
kubernetes
small
kubernetes
distribution.
In
our
case,
I'd
I've
chosen
k3s
and
run
Prometheus
on
that.
B
So
we
still
have
the
problem
of
like
how
do
we
get
the
metrics?
How
can
we
see
the
metrics
yeah?
So
we
we
want
to
solve
that,
so
the
devices
themselves
like
for
alerting
they
they
get
all
the
alerting
roots
on
the
Prometheus
itself
and
we
can
configure
those
to
send
the
individual
alerts
to
a
central
alert
manager
and
we
can
populate
those
alerts
with
a
device
ID.
So
we
know
from
which
device
this
alert
is
coming
from
and
then
on
the
server
side.
Whatever.
B
So
we
kind
of
solved
alerting,
but
we
serve
owners
of
the
metrics
problem
right.
We
want
to
see
what's
going
on
ourselves,
so
idea
number
one
is
we
run
a
Prometheus
on
these
services
and
we
could
have
a
reverse
proxy
and
we
could
access
that
and
we
could
also
maybe
SSH
into
the
device
itself.
So
there's
our
user
wondering
what's
going
on,
but
we
can't
really.
We
can't
really
access
this,
these
Promethea
so
yeah.
B
We
have
a
problem
and
we
can't
solve
it
that
way
and
we
still
can
only
access
one
of
the
Prometheus
and
not
like,
like
aggregate
or
like
query
across
the
entire
fleet.
The
idea
number
two:
we
still
run
Prometheus
on
each
device.
We
might
still
have
a
reverse
proxy
and
we
could
have
a
central
Prometheus
that
federates
from
that
central
place
to
other
devices.
B
However,
we're
still
not
able
to
do
that
because
we
can't
like
go
into
the
into
the
individual
devices
from
from
that
central
place,
so
we
improved
the
aggregation
problem,
so
we
can
query
across
all
devices
with
a
central
Prometheus.
However,
we
are
still
exposing
things
to
the
outside
and
that
might
not
be
possible
and
we
might
be
able
to
federate
every
so
often,
so,
maybe
only
every
like
two
minutes,
but
we
have
a
scrape
interval
for
the
individual
individual
targets
of
15
seconds.
B
So
we
are
losing
like
a
bunch
of
samples
in
between
so
another
idea.
We
could
have
something
like
a
small,
app
or
a
client
running
next
to
Prometheus
that
then
queries,
Prometheus
and
actually
just
takes
everything
it
sees
into
memory
and
then
forwards
this
to
a
central
Prometheus.
So
we
inverted
the
problem
and
we
can
finally
do
this
because
we
know
from
the
individual
Prometheus
talked
to
the
central
Prometheus.
B
However,
so
we
don't
expose
Prometheus
to
the
outside
anymore.
However,
we
still
only
collect
metrics
every
so
often
and
we
still
lose
samples
in
between
and
then
the
other
problem
is
this,
probably
something
that
is
a
custom
bird.
So
we
need
to
maintain
that
we
need
to
make
sure
it
works.
We
need
to
test
this
and
it
might
not
be
the
best
solution.
B
What
a
thermal
sight
car
does
is,
it
writes
blocks
to
its
local,
TS,
TV
or
like
to
the
to
the
local
disc,
and
we
have
an
immutable
block
and
we
could
use
that
to
ship
this
off
to
object,
storage,
so
we're
introducing
one
more
component
and
and
not
like,
have
a
sight
card
that
we
run
ourselves
and
build
ourselves,
but
we
can
make
use
of
ethanol
component
that
is
tested
and
better
tested.
So
we
could
do
that.
However,
we
still
have
a
lack
of
metrics
showing
up
so
by
default.
B
So
we
need
to
wait
2
hours
until
like
in
worst
case
scenario
until
we
can
see
things,
and
we
also
need
to
expose
the
object
storage
to
these
devices,
so
that
might
not
be
the
best
not
not
be
the
best
option
that
we
that
we
have
the
idea
of
number
5
now
we're
getting
like
into
into
the
interesting
things
we
could
use
something
called
remote
write
and
that
is
essentially
database.
Replication.
Boot
into
Prometheus,
and
we
were
replicate
the
right
ahead,
look
so
Prometheus
whenever
it
scraped
something
it
writes
what
it.
B
What
if
what
it
has
seen
into
into
this
right,
a
headlock,
and
we
could
simply
replicate
this
to
the
center
of
Prometheus
and
just
like
a
brief
introduction
to
remote
write
how
that
works.
We
have
the
write,
a
headlock
and
each
URL
that
are
each
destination
that
we
want
to
write
to
gets
its
own
cue
and
then
we
can
buffer.
B
B
B
So
what
have
we
improved
with
this
idea?
We
don't
need
to
expose
objects,
charge
anymore.
We
use
a
built-in
feature
of
Prometheus,
but
we
have
that
Central
Prometheus
that
might
be
flooded
with
requests
and
it
doesn't
really
scale
above
a
certain
amount
of
devices
and
samples
that
we
ingest
so
I,
don't
know
we
need
to
do
something.
I
guess.
B
Yeah
we
could,
we
could
create
a
component
that
would
be
able
to
receive
these
these
metrics,
these
samples,
just
like
Prometheus,
does,
but
we
could.
We
could
have
something
that
then
receives
these
metrics.
The
query
is
able
to
like
the
fairness,
query
is
able
to
query
that,
and
if
it
is
just
one
component,
it
will
blow
up
again.
So
we
could
have
this
simple
component
or
not
simple
component,
but
we
have
could
have
this
receive
component.
That,
essentially,
is
a
hash
ring.
B
So
in
this
case
this
the
farmers
receive
component
that
is
able
to
create
a
hash
ring
and
upon
receiving
metrics.
We
will
actually
take
the
entire
labour
set
that
we
receive.
We
were
sorted
to
make
it
deterministic
and
then,
with
the
help
of
the
tenant
ID
we
can.
We
can
even
distribute
this
a
bit
more
to
be
able
to
distribute
the
the
metrics
and
and
the
load
across
all
the
received
components,
and
it
is
using
a
hash
function
that
also
Prometheus
users.
So
it's
like
again
building
on
top
of
the
shoulders
of
the
Giants.
B
Basically,
here's
the
snippet
from
from
the
receive
component
itself
this
go
and
you
can
see
we
put
in
the
tenant
as
a
string
and
the
time
series,
and
from
that
we
get
all
the
we
get
all
the
we
got,
all
the
we
got
all
the
sum:
the
labels,
the
labor
set.
We
hash
this
down
here
and
then
to
make
to
make
it
able
to
be
mapped
on
on
the
individual
devices
on
the
individual
receives
replicate
sorry,
the
individual
replicas
that
we
have
in
the
hash
ring.
B
We
need
to
model
over
this
with
a
number
of
replicas
we
have
so
we
can.
We
can
be
able
to
tell
which
end
point
because
there's
a
map
in
the
end
we
get
a
URL
we
can,
which
is
the
receive
component
that
is
responsible
for
storing
this
specific
metric.
So
this,
basically
what
it
looks
like.
So
we
are
we're
now
able
to
remote
write
to
a
hash
ring
and
we
can
scale
it
up
and
down
as
we
want
and
yeah.
B
So
we
have
this
trade-off
like
everything
that
is
super
new
is
in
a
receive
component,
we
can
distribute
the
load,
but
for
long
term
storage
we
can
make
use
of
the
object,
storage
again
complete
the
picture
we
wanted
to
be
able
to
basically
tell
the
like.
In
our
case,
I
said:
we
want
to
have
the
room
temperatures
across
the
entire
house,
and
now
we
can
have
a
furnace
rule
component
that
is
able
to
query
the
founders.
B
Query
that
then,
fans
out
to
the
store
and
the
receive
components-
and
we
can
see
you
I-
don't
know
like
the
temperature
of
the
past
week
and
across
all
the
raspberry
pies
that
we
have
in
our
house
to
make
it
even
a
bit
more
complicated
or
not
complicated
but
sophisticated.
We
are
even
able
to
run
multiple
stateful
sets
or
receive
hash
rings,
so
we
have
kind
of
a
tenancy
system
in
the
furnace
receive
component
and
to
to
configure
this,
we
have
a
contact
map.
B
That
kind
of
looks
like
this,
so
we
have
a
hash
ring
as
a
name
and
the
tenants
that
should
go
into
this
hash
ring
and
the
name
of
the
hash
ring
is
actually
the
same
name
that
a
stateful
set
has.
So
we
are
mapping
the
state
for
said
to
the
tenant
in
this
case,
and
then
the
farmers
receive
controller
is
a
little
component
that
we
wrote
that
is
able
to
simply
see
that
the
amount
of
replicas
in
each
hash
ring
and
generate
these
endpoint.
B
B
Usually
I
those
two
arm
and
I
plan
on
contributing
this
back
upstream,
so
it
should
be
totally
seamless
to
deployed
the
entire
stag
to
a
Raspberry
Pi
in
the
future,
and
then
we
can
see,
for
example,
the
load
five-four
for
that
Raspberry
Pi
and
then
for
the
same
Raspberry
Pi.
Oh.
If
we
put
the
cluster
equals
metal,
metal,
metal
mats
of
Pi
foreign
there,
we
can
see
that
we
get
the
same
data
on
this
remote
remote,
remote
right
receive
component,
which
is
a
fan
of
receive
hash
ring
in
this
case.
B
So
yeah
like
to
prove
the
point.
We
are
actually
running
this
on
a
cluster
and
then
we
are
able
to
query
this.
Let's
go
a
bit
further
back
a
bit
further
and
we
can.
We
can
basically
look
at
the
last
week.
For
example,
we
can
see
that
that
we
get
metrics
from
from
the
last
week,
even
though
the
Raspberry
Pi,
for
example,
it
doesn't
even
have
that
data
anymore,
just
to
just
to
show
the
namespace
that
that
is
running
these
components.
B
Right
so
today,
there's
gonna
be
a
finest
deep
dive
that
is
going
into
the
receive
component
a
bit,
but
also
into
every
every
other
component,
and
that
Thanos
has,
and
you
can
find
that
at
520
in
room,
6,
C
and
just
to
summarize
the
stock
again.
We
are
now
able
to
run
a
very
lightweight
mana
monitoring
system
on
the
edge
and
we
can
turn
on
the
retention
of
the
prometheus
to
just
a
few
hours
and
like
remote,
write
everything
to
a
central
place.
B
We
can
make
use
of
the
Prometheus
built
in
replication
functionality,
and
the
famous
receive
component
in
this
case
helps
us
with
our
GAAP
deployments
and
we
can
scale
the
ingestion
the
ingestion
load
just
like
we
need
it
right.
So
we
we
also
are
able
to,
for
example,
use
horizontal
pot
autoscaler
based
on
custom
metrics,
for
example,
to
whenever
we
receive
more
metrics.
We
could
scale
the
Seifert's
head
up
and
down
just
just
just
like.
B
If
you
want
to
learn
more
about
tongues
itself,
there's
a
kata
Kota
that
you
that
you
can
basically
do
a
walk-through
of
the
Pharos
project
itself
and
I.
Invite
everybody
to
work
with
us
and
the
redhead
team
is
hiring
everywhere.
So
just
come
and
talk
to
us.
If
you
want
to
work
on
these
kind
of
of
topics
and
thank
you
so
I
think
we
have
a
lot
of
time
for
questions.
C
B
So
I
think
in
our
case,
what
we,
what
we
do
is
we
have
alerting
rules
on
the
individual,
Prometheus
components
on
the
individual.
For
me,
Promethea
running
on
the
devices,
and
if
those
like
stop
remote
writing,
we
can
make
use
of
the
central
alert
manager
that
we
can
send
alerts
to
yeah.
If
there's
no
connection
at
all,
it
is
really
difficult
to
tell
if
it's
remote
right
still
able
to
remote
write
or
not.
B
B
We
do
have
a
so
we
can
I
can
show
you
this,
so
we
do
have
something,
for
example,
where
we
can
do
the
count
of
up
mount
of
up
and
then
by
cluster,
for
example.
So
we
can
see
that
indeed
we
are
receiving
from
multiple
classes
here,
I'm
just
doing
in
query
to
make
it
faster,
so
yeah
we
can
see
we
have
like
for
for
Promethea
remote
writing
the
entire
time,
and
this
is
kind
of
how
we
can.
We
can
see
what
our
remote
writing
and
then
we
could
as
you
as
previously.
D
Hi,
you
mentioned
a
little
bit
about
collecting
the
metrics
from
air
gapped
devices
yeah,
it
kind
of
seemed
like
there
was
an
assumption
that
eventually
the
device
will
be
able
to
reach
out
to
thanos,
receive
and
and
report
at
some
interval.
But
how
would
you
handle
systems
where
that
air
gap
is
permanent?
So,
for
example,
you
have
an
environment
that
will
never
be
allowed
to
connect
to
an
external
network,
and
so
essentially,
the
only
way
to
get
metrics
out
of
that
environment
is
to
have
someone
with
access
to
that
environment.
B
Words,
yeah
I
mean
it
is
not
not
automated,
as
you
said
like
somebody
needs
to
do
it
manually
and
you
need
to
probably
increase
the
retention
of
the
Prometheus
itself.
So
you
probably
want
to
have
like
a
week
and
then
do
it
once
a
week
and
going
into
into
that
device
and
Prometheus
just
writes
these
individual
immutable
blocks
on
disk,
and
you
can
literally
copy
paste
them
into
somewhere
else,
and
you
could
paste
them
into
like,
like
from
from
that
device.
B
D
E
Hey
so
I
might
just
not
be
thinking
about
this
correctly,
but
query
is
basically
like
a
pull
and
then
the
remote
right
is
the
push
portion
right
right.
But
is
there?
Is
there
ever
a
time
where
you're
like
duplicating
data?
Because
of
that
because,
like
maybe
the
query
has
already
stored
and
pushed
object
store
and
then
the
right
ahead
logs
pushed
to
the
object
store,
but
it's
the
same
thing
or
am
I
just
not
thinking
about
it
correctly.
So.
B
The
individual
Prometheus
they
they
might
have
the
same.
They
might
have
the
same
data,
yes,
and
that
that
is
something
that
that
can
occur
or
like
the
receive
components.
They
might
only
differ
by
one
labor,
which
is
like
replica
one,
two
and
three
for
example,
and
so
you
have
duplication
in
there
and
the
famous
query
is
actually
able
to
deduplicate
metrics
based
on
specific
labor
names
that
you
give
it
and
then
filter
those
out
and
do
some
heuristics
on
deduplicating
those
metrics
that
what
you're
asking
for
yeah.
E
B
The
final
store
is
just
basically
cashing
and
making
making
sure
that
we
have
the
G
RPC
interface
of
the
store,
like
an
internal
G,
RPC
interface
available
for
querying
object,
storage,
and
that
is
also
used
by
the
famous
query
ax
to
then
query
whatever
is
in
an
object,
storage
and
that
very
same
interface
is
also
used
to
query.
What's
in
the
receive
components,
for
example,
so
the
query
only
does
the
streets
yeah,
okay,.
F
To
make
sure
I'm
understanding
correctly,
the
edge
nodes
are
the
edge
devices
are
making
the
initial
connection
up
correct.
All
right
are
stateless
in
the
connection.
Is
there
a
way
that
the
centralized
environment
can
determine
whether
or
not
that
the
device
is
being
swapped
out
with
another
device?
If,
for
like
example,
the
Raspberry
Pi
a
go
and
goes
down
and
they
go
find
another
one
and
we
reprovision
another
piece
of
hardware
right,
I,.
B
Mean
if
you
can
make
sure
that,
for
example,
this
cluster
level
would
be
the
same
one
then
you
could
could
just
do
it
like
that
and
just
like
give
it
the
same
cluster
label
again
like
the
same
cluster
name
other
than
that
I.
Don't
I,
don't
really
know
like
it.
I
think
it.
There
is
specific
to
your
domain
in
this
case
and
what
you
want
to
achieve.
I
can't
really
give
an
answer
other
than
it
depends.
Maybe
I
don't
know.
We
will
need
to
talk
a
bit
more
in
depth
about
what
you
are
doing.
G
B
There's
yeah
I
mean
that
is
basically
a
question
of
comparing
m3,
DB
and
Thanos
and
card
headaches
and
all
of
these
components
right
from
from
the
perspective
of
the
individual
devices
and
the
Promethea.
There
there's
not
really
a
difference
because
it
just
uses
the
same
remote
right
mechanism.
Okay,.
A
Okay,
you
just
want
a
few
minutes
starts
for
the
lunch,
I
guess
yeah.
So
there
are
two
bond
things.
First,
please
rate
the
session
on
scat
and
the
other
one.
We
have
like
lunch
load
balancing
happening
and
you
were
supposed
to
go
to
the
West
Terrace
like
over
there
approximately
how
it's
not
too
wet
okay
applause
for
my
tears
again
and
thank
you.