►
From YouTube: “If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus
Description
“If you Don’t Monitor your Infrastructure, you Don’t Own it!” Regain Control Thanks to Prometheus [I] - Etienne Coutaud & Guillaume Lefevre, OCTO Technology
In the French FedEx company we used Prometheus to monitor the infrastructure. It hosts a CQRS Architecture composed with Kafka, Spark, Cassandra, ElasticSearch, and microservices APIs in scala.
This presentation is about using Prometheus in production, you will see why we choosed Prometheus, how we integrated it, configured it and what kind of insights we extracted from the whole infrastructure.
In addition, you will see how Prometheus changed our way of working, how we implemented self-healing based on Prometheus, how we configured systemd to trigger AlertManager API, integration with slack and other cool stuffs.
Some demonstrations will be performed in addition of the presentation.
About Etienne Coutaud
Etienne Coutaud is a French DevOps Engineer working in OCTO Technology for 2 years in Paris. Etienne worked of the implementation on Openshift in production for the health insurance agency. Currently working for the French Fedex he participated on the cloud infrastructure automation, continuous integration and monitoring system. Etienne is enthusiast about all cloud technology which change our way of working.
About Guillaume Lefevre
Guillaume Lefevre is a French DevOps Engineer at OCTO Technology for a year now. He worked in the networking field for various company before moving to DevOps. Currently working for the French Fedex he participated on the cloud infrastructure automation, continuous integration and monitoring system. Guillaume love to automate everything that can simplify workflow and empower people.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
A
Okay,
if
you
few
of
them
who
does
anybody?
Does
anybody
run
it
on
a
picture
with
no
container
just
simple
key
teacher?
No
one?
Okay.
So
today
we
are
going
to
show
you
we
run
primitives
or
we
configure
it
and
we
use
it.
You
know
daily
job
at
a
company
called
what
we
call
a
French
French
FedEx
company,
so
this
is
a
equivalent
of
the
FedEx
here
in
a
in
u.s..
A
So
that's
us!
You
can
find
us
on
Twitter
on
LinkedIn
and
you
can
find
us
even
the
rest
of
this
week's
so
a
little
bit
of
context
about
French
flex
company.
So
the
main
goal
is
to
move
package
from
point
A
to
point
B
and
to
track
every
evens
of
a
package
lifecycle.
So
every
time
package
is
moved
off
touch.
This
is
fast
and
an
event
is
generated
and
injected
into
our
architecture.
A
B
A
A
So
to
sustain
this
goal,
we
use
basic
message:
architecture
message,
messages
based
architecture,
it's
composed
of
almost
ninety
servers.
So
it's
not
container.
It's
all
instances
running
on
a
private
cloud
under
new
techniques
and
for
structure.
So
every
time
and
even
generates
a
packet
generates
and
evens.
It
is
going
to
go
to
Kafka
topics.
Then
you
have
spark
swimming
application.
A
Reading
these
topics
in
into
Kafka
computes
the
evens
and
stir
it
into
a
Casanova
databases.
Then,
behind
this
we
have
some
front
and
a
PA
for
our
customers
who
retrieve
the
wizard
in
the
event
and
to
reenact
some
messages
into
the
Kafka
topic.
So
the
all
infrastructure
is
managed
with
ansible
and
at
the
beginning
of
the
project
we
didn't
have
any
monitoring
architecture.
We
were
blind
on
the
first
lecture.
We
could
not
see
if
it
was
working
properly
if
any
of
the
component
was
failing
or
not.
A
So
the
first
thing
we
have
done
is
to
begin
to
set
up
on
our
architecture
of
a
structure
and
to
retrieve
some
metrics.
The
first
layer
that
we
began
to
watch
was
the
system
matrix.
So
we
used
a
primitives,
not
exporter,
that
we
setup
on
every
server
of
our
architecture
to
retrieve
system
metrics
like
a
CPU
run
in
everything.
So
we
do
this
by
ansible
again,
then.
The
second
layer
we
we
watch
is
what
we
call
the
middleware
layer.
A
So
this
is
where
we
are
going
to
monitor
every
of
our
component
life,
Kafka,
spar,
Cassandra
or
zookeeper,
and
we
have
several
different
exporter.
We
are
jmock
sporter
for
Java
Java
based
middleware,
like
Kafka
or
Park.
We
have
specific
exporter
like
elasticsearch
exporter
or
h0z
exporter,
and
of
course
we
have
some
homemade
exporters
that
we
will
ourself
and
the
last
layer
that
we
end
up.
Monitoring
was
the
application
layer,
so
that
was
a
spark
swimming
application
and
the
IPA
that
was
developed
in
Scala.
A
So
this
is
where
the
developer
are
running
and
publishing
their
own
metrics
for
giving
us
business
data
like
the
architecture.
The
simple
architecture
for
monitoring
that
we
set
up
is
that
we
only
have
one
primitive
server,
which
is
plugged
to
an
alert
manager
and
Agrafena
for
dashboarding.
Then,
beyond
this,
we
have
all
of
our
communication
tools
and
devops
tools
like
stack
for
receiving
at
manage
on
notification,
mail
and
some
jenkins
to
try
group
remotely
jobs.
A
B
As
DM
said,
the
first
step
when
we
want
to
start
to
build
a
monitoring
infrastructure
is
to
have
some
basic
metrics
about
your
system,
some
metrics
relative
to
your
CPU
Ram,
for
example,
to
do
that.
Primitives
provide
different
exact
exporter
and
the
first
exposure
is
the
node
exporter.
This
is
a
binary
written
in
go
and
just
simple
at
that.
You
are
running
it
into
your
server
and
you
can
have
some
basic
metrics
about
it.
B
We
can
have
RAM,
CPU
or
file
system,
so
the
exporter
you
can
have
the
ability
to
extend
the
component
of
so
your
exporter
by
using
correct
of
flags
in
our
system
were
based
on
sent
OS.
So
everything
is
wrapped
using
system
D.
With
that
flag,
collector
system
D,
we
enabled
to
squat
metrics
relative
to
system
D.
For
example,
we
are
able
to
have
the
states
of
the
of
different
units
is
very
useful
to
have
monitoring
on
system
D,
so
the
second
layer.
What
we
want
to
monitor
is
the
middleware
layer
to
do
that.
B
You
have
two
different
ways.
The
first
way
is
to
use
specific
exporter.
We
have
Alexis
Gorge
exporter,
H,
a
poxy
exporter
of
zookeeper.
This
exporter
are
scrapping
metrics
and
then
translate
metrics
into
primitive
metrics,
and
you
have
the
second
way
and
when
qu
you
have,
when
you
have
some
Java
component,
you
need
to
use
a
Java
agent.
B
This
agent
is
provide
by
primitives,
you
stick
it
to
your
component
and
when
it
runs
it
expose
your
metrics
relative
to
your
component
and
then
primitives
will
scrub
your
metrics
and
you
will
have
metrics
from
your
Java
middleware
into
your
monitoring
system.
So
now
we
are
monitoring
systems,
while
monitoring
meter
where,
as
the
last
layer
is
monitoring
your
application.
B
So
it's
your
dev
job
primitives
provide
different
different
client
library,
so
you
can
find
a
brewery
in
Ruby
in
go
or
a
pint
in
Titan
and
the
vast
library
give
to
your
developers
the
ability
to
have
own
metrics
to
build
their
own
business
that
are
directly
from
the
application.
In
our
case,
our
developer
are
building
application
on
fact
streaming
soon.
They
are
using
scalar
client
library
and
in
using
that
client
library.
They.
B
In
our
stack,
we
are
using
the
highlife
manager
and
the
hearth
manager
role
is
to
fire
event
relative
to
primitive
root.
So
when
we,
when
primitive
food
is
fire,
the
heart
manager
will
match
with
shallot
using
labels,
and
you
can
send
the
heart
to
different
receiver.
You
have
receiver
for
slide,
for
example,
you
Jennsen
email
and
you
can
send
to
christen
web
to
using
that
example.
We
have
sent
you
are,
we
are
triggered
Jenkins
jobs
remotely
and
it's
the
changing
job
have,
for
example,
some
constable
run
someone
Sybil
playbook
or
chess
PlayBook
you.
B
So
no
we
have
metrics
among
about
the
third
layer
on
our
infrastructure.
We
have
thousand
of
metrics
is
tough
to
using
it's
tough,
to
build
some
dashboard.
To
do
that,
you
will
have
to
query
your
primitives
and
primitives
provide
different
function
to
travel
your
system.
So
we
love
particularly
two
functions.
The
first
one
is
top
K
top
J
is
very
useful
to
extract
some
data
when
you
have
a
lot
of
Fame
point.
The
second.
The
second
query
is
pretty
cleaner,
Freddie
cleaner.
B
A
Everything
was,
it
was
a
lot
of
theory.
Obviously
we
don't
have
access
to
the
production
of
a
women
from
here,
so
we
prepared
digital
demonstration.
You
can
see
the
schema
here,
so
we
have
a
little
bit
on
application,
which
is
reading
an
injection
into
a
single
calf.
Cannot
then,
beyond
this,
we
have
all
of
monitoring
a
pass
to
true,
so
we
have
one
formative,
server,
monograph,
Anna,
again
dashboarding,
so
let
manager
that
manager
is
blood
to
a
Jenkins
into
slack
system
and
on
this
architecture
we
have
several
endpoints
some
to
simulate
system
matrix.
A
B
The
first
step,
when
you
define
when
you
set
up
your
primitives,
is
to
define
endpoint.
So
you
have
different
way
to
define
your
hand
point,
and
on
that
demo
we
have
different.
We
have
defined
our
endpoint
using
static
configuration
file,
so
you
can
size
the
configuration
we
have
defined
different,
an
endpoint,
so
we
have
metrics
from
the
different
layer.
We
have
metrics
from
a
business
application
of
Python
application.
We
have
metrics
from
the
middleware
capture
and
we
have
metrics
for
node
exporter
know
we
have
different
endpoint.
We.
A
B
B
So
don't
judge
me
I'm,
not
a
newbie
in
Python,
so
so
I
used
a
client,
a
Python,
client
library
and
then
I
define
a
matrix
like
that.
So
I'm,
how
about
you
have
my
matrix,
I'm
increment,
my
control,
then
the
library
will
expose
matrix
on
an
HTTP
server.
So
in
my
in
my
in
my
primitive
system,
when
I
look
up
for
the
matrix,
I
can
find
my
matrix
directly
from
my
patent
application.
B
This
patent
application
is
injected
data
in
reading
me
in
reading
data
into
Kafka,
so
we
are
able
to
see
the
company,
the
behavior
of
the
Kefka
when
it
is
hit
by
my
application
and
the
application
every
45
second
raised
in
error.
This
error
is
catch
by
my
primitives,
its
raison
elect,
so
we
can
find
in
the
primitives.
B
So
every
45
second,
the
heart,
an
area
of
this
trigger
it's
sent
to
the
hideout
manager,
the
heart
and
then
the
heart
manager
will
will
trigger
remotely
a
Jenkin
jobs,
and
this
jenkin
job
simply
will
restart
the
application
and
send
an
alert
into
our
slack.
But
yes,
so
we
can
see
it's
like
is
the
the
application
is
running
and
it's
triggered
some
some
higher.
A
So
you
can
see
that
it's
really
simple
to
have
advanced
teacher.
So
right
now
we
have
a
full
stack
monitoring
on
our
architecture,
so
we
are
monitoring
the
three
layer,
application
system
and
middleware.
We
have
some
alerting
and
health
eating,
but
it's
so
much
more
than
just
watching
that
ball
and
having
metrics
it.
Otherwise,
to
do
some
other
thing
you
know
daily
by
these
jobs
and
to
change
our
way
of
working.
A
With
some
of
the
queries,
we
show
you,
we
can
credit
failure
and
correct
it
before
the
rootage
is
present,
so
it
means
less
business
loss.
Of
course,
this
means
faster
troubleshooting
and
what
could
spin
pointing
because
we
can
use
a
dashboard
and
the
metrics
we
have
and
not
login
to
the
server
and
do
some
manual
steps
so
with
elapsed,
troubleshoot
faster.
The
fourth
point
is
to
trillion
improve
configuration
of
indirect
parameter,
because
we
can
see
all
of
Amida.
A
We
are
working
properly
and
alive
into
our
dashboard,
so
we
can
see
which
one
is
overloaded
or
which
one
is
less
used.
So
we
can
change
the
configuration
to
improve
the
overall
advantage,
famines
and,
of
course,
because
we
can
see
all
of
the
logic
flow
of
often
even's
into
our
architecture
we
can
see
which
component
is
a
bottleneck
or
not.
A
B
Other
notes,
obviously
I
use
my
and
I
use
my
dad
brought
every
day
for
everything,
but
when
we
built
the
system,
we
decided
to
happen
the
dashboard
for
everything
for
every
people
involved
in
the
project.
It
was
for
the
developer
for
the
entrance
with
business
team
to
to
give
to
you
developers
the
hability
to
watch
your
dashboard.
It
will
bring
some
benefits.
The
first
benefit
is
for
your
developer
to
improve
the
testing
of
our
application,
I'm
as
a
developer.
I
can
see
the
impact
of
my
released
directly
on
the
system
by
looking
the
dashboard.
B
B
If
I
see
the
dashboard
is
that,
if
I
see
the
status
of
the
infrastructure,
I
am
more
confident
and
I
have
more
visibility
into
the
platform
having
the
relevant
metrics
available
inside
from
the
platform,
it's
a
good
way
to
involve
the
top
management
to
to
involve
every
people
to
give
a
comprehensive
view
of
your
system
and
then
using
primitives
you
develop,
you
are
able
to
inject
derivatives,
are
how
matrix
and
they
can
build
their
holiday
drop.
So
this
is
a
good
way
to
give
to
your
developer,
more
responsibility.
A
It's
time
consuming
to
continuously
improving
your
metric,
your
triger
and
your
audit
to
match
it
to
useful
things.
So
maybe
some
time
we
you'll
be
trouble
shot,
something
you
will
not
have
the
matrix
in
your
dashboard
and
not
the
right,
another
right,
dashboard
and
maybe
another
it
will
not
have
been
configured.
So
this
is
a
walk
you
need
to
do
continuously
to
otherwise
matrix
like
white
racial,
a
new
prevent,
although
what
we
call
the
I
left
fatigue.
A
So
maybe
at
the
beginning,
you
will
want
to
have
an
alert
on
everything,
so
you
can
saw
some
great
things
blinking
everywhere,
but
at
the
time
you
will
not
watching
it
anymore
because
it
will
be
useless
or
not
meaningful.
So
you
have
to
continuously
improve
and
change
what
you
have
already
the
second
tips,
it's
an
obvious
one,
okay,
but
you
all
of
your
applications.
Several
and
many
paths
and
monitoring
platform
need
to
be
in
sync
in
terms
of
NTP,
also
where
you
will
have
some
problems
in
terms
of
time
stamp
on
your
matrix.
A
But
the
second
point
is
more
important
is
this
is
when
you
are
watching
your
graph
and
a
server
the
browser
you
use
and
the
graph
and
a
server
the
clock
must
not
be
too
far
apart.
Other
way,
if
you
want
to
query
a
specific
time
line
and
the
clocks
are
not
in
sync,
maybe
you
will
do
that.
Do
data
point
available
because
so
requested
time
line
will
not
be
the
same
on
your
border
in
on
your
retina
server.
So
we
need
to
be
careful
with
this
and
you
need
to
manual
NTP.
So.
A
So
you
just
have
everything
is
done,
but
okay,
so
just
the
clock
far
apart.
The
third
thing
is,
is
what
we
have
called
the
skyping
and
have
a
trade
off.
So
again,
this
is
a
true
story,
but
on
the
left
you
can
see
that
at
the
beginning
we
set
up
skyping
interval
of
five
seconds,
so
the
swapping
interval
is
yeah.
B
A
Five
seconds,
so
the
squatting
interval
is
often
our
primitive
use
with
queries
our
endpoint,
so
our
250
endpoint.
So
you
can
see
that
the
consumption
in
terms
of
CPU
is
at
almost
75,000
and
sometimes
peaking
at
100%.
So
this
is
because
we
use
primitives
1.5,
not
permit
is
2.0
yet,
but
then
what
we
choose
to
do
is
to
set
I
or
scrapping
it
have
a
and
who
put
it
to
a
15
seconds,
and
you
can
see
that
it
reduce
almost
buy
our
of
satirical
summation.
So
this
is
why
we
call
it
ascribing
interval
trade-off.
A
If
you
have
a
loss,
popping
interval
of
5
seconds,
we
will
have
almost
real-time
data,
because
the
lag
effect
will
only
be
of
5
seconds
between
so
data
on
your
server
and
the
data
on
your
primitives,
but
it
will
be
CPU
intensive
if
you
have
like
us
only
one
for
me
to
server.
So
we
need
to
have
a
robust
per
minute,
primitive
server
or
a
bigger
architecture
like
using
aggregation,
HD,
poxy
and
everything
under
control.
We,
if
you
use
the
I
of
scrapping
interval.
Of
course
you
will
have
what
we
call
the
matrix
lag
effect.
A
A
B
So
the
next
step
in
the
run
map
of
our
project
is
obviously
it's
to
upgrade
to
primitives
to
that
how
to
improve
the
performance
of
the
system
monitoring.
The
second
one
is:
we
don't
have
availability
for
Allen,
for
instance,
we
have
already
run
several
parameters,
so
we
want
to
each
province
feature
and
especially
high
availability
has
Guillaume
said
every
day
we
are
continuously
improving,
triggering
improving
traceur
dejara.
So
we
will
keep
that
work
to
improve
the
investigation.
B
We
will
starting
to
set
up
graph
an
adult,
Bob
Sweeney,
so
it
will
a
better
way
to
troubleshoot,
faster
and
with
more
efficiently.
We
have
a
beginning
of
safe
feeling.
We
are
with
Howard
manager
and
the
Jenkins,
so
we
will
keep
keep
on
to
improve
it,
because
we
are
lazy
people
and
we
said
that
our
infrastructure
is
running
on
Nutanix
infrastructure,
and
yet
we
do
not
monitor
that
layer.
B
A
B
So
the
takeaway
for
our
presentation,
it's
the
first
one-
is
using
primitives.
You
are
able
to
have
a
lot
of
metrics.
You
are
about
to
have
a
lot
of
data,
so
do
not
hesitate
to
instrument
your
war
infrastructure
and
you
will
have
a
full
stack
monitoring.
The
second
way
is
this
is
very
easy
to
to
start
in
an
exporter
to
studied
primitives,
and
you
have
beginning
of
data,
so
a
very
fast
to
dip,
but
it's
very
long
to
master
continuously.
B
B
Business
business
team
and
many
top
management
will
be
more
involved
into
your
project
and
it
will
improve
quality
of
the
code
of
the
developer,
because
the
developer
will
be
more
confident
into
your
platform
and
they
can
see
directly
the
impact
of
the
realism
of
the
code
so
having
a
full
stack
monitoring.
It
will
change
the
way
of
working
for
the
hot
stream
and
for
the
depth
into
primitives
as
a
lot
of
exporter
for
each
component.
B
So
obviously
you
will
find
your
exporter
that
you
need,
and
if
you
don't
find
the
exporter
that
you
need,
you
can
easily
code
your
home
because
primitives
forma
is
just
the
key
and
value
and
you
have
a
matrix.
So
if
you,
if
you
don't
from
your
exporter,
do
not
hesitate
to
write
your
home
and
to
open-source
is
for
the
whole
community.
So.
A
A
Know
a
bit
on
application
raise
an
error,
so
the
system
status
change
to
fail
and
we've
the
node
exporter.
We
can
watch
the
service
status
so
many
it
failed
and
Ally
artist.
Why
go
into
the
rat
manager
and
this
alia
triger,
or
a
stack
notification
and
remotely
jobs
into
a
Jenkins
as
in
Jenkins,
was
just
an
instable
playbook
to
start
it
again.
Does
that
just
basic.
B
A
B
In
the
demo,
it
justice
system
system
lira
start
so,
but
in
the
real
life
we
are
running
on
Sybil
playbook
for
different,
for
example,
where
the
job,
but
that
shake.
If
if
the
file
system
is
full
or
not
so
we
have,
we
have
playbook
that
ran
on
on
the
server
on
the
target
service
to
clean
to
clean
the
log,
for
example,.