►
From YouTube: A Practical Guide to Prometheus for App Developers [B] - Ilya Dmitrichenko, Weaveworks
Description
A Practical Guide to Prometheus for App Developers [B] - Ilya Dmitrichenko, Weaveworks
Ilya will first briefly outline how Weaveworks run cloud-native apps in production on Kubernetes, and how they use Prometheus for monitoring, as well as some of the open-source tools the team has built to implement continuous delivery.
In the main section Ilya will turn the spotlight on Prometheus and demonstrate step-by-step how simple it is to instrument an app, using a very generic Node.js app as reference.
About Ilya Dmitrichenko
Twitter Tweet Facebook Message Websitehttps://weave.works
Ilya is a Developer Experience Engineer at Weaveworks, focused on making the adoption of microservices easier. Prior to Weaveworks, Ilya worked at Xively, where he personally experienced the shift to a true DevOps culture. He began to shift focus down the stack, becoming one of the early evangelists of and contributors to open source projects in the emerging Docker/container ecosystem.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
my
name
is
Ilya
I
work
in
London
company
called
vivre
works.
She
started
by
RabbitMQ
co-founders.
Perhaps
perhaps
some
of
you
are
familiar
with
rabbitmq
and
we've
been
living
around
since
2014
and
that's
around
the
time
I
started
working
on
cloud
native
tools
had
feed
worked
right,
the
first
release
day
a
network
for
docker.
That
was
really
the
first
network
that
actually
worked
out
of
the
box
for,
like
you,
know,
containers
running
and
multiple
hosts
without
having
to
run
containers
in
the
host
name,
space,
etc.
So
that's
that's
one
of
our
flagship
products.
A
That's
called
the
net.
It's
2.0
now
2.1,
actually
and
more
recently
been
contributing
to
to
Prometheus
project
and
Copernicus.
Aero
tools
such
as
cube,
ATM,
have
a
few
projects
of
my
own
and
the
contribution
that
we
made
towards
the
Prometheus
project
was
was
was
done
by
Julius
waltz.
So
maybe
here
maybe
not
today-
and
this
is
the
the
remote
write
feature
and
also
remote
read
that
allows
us
to
to
store
data
prometheus
data
in
in
the
cloud
service
right.
A
We
run
a
cloud
service
that
provides
a
remote,
read
and
remote
bright
endpoints
for
your
preview.
Server,
it's
pretty
easy
to
unable
in
existing
Prometheus
file,
but
you
could
just
use
our
out-of-the-box
configuration.
So
this
is.
This
is
just
a
bit
of
an
intro
about
myself
and
the
company
I
work
for
I'll.
Show
you
a
little
bit
of
the
product,
just
just
by
just
as
means
of
you
know:
visualizing
the
data
I
have
in
Prometheus
and
I'm
doing
from
ql
examples
and
and
stuff.
A
A
I'm,
an
app
developer,
I
build
apps
right,
that's
right!
Yeah
I
do
micro
services
as
everyone
right
and
as
I
do
those
things.
I
run
them
on
kubernetes
I.
Let
Kaveri
this
random
for
me.
So
I
don't
have
to
worry
about
silly
things,
anymore,
I!
Let
kubernetes
run
my
market
services
for
me
and
that's
that's
that's.
What
most
of
you
do
right?
How
many
of
you
don't
do?
Kubernetes
brilliant
great?
We
had
to
keep
going
after
all.
A
Yeah,
you
can
guess
so.
Observability
is
pretty
essential
and
as
a
developer
in
the
age
of
micro
services,
you
should
embrace
observability
and
and
do
your
best
job
towards
that
and
instrument.
Your
rod
with
Prometheus
metrics
we're
no
longer
have
to
try
and
you
know,
foreign.
The
tool
that
will
like
look
at
the
packet
of
system
calls
and
tell
you
what
the
app
is
doing
right.
A
A
A
Well,
perhaps
a
good
question
to
us
great.
Why
well
I
think
this
is.
This
is
mostly
to
do
with
marker
services
being
a
culture
and
not
to
be
mistaken,
with
architecture
right
culture,
not
an
architecture.
You
build
your
app.
Your
rodded
team
takes
responsibility
to
run
the
app
that
they
ship,
ops
team
run.
Infrastructure
developers
run
their
apps,
so
they
can
actually
put
the
work
in
to
make
it
more
observable.
They
can
put
those
Prometheus
metrics
in
just
just
to
make
it
easier
for
themselves
to
run
the
app
to
have
decent
monitoring
in
place.
A
Each
team
has
taken
on
cold
responsibility,
I
better,
do
that
right,
because
because
otherwise,
what
they're
gonna
second-guess
what's
going
on?
Is
you
know
it's
them?
We're
running
the
up
now,
I
think
that's
kind
of
one
of
the
the
big
culture
changes
with
brim
eaters
micro
services.
So
it's
a
big
culture.
Change
we're
with
micro
services
and
prometheus,
is
a
unobservable
team
tool
that
is
built
for
micro
services
in
a
day
and
age
of
the
container
orchestration
it
has
native
support
for
kubernetes
and
other
orchestrators.
A
So
you
don't
have
to
configure
your
app
until
it
like
hey
you're
monitoring
thing
is
over
there
and
then
oh
I
got
to
change
that
because
there
is
a
new
monitoring
thing
or
what
I'm
gonna
do.
Oh,
my
client
librarian
sports
one
at
a
time.
So
previous
doesn't
have
that
problem
at
all
and
you
can
run
multiple
versions.
A
A
A
So
here's
an
example
of
sort
of
different
metrics
that
that
a
metric
standpoint
of
my
app
may
output,
what's
its
uptime
seconds,
there's
a
that's.
That's
a
platform,
specific
metric,
HTTP,
request,
duration
seconds
count.
That's
a
that's!
A
middleware
type
metric
your
HTTP
framework
web
app
framework.
Whatever
you
call
that
thing,
the
new
users
count,
that's
the
sort
of
business
metric,
so
you
can
put
all
of
those
in
this
is
a
very
simplified
version.
A
You'd
probably
have
labels
a
lot
of
place
and
you
know
like
we
will
see
an
example
later
on
and
here's
a
more
specific
case
of
HTTP
middle
via
metric,
where
we
have
get
type
request
and
put
type
requests
and
error
codes
associated
with
those
and
yeah.
This
particular
app
has
only
seen
one
of
each
of
those
you
can
find
out
more
about
few
more
advanced
metric
types
such
as
histograms
summaries
and
such
things
on
the
Prometheus
website.
In
the
documentation,
these
things
are
called
metric
types
and
we're
kind
of
like
there
are.
A
There
are
quite
a
few
more
advanced
topics
there,
so
haja
used
the
funny
fuse
to
improve
my
app
as
I
mentioned.
There's
this
prom
cue
language
which
allows
you
to
Kure
the
the
metric
data
you
stored
over
time,
and
you
could
do
some
pretty
advanced
theories.
I'm
still
learning
myself
so
bear
with
me
here.
So
here
is
a
very
simple
one.
You
just
say
a
metric
name
and
you
get
all
the
all
the
things
there
are
for
process
uptime
seconds,
easy,
okay!
A
Well
that
maybe
not
quite
useful!
You
actually
want
to
look
at
a
specific
job,
and
here
Pierre
would
be
looking
at
my
app,
and
this
will
give
me
the
EPA
uptime
of
all
instances
of
my
app
for
the
HTTP
stuff.
You'd,
look
at
the
duration
seconds
count
and
you
can
do
something
more
specific.
Where
you'd
look
at
my
apt
method,
get'
puff
your
eye
flash
right
and
you
can
sum
those
up
sweetu
sort
of
aggregate
over
all
of
the
instances
of
that.
A
My
app
thing
and
you
could
do
other
things
such
as
rates
and
we'll
look
at
some
of
this
momentarily
in
the
tutorial
part
and
there's
another
nice
metric
to
be
aware
of
as
the
app
metric.
This
is
the
metric
that
for
me
to
assess
internally
when
it
finds
a
given
job,
so
it
looks
in
Copernicus
and
it
finds
a
service
and
it
goes.
A
So
you
could
look
at
whether
we're
the
home
of
you,
so
that
essentially
indicates
the
health
of
your
up
if
the
app
is
instrumented.
So
if
you,
if
you
look
at
my
app
like
this,
you
tell
whether
whether
all
of
instances
of
your
app
are
healthy
and
able
to
report
prometheus
metrics
at
the
very
least,
and
you
can
do
a
sum
over
those.
A
A
A
So
it's
the
end,
if
you
have
time
left,
but
we
can
see
that
that
beef
cloud
agents
are
connected
to
this
cluster,
so
we're
good
to
go
and
let's
check
out
the
first
version
of
the
app
well.
Let
me
tell
you
about
this
up.
First
right,
so
we're
launching
a
pretty
awesome,
API
I'm
sure
how
can
use
readers
will
be
all
over
it
as
soon
as
we
launch
it,
it's
called
string
Lee.
It's
pretty
amazing,
it
does
one
thing
and
does
it
well
to
reverse
the
strings,
very
useful,
isn't
it
so?
A
A
Can
show
you
a
little
brief
cloud
explore
perhaps
in
the
meantime,
so
what
can
I
do
here?
Oh
this
is
a
previous
version
of
the
thing
running
right
now,
so
that's
kind
of
what's
so
interesting,
I
guess
I'm
gonna
narrow
down
into
a
default
namespace
there's
only
one
thing
that
I
expect
to
leave
there
to
see.
It
is
expected
to
be
there
right
now
then
og
is
up.
Okay,
well
conference
Wi-Fi!
A
A
It's
working:
okay,
it's
Prometheus,
I,
don't
work!
That's
good!
Okay,
great!
So
I'm
we
were
gonna
say
you
know:
q-quit,
we'll
get
serviced,
no
cheers
demo
get
its
external
IP!
Yeah!
That's
been
assigned,
that's
good!
We
can.
We
can
save
the
external
IP
as
a
variable,
so
we
can
use
that
later
so
yeah,
let's,
let's
check
out
string
Glee
app
right.
Can
everybody
see
the
console
yeah?
It's
pretty
good?
Okay,
I!
Couldn't?
Oh,
oh!
No!
That's
that's
pretty
big,
but
maybe
one
bump,
okay,
that'd
be
okay,
I
guess
so!
A
Yeah
string,
Glee
app
is
like
this
tells
me
what
I
can
do.
Is
this?
And
yes,
you
traverse
a
string.
That
does
one
thing:
it
does
it?
Well,
that's
great!
We'll
have
the
UNIX
philosophy
right,
so
yeah
I've
done
that
already
so
I'm
gonna
run
Apache
bench
as
you
do
really
good
a
like.
You
find
it
on
hacking.
You
so
you're
gonna
have
fun
whether
these
guys
can
actually
stand.
A
This
up
still
stands
after
after
your
like
Apache
bench,
for
a
little
while
run
it
again
there.
It's
still
running
well
but
like
back
back
back
back
back
at
the
start
up
at
the
the
coffee
shop
we
thinking
well,
we're
gonna
have
to.
You
know,
make
sure
that
we
can
withstand
the
the
hockey
stick:
growth
and
we're
gonna.
A
We're
gonna
go
and
do
some
more
benchmarking,
and
we
we
figured
as
I
like
to
keep
things
very
simple
and
minimal.
We
figured
we're
gonna
get
a
right.
Little
load
test,
think
well,
we're
gonna
use
Apache
bench
and
we're
gonna,
we're
gonna.
Add
this
hits
endpoint,
which
is
nothing
really
clever
at
all.
So
we
essentially
just
gonna,
have
a
little
counter
here
for
anybody
who
accesses
the
API
and
we're
gonna
render
that
/kids
right,
which
is
gonna,
render
that
number.
So
if
I
go
run
draft
up
again.
A
A
A
That
you
know
reads
that
slash
hits
and
stores
out
in
the
CSV
file
that
I
could
potentially
plot
later.
That's
this!
That's
pretty
awesome.
Isn't
it
very,
very
simple
load
testing
framework
as
I'd
call
it
and
yeah
framework
without
the
painting?
It's
just
a
frame.
It's
yeah,
we're
gonna,
we're
gonna
run
that,
but
first
we're
gonna
run
the
Apache
avenge.
A
A
Hello,
cube
Khan
and
well.
Let
me
just
double
check
with
it.
This
is
this
even
board
so
that
I
forgot
yeah,
okay,
Brennan,
that's
an
interesting
word:
yeah,
okay
and
I'm.
Just
gonna
double
check
whether
he'd
send
point
is
working.
Oh
yeah,
cool
right
I
saw
two
hits
on
the
API,
so
we're
gonna,
we
gonna
run
a
be
300
C
hundred
and
we
we
run
that
and
we
opened
another
shell
collect
samples
here
and
in
the
very
first
one,
I'm
gonna
run
Apache
bench
bunch
times
and
yeah
well,
yeah
I
think
this.
A
This
is
a
pretty
cool
thing.
I've
got
this
some
pulse
file
now
I've
got
like
all
these
numbers
here
and
as
I'd
be
looking
at
it.
I'll
think
oh
well
hold
on
a
moment,
and
is
this
looking
right?
Yeah,
it
does
look:
okay,
okay,
how
about
I
scale
up
my
a
friend,
I
go
and
scale
this
up
and
you
could
all
scale
deployment.
A
A
This
is
strange,
why
is
it?
Why
is
it
zero
here?
Yeah
so,
like
I,
have
no
idea
right,
the
my
my
my
CSV
father's
and
doesn't
let
me
do
much,
it
doesn't
scale.
Is
it
okay,
so
I
go
on
Google
and
I
go
to
a
conference
and
find
out
like
or
meet
up
a
meet
up.
You
know:
I'm
I
got
to
launch
it
tomorrow,
I
get
to
local
meetup
and
I,
hear
about
good
prometheus.
It's
a
thing.
A
So
I
go
I,
get
home
and
I
I
figure
well,
I
mean
I'm
a
minimalist
that
person
I'm,
gonna,
I'm,
gonna,
try
and
read
the
docs
about
prometheus
and
I
I
write
this.
This
version
was
really
basic,
Prometheus,
metrics
and,
and
what
do
I
do
here?
Well
actually
this
this
is
all
I
kinda
need
to
do.
Prometheus,
metrics
right,
so
I
changed
from
just
hits
to
be
request,
total
and
yeah.
That's
it
cool!
Okay!
Let's
try
that
so.
A
A
A
A
A
Yeah,
so
if
I
look
at
it
as
it-
oh
that's,
that's!
That's
not
just
my
app
I
got
a
look
at
you
know
GS
demo,
and
so
these
are
two
instances
of
my
app
I
can
see
that
each
of
them
has
seen
quite
a
few
requests
already
I'd,
rather
look
at
it
as
a
graph.
Look,
here's
here's
the
hockey,
stick:
growth,
right,
brilliant
and
yeah!
A
Well,
this
is
kind
of
cool,
but
you
know
I
also
want
to
make
sure
that
all
the
users
get
pretty
decent
user
experience
so
whenever
they
want
to
reverse
the
stray
ad
string
they
when
they
will
get.
You
know
fairly
quick
response.
Well,
well,
we'll
reverse
their
strings
and
we'll
do
that
very
fast.
That's
you
know!
That's
what
what
the
guy,
who
wrote
the
million
dollar
method
promised
so
so
yeah,
okay,
well
I
thought
about
it
for
a
while
and
I
thought.
Well,
actually,
you
know
there
are
a
few
reasons.
A
I
could
actually
improve
this
up.
A
little
bit
more
and
and
I
figured
I
could
use
a
Prometheus
client
that
integrates
where
it
was
one
of
no
GS
web
framework
like
app
frameworks
for
REST
API
s
and
my
favorite
one
is
yet
the
mineral
list,
one
the
Resta,
Phi
and
I'm.
Using
this.
This
Prometheus
client
here
and
and
instead
of
doing
all
of
that
stuff
by
by
hand
like
I,
did
before
I
kind
of
have
nice
framework
situation
going
on
here
right,
so
I
got
like
my
routes
defined
pretty.
A
Clearly,
here's
the
API
route
for
Emperor's
I
already
have
a
to
do
line
here,
but
that's
okay,
I
keep
my
million-dollar
algorithm
going
on
here.
Where
is
that
it's
here
still
same
I
think
I
never
understood
what
that
line
does,
but
they
said
it's
it's
good,
so
yeah
and
now
I
don't
have
to
do
the
metrics
manually
anymore
and
if
draft
is
still
running
in
conference,
Wi-Fi
that
looks
good.
I
should
be
able
to.
A
See
the
the
metrics
that
Prometheus
client
library
has
given
me
and
I:
don't
have
this
IP
address
I
need
to
run
this
thing
and
yeah,
so
I
should
be
able
to
to
see
all
the
the
metrics
that
whoa
a
lot
cool.
Alright,
so
I'm
not
sure
what
I'm
gonna
do
with
all
of
these
like
I.
Don't
really
care
too
much
right
now,
I
kind
of
care
about
the
user
experience
and
we
just
get
requests
right.
So
I'm
gonna,
look
at
some
of
these
things
and
I'm
going
to
look
for.
A
Oh
yeah
that
looks
cool
I
can
I
can
actually
see
what
ninety-five
percent
of
people
are
saying
right,
so
I
can
go
and
check
how
well
most
of
my
users
are
served.
So
if
I,
if
I
look
at
this
notebook
that
I
saved
earlier,
we
load
that
and
I
can
see
here
that
oh
hold
on
I
need
to
deploy
a
load
test.
So
I'll
stop
abusing
conference
Wi-Fi
with
Apache
bench
and
I'm
gonna
like
deploy,
deploy
a
little
test
in
the
cluster.