►
From YouTube: Overcoming (organizational) scalability issues in your Prometheus ecosystem - Jürgen Etzlstorfer
Description
Prometheus is considered a foundational building block when running applications on Kubernetes and has become the de-facto open-source standard for visibility and monitoring in Kubernetes environments.
Your first starting points when operating Prometheus are most probably configuring scraping to pull your metrics from your services, building dashboards on top of your data with Grafana, or defining alerts for important metrics breaching thresholds in your production environment. in your production environment.
As soon as you are comfortable with Prometheus as your weapon of choice, your next challenges will be scaling and managing Prometheus for your whole fleet of applications and environments. As the journey “From Zero to Prometheus Hero” is not trivial you will find obstacles on the way. In this presentation, we are highlighting the most common challenges we have seen and provide guidance on how to overcome them. Finally, we are discussing a solution to get you there more quickly to build automated, future-proof observability with Prometheus showing Keptn as one possible implementation.
A
A
A
I
think
this
is
the
only
time
I
will
mention
dynatrace,
so
the
rest
of
the
talk
I
will
be
either
mentioning
prometheus
or
the
captain
open
source
project
will
which
I'm
using
at
the
end
of
the
talk
or
like
at
the
the
start,
starting
from
the
middle
of
the
talk
to
the
end
of
the
talk
to
show
you
one
option
or
one
possibility
to
overcome
this.
These
challenges
that
I
will
sketch
in
the
beginning,
but
I
will
also
give
options
where
you
how
to
build
it
yourselves.
A
A
A
So
let
us
get
started
and
when
we
talked
to
organizations
that
are
using
prometheus,
we
we
asked
them.
What
what
do
you
want?
Why
are
you
using
prometers?
What
is
like
the
main
benefit
and,
of
course
it's
the
whole
observability
part,
but
it's
also
that
they
are
saying
I
want
to
have
like
automated
dashboards.
I
want
to
for
each
application
that
I'm
on
boarding.
I
need
a
couple
of
predefined
or
a
couple
of
standard
dashboards.
A
I
need
my
alerts
in
my
alert
manager
setup
based
on
some
rules
that
we
define
inside
of
our
company
or
we
define
for
some
of
our
products,
some
of
our
services
and,
of
course
I
want
to
have
like
everything
automated.
I
want
to
have
automated
monitoring
of
all
those
applications,
so
this
is
what
folks
usually
want,
but
of
course
this
is
not
where
they
get
in
the
very
beginning.
A
So
how
can
we
get
there
and
for
today
I've
prepared
a
little
bit
of
the
agenda,
so
I
want
to
start
with
basic
kubernetes
building
blocks
very
simplified,
but
they
will
guide
us
through
those
challenges
and
I've
prepared
three
challenges
and
options
how
to
overcome
them.
And
then
I
will
talk
about
a
little
bit
about
this
open
source
project
that
I'm
involved
in
called
captain
and
how
to
overcome
these
challenges.
Also
with
captain
I've
also
written
a
blog
on
this.
A
So
if
you
want
to
to
to
read
this
blog,
I
put
in
here
the
link.
It's
it's
a
so
called
friends
link
on
medium,
so
you're
not
wasting
your
three
stories
a
month,
I'm
not
even
sure
about
this
whole
thing
on
medium,
but
some
of
those
stories
they
are
hidden
behind
the
paywall.
But
if
you
go
with
this
friends
link
it's
it
will
put
it
in
front
of
the
table,
so
it
will
be
for
free.
A
So
what
do
we
have?
Usually
in
kubernetes
we
have
our
application
clusters,
so
we
have
one
or
more
clusters.
We
have
our
pods
running
in
our
clusters.
These
are
exposing
metrics
on
this
endpoint
and
prometers
is
basically
scraping.
All
these
metrics
fetching
all
these
metrics,
storing
it
and
then
providing
it
to
users
directly
by
prometheus
or
most
often,
by
grafana
dashboards.
A
So
you
can
take
a
look
and
see
what's
really
going
on
and
you
have
scaling
inside
all
these
applications
and
of
course,
you
also
might
have
scaling
on
the
prometers
clusters,
both
premiere
size
and
profana.
They
need
configuration.
Obviously
it's
not
that
everything
is
predefined
out
of
the
box,
so
you
have
to
configure
this
and
this
configuration
will
be
kind
of
how
to
scale
configuration
for
these
systems
will
be
the
core
of
this
presentation.
It's
not
so
much
that
how
to
scale
prometheus
clusters.
There
are
other
projects
out
there.
A
A
To
scale
so
I
won't
be
touching
these.
These
are,
I
think,
great
projects
out
there,
they've
also
been
presented
at
kubecon
and
at
various
other
options,
and
you
can
find
them
online,
so
it's
stainless
victoria,
metrics
and
maybe
others,
but
I
won't
touch
this.
This
part.
I
will
be
more
touching
the
part
of
really
having
scaling
the
the
whole
configuration
and
to
enable
more
teams
to
use
prometheus
and
to
use
the
whole
prometers
ecosystem
with
the
alert
manager
in
grafana
and
other
tools
that
you
might
have
so.
A
First
of
all,
one
problem
that
we
have
seen
is
that
all
those
configurations
that
you
need
to
write
for
promethos
and
alert
manager
and
grafana,
they
you
write
them.
You
apply
them,
but
still
you
have
to
make
sure
that
you
have
you
version
them
and
to
make
sure
that
the
version
that
it's
like
in
your
git
repository
is
actually
the
version
that
it's
applied
in
prometheus.
Sometimes
you
have
like
it's.
A
A
They
need
to
have
deep
knowledge
of
all
these
configurations.
So
writing
all
those
configurations,
possibly
also
in
different
languages
or
like
in
yaml
or
json
or
jsonnet
or
whatever
you're.
Using
take
some
time
and
be
aware,
it's
not
only
one
configuration
that
you
have
to
write,
but
it's
really
it's
a
couple
of
configurations
and
we've
seen
large
systems
where
they
are
generating
parts
of
these
configurations.
A
A
I
think
it's
good
to
use
them
because
they
really
take
away
this
burden
of
having
like
making
mistakes
in
all
these
configurations
and
finding
like
in
the
ammo
that
you
have,
the
intendation
is
not
correct
and
these
kind
of
things
so
code
generators
really
provide
the
power
that
you
just
feed
them
with
the
input
and
they
are
generating
the
configuration
file
that
you
need.
I've
listed
a
couple
of
them
at
the
end
of
the
presentation.
A
A
A
These
code
generators
that
I
found
and
we
that
I've
discussed
also
with
with
parts
of
the
community
they
most
often
solve
one
challenge,
or
maybe
they
are
one
code
generator
for
graphana
dashboards,
but
they
are
not
the
same.
It's
not
the
same
code,
generators
for
prometheus
configuration
or
the
other
way
around.
A
So
what
you
really
need
is
some
way
to
base
everything
on
one
concept:
to
have
like
one
kind
of
input,
file
and
feeds
all
these
code,
generators
with
the
same
kind
of
input,
and
what
we
have
seen
is
the
concepts
from
the
site.
Reliability,
engineering
community
are
really
a
good
fit
for
these
kind
of
things.
So
it's
service
level
indicator
service
level,
objectives,
error,
budget
service
level
agreements.
A
So
you're
raising
the
abstraction
level,
basically
that
you
define
everything
in
based
on
concepts
that
you
can
it's
more
easy
to
to
discuss,
based
on
slos
than
to
discuss,
based
on
pronquill
that
you
need
to
write
because
for
pronquill
you
need
a
specialist
and
for
sre
concepts,
they're
more
easy
to
understand
and
based
on
those.
You
can
then
generate
the
prom
ql
and
you
can
validate
if
it's
correct
and
then
you,
then
you
put
it
in
the
git
repository.
A
A
So
with
this,
it
was
kind
of
side,
side,
product
and
with
captain
you
can
also
automate
parts
of
your
operations.
I
think
I've
written
it
also
in
the
in
another
slide
what
what
captain
is.
But
let
me
show
you
in
this
picture
that
I've
discussed
before
where
captain
comes
into
place.
So
it's
basically
captain
takes
is
an
input,
slo
files,
service
level,
objectives
that
describe
the
service,
the
goals
of
the
service
in
terms
of
quality
criteria
like
a
response
time
or
an
error
rate
or
throughput
or
memory
consumption,
these
kind
of
things.
A
So
this
will
be
the
input
file
for
captain
and
captain.
That
itself
has
its
own
internal
components
that
will
generate
all
the
configuration
for
the
attached
monitoring
tool.
So
in
the
beginning
I
said
I
won't
mention
the
company
I'm
working
for,
but
it's
the
monitoring
tools
can
be
either,
for
example,
promethor,
so
we
are
generating
prometheus
configuration
or
we
can
also
generate
configuration
for
dynatrace
or
for
other
monitoring
tools,
and
captain
also
has
its
internal
git
repository.
A
So
you
also
get
this
synchronization
or
this
version
management
of
all
this
configuration
files
and
then
the
application
of
these
configuration
files
to
the
different
components
in
your
permitted
ecosystem.
So
I've
talked
a
little
bit
now.
Let
me
just
show
you
a
like
a
five-minute
demo,
what
I
mean
with
configuring,
these
kind
of
things.
What
I
do
have
is.
A
Right
microphone,
so
what
I
do
have
is
I
set
up
a
grafana
instance:
it's
fairly
new,
so
there
shouldn't
be
anything.
There
are
no
dashboards.
There
are
no
data
sources,
so
it's
it's!
It's
kind
of
a
fairly
new
rafana
installation
same
is
true
for
for
prometheus
and
I've.
My
captain
installation
already
up
and
running
on
this
kubernetes
cluster.
So
it's
like.
If
I
take
a
look
at
all
the
parts
that
are
running
already
in
my
captain
installation.
A
I
would
see
a
couple
of
services
that
and
like
all
these
services
they
make
up
the
whole
captain
control
plane.
I
will
talk
about
this
in
a
minute,
but
what
would
see?
I
have
one
service
for
grafana
and
I
want
I
have
a
couple
of
services
for
prometheus
and
they
will
be
responsible
for
taking
the
input
and
then
doing
something
with
all
this
input
and
configuring,
the
promethors
and
and
profile.
So
we've
seen
everything
is
up
and
running,
and
let
me
now
do
the
configuration
of
prometheus.
A
So
captain
is
event
based
and
in
this
sense,
with
the
cli
I'm
sending
one
event
to
my
captain
control
plane.
The
event
will
hold
information
about
that.
I
want
to
configure
my
monitoring
tool.
Promethor
is
for
one
specific
project,
basically
the
application
and
for
one
specific
microservice,
it's
my
shopping,
cart
called
carts,
and
I
would
just
execute
this
command.
It
will
take.
It
will
res
the
captain.
Cluster
will
receive
a
cloud
event
and
we'll
process
this
cloud
event
like
both
services
will
process
the
cloud
event.
A
First
prometheus
and
second
graphana
will
process
this
cloud
event.
It
can
be
more
services
that
can
subscribe
to
this
kind
of
events,
but
in
my
use
case
it's
only
these
two
services
that
are
interested
in
all
this
configuration
of
promethouse
information
and
what
I
like,
what
was
happening
in
the
background
is
that
first
promethos
was
configured
like
all
the
screenshots
for
my
project.
A
Sock
shop
and
my
service
cards
have
been
created.
Alerting
rules
have
been
created
for
based
on
my
slo
file.
I
have
one
slo
file
edit
here
it
says
no
slo
file
found
for
stage
production.
I
only
have
one
slo
file
for
my
stage.
I
think
it's
called
pre-production,
but
we'll
see
it
in
a
minute.
So
these
files
they
have
been
taken
into
consideration
for
then
creating
all
those
rules,
the
alerting
rules,
the
scrape
jobs
and
also
microfinance
dashboards.
A
So
if
I
go
back
to
my
profile,
I
should
now
see
at
least
the
data
source
we
have
to
reload
yeah.
So
I
see
now
my
my
data
source,
it's
on
the
same
cluster,
so
it
was
configured
to
be
on
the
same
cluster
as
as
micro
as
as
the
service
where
it's
running,
and
I
should
be
also
able
to
see
some
dashboards,
probably.
A
Here
we
go,
we
have
a
captain
dashboard
and
we
can
already
see
some
data
coming
in
so,
and
I
can
see
I
have
data
coming
into
my
production,
environment,
staging
and
depth
environment.
So
it's
very
simplified,
but
it
will
it.
It
should
give
you
the
idea
that
you
can
com
for
each
new
application
that
you're
on
board.
A
So
let
me
show
you
how
we
envisioned
this
and
how
and
how
we
how
we've
done
this
so
asset
captain
is
an
event-based
control
plane
for
continuous
delivery
and
automated
operations.
It's
a
very
long
sentence
and
it
has
a
couple
of
passwords.
We
know,
but
still
we
like
the
sentence
because
it
kind
of
really
says
what
it
is:
it's
a
control
plane.
So
it
will
connect
different
tools
and
it
will
kind
of
orchestrate
different
tools
for
continuous
delivery
and
automated
operations.
A
So
captain
itself
can
do
continuous
delivery,
but
we
can
also
control
others
like
argo,
cd,
argo,
rollout
or
jenkins,
or
other
tools
that
or
azure
devops,
so
other
tools
that
you
might
that
you
might
already
be
using.
You
can
put
captain
on
top
of
it
so
that
captain
can
control
like
the
deployment
of
the
of
the
service,
but
captain
again
trigger
afterwards
the
tests.
So
you
don't
have
to
trigger
the
tests
manually
after
you
do
the
deployment
after
the
tests
captain
can
trigger
the
evaluation.
A
So
you
don't
have
to
do
any
evaluation
manually
after
the
evaluation
captain
can
decide
if
it's
a
thumbs
up
or
comes
down
and
promote
it
to
the
next
stage.
So
these
kind
of
things
captain
can
control
as
a
control,
plane
and
attached
to
the
control.
Plane
are
all
these
services
that
you
have
seen
in
my
console
before.
A
So,
for
example,
I
have
services
for
configuring,
promethos
and
scrafana,
but
I
have
also
other
services
like
connecting
it
to
a
slack
bot
so
that
you
can
control
captain
with
a
slack
bot
or
you
get
all
the
notifications
of
all
the
failed
or
past
evaluations.
Or
you
get
notifications
if
new
services
get
on-boarded
so
that
you
can
take
a
look
at
the
dashboards
in
grafana
every
time
there
is
a
new
application
onboarded
to
to
create
a
specific
profile.
A
So
captain
lives
in
your
kubernetes
cluster
and
it
will
be
acting
upon
of
like
when,
whenever
captain
is
receiving
events,
it
will
act.
So
it's
not
an
operator.
That
is
also
like
it's
a
prominent
way
to
do
things
in
kubernetes.
To
do
it
as
an
operator,
captain
itself
is
not
an
operator,
but
captain
is
reacting
on
cloud
events,
so
it
can
easily
interact
with
other
tools
because
they
can
just
send
cloud
events
to
captain
also
from
outside.
A
A
First
service
level
indicators
are
slis
and
we
can
basically
think
of
it
as
a
metric.
So
something
like
an
error
rate
of
login
requests
for
the
last
five
minutes,
then
this
metric
would
be
the
sli.
The
slo
is
some
goal.
Some
objectives:
objective,
sorry
that
we
defined
upon
the
sli.
So
we
are
using
the
sli.
A
So
this
would
be
something
we
can
evaluate
and
based
on
this,
we
can
either
even
define
slos,
but
for
captain
itself
we
are
only
using
slis
and
slos,
since
these
are
more
than
technical
parts
and
the
service
level
agreements
are
more
the
business
agreements.
There
is
a
nice
very
short
youtube
video
I've
also
linked
here.
So
I
think
it
will
explain
it
like
roughly
five
minutes
the
whole
concept
behind
it.
A
So
how
does
it
work
based
on
those
files
to
create
the
alerts
and
dashboards?
So,
first
of
all,
you
can
either
define
the
indicators
by
yourself
or
you
can
reuse
the
indicators
that
the
captain
has
already
built
in
so,
for
example,
we
can,
for
example,
the
number
of
database
calls.
That's
not
something
that
is
already
built
into
captain,
but
you
can
just
define
it
and
you
would
map
it
the
the
name
of
this
indicator.
A
You
would
just
define
it
with
the
prom
qr,
so
this
will
be
evaluated
whenever
you
are
asking
for
the
number
of
of
the
database
calls
the
error
rate.
The
response
time
is
already
baked
in,
but
if
you
want
to
overwrite
it,
you
can
do
so
in
this
indicator
file
and
you
can
then
reuse
those
indicators
in
your
objective
file
and
the
objective
file
holds
a
list
of
service
level
indicators
and
defines
one
objective
or
criteria
upon
the
service
level
indicator.
A
A
We
also
have
like
warning
criterias
that
say
if
you're,
if
you're,
not
within
the
past
criteria,
then
also
warning
will
be
evaluated
and
you
could
have
to
score
for
the
whole
sli
and
in
total,
then
captain
can
generate
a
score
for
you
and
you
have
like
a
total
score
for
this
service
level.
Objective
evaluation-
and
this
is
the
basic
part
for
for
the
captain
quality
gate.
A
We
can
use
it
for
the
captain
quality
gate
for
your
continuous
delivery,
but
we
can
also
use
it
to
to
define
your
your
alerts
and
thresholds
in
your
dashboards
in
this
kind
of
thing.
So
it's
we.
We
can
use
it
for
for
multiple
things,
and
this
is
where
I
spoke
in
the
beginning
that
everything
is
based
on
some
common
concepts.
These
are
our
slis
and
slos.
A
So
when
we
are
executing
our
command
to
configure
our
monitoring
tool
like
prometheus,
this
will
be
first
sent
to
the
action
control
plane.
The
control
plane
will
distribute
this
event
to
all
the
services
that
are
listening
for
these
kind
of
events.
As
said,
it's
a
cloud
event,
that's
a
cncf
project
and
we
are
using
the
the
syntax
of
a
cloud
event
here,
also
in
captain,
so
it
will
be,
in
my
case,
it
was
distributed
to
our
farmer
service
and
to
our
promethor
service.
They're.
A
Now
interpreting
this
cloud
event
in
a
way
to
to
generate
some
tool,
specific
interpretation
or
to
to
generate
the
api,
calls
that
are
needed
for
those
tools
and
are
executed.
So
the
graphana
service
was
talking
to
microphone
installation.
My
prometer
service
was
talking
to
my
promoters,
installation
and
they
were
doing
the
configuration
for
both
of
them,
based
on
the
same
input
file
that
we
can
see.
A
I
think
it
was
not
the
same
as
I've
put
here
on
the
slides,
but
it
was
the
same
input
file
that
both
of
them
are
receiving
at
the
same
time
and
they
are
doing
this
generation,
so
we
can
also
use
the
same
files
on
the
left
hand,
side,
it's
again
the
same
file,
but
you
can
also
use
them
for
evaluating
the
service
quality
of
a
microservice.
So
we
can
use
the
captain
cli
or
api
to
start
some
evaluation,
so
captain
will
reach
out
to
the
data
provider
that
is
configured
in
the
previous
command.
A
So
captain
now
knows
how
to
reach
also
how
to
reach
prometheus
or
others
it
will
reach
out
to
those
data
providers
will
fetch
all
those
slis
will,
of
course,
execute
all
this
promptly
to
fetch
all
these
metrics,
we'll
evaluate
them
in
the
time
frame
that
that
is
given
either
that
is
given
here
or
I
said
before.
Captain
can
be
used
as
a
control
plane.
A
So
if
you're
using
captain
as
a
control
plane
to
trigger
also
tests
like
performance
tests
or
load
tests,
captain
knows
when
you
treated
the
test
and
when
the
tests
have
been
finished.
So
you
don't
have
to
to
define
the
time
frame
because
captain
already
knows
so,
it
will
be.
It
will
fetch
the
data
for
the
correct
time
frame,
evaluate
the
data
score.
All
these
slis
will
give
you
a
score
between
one
and
zero
for
each
sli
and
then
come
up
with
the
total
score,
like
90
80.
A
Then,
after
you
executed
the
translation
action,
you
can
again
use
the
captain
quality
gate
to
evaluate
if
the
service
quality
is
now
satisfying
again
and
then
you
can
either
actually
do
the
next
termination
action
execute
the
next
regulation.
Action
or
you
can
close
the
problem
since
it
has
been
remediated,
so
these
sli
concepts
and
slo
contents
and
files
can
be
reused
a
lot
of
time.
A
So
how
can
we
really
use
this
within
captain
and
when
do
they
come
into
play?
So,
first
of
all,
in
captain
everything
is
organized
inside
projects
and
services.
So,
first
you
basically
create
the
project
we
can
say.
The
project
is
something
like
a
it
gives
repository
where
we
store
all
of
our
files
and
inside
captain.
It's
also
each
project
will
get
its
own
name
space
with
different
stages.
A
A
The
shipment
file
basically
describes
your
environment
describes,
which
stages
you
want
to
have
like
a
pre-production
or
staging
stage
in
a
production
stage
which
kind
of
deployment
strategies
you
want
to
have
which
kind
of
testing
strategies
it
does
not
define
the
tooling,
like
which
kind
of
monitoring
you
would
not
define
it
in
the
shipyard.
You
would
define
in
the
captain's
uniform
that
will
then
connect
the
tooling
to
the
actual
process,
so
we
really
have
a
distinction
between
like
what
you
want
to
have
here
and
who
is
responsible
for
it.
A
A
I
can
also
show
you
this
this
in
an
example
later
and
in
the
next
step,
you
start
to
configure
your
monitoring
either
prometheus
or
other
monitoring
tools
that
will
basically
create
for
the
service
that
is
already
on
board
and
based
on
the
sli
and
so
files
that
you've
already
given
the
captain.
They
will
configure
your
prometheus
and
grafana
instances,
and
with
this
you
can
start
to
deploy
services
with
captain
to
send
an
event
about
a
new
artifact.
A
A
new
artifact
is
basically
a
new
docker
or
a
new
container
image
doesn't
have
to
be
talked
about.
A
new
container
image
and
captain
will
start
its
deployment
pipeline
and
triggering
all
these
different
tools,
for
example,
triggering
helm,
actually,
first
updating
the
git
repository
then
triggering
helm
to
do
the
deployment.
If
it's
configured
as
a
blue-green
deployment,
captain
will
do
a
blue-green
deployment
with
helm,
triggering
the
tests
that
have
been
that
have
been
defined
either
performance
tests
or
just
some
some
basic
tests.
A
The
test
tool
itself
again
is
defined
in
the
captain's
uniform.
So
in
this
case
I
select
the
chain
meter.
So
c
meter
will
be
triggered.
Once
g
meter
is
finished,
it
will
come
back
with
the
evaluation
to
captain.
Captain
will
start
to
evaluate
it,
based
on
the
sli,
so
files
and
then
decide
to
promote
it
or
not
promote
it
to
the
next
stage.
A
All
the
information,
or
maybe
just
the
information
about
the
promotion,
can
be
sent
by
a
slack
or
teams
or
whatever
you're
using
and
then,
if
you
decide,
if
captain
decides
or
you
decide
to
promote
it,
you
can
move
it
to
the
next
stage.
You
can
also
skip
tests.
Captain
will
try
to
reach
a
testing
tool
if
no
testing
tool
is
available.
Capture
will
move
on
to
the
next
stage.
So,
for
example,
in
this
case
I've
decided
in
production.
A
I
don't
want
to
execute
performance
tests
or
load
tests.
Since
I
have
basically
my
end
users
testing
my
website,
my
webshop
all
the
time
by
just
using
it.
So
I
can
just
wait
for
a
couple
of
minutes
hours
whatever
and
then
do
it
up.
Another
evaluation
score
it
again
and
then
keep
it
or
roll
it
back
to
the
previous
version.
A
A
It's
the
reason
behind.
It
is
just
that
the
prometheus
service
is
basically
responsible
for
configuring,
chrome
ethos
and
like
talking
to
a
prometheus
instance
and
configuring
this
instance
the
other
one.
The
prometers
sli
service
is
more
responsible
for
fetching
all
the
data
from
prometheus,
and
then
we
have
a
graphana
service,
all
three
services
or
even
more
services.
They
can
listen
on
the
same
events
same
kind
of
events,
and
they
can
they
can
integrate
the
events
in
a
way
that
is
up
to
up
to
the
implementation
of
all
the
services.
A
A
If
you
have
more,
I
would
be
happy
to
include
them
if
you
have
something
like
built
by
yourself
but
by
your
company,
if
it's
open
source,
I
would
be
very
curious
to
take
a
look
at
it
and
I'll
probably
include
it
also
here,
if
you
think
that,
like
captain
might
solve
some
of
your
challenges,
please
go
ahead
and
give
it
a
try.
Our
website
is
captain.sh
and
all
the
source
lives
on
github.com
captain
slash
captain,
that's
our
main
repository.
A
If
you
want
to
give
it
a
try,
and
if
you
want
to
have
like
hands-on
tutorials,
it's
tutorials.captain.sh,
that's
also
an
easy
way
to
get
started,
but
yeah.
Just
let
me
know
if
you
are,
if
you
think
that
might
be
something
that
will
help
you
in
your
journey
of
onboarding,
more
services
to
like
using
prometheus,
to
monitor
and
to
get
observability
for
more
services
in
your
organization.
A
That
would
be
great
and
yeah.
I
think
that
was
it.
What
I
wanted
to
show
you,
since
I'm
at
the
end
of
my
slides,
but
what
I
wanted
to
show
you
is
like
how
we
how
the
hit
repository
for
captain
looks
like
and
then
I
will
take
a
look
if
there
are
already
some
some
questions
so
going
back
to
or
going
to
my
hit
repository,
so
captain
holds
its
own
git
repository
inside
the
kubernetes
cluster,
but
in
this
case
I've
linked
it
to
github.
A
So
I
can
show
you
what's
really
going
on
so.
First
of
all,
I
have
all
the
different
stages
defined
as
branches
in
my
git
repository.
So
I
can
take
a
look
at,
for
example,
my
staging
branch
and
I
can
see
which
services
are
on
board
it.
So
I
have
two
different
services,
the
cart's
database
and
then
the
shopping
cart
service
itself.
So
it's
a
very,
very
small
sock
shop.
A
It's
not
even
the
full
stock
shop,
it's
just
the
shopping,
cart
and
no
search
no
front
and
nothing
else,
but
just
a
shopping
cart
by
the
way.
The
source
code
of
the
shopping
cart
does
not
live
in
this
git
repository.
It
lives
wherever
you,
you
have
it,
because
captain
itself
was
never
intended
to
to
version
the
source
code
and
to
build
this
microservice.
A
But
we
start
when
the
microservice
is
already
built
as
a
container
image,
and
then
we
start,
then
captain
can
kick
in
basically
and
we
have
all
the
services
that
are
all
the
files
that
I
was
talking
earlier.
We
have
them
here.
We
have
our
slo
file,
we
have
the
testing
instructions
here.
We
have
the
help
charts
here
and
so
taking
a
look
at
our
slo
files.
A
We
can
see
how
the
service
level
quality
will
be
evaluated,
and
this
is
the
file
that
will
be
used
also
to
generate
the
alerts
and
parts
of
the
dashboards
so
and
moving
to
another
stage
like
to
production.
I
think
I've
not
edited
yet
for
production.
I've
not
added
an
slo
file,
so
I
won't
be
creating
any
alerts
for
production
which
is
in
the
demo
it's
okay,
but
for
a
production
use
case.
Of
course
you
also
want
to
have
this
slo
file.
Good
part.
A
Is
you
can
easily
reuse
those
slo
files
since
in
the
slo
file
itself
it
doesn't
talk
about
any
service,
it
is
responsible
for
because
we
put
it
in
the
right
structure
in
the
git
repository.
So
captain
knows
where
to
look
for
this
kind
of
files.
So
if
there
is
a
file
inside
the
cards
folder,
then
capture
will
find
it.
If
there
is
no
file,
the
capital
will
just
skip
it.
So
in
this
case
this
is
how
captain
manages
and
versions
these
files.
A
There
is
one
question:
if
I
understand
it,
you
need
to
know
the
metrics
names
up
front.
Is
there
a
way
to
create
recording
rules?
Actually,
yes,
the
the
metric
names
like
in
the
service
level
indicator
files.
Maybe
we
have
one
here
edit
or
I'm
reusing
the
indicators
that
are
already
built
in
I'm
afraid,
I'm
reusing
the
ones
that
are
already
built
in.
Let
me
just
take
a
look,
so
we
have
to
go
back
to
the
slides.