►
From YouTube: Building advanced SLIs and SLOs with PromLens for Keptn
Description
Join us in this webinar to learn more about how to build advanced service-level indicators and service-level objectives with PromeLens for Keptn.
Your hosts: Julius Volz (Co-Found Prometheus and Founder of PromLabs), Jürgen Etzlstorfer (Keptn / Dynatrace)
A
Hi
and
welcome
everyone
to
another
captain
webinar
and
today,
I'm
very
honored
to
have
julius
with
me
hi,
julius.
A
It's
really
a
pleasure.
Julius
is
a
co-founder
of
prumitoys
and
the
founder
of
prom
labs,
and
today
he
will
show
us
how
to
build
advanced,
slis
and
slos
with
prom
lens,
and
we
will
be
using
these
slis
and
slows.
Of
course,
in
captain
I
will
be
doing
a
little
bit
of
introduction
to
set
the
stage
for
julius.
I
will
talk
about
why
we
need
it
and
then
I
will
hand
over
to
yurios
and
he
will
share
how
to
really
build
advanced
slis
with
prom
lance.
A
If
you
have
any
questions,
it's
the
first
time
that
we
are
streaming
directly
to
youtube.
Normally
we
are
on
soon,
but
we
decided
to
go
directly
for
youtube
this
time.
If
you
have
any
questions,
please
put
them
in
the
youtube
comment
section.
I
will
be
actively
monitoring
this
section
and
I
will
forward
the
question
after
the
presentation
of
julius.
A
So
with
this
I
will
be
jumping
into
the
first
part
of
the
presentation.
Give
you
a
little
bit
of
background
julius.
I
will
just
put
you
to
the
backstage
and
I
will
bring
you
back
up
to
the
stage
really
soon
so
see
you
in
a
minute.
A
Well,
let's
start,
why
do
we
need
slis
and
slos?
What
is
it?
Actually?
I've
not
even
explained
it
yet,
but
let's
start
a
little
bit
earlier.
Actually,
let's
start
with
the
common
cicd
pipeline,
this
is
really
more
a
cd
pipeline.
I
think,
but
when
we're
thinking
about
these
kind
of
pipelines,
they
start
with
building.
They
start
with
deploying
we
want
to
run
some
tests
and
we
want
to
make
sure
that
our
microservice
at
our
application,
that
we
run
into
production,
that
it
really
satisfied,
satisfies
a
given
quality
criteria
or
more
criteria.
A
Actually,
so
we
need
to
have
some
kind
of
approval,
and
typically
this
is
done
by
analyzing,
the
time
frame
of
the
of
the
test
period
by
taking
a
look
at
the
performance,
is
it
slower
than
the
previous
run?
Is
it
equally
good?
Is
it
faster?
Why
is
it
faster
and
we
have
to
take
a
look
at
a
lot
of
data
and
we
have
to
separate
the
data
that
is
really
affected
by
this
test.
Run.
B
A
Other
data-
that's
usually
just
some-
let's,
let's
call
it
some
noise
in
our
environment,
so
we
really
have
to
make
sure
to
take
a
look
at
the
right
data
for
this
approval
process,
because
what
we
don't
want
to
have
is
that
we
ship
something
into
production.
That
is
actually
not
a
good
build.
So
we
really
want
to
make
sure
that
everything
we
want
to
ship
in
production
does
satisfy
some
given
quality
criteria,
and
this
process
usually
takes
quite
a
while.
A
A
We
could
actually
gather
the
metrics
automatically
for
the
test
period.
We
can
automatically
generate
a
score
and,
based
on
this
score,
we
can
automatically
make
a
decision
if
we
want
to
promote
it
to
the
next
stage.
If
we
want
to
release
it
to
production
or
if
we
want
to
roll
it
back
or
hold
it
back
in
this
stage
or
even
roll
it
back
to
the
previous
version,
and
this
can
be
done
automatically
in
captain,
we
also
call
this
a
quality
gate,
and
this
mechanism
should
not
be
done
only
once,
but
this
should
be
done
continuously.
A
So
if
we
have
this
concept,
we
can
put
it
in
our
cd
pipeline
and
the
actual
quality
gate,
the
quality
criteria
we
are,
we
define
them
as
service
level
objectives,
we
call
them
slos
and
I
will
share
a
little
bit
more
detail
just
in
a
second.
But
let
us
just
think
about
the
concept
that
I
just
explained
how
we
can
put
this
into
our
city
pipeline.
A
So
we
still
do
our
building
our
testing,
but
after
the
tests
we
are
triggering
this
quantity
gate
and
basically
the
quality
gate.
What
it
does
it
will
pull
the
metrics.
You
will
pull
the
data
for
the
testing
time
frame.
We
will
pull
the
right
data
that
is
labeled.
That
is
tagged
that
we
already
know
which
data
is
affected
by
the
tests
that
we
have
been
executing
on
our
on
our
application
that
we
want
to
deploy
or
that
we
want
to
release.
Actually.
A
A
We
can
even
give
it
either
give
it
a
thumbs
up
or
a
thumbs
down,
and
the
whole
process
can
be
now
really
decreased
in
terms
of
the
time
it
takes
from
a
couple
of
minutes
to
around
one
minute,
because
the
only
thing
what
we
have
to
do
is
automatically
fetching
all
the
data,
doing
the
evaluation
of
the
data
and
then
making
a
decision,
and
this
can
be
done
with
captain
quality
gates.
A
So
how
does
captain
quality
gates
work?
So,
first
of
all,
we
are
using
two
prominent
concepts
of
the
site:
reliability,
engineering,
community.
There
is
also
a
great
book
on
this,
the
sre
handbook.
It's
called
our
google
sr,
e-book,
it's
actually
from
google
engineers,
and
you
will
find
it
online.
You
can
read
it
online
for
free
and
I
will
just
explain
two
concepts
of
this
sre
community.
A
The
first
one
is
service
level
indicator
and
we
can
think
of
this
service
level
indicator
really
as
a
metric,
something
like
the
error
rate
of
the
login
requests
and
for
this
metric
we
want
to
define
a
service
level
objective,
and
this
objective
is
some
target
for
the
indicator.
Something
like
the
error
rate
of
the
login
requests
must
be
less
than
two
percent
and
we
can
even
say
within
the
last
15
minutes
or
within
the
last
test
run.
A
A
A
A
I
have,
for
example,
the
error
rate
the
number
of
database
calls
or
the
response
time
for
the
90th
percentile,
and
we
will
see
just
an
excerpt
of
a
prongul
here
how
this
is
done
internally
in
captain
or
how
you
would
define
it.
The
captain
can
use
it
and
in
the
slo
file
we
take
the
name
of
the
indicator
and
we
can
use
this
indicator
name
and
add
an
objective,
a
target
for
this
indicator
name.
A
So,
for
example-
and
I
hope
you
can
see-
also
my
mouse
moving
here-
for
example,
the
error
rate-
I
want
to
define
an
objective
for
my
error
rate.
For
example,
it
has
to
stay
lower
than
one
percent.
That's
basically
what
it
says
here:
it
has
to
be
lower
or
equal
than
one
percent.
A
So
with
this
I
can
define
the
criteria
or,
for
the
number
of
database
counts.
The
criteria
is
it's
only
allowed
to
increase
by
a
maximum
of
two
percent
to
the
previous
runs,
run
or
runs.
That's
just
how
you
want
to
define
it,
and
with
this
you
can
make
a
very
strong
quality
gate.
You
can
mix
different
criteria.
A
You
can
have
a
combination
between
relative
thresholds
and
absolute
thresholds,
something
like
we
have
here.
We
allow
it
to
pass
if
it
stays
within
an
increase
of
two
percent
to
the
previous
runs.
If
it
does
not
stay
within
this
increase,
captain
will
go
ahead
and
validate
also
the
warning
criteria.
A
A
This
is
all
orchestrated
and
triggered
and
executed
by
captain,
and
you
can
define
this
quality
gate
in
plain
ammo
and
you
will
define
and
add
this
quality
gate
to
captain.
So
how
can
you
execute
it
either
directly
in
your
city
pipeline,
orchestrated
by
captain,
or
you
can
also
do
it
via
the
cli
and
just
send
an
event
to
start
the
evaluation
and
what
the
captain
control
plane
will
do.
It
will
reach
out
to
the
providers
of
the
to
the
to
the
data
providers.
A
A
Defined
in
captain,
but
there
might
be
more
complicated
slis
that
you
want
to
write
and
writing
prompt
url
is
actually
not
that
easy,
so
I
will
be
handing
over
now
to
euless
and
he
will
explain
us
how
to
actually
build
advanced,
slis
and
yeah.
The
stage
is
all
yours.
Let
me
just
put
you
back
on
the
stage
here
and
here
we
go.
B
Yeah,
thank
you,
you're,
a
great
explanation,
yeah.
So
let's
look
at
one
example
of
a
bit
more
of
an
advanced
sli
and
how
we
might
go
about
building
it.
We
will
see
some
challenges
during
the
prom
ql
building
process
and
then
I'll
briefly
talk
about
prominence,
a
tool
by
my
company
promlabs
that
aims
to
make
building
of
queries
simpler.
B
So
just
as
an
example,
if
you
want
to
get
a
bit
more
of
a
complex
sli
than
the
ones
that
are
built
into
captain,
let's
imagine
we
want
to
look
at
the
500
error
rates
of
a
given
service
that
we
care
about,
and
we
want
to
aggregate
that
error
rate
over
multiple
different
service
instances
of
that
demo
service.
But
we
want
to
still
split
it
up
into
different
combinations
of
paths
and
methods.
So
we
want
to
check
each
of
those
individually
if
their
error
rate
percentage
is
pretty
large
for
that
combination.
B
So
this
should
be
easy
right,
as
you
haven't
already
mentioned,
promql
is
not
always
super
easy.
It
can
be
a
bit
tricky
a
bit
tough
to
learn,
but
also
once
you
know
how
it
works.
It's
not
always
super
easy
to
actually
work
with
it
and
build
queries
in
the
interface
in
prometheus.
So
a
query
that
I
just
described
could
look
like
this
without
going
too
deeply
into
the
prometheus
query:
language
I'll
just
describe
on
a
high
level
what
it
does
here.
So
on
one
hand
we
are
getting.
B
We
are
summing
up
the
500
error
rates
over
all
of
the
instances
in
our
service,
but
still
keeping
method
and
path
labels.
They
are
on
this
metric
here,
but
we
don't
see
them
and
then
we're
dividing
every
path
and
method,
500
combination
by
its
total
rate,
so
we're
getting
a
ratio
of
the
500
errors,
divided
by
the
total
rate
for
the
same
combination
for
that
path
and
method
to
get
from
a
ratio
to
a
percentage.
B
We
multiply
everything
by
100,
and
now
we
still
wanted
to
add
that
additional
criterium
to
say
we
want
to
exclude
any
path
method,
combinations
with
a
total
request
rate.
That
is
not
at
least
one
per
second,
so
I'm
doing
that
down
here
with
this
and
binary
operator,
a
set
operator,
just
you
know,
excluding
combinations
that
don't
at
least
have
that
high
of
an
error
rate.
B
So
what
can
go
wrong
first
thing
that
can
go
wrong
building.
This
kind
of
expression
is
that
you
don't
get
any
query
output.
B
The
result
in
the
table
view
in
prometheus
will
just
be
completely
empty
and
you
don't
know
why.
So
this
might
actually
be
totally
fine.
So,
in
the
case
where
there
are
no
well
where
there
are
just
simply
no
expressions
that
any
path
method
combinations
that
that
match
any
total
rate
that
has
at
least
one
request
per
second,
you
will
also
you
will
just
not
see
any
result,
but
there
could
be
many
other
reasons.
B
So,
for
example,
you
may
be
selecting
an
incorrect
metric
name.
Maybe
it
has
changed
over
time.
Maybe
it
was
correct
once
maybe
one
of
your
label
matches
like
the
job
demo
or
the
status
500
is
broken
in
some
way.
Maybe
the
label
names
are
not
quite
as
you
expect.
Maybe
the
label
values
aren't.
B
Maybe
you
are
calculating
rates
over
a
too
small
window.
A
rate
needs
at
least
two
data
points
to
exist
under
a
time
window
be
able
to
be
able
to
calculate
a
rate.
So
that
could
be
one
thing
where
just
produces
empty
output.
Maybe
you
got
the
binary
operator
matching
incorrect
here.
B
So
if
you
didn't
have
any
modifiers
on
this
binary
a
binary
operator,
you
would
also
just
get
an
empty
result.
So
you
see
like
a
lot
of
little
things
that
can
create
anxiety
and
are
tough
to
get
right
and
you
kind
of
need
to
know
what
underlying
data
you're
working
with
at
each
of
these
different
sub
expressions
to
be
able
to
formulate
the
right,
binary
matching
correctly
and
so
on,
and
so
on.
B
So
there's
one
more
issue:
for
example,
if
you
create
an
error
somewhere
in
a
prom
ql
expression,
this
is
a
shorter
one.
Just
to
demonstrate
this,
you
will
get
an
error
message
saying
that
something
is
wrong
somewhere
in
this
expression,
but
it's
kind
of
tough
to
figure
out
where
exactly
this
problem
is
happening
and
then
what
you
know
what
is
even
wrong.
What
do
certain
constructs
in
here
mean
like
group,
write
and
ignoring
and
so
on?
B
Also,
you
know,
even
if
you
get
your
queries,
correct,
initially,
metric
names,
change
over
time
label
names
change
over
time,
as
people
change
the
way
they
monitor
things,
the
the
instrumentation
within
services
and
so
on.
So
even
if
you
do
have
something
correct,
initially,
it's
nice
to
have
a
tool
to
be
able
to
debug
queries
over
time,
whether
that's
an
alert
query,
a
dashboarding
query
or
an
sli
in
captain.
B
Also
yeah
there's
not
real
auto-complete
syntax,
highlighting
deeper
code,
intelligence
and
so
on
in
in
the
built-in
prometheus
ui
so
enter.
Prom
lens
promlence
is
a
query
builder
analyzer
in
visualizer
from
the
for
the
prom
ql
query
language
and
you
can
try
it
out
on
promlands.com,
but
it's
really
meant
to
be
deployed,
on-premise,
very
simple,
docker
installation,
you
run
it.
You
point
it
at
your
prometheus
server
and
voila.
You
get
a
better
ui
for
working
with
promql.
A
B
B
The
core
aspect
is
that
it
really
shows
you
the
structure
of
each
query
that
you
type
and
shows
you
at
every
sub
query
what
results
you
have,
how
many
results,
what
are
the
different
label
names
and
what?
How
many
different
values
are
there
for
every
different
label
name
so
eight
for
the
mode
label.
Here,
for
example,
as
I
said
earlier
like
this,
would
show
you
really
quickly
where
you
actually
produce
empty
results
somewhere
in
your
expression.
B
It
can
also
show
you
where
exactly
certain
prong
ql
errors
are
happening
in
that
tree
view.
So
you
get
a
you
know
if
it
if
it
can't
find
them
with
the
linter
yet,
and
they
actually
happen
during
the
execution
you
can
find
it.
You
can
pinpoint
it
in
this
tree
and
then
for
every
type
of
note
that
you
click
on
in
this
tree.
It
can
visualize
a
bit
for
you
or
explain
what
exactly
is
happening.
What
is
this
note
doing
in
this
case?
It's
a
binary
operator.
B
So
this
helps
you
get
binary
operators
correct,
not
everyone
will
know
all
the
syntactical
constructs
of
prom
ql,
so
prom
lens
also
has
a
full
form
based
editor
built
in.
You
can
create
any
promptql
query
just
by
using
a
form
editor.
If
you
really
want-
and
this
is
inline
so
you
have
this
tree
and
you
can
just
switch
any
node
to
a
form,
editing
mode
and
then
see
what
this,
what
the
different
aspects
of
that
subquery
actually
mean,
and
you
can
change
them
and
apply
them.
B
Similarly,
you
can
actually
jump
into
any
sub-node
and
edit
it
directly
as
prongql
as
well,
if
you're
comfortable
with
that
and
yeah.
So,
let's
actually
just
you
know
without
looking
further
at
gifs,
let's
just
jump
over
to
prom
lens
itself
and
start
building
that
query
that
we
saw
earlier
just
so.
You
get
an
impression
of
how
you
would
work
with
this
tool
and
what
it
feels
like.
B
B
Great,
so
how
people
typically
start
is
just
hey.
I
select
some
data
and
in
this
case
you
know
this
is
the
actual
request
count
I
get
over
my
service
and
then
it
has
all
these
different
labels,
path,
status,
method
and
so
on,
and
you
know
already
is
actually
suggests
me
hey.
This
is
a
counter,
so
you
might
want
to
add
a
rate
over
this,
so
I'm
actually
just
going
to
follow
that
and
update
the
text.
View
up
here
just
follow
that
recommendation.
B
Now
I
probably
want
to
get
at
first.
Let
me
just
get
the
status
500
errors,
so
I
could
actually
just
use
the
autocomplete
up
here
and
now
I
have
only
the
bad
ones
only
the
500
requests-
and
I
want
to
get
those
not
for
every
instance
individually
in
the
in
this
case,
I
can
see
all
the
three
different
demo
service
instances
individually.
I
want
to
aggregate
that
away,
so
I
could
do
that
over
here.
B
I
could
also
actually
just
click
on,
add
some
and
then
I
could
use
the
form
builder
to
say
I
want
to
preserve
all
labels
except
the
instance
label.
I
want
to
aggregate
that
away.
So
oops
I
mistyped
it
instance
see
a
preview
up
here,
apply
the
change,
close,
the
form
editor,
and
now
we
get
the
500
error
rate
for
every
path
and
method
combination.
The
job
is
always
the
same,
so
we
can
ignore
that
great.
So
that's
good!
Now
I
want
to
divide
that
for
each
path
method
by
the
total
rate.
B
So
what's
the
total
rate,
we
could
actually
start
building
that
as
a
separate
query,
maybe
I'll
just
like
copy
this
over
here.
The
total
rate
would
then
be
all
the
different
status
codes
and
I
then
actually
want
to
get
rid
of
the
status
code
dimension,
because
you
know
I
want
to
sum
over
all
the
different
status
codes.
Currently,
it
would
give
me
this
per
status
code,
so
here
in
the
without,
I
will
also
say
I
want
to
do
without
the
status
code.
B
I
want
to
have
a
binary
operation
between
my
initial
query,
I'll,
keep
it
naive
for
now
and
just
say
like
some
divided
by
and
I
can
actually
just
drag
this
expression
from
the
bottom.
Over
here
I
hope
I'm
not
actually
making
it
more
confusing
right
now,
but
right
now,
I'm
trying
to
divide
this
500
error
rate
by
basically
the
total
rates-
and
I
see
already
I
get
zero
results
here
in
this
binary
operator.
B
Although
I
have
four
results,
on
the
left
hand,
side
right
for
all
these
patent
method,
combinations
and
five
results
on
this
one
there's
an
additional
path
on
the
total
side
that
doesn't
have
any
5x
500
errors,
so
it
doesn't
appear
in
these
four
results.
So
why
doesn't
this
binary
operator
do
anything?
Let's
explain
it
oops.
I
think
I
have
a.
I
broke
something.
A
B
Okay,
let
me
interesting
so
that's
the
case.
I
need
to
check.
Let
me
get
there
again
quickly,
so
we
have.
A
It's
really
like,
so
we
can
see
it
was
really
alive
or
it's
really
interactive.
B
Yeah
you
can
imagine,
though,
that
I
will
fix
that
very
quickly
now
I
thought
I'd
get
rid
of
those
okay,
so
again
like
this
and
our
here
on
the
right
hand,
side
I
want
to
get
rid
of
the
status
code.
B
Yup
and
okay,
interesting
I'll,
try,
I
won't
click
there
again.
I
have
to
see
what's
actually
up
there
so
now
I
will
actually
maybe
just
look
at
the
two
sides
here.
So
one
reason
why
these
don't
match
up
is
because
the
left
hand
side
still
has
a
status
code
label
and
by
default
this
kind
of
binary
operator
matching
would
try
to
match
exactly
identically.
B
Labeled
objects
from
the
left
and
right
hand,
side,
but
all
the
left
hand,
side
objects,
have
the
status,
500
label
or
the
right
hand
side
do
not
so
nothing
will
match.
So
I
can
actually
just
tell
this
thing
to
say:
hey
only
match
on
or
actually
the
opposite
ignore
some
labels
in
the
matching.
So
I
want
to
ignore
the
status
matching
and
yeah.
So
now
I
actually
do
get
results
here
for
good
measure.
For
cardinality
reasons.
I
could
add
a
group
left
here.
B
I
don't
think
I
actually
need
it
here,
so
you
can
see.
I
jump
around
a
bit
wildly
like,
depending
on
where
it's
currently
most
convenient
for
me
to
edit
things
like
with
a
form
inline
drum
ql.
I
can
always
switch
to
that.
If
I
want
to,
I
can
edit
the
whole
subtree
in
line
now.
Let
me
just
save
this
link
just
in
case
that
bug
is
still
online.
So
if
I
explain
this
yeah,
no,
no,
this
actually
works.
B
So
you
can
see
that
the
left
and
right
hand
side
are
paired
up
correctly
for
all
these
different
for
the
500
rates,
on
the
left
hand,
side
and
the
total
rates
on
the
right
hand,
side
and
it's
ignoring
the
status
label
and
then
there's
this
additional
non-existent
api
endpoint
that
only
produces
404.
So
it
doesn't
have
a
500
error,
so
this
doesn't
find
a
match.
B
Okay,
so
the
last
condition
that
we
wanted
to
add.
Oh
well,
there's
a
couple
more
things
right,
so
I
wanted
to
get
to
a
percentage
now
just
multiply
everything
by
100
and
then
I
want
to
get
to.
I
want
to
filter
away
the
those
combinations
of
method
and
status
that.
B
In
this
case,
I
also
again
like
the
and
has
a
problem
here.
It
produces
zero
results,
although
I
have
four
results
on
the
left
and
five
results
on
the
right
and
the
reason
here
is
again
the
matching.
So
I
I
do
have
a
status
code
on
the
left
hand,
side
and
I
do
not
have
the
status
code
on
the
right
hand
side,
but
otherwise
the
cardinalities
actually
match
up
quite
nicely.
B
So
this
was
just
a
demo
how
to
work
with
prom
lens,
and
you
know
it
has
a
couple
more
features
that
haven't
shown
like
you
know,
the
linter
will
show
you
where
something
is
broken
and
so
on
cool.
Okay
back
to
the
actual
presentation.
B
There's
one
more
gotcha
that
I
haven't
talked
about
at
all,
so
currently
this
expression
that
we
looked
at
outputs,
maybe
no
paths
at
all.
Actually
I
do
have
a
bug
on
the
next
slide
here.
I
will
talk
about
that,
but
it
might
it
might
not
output
any
paths
at
all.
If,
for
example,
there
is
no
path
method
combination
that
has
more
than
one
per
second
right
or
it
might
output
multiple
outputs.
B
Yeah
for
for
any
so
basically
one
output
for
any
of
these
paths
that
do
have
a
larger
than
one
per
second
rate,
and
I
just
got
these
headings
wrong.
That's
what
what
I
meant
with
the
bug.
So
it
will
show
you
yeah.
It
will
show
you
all
the
paths
that
do
that
do
have
a
high
enough
rate,
because
we're
not
filtering
by
the
actual
outcome
value.
Yet.
B
And
yeah,
so
basically
yeah
sorry,
jung
actually
did
have
a
bug
in
here.
I
wanted
to
get
into
it,
so
I
wanted
to
get
to
a
state
where
I
actually
not
have
just
the
not
as
the
final
output
not
have
the
the
the
ratio,
the
error
ratio
for
each
path,
but
only
have
it
when
if
the
error
ratio
itself
is
also
high
enough,
so
we
would
have
to
add,
like
one
more
filter,
to
the
actual
query
which
I
actually
totally
forgot.
B
B
You
know
show
me
the
paths
that
match
all
these
criteria
and
that
have
a
larger
than
let's
say
two
percent
error
ratio,
and
then
we
get
one
output
yeah.
So
imagine
we
had
that
so
problem
with
captain
is
that
in
those
sli
specifiers
it
expects
exactly
one
single
output
number
and
not
zero
numbers
or
a
whole
vector
of
time
series
of
numbers.
B
So
we
need
to
somehow
condense
down
that
filtered
list
of
of
method
path,
combinations
to
a
single
number.
So
what
could
we
do?
For
example,
we
could
wrap
a
query
around
all
of
it
and
make
it
such
that
if
it
outputs
a
zero,
all
those
combinations
of
path
method
are
good
and
otherwise
it
would
produce
one
two
or
three
like,
however,
many
bad
of
endpoint
combinations,
there
are
so
the
way
we
can
achieve
that
is
this.
The
following.
B
We
wrap
the
entire
expression
that
we
do
have,
including
that
last
filter
that
I
just
ad
hoc
built
in
there.
We
wrap
it
in
a
count
aggregator
in
prom
ql,
and
this
will
count
how
many
output
series
there
are.
So
this
will
give
us
a
single
number.
In
the
end.
There
is
one
gotcha,
though,
if
there
isn't
even
an
input
number,
because
there
is
currently
no
bad
path
at
all,
then
the
count
aggregator
will
also
create
just
an
empty
result,
which
captain
will
also
not
be
able
to
deal
with.
B
So
we
need
to
make
sure
that,
in
the
case
where
everything
is
fine,
we
actually
join
in
using
the
or
set
operator
we
join
in
a
default
value
of
zero,
and
then
that
will
make
it
into
captain
and
so
in
captain,
then
we
could
just
have
a
actual
comparison
to
that
output
and
have
some
kind
of
threshold.
Maybe
if
it's
you
know
larger
than
zero,
we
don't
roll
out
further
great,
so
yeah
try
out
prominence.
You
can
actually
try
out
the
demo
on
promnets.com.
B
A
Cool,
thank
you
so
much.
It
was
really
great
to
see
how
you
start
from
very
small
and
easy
prom
ql
to
get
it
more
and
more
complicated.
So.
B
Really,
thanks
for
this.
A
Before
we
move
on
to
the
questions,
I
will
just
highlight
how
to
to
get
started.
If
you
want
to
try
captain
or
if
you
want
to
try
prom
lance
or
for
captain,
it's
captain.sh
for
prominences
prominence.com,
and
if
you
want
to
join
the
captain
community,
I
have
listed
a
couple
of
links
here.
I've
also
listed
the
integration
for
promethos
and
the
prometers
sli
service
that
you
might
want
to
install
when
you're
running
captain
together
with
prometheus.
A
We
are
doing
this
kind
of
webinars
regularly.
So
please
join
the
captain.
Community,
join
our
or
follow
us
on
twitter
and
join
our
google
group,
so
you
always
get
updated
on
when
we
are
hosting
another
captain
webinar.
So
let's
take
a
couple
of
questions
here,
so
we
received
a
couple
of
questions.
I
would
just
throw
them
directly
at
here
julius.
A
It
depends
on.
I
think
a
couple
of
questions
are
more
captain
related.
So
there
was
one
question
I
think
you
just
answered
it.
It
says:
do
I
need
to
sign
up
to
use
promlance
and
yeah,
but
I'll
just
hand
over
the
question
to
you.
B
Yep,
so
you
can
use
the
free
version,
so
it's
not
open
source,
but
it
it
comes
in
a
docker
container
which
you
can
run
with
or
without
a
license
key
and
without
it
has
some
basic
features.
So
you
can
manually
specify
a
prometheus
server
to
connect
to
at
the
top
and
then
enter
your
queries.
B
B
You
can
also
create
shared
query,
links
and
yeah
there's
some
other
features
that
are
or
that
will
be
only
available
with
a
license
and
yeah.
You
can
go
to
promnence.compricing
to
see
the
current
pricing
I'm
experimenting
with
it.
It's
still
pretty
early
on.
So
if
you
feel
like
that
middle
tier,
that
has
a
fixed
pricing,
isn't
really
working
for
you
just
talk
to
me
and
we
can
explore
this.
A
Cool,
thank
you.
So
there
is
a
free
option
and
yeah
so
give
it
a
try
on
prominence.com.
There
was
one
interesting
question:
can
I
deploy
prom
lens
with
captain?
I
think
it's
it's
actually
made
that
you
can
just
run
it
on
your
local
machine,
but
so
from
a
technical
perspective.
I
think
it
just
runs
in
a
darker
container
right.
B
Yeah
I
mean
like
typically,
you
would
deploy
prom
lens
even
somewhere
in
your
kubernetes
cluster
and
just
connect
it
to
the
grafana.
So
it
wouldn't
be
something
that
typically
users
would
deploy
on
their
laptop
or
so
it
would
just
run
somewhere
as
a
very
simple
service.
It's
mostly
stateless,
except
for,
if
you
have
some
link,
sharing
backend
configured
and
yeah,
and
then
it's
just
a
better
ui
for
your
existing
primitive
servers.
A
A
Yeah,
but
I
think,
because
we
had
another
question:
how
to
add
the
sli
to
captain
yeah.
That's
exactly
the
answer!
If
you,
if
you
write
your
sli
in
prom
lens,
what
you
can
then
just
do,
is
you
copy?
I
think
the
the
editor
that
you
showed
us.
It's
it's.
Basically,
a
text
editor,
so
you
can
copy
from
there
and
then
you
put
in
the
sli
file
in
captain
just
make
sure
there
are
no
blank.
There
are
no
new
lines
in
there.
A
You
copy
it
in
the
sli
file,
you
save
the
sli
file
and
with
the
captain
cli
you
just
do
a
captain,
add
resource
and
you
take
the
sli
file
and
put
it
on
into
your
captain
control
plane
and
it
will
overwrite
the
previous
version.
A
Everything
is
github
space,
as
you
know,
so
you
still
have
the
the
the
previous
version
in
the
history.
This
kind
of
all
these
kind
of
things
you
still
have,
and
you
can
just
an
edit
yeah
and
if
you
want
to
run
prom
lens
in
the
kubernetes
cluster
and
the
way
to
to
bring
it
to
the
kubernetes
cluster,
you
can
also
use
captain
to
deploy
there
and
you
can
even
have
a
quality
gate
for
prom
lens
to
check.
A
If,
if
some
of
the
prominence
internals
are
are
fine,
the
other
questions,
I
think
that
we
covered
already
so
with
that.
Once
again,
thanks.
B
A
For
joining
us
today,
thanks
to
euler's
thanks
to
to
the
audience
joining
us
on
youtube.
Thank
you.
Please
check
out
promlands.com,
give
it
a
try,
give
feedback
and
also
check
out
captain
on
captain.sh
and
check
out
the
reverse
integration
and
write
your
slis
and
slos
with
prominence
thanks
so
much
julius
and
see
you
soon
and
have
a
great
day
bye.
Thank
you.