►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Prometheus Monitoring Mixins: Using Jsonnet to Package Together Dashboards, Alerts and Exporters - Tom Wilkie, Kausal (Intermediate Skill Level)
Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configure monitoring configuration hard to build. Simultaneously, its becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration. In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” - extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul and Cassandra.
Prometheus offers powerful open source monitoring and alerting - but that comes with higher degrees of freedom, making pre-configure monitoring configuration hard to build. Simultaneously, its becoming accepted wisdom that the developers of a given software package are best placed to operate said software, or at least construct the basic monitoring configuration. In this talk we present a technique for using Jsonnet (a configuration language from Google) for packaging and deploying “Monitoring Mixins” - extensible and customisable combinations of dashboards, alert definitions and exporters. This technique allows developers of open source projects to publish best-practice monitoring configurations alongside their code, and for users to consume it, customise it and stay up to date. We will present example Mixins for Kubernetes and other services such as Consul and Cassandra.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hello,
everybody,
my
name
is
Tom
I
work
for
the
final
apps.
Now,
although
every
cube
can
I
seem
to
be
a
different
company,
I'm
still
working
on
Prometheus
and
and
today,
I'm
come
to
talk
about
Prometheus
monitoring
mix-ins.
So
this
is
a
way
of
packaging,
together,
dashboards
and
alerts
in
kind
of
reusable
redistributable
way
and
I'm.
Here
to
talk
to
you
about
why
the
story
behind
this
is,
we
were
doing,
we
were
helping
people
adopt
Prometheus
in
Crafar,
know
and
do
their
monitoring
setups
for
them.
A
We
found
we
were
copying
and
pasting
the
alerts
we
defined
for
kubernetes,
for
you
know
for
Cassandra
for
console
for
CD.
We
were
copy
and
paste
them
between
customers
and
doing
slight
tweaks,
and
when
we
got
to
like
the
sixth
or
seventh
time
we're
like
you
know,
this
has
got
to
stop
so
we've
been
working,
a
bunch
of
has
been
working
over
the
past
few
months
on
a
way
of
making
this
much
more
reusable.
A
A
So
if
10
of
you
could
follow
me,
make
me
really
happy
so
there
is
already
a
way
of
sharing
dashboards,
and
this
is
fun
labs
just
come
here
and
I
work,
for
they
have
an
online
system
that
allows
you
to
upload
dashboards
and
then
other
people
can
really
easily
download
them
and
I
can't
I,
can't
say
nasty
things
because
they're
now
my
employer,
but
there
there's
some
challenges
with
this
approach.
Most
of
the
dashboards
that
I've
got
from
Griffin
comm
give
me
things
that
look
like
this.
They
basically
just
just
haven't
quite
worked.
A
Every
time
I've
got
them,
I've
had
to
go
in
and
tweak
with
them
and
play
with
them
to
make
them
work
for
my
install
of
communities
or
my
install
of
of
anything
I'm
going
to
turn
that
off
and
normally
the
things
we've.
The
problems
we've
had
is
kind
of
the
queries
that
they've
used
have
made
assumptions
about
the
labels
that
are
being
used
for
the
targets
and
for
the
time
series.
A
So
this
is
trying
to
tell
me
how
much
memory
I'm
using
in
my
entire
cluster,
the
sum
of
all
of
the
container
memory
usage
divided
by
the
sum
of
all
the
machine,
and
it's
trying
to
do
it
for
a
particular
node.
But
in
my
prometheus
kubernetes
setup,
I,
don't
put
the
hostname
of
the
pods
and
containers
as
a
label,
because
that's
actually
a
bad
practice,
and
you
shouldn't
do
that.
You
should
be
doing
it
through
joins
and
other
methods,
and
so
this
this
is
really
common
most.
A
For
me,
at
least
most
of
the
dashboards
I've
downloaded
this
way
haven't
worked,
and
so
the
take-home
here
is
that
to
fix
this
problem,
dashboard
should
not
be
making
opinionate
should
not
be
having
opinions
about
the
label
sets
you
use.
This
should
be
configuration
that
I
can
inject
after
I've
downloaded
the
dashboard,
and
that's
not
the
only
problem.
A
We
want
to
encourage
people
who
write
software
to
also
tell
us
how
to
monitor
it.
Normally,
the
author
of
the
software
is
the
best
person
to
do
this
and
they
know
how
it
can
fail.
They
know,
you
know
what
the
error
cases
are,
and
so
really
we
want
people
to
distribute
alert,
definitions
and
dashboards
with
their
code
and
and
chorus
and
SCD
do
this,
which
is
really
good
right,
really
really
should
be
rewarded,
but
the
problem
is,
you
know:
I
come
in
and
I
copy
and
paste.
A
This
and
I
put
it
into
my
cluster,
and
then
it
don't
quite
work.
You
know
inevitably,
and
so
I
fix
a
few
things
and
I
make
a
PR
and
I
share
them
and
everyone's
doing
this,
but
there's
not
really
a
way
to
consume
the
iterative
improvements
that
are
happening
on
open
sourced
monitoring
configurations,
there's
no
easy
way.
I'd
have
to
keep
going
and
checking
and
copying
and
pasting
and
reapplying.
My
label
sets
to
this
config,
which
is
just
you
know.
It
just
doesn't
happen.
A
So
we
want
a
way.
That's
that
we
can
configure
these
labels.
We
want
a
way
that
we
can
extend
them.
So
Jack
here
gave
me
a
really
good
example.
Just
earlier
he
wants
to
inject
into
the
alerts
the
ability
to
add
URLs
to
his
playbook.
You
know
so
the
sed
alerts
they
distribute.
Don't
have
that,
obviously,
because
it's
his
playbook,
and
so
he
wants
to
be
able
to
extend
them.
A
I
want
them
to
be
reusable
I
want
them
to
be
easy
to
install
and,
more
importantly,
easy
to
stay
up-to-date
with
you
know,
I
want
to
treat
all
of
this
like
code.
You
know
code
is
constantly
improving
and
if
your
vendor,
in
or
if
you're
using
dependencies
properly
then
you're
consuming
those
up
and
getting
those
improvements,
so
I
want
to
I
want
to
do
this.
A
A
Yeah,
so
let's
look
at
the
first
one,
configurable
and
extensible
tends
out
actually
profound.
It
does
have
a
way
of
doing
this.
Bhuvana
has
templates.
I
didn't
know
about
this
for
a
really
long
time,
I'm
using
graph
honor
for
maybe
three
or
four
years
and
I
only
found
out
about
this
in
the
last
year.
So
maybe
I'm
slow,
but
you
can.
You
can
define
templates
at
the
top
of
your
dashboard
and
then
from
there.
A
You
can
use
dropdowns
to
select
and
you
can
substitute
them
into
the
queries
and
this
actually
souls
most
of
the
problems
for
graphing
our
dashboards.
Unfortunately,
graph
on
accom
doesn't
support
PRS,
otherwise,
I'd
be
going
through
and
making
these
changes
to
the
dashboards
on
there.
So
that
kind
of
comes
to
my
second
point:
I
was
making,
but
also
this
approach
doesn't
work
for
Prometheus,
recording
rules
and
Prometheus
alerts,
because
there's
no
UI
for
those.
You
know
that
they're
their
files
on
disk.
A
That
way,
if
you've
ever
exported
a
dashboard
from
grifone
you'll
see
it's
a
massive
blob
of
JSON
and
it's
really
hard
to
code
review
and
it's
really
hard
to
enforce
standards
on
so
so
yeah.
We're
not
we're
not
going
to
do
it
like
this,
so
we're
gonna
use
a
config
language
and
then
you
get
into
the
big
debate
of
which
is
the
best
config
language.
So
this
is
the
next
30
minutes
of
the
talk,
which
is
the
best
conflict,
language,
I'm,
gonna
review,
all
of
them.
A
No,
not
really
I'm
going
to
jump
straight
to
the
answer.
You
can
do:
yeah,
Mel
and
Jason,
but
they're
not
really
dynamic
enough
they're.
Just
data
data
structure,
definitions
effectively
and
there's
no
way
of
stop
shooting
I,
know.
Yeah
Mel
has
some
extensions
to
do
this,
but
yeah
not
going
to
do
that,
and
actually
quite
common
in
most
of
the
places.
I
see
some
combination
of
this
for
at
least
for
doing
their
kubernetes
deployments
and
so
on,
they're
using
em
mm
for
Ginga
to
template
out
the
differences
between
different
environments-
and
this
is
fine.
A
This
is
better
than
just
having
the
raw
yeah
Mille,
but
the
problem
here
is
that
doesn't
give
you
any
kind
of
abstract
avow.
You
can't
define
some
defaults
for
all
deployments
or
you
can't
define
you
know
in
this
world.
You
can't
define
defaults
for
all
dashboards
and,
what's
more,
I
want
to
be
able
to
build
functions
which
allow
me
to
say
generate
recording
rules,
for
my
histograms
I,
probably
should
have
asked
at
the
very
beginning,
like
is
everyone
familiar
with
Prometheus
I'm,
using
recording
rules
and
histograms
like
hands
up
if
you're,
using
Prometheus?
A
Okay
and
just
oh,
you
all
happy
with
these
terms
good
and
hands
up
refusing
core
fauna,
yeah?
Okay,
so
so
I
want
functions.
Basically,
I
want
some
sort
of
abstraction
and
what
what
I
think's
actually
becoming
quite
popular,
is
using
like
real
programming
languages
for
our
configuration,
a
company
called
improbable
in
London
uses,
go
for
all
of
their
configuration
for
their
monitoring
for
the
deployment
objects
in
kubernetes.
A
A
But
I,
don't
like
this
approach.
I,
don't
like
this
approach,
because
this
is
a
full-blown
programming
language.
This
feels
like
I'm,
writing
a
program.
It
doesn't
feel
like
I'm
writing
code.
It's
a
bit
intangible,
but
also
like
python.
Has
global
state
I
hate
global
state.
I!
Don't
want
side-effects
in
my
config,
that's
really
strange
to
me
and
and
I
wanna
I
want
a
bit
more
predictability
and
I
worked
at
Google.
So
what
I
want
is
JSON
it
JSON.
It
is
a
configuration
language.
It
comes
out
of
Google,
but
it's
not
actually
used
by
Google.
A
A
So
this
is
the
language
internally
within
Google
that
they
used
to
describe
jobs
running
on
Borg,
which
is
I,
guess
the
predecessor
to
kubernetes,
what's
more
J
sonic
can
manifest
out
can
render
out
Yama
files
any
files,
JSON
files
in
you,
basically
name
it
all
just
strings
and
just
normal
files
written
by
a
chap
called
Dave
Cunningham.
Who
I
don't
think
is
here,
but
he's
a
good
guy
to
give
you
a
kind
of
idea.
A
What
what
JSON
adds
I've
just
taken
this
straight
off
their
website,
Jason
as
I
said,
is
just
kind
of
the
data
definition
and
JSON.
It
adds
variables,
conditionals,
arithmetic
functions.
We
can
read,
but
mainly
adds
functions
and
add
string
substitutions,
so
I'm
now
going
to
go
to
his
website
and
give
you
a
bit
of
a
demo
if
I
can
fiddle
with
my
screens
good.
A
So
the
websites
recently
been
rim
that
revamped
and
is
pretty
good
and
he's
included
a
little
JSON
JavaScript
compiler.
So
you
can
see
you
know
you
can
edit
this
and
it
compiles
out.
So
what
we'll
start
from
from
scratch?
Okay,
an
empty
file
is,
is
an
error
which
is
good,
so
JSON
it
you've
got
JSON
objects,
you
don't
need
the
quotes
around
the
keys
or
cheshire
I.
A
Don't
know
whether
you
need
an
Jason
I,
don't
rent
a
son
for
a
while,
okay,
and
so,
if
you
do
that,
you'll
see
it
renders
an
object
called
hello
world
right.
This
is
trivial,
but
if
I
don't
want
to
just
say
hello
to
the
world,
I
can
do
things
like
hello,
substitution,
name
and
then
I
can
do
local
name
equals
Tom,
okay,
hello,
Tom,
even
better.
A
A
Okay
and
then
here,
I'm
gonna
refer
to
name.
So
this
is
the
first
kind
of
bit
of
magic,
which
is
called
self
knit
refers
to
the
dictionary
in
basically
so
you
see
this
is
saying:
I
must
supply
a
name,
so
I'm
gonna,
I'm
gonna,
merge
this.
This
is
the
second
bit
of
magic
in
JSON.
It,
which
is
the
merge
operator
and
I'm
gonna,
have
to
quote
that.
A
Okay,
so
now
I
can
say
hello
to
Tom,
but
the
output,
spitting
out
name
Tom
well,
I,
don't
want
that
in
the
output
like
cuz,
that's
kind
of
a
hidden
variable
right.
So
if
we
do
a
double
double
colon
there
double
column
means
hide.
You'll
see
this
a
lot
in
the
future.
So
this
is
why
I'm
putting
it
in
you
can
also
put
arbitrary
stuff
in
here.
You
know
you
could
do
be
good
by
and
you'll
see
how
the
merge
operator
works.
Okay.
A
So
this
is
the
basics
of
JSON
it,
and
you
can
see
from
these
building
blocks.
You
can
build
up
quite
a
lot
of
complexity.
I'm
just
gonna,
though
some
other
stuff
I
wanted
to
show
you
done.
String
substitution
done
hidden
fields,
done
merging
and
there's
syntactic
sugar
around
the
merge
operator,
which
is
kind
of
cool,
so
I'll
call
this
local,
hello,
err,
yep,
definitely
spelt
right
and
then
I'll
do
hello.
A
Okay
and
you
can
see
how
I
don't
add
to
put
the
plus
in
there
yet
because
it's
it's
syntactic
sugar,
basically
and
this
kind
of
makes
it
feel
like
you're
instantiate,
a
Hello.
If
you
see
what
I
mean
you're,
not
you're,
just
merging
the
dictionary
name
equals
Tom
with
a
Hello
a
dictionary,
and
it
becomes
this
thing.
A
Okay,
another
example:
merging
references
are
there's
one
extra
special
reference
that
I'm
going
to
do
so
be
it's
gonna,
be
another
object,
and
inside
B
we're
gonna
put
C
and
we're
gonna
make
that
a
list
and
inside
here
we're
gonna,
put
a
string.
Well,
just
a
string
okay.
So
this
is
kind
of
what
we
expect
yep
and
then
we
can
sub
the
name
in
but
self
dot
name
wouldn't
work
here
right
because
we're
in
a
string
we're
in
a
list
right
Phil
does
not
exist
name.
A
So
what
you
can
do
is
you
can
do
super
nope,
that's
wrong!
You
can
do
dollar
dollar
name
dollar
refers
to
the
utmost
object.
So,
in
this
case,
dollar
is
referring
to
this
object
here
and
fingers
crossed
it
works
yes,
I
seem
to
know
what
I'm
doing
so
anything
else
to
show
you.
Yes,
there's
one
more,
and
this
is
where
it
starts
to
get
really
powerful.
You
can
do
things
like
we're:
gonna
go
a
super
dot,
a
plus
I,
don't
know
right
and
so
super
refers
to.
A
In
this
case
the
thing
you're
being
merged
with
no,
it
doesn't
oh
yeah
it
does
it's
my
name
yeah
hello,
Tom,
dude,
excellent.
So
that's
basically
JSON
it.
It's
really
straightforward.
In
my
opinion,
it's
got
a
standard
library
for
doing
all
sorts
of
functions
and
things
and
see
it's
got
maths
and
and
so
on,
but
really
it's
that
merge
operator.
That
is
the
important
thing,
because
that's
the
that's
what
you
that's,
what
you
use
for
composition,
which
is
what
you
use
to
build
up
your
monitoring,
config.
A
There's
a
few
extra
examples
on
here
to
encourage
you.
You
know
it
shows
how
to
do
a
function,
which
is
just
how
you'd
expect
it
to
work
and
then
down
here.
It
shows
you
how
to
manifest
out
ini
files
and
other
types
of
files,
they're
less
interesting
in
my
opinion,
so
on
mirror
they
used
to
be
away
for
keynote
to
automatically
mirror
a
nun,
no
I'm,
pretty
sure
or
I.
Just
dreamt
it
okay.
A
So
that's
that's
JSON
and
that's
the
language,
we're
gonna
use
and
now
I'm
going
to
show
you
how
we're
gonna
use
it,
because
it's
not
enough
for
everyone
to
go
off
and
and
just
dive
in
and
start
using
JSON
it
to
do
this,
because
the
idea
is
that
these
mix-ins
were
going
to
produce,
are
gonna,
be
reusable
by
you
know:
people
who
didn't
write
them,
so
we
want
some
kind
of
standard.
You
know
a
lightweight
standard,
at
least
to
say
what
the
keys
are
going
to
be
able.
So
this
is
it.
A
This
is
the
standard
all
five
lines
of
it.
You
put
configuration
in
a
thing
called
underscore
config
and
it's
hidden.
The
reason
we
say
underscore
config
is
just
to
make
it
really
clear
that
this
thing's
reserved
and
that
you
shouldn't
use
this.
The
underscore
is
not
meaningful.
The
plus
here
means
when
this
whole
dictionary
is
merged
with
another
dictionary
that
merge
will
become
recursive,
so
you
can
have
an
optional
plus
here.
If
it
doesn't
have
the
Plus,
this
configure
would
overwrite
the
config
of
anything
you're
merging
with
which
is
not
what
you
want.
A
You
basically
always
want
Plus
:.
When
you're
defining
these
things,
then
we've
got
a
dictionary
full
of
dashboards,
and
then
we've
got
a
list
full
of
alerts
and
a
list
full
of
coop
recording
rules,
and
they
could
be
the
same
thing.
I
guess
for
the
parents
out
there,
the
config.
Normally
this
is
Frederick
who
I've
been
working
with
on
this
I
should
mention
him
an
earlier
sorry.
Is
he
here?
Oh
he's
doing
the
deep
dive?
Okay,
well,
he's
been
working
on
me
with
this.
A
It's
a
real
shame
he's
not
here,
but
we've
settled
on
this
format
for
providing
the
config.
You
basically
have
to
provide
a
the
key
value
pair
that
gets
substituted
into
your
Prometheus
query.
You
put
it
in
config,
call
it
a
selector
and
then,
when
you
define
an
alert,
it
gets
pretty
coded
code
heavy
from
here
on
in
I'm
afraid
guys
when
you
define
an
alert
you
substitute
in
the
selector.
You
want
with
the
config
like
that.
This
is
string
substitution
in
JSON
it
it's
pretty
good,
triple
a
triple
line.
A
Triple
bars
are
just
for
doing
big
multi-line
strings
which
I
had
to
do
to
fit
them
on
the
slides,
and
this
is
a
cube
pod
crash,
looping
alert
which
you
can
consume
and
run
against
any
kubernetes
cluster
and
any
cube
state
metrics.
You
just
have
to
provide
the
label
that
selects
your
cube,
State
metrics.
A
So,
for
instance,
it
I
think
if
you're
deploying
corer
s,
this
is
called
name
equals,
cube,
State
metrics,
where
and
if
you
deploy
on
my
clusters
at
least
I,
tend
to
include
the
namespace
in
the
job
name,
so
I,
don't
accidentally
aggregate
jobs
across
namespaces,
which
is
almost
never
what
I
want
to
do.
You
can
see
when
you
render
this
out.
It
comes
out
like
this,
which
is
for
everyone
here,
who's
ever
written
a
Prometheus
alert
definition.
A
This
is
a
Prometheus
alert
definition,
I've
made
it
nice
and
bold,
so
you
can
see
where
the
substitutions
come
in,
so
that's
the
JSON
it
and
that's
the
rendered
yeah
more
and
I'll
show
you
how
to
render
it
next
so
reasonable.
So
here
we're
going
to
look
at
how
do
we
satisfy
the
second
bit
of
the
requirements
which
is
being
able
to
download
easily
install
and
stay
up
to
date?
It's
really
key
that
we
can
stay
up
to
date,
otherwise
those
hold
all
those
drive-by
commits
on
github
are
for
nothing.
A
So
how
do
we
do
that?
Well,
this
is
actually
where
I
was
hoping.
Fredrik
would
would
join
me
and
say
this,
but
turns
out
he
can't
we've
written
a
tool
called
JSON.
It
bundler,
you
don't
have
to
type
JSON
at
bundler
every
time
you
want
to
use
it
because
JB
wasn't
was
not
taking
on
most
of
our
machines,
so
it's
just
called
JB,
and
this
is
a
package
manager
for
JSON.
A
A
I
have
no
idea
what
that
is,
because
I
don't
use
Ruby
but
I
do
know,
go
lang
and
we've
tried
to
avoid
the
horror
show
that
is
going
venn
during
and
package
management
you
install
it
with
go
because
it's
written
and
go
and
then
you
have
to
do
it
in
it,
which
just
drops
off
a
couple
of
empty
files.
There
then
to
install
a
package,
so
we've
we've
got
this
cube.
A
Letty's,
monitoring,
github,
org
and
inside
there
off
here
is
the
kubernetes
mix
in
which
we've
slowly
been
working
on
together,
and
this
will
dump
it
into
your
vendor
directory
and
inside
you'll
notice.
It's
installed
this
thing
called
graph
on
it
graph
on.
It
is
a
json
library
for
building
graph
on
our
dashboards
and
it's
also
a
mouthful
to
say
it's
noticed
that
that
is
a
trend
transient
dependency
of
kubernetes
mixin
installed
it.
That
way,
and
that's
really
like
the
Minimum
Viable
Product,
we
thought
for
a
for
a
package
manager.
A
You
can
install
stuff
it'll
install
dependencies
and
if
you
run
JB
installed
without
any
parameters,
it'll
update
everything
which
are
the
three
things
we
wanted.
The
next
thing
you
can
do.
If
you
want
to
consume
this
and
start
customizing
these
mix-ins
and
say
you
know,
combine
mix
in
three
or
four
of
them.
You
dumped
a
little
file
in
your
in
your
directory,
called
monitoring,
monitoring,
JSON
it
which
can
be
called
whatever
you
want.
A
Actually,
but
inside
there
you
see
you
can
override
the
config,
and
this
is
where
we
get
into
the
JSON
it
right,
so
we're
taking
the
kubernetes
mixin
we're
importing
it
we're
merging
it.
With
this
dictionary
that
contains
a
config
dict,
which
is
merged
into
the
other
conflict
and
overrides
the
see
advisor
selector
cube
state
metrics.
A
Here,
we've
changed
it
to
job
equals,
cube
State
metrics
instead
of
job
equals,
default,
cube
sabermetrics,
and
then,
if
you
render
it
out
with
this
incredibly
long
and
complicated
command,
you
can
see
the
job
is
changed
to
cube
State
metrics
and
there
we
go.
It's
done,
that's
how
you
supply
your
opinions,
about
your
labeling
in
your
cluster
to
the
Nixons,
so
I
think
that's
pretty
cool
one
of
the
things
we
really
want
to
do.
A
We
didn't
get
time
to
is
package
up
all
of
these
commands
into
some
kind
of
script
or
little
like
mix
tool
that
allows
you
to
Lin
and
check
your
mix-ins
work
and
allows
you
to
deploy
them
a
bit
easier,
but
these
will
basically
render
out
into
the
ammo
files
that
you
can
then
load
into
Prometheus
or
the
JSON
files
you
can
load
into
gravano
and
the
final.
The
third
thing
I
really
wanted
right
was
deployment
agnostic.
We
said
we
want
to
support
more
than
just
kubernetes
or
though
we
want
a
really
first-class
experience
on
kubernetes.
A
So
we
want
to
you
know:
we
need
a
way
of
building
normal
files,
which
you
can
integrate
into
your
config
management
system.
So
he's
the
complete
commands.
It's
just
JSON
it,
this
one's,
quite
cool.
The
command
will
generate
out
a
subdirectory
full
of
dashboards,
so
I'm
gonna
just
give
that
a
show
I
give
that
a
go
see
if
it
works
and
my
screen.
A
A
We
can
I'm
supposed
to
have
there,
we
go,
we
can
JB
in
it
and
then
we
can
install
cube.
Letty's,
mixin
I
know
I'm,
cheating,
so
I'm
using
my
bash
history,
but
I
typed
like
a
five-year-old.
So
there
we
go,
we've
installed
it.
If
we
do
tree
on
vendor,
you
can
see
all
the
stuff
that's
been
included
and
the
transient
dependencies
and
then,
if
we
touch
monitoring,
dot,
JSON
it
and
then
we
go
back
to
the
slides
and
copy
and
paste.
What's
in
there.
A
I
already
have
it
here:
that's
what
I
was
supposed
to
do
now.
If
I
save
that,
then
I
should
you
can
see
monitoring,
dot
JSON,
it
has
what
I
wanted
and
if
I
do
JSON
its
to
do
so.
This
one
is
doing
the,
including
the
vendor
directory,
manifesting
everything
out
into
a
folder
called
dashboards
and
then
importing
and
and
sorry
executing
they'll
go
find
the
dashboards.
A
A
A
A
Live
demos:
why
do
we
put
ourselves
through
this
right?
Fingers
crossed
yay,
there's
some
alerts
and
you
should
be
able
to
see
that
all
of
the
job
names
in
these
alerts
have
been
they've
removed.
The
namespace
prefix
that
I
that
I
specified
my
clusters,
because,
let's
say
this
cluster
doesn't
have
them
so
almost
done.
There
is
one
more
thing
in
in
good
presentation
style,
and
this
is
case
on
it.
I
touched
on
case
on
it
earlier.
A
This
is
the
the
taller
Brian
down
here
works
on
and
it's
a
library
and
a
tool
for
building
kubernetes
objects
with
case
on
it,
though,
with
JSON
it,
and
it's
really
good,
we
use
it.
We
use
it
with
all
our
customers
and
the
actual
reason
we
chose
Jay
sonic
to
use
the
mix-ins
is
because
once
you've
got
the
same
language
for
your
kubernetes
deployments
and
objects
and
the
same
language
for
your
dashboard,
you
can
start
sharing
the
configuration
between
them,
and
so
this
is
really
cool.
We've
built
well,
I
think
is
cool.
A
At
least
we've
built
Prometheus
case
on
it,
which
is
a
library
a
case
on
it,
library
that
contains
deployments
for
Prometheus
Rivanna,
node,
Explorer,
2,
cube
state
metrics,
and
then
it
depends
on
those
keys
that
are
included
in
the
mix-ins
to
manifest
out
all
of
the
dashboards
and
alerts
into
config
maps
and
then
apply
them
into
your
cluster.
All
in
one
step,
you
don't
have
to
compile
them
or
anything,
it's
worth
noting
that
we
do
it
in
a
particular
opinionated
way.
Here.
A
If
you
use
the
Prometheus
operator,
for
instance,
and
so
this
I
think
this
is
super
cool
I'm,
going
to
just
give
you
a
quick
example:
no
I'm
not
actually
to
have
time
come,
find
me
afterwards.
I'll,
give
you
a
quick
example,
but
it
effectively
means
there's
a
single
single
layer,
a
single
namespace,
a
single
object
domain
for
all
of
our
configuration.
We
can
just
add
stuff
in
and
it
becomes
incredibly
easy
to
manage.
So
that's
Prometheus,
monitoring,
mix-ins
I,
haven't
got
my
notes
because
I
didn't
turn
military
walk.
Give
me
a
second
there.
A
That's
monitoring
nixon's.
We
have
built
a
whole
bunch
of
them
already.
This
basically
has
taken
the
stuff
we've
worked
on
with
our
customers
over
the
past
six
months
and
turn
them
into
mix-ins.
We've
got
the
one
going
into
chorus
fret,
CD
and
frederick
super
supportive
of
this
loads
of
improvements
going
in
there
I've
built
one
for
console,
hoping
to
merge
that
it
mean
it's
in
in
my
old
companies,
public
mono
repo,
but
we're
hoping
to
merge
that
into
the
console
exporter.
A
If,
if
they
let
me
I
own
the
volt
exporter
and
the
company,
grapeshot
sponsored
all
my
work
on
that.
So
we've
already
put
the
voltage
Porter
mixing
in
with
that
the
kubernetes
mixing
as
we
discussed.
This
is
in
this
new
organization
and
we've
also
got
a
semi
prometheus
mix
in
that
kind
of
for
self
monitoring,
Prometheus
and
and
dashboards
as
part
of
Prometheus
case
on
it.
I'm
gonna
break
that
out
and
put
it
somewhere
a
bit
more
official
working
on
a
node
exporter
one.
A
So
this
is
effectively
going
to
take
a
whole
bunch
of
the
alerts
that
we've
defined
in
the
Kuban
eTI's
one,
but
that
are
more
generic
and
and
would
apply
to
pretty
much
any
linux
machine
and
we're
going
to
hopefully
include
that
in
the
node
exporter
internally,
I'm
working
ones
for
Cassandra
ones.
For
my
sequel
and
you
name
it
basically
any
system
to
monitor
mix-ins
configurable,
an
extensible,
reusable,
hopefully
easy
to
install
deployment.
Agnostic
I've
got
a
little
bit
of
work
to
do
want
to
put
yours
in
playbooks.
A
Currently,
we
we
haven't
figured
out
way
of
doing
that.
I
want
to
build
this
tool,
so
you
don't
have
to
have
those
cumbersome
commands.
We
did
this
all
in
the
open,
Frederick
and
I.
So
if
you've
been
on
the
Prometheus
mailing
list
or
if
you've
been
on
I,
think
the
case
on
its
slack
a
whole
bunch
of
different
places,
we've
been
posting
the
design,
Docs
they're
out
there
publicly
readable
and
comments
on.
Please,
you
know
tell
me
if
you
think
this
is
a
good
idea
or
not.
Similarly,
Frederick's
put
the
package
management
proposal
up.
A
There
I'll
put
these
slides
online
so
you'd.
After
do
this,
I
wanted
to
call
out
a
few
people
who
have
really
helped
me
on
this
project.
This
is
Dave
looking
very
thoughtful.
You
can
find
him
at
spark,
prime
on
github
as
well
he's
a
really
cool
guy
he's
really
been
really
supportive
of
our
work
here
and
like
privately.
He
told
me
probably
shouldn't
repeat
this
I
guess.
A
Well,
no,
he
told
me
that
this
is
really
what
he
envisions
for
JSON
it
when
he
built
it
you
know
years
ago,
and
that
he
never
managed
to
get
this
far
because
he
got
bogged
down
in
other
things,
but
that
he's
really
happy
that
JSON
is
kind
of
this
configure
and
configuration
language
that
should
be
able
to
combine
multiple
different
domains.
Frederic,
obviously,
has
been.
You
know
been
my
partner
on
this
he's
built
the
package
management
he's
been.
You
know
he
owns
all
of
this
with
me.
A
Julius
I,
don't
know
whether
he's
here,
but
he
had
a
previous
design
doc
suggesting
you
know
a
lot
of
the
problems
we've
tried
to
solve
were
motivated
by
the
design
dock
that
he
circulated
and
again
super
great
feedback
from
him.
David
I
stole
his
slides
because
he
gave
this
talk
or
over
version
of
this
talk
at
the
Berlin
cloud
native
meetup
a
while
ago,
as
this
was
much
early
on
and
grapeshot
I
wanted
to
give
a
shout
out
to
grapeshot,
who
sponsored
a
lot
of
work
on
this.
Yes,
thank
you
very
much.
A
That's
a
great
question,
so
the
question
is:
what
do
I
do
with
my
existing
dashboards
and
how
do
I
translate
them?
Is
there
at
all?
So
the
nice
thing
is
JSON
is
a
superset
of
Jason,
so
you
can.
Just
in
you
can
just
include
your
old
dashboards
as
big
JSON
blobs
in
your
mix-ins
and
then
you
know
put
a
little
bit
of
effort
into
maybe
tightening
them
up
and
adding
in
the
substitutions
but
JSON
it
is
just
Jason,
so
you
don't
really
need
there's.
No
migration
necessary
effectively.
A
Obviously,
there's
static
and
you
don't
get
the
benefits
of
this.
We
don't
have
at
all
I.
Think
it's
a
great
suggestion.
Would
you
like
to
write
one
I
mean
I
ride,
encourage
I'd,
help
I,
think
I,
remember,
John
relief
works
was
saying:
there
is
a
he's
working
on
a
tool
which
is
more
like
a
set
of
rex's
I.
A
Think
for
translating
them
into
graph
on
it,
and
if
that
approach
works,
then
sorry
into
refine
a
Lib,
and
if
that
approach
works
we
can
potentially
use
the
same
tool
to
translate
them
into
a
graph
on
it
and
JSON
it.
Maybe
I
don't
know,
but
I
think
it's
a
great
suggestion.
Thank
you.
Other
question
I
hear
anyone.
It's
a
big
audience,
so
I'm
gonna
go
over
there
at
the
back.
A
Are
we
going
to
have
any
curated
lists?
I,
don't
know
I
want
to
I
want
this
to
be
a
big
open
thing
like
I.
Don't
want
to
I,
don't
want
to
rule
this
with
an
iron
fist.
You
know
I've
gotta,
let
it
fly
if
it's
gonna
flourish,
so
I
kind
of
like
how
we
go.
There's
no
go
package
manager,
there's
no
like
central
go
repository
and
you
can
just
publish
anything
on
github
and
that's
why
we
went
with
that
model
for
JB.
You
know
they
can
just
be
anywhere
I
suspect,
like
with
with
Prometheus.
A
There
is
a
a
list
of
exporters
and
it's
relatively
easy
to
add
to
that.
You
know
we.
Don't
we
don't
stop
you
adding
to
it.
We
just
stop
you
like
having
two
or
three
entries
that
are
the
same.
So
I
said
you
know.
Maybe
we'll
do
something
like
that,
but
we
want
to
encourage
this
to
grow
like
I'm,
not
going
to
have
a
central
list
that
you
must
add
a
PR
to
to
have
an
official
mixin
or
anything
like
that
like.
A
A
Okay,
so
it
does
have
a
package
manager.
We've
been
talking
to
Brian
about
what
we're
gonna
do
about
that
cuz,
they're,
they're,
very
similar
and
there's
slight
differences.
I
would
hope
we
can
not
have
two
and
we're
both
being
flexible
about
it
and
Brian
said
something
to
Bob.
That
I
didn't
hear
we're
looking
at
JB,
great
yeah,
so
I
mean
I.
Hopefully
we
can
work
together
like
we're
talking
about
it.
Anyone
else
over
here,
yep.
A
Where's
Frederick,
so
the
question
was:
is
there
a
good
solution
for
brought
Jason's
being
too
large
for
config
maps?
There
is
a
tool
in
maybe
in
the
prometheus
operator.
Maybe
it's
in
the
cubical
fauna
replacing
one
of
Frederick's
repos.
It
doesn't
work
with
this
yet,
but
it's
on
our
it's
on
our
like
radar
of
having
a
version
that
works
or
this
I
would
actually
hope
that
we
could
just
do
it
in
case
on
it
because
we've
got
you
know
it's
quite
a
powerful
language.
A
There's,
no
reason
why
we
shouldn't
be
able
to
write
some
sort
of
tool
to
do
that
and
I
guess
we
could
with
with
with
especially
with
the
with
the
case
on
its
support.
We
could
just
have
a
config
map,
a
dashboard.
At
this
rate,
it
would
be
pretty
straightforward
to
do,
but,
like
yes,
we're
aware
of
this
problem
and
we've
got
some
thoughts
around
it
and
we
would
love
help
up
there.
A
I'm
not
going
to
answer
questions
on
helm,
sorry,
I've
been
told
to
be
nice
now
I
mean
I,
realize
helm
has
a
lot
more
use
than
case
on
it
and
I
know.
Brian
was
telling
me
that
they
want
to
find
a
way
that
that
these
can
be.
You
know
more
compatible
together.
I
guess,
I
should
probably
look
at
a
way
of
consuming
nixon's
in
helm
as
its
helm
is
so
widely
used.
I
would
make
sense,
really
wouldn't
it
but
I
don't
use
helm.
So
anyone
else.
A
No,
oh
one
more
hi:
this
is
a
old-fashioned
I,
think
and
I
don't
drink
them,
because
I
can't
take
alcohol
that
strong.
This
is
the
this
is
why
I
thank
David
because
he
sent
me
his
slides,
and
that
was
his
front
page
and
it
was
much
better
than
the
one
I
had
so
I
still
look
cool
but
no
I,
don't
drink
I,
don't
drink
spirits,
beer
for
me,
Bob
last
one.
A
Would
you
use
this
to
set
up
your
real
able
rules
well
Prometheus
case
on
it,
so,
which
is
my
opinionated
set
of
deployments
for
Prometheus
on
kubernetes
has
embedded
in
it
a
set
of
relabeling
rules
and
they're
the
best
practices
that
I've
learned
in
the
two
or
three
years
I've
been
doing
this
they're
different
than
the
Prometheus
operator
rules,
so
the
mix
in
mean
relabeling
rules
are
definitely
outside
of
the
scope
of
the
mix-ins
and
the
whole
point.
I
guess
of
the
mixes
is
to
be
able
to
support
people
with
different
relabeling
rules.
A
Yeah,
so
I
guess
answers
questions.
No
this
doesn't
this
is
you
know,
relabeling
rules
are
beyond
the
scope
of
this
and
really
the
way
you
deploy
it
would
dictate
your
the
real
Eamonn
was
cool,
I
think
we're
out
of
time
if
you've
got
any
questions,
feel
free
to
come
up
front
and
we'll
answer
later.
Thank
you
very
much.