►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes has entirely changed the way we build our infrastructure, enabling organizations to ship more changes faster. But, with all that speed comes challenges — containers introduce a non-trivial level of complexity when it comes to maintaining visibility into your infrastructure. Just as Kubernetes has required us to rethink how we build and deploy our applications, so too must we rethink the traditional approach to monitoring.
In this webinar, Sensu CEO and co-founder Caleb Hailey will review some of the existing and popular patterns for monitoring Kubernetes (like Prometheus), and what gaps remain outstanding, especially when it comes to monitoring the applications running on Kubernetes. He’ll go over best practices for monitoring workloads on K8s, including the sidecar pattern for monitoring.
A
Like
thank
everyone.
Who's
joining
us
today.
Welcome
to
today's
CN
CF
webinar
monitoring,
kubernetes
workloads,
the
citecar
pattern,
I'm
Chris,
short
CN
CF,
ambassador
I,
also
write
devops,
ish
and
I
worked
at
Red
Hat
I'll
be
moderating
today's
webinar
and
like
to
welcome
our
very
special
presenter.
Today,
Caleb
Halley
CEO
at
Sims,
ooh
I,
think,
is
Chris
excused
housekeeping
items
before
we
get
started.
Caleb
during
the
webinar
attendees
are
not
able
to
talk.
There
is
a
Q&A
box
on
your
screen.
A
Please
feel
free
to
drop
your
questions
in
there
and
we'll
get
as
many
answer
here
as
we
can
also
there'll
be
time
for
Q&A
during
the
discussion
and
at
the
end.
So
if
we
don't
get
your
questions
answered,
we
will
attempt
to
get
them
answered
after
the
fact,
as
well
without
further
ado.
Take
it
away.
Caleb
awesome.
B
Thanks
for
the
introduction
and
I'm
Caleb
CEO
and
co-founder
at
10
sue,
that
also
makes
me
kind
of
one
of
the
biggest
cheerleaders
for
sensu
happy
to
be
on
the
CN
CF
webinar
today,
and
we're
gonna
get
caught
up
with
our
overview,
for
the
presentation
today
is
to
get
caught
up
a
little
bit
since
a
guest
blog
post.
We
had
on
the
CNC
F
blog
a
few
months
back.
That
was
the
beginning
of
our
monitoring,
kubernetes
blog
series
and
so
we'll
kind
of
recap.
B
Some
of
the
main
points
that
we
covered
there
and
then
continue
that
conversation
with
a
discussion
today
about
going
beyond
monitoring
kubernetes
to
monitoring
workloads
on
kubernetes
and
we'll
do
that
with
the
sidecar
pattern
and
I
will
attempt
the
demo
gods
with
a
live
demo,
so
containers
cloud
native
computing.
These
are
our
new
reality.
B
Businesses
certainly
are
adopting
the
cloud
native
movement,
and
we
think
that
that
means
things
like
strategies
like
multi-cloud
are
here
to
stay
they're
necessary
for
the
modern
business
to
stay
competitive
and
our
industry
I
think
for
a
long
time
been
relying
on
micro-service
based
architectures
in
order
to
to
compete.
Everything
is
about
moving
faster
right,
so
micro
service
space,
architectures
have
been
big
movement
there,
and
this
Advent
and
and
near
ubiquity
of
micro
services,
kind
of
naturally
paved
the
way
for
container
technology.
B
You
have
to
take
into
consideration
when
monitoring
communities
you
have
data
coming
from
the
kubernetes
cluster
itself
and
projects
like
the
cube
state
metrics
each
individual
cubelet,
the
engine
behind
kubernetes
that
runs
on
each
of
the
notes
in
your
kubernetes
cluster,
the
underlying
compute
resources.
Of
course,
these
are
all
abstractions,
so
we
do
need
in
order
to
get
some
proper
visibility
into
the
health
and
performance
of
our
system.
We
need,
you
know
we
need
access
to
monitoring
data,
all
the
way
down
the
stack.
B
So
those
were
some
of
the
things
we
focused
on
in
the
first
part
of
the
blog
series
and
then,
as
that
blog
series
continued
on,
we
we
got
into
some
of
the
strategies
and
in
that,
in
this
parts,
two
three
and
four
of
the
blog
series.
We
really
talked
about
Prometheus
as
one
of
the
more
popular
solutions
for
monitoring
kubernetes.
It
makes
a
great
companion
for
as
telemetry
based
solution,
but
we
didn't
get
to
kind
of
segwaying
beyond
those
blog
blog
series
is
some
of
the
other
monitoring
Strad
that
exists.
B
So
we
did
talk
about
sensu
at
how
it
complements
a
Prometheus
or
telemetry,
based
approach
to
monitoring
and
specifically
to
monitoring
kubernetes
and
whether
we
don't
really
talk
about
what
those
strategies
are.
So
we
think
kind
of
a
an
observability
strategy
for
kubernetes
involves
some
amount
of
remote
polling.
This
would
be
a
traditional.
B
This
pattern
that
is
sidecars
and
well,
first
answer
the
question
in
what
are
our
sidecars
and
preparing
these
slides
I
kind
of
paused
for
a
moment
and
acknowledged
or
or
realize
that
sidecars
in
and
of
themselves
as
a
pattern,
are
not
actually
a
formal
convention
of
kubernetes
and
so
a
little
bit
of
research
via
the
Google's
found.
You
know
the
the
only
canonical
reference
to
a
sidecar
is
here
with
air
quotes.
B
Thankfully,
I
have
my
video
on
so
we
can
show
the
air
code
air
quotes,
but
this
concept
of
pods
that
are
running
together
are
a
number
of
containers
that
you
would
compose
together
in
a
pod
where,
where
one
or
more
of
those
containers
are
providing
some
kind
of
infrastructure
service,
this
is
where
this
concept
of
the
sidecar
comes
from.
You
have
a
comment.
A
I
did
it's
it's
funny
to
me
like
there
might
even
be
people
on
this.
This
call
right
now
that
don't
even
understand
like
what
a
sidecar
like
what
is
named
after
right,
like
right,
motorcycles
used
to
have
cars
attached
to
them
like
little
carts
that
people
said
in
and
they
did
nothing
in
the
other
person
drove
a
motorcycle.
Well,
now
sight
cars
are
actually
doing
a
ton
of
things
and
yeah.
Let's
keep
going
I
like
it,
yep.
B
This
that
I
think
one
of
the
earliest
references
to
the
sidecar
pattern.
This
may
even
have
been
on
the
CNC
F
blog
goes
a
few
weeks
ago
and
I
put
these
slides
together
and
I.
Didn't
leave
myself
a
speaker
of
milk,
so
I
thought
for
sure.
I'd
remember
this,
but
it
was
like
2015
I
want
to
say
and
the
other
in
this
blog
post.
It
mentioned
two
or
three
other
patterns
that
were
popular
at
the
time
that
were
not
baked
into
the
kubernetes
platform
and
I.
B
Think
one
of
them
is
now
kind
of
like
a
proper
supported,
workflow
or
pattern
and
kubernetes.
The
other
two
have
long
since
been
left.
You
know
in
the
past,
where
the
the
the
community
around
community
is
is,
is
moving
fast,
obviously
experimenting
with
various
different
workflows.
It's
still
early
days,
I
think,
and
so
not
everything
sticks,
but
sidecars,
it
seems,
are
here
to
stay
yeah,
which
is
this
kind
of
interesting
basis
for
our
discussion
today.
So
what
are
some
of
the
benefits?
I'll
be
curious
to
get
your
thoughts
on
this
and
the
next
slide
Chris.
B
But
you
know
from
our
perspective,
we
think
you
know,
could
you
could
you
bake
some
of
the
functionality
or
contents
or
services
of
a
service
that
you
might
run
in
a
sidecar
into?
You
know
a
single
container?
Are
we
then
trying
to
like
build
these
little
mini
monoliths
inside
of
our
micro
service
architecture?
Certainly
you
could,
but
breaking
those
things
apart.
You
get
this
nice,
like
a
modular
composable
and
all
the
all
the
nice
attributes
that
you
about
when
you
think
about
micro-service
architectures,
apply
I,
think
one-to-one
to
this
sidecar
pattern
in
kubernetes.
B
Cool,
so
then
what
are
some
examples
of
things
that
happen
inside
cars,
and
this
is
certainly
not
the
exhaustive
or
canonical
reference
for
things
that
you
can
do
with
side
cars,
but
some
of
the
more
popular
or
obvious
ones
that
we've
seen
are,
if
you've
heard
about
the
service
mesh.
Very
often
the
service
mass
implemented
as
a
sidecar,
and
there
are
various
logging
solutions
or
logging
platforms
that
have
agents
that
want
to
be
run
as
side.
B
Cars
and
sensu
is
by
far
not
the
only
monitoring
solution
that
provides
an
agent
which
can
be
run
as
a
side
car.
So
you
have
this
nice
one-to-one
pairing
of
a
monitoring
agent
per
you
know,
collection
of
services
that
works
pretty
well
so
before
I
get
before
I
continue.
I
promised
I
would
pause
momentarily
as
we
went
through
the
presentation
and
look
at
the
Q&A.
So
let
me
see
four
questions
yet
none!
Yet
that's
good!
C
B
Okay,
so
let's
do
the
old,
you
know
tell
them
what
you're
gonna
show
them
show
it
and
then
talk
about
it
things
so,
for
the
first
part
of
our
demo,
I'm
just
going
to
deploy
an
example
application
into
kubernetes
using
deployment,
Hayes
deployments
all
well.
Look
at
the
deployment
you
animal,
the
example.
Application
is
just
going
to
be
an
inch
and
x-default
nginx
web
site,
so
look
at
the
deployment
yeah
mo
will
deploy.
B
It
will
then
go
see
our
live
running
app,
and
this
can
kind
of
bring
us
to
the
place
that
maybe
a
lot
of
the
people
who
are
are
attending
today
and
people
who
may
be
interested
in
this
particular
topic
may
have
reached,
which
is
cool,
I've
deployed
my
app
or
service
into
communities.
Now
I
have
to
figure
out
how
to
monitor
this
thing,
and
maybe
you
had
a
different
workflow
or
different
tools
for
doing
that
outside
of
this
kind
of
cloud
native
context
that
don't
necessarily
work
very
well
in
that
highly
ephemeral
compute.
B
A
B
This
is
intentionally
extremely
simple
here
we
have
a
service.
The
nginx
container
by
default,
wants
to
run
on
port
80
I'm
gonna
expose
that
outside
of
the
communities
cluster
at
port,
8000
and
then
I
have
an
extremely
simple
deployment,
and
this
deployment
is
just
running
the
the
upstream
nginx
container,
just
the
latest
iteration
of
it
and
configuring
that
port.
So
let's
go
ahead
and
deploy
this,
and
you
can
see
on
the
bottom
here
bottom
pane
of
my
terminal,
I'm
just
running
the
Linux
watch
command.
B
So
we
can
see
live
the
current
state
of
the
pods
running
in
this
kubernetes
cluster
and
on
the
top
I
will
be
issuing
my
commands.
You
can
also
see
it's
outside
of
scope
of
our
demo
today,
but
we
will
provide
as
one
of
the
follow-up
action
items
the
materials
not
sure
at
the
end,
instructions
for
how
to
deploy
sensu
into
kubernetes,
but
I
have
my
sensor
here
running
in
kubernetes
and
I
actually
have
as
a
companion
to
sensu
in
Flex
DB
as
time
series
database
and
graph
on
ax
for
building
custom,
graphing
dashboards.
B
So
I
have
like
a
reference
architecture,
telemetry
pipeline
since
you
as
the
collection
and
then
the
other
components
there.
Okay.
So
let
us
to
make
to
help
myself
avoid
any
embarrassment.
I
created
these
little
scripts,
but
let's
just
all
do
the
verbose
way
so
I
have
created
a
namespace
in
community
is
called
webinar.
That's
where
I
will.
B
B
We
will
go
to
our
browser
and
observe
at
port
8000
wallah
I've
just
launched
my
you
know:
fancy
new
service
I'm
live
on
the
Internet,
all
power
by
kubernetes,
very
cool.
So
again,
the
the
purpose
of
this
step
in
our
demo
is
just
to
kind
of
find
common
ground
with
user.
So
if
you
have
reached
this
stage
in
your
your
kind
of
application
operations,
lifecycle
onboarding
onto
kubernetes
are
now
interested
in
in
methods
to
monitor
that
service.
B
This
is
where
we
will
try
to
make
interesting,
so
I
have
a
secondary
I
have
a
secondary
communities
deployment.
Oh,
this
is
identical
to
the
first
one
we've
shown,
but
this
is
to
show
what
is
involved
in
in
adding
the
sensu
sidecar
to
your
application
deployments,
and
this
works
with
kubernetes
deployments.
It
works
with
all
of
the
all
the
controllers,
so
it
works
with
deployments.
Stateful
sets
daemon
sets
and
so
on.
B
So
to
add,
since
agent
sidecar
I
just
add
another
container
in
the
spec
template,
spec
dot,
containers,
scope
of
our
deployment,
yamo
and
I'm
gonna
use
the
latest
version
of
sensu
and,
as
you
can
see,
a
hundred
percent
of
sensitives
configuration
is
exposed
in
a
very
cloud
native
friendly
manner.
So
that
sense
we
agent
you-
can
configure
it
via
like
command
line
flags,
so
I
can
provide
some
of
the
configuration
that
way.
B
I
can
pride
all
of
it
that
way
if
I
wanted,
but
also
we
expose
100%
of
the
since
we
agent
configuration
as
environment
variables.
So
you
have
this
very
communities
native
declarative
configuration
workflow
available
to
you
to
configure
how
you
are
running
the
sensory
agent
in
how
you're
configuring
the
sensor
agent.
So
this
is
the
configuration
I
need
to
for
my
adding
my
sidecar
and
now,
let's
deploy
it
so
I
will
you'll
notice
too.
That
I
have
this
rolling
update
strategy
here.
Let
me
actually
slow
this
down,
so
we
could
talk
about
it.
C
B
B
B
B
Ok,
now
the
other
cool
thing
about
this
is
now
I
have
now
that
I
have
this
configuration
in
place.
I
can
also
scale
this
thing.
So
let
me
scale
my
deployment
and
I'm
going
to
say,
let's
say:
10
replicas
as
I
do
that
I
can
see
live
again.
Kubernetes
is
provisioning
new
compute
instances.
This
is
all
pretty
easy,
I
think
from
a
monitoring
perspective,
a
lot
of
tools
will
do
this
very
well.
The
thing
that
sensor
does
exceptionally
well
is
not
only
the
automated
registration
but
also
the
automated
D
registration
of
systems.
Out
of
monitoring.
B
So
again,
this
works
cross-platform.
So
for
your
traditional
bare-metal
compute,
traditional
on-premise
virtualization,
multiple
public
cloud
providers
and
even
in
cloud
native
environments,
we
can
tell
the
difference
between
a
node
being
down
and
one
that's
been
deep
provisioned.
So
that's
a
whole,
probably
topic
for
separate
conversation.
B
Ok,
I
see,
there's
a
question
that
looks
like
a
great
question
to
answer
now,
instead
of
waiting
for
the
end.
So
the
question
is:
what
kind
of
monitoring
agent
do
in
a
sidecar
that
it
cannot
do
in
its
own
pod?
So
some
great
question,
and
probably
one
that
you
can
help
me
with
Chris
and
the
answer.
The
question
is
that
you
you
get
this
one-to-one
mapping,
so
you
so
so
part.
B
One
of
the
answer
is
for
every
instance
of
an
applicative
APOD
which
can
be
a
collection
of
services
and
in
some
ways
you
can
almost
mindmap.
You
know
a
pod
to
what
you
might
have
provisioned
VM
for
previously
right.
So
if
you
had
a
microcircuit
architecture
that
said
we
have
I
did
we
have
compute
instances
that
have
a
role
that
role
says
it's
going
to
run.
You
know
XY
and
Z
service.
You
would,
of
course,
have
had
things
like
monitoring
agents
and
whatever
else
on
those
compute
instances
as
a
one-to-one
mapping.
B
So
even
in
a
clustered
service
you
would
have
you
know
this
complete
set
of
of
services
to
make
up.
You
know
an
individual
instance
of
a
you
know
of
a
service
and
that
concepts
you
can
kind
of
carry
over
into
communities
using
the
pod
concept
and
the
reason
why
it's
beneficial
and
why
it
works.
Is
that
all
the
containers
in
a
pod?
This
is
the
part
that
you
can
help
me
with
Chris.
All
the
containers
in
a
pod
have
a
shared
network
address
space,
so
they
could
talk
to
each
other
via
localhost
right
run.
A
A
You
know
thing
to
do
long
term
strategy
wise,
so
putting
these
side
cards
in
or
putting
a
collection
of
pods
to
putting
a
collection
of
containers
in
a
single
pod
allows
you
to
say:
oh
I,
have
this
tiny
little
cash
like
you
know,
whatever
bread
is
cash
running
with
this,
you
know.
Go
application
and
I
need
to
make
sure
that
everything
is
running
within
a
certain.
A
You
know
request
for
a
second
ratio,
the
best
way
to
get
that
is
from
that
pot
itself,
not
to
add
some
latency
to
that,
then
you
know
automatically
scale
up
and
down
the
size
of
your
pods,
so
this
provides
a
real-time
or
near
real-time
and
then
be
very,
very,
very
close
and
granular
access
to
what
is
running
in
a
pod,
yep
well
said,
and
I
just
went
back
and
checked.
The
children's
illustrated
guide
to
kubernetes
did
not
mention
sidecar,
but
it
did
mention,
contain
pods
can't
contain
multiple
containers,
so
don't.
B
B
I
see
I
see,
one
of
the
questions
was
volunteer.
Answer
from
the
attendees
to
the
question
we
just
answered.
I
think
we
covered
that
there's
another
one.
That
says:
is
there
any
way
we
can
separate
sensitive
configuration
from
the
original
app
pod,
so
the
developers
do
not
think
need
to
think
about
the
monitoring
configuration
there's
a
couple
different
ways
to
approach
this.
B
If
I
go
back
to
look
at
that
deployment,
yamo
I
think
what
the
question
is
saying
is:
if
I'm,
a
developer
and
I'm
responsible
for
you
know
deploying
my
service
on
kubernetes
I'm
gonna
be
responsible
for
this
part
of
my
provisioning
template
right.
This
is
all
the
configuration
that's
needed
actually
to
run
my
app
or
my
particular
service
and
the
contract
between
the
you
know:
application
engineering
teams
or
service
engineering
teams
and
the
operations,
engineering
teams
or
your
site
reliability.
B
Engineers,
sometimes
is
that
you
know
that
the
contract
is,
we
will
take
care
for
the
monitoring.
You
just
give
us
the
the
thing
to
deploy
and
the
they.
There
are
a
couple
different
answers
there.
One
of
them
is
on
our
roadmap
doesn't
exist
yet
today,
but
that
is
the
concept
of
I
think
it's
called
an
admission
controller
in
kubernetes.
So
what
this
would
do
is
you
would
have
a
service
that
lives
inside
of
your
kubernetes
cluster.
B
That
says
that
it's
kind
of
like
the
gatekeeper
for
things
that
are
provisioned
into
your
into
kubernetes
and
it
can
apply
rules
to
the
services
that
are
provisioned
and
one
of
those
rules
is
hey.
We
want
a
sense
of
agent
to
be
a
central
agent
sidecar
to
be
present
in
every
single
pod
within
these
namespaces
or
something
along
those
lines,
and
then
you
can
apply
the
configuration
of
the
sensu
agent
in
that
way.
B
Some
other
things
that
we
can
do
are
you
can
move
the
what's
available
today
is
all
this
configuration
can
also
live
in
a
config
file,
so
you
can
you
can
make
that
configuration
file
available
as
a
config
map
inside
of
the
kubernetes
cluster,
and
then
you
can
volume
mount
that
config
map
into
the
sense
of
agent,
so
that
would
eliminate
a
lot
of
this
configuration
you
have
about.
You
know
four
or
five
lines
of
boilerplate
of
a
boilerplate
here,
just
saying:
I'm,
adding
the
sensory
agent
and
I'm
volume
mounting
this
config
file.
I,
don't
know!
B
What's
in
that
config
file,
cuz,
that's
maintained
by
someone
else,
but
that's
what
I've
been
told
to
do
to
add
monitoring
to
my
apps
so
and
there's
quite
a
few
other
strategies
we
could
discuss,
but
probably
out
of
the
scope
of
the
conversation
today.
So
hopefully
that's
a
helpful
answer
to
that
question.
Okay,
let's
resume
the
demo
I
see
that
questions
are
continuing
to
pour
in.
Hopefully
that
means
where
we've
found
ourselves
a
nice
relevant
conversation
because
I
haven't
actually
monitored
the
thing
yet
right.
All
I
did
was
just
say
how
to
get
started.
B
C
B
I
turned
it
on
I
turned
it
on,
but
I'm,
not
actually
yep.
Okay,
my
work
is
done,
go
at
home.
So
now,
let's
talk
about
actually
monitoring
that
thing.
So,
let's
go
to
another
demo
and
in
this
demo
I'm
going
to
talk
about
a
few
things.
One
is
and
I'm
now
realizing
in
real-time
I,
probably
could
have
provided
a
little
bit
of
an
architectural
diagram
here,
but
one
is
I
want
to
talk
about
the
the
architecture
of
sensu
and
the
compile
the
server
systems
communicate.
B
So
our
sense
you
agent,
communicates
with
the
sensitive
back-end
in
a
pub
sub
model
of
communication.
The
agent
actually
only
requires
outbound
network
access,
so
we
don't
have
to
open
ports
in
the
agent
or
on
the
pod
or
on
a
virtual
machine
or
whatever
in
order
to
for
the
monitoring
back
in,
like
the
monitoring
control
plane
to
have
access
to
the
individual
agents.
So
the
agents
just
have
an
outbound
connection.
B
So
if
it
loses
connectivity
with
one
back-end,
it'll,
just
failover
to
another
one,
and
when
that
agent
connects
the
eight
it
says-
hey
I'm,
a
you
know,
it
kind
of
describes
itself
the
roles
of
that
particular
agent
and
in
the
case
of
a
sidecar,
it
might
say,
hey
I'm
running
on
I'm
on
kubernetes,
I'm
a
docker
container
and
the
role
of
this
thing
is,
in
our
case
a
nginx
service
so
or
a
web
service
or
hopefully
are
not
running.
Stateful
workloads
and
kubernetes
is
also
a
topic
for
another
day.
B
But
you
know
on
a
virtual
machine,
you
might
say:
hey
I'm
a
my
sequel,
server
or
postgrads
database,
or
you
know
whatever
the
case
my
so
the
agents
will
subscribe
to
topics
for
each
of
those
we
call
them
subscriptions.
So
the
agent
will
subscribe
to
a
number
of
subscriptions
on
the
back
end
and
then
that
back
end
as
your
control
plane
is
where
we
will
configure
the
monitoring
to
say:
hey
I
want
to
collect
this
monitoring
data
from
you
know
from
all
the
members
of
my
nginx
subscription.
B
That's
the
demo
we're
going
to
show
today
and
what
else
do
I
have
we're
gonna
configure
a
very
simple
check
and
in
fact
we
are
I'm
going
to
use,
which
is
hopefully
a
little
bit
surprising,
we're
going
to
set
to
provide
support
for
a
wide
variety
of
plugins
from
all
the
way
from
old,
Nagios
style
collection
plugins
to
we
have
in
our
agent
stats
the
socket.
So
if
your
apps
are
configured,
let's
say
for
custom
metrics
on
data
dog
and
you
need
to
use
something
other
than
data
dog
or
something
behind
the
firewall.
B
You
just
point
those
things
at
your
localhost
stats,
a
socket.
They
each
collects
those
metrics.
We
also
have
support
for
collecting
metrics
from
Prometheus
endpoints,
but
for
the
demo
today,
I'm
actually
gonna
use
an
old
Nagios
plugin
and
the
old
check
HTTP
plugin.
We
think
a
lot
of
a
lot
of
operators
and
engineers
have
some
background
with
Nagios
for
for
better
or
for
worse
and
have
some
familiarity
with
that
plugin.
So
I'm
gonna
show
how
that
can
be
used,
even
in
this
modern
context,
both
to
check
the
health
of
our
service
and
to
collect
metrics.
B
To
make
a
nice
pretty
telemetry
dashboard
here,
ok,
so
we
already
have
those
subscriptions
in
place
so
now
I
just
need
to
provide
the
sense
of
configuration.
So
let's
look
at
what
that
looks
like
so
in
7
to
go.
If
there's
any
attendees
who
are
familiar
with
the
older
version
of
synthesis,
this
is
gonna,
be
new
in
sense.
You
go.
We
have
a
couple
of
new
features
to
talk
about,
one
of
which
is
also
the
sensu
configuration
file
format,
which
looks
should
look
very
familiar
to
kubernetes
users.
B
Our
configuration
can
be
provided
as
gamal
or
json
files.
There
are
four
top-level
attributes
for
every
single
sensu
resource
type,
an
API
version,
metadata
and
spec
I.
Think
if
you
change
this
from
type
to
kind,
it
would
look
like
kubernetes,
Yambol,
and
so
here
in
this
configuration
file,
I
am
registering
a
handful
of
monitoring
plug-ins.
These
are
called
assets
in
census.
So
these
are.
These
are
any
HTTP
vegetable,
tarball,
optionally
gzipped.
You
have
to
require
sha-512,
so
we
can
verify
the
integrity
of
the
asset
that
we're
fetching.
B
We
register
these
in
the
control
plane
in
the
sense
you
back-end,
and
then
we
associate
them
with
our
monitoring
check
configuration.
So
this
is
monitoring
check
in
sense.
You
and
you
can
see
I'm,
giving
it
a
name.
I'm
saying
this
is
the
command
that's
going
to
be
executed,
so
the
sense
of
agent
is
actually
just
going
to
execute
a
service
check.
I'm
publishing
this
request
on
a
10-second
interval
to
the
nginx
subscription,
so
any
weather.
B
If
it's
a
container
or
a
virtual
machine
or
a
bare-metal
host
any
agent,
that's
a
member
of
that
subscription
will
receive
this
request
every
10
seconds
and
execute
the
check
and
then
I'm
referencing
those
plugins.
That
will
be
needed
for
the
agent
to
execute
this
check.
I
don't
have
these
installed
in
my
sensor
agent,
so
they
will
need
to
download
these
assets
one
or
more
of
these
assets.
B
In
order
to
to
complete
this
check
and
then
I'm,
informing
sensu
that
hey
the
output
of
this
check
contains
some
salami
tree
data
in
Nagios,
perfect
data
format
and
I
would
like
to
convert
it
and
write
it
into
influx
dB.
So
that
is
an
overview
of
the
since
you
check
configuration
and
now
let's
go
look
in
our
since
we
dashboard
we'll
see
I,
don't
have
any
sense,
you
checks,
configured
and
so
I
will
now
say
sensu
cuttle,
which
this
allows
me
to
also
to
introduce
our
command
line.
B
You
utility,
which
again
should
look
and
feel
very
familiar
for
communities,
users,
and
since
you
we
also
have
the
similar
concepts
of
namespaces
and
our
back
that
are
back
includes
rolls
and
cluster
rolls
and
roll
bindings
and
cluster
roll
bindings.
So
again,
some
familiarity
there
and
I'm
gonna
use
this
same
even
create
F
workflow
here
and
go
ahead
and
create
that
nginx
monitoring
check
and
so
in
real
time.
I
see
that
configuration
appear
in
the
sense
of
dashboard.
I
could
come.
B
You
know
if
I'm,
a
different
kind
of
user
that
doesn't
like
the
Grand
Line
I
could
actually
come
here
and
edit.
This
check
configuration
or
create
new
ones,
modify
it
in
some
way.
But
if
I
go
to
my
sensory
dashboard
and
I
say
show
me,
you
know
all
the
results
of
this
check.
It's
already
been
executed
on
all
three
of
those
pods
and
I
see
this
old-school.
Nagios
output
here
Chris
looks
like
I,
see
a
little
smirk
on
your
face,
like
maybe
you
you've
seen
this
movie
before.
C
A
B
There's
there's
good
in
bed,
you
know
I,
think
a
tool
like
Nagios
couldn't
be
around
for
20
years.
If
it
didn't
do
something
right,
exact
and
the
thing
that
we
think
that
Nagios
got
very
right.
Was
this
concept
of
the
service
check
right?
It's
very
easy
to
reason
about
how
to
write
a
little
monitoring
script,
and
now
you
know
we
wrap
some
nice
ease
around
it
like
a
simple
packaging
and
distribution
mechanism
that
we
provide
and
sense
you
and
you
can
do
cool
things
with
it.
B
Okay,
so
I
want
to
eventually
have
a
dashboard
to
show
for
all
this
and
incentive.
We
do
that
with
with
what
we
call
handlers
so
Senzu
we
think
about
as
like
a
monitoring
event
pipeline
and
so
that
the
event
reference
there,
it's
kind
of
a
nod
to
sensors
architecture
and
that
event
data
can
be
service.
Health
checks,
like
is
this.
Its
availability
information
right,
like
is
this
thing
up
or
down,
is
healthy.
It
can
also
be.
B
It
can
also
be
telemetry
data
and
the
discovery
data
we
already
saw
right.
So
since
you
kind
of
keeps
track
of
hey
here's,
a
new
node
or
that
node
has
gone
away,
and
then
we
can
use
handlers
to
process
that
data.
We
have
a
in
flux,
DB
Handler,
we're
going
to
configure
here
to
pipe
the
telemetry
data
that
we're
collecting,
even
though
it's
in
Nagios
format
convert
it
to
influx
DB
format
and
write
it
in
to
influx
DB.
So
I'll
do
that
here,
like
this
webinar.
B
Now
that
I
have
registered
the
configured
sense,
you
registered
those
resources
and
configured
sensu
to
start
sitting
on
telemetry
data.
I
will
start
to
see
metric
data
points
in
my
dashboard.
So
we
give
this
a
second.
My
dashboard
is
refreshing
on
a
five-second
interval.
Our
metrics
are
being
collected
on
a
10-second
interval.
I
have
now
a
complete
end-to-end
telemetry
pipeline
in
this
very
modern
context
that
is
kubernetes
using
you
know.
I
did
it
didn't
require
you
know
heart
transplant
surgery.
B
So
if
I'm
coming
from
a
place
where
I
was
using,
you
know
Nagios
still
or
you
know
any
of
the
Nagios
derivatives
that
have
popped
up
over
the
years,
which
is
a
surprising
number
of
organizations
and
now
I'm
moving
those
workloads
into
the
cloud
native
compute
environments
and
need
a
solution
for
wait.
A
second
Nagios
isn't
going
to
work
in
this
environment,
but
I
got
to
throw
the
baby
out
with
the
bathwater.
B
So
now,
when
I
do
that
again,
those
I
get
those
seven
additional
instances
of
this
app
making
my
service
more
highly
available,
and
that
means
I
have
sidecars
in
all
of
those
places
and
as
the
sensory
agent
side
cars
come
online,
they
don't
have
the
plug-in
I
need
to
perform
this
monitoring
check
yet,
but
since
it
takes
care
for
all
that,
so
the
agents
receive
this
request
to
perform
this
healthcheck.
They
execute
the
request.
B
They
send
the
results
back
to
the
sent
to
back
into
this
pipeline
and
we
go
and
take
that
data
and
do
as
you
wish
with
it.
Some
of
the
other
handler
things
you
can
do
with
sensu
is
obviously
send
alerts,
so
we
have
like
eh
duty
handlers
that
would
just
notify
you
up.
A
thing
is
down.
If
you
want
to
write
data
into
elasticsearch
or
Splunk
instead
of
time-series
database
like
in
Flex
DB,
we
have
solutions
for
that.
The
the
possibilities
are
really
very
endless
and
but
that's
also
a
topic
for
a
different
day.
A
B
So
the
agent
actually
in
receiving
the
output
of
executing
the
script
has
been
informed
that
hey
that
output
can
this
output
string
I'm
seeing
here
contains
some
symmetry?
Did
that
I
know
how
to
parse
so
its
extracting
those
two
metrics
into
a
sense,
ooh
internal
format?
And
then
we
send
that
that's
into
like
a
JSON
data,
payload,
that's
how
the
data
flows
through
the
sensu
pipeline
and
then
on
the
back
end.
We
have
a
handler
so
the
if
we
go
back
and
look
at
our
check
configuration.
B
So
we
told
sensu,
hey
I,
want
to
use
the
influx
DB
Handler
and
then
we
configured
in
flux,
DB
handler
here.
This
is
the
name
of
it,
which
is
this
version
of
it
is
just
actually,
you
know,
piping
out
to
execute
a
plug-in
that
is,
writing
data
to
the
to
my
influx
DB
back
in
with
super
secure
credentials.
So
please,
no
one
hack,
my
demo,
environment
and-
and
you
know,
I-
can
also
further
apply
since
it
provides
a
like
a
transformation
step
in
the
pipeline.
B
So
if
I
want
to
apply
a
mutator
to
further
mutate,
the
payload
that's
flowing
to
the
pipeline
before
it
reaches
my
handler.
That's
going
to
process
that
data
by
writing
it
to
the
datastore.
Then
I
could
do
that
which
I
have
configured
here.
These
resources
are
available
in
a
new
service.
That's
in
Veda,
currently
that
we
call
bonsai.
So
this
is
kind
of
like
the
the
docker
hub
for
for
sensu
and
if
I
come
here
and
search
for
in
flux,
DB
I'll
see
that
in
flux,
DB
Handler
and
which
gives
me
some
more
documentation.
B
That
explains
exactly
how
the
thing
works
and
what
configuration
is
available
for
it.
But
then,
if
I
come
and
use
the
query
editor
here
in
Prometheus
and
I'm,
looking
at
which
metrics
are
available
in
I'm.
Sorry
on
Prometheus
in
grow
fauna
and
this
graph
fauna
is
connected
to
my
influx
DB
back
end
when
I
go
use.
B
A
B
So
if
there
are
other
questions,
I
can
answer
them
now,
let's
see
in
yeah
in
your
example,
what
kind
of
health
checks
are
you
doing?
I
want
to
understand
what
these
can
complement
on
top
of
the
cube
state,
metrics.
So
a
great
question,
the
cube
state
metrics
project,
and
let
me
see,
if
going
to
the
googles
in
a
webinar,
it
could
be
dangerous,
but
let's
try
it.
You
think
sorry
metrics,
so
the
cube
state
metrics
project
is
a
Prometheus
exporter.
Oh
yeah
good
call.
B
Going
whoa
there,
you
go
little
delay
there
extra
large,
so
this
is
an
add-on
for
kubernetes
that
exposes
metrics
about
the
communities
cluster.
So
each
of
the
cubelet
backends
has
a
slash,
metrics
API
I
forget
what
the
port
that
it
runs
on
is,
but
that
exposes
metrics
about
the
individual,
kubernetes
node
and
that's
you
know,
resources
a
number
of
jobs.
It's
running
and
this
sort
of
thing
all
telemetry
data
cube
state
metrics,
is
gonna
expose
that
expose
other
data
about
kubernetes
resources
across
the
expanse
of
the
cluster.
So
this
is
at
the
control
plane
level.
B
It's
giving
me
performance
data
about.
You
know
the
total
number
of
jobs,
the
you
know
all
the
kinds
of
metrics
about
the
duration
of
those
jobs
on
Canaries
and
whatnot,
but
it's
not
actually
giving
me
a
very
granular
telemetry
data
or
metrics
about
their
workloads
that
I'm
running
on
top
of
kubernetes.
Let's
see
does
maybe
this
even
what
metrics
does
this
give
us
I.
C
C
A
Know
if
I'm,
that
DevOps
engineer
of
old,
that
I
used
to
be
and
I
need
to
toe
they
hey
I've
got
a
bad
pod
somewhere,
which
ones
performing
poorly.
That's
not
going
to
come
from
that
right,
like
it's
correct,
something
like
Senzu
or
I'm.
Gonna
have
some
other
thing
in
place
to
say.
Oh,
this
is
a
noisy
log
or
a
low
number
of
connections
over
here
correct.
C
B
And
what's
nice
about,
you
know
and
I'm,
not
gonna,
try
to
repeat
everything.
We
said
in
our
blog
series,
we're
big
fans
of
Prometheus
actually
at
sensu,
maybe
not
something
you
would
expect
from
what
appears
to
be
competing
open-source
projects.
But
you
know
Prometheus
is
a
wonderful
telemetry
folk
monitoring
solution
and
the
model
is
actually
quite
interesting.
B
How
and
there
are
some
nice
things
about
it
that
make
a
lot
of
sense
in
a
cloud
native
computing
environment
like
kubernetes,
we,
you
know
you
have
this
orchestration
and
abstraction
between
the
you
know
the
apps
the
way
you're
developing
in
running
the
thing
and
a
great
answer
in
some
cases
to
the
question
that
we
answered
earlier,
that
was
that
was
about
hey
I.
Don't
want
the
developers
to
have
to
think
about
modifying
their
deployment.
Yeah
well
to
know
like
they
don't
need
to
know
we're.
Monitoring
is
right.
B
Prometheus
is
great
for
that,
because
they
could.
You
can
instrument
your
applications.
You
expose
metrics
and
you're
done,
like
you
can
say,
hey
I
did
my
job
the
metrics
for
my
app
are
available.
Something
else
is
gonna
come
along
and
scrape
them.
That's
great
for
again,
but
that's
telemetry
focus.
So
if
you
need
also
that
kind
of
functional
service
health
checking,
you
sometimes
you
know
sometimes
try
and
do
that's
like
that.
You
know:
I
have
this
hammer.
I'm
gonna
smash
all
the
problems
with
it.
Sometimes
it
doesn't
make
sense
to
map
you
know.
B
A
B
If
you
need
a,
if
you
need
a
functional
survey,
self
checking
now
you
you
have
to
look
elsewhere,
the
other
trade-off
between
a
Prometheus
and
sensors
approach.
I
think
is
since
who
gives
you
a
a
single
sidecar
solution,
whereas
if
you
had
services
running
as
containers
in
a
pod
and
each
one
of
them,
you
had
a
you
had
a
Prometheus
exporter
for
those
are
like
single
purpose
agents
right.
B
But
opening
ports
in
our
network
is
a
non-starter
for
various
compliance
or
security
reasons
and
whatnot,
and
they
would
have
to
audit
the
security
of
each
individual
exporter,
as
opposed
to
you
know,
there's
a
smaller
surface
area
when
you
have
a
single
like
multi-purpose
agent,
like
you
get
from
sensu,
and
sometimes
in
our
blog
series.
We
talked
about
there's
like
a
very
good
companion
solution
there,
which
is
use
the
Prometheus
exporters
and
use
this
local
sensory
agent
to
locally
scrape
the
you
know
so
locally
scrape
those
metrics
shove
them
onto
the
pipeline.
A
B
There
is
not
assess
available
work
and
sensor
has
only
been
on
premise.
This
is
a
funny
question
for
us,
because
95
plus
percent
of
our
customers
are
running
sensu
in
the
public
cloud
somewhere.
So
on
premise
for
incentives
use
case
because
it
was
you
know,
since
he
was
designed
as
a
you
know,
since
he
was
cloud
native
before
there
was
that
the
term
cloud
native
right,
since
he
has
only
known
a
world
in
which
public
cloud
arrested
and
that
was
kind
of
a
problem.
B
B
But
the
nice
thing
is,
you
know
solving
that
original
problem
we
sought
to
solve,
which
was
how
do
I
monitoring
is
kind
of
like
hey?
Is
this
thing
working
right?
It's
like
Big
Brother
tapping
on
your
shoulder
and
in
when
you
have
this
ephemeral
compute
and
that
thing
can
suddenly
go
away
the
old-school
style
of
marketing.
That's
like
crap!
This
thing
went
away.
I
need
to
go
wake.
Somebody
up
doesn't
really
make
sense.
They
need
monitoring
that
understands
this,
like
ephemeral,
compute,
environment
and
this
infrastructure,
as
code
development
workflows
and
how
to
go.
B
You
know,
since
we
can
do
things
like
hey
I,
think
this
node
might
be
down,
I
should
probably
wake
somebody
up.
Let
me
go
ask
an
external
source
of
truth
first
and
go
to
the
kubernetes
api
or
the
ec2
api
and
say
hey
external
source
of
truth.
What
do
you
think?
Do
you
think
this
thing
should
still
be
running
and
easy
to
says
back?
Oh
no.
I
terminated
that
instance.
I
don't
think
it
should
be
running,
so
we
can
safely
remove
that
system
from
monitoring
and
not
go
awake.
B
B
That's
our
sense
of
community
slack
or
our
discourse
channel
is
probably
the
best
place.
You
can
go
there
post
a
question
or
usually
pretty
timely,
about
responding
to
those
questions.
If
there
are
unanswered
questions
at
the
end
of
this
webinar,
please
feel
free
to
go
post
them
there.
If
you
or
we
might
beat
you
to
it,
so
we
will
usually
take
the
unanswered
questions
from
our
webinars
and
post
the
question,
along
with
an
answer
to
create
some
opportunity
for
follow-up
discussion.
I
showed
briefly
our
new
hub
for
sharing.
B
Since
you
plugins,
there's,
there's
a
lot
of
content
to
be
published
there.
Leading
up
to
our
event,
will
have
some
announcements
around
that
at
our
upcoming
event
and,
of
course,
there's
Q
Khan
coming
up
so
not
only
will
sent
to
you
the
company,
be
there
I
myself
also
will
be
president
cube
Khan.
So
if
you
have
follow-up
questions
that
you
want
to
ask
me
in
person,
I
would
love
to
hang
out
with
you
and
say
in
sunny
San,
Diego,
I
I.
B
A
Yep
yeah
I
think
you're
good
yeah
get
your
education
on
show
up
for
the
contributors
summit
early.
If
you
want
that's
my
contribute,
X
plug
how
about
that
awesome.
Cool
thanks
for
thanks
for
everybody
for
joining
us
today,
really
appreciate
your
time
today.
Caleb
thank
you
for
educating
everyone
on
Senzu
sidecars
and
many
many
things
of
metrics
and
telemetry.
The
webinar,
recording
and
slides
will
be
online
later
today.
Take
some
time
to
convert
the
zoom.