►
From YouTube: From New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes - Eddie Zaneski
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
From New Cluster to Insight. Deploying Monitoring and Logging to Kubernetes - Eddie Zaneski, DigitalOcean
The question that most people ask after spinning up their first Kubernetes cluster is "how do I do monitoring and logging". In this session we'll utilize open source tools like Prometheus, Helm, Grafana, and Loki to quickly go from being in the dark to having full visibility into the happenings of our clusters and workloads. We'll deploy an entire monitoring and logging stack to a fresh cluster live and discuss the pros and cons of such tools. You'll leave this session with a playbook on how to bootstrap your cluster observability and some gotchas and lesson's learned from our monitoring of large workloads at DigitalOcean.
https://sched.co/MPba
A
Hello
again,
everyone
good
to
see
so
many
people
here
today.
We're
going
to
hear
about
monitoring
and
logging
in
kubernetes
I
was
on
the
program
committee
and
was
super
keen
on
this
talk
because
spoiler
alert
well,
it's
in
the
abstract,
so
you've,
probably
all
seen
it
references
a
project
that
I
started
called
Loki,
so
Eddie's,
gonna
kind
of
show
you
how
to
use
my
code,
which
I'm
also
super
keen
so
round
of
applause,
Eddie.
B
Thank
you,
Tom
how's,
it
going
everyone.
My
name
is
Eddie's
an
esky
and
I
served
the
developer
community,
a
company
called
digital
ocean
quick
show
of
hands.
Who
here
has
used
a
digital
ocean
tutorial
before
yeah
fun
fact
we
are
not
just
a
tutorial
company.
We
are
a
cloud
provider.
So
thank
you
for
that.
B
So
today,
like
Tom
said
I'm
here
to
talk
about
logging
and
monitoring,
kubernetes
I
just
want
to
get
a
real,
quick
gauge
from
the
audience
will
do
like
a
one
to
five
one
being
you
are
very
new
to
kubernetes
or
not
have
used
it
before
and
five
is
your
name
is
Kelsey
Hightower,
so
can
can
the
ones
raise
their
hand,
real,
quick
level
of
kubernetes?
Okay,
we
got
a
couple
all
right,
twos,
please,
cool
threes,
okay,
fours,
okay,
fives
Kelsey
is
not
here.
Damn
it
cool.
Alright,
that's
super
helpful.
B
So
there's
a
lot
of
like
mid-level,
so
I
want
you
to
take
a
step
back
to.
When
you
were
a
you
know,
a
new
developer.
You
might
have
heard
of
this
thing
called
kubernetes
before.
Maybe
your
boss
walked
into
your
office
and
dropped
a
book
on
your
desk.
Like
my
dad's
boss
did
when
UNIX
first
came
out,
he
said
here:
here's
Unix,
you'll,
learn
it
so
I
imagine
you
might
have
had
some
similar
interaction
like
that
and
I.
B
Just
really
want
you
to
think
back
to
like
how
new
the
world
was
then,
and
how
scary
and
confusing
it
could
have
been
right.
And
so
your
boss
is
like
hey
we're
gonna
start
using
this
thing
called
kubernetes
I
want
you
to
learn
about
it
and
you
know:
go
spit
up
a
cluster
somewhere,
and
so
you
might
reach
for
your
favorite
cloud
provider.
So
maybe
Google
Amazon
as
your
digital
ocean
really
doesn't
matter.
You
reach.
B
So
I'm
just
gonna
pick
London
here,
let's
get
like
a
beefy
five
nodes
cool
all
right
so
that
I'm
going
to
copy
this
ID
and
so
we'll
use
this
in
a
bit
should
be
ready
and
then
I
really
want
you
to
think
after
day.
One
is
over
right,
so
you've
spun
up
your
cluster.
You
found
your
cloud
provider
you're
good
to
go.
You've
deployed
your
first
app
your
hello
world.
Maybe
you've
got
a
couple
containers
running
in
there.
You've
spun
up
some
load,
balancers
and
then
really
comes.
B
You
know
the
dawn
of
the
second
day,
and
this
is
where
you're
like.
Oh
I,
have
a
giant
black
box
that
I
have
no
idea
how
to
look
inside
right.
You
have
no
idea
you're,
your
old
methods
of
you,
know,
logging
and
monitoring
and
having
observability
to
kind
of
get
tossed
out
the
window.
Some
tools
might
be
familiar,
but
you
really
have
to
approach
this
from
a
very
different
place.
You
know
you,
you
might
be
shipping
off
to
a
log
server.
So
really
just
think
back
to
like
that.
B
First,
you
know
day
two
when
you
had
no
idea.
So
maybe
maybe
you
wind
up
on
this
awesome
landscape
that
CN
CF
dot,
IO
website
and
you're.
Faced
with
this
really
largely
confusing
map-
and
you
start
looking
around-
maybe
you
click
some
stuff
and
you're,
just
like.
Oh,
my
god
like
where
do
I
start,
and
thankfully
you
see
over
on
the
right
side,
there
there's
an
observer
ability
and
an
analysis
section
right,
and
so
this
is
broken
up
into
monitoring
and
logging.
B
And
you
see
you
know
you
might
see
a
few
things
you
recognize
from
the
past.
So
you
know
we
got
like
data
dog
in
here,
New,
Relic
and
Nagios,
some
other
stuff
down
there.
And
then
you
know
when
it
comes
to
logging.
You
see
maybe
Splunk
right.
That's
probably
what's
familiar
to
a
lot
of
people,
maybe
elastic,
but
some
of
these
tools
are
definitely
new
and
scary
and
you
definitely
have
to
pick
up
a
whole
new
set
of
you
know.
B
What's
on
your
tool
belt,
to
get
this
to
work,
and
so
I'm
just
gonna
break
this
down.
We'll
split
this
up
into
two
sections:
well,
we'll
talk
about
monitoring.
First
and
we'll
talk
about
logging,
and
so
you
know
I
guess.
The
first
thing
you
got
to
do
with
monitoring
is
take
a
step
back
and
figure
out
what
you
want
to
monitor.
B
So
you
know
just
like
before,
when
you
had
those
you
know:
fitme
metal
servers
deployed
or
those
VMs
in
the
cloud
you
know
at
some
place,
you
have
you
still
that
concept
right,
so
you
have
your
nodes
somewhere,
and
so
you
need
to
get
metrics
out
of
those
nodes.
Like
your
CPU
usage,
your
disk
usage,
your
memory
usage,
your
network
traffic,
your
packets,
that
are
dropped
right
all
of
these
things
matter,
so
your
nodes
are
definitely
where
you
need
to
start
to
start
pulling
some
of
that
data.
B
The
way
we
do
that
right
now
we
have
like
exporters
that
we
put
on
the
nodes.
We'll
talk
about
in
a
second
pods
is
probably
the
next
step
up
right.
These
are
actual
like
containers
and
your
pods
that
are
running
on
your
clusters.
You
want
metrics
out
of
those.
You
want
to
know
how
many
pods
are
running.
How
many
are
supposed
to
be
more
running?
You
could
be
watching
these
things,
setting
alerts
if
that
number
differs
ever,
and
so
you
know
taking
a
further
step
down.
B
You
have
your
cube
system
namespace
right,
so
this
is
the
namespace
was
all
kubernetes
core
components
are
running
like
the
cube
api
server
and
the
scheduler
and
SCD.
So
these
things
actually
are
really
important
right
that
you
need
to
have
inside
and
observability
into,
and
then
you
know
taking
a
step
further
into
your
actual
application.
You
have
your
APM,
your
application
performance
monitoring.
B
This
is
where
you're
gonna
care
about
things
like
request
per
second,
you
know
the
requests
that
are
coming
into
your
app
and
error
rates
that
are
being
thrown
from
your
API
or
something
right,
so
we're
drilling
down
a
little
further
and
then
you
might
have
very
specific
app
metrics
that
you're
exporting.
So
let's
say
you
have
a
webstore,
and
this
is
your
top-selling
product
or
something
right.
B
So
you
can
expose
that
as
a
metric,
somehow
that
you
can
watch
on
a
dashboard
so
well,
you
know
it
depends
on
where
you
are
in
the
company
where
you
are
in
the
stack.
You
know
the
system.
People
obviously
really
want
to
know.
What's
going
on
with
a
lot
of
those
nodes,
the
business
people
want
to
know
which
product
is
selling
the
most
of
so
this
whole
range
of
different
things
that
you
really
want
to
get
data
out
of.
B
So
these
are
some
of
the
tools
that
I've
worked
with
in
the
past
data
dog
light
step,
cystic
honeycomb,
I/o
influx
data-
these
are
all
really
great
commercial
tools.
Some
of
these
have
open
source
components
that
are
out
there.
Data
dog,
you
know,
has
an
agent
that
runs
it's
a
really
great
product.
It
can
get
really
expensive
too.
So
it's
a
favorite
amongst
a
lot
of
people
in
you
know
honeycomb
I/o,
it's
it's
definitely
a
newcomer
from
the
scene,
but
I
know
a
lot
of
bigger
companies
are
starting
to
use.
Something
like
that.
B
So
you
know
some
of
these
might
be
familiar
light.
Step
is
actually
we
use
a
digital
ocean,
we're
a
big
fan
of
it.
I
got
a
page.
I'll
show
you
later,
but
really
like
the
the
core
project
that
you
know.
A
lot
of
these
metrics
and
the
whole
kubernetes
ecosystem
runs
around
is
called
Prometheus.
Anyone
heard
of
Prometheus
before
yeah
got
a
bunch
of
people
right.
So
Prometheus
is
a
you
know,
a
metrics
system
that
we
it
works
in
a
different
way
than
most
have
worked
in
the
past.
B
It
introduces
some
CR
DS
that
are
like
custom
resources,
so
you
can
create
like
a
service,
monitor
and
some
annotation
magic,
and
basically
what
this
means
is
Prometheus
can
just
kind
of
go
out
there
and
start
scraping
anything
that
gets
added
to
your
cluster.
It
works
really
well,
it's
really
easy
to
set
up.
So
this
is
where
we're
gonna
dive
into
in
a
bit
and
then
obviously
you
need
some
place.
You
know
display
those
metrics.
The
gold
standard
is
graph
on
ax
people
use
for
fauna
before
it
yeah
yeah,
that's
a
big
room
right.
B
So
Griffin
is
a
pretty
awesome.
Metrics
aggregation
dashboard
type
daily,
so
you
can
ship
all
of
your
metrics
and
data
into
a
graph
on
ax
and
display
them
on
very
nice.
Dashboards.
I
haven't
worked
at
a
company
that
has
used
for
fana
before
so.
This
is
a
quick
high-level
view
of
our
metrics
pipeline
inside
digital
ocean.
You
probably
can't
see
any
of
this
stuff.
B
The
important
thing
is
to
call
out,
though,
is
we
have
like
4.5
million
metric
samples
a
second
right
now,
that's
actually
doubled
we're
doing
9
million
a
second,
because
we're
running
two
different
hypervisor
versions.
Right
now,
as
we
upgrade,
and
so
this
all
magically
works
around
this
pretty
cool
thing
called
Pandora.
It
does
some
really
awesome
service
discovery
throughout
all
of
do.
Tim
Simmons
gave
a
talk
last
year
at
cube,
Connie
you
on
how
all
this
works.
B
So,
if
you're
interested
check
that
out
but
like
so
we've
built
everything
around
Prometheus,
we
absolutely
love
it,
and
then
I
also
mentioned
we
use
light
step.
We
use
this
for
APM
and
tracing
this
product
is
super
awesome.
I
think
we
were
one
of
the
earliest
light
step
customers.
This
is
just
a
quick
view
of
our
kubernetes
service
and
it
kind
of
like
breaks
down
really
well.
B
All
of
your
function
calls
and
all
of
your
spans
and
traces-
and
it's
we're
just
super
happy
with
it,
so
absolutely
check
them
out
for
sure,
and
so
that's
really
a
high
level
of
logging
right.
So
you
have
little
data
points
that
you
need
to
put
somewhere
and
then
obviously
the
next
thing
is
logging,
and
so,
when
we
think
about
logging,
you
know
what
are
the
things
you
need
to
log
all
the
things
right,
the
12
factor
app
says
when
it
comes
to
logging.
You
should
be
logging
anything
that
considers
an
event.
B
You
know
something
happened
that
you
really
want
to
take
care
of.
We
have
different
log
levels
for
different
things,
and
so
you
know
the
way
kubernetes
does
logging.
Is
you
really
want
to
write
to
standard
out?
So
you
know
in
the
past,
you
might
have
write
into
a
specific
file
somewhere
for
your
application,
and
then
some
aggregator
would
have
gone
and,
like
scraped
all
those
files
individually,
but
the
way
that
kubernetes
is
really
built
to
work.
B
Well
is,
if
you
just
write
to
standard
out
and
all
of
those
logs
get
funneled
into
like
an
actual
cube
system.
Log
like
it's
written
out
to
the
disk,
and
then
you
can
have
daemon
sets
or
some
other
service
that
actually
pulls
in
that
thing
log
or
that
stream.
So
you,
you
really
want
it
right
to
stand
it
out
when
it
comes
to
this.
B
There
are
some
log
services
that
you
can
ship
directly
into
that
we'll
take
a
look
at
and
it
kind
of
goes
against
the
way
kubernetes
works,
but
there
is
an
architecture
to
support
that.
So
just
log
all
the
things
until
your
team
starts
yelling
at
you
and
then
keep
logging
more.
So
some
other
loggers
you
might
be
familiar
with.
So
we
break
this
down
into
shippers
and
in
jesters,
write
shippers
or
what's
actually
getting
those
logs
out
of
your
cluster
again.
B
Data
dog
I
think
so
feel
free
to
call
me
out
on
this
on
the
Twitter's,
but
I'm,
pretty
sure
data
dog
is
the
only
provider
out
there
that
does
metrics
and
logging
in
one
single
agent
that
runs
on
your
server
and
it'll
ships
it
off
magically
to
the
same
place,
so
I
think
to
get
logs
that
a
date
at
all.
You
just
stick
an
extra
line
and
your
config
and
it
starts
doing
it
for
you
magically
so
big
shout
out
to
them.
B
I,
don't
know
of
any
others
that
do
that
file
beats
another
one,
that's
really
popular
in
the
kubernetes
world.
This
is
how
the
elastic
folks,
so
this
is
like
a
lightweight
service
that
will
find
that
log
ship
it
out
to
somewhere
wherever
you
want,
it's
really
a
thin
footprint
there.
Our
syslog
is
what
we
use
inside.
Do
we
actually
have
a
really
huge
our
syslog
collection,
it
pulls
it.
B
Our
Sicily
has
the
concept
of
inputs
and
outputs
and
we
pull
from
a
bunch
of
different
sources
and
we
spit
it
out
to
a
bunch
of
different
pipelines
and
transforms
in
different
places.
So
these
are
some
common
ones
that
you
might
see
some
in
gestures
that
might
be
familiar
again,
data
dog.
Does
it
all
log
vio
Splunk
I'm,
a
huge
fan
of
Splunk
I've,
used
it
a
lot
in
the
past.
But
if
you,
google,
like
kubernetes
Splunk,
there's
really
not
allowed
out
there
so
I.
Imagine
they're
coming
out
with
something
cool
eventually,
hopefully,
and.
C
B
Even
using
other
ones,
so
really
great
tools,
but
again
data
dogs,
the
only
one
that
does
all
three
of
these,
the
biggest
open
source
stack
that
you
might
be
familiar
with
is
the
efk
stack
used
to
be
the
ALK
stack.
The
L
stack,
you'd
swap
out
this
thing
called
fluent
D
with
logs,
but
elasticsearch
is
where
you're
going
to
ship
your
logs
out
to
you
know:
you'd
write
you
run
this
in
your
cluster
or
you'd
have
a
managed
service
somewhere,
fluent
D
is
actually
the
log
shipper.
This
is
a
process
that
runs
in
Ruby.
B
It
takes
a
bunch
of
plugins.
It's
got
a
huge
Arco
system
out
there.
There
are
complaints
about
you
know
it's
performance
and
how
much
memory
it
actually
takes
up.
There's
a
lighter-weight
version
fluent
bit:
that's
written
in
C
that
might
be
useful,
but
fluently
really
does
all
the
full
indexing
and
ships
everything
out
for
you.
So
if
you
have
like
high
quality
needs
for
all
of
those
different
bits,
indexed
check
out
fluent
D
and
then
Cabana
is
you
know
the
historic
dashboard
in
the
past
for
displaying
log
who
uses
Cabana
who's
yeah.
B
This
stacks
very
familiar
to
folks
right,
so
you
know
this
is
this
is
pretty
much
one
of
the
you
know
the
standards
for
kubernetes
right
now,
but
there's
actually
a
new
challenger
that
is
approaching
Griffen
a--.
Actually,
as
Tom
mentioned,
has
a
pretty
cool
thing.
They
put
out
last
cube
con
called
low
key.
It
works
really.
Well,
it's
really
light.
It
doesn't
do
things
it's
a
bit
different
than
how
a
lot
of
the
other
investors
work.
It
doesn't
do
all
the
parsing
and
as
much
indexing.
B
So
if
you
don't
need
all
that,
like
full-text
individual
data
searching,
it
works
really
well,
it's
super
lightweight.
It's
composed
of
two
different
components:
really
one's
called
prom
tail,
it's
basically
the
reverse
of
Prometheus
right.
So
this
is
prometheus
for
logs
prom
tale
actually
will
push
out
logs
into
low
key
instead
of
pulling
it.
So
it's
a
bit
flip
there,
but
so
prom
tale
will
run
as
a
daemon
set
on
all
of
your
nodes
and
it
will
just
ingest
all
those
logs
and
ship
them
out
to
Loki
Loki's
running
somewhere
in
your
cluster
ingesting.
B
All
of
that,
for
you
and
actually
graph
on
a6,
came
out
with
a
pretty
cool
view,
called
explore
that
we'll
take
a
look
at
and
this
actually
lets.
You
look
at
your
logs
right
next
to
your
your
metrics
and
dashboards
and
your
Prometheus
queries.
So
you
actually
do
like
one
to
one
correlation
and,
like
click,
one
thing
and
it
pops
over
so
yeah.
It's
really
cool,
so
I'll
show
you
that
in
a
bit
just
quick
pipeline
of
our
do
thing.
I
mentioned
we
use
our
syslog.
B
We
actually
have
a
hundred
percent
availability
for
log
ingestion,
which
is
pretty
cool.
This
is
across
ten,
you
know,
plus
regional
aggregators,
and
we
do
about
30,000
log
lines
ingested.
A
second
right
now
we're
pretty
happy
with
the
setup
with
our
syslog.
It
works
really
well,
it
kind
of
goes
through
a
bunch
of
different
transforms,
like
I
mentioned,
for
different
reasons,
so,
and
so
now,
we'll
just
pop
right
into
the
demo.
So
I'm
gonna
use
helm
for
this
people
use
helm,
yeah
little
familiar
helms
like
a
package
manager
for
kubernetes.
B
It's
got,
you
know
some
different
manifests
that
we
can
deploy
those
templating
really
well.
There's
a
big
open
standard,
helm,
repo
called
the
the
charts
repo
with
stable.
It's
like
all
these,
like
packages
that
are
community
managed.
A
lot
of
these
can
be
configured
so
you
definitely
want
to
check
out
the
readme
zhan.
B
B
If
you
want
to
grab
this,
but
after
spending
you
know
probably
the
past,
you
know
year-and-a-half
doing
nothing
but
kubernetes
working
with
people
who
are
brand
new
to
kubernetes
I've,
seen
a
lot
of
folks
struggle,
different
parts
right
that
the
number
two
questions
I
get
answered,
or
how
do
I
do
logging
and
had?
Are
you
monitoring?
So
this
is
definitely
where
a
lot
of
people
are
getting
stuck.
This
is
a
pretty
simple
stack
that
I
put
together
through
trial
and
error.
I
played
with
a
lot
of
different
products.
It'll
get
you
started.
B
You
definitely
want
to
read
the
docs.
You
definitely
want
to
read
all
the
values
that
you
can
configure
in
the
helm
charts,
but
this
is
definitely
a
great
place
to
get
some
insight
into
this
black
box
that
you
have
so.
The
first
thing
we
got
to
do
is
we
got
to
configure
helm,
so
I've
got
a
little
command
here.
B
If
in,
if
you've
used
helm
before
it
probably
looks
familiar,
you
have
to
actually
create
like
a
service
account,
you
got
a
cluster
role
binding
and
you
got
to
create
that
role
and
initialize
helm
on
your
your
cluster
right.
So
this
is
going
to
install
tiller,
which
is
like
a
service
side
component
that
will,
you
know,
listen
to
API
requests,
put
all
the
charts,
compile
the
yam
bowls
spit
it
out,
and
then
he'll
merge
in
three
actually
gets
rid
of
tiller.
So
a
lot
of
these
steps
will
actually
be
emitted,
which
is
cool
so.
D
B
Should
be
good,
but
now
it's
yep
there
we
go
so
cool,
so
tillery's
done
at
the
bottom
there
and
then
the
next
thing
we'll
do
is
we'll
install
the
prometheus
chart
here
and
so
I've
got.
You
know
just
using
the
very
standard
stable,
prometheus
chart.
I
got
this
custom
values
file.
There's
just
a
couple
things
in
here
that
I
configured
to
make
things
work
a
little
better.
B
So
we
got,
we
got
to
turn
off
at
CD
monitoring.
Generally
cloud
providers
will
lock
down
their
master
node,
so
you
can't
get
on
there
and
at
CDs
a
side
effect
of
that.
So
most
cloud
providers
don't
give
you
direct
access
to
at
CD.
So
we
just
turn
that
off
I
got
to
change
a
couple
ports
in
here
we
put
firewall
rules
on
our
clusters
and
so
I
just
have
to
bump
up
the
the
Prometheus
node
exporter.
So
this
is
actually
what
runs
on
those
nodes.
B
That's
going
to
be
shipping
out
actual
hardware
data,
so
we
just
got
a
bump
to
pour
it
up
into
a
range.
That's
in
that
firewall
I
got
a
set,
a
graph
on
a
password
I'm.
Turning
on
persistence,
that's
gonna
set
it
like
a
five
gigabyte
disk
and
then
up
here
same
thing
with
Prometheus.
So
this
is
the
the
disk
for
Prometheus.
B
And
that
shouldn't
take
too
long,
it's
a
while
it's
running
we'll
take
a
look
at
the
next
thing,
so
Loki's.
What
we're
gonna
do
next,
so
right
now,
Loki's
not
part
of
the
stable
charts,
because
it's
like
the
Alpha
Tom
yeah,
not
even
alpha,
so
he's
not
even
alpha.
It
works
really
well,
there's
a
bunch
of
people
in
the
Griffons
slack
that
are
actually
running
in
a
production.
The
community
is
pretty
great,
so
hop
on
the
Griffons
slack.
If
you
actually
want
to
get
involved
there,
I
actually
haven't
run
into
any
problems.
B
It's
been
very
stable
for
me,
so
we're
gonna.
We
got
to
pull
down
the
actually
I
did
in
the
helm
thing
you
got
to
pull
down
the
Griffin
Oh
Loki
shark
from
above
and
so
we'll
just
install
that
we're
gonna
set
a
couple
things
like
persistence,
like
I
mentioned
before,
and
then
that's
really
it
so
we'll
do
a
make
Loki
here.
B
And
so
it's
gonna
create
some
of
those.
This
it's
gonna
deploy
some
of
those
pods
shouldn't
take
too
long
to
get
through
here,
while
that's
running
I'm,
just
gonna,
we
actually
just
shipped
an
insights
part
and
built
into
our
managed
community
stuff.
Every
cloud
provider
has
one
they
work
really
well.
So
this
is
like
some
super
high
level
metrics
that
you
can
take
a
look
at
these.
These
are
exported
right
from
the
nodes.
You
can
also
install
like
linker
D
and
some
cube
state
metrics
in
here,
but
it
just
gives
you
a
quick
base.
B
B
C
B
That
works
really
well,
so
I'll
run
that
so
we
can
just
get
some
metrics
being
deployed.
So
cool
is
Loki's
done,
I'm
gonna
log
into
this
cluster
with
admin
and
tacocat,
which
is
the
best
palindrome
of
all
time.
If
you
didn't
know-
and
now
we
can
actually
the
first
thing
we
got
to
do.
Is
we
got
to
configure
Loki
as
a
data
source
I.
Think
there's
a
way
to
do
this
automatically
in
the
chart.
B
I
haven't
figured
it
out
yet,
though
we
just
come
in
here
and
you
pop
in
Loki
with
the
service
discovery,
so
just
the
service
name
and
then
port
3100,
so
what
it
runs
on
click,
Save
and
test-
and
it's
just
connected
like
that
and
then
so
we'll
just
take
a
look
at
some
of
the
metrics
that
we
get
out
of
the
box
for
free
I'm
actually
going
to
switch
over
to
a
different
cluster.
But
I've
already
done
this
song.
That's
generated
a
bunch
of
data
already.
B
B
And
I
haven't
I,
haven't
configured
this
any
differently,
so
there's
some
really
great
dashboards.
That
work
out
of
the
box
in
here.
So
like
this
just
use
method
word
that
apparently
Tom
built
it's
got.
You
know
your
full
cluster
overview
pulling
from
Prometheus.
You
know
you
can
take
a
look
at
all
these
bits
here.
I'm
actually
I'm
also
going
to
run
a
scaling
script.
That's
just
gonna
scale,
some
services
up
and
down
to
generate
some
more
load
all
right.
So
these
dashboards
work
really
well
the
box.
B
You
can
take
a
look
at
the
like
the
namespace
one
in
here.
So
I
have
a
couple.
Different
micro
service
demos
built
out.
Google's
got
one,
that's
my
favorite!
Actually,
it's
their
hipster
shop,
it's
under
Google
cloud
platform,
slash
micro
service
demo,
it's
built
with
a
bunch
of
G,
RPC
stuff,
really
cool
demo
to
go
through
so
I've
got
that
running
under
default
here
and
then
sock
shops
by
the
awesome.
We've
worked
folks
and
then
emoji
voto
is
by
the
boolean
people
right.
So
you
can
take
a
look.
B
So
the
dashboards
are
cool.
You
can
take
a
look
around
in
there
to
take
a
look
at
the
logs.
We
actually
just
go
to
this
explore
tab.
That
I
mentioned
you
make
sure
loki's
picked
as
a
data
source
up
here
and
then
you
can
come
down
and
let's
just
grab
all
the
logs
out
of
the
namespace,
we'll
grab
the
emoji
vote,
a
link
name,
space
right
and
so
boom.
There's
all
of
our
logs
aggregated
right
there.
You
can
take
a
look.
You
know,
there's
some
votes
going
in
here.
B
B
Container,
alright,
and
so
let's
say
we
just
want
to
grab
this
query
they'll
warn
us.
We
need
to
like
throw
a
raid
on
there
right.
So
if
you
had
a
very
specific
thing,
so
if
a
moji
voter
was
exporting
Prometheus
metrics,
you
know
we
could
take
a
look
at
the
Prometheus
metrics
that
are
exported
and
then
matching
them
up
with
the
logs
to
have
like
side-by-side
insight
right
there
right,
so
you
don't
actually
have
to
like
open
another
tab.
Go
someplace
else.
Copy/Paste,
like
this
view,
is
really
awesome.
B
It's
got
everything
here
and
then,
if
we
pop
back
we'll
close
this
one
look
at
a
mode,
you
can
turn
on
the
labels
for
the
logs
here
and
you
can
see
oh
like.
Maybe
this
error
is
only
coming
from
this
individual
pod.
It
might
be
hard
to
see.
So
it's
got
this
little
magnifying
glass
that
you
can
click
here
and
it
will
add
it
to
the
top.
So
now
we've
drilled
down
only
into
you
know
this
individual
pods
vote
bots,
and
so
you
can
see
it's
got
a
bunch
of
errors
that
are
logged
out
there.
B
It's
voting
for
donut
quite
a
bit
right,
and
so
it's
a
pretty
powerful
system.
It
works
really
well.
I
haven't
had
any
trouble
and,
like
I
said,
the
real
power
is
how
this
really
plays
together.
So
from
there
you
know
you
could
configure
Prometheus
has
the
alert
manager,
that's
how
we
trigger
load
set.
Do
you
could
configure
the
alert
manager?
It's
I'm
pretty
sure
it's
part
of
the
the
prometheus
operators
chart
by
default.
If
you
want
it,
and
then
you
can
go
in
there
set
up
alerts.
B
Service
discovery
for
alerts
is
something
that
we
definitely
want
to
work
on
in
the
future.
We
would
love,
for
you
know,
alerts
to
be
created
really
based
off
service
discovery
and
some
other
stuff.
We
do
it
a
bit
right
now.
The
real
powerful
bit,
though,
like
I
mentioned,
was
just
having
this
high-level
overview,
and
so
you
know
if
you're
getting
started
with
kubernetes,
and
you
don't
really
know
where
to
begin.
When
it
comes
to
taking
a
look
inside
this
black
box.
B
This
is
really
a
great
place
to
start
so
I
have
it
like
I,
said:
I
have
a
link
to
the
github
repo
and
flip
that
up
real
quick.
So
this
is
a
link
to
the
github
repo.
It's
got
that
make
file
in
there.
It
really
goes
through
all
different
bits.
You
know
feel
free
to
grab
it
play
with
it
comment
and
really
take
a
look
at
that
helm.
Charts
though
they're
like
I,
said
they're
very
configurable
I
got
a
bunch
of
data
in
there
and
then
the
there's
a
link
here.
B
If
you
can
go
to
this,
do
that
Co
slash
see
this.
Is
our
current
survey
a
little
plug?
If
you
could
take
this
survey,
it
would
help
us
alot
it's
about
remote
work
culture.
So
if
you
work
remotely
or
want
to
we're
just
collecting
a
bunch
of
data,
we
publish
all
the
results
and
we'll
send
you
this
really
cool
Sammi
remote
sticker.
If
you
take
it
so
comms
team
asked
me
to
do
that.
So
please
please
do
that,
but
that's
really
all
I
have
for
the
demo
I
love
to
answer
any
questions.
A
E
Hi,
my
name
is
Sergio,
so
you
actually
said
that
we
should
lock
all
the
things
I
would
like
to
know
how
digitalocean
deals
with
this
or,
what's
your
take
on
because
I
mean
I
understand
as
a
developer.
I
want
to
see
basically
everything
or
as
much
as
I
can
but
then
as
a
cease
admin
or
DevOps.
One
of
yours,
sorry,
I,
don't
know
what
they
are
calling
in
nowadays,
I
I
need
to
I
need
to
not
spend
so
much
money
right.
E
B
So
the
question
is
how
to
really
make
a
decision
on
what
to
log
and
how
much
it
really
comes
down
to
what
your
business
needs
are.
If
you
know
you
have
high-level
auditing,
where
you
need
to
log
every
single
interaction
ever
you
know
to
get
some
data
out
of
it.
Maybe
that's
what
you
need
for
business,
but
in
terms
of
like
the
12
factor
app,
they
really
suggest
to
log
all
of
events.
So
you
know
whatever
something
that
triggers
that's
considered
to
an
event
that
should
be
logged
at
the
Oh.
B
Our
our
team
has
this
I've
heard
many
times
that
the
observability
team
has
said:
hey,
stop
logging
so
much.
It
costs
us
a
lot
of
money
in
storage.
It's
really!
Obviously,
if
you're
logging-
something
that's
not
super
useful,
you
probably
shouldn't
be
doing
that,
but
if
it's
at
an
actual
event
or
something
that
would
help
you
solve
a
problem
right,
because
when
something
goes
wrong,
you
want
to
get
to
identified
of
the
issue
as
quick
as
possible
right.
B
F
C
F
B
So
a
question
is
you're
running
your
observability
inside
your
cluster.
What
if
everything
breaks
and
goes
to
hell
right?
So
you
have
no
idea
so
Seth
Vargo
from
the
Google
cloud
team
spoke
at
hashey
conf
last
year,
and
he
mentioned
that
if
you're
running
vault
in
production,
you
should
absolutely
be
running
vault
in
a
separate
kubernetes
cluster,
and
so
your
main
cluster
should
talk
to
your
vault
cluster
and
that's
in
terms
of
security
and
policy.
There
right
I,
imagine
the
same
carries
over
to
your
observability.
B
So
if
it's
in
your,
if
you
run
your
observability
your
pipeline
and
a
separate
cluster,
you
know
you're
not
having
all
your
eggs
in
one
basket
there.
All
of
these
tools
support
pretty
high
availability.
You
know
you
can
run
these
on
different
nodes.
You
can
distribute
them
there.
You
can
pay
for
a
service
to
ship
everything
off
and
you
know
store
it
in
there.
So
yeah
you
definitely.
B
C
Hello
Adam.
Thank
you
for
great
presentation.
First
of
all,
the
second
thing
I
have
two
simple
questions
about
the
locket.
If
you
didn't
mention,
I
tried
this
system
some
time
ago,
like
a
POC
and
I,
didn't
know
how
to
make
the
retention
policies.
The
rotation
of
Mavericks
indeed
days
is
real
and
the
second
thing
I
thought
the
guy
who
just
asked
about
the
how
to
decide
what
metrics
are
really
the
the
most
important.
This
is
the
two
questions.
Thank
you
yeah.
So.
B
First
question:
if
I
understood
correctly
you're
asking
how
do
you
handle
like
log
rotation,
yeah
so
kuba?
If
you're
writing
the
standard
out,
kubernetes
actually
runs
log
rotate
itself,
so
either
the
cube,
lid
or
the
API.
So
every
something
on
a
kubernetes
node
will
run
log
rotate
for
you
and
swap
those
logs
out
for
you
under
the
hood.
So,
but
that's
why
you
really
want
to
log
to
stand
it
out.
Kubernetes
will
take
care
of
everything
and
the
other
question
was:
how
do
you
know
what
metrics
you
want
to
grab?
B
It's
really
it
comes
down
to
your
business
needs.
Some
folks
need
every
single
metric
reported.
You
know,
obviously
your
node
and
your
hardware
metrics
every
that's
a
base.
Everyone
wants
to
know
how
much
ram
they're
using,
but
in
terms
of
like
your
app
specific
metrics
things
that
most
of
the
permits,
so
preemies
has
two
parts
of
like
getting
data
once
called
instrumentation
and
one's
called
exporters.
So
exporters
are
something
that
will
run
on
your
node,
so
get
the
data
out
of
your
node,
it's
a
knowed
exporter
and
then
instrumentation
is
code.
B
You
can
write
into
your
apps,
and
so
you,
when
you,
you
know
separate
these
two
things.
Instrumentation
comes
down
to
you
know.
Maybe
you
want
the
error
rates
that
get
failed.
Maybe
you
have
most
most
of
the
there's
like
interceptors,
4G
or
PC,
and
kubernetes
promise.
Prometheus
prom,
HTTP
out
of
the
box,
will
tell
you
your
your
HTTP
error
rate.
It
works
magically
pretty
well,
and
so
those
things
are
pretty
core
staples
outside
of
that.
It's
Prometheus
as
a
time
series
database,
so
the
met
when
ever
the
scraper
reaches
out.
It
hits
the
exporter
it.
B
The
exporter
doesn't
have
to
have
any
concept
of
time.
It
just
has
to
respond
with
the
current
metrics
and
then
the
Prometheus
query.
Language
is
what
you
make
a
query:
over
till
I
get
the
rate
over
five
minutes
or
something
so
Prometheus
just
takes
all
the
metrics.
It
stores
everything
as
much
space
as
it
has
any.
D
Hi
I
mean
here
the
Prometheus
operator,
I
didn't
use
it
yet
is
making
it
should
it's
supposed
to
be
making
it
easier
to
deploy.
Prometheus
right,
so
I
would
use
suggests
to
deploy
one
Prometheus
instance
per
application,
for
example
per
namespace,
and
then
have
one
Prometheus
that
collects
the
data
from
all
the
other
Prometheus's
or
what
would
be
the
pattern
to
do
it
right?
That's.
B
A
great
question
so
he's
asking
so
Prometheus
operator
makes
it
very
easy
to
deploy
multiple
Prometheus,
no
Prometheus
Prometheus
in
your
cluster
right,
and
so
it
keeps
watching.
That's
what
you
know
a
operator
really
does
I.
Don't
actually
have
an
answer
to
that.
I
really
don't
know.
I'd.
Imagine
that
you'd
there'd
be
no
harm.
Prometheus
is
a
pretty
light
footprint.
You
know.
Storage
is
the
biggest
thing
there.
So
as
long
as
you're
attaching
volumes
I
guess
in
terms
of
your
like
security
policies.
B
If
you
want
things
but
I've
heard
many
people
say
they
just
give
every
different
team
in
their
organization
a
namespace
and
they
let
them
run
wild
right.
So
I.
Imagine
like
your
core
platform
team,
would
install
the
operator
and
make
it
available
to
everyone
to
pull
into
the
namespace
if
they
want
so
good
question,
though
I
don't
actually
know
sorry.