►
From YouTube: TGI Kubernetes 159: Exploring Kuberhealthy
Description
Notes: https://github.com/vmware-tanzu/tgik/tree/master/episodes/159
- 00:04:47 - Welcome to TGIK!
- 00:07:20 - Week in Review
- 00:25:28 - Looking at Kuberhealthy
- 00:28:59 - Installing Kuberhealthy
- 00:43:27 - Connecting to console
- 00:50:26 - Finding prom metrics (`/metrics')
- 00:55:15 - Exploring external checks
- 01:01:11 - Creating your own checks
- 01:03:28 - Continuing to debug cross-namespace checks
- 01:10:12 - Looking at HTTP Check
- 01:13:16 - Debugging and filing bug
- 01:48:22 - Wrapping up
Come hang out with Joe Beda (@jbeda) as he explores something new in the Kubernetes ecosystem. Hang out as we explore something NEW for the first time.
Continuing the CNCF sandbox series, this week we will be looking at kuberhealthy! (https://github.com/kuberhealthy/kuberhealthy)
A
Hello,
everybody
and
welcome
to
tgi
kubernetes,
I'm
your
host
for
our
weekly
weekly-ish
live
stream
where
we
play
around
with
any
part
of
kubernetes,
sometimes
things
that
are
new
in
the
ecosystem.
Sometimes
new
parts
of
the
project
are
itself
oftentimes,
we'll
have
experts
outside
their
comfort
zone.
So
you
know
just
for
context.
My
name
is
joe
bida,
I'm
one
of
the
founders
of
kubernetes.
A
So
I'd
like
to
start
out
by
saying
hello
to
all
the
people
that
are
joining
us,
I'm
continually
surprised
we're
on,
like
you
know,
episode
159
here,
I'm
continually
surprised
with
folks
tuning
in
from
all
around
the
world,
sometimes
at
very
inconvenient
hours.
You
know
friday
afternoon
is
great
for
me
not
necessarily
great
for
everybody
else,
so
yeah
I
just
want
to
start
with.
A
A
I
really
need
to
learn
how
to
pronounce
your
name,
sir
and
eric's
from
dallas
phillip
yeah
good,
to
see
you
all
philip.
We
have
a
phillips
from
berlin
and
then
pie
from
paris.
Wonderful,
well,
the
way
that
I'd
like
to
work.
This
is
we're
going
to
start
off
with
doing
a
little
bit
of
a
review
of
some
of
the
stuff.
That's
been
new
over
the
week
and
you
can
go
to
tgik.io
notes
and
that's
where
we
have
essentially
hackmd,
which
is
this
awesome
service.
A
Some
stuff
that
you
think
is
interesting
and
and
relevant
to
both
what's
been
happening
over
the
last
week
or
two,
and
also
the
the
topic
that
we're
going
to
be
discussing
today,
oh
and
by
the
way
after
we
go
through
the
news
of
the
week,
we're
going
to
be
looking
at
this
project
called
cooper
healthy,
which
has
been
around
for
a
little
bit
really
interesting
project
for
essentially
doing
running
containers.
That'll,
do
sort
of
essentially
probes
like
test.
A
A
Let's
see
if
I
can
make
this
a
little
bit
bigger
here
and
the
way
that
kubernetes
releases
go-
and
this
is
a
great
way
to
get
involved
in
the
upstream
project-
is
to
start
tagging
along
there's
a
ton
of
different
roles.
They
always
need
help
and
it's
a
great
way
to
really
see
how
the
sausage
is
made
for
for
kubernetes.
But
this
is
a
great
way
to
to
to
go
through
that
now
over
time.
A
A
Fixing
phase
before
things
are
called
done
and
actually
go
ahead
and
they're
shipped,
and
so
that's
where
we're
at
now,
and
that
just
happened
when
that
that
happened
yesterday
and
the
other
interesting
thing
as
you
go
through.
The
the
the
release
process
is
how
we're
handling
enhancements
to
kubernetes
these
days.
There's
these
kubernetes
enhancement,
proposals
and
just
ways
for
us
to
get
very.
You
know,
or
at
least
much
more
thoughtful
about
the
new
features
that
are
being
added.
A
Make
sure
that
everybody's
on
on
the
same
page,
knows
what's
coming
down
the
pike
and
and
is
able
to
to
make
sure
that
we're
building
the
right
stuff
and
we're
doing
it
in
the
right
way,
and
so
the
enhancements
team
there
actually
helps
to
deal
with
making
sure
you
know
what's
going
to
make
it
into
a
milestone
and
what's
not
so
that's
some
really
cool
stuff
going
on
there.
It's
good
to
see
that
1.22
getting
moving
or
continuing
to
move
forward.
A
Let's
see
the
next
thing
here
is
you
know
a
project
that's
near
and
dear
to
my
heart
is
contour.
This
is
something
that
we
started
at
heptio
and
contour
1.17
is
released.
What
I
think
is
really
interesting
here
with
contour
is:
it
is
still
very
much
focused
on
being
an
envoy
based
ingress
controller,
and
so
the
the
the
feature
set
is
very
narrow,
or
at
least
focused.
A
We
also
supported
essentially
a
new
set
of
of
crds
that
were
contour
specific
apis,
and
it's
been
really
interesting
to
see
those
apis
and
the
people
that
helped
to
make
those
things
work.
Nick
young
who's,
who's
based
out
of
australia
is
is
carrying
a
lot
of
that
work
now.
But
it's
really
great
to
see
those
things
really
influence
and
help
inform
some
of
the
work.
A
So
you
can
go
ahead
and
look
at
there's
contour
and
then
there's
an
operator
for
helping
to
install
and
manage
it.
So
a
lot
of
good
stuff
going
on
there.
It's
great
to
see
that
continue
to
continue
to
make
progress,
so
yeah
very
cool
stuff.
Let's
see
next
all
right,
so
this
was
a
company.
I'm
gonna
spend
a
little
bit
of
time
on
this
that
I
met
with.
At
the
you
know,
I
have
a
a
bunch
of
people.
You
know
want
me
to
meet
with
various
companies.
A
And
this
company
paretus.ai
now
they're,
not
open
source,
but
one
of
the
things
that
they're
working
at
doing
is
actually
creating
a
lot
of
value
for
open
source
projects
and
especially
user
communities
for
open
source
projects.
With
the
idea
of
how
do
you
get
people
answers
to
questions?
You
know
in
a
more
effective
way,
and
I
think
this
is
something
that
we
struggle
with
with
community
with
the
kubernetes
community.
A
You
can
take
the
open
source,
you
can
run
it
yourself
and
we
want
kubernetes
to
be
useful,
even
if
you're
not
paying
somebody
like
vmware
to
help
you
get
going
with
it
and
but
that
doesn't
scale
I
think.
So.
What
we
find
is
that
people
get
in
trouble,
they
have
questions,
they
have
problems,
and
you
know
any
sort
of
help
that
they're
going
to
get
from
the
community
is
really
best
effort.
We
have
a
great
community,
but
the
using
community,
the
user
community,
is
much
much
larger
than
the
contributing
community
and
oftentimes.
A
How
do
we
learn
over
time
to
provide
the
right
answers
for
various
questions?
And
you
know
I
like
folks
like
me,
who
are
deep
in
the
community,
I
kind
of
know
all
the
keywords
to
use.
I
can
search
google.
I
can
probably
find
something
out
there
if
I
can,
but
a
lot
of
people
like
you,
don't
even
know
what
to
search
for
when
you're
looking
for
help
and
that's
where
I'm
really
hoping
oh
steve's,
showing
up
here
too
contour,
but
that's
where
I'm
really
hoping
that
tools
like
this
will
really
help
folks
out.
A
So
I
just
want
to
give
them
a
shout
out,
because
I
think
this
is
really
interesting
and
they're,
making
an
effort
to
make
sure
that
they
can
really
index
a
lot
of
kubernetes
resources
and
make
this
stuff
be
be
much
more
accessible.
So,
like
here's,
one,
which
is
sticky
sessions
for
multiple
back
ends
on
the
same
domain
on
different
paths
like
I
don't
even
know
what
this
is
but
like,
and
this
is
in
the
the
you
know,
ingress
engine
x
looks
super
relevant.
I
don't
know
if
this
has
the
answers
there.
A
So
that's
really
cool,
and
I
just
you
know,
I
think
you
know
anything
we
can
do
to
make
more
people
successful
with
kubernetes
and
really,
you
know
not
overload
so
many
of
the
people
that
are
putting
their
time
in
their
their
volunteer
hours
and
seems
really
worthwhile
here
so
yeah.
So
quick
shout
out
for
them
for
for
doing
some
interesting
work
there
again,
the
this
was
paratus.ai,
it's
worth
checking
out
and
I'm
sure
they
would
love
to
hear
any
feedback
that
you
all
might
have
about
about
the
service
too.
A
So
cool
all
right
couple,
other
things
that
I
found
interesting
over
the
last
week.
So
the
first
is
a
really
interesting
article
on
schema
validation
tools
for
kubernetes,
and
I
think
you
know
when
really
looking
at
cubaval,
cubeconform
and
then
kubernetes
dry
run
in
sort
of
like
both
client
and
server
mode.
These
things
are
really
interesting
to
me.
A
A
In
some
way,
and
then
something
tries
to
apply
it
to
your
cluster
and
if
it
succeeds,
everything
is
great
if
it
fails
well,
you
know
now
what
what
am
I
going
to
do
right?
How
do
I
communicate
that
to
users?
How
do
I
actually
close
that
loop
and
so
just
like
any
sort
of
ci
cd
system
oftentimes?
You
can
get
yourself
into
this
point
of
view
where
it's
like.
Oh
try
number
two
try
number
three
and
it's
like
you
know
the
best
way
to
debug.
A
It
is
to
make
a
ton
of
commits
to
your
project,
which
is
not
great,
whereas
I
think
looking
at
some
of
these
tools
here,
whether
it's
cube
control,
dry
run
or
whether
it's
something
like
cuba,
val
or
or
cube
conform.
These
things
really
help
you
catch
things
earlier
in
your
pipeline,
which
I
think
is
really
interesting.
A
If
you're
looking
to
figure
out,
how
do
I
actually
check
the
validity
of
my
configuration
earlier
on
in
my
process,
whatever
my
process
might
be
all
right
and
then
the
last
thing,
yeah
flux
for
the
win-
and
I
think
you
know
there's
definitely
tools
like
flux
that
can
provide
dashboards
that
help
to
notify
you
of
this
stuff,
and
so
there's
things
you
can
do
to
really
start
closing
that
loop.
But
you
know
it's
always
there's
a
little
bit
of
a
like.
A
A
The
question
then,
is,
is
how
do
you
notify
people
that
this
happened
and
then
how
do
you
make
sure
that
they
can
make
sure
that
they
can?
You
know
the
next
time
they
check
in
a
config.
They
can
feel
confident
that
it's
not
going
to
get
auto
reverted
again,
because
you
don't
want
to
get
into
that
loop
where
it's
like.
Let
me
check
something
in
and
then
wait
five
minutes
to
see.
A
If
it
works,
because
that's
just
a
painful
experience
for
everybody
involved
and
then
there's
this
other
project-
and
I
found
this
one
on
the
the
kubernetes
subreddit
called
another
scheduler-
and
I
just
I
love
these
simple
little
tools
here,
because
this
thing's
really
interesting.
So
one
of
the
things
I'm
not
sure
the
name
is
great
because
it's
a
different
type
of
scheduler
here,
but
one
of
the
things
that
I
think
is
interesting
is
auto.
Scaling
is
great,
except
auto.
Scaling
is
really
really
hard.
A
It
turns
out
that
you
know
we
should
probably
do
an
episode,
or
at
least
I
would
love
to
do-
an
episode
on
cada
at
some
point
which
which
has
some
of
the
similar
functionality,
but
finding
the
right
things
to
cue.
Your
auto
scaling
off
of
is
really
difficult.
If
you
know
you
go
off
of
cpu
usage,
then
that's
great
but
oftentimes.
Your
bottlenecks
are
not
in
your
particular
service.
It
might
be
at
the
database.
A
It
might
be
something
that's
two
or
three
hops
down
the
line,
and
so
the
thing
that
you
need
to
scale
is
not
always
obvious,
and
you
can
probably
get
you
know.
This
is
one
of
those
things
that
I
love
talking
about,
where
you
know
where
you
can
get
probably
90
80
90
of
the
benefit
of
auto
scaling
with
a
simple
time-based
scaler
that
says
hey,
you
know,
most
services
have
diurnal
cycles,
you
know
scale
it
up
during
the
day
scale
it
down
at
night.
A
You
watch
some
graphs
and
you,
you
know,
adjust
it
every
once
in
a
while,
once
in
a
while
to
make
sure
that
you
have
the
appropriate
scaling
going
on
here,
that's
a
totally
reasonable
way
versus
actually
taking
the
risk
of
full-blown
auto
scaling,
and
that's
essentially
what
this
tool
is.
You
run
this.
You
put
a
couple
of
annotations
on
your
on
your
your
deployment
say,
and
it
automatically
goes
through
and
scales.
This
thing
for
you,
which
I
think
is
really
interesting.
A
So
that
is
super
easy
to
use,
and
this
is
essentially
saying,
like
you
know,
starting
in
1400
hours,
utc
go
up
to
five
replicas
and
then
in
15
go
down
to
one,
and
I
think
they
also
support
you
know.
When
do
you
want
to
like
do
a
restart?
You
know
it's
like
you:
have
a
memory
leak,
just
restart
it,
how
hard
could
it
be,
but
that's
essentially
what
it
does
also
so
yeah.
I
think
it's
it's.
You
know
this
is
super
focused.
A
You
know
fun
little
utility,
that's
that
I
think
you
know
really
does
some
something
useful
in
a
in
a
super
simple
way.
Relatively
new
project
has
been
around
for
like
looks
like
eight
days,
but
this
is
the
type
of
stuff
that
I
think
is
really
interesting.
Now
in
the
in
the
reddit
thread,
people
were
pointing
at
the
the
keta
cron
scheduler,
also,
which
I
think
is
really
interesting,
and
this
is
a
little
bit
heavier
weight.
It's
part
of
a
larger
set
of
tools.
A
Is
that
for
one
of
these
scales
you
have
a
start
and
an
end
and
as
desired
replicas,
but
what
does
it
set
replicas
to
at
the
end
right?
Maybe
it
sets
it
to
what
it
was
before
and
it
uses
annotations
to
actually
keep
track
of
what
had
been
going
on
before,
but
I'm
not
sure
oh
has
kate
have
been
pushing
duck
types
into
that.
That's
super
cool.
If
that's
the
case,
I
love
you
know.
A
I
love
the
idea
of
duck
types
in
this
stuff
because,
like
here
you
know,
the
the
the
the
target
ref
is
is
my
deployment.
So
I
assume
that
this
assumes
that
you're
targeting
a
deployment,
but
I
think
over
time
there
may
be
other
things
that
you
want
to
actually
go
and
manipulate
here,
and
I
think
this
is
you
know,
there's
an
open
question
here
and
I
would
probably
want
to
look
at
the
code
for
another
scheduler
around.
A
What
are
the
types
of
objects
that
it's
looking
for
here?
Is
it
just?
Is
it
just
deployments
that
it's
looking
at?
Are
there
other
objects
that
they're
looking
at
too
awesome?
Okay,
cool
yeah
so
way
easier
than
vp
or
hpa.
I
think
you
know
vpa
is
one
thing
really.
This
is
about
setting
resources
appropriately.
Based
on
this
is
really
be,
would
be
versus
hpa
or
something
like
keta.
That's
doing
you
know
horizontal
scaling
of
deployments
based
on
some
metrics.
A
This
is
just
you
know,
a
dirt
simple,
time-based
scaling
thing,
which
I
think
it's
like
you
know.
It's
like
keep
it
simple
stupid
right.
This
is
very
much
in
that
sort
of
keep
it
simple
type
of
tool,
which
I
think
is
is
very
unix
philosophy,
and
I
love
seeing
stuff
like
that
really
come
up
so
yeah.
I
thought
that
was
a
really
interesting
project
that
might
be
fun
to
play
with
at
some
point
all
right,
any
other
things
that
you
all
are
excited
about.
A
That
you've
seen
over
the
last
week
that
I
missed
here,
I'm
sure,
there's
stuff,
that
I've
missed,
but
I'd
love
to
you
know
always
want
to
solicit
a
little
bit
of
audience.
Participation
here
so
feel
free
to
to
throw
something
up
on
the
on
the
note
stock.
If
there's
something
else
that
you
know
otherwise
we're
gonna
jump
in
to
cuber
health,
which
is
gonna,
be
where
we're
heading
so
yeah.
A
A
So
as
we
were
bringing
up
google
compute
engine,
we
had
a
prober
that
would
say
hey.
I
should
always
be
able
to
like
every
you
know,
every
five
or
ten
minutes,
I'm
going
to
create
a
new
vm
test
to
make
sure
that
vm
comes
up
successfully
and
then
delete
the
vm,
and
if
that
stops
working,
you
want
to
find.
Like
I
mean
a
big
part
of
this
is:
if
your
service
fails,
you
want
to
find
out
about
it
before
your
users
find
out
about
it.
A
The
last
thing
you
want
to
do
is
have
a
user
file
a
support
ticket
and
have
you
go?
Oh,
my
goodness.
I
didn't
know
that
this
was
happening,
and
so
you
know
this
is
kind
of
a
synthetic
user.
How
do
I
actually
pretend
that
I'm
a
user
to
be
able
to
create
one
of
these
things
and
to
do
a
user
type
workload
and
make
sure
that
it
works
and
that's
what
cuber
healthy?
A
Is
it's
a
more
generic
version
of
what
we
had
as
as
prober
or
the
synthetic
tests
or
synthetic
checks
that
we
had
at
google?
And
so
I
think,
that's
super
cool
and
I
love
seeing
something
fill
that
gap
there,
and
so
I
didn't
coordinate
with
any
of
the
authors
around
this.
So
if
any
of
the
authors
of
cuber
healthy
are
actually
joining
us,
I
would
love
to
you
know,
love
for
them
to
say
hi
and
say
thank
you
for
doing
this
stuff.
I
think
you
know
a
lot
of
them.
A
I
believe
you
know
so
one
of
the
I
was
doing
a
little
bit
of
there's
a
talk
from
kubecon
north
america
here
from
a
couple
of
comcast
engineers,
jessie
lynn
and
chilla,
and
so
I
watched
this
beforehand.
Great
introduction
they'll
probably
do
a
better
job
of
it
than
I
will.
But
you
know
you,
you
all
come
to
see
me
flail
around
with
new
things
and
there's
also
a
blog
post
here,
around
sort
of
kubernetes
kpis
with
this.
A
A
Three
node
kind
cluster,
and
for
those
who
who
may
not
be
familiar
with
kind,
it
is
a
kubernetes
and
docker.
It's
kind
of
like
mini
cube
in
some
ways,
but
much
easier
to
in
some
ways.
Automated
and
ship.
Your
own
images
right,
creating
a
new
base
image
for
for
a
mini
cube
is
actually
really
hard,
or
at
least
I
was
never
able
to
figure
out
how
to
do
it.
A
So
when
you're,
actually
working
on
kubernetes,
you
can
actually
sort
of
create
a
new
image
for
kind
really
easily,
and
so
it
it
often
ends
up
being
a
target
that
a
lot
of
the
developers
use
when
they're
when
they're
playing
with
stuff.
But
the
way
that
you
can
bring
up
a
cluster
that
has
multiple
nodes
is,
and
the
kind
documentation
goes
through
this
stuff.
Is
you
essentially
have
this
little
yemo
file
that
configures
your
kind
cluster
and
you
can
pick
which
version?
A
Otherwise,
it's
going
to
pick
the
latest,
so
this
is
1.21.1,
or
at
least
the
latest
that
that
it
knows
about,
and
here
we're
saying
that
we
have
three
nodes.
One
is
a
dedicated
control,
plane
and
then
two
other
as
workers
and
then
other
under
the
covers.
This
uses
cube
admin
to
bring
these
things
up
and
stitch
them
together,
and
so
we
got
that-
and
I
don't
have
anything
else
running
there.
So
like
it's,
it's
a
clean
slate
cluster,
so
we
don't
have
prometheus.
A
We
don't
have
anything
like
that
running
on
it
and,
and
so
we're
just
gonna,
go
ahead
and
start
now,
one
of
the
things
that
this
does
have
like
a
simple
json
status,
page,
but
also
a
prometheus
x,
endpoint
and
influx
db,
metrics,
forwarding
and
stuff.
So
that's
actually
pretty
cool
and
yeah,
and
so
this
requires
kubernetes
1.11
above
and
helm3
to
do
it
that
way.
They
also
I'm
I'm
not
a
huge
helm
fan.
A
We
can
go
into
the
details
there,
but
I
like
to
really
like
when
I'm
installing
something
like
this,
I
like
to
to
to
really
dig
in
to
some
of
the
underlying
some
of
the
underlying
configs
that
we're
actually
pushing
there.
So
my
understanding
is
that
in
deployment
here
they
actually
do
have
a
bunch
of
deployment
yaml
that
we
can
actually
go
ahead
and
deal
with.
A
A
Look
at
what
we
got
going
on
here,
all
right,
so
there's
a
there's,
there's
a
bunch
of
stuff
going
on.
So
what
we
have
is
we
have
the
crd
for
the
kh
checks,
cage
jobs
states.
So
one
of
the
things
that
that
cuber
healthy
does
is
it
stores
its
state
as
crds,
and
so
you
can
it
and
the
multiple
instances
will
actually
coordinate
with
each
other.
There
pod
disruption
budget
for
this
stuff,
and
then
it
has
its
own
config
map
that
says
that
it's
listening
on
8080.
A
And
then
you
know
influx
stuff.
If
it's
going
to
be
pushing
stuff
and
then
it
you
know
some
sort
of
gc
stuff
that
it's
actually
dealing
with
there.
So
that
looks
pretty
cool
and
then
it
uses
a
daemon
set.
Okay,
so
so
deployment
service
account
damon
set.
So
I
didn't
know
it
used
a
daemon
set.
That's
interesting!
I
wonder
what
the
damon
set
is
used
for.
So
maybe,
if
you're
running
a
a
check
on
every
node,
it
uses
the
daemon
set
to
go
ahead
and
do
that
and
then.
A
Family
texted
me
here
so
alex
is
saying
they
had
trouble
installing
the
cellium
cni
load
balancer
on
on
kind
yeah.
We
should
let
duffy
know
about
that,
because,
because
duffy's
now
working
on
psyllium
stuff
and
he's
a
huge
kind
fan,
so
my
guess
is
that
he's
all
on
top
of
making
sure
that
psyllium
and
kind
work
well
together.
A
Auto
mount
the
service
account
token,
which
is
great,
and
then
it
has
an
image
with
all
the
probes
man.
This
is
a
great
example
of
a
it's.
You
know
you
read
some
of
the
stuff
and
you're
like
clearly
somebody's
running
this
in
production,
because
they've
hit
their
heads
on
all
the
different
things,
and
you
know-
and
you
know
how
to
actually
run
this
stuff.
You
know
for
real
is
really
really
interesting.
A
A
So
it
looks
like
they
have
a
bunch
of
their
own
checks
that
get
installed
by
default
here.
So
there's
one
around
creating
daemon
sets
one
around
creating
deployments
and
then
one
around
dns.
It
looks
like
so
not
only
do
they
install
this
stuff,
but
they
actually
have
some
stock
probes
that
they're
going
to
run
out
the
gate.
So
let's
go
ahead
and
make
that
work.
So
I'll
do
so.
I
like
using
cap
for
doing
this
stuff
because
again
it
shows
you
a
lot
more
of
what's
actually
happening
so
cap
deploy.
A
The
so
I
always
have
to
do
help
here.
This
is
part
of
the
carvel
tool
chain
that
we're
using
for
deploying
stuff
and
so
yeah.
So,
okay,
so
cap
deploy
we're
gonna
name,
it
uber
healthy
as
the
app,
and
so
this
is
sort
of
the
group
of
of
resources
that
you're
talking
about
and
then
f
is
cooperhealthy.yaml
and.
A
A
A
A
B
A
A
A
A
Yeah,
so
this
is
something
new
and
and
so
yeah.
I
want
to
dig
into
that
and
understand.
So
this
is
great.
This
is
one
of
the.
This
is
why
I
like
using
cap
is
that
it
shows
you
this
sort
of
terraform
level,
view
of
what
you're
about
to
change
and
we're
going
to
create
a
whole
bunch
of
stuff.
Here,
a
daemon
set
some
of
the
cage
checks,
a
deployment,
a
whole
bunch
of
roles,
a
deployment
for
cuber
healthy
itself.
So,
let's
just
let
her
rip
and
actually
let
this
stuff.
A
A
A
So
it
said
I
was
launching
a
deployment
and,
oh
I
oh,
this
is
the
name
of
the
thing.
This
is
a
cuber
healthy
check,
okay,
so
this
is
it's
adding
some
checks
and
we're
doing
this
okay
cool.
So
we
got
that
up
and
running
okay.
So
it's
doing
its
thing.
So
that's
cool!
Now,
let's
go
through,
and
so
we
got
the
thing
deployed
to
some
definition
of
deployed
and
after
insertion
will
only
be
available
from
within
the
cluster
at
the
service
plot
to
expose
it,
you
can
do
external
checking
service.
A
A
Do
not
disturb
here
because
scott's
actually
was
hitting
me
on
our
internal
slack
here.
What
are
you
saying
here,
scott
trying
to
tell
me
what
I'm,
what
I'm
hitting
oh
okay
you're
bumping
in
not
being
able
to
use
a
slash
and
the
value
of
a
label?
I
think,
okay,
that
may
be
what
was
being
hit
there
but,
like,
I
think,
we're
good.
Now.
I
think
we're
good
okay.
So
here
we
have
so
this
thing
is
up
and
running.
A
We
have
cuber
healthy
with
services,
okay,
so
now
what
I
can
do
here
is
why
don't
I
the
pods?
Oh
this,
isn't
the
default
name
space
we'll
go
to
the
cooper,
healthy
namespace!
There
we
go,
and
now
we
can
do
namespace
overview
here,
and
here
we
see
the
pods
the
deployments
and,
I
believe,
config
maps
secrets,
so
all
sorts
of
good
stuff.
Now
what
do
we
have?
Let
me
make
sure
that
we
didn't
have
other
stuff
accidentally
show
up
in
default.
A
A
Yeah,
so
I'm
not
sure
what's
going
on
here,
so
let
me
go
through
and
I
want
to
make
sure
that
we
don't
end
up
with
things
screwed
up
here.
So
let
me
go
through
and
what
I'm
going
to
do
is
I'm
going
to
go
back
here
and
I'm
going
to
deploy
cap.
I
want
to
actually
bring
it
back
down
and
then
redeploy
it
so
I'll
delete.
A
B
A
And
that
looks
right.
There
may
be
all
right,
so
there
may
be
other
stuff,
that's
sort
of
like
stuff
random
stuff
that
was
left
around,
but
I
think
that'll
at
least
get
us
going.
Okay,
sorry
about
that,
all
right!
So
now
yeah
because
there's
like
a
root
ca
there
and
everything
in
the
default
cuber
healthy
there
we
go
okay,.
A
Oh,
it's.
Actually
it's
it's
running
a
a
probe
already,
so
that's
pretty
cool
okay.
So
it's
already
ran
some
stuff
on
us
and
there's
a
deployment
here,
and
this
thing
is
no
replicas.
Oh,
this
is
exist,
so
this
is
like
one
of
the
tests
that
it's
already
running,
okay,
so
we're
going
to
click
on
cuber,
healthy
and
one
of
the
things
is,
we'll
start
a
port
forward
to
it.
This
is
I
love
this
in
octane,
so
maybe
come
on
octane.
You
can
do
it.
A
B
A
A
It
does
some
some
master
election,
oh
and
then
here's
the
deployment
so
okay.
So
this,
then
it
actually
ran
all
three
of
the
checks
that
we
identified
here.
Let
me
make
this
a
little
bit:
bigger,
hey
vlad
how's
it
going
man,
so
it
ran
all
three
of
these
checks
and
then
here's
the
latest
status
of
this
and
there's
probably
a
you,
know
other
metrics
that
we
can
actually
take
from
it.
But
this
is
really
cool,
so
yeah,
so
this
is
essentially
what
we're
looking
at
here.
A
So
security
considerations.
This
thing
that
I'm
hitting
this
thing
that
I'm
hitting
here
is
this
is
all
unauthenticated.
This
is
plain
old
http,
so
you
don't
want
to
expose
this
outside
of
your
cluster,
but
this
is
really
cool,
so
we
got
it
working.
This
is
already
adding
value.
Clearly,
you'd
want
to
set
this
up
to
sort
of
a
monitoring
system.
A
So
if
some
of
these
start
failing,
you
actually
get
you
actually
get
notified
of
it
clearly,
but
that's
huge
right
just
actually
getting
and
going
there,
and
so
that
is
sort
of
the
quick
getting
started.
I
want
to
dig
a
little
bit
deeper
here,
so
prometheus,
metrics
and
point
support.
I
want
to
actually
at
least
look
to
see
what
that
looks
like
and
let's
see
where
their
documentation,
how
okay
so
cool.
A
Oh,
I
see
the
job
runs
only
once
so.
It's
a
manually
triggered
thing,
whereas
the
checks
actually
run
on
a
cron
job.
So
here
you
can
see.
Okay,
I
want
to
run
every
30
seconds
with
the
two
minute
timeout
and
then
I
want
to
add
this
to
the
pod
and
then
it'll
just
launch
a
pod
here
and
then
how
does
it
figure.
A
A
A
A
What
what
don't
you
like
about
their
capitalization?
The
fact
that
some
of
it
is
capitalized
and
some
of
it
isn't
casing
choices
on
that
json
output?
I
think
the
only
thing
is
that
uuid
isn't
capitalized.
The
rest
looks
pretty
pretty
normal
to
me.
These
things
are
actually
coming
from
the
the
actual
namespace
and
then
the
name
of
the
check
that
they're
actually
dealing
with
there.
A
A
A
Each
of
the
checks,
including
the
namespace
that
they're
in
with
this,
is
a
whether
they're,
healthy
or
not,
so
this
will
be
one
or
zero,
whether
they're
healthy,
that's
what
it
means
when
it
says
that
it's
a
gauge
and
then
this
is
duration
in
seconds.
Just
the
check
run.
How
long
is
it
actually
taking
to
be
able
to
do
this
stuff
and
then
same
thing
for
job
and
okay,
cool
yeah?
All
right
that
is
super
cool.
That's
already
useful
right
off
the
bat
there,
let's
see,
here's!
A
A
Yeah,
it
looks
like
it
came
out
of
comcast,
it's
now
a
cncf
sandbox
project,
but
one
of
the
things
that
you
can
do
is
you
can
do
an
image
pull
check
which
is
like
this.
You
know
this
just
checks
to
make
sure
that
you
can
pull
a
particular
image
and
will
already
run
that,
and
I
love
that
you
know
they
have
essentially
sample
yaml,
that
you
can
just
sort
of
copy
and
paste
to
make
this
stuff
happen,
which
is
great,
and
so
you
know
you
can
do
specs
for
a
event.
A
Reason
for
cron
jobs,
so
yeah
all
sorts
of
interesting
checks,
and
you
could
imagine
that,
like
hey,
you
could
always
add
your
own
checks
here
to
do
stuff.
So
there's
a
dns
static
check.
You
know,
hey
is
dns
working
here
and
so
here's
an
internal
one,
here's
an
external
one
for
like
google.com,
so
yeah.
So
let's
try
and
create
this
and
actually
see
what
happens.
A
A
A
Oh
yeah,
it's
running
it
all
right,
so
this
is
an
interesting
thing
from
a
security
point
of
view.
Is
that
wherever
the
the
kh
check
crd
is
that's
where
it
goes
ahead
and
adds
the
pods
or
runs
those
checks,
and
so
it
won't
do
anything.
It
looks
like
cross
namespace,
which
is
important
from
a
security
point
of
view.
A
So
essentially
what
you're
saying
is
that
if
you
give
somebody
our
back
permissions
to
create
checks
within
a
namespace,
then
they
essentially
are
launching
stuff
in
that
namespace,
and
so
that's
kind
of
like
giving
them
root
on
that
namespace,
because
you
could
write
a
check
that
points
at
a
pod
that
does
stuff
so
like
it's
just
like
you're,
giving
them
permission
to
create
pods,
but
it's
all
still
contained
within
a
namespace,
which
is
a
good
thing
all
right.
So
it
looks
like
that
that
ran
and
then
yeah
here
we
see
pod
deleted.
A
A
A
A
A
You
know
how.
How
often
are
we
doing
these
things
every
two
minutes
with
a
15
minute
timeout,
so
I
don't
know.
I
don't
think
that
there's
a
way
to
explicitly,
I
don't
think,
there's
a
way
to
explicitly
trigger
it.
Okay,
so
let's
let
that
go
and
see
if
it
happens
again
or
whether
that
was
a
transient
thing
that
happened
and
if
it,
if
it
happens,.
A
Containers
yeah:
if
it
happens
again,
then
we
can
try
and
debug
it,
but
it's
gonna
take
a
while
before
it
does
that.
Okay,
I
want
to
start
reading
some
more
documentation
here
on
this
stuff.
So
here's
all
these
different
checks.
What
does
it
take
to
actually
write
your
own
check?
What
is
sort
of
the
contract
there,
and
so.
A
A
A
So
do
check
stuff
and
then
oh
and
then
there's
a
check
client
here,
oh
fascinating,
so
they
actually
have
their
own
client
for
reporting
things
back
and
so
there's
a
client
package
there
and
then
you
can
similarly
do
stuff.
Oh,
they
have
a
client
package
for
javascript
and
you
can
do
it
another
language.
This
is
what
I
want
to
see
like
what
is
the
protocol
here.
So
so
there's
a
kh
reporting,
url
environment
variable.
You
send
a
post
to
that
with
the
following
body.
A
A
So
that's,
pretty
cool,
I'm
wondering
like
I'd,
be
interested
to
sort
of
brainstorm.
What
are
some
other
things
that
you
might
want
to
include
from
the
check
right
like?
If
you
have
an
error?
Do
you
want
to
maybe
have
a
pointer
to
a
log
file
a
way
to
collect
the
logs?
So
one
of
the
things-
and
I
was
talking
to
to
vlad
earlier
today-
is
that
you
know
looking
at
this
compared
to
something
like
sauna
boy
or
crash
d,
one
of
the
things
we
do
with
sono
boy
and
crash
d.
A
Is
it
in
a
lot
of
ways
it's
similar
to
this,
but
we
collect
a
heck
of
a
lot
more
sort
of
debugging
information
about
what's
happening,
so
the
output
is
not
just
sort
of
a
go
no-go
decision.
It's
really
like
a
lot
more
details
in
terms
of
what
happened
and
what
went
wrong
when
you
go
ahead
and
hit
an
error,
you
can
pass
in
things:
okay,
yeah
that
looks
pretty
cool
and
then
like,
and
then
it
looks
like
the
go
package
for
doing
this
stuff
is
pretty
dead,
simple.
A
It
just
sort
of
you
report
failures
or
your
report
success
one
way
or
the
other
and
that'll
actually
do
the
post,
and
it's
one
or
the
other,
pretty
darn
simple
here
I
you
know,
I
think
there
is.
I
wonder,
you
know
my
security
brain
always
comes
into
here.
It's
like
okay,
like
like,
if
you
like
how
secure
are
those
urls
like
one
of
the
things
that
they
like
signed
like
cookies?
You
know
that's
one
of
the
things
that
I
would
do.
There
is
to
really
check
that.
A
A
I
wonder
why
it's
failing.
I
wonder
why
it's
failing
and
then
it
goes
ahead
and
it
deletes
the
pod.
So
it's
a
lot
harder
for
us
to
actually
figure
out
what
happened
there,
heroin
waiting
for
pod
to
start
gauge,
okay,
false.
So
this
is
what's
going
on
there
and
we
don't
have
the
like.
The
pod
is
long
gone
here.
A
Yeah,
so
it
seems
like
it
might
be
something
like
at
the
cube
level.
So
like
yeah,
I
scott,
I
I
think
you
showed
up
late.
I
was
saying
in
some
ways
this
feels
a
little
bit
like
you
know:
prober
external
probes,
at
google.
I
do
want
to
debug
what's
going
on
here
and
I
I'm
wondering
all
right,
let's
go
through
and
I
want
to
look
at
the
code,
for
maybe
we
can
figure
out
what's
happening
here.
A
A
A
A
A
Okay,
I
wonder
if
there's
okay,
so
it
looks
like
one
of
the
key.
The
key
problems
here
was
that
we
had
an
owner
ref
that
that
actually
crossed
namespaces
and
that's
a
problem
here,
because
it
was
trying
to
set
an
owner
ref,
it's
okay.
That
was
a
warning,
though
okay,
so
that's
actually
not
going
to
be
a
problem
and
then
there's
a
failed
mount
warning,
but
I
wonder
if
that
actually
caused
things
to
actually
really
break
down.
Okay,
so
let
me
go
through
and
let's
see,
if
let
me
delete
this.
A
A
A
A
Okay,
I
don't
see
a
bug
on
this.
I
want
to
file
a
bug
on
this
bottom,
wait
until
after
tgik
to
actually
do
it,
because
I
want
to
make
sure
that
I
can
do
a
little
bit
more
debugging
before
I
go
ahead
and
file
a
bug
for
them.
But
my
guess
is
that
this
is
something
that
got
locked
down
in
a
later
version
of
kubernetes,
because
I
wouldn't
be
surprised
if
you
know
they're,
not
testing
this
on
121.
A
A
Oh,
let's
see
ortel
is
asking
about.
How
do
I
get
like
this
stuff
on
my
mac?
This
is.
This
is
called
menu
stats
this
this
utility
here
so
yeah
you
can
get
to
pay
for
it,
but
it
has
all
of
these
cool
little,
like
you
know,
stats
that
you
can
get
on
your
mac,
so
that's
kind
of
cool,
so
yeah.
So
what
else
can
we
play
with?
I
think
you
know
we
could
try.
A
You
know
running
some
more,
let's,
let's
try
and
configure
another
useful
check
or
we
could
try
and
actually
debug,
what's
going
on
and
actually
file
a
real
bug
and
maybe
actually
contribute
back
to
them.
What
do
you
guys
think
so
yeah?
So
writing
a
custom
check
for
an
app
so
like
what
I
would
actually
do
here
for
a
custom
check
for
an
app
is
you
know
you
could
definitely
do
like
an
http
check.
A
A
A
But,
like
you
know,
if
we
go
and
if
we
look
at
the
you
know
what
it
takes
to
write
your
own
check,
container
you're,
essentially
returning
either
an
error
or
an
okay,
I
would
love
to
be
able
to
return
another
snippet
of
yaml
here
and
then
have
you
know
some
way
for
cuber
healthy
to
actually
keep
track
of.
Like
here
are
the
past
five.
A
You
know
runs
of
this
thing
or
the
past
end
runs
of
these
things
and
then
actually
sort
of
be
able
to
say
well,
you
saw
success,
failure,
failure,
success,
success,
success
and
then
start
digging
into
that
by
actually
pulling
out
some
more
data
from
it.
I
think
that
would
be
really
really
cool
because
just
like
hey,
it
worked
okay
equals
true
like
sometimes
you
want.
You
know.
Why
did
it
work?
What
was
the
response?
How
long?
A
I
guess
the
how
long
it
takes
we
got
but
yeah
so
that
that
would
actually
be
really
cool,
because
visualizing,
the
history
of
these
things
can
often
be
really
important
when
you're
trying
to
debug
these
systems,
but
you're
going
to
get
some
of
that.
I
guess
if
you're
scraping
this
with
prometheus,
if
you
see
your
check
start
to
fail
at
the
same
time,
you
see
some
other
metrics
start
to
go
wonky,
that's
actually
a
really
good,
really
good
indication
too.
A
A
A
A
A
A
A
Annotations
all
right,
one
of
the
things
that
I'm
going
to
actually
see
here.
Okay,
so
here's
what
I'm
going
to
do
is
they
do
have
a
let's
see
they
had
command
line
flags.
One
of
these
things
is
debug,
let's
enable
debug
logging
on
this
thing,
because
I
think
that'll
definitely
help
us
to
be
able
to
figure
this
stuff
out.
So
let's
look
at
deployments
in.
A
A
So
one
of
the
dangers
here
like
let's
say
that
you
have
something.
That's
like
hey.
Let
me
test
that
I
can
launch
a
pod.
Is
that
if
your
check
self
fails,
it
may
actually
leave
things
around
and
actually
leave
them
running,
and
so
you
know
if
you're
writing,
good
checks.
One
of
the
things
you
may
want
to
do
is
actually
have
your
check
the
next
time.
A
You
leave
that
pod
there
you
create
another
pod,
something
fails,
you
leave
it
there,
and
so
you
know
that
is
like
a
danger
here
and
then
at
some
point
you,
like
you
know
your
cluster
fails
and
you
show
up-
and
it's
like.
Why
do
I
have
like
you
know
a
thousand
abandoned
pods
here
that
I'm
actually
having
to
deal
with,
and
so
I
I
don't
know
how
much
this
thing?
A
A
A
A
Pod,
okay,
but
it
doesn't
actually
say
what
it
did
to
check
that
to
actually
create
that
pod.
Okay,
well,
let's
go
through
and
let's
try
and
actually
add
this
to
namespace
default
again.
So
what
we're
going
to
do
is
I'm
going
to
go
to
custom
resources
here
and
I'm
going
to
delete
my
external
resource
that
we
had,
because
I
don't
want
to
have
two
of
these
things
sitting
around
that'll
be
confusing.
A
A
A
A
A
A
A
A
A
Kuber
healthy
deployment
and
look
at
this
pod,
I
should
be
able
to
do
a
start
four.
Okay.
So
now
we
have
at
least
something
that
we
can
hit
here.
Okay,
error:
when
waiting
for
pod
to
start
the
cage
checkbox
is
deleted.
Waiting
for
start
failed
okay,
so
we
got
that
so
now,
let's
actually
go
through
and
we'll
look
at
the
okay.
We
got
a
lot
of
stuff
scrolling
here.
We
want
to
look
for
dns
status
external
as
our.
A
A
A
Oh
yeah,
what
do
we
got?
It's
a
watch
and
I
think
this
will
actually
dump
the
yaml
or
any
changes
to
any
yaml
as
we're
actually
doing
stuff.
So
now,
if
we
actually
go
through
and
we
wait
for
this
thing
to
run
again,
we
should
see
odds
come
as
we
go
as
a
you
know.
We
should
see
the
yam
will
get
dumped
as
they.
A
A
A
And
then
downward
api,
which
is
the
name
space
that
you're
running
in
this
thing,
this
config
map
is
from
the
cube
root.
Ca
crt
is
that
the
name
of
the
config
map
that
it's
actually
trying
to
yeah?
Okay?
So
it's
pulling
that
in
so
that
looks
fine.
Maybe
it's
the
service
account
which
service
account
service
account
name
is
default,
so
that
shouldn't
be
a
problem.
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
B
A
B
A
A
A
A
A
A
A
A
B
A
A
A
A
A
A
Yeah,
that's
the
problem,
because
here's
the
thing
running
error
waiting
for
node
to
reach
minimum
age
nodes
blah
is
forbidden.
The
access
in
dns
checker
determine
that
that
it
was
okay
and
so
now.
The
summary
here
is
that
this
thing
is
running
and
doing
its
thing.
Okay,
so
that's
the
problem.
Okay
yeah!
I
may
have
completed
it
back,
okay,
so
what
I'm
going
to
do
here
is
I'm
going
to
file
an
issue
here
so
issue
new
issue,
bug.
A
A
A
A
A
A
A
Yeah,
that's
what
I
that's.
What
I
was
guessing
is
it's
probably
something
with
with
a
newer
version
of
kubernetes,
locking
this
stuff
down
a
lot
more
than
than
previous
ones,
which
is
probably
good
right,
so
you
are
johnny
dog,
so
yeah,
okay,
I
really
like
the
project
man.
It's
there's
a
lot
of
really
great
stuff
here.
You
know.
Let
me
go
ahead.
I
think
we're
running
out
of
time.
Let
me
wrap
up
and
and
thanks
for
joining
us.
I
really
appreciate
that.
A
The
I
love
checks
like
this
and
it's
clear
reading
through
the
code
reading
through
the
ammo
that
this
is
something
that's
being
used
in
production,
that
you
know
a
lot
of
the
the
sort
of
the
the
places
where
you
can
stub
your
toe
they've
been
dealt
with
you
del
del,
for
you
already
dealt
with
already
for
you.
I
love
the
the
ability
to
do
all
these
these
checks.
I
would
love
to
see
richer
history
around
what
happened
with
these
checks
more
ways
to
debug
it.
A
I
think,
especially
with
http
checks.
What
you
know
what
we
call
the
google
probes
having
the
history
of
when
probes
were
run,
what
the
status
was
the
hbo
that
was
sent
and
then
received
is
often
really
useful.
Doing
http
checks
like
that.
Sometimes
you
want
to
maybe
have
a
regular
expression
that
you're
checking
against
versus
just
the
explicit
content
oftentimes
when
you
have
a
check
that
fails
for
an
http
check,
the
content
and
the
quality
of
that
failure
will
really
help
you
zero
in
on.
A
It's
using
crds
to
actually
sort
of
you
know
smart
way,
store
all
the
metadata
about
this,
but
having
some
way
to
maybe
have
this
actually
throw
some
logs
at
a
logging
system
with
trace
ids
so
that
you
could
actually
then
start
saying
you
know
and
then
having
some
way
to
go
from
like
oh
here
was
the
check.
Here's
all
the
runs
of
that
check,
and
then
this
particular
you
know
and
here's
the
history
of
when
they
were
successful
versus.
A
Not
let
me
drill
into
that
failure,
somehow
being
able
to
start
providing
that
experience
would
be
really
really
useful.
You
know,
especially
as
people
start
using
these
things
more
and
more
for
application
checks
right.
I
think
the
it's
clear
that
this
was
built
to
make
sure
a
cluster
is
healthy,
but
you
can
definitely
see
that
this
could
be
used
to
ensure
that
an
application
is
healthy
and
once
you
start
doing
stuff
at
the
application
level,
you
really
want
to
make
sure
that
you
can
make
it
as
self-service
as
possible
for
those
application
teams.
A
If
somebody
hits
a
problem,
you
want
them
to
be
able
to
debug
it
themselves,
figure
it
out
and
not
be
like
hey
your
tools,
broken
right
and
so
providing
that
you
know
the
capability
of
that
dashboard,
so
the
application
team
can
debug.
It,
I
think,
is
a
really
interesting
direction
to
take
this,
especially
with
the
http
probes,
if
you're
in
a
multi-cluster
world
being
able
to
one
really
interesting
case
would
be
setting
this
up
to
check
things
across
cluster
right.
A
A
I
mean
the
the
you
know
the
this
idea
of
like
a
contract
with
a
container
where
it's
like,
hey
you
read
this
thing.
You
do
this
and
you
now
have
this
an
extensibility
mechanism
where
we'll
run
pods
with
this
container
image.
I
think
that
that's
really
cool.
I
would
love
to
see
more
examples
of
us
using
you
know,
sort
of
interfaces
and
sort
of
contracts
with
containers
to
do
this
type
of
extensibility.
I
think
that
that's
super
cool,
so
that's
sort
of
quick
thoughts
there.
A
Thank
you,
everybody
for
joining
me
as
I
sort
of
delved
into
this
and
figured
out
what
was
going
wrong,
really
cool
tool.
I'm
sure
a
lot
of
folks
will
find
ways
to
to
make
it
make
it
work
for
them
and
also
super
clear
that
it's
being
used
in
anger
and
actually
bringing
some
value.
I
encourage
everybody
to
watch
the
the
kubecon
talk
that
I
that
I
listed
here
from
kubecon
north
america
2019..
A
I
think
that
also
gives
a
really
great
overview
of
what
the
project
is
and
why
they
did
it,
and
you
know
it's
probably
a
little
bit
long
in
the
tooth
now.
So
I'm
sure
things
have
moved
forward
since
then.
So
with
that
I'm
going
to
go
ahead
and
sign
off.
Thank
you
everybody
for
joining
me
for
yet
another
tgik.
I
love
doing
this.
It's
always
it
takes
a
lot
of
energy
to
get
going,
but
I
always
feel
energized
at
the
at
the
end.