►
Description
Off the Beaten Path: An Explorer’s Guide to Kubernetes - Taylor Thomas, Intel
In the past year, Kubernetes proved itself as a powerful tool for creating and managing service based architectures. However, as with anything new, it is easy to find yourself lost when the complexity of a service increases and you enter into uncharted territory. Using our team’s work with Jenkins, Snap, and Cassandra in Kubernetes as a backdrop, this presentation will deep dive into specific pain points and how we solved them. Coupled with code examples and demonstrations, this presentation will show a pattern of how to solve undocumented issues and invite others to the discussion of documenting solutions to similar problems.
About Taylor Thomas
Taylor Thomas is a Cloud Software Engineer for the Software Defined Infrastructure team at Intel working on Kubernetes, CI/CD, and the Snap open telemetry framework. Taylor has a background in IT Operations and API development and is one of the co-organizers for the Docker Meetup in Portland. He is a Gopher by way of Node.js and enjoys working in DevOps. You can find him on Twitter (@_oftaylor) or on GitHub (thomastaylor312).
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Okay,
we're
gonna
go
ahead
and
get
started,
then
so
welcome
everyone.
As
you
can
see,
my
talk
is
titled
of
off
the
beaten
path
and
explores
guide
to
Coober.
Nettie's
I
actually
was
influenced
by
the
title
by
one
of
my
co-workers,
because
that's
how
he
described
what
we
were
doing
with
some
of
the
stuff
we're
trying
to
figure
out
inside
a
coup
brunettes,
but
my
name
is
Taylor
Thomas
I'm,
a
cloud
software
engineer
engineer
at
Intel,
I
work
with
their
continuous
integration,
continuous
delivery
pipeline
on
Cooper
Nettie's.
A
Of
course,
some
direct
work
with
Cooper
neighs,
and
also
the
snap
telemetry
framework.
She'll,
explain
in
a
second
and
I'm
one
of
the
meetup
organizers
for
the
docker,
meet
up
in
Portland
and
I'm,
also
a
home
automation,
amateur
much
to
the
annoyance
of
my
wife,
a
lot
of
times
and
a
hiker.
And
if
you
want
to
see
how
some
of
the
other
stuff
I'd
and
there's
my
twitter
and
github
handles.
A
So
I
just
want
to
give
an
overview
of
what
I'm
going
to
try
to
go
over
here
and
the
goal.
What
we're
going
over.
So,
just
as
a
note,
this
is
for
intermediate
to
advanced
users
of
cooper
Nettie's,
I'm
expecting
that
you've
used
it
and
have
an
idea
of
what
the
different
objects
are
and
how
to
use
them.
A
So
just
a
note
about
what
we're
doing,
we
are
still
in
a
nascent
stage
of
in
terms
of
like
size,
but
we
have
about
17
different
repos.
That
goes
up
about
once
every
two
weeks
because
of
all
the
plugins
that
we
use
with
snap
all
the
different
Cooper
Nettie's
applications
we
test
John
Jenkins
before
we
deploy
them.
We
have
about
thirty
two
jobs
a
day
that
was
about
two
weeks
ago,
so
it's
probably
gone
up
since
then.
That
goes
up
quite
often,
and
then
we
have
a
CI.
A
Cd
system
has
to
be
able
to
run
bare
metal
tests,
VMs
and
containers
for
unit
tests
or
or
large
integration
tests
between
different
parts
of
an
application.
We're
also
running
Cassandra
to
save
different
data
sets
from
some
of
the
other
projects
that
we're
working
on,
and
we
do
a
lot
of
our
telemetry
gathering
with
snap
now
just
a
quick
overview
of
what
snap
is.
If
you
want
to
find
out
the
full
details,
you
can
come
talk
to
one
of
us
of
the
intel
booth
or
talk
to
really
anyone
here
from
intel
right
now.
A
But
snap
is
a
really
cool
tool
that
makes
data
gathering
really
easy,
instead
of
it
being
very
specific
to
a
specific
type
of
data.
It
puts
an
abstract
interface
in
front
of
it,
where
it'll
expose
all
the
metrics.
A
machine
can
get
whether
those
be
application,
metrics,
CPU,
metrics
memory
or
things
like
docker
instruction,
docker
statistics,
and
so
snap
is
a
tool
that
we
use
for
monitoring
in
Coober,
Nettie's
and
I'll
show
how
we
do
use
that
later.
A
So
I
said
in
the
beginning,
we're
going
to
go
over
why
we
chose
Kuber
Nettie's,
some
things
about
configuration
using
the
KU,
béarnaise,
api
volumes,
containers
and
technical
debt,
and
so
the
question
why
we
chose
cabernets.
It's
actually
really
interesting
to
see
why
people
chose
to
use
cooper
Nettie's,
because
most
of
us
are
here
because
we're
either
using
it
or
interested
in
it.
So
I'm
not
like
trying
to
convince
anybody,
I'm,
just
I'm,
just
given
the
reason
why
we
did
so
the
thing
that
really
got
us
for
it
was
we
can
iterate
and
deploy
quickly.
A
Now
that
took
some
work
to
get
to
it's,
not
something
magical
that
happens
when
you
start
in
Cooper
Nettie's,
but
once
you
have
it
up
and
running,
we
can
deploy
things
very
quickly.
It
also
allows
you
some
really
fine
grained
control
of
your
applications.
If
you
are
here
for
the
last
presentation,
we
talked
a
little
bit
about
the
scheduling,
resources
and
things
I
mean
you
can
get
some
very
fine
grain
detailed
tweaking
of
your
applications
and
coo
brunette
ease
and
the
different
API
objects
are
very
logical,
so
well.
A
It
also
is
just
really
when
I
talk
about
fast
spin
up
speeds
here,
I'm
talking
like
we
can
spin
up
Cassandra
or
Jenkins
in
about
two
minutes,
I'm
a
minute
to
two
minutes.
So
it's
a
really
quick
process.
Once
we
get
it
all
set
up,
so
first
thing
configuration
and
we
have
a
set
up
Jenkins.
Now,
just
two
out
of
curiosity,
how
of
you
have
set
up
or
configure
to
Jenkins
server?
Okay,
yeah!
That's
what
I
thought?
Okay!
So
it's
a
little
complex
like
this,
or
maybe
this
or
this
one,
that's
not
even
the
whole.
A
Let's
lie
like
I
can
keep
going
for
about
three
or
four
more
I,
think
I
tried
just
guessing
but
based
on
how
much
I
was
grabbing
and
that
doesn't
even
include
all
the
security
settings
or
any
of
that
in
other
stuff.
You
have
to
configure
for
Jenkins
to
work,
and
so
how
do
we
solve
this?
In
our
case,
so
we
had
to
use
the
coup
burn,
ASAP
I'm,
going
to
explain
a
little
bit
about
what
we
did
with
Jenkins
and
some
of
the
other
things
we
did
as
well.
A
So
the
issue
we
ran
into
with
Jenkins
is
that
to
bootstrap
things
in
Jenkins
we
hit
you
have
to
have
we
have
to
password,
so
we
use
github
Oh
off.
We
need
an
OAuth
token.
We
need
a
password
to
be
able
to
access
repos.
We
need
a
private
key
and
you
can't
just
shove
that
stuff
around
really
easily
in
Jenkins
you
can't
just
set
it
to
an
environment
variable
or
or
that
you
have
to
have
in
a
config,
and
so
it
was.
A
We
really
couldn't
do
it
just
by
putting
in
a
config,
mappers
or
a
secret,
and
so
we
had
to
use
a
bootstrapping
script
which
I'll
show
how
we
did
that
to
get
all
those
things
set
up.
We
also
had
to
use
it
to
get
information
and
snap
because
snap,
when
it's
running
in
the
daemon
set
to
gather
all
the
data,
it's
running
on
a
host
network
and
if
you
haven't
ran
into
this
before,
if
you
have
something
running
on
a
host
network,
it
can't
resolve
the
dns
address
of
services.
A
So
we
had
to
use
the
API
to
get
those.
So
let
me
just
explain
how
many
did
it
so
tokens?
This
is
a
lot
of
code.
I
have
my
slides
up
on
there,
so
you
can
look
at
it,
but
I
also
have
some
snippets
that'll.
You
can
access
online
later,
but
the
first
section
up
at
the
top,
the
blue
code.
Basically,
every
single
pod
inside
of
every
single
container
inside
of
a
pod
has
tokens
mounted
by
the
bike.
Uber
Nettie's
inside
of
it
at
the
VAR
run
secrets.
Kubernetes
io
service
account
inside
of
there.
A
So
once
we
do
that,
we
set
up
the
connection,
and
the
next
big
part
is
that
green
line
where
we
have
the
ad
request
property.
This
is
where
you
add,
the
authorization
header
with
that
token,
so
it
allows
you
to
do.
Is
just
a
basic
HTTP
call
I'm?
Using
that
token,
you
just
have
to
make
sure
you
put
it
in
the
in
the
proper
header
so
that
it
actually
works
for
you
and
in
case
you're
interested
in
doing
this
in
Jenkins.
A
I
also
included
that
that's
right
down
here,
I've
once
again,
full
code
snippets
online,
but
you
basically
have
something
called
a
username
and
password
credentials.
Implementation
have
one
of
those
for
private
key
and
a
bunch
of
other
stuff,
but
you
can
set
the
ID
which
I
just
set
to
blah
blah
and
then
the
username
and
password
which
I
got
out
from
the
JSON
response
from
the
API,
and
so
that
was
the
first
use
we
had
of
it
now.
A
This
tok
using
those
tokens,
are
very
helpful
in
case
you
have
to
do
some
custom
configuration
or
setting
some
setting
a
value
in
the
API
or
retrieving
a
value
from
the
API.
The
next
thing
is
the
go
library.
So,
as
I
said,
with
snap,
we
couldn't
resolve
the
snap
service
IP
address,
and
so
what
we
did
instead
is
we
had
to
go
grab
all
of
this.
We
grabbed
the
services
from
the
API
now
this
is
just
a
really.
A
This
is
like
the
first,
the
first
version
we
could
have
filtered
it
because
you
can
see
in
that
green
line
right
there
that
I,
all
you
have
to
do
is
do
a
label
selector
and
you
can
actually
select
the
service
you're
looking
for
using
that
label,
but
creating
a
client
is
really
easy,
because
the
go
library
knows
to
look
for
the
tokens
in
that
directory.
That
I
was
talking
about
earlier.
A
So
all
you
have
to
do
is
client,
new
and
cluster,
and
you
can
connect
straight
up
to
the
API
and
in
our
case
we're
able
to
just
look
for
the
service
and
then
access
the
service
IP
address
and
set
that
as
the
IP
address
that
snap
needed
to
connect
to
to
get
everything
bootstrapped
and
set
up.
So
it's
very
useful
if
your
code
is
already
and
go
like
we
had
here,
but
if
it's
not
and
go
there's
other
options
like
I'm
showing
before
now
last
one
is,
you
can
also
saw
coup,
control
or
coop
cuddle.
A
So
the
next
thing
volumes
now
volumes.
First
thing
we
learned
about
this-
is
that
lots
of
different
things
can
be
volumes.
Config
maps
and
secrets
are
very
similar
in
how
they
behave
when
they're
mounted.
You
can
also
mount
different
types
of
persistent
disks
and
a
couple
different
things
with
that.
So
a
lot
of
times
when
you
just
mount
a
config
map,
if
you
have
multiple
keys
inside
that
config
map
is
going
to
mount
each
Keys
its
own
file.
A
If
you
want
to
map
only
one
of
those
keys
as
a
file
or
specific
ones,
you
can
actually
pass
what's
in
that
top
example
of
an
items
array
and
the
items
array
can
contain
the
key
that
you're
looking
for,
and
you
can
actually
rename
what
that
file
is
going
to
be
called
inside
of
the
container.
In
our
case,
this
is
for
our
Jenkins,
and
so
we
renamed
the
config
key
to
config
dot
XML,
so
it
went
new
would
be
an
XML
type,
then
read
it
in
in
the
Jenkins
startup
process.
A
Now
also
one
thing
I
didn't
include
here,
but
this
is
a
very
recent
addition
to
one
dot
for
so,
if
you're
on
one
got
for
now,
you
can
take
advantage
of
it.
One
of
the
problems
that
we've
had
was
that,
when
you're
setting
especially
secrets
like
key
values
inside
of
a
container,
it
doesn't
work
unless
it
has
a
permission
set
of
six
hundred,
and
so
you
can
actually
do
that
now
before
you
couldn't
so
there's
two
ways
to
do
it.
A
The
first
one
is:
you
can
give
a
default
mode
to
every
single
file
in
the
config
map.
That
goes
at
the
same
level
as
name
and
items,
and
so,
if
you
put
a
default
mode
in
there,
you
can
say
default
mode,
600
and
you'll
and
everything
will
be
600.
If
you
want
to
set
it
individually,
you
have
to
go
and
break
everything
out
into
the
items
array
and
so
inside
of
items
there's
a
mode
key.
A
That
would
go
right
there
and
you
can
say
that's
600,
so
that's
actually
really
knew
it
rolled
out
with
one
dot
for
just
recently.
Now
there
can
be
problems
when
you're
trying
to
mount
some
of
these
config
maps
in
the
same
directory.
In
our
case,
we
had
there's
a
there's,
a
key
called
sub
that
allows
you
to
do
it,
but
in
Jenkins
at
least
it
doesn't
like
that
very
much.
A
It
tries
to
think
that
the
it
thinks
the
file
has
a
lock
on
it
for
some
reason
which
we're
still
troubleshooting,
but
just
be
careful
when
you're,
when
you're
putting
things
in
the
same
directory
and
be
aware
that
you
might
have
to
have
some
workarounds.
In
our
case,
we
had
to
just
do
a
dead,
sim
link
that
points
to
the
location
where
we're
mounting
the
volume
using
kuber
Nettie's.
So
that's
just
a
couple,
the
gotchas
for
you.
If
you,
if
you
see
any
others,
it's
really
good
to
document
them.
A
So
people
know
what's
going
on
okay,
so
just
a
couple
words
about
containers.
This
is
the
cloud
native
conference
here,
and
that
has
a
lot
to
do
with
how
we
build
containers-
and
we
learned
a
bunch
of
different
lessons
from
building
jenkins
snapping
cassandra
for
working
inside
of
a
container
orchestration
system
like
kuru
Nettie's,
and
so
first,
I'm
going
to
go
through
some
dues.
No
just
remember
a
lot
of
these
dues
have
a
corollary,
don't
that
we
learned
from
to
learn
how
to
do
this
properly.
A
So
the
first
thing
do
allow
external
configuration
and
what
I
mean
by
that
is
that
your
container
should
allow
you
to
set
everything
with
a
basic
config
file
mounted
somewhere
or
through
environment
variables,
preferably
both
that's
what
we
learn
from,
especially
from
Jenkins
like
you
can't
have
all
this
config
have
to
be
their
pre-existing
and
then
mount
different
values,
and
you
just
need
to
be
able
to
configure
them
using
environment
variables.
Another
one
this
one.
Sometimes
we
missed
you
just
have
to
have
sensible
defaults
when
you're
creating
your
images.
A
In
our
case,
these
are
things
like
basic
security,
timeouts
other
values
of
the
application
startup,
so
timeouts
in
our
case
were
for
Jenkins
and
how
much,
how
much
time
it
had
to
connect
between
different
agents.
We
also
have
the
idea
of
having
a
readme.
Now
I
show
an
example
of
what
I
mean
by
read
me.
This
is
a
pod
monitor.
We
created
that
a
lot
that
I'll
show
some
of
the
code
for
later,
but
I
wanted
to
show
what
we
did
in
this
case.
A
It's
very
helpful
and
the
best
containers
that
you
use
out
there
have
a
very
good
read.
Neither
you
can
find
generally
on
the
on
docker
hub
or
wherever
you're,
storing
your
containers,
and
you
can
see
exactly
how
to
configure
each
of
those
another
one
is
to
remember
is
just
for
the
love
of
all
that
is
holy,
make
it
log.
Okay,
now
I
I
can't
stress
this
enough.
We
have.
We
have
problems
when
things
not
logging
right
or
being
able
to
change.
The
logging
level
is
a
really
painful
process
of
setting
like
Jenkins
and
Cassandra.
A
You
have
to
go
in
and
set
a
different
log
at
a
command-line
flag
and
do
a
bunch
of
different
configuration
just
make
it
log
and
what
I
mean
by
a
good
logging.
Is
this
tell
you
tell
the
user
exactly
what's
going
on,
because
this
is
how
we
troubleshoot,
when
you
go
in
first,
look
at
like
probably
coop
control
describe
pods,
and
then
you
look
at
coop
control
logs
and
right
here.
I
can
see
exactly
what
this
one's
the
pod
monitor
is
talking
about.
A
I
can
see
that
there
is
no
slack
webhook
specified,
so
it's
logging,
instead
of
sending
it
to
slack
it's
starting
a
probe.
It
didn't
find
any
errors
and
how
long
it's
waiting
for
same
thing
down
here
with
another
alerting
tool,
we're
using
it
says
how
long
it's
sleeping
for
what
it
queried
did
it
get
results.
Were
there
any
hits
so
I
know
exactly
what's
going
on
inside
of
the
container
and
being
able
to
configure
that
log
level?
Another
thing
is
to
remember:
it's
ok
to
install
the
things
you
need.
A
A
Once
again,
this
is
one
we
learned
a
lot
from
from
Jenkins
and
also
from
snap
and
installing
some
of
the
things
we
did
is
that
if
you
try
to
let
Jenkins
install
its
plugins
every
single
time,
it
spins
up
that's
a
lot
of
time,
because
Jenkins
already
takes
a
minute
or
two
minutes
to
spin
up
by
itself
with
plugins
already
installed,
so
adding
something
else.
On
top
of
that,
it's
worth
the
hit
for
having
a
bigger
image
then
trying
to
install
those
every
time
so
now
on
to
some
don'ts.
A
So
first
thing
is:
don't
keep
credentials
and
configs.
What
I
mean
by
that
is.
You
should
be
able
to
specify
credentials
in
a
in
a
separate
location
outside
of
the
config.
So
you
shouldn't
have
to
specify
your
password
in
the
config
file
you
have
to
mount.
You
should
be
able
to
specify
here's
a
file
that
contains
the
password
or,
like
I,
said
before,
specify
in
an
environment.
Variable
now
don't
put
stack
traces
and
logs.
This
is
to
an
extent.
Obviously,
if
it
crashes
or
there's
some
really
serious,
unrecoverable
error
stack.
Traces
are
good.
A
I
didn't
want
to
splat
it
up
here
because
it's
really
hard
to
read,
but
we've
we've
had
those,
especially
in
Java
applications
where
it
spits
out
the
stack
trace
that
goes
on
for
like
two
pages
and
how
there's
one
line
of
information
I
need
in
there.
Then
you
try
to
throw
that
to
elasticsearch
and
Lassiter's
I,
don't
know
what
this
is
and
there's
20
entries
for
that
single
thing
that
went
on.
That's
not
helpful.
A
Another
thing
is:
don't
over
customize.
We
learn
this
really
the
hard
way.
Just
saying
how
many
things
we
add
it
on
because
then
you
have
more
layers
that
could
fail
on
you.
If
you
can
use
a
default
image
and
customize
it
just
using
the
config.
Do
that
it's
much
better
than
customized
we've
been
going.
We
started
with
actually
a
lot
of
customization
to
Jenkins.
We've
been
slowly
rolling
that
back
as
we
have
figured
out
how
to
do
it
with
better
methods.
A
So
just
a
note
on
technical
debt,
I
so
really
as
you're
doing
Cooper,
Nettie's
and
spinning
things
up
in
it.
You'll
have
to
refactor
things
anyway,
so
it's
okay
to
take
on
some
technical
debt,
and
you
don't
want-
and
you
don't
want
to
leet,
make
it
so
hard
to
get
started
because
you're
worried
about
taking
on
technical
debt.
Two
examples
that
we
just
we
did
to
get
started
was
credential
baking
and
token
mountain.
So
like
bad
security
practice
mount
is
mounting
credentials
and
putting
credentials
inside
of
a
container.
A
We
took
that
hit
for
a
little
bit
to
get
started
because
we
had
a
secure
container
registry,
that
was
private
and
it
would
allowed
us
to
spin
it
up
and
get
going
quicker.
We've
gone
through,
we've
removed
that
and
we've
been
able
to
mount
it,
but
it
allowed
us
to
get
through,
especially
with
Jenkins
just
getting
it
started
up,
and
so
it's
okay
to
take
on
technical
debt
like
that,
the
other
one.
This
is
actually
one
that
has
a
good
code
snippet
along
with
it
I,
for
example.
A
If
you
do
not
want
to
have
a
pod,
have
access
to
the
API,
because
that
token
that
I
talked
about
earlier
is
mounted
by
default
in
every
single
container,
and
sometimes
you
don't
want
that
container
to
have
access
to
the
API.
You
could
the
only
way
around
that
right
now
they're
working
on
a
solution
for
this
a
better
solution,
but
right
now
you
can
actually
mount
an
empty
volume.
There
that's
a
little
hacky
but
yeah
question.
A
A
So
we
got
it
working
and
we
love.
We
all
love.
This
meme
works
fine
in
dev
right,
but
the
problem
is,
is
we
do
DevOps?
So
the
problem
is,
we
are
the
ops
people
who
just
developed
ourselves
into
a
corner,
so
we
got
to
this
question
of
things
work
now
what
we
had.
What
did
we
do
to
ourselves?
We
had
some
problems
to
to
fix
and
I'll
go
over
those
and
how
we
fix
them.
So
the
first
thing
we
had
was
secrets
all
over
the
place.
A
So
we
wanted
to
get
out
of
that
method
of
sharing
secrets
around,
and
we
also
had
dozens
of
manual
changes
required
for
each
one.
All
the
files
were
very
close
to
each
other,
and
so
you
had
to
just
change
a
little
bit
for
each
one
for
public
for
private,
for
staging
for
dev
and
that
also
led
to
our
copy
paste
reuse,
where
we
copy
pasted
a
lot
of
the
code
and
and
change
just
those
little
things.
A
This
was
especially
with
the
Jenkins
agents
that
we
we
have
spent
up
because
they're
very
close
to
each
other
just
need
some
very
small
changes
for
each
environment
and
we
had
it
to
the
point
where
no
one
person
could
pick
it
up
and
run
it
and
my
co-workers
actually
in
the
audience.
I
had
done
a
lot
of
the
Jenkins
work.
He
couldn't
get
it
working
for
the
life
and
he'd
done
mostly
Cassandra
work.
A
I
couldn't
get
Cassandra
working
for
the
life
of
me,
even
though
we
could
do
it
just
fine
on
our
own,
at
our
laptops
and
so
to
solve
that
q.
Your
old
friend
configuration
management,
I,
don't
know
about
all
of
you,
but
for
me
at
least
my
initial
thought.
When
you
start
doing
the
whole
orchestration
of
containers
like
hey
I,
don't
have
to
spin
up
whole
clusters
anymore.
I,
don't
have
to
do
all
the
ansible
configuration
or
puppet
or
chef
and
apply
all
these
things,
because
I'm
not
spinning
up
a
whole
server.
A
It's
just
a
container
and
I
learned
that
what's
wrong.
In
our
case,
we
use
we
use
ansible,
and
we
use
that
because
it
has
the
ability
to
encrypt
secrets
and
keep
them
with
the
repo
and
also
to
template
things
really
well,
and
so
just
an
example
of
what
that
looks.
Like
is
right
here.
These
are.
This
is
like
what
the
PlayBook
file
looks
like.
We
have
specific
templates
that
we're
using
for
our
secrets
and
for
the
name
states
that
we're
creating,
and
it's
all
copied
into
a
specific
path
that
we
set
up
above.
A
So
this
is
just
a
snippet
of
inside
of
a
configuration
file
for
cooper
Nettie's,
so
we're
able
to
configure
our
limits
and
requests
and
also
the
Jenkins
type,
so
public
or
private,
and
whether
or
not
we're
using
a
persistent
disk
which
we
use
for
our
staging
and
prod
or
something
like
dev,
where
we
use
our
empty
directory
and
just
as
an
example,
I'll
actually
show
this.
So
this
makes
it
so
easy
to
deploy
jenkins
and
I'll
just
show
an
example.
A
So
in
this
case,
because
I'm
in
a
dev
environment,
I'm
actually
adding
a
namespace
suffix,
so
we
can
dynamically
change
what
namespace
for
putting
these
into
and
I'm
populating
the
files
for
Jenkins,
and
so
it
just
takes
a
little
bit.
Basically,
it's
going
through
and
running
all
that
different
code
skipping
some
of
the
stuff
that
we
don't
need
for
the
dev
environment
and
creating
the
different
deploy
scripts
and
configuration
files
for
cooper
Nettie's
as
it
goes
through
here,
and
you
can
see
it
going
through
now,
it's
done
now,
all
I
have
to
do
to
spin.
A
A
So
I'm
going
to
look
for
all
of
our
Jenkins
pods
and
you'll
actually
see
other
people
who
work
on
the
team.
You
can
see
all
the
ones
that
have
spun
up
with
my
name
on
it
and
it's
on
container,
creating
and
running
in
different
ready
statuses,
but
once
Jacon
spins
up
it's
all
up
in
about
a
minute
to
two
minutes,
every
single
time
makes
deployments
extremely
easy
now
this
is
what
came
from
having
configuration
management
I,
really
don't
care
what
what
people
use
like
if
you
want
to
use,
ansible
or
puppet
or
chef
or
whatever.
A
This
is
something
that
really
helped
us
kind
of
bring
that
chaos
under
control
and
have
something
repeatable.
It
also
helped
us
in
testing
which
I'll
show
a
little
bit
later,
so
the
next
thing
we
get
to
is
now
we
had
things
working
are
things
actually
working?
We
could
deploy
things
and
then,
an
hour
later,
sometimes
the
next
day
find
out
was
running
in
a
degraded
state
for
24
hours,
so
kuru
Nettie's
can
even
say
that
something
is
running.
A
But
how
do
we
know
that
things
are
actually
running
and
so
I'm
going
to
go
over
how
we
managed
to
do
this?
We
did
it
with
monitoring
tools.
Writing
this
probes
and
a
lot
of
testing.
So
first
thing
is:
when
monitoring
we
did.
Three
different
things
here
to
first
is
node
and
application
information
using
snap,
and
so
snap
is
basically
doing
see.
It's
that
one,
no
no
section
so
snap
like
I
explained
earlier,
allows
us
to
gather
data
pretty
much
remaining
source.
A
If
you
have
another
tool,
you're
using
there's
other
databases
that
snap
support,
but
in
our
case
we've
found
capacitor
be
very
helpful
to
give
us
those
alerts
when,
when
things
were
going
wrong
with
the
nodes,
we
also
used
elasticsearch
with
fluent
d
I'll,
have
a
link
with
that.
At
the
end,
that's
actually
an
add-on
to
the
kubernetes
cluster.
You
can
spin
it
up
with
and
we
use
a
handy
tool
that
we
found
it's
an
open
source
tool
by
I.
A
Think
he
yelped
called
the
last
alert
and
elastomer
allows
you
to
do
just
elastic,
search,
queries
and
alert
based
on
spike
of
CPU
or
blacklists.
If
they
see
a
certain
value,
that's
not
supposed
to
be
there
to
alert
for
you.
That's
also
very
handy
and
that's
actually
where
we
get
a
lot
of
our
learning
from.
We
know
if
something
goes
wrong
with
Jenkins,
because
we
get
a
bunch
of
severe
exceptions
in
the
logs
and
so
that
that
helps
us
monitor
things
through
the
logs
as
well.
A
Now,
the
last
thing
we
did
is
we
built
just
a
pretty
drop
dead,
simple
pod,
monitor
it's
a
Python
script
inside
of
a
container,
that's
running
and
it
uses
and
leverages
label
queries.
But
I'll
show
what's
going
on
here.
So
once
we
get
a
list
of
pods
from
the
API,
we
can
actually
use
this
a
little
block
of
code
where
we
loop
over
the
container
statuses
because
inside
of
a
pod
you,
your
pod
can
be
running,
but
the
containers
are
not
actually
running
a
running.
A
Pod
means
the
pod
has
been
scheduled
and
the
containers
are
created
and
that's
all
it
means
you
can
even
look
at
it
in
the
documentation.
There's
a
stern
warning
in
there
about
that,
and
so
what
you
can
do
with
this
is
you
can
actually
loop
through
each
container
center.
So
we'd
first
check
to
see
if
the
status
is
not
running
in
the
pot,
because
if
it's
not
running,
then
obviously
it's
not
ready.
A
But
then
we
check
each
container
inside
of
the
pod
to
see
if
that
pot
is
ready,
and
then
we
start
adding
data
into
basically
a
small
object
that
we
have.
That
shows
exactly
which
containers
are
broken
and
which
namespace
they're
in
and
a
couple
other
details,
and
so
I
actually
have
a
full
script
of
this,
which
I'll
show
a
link
to
at
the
end.
A
So
in
our
case
we
have
Jenkins
and
Cassandra
and
snap
and
a
couple
other
things,
and
so
it
goes
through
here
and
just
joins
all
the
deploy
types
into
a
string
into
a
label
query
now
to
make
that
a
little
bit
clearer
what
it
looks
like
if
I'm,
just
using
as
a
normal
command.
It
looks
like
this
so
right
here
you
can
actually
use
set
based
matches
with
label
queries,
and
this
is
very
helpful
for
monitoring
and
debugging.
A
This
is
actually
one
of
the
favorite
things
I've
done
in
terms
of
monitoring,
because
it's
so
simple,
there's
just
some
gotchas
that
we'll
talk
about
so
there's
liveness,
probes
and
liveness
probes
will
kill
your
pod
if
they
return
false.
Basically,
if
they're
show
that
it's
not
working,
it
will
kill
your
pot
and
restart
it.
That's
not
the
behavior
we
wanted,
especially
with
Jenkins,
because
Jenkins
could
be
not
ready,
but
still
like.
A
We
don't
want
to
kill
a
running
job,
that's
on
an
agent
or
something
else
that's
going
on,
and
so
the
readiness
probed
are
what
trigger
back
to
that
code.
I
was
showing
earlier
with
the
pods
at
being.
Are
the
containers
in
a
pod
being
in
a
ready
state?
In
our
case,
this
is
just
a
simple
cat
of
a
file
that
we
put
on
the
file
system
once
it's
once.
A
Jenkins
is
up
and
ready,
and
you
can
also
do
an
HTTP
GET
instead
of
an
exec
that
documentation
is
available,
and
then
you
have
an
initial
delay
seconds.
In
our
case,
we
wait
about
15
seconds
for
Jenkins
to
have
some
time
to
spin
up,
because
we've
never
seen
it
spin
up
really
faster
than
that,
and
so
once
that's
running
it
just
keeps
running
out
and
he'll.
Tell
you
if
it's
failed
or
not
it'll
actually
show
up
in
the
coop
control
described.
A
Pods
it'll
show
it
as
an
event
that
occurs
that
it
failed
that
check
and
how
many
times
now
just
a
note.
If
a
service
is
not
I
mean
if
a
container
is
not
in
a
ready
state,
a
service
cannot
forward
traffic
to
it.
That's
actually
good,
because
we
don't
want
that
to
happen
in
with
us
and
Jenkins
like
we
don't
want
the
the
jobs
going
to
the
Jenkins
server
and
the
Jenkins
server
is
not
ready
to
receive
them,
and
so
this
works
great.
A
However,
when
you
get
to
things
that
require
state,
whether
you're
doing
in
a
pet
sitter
or
deployment,
in
our
case
it
was
Cassandra
and
what
happened
with
ours
is
what
I
did
this
in
Jenkins
I
was
like
wow
that
was
easy
and
I
went
over
to
Cassandra
and
created
a
Cassandra
health.
Sorry,
a
readiness
probe
and
then
all
of
a
sudden,
Cassandra
just
would
die
and
I
was
like
okay.
Why
is
it
dying
and
I
found
out?
A
It
was
because
the
the
service,
it
wasn't
forwarding
the
traffic
to
for
the
inner
cluster
communication
for
that
to
spin
up
and
set
up
the
cluster.
So
after
a
little
bit,
I
found
this
right
there.
This
is
your
friend
in
case
you're,
using
readiness
probes.
It
is
an
annotation
that
you
can
add
in
that
says,
you
can
tolerate
unready
endpoint.
So
you
can
see
these
are
the
intra
node
community
of
this
week.
A
It
should
be
internet,
intrepid
internal
communication,
ports
right
here
that
allow
it
to
set
up
this
cluster
and
we've
only
exposed
those
without
the
service
being
ready.
But
then,
once
the
services
is
up,
then
the
other
one
that
allows
CQ,
CQ
LSH
to
connect
allows
the
traffic
to
go
through
to
Cassandra.
A
So
the
other
thing
we
did
was
testing
now.
I
know
that
sounds
stupid.
Believe
it
or
not,
testing
is
important.
Even
inside
of
the
inside
of
Coober
Nettie's.
There
are
a
couple
times
we
brought
things
down
when
we
launched
them,
because
we
didn't
have
a
gating
test
for
our
different
kuber
Nettie's
files
in
the
different
images
we've
created,
and
so
we,
after
we
learned
that
we
decided
we
ke,
need
to
get
these
gating
tests
and
and
get
them
working,
and
so
these
actually
do
a
full
deploy
of
Jenkins
or
Cassandra
or
snap
on
kuber
Nettie's.
A
So
it's
actually
just
running
it's
coo
Brunetti,
spinning
up
another.
It's
jenkins
running
on
Corinne,
a
spinning
up
Jenkins,
and
so
it
works
really
well
for
those
unit
tests
now.
Koo
béarnaise
makes
this
really
easy
because
of
that
that
coop
control
the
ability
to
access
the
API
and
do
other
things
inside
of
a
pod.
Now,
because
we
automated
things
that
actually
made
it
really
easy
I'll
show
you
what
that
job
looks
like
I
notice
is
actually
me
see.
We
can
zoom
in
so
right
here.
A
So
this
is
a
test
for
an
influx
DB,
snap
plugin
and
it
creates
a
service
and
then
waits
for
it
to
come
up
and
checks
to
make
sure
all
the
pods
are
ready
and
once
all
the
pods
are
ready,
it
actually
runs
that
command
inside
of
the
specified
container
and
runs
these.
In
this
case,
it's
running
the
go
unit
tests
for
this
to
make
sure
everything's
running,
and
then
it
succeeds
and
cleans
up
that
namespace
that
was
created.
A
That's
all
done,
because
we
had
that
coop
control
installed
and
that
agent
of
Jenkin
inside
Jenkins
can
run
and
spin
up
the
the
necessary
containers
for
things
to
work.
So
just
in
review
I
about
what
we
went
over.
So
the
first
thing
is
just
design
your
containers,
while
it
can
also
deal
with
like
designing
your
application
and
well,
but
just
make
sure
your
containers
or
design
well
and
can
work
inside
of
kuru
Nettie's
know
how
to
use
the
API.
It
will
be
your
friend
because
you
can
either
use
it
inside
I.
A
We
actually
found
out
talking
to
the
sed
people
that
their
operator
stuff
is
using
the
API
to
make
sure
they
set
up
everything.
You
will
have
to
do
this
at
some
point,
as
you
create
more
applications
from
that
you're
going
to
be
running
in
coover
Nettie's,
don't
forget
the
role
of
config
management
make
sure
you
are
monitoring
and
testing
all
the
things
know
the
gotchas
some
of
these
different
examples
we
gave
and
then
just
one
last
problem
to
go
over
the
need
to
document.
A
There's
actually
been
some
mention
about
the
state
of
documentation
inside
of
Coober
Nettie's,
now,
actually,
I
find
the
documentation,
useful
and
but
there's
still
some
big
gaps,
so
I'm,
hoping
that
you
guys
at
least
learned
one
thing
from
this
that
can
help
you
with
some
of
your
deployments,
but
I
just
wanted
to
share
an
example
of
why
this
documentation
is
important.
So
that
example,
I
gave
of
the
tolerate
unready
end
points.
So
that
came
after
a
lot
of
a
lot
of
research.
A
So
after
I
discovered
the
Cassandra
gone
down,
first
I
had
to
figure
out
it
wasn't
a
problem
with
Cassandra
or
something
else,
I
upgraded
that
took
about
like
an
hour,
so
no
Mike,
okay,
so
it
can't
connect
to
the
service.
But
then
for
the
next
half
of
half
of
the
day,
I
had
to
search
around
I
was
googling
and
you
didn't
know
what
term
to
search
for
and
I'm.
A
Looking
for
all
these
everything's
and
finally
I
found
one
issue:
the
hook
to
another
issue,
the
hook
to
a
PR
that
had
that
code
in
it
there's
no
documentation
on
it.
And
so
we
like
that's
that's
what
we're
trying
to
avoid
here
is
we're
all
trying
to
get
these
things
spinning
up
in
production
and
it's
very
helpful
if
we
share
what
we've
done
and
document
what
we've
done.
So
my
invitation
is
that
we
can
make
this
running
a
production
applications
and
Kure
is
much
easier
by
documenting
and
to
start
off.
A
I
created
a
blog
post
with
my
team
that
has
all
like
these
code
snippets
and
more
about
what
we
did.
I
mean
chock-full
of
code
snippets
and
you
can
go
visit
that,
along
with
some
other
posts
like
we
have
some
things
about
coop
snap,
which
I'll
show
here
and
we'll
get
two
questions,
but
there's
the
other
links
that
I
was
talking
about.
A
If
you
want
to
see
more
about
how
we
ran
snap
in
Coober
Nettie's,
you
can
go
to
cubed
snap
dash
telemetry
do
and
then
that's
the
code
inside
of
the
add-ons
for
the
coup,
bearnaise
repo
for
the
elastic
search
with
fluid
d.
So
with
that,
are
there
any
questions
and
if
you
could
come
up
and
use
the
mic?
Oh.
A
A
You
can
download
these
and
actually
get
all
the
different
code,
and,
as
I
said
this,
this
blog
has
the
detailed
examples
of
everything
I
showed
in
here
plus
more
so
you
can
go
on
there
and
see
and
just
start
documenting
and
sharing
these
things
around,
because
we
could
use
it
and
I'm
hoping
you
guys
have
been
able
to
use
this
and
by
helping
each
other
out,
will
kind
of
improve
that
documentation
that
we
have
and
be
able
to
make
it
easy
for
people
to
run
these
things
in
production
so
well.
Thank
you.