►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
Managing Kubernetes: What You Need to Know About Day 2 - Craig Tracey, Heptio (Intermediate Skill Level)
OK, so you have successfully deployed Kubernetes, but now what? This presentation will provide insights on how to best leverage your "Day 2" software deployment and operations with Kubernetes. With dozens of real-world customer use cases to dissect from, we will cover the commonalities amongst deployments, the unforeseen failures, the most-often misunderstood features, and a redux of top-ranked best-practices that will result in a successful production roll-out. From user management to labelling policies and all the way through to upgrades, we will show you how our customers "went live." Beyond just the technical, this talk will also cover the organizational constructs that have yielded successful outcomes.
About Craig
For the last 20 years Craig has helped build the infrastructure that powers the Internet. In this time he has had the opportunity to develop everything from kernel device drivers, to massive-scale cloud storage services, and even a few distributed compute platforms. Now as a Software Engineer turned Solutions Engineer at Heptio, he helps organizations accelerate their adoption of Kubernetes by teaching the principles of cloud native architectures through code. Craig is based in Boston, Massachusetts; spending the winters playing hockey and the summers getting lost in Europe.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
So
today
we're
gonna
talk
about
managing
kubernetes,
specifically
around
day
two
operations,
we're
going
to
take
a
slightly
different
take
than
what
you
might
be
expecting,
but
hopefully
this
ends
up
being
a
a
good
resource
for
you,
so
my
name
is
Craig.
Tracy
I
should
probably
turn
this
thing
on.
My
name
is
Craig.
Tracy
I
have
been
in
infrastructure
for
a
little
over
twenty
years.
Now
you
know
time.
I've
worked
at
a
whole
series
of
different
places.
A
Some
of
these
may
be
household
household
names
depending
on
your
household,
but
in
that
time
I've
done
a
whole
bunch
of
stuff.
I
first
started
off
as
a
kernel
developer
of
all
things,
working
on
very
specific
and
proprietary
network
drivers
even
worked
on
an
ASIC
in
that
time
and
basically
decided
that
during
that
time,
the
difficulty
that
I
was
having
with
that
my
career
choices
at
that
point
was
that
working
on
something
that's
very
bespoke
like
that,
are
very
specific,
like
that,
you
don't
always
get
to
see
what
a
customer
sees.
A
You
don't
get
to
see
like
your
work
in
the
real
world,
so
I
eventually
worked
my
way
up.
The
stack
eventually
working
on
cloud
infrastructure,
both
from
a
consumer
side,
as
well
as
as
a
vendor
from
the
vendor
side
and
found
that
I
was
actually
working
a
lot
closer
to
customers.
That's
something
that
I
really
loved,
because
this
is
where
the
rubber
meets
the
road.
This
is
where
you
actually
get
to
see
all
the
hard
work
that
we're
putting
in
with
things
like
kubernetes.
A
Is
it
just
a
science
project
or
it
can
we
make
it
real
change
with
it,
and
that's
why
I
joined
tempio,
so
I
turned
software
developer
into
field
field,
engineer
at
Papio
and
I
work
with
customers
every
single
day.
Those
customers
range
from
small
startups,
all
the
way
through
fortune,
100
banks
and
a
lot
of
the
patterns
that
I've
seen
along
the
way,
are
very
similar
across
all
these
organizations
and
actually
also
mentioned
these.
These
customers,
I
work
with
are
spread
across
many
different
verticals,
so
I
think
it's
a
good
representation
of
what
we
see.
A
Actually,
you
know
how
Koreans
is
being
adopted
in
a
while
and
by
the
way,
I'll
it
right
now.
So
this
is
something
I
really
enjoy
if
you're
also
enjoy.
If
you
also
enjoy
working
with
close
with
customers,
we
are
hiring.
So
please
see
me
after
and
then
in
my
free
time,
I'm
doing
something
else.
I'm
writing
a
book
you'll
be
surprised
to
learn
that
it's
called
managing
kubernetes.
A
It's
not
quite
done
yet.
It
should
be
out
sometime
this
summer,
but
I'm
working
with
Brendan
burns
on
this,
and
it's
a
lot
of
the
topics
that
we'll
talk
about
here,
or
at
least
some
of
the
topics
that
we'll
talk
about
today
are
in
that
book.
But
please
look
for
that
sometime
this
summer,
all
right,
let's
start
with
some
questions
who,
in
the
room
would
consider
themselves
to
be
a
user
of
Koreans
and
what
I
mean
by
user
is
a
user?
A
A
Okay,
it's
hard
to
tell
because
you
guys
didn't,
raise
your
hands
too
high,
but
and
say
that's,
maybe
about
50%
how
about
operators,
operators,
the
Koreans
would
be
cluster
administrators,
okay,
I'd
say
that's,
probably
also
about
50%
and
all
right.
I
want
you
to
be
honest
on
this
one
as
well,
so
who
has
been
using
Koreans
for
more
than
two
years.
A
Okay,
do
you
guys
want
to
give
this
done
and
how
about
one
year
to
two
years,
okay
and
less
than
one
year?
Okay,
this
is
kind
of
the.
This
is
kind
of
what
I
expected
so
I
think
right
where
we
are
right
now
with
kubernetes.
Is
we
see
a
lot
of
people?
We
actually
work
with
customers
who
are
pretty
well
adopted
at
this
point,
but
also
work
with
customers
who
are
very
much
in
the
phases
of
their
deployment,
and
this
talk
is
really
geared
towards
that.
A
It's
more
of
a
beginner
to
intermediate
talk
and
the
thing
I
so
part
of
this
when
I
think
about
giving
this
talk,
I
had
all
these
ideas
in
my
head
and
then
I
realized.
It's
only
35
minutes.
That's
why
we
started
a
little
bit
early
and
that's.
Why
also
that's
also
why
I'll
be
talking
very
quickly,
but
a
lot
of
the
day
to
operations
talks
that
you
encounter
in
the
wild
they're,
very
tactical.
They
talk
about
hey.
A
If
you
want
to
monitor
your
platform,
this
is
the
tool
that
you
use,
and
this
is
how
you
use
that
tool
to
do
it.
I
want
to
take
a
slightly
different
approach,
because
those
are
very
reactive
approaches
right.
This
is
something
happened
and
I
need
to
know
how
to
fix
that
particular
thing,
and
this,
what
we'll
talk
about
is
the
more
proactive
ways,
the
ways
that
you
should
think
about
deploying
kubernetes
and
the
ways
that
your
the
way
that
you
think
about
cadres
is
going
to
affect
day.
Two.
A
So,
whenever
I
first
engage
with
customers,
it's
typically
on
the
phone
before
I
get
on
site
and
the
one
thing
I
always
say
them
is
that
if
you're
adopting
kubernetes,
it
is
is
just
as
much
cultural
as
it
is
technical
and
a
lot
of
people
don't
seem
to
recognize
that
or
maybe
discount
that
to
some
degree
when
just
like
with
when
we
did
when
we
moved
from
like
into
the
cloud
era,
you
know
cloud
deploying
infrastructure
on
cloud
infrastructure,
I
start
deploying
applications
on
cloud
infrastructure.
There
was
a
mindset.
A
You
have
to
have
a
different
mindset
about
how
you
deploy
your
application.
Application
has
to
be
tolerant
of
failure.
We
were
placing
more
of
the
burden
on
being
having
that
awareness
or
that
graceful
restart
into
the
application
versus
waiting
for
the
hardware
or
the
infrastructure.
To
save
us.
The
same
is
true
for
kubernetes,
and
even
though
it's
still
adopting
some
of
those
same
cloud
properties,
there
are
different
ways
that
we
have
to
think
about
deploying
micro
services.
A
The
first
thing,
I
always
tell
customers
when
I
work
with
them,
is
to
think
simplicity.
You
think
about
simplicity,
and
you
almost
invariably
when
I
work
with
a
very
new
customer.
The
first
two
questions
I
get
are:
how
fast
will
it
run
and
how
big
what
scale-
and
these
are
precisely
the
wrong
types
of
questions
that
you
should
be
asking
these.
Those
questions
imply
that
you
want
to
at
some
point
just
dry
this
thing
into
the
ground
or
to
make
it
perform.
Perhaps
beyond
its
means.
A
Kubernetes
itself
is
very
simple
like
if
you
think
about
the
frameworks
of
the
past,
we
have
frameworks
that
are
massively
complex.
They
were
hard
to
operate
and
I.
Think
those
reasons
alone,
or
those
reasons
to
a
large
part
played
played
into
the
fact
that
we
didn't
adopt
those
very
heavily
I
guess
kibriya,
the
architecture
of
quarries
is
very
simplistic.
You
should
think
about
the
simplicity
of
kubernetes
and
how
you
can
adopt
those
same
patterns
for
your
infrastructure.
Don't
try
to
do
everything
at
once.
A
You
know
typically
you'll
see,
customers
come
in,
our
customers
come
to
us
and
they
say:
okay,
we're
deploying
kubernetes,
and
we
have
this
monitoring
platform
we're
using
Nagios
we're
using
whatever
tools
we
have
in
place.
And
you
know
we
decided,
like
you
know:
everything's
gonna
be
new,
we're
just
gonna
go
prometheus
and
don't
get
me
wrong.
Using
Prometheus
is
is
great,
but
at
the
same
and
it
will
get
you
really
far
along
your
your
path.
A
A
This
is
late
last
year
and
I
walked,
I
walked
on
site
and
I'm
staying
there,
I
go
to
engage
with
the
the
you
know,
engagement
lead,
and
he
comes
up
to
me
and
he
says
you
know
we
can't
do
anything
that
I
would
we
have
a
p0
outage
and
like
we
need
to
work
on
this
and,
like
we'll
see
you
like
later,
even
maybe
this
afternoon
tomorrow
and
I
said
well,
ok,
but
like
maybe
how
can
I
help
in
this
scenario?
I
didn't
I
didn't
know
if
it
had
do
anything
with
kubernetes
or
what
not.
A
It
turns
out
that
it's
a
Java
shop
and
their
Java
stack
traces,
their
multi-line,
Java,
Java
stack.
Traces
were
actually
showing
up
as
single
lines
and
Splunk,
and
this
was
all
coming
from
kubernetes
and
I'm
sitting
there
scratching
my
head
and
like
this
doesn't
really
sound
like
a
p0
outage,
but
the
whole
team
was
working
out.
They
were
frantic
about
it
and
the
reason
why
they
were
frantic
about
it
was
precisely.
This
is
the
optics.
They
were
their
internal
to
that
organization.
A
They
were
trying
to
push
kubernetes
and
if,
if
their
users
didn't
have
the
same
tools,
didn't
have
the
same
debug
or
or
capabilities
that
they
had
with
the
existing.
What
I'll
call
legacy
ec2
deployment
pattern,
then
it
just
wasn't
going
to
work
so
keep
things
simple
and
focus
on
the
user
experience.
A
Unless
you
talk
about
day
one
and
in
fact,
I
think,
there's
probably
no
distinction
between
day
one
day,
two
and
day,
and
it's
just
really
continuum
how
you
think
about
deploying
your
systems
and
at
the
same
time
people
say
that
you
know
there's
dozens
of
installers
out
there
and
people
say
that
you
know
installation
is
a
solved
problem.
Kubernetes
I
would
say.
That's
not
quite
the
case.
I
think
I,
think
that
shows
that
this
is
not
a
solved
problem
and
with
regard
to
installation
how
you
install
effects
day.
A
So
in
almost
invariably
when
I
work
with
customers,
they
they
get
to
install
point
or
they've
already
installed
using
some
tool,
and
it's
like
mission
accomplished
we're
ready
to
go,
and
that's
not
the
case
at
all.
The
decisions
that
you
make
during
installation
time
will
affect
date
two,
and
we
specifically
see
these
things
in
is
in
areas
like
Hardware
tuning
and
people
say
like
well.
A
What
hardware
tuning
do
I
really
have
to
do
if
you're
using
large,
large
instances,
whether
they
be
on
or
in
the
cloud
there
are
cubelet
configuration
parameters
that
you'll
want
to
adjust,
for
instance,
max
pods,
or
this
the
amount
of
reserved
space,
so
the
reserved
space
is
memory
and
disk,
etc.
That
you're
gonna
reserve
for
things
like
the
cubelet
processes,
as
well
as
system
system
demons
like
SSH
and
you
def,
etcetera
and
unfortunately,
today
we
don't
have
a
capability
to
really
change
those
things
in
production.
A
This
is
coming,
so
the
cubelet
will
soon
support
dynamic
configuration,
but
we
don't
have
that
just
quite
yet,
and
the
other
thing
to
think
about
with
how
you
deploy
kubernetes
from
the
platform
side
is
the
component
life
cycles.
If
you
look
at
some
tools
out
there
with
with
that
perform
installations,
like
cops,
does
this
today
I'm
not
picking
on
cops?
The
cops
is
actually
a
really
great
tool,
but
cops
will
deploy
at
CD
to
the
same
nodes
as
the
control
plane
well
at
CD
and
Cabrini's
have
very
different
component
life
cycles.
A
If
you
think
about
the
migration
from
Etsy
d2
to
d3,
it's
not
the
same
as
the
migration
from
let's
say
a
net
CD.
Sorry,
a
kubernetes,
1.92
1.10.
We
can
think
about
the
coronaries
control
plane
and
even
the
workers
as
being
mostly
ephemeral.
We
should
be
able
to
kill
them,
there's
no
real
assets
on
those
machines.
There
may
be
some
assets
like
PKI
assets
on
the
masters,
but
for
the
most
part
they
are
basically
ephemeral
instances
and
we
should
start
thinking
about
immutable
infrastructure.
In
that
way.
On
the
flip
side,
we
have
Etsy
D.
A
This
is
the
thing
you
care
about.
This
is
the
pet
in
your
environment.
This
is
the
thing
that
you
should
care
and
feed
for.
So
we
had
happy.
I
would
recommend
that
you
keep
Etsy
D
external
for
the
cluster,
still
deploy
in
the
same
manner,
I
like
at
least
three
to
five
@
çd
nodes,
but
that
they
remain
external
to
the
actual
kubernetes
cluster
and
a
lot
as
long
as
I
talk
about
platform
as
well.
A
You
should
start
to
think
about
those
services
that
you'll
you'll
Balt
on
the
side
of
kubernetes
that
will
start
to
help
you
manage
kubernetes
towards
de
so
I'd
have
to
deal.
We
have
two
projects
in
particular
sonobuoy,
sonobuoys
and
and
conformance
testing
for
clusters.
So,
as
you
deploy,
you
might
want
to
run
sonobuoy
to
make
sure
that
the
things
you
deployed
make
sense
and
arc
which
is
backup
in
recovery
for
kubernetes.
A
The
other
question
that
we
invariably
get
asked-
and
this
also
affect
your
day
to
operations,
is
availability.
There's
certainly
use
cases
in
the
wild
where
high
availability
does
not
matter
and
I
want
to
caution
a
lot
of
people
in
the
room
or
people
watching
this
that
be
very
precise
in
what
you
require
from
high
availability.
Oftentimes,
it's
just
a
checkbox.
It's
like
okay,
we're
highly
available,
and
therefore
you
know
we
must
be
good.
A
We
had
a
customer
late
last
year
as
well,
they
had
very
few
resources
and
they
were
using
their
cluster
for
a
CI
CD
platform
and
they
started
talking
about.
They
need
AJ
and
all
these
sorts
of
things,
and
we
said
well,
it's
a
CI
CD
platform
like
if
this
thing
goes
down
the
middle
of
night.
Is
it
really
gonna
be
the
end
of
the
world
right?
Your
cash
registers
not
closed.
A
If
that
happens
so
think
about
you
think
about
it
holistically
and
then
wouldn't
you
know
when
we
we
actually
have
this
internal
debate
the
whole
time
inside
inside
hefty.
It
was
like,
which
is
the
right
pattern
to
use
when
you
go
into
a
cloud
environment.
Is
it
do
you
spray
kubernetes
across
multiple
availability
zones
within
the
same
region,
or
do
you
set
up
multiple
clusters
within
the
same
region
in
single
availability
zones,
so
that
would
be
single
cluster
multi
and
cluster
prasing
I
think
both
are
viable
options,
but
it
depends
what
you're
optimizing
for
right.
A
So,
if
you're
optimizing
for
always
being
up-and-coming,
want
to
leave
things
alone
and
hope
things
that
work
well,
the
one
you
know,
if
you
think
about
most
cloud
providers,
they
have
these
guarantees
around
that
one
easy
you
will
only
one
AZ
will
fail
at
a
time,
but
remember
when
you,
like
things,
will
slow
down.
Things
will
not
be
the
same.
So
keep
that
in
mind.
A
A
Okay,
so
we
also
talked
this.
The
second
box
on
the
previous
slide
was
or
the
three
main
areas
that
we
talked
want
to
talk
about.
Are
the
API
server
API
servers
where
you
can
wield
a
ton
of
control
and
when
I
this
is
also
happens
when
I
walk
into
a
customer
when
we
walk
into
a
customer
one
of
the
first
questions
they
say
is:
how
do
we
set
up
our
back?
A
How
do
we,
you
know,
make
sure
that
we're
doing
this
right
when
you
use
this,
the
installers
that
are
out
there
today,
you
have
things
like
you
know,
cube
ATM.
Does
this
cops
those
this?
They
all
do
it?
They
give
you
an
x.509
client
certificate
and
that's
the
thing
that
typically
most
customers
that
I
engage
with
end
up
using
for
everything
right.
It's
the
administrator
and
Bob
over
here
is
using
the
same
certificate.
There's
no
real
authentication
strategy
when
they're
talking
when
they're
setting
up
their
cluster,
so
authentication
we're
going
to
be
asking.
A
Has
this
users
proven
their
identity,
an
access,
control
or
authorization
will
ask
like?
Is
the
user
allowed
to
perform
this
task
and
in
admission
control
we're
going
to
ask?
Does
this
request?
Look
like
something
that
I
should
accept
or
if
it
doesn't
look
like
something
I
should
accept.
Can
I
mutate
it
in
such
a
fashion
that
I
can
make?
A
It
looks
like
look
like
something
I,
so
I
just
started
talking
about
authentication
in
a
second
ago,
but
one
thing
that
users
also
don't
understand
in
kubernetes
is
that
there
are
there's
no
resource
or
you
call
the
user
right
they
and
they
find
it's
very
weird
that
you
cannot
run
cube.
Ctl
get
users,
there's
no
such
thing.
What
there
is,
though,
inside
kubernetes
is
this
user
info
resource
and
what
kubernetes
does
this
is
one
thing
I
really
like
about
kubernetes.
Is
that
kubernetes,
basically
delegates
handling
of
users
to
some
other
process?
A
There
are
a
bazillion
ways
to
handle
this
within
kubernetes
from
authenticating
proxies
OAD
see
the
static
tokens,
you
name
it,
the
one
that
we
do
recommend
now
is
OID
C
and
what
oh
I
D
C
gives.
You
is
the
ability
to
basically
manage
users
external
to
the
cluster.
It
usually
typically
provides
you
a
way
to
manage
them
in
a
consistent
manner
across
your
organization,
almost
invariably
any
organization.
A
They're
gonna
have
some
sort
of
user
management
scenario
or
system,
whether
it
be
LDAP
ad,
you
name
it,
unfortunately,
AD
more
I,
guess,
yeah
I
think
it's
actually,
fortunately,
and
I
don't
want
Koreans
to
be
managing
users.
Cabrini's
will
not
talk
directly
to
ad
or
LDAP.
They
are
there's
a
project
out.
A
There
called
Dex
by
core
OS,
which
acts
as
a
user
authentication
broker
for
these
back-end
services,
and
it
provides
you
an
OID,
C
front-end
having
an
OID
front-end
makes
it
really
easy
for
you
to
onboard
and
off-board
users,
and
then
we
have
a
project
called
gangly.
This
is
a
project
that
I
developed
with
a
customer
late
last
year,
and
the
idea
behind
gangway
is
very
simple:
it
provides
a
a
web
UI
for
someone
to
easily
log
them
or
easily
onboard
themselves
into
your
kubernetes
cluster.
So
you
point
them
at
the
URL.
A
They
get
all
the
commands
of
the
need
to
basically
set
up
their
cube,
kubernetes,
environment
and
away
they
go.
It
leaves
you
out
of
the
picture
of
having
to
manage
them
on
a
one-off
basis
and
always
remember
the
dashboard
people
deploy
the
dashboard.
Remember
you
don't
want
to
end
up
like
a
very
famous
automotive
company
that
was
compromised
by
leaving
their
dashboard
wide
open
with
the
dashboard.
You
can
basically
do
anything,
make
sure
that
you
are
locking
down
your
dashboard
use
an
authenticating
proxy
or
something.
A
The
other
thing
that
you
see
a
lot
of
clusters
is
that
people
do
not
implement
our
back.
You
should
implement
our
back
and
the
way
that
our
back
works
is
very
simple.
We're
basically
talking
about
verbs
against
resources.
So
if
we
look
in
the
top
right
there,
when
we
a
lot
of
people,
don't
understand
how
our
back
works-
and
this
is
basically
how
it
works.
It's
just
working
on
rest,
where
we
have
resources
which
in
this
case
would
be
pods
and
the
verbs
that
a
user
would
be
able
to
affect
on
those
particular
resources.
A
A
This
is
the
output
of
a
jdl
ET
token
that
we've
gotten
from
auth0
in
this
case,
which
is
by
the
way
as
a
fantastic
service,
it
gets
mapped
to
a
user
info
struct,
and
this
user
info
struct
is
then
used
to
basically
apply
this
role,
which
is
a
policy
and
bind
it
to
a
user.
In
this
case
we're
binding
it
to
Jane
so
think
about
it.
That
way,
it's
very
simple,
don't
overthink
it
and
kubernetes
makes
us
very
very
like
someone
obfuscated
and
hard
to
understand.
A
A
Actions
that
a
user
can
take
on
a
resource
but
also
understand
that
some
of
these
are
at
some
of
these
are
implicit.
So
if
you
give
user
the
ability
to
basically
create
a
pod,
they
would
also
have
access
to
secrets.
So
don't
think
that
you
are
doing
anything
funny
there
by
just
giving
them
pods
and
not
secrets.
A
The
third
part
of
the
API
flow
is
admission
control,
and
this
is
where
we
can
also
start
to
affect
a
lot
of
business
logic.
This
is
where
you
can
start
to
make
it
make
decisions
around
what
you
demand
from
your
users
in
your
environment.
So,
first
and
foremost,
if
you
intend
to
do
any
sort
of
bin
packing,
like
heavier
utilization
you're
on
your
clusters,
you're
going
to
need
to
do
resource
quotas
and
if
you're,
not
even
if
you're,
not
doing
resource
quarters,
there
are
some
usage
patterns
like
I'm,
using
Java,
8
or
less
Java.
A
8
does
not
respect
resource
limits,
so
you
should
have
mechanisms
place
that
are
basically
ensuring
that
you're,
a
single
pod
cannot
basically
take
control
of
a
worker
node.
The
other
thing
that
we
get
through
a
Mission
Control
is
pod
security
policies.
You
should
be
also
implementing
these
and
make
sure
that
users
not
doing
things
that
you
don't
consider
to
be
within
spec.
A
It
has
brought
their
there
a
broad
set
of
rules
that
you
can
apply
for
pod
security
policies,
but
if
they,
if
those
don't
exist,
we
also
have
dynamic
admission
control,
and
this
is
where
you
can
actually
write
things.
There's
very
specific
to
your
organization,
for
instance,
do
I
require
a
specific
set
of
labels
on
a
pod.
If
do
I
want
to
apply
labels
automatically
to
a
pod?
You
can
write
this
in
any
language
that
you
want,
and
basically
API
server
will
call
this
and
mutate
an
incoming
request
on
the
fly.
A
A
And
the
third
area
that
I'd
like
to
instruct
users
to
think
about
is
the
the
use
of
custom
controllers
and
I
can
give
another
example
here.
But
let's
talk
about
why
this
is
important.
So
the
way
the
Cabrini's
works
is
very
simple.
We
declare
state
and
kubernetes
will
reconcile
until
you
achieve
that
state,
and
this
is
how
you
should
be
thinking
about
your
deployments
to
kubernetes
I
went
to
a
customer
again
late
last
year
and
they
had
this
there.
A
A
Basically
was
a
migration
path,
you
know,
and
they
had
this,
what
they
had
done
is
they
had
basically
written
a
container
such
that
when
the
container
came
up,
it
was
starting
to
do
this
service
discovery,
but
the
problem
with
that
is
then,
when
you
launch
these
containers
horizontally
right,
all
these
containers
are
trying
to
basically
update
service
discovery
and
that
doesn't
really
make
a
lot
of
sense.
It
also
doesn't
handle
the
case
where
you're
turning
down
the
pod,
and
you
want
to
basically
remove
from
you,
remove
your
entries
from
service
discovery.
So
what
I?
A
What
we
talked
about
is
you
should
be
thinking
about
as
if
you
can
deploy
your
code
as
a
controller,
and
the
controller
is
just
basically
taking
the
the
state
that
it
sees
within
the
koreans
cluster.
These
can
be
deployed
just
as
pods
just
like
anything
else.
They
just
have
access
to
the
to
the
koreans
api
and
can
affect
change
based
upon
events
that
happen
across
the
koreans
api,
and
we
took
their
use
case
and
we
wrote
a
custom
controller.
A
Custom
controller
was
roughly
about
20
lines
of
code
and
what
it
does
is
it
just
sits
there
and
listens
for
a
new
pod,
a
new
service
to
come
online
and
register.
Is
it
with
rica,
eureka,
and
when
that
thing
goes
away,
it
removes
it
from
eureka.
Very
simple,
so
again,
back
to
simplicity
back
to
user
experience,
you
should
be
thinking
simply
and
controllers.
Let
you
do
that.
A
Controllers
can
also
be
written
in
language.
There
are
put
there
by
programming
bindings
for
all
kinds
of
languages,
whether
it
be
Ruby,
Java,
Python
go.
What
I
will
say,
though,
is
that
these
bindings
are
built
from
swagger
and
they
are
are
primarily
auto-generated
code.
So
there
are
some
limitations.
There
just
understand
that
there
are
limitations
there.
If
you
have
a
preference
per
go
or
are
you
inclined
to
use,
go
or
think
about
using
go?
A
That's
definitely
the
most
supported
path,
and
the
other
thing
about
to
understand
understand
about
custom
controllers
is
that
there
are
different
use
usage
patterns,
so
the
most
simplistic
usage
usage
pattern
would
be
through
CR,
DS
and
custom
controllers,
in
this
case
you're
using
the
something
native
to
kubernetes
called
custom
resource
definitions,
and
you
are
basically
affecting
change
on
some
object
that
you
have
defined
a
limitation.
There,
though,
is
that
it
uses
open,
API
spec
as
for
validation.
So
if
you
have
complex
validations,
you
probably
won't
be
able
to
do
them.
A
You'll
be
able
to
do
simple
ones
like
string
an
integer,
but
something
like
this.
If
this
field
in
this
field,
you
can't
do
that
sort
of
thing,
and
the
other
thing
is
that,
with
custom
resource
custom
resources
you're
actually
affecting
those
objects
against
your
local
@
çd
source.
So
if
you
expect
a
high
degree
of
churn,
you
should
probably
think
about
using
something
else,
and
fortunately,
Cabrini's
provides
a
different
mechanism
which
is
called
aggregate
api.
The
difference
is
that
it's
a
little
bit
more
complex
to
set
up
with
an
aggregate
api.
A
You
still
define
your
own
objects,
but
what
ends
up
happening
is
that
in
the
API
flow
it
comes
through
the
API
server
here
and
then
when
it,
when
API
server
sees
hey.
This
is
an
object.
I
don't
know
about
it,
actually
proxies
it
down
to
your
custom,
API
server
that
knows
about
it,
but
you'll
notice
that
this
custom,
API
server,
has
its
own
sed.
So
this
is
a
complication
during
the
installation
that
you'll
need
to
think
about
and
the
so.
A
This
is
not
very
very
hard,
but
it
is
something
more
complex
than
the
standard
set
up.
The
other
thing
you'll
want
to
think
about
here
is
that
these
should
all
be
encrypted
through
TLS
and
that's
again,
another
deployment
challenge,
I'll
call
it,
and
you
should
use
this
use
case.
If
you
have
a
lot
of
churn
or
you
have
a
very
unique
use
case,
I
guess
what
does
the
future
look
like?
So
I
think
you
know
there
are
a
lot
of
limitations
now
and
how
we
can
manage
clusters
at
scale.
A
We
have
tools
out
there,
but
the
tools
could
use
a
little
bit
of
I
guess
additional
help.
So
one
thing
that's
I,
think
very
exciting
that
will
help
us
through
both
day
1
and
day
2.
If
we
want
to
call
it,
those
things
is
the
cluster
API.
This
is
a
new
initiative,
that's
happening
in
open
in
the
community,
and
the
idea
is
that
we
can
start
to
manage
kubernetes
clusters
with
kubernetes
itself.
So
in
the
cluster
API,
there
are
two
primary
routes:
primary
resources.
A
They
are
the
cluster
and
the
machine,
and
what
we
do
with
that
is
we
basically
define
a
cluster
just
like
we
would
define
a
pod
or
anything
like
that
through
llamó
interface.
Push
that
to
the
API
server,
and
then
we
define
a
couple
machines
that
are
part
of
that
cluster,
whether
they're
lit
whether
they
are
work
or
knowns
or
they're
part
of
control
plane
and
then
from
there.
We
have
the
normal
control
flow
right.
A
The
thing
that's
nice
about
this
is
that
we
start
to
commoditize
deployment
like
we
don't
have
to
start
thinking
about
how
to
deploy
specifically
to
GC
or
ec2.
We
can
start
to
develop
a
common
frameworks
across
all
these
and
strategies
for
how
we
manage
clusters,
and
we
also
standardize
the
operations.
So
cluster
api
is
capable
of
doing
things
like
upgrades
and
if
we
in
fact
they're
introducing
it
right
now
into
cluster
8,
but
as
to
how
we
define
strategies
for
what
the
upgrade
looks
like
it's
it
in
place.
Is
it
blue
green?
A
How
do
we
do
that
and
one
other
really
nice
thing
about
the
cluster
API
is
I.
Think
it's
going
to
start
to
ring
us
more
towards
the
idea
of
a
registry,
so
there
are
other
efforts
out
there
around
cluster
registries,
but
I
think
the
the
having
cluster
API
have
knowledge
of
the
environment,
the
clout,
the
cluster
versions,
the
machine
versions.
Whatnot
will
get
us
closer
to
be
able
to
do
things
like
Federation,
and
this
has
been
a
relative
pain
point
in
the
community
I
think
to
date.
A
One
thing
I
didn't
put
on
here
by
dimension
for
it
to
is
the
future
I
think
with
feature
will
also
start
to
see
that
more
of
these
more
of
the
native
kubernetes
components,
components
will
start
to
have
dynamic
configuration
so
we're
getting
that
with
the
cubelet
I
think
the
API
server
does
not
yet
support
that.
But
I'd
have
to
know.
This
is
something
that
we
are
particularly
interested
in
because
it
start
is
going
to
start
to
ease
some
of
those
day.
A
Two
concerns
so
to
wrap
up
I
totally
stole
this
Voltron
moment
from
our
CEO
Craig,
who
mentioned
last
week.
So
this
is
where
all
the
pieces
start
to
come
together,
like
those
five
things
that
you
should
think
about.
Number
one
is
simplicity.
Number
two
is
user.
Experience
make
sure
that,
with
these
two
cultural
aspects,
you're
giving
users
what
they
want
in
a
simplistic
way,
this
will
get
you
the
first
way
into
your
adoption
of
kubernetes
and
then
from
a
very
tactical
point
of
view.
We
should
we
talked
about
the
platform.
A
A
Yeah,
so
you
can,
there
are
other
things
to
think
about
when
you
get
there
as
well,
so
we
didn't
cover
that
I
mean
so
there
are
I'm.
Sorry,
yes,
I
apologize.
The
question
is
when
you,
when
we
talk
about
multiple
availability
zones,
have
we
seen
things
like
network
latencies,
etc
between
availability
zones?
Yes,
you
can
certainly
have
late
sees,
but
typically
with
most
cloud
providers,
you
do
have
some
guarantee
and
it's
pretty
it's
pretty
good.
A
You
know
again,
you
sorry,
the
skaters
were
talking
about
here,
right
yeah,
so
we,
this
is
actually
something
we've
discussed
internally.
You
will
have
some
of
that
and
I
think.
This
is
why
the
recommendation
that
we
made
is
like
when
you
have
when
you
experience
these
AZ
failures,
that's
something
that's
typically
a
time
where
services
are
trying
to
restart
something
is
happening
in
in
the
environment
that
may
affect
latency.
So
that
is
definitely
something
to
consider
when
you
decide
between
these
two
usage
patterns.