►
Description
Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).
What Does “Production Ready” Really Mean for a Kubernetes Cluster? - Lucas Käldström, Individual (Advanced Skill Level)
How would you describe and set up a “production ready” Kubernetes cluster? How are the buzzword terms “production ready” and “highly available” defined anyway? Can a cluster be created so that it’s end-to-end secured, has no single points of failure, is upgradable without control plane downtime and is conformant? If you have access to automated infrastructure, e.g. via a Cluster API controller, you should be able to do CI testing of your cluster, as well as CD of new configuration and versions. Some call this pattern “GitOps”; to write the desired cluster state declaratively and let a controller reconcile the cluster state. By the end of this talk, you should be able to tell: - What you may consider a “production ready” cluster to be and identify the moving parts - How to secure cluster component traffic - How to minimize failure points - How to manage clusters using the Cluster API
"About Lucas
Lucas is a passionate Kubernetes Maintainer and Certified Kubernetes Administrator that is excited about all things cloud native. Lucas has been engaged in Kubernetes work for about two years now and been involved in work like porting Kubernetes to multiple platforms, getting Minikube off the ground, being a core contributor in SIG Cluster Lifecycle and maintaining kubeadm. Besides Upper Secondary School Lucas runs a consulting company for Cloud Native tech programming tasks and runs the official CNCF & Kubernetes meetup in Finland. Lucas has been speaking at KubeCon in Berlin and Austin previously, and was awarded the Top Cloud Native Ambassador of 2017 together with Sarah Novotny."
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
Join us for KubeCon + CloudNativeCon in San Diego November 18 - 21. Learn more at https://bit.ly/2XTN3ho. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.
A
Hi
and
welcome
to
this
session
on
production
already
kubernetes.
What
is
it
it's?
It's
kind
of
this
bus
was
that
everybody
throws
around
so
we'll
well
dive
into
the
technical
details
of
what
could
be
considered
production
ready
cluster
and
go
from
there.
Two
different
aspects
of
that
I'm
Lukas
Kallstrom
from
Finland
I'm,
a
high
school
student
graduating
this
year,
actually,
which
is
a
lot
of
fun
I've,
been
to
three
years
active
in
the
governess
community
and
I'm,
a
CNCs
ambassador,
I'm
contracting
independently
this
time
for
other
currently
for
weworks.
She
working
well
and
yeah.
A
A
That
is
documented
in
our
official
company
experimentation
and
will
show
diagram
there
and
then,
as
a
fourth
thing,
will
look
into
the
cluster
API,
which
and
with
which
is
an
ongoing
effort
in
Sigma
life
cycle.
As
work
group.
We're
trying
to
define
declarative
approach
to
cluster
management,
and
we
hope
it
to
be
an
emerging
standard
in
in
this
space.
A
So
yeah
and
to
just
be
clear,
there's
a
lot
of
there's
many
layers
in
kubernetes
clusters:
application
management.
So
this
talk
is
focusing
only
on
the
convenience
layer.
You
could
have
another
talk
on
how
to
do
production
deployment
or
through
blip
application
deployments
in
a
production-ready
manner
or
production-grade
manner,
or
how
to
provision
your
infrastructure
in
a
production
grade
manner.
But
this
is
talking
about
the
kubernetes
cluster,
how
we
can
set
that
up.
A
So
it's
resilient,
and
here
we
go
one
possible
definition
that
we
could
think
of
is
that
the
cluster
is
production
ready
when
it's
in
a
good
enough
shape
for
the
user
or
the
administrator
to
serve
real-world
traffic.
This
can
be
very
large
deployment
with
thousands
of
nodes
and
high
availability
and
everything,
or
it
could
be
like
a
one
master
thing
that
is
having
a
blog
either
case.
So
it's
it's
kind
of
it's
facing
the
real
world
or
the
users,
customers
and
good
enough.
A
I
wrote
good
enough
there,
because
it's
all
about
trade-offs,
so
you
have
to
you,
have
to
draw
the
line
somewhere.
Another
possible
definition
which
caught
the
Moga
from
Google
had
in
another
relevant
talk,
is
your
offering?
Is
production
ready
when
it
slightly
exceeds
your
customers
expectations
in
a
way
that
allows
for
business
growth?
So
it
should
be
slightly
more
than
you
expect,
but
not
too
much
so
we
have.
We
are
we're
on
the
other
side,
and
these
are
two
possible
definitions.
Less
probably
many
more.
A
Again,
it's
a
lot
of
trade-offs.
We
have
to
do
this.
As
you
might
know,
kubernetes
has
like
hundreds
of
command
line
flags
and
there's
probably
thousands
of
ways
you
can
set
up
your
cluster,
like
everyone
here
has
their
own
kubernetes
offering
and
the
trust
of
lifecycle
space
is
actually
kind
of
fragmented,
so
simplest
life
cycle
is
trying
to
unify
these
efforts
and
provide
a
better
base
to
run
kubernetes,
and
in
this
we
have
to
find
some
some
kind
of
balance
between,
for
example,
security,
high
availability
and
other
things
that
affect
the
cluster.
A
A
So
I
have
five
items
and
the
first
one
is
the
cluster
is
reasonably
secure.
The
security
features
in
kubernetes
are
still
evolving,
especially
a
lot
latest
year.
That
has
been
a
lot
of
focus
on
security
and
our
back,
for
example,
graduated
stable
last
year,
which
is
a
great
and
essential
effort
in
having
any
kind
of
security
in
your
kubernetes
cluster
and
but
again
reasonably
here
is
you?
A
A
What
we
really
nice
to
have
in
my
class
to
setup
is
that
all
elements
like
the
API
server
controller
manager,
scheduler
all
the
cubelets
and
so
forth,
are
declaratively
controlled
by
the
system
that
manages,
for
example,
infrastructure.
Has
this
whole
world
view
of
the
cluster
instead
of
imperatively,
setting
things
up,
executing
commands
on
the
command
line,
SS
aging
into
nodes
so
and
so
on
and
so
forth?
Instead,
it
would
be
nice
to
just,
for
example,
for
upgrades,
go
in
some
kind
of
resource
that
is
centrally
stored
and
bump.
A
The
version
number
to
the
next
minor
or
patch
version,
and
then
some
kind
of
controller
kicks
in
and
make
sure
this
at
this
desired
state
is
the
actual
state
and
axon
that
sounds
familiar.
That
is
basically
what
kubernetes
is
all
about
fat
occasions,
but
can
we
find
something
like
this
in
a
Cabana
centric
way?
That
is
for
the
cluster
itself?
Maybe-
and
this
also
makes
it
easier
to
upgrade
rollback
and
perform
normal
operations.
A
The
operator
pattern,
driven
by
chorus
and
similar
folks,
is
really
kind
of
this
exact
this
thing,
but
can
we
expand
it
to
our
own
notes
as
well,
then
changes
to
cross
the
state
can
be
safely
applied.
That
is
one
of
the
blue
bars
for
me,
at
least
for
a
production
run
a
cluster.
If
we
can,
we
know
that.
Oh
here's,
a
new
patch
version
of
kubernetes.
We
know
that
we
can
actually
apply
this
in
a
safe
manner
and
not
suffer
from
downtime.
A
And
the
last
thing
is
touching:
on
conformance
so
I
see
no
cabinet,
CN
n
CF
launched
the
kubernetes
conformers
program
last
year
and
Cabrales
has
these
end-to-end
tests.
That
is
verifying
that
the
cluster
works
as
it
should.
Currently,
we
have
there's
a
lot
of
entrants.
A
synchronous,
a
small
set
of
them
are
marked
with
the
conformance
label,
and
this
conformance
label
is
good.
It's
verifies.
We
have
a
some
kind
of
standard
between
different
deployments
of
kubernetes,
for
example,
iki
sa
KS
g,
ki
e
cubed
m
and
all
cops
all
the
providers.
A
We
know
that
if,
if
all
of
these
deployments
passes
past
conformance,
we
can
move
our
applications
from
place
to
place
and
rely
on
the
same
api's.
Unfortunately,
the
the
current
conformance
tests
have
pretty
low
coverage
of
all
the
convenience
features,
so
we
need
to
work
as
a
team
in
the
community
hi
hi.
Oh
this
bar.
A
Meanwhile
there's
a
lot
of
end-to-end
tests.
You
can
optionally
enable
for
your
inten
tests,
flow
and
I
would
say.
If
you
have
a
bananas
or
production-ready
cluster,
then
try
to
to
a
higher
the
bar
make
the
bar
higher
when
running
these
end-to-end
tests.
So
you
actually
show
that
more
is
covered
than
just
a
conformist
bar.
A
And
here
is
a
slide.
I
often
show
in
my
presentations
trying
to
visualize
what
kubernetes
actually
looks
like
and
the
architecture.
So
in
the
heart
of
the
system,
we
have
the
API
server.
That
is
a
web
server
that
talks
to
its
database
at
CD,
which
can
be
cluster,
and
it's
the
where
every
everything
is
stored.
All
the
data
and
API
server
can
be
horizontally
scaled.
That's
not
a
problem,
and
the
controller
manager
and
schedule
talk
only
to
the
API
server
and
go
that
way
access
and
are
accessing
that
way.
A
The
data
at
CD
and
then
we
can
have
all
the
policy
enforcement
and
similar
API
conventions
in
the
API
server.
The
controller
manager
makes
reconciles
all
time
make
sure
the
actual
state
is
the
desired
state
and
the
scheduler
finds
pods
that
are
not
bind
turnout
and
binds
the
pod
to
the
node
of
choice
in
in
the
pool
in
the
cluster.
On
a
node
side.
A
Again,
we
have
first
iOS
which
can
be
Linux
or
Windows,
and
we
have
some
kind
of
container
runtime
which
can
be
docker
container,
DCI,
Rio
or
any
CRI
compliant
runtime,
which
you
see
there.
The
different
colors
represent
different
interfaces
for
communication
between
components
and
then
cubelets,
which
is
the
node
agent
of
kubernetes
talks
to
the
container
runtime
and
says
what
what
containers
to
execute
and
the
container
runtime
executes
this
container,
which
then,
of
course
runs
on
top
of
the
operating
system.
A
Eventually,
the
cubelet
pulls
the
API
server
were
actually
has
a
watch
through
API
server
and
receives
these
new
pods
bind
it,
and
when
it's,
when
it's
now
knows
that,
I
should
run
this
application.
It
executes
call
we
ser
I,
but
it
also
executes
a
call.
We
are
CNI
to
allocate
an
IP
and
a
unique
IP
address
for
a
pod
in
a
cluster,
and
this
networking
layer
is
third
party,
so
Cabrales
doesn't
ship
with
any
defaults.
A
A
A
You
might
have
heard
of
the
CSR
API
in
kubernetes,
that
is,
certificate,
signing
request
API.
So
that
means
there's
a
controller
run
in
the
controller
manager
or
somewhere
else
that
signs
certificate,
signing
request,
s--
uploaded
to
the
api
server
and
then,
for
example,
this
is
used
for
for
cubelets.
A
Then
the
cubelet
will
get
an
unique
identity
on
demand
more
about
this
later,
but
if
you're
on
a
cloud
or
somewhere
that
way,
you
have
the
possibility.
You
should
use
an
external
key
signer
and
have
this
key
CI
key
external
to
the
cluster,
not
inside
the
cluster,
API
authentication
and
authorization.
A
Localhost:8080
was
a
thing
in
many
clusters,
maybe
a
year
ago,
it's
more
maybe
and
it
provided
you
its
root
access
to
all
the
nodes
in
the
cluster
and
master.
Anything
access
to
the
API,
server,
unsecured
or
like
unauthorized,
gives
you
root
access
to
every
node.
Also
anonymous.
You
should
disable
an
investigation,
this
or
or
the
local
of
say.
The
editing,
for
example,
was
disabled
by
a
year
over
a
year
ago
by
cubed
m
and
similar
deployments
enforce
our
back
and
the
node
authorizer
are
also
good
things
to
to
keep
in
mind.
A
If
you
have
our
back
on
then
and
and
have
deny
all
by
default,
you're
pretty
sure
that
a
normal
workload
in
the
cluster
can't
write
root
access
to
the
API
server,
which
is
kind
of
bad.
The
node
wrote
riser
is
basically
making
sure
that
nodes
can't
do
anything.
Instead,
they
can
only
handle
pods
and
similar
resources,
access
ports
and
similar
resources
that
they
actually
should
run.
A
The
couplets
in
the
cluster
have
sometimes
fun
defaults,
read-only
port
is
and
the
C
advisor
ports
are
by
default,
open
to
the
whole
world.
So
if
you're
running,
for
example,
a
public
digitalocean
droplet
or
another
William
in
any
cloud
which
is
accessible
by
the
internet,
everyone
can
read
your
like
top
three
and
be
like
PS
everything,
C
CPU
usage
memory
usage
check
out
the
processes
read
what
all
the
ports
you
you
have
in
kubernetes
and
similar
things.
A
So,
of
course,
it's
recommended
to
disable
these
ports.
We
disable
the
public
service
port
already
a
year
ago
in
Cuba
I'm.
The
read-only
port
is
about
basic
lockdown
by
default
in
111
I
guess
it
might,
there
might
be
some
consumers
of
it,
but
since
last
life
cycle
and
single
trailer
recommends
consumers
of
the
read
cubelet
read-only
port.
A
Try
to
get
a
client
certificate
to
talk
to
the
secured
port
instead
and
each
cubelet
should
have
its
unique
identity,
will
get
rough
in
a
bit
the
dashboard
and
helm
or
have
been
like
disgusts
pretty
much
about
as
well.
Please
don't
give
cluster
admin
power
to
everyone
using
the
dashboard
or
everyone
using
helm.
That
can
then
it's
possible
that
your
cluster
is
mining
bitcoins
after
some
time.
A
So
instead,
the
dashboard
has
actually
executed
on
this
and
since
1/7,
it's
as
a
login
screen
and
let's
user
of
the
web
api
authenticate
and
the
dashboard
just
delegates
this
these
credentials,
because
if
you
ever
go
and
like
run
the
thing
in
the
kubernetes
box,
that
says
has
like
huge
warning
signs.
Do
not
run
this
in
case
you're,
really
sure
and
want
to
disable
our
Mac.
A
If
you
do
that,
any
any
work
loading
commanders
can
do
anything
or
if
you
do
that
for
dashboard,
any
user
dashboard
can
do
anything.
The
same
goes
for
helm.
You
should
set
up
some
kind
of
lists,
a
cluster
row
and
specify
which
operations
in
our
back,
which
operations
tiller
the
server
side
of
helm,
may
perform
and
there's
some
there's
some
good
stuff
in
this
blog
post
as
well.
You'll
get
the
slides
from
this
later.
So
you
can
follow
the
link
then
I
by
default.
A
There's
some
good
good
API
is
here
are
back,
which
we
talked
about.
Network
policing
is
another,
so
by
default
all
ports
can
talk
to
each
other.
If
you
don't
like
that,
specify
a
network
policy-
and
here
you
need
a
third-party
network
provider
that
supports
a
network
policy
and
deny
all
communication
by
default
and
then
just
whitelist.
Oh,
my
web
service
should
be
able
to
talk
to
the
backend
and
then
certipur-us
restrictive,
odd
security
policy
after
default.
A
So
again,
in
the
hardest
system,
we
have
an
API
server,
it
has
a
serving
cert,
and
this
has
a
cube
of
fines.
Will
talk
about
the
cube
list,
client
sue,
the
controller
manager
and
the
scheduler
have
client
certs
to
the
API
server
and
has
a
have
a
common
common
name.
The
search
should
have
these
common
names
in
order
for
it,
oughtn't
ik
8
in
the
right
way
to
the
API
server
with
our
back.
A
A
If
it
satisfies
some
criterias
and
then
the
csr
signer,
which
is
the
only
thing
that
should
have
access
to
the
CI
key,
if
it's
approved,
then
the
csr
signer
goes
ahead
and
sign
this
csr,
which
makes
it
possible
for
the
cubelet
to
get
its
own
unique,
client
search
on
demand.
We
didn't
have
to
pre
allocate
this
then
when
you
do
cube,
CT
log
or
cube
CTL,
exec,
you're,
actually,
first
talking
today,
api
server
and
then
the
api
server
with
a
client
cert
talk
to
the
nodes
to
the
cube
ceiling
or
cube
latin
question.
A
And
here
we
have
a
diagram
of
what
of
the
different
aspect:
layers
of
cubanelle,
our
kubernetes
cluster,
so
we
have
in
infrastructure.
Cubed
M
is
running
on
each
of
these
locally.
On
each
of
these
nodes
machines,
then
they
form
a
kubernetes
layer
and
then,
on
top
of
that
we
run
I,
don't
like
Obama's
as
monitoring
logging,
cloud
extensions
and
so
so
on
and
so
forth.
We'll
talk
about
more
about
the
cluster
API
later
Cuba
names
task
is
to
set
up
a
Minimum
Viable
best-practice
kubernetes
cluster.
A
A
Using
your
preferred
method,
then
you
provide
some
kind
of
external
load.
Balancer
like
a
cloud
cloud
load.
Bonus
are
using
DNS,
result
resolving
and
then
you
queue
barium
join
the
nodes,
and
then
it
will
look
like
this,
so
this
is
possible
and
it's
making
the
life
easier
for
the
operators
when
you
use
incubate
them.
It's
still
some
tasks
to
do
manually
and
depending
on
what
environment
you're
in,
but
still
you
have
a
lot
of
common
functionality
in
one
piece:
the
building
block,
which
is
cubed
M
and
will
will
thrive.
A
Signals
lifelike
strives
to
make
this
easier
all
the
time
and
then
something
I've
seen
when
talking
to
people
a
common
misunderstanding
in
some
cases
or
the
difference
is
the
difference
between
multi-master
and
highly
available.
So
it's,
for
example,
this
setup
highly
available.
Anyone
know
exactly
because
here
we
have
a
single
point
of
failure
in
this
master
D.
So
this
cluster
is
multi
master,
yes,
but
it
has
some
kind
of
single
point
of
failure.
It
could
be
like
this
is
just
a
common
example
or
like
silly
example,
but
there's
there's
even
more
corner
cases
to
this.
A
A
A
For
example,
the
service
account
signing
key
for
the
controller
manager
if
you
have
different
signing
Keys
on
different
masters
on
the
controller
for
the
control
manager,
you're
gonna
end
up
its
you're
gonna
have
trouble
and
that
it
also
makes
it
complicated
when
you're
doing
upgrades
and
certificates
rotation,
especially
in
an
age
a
cluster,
so
the
class
life
service,
hopefully
looking
into
providing
some
best
practice
documentation
for
such
cases.
If
you
want
to
contribute
to
our
efforts,
please
do
get
in
touch
with
us
and
help
us
improve
the
documentation
on
kubernetes
talks.
A
A
And
the
last
last
piece
is
declarative
control
with
the
cluster
API.
This
is
still
in
alpha,
as
I
said,
we're
expecting
to
graduate
to
be
the
later
this
year,
but
it's
really
really
interesting
progress
and
chris
nova
and
robert
bailey,
who
is
leading
the
workgroup,
are
more
than
happy
to
have
you
there
and
comment
engage
with
the
group,
for
example,
or
this
is
a
declarative
way
to
configure,
create
manager
cluster
using
Bernina
CM,
normal
kubernetes
api
conventions.
A
We
all
kind
of
agree
on
at
this
point
and
then
reconcile
in
the
same
way
as
the
normal
Cabrera's
controllers,
that
do
cloud
providers
will
implement
this
small
small
controller
loop,
small
shim,
that
check
that
do
I.
Well,
now,
I'm
expected
to
run
a
machine
that
ID
don't
in
the
cluster
or
only
in
the
cloud.
A
So
this
goes
sounds
like
a
topic,
that's
being
discussed
widely
in
in
different
forums
which
is
get-ups.
So
with
this
new
cluster
api,
you
can
technically
then
have
used
git
as
your
single
or
your
the
the
through.
Let's
say
sorry,
where
you
store
the
truth,
and
here
we
have
a
machine
set
that
is
declaring
how
many
replicas
of
the
nodes
machines
should.
We
have
and
there's
also
a
provider
config.
That
should
also
have
the
satisfied,
a
kubernetes
api
conventions
and
similar
similar
stuff.
A
A
So
we
have
two
minutes
left
and
he
has
to
recap
identify
the
needs
of
your
business.
How
much
do
you
want
to
put
into
how
much
effort
and
money
do
you
want
to
put
into
a
che
which
adds
complexity
and
how
much
do
you
want
to
focus
on
security?
I
really
encourage
you
to
do
all
these
things,
but
just
try
to
find
the
balance,
and
if
you
do
put
a
lot
of
efforts
here
in
insert
in
these
things,
please
give
us
feedback
on
the
OSS
components
we
have
like
cubed
em
in
the
cig
meetings.
A
Higher
availability
is
not
the
same
as
multi
master,
which
we
talked
about.
The
certificate
identities
are
really
important
for
the
cluster
to
function.
Well,
and
it
should
be.
You
should
be
considered
with
our
back,
for
example,
try
to
disable
the
unnecessary
ports
if
you
can
etc,
and
the
clarity
of
control
over
your
cluster
is
at
least
in
my
opinion,
better
than
imperative.
So
you
can
have
some
kind
of
reconciliation
loop
that
that
doesn't
thing
this.
That
applies
the
state
and
still
offer
please
contribute,
so
we
can
get
it
to
be
now
and
that
is
it.