►
From YouTube: Kubernetes Security and How to Fix K8s Cluster at Scale - Simon Pearce & Sebastian Scheele
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Kubernetes Security and How to Fix K8s Cluster at Scale - Simon Pearce, SysEleven & Sebastian Scheele, Loodse
As a hosting provider, we have the challenge to run and manage multiple Kubernetes clusters for various customers on our infrastructure, similar to e.g. Google or Azure in a secure way. The majority of these clusters are fully managed by us. Our customers want to build and run containers. Not maintain and upgrade Kubernetes clusters. In this talk, we will give you a breakdown on how we help our customers to secure their clusters and how we can force Kubernetes upgrades to all clusters in a scalable way. We will use the Kubernetes API bug occurred in December as an example to show how we could fix all Kubernetes clusters in a very short time frame. This talk focus on the secure operation of multiple Kubernetes clusters and the requirements a Cloud Provider have to it.
https://sched.co/MPYO
A
B
Morning,
Hugh
Khan,
my
name
is
Simon
Pearce
and
all
a
warm
welcoming
to
Barcelona
nice,
sunny
city,
hope,
you're,
enjoying
the
keynotes
and
everything
one
of
the
first
talks
is
today
very
excited
to
be
here
today
and
basically
I'm
from
a
company
called
sis
11,
a
web
hosting
company
based
in
Berlin
in
Germany
and
sebast
Marie
said.
We
would
like
to
talk
to
you
today
about
how
we
manage
security
issues
at
611
and
how
we
run
kubernetes
clusters
at
scale.
So,
let's
get
started.
A
B
With
the
moss
sorry
what
much
okay
so
first,
first
of
all,
why
are
we
doing
all
this?
It's
basically
I
want
to
take
you
on
a
journey
here.
Tell
us
a
bit
about
our
story
about
what
we're
trying
to
achieve
here.
So,
first
of
all,
it's
definitely
important
to
know.
Why
are
we
actually
doing
this
we're
doing
this
because
we're
basically
a
service
provider?
So
we
offer
kubernetes
as
a
service
to
a
lot
of
customers
with
various
different
use
cases.
B
We
basically
have
run
multiple
clusters
and
they're
also
distributed
between
OpenStack,
and
also
we
run
them
on
AWS
too.
So
customers
have
got
the
freedom
of
choice
to
choose
between
these
two
platforms
that
they
would
like
to
use
and
basically
use
them
as
efficiently
as
possible.
Different
platforms,
different
features.
B
The
majority
of
these
clusters
are
managed
clusters.
In
all
cases,
we
definitely
take
the
full
management
and
responsibility
of
all
the
master
components
that
we
run
in
the
kubernetes
cluster,
because
we
feel
that
the
majority
of
the
customers
do
not
want
to
take
care
of.
Is
my
etcd
running?
How
is
my
API
server
doing?
What
is
my
controller
manager
doing?
The
major
interests
of
all
of
our
customers
is
running
containerized
applications
at
scale.
B
B
So
what
is
our
motivation?
Why
are
we
actually
trying
to
do
this?
What
is
basically
what
we're
trying
to
achieve
here?
One
is
to
be
one
of
the
best
managed
humanity
services
in
Germany,
at
least,
maybe
which
is
kind
of
looking
at
something
which
is
quite
difficult
to
become,
but
not
impossible.
I
would
say
also
a
great
motivation
is
having
to
deal
and
fix
security
issues,
so
basically
have
a
check.
What
security
issues
are
there
and
how
can
I
deal
with
them
and
what
do
I
actually
have
to
do.
A
B
A
You
could
escalate
your
your
calls
so
that
you
can
complete
access
to
the
API
when
you're
running
an
API
and
react
aggregated
at
the
API
server
and
like
in
the
default
setup
on
most
cuban
ITA's
systems.
The
metric
server
was
running
there.
So
this
was
the
case,
and
so
especially
if
your
queue
benitez
api
server
is
publicly
exposed,
you
must
act
quite
fast
and
it
Simon
said
the
node,
for
example.
A
At
this
level,
they
don't
must
be
on
a
public
neck,
but
but
somehow
the
masters
are
on
a
public
network,
so
you
must
act
fast
and
it's
much
more
important.
First
of
all,
you
need
to
know
which
systems
you
are
patching
because
when
you
know
ok,
there
potentially
50
systems
in
the
in
your
organization,
I
assure
which
versions
they
are
running
so
to
really
have
a
centralized
view
on
it
to
know.
Ok,
this
classes
are
affected,
this
classes
I
need
to
patch
and
how
to
do
this.
B
So
next
is
basically
the
challenge.
What
challenges
are
we
facing
here
and
I
would
like
to
talk
to
you
a
bit
about
what
kind
of
challenges
you
have
when
you're
running,
multiple
hundreds
of
communities
clusters
so
basically
we're
running
hundreds
of
clusters,
different
cloud
providers
for
a
vast
amount
of
customers
and
a
large
majority
of
these
customers
also
run
production
workloads.
We're
not
just
talking
about
testing
def
staging
stuff,
we're
actually
talking
about
production
workloads.
B
These
are
distributed
as
I
said
earlier
across
multiple
clouds,
which
also
gives
us
more
issues
that
we
need
to
face
an
encounter,
because
the
way
you
may
have
to
roll
out
surfs
and
things
you
may
need
to
change
things
like
base
images.
Network
configurations
are
different
across
different
cloud
providers,
as
the
majority
of
you
probably
already
know.
B
B
So
number
one
would
be
like.
How
do
you
go
about
upgrading
production
clusters,
so
you've
got
like
workload
on
there
and
you
somehow.
You
need
to
shift
this
workload
from
maybe
one
cluster
to
another
onto
another
worker
nodes.
There's
lots
of
stuff
that
you
need
to
think
about
and
to
consider
when
making
these
decisions,
and
one
of
the
major
problems
is
also
that
this
in
the
majority
of
times
needs
to
be
done
as
soon
as
possible,
depending
on
the
case.
A
So
I
mean
if
you're
running
more
than
20
classes.
Fifty
clusters-
you
cannot
do
this
manually.
You
need
to
out
automate
this,
so
one
possibility
is
to
build
an
operator
for
cumin
eaters
which
manage
your
queue
bonitas
cluster,
so
that
you
can
really
leverage
all
the
functions
functions.
Kubernetes
provides
you
to
upgrade
and
to
run
across
your
whole
infrastructure
to
upgrade
the
clusters
and
the
nodes.
So,
and
how
can
you
do
this?
Like
one
building
block,
you
could
currently
look
like
it's
quite
early.
A
It's
currently
still
alpha
and
there's
a
lot
of
discussion
around,
but
you
can
at
least
getting
ideas
if
you
want
to
build
something
like
this
is
cluster
API,
which
is
a
part
of
the
sig
life
cycle
group,
where
the
intention
is
building
building
blocks
for
different
use.
Cases
like
how
to
spin
up
a
control
plane
up
how
to
manage
my
my
worker
nodes,
how
to
upgrade
my
worker
nodes
and
things
like
this,
so
so
really
looking
into.
A
So,
of
course,
one
of
the
first
parts
is
classic
creation.
Cluster
updates.
So
before
you
can
manage
anything
or
can
manage
a
class
that
you
need
to
upgrade
the
class
as
creates
a
cluster
and
then
later
also
have
the
possibility
to
upgrade
the
cluster
and
somehow
have
a
centralized
registry
where
you
can
register
all
your
clusters,
which
are
running
in
your
organization,
so
that
you're
not
running
in
the
problem.
A
Okay,
if
you
develop
a
set
of
clusters
there
and
some
other
developer
set
up
a
class,
are
there
and
from
a
security
perspective,
you
don't
even
know
what
version
of
Cuban
Eaters.
You
are
running,
what
version
of
operating
system
container
runtime
you're
running,
and
so,
if,
if
a
back
is
coming
up,
you
start
searching
in
your
organization
what
clusters
do
any
to
upgrade
or
inform
your
developer
and
in
hopes
that
you
develop
our
upgrade
or
the
cluster
and
don't
forget
some
small
development
just
somewhere
in
the
back,
which
is
then
affected.
Potentially
that's
one
point.
A
A
This
manages
and
controls
us
and
last
but
not
least,
your
machines
itself,
so
the
worker
nodes
and
the
nodes
of
c.c
master
in
the
best
case
yeah,
you
want
to
control
the
queue
Benitez
version,
the
darker
version
and
all
kinds
of
stuff,
and
one
thing
which
is
currently
developed
in
the
cluster
API,
for
example,
is
a
machine
deployment
object.
We
should
then
can
use
to
do
rolling
upgrades
of
your
machine
so
that
you
can
upgrade
the
machines
automatically
without
a
lot
of
interactions.
A
Yeah
and
what
we
do
to
make
this
even
easier
together
was
this
11.
We
run
queue
Benitez
on
cue
Benitez,
so
because
we
said
ok,
we
don't
want
to
for
every
class.
I
want
to
have
the
overhead
of
running
a
complete
master
with
this
VMs.
So
we
put
the
complete
master
in
container
and
run
only
the
container
where
this
makes
it
already
easier,
because
we
don't
care
for
the
master
on
the
operating
system
layer.
B
B
Of
course,
for
us
offering
a
managed
service
is
monitoring.
How
can
we
actually
monitor
find
out
what
are
people
running
in
their
clusters
and
one
specific
thing
is:
what
version
of
communities
are
they
running?
Are
they
actually
running
a
version
which
is
vulnerable
which
could
cause
us
problems
and
in
this
graph
on
a
graph?
B
What
I'm
gonna
have
to
do
I'm
going
to
rethink
this
I
mean
kubernetes
is
a
fast-moving
piece
of
software.
So
there's
going
to
be
a
lot
of
updates
to
be
done,
something
you
need
to
consider
when
you're
running
these
sort
of
systems
in
production.
So
how
do
we
actually
do
this?
We
can
basically
patch
the
cluster
objects.
We
have
a
cluster
object
which
basically
defines
the
API
servers,
the
kubernetes
version.
B
We
would
like
to
run
and
all
those
sort
of
things
and
we
can
use
a
standardized-
a
cube,
CTL
patch
method
to
actually
patch
these
resources
and
provide
each
customer
individually
with
an
upgrade.
So
it's
also
possible
to
do
canary
upgrades
and
other
things
to
actually
find
out
if
we
are
going
to
run
into
any
potential
issues
while
doing
this,
and
then
we
basically
come
to
the
upgrade
process
as
in
such
what
do
we
have
to
do
to
actually
accomplish
this?
B
So
first,
of
course,
when
we're
talking
about
security
issues,
we
need
to
know
what
is
affected
and
who
is
affected.
So
often
this
is
done.
If
you,
if
you
find
out
what
how
severe
is
the
impact
going
to
be
so,
if
we're
talking
about
something
public
facing
like
the
API
server,
for
instance,
and
if
we
look
back
at
one
of
the
bugs
which
Sebastian
showed
us
where
you
could
basically
use
anonymous
requests,
you
could
basically
use
the
API
server
to
proxy
requests
through
to
the
metrics
endpoint
and
to
other
things.
This
can
be
very
severe.
B
One
of
the
instruments
and
one
of
the
tools
that
we've
introduced
at
sister
Levin
to
overcome
these
sort
of
situations,
is
a
so
called
change
advisory
board.
So
we
have
like
a
group
of
kubernetes
engineers,
some
security
engineers
they're
all
on
the
board.
Most
of
these
all
these
people,
of
course,
have
a
great
idea
about
how
kubernetes
runs,
which
moving
parts
we're
talking
about
and
how
severe
this
issue
is.
B
We
basically
are
always
reading
the
mailing
list,
checking
out
what
kind
of
issues
arise
on
github
our
partner
loot,
so
here
a
great
help
doing
that,
of
course,
because
they
do
a
lot
of
engineering
within
the
kubernetes
ecosystem
so
automatically
you
find
out
about
these
sort
of,
hopefully
a
bit
quicker
than
the
general
public.
So
we
can
basically
kind
of
address
these
issues.
B
Then
we've
got
a
notifying
system,
so
we
can
find
out
what
version,
as
we
saw
on
the
graph
on
our
dashboard,
are
people
actually
running,
and
then
we
can
basically
through
our
notifier.
We
can
inform
these
people
individually
and
tell
them
okay.
Your
system
is
affected.
We
have
a
problem
here
and
if
it
is
an
urgent
severe
issue,
then
we
are
going
to
fix
this
issue
today.
B
We're
not
going
to
wait
another
day,
we're
going
to
start
off
and
we're
definitely
going
to
patch
your
API
server
and
make
sure
it
is
in
a
good
state
again,
then
we
go
and
we
start
patching
all
or
the
majority.
Normally
it's
like
that.
We
do
it
for
one
region
first
and
then
we
would
go
to
another
region.
Maybe
we
can
pick
certain
customers
which
might
be
more
vulnerable
than
others,
and
definitely
look
at
the
targeted
versions
that
we're
looking
at.
B
B
Then
we
can
go
and
upgrade
the
machine
deployments.
We
now
have
the
new
base
image.
We
can
switch
the
current
machine
deployment,
which
is
basically
a
rolling
upgrade
so
similar
to
you
would
do
when
you
you're
deploying
your
application.
We
can
do
exactly
the
same
for
all
of
our
worker
nodes.
We
basically
tell
cumin
register,
ok,
there's
a
new
version
of
this
image
available.
There's
a
new
version
of
kubernetes
available.
B
We
have
exactly
the
same:
Mac's
unavailable,
Mac
search
configuration
possibilities
available
to
dynamically,
upgrade
the
nodes
and
also
train
the
nodes
at
the
same
time,
so
that
we
basically
shift
the
container
work
Road
from
one
node
to
another
one
by
one
to
make
sure
none
of
the
customers
are
affected.
During
that
time,
we
don't
want
any
of
the
customers
to
have
any
outages.
While
we're
doing
this
so
rolling
upgrade
of
core
as
a
kubernetes
feature
is
one
of
the
key
components
to
be
able
to
do
this
here.
B
We
definitely
need
to
look
at
upgrading
the
docker
daemon.
If
we're
looking
at
some
of
the
issues
we
were
talking
about
earlier,
the
docker
daemon
was
in
a
state
where,
if
people
were
running
containers
with
a
UID
0,
so
as
a
root
which
I'm
afraid
a
lot
of
people
still
do,
then
it
was
possible
to
break
out
of
the
actual
container
and
to
obtain
root
access
on
the
host
system,
which
of
course
is
definitely
a
no-go.
So
it's
definitely
also
something
that
needs
to
be
addressed
fairly
quickly.
B
A
I
mean
what
we
learned
during
the
last
2
to
3
years.
First
of
all,
what
really
helps
us
automate
all
your
upgrade
processes,
especially
if
you
are
over
20
systems
clusters.
You
don't
want
to
do
this
manually
anymore.
You
need
to
do
automation
and
everything
should
be
built
up
process,
so
they
easily
can
deploy
new
versions
without
any
manual
intervention
so
that
you
know.
First
of
all,
you
know
that
your
process
is
running
and
that
really
no
one
needs
to
interact.
A
When
you
know,
ok,
I
need
to
upgrade
all
my
clusters
now
that
it's
only
like
you're
putting
somewhere
version
and
then
the
control
plane
starts
and
doing
this.
Another
important
stuff
is
what
we
learned
is
like
we're
doing
a
lot
of
e
to
the
e
tests
on
all
supported
cloud
providers.
So
what
you
see
on
this
slide?
A
We
support,
like
all
the
majors
AWS,
Asia
Google,
but
also
OpenStack
VMware
and
many
more,
and
so
we
really
want
to
need
to
ensure
when
there's
a
new
version,
we
can
roll
this
out
and
it's
not
breaking
so
the
only
thing.
What
we
currently
only
need
to
do
now
is
like
we
putting
the
new
version
into
our
CI
system
in
our
version
file
and
then
the
our
test
suite
starts
running
and
tests.
First
of
all,
can
we
spin
up
a
cluster,
but
also
things
like?
Can
we
upgrade?
A
Can
we
upgrade
from
Graham
version
to
the
next
version
and
like
after
two
three
hours,
we
know
okay,
yeah.
The
new
versions
are
running
with
all
the
different
combinations,
also
with
different
potentially
combinations
of
the
operating
system.
So
and
then
we
know
okay
yeah.
Now
we
can
ship
this
to
our
customers
and
they
can
bring
this
into
production.
A
A
next
thing
is,
if
you're
doing
this,
automated
running
your
CI
system
testing
if
subversions
are
running
running
the
conformance
test,
so
that
you
everytime
know:
okay,
there's
nothing
braking,
but
the
conformance
tests
are
not
enough.
Look
into
what
additional
things
you
need
like
the
conformance
test.
Don't
cover
your
storage
and
don't
cover
your
load
balancer,
so
putting
tests
additional
tests
around
this,
but
that
at
the
end
you
know.
A
And
so
then
you
know:
okay,
everything
is
working
and
you
can
bring
the
new
version
into
production
what's
currently
quite
new
and
which
makes
sense
to
look
into
port
security
policies
so
that
you
can
enforce
the
users
to
potentially
not
running
specific
things,
for
example,
that
the
container
is
running
not
as
a
root
user,
so
they
already
prevent
you
from
the
run,
see
CVE
we
saw,
and
when
you
have
this
activated,
there's
one
thing
less.
Of
course
you
should
upgrade
the
and
purposes,
but
it's
not.
A
A
So,
potentially
you
want
to
prevent
the
developers
to
do
specific
things
or
have
more
control
what
you
do
in
the
cuban
etus
api
yeah
and
then
the
SE
c--,
as
I
benchmark,
which
describes
security
perspectives
from
cuban
ages
and
there's
also
some
tools
available,
like,
for
example,
cube
bench
which
you
can
run
on
your
cluster
and
which
gives
you
at
least
some
indications
where
things
could
be
where
you
should
look
into
it,
and
then
you
can
dedicate
it
look
into
it
and
decide.
Okay.
A
A
Security
problems,
so
this
you
can
join
a
list
where
they
are
the
humanities
announcement
and
they
you
get
an
email
also
about
security
issues.
Of
course,
also
you
can
get
this
information
from
Twitter
or
in
this
leg
from
the
communities
community,
but
you
should
be
watching
this
actively
and
know.
Ok,
when
something
comes
out,
you
get
be
informed
because
that's
the
first
point,
you
need
to
be
informed
that
you
need
to
upgrade
before
we
can
take
any
actions.
B
Yeah
so
I
think.
Thank
you
very
much.
I
think
the
most
important
thing
about
here
for
the
this,
for
you
to
take
away
is
that
security
should
be
a
concept
in
your
company,
no
matter
if
you're
running
one
or
multiple
kubernetes
clusters
do
your
fire
drills
be
prepared,
security
issues
are
going
to
arise
and
they're
going
to
be
more
to
come
in
the
future.
I'm
most
probably
they're,
going
to
get
worse
so
definitely
be
prepared.
B
C
B
Great
question,
of
course,
it
would
be
fairly
simple
to
basically
adapt
something
like
eks
and
unused
it.
That
was
not
our
intention.
When
we
decided
to
go
out
and
build
a
service,
we
have
a
vast
amount
of
software
engineers
on
our
teams.
We
wanted
to
make
humanity's
feel
in
a
way
that
it
was.
It
was
built
by
us
that
it
had
like
the
additional
add-ons
that
we
wanted.
We
include
like
backups,
we
got
a
federated
from
ether,
it's
included.
Of
course.
B
D
B
So
you're
definitely
right.
We
encountered
a
few
issues
doing
this
number.
One
is
like
run
all
your
master
components
in
pots
which
helps
us
a
lot.
We
can
run
like
easy
to
replicas
of
things
like
API
servers,
so
it's
like
fairly
simple
to
actually
get
a
service
which
is
not
interrupted,
or
there
are
definitely
and
one
of
the
things
that
you
definitely
mentioned.
Pot
disruption
policies
can
cause
problems.
B
We
were
on
etcd,
for
instance,
as
stateful
sets
for
our
customers,
and
if
a
number
of
worker
nodes
are
giving
you
problems
or
you
have
any
network
outages,
then
definitely
things
like
pod
disruption
policies
can
cause
problems,
especially
if
they're
wrongly
configured
so
at
the
end,
it
all
boils
down
to
monitoring
and
good
engineers
and
know
what
they're
doing
I
mean
you've
got
to
consistently
look
at
these
services.
You've
got
to
find
out
what
kind
of
reaction
times
and
my
customers
have
what
kind
of
experience
they
have
and
also
our
company
ourselves.
B
E
A
Mean
there
are
different
different
options
on
the
one
side,
I'm
doing
the
classic
creation
we
can
decide,
okay
or
as
a
developer
can
decide
if
if
he
wants
to
have
what
security
policies,
but
on
the
other
side,
it's
also
a
question
how
you
shape
your
service.
So
if
you
say
ok,
my
service,
we
want
to
provide
you
the
best
experience
in
the
world
and
to
avoid
outages
why
your
expectation
is
a
complete,
manage
q
Benitez.
A
This
is
a
minimum
requirement
you
need
to
provide,
and
if
this
is
not
availability
application
not
in
that
way,
it
cannot
run
in
in
the
class,
and
this
is
why
I
work,
for
example,
looking
heavily
into
admission
controllers
so
that
it
really
can
detect
before
deployment
that
potentially
this
application
should
not
run
there,
because
we
know
ok,
it
could
cause
problems
and
inform
the
user
early
at
the
beginning
and
potentially
not
even
start
the
application.
Because
then
he
see.
A
B
F
B
There's
definitely
a
few
things
to
keep
in
mind
when
you're
running,
like
50
plus
worker
nodes
on
your
cluster,
because
basically
you're
talking
about
probably
shifting
a
lot
of
persistent
volumes
and
a
lot
of
parts
from
A
to
B.
So
probably
one
of
the
thing
is
what
you
could
do
is
like
put
them
into
individual
machine
sets.
B
Well,
not
so
much
as
they
would
during
the
day
time.
The
impact
is
not
quite
as
great
but
I
would
definitely
say
like
distributing.
Smaller
sets
of
machines
is
definitely
the
way
that
you
really
want
to
get
around
that
if
it's
possible
and
that's
we're,
definitely
using
work
scene,
worker
nodes
and
clusters
as
an
API
object.
It
definitely
helps
doing
that.
Okay,.
G
H
I
want
to
ask
if
the
urn
ability
that
we
have
is
outside
the
clustering
group,
so,
for
example,
of
if
the
notes
I
mean
on
the
level
of
the
notes
or
operating
system,
we
have
variabilities.
So
what
is
the
most
acceptable
policy
is
to
start
to
migrate,
the
the
current
cluster
to
another
infrastructure
or
to
repair
the
infrastructure
itself,
assume
that
the
all
nodes
in
the
infrastructure
had
the
same
in
has
the
same
their
nobility.
B
H
Now
suppose
we
have
thin
nodes
that
indistinct
a
star
and
workers,
and
these
two
not
suppose
that
it's
the
same
operating
systems,
infrastructure
specification
and
we
discovered
zero.
There
is
variability
in
level
of
infrastructure
on
the
veil
on
the
operating
system,
I
mean
outside
the
cluster,
so
how
we
can
manage
to
resolve
the
security
issues
outside
cluster
groups
is
the
best
way
to
start
managing
and
keep
the
cluster
itself
with
the
same
node,
and
they
start
to
solve
it,
not
by
nod
or
to
migrate
it
to
another
nodes.
H
A
I
I
think
I
think
we
can.
We
will
discuss
afterwards,
so
I
think
we
are
end
of
time.
So,
thank
you
all.
If
you
have
further
questions,
I
mean
we
are
still
here
and
can
answer
your
question.
Otherwise,
this
xi
has
a
booth
at
the
conference
and
we
also
have
a
boost
Lutzer
Lukas's
logo
and
there
you
can
ask
all
your
questions
thanks
a
lot.
Thank
you.