►
From YouTube: Sponsored Keynote: What I Learned Running 10,000+ Kubernetes Clusters - Jason McGee, IBM Fellow, IBM
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Sponsored Keynote: What I Learned Running 10,000+ Kubernetes Clusters - Jason McGee, IBM Fellow, IBM
After years of running Kubernetes, I've learned a few things about scale. When you have one cluster, you can manage it by hand. For 2-10 clusters, familiar tools work OK. But for more than 10, it's time for help. In my Kubernetes journey, I've seen stable tools fail under pressure. Even with good intentions, developers can throw a wrench into ideal consistent ops. This keynote covers how to scale to thousands of clusters worldwide. You'll also hear why observability for every cluster change is critical. And don't forget about policy enforcement for cluster declarations. Bonus! A demo of Open Razee (a new open source project) shows how you can control and scale daily cluster updates.
https://sched.co/MRzX
A
A
What
I
would
do
today
in
in
the
few
minutes
that
I
have
is
just
share
some
of
the
lessons
that
we've
learned
running
kubernetes
at
scale
now
for
a
little
bit
of
context,
I'm
going
to
focus
in
on
our
public
cloud
platform
in
IBM
cloud
we
have
a
managed
kubernetes
service
called
IKS
in
that
service,
we're
managing
Hueber
Nettie's
clusters.
On
behalf
of
our
users,
we've
created
hundreds
of
thousands
of
clusters.
Since
the
service
launched
on
average,
we
run
tens
of
thousands
of
active
clusters
on
a
given
day,
we're
running
that
globally.
A
In
six
regions
and
35
data
centers
around
the
world,
we
actually
used
cuber
Nettie's
to
run
kubernetes.
So
we
run
thousands
of
kubera
Nettie's
masters
on
control,
planes
that
are
themselves
running
a
kubernetes
and
we're
running
and
a
really
diverse
set
of
workloads
on
cube,
we're
running
web
and
API
applications,
we're
running
databases
and
data,
warehouses,
we're
running
machine
learning,
blockchain,
IOT
applications
and
high-volume
websites
like
whether
comm
so
a
really
diverse
platform
at
running
at
large
scale
all
around
the
world.
A
Now
what
problems
did
we
face
when
doing
this
and
I
think
the
core
problem
was
how
the
heck
do
we
manage
all
this?
You
know
we
have
a
25-person,
sre
organization,
who's
kind
of
on
the
front
line
running
this
service
every
day.
That
team
actually
has
been
the
same
size
since
we
launched
our
service.
So
how
do
they
deal
with
that
growth
curve?
A
You
know
how
did
they
survive
the
scale
as
more
and
more
users
come
on
board
and
start
creating
kubernetes
clusters
and
I
think
there's
kind
of
two
key
problems
that
we
had
to
tackle.
The
first
was:
how
do
we
help
the
team
manage
the
environment
and
I?
Think
the
key
lesson
here
for
us
was
to
adapt
the
system
not
change
how
the
team
worked
at
IBM.
We
live
every
day
on
slack.
A
All
of
our
kind
of
daily
communication
happens
in
slack,
and
so
one
of
the
core
things
that
we
decided
to
do
was
build
our
operational
system.
In
the
same
place,
our
developers
were
working
everyday,
so
we
built
all
of
our
operational
tools
into
slack.
We
created
a
set
of
bots
so
that
we
can
do
chat
operations.
Everyone
can
talk
with
each
other
in
slack,
but
we
can
also
deploy
updates
deal
with
incidents
at
get
access
to
the
system,
manage
compliance
and
deployment.
A
All
of
that
can
happen
automatically
using
a
set
of
bots
that
we've
integrated
into
slack.
So
that
was
our
first
lesson:
work
where
the
team
is
don't
make
the
team
go
to
the
system.
The
second
big
problem
was:
how
do
we
know
what's
deployed,
and
how
do
we
update
this
system
in
our
kubernetes
service,
on
behalf
of
our
users?
A
When
we
started,
we
had
a
pretty
traditional
Jenkins
base,
CI
CD
structure,
and
we
found
that
as
we
scaled,
that
model
didn't
work.
It
was
too
slow
to
deploy.
It
was
fragile.
The
rules
about
where
you
wanted
to
deploy
things
got
really
complex,
so
we
built
a
different
system
to
help
us
manage
inventory
and
deployment
at
scale
and
that
system
had
kind
of
four
key
ideas
in
it.
A
First,
we
switched
to
a
poll
based
self-updating
cluster
model,
so
instead
of
pushing
changes
into
production,
all
the
clusters
running
in
the
world
could
pull
changes
from
a
central
system
and
update
themselves,
and
that
allowed
us
to
do
it
very
quickly.
We
could
scale
very
easily
and
a
lot
of
stab
more
control
over
what
was
running.
The
second
idea
was
flexible
rule
and
label
based
configuration.
If
you
have
tens
of
thousands
of
clusters,
you
can't
do
anything
on
an
individual
cluster.
A
You
have
to
be
able
to
do
it
on
fleets
of
systems
and
so
using
rules
to
decide
where
things
ran
and
using
labels
within
the
environment
gave
us
the
fine-grained
controls
we
needed
over.
The
system.
Third
feature
flags
four
deployments,
so
we
use
the
idea
feature
flags
to
control
what
we
wanted
running
in
different
environments
and
we
used
a
feature
fight
system.
We
actually
use
launch
Darkly
to
control
what
capabilities
we
would
deploy
in
those
clusters
and
then
finally,
dynamic
inventory.
A
We
needed
the
systems
to
report
for
themselves
what
was
running
in
every
cluster
and
what
capabilities
were
deployed
in
individual
systems
around
the
world.
Let
me
show
you
a
couple
of
examples:
here's
a
picture
of
a
cluster
overview,
so
we
can
look
at
any
cluster
anywhere
in
the
world
and
see
exactly
which
services
are
running
in
that
cluster,
what
versions
of
those
services
and
when
they
were
last
updated?
A
We
can
flip
it
around
as
well
and
look
at
an
individual
micro
service
and
understand
where
in
the
world
is
that
running
which
clusters
is
it
on
and
what
versions
are
deployed
in
the
fleet.
So,
as
we're
doing
updates,
we
can
get
a
quick
view
of
everything,
that's
deployed
in
the
world
and
where
we
might
need
to
make
changes
and
then
finally,
here's
a
view
of
the
rules.
So,
instead
of
pushing
to
all
the
clusters,
we
can
actually
define
rules,
let's
say
in
the
asia-pacific
region.
A
A
We
can
do
updates
very
easily
and
we
thought
this
was
something
that
we
could
share
with
the
world,
and
so
today
we're
launching
a
new
open
source
project
called
Razzie
that
takes
this
continuous
delivery
system
that
we've
build
and
makes
it
available
in
open
source
I,
encourage
you
to
go
to
a
Razzie,
I/o
and
check
it
out
and
that
work
with
us
on
how
we
do
micro
service
continuous
delivery
of
scale.
Thank
you
very
much.