►
From YouTube: Hardening Kubernetes Setups: War Stories from the Trenches of Production - Puja Abbassi, Giant Swarm
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Hardening Kubernetes Setups: War Stories from the Trenches of Production - Puja Abbassi, Giant Swarm
When you run Kubernetes in production and at scale, you encounter many issues both on the infrastructure side as well as in user-space. Some of these issues come with time and increased usage and size of clusters as well as amount of workloads, some might only come once you go global and into regions that have vastly different technology landscapes like China. This talk goes into detail on learnings from concurrently operating 100+ clusters for big enterprises in production on different clouds and data centers around the globe. Over the years we have fixed 100s of post mortems and want to share both operations and development best-practices that can help avoid the issues we ran into. A big focus of this talk is getting towards a hardened and reliable cluster setup and the handling of multi-tenancy in clusters that are used by a multitude of teams.
To Learn More: https://sched.co/GrZT
A
Okay,
let's
start
in
the
back
guess:
there's
some
seats
here
in
a
bit:
no
in
the
next
half
an
hour,
I'm
gonna
talk
a
bit
about
post-mortems
and
what
we
can
learn
from
them
before
I
start
a
bit
about
me.
My
name
is
puia
I'm,
a
developer
advocate
at
Giants
forum
and
a
product
owner
I,
see
myself
kind
of
in
this
triangle
between
the
community
that
is
around
here,
and
online
customers
or
potential
customers
and
end
users
and
our
product.
A
And
what
I'm
going
to
talk
about
today
is
I'm
gonna
start
a
bit
about
the
context
where
we
did
this.
There
made
these
learnings
I'm
gonna
talk
a
bit
about
the
post-mortem
process
and
some
war
stories
that
I
promised
and
then
I'm
gonna
go
into
the
part
where
we
talk
about
hardening
and
how
we
can
avoid,
or
at
least
how
we
can
cope
with
these
kind
of
issues
in
our
clusters.
A
So,
let's
start
with
the
setting,
we
run
around
a
hundred
and
something
clusters
as
giant
swarm,
and
these
run
on
different
clouds.
They
run
in
different
regions
all
around
the
world.
They
run
on
premise,
some
of
them
and
I
list
China
separately
here,
because
it's
kind
of
a
different
beast
could
give
a
full
talk
just
on
China,
but
hit
me
up
afterwards.
If
you
want
to
hear
your
war
stories
from
there
and
then
there
is
a
lot
of
diversity
in
these
clusters
because
they
run
for
different
companies.
Darry.
A
These
companies
come
from
different
industries
from
like
I
know,
from
insurance
or
banking
to
retail
and
inside
these
companies.
There's
different
kinds
of
users
is
like
end-user
developer:
the
rights
code.
There
is
an
admin.
There
is
architects,
there's
all
kinds
of
end-users
that
using
these
clusters,
or
using
or
at
least,
are
somehow
in
touch
with
the
kubernetes,
and
then
there
is
different
use
cases.
We
have
like
IOT
use
cases,
there's
machine
learning,
use
cases,
there's
just
a
very
big
ecommerce
use
cases
so
handling
this.
A
We
go
and
be
opted
for
for
more
freedom,
so
you
could
go
and
say
you
lock
everything
down
and
we
don't
know
anything,
but
that's
not
how
we
work
it's
I
kind
of
believe
in
trusting
your
users
and
I
think
us
as
operators
of
kubernetes
and
infrastructures.
We
should
rather
up
for
freedom
and
give
developers
the
freedom
that
they
wanted
to
have
with
all
these
new
technologies
that
we
built
in
the
last
five
six
years.
So
what
you
need
to
do
once
you
give
users
a
lot
of
freedom?
A
Is
you
need
to
educate
them
a
lot?
So
you
need
to
talk
a
lot.
You
need
to
write
a
lot
of
documentation
and
on
the
other
side,
you
can't
avoid
misuse
you.
You
can't
avoid
it
in
this
configuration,
so
you
need
to
kind
of
harden
up
annette's
the
post-mortems
come
in
failures,
happen
and
they
happen
a
lot.
A
This
post-motor
philosophy
I'm,
not
sure
if
it's
the
original
kind
of
source
but
Google
puts
it
pretty
well.
The
goal
is
to
document
incidents
to
find
all
the
root
causes
that
contribute
to
this
incident
and
then
basically
be
able
to
fix
it
or
at
least
avoid
that
it
spreads
too
far,
and
how
does
it
work
a
transform?
It's
basically,
we
gather
issues,
we
fix
them
in
code
and
we
roll
it
out
to
production
and
what
helps
us.
There
is
reused
issue
templates
in
github
you're,
not
on
JIRA
or
something
we
just
use.
A
The
postman
comes
in,
it
gets
to
a
team,
and
the
product
owner
of
the
team
assigns
it
or
talks
to
their
two
team
and
assigns
it
to
someone
and
whatever,
as
a
as
a
reason,
was
mentioning
this
guy
in
the
end
or
this
girl,
they
should
be
actually
unhappy
to
get
this
postmortem
because
I.
Sometimes
it's
really
frustrating
right.
You're
like
we're
a
bit
holding
on
this
issue.
You
can't
find
the
root
cause.
You
can't
find
a
fix,
it's
really
hard,
but
sometimes
it's
also
really
interesting,
because
you
find
you
have
stories
for
afterwards.
A
You
have
stories
to
share
like
I'm,
going
to
share
here
and
you
feel
kind
of
like
a
detective
like
that's,
that's,
that's
a
fun
part
of
it
and
we
see
a
lot
of
issues.
This
is
roughly
a
year
because
in
November
last
year
we
added
this
label
and
our
github
post-mortem.
So
roughly
a
year
of
post
mortems
for
us
means
more
than
500
of
them.
I
mean
some
of
them
are
really
small,
might
be
like
some
UX
small
cleric
or
like
locks
filling
up
a
disk,
and
it's
fixable
really
fast.
A
Some
of
them
might
be
a
bit
harder
and
I'm
gonna
share
some
of
them
here.
If
you
want
to
hear
more,
just
hit
me
up
afterwards
and
I'm-
maybe
gonna
blog
about
this
and
add
more
to
this,
but
let's
see
into
some
of
them
and
what
we
can
learn
from
them
and
I'm,
not
sure.
If
some
of
you
ran
into
this
issue,
111
kubernetes
had
a
memory
leak
and
was
only
fixed
in
111
for
and
we
had
it
in
production
since
Wallen
having
one
for
some
customers.
A
So
until
the
fix
landed
upstream
and
could
be
released,
we
had
to
actually
patch
our
own
hypercube
image
and
roll
it
out
to
production
for
some
customers.
So
that
is
something
that
you
need
to
be
comfortable
with,
as
operators
and
sres
for
kubernetes
and
for
cloud
native
technologies,
sometimes,
and
especially,
if
you're
running
and
production,
you
need
to
be
ready
to
get
into
the
code
ready
to
fix
upstream,
even
if
you
want
to
or
in
and
if
you
need
to
so.
A
This
is
something
that
you
should
be
prepared
for,
and
then
I
mean,
if
you
run
ingress
controllers,
you've
maybe
seen
that
they
can
be
issues,
especially
if
you
run
complex
software
like
nginx.
We
usually
run
and
you're
next
we're
very
happy
with
it,
but
in
certain
versions
and
older
versions,
for
example,
a
faulty
ingress
like
having
a
wrong
certificate
in
your
ingress
would
break
the
whole
ingress
controller,
and
that
means
like
even
if
you're
monitoring
front
end,
has
a
runs
certificate.
A
It
could
break
your
production
website
because
it's
running
on
the
same,
invest
controller,
usually
right
and
then
remember
what
I
said
in
the
beginning.
We
have
lots
of
teams
working
on
these
classes
and
like,
if
you
give
them
lots
of
freedom.
You
get
lots
of
issues
all
right,
so
you
need
to
find
a
way
how
to
harden
against
this,
and
in
in
this
case
it
will
basically
update
to
a
newer
version
that
is
better
like
or
less
prone
towards,
miss
configuration,
and
this
is
like
0,
15,
0
17
upwards
of
nginx.
A
A
We
love
the
Calico
people
and
they
fix
this
issue
actually,
but
there
was
a
for
a
very
short
time.
There
was
this
issue
where
IP
IP
tunnels
were
just
closing
after
a
while,
so
we
built
this
very
small
hacky
tool,
which
was
basically
pinging,
the
full
mesh
of
IP
IP
tunnels,
all
the
time
for
them
to
be
kept
alive.
All
right-
and
this
is
something
that
sres
are.
Basically
that's-
that's
a
day-to-day.
A
A
If
you
cannot
fix
the
issue,
so
you
need
to
fix
it
somehow
and
I
don't
know
if,
if
you
guys
run
no
test
load
tests,
load
tests
are
funny,
especially
if
you're
not
the
person
doing
them,
and
if
you
knew-
and
this
was
like
two
years
ago,
where
like
kubernetes
at
this
customer
was
at
least
quite
new,
and
it
was
I
got
it's
new
software
like
calico
and
cute
proxy
and
English
controllers
and
the
load
test
went
wrong.
It
was
like
three
hundred
thousand
requests.
A
A
second
should
have
been
100
thousand
and
it
broke
the
whole
cluster
like
it
like
traffic
went
down,
went
away
and
everything
yeah,
it
must
be.
Your
calico
must
be
your
new
software.
There
turns
out
was
ec2,
they
had
very
few
machines.
It
was
very
few
machines
that
were
running
ingress
controllers
and
there
were
small
machines
too.
A
So
network
would
like
cut
off
at
seven
gigabit,
so
the
solution
was
basically
just
scale
it
up,
but
sometimes-
and
in
this
case
the
metrics
were
somehow
lying,
because
if
you
run
calico
and
pots
and
containers,
you
would
run
it
an
host
Network.
And
if
you
look
at
the
metrics
for
this
container,
you
see
the
host
metrics
of
the
network
right.
So
you
think,
okay,
this,
if
you
see
the
metrics,
are
cutting
off
there.
So
you
think
oh,
it
must
be
this
container,
but
it
was
not
so.
A
This
is
just
like
a
very
small
glimpse
into
it.
I've
had
time
I
would
like
share
a
lot
more
with
you,
but
let's
get
to
the
gist
of
it
and
try
to
find
like
how
what
can
we
do
against
it
and
how
can
we
like
in
general,
try
to
be
better
and
and
like
better
prepared
for
these
issues
and
I
tried
to
then
go
into
the
data
and
I'm?
A
Not
super
data-driven,
but
I
went
just
into
our
post
mortems
head
look
and
looked
what
are
the
hot
spots
and
one
of
the
hot
spots
is,
and
we
actually
have
that
even
down
in
the
in
the
contract
that
you
need
to
upgrade.
But
old
versions
are
usually
more
prone
to
issues
and
then
the
other
issue
is
ingress
controllers
or
ingress
by
itself,
because
I
mean
that's
where
traffic
comes
in,
that's
where
uptime
is
kind
of
happening.
A
So
there's
a
lot
of
issues
around
that
the
networking
DNS
I
mean
most
people
know
there's
some
issues
around
networking
somehow
happening
all
the
time
and
then
one
thing
that
is
I
guess
pretty
common
around
kubernetes,
especially
if
you
have
newer
users
and
users
who
are
like
not
very
deep
into
the
matter.
Yet
is
resource
pressure
and
I'm
gonna
talk
about
like
each
of
these
in
detail
in
a
bit
and
one
the
last
one
is
multi-tenancy
basically
and
there's
certain
issues
that
are
just
happening.
A
If
you
have
a
lot
of
them,
tenants
lot
of
teams,
and
so
let's
get
into
the
meat
old
versions,
basically
by
definitions,
if
you
see
like
change,
locks
of
new
components-
and
it's
not
only
kubernetes
right,
it's
also
all
your
critical
components.
The
the
dns
that
you're
running,
maybe
you're,
running
issue.
All
these
things,
usually
change
locks,
have
a
lot
of
issues
fixed
right,
so
running.
A
A
newer
issue
is,
is
often
recommended,
and
especially
if
you
have
a
good
security
department
that
will
force
you
to
run
certain
versions
and
then
just
look
at
CVS
and
where
they
are
fixed,
like
in
nginx
ingress.
For
example,
there
was
CVS,
they
were
fixed
in
0:21
if
you
were
stuck
on
a
very
old
version
and
we're
just
lazy,
because
everything
works
and
you
didn't
want
to
upgrade,
because
why
upgrade
the
working
system
right?
But
now
you
need
to
upgrade
to
the
year
21
just
to
fix
the
CVE.
A
So
you
have
a
lot
of
route
like
a
lot
of
things
to
catch
up
on,
there's
may
be
configuration
changes,
there
might
be
a
flag
change.
There
might
be
changes
in
the
annotations
that
you
need
to
do.
It
is
this
lot
of
work
and
if
you
need
to
roll
it
out
through
a
day
like
if
your
security
department
heard
someone
says
hey,
you
need
to
fix
it
now
and
we're
in
production.
It's
like
Black
Friday.
A
We
need
to
do
this,
then
you
have
a
lot
of
work
and
you
need
to
force
every
all
the
teams
to
do
all-nighters
and
work
all
weekend
just
get
to
the
new
version.
That's
that
you
need
to
test
upgrades
extensively.
Even
if
there
is
no
breaking
change,
everything
should
be
fine.
There
might
be
some
configuration
that
you
are
doing,
that
you
have
that
might
not
be
tested.
A
You
should
be
able
to
spin
something
up
and
test
it
around
before
you
do
the
upgrade
in
the
actual
production
environments
and
then
try
to
automate
your
processes
automate
the
upgrades,
or
at
least
have
some
process
where
you
say
just
maybe
subscribe
and
slack
to
the
github
release
feet
and
then
have
a
team
responsible
for
each
component.
Have
someone
whose
things
okay?
This
is
my
job
to
keep
this
up-to-date
to
keep
at
least
very
close
to
the
latest.
I
mean
you
don't
need
to
run
zero.
What
is
it
dot?
A
A
The
other
thing
is
ingress
and
we
do
like
the
nginx
ingress
and
our
customers
like
it
a
lot,
because
it's
very
configurable
there's
lots
of
options.
It's
everyone
knows
nginx,
and
it
has
a
lot
of
power,
but
it's
kind
of
like
with
great
respect.
Our
comes
great
responsibility.
You
can
break
things
right
and
if
you
don't
want
to
take
away
all
the
features
from
your
from
your
users,
you
need
to,
on
the
one
side,
run
newer
versions,
and
sometimes
you
need
to
think
about.
A
Maybe
you
want
to
separate
controllers
like
in
the
case
where
the
Prometheus
cert
was
taking
down
production.
You
could
have
run
an
engine
English
controller,
just
for
your
workloads
for
your
production,
your
user
facing
workloads
and
then
one
for
admin
right
this
way
you
cannot
influence
each
other
that
much
and
then
you
should
do,
and
this
is
not
only
for
ingress.
This
is
just
in
general.
You
should
do
let
load
and
failover
testing,
and
this
failover
testing
I
mean
try
what
happens
if
I
take
half
the
ingress
controller
cell?
A
What
happens
if
I
take
a
full
node
down
and
try
to
see
also
like
what
happens
when
I
upgrade
in
production,
like
upgrade,
my
parts
do
I,
have
like
lost
connections
in
between
them.
This
way
you
find,
if
you're,
really
graceful.
Also
in
your
software,
if
your
settings
for
the
ingress
controller
or
for
some
load
balancers
are
graceful
enough
to
really
have
zero
to
zero
downtime.
A
At
the
same
time,
if
you
have
that,
then
try
that
most
English
controllers
won't
work
for
you
just
use
a
load
balancer
directly
for
that
service
might
be
a
bit
more
expensive,
but
it's
worth
it
right.
On
the
other
side,
networking
DNS
there
is
no
like
golden
rule
there.
The
main
thing
is:
do
monitor
and
alert
on
those
things
you
might
see
issues
before
they
actually
happen,
and
this
is
like
really
important
to
see
to
have
like
preventive
monitoring
and
alerting
on
things.
A
A
A
If
you
create
pressure,
what
goes
down
first,
if
your
cubelet
goes
down
under
pressure
or
your
proxy,
then
you
have
a
really
big
issue
right,
because
then
you
have
a
full
note
going
away
and
then
that
just
like
propagates
through
your
cluster
and
you
take
the
whole
cluster
that
so
you
need
to
protect
these
and
you
don't
also
need
to
protect
your
critical
adults.
Yet
your
dns,
your
networking,
if
those
get
killed
before
your
word
loads,
you
haven't
protected
your
workloads.
You
it's
really
important
in
the
last
button
least
in
the
multi-tenancy
use
case.
A
It's
not
only
about
like
isolating
on
namespaces
in
our
back
and
I've.
Seen
these
things
where
it's
like,
we
have
our
back
and
PSP
turned
on,
but
every
one
is
a
cluster
admin
wasn't
really
useful
right,
but
it
happens
and
they're,
like
our
security
department,
is
ok
with
that.
I
wouldn't
do
that,
and
especially
even
if
you
trust
all
your
users,
and
you
should
don't
do
it
just
because
you
might
run
into
issues
just
like
everyone
is
tired
ones
or
runs
the
wrong
command.
A
No
manual
ops
try
to
have
a
pipeline
into
it,
and
everyone
uses
the
pipeline
at
least
in
production,
and
then
you
need
maybe
a
few
sree
accounts
to
get
in
in
the
worse
case.
But
then
you
don't
need
to
do
that.
Many,
like
all
the
are
back
users
and
configuration
and
production
anymore,
and
you
have
a
lot
less
issues
and
you
will
have
reviews
when
you
do
get
ups.
You
can
really
review
every
change
that
you
do
and
you
don't
go
in
and
change
something
manually,
I
think
Kelsey
tweeted
out.
A
Yes,
no
no
last
week
about
cube
CTL
being
the
new
SSH
and
I'm
pretty
much
on
that
in
prod,
at
least
because
if
you
cube
the
CTL
into
your
product,
lustre
you're
doing
something
wrong.
Usually
last
person
needs
some
best
practices
here
do
have
a
good
monitoring
and
alerting
setup
promises
and
alert
manager
are
great,
have
something
like
pager
duty
or
ops
Genie
that
wakes
people
up.
A
If
there
is
something
wrong,
if
you
don't
fix
it
during
the
night,
even
if
important
for
you
make
might
not
might
only
be
office
hours,
if
you
leave
it
overnight
in
the
morning,
you
were
gonna
have
a
bigger
issue
and
that
might
be
worse
to
to
handle.
Then
just
fixing
it
during
the
night
in
five
minutes,
because
it's
gonna
take
an
hour.
A
Maybe
then,
or
it's
gonna
be
even
worse
and
then,
while
you're
debugging
and
you
need
to
debug
a
lot,
maybe
have
a
good
logging
setup
have
maybe
even
tracing
if
you
can,
and
sometimes
you
see
these
issues,
they
occur
and
they
lead
to
downtime.
But
maybe
you
you
were
having
these
issues
even
the
last
months,
but
you
just
didn't
see
them
because
they
didn't
like
there
was
some
something
in
the
environment
that
didn't
make
them
really
influence
your
your
production
workloads,
for
example.
A
It
may
be
we're
just
influencing
some
other
things
and
you
didn't
see
them,
but
at
least,
if
you
have
some
historical
data
and
you're
monitoring
and
logging,
you
will
find
patterns
better
and
find
issues
that
you
might
not
have
seen
before
and
as
I
said,
fix
issues
fast.
If
you
can
I
mean
some
issues,
you
just
take
a
while
and
need
to
go
upstream
and
to
ask
people
and
find
find
solutions,
but
in
general
try
to
be
as
fast
as
possible
and
give
give
these
issues
just
priority
and
then
for
me
personally,
very
important.
A
You
need
to
educate
users
and
really
talk
to
everyone
and
see
that
people
understand
what
they're
doing,
because,
especially
in
if
you
have
a
lot
of
teams.
We
had
this
issue
where
there
was
20
teams
or
10
teams
on
a
cluster,
and
there
was
an
inverse
controller
and
one
team
decided
to
have
a
second
ingress
controller
in
that
cluster.
But
they
didn't
tell
all
the
other
teams.
So
there
was
no
ingress
classes
on
the
ingress
objects,
it
doesn't
work.
A
There
should
have
been
communication,
but
sometimes
there
is
not,
and
you
need
to
foster
that
communication
and
if
you
see
things
then
try
to
talk
to
the
users
and
see
if
they
know
about
it,
because
sometimes
even
like
management
or
operations,
person
now
might
not
know
about
some
things
that
are
happening
in
there,
because
they
didn't
like
look
every
every
hour
and
every
minute
into
your
cluster
and
then
have
a
process
for
post-mortems.
It's
super
important.
If
you
don't
have
a
process,
you're
not
gonna,
stick
to
it.
You
need
to
write
something
down.
A
You
don't
need
to
use
our
process
or
something
that
is
in
the
Google
sre
book.
You
need
to
find
your
own
process
with
your
own
tooling,
but
have
a
process
and
try
to
stick
to
it
and
then,
most
importantly-
and
this
is
something
that
is-
that
is
something
a
lot
of
people
have
backups
and
use.
I,
don't
know
happy,
you
are
maybe
and
all
these
things
to
recover,
and
but
they
don't
train
the
recovery
and
you
should
train.
You
should
see
how
fast
can
I
get
back
because
we
really
like
in
micro-services
world.
A
A
That
is
not
in
my
cluster
and
that's
super
important
and
stand
on
the
shoulder
of
giants.
Like
there's
a
lot
of
people
out
there
I
mean
from
data
doc
to
open
AI,
two-bit
mix
that
are
sharing
what
they're
finding
at
scale,
sometimes
in
their
clusters.
There's
a
lot
of
people
on
this
conference
and
on
other
conferences,
sharing
their
experiences
in
blog
posts
and
talks.
A
A
You
may
be
point
you
to
the
right
people
point
you
to
the
right
direction,
where
you
should
look
what
kind
of
logs
and
maybe
will
help
you
afterwards,
even
even
more
in
Sector,
overflow
or
somewhere,
and
that's
pretty
important
too
for
me
to
be
able
to
have
this
community
and
stand
on
the
shoulder
of
this
community
to
use
it
as
a
resource
for
yourself.
This
is
like
something.
If
you
take
one
thing
out
of
this.
All
of
this
talk
is
you
don't
need
to
reinvent
the
wheel?
A
You
can
stand
on
the
shoulder
of
giants
here
and
before
I,
open
up
towards
questions.
I
want
to
thank
everyone
who
gave
me
feedback
to
this
talk,
and
especially
our
team.
A
giant
swarm,
it's
a
crazy
Bunch,
it
had
a
lot
of
people
and,
if
it
weren't
for
them,
I
wouldn't
be
here.
I
couldn't
share
the
war
stories
because,
most
of
the
time,
I'm
not
the
one
who
wakes
up
at
3
a.m.