►
From YouTube: What We Learned from Reading 100+ Kubernetes Post-Mortems - Noaa Barki & Shimon Tolts, Datree
Description
Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
What We Learned from Reading 100+ Kubernetes Post-Mortems - Noaa Barki & Shimon Tolts, Datree
A smart person learns from their own mistakes, but a truly wise person learns from the mistakes of others. When launching our product, we wanted to learn as much as possible about typical pains in our ecosystem, and did so by reviewing many post-mortems (100+!) to discover the recurring patterns, anti-patterns, and root causes of typical outages in Kubernetes-based systems. In this talk we have aggregated for you the insights we gathered, and in particular will review the most obvious DON'Ts and some less obvious ones, that may help you prevent your next production outage by learning from others' real world (horror) stories.
A
B
Yeah
yeah,
but
first
I
think
it's
the
first
time
that
we
meet
so
hi
everyone.
My
name
is
noah
noah
barki.
I
am
a
full
stock
developer
for
more
than
five
years
and
I
am
also
a
tech
writer
and
one
of
the
leaders
of
github
israel
community,
which
is
the
largest
github
community
in
the
whole
universe,
universe.
A
Universe
so
hi
everyone,
I'm
shimon,
I'm
one
of
the
co-founders
and
the
ceo
of
the
tree,
I'm
also
an
aws
community
hero,
but
we're
here
in
cncf.
So
I'm
one
of
the
core
organizers
of
cncf
tel
aviv
here
out
from
israel,
we
have
a
vibrant
community
and
if
you
happen
to
stop
by,
you
should
definitely
come
and
see
the
cncf
in
tel
aviv.
A
A
Is
we
help
companies
prevent
misconfigurations
from
reaching
production
in
kubernetes
environments?
So
this
means
we
have
an
open
source
cli
that
actually
can
run
on
your
laptop
or
in
your
ci.
Cd
against
kubernetes
manifests
and
help
charts,
and
it
can
detect
misconfigurations
such
as
missing
a
cpu
limit
or
a
liveness
probe.
B
A
B
A
A
A
B
Try
more
than
4
000,
more
pods
that
were
constantly
restarting
yeah.
That's
what
happened
to
them!
They
had
one
failing
crown
job
that
were
constantly
restarting
new
parts
and
not
only
that
it
immediately
took
their
cluster
down,
but
it
also
cost
them
a
lot
of
money
because
their
api
server
accumulated
thousands
of
cpu
wasting.
B
B
Yes,
yes,
just
because
kubernetes
allows
you
to
deploy
a
specific
resource
with
a
specific
configuration.
It
doesn't
mean
that
this
is
what
you
should
do
and
from
my
experience
most
of
the
in
most
of
the
cases,
this
is
not
your
con.
This
is
not
the
configuration
that
you
should
work
with,
but
let's
move
forward
to
the
next
one.
B
B
A
B
Yeah
yeah,
and
let
me
tell
you
a
bit
about
the
landows:
orlando
is
an
online
fashion
company
with
over
6
000
in
place.
So
again
you
can
only
imagine
how
much
money
it
costs
to
them,
but
the
real
lesson
that
we
learn
here
is
to
always
make
sure
that
you
define
clearly
define
the
policies
in
your
organization
and
the
best
practices
in
in
your
organization.
It's
super
super
important
and
people
think
that
having
a
correct
yaml
structure
is
really
basic,
but
it's
not.
B
B
A
B
B
Of
traffic
and
that's
what
happened
to
also
target
actually
was
their
first
incident
when
they
started
to
use
kubernetes.
So,
especially
specifically,
what
happened
to
to
target
is
that
one
of
the
their
developers
they
put
a
star
in
the
ingress.
Nobody
was
watching
and
immediately
took
their
entire
cluster
down
and.
A
B
Yeah
yeah
exactly
and
the
real
lesson
that
we
learned
from
target
is
to
delegate
the
knowledge.
It's
really
important
when
you
start
to
use
kubernetes
in
your
organization
to
delegate
the
knowledge
to
the
entire,
the
entire
organization.
You
have
developers
team
and
you
have
devops
team
and
you
have
many
many
department
in
your
organization
and
you
want
to
make
sure
that
everyone
understand
how
to
work
with
kubernetes
and
what
you
should
do
and
what
you
should
not
do.
A
A
B
A
B
And
you
can
tell
that
to
blue
matador.
Blue
matador
had
one
pod
that
served
third
party
application
and
apparently
that
application
containers
were
memo
hawks.
So
one
day
one
of
their
devops
engineer
noticed
out
of
memory
issues.
He
looked
a
little
deeper
and
he
found
out
that
this
container
didn't
have
any
memory
which
immediately
took
the
cluster
down
out
of
a
memory
lick,
because
those
containers
took
all
the
memory
in
the
production,
node
and.
A
Well,
it
makes
sense
you
know,
because
when
you
use
third-party
software,
it's
not
even
from
your
company,
so
you
don't
know
how
it's
going
to
behave
like
I
remember
we
were
using
like
a
rabbit
mq,
which
is
like
a
queuing
service
yeah
and
the
default
behavior
of
the
queue
is
to
accumulate
as
much
memory
as
it
can
in
order
for
it
to
be
ready
to
serve
queuing.
So
if
you
don't
limit
this
container,
it
will
just
automatically
take
everything
and
there
are
like
some
java
applications
and
stuff
that
are
very,
very
like
memory
heavy.
B
A
B
That
it
can
really
happen
to
anyone,
because
bloomerador
is
a
small
startup
company.
It's
not
a
big
company,
it's
not
orlando.
It's
not
6000
employees
and,
while
I
kept
reading,
I
found
out
that
it
can
really
happen
to
anyone
and
when
I
say
anyone
I
mean
anyone:
google,
spotify,
airbnb
sky
scanner,
datadog
toyota,
so
many
companies
that
it
happened
to
them.
It's
kubernetes,
it's
not
simple
at
all,
and
it
can
really
happen
to
anyone.
A
Yeah
well,
as
we
said
like
it,
kubernetes
brings
a
lot
of
flexibility,
but
with
that
you
know
you
sometimes
you
need
some
sort
of
like
guardrails
or
something
to
help.
You
do
the
right
thing,
because
if
you
have
a
lot
of
options
and
a
lot
of
things
you
can
do,
and
especially
if
you're,
trying
to
delegate
infrastructures
code
responsibilities
to
developers
and
engineers
and
not
to
only
have
like
the
centralized,
ops
or
devops
team,
like
babysit
all
the
developers,
you
know
and
they
go
over
like
be
human
debuggers
for
yammels.
A
B
I
completely
agree
with
you,
and
I
want
to
I
want
to
add
something
for
the
favor
in
developers
in
the
audience,
because
as
a
developer,
when
it
comes
to
kubernetes
when
it
comes
to
devops,
it's
it's
not
my
field,
it's
something
that
I
might
be
afraid
of,
and
it's
not
it's
not
my
pipeline
because
I'm
a
developer,
I'm
I'm
writing
my
code
and
I'm
shipping
it
to
production.
And
yes,
I
want
to
make
sure
that
I
would
never
never
harm
production
but
doing
stuff
that
related
to
devops
to
infrastructure
code.
B
A
B
It's
a
process,
it's
not
something
that
is
done
in
one
day.
It's
not
one
take
and
developers
and
devops
are
speaking
entirely
different
languages.
I
am
at
the
devil,
the
development
side
of
the
pipeline,
and
you
are
in
the
production
we
wake
up
every
morning
with
different
goals.
I
want
to
be
as
I
as
I
like
to
call
it
the
best
feature
machine.
I
would
like
to
deliver
the
my
features
way
before
my
product
manager
even
thought
about
it
and.
A
B
B
A
B
And
I
want
to
end
to
define
your
policies
that
what
is
a
policy,
because
as
a
developer,
when
I
thought
about
policy,
I
I
said-
and
I
thought
what
is
policy
in
my
area
in
my
field
policy-
is
basically
what
I
need
to
do
to
feel
confident
in
order
to
ship
my
code
to
production.
So
what
do
I
do?
I
write
tests.
I
write
integration
tests.
I
do
qa
tests.
Basically
it's
it's
a
bunch
of
tests,
but.
B
But,
but
that's
what
I
do,
I
obey
clean
code
standards
and
I
have
my
own
best
practices
and
I
always
make
sure
that
I
read
for
I
don't
know
if
I
use
golang
what
are
the
conventions
and
what
are
the
best
practices
when
you're
writing
golang,
and
this
is
what
I
do
in
order
to
feel
confident
with
my
code.
So
policies
is
something
that
is
as
a
developer
and
as
a
devops.
We
we
all
have.
We
all
share
the
policies.
B
I
think
that,
as
developer,
my
policies
are
when
I,
when
I
write
my
unit
tests-
and
I
add
them
to
the
ci
cd,
so
I'm
this
is
the
part
of
the
pipeline
and
we
all
in
the
pipeline,
and
I
want
to
add
them
in
the
cicd
pipeline,
and
I
make
sure
that
before
I
push
my
changes,
I
run
npm
tests.
Yeah
everything
is
okay,
good
good.
I
feel
confident
and
confident
with
my
with
my
code
and
I
think
that
policies
this
is
the
key.
B
B
I
believe
that
devops
engineer
would
like
to
make
those
policies
on
every
change,
so
we
want
to
validate
that,
ideally
through
automated
checks
like
having
it
on
in
within
your
cicd
pipeline
or
even
as
a
pre-commit
hook
as
a
local,
a
testing
in
your
local
dev
environment.
This
is
this
is
practically
what
you
do,
what
you
already
do,
so,
what's
the
difference
between
kubernetes
resources
than
the
rest
of
your
code,.
A
Absolutely
and
I
think
that
seamlessly
integrating
within
the
development
workflow,
whether
it's
a
vs
code
extension
or
a
cli
utility
or
a
pre-commit
hook
or
just
having
it,
you
know
enforced
and
mandatory
yeah.
It's
like
it's
a
hard
word
to
stay
in
force,
but
in
a
way
you
know
security
and
stability,
and
there
are
some
tests
that
are
like
enforced.
So
like
every
job,
and
maybe
you
have
like
500
git
repositories.
B
B
A
Yeah,
so
this
is
a
very
important
step,
which
I
think
it's
you
know.
It's
absolutely
crucial
to
be
able
to
dynamically,
adjust
your
policies
because,
okay,
so
let's
say
we
sit
down,
we
define
which
policies
we
want.
Maybe
we'll
show
you
some
tools
that
you
can
use
in
order
to
do
it
now,
but
maybe
you
put
it
in,
but
then
you
run
for
one
week
two
weeks
three
weeks
you
might
have
a
new
policy,
but
then
what
are
you
going
to
do
now
going
to
deploy
a
policy
change
to
500
git
repos?
A
So
it
becomes.
You
know
it's
a
it's
like
sort
of
a
burden
now,
because
in
order
to
to
change
it,
I
need
a
way
to
have
like
a
centralized
management
place
where,
like
all
of
my
repos,
we
pulled
the
policies
from
a
centralized
location
and
then
execute
them.
So
I
can
modify
the
policies
in
one
place
and
it
will
propagate
to
all
of
my
workloads
on
another
place
and
secondly,
you
want
a
place
to
that.
That
gives
you
the
ability
to
monitor
which
tests
are
running
which
tests
are
failing.
A
B
A
B
I
want
you
to
start
thinking
about
where
would
be
the
most
suitable
place
to
you
to
your
organization
to
put
an
application
that
will
validate
your
kubernetes
resources
in
your
pipeline,
because,
where
you
will
choose
to
put
that
application,
it
might
affect
your
entire
organization
makes
sense.
So,
let's
start
with
gatekeeper,
gatekeeper
is
a
policy
controller
for
kubernetes.
B
A
Meet
the
criteria
yeah,
it
will
kick
you
off,
yeah,
okay,
so,
but
this
is
like
at
the
end
right
it's
in
the
production,
so
it's
not
shift
left
at
all.
It's
like.
I
already
developed
my
code.
It
was
built
and
everything
and
it's
shipped-
and
it's
like
the
last
mile,
but
maybe
I
want
to
do
you
know
if
you
want
to
check
it
prior
to
that.
A
B
It
was
really
really
interesting
because
it
was
the
first
time
that
I
ever
used
like
local
testing
like
this
is
my
unit
testing
for
kubernetes
resources,
and
I
remember
that
yeah.
Okay,
I
understand
what
unit
testing
are
okay,
I
started
to
love
kubernetes
it's
fun
and
I
think
it's
a
great
power
when,
when
it
comes
to
using
confidence,
because,
as
I
said
before,
this
is
this
is
the
key
point
to
the
community,
the
communication
between
devops
engineers
and
developers-
and
I
think
it's
really
important.
B
A
A
Yeah
cool,
so
those
are
the
three
main
tools
that
you
wanted
to
talk
about.
You
might
ask
yourself
which
one
is
for
me:
maybe
all
of
them
are
for
me.
What
are
the
differences
so
we're
like
at
the
end
of
our
talk,
the
main
difference
is,
first
of
all,
the
first
one
is
define
your
policies,
so
the
tree
comes
with
dozens
of
prebuilt
rules
like
from
all
those
postmortems
that
we
talked
to
you
about.
A
Of
course,
if
you
do
have
a
list
or
want
you
can
use
conf
test
or
gatekeeper
to
do
it.
Secondly,
you
want
to
integrate
your
policies
inside
your
organization.
So,
as
we
said,
the
tree
comes
with
a
lot
of
plugins
and
web
hooks
and
you
know
to
github
actions
and
any
ci
cd
provider
in
pre-commit
hooks,
and
you
can
run
it
in
your
id
and
everywhere
you
want
and,
of
course,
in
your
ci
cd,
while
other
ones
require
a
manual
configuration
and
third,
which
I
think
is
very
important.
A
You
know
for
organizations,
because
you
know
I
really
experienced
this
in
my
previous
role,
where
we
had
400
engineers
and
it's
really
really
hard
to
control.
You
know
this
amount
of
engineers,
thousands
of
git
repos,
and
I
think
that
the
final
part
is
the
control
review
and
monitor
which,
with
the
tree,
we
provide
a
centralized
policy
management
solution.
A
So
I
can
go
and
define
my
policies
in
one
place
and
it
will
dynamically
adjust
a
in
run
like
you,
don't
need
to
change
anything
on
your
ci
cd
on
your
like
you
don't
need
to
modify
the
binary
because
the
policies
are
streamed
into
the
ci
into
the
cli
solution
and
then
they
are
applied.
So
it's
really
really
suitable
for
organizations
that
want
to
use
it.
B
And
the
takeaways
that
I
we
want
you
to
to
take
from
this
session
is
to
first
to
understand
how
important
our
policies
and
really
how
you
should
use
it
and
how
you
should
why
you
should
define
it
in
your
organization.
And
if
we
talk
about
policies,
then
to
never
trust
the
default.
Configuration
define
the
policies
in
your
organization,
as
we
talked
before,
delegate
the
knowledge
to
the
rest
of
the
team
and
across
your
organization,
and
it
can
really
happen
to
anyone.
A
Thank
you
very
much.
It
will
stop
sharing
now
and
we
really
want
to
thank
you
very
much
for
attending
our
talk,
feel
free
to
reach
out,
and
you
know,
come
talk
to
us
open,
pull
requests
on
github
and
we're
available
on
linkedin
and
anywhere
and
we're
looking
forward
to
hearing
from
you
and
we're
working
very
closely.
You
can
open
issues
and
we're
adding
support
for
more
and
more
platforms,
and
it's
been
a
pleasure.
No
thank
you
very
much.