►
From YouTube: We Didn’t Start the Fire: Communication Breakdowns and How to Preven... Ian Coldwater & Kat Cosgrove
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
We Didn’t Start the Fire: Communication Breakdowns and How to Prevent Them - Ian Coldwater, Twilio & Kat Cosgrove, JFrog
When the Kubernetes project was smaller and fewer people used it, core maintainers could assume shared context, and we didn't have to think about communication as much. We've grown a lot since, but we haven't always adjusted these assumptions. We don't all know each other anymore, and can no longer rely on everyone sharing the same context or concerns. With more growth, communication becomes more critical.
Design decisions and the way we communicate about them are affected by the assumptions we make about each other. If we aren't careful, problems can arise for everyone. The Dockershim deprecation conflagration is one example of how this can go wrong, but it wasn't the first or the last communication breakdown of its kind. Let's look at this history together, learn from it, and figure out how we can prevent such breakdowns in the future.
It's on all of us to work together to address this. We can do better when we come together, communicate more, and check our assumptions. We've got this.
A
B
Once
upon
a
time
way
back
when
you
could
count
the
cards
on
the
landscape,
the
kubernetes
project
was
smaller
and
newer
and
communication
wasn't
really
a
thing
that
we
had
to
think
about
as
much
a
lot
of
people
knew
each
other.
People
were
more
likely
to
come
from
similar
backgrounds
and
assumptions
about
shared
context
between
maintainers
and
users
were
more
likely
to
hold.
True
kubernetes
is
a
much
bigger
community.
Now,
though,
many
of
us
don't
share
the
same
context
anymore
or
even
the
same
concerns
our
backgrounds
and
our
background.
Knowledge
can
vary
pretty
widely.
A
We've
kept
learning
this
lesson,
or
at
least
we've
kept
needing
to
communicating
changes
to
users
without
understanding
each
other's
needs
and
concerns
has
had
a
series
of
unexpected
results.
We've
kept
making
assumptions
and,
in
turn
messages
keep
being
received
in
ways.
We
might
not
have
thought
they
would.
B
B
B
A
So
it
was
announced
a
couple
of
versions
before
this
would
happen
that
in
kubernetes
1.16
for
the
first
time
there
would
be
breaking
changes
for
users.
Some
deprecated
apis
were
going
to
be
removed,
endpoints
that
had
ended
in
v1
beta
1.
Some
of
them
were
going
to
be
changing
to
v1
and
ones
that
might
have
started
with
extensions
before
we're
now
going
to
be
starting
with
things
like
policy
v1
or
appsv1.
A
This
was
going
to
cause
breaking
changes
for
users,
especially
ones
who
hard
coded
these
endpoints
into
their
code
bases.
We've
all
seen
spaghetti
code,
we've
all
been
there
for
some
users.
This
was
going
to
be
a
big
deal,
but
we
didn't
really
realize
that
entirely
at
first,
we
started
by
communicating
those
details
the
way
that
we
communicated
details
about
any
other
change
in
kubernetes
in
the
changelogs
and
the
release
notes.
A
A
B
A
And
as
it
turned
out,
the
population
of
end
users
by
then
looked
different
than
it
used
to
the
newer
population
was
larger.
They
had
more
varied
backgrounds
and
more
varied
levels
of
background
knowledge,
and
often
they
were
more
disconnected
they
weren't
plugged
into
the
community,
maybe
didn't
know
where
to
find
those
community
resources.
A
If
we
had
talked
to
end
users
more
to
understand
their
use
cases
and
concerns,
we
might
have
anticipated
better
in
advance
how
these
changes
might
have
been
received,
as
it
turns
out,
more
users
had
been
hard-coding
their
endpoints
than
I
think.
A
lot
of
maintainers
had
originally
realized
and
a
lot
of
users
got
kind
of
freaked
out
by
this.
A
A
lot
of
users
didn't
want
to
update
in
order
to
avoid
these
breaking
changes
and
the
fact
that
a
lot
of
users
were
really
hesitant
to
update
for
breaking
changes
caused
ripple
effects
on
the
project
that
continued
today.
Officially
kubernetes
has
never
had
a
long-term,
stable
version,
but
functionally
in
some
ways.
By
now,
it's
kind
of
played
out
that
way.
A
People
are
still
running
1.15
in
the
wild,
which
is
way
out
of
patch
cycle.
For
us,
we
support
n
minus
three.
This
not
only
has
security
considerations,
an
issue
near
and
dear
to
my
heart,
as
kubernetes
security
co-chair,
it
has
had
effects
on
the
decisions
made
by
public
clouds
and
manage
kubernetes
distributions.
A
A
A
B
Now,
if
you
aren't
aware
of
what
the
docker
shim
is,
which
would
not
be
surprising
considering
the
way
this
story
plays
out,
here's
a
little
bit
of
context
for
you
when
you're
using
kubernetes,
you
need
to
specify
a
container
runtime
in
a
lot
of
cases.
People
just
go
with
docker
as
the
runtime,
because
it's
familiar
and
that's
what
they
use
in
development
on
the
surface.
That
seems
fine,
but
docker
isn't
actually
just
one
thing.
It's
a
whole
tech
stack
and
inside
of
the
docker
stack
is
a
container
runtime
called
container
d.
B
B
This
is
an
entirely
separate
thing
that
people
have
to
maintain
and
it's
been
a
pain
point
for
quite
a
while,
so
it
was
decided
that
support
for
the
docker
shim
would
be
deprecated,
forcing
people
to
use
container
d
or
cryo
or
another
compatible
run
time
directly.
Instead,
this
change
is
ultimately
good
for
the
community,
and
some
people
thought
it
wouldn't
be
a
big
deal.
A
One
thing
that
a
lot
of
us
really
misunderstood
as
core
maintainers
around
this
was
the
level
of
understanding
of
a
lot
of
end
users.
If
all
of
the
things
that
kat
just
said
sounds
like
deep
technical
arcana
to
you,
that's
understandable.
It
sounds
like
that
to
a
lot
of
people.
A
lot
of
people
don't
actually
know
the
difference
between
a
docker,
shim
and
a
docker
file,
or
for
that
matter,
between
docker
and
kubernetes.
A
A
That
was
not
quite
what
we
were
going
for
so
okay,
but
let's
go
back
a
little
bit.
What
happened
here?
The
person
who
originally
put
in
the
deprecation
pr
for
docker
shim
announced
on
twitter
one
morning
that
that
was
going
to
happen
and
he
tweeted
out
that
docker
was
going
to
be
deprecated
in
kubernetes.
He
did
not
mention
docker
shim
by
name.
A
Probably,
I
would
guess,
assuming
that
people
would
know
what
he
meant
he
asked
for
that
tweet
to
be
amplified
and
first
thing
in
the
morning
before
coffee,
I
got
bright
ideas
and
I
thought
I
would
come
help
somebody
with
a
platform
on
twitter.
I
could
amplify
it,
I
would
get
attention
on
it,
users
would
see
it
and
they
could
make
a
plan
so
ralph,
wiggum
style.
A
I
helping
I
went
and
quote
tweeted
this
with
a
kind
of
intentionally
click-baity
headline,
which
said
this
isn't
going
to
affect
all
users,
but
it
is
going
to
affect
some
pay
attention
to
this.
If
it
does
affect
you
all
caps,
it
will
break
your
clusters
and
I
meant
well.
I
wanted
to
get
eyes
on
it.
I
wanted
users
to
see
it.
It
was
asked
to
be
amplified.
I
amplified
it,
I
did
mean
well,
but
it
maybe
didn't
quite
go
the
way
that
I
thought
it
would.
A
A
Some
of
them
maybe
sounded
kind
of
scared,
and
I
knew
that
I
wasn't
going
to
be
able
to
answer
their
questions
immediately
because
I
had
to
go,
and
so
I
went
on
to
the
cc
and
cncf
ambassadors
channel
to
ask
if
some
ambassadors
could
come
help
field.
These
questions
from
end
users,
since
I
had
to
go
for
a
while.
I
left
for
work
hoping
for
the
best,
and
then
things
got
weird.
B
B
B
A
A
A
I
figured
I
would
you
know,
try
to
amplify
this
to
get
attention
on
it
so
that
end
users
could
see
it
and
plan
around
it,
but
I
wasn't
really
very
thoughtful
about
the
way
that
I
put
that
it
certainly
did
get
attention
in
the
way
that
it
would,
if
I
had
shouted
fire
in
a
crowded
theater,
but
as
these
situations
do
that
got
pretty
out
of
control
and
that
wasn't
what
I
meant
to
do
at
all.
I
felt
so
bad.
I
felt
terrible
for
scaring.
A
A
B
Ian
and
I
definitely
both
made
mistakes
communicating
here.
I
wanted
to
answer
a
few
people's
questions
and
I
wanted
to
save
myself
some
time
while
doing
it,
but
I
forgot
that
most
developers
actually
don't
touch
kubernetes.
They
just
know
that
they
use
docker
and
development
and
kubernetes
is
used
somewhere
else
at
their
company.
So
the
tweets
made
them
think
that
docker
was
dead
and
needed
to
be
replaced
as
a
whole.
B
A
The
intent
of
the
docker
shin
deprecation
announcement
as
it
originally
occurred,
was
good.
We
wanted
to
avoid
the
problems
from
v1
beta
1
by
giving
users
lots
of
notice
that
a
change
was
occurring,
so
they
could
plan
around
it
and
we
wanted
to
make
sure
they
would
see
it.
We
did
achieve
that,
but
we
missed
the
mark
in
some
other
key
ways
and
from
there
we
learned
that
we
actually
have
to
have
a
better
understanding
of
what
other
people's
understandings
are.
We
have
to
talk
to
others.
B
But
before
we
wrap
up
docker
shim,
we
should
mention
the
cold
watering.
It's
fun.
I
promise
a
couple
of
weeks
before
the
docker
shim
deprecation
took
over.
I
had
cosplayed
as
ian
for
a
panel
at
cubecon,
north
america.
It
was
very
funny
at
the
time,
but
it
didn't
really
go
anywhere.
However,
for
some
reason,
the
two
of
us
being
involved
in
twitter
drama
together
reignited
that
and
the
result
was
several
days
of
people.
Cosplaying
me
cosplaying
ian.
There
were
wigs
on
everything
from
carps
to
pokemon.
B
A
I
really
appreciated
that
moment
of
wholesome
silliness.
I
feel
like
the
internet
needs
more
of
that,
and
that
moment
of
levity
was
nice,
but
we
as
a
project,
still
had
a
lot
of
work
to
do
coming
out
of
that.
So
collectively
we
thought
about
how
we
could
improve
again.
How
could
we
avoid
a
situation
like
this
happening
next
time?
A
A
The
community
got
together
to
figure
out
what
we
could
do
better
in
the
future
and
from
there
we
got
the
contributor
comms
committee.
The
contributor
comms
committee
was
a
committee
that
was
going
to
get
together
to
try
to
figure
out
how
to
craft
messaging,
to
users
around
changes
and
deprecations,
so
that
users
could
get
the
information
that
they
needed
without
causing
panic
or
trouble
for
anybody.
This
was
a
great
idea
and
an
important
effort
and
maybe
not
an
entirely
perfect
one.
A
You
might
not
have
heard
about
this
entirely
and
you
might
not
have
heard
about
some
of
it,
because
actually
it
ended
up
going
well,
so
pod
security
policies.
What's
up
with
this,
what
happened
there?
Pod
security
policies
are
a
very
old
feature
in
kubernetes
they've
been
around
for
a
long
time,
they're
really
important.
They
allow
users
to
set
security
controls
on
their
pods
and.
A
While
they're
important
they've
also
had
problems
for
years,
they
have
usability
problems,
users
have
found
them
confusing
and
they've
had
problems
from
a
maintainer
perspective,
as
we
found
out
more
about
what
isn't
working
for
users
in
them.
We've
also
found
out
that
we
can't
actually
make
meaningful
changes
to
fix
those
problems
without
breaking
the
api.
A
A
A
However,
people
started
talking
about
that
before
there
was
actually
a
plan,
and
this
had
the
effect
of
lighting
a
fire
under
us,
because
we
knew
we
had
to
come
up
with
one,
but
for
a
while
there
was
kind
of
a
question
mark.
They
were
going
to
go,
we
didn't
know
what
we
were
going
to
replace
them
with.
A
Contributor
comps
got
together
and
figured
that
this
was
a
recipe
for
a
scenario
that
might
cause
confusion
and
panic
for
end
users
and
fair
enough,
something
that
was
really
important
to
them
that
affected
the
security
of
their
clusters
that
affected
a
lot
of
users
was
going
to
be
deprecated
and
we
didn't
have
a
plan
to
replace
it.
Oh
no,
so
contributor
comms
figured
that
they
would
get
together
and
get
in
front
of
the
narrative
before
people
started.
A
Talking
about
this
more
publicly
and
causing
panic
for
users
and
in
order
to
get
in
front
of
the
narrative
contributor
comms
figured
that
they
were
going
to
get
together
and
write
a
blog
post.
They
talked
amongst
themselves,
they
figured
out
what
was
going
to
be
in
it
and
and
they
wrote
a
draft
post
talking
about
the
fact
that
this
was
going
to
be
happening
in
an
effort
to
try
to
get
information
out
to
users
so
that
they
would
know
what
was
going
on
now.
This
was
well
intentioned,
but
it
fell
down.
A
A
little
bit
because
contributor
comms
were
talking
amongst
themselves
about
what
was
going
on.
They
wanted
to
get
that
out
quickly
in
the
interest
of
speed
and
in
the
interest
of
speed.
They
decided
to
do
that
themselves,
but
they
didn't
actually
talk
to
people
who
were
closer
to
the
process,
and
so
the
information
that
they
were
going
to
bring
to
users
was
outdated
and
actually,
I
would
argue,
a
little
bit
scarier
than
what
was
actually
going
on
because,
on
our
end,
sigs
in
security.
A
We
actually
didn't
know
that
that
was
happening
over
by
contributor
comms,
but
what
we
were
doing
was
actually
coming
to
a
lot
of
progress
and
pretty
good
conclusions
about
what
was
going
to
come
next
to
replace
pod
security
policy.
We
were
feeling
really
optimistic
about
this,
and
that
was
going
great
on
our
end.
A
We
had
no
idea
that
was
going
on
over
there
and
they
wouldn't
actually
really
know
what
was
going
on
on
our
end
because
they
didn't
ask
us
so
the
post
that
was
going
to
go
out
had
outdated
information
about
how
they're
not
really
being
a
replacement
plan.
Yet,
while
on
our
end,
we
were
making
one
eventually,
we
heard
that
this
was
happening
and
we
scrambled
amongst
ourselves
to
try
to
figure
out
what
to
do.
A
We
knew
it
was
important
to
get
communications
about
this
out
to
users,
but
we
were
about
to
come
to
a
conclusion
that
we
thought
was
pretty
good,
so
we
put
a
hold
on
the
blog
post
talked
amongst
ourselves.
Looked
at
the
blog
post
draft,
it
needed
some
work.
It
had
outdated
info
in
it
that
we
thought
might
actually
freak
users
out
more
and
in
talking
amongst
ourselves.
We
actually
ended
up
making
the
same
mistake
that
contributor
comms
did
we
figured.
A
We
would
rewrite
the
blog
post
to
better
reflect
current
events,
but
we
didn't
tell
contributor
comms
that
we
were
doing
that
and
so
contributor
comms
very
fairly
were
like
what
is
happening
here.
This
was
supposed
to
be
out
weeks
ago.
What
are
you
doing
on
our
end?
We
could
have
communicated
better
with
them
too.
A
So
what
we
learn
from
this,
we
can
learn
that
it's
really
important
for
people
to
talk
to
the
people
who
are
closer
to
the
process,
whether
it's
closer
to
the
process
of
making
that
change
or
whether
it's
closer
to
the
process
of
communicating
out
to
users
about
that
change.
As
it
turns
out,
this
actually
turned
out
pretty
well.
A
Eventually,
we
did
rewrite
a
new
post
to
better
reflect
current
events
and
we
got
it
out
to
contributor
comms
who
liked
it.
It
was
maybe
a
little
slower
than
we
thought
it
would
have
been,
but
it
turned
out
well,
the
information
that
we
ended
up
putting
out
to
users
was
clearer
and
more
actionable
and
did
actually
better
empower
them
to
be
able
to
make
plans
in
the
future,
as
psps
did
get
deprecated.
A
So
what
we
can
learn.
Everybody
needs
to
talk
to
each
other
more
because
everybody
messed
this
up
a
little
bit,
but
also
everyone
learned
we're
all
still
friends,
it
all
turned
out.
Okay
and
I
think
the
change
ultimately,
although
a
little
slower
ended
up
being
communicated
better
speed,
isn't
really
the
only
factor
at
play.
B
So
docker
shim
taught
us
that
we
needed
to
talk
to
our
community,
but
this
showed
us
that
we
forgot
to
talk
to
one
another.
Remember
we're
used
to
maintainers
being
a
really
small
community,
and
it
just
is
not
true
anymore.
Kubernetes
is
big.
Now
a
lot
of
maintainers,
don't
know
each
other
much
less
know
what
each
other
is
thinking.
B
B
If
we
had
talked
to
the
relevant
cigs
before
getting
to
work,
we
might
have
saved
ourselves
a
lot
of
time
and
grief.
We
did
avoid
it,
though,
and
that's
important,
because
we
learned
from
the
last
two
disasters.
We
saw
the
problem
and
we
course
corrected
before
the
ship
hit
the
iceberg.
Instead,
we
just
kind
of
clipped
it.
There
was
some
damage,
but
it's
just
scratched
paint
and
if
we
learn
from
this
one
too,
we
get
closer
to
being
able
to
just
sail.
A
And
we
learn
a
little
more
every
time.
I
want
to
point
that
out
and
keep
pointing
it
out,
because
it's
true
in
the
interest
of
continuous
improvement
in
the
grand
devops
tradition
of
learning
from
incidents
and
failures,
so
we
can
do
better
next
time
we're
doing
that.
We're
doing
it
right
now
and
that's
awesome,
we're
doing
it
together.
A
We've
gone
through
a
brief
history
here
of
some
communication,
breakdowns
between
maintainers
and
users,
maintainers
and
maintainers,
about
how
to
communicate,
changes
and
we're
hoping
that
we
can
take
these
lessons
and
learn
how
to
do
that
better
in
the
future.
So
how
can
we
do
that?
How
can
we
take
the
lessons
that
we've
learned
from
this
to
improve
going
forward?
A
A
We
need
to
talk
to
developers
if
we're
administrators
or
maybe
the
other
way
around,
really
the
whole
community
needs
to
be
understanding
each
other's
perspectives
and
talking
together.
This
is
especially
the
case
for
the
understanding,
the
perspectives
of
the
people,
who
are
most
affected,
which
could
be
any
number
of
people
depending
on
what
we're
talking
about,
but
it's
especially
important
to
hear
from
and
understand
where
they're
coming
from.
A
B
B
This
is
a
totally
natural
result
of
a
community
growing
very
quickly,
and
nobody
should
leave
this
talk
feeling
bad.
Ultimately,
we
all
want
the
same
thing.
We
want
to
build
better
tools
for
our
community
and
minimize
negative
impact.
So
let's
learn
to
recognize
communication
breakdowns
earlier
before
they
become
disastrous.
B
A
A
Mistakes
do
happen,
but
we
can
make
them
less
likely
to
occur
and
less
destructive
when
they
do
occur.
It
just
takes
sometimes
taking
a
moment
to
think
about
where
other
people
are
coming
from
and
to
think
about
how
we're
communicating
to
people
in
relation
to
that
we
have
built
so
many
amazing
things
together,
we're
going
to
continue
to,
and
we
believe
that
if
we
work
better
together,
we
can
do
better
together
and
we
can
build
a
stronger
community
and
a
better
platform.
A
A
It
feels
really
important
to
us
to
talk
about
these
mistakes
so
that
we
can
learn
from
them
going
forward,
but
I
think
we
can.
I
think
we
already
have,
and
I
think
we
will
we
got
this-
I
believe
in
us.
Thank
you
so
much
for
coming,
I'm
kat
cosgrove
and
I'm
ian
coldwater
and
thank
you
for
coming
to
our
kubecon
talk,
bye.