►
From YouTube: SIGs Aren’t Silos: A Case Study Into Solving Inter-Domain... Swetha Repakula & Antonio Ojea Garcia
Description
Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
SIGs Aren’t Silos: A Case Study Into Solving Inter-Domain Problems In Kubernetes Development - Swetha Repakula, Google & Antonio Ojea Garcia, RedHat
The Kubernetes project development activity is organized into Special Interest Groups (SIGs). Each SIG is composed of members from multiple companies and organizations, with a common purpose of advancing the project with respect to a specific topic, such as Networking, Testing, Node or Documentation . Most of the time, tasks fall neatly within a SIG that is vertically focused on a particular component or domain area. However, what happens when those changes have an impact broader than that SIG? What happens because SIGs operate as silos? This talk is a case study in a recent cross-sig bug where a subtle behavior change by one SIG led to an outage causing bug in another. Due to the subtlety and cross-SIG nature of the bug, it went unnoticed for 6 months. Antonio and Swetha will walk through the incident and share the lessons learned.
A
This
is
a
developer
talk,
as
you
can
see,
we
are
I.
Imagine
that
all
of
you
are
contributors
or
willing
to
contribute,
or
people
interested
in
kubernetes
is
anybody
that
is
a
developing
kubernetes
can
raise
the
hand,
okay
and
people
that
are
going
to
contribute
to
kubernetes
ever
here.
Okay,
so
you
are
going
to
find
this
issue,
because
this
is
a
talk
about
a
Horror
history
that
we
we
have
to
suffer,
and
you
may
think
that
kubernetes
is
this.
A
A
A
Side
is
based
on
a
special
interest
group,
so
we
can
have
this
groups
or
six
that
are
horizontal,
like
API,
Machinery
or
excalibility,
or
we
have
another
internet
or
another
group
that
are
vertical
like
network
node
scheduler
most
of
the
people
are
used
to
these
verticals
in
their
companies
and
to
the
problems
that
this
represent.
What
is
the
problem,
this
General
problem
in
the
network?
Not
this
is
a
problem
in
your
site
and
you
have
this
wall
going
back
and
forth,
and
this
happens
in
kubernetes
too.
We
have
people
that
is
in
signaled.
A
We
have
people
at
least
six
Network
and
we
have
bikes
and
we
never
know
who
is
the
who
is
the
responsible,
but
instead
of
fighting
the
ball,
we
try
to
collaborate
to
solve
the
problem,
and
this
this
talk
is
about
one
of
these
examples.
Where
about
in
a
component
that
is
responsible
from
when
she
was
affecting
another
six,
and
how
do
we
solve
it
in
collaborate?
A
So
this
is
how
the
we
talked
about
how
the
project
is
organized
in
terms
of
development.
But
this
is
how
the
kubernetes
components
look
like.
We
have
accumulate,
we
have
a
API
server,
we
have
a
controller
manager,
we
have
q
proxy
and
the
components
are
more
or
less
related
to
one
seek,
but
doesn't
mean
that
the
seek
is
the
owner.
So,
for
example,
the
qlip
most
of
the
code
is
responsibility
or
responsibility
of
signal,
but
Sig
network
has
a
lot
of
bits.
A
A
And
at
the
end,
we
have
all
this
shutter,
but
at
the
end
we
are
coming
to
this
conference.
We
are
having
fun
together
and
we
are
just
people
most
of
us
are
paid
and
we
have
companies
paying
what
services,
but,
but
not
all
of
us
and
I
would
say
that
not
most
of
us
are
paid
just
for
working
in
kubernetes
is
asking.
A
Let's
start
to
explain,
what's
happening
so
I,
don't
remember
when
it
was
so
in
122
there
was
a
big
refactor
on
the
on
the
route
on
the
signal
on
the
qubit
about
the
pole,
life
cycle-
and
this
was
132
I
think
was
one
year
ago,
but
six
months
later,
or
because
she
will
release
until
the
uses
start
to
use
the
the
kubernetes
release.
This
is
always
a
gap.
A
This
is
most
I,
don't
know,
but
suddenly
we
start
to
receive
a
lot
of
bugs,
as
you
can
see
back
that
pops
is
failing
to,
but
we
failed
status.
Ip
address
to
use
the
new
parts,
but
traffic
is
still
going
to
our
Port
accomplendent
faces
and
people
say
well,
we
have
five
IPS.
This
is
signal.
Okay
and
then
you
have
another
one
that
is
terminated
forward,
should
download
listed
in
servicing
Point.
A
Okay,
servicing
point:
this
is
thing
number
two,
so
kubernetes
sending
traffic
to
draining
North
draining
knows
this
is
going
to
be
signal,
and
this
is
how
we
react.
I
mean
you
don't
really
know
this
is
not.
We
don't
have
a
support
people,
these
people
open
the
bag
and
we
just
go
there
and
we
try
to
figure
out
why
who
can
help
here
and
just
tag
it,
and
sometimes
we
attack
a
lot
of
people
and
sometimes
we
don't
tag
in
the
on
the
invisible
orphan.
B
B
Traffic
was
being
routed
to
non-existing
pods,
so
we
would
just
get
a
bunch
of
complaints.
Why
is
my
traffic
going
to
something
that
doesn't
exist
has
been
black
hole
here?
Load
balancers
are
broken,
then
the
next
one
was
oh.
Not
only
is
your
traffic
not
going
to
the
right
place
it's
going
to
the
wrong
pond,
which
can
be
a
pretty
big
security
problem.
B
B
Familiar
with
this,
if
you've
deployed
a
pod
before
you
enter
in
at
the
top,
and
it
goes
into
that
pending
State
before
it's
scheduled
once
it's
scheduled
for
a
little
bit,
but
once
you
know,
cubelet
has
a
probe
and
it's
getting
Marcus
ready.
We
get
moved
into
this
running
State
and
those
top
three
states
over
there
could
be
in
a
cycle
right
like
a
pocket
restarted,
it
gets
rescheduled,
it
crashes,
you
know
all
sorts
of
things.
B
D
B
Probe,
it
goes
checks
in
the
on
the
Pod.
Are
you
ready?
It
says?
Yes,
I
am
cubelet
news
is
that
it
says
hey.
Let
me
update
the
API
server
and
then
the
bottom
left
component.
There
is
kind
of
the
controller
manager,
so
it
has
the
endpoints
and
the
endpoint
slice
controllers
that
are
kind
of
packaged
into
that,
and
we
read
that
information
to
determine
and
make
service
decisions
or
endpoint
decisions,
so
I'm
going
to
Deep
dive
in
a
little
bit
more.
B
So
the
first
thing
is
to
talk
about
odd
shutdown,
and
this
is
kind
of
before
122
and
before
the
refactor.
The
typical
scenario
is
a
user.
Will
user
will
delete
the
Pod?
So
it's
like
an
API
call.
There
is
a
deletion
stamp
set
on
the
Pod
that
will
get
signaled
to
the
cubelet.
I
have
simplified
this
diagram.
There
are
other
steps
involved,
but
I'm
going
to
make
it
very
simple.
Cubelet,
basically
signals
to
shut
down
the
Pod
through
the
container
runtime.
The
Pod
goes
away.
Cubelet
will
also
update
the
status
saying
hey.
B
This
is
no
longer
a
pod,
it
usually
goes
into
unknown
and
the
another
thing
it
does
is
it
removed
the
Pod
IP.
There's
a
slight
variation
of
this,
and
that
happens
when
you're
doing
evictions.
We
skipped
kind
of
the
first
few
steps
and
cubelet
usually
will
make
a
decision
and
say
hey.
This
pod
needs
to
go.
Maybe
it's
because
there's
no
more
memory,
we
need
more
CPU,
it's
made
that
decision,
so
it
will
directly
shut
down
the
pot
and,
in
those
cases
again
the
Pod
IP
is
removed.
B
In
those
cases
when
a
pod
is
evicted,
it's
also
considered
a
terminal
state.
So
it
is
in
that
failed.
State,
usually,
and
if
you
look
at
the
diagram
at
the
bottom
right,
we
have
three
different
types
of
terminal
pods,
so
the
top
one
is
completed.
So
that
means
it
would
have
been
in
that
succeeded,
State.
The
second
one
is
an
error,
so
that
would
be
in
that
failed,
State
and
then
the
last
one
is
eviction
is
also
in
the
failed
State.
B
B
Ip
shouldn't
be
removed.
The
other
key
thing
that
changed
is
probes
are
shut
down
before
the
Pod
is
marked
terminal,
so
that
will
be
kind
of
key
a
little
bit
later
on
yeah,
so
that
this
is
basically
the
change
that
has
happened
and
so
far,
if
you
look
at
it
from
a
pod
life
cycle
standpoint,
this
makes
a
lot
of
sense
and
it's
actually
kind
of
cleaned
up,
some
of
the
confusion
or
Clarity
in
some
ways,
because
now
you
know
what
the
Pod
life
cycle
should
look
like.
B
Move
into
the
Network
side
of
things
so
services
and
endpoints
and
what
happens
when
we're
processing
these
events,
the
controller
manager
which
has
our
endpoints
controllers,
they
watch
for
these
pod
updates
and
when
it
gets
a
pod
update.
We
see.
Oh
look.
This
pod
got
deleted.
We
should
remove
the
endpoint
from
our
endpoint
list.
B
E
B
However,
if
we
go
into
the
buck,
hopefully
you
can
read
that
there's
two
kind
of
key
pieces
here
and
so
I'm
kind
of
digging
into
the
endpoints
code
now
to
kind
of
just
show
you
the
subtlety
of
this
book.
The
first
line
is
telling
you:
if
there's
no
IPS
on
the
Pod
object,
skip
this
pot.
We
don't
care
about
it,
we're
not
going
to
include
it.
The
second
one
we're
going
to
ignore
the
tolerated,
the
yeah
tolerate
unready
endpoints.
But
the
next
piece
is
the
deletion
tax
stamp.
B
If
the
Pod
is
deleted,
obviously
we
don't
want
to
include
it.
We
don't
care
about
it.
So
let's
skip
it.
So
if
we
go
back
to
our
diagram,
what
this
really
means
is
if
the
cubelet
is
now
actually
only
marking
pod
States
as
terminal
in
an
eviction
case,
that
means
that
the
Pod
actually
doesn't
get
deleted,
and
instead
we
also
now
don't
remove
the
IP
and
a
key
thing
about
evictive
pods.
B
Is
they
stick
around
in
the
API
and
they
stick
around
until
you
get
a
threshold
of
it
and
then
pod
GC
goes
and
looks
and
finds
all
these
orphan
pods
and
removes
them.
But
there
is
some
time
when
these
terminal
pods
can
exist
in
the
ecosystem
and
that's
kind
of
what's
Happening.
Here
is
because
now
pod,
the
Pod
state
is
terminal
not
deleted,
and
it
has
an
IP.
What
ends
up
happening
is.
If
we
look
back
at
the
code,
is
the
slide
yeah?
What
ends
up
happening
is
it's
actually
failed.
B
It
passes
both
those
checks,
and
now
this
endpoint
is
now
included
into
our
endpoint
list,
and
if
we
go
dive
a
little
bit
more
deeper
into
the
endpoint
code,
we
do
have
a
few
more
checks
that
we're
trying
to
do,
namely
it's
this
little.
If
condition
that
is
checking
if
a
pod
is
ready
or
if
the
Pod
is
not
ready,
basically,
and
when
we
get
to
that
ready
state
that
is
purely
based
off
of
that
cubelet
update
right,
the
pro
Buffet,
the
second
piece,
which
is
that
should
fog
be
in
endpoint.
B
B
B
B
Shouldn't
really
affect
your
routing
decisions
unless
you're
consuming
endpoints
and
endpoint
slices
by
yourself
and
making
other
networking
decisions
based
on
it.
So
overall,
this
is
kind
of
this
is
a
bug
that
exists
but
doesn't
affect
us
too
bad.
However,
the
story
kind
of
gets
a
little
worse,
so
we'll
pass
it
back
to
Antonio.
A
Okay,
so
I
think
that
right
now
is
more
or
less
clear
how
this
works
right.
Everybody
uses
Services,
everybody
uses
employees
and,
as
can
as
you
can
see,
there
is
a
lot
of
chain
reactions
in
between
the
other
consequence
of
this
change
on
the
public
cycle
was
that
we
have
another
nice
feature
for
for
services
endpoint,
that's
called
terminating
in
part,
so
most
of
you
that
goes
to
deploy
services
and
want
to
have
a
Serial
destruction,
or
something
like
that.
You
just
don't
want
the
important
to
go
out
of
the
of
the
service.
A
Just
when
you
delete
you
want
to
have
some
Grace
periods
so
that
part
can
receive
traffic.
Okay.
So
for
that
in
sync
Network,
we
we
implemented
the
Terminator
endpoint
feature.
What
does
it
mean?
It's
just
the
body
state
is
no
longer
by
so
it's
not
ready
and
ready.
It's
ready
he's
unterminated.
You
can
send
me
traffic
if
you
are
ready.
So
we
have
this
special
condition
in
the
point:
analyze
controller
that
is
allowing
to
send
traffic
to
to
post
that
are
terminating
and
one
of
the
consequences
of
beautiful
Factor
was
another
another
bar.
A
The
the
problem
was
okay,
the
problem
was
that
the
interior
Factor,
when
the
pulse
was
going
down.
We
forget
to
to
to
keep
the
product,
so
the
call
was
was
when
the
public
is
going
down.
You
have
the
crop,
that's
what
I
supplemented
before
to
say.
I
is
the
pulse
ready
and
the
pope
Sudan
was.
There
am
I
ready,
but
the
problem
is
that
during
this
shutdown,
the
approach
will
remove
it.
So
the
cubelet
couldn't
know
what
is
the
state
of
the
pot.
So
what
happens
in
this
situation?
A
So
the
problem
is
that
in
this
situation,
the
important
controller
as
soon
as
that
the
producer
will
ready,
meanwhile,
is
terminated
and
what?
If
the
pulse
was
already
there
or
is
not
ready,
he's
not
reflected
so
you
end
sending
traffic
to
about
30
step,
and
this
means
for
people
with
ingresses
or
other
kind
of
services.
You
are
going
to
have
four
zero
four
or
connecting
dropping.
A
The
fix
was
really
simple
for
the
people.
That
know,
of
course-
and
it's
got
just-
don't-
just
keep
the
props
working
and
moving
while
the
posts
are
terminated.
Okay,
so,
and
after
this,
what
are
the
lessons?
Regular
I
think
that
we
are
going
too
fast.
Okay,
we'll
have
more
time
for
questions.
So
the
last
thing
that
we
learn
is
all
the
all.
These
Behavior
changes
come
break
holders
and
what's
important
with
this.
A
The
important
with
this
is
that
we
need
to
to
establish
contacts
on
the
call
and
the
best
way
to
start
this
contest
is
adding
tests
right
now,
until
122
is
watching
clear
how
the
polite
Cycles
will
work
the
concept
of
Port
Terminal,
those
Concepts.
Maybe
they
were
clear,
but
we
didn't
have
tests
and
the
rest
of
the
people.
D
A
C
A
A
F
A
A
Just
ask
I'm,
for
example,
are
very
verbose
people,
so
I'm
go
to
a
channel
and
I
keep
asking
on
the
channel
and
they
just
start
with
one
person
that
is
going
to
reply,
and
if
it
is
not
going
to
reply,
you
need
to
think
maybe
I'm
doing
the
wrong
question.
Maybe
this
is
not
the
right
channel,
but
I.
Don't
think
that
it
is
a
is
people
we
are
people
just
know,
each
other
just
ask
questions
and
I,
don't
think
nobody's
afraid
of
answering,
but
there
is
no
process.
A
B
I
had
the
same
answer.
You
know
you
have
to
kind
of
I
mean
it's
both
a
blessing
and
a
curse.
Because.
Unless
you
know
people
you
don't
know
who
to
reach
out
to
but
I
think
I
really
like
what
Antonio
said.
Just
ask
questions.
I
have
not
met
a
single
kubernetes
contributor.
Who
did
not
want
to
answer
my
questions.
E
A
The
next
question:
if
you
are
uses
of
kubernetes
other
thing,
that's
important
for
us,
is
we
don't
have
a
qat,
we
have
users
and
I
mean
it's
sad
to
say,
but
uses
are
the
first
one.
That
CI
is
is
something
that
I
don't
want.
We
have
a
seat
testing
meeting
tomorrow
that
I'm
going
to
talk
about
ci2,
but
I
was
taking
for
the.
E
D
B
Love
you
so
I
mean
I.
Think
every
project
is
a
little
different
for
me.
I
think
the
I'm
relatively
newer
to
sign
Network
than
Antonio
is
and
I
just
started
attending
the
meetings.
You
start
learning
some
things.
I
will
say
you
know
once
you've
learned
some
names
who
come
often
you
kind
of
feel
like
oh
I
can
reach
out
to
them.
B
Ask
them
questions
and
Tony
also
reviewed
quite
a
few
of
my
PRS
and
and
then
you
just
start
reaching
out
on
Slack
and
once
you
develop
a
relationship,
it
kind
of
continues
like
it's
not
going
to
end
and
then
through
Antonio
I
met
other
people
also
working
at
Google.
There
are
other
contributors,
so
I
do
you
know
you
can
meet
someone
who'll
be
like
oh
talk
to
so,
and
so
they
might
help
you.
B
So
if
you
are
not
a
contributor
who
has
this
big
group,
I
would
say
first,
you
know
go
to
that
meeting
and
just
kind
of
see
who
are
the
face
is
see.
Who
are
the
names
typically
I?
Think
most
of
them
start
off
with
triage
and
like
you'll,
see
people's
GitHub
names,
you'll
I.
Think
a
lot
of
people
also
try
to
make
it
really
easy
to
find
them
on
slack
and
then
the
name
that
they
have
on
the
meeting.
B
A
Yeah
I
mean
I
have
seen
that
all
of
us
in
in
kubernetes
and
in
most
of
the
offensive
projects,
you
really
don't
look
at
the
company
of
the
people.
I
mean
I,
only
know
the
people
by
the
GitHub
countries
so
I.
That's
not
the
right
size
of
that.
I
really
don't
know
the
company
of
the
people
or
I
really
don't
care.
I
mean
we
have
the
same
goal.
It's
fixing
the
bag.
Actually,
this
bag.
A
It
happened
in
a
record,
it
was
reported
internally,
it
happened
and
that
we
were
reported
internally
and
we
happened
to
work
in
the
same
back
together.
I
mean
with
you
know,
different
origin,
but
with
the
same
goal
that
we
speaks
in
kubernetes,
not
fixing
Network
a
company
I
mean
we
have
to
fix
our
company
things,
but
this.
The
conclusion
is
that
do
you
forget
about
companies,
and-
and
this
is
about
contributing
and
making
things
better
for
users
and
everybody.
G
All
right,
first
of
all,
awesome
presentation,
and
it
gets
really
awesome
summary
of
this.
Well
of
all
the
issues
here.
I
had
a
quick
question,
so
one
of
the
interesting
things
I
think
this
arises
is
like
there's
certain
invariants
and
kind
of
guarantees
and
kubernetes
around
different
life
cycle
right,
whether
it's
Network
endpoint
slices,
pod
life
cycle,
Etc,
I'm
kind
of
wondering
what
your
suggestion
is
and
what
you
would
find
helpful
is
like
how
we
communicate
those
guarantees
across
different
six
and
broadly
in
the
community.
G
B
So
I
don't
think,
there's
an
easy
answer,
just
because
I
I
do
think.
Documentation
is
the
first
step
if
you
components
you
own,
and
this
is
the
behavior
that
you've
defined
at
least
there's
Clarity
there.
If
somebody
asks
you
something,
you
point
to
them
and
say
hey,
this
is
the
thing
and
it's
great.
If
documentation,
States
code.
B
D
B
C
B
So
we
added
a
test
so
that
way,
if
an
invariant
changes
we
can
either
like
you
know,
say
no
no
or
we
can
we're
aware
when
we're
coding
and
I
think
it
unfortunately
Falls
a
little
bit
to
the
consumers
right.
You
need
to
be
aware
of
the
apis
you're
using
and
you
need
to
make
sure
if
you
make
an
assumption,
you
either
have
it
tested
or
you
have
it
very
clear
and
obviously
now
I
can
tell
and
say,
hey
David.
Can
you
go
and
make
sure
any
changes
to
these
areas?
B
A
I
am
more
radical
with
tests
and
test
and
test,
and
you
have
something
you
put
a
test
if
somebody
wants
to
use
something,
that's
not
supposed
to
wait
to
work,
it's
going
to
fail.
The
test
and
documentation
is
great,
and
we
have
this.
Spotlight
cycle
is
well
documented.
I
mean
it's
pretty
well
documented,
but
the
problem
that
we
have
here
this
is
the
contact,
wasn't
enforce
and
One
controller
was,
you
know,
making
the
interpretation
of
of
that
content,
but
wasn't
enforcing
so
I.
A
Think
that
it's
in
in
this
case,
in
in
hindsight,
is,
is
a
a
problem
in
sync
Network.
We
didn't
have
a
test
to
cover
this.
We
assume
that
in
points
and
normal
cycle
was
this
ideal
thing
and
I
mean
we'll
have
to
do
both.
We
have
to
document
things
to
know
how
it
works,
and
then
we
need
to
add
test
to
to
know
that
we
don't
regress,
that's
what
it
says:
no
magic,
his
test
and
documentation.
C
E
Fine
yeah,
thank
you.
This
was
a
really
cool
presentation
and
I.
Think
that
you
mentioned
that
you
know
there
was
a
disconnect
between
the
like
testing
in
network
six
here
and
I'm,
just
kind
of
curious,
like
how
many
people
are
active
in
multiple
cigs
at
once,
and
then
how
do
different
things
like
prioritize,
like
their
issue,
backlog
based
on
things
that
other
things
bring
up.
A
You
you
don't
have
and
that's
the
problem,
it's
more
on
the
people
gut
feeling
I
started
to
working
working
on
this
in
Valencia
I'm,
with
some
people
from
here.
I
saw
these
box
and
I
started
working
on
this,
and
some
of
them
say
well,
we
had
similar
bags
and
then
we
started
working
more
on
this
because
we
knew
that
something
was
going
on
and
it's
just
I
mean
you
know
something
is
bad
and
you
want
to
fix
it
and
that's.
This
is
something
that
is
impacting
the
project
today.
A
So
I
think
that
you
can
come
good
idea.
I
mean
this
is
important,
I'm
going
to
delay
other
things.
But
if
you
tell
me,
we
have
a
backlog
within
some
points
and
priority
I
mean
I.
Personally,
don't
have
that
I.
We
try
to
do
that
on
the
sixth.
We
try
to
prioritize,
but
at
the
end,
I
think
that's
more
about
the
people
and
they
got
feeling
and
their
willingness
to
fix
things.
A
Of
course,
if
you
have
a
bug
that
is
affecting
your
product
in
your
company,
that's
going
to
be
prioritized,
but
you
know,
because
you
are
working
at
the
finger
flow.
If
we
are
talking
about
a
vacuum
kubernetes
that
is
not
affecting
the
company,
this
is
about
people,
I
mean
I
mean.
Maybe
you
are
my
friend
as
you
tell
me,
this
is
affected
me
and
in
the
chat.
Okay,
let
me
I
have
one
hour.
Let
me
check
it
and
I
fix
it.