►
From YouTube: Becoming an Expert of Diagnosing a Kubernetes Control Plane Problem - Wenjia Zhang & Joe Betz
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Becoming an Expert of Diagnosing a Kubernetes Control Plane Problem - Wenjia Zhang & Joe Betz, Google
Debugging a kubernetes control plane can be a challenge with so many pieces in the puzzle. What and how is your kubernetes control plane doing? Are your kubernetes clusters happy? Can they be happier? If they are not happy, what went wrong? How can you tell? In this talk, Wenjia and Joe will walk you through a complete spectrum of metrics that you need to debug and tweak the performance of kubernetes control plane.
To learn more click here: https://sched.co/FuK7
A
And
that's
where
you
all
come
in
we'd
like
to
share
our
experiences
with
you,
our
learnings,
so
that
we
can
make
kubernetes
better
together,
I'm
Joe,
Betts
I'm,
the
lead
engineer
for
sed
at
Google
I'm,
also
a
maintainer
of
the
Exedy
project,
and
it
contributed
her
to
the
kubernetes
project.
Joining
me
today
is
when
Jay
Zhang,
who
is
also
a
contributor
to
both
kubernetes
and
@cd,
will
go
over
a
brief
recap
of
the
kubernetes
control
plane
remind
you
of
some
of
the
components
that
are
in.
A
It
will
then
walk
through
some
general
diagnostic
strategies
that
you
can
use
with
the
kubernetes
control
plane
and
then
we'll
get
to
the
heart
of
our
talk
where
we'll
go
through
real-world
examples
of
an
actual
issues
with
kubernetes
and
we'll
pair.
Each
of
those
issues
with
a
key
concept
from
kubernetes
that
you
can
use
to
inform
your
diet.
Diagnostics
work
will
then
close
it
up
with
some
guidance
on
how
you
can
keep
your
kubernetes
healthy,
your
clusters,
healthy
proactively.
A
So,
let's
just
briefly
recap:
the
control
plane,
the
kubernetes
control
plane
is
inspired
from
the
shape
of
a
ship's,
helm.
Kubernetes
literally
means
helmsman
and
a
home
has
a
hub
in
the
center
and
spokes
that
go
out
in
every
direction
from
it.
The
kubernetes
control
plane
has
a
central
component
and
then
a
series
of
components
that
interact
with
it
to
do
work.
A
That
central
component
is
called
the
cube
API
server,
and
it
provides
a
restful
interface
that
the
other
components
communicate
with
to
determine
the
state
of
the
cluster,
apply
their
business
logic
and
then
apply
the
changes
they
want
to
make
to
the
cluster
back
to
it.
Nestled
beneath
the
API
server
is
a
distributed.
Key-Value
store
called
that
CD.
There's
a
lot
I
could
say
about
this,
but
for
now
we'll
just
remember
that
this
is
the
persistent
storage
system
for
the
control
plane.
Let's
go
around
some
of
the
spokes.
The
first
is
the
controller
manager.
A
This
is
a
bundle
of
controllers
that
run
in
a
single
process.
Some
of
those
controllers
you
might
be
familiar
with
because
they
map
directly
to
a
concept
in
kubernetes.
For
example,
the
service
controller
is
responsible
for
the
business
logic
of
service
resources,
and
so
you
can
kind
of
inform
your
understanding
of
what
it
does.
Based
on
that,
some
of
the
controllers
provide
a
function.
For
example,
the
garbage
collection
controller
is
responsible
for
cleaning
up
resources
that
aren't
needed
by
the
system
anymore,
based
on
policies.
A
Okay,
we
also
have
the
cloud
controller
manager.
This
is
responsible
for
integrating
with
the
various
cloud
providers
that
kubernetes
supports
it's
the
primary
integration
point
for
them,
and
it's
the
interface
that
hides
the
differences
of
the
cloud
providers
from
kubernetes.
So
you
get
a
uniform
experience.
A
Of
course,
kubernetes
has
a
scheduler
I
like
to
think
of
the
scheduler
at
a
very
high
level,
as
being
primarily
responsible
for
mapping
pods
onto
nodes.
It's
the
responsibility
of
the
other
components
to
get
the
pods
created
so
that
the
scheduler
can
then
schedule
them
most
kubernetes
clusters
have
some
extensions
installed,
maybe
they're
installed
as
part
of
the
kubernetes
distro
you
use
with
your
cloud
provider.
Maybe
there's
something
you've
installed.
Custom
on
to
your
cluster
I'd,
encourage
you
to
take
some
time
to
understand
the
extensions
that
exist
on
your
cluster
when
you're
diagnosing
it.
A
A
This
is
important
because
first
of
all,
panicking
doesn't
help
and
also
kubernetes
design
so
that
the
control
plane
is
not
a
single
point
of
failure.
All
the
nodes
will
continue
to
run
the
pods
that
have
been
scheduled
on
them
until
the
control
plan
recovers.
So
you
can
continue
to
run
your
mission
critical
software.
A
While
you
recover
the
control
plane,
a
common
question
I
get
when
people
ask
me
about
how
to
diagnose
control
problems
is,
is
there
some
kind
of
logic
or
mapping
that
I
can
apply
to
the
various
problems
that
I
might
encounter
and
the
locations
in
the
system
where
that
issue
is
happening?
And
surprisingly,
sometimes
the
answer
is
yes.
A
So,
for
example,
in
the
kubernetes
api
server,
if
you
get
a
500
series
response
from
that
in
HTTP,
that
literally
means
the
API
server
is
telling
you
that
it
knows
it
has
an
internal
server
error
and
it
will
give
you
a
message
telling
you
some
more
details
and
you
can
go
directly
to
that.
Opponents
are
inspecting
it,
but
in
many
cases
a
single
symptom
can
map
to
multiple
causes
and
for
these,
even
if
you've
seen
the
symptom
before
and
remember
what
happened.
A
This
is
the
template,
we'll
use
in
the
examples
we're
about
to
show
we'll
start
with
some
symptoms
and
then
we'll
work
our
way,
all
the
way
down
to
a
root
cause
and
then
we'll
take
it
one
step
further
and
look
at
mitigations
and
things
that
you
could
do
immediately
to
get
your
cluster
working
again.
So,
let's
walk
through
some
examples,
we'll
start
with
the
first
one.
A
Our
key
concept
here
is
that
all
component
and
interaction
and
kubernetes
is
observable,
and
what
that
means
is
that
the
coop
API
server,
the
hub
of
that
helm,
performs
a
business
logic
of
its
own
and
the
components
that
communicate
with
it.
Everything
from
the
master
components
to
the
extensions
to
the
nodes
themselves,
and
even
you,
the
user,
must
interact
only
with
the
coop
API
server
and
cannot
directly
communicate
with
each
other.
As
a
result,
all
communication
between
all
those
components
is
going
through
this
central
point
and
you
can
inspect
all
of
that
interaction.
A
You
can
do
that
quite
simply,
just
with
the
coop
cuddle
command-line
tool,
for
example,
if
you
use
a
kit,
you'll
see
a
pod
and
if
user
describe
you'll
see
that
same
pod
with
more
detail,
but
it's
important
to
know
that
describe
doesn't
show
all
the
details
of
a
pod.
It
shows
the
most
essential
information.
If
you
want
everything
you
really
need
to
drop
down
to
outputting
the
resource
in
either
yamo
or
json,
and
then
you
will
get
each
and
every
field
that
exists
in
that
resource.
Sometimes
the
change
will
only
be
visible
here.
A
If
you
needed
to,
you
could
use
that
same
command
without
put
llamó,
and
you
would
literally
get
an
entire
copy
of
that
resource
every
time,
a
change
anyway.
So
this
gives
you
extensive
details
of
what's
happening.
Let's
use
that
to
look
at
an
actual
problem
here.
We've
got
a
pod
that
has
been
stuck
in
a
terminating
state.
Pods
in
kubernetes
should
only
be
take
a
couple
minutes
or
a
couple
seconds
to
terminate
at
most,
but
this
one's
been
stuck
in
the
state
for
over
35
minutes.
A
There's
a
couple
ways
that
you
could
find
the
next
thing
you
could.
You
could
search
the
web
and
ask
what
are
the
common
causes
of
pods,
getting
stuck
in
terminating
state
and
you'll
get
a
bunch
of
hits
back,
and
some
of
them
will
talk
about
finalized
errs
or
you
could
just
go
directly
and
look
at
the
resource
and
see
that
there's
a
finalizar
like
the
one
shown
here
on
it.
A
The
next
step
might
be
to
learn
a
little
bit
more
about
what
finalized
errs
do
and
what
you'll
learn
is
that
controllers
create
finalized,
errs
and
then
are
responsible
for
moving
them
when
they're
done
with
whatever
work
they
needed
to
do
and
it's
only
after
all,
the
finalize
is
removed
that
a
pod
is
removed.
So
that's
clearly
our
problem:
we've
got
a
pod,
that's
not
being
removed
because
there's
the
finaliters
are
still
on
it.
So
why
is
there?
Why
is
it
our
final
Iser
and
what
controller
was
responsible
for
it?
A
Well,
if
you
dig
a
little
further,
you
might
find
that
the
finalizar
includes
the
name
of
the
component,
so
you
can
use
that
to
do
some
searches.
Ask
your
colleagues
and
find
that
this
is
a
third-party
component
that
was
installed
in
your
cluster
and
there's
also
a
known
issue
with
it,
where
sometimes
under
certain
conditions,
it
doesn't
remove
its
finalized
errs
and
that's
been
fixed
in
or
later
release.
So
you
could
then
now
start
to
actually
sort
sort
out
the
problem.
A
Let's
look
at
one
more
example.
This
one
is
based
on
the
concept
that
all
state
is
persisted
in
that
CD.
This
is
really
useful
to
know,
because
if
your
cluster
becomes
unresponsive
to
the
point
where
you
can't
interact
with
it
through
the
API,
you
can
still
go
down
and
look
at
the
data
and
see
these
stores
a
history
of
data.
It
doesn't
just
store
the
latest
version
of
objects
when
you
update
them,
it
keeps
track
of
history.
It
uses
this
to
provide
a
watch
mechanism
so
that
you
can
see
changes
over
because
of
this.
A
You
could
imagine
well
if
we
keep
updating
object,
the
storage
space
will
grow
indefinitely
over
time,
so
sed
provides
a
way
to
compact
that
old
history
away
with
kubernetes.
That's
done
from
the
coop
api
server,
using
the
flag
shown
here
and
by
default
to
compacts
every
five
minutes.
So
every
five
minutes
everything
older
than
five
minutes
is
deleted,
which
means
right
before
compaction
occurs.
You
can
have
up
to
10
minutes
of
historical
data.
A
So,
what's
up
look
at
an
issue
here:
we've
got
a
cluster,
that's
become
unresponsive
and
we
look
at
the
logs.
We
see
this
at
CD
data
space
exceeded
air.
Some
of
you
may
have
seen
this
one
before
my
favorite
tool.
To
do
diagnosis
is
called
augur.
It's
a
tool
that
we've
written
an
open
source.
It
allows
you
to
take
an
sed
data
file.
Even
if
the
cluster
isn't
running
and
analyze
it,
you
can
look
at
specific
objects
and
see
them
in
yamo.
Even
if
they
were
written
in
protobuf
you
connect.
A
You
can
examine
the
coal
key
space
in
aggregate
as
well
here
I'm
going
to
ask
for
three
main
fields:
I'm
gonna
ask
for
the
total
size
of
each
object,
I'm,
including
all
historical
revisions,
I'm
gonna
ask
for
the
current
revision
size,
I'm
gonna
ask
for
the
number
of
revisions
for
that
object
and
then
I'm,
gonna
sort
it
by
the
biggest
ones
first,
and
what
I
get
back
and
I
just
show.
The
sample
here
is
about
five
thousand.
These
minion
objects,
which
are
about
fifteen
kilobytes
each
well.
A
That
space
adds
up,
especially
considering
you've,
got
120
revisions
of
each
of
them,
so
you
could
do
a
back-of-the-envelope
calculation
sort
out
that
well,
if
all
of
them
took
up
that
much
space,
we'd
be
well
over
8
gigabytes
of
storage
space,
clearly
we're
not
because
we
hit
the
4
gigabyte
limit.
So
what
happened?
Well,
in
this
case
our
cluster
just
been
resized
from
about
a
thousand
nodes
to
about
five
thousand
nodes
and
wide
it
was
sizing
up
when
it
failed.
A
Another
question
you
might
ask
is
well:
why
is
it
being
updated
12
times
a
minute?
Well,
if
you
do
some
research-
and
this
is
a
little
advanced-
you
might
find
that
all
the
nodes
in
kubernetes
do
a
heartbeat
update
to
the
cake,
cube,
API
server,
every
5
seconds,
and
so
that's
where
those
rights
are
coming
from.
You
can
calculate
all
this
together
and
sort
out.
What's
going
on
the
last
remaining
bit
is:
why
is
each
node
object?
A
15
kilobytes,
you
might
examine
that
by
hand
and
what
you'll
see
is
a
large
list
of
docker
image
names.
It
takes
up
almost
all
the
space
and
it's
in
every
node
and
it's
the
same
list.
Well.
This
is
a
pre-loaded
image
list
that
you
can
put
in
kubernetes
and
it
allows
each
node
to
load
docker
images
before
pods
are
scheduled.
They
need
those
nodes,
and
that
means
that
when
those
are
when
pods
are
started,
it's
very
efficient,
but
this
is
taking
up
a
bunch
of
space.
There's
three
things
you
could
do
to
resolve
these
problems.
A
You
could
reduce
the
compaction
interval
that
will
cut
your
storage
space
in
half,
because
you'll
be
storing
half
so
much
history.
You
could
increase
the
HDD
storage
size,
there's
no
reason
it
has
to
stay
up
for
four
gigs.
You
could
go
to
eight
or
sixteen
or
you
could
reduce
that
preload
image
list
and
you'd
be
back
to
a
reasonable
size.
I
will
hand
it
off
to
Windsor
for
the
next
examples.
B
B
B
That
way,
you
turned
vanilla,
vanilla,
note
into
a
master
note
with
all
the
different
components
running
on
it,
and
that
master
note
is
the
control
plane
for
your
coronet
YZ
cluster,
so
things
the
master
components
themselves
are
managed
using
pod.
It
gets
all
the
benefit
out
of
it,
for
example,
the
master
couplet,
who
use
the
liveness
in
readiness
probe
to
check
the
house
nests
of
each
component
and
if
any
of
them
is
crashed
or
unresponsive,
the
master
couplet
can
restart
it.
B
So
the
the
control
plane
is
like
any
of
your
workload,
node
it's
predictable,
repeatable
and
immutable,
and
when
it
comes
to
it
down
to
diagnosing
a
control,
plane
issue,
it's
become
very
handy,
and
now
is
that
concept
in
mind
before
we
get
into
any
of
the
real-world
issues.
I
want
to
give
you
a
list
of
nice
tools.
I
can
use
for
sanity
check
the
first
one
is
coop
kado,
get
notes.
B
It's
really
just
used
to
check
if
all
expected
notes
are
present
and
in
ready
status-
and
you
can
see
their
master
is
just
another
node
right
from
that
point
on
you,
you'd
better,
have
access
to
your
master
VM.
Once
you're
on
your
master,
you
can
use
docker
PS
to
check
the
process
status
of
your
docker
containers,
which
are
the
master
components
and
you
can
see
we
have
a
seat,
a
scheduler,
API
server,
controller
manager.
B
Turn
out
that
the
controller
manager
crashed,
because
there
is
error,
starting
this
resource
quota
controller
and
you
keep
reading.
There
is
something
suspicious.
This
blah
blah
thing
might
have
some
issue.
Is
it
you
probably
want
to
either
remove
this
or
fix
whatever
problem
with
it?
But
in
this
case,
what
were
similarly
even
more
is
that
the
controller
manager
should
not
fail
this
easily
with
any
transient
error
right,
no
way
that
you
could
shoo
function
jump
instantly,
no,
they
the
our
control
plan.
The
controller
manager
should
have
a
better
way
to
handle
errors
like
this.
B
B
So
there
is
a
coronaries
issues
and
there
is
fix
on
it
and
once
you
get
the
root
cause,
the
solution
is
very
easy.
The
short
term
you
want
to
remove
or
fix
this
service
on
the
node
side
and
long
term.
You
do
want
to
upgrade
your
control
plane
to
whatever
version
that
have
that
fix.
Now.
In
this
example,
you
can
see
the
scope
kado
get
components
that
is,
this
is
very
powerful.
You
can
feel
like.
You
got
the
ultimate
weapon
to
diagnose
any
control
playing
issues
today.
B
Sometimes
it's
not
working
hmmm,
that's
not
good.
Well,
what
does
that
mean
if
I
remember
correctly?
Joe
just
mentioned
that
everything
is
talking
to
API
server
and
guess
who
is
Coco
talking
to
API
server?
So
if
your
coop
kado
is
not
working,
sometimes
it
means
your
API
server
is
not
working
and
a
DAT
is
actually
the
default
part
of
the
API
server
and
we
check
the
API
server
log.
What
do
we
see?
It
panicked?
That's
not
good.
B
Do
we
panic,
don't
panic,
remember
joe
said:
don't
panic
and
we're
software
engineers?
We
can
resect
race
and
turn
out
there's
something
wrong
with
this
aggregator
controller
open
API.
Some
of
you
might
already
know
what
that
is,
somebody
feel
am
I,
not
I,
don't
care
it
doesn't
matter.
You
can
always
check
online
and
apparently
this
is
a
no
issue
again
and
there
is
issue
opened
in
coop,
open,
API
ripple.
B
There
is
fix
with
it
and
there
is
kubernetes
fix
that
pick
up
that
fixing
open
API.
With
this
example,
if
I
remember
correctly,
everything
happened
like
wheezing
two
three
days
and
I
did
a
little
survey
before
this.
Before
this
speak.
I
know
there
are
a
lot
of
contributors,
so
I
want
to
say
that
open
source,
kubernetes
communities
rocks
run
applause
to
all
the
contributors.
B
Thank
you.
Thank
you.
We
did
a
good
job.
Let's
continue
doing,
we
still
have
to
fix
this
issue
and
it's
easy
just
upgrade
your
control
plane
to
the
next
version
version
that
have
this
fix
for
the
this
two
examples:
when
you
do
you,
you
have
a
control
plan
is
not
working
right
for
both
example.
You
have
a
broken
components.
What,
if
you
well
from
this
to
fix,
you
can
see
that
we
tried
really
hard
to
make
sure
our
control
plane
do
not
fail
because
of
transient
error.
B
B
This
is
when
a
good
monitoring
alerting
system
is
very
necessary
and
remember
joe
said
that
HTTP
500
error
is
a
very
good
indicator.
There
saying
that
ABI
server
is
wrong
in
this
case.
In
our
system,
we
have
a
monitor,
counting
how
many
a
500
errors
we
got
from
the
API
server
log,
and
if
it's
too
many
we
gotta
learn
it.
B
Turn
out,
it
failed
to
least
this
HPA
under
this
version-
extensions
fee,
1
beta
1.
What
do
we
do?
Next?
We
got
the
keyword
we
can
search
online
and
in
this
case,
instead
of
finding
the
exactly
issue
that
we
see,
we
find
why
we
had
that
issue.
Turnout
from
one
point,
horizontal,
pod
or
the
scalar-
is
no
longer
supported
under
this
version,
but
using
a
new
version
called
all
the
scaling
v1
and
in
this
particular
cluster
it
still
have
some
old
version.
Hpa
and,
of
course,
API
server
cannot
recognize
it
anymore.
B
We
didn't
find
the
fix
for
that
issue,
but
still
we
don't
need
to
panic.
All
we
need
to
do
is
just
be
a
little
more
creative
here.
The
safe
approach
would
be
to
delete
the
existing
old
version,
HPA
right
and
create
a
new
one
with
the
correct
version.
Of
course,
that
would
come
with
some
dumb
time.
Let
me
remind
you
guys
again
control
plan.
Downtime
does
not
mean
cluster
downtime.
B
Your
father's
still
running
your
application
is
fine,
but
if
you
really
do
not
want
a
control
plane
with
down
time,
there
is
menu
STD
surgery
that
you
can
do,
but
keep
in
mind
that
messing
up
with
the
raw
data
is
always
dangerous.
You
better
make
sure
what
you're
doing
have
a
back-up
plan
all
right.
B
We
talked
about
API
server,
we
talked
about
controller
manager
and
you
Vanessa
D
I
promised
some
of
the
guys
here
that
we're
going
to
talk
about
scheduler,
and
here
we
are
I'm,
actually
going
to
spend,
and
so
section
talk
about
scheduler,
which
is
we
should
trust
scheduler.
If
anything
happens,
it's
our
scheduler
is
unhealthy.
Instead
of
thinking
that
scheduler
is
wrong.
B
Actually,
look
at
the
scheduler
message
and
see
if
it
can't
give
you
the
control
the
clue,
because
our
scheduler
engineering
engineer
did
a
really
good
job.
The
scare
is
very
robust,
so
I
want
to
start
with
some
very
important
scheduler
concept
that
would
help
you
interpret
scheduler
issues.
The
first
one
is
note
selector.
It
provides
a
very
simple
way
to
considering
pods
to
notes
with
particular
labels.
For
example,
you
can
say
that
you
can
ask
the
scheduler
or
say
please
scheduler,
please
schedule
a
spot
on
note
that
have
GPU
right
and
then
followed
by.
B
That
is
in
note,
affinity
and
note
Tings.
If
no
affinity
is
a
property
of
pots
that
attract
them
to
certain
node
and
tank
is
the
opposite.
It
can
repair
the
note
from
a
set
of
hot.
Remember
that
note
that
have
GPU
on
it.
It
can
tell
the
scheduler
only
the
pot
that
need
the
GPU
can
be
running
on
me.
So
if
you
have
some
scheduler
issue,
those
are
the
things
that
you
want
to
check
like
if
there's
they're
set
properly
and
then
there
is
possible
court
down
notes
when
a
note
is
cordoned,
it's
unschedulable.
B
B
Instead
of
talking
about
one
real
issue,
I
want
to
give
a
summary
of
how
to
interpret
unschedulable,
so
the
symptom
could
varies
from
sanity.
Check.
Tells
you
that
scheduler,
it's
not
working
to
schedule
or
use
up
too
much
resource
too
many
painting,
mods
or
no
paws
get
scheduled,
and
then
there
are
some
analysis
tools
you
can
use.
For
example,
scheduler
metrics.
There
are
a
lot
of
useful
informations
in
there.
I
know
they
added
a
lot
of
new
useful
metrics.
B
Lately,
then,
you
can
also
check
the
scheduling,
related
events
for
specific
parts
at
York
you
care
about
and
see.
If
everything
looks
fine
and
then
you
can
always
check
scheduler
log
as
well,
and
the
route
costs
could
be
from
any
of
the
things
that
we've
just
talked
about
and
last
one.
This
is
a
message
that
I
want
to
delivered.
I
just
want
to
repeat
it
again.
B
If
the
race
scheduler
reports
your
pot
as
unschedulable,
it's
really
right
and
we'll
try
it's
best
to
tell
you
why
not
the
trick
is
to
listen
to
it
and
use
that
as
a
clue
to
find
out
why
your
control
plan
is
acting
funky
and
there
is
a
very
good
talk
given
by
Marek
hrabowski.
He
saw
a
colleague
he
is
a
real
control
plan
expert.
He
had
the
from
last
year's
coupon
on
the
scheduling
features
you
can
check
online
and
with
all
this
nice
tricks
that
Joe
and
I
just
talked
about.
B
Let's
go
back
to
the
very
beginning,
but
won't
you
sing
with
me
Ally
to
the
talk.
Why
do
we
need
to
come
to
this
top
because
we
want
to
keep
our
control
plane
healthy
right?
What
if
we
do
not
need
to
use
any
of
the
trick
at
all
and
we
have
a
healthy
control
plane
to
run
all
the
time.
So
what
are
the
puzzle
pieces
that
we
could
put
together
beforehand
to
have
a
healthy
control
plane
running?
B
We
won't
have
time
to
talk
about
each
in
detail,
but
at
least
give
you
some
basic
resource
you
can
check.
After
this
security
is
definitely
play
big
part
in
this
picture,
and
you
can
check
this
talk
about
kubernetes
for
enterprise
security
requirements.
It's
given
in
the
Google
cloud
next
conference
and
the
next
one,
as
I
mentioned
in
one
of
the
example
and
an
accountable
monitoring
alerting
system
is
very
important.
B
There
are
a
lot
of
nice
product
outside
that
you
can
find
one
that
fits
your
needs
or
you
can
read
this
coronet,
it's
official
documentation
and
build
your
own
resource
called
some
people
talk
about
resource
or
coder.
Before
this
talk,
so
there
is
a
series
of
video
video
called
kubernetes
best
practice.
One
of
it
is
to
teach
you
how
to
set
resource
requests
and
limits
in
corollaries
and
then
for
scalability.
B
You
do
want
to
have
a
proper
size
of
master
and
cluster
or
your
notes,
and
in
the
kubernetes
official
documentation
there
are
some
informations
about
that
and
then
last
but
not
least,
upgrading
and
backup
is
important.
You
hear
me
mention
a
couple
of
times
in
our
talk.
Upgrading
upgrading
and
the
good
news
is
the
best
resource
is
hearing.
This
could
come
tomorrow
at
11:30.
Our
colleague
Chow
is
gonna.
Give
the
talk
here
is
everybody,
so
it's
about
it's
purely
about
safely.
B
C
I
have
a
question:
we
always
find
some
problem
by
the
condition
and
that
you
went
on
the
cabarrus
object
path,
sometimes
some
coral
components,
errors
hide
in
the
lock,
not
in
events
or
conditions.
So
it's
very
hard
to
find
the
relationship
between
the
between
the
conditions
and
the
component
I
think.
Maybe
we
should
define
a
different
kind
of
event,
such
as
cluster
or
component
to
attach
the
errors
in
the
coral
component
to
the
specific
event.
What's
your
opinion
about
this.
A
Still
processing
but
I
think
it's
worth
talking
about.
I
think
one
of
the
reasons
why
we
give
presentations
like
this
is
because
we
want
the
shelf
life
of
them
to
be
short,
I
would
like
this
presentation
become
obsolete
because
we
make
kubernetes
better
so
yeah
I'd
love
to
talk
about
things
like
that.
I
think
what
you've
mentioned
sounds
like
a
gap
to
me,
something
we
could
improve.
D
So
I
think
you
guys
might
have
a
lot
of
experiences
with
maintaining
an
operation
for
a
very
large
clusters
and
also
very
complex
system.
So
what
is
the
best
practice
when
you
have
a
very
tricky
problems?
And
you
don't
know
where
to
look
to
look
at?
Maybe
it's
a
network
or
its
infrastructure
or
anything
within
the
class
or
so
any
best
practice
for
that
part,
including
monitoring
I,
would.
A
A
What
one
one
piece
of
feedback
I
might
give
to
people
is,
if
you
run
into
an
issue
where
you
end
up
spending
a
lot
of
time,
just
fried
trying
to
find
that
first
breadcrumb
spend
some
time
afterwards.
Thinking
about
why
that
was
what
monitoring
was
missing.
What
infrastructure
was
missing
to
trace
that
problem?
Basically
do
kind
of
a
post-mortem
on
that
issue
and
use
that
to
improve
your
monitoring
and
your
your
your
first
stage
of
identifying
an
issue
and
tracing
it
so
that
next
time
you're
in
you're
in
better
shape.
B
One
thing
I
want
to
add
is
that
you
can
see
there
are
so
many
different
and
enterprise
using
it
and
we
people
from
STAR
Labs
all
the
way
to
the
big
company
like
Google.
We
even
have
internal
use
cases
for
kubernetes,
so
the
use
case
are
so
different
that
it's
really
hard
to
find
a
standard
to
like
how
to
use
kubernetes
in
a
certain
way.
B
But
one
thing
that
is
that
you
can
definitely
find
other
people
who
use
it
the
same,
probably
similar
way
as
you
are,
and
if
you
post
questions
on
the
kubernetes,
github
or
Stack
Overflow
you're,
probably
going
to
find
some
answers,
and
during
the
issue
discussion,
you're,
probably
going
to
find
the
fix.
You
can
contribute
to
the
community.
E
A
Gonna
give
a
talk
about
this
at
coop,
can't
see
at
all
so
watch
the
slides
when
we
present
those
I
think
a
lot
about
high
availability
at
CD.
There
are
certain
minimum
same
versions
that
I've
talked
to
the
community
about
before
and
I'm
trying
to
think
at
the
high
level.
What
I
can
tell
you
it's
it's
a
detailed
topic,
maybe
maybe
we'll
save
that
for
the
Seattle
conference.
Okay,.