►
From YouTube: Debugging etcd - Joe Betz & Jingyi Hu, Google
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Debugging etcd - Joe Betz & Jingyi Hu, Google
Oh no, something’s wrong with my Kubernetes cluster! Was it something I did? Or maybe one of those controllers we installed? The answers might be hidden in the etcd’s logs or data files. We’ll walk through real-world examples of how to make sense of etcd logs and correlate issues back to the commands or controllers that initiated them. We’ll cover tools we use to get answers from etcd’s data files, what to look for, and how to quickly get a cluster healthy again.
To Learn More: https://sched.co/GrYJ
A
A
We'll
start
the
talk
today
with
a
brief
recap
of
that
CD
I
won't
spend
too
much
time
on
that
and
then
we'll
dive
down
a
little
more
into
some
components
of
how
Etsy
DS
serves
and
stores
data.
It's
essential
to
understand
that
to
be
able
to
inform
yourself
when
you're,
looking
at
error
messages
and
logs
and
metrics
to
be
able
to
make
sense
of
what
they're
trying
to
tell
you
well
then
spend
a
few
minutes
on
some
of
the
main
tools
that
we
use
for
debugging
at
CD.
A
Some
of
the
things
have
been
really
helpful
for
us.
Some
things,
we've
built
or
improved,
will
then
dive
into
some
techniques
that
we
found
to
be
most
effective
at
getting
from.
You
know
the
initial
issue
report
down
to
the
cause
of
a
problem
and
we'll
close
out
the
talk
with
some
proactive
measures.
You
can
use
to
keep
your
clusters
healthy.
A
So,
let's
just
start
with
a
brief
recap
of
that
CD.
This
is
my
opinion.
You
could
say
that
CD
is
a
distributed,
key-value
store,
and
that
would
be
true,
but
it
doesn't
really
capture
much
detail
about
what
we're
trying
to
say
at
CD
is:
there
are
many
distributed,
key-value
stores
some
does
are
designed
to
store
petabytes
of
data.
That's
not
at
CDs
purpose.
A
Another
thing
you
could
say
about
it
is
that
it's
a
consensus
data
store
and
that
refers
to
the
fact
that
it
relies
on
the
RAF
consensus
algorithm
to
replicate
the
state
to
all
the
members
of
a
cluster
for
high
availability,
but
again
we're
not
totally
getting
to
the
intent
of
the
system.
So
the
way
I
like
to
think
about
it
is
it's
a
system
to
reliably
manage
the
coordination
state
of
a
distributed
system.
A
So
what
we're
really
saying
here
is
that
we're
gonna
build
bigger,
distributed
systems
on
top
of
a
TD,
and
it's
gonna
provide
that
core
coordination
in
the
example
of
kubernetes.
It's
storing
all
of
your
resource
objects
and
providing
a
way
so
that
users
and
the
various
components
of
kubernetes
can
all
interact
with
those
resources
to
share
state
and
coordinate
their
behavior.
It's
not
the
first
of
these
systems.
A
A
They
are
being
used
for
coordination,
so
they
require
a
strong
consistency
model
and
typically,
the
distributed
system
has
many
components
in
it,
all
of
which
need
to
observe
some
subset
of
the
state
in
this
store,
and
so
there's
usually
some
kind
of
watch
mechanism
that
allows
them
to
subscribe
and
receive
near
real-time
updates
to
some
set
of
that
data
so
that
they
have
fast
access
to
it.
Lastly,
these
systems
typically
include
a
variety
of
concurrency
control
primitives.
So
this
could
be
a
check
and
set
operation
to
allow
you
to
do
optimistic,
concurrency
control.
A
It
could
be
leases
to
allow
you
to
build
a
leader
election
things
like
that.
Just
really
briefly
about
raft.
This
is
a
whole
topic
and
I
can't
cover
it
in
this
in
this
presentation
and
give
it
full
justice,
but
raft
is
the
concurrency.
Is
the
consensus
algorithm
that
city
uses
and
its
leader
election
based?
So
when
you
see
things
in
that
city
about
leader
election,
we're
talking
about
raft
and
the
way
that
it
basically
works
is
that
you
require
a
cluster
to
have
a
majority
vote
on
who
the
leader
is.
A
So
if
you
have
a
cluster
of
size
3,
the
majority
would
be
too
and
as
long
as
two
members
can
vote
and
agree
on
the
leader,
the
system
is
available.
One
of
those
members
could
fail.
It
could
be
because
it's
you
know
been
partitioned
from
the
network
or
the
machine
is
crashed
and
the
system
will
can
still
continue
to
function.
Sometimes,
and
so
three
is
a
common
number
of
members
in
that
city
cluster
that
you
would
run
the
other.
A
Most
common
number
is
five,
because
now
you've
gotten
up
to
a
failure,
tolerance
of
two
that
could
allow
you
to
have
one
planned
failure.
So,
for
example,
if
you're
planning
an
upgrade,
you
know,
you'll
lose
one
member
ask
you
do
that
rolling
upgrade,
and
then
you
can
also,
at
the
same
time,
support
one,
unplanned
failure
and
still
be
available.
A
A
The
major
API
endpoints
are
provided
through
G
RPC,
and
it's
also
worth
noting
that,
if
you
want
to
explore
those
directly
there,
there
re-exposed,
in
addition
to
the
high-performance
binary
protocol,
they're
available
at
the
JSON
HTTP
endpoint
and
then
the
last
major
subsystem
is
both
DB
SED
needs
a
backing
sort
of
persistence,
data
to
disk
and
it
uses
the
system
for
that.
We'll
talk
a
little
bit
more
about
it
later.
A
Let's
talk
briefly
about
how
sed
serves
and
stores
data.
This
is
a
sampling
of
a
key
space.
This
is
from
kubernetes
and
the
key
space
is
organized
hierarchically.
You
can
see
that
there
are
the
first
prefix
you
can
kind
of
ignore,
but
the
second,
the
second
directory,
is
the
kubernetes
types
and
then
followed
by
the
namespaces
and
the
resource.
A
Names
and
kubernetes
chose
this
because
at
CD
provides
an
index
over
the
key
space
which
is
lexically
ordered,
and
that
allows
you
to
do
what's
called
a
range
read
which
you
can
get
you,
basically
anything
that
has
the
same
common
prefix
or
anything
it's
in
in
this
example.
Anything
that's
in
the
same
subdirectory
that
allows
you
to,
for
example,
here
you
could
get
all
of
the
replicas
sets
with
a
single
query
or
you
could
get
all
of
the
replicas
sets
for
a
particular
namespacing.
We're
note.
A
A
Here's
a
couple
of
the
other
major
at
CD
operations
to
watch
out
for
there's
a
single
write,
operation
called
put
delete,
works
kind
of
like
the
range
read
and
that
you
can
delete
multiple
entries
in
a
single
request.
If
you
need
it
and
then
there's
an
interesting
operation
called
transaction,
this
is
a
multi-purpose
operation.
What
it
includes
is
a
condition
and
zero
or
more
operations
that
you
were
going
to
execute.
A
If
that
condition
is
true
and
zero
more
that
you
can
execute
if
it's
false,
so,
for
example,
you
could
use
this
as
a
check
and
set
operation.
You
could
say
if
the
value
of
some
key
is
what
I
expect
it
to
be.
Only
then
updated
which
would
prevent
you
from
accidentally
clobbering
anybody
else's
rights,
or
you
could
use
it
as
a
way
of
performing
multiple
operations.
Atomically,
you
could
just
hardwire
the
condition
to
be
true,
and
then
you
could
put
to
put
rights
in
that
operation.
A
All
of
these
operations
are
what
we
call
linearize
ibly
consistent,
it's
kind
of
a
big
word
another
term,
for
it
is
external
consistency
and,
if
you're
unfamiliar
with
that,
that
means
I'll
just
give
you
a
brief
intuition.
It's
a
fairly
strong
form
of
consistency.
Where
say,
you
have
a
client
and
it
performs
a
put
on
one
member
in
that
CD
cluster
and
then
subsequently
it
gets
a
response
back
acknowledging
that
that
put
was
successful.
It
then
does
a
read
on
maybe
a
different
member
of
that
cluster.
A
Is
it
guaranteed
to
see
the
write
that
it
performed
previously
and
the
answer
is
yes,
it's
guaranteed
to
see
its
own
writes
in
the
absence,
of
course,
any
other
clients
interacting
here,
because
those
are
not
atomic
operations.
So
if
another
client
jumped
in
and
wrote
in
between,
it
could
but
it's
guaranteed
to
logically
see
a
view
of
the
world
from
after
its
own
right.
Those
things
happen
in
sequence,
a
other
type
of
major
type
of
operation.
That
HCD
house
is
a
watch.
A
This
is
a
streaming
operation
and
what
at
CD
does
is
it
establishes
a
G
RPC,
bi-directional
stream
with
a
server,
and
then
it
sends
requests
one
way
and
receives
updates
in
the
other
direction.
So
it
can
ask
to
establish,
watches
over
a
particular
agents
of
the
key
space,
and
then
it
will
eventually
receive
acknowledgments
that
the
watch
has
been
created
and
then
receive
events
for
any
changes
to
that
region
of
the
key
space
in
order.
A
This
is,
of
course,
is
eventually
consistent,
so
just
to
follow.
Through
with
that
example
say
we
have
a
client
that
client
has
a
cache
and
we've
configured
that
cache
to
be
updated.
Using
a
watch
from
Etsy
D,
a
particular
member,
but
then
if
we
were
to
again
perform
a
right
to
a
different
member
and
receive
an
acknowledgement
that
that
rights
been
completed.
A
If
we
go
to
our
cache
or
guaranteed,
to
see
that
right
well,
know,
there's
a
race
right,
it
depends
if,
if
the
event
processes
through
the
system
in
time
before
I
read,
is,
is
mix
it
to
the
cache,
then
we
are
guaranteed.
Then
we
will
see
the
right,
but
that's
not
guaranteed.
It
could
be
that
the
read
happens
first,
so
you
need
to
be
aware
of
those
different
consistency
models
for
these
different
operations.
A
I'm
gonna
dive
down
into
just
a
couple
elements
of
storage
and
we'll
talk
a
little
more
about
debugging
when
debugging
sed
it's
important
to
understand
that
it
retains
a
global
revision
number
and
every
time
you
perform
an
operation,
it
increments
that
number
and
it
keeps
the
history
of
all
the
objects.
So
it
doesn't
do
in
place
updates
of
objects,
it's
just
kind
of
a
copy-on-write
mechanism,
and
we
refer
to
this
the
the
this
ability
to
keep
a
history
and
be
able
to
read
things
in
history.
A
As
a
multi-version
concurrency
control
system-
and
it
exact
aspect
of
the
system
exists
in
two
layers,
the
most
obvious
one
is
up
in
the
ad
key
space.
Well,
you
can
actually
see
the
old
revisions
and
where
the
watches
can
be
replay
history
by
traversing
that
history
of
objects
there's
also
a
second,
more
subtle
layer
of
multi-version,
concurrency
control,
hidden
down
on
both
DB.
This
enables
the
storage
layer
to
execute
well
at
least
one
read
or
one
write
and
as
many
reads
as
it
needs
concurrently,
most
data
systems
would
be
exclusive.
A
You
could
do
one
read
or
in
right
or
sorry,
one
write
or
n
reads,
but
not
one
write
and
and
reads,
and
it
does
that
by
keeping
a
history
of
objects
that
it
compacts.
Why
am
I
telling
you
this
well
there?
This
actually
shows
up
in
that
CD
when
you're
debugging
it
there's
two
major
operations
that
show
up
one
is
called
compaction
and
compaction
is
the
mechanism
by
which
sed
goes
through
and
removes
old
version
of
revisions.
Once
you
no
longer
need
them,
you
have
to
tell
it
what
you
want
it
to
do.
A
It'll
keep
everything
by
default
and
you
can
configure
it
to
automatically
compact
all
the
revisions
or,
if
you're
using
kubernetes,
the
client
will
actually
request
the
compactions
automatically
in
the
case
of
kubernetes
every
five
minutes,
it
will
request
that
everything's
older
than
five
minutes
is
removed
and
kubernetes
is
tolerant
to
not
having
a
full
revision
history
so
that
works
fine
down
at
the
bolt
DB
layer.
There's
some
potential
for
fragmentation.
A
If
you
look
at
the
sed
data
directory,
you'll
see
this
and
you
might
get
a
little
bit
intimidated,
but
when
debugging,
this
is
a
great
way
to
get
answers
to
questions
so
I'm
going
to
point
out
a
couple
things
one
is
that
main
DB
file
under
snap?
That
is
the
mean
file
that
live
reads
are
performed
from
and
then
there's
these
snap
files
and
wall
files
and
it's
better
to
explain
those
through
kind
of
a
logical
view
of
the
system.
So,
let's
start
with
the
wall
file,
the
wall
file
stands
for
right
ahead
blog.
A
A
Of
course,
you
can't
really
effectively
serve
rights
from
a
file
light
or
reads
from
a
file
like
that.
So
at
CD
immediately,
then
also
writes
that
to
the
main
DB
file,
which
is
efficient
for
performing
reads,
of
course,
at
CD
could
crash
between
when
that
first
write
occurs
and
the
second
one
does.
So,
if
it
does,
it's
gonna
need
to
replay
the
wall
file
to
recreate
the
DB
file.
That's
how
it
recovers.
A
There's
another
problem
with
this,
which
is
that
wall
file
would
grow
indefinitely
over
time
and
that's
not
acceptable.
So
what
we
need
to
do
is
we
need
to
keep
it
from
doing
that
and
Etsy
does
that
by
periodically
introducing
snapshots
yeah,
it
writes
a
copy
of
the
key
space
to
a
file,
a
periodic
cleaning
associated
that
with
the
exact
position
in
the
log
and
then,
if
it
needs
to
recover,
it
finds
the
newest
snapshot
loads
that
and
then
replays
the
remainder
of
the
wall.
A
To
recreate
this
state
all
right
now
we
have
some
intuition
about
some
details
in
Etsy
dee.
Let's
talk
about
a
couple
of
the
main
tools
that
we
use
at
CD.
Cuddle
is
one
of
the
most
well
known
tools
and
it's
actually
pretty
useful
for
debugging
as
well.
You
can
use
it
to
examine
the
state
of
a
key
space,
see
what's
in
there
oftentimes,
when
you're
untried,
to
understand
why
it
cities
behaving
on
a
certain
way
under
certain
systems.
A
Knowing
you
know
how
many
keys
you
have
do
you
have
like
a
million
jobs
stored
in
a
kubernetes
database
and
expensive
ragin
trees
against
those
is
that
the
problem
with
just
looking
at
the
key
space
and
glassy
and
I
can
really
helpful
about
kubernetes
is
efficient.
It
stores
things
in
in
a
binary
protocol.
So
if
you
actually
try
and
look
at
a
particular
object,
you
might
run
run
into
trouble.
You
might
see
something
like
this,
so
we've
written
a
tool
called
augur.
A
It's
a
pretty
simple
tool
and
what
it
does
is
it
allows
you
to
examine
kubernetes
objects
in
sed,
it's
aware
of
all
the
kubernetes
datatypes
and
you
can
easily
just
add
it
to
that
previous
command,
that
we
ran
and
decode
that
back
into
Hamill
or
JSON
whatever
you
want.
That's
handy,
it's
even
more
handy.
Sometimes,
if
say
you
have
a
user
who
accidentally
deleted
an
object
in
the
kubernetes
cluster,
they
might
want
that
object
back.
A
One
thing
you
could
do
is
find
the
newest
backup
you
have
where
that
object
hasn't
been
deleted
yet
and
use
augur
to
just
directly
extract
that
one
object
out
of
that
backup
file
for
them.
It's
a
great
way
to
remediate
an
issue
without
too
many
steps.
The
other
tool
I'll
mention
is
called
Etsy
dee-dum
plugs.
This
is
a
great
way
to
look
at
the
wall
file,
the
other
of
the
two
main
file
types
we
had.
You
can
see
every
operation
that
occurs
in
Etsy
D
in
the
order
that
they
occurred.
A
This
is
going
to
be
useful
for
trying
to
exactly
what's
gone
wrong.
We've
even
added
a
utility
to
this
that
I
haven't
shown
here
where
you
can
have
it
decode
the
value
types
using
augur,
so
you
can
see
like,
for
example,
we've
got
some
severely
escaped
kubernetes
data
in
here
you
could.
You
could
see
that,
in
its
in
a
more
readable
format,.
B
Thanks
sure,
so,
I'll
talk
a
little
bit
about
the
debugging
approaches.
I'll
offer
show
you
this
graph.
It's
something
I
created
when
I
first
started
working
on
at
CDI
feel
like
I
need
some
systematic
way
of
debugging
when
a
cluster
is
not
functioning
very
well.
So
I
started
to
create
this
debugging
flow
graph,
initially
in
my
head,
but
now
it
ends
up
to
become
something
like
this.
It's
been
useful
to
me
so
I'm
happy
to
share
it
with
you.
You
can
see
starting
from
the
left
side
of
the
graph.
B
It
starts
with
some
some
of
these
sanity
checks
and
as
it
flows
towards
the
right-hand
side
of
the
graph,
it
leads
you
towards
the
root
cause
of
the
issue.
So,
let's
start
with
the
sanity
checks
is
at
city
running.
That's
probably
the
first
thing
I
want
to
check.
There
are
lots
of
ways
you
can
track.
The
first
thing
I
usually
do
is
the
example
here
at
cities
running
in
containers,
so
I'll
just
do
a
dark,
IPS
and
grab
the
keyword
you
can
see
you
can
see.
B
The
container
is
good
was
created
two
days
ago
and
spin
up
and
running.
Since
then,
the
next
example
is
sort
of
like
at
a
bad
example
where
you
can
see
it
was
created
two
days
ago,
but
it's
no
longer
running,
oh
by
the
way,
so
I
highlighted
issues
or
the
bad
examples
in
the
red
color
and
just
for
reference.
Another
check
you
can
do
is
so
the
LCD
has
the
health,
its
HTTP
endpoint
check.
If
you're
familiar
with
kubernetes,
you
use
the
cube
Caudill.
Some
people
call
it
cube.
B
Control
get
component
statuses,
as
city
is
one
of
the
component,
and
you
can
see
it
actually
carries
the
health
and
point
of
debt
CD,
and
in
this
case
it
returns
health.
True.
Alternatively,
you
can
just
require
that
endpoint
directly
using
curl
again.
This
is
a
good
example.
The
last
line
is
the
bad
example
where
the
client-
or
you
cannot
even
talk
to
the
server
you
can
see
the
connection
refused.
B
B
It
is
sort
of
like
a
weaker
check
as
compared
to
the
health
check
I
just
mentioned
because
as
part
of
the
health
check,
actually,
your
cluster
has
to
have
a
leader
and
it
has
quorum
as
part
of
that
check
actually
tries
to
do
a
quorum
right
so
for
that
to
success,
but
this
just
gives
you
the
status
of
the
individual
at
city.
Node.
Another
check
you
can
do
is
add
city
control,
member
list,
it
sorts
of
gives
you
the
view
of
from
each
at
City
node
in
the
cluster.
B
So,
for
example,
when
you
have
like
bad
connections
between
our
city
node
you,
you
suspect
that
there
might
be
a
network
or
partition.
You
can
use
this
to
check
each
one's
view
of
the
cluster.
So
what
might
be
the
so?
These
are
the
possible
outcomes.
After
you
run
those
checks,
you
might
find
that
the
server
fail
to
start.
B
If
you
follow
the
arrows
think
in
most
cases
it
could
be
due
to
you
have
a
bad
configuration
when
you
start
a
server
or
it
could
be
that
if
there
is
already
data
file
under
your
at
City
data
directory
at
when
at
City
stars
will
try
to
read
a
file
and
catch
up
to
state.
If
that
file
itself
is
bad
corrupted,
then
the
server
will
fail
to
start.
B
Another
possible
scenario
is
that
the
server
was
able
to
start.
You
can
see
it's
running,
but
then
crashed
I'll.
Give
you
an
example
here.
If
you
look
at
the
server
log,
I
sort
of
highlighted
in
red
the
capital
letter
C,
that
means
crash.
When
you
see
that
in
a
log
it's
a
bad
thing,
and
here
it's
an
example
where
it's
probably
too
much
detail,
but
it's
a
free
list.
Corruption
in
the
both
DB
data
data
database,
that's
been
used
by
@cd.
B
If
you're
interested
in
details,
I
have
included
a
link
to
the
actual
fix
of
that
issue.
Another
type
of
sanity
checks,
I,
usually
do
is
to
see
if
the
space
quota
is
exceeded.
There
are
a
couple
of
ways
you
can
see
that,
for
example,
the
first
command.
We
just
use
that
it's
a
city
control
endpoint
status,
one
of
the
field
it
returns
is
the
database
size.
You
can
see
here.
It
says
8
gigabytes,
that's
the
default
maximum
space
called
a
allowed
in
at
CD.
B
If
you
see
that
it
sort
of
tells
you
you
exceeded
or
you're
reaching
the
maximum
space
quota
or,
alternatively,
you
can
just
directly
go
to
the
ED
CD
data
directory
and
check
the
size
of
the
DB
file.
In
case
your
server
is
not
running,
or
it's
not
responding
to
your
to
your
query.
The
end
point
status
Kure.
You
can
just
check
the
size
of
that
file
directly.
There
are
a
couple
more
places
where
you
can
see
this
issue.
B
For
example,
if
you
do
STD
control
alarm
list
normally,
if
the
city
cluster
is
running
correctly,
there
will
be
no
alarms,
but
here
is
so
it
shows
no
space
alarm
or
if
you
are
issuing
a
command
request
from
the
client
to
the
server
and
the
server
responded
to
you
with
an
arrow
says
database
space
exceeded.
It
basically
means
what
it
says.
B
Another
type
of
sanity
checks,
I,
usually
do
is
to
see
if
there
are
requests
taken
to
taking
a
server
too
long
to
response
a
couple
of
ways
to
see
that
indirectly,
like
we're
issuing
we're
using
the
same
example
a
same
command,
sed
control,
endpoint
status.
In
this
case,
what
happens
is
that
your
command
actually
times
out?
So
there
is
a
default
of
five
seconds
timeout
for
each
of
these
commands.
B
In
this
case,
the
server's
not
responding
to
you
in
time,
so
your
clients
or
your
command
actually
times
out
or
alternatively,
if
you
check
the
the
server
log
and
search
for
the
keyword,
apply
entry
stick
alone,
you
can
see
here's
the
facts
and
hole
where
it's
applying
one
entries
but
took
like
12
seconds,
which
is
really
long,
so
these
are
I've
listed
for
most
likely
next
steps
or
for
most
likely
reasons
that
could
cause
this
issue.
They
are
actually
corresponding
to
the
graph.
If
I,
just
briefly
go
back
to
this
graph.
B
If
you'll
follow
the
arrows,
there
were
four
arrows,
which
are
basically
the
four
things
I
listed
here.
So
one
possible
reason
is
that
your
request
itself
is
too
large,
meaning
you're
reading
a
lot
of
things
or
you're
modifying
a
lot
of
things.
Another
possible
reason
is
that
the
server
is
overloaded,
so
your
your
request
might
be
small,
but
there
are
other
large
expensive.
B
That's
using
authorizing
all
the
server
resource.
Another
possible
reason
could
be
due
to
pure
disk
performance.
A
way
to
check
that
is
so
at
CDE.
Has
this
slash
matrix
endpoint
that
you
can
use,
expose
lots
of
matrix
I've
listed
two
matrix
that
are
directly
related
to
this
performance
here
and
it
could
be
networking
issue
so
I
think
so
far.
We've
talked
about
all
of
the
sanity
checks
listed
on
the
left
side.
Now
I
would
like
to
go
through
two
things
that
I
think
to
debugging
approaches.
I
think
are
mostly
common,
mostly
used.
B
First,
one
is
trace
the
request.
For
example,
you
might
start
with
the
sanity
checks
and
you
found
all
the
space
quota
it's
exceeded.
Then
you
check
the
data
stored
in
your
@
çd.
You
couldn't
understand
why
there
were
so
many
things
stored
in
in
it
or
you
can
understand.
Remember
each
each
right
request
actually
create
a
new
version
of
the
data
instead
of
in-place
update,
so
you
might
found
someone
is
frequently
updating
the
data
that
uses
all
the
space,
but
you
don't
know
who's,
actually,
writing
or
updating
that.
B
In
that
case,
you
might
need
to
trace
the
rec
request.
Another
example
would
be
your
starting
from
the
sanity
checks
and
you
found.
Oh
one
of
the
requests
is
taking
too
long
to
respond
and
the
request
itself
is
actually
really
large,
like
it's
probably
reading
a
lot
of
things,
and
you
don't
know
who
is
reading
that
in
that
case,
you
also
want
might
want
to
trace
the
request
here.
I
have
a
concrete
example
of
how
to
do
the
tracing.
B
This
is
an
example
from
the
kubernetes,
so
in
kubernetes,
the
only
component
that
talks
directly
to
at
CD
is
QB
API
server,
so
in
this
case,
naturally
we're
tracing
to
the
cube,
API
server
log.
So
in
the
log
I'll
describe
what
happens
here.
So
it
basically
says
there
is
a
single
request.
Take
the
API
server
33
seconds
to
finish,
and
there
are
two
major
parts
of
this
time.
The
first
part
is
takes
like
20
21
seconds,
which
is
server
reading
stuff
from
dead
CD.
B
The
next
step
is
transforming
those
readings
from
at
CD
to
the
HTTP
response,
which
is
then
sent
back
to
the
server
client.
So
the
first
step
is
kind
of
problematic.
You
can
see
it
takes
21
seconds
for
dead
city
server
to
respond,
and
if
you
look
carefully
closely,
you
can
see
the
request
actually
reads:
320
KN
items
from
at
CD
in
this
single
request.
B
So
this
is
how
you
can
tell
the
request
is
very
large,
and
if
you
want
to
trace
little
further,
the
log
actually
tells
you
it
is
from
a
controller
called
crown
job
controller.
So
so
the
request
is
from
the
cron
job
controller
to
the
kewpie
api
server.
Then
to
that
CD,
which
is
a
really
large
request.
If
you
want
to
trace
even
further,
you
can
look
at
the
source
code,
I've
included
them
here,
you
can
see.
This
line
is
actually
calling
the
listing.
B
So
if
you,
by
the
way,
if
you're
familiar
with
kubernetes,
you
can
see
at
this
line,
probably
can
be
optimized
a
little
bit
because
this
controller
here,
it's
not
using
sharing
former
it's
just
listing
directly
from
LCD
and
is
not
using
pagination
here.
This
is
an
example
of
how
do
you
trace
the
request?
B
Another
debugging
approaches
I
want
to
talk
about
is
inspect
data
chill
actually
already
mentioned
in
the
previous
slides.
To
use
augur
to
inspect
data
here,
I
guess
I
just
want
to
emphasize
that
it
really.
It's
really
important
to
remember.
There
are
like
three
dimensions
in
the
database.
How
many
objects
do
you
have
in
our
database?
B
How
big
are
those
objects
and
also
the
third
one
might
be
not
so
obvious,
but
majorities
try
to
emphasize
on
that
the
revisions
so
whenever,
whenever
you
try
to
update
an
object,
even
though
the
objects
already
in
@cd,
but
it
will
not
update
that
object
in
place,
it
will
create
a
new
version
or
a
new
copy
of
that.
So,
even
though
you
don't,
you
might
not
have
a
lot
of
objects
and
they
individually
they're,
not
so
big,
but
if
we
were
updating
them
very
frequently,
you
can
still
use
a
lot
of
space
to
be
precise.
B
B
This
this
part
I,
want
you
to
talk
about
the
relation
between
the
workload
and
the
data
volume.
The
workload
I
mean
the
server
requests.
Like
the
example
requests
we
just
saw
earlier
so
I
think
it's
I
want
to
I
want
to
mention
that
the
workload
can
increase.
The
volume
like
I
just
described.
Each
right
operation
creates
a
new
version
and
it
might
not
be
so
obvious
in
the
reverse.
Data
volume
can
can
also
increase
the
workload.
B
For
example,
if
you
have
lots
of
stuff
stored
in
at
CD
and
either
intentionally
or
accidentally,
you're
reading
a
lot
of
a
lot
of
them,
then
it's
is
creating
a
huge
large
request,
for
this
is
the
same
example
I
just
copied
from
the
previous
slides,
I
sort
of
highlight
a
different
part.
It's
it's
enlist
request.
It
actually
requests
to
list
all
the
job
objects
in
at
CD,
but
you
might
not
aware
that
there
are
320k
job
objects
in
your
ad
CD,
so
accidentally
in
you
might
create
a
huge
request.
B
All
right
so
now,
I
want
to
briefly
talk
about
keeping
your
ad
City
healthy.
These
are
some
I
guess
some
recommendations
from
us.
First,
as
Astro
mentioned
previously,
as
CDE
support
promises
and
I
mentioned
earlier,
there
is
an
matrix
endpoint
which
you
can
use.
We
recommend
that
you
monitor
these
metrics
carefully,
maybe
select
the
the
ones
that
are
most
important
to
you
and
create
a
dashboard
like
this.
B
We
also
recommend
that
you
use
the
officially
maintained
versions,
we're
currently
maintaining
three
versions:
three
minor
versions-
3.1
3.2
and
3.3-
and
there
is
a
minimum
recommended
versions
sent
to
the
at
City
dev
mailing
list.
Earlier
this
year,
backups,
we
recommend
you
do
backups
for
discovery
out
for
disaster
recovery
purpose.
We
also
recommend
that
you
do
per
backup
check,
meaning
that
whenever
you
got
a
backup
new
back
hub,
you
want
to
do
some
integrity
check
on
that.
B
If
you're
using
the
newer
at
city
control
versions,
you
can
just
do
sed
control
snapshot
status
as
part
of
it.
It
will
do
this,
the
track
for
you
or
because
the
underlying
database
is
ball.
Db
you
can
use.
There
is
a
command
line
tool
called
B
bolt.
You
can
also
use
that.
We
also
recommend
that
you
regularly
verify
you
can
actually
restore
your
cluster
from
the
backup
file
created
upgrades
again.
We
recommend
you
upgrade
to
the
officially
maintained
versions.
B
Upgrade
itself
is
not
trivial,
but
there
is
a
well-documented
process
on
the
github
document
downgrades.
Currently,
the
only
possible
way
to
downgrade
is,
if
you
back
up
everything
before
upgrade,
but
there
is
ongoing
support
for
one
minor
version
downgrade
if
you
want
to
track
the
process,
it's
in
that
github
issue.
B
One
last
thing:
it's
very
important,
I
think
at
City
Community
needs
your
help.
If
you
want
to
help
there
are
some
ways
we
can
reach
the
current
people
who
are
working
on
on
CD
I
want
to
mention
we
have
a
community
meeting
where
I
think
it's
currently
Paul's
during
the
holiday
seasons,
but
we're
going
to
resume
it
early
next
year,
early
January
with
that.
Thank
you.
A
A
What's
the
recommended
frequency
of
@cd
backups
I'm,
not
sure
I
can
give
one
number
to
that.
I
think
it's
a
little
use
case
dependent
but
I
will
say
a
couple
things
less
than
once.
An
hour
seems
rare
to
me
and
there
are
critical
moments
where
you
will
most
want
backups
right
before
you
do
an
upgrade,
probably
right
before
you
do
a
rollback.
If
you
have
to
do
a
rollback
from
an
upgrade
and
the
upgrade
I
would
include
not
only
a
net
CD
upgrade,
but
a
system
upgrade
on
top
of
it.
B
I,
don't
think
currently
at
CD
is
emitting
a
lot
of
things
into
the
Lord.
Maybe
you
turn
on
the
debug
flag.
I
don't
know,
but
usually,
if
you
look
at
the
starting
from
the
line,
you
will
see
if
it's
information
which
it's
I
or
a
warning,
a
w.
Those
are
I
mean
more
worse.
Thing
is
II
era
or
even
worse.
Ii
crash.
B
A
Fortunately,
that
is
very
credible
because,
right
at
the
beginning,
right
before
that
pipe
operator,
you've
got
that
single
character.
It
identifies
it,
so
you
could
use
that
or
write
your
own
tool
to
pipe
through
it.
I
believe
there
are
also
some
log
level
settings
that
you
could
tie
with
a
net
CD
if
you
need,
but
that's
typically
to
get
more
and
not
less
and
I.
Think
you
want
less
so
you
might
want
to
just
pipe
it
through
a
tool
that
filters.
If
that's
what
you
need.
A
C
B
A
The
question
was
for
the
disc
to
slow.
How
do
you,
how
do
you
know
if
your
disc
is
to
slow?
What
could
you
look
at?
How
would
you
learn
more
about
that
yeah
I
would
say
the
the
the
ops
metrics
are
one
of
the
great
things
to
watch.
If
you
see
that
latency
reaching
a
limit
there,
that
is
a
pretty
strong
indicator
and
that's
a
metric
that
you
could
alarm
on
that.
You've
probably
exceeded
the
limits
of
your
disk
there's.
Also
an
additional
section.
A
A
Yeah,
the
question
was
you:
can
you
can
create
a
lot
of
pods
in
kubernetes
and
that
can
end
up
overloading
at
CD?
Is
there
work
being
done
to
improve
things
like
this?
The
answer
is
yes,
that's
why
some
of
us,
or
both
of
us
actually
are
commuters
on
both
projects?
In
that
specific
example,
the
pod
sent
heartbeats
to
kubernetes,
which
puts
load
on
at
CD.
There
is
an
alpha
status,
fix
to
move
that
out
into
a
separate
object.
A
That
is
much
cheaper
to
update,
and
that's
that's
really
how
we
fix
some
of
these
things
you
can.
We
could
try
and
get
more
capacity
out
of
that
city
and
improve
its
scalability
and
performance,
but
if
the
load
that's
coming
to
it
is
unreasonable.
We
need
to
go
back
and
improve
that
load.
Absolutely.
A
A
You
can
reconfigure
the
Kubb
API
server
to
compact.
More
often,
if
you
need,
you
can
also
do
your
own
manual
compactions,
but
I
would
kind
of
recommend
if
you're
using
kubernetes,
to
work
with
it
to
do
the
compactions
it's
it
can
tolerate
the
loss
of
historical
data,
so
it
should
be
safe,
but
I've
never
done
it.
A
A
Well,
one
of
the
reasons
is,
if
you,
if
you
do
a
snapshot
right
at
the
tail
end
right
before
compaction,
happens
and
there's
a
lot
of
activity
in
that
cluster,
there
could
be
a
lot
of
revision
counts
and
then,
if
you
did
one
right
after
the
compaction
happen,
you
could
see
a
drastically
lower
number,
because
it's
you
know
if
you're
doing
it
every
five
minutes,
removing
everything
all
there
five
minutes,
that
could
be
a
2
X
interval
and
if
you're
doing
something
different,
it
could
be
even
bigger.
Take
one
more.
A
Yes,
large
compactions
can
get
expensive.
There's
been
work
to
make
those
increasingly
concurrent,
but
I
would
still
say
that
is
not
an
entirely
solved
problem
in
NCD.
Making
those
fully
concurrent
and
low
priority
with
so
that
they're
completely
invisible
is,
is
an
aspirational
goal.
That's
not
been
achieved
and
yeah.
So
if
you
are
seeing
large
compactions,
you
will
see
them
written
in
the
logs
and
you
will
see
how
long
they
took
and
you
may
see
latency
spikes
that
are
visible
up
through
the
system
from
there.
So
I
would
keep
an
eye
on
that.