►
From YouTube: Production Cluster Monitoring and Remediation for High Reliability - Shijun Qian & YingKe Liu
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Production Cluster Monitoring and Remediation for High Reliability at eBay - Shijun Qian & YingKe Liu, eBay
eBay runs dozens of Kubernetes clusters across global data centers in different regions. Tens of thousands of nodes support eBay core services such as search and big data. Complex large cross-regional production clusters and the extremely high cluster stability required workloads make monitoring and remediation a huge challenge for us. Based on Prometheus federation, component assertions, metric exporters and our own monitoring tools, we built a series of clear dashboards, and then we implemented a complete cross-clusters remediation flow, incident management, and monitoring automation. In this talk, we hope to share our large-scale Kubernetes production clusters monitoring experience and future thoughts.
To learn more click here: https://sched.co/FuK6
A
A
Okay,
here
is
the
agenda.
There
will
be
two
parts:
the
first
one
I
will
introduce
the
eBay
production
monitoring
in
our
drawing
turbinates
clusters,
then
in
her
will
introduce
hobby
remediation
based
on
over
monitoring
at
first.
Let
me
briefly
introduce
the
I
paid
for
manages
clusters.
Ebay
was
early
birth
to
adopt
provenance
from
the
beginning
of
2015.
We
want
to
use
the
turbinates
as
a
unified
cloud
infrastructure
to
manger
all
of
the
fleets
in
eBay.
Today
we
have
built
more
than
13
clusters,
including
over
8000
irnos,
including
the
PM's
and
the
VMS.
A
We
support
over
3000
eBay
applications
and
the
100,000
pods,
so
how
to
mention
this
amount
of
clusters
and
pose
a
huge
challenge
for
us.
What
we
need
is
a
reliable
scalable
and
the
intelligent
monitoring
system
here
is
over
mentoring,
girls,
the
first
growth
we
should
support
the
control
plane
management,
including
a
pair
server,
etcd
stage
error
and
the
loss
of
all
controllers.
There
is
a
data
plan
management
we
should
including
the
node
left
side.
Whole
pole,
left
central
and
the
other
objects
lost
lifecycle
management.
The
third
girl
is
a
lotta
manager.
A
A
Okay
in
order
to
reach
this
course
here
is
the
overview
of
our
monitoring
system.
The
best
part
of
monitoring
is
logging
and
matrix.
We
Clara
team
was
formed
in
Japan
and
at
airplane.
Based
on
these
two
parts,
we
still
have
a
serious
of
a
lotta
rules
to
notify
our
DevOps.
The
trust
events
best
are
beside
offices
with
the
drawing
of
trousers
manger.
Monitoring
or
close
the
fastest
is
a
bit
burden,
so
we
implement
a
way
to
automate
it
manager,
overload
rules
and
monitoring
rated
at
operations
and
specs.
A
At
last,
we
have
some
practice
in
aerobes
based
on
loads
and
matrix.
Next
I
will
introduce
them
one
by
one.
Okay,
the
first
part
is
a
hobby
loading.
Everybody
knows
that
the
importance
of
loading
without
a
lot.
It
will
be
very
hard
to
d2
the
load
cause
of
each
program
so
and
eBay
we
deployed
enhancement
to
the
fire,
beats
as
demo
set
running
in
each
nose
Clayton
loads.
There
are
two
kinds
of
lots
in
faster.
The
first
hand
is
lots
from
containers
which
can
write
easier
lots
in
to
send
out
or
say
narrow.
A
These
loads
can
be
easily
related
by
therapies,
because
verbs
can
automatically
describes
pores
and
containers
in
the
node
and
finding
the
Lord's
passed
and
it
central
atom
the
other
hand
of
Lords
our
faith
student
Lord,
including
a
country
enlarge
to
greater
rod,
an
eternal
Lord.
In
addition,
if
container
cannot
write
lots
into
the
stand
out
of
thin,
air
user
can
also
speed,
find
bits.
The
Lord
passed
as
a
notation
in
the
pod
and
the
way
we
are
mounted
it
into
the
host.
A
The
the
therapies
can't
relate
them
as
well,
by
the
way
in
Aden
to
prevent
the
Lord's
to
take
up
too
much
disk.
We
enable
it
a
lot
of
rotate
to
automatically
change
order
loads.
So
now
we
can't
alleged
Lords
from
knows
and
Ron.
He
knows,
then.
We
push.
The
Lord's
include.
Half
turn
and
there
will
be
a
streaming
process
based
on
the
Rings
to
do
some
data
processing,
language,
Raj
summarize
and
other
words.
After
this
process
there
will
be
a
resource
manager
to
tell
us
where
the
Lord
should
be
stored.
A
Currently,
we
support
two
kinds
of
saw:
Reggie
Benton's
alert
research
was
use
the
tool
storage,
the
recently
launched
under
HTTPS,
which
used
for
long-term
storage,
patient.
Ok,
there
is
a
how
we
show
these
Lots.
We
build
a
load
console
based
on
Cabana
when
user
Prairie,
Lourdes
Lourdes
from
Sofia,
or
also
finds
a
storage
from
the
resource
manger
and
display
them
in
a
friendly
way.
For
example,
this
is
over
lord
console.
We
can
choose
the
type
of
rods
we
want
like
to
stand
out
or
say
an
arrow
or
both
of
them.
A
Then
we
can
also
choose
which
transfers
which
namespace,
which
port
and
which
containers
then
we
can
get
their
lords.
We
want
is
right
all
right.
Okay,
now
we
can
tear
Lots,
but
only
loss
is
not
enough
for
monitoring.
If
we
want
to
rotate
the
trustor
wrong
time
status,
we
must
also
collect
matrix
from
the
previous
slides.
We
already
knows
at
eBay
monitoring,
ship's
port
concha
prime
management,
which
needs
to
know
the
ApS
server
and
stay
during
latency,
etcd,
disuse,
age
and
so
on.
Next
is
an
old
website
arrangement.
A
We
should
a
nodes
and
not
already
knows
stay
tuned
in
disable,
honest
and
northern
resource
usage
and
other
status
and
the
Danes
the
pot
life
center
management.
In
this
area
we
need
to
Delta
pod
resource
request
and
the
day
every
usage
and
some
latency.
This
is
very
important
for
hybrid
deployment
and
HPA.
A
Assertion
is
actually
some
ranchers
that
simulate
water
loss
or
to
real-time
analysis
of
workloads
to
decide
the
jersey
of
a
cluster,
at
least
a
possible
assertion
running
over
trusters.
They
were
chief
api
server,
etcd,
FM,
whip
usage
and
so
on.
This
assertion
is
very
important
about
here
us
whether
the
key
component
voting,
as
is
mated,
and
whether
the
networking
is
good
enough
for
over
were
close.
A
Okay.
Now
we
have
a
matrix
producers
so
how
we
relate
them
in
each
clusters
we
deployed
to
promises
for
HJ
to
take
all
matrix
from
the
matrix
producer.
Besides
of
this
row,
matrix
Malamud
matrix,
we
set
up
a
series
of
all
recording
rows
to
summarize
the
resource
usage
utilization,
saturation,
/
namespace
and
the
node.
It
will
help
us
to
piece
the
overview
of
all
clusters
moderately
above
the
transfer
level
promises
in
order
to
creates
a
global
view
of
old
houses.
We
also
deployed
Federation
promises,
which
we
are
collected
at
key
metrics
from
each
cluster
promises.
A
A
Ok
for
showing
the
matrix,
we
need
to
carefully
designed
what
we
really
want
and
the
water
automated
easily
to
check
problems,
how
to
take
the
overview
status
or
for
so
many
transfers.
This
is
an
analyst
working
and
we
are
keeping
improve
it
by
feedback
alternately.
We
have
two
kinds
of
boss
trusted
boss
and
robot
boss.
A
In
global
view,
we
build
several
boss
to
show
a
robot
faster
here,
see
faster
from
city
alert,
status,
trustor,
components,
version,
and
so
when
something
goes
wrong
in
these
dashboards,
which
he
then
jumped
into
the
cluster
level
boss,
in
trust
view
with
build
dashboards
for
each
aspect
of
cluster,
including
API,
server,
edct
and
node,
and
other
kinds
of
dominance.
Proms
at
ease
trust,
her
boss,
we
it's
the
detailed
charts
in
space
field
area
and
the
discharge
can
help
us
to
pretreat.
A
Had
it
the
root
cause
of
each
problems
and
now
I
will
show
you
some
was.
The
first
stage
was,
is
over
trouble.
Healthy
was
this
was
held
for
parts
and
the
first
part
is
showing
the
issue.
Pulpiness
version
running
over
trousers
is
a
great
help
for
us
to
know
the
exactly
number
of
for
dominance,
which
is
out
of
date
and
the
weekend
fits
them
one
by
one.
The
other
three
parts
shows
the
trouble
alert,
status,
node
status
and
port
status.
A
When
some
clusters
have
problems-
and
we
can
find
it
in
this
dashboard
and
assume
we
had
jumped
to
the
party
placer
health
dashboard
from
the
roster
health
dashboard.
We
can
paste
the
detailed
information
about
it,
unhealthy
nose
and
also
condensed
contouring
healthy
and
the
pot
healthy.
Here
we
using
no
DLC
as
an
example.
If
we
catched
one
node
is
unhealthy,
we
can
quickly
click.
The
hyperlink
puts
a
detailed
node
status
dashboards
from
the
node
state.
A
motor
must
adhere
the
node
health
dashboard.
A
What
we
need
is
a
automatic
alerting
when
the
transfer
has
rarely
some
problems
at
eBay,
we're
using
a
lot
of
control,
alert
manager
to
fire
alerts
based
on
some
predefined,
a
lotta
loose.
We
have
a
pertain
to
this
alert
rules
for
Patriot
management,
and
there
will
be
two
part:
two
labels,
the
first
one
is
a
component.
It
will
help
us
to
know
which
one
should
be
responsible
for
this
alert
and
the
energy
will
be
notified
and
the
second
that
label
is
the
sorority.
A
A
Then
each
alert
should
also
have
four
annotations
to
describe
what
is
the
alert
which
trusters
should
this
alert
appeal
are
applied
and
the
rampid
linked
to
tea
with
solutions
step-by-step?
This
one
boots
is
a
world
of
helpful
for
people
who
is
not
familiar
with
this
alert,
and
it
can
help
them
to
reserve
this
a
lot
okay,
when
a
real
alert
felt,
we
will
get
a
phone
call
from
PG,
Duty
or
receiving
email
or
message
about
these
alerts.
A
But
just
a
notification
is
not
enough
to
manger
laws,
though
they
also
create
about
to
automatically
create
dirty
tips.
That
is
a
form
including
the
load,
cause
analysis,
extrusion
plan
and
a
lot
Anam
consumption.
This
katie
will
be
assigned
to
the
responsible
people,
and
here
we
are
filled
out
this
form
during
the
resolving
process
of
this
cottage,
where
be
reviewed
by
all
of
our
team,
and
the
issue
will
help
us
to
find
out,
which
part
is
the
most
unstable
and
how
to
improve
it.
Though.
A
After
this
step,
we
have
an
end-to-end
alert
management
and
above
this
we
also
build
some
dashboards
to
show
of
alerts.
For
example,
this
is
overdrove
alerts
dashboards.
This
change
sure
how
many
alerts
that
are
a
very
felt
in
over
clusters
and
as
a
detailed
metadata
about
each
cluster.
Each
alerts.
A
Now
we
already
have
discussed
about
Lourdes,
dashboards
and
alerts.
Then
what
else
do
we
missing?
And
it
is
a
whole
week,
an
audit
or
closed
classes,
for
example,
if
a
user
on
name
space
across
five
clusters-
and
he
wanted
to
have
a
picture
about
how
many
posts
running
how
many
resource
are
using
in
all
of
these
clusters,
so
hot
for
the
he
found
out
he'd.
This
require
us
to
provide
accomplished,
accomplished,
Prairie
ability
to
find
out
this
result
and
it
can
be
easily
used
by
our
customers.
A
A
The
data
in
English
search
was
used
to
build
dashboards.
For
example,
the
Left
chart
is
showing
the
CPU
and
the
main
memory
code.
High
usage
per
namespace,
which
is
based
on
over
the
source
code
object
and
the
pillow
chart
shows
a
nose
by
switch
and
a
wrench.
The
switch
in
the
right
information
was
stored
as
an
annotation
in
our
node
object,
so
you
can
see
is
very
easily
to
build
more
and
more
boss,
based
on
overt
religious
objects
and
over
CRD
objects.
The
other
data
storage
is
a
Postgres.
A
By
Postgres
we
trying
easily
r2
using
solo
to
carry
what
we
want.
For
example,
if
we
want
to
get
the
old
post
of
a
namespace
coverage
or
troop
system
across
these
many
clusters,
we
can
easily
to
write
a
Soto
to
get
what
we
want
and
at
the
same
we
can
also
get
all
services
which
type
is
loaded.
Venison
is
very
easy
right.
It
really
save
all
time
when
you
want
to
find
something
quickly.
A
Ok
next
I
want
to
share
is
a
whole
image
over
many
configurations
from
the
previous
slides.
We
already
knows
that
monitoring
has
a
loss
of
dominance.
Lies
promises
a
lot,
a
manger
to
watch,
so
there
will
be
a
lot
sort
of
operations
needed
need
us
to
maintain.
When
we
only
have
a
few
clusters,
the
sony
is
not
difficult
to
manage
them,
but
with
the
cloud
cluster
growing
manually
manage
this
population
is
a
disaster,
so
at
eBay
we
build
each
dreaming
process,
manger
/
monitoring
corporations.
All
corporations
was
maintained
in
arbitrable.
A
If
we
want
to
make
some
change,
we
need
to
submit
a
PR
and
the
reviewed
by
our
peers.
Then
a
CI
job
will
sink
the
lattice
or
a
pole
to
another
Chartist
little
lab
repo,
which
can
be
accessed
from
all
of
our
trousers,
including
tab
and
production,
and
how
to
make
it
a
defect
in
our
real
clusters:
less
using
the
plum
juice
operations
and
Allerton
rose
as
an
example
for
these
corporations
repute
a
lot
controller
which
running
in
each
classes.
This
controller.
A
We
are
digital
new
list,
recovery
agents
from
overdue
library
book
and
the
validated
and
then
compare
them
with
the
wrong
time,
corporations
and
other
a
lotta
loose,
plum
diploma
shorting
instance.
In
this
step,
the
alert
plancherel
we
are
smart
to
keep
the
a
lotta
rules
which
is
should
not
applied
in
these
clusters
based
on
the
cluster
annotation,
which
you
mentioned
before
after
this
step,
if
contrasting
is
name
is
needed
to
update
these
configurations,
which
we
are
updated,
the
company
map
and
at
LS
at
the
last
interval
reboot
the
promises
to
maintain
it
event.
A
Okay,
now
make
it
loss
of
monitoring
data
from
Lourdes,
matrix
and
reproach.
So
the
next
step
we
consider
is
how
to
utilize
this
data
to
help
us
to
manger
over
clusters.
Most
motley,
especially
we
wanted
to
do
a
real-time
analysis,
predict
trust
issues
and
the
inventory
reducing
the
meantime
to
detect
a
problem
and
the
mean
time
to
repair
it.
A
So,
in
the
last
of
my
monitoring,
sharing
is
a
brief
introducing
about
our
practice
in
the
Arabs.
The
basically
process
of
our
Arabs
has
tools
are
four
steps.
The
first
step
is
data
pre-processing,
for
example,
which
would
apply
some
smoothing
algorithm
like
autumn
winter
approach
or
matrix,
which
is
a
time
series
data
data.
Then
we
can
apply
the
D
tensor
algorithm
to
find
the
program
like
three
cinema
is
original
forest
and
the
other
algorithm.
A
This
algorithm
can
help
us
to
attach
the
anomaly
events
from
so
many
matrix
data,
but
after
this
step,
the
anomaly
events
are
too
austere
too
many
to
us.
So
we
need
to
do
correlation
or
grazing
analysis
for
this
loop
root
cause
analysis.
For
example,
we
should
apply
FP,
Drowsy
algorithm,
then.
Finally,
we
can
get
a
small
subset
of
anomaly
events,
and
this
events
should
be
highly
valued
by
us.
A
We
could
wish
you
the
automatically
flying
others
to
our
DevOps
and
it
would
dealing
with
this
problem
and
fixing
okay,
that's
all
what
I
want
to
share
about
eBay
production
monitoring,
including
heavy
loading,
Hobbit
collapse,
matrix
how
we
build
dashboards,
how
we
manger
alerts
and
the
how
we
do
monitoring
automation
and
the
Arabs.
Thank
you.
The
next
will
come
in
her.
He
will
introduce
our
remediation
solution
based
on
over
production
monitoring.
B
B
We
have
more
than
30
clusters,
including
soap.
They
have
pre-production
clusters
as
well
as
some
dedicated
clusters,
such
as
payment
clusters,
search
Hadoop
clusters.
Now
we
are
managing
both
Pimentel
and
virtual
machines.
We
have
thousands
nodes,
but
our
mission
is
using
criminales
to
manage
all
the
nodes
in
eBay,
so
we
need
to
have
automated
remediation
flow
or
know
the
level
for
some
problem
notes.
So
here's
the
aggro
of
our
flow
I
will
introduce
them
each
component,
one
by
one.
B
The
first
know
the
prominent
attacker,
which
is
upstream
to
oh,
but
we
have
some
customization
or
for
fight.
It
is
a
demon
running
a
each
node.
We
detect
the
memory
disk
and
a
container
around
the
house
and
report
status
to
compute
node,
don't
confuse
it
can
be
note
it
isn't.
It
is
not
same
as
the
Canaries
and
other
young
cluster
communities,
a
CRD
object
that
represents
a
node.
It
is
differently
from
from
an
S
note.
It
includes
operational
information
such
as
memory,
CPU,
disk
partitions
and
all
kinds
of
node
status.
B
Although
the
Status
are
reported
to
this
object,
limitation
controller
watched
this
object
and
we
have
many
sub
controllers
in
the
remediation
controller.
Each
controller
is
a
response
for
each
status.
The
controller
makes
the
decision
for
node
remediation
based
on
the
status
and
some
magic
speak
lag
data.
Also,
we
have
a
data
controller
which
caused
the
rest
api
of
reparo
to
carry
sensor
magicus.
Also.
What
is
mad?
What
is
repair
Oh
repair
is
a
system
that
managed
all
the
hardware's
of
ebay
data
center.
B
It
exposed
REST
API
for
horrible
magicus,
so
from
the
flow
we
can
see.
If
the
node
problem
is
detected
by
a
node
problem,
detector-
and
it's
reported
to
node
come
here-
know
the
status
and
the
limitation
controller
watch
these
status
and
make
the
decision
to
do
some
remediation
and
if
the
remediation
has
been
success
has
been
remediated
as
successfully
the
node
should
be
become
health
again,
and
another
problem
detected
the
good
status
and
the
report
good
status
to
the
communal
again.
B
So
we
know
the
two
statuses
is
changed
from
bad
to
good,
but
you
know
we
know
this
controller
has
a
high
risk.
Why?
If
there
is
a
body
in
this
controller
and
all
nodes
in
the
cluster
might
be
impacted,
that
will
be
a
disaster,
so
we
would
design
the
disrupt
his
rupturing
budget,
to
name
it
to
the
change
based
on
the
know,
the
number
and
the
change
timestamp.
It
is
a
admission
Rebeck.
The
changes
will
be
rejected
if
the
change
in
number
exceeds
the
budget.
B
For
example,
in
an
hour,
you
can
only
change
ten
notes
or
is
a
configurable.
So
this
is
the
hardware
Halliwell
faders
which
detect
the
first
stage,
tear
and
non-intrusive
detection,
as
I
mentioned
in
my
previous
slide.
We
get
the
sensor
matrix
from
reparo,
the
matches
can
be
temperature,
CPU
power
supply
unit
and
so
on.
In
wise
we
have
oh,
we,
we
use
node
problem
detector
to
analysis,
kernel,
message,
MSE
message
and
the
disk
checker.
B
That
is,
as
we
have
different
X
values
we
to
the
disk
health
charity,
based
on
different
banners
used
by
using
different
tools.
So
we
have
automatics.
We
need,
as
Daniel
mentioned,
err
which
had
pretty
good
photos
based
on
the
matrix
and
logs
called
air
ops,
but
for
now
not
never.
We,
we
didn't
apply
air
ops
for
now.
Viseotech
action
after
failures
have
happened
for
each
faders,
we
define
a
corresponding
remediation
pattern
to
try
to
fix
the
specific
failures.
B
As
mafias
happened,
more
talents
are
added,
some
faders
can
be
tolerant
and
to
remediate
it,
but
for
some
hardware
fellows
when
you
descend
to
repair
Oh
to
break
fix.
Luckily,
the
whole
lifecycle
is
automated.
We
will
get
notified
after
the
halwa
has
been
fixed
and
it
will
be
a
great
provision.
This
note
in
our
cluster
okay,
this
is
sort
of
our
videos
and
here's
some
jicama
notes.
We
really
care
about
it.
B
First
mokuba's
relates
the
components
such
as
proxy,
her
blade
and
also
we
have
so
many
configurations
in
your
node,
such
as
kernel
parameters,
even
as
network
configurations
and
then
a
storage,
back-end
configurations
inside
sure.
You
know
we
have
a
sort
operator
which
is
also
controller
to
make
sure
the
configurations
are
same
in
a
specific
group
or
container
runtime.
B
We
need
to
make
sure
it
can
poor
image,
it
can
create
start/stop,
delete
containers
and
it
should
try
to
bring
in
fact,
if
one
or
more
one
or
more
of
them
field
or
some
key
services,
we
also
monitor
them,
such
as
NTP
e
under
services.
So
to
summarize,
we
focus
on
another
level,
author
elimination.
Now
we
have
eliminating
controller,
which
acts
as
a
reminiscing
engine
to
a
necessity
of
failures
and
remediated
failures
in
future.
B
A
You
know
uses
was
all
over
Malta.
We
had
some
more
modest
about
it
into
the
remediation
and
the
repair
right
and
currently
we
are
in
the
research
phase
in
over
Arabs
and
we
have
the
four
steps
to
for
our
Arabs,
which
I
mentioned
before
and
intently
in
over
production.
We
are
researching
hopeful
using
the
Arabs
to
monitor
over
disk
problems,
which
also
we
support
a
lot
of
for
distal
usage,
very
highest
appreciation,
light
at
the
DB
or
something
else
so
they
will
reach
out
and
share
about
the
disk
healthy
and
the
butters
are
modest.