►
Description
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Anomaly Detection for Cloud Native Storage - Seiya Takei & Xing Yang
Integrating with heterogeneous storage in the Cloud Native environment has always been a challenge. How to detect problems and fix them in a timely fashion is important for mission critical workloads. In this session, Takei-san and Xing will describe a common volume metrics model designed to retrieve data from heterogeneous storage in the Cloud Native environment. They will also illustrate a ML module that analyzes the data to detect anomalous behavior, and discuss how it helps Yahoo Japan identify problems early to keep the Cloud Native storage system healthy. Volume metrics such as IOPs, bandwidth, latency, and capacity are collected from storage backends serving workloads running on Kubernetes, and emitted to the Prometheus server. The ML module retrieves data from Prometheus and applies algorithms to do anomaly detection. Results are evaluated and alerts are issued when needed.
https://sched.co/Nrnb
A
B
A
Here
is
the
agenda
for
today.
So
first,
what
you
were
brief
introduction
to
anomaly
detection
and
our
focus
is
on
doing
anam,
detection
in
storage
performance
and
Takizawa
talk
about
Yahoo,
Japan's
use
cases,
we'll
talk
about
help
me
tree
a
project
that
integrates
with
Prometheus
and
graph
Anna
collecting
performance,
metrics
from
storage
backhands
and
we're
doing
anonyme
detection
using
those
data
collected
from
the
metrics
module
and
what
your
demo.
A
A
A
The
first
one
is
point
anomalies,
which
means
the
single
data
point
is
very
far
away.
From
the
rest
of
example,
you
can
detect
a
fraud
detection
just
by
looking
at
the
amount
spent.
Second
category
is
contextual
anonymous,
so
the
anomaly
is
context
specific.
This
is
the
typical
in
time
0
data,
so
you've
spent
a
lot
of
money
on
food
every
day
during
holiday
season
may
be
normal,
but
otherwise
he
may
not
be
normal
and
the
third
category
is
collective
anomalies,
so
meaning
that
a
group
of
data
collectively
identifies
them
an
army.
A
But
if
you
look
at
a
single
data
point
in
that
group,
that
may
be
normal.
So
if
you're
spending
money
to
buy
our
large
number
of
expensive
items
on
single
day,
then
maybe
that's
not
normal
that
if
you
just
look
at
a
single
transaction
that
may
be
okay,
so
there
are
different
use
cases
in
anomaly.
Detection.
It's
used
in
intuition
detection
to
find
out,
if
you
know,
is
a
hack
by
looking
at
the
network
traffic
patterns.
A
It's
used
in
medical
house
to
see,
if
you
know
the
tumor
by
doing
a
scan,
it's
used
in
fraud
detection.
So
if
you
really
don't
travel
abroad
and
suddenly
your
credit
card
is
used
in
a
different
country,
then
that
chunk
section
is
likely
to
be
rejected.
And
then,
until
you
confirm
that
it
is
really
new,
then
you
wouldn't
be
able
to
use
your
credit
card
and
it's
used
in
fault.
A
So
it's
challenging
to
detect
issues
in
story
performance
because
many
things
can
go
wrong.
You
could
have
these
failures,
you
could
have
a
rewrite.
I/O
arose
or
some
port
problems
configuration
issues
that
were
problems,
so
we
need
to
look
at
metrics
collected
from
different
ways,
look
at
the
IRS
penguin
and
they
look
at
the
cash
head
and
so
on.
We
need
to
collect
data
from
multiple
dimensions
and
correlate
them
and
identify
bottlenecks.
B
In
OpenStack
environment
we
are
using
storage,
it's
for
cinder,
Swift
Manila
in
kubernetes
environment.
We
we
are
using
strategies
for
persistence,
volume
OpenStack
in
kubernetes
classes
have
each
stretch.
So
if
the
number
of
addresses
is
increases,
the
number
of
stress
also
increases,
and
moreover,
there
is
in
there
is
bias
in
in
all.
There
is
bias
in
kubernetes
class
in
capacity
between
OpenStack
our
cumulative
classes.
This
means
some
clusters
have
enough
capacity
in
a
pre
capacity
in
storage,
but
some
clusters
have
not
enough
free
capacity
in
storage.
B
B
A
So
tacky
Sam
has
talked
about
it
use
cases,
and
then
they
want
to
use
the
colony
tree
and
the
anomaly
detection
of
open
ideas.
The
open
SDS
is
an
open,
autonomous
data
platform,
so
this
shows
the
project
scope
soon
as
the
northbound
plug-in
to
support
container
orchestration
systems
and
it
provides
the
common
control
plane
to
manage
log
file
and
object
storage.
Our
focus
today
is.
A
This
is
the
architecture
diagram
of
how
naturey,
so
here
we
have
a
API
server
that
has
a
post
and
get
it
nice
and
then
is
a
controller
and
then
of
the
talk
that
is
like
a
docking
station
at
hosts
metrics
drivers
for
lbn
staff
and
other
third
party
metrics
drivers
on
the
dock.
There
is
a
collector
collector,
collect
metrics
from
different
metrics
drivers
and
then
that
data
will
be
then
to
will
go
here.
A
This
is
know
the
adapter
here
that
includes
a
converter
and
exporter,
so
converter
basically
converts
the
matrix
into
a
format
that
can
be
understood
by
the
destination
and
exporter,
and
it's
the
data
in
a
format
that
can
be
we
retrieved
by
the
destination.
So
like
Prometheus,
we
also
have
integration
with
the
popke.
B
Metrics
quarter
gets
iOS
bandwidth,
latency,
CPU
usage
and
custom
metrics
for
volume
per
disk
controller
resources,
for
example,
I
opted
of
Orion
and
Vaughn
beast
of
disks
and
so
on
on
premise
bridge
the
on
premise:
threads
such
as
direct
attach
software
design,
invite
Enterprise,
is
supported
and
also
cross
bridge,
such
as
safest
we
get--we
AWS.
A
dual
Google
cloud
platform
is
also
supported.
A
Since
we
integrated
with
the
Prometheus
I
just
want
to
quickly
go
through
this
Prometheus,
a
potential
diagram
prometheus
as
a
server
that
includes
three
components,
is
the
retrieval
component,
that
periodically
scrapes
data
frying
HTTP,
endpoint
copy
girl
on
the
exporter
and
then
saves
dad
in
the
database,
which
is
this.
A
TSG
database
saves
the
matrix
with
a
timestamp
and
then
the
third
component
is
this
HTTP
server
that
can
handle
HTTP
based
requests
using
prompt
you
out,
which
is
the
Prometheus
security
language
and
then
also
a
permit.
It
has
a
alert
manager.
A
So
this
shows
a
little
bit
detail
of
our
tell
me:
trees,
integration
with
Prometheus.
So
if
it's
a
close-in
request,
API
server
gets
it
pass
on
to
the
controller
controller.
We'll
send
that
you,
the
collector
from
the
collector,
communicates
with
the
MAGIX
drivers
data
matrix
and
then
the
exporter
will
emit
that
matrix.
A
Prometheus
server
will
be
pulling
those
metrics
and
if
it's
a
get
request,
so
that
assumes
the
data.
The
metrics
is
already
there
in
a
primitive
liver
disease.
So
the
controller
registered
your
conversion
using
the
prom
pol
and
get
that
metrics
from
the
Prometheus
server.
So
tell
me
Jim
what
it
doesn't
have
a
database
itself.
We
just
use
the
database
in
Prometheus,
but
our
own
on
detection,
what
it
does
have
a
bit
of
database,
which
I
will
cover
in
the
later.
B
Type
of
driver
such
as
LVM
set
and
so
on,
get
matrix
fit
as
SSID
image.
Name
start
time
and
end
time
in
society
is
volume
ID
in
example,
swap
matrix
that
has
instanceid.
It
was
named
job
another
component
name,
unique
aggregate
type
and
metric.
Both
bodies
after
metrics
is
cert
me
trick
it
had.
It
has
only
time
stamp
and
value.
B
Instead,
the
set
father
is
already
exist
in
primitive
website.
In
this
thing,
set
its
folder
in
context
is
used
to
correct
step.
Metrics
only
crosser
metric
matrix
is
printed
in
slide
as
supported
matrix
because
the
site
has
no
enough
space,
but
others
matrix
can
be
corrected.
Third
volume
step
components
such
as
OSD,
monitor
and
so
on.
B
B
A
Going
back
to
you
this
diagram,
so
our
canvas
module,
export
the
matches
and
then
Prometheus,
where
we
achieve
those
metrics
Prometheus
also
had
to
alert
manager.
So
you
can
use
the
Prometheus
of
our
major
in
two
ways.
You
can
configure
a
lot
of
rules
in
Prometheus
and
then,
when
it
passes
the
threshold
Prometheus
worries
an
alert
on
the
other
manager.
A
So
I
think
this
example
showing
here
needs,
maybe
code
you
see
by
this
one
disk
space,
10%
free,
then
always
an
alert
and
then
the
second
way
is
to
use
the
REST
API
interface
of
our
manager.
You
can
retain
custom
alert,
so
our
anomaly
detection
motor
uses
that
so
the
example
shown
here.
That's
the
thing
this
running
for
that's
a
custom.
A
There
are
different
algorithms
used
in
anomaly.
Detection
means
the
classification
based
it
basically
uses
in
a
classifier
to
distinguish
between
normal
and
abnormal
data
points.
There
is
a
nearest
neighbor
based
algorithm,
which
assumes
normal
data
reside
in
a
cluster
in
a
a
neighborhood
and
then
the
anomalies
occur
in
place,
far
away
from
closest
neighborhood
and
then
there's
a
clustering
based
assuming
that
normal
data
beside
in
a
cluster
and
then
the
anonymous
stare
points
does
not
beam
onto
any
plaster
and
they're
also
statistical
models
like
caution
and
regression,
and
also
information,
theoretic
and
the
spectral.
A
So
a
Gaussian
model
is
the
classic
of
statistical
model,
assumes
that
data
forms
in
this
bow
shape,
and
you
should
use
a
to
specify
a
threshold
to
differentiate
between
your
normal
data
points
and
anomalies
and
DP.
Skin
means
the
density
faces,
spaceship,
posturing,
application
with
noise,
so
we
have
a
diagram
here
so
carefully.
The
DB
scan
algorithm
takes
two
input.
1
is
the
radius
of
the
neighborhood
around
the
core
point
and
then
the
second
one
is
the
minimum
data
points
in
the
cluster
and
then
based
on
this
two
input
parameters.
A
It
divides
the
data
into
three
categories,
core
points
for
the
points
and
outlier.
So
this
is
a
blue
thought
in
the
middle.
Here,
that's
a
core
point,
because
it
is
in
cluster
with
more
than
five
data
points
and
then
the
green
dot.
There
is
a
order
point
because
it
is
in
a
cluster
of
the
less
than
five
data
points
and
then
the
red
one
is
an
outlier,
because
it
is
not
in
the
cluster
with
any
other
data
points.
A
So
we
run
the
Gaussian
model
ways
the
data
we
collected
from
lbn,
so
we
collected
60
tons
and
that
tricks
from
lbn,
but
to
be
only
use
the
to
show
this
it's
easier
to
display
here.
So
we
use
the
volume
level
metrics,
IAP,
sender,
latency
here,
so
we
run,
we
run
the
reverse.
It
collected
back
in
two
days
and
then
we
got
six
thousand
data
points
after
the
movie
from
zeros,
and
then
we
also
use
the
TV,
and
then
we
use
the
boney
to
generate
some
workloads.
A
So
yet
so
the
blue
data
point
those
are
in
the
those.
Those
are
normal
data
points
and
then
the
red
ones
are
a
liar,
and
then
this
is
the
from
the
Devi
scan
using
the
exact
same
data.
So
this
shows
so
the
the
bigger
story
in
red.
Those
are
core
point
and
the
small
point
in
red.
That's
the
water
point
and
the
black
dots
are
anomalies.
A
A
We
also
run
those
tests
using
SEF,
so
we
set
up
kubernetes
cluster
and
we
have
a
open
STS
in
store
there
and
then
also
set
up
a
MongoDB.
So
MongoDB
is
running
as
a
staple
set,
there's
a
3d
graphic
class
and
it
is
running
a
avoiding
provision
to
buy
open
access.
Yes,
that
plug-in
and
the
pack
can
use
that.
So,
yes,
it
sure
is
just
sort
of
class.
Is
that
ok,
it's
a
plugin,
and
then
we
collect
metrics
from
the
SAP
exporter,
node
X
quarter
and
the
DB
exporter.
A
A
The
proponent
test
tool
will
use
those
to
generate
some
more
clothes
to
you
to
you,
some
invert
deleting
update
and
find
so
we
get
five
clusters
here,
but
most
of
them
are
three
in
one
large
cluster,
that's
in
the
color,
the
red,
color,
the
rest
of
them
so
like
there's,
some
smaller
class,
lettuces,
yellow
orange
and
green
there's,
a
smaller
classroom
and
also
them
are
at
the
bottom.
So
there
are
a
small
number
of
what
are
some
anonymous
anonymous
data
point.
Those
are
the
black
dots
there.
A
So
this
is
actually
the
same,
the
same
data,
but
this
is
just
a
one
section
of
that.
This
is
the
only
one
thousand
data
points.
So
when
we
use
this,
if
just
the
company
has
one
cluster
and
then
they
are
most
of
them
are
in
this
one
cluster
and
then
there
are
some
the
black
data
points.
Those
are
those
are
anomalies
and
the
by
the
way.
This
is
the
SEC
line
at
UPS,
because
we
don't
get
the
the
water
level
proposed
metrics
that
will
be
used
for
lbn,
so
for
staff.
A
And
this
is
a
data
collected
from
the
node
exporter.
This
is
the
node
memory
available
in
megabytes,
so
this
also
shows
just
one
large
cluster
and
then
there's
small
number
of
anonymous
data
points.
So
this
is
from
MongoDB.
This
is
the
global
log
current
queue.
This
is
actually
not
what
happen
if
left
expected,
because
there
is
a
gap
there,
so
it
forms
two
clusters
and
then
there
was
a
few
anonymous
data
points.
So
my
assumption
is
because
we
generally
allotted
for
load.
B
B
B
B
C
A
So
what
are
we
are
going
to
do?
Next?
We
of
course
need
to
collect
more
data,
so
we
can
get
some
good
results
and
we
want
to
correlate
the
metrics
the
collapse,
data
from
the
store
to
back-end
from
the
node
exporter
and
from
applications
and
trying
to
find
out
the
problems
using
this
data,
and
there
are
some
other
algorithms
that
we
can
consider
in
the
future
like
random
forests
and
the
autoregressive
integrated
moving
average
and
so
on,
and
we'll
continue
the
journey
to
the
autonomous
to
data
platform.
A
C
People
talk,
it's
really
interesting.
We
are.
We
are
also
working
on
the
animal
attraction
in
Red
Hat,
so
I
would
be
interested,
whether
you're
tooling,
that
you
showed
the
whole
system
from
issues
and
the
animal
detection
training,
except
how
general
it
is,
how
reusable
it
is
for
different
kinds
of
metrics,
maybe
not
just
storage
or
combining
metrics
from
storage
and
other
metrics
from
the
cluster
for
Prince,
Buster
and
applications
and
stuff
like
that.
If
there
is
a
potential
for
reusing
it
for
other
use
cases
as
well,
yeah.
A
So
right
now
we
have
just
a
focus
on
storage,
so
whatever
collecting
on
keep
I
switch
it
back
in,
but
I
also
mentioned
that
we
are
also
trying
to
collect
matches
from
application
like
the
MongoDB
right
yeah.
So
so
we
and
I
was
fond
note
exporter.
So
we
do
like
to
look
at
the
data
from
different
angle.
Try
to
correlate
them.
D
A
So
yeah,
no,
we
haven't
really
get
you
backstage
here.
So
what
are
we
you?
What
do
we
have
right
now?
It's
a
suggest
that
you
collect
metrics
and
then
the
root
cause
analysis
will
be
coming
next
and
in
trying
to
find
out
problems
and
then
trying
to
figure
out
how
to
improve
the
system
that
will
come
next.
So
we
haven't
really
get.
D
A
So
that's
a
second
yeah,
so
yeah.
Definitely
we
will
get
there
so,
but
we
are
really
not
the
idea.
So
if
you
look
at
here
as
though
we
are
right
now
are
right
here
so
just
to
collecting
the
data.
But
then
you
know
anomaly,
so
that's
the
next!
So
how?
What
do
you
do
with
those
data
right
so
that,
right
now
we
are
just
stood
at
the
data
collect
collected
and
we
got
alerts
and
then
but
I
definitely
get
your
point.