►
From YouTube: Subutai Ahmad, VP of Research, Numenta @ MLconf SF
Description
Real-time Anomaly Detection for Real-time Data Needs: Much of the world’s data is becoming streaming, time-series data, where anomalies give significant information in often-critical situations. Examples abound in domains such as finance, IT, security, medical, and energy. Yet detecting anomalies in streaming data is a difficult task, requiring detectors to process data in real-time, not batches, and learn while simultaneously making predictions. Are there algorithms up for the challenge? Which are the most capable? The Numenta Anomaly Detection Benchmark (NAB) attempts to provide a controlled and repeatable environment of open-source tools to test and measure anomaly detection algorithms on streaming data. The perfect detector would detect all anomalies as soon as possible, trigger no false alarms, work with real-world time-series data across a variety of domains, and automatically adapt to changing statistics. These characteristics are formalized in NAB, using a custom scoring algorithm to evaluate the detectors on a benchmark dataset with labeled, real-world time-series data. We present these components, and describe the end-to-end scoring process. We give results and analyses for several algorithms to illustrate NAB in action. The goal for NAB is to provide a standard, open-source framework for which we can compare and evaluate different algorithms for detecting anomalies in streaming data.
A
Right,
thank
you
actually
I'm
curious.
How
many
of
you
actually
have
heard
about
new
mentor
or
no
of
us
before
you
name?
Okay
great,
so
you
might
know
that
we've
been
working
in
machine
learning,
algorithms
in
terms
of
applications,
we've
been
focusing
on
streaming
analytics
and
we
work
with
a
number
of
customers
in
that
area,
and
we've
really
been
focusing
on
anomaly
detection
within
that
field
and
when
it
came
to
kind
of
evaluating
different
anomaly
detection
algorithms,
we
came.
A
We
found
that
it
was
actually
really
hard
to
do
that
and
there
weren't
really
good
benchmarks
for
it.
The
field
of
machine
learning
is
full
of
lots
of
good
benchmarks,
but
in
the
world
of
streaming
anomaly
detection
there
seems
to
be
very
few,
if
not
none,
and
so
what
we
did
is
we
created
an
open-source
benchmark
for
the
community
and
that's
what
I'm
going
to
talk
about
now?
A
Okay,
so
if
you
think
about
kind
of
the
world
of
streaming
analytics,
as
you
all
know,
there's
an
exponential
growth
in
the
number
of
sensors
with
Internet
of
Things
and
real-time
data
collection
is
a
very
large
set
of
explosion
of
data.
That's
coming
out:
there's
not
a
lot
of
machine
learning.
Technologies
actually
focus
in
this.
A
Most
of
the
work
has
been
on
databases
and
visualizations
and
so
on,
but
in
the
you
know,
in
the
realm
of
learning
anomaly,
detection
is
one
of
the
biggest
applications,
if
not
the
biggest
application
in
in
streaming
analytics,
so
you
can
think
about
things
like
monitoring
the
health
of
your
data
center.
You
want
to
know
when
servers,
you
know
something
going
wrong
with
your
servers
monitoring.
You
know
fraudulent
transactions
in
streaming,
financial
data,
energy.
A
You
know
strange
things
happening
in
energy,
tracking,
a
fleet
of
vehicles
or
you
know,
even
looking
forward
into
real-time
kind
of
health
monitoring
and
trying
to
figure
out.
When
are
you
going
to
become
sick?
Are
you
becoming
sick
right
now
in
real-time?
One
of
the
interesting
things
is
that
although
we
use
the
word
anomaly
detection,
it's
not
really
detection.
When
you
think
about
streaming
analytics,
it's
more
about
prevention.
You
want
to
find
you
want
to
get
understand
with
something
unusual
is
going
on
before
some
sort
of
a
catastrophic
failure.
A
You
know
it's
much
more
valuable
to
know
that
you're
going
to
have
a
heart
back
in
three
minutes,
then
oh
you're
having
a
heart
attack
right
now
that
little
bit
of
difference
makes
a
big.
So,
let's
look
at
an
example,
so
this
is
a
preventive
maintenance
scenario.
So
this
is
a
very
large
industrial
machine.
That's
in
some
remote
location-
and
this
is
the
temperature
of
one
of
their
sensors
and
there's
three
different
anomalies
in
here.
If
you
look
at
the
first
one,
the
left
one
well,
that
was
a
planned
shutdown
of
the
machine.
A
So
that's
fine.
If
you
look
at
the
right
hand,
one
this
was
actually
a
catastrophic
failure
that
cost
this
company
millions
of
dollars
because
they
didn't
know
about
this
in
advance.
Now,
if
you
look
at
the
middle
one
that
occurred
several
weeks
before
the
catastrophic
failure,
it
was
actually
a
strange
temporal
fluctuation
in
the
data
and
an
expert
looking
at
that
actually
told
us
that
that
was
the
actual
cause
that
then
several
weeks
later
led
to
the
catastrophic
failures.
If
you
could
detect
that
you
know,
middle
anomaly
could
have
saved
the
company
millions
of
dollars.
A
Ok,
so
it's
an
example
of
why
detecting
something
early
is
really
valuable
in
streaming
analytics
so
I
mean.
Do
we
really
need
yet
another
benchmark?
Well,
you
know
what
is
a
benchmark
so
benchmark
consists
of
you
know.
Labeled
data
sets
a
well
thought
out
scoring
mechanism
and-a
versioning
system,
so
you
can
compare
different
results.
It
turns
out
in
most
of
the
benchmarks
in
anomaly.
Detection
are
really
designed
for
batch
data,
not
streaming
data.
The
second
thing
where
you
know
practitioners
in
machine
learning.
A
You
know
we
really
want
real
world
data
and
it's
extremely
hard
to
find
benchmarks
containing
real-world
data
with
labeled
anomalies
and
the
combination
of
those
two
just
doesn't
exist,
and
so
we
saw
a
need
for
a
benchmark
that
is
really
designed
to
test
anomaly:
detection,
algorithms
on
real
time
streaming
data,
and
we
felt
that
a
standard
community
benchmark
would
really
spur
innovation
and
really
help
the
community
yeah
okay.
So
what
do
we
do
so
so
we
created
the
new
meant
anomaly,
benchmark
or
dad.
A
This
is
a
rigorous
benchmark
for
anomaly
detection
in
streaming
applications
and
there
are
three
basic
pieces
to
it.
So
first
is:
we
include
a
real
world
benchmark
data
set.
There
are
about
300
over
3
or
50,000
individual
data
points
there,
47
real-world,
datasets
and
11
artificial
streams,
there's
a
scoring
mechanism
which
I'll
describe
in
a
couple
of
minutes.
So
the
scoring
mechanism
rewards
early
detection,
consists
of
generous
windows
around
and
anomaly.
So
you
have
some
slop
in
how
you
detect
things.
A
It's
got
a
scoring
function
and
we
have
a
notion
of
application
profiles
which
control
kind
of
the
balance
between
false
positives
and
false
negatives.
I'll
talk
about
that
the
second
and
the
third
thing
is
that
it's
a
completely
open
resource,
it's
a
GPL,
a
repository
that
contains
all
of
the
data
all
of
the
source
code,
the
documentation
everything's
up
on
that
github
repository
there.
A
So
here's
some
of
the
example
data
sets
that
are
in
there
all
right.
This
particular
one
is
looking
at
the
latency
of
load
balancers
on
a
website,
so
you
can
see
that
most
of
the
time
it's
pretty
low
every
once
in
a
while,
you
get
spikes
up
and
that's
totally
normal.
This
is
a
really
unpredictable
hard
data
set
and
in
the
middle
or
the
right
hand,
side
there's
an
anomaly
where
the
Layton
sees
ended
up
being
a
lot
higher
than
normal.
Okay.
A
So
that's
an
example
of
an
anomaly
that
you'd
want
to
detect
us
as
soon
as
possible.
If
you're
doing
a
production
website
he's
a
very
different
example,
it's
hourly
demand
from
some
service.
In
this
particular
case,
it's
New
York
City
Taxi
data.
So
this
is
counting
how
many
taxi
rides
are
going
on
every
hour
and
you
can
see
it's
got
a
very
periodic
shape
to
it,
but
there
are
some
anomalies
in
there.
A
So
in
the
middle,
on
the
left
hand,
side
there's
a
spike
in
demand
that
was
actually
the
New
York
City
Marathon
and
then,
on
the
right
hand,
side.
You
see
a
very
different
type
of
anomaly.
You
have
unusually
low
demand,
so
the
values
here
are
completely
within
range.
A
threshold
wouldn't
catch
it.
An
outlier
based
method,
probably
wouldn't
catch
it,
but
it's
it's
definitely
an
anomaly
that
you
want
to
go
on.
A
Here's
the
third
example-
and
this
is
looking
at
the
CPU
usage
on
a
production
server,
the
the
pattern
is
extremely
spiky
again
and
in
the
beginning
it
was
not
very
spiky
that
was
pretty
low
and
then
it
starts
to
spite
so
there
a
kind
of
a
new
behavior
that
happened.
That
was
an
anomaly,
but
then
this
behavior
actually
continues.
So
maybe
some
new
software
was
installed
or
new
service
was
installed
or
something
and
this
even
though
there
was
an
anomaly
there,
this
behavior
is
now
the
new
normal.
A
So
you
know
those
examples,
kind
of
show
the
range
of
anomalies
that
you
tend
to
see
in
streaming,
data
and
they're
pretty
complicated
to
score.
So
how
should
we
score
them?
Well,
let's
define
what
we
mean
by
an
ideal
detector,
so
the
perfect
detector
here
would
detect
every
single
anomaly
and
it
would
detect
every
anomaly
as
soon
as
possible.
So
if
two
algorithms
detect
the
same
anomaly,
the
one
that
detected
it
earlier
should
get
preference.
It
should
provide
the
detections
in
real
time.
So
it
means
no
look
ahead.
A
You
have
as
you're
getting
the
data
you
have
to
be
able
to
give
a
decision.
The
ideal
detector
would
have
no
false
positives,
and
one
thing:
that's
really
important
in
streaming
analytics
is
you're
dealing
with
thousands
of
metrics
a
pretty
large
deployments.
Often,
and
you
cannot
have
manual
parameter
tuning.
You
know
you
can't
ship
a
machine
learning
person
with
every
sensor,
so
whatever
is
then
has
to
be
done
automatically.
A
The
last
thing
is
that
chain
sophistic
change
all
the
time
in
streaming
data,
and
so
you
have
to
be
able
to
automatically
adapt
to
changing
statistics,
and
the
scoring
mechanism
should
take
all
that
into
account
and
storing
mechanism
eckin
isms
in
traditional
benchmarks
that
completely
insufficient
for
this.
So
the
standard
notions
of
precision
and
recall
that
we're
familiar
with
they
don't
incorporate
that
notion
of
early
detection.
There's
often
an
artificial
separation
between
training
and
test
sets,
which
makes
sort
of
things
like
continuous
learning,
really
hard
to
measure
and
then
batch
data
files.
A
Allow
look
ahead
and
multiple
passes
through
the
data.
Well,
you
just
can't
do
that
in
real
time
street
in
cases
you've,
given
the
data
that
you
have
and
you
have
to
make
a
decision
before
you
move
on.
So
let's
look
at
an
example
here
of
how
we
score
it.
So
here's
the
metric
and
it
was
there-
was
an
now
anomaly
that
was
labeled
there
right
at
the
bottom.
But
if
you
think
about
it,
where
exactly
is
anomaly?
A
It's
probably
there
was
something
happen
in
them
in
the
in
the
system
that
caused
the
metric
to
slowly
start
falling
and
then
spike
down.
So
ideally,
what
you
want
to
do
is
detect
the
anomaly
as
soon
as
that
happened.
The
other
hand,
if
you
can't
do
that
anywhere
in
the
middle,
is
fine
too
that's
better
than
not
detecting
it.
So
we
do
an
AB.
Is
we
define
what
we
call
anomaly
windows?
A
So
we
have
a
very
generous
window,
that's
around
each
anomaly,
and
the
idea
is
that
the
earlier
you
detect
the
anomaly
in
the
window,
the
better
your
score
is
going
to
be
so.
We've
defined
a
scoring
function.
So
we
look
at
every
single
detection
that
your
algorithm
does
and
effect
of.
Each
detection
is
scaled
relative
to
position
within
this
window.
So
we
use
a
sigma
sigma
idols
scaling
function
within
the
window.
Detections
that
are
outside
the
window
are
considered
false
positives,
and
this
is
automatically
handled
in
the
scaling
function.
A
The
idea
here
is
that,
within
a
window,
there's
lots
of
unusual
things,
probably
going
on
we're
going
to
give
the
algorithm
the
benefit
of
the
doubt
and
just
pick
the
earliest
one
and
use
that
as
your
detection,
okay
and
then
the
complete
score
for
the
data
set
is
basically
the
sum
of
all
of
these
scale
detections
and
then
we
also
count
all
of
your
false
negatives.
So
if
there
any
windows
for
which
there
are
no
detections,
those
are
lots
of
false
negative.
So
we
count
up
all
of
them
and
add
in
a
scaled,
a
bearable.
A
Okay.
So
that's
the
scoring
function.
A
couple
of
other
details:
we've
defined
a
notion
of
application
profiles,
so
the
idea
behind
this
is
that
what
is
the
effect
of
a
false
positive,
with
the
importance
of
a
false
positive
versus
a
false
negative,
really
depends
on
your
application.
So
if
you
think
about
like
heart,
patients
again
and
eqg
EKG
data,
it's
okay
to
have
a
few
false
positives,
but
you
really
don't
want
to
have
a
false
negative.
That
would
be
very
bad.
A
In
the
other
hand,
if
you're
an
IT
or
in
DevOps,
if
you
work
with
those
folks,
they
really
hate
false
positive
they're
going
to
get
called
at
22.
Am
it
woken
up
there?
Really,
if
there's
too
many
false
positive,
you'll
just
turn
off
the
alarm,
so
in
that
case,
false
positives
are
really
bad.
They're,
actually,
okay,
with
a
few
false
negatives,
because
usually
systems
are
fairly
robust
to
individual
server
failures
and
so
on.
So
exact
ratio
here
really
depends
on
the
application.
A
So
what
we
did
is
we
created
a
system
where
you
can
kind
of
design
what
those
weights
are.
Nab
comes
with
three
application
profiles:
there's
a
standard
one
that
kind
of
equally
weights
those
two
there's
one
that
favors
low,
false
positives
and
there's
one
that
kind
of
favors
low,
false
negatives,
but
you
can
kind
of
tune
it
if
you
want
to
really
understand
the
impact
of
that
a
couple
other
details
in
that
we
really
want
to
make
sure
that
we
emulate
a
real
time
practical
application
scenario.
A
So
when
you
run
an
application
and
algorithm,
you
are
not
allowed
to
look
ahead.
Detections
must
be
made
on
the
fly
as
you
go.
Okay,
there's
no
separation
between
training
and
test
sets.
Every
metric
is
independent.
So
what
we
do
is
we
invoke
a
model.
We
start
streaming
the
data
to
that
detector
and
you're
supposed
to
detect
anomalies
as
you
go,
and
that
reflects
real
scenarios.
A
Another
really
important
thing
is
that
there
should
be
no
back
/
data
set
parameter
tuning.
You
can
tune
parameters
overall,
but
it
must
be
the
same
set
of
parameters
that
are
used
for
all
the
metrics.
If
you
do
any
parameter
tuning,
it
must
be
automated
and
done
on
the
fly
again.
This
reflects
practical
scenarios.
Now,
if
you've
deployed
machine
learning
algorithms,
you
know
these
are.
This
is
pretty
tough
criteria.
It
is
hard
to
do
that
in
practice,
but
that's
what
the
world
is.
A
Ok,
so
we've
tested
a
couple
of
different
algorithms
with
Knab
just
to
seed
it.
The
thing
to
remember
is
NAB
as
a
community
effort.
It's
open
the
goal.
Our
goal
is
to
have
researchers
independently,
evaluate
a
large
number
of
algorithms.
We
want
to
spur
the
industry
to
focus
on
streaming
analytics,
and
so
we
designed
it
so
it's
very
easy
to
plug
in
and
test
new
algorithms,
regardless
of
the
programming
language,
our
scenario
and
all
of
that's
kind
of
documented
repository.
A
What
I'll
show
is
some
seed
results
with
three
different
algorithms,
so
there's
hierarchical
temporal
memory,
which
is
our
algorithm
and
we've
been
working
on
streaming
and
anomaly:
detection
for
a
while
HTM
model,
temporal
sequences
and
data
and
they're
continuously
learning.
We
also
tested
etsy
skyline.
This
is
a
very
popular
open-source
anomaly
detection
technique.
It
uses
a
mixture
of
statistical
experts
and
is
also
continuously
learning
Twitter
released
an
out
nominee
detection
algorithm
early
this
year
and
also
made
it
open
source.
A
So
we
have
that
and
they
what
they
do
is
try
to
do
a
robust
of
outlier
statistics
and
they
also
do
a
little
bit
of
piecewise
approximation
to
get
a
sense
of
temporal
sequences.
Okay,
so
those
are
the
three
algorithms
that
we
tried
this.
You
know
we
were
happy
to
see,
perhaps
not
a
surprise
that
HTM
says
really
well
on
this
particular
data
set
on
this
benchmark
against
these
other
two
algorithms.
A
So
what
we
thing
to
look
at
here
is
that
the
perfect
detector
gets
a
score
of
hundred,
so
we
scaled
it
to
be
that
way.
New
menta
gets
about
64
on
the
standard
application
profile.
Twitter
is
a
little
bit
worse
and
then
skyline
was
the
worst.
All
three
are
quite
a
bit
better
than
chance,
which
is
at
16.8
now
there's.
None
of
these
detectives
are
perfect.
This
is
a
hard
data
set
and
there's
quite
a
room,
a
lot
of
room
between
the
performance
of
the
momenta
detector
and
perfect.
A
So
there's
a
lot
of
room
here
to
still
to
innovate.
There
what's
interesting
is
looking
through
some
of
the
results.
You
get
some
insights
into
some
of
the
challenges
here,
so
here's
an
example
data
set
which
is
CPU
usage
on
another
production
server.
There
are
two
anomalies
here
on
the
Left
there's
a
very
easy
spike
anomaly.
All
three
algorithms
detected
on
the
right
hand
side.
There
is
a
shift
in
the
CPU
usage.
A
A
Twitter
actually
doesn't
detect
that
until
much
later,
but
you'll
see
that
this
new
usage
is
now
the
new
normal
in
the
case
of
Twitter,
they
keep
giving
false
positives
for
quite
a
while
skyline
actually
does
really
well
there's
no
false
positives.
Afterwards,
the
new
mented
detector
does
give
a
couple
of
false
positives
afterwards
and
then
adapt
to
this
new
normal,
so
the
skyline
actually
work
best
on
it.
On
this
data
set
here's
another
one,
which
is
the
Machine
temperature
scenario.
A
There
are
two
focusing
on
the
last
two
anomalies,
all
three
of
them
detect
the
catastrophic
failure.
Only
the
new
menta
HCM
detector
detects
that
temporal
anomaly.
This
is
not
a
surprise,
because
the
HTM
actually
looks
at
the
temple
sequences,
not
just
in
I
ID
statistics.
In
this
case,
HTM
gives
one
false,
positive
and
and
skyline
gives
another
false
positive.
A
But
if
you
look
closely
there's
often
a
subtle
shift
that
proceeds
that
that
that
big
spike
so
there's
an
example
of
that,
in
this
case,
skyline
and
Twitter
both
detect
the
spike
but
the
HTM,
because
it's
looking
at
temporal
sequences
detect
that's
detects,
that's
up
in
a
subtle
shift.
This
is
really
hard
to
do
with
you're,
just
looking
at
outliers
and
statistics,
it's
very
hard
to
detect
that
subtle
ship
without
getting
a
lot
of
false
positives,
and
you
know
so.
A
Okay,
so
in
summary,
anomaly
detection
fail
is
the
most
common
current
application
in
streaming
analytics
as
an
AB
is
a
community
benchmark
for
a
streaming
anomaly
detection.
It
includes
a
comprehensive
labeled
data
set
with
real
data.
We
have
a
scoring
methodology,
that's
designed
for
practical,
real
time
applications
when
the
fully
open
source
code
base
and
a
looking
head.
A
You
know,
we've
seeded
this
with
a
few
algorithms,
but
we
really
hope
to
see
researchers
test
a
lot
of
other
algorithms
and
we
really
hope
to
see
more
focus
on
the
world
of
streaming
analytics
from
the
machine
learning
community.
We
hope
to
spark
improved
algorithms
for
streaming
and
for
version
2,
we'd
love
to
get
more
data
sets
I
know
many
of
you
have
data
sets.
A
Please
come
contact
us
if
you
get
willing
to
donate
the
data
set,
we
want
to
get
data
set
with
labeled
anomalies
in
there
and
if
your
machine
learning
practitioner
you
know
that's,
that's
gold,
there's
a
possibility
of
incorporating
some
from
UC
Irvine.
They
have
some
nice
streaming.
Data
sets
that
we
could
incorporate
yahoo
labs
released
a
streaming
data
set
earlier.
Unfortunately,
it's
not
currently
open
yet
so
we
can't
incorporate
it,
but
if
they
do
make
it
open,
then
we'd
be
able
to
do
that.
A
And
currently
you
know,
there's
a
lot
more
features
we
can
add
to
nab
as
well.
So,
for
example,
currently
it's
a
the
metrics
are
a
single
metric,
scalar
metric
value
plus
a
timestamp,
so
one
possible
extension
of
NAB
is
to
include
multivariate
anomaly
detection
again
this
is
dependent
on
us
getting
good
data
sets,
but
that's
another.
We
want
to
add
additional
features
to
nab
as
well
over
time.
A
Okay.
Finally,
here
are
some
resources.
We
do
have
a
table
and
we're
more
than
happy
to
answer
your
questions
and
have
you
participated
in
this?
The
repositories
up
there
there's
a
paper.
That's
going
to
peer
in
I
triple
e
IC
mla
in
december,
with
Alex
Lavin
and
myself.
There's
the
link
to
that
is
on
the
nav
repository,
but
the
PDF
of
that
is
up
in
archive
and
then
there's
our
contact
info
there.
Thank.