►
From YouTube: Realtime Data Analytics at Datadog by Alexis Le-Quoic
Description
More on realtime data analytics here: http://www.hakkalabs.co/articles/realtime-data-analytics-at-datadog
realtime data analytics
real time data analytics
ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.
FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520
A
A
So
what
I
figured
we'd
talk
about
tonight,
there's
a
few
things.
So,
first
of
all,
and
that's
not
going
to
be
product,
you
know
not
a
product
pitch,
but
just
explain
what
the
problem
space
is,
what
it
is
that
we're
dealing
with.
So
what
we
do
and
who,
who
is
the
user
of
datadog,
then
from
there
we'll
go
into
a
little
bit
of
domain
constraints
so
with
based
on
what
the
problem
space
is,
what
constraints
it
entails
and
what
constrains
it
doesn't
tell
us
what
kind
of
trade-offs
you
can
make.
A
You
know,
authorities
or
anything
like
that-
also
look
at
the
tools
that
we
use:
data
stores,
languages,
libraries
and
so
on
and
so
forth
and
last
a
few
challenges
that
that
that
we
faced
so
we'll
kind
of
you
know
just
discuss
a
few
technical
challenges
that
is
okay.
So
what
data
dog
is?
Is
it's
infrastructure
monitoring
as
a
service?
So
it's
a
monitoring
tool
and
I'm
so
who?
Who
here,
uses
a
monitoring
tool?
You
know
as
an
infrastructure
monitoring
tool?
A
A
Web
web
front
ends
load
balancers
web,
so
web
tiers
application
servers,
it's
even
bare
metal,
you
know,
aws
or
and
so
on
and
so
forth.
So
the
idea
is
that
you
have
one
place,
because
your
application
is
typically
built
on
top
of
a
bunch
of
these
components.
You
have
one
place
where
you
can
see
how
the
app
is
is
performing
so
so
far,
so
good
all
right
now.
So
what
is
really
monitoring
as
a
highest
level?
What
what
does
it
mean?
So
really
it's!
A
So
it's
a
tool.
It's
a
system,
it's
whatever
you've
installed,
whatever
wakes
wakes
you
up
at
night
when
something
goes
wrong,
but
really
it's
the
primary
interface.
You
have
to
understand
your
application.
So,
if
you're
running
an
app,
whether
it's
if
it's
non-trivial,
if
it's
just
distributed
application,
you'll
end
up
having
something
that
tells
you
how
well
it's
doing
where
it's
failing
the
bottlenecks
and
so
on,
and
that's
your
monitoring
system.
So
it's
really
it's
the
primary
interface.
We
can
learn
and
understand
how
your
system
is
doing
you.
A
Can
I
mean
if
you
go,
if
you
say,
if
you
have
a
data
center,
you
can't
just
go
walk
in
the
data
center.
Look
at
the
racks
and
kind
of
understand
what's
happening.
Obviously
so
so
you
need
something
to
sort
of
to
mediate
that
and
that
is
in
essence,
your
monitoring
system.
Now,
in
the
worst
cases,
your
monitoring
system
is
your
customers,
because,
basically
they
they
tell
you
when
something
goes
wrong
and
you
know
you
fix
it,
that's
not
the
I
mean
it's
one
way
to
do
it.
A
It's
not
the
way,
I'd
recommend,
but
it's
been
done
in
the
past.
In
the
better
case,
is
you
have
something
that
kind
of
sucks
performance
metrics
all
the
time
you
know
around
the
clock
from
your
various
components
from
your
software
and
tells
you
hey.
This
is
going
well.
This
is
you
know,
there's
a
problem
here,
but
that's
that's
essentially
what
it
is.
Two
quick,
quick
taxonomy
about
metrics,
so
the
monitoring
system,
what
it
consumes
first
and
foremost,
is
metrics.
It's
and
metrics.
A
So
an
example
of
a
metric
is,
for
instance,
at
the
lowest
level,
could
be
how
much
cpu
a
particular
machine
is
or
how
busy
is
a
cpu
in
a
given
machine
at
the
highest
level.
It
could
be
how
many
users
on
the
site
at
a
given
time,
so
it
really,
it
really
varies.
The
taxonomy
is
relatively
simple,
though,
so
you
have
in
terms
of
metrics
that
you're
going
to
care
about
it's
either
about
resource
or
workload,
so
resource
metrics
is
stuff
you
consume,
so
it
could
be.
You
know.
A
Cpu,
network
memory,
storage
and
so
on
and
workload
is
what
you
produce,
what
your
application,
what
it
is
that
you're
running
on
the
box
is
producing
or
or
the
application
as
a
whole
is
producing
and
really
to
understand
how
things
are
going,
how
your,
how
your
application
your
system
is
performing
and
that
by
the
way
applies
to
could
be
a
factory.
It
could
be
anything
you
know
in
the
case
of
a
factory
you
have,
you
know
you
get
stuff
in
you,
produce
widgets
and
workloads
how
many
widgets
you
produce
an
hour.
A
So
that's
a
physical
work
in
our
world.
The
software
world
so
resources.
Typically,
you
know
with
the
machines
the
resources,
the
machine
and
workload.
Is
you
know
whatever
it
is
that
you're
producing,
so
this
taxonomy
applies
kind
of
regardless
of
of
the
application
that
you
you've
written
or
you
use,
and
that's
going
to
be.
You
know
that.
That's
that
that's
one
way
to
look
at
the
metrics,
and
so
the
monitoring
system
will
first
and
foremost,
consume
these
metrics
and
make
some
judgment
or
display
display.
That
will
make
some
judgment
about.
B
A
A
What,
when
you
use
the
monitoring
system,
first
and
foremost,
you're,
basically
graphing
a
lot
of
stuff,
so
this
is
one
example,
and
the
idea
is
that
you
can
you're
not
graphing,
because
you
like
colors
and
you
like
stuff
that
moves-
and
you
know
it's
cool,
you're
graphing,
because
really
quickly
with
you
know,
you
can
see
interesting
patterns.
You
know
here
and
there
there
are
different
metrics,
but
they
exhibit
the
same
pattern,
whether
it's
kind
of
flat,
whether
this
is
getting
bad
here
or
so,
whether
there's
a
spike
here.
A
So
the
idea
is
that,
let's
walk
through
an
example
of
monitoring,
so
you
have
there
are.
I
have
here
two
metrics
one
of
throughput
and
one
of
contention,
so
it
could
be.
You
know
anything
could
be
number
of
pages
served
or
whatever
number
of
people
walk
in
the
subway
at
a
given
station
and
contention
is
in
so
the
measure
of
how
many
unique
things
are.
A
A
A
So
I
mean
this
is
this:
is
the
this
in
particular?
Is
the
number
of
locks
in
a
relational
database
so
and
and
that
it
could
be
anything?
But
in
this
case
that's
what
it
is
and
that
that
application,
the
throughput
of
that
application,
depends
on
that
relational
database,
and
so
whenever
it
starts
to
lock
up
everybody
in
that,
in
that
application
starts
to
wait
for
our,
you
know,
for
their
turn
to
grab
the
lock
or
the
locks
you
know,
as
it
were,.
B
A
All
right
so
the
second
thing,
so
you
do
graphing
and
you
know
you
manipulate
the
data.
The
second
thing
you
want
to
do
with
the
monitoring
system
is
you
do
a
lot
of
alerting
and
alerting
is
really
automatic
detection
because
you
don't
have
time
to
watch
a
thousand
metrics.
So
that's
you
know
defined
alerts
threshold
I
mean
you
can
do
a
number
of
different
things,
but
that's
that's
the
other
thing.
A
Another
very
common
use
case
of
of
monitoring,
and
the
third
use
case
is
more
of
a
what
I
call
team
sport
is
that
whenever
you
have
a
complex
application,
you
need
to
be
able
to
talk
about
the
data.
You
know
you
just
you
have
a
lot
of
back
and
forth,
so
this
is,
for
instance,
one
discussion
we
had
on
the
facebook
metric,
that's
kind
of
actually
talking
about
this
improvement.
A
So
that's
really
the
three
main
use
cases
of
a
monitoring
system.
It's
so
you
do
a
lot
of
graphing
it's
real
time.
Of
course
I
mean
hence
the
hence
the
title
of
the
talk.
It
is
you
so
you
graph,
you
do
a
lot
of
graphing.
You
alert!
You
want
to
learn
from
the
metrics
and
then
you
do
a
lot
of
collaboration
around
the
data
to
try
to
understand
really
understand
what
the
application
is
doing,
because
if
you
just
look
at
the
architecture
diagram,
you
don't
really
understand
much.
That's
the
sad
truth.
A
So
knowing
that,
what
are
no
that's
a
problem
in
space,
that's
how
that's
the
use
cases
if
you
will
so
the
highest
level
of
the
description
of
the
problem.
So
what
kind
of
constraints
does
that
does
that
entail?
And
the
idea
again
is
that
we
start
from
kind
of
the
highest
level
use
cases
and
we
look
at
the
constraints,
and
then
it
tells
us
what
kind
of
trade-offs
we
can
do
and
then
that's
going
to
help
us
find
you
get
to
some
architecture
so
domain
constraints.
A
The
first
one
is
the
is
latency.
So
when
I'm
in
real
time
there's
a
lot
of
understanding
of
real
time,
I
mean
real
time
in
the
sense
that
this
is
a
10
to
30.
Second
latency,
between
the
time
you
collect
a
given
metric,
for
instance,
how
much
cpu
a
machine
is
using
to
the
time
you
can.
You
can
see
it
on
on
a
graph,
so
it's
it's
10
to
30
sec,
it's
typically
within
10
or
30
seconds.
There's
a
number
of
factors.
It
could
be
one
second
or
half
a
second
there's.
A
A
number
of
factor
that
play
there
could
be
time
it
takes
to
send
the
data
and
time
takes
to
process
it
and
so
on
so
forth.
The
interesting
thing
is
that
so
it's
really
time
to
signal
so
time
from
the
moment
you
collect
there's
a
program
somewhere
that
collects
the
metric
to
the
time
you
receive
it,
and
your
brain
can
do
something
with
it.
It's
not
really
page
load.
So
the
page,
the
page
is,
you
know
a
little
bit
faster
than
that.
Hopefully,
but
that's
one
key
thing.
A
So
the
idea
is
that
if
you,
so
it's
not
200
milliseconds,
it's
not
one
microsecond
and
because
for
this
particular
domain,
you're
not
going
to
be
able
to
react
to
there's
a
cpu
spike
of
a
microsecond,
because
what
are
you
going
to
do?
There's
no
there's
nothing
actionable
there.
It's
also
not
10
minutes
in
the
sense
that
you
don't
want
to.
You
know,
find
out.
You
don't
want
your
customers
to
call
you
and
say:
hey
the
website
is
slow
and
because
they
in
10
minutes,
they
can
figure
out
that
there's
a
problem.
A
So
that's
where
10
to
30
seconds
kind
of
maps
to
how
fast
somebody
and
people
can
look
at
the
graph
rock
it
and
then
react
on
it.
So
that's
first
first
constraint.
Second
constraint
is
it's
availability,
I
mean
that's
kind
of
standard,
but
in
this
case
we
get
data.
When
you
run
a
monitoring
service,
you
get
data
24,
7,
365,
blah
blah,
there's
no,
there's
no
seasonality,
because
it's
just
machines
talking
to
the
machine.
So
you
know
it's
thanksgiving
it'll
be
as
busy
as
you
know.
Tomorrow
it
doesn't
really
matter.
A
It
just
increases
ever
so
slowly
as
we
have
more
uses,
more
machines
reporting
to
the
system,
and
the
third
key
constraint
is
that
you
want
to
have
access
control
and
that's
very
generic
to
any
multi-tenanted
system.
So
if
you
run
something
as
a
service,
usually
you'll
have
all
your
data
in
one
place,
but
you
have
multiple
users
that
don't
necessarily
want
to
see
each
other's
data.
So,
in
our
case,
that
means
obviously
that
you
know
the
data
that
we
receive
has
to
be
partitioned
by
account.
A
So
now,
let's
look
at
degrees
of
freedom
and
I'll
try
to
make
it.
You
know,
as
the
talk
advances,
it
gets
more
and
more
concrete,
so
fair
enough,
so
degrees
of
freedom.
What
kind
of
trade-offs
you
can
make?
So
I
talked
a
little
bit
about
the
100
millisecond
round
trip.
I
don't
care
if
I,
if
it
takes
100,
you
know
50
milliseconds
between
the
time
I
gather
metric
on
a
machine
at
the
time.
I
can
see
it
because
if
it
takes
two
seconds,
it'll
be
the
same
result.
A
You
know,
that's
there's
not
a
time,
so
there's
no
point
in
over
engineering
to
get
like
super
fast
latency.
So
this
is
not
a
financial
trading
application.
This
is
really
monitoring
monitoring
infrastructure.
So
that's
we
can
get
away
with
that.
The
second
thing
is,
we
don't
have
to
engineer
for
crazy
elasticity.
You
know
certain
bursts
black
friday,
cyber
monday,
etc,
etc.
That's
not!
We
don't
really
care
in
our
case,
what
really
drives
the
amount
of
volume
data.
A
The
volume
of
data
we
get
is
how
many
customers,
or
how
many
users
connect
their
machines
to
data
logs.
So
we
can
start
streaming
streaming
metrics
to
us.
The
third
one
we
can
get
away
with
to
some
extent
is
precision
and
that's
an
interesting
one,
so
precision
kind
of
acts
as
a
counterbalance
or
not
a
counterbalance,
but
can
be
antagonistic
with
latency.
A
The
problem
is
it's
late,
so
I
mean
the
the
the
x
axis,
which
is
time,
doesn't
really
matter
what
it
is,
but
you
know
the
actual
scale,
but
it's
not
very
useful
because
here
you'd
have
to
basic
decision
something
that
happened
maybe
an
hour
ago,
ten
minutes
ago,
it's
much
more
useful
to
have
something,
that's
less
precise,
but
has
lower
latency
and
that's
kind
of
a
key
thing.
I
mean
that's,
that's
speaking
related
to
the
real-time
aspect
of
the
of
the
of
the
architecture
and
of
the
talk
is
that
I'd?
A
Much
rather
have
this
than
that.
I
don't
care
for
a
high
level
precision,
and
so
that's
an
interesting
trade-off,
because
that
means
that,
if
you're
as
as
you're
processing
data
as
you're
receiving
data,
you're
graphing
it
if
your
processing
systems
start
to
slow
down,
if
you,
if
your
pipeline
starts
to
slow
down,
you
can
say
all
right,
I'll
skip
a
bunch
of
points
to
just
catch
up,
because
I
don't
care
of
having
every
single
point
there.
I'd
rather
make
sure
that
I
can
catch
up
and
be
be
up
to
date.
A
A
A
Okay,
cool
yeah.
Do
do
tell
me
if,
if
it's
not,
if
I'm
not
making
sense.
So
having
said
that,
I
want
to
take
a
quick
look
at
what
the
monitoring
data
what
it
looks
like
really
from
you
know,
maybe
even
more
concretely.
So
there
are
two
types
of
events:
two,
I'm
sorry
two
types
of
data
in
monitoring
really
so
you
have
metrics
and
events,
and
so
metrics
are
really
basically
time
series.
So
it's
the
metric
could
be
the
temperature
of
this
room.
You
know
hour
after
hour.
A
It's
usually
anything
it
could
be
number
of
bytes
processed
by
an
interface
inside
a
machine.
It
could
be
number
of
visitors
to
your
side.
It
can
be,
can
be
pretty
much
anything
anything,
that's
a
that's
a
time
series
so
and
events
are
just
annotation
I'll.
Show
you
a
little
bit
more
detail
so
metrics
time
series,
so
at
the
most
basic
level
is
basically
it's
time
stamps
and
values.
That's
that's!
The
basic
representation
has
has
a
bunch
of
metadata.
It
could
be
the
name
of
the
metrics.
A
A
That's
pretty
much
it
I
mean
in
rc7,
datadog,
that's
kind
of
the
core
data,
that's
the
core
data
model.
If
you
will
so
it's
very,
very
simple,
it's
very,
very
uniform,
there's,
not
a
ton
of
variations.
The
resolution
we
chose
is
that
of
a
second
and
that's
really
again
back
to
what
I
was
saying
in
terms
of
it
takes.
You
know,
at
least
you
know
10-30
seconds,
to
interpret
a
graph
to
be
able
to
be
able
to
react
to
it
to
to
understand
what
the
data
is
telling
you.
A
So
there's
no
point
in
measuring
things
at
a
resolution
lower
than
100,
then
for
one
second
and
it's
adequate
to
capture
those
signals
of
less
than
0.5
hertz
and
now,
when
you
think
about
it
the
way
so
the
way
it
really
works.
If
you
set
up
data
dog,
you,
for
instance,
deploy
it
on
your
laptop
you'll
have
a
little
program.
Little
agent,
that's
going
to
run,
collect
metrics
about
how
what
your
cpu
is
doing,
what
your
memory
is
doing,
which
program
are
running
and
so
on
and
so
forth.
A
That
typically
is
going
to
sample
data
every
10
to
20
seconds.
It
could
sample
more
often,
but
you
don't
want
to
you,
don't
want
to
waste
cycles
to
measure.
You
also
want
your
laptop
or
your
server
to
do
interesting
things
useful
things.
So
that's
why,
even
though
we
can
measure
things
down
to
a
second
typically,
if
you
look
at
at
graphs
we're
giving
you
there
you'll
have
a
point
every
10
to
20
seconds.
A
If
you
have-
and
this
is
maybe
slightly-
I
would
say
off
topic-
but
in
case
you
want
to
measure
higher
frequency
signals,
you
can
use
something
called
statsd
which
will
pre-aggregate
whatever
metrics
you
want
to
send
it
and
we'll
consume
that
so
overall
metrics
the
interesting
thing.
There
are
two
interesting
things:
one
they're
pretty
high
signal
noise
ratio
in
the
sense
that
if
you
have
a
very
noisy
metric,
something
that
goes
all
over
the
place-
it's
probably
not
very
useful
in
the
first
place,
so
you
need
to
derive
it
to
get
something
useful.
A
So
it
also
speaks
to
the
fact
that
you
can
lose
some
precision
and
still
kind
of
be
able
to
understand.
You
know
what
the
metric
is
telling
you,
what
the
performance
of
the
system,
for
instance,
is,
and
the
second
thing
which
is
great.
It's
is
that
metrics
are
immutable
so
because
there's
a
time
stamp
and
nobody's
figured
out
time,
travel
yet
will
be
able
to
you
to
persist,
metrics
and
then
they're
never
going
to
change
because
they
measure
something
that
happened
in
the
past.
A
Essentially
so
quick
example,
if
you
want
to
measure
so
io
8
is
the
I
use
in
this
case
resource
metrics.
So
what
what
the
machine
is
doing?
What
the
cpu
is
doing?
This
is
number
of
cycles
in
percentage
that
are
spent
waiting
on
on
some
kind
of
disk,
for
instance.
So
we
have.
This
is
the
example
of
a
query.
A
So
now
the
result,
if
you,
if
you
send
this
query
into
data.,
because
use
case
number
one
graphing,
we're
mostly
interested
in
displaying
data,
there's
no
point
in
giving
you
a
million
data
points
back
if
we're
going
to
compress
that
to
a
graph,
that's
going
to
fit
in
your
screen.
So
it's
typically,
you
know,
200
to
600
points
for
display,
and
these
points
are
essentially
times
that
value
times
the
value
times
the
value
and
when
you
graph
them.
This
is
what
you
get
you
know
you
can.
A
I
mean
this
is
something
that's.
This
is
actually,
I
think,
a
measure
on
on
a
machine.
So
that's
something
I
took
I
think
this
morning.
So
the
interesting
thing,
because
graphing
is
one
of
our
main
use
cases.
The
resolution
isn't.
The
resolution
is
time
independent.
By
that
I
mean,
if
you
look
at
one
hour
data
you
get
say,
600
points.
If
you
look
at
one
month
of
data,
you
also
get
600
points,
so
there's
clearly
some
time
there's
a
time
aggregation
that
takes
place
so
that
we
don't
the
system,
doesn't
query.
A
If
you
look
at
a
month
of
data
and
you
have
two
million
points,
we
don't
query.
Two
million
points
sent
back
to
the
browser
two
million
points
and
wait
for
the
browser
to
render
two
million
points.
Of
course,
we
we
act.
We
time
aggregate
as
we
query
and
return
to
you
just
200
to
600
points.
So
you
have
a
nice
graph.
A
So
if
you
want
to
get
a
little
bit
more
complex
and
I
want
to
explain
a
little
bit
also
kind
of
some
of
the
intricacies
of
of
time
series
and
monitoring-
let's
say
you
want
the
same
metric
but
average
across
the
number
of
machines,
a
number
of
sources,
if
you
will
so
the
query
is
slightly
different.
I
don't
it's
not
that
important.
A
A
So
what
we
do,
what
we
call
spatial
aggregations,
so
you
basically
take
this,
which
is
the
metric
for
all
the
machines
and
you
compute
the
average
so
because
at
the
end,
this
is
what
you're
asking
the
system
to
produce,
and
so
one
interesting
thing
is
that,
because
not
all
data
is
sampled.
At
the
same
time,
you
have
to
do
some
form
of
interpolation
so
that
you
can.
You
can
basically
compute
the
average
on
on
the
same
on
the
same
timestamps.
A
Does
that
make
sense
that
part
okay,
cool?
If
it
doesn't?
You
know
please
raise
your
hand,
so
so
this
this
time
aggregation
this
space
aggregation
happening.
So
that's
these
are
examples
of
metrics
and
the
kind
of
processing
we
do
on
them
and
the
kind
of
queries
you
can
do.
Often
events
are
really
so.
This
is
much
more
simple,
so
the
kind
of
stuff
we
collect
is
a
really
time-based
textual
annotation.
So
it
could
be.
A
Oh,
I
released
a
new
version
of
the
software
or
oh
somebody
got
a
page
because
a
web,
a
web
machine
went
down
or
anything.
Oh,
we
started
a
new,
you
know
marketing
campaign
and
so
on
so
forth.
So
the
data
model
is
very
simple:
it
has
title
body
and
then
some
metadata
author
attacks
as
an
example.
This
is
so
here.
These
are
events,
so
nagios
is
a
is
a
sort
of
legacy
monitoring
system
we're
collecting
just
so
this
is
the
metadata
is.
What
is
what
it
is?
A
There's
a
title:
there's
a
body
and
timestamp
of
course,
and
then
that's
it.
It's
interesting
to
you
to
mix
events
and
metrics
because
it
lets
you
mark
it
lets
you
annotate
the
data
because
a
lot
of
times,
for
instance,
you,
let's
say
you
start
a
new
marketing
campaign,
and
so,
if
you
don't
know
that,
and
you
see
a
huge
spike
in
your
web
traffic,
you
like
shoot,
you
know.
Maybe
the
cdn
is
messed
up
and
we
get
all
the
traffic
back
to
origin
server
and
we're
going
to
just
die.
A
You
know
everything
is
going
to
burn
burn
down
to
to
the
ground,
but
maybe
it's
actually
because
we,
you
know,
there's
a
new,
really
successful
marketing
campaign,
that's
just
driving
more
traffic
and
just
the
reaction.
Obviously
you
know
it's.
You
you'll
be
careful
in
both
cases,
but
in
in
the
second
case,
where
you
know
you
can
you
know?
A
Oh,
this
is
because
there's
that
or
it's
probably
because
there's
this
new
marking
campaign
being
able
to
sort
of
mix
and
match
it's
really
not
very
visible,
but
there
are
markers
where
I
have
the
the
origin
of
the
the
arrows
there
markers
it's
really
useful,
because
now
you
have
a
sense
of
oh,
it's
a
brand
new
event.
I
have
no
idea
this
we're
sort
of
in
twilight
zone.
I
don't
know,
what's
going
on
or
yes,
okay,
this
is
expected
and
then
this
is
what
how
we
can
react.
A
So
that's
it.
I
mean
that's
the
core
data
model
in
our
in
our
in
our
architecture,
in
the
prominent
spaces.
That
and
the
rest
is
just
it's
not
extra,
but
it's
not
not
that
important.
Okay.
So
now
we
know
what
kind
of
data
we
produce
or
gets
materialized
in
the
system.
Let's
look
at
the
way,
they're
accessed
how
how
how
data
flow.
So
in
most
cases
I
mean
this
is
a
the
simplified
workflow.
A
You
have
things
so
agents
being
small
machines,
small
programs
on
a
machine
collecting
metrics,
sending
them
through
data
or
just
so
programs
hitting
our
api
just
send
data,
so
it
gets
into
something
that
we'll
call
intake,
gets
queued,
gets
processed,
get
persisted
and
then
that
data
is
accessed
basically
either
because
you're
looking
at
a
graph
via
a
browser,
and
that
graph
may
be
the
last
hour
of
data,
for
instance,
which
is
very
common.
Is
you
want
to
know?
What's
happening
now
and
once
in
a
while
you'll
go
back
and
say?
A
Oh,
let
me
look
at
what
happened
a
week
ago
or
last
month
or
something
like
that,
but
the
vast
majority
of
the
traffic
outbound
so
inbound
is
all
the
telemetry
all
the
metrics
coming
from
the
servers
and
the
applications.
Outbound
is
it's
a
little
bit
of
recent
data
and
then
very
frequently
kind
of
the
more
random
queries.
A
By
that
I
mean
all
the
customers
send
all
data
all
the
time.
There's
no!
Oh
today's
tuesday,
I
don't
know
it's
customer
a
it's
going
to
send
data,
so
it's
not
batch.
It's
always
like
constant
flow
of
stuff
coming
in
the
latency
constraint
that
I
mentioned
about
so
actually
so
here
I
have
30
60
seconds
more
attention,
30
seconds.
So
that's
again
between
the
time
the
data
is
collected
to
the
time
it
flows
into
the
intake.
A
It's
gonna
be
in
the
order
of
you,
know
tens
of
seconds,
not
minutes,
not
milliseconds,
and
so,
when
you
have
this
kind
of
stuff,
particularly
when
you
have
you're
ingesting
data,
you
know
at
a
very
constant
rate
and
you
have
a
sort
of
fairly
sort
of
constant
upper
bound
for
latency.
You
end
up
basically
having
just
a
bunch
of
queues.
That's
that's
what
I'll
show
in
a
little
bit.
A
A
An
interesting
thing
is
that,
and
that's
maybe
a
little
bit
so
I'll,
explain
exactly
the
the
queries
when
you
look
at
the
parameters
of
the
queries,
so
this
time
span
there's
a
metric
you're
interested
in
the
let's
say
the
machine
where
it's
collected.
These
are
when
you
render,
when
you
look
at
dashboards,
that
people
put
up
of
graphs,
they
tend
to
be
slow,
moving
dimensions,
that
is
to
say
either
you
look
at
a
bunch
of
different
metrics,
but
it's
the
same
time
span.
So
it's
the
time
is
fixed.
A
Basically,
you
do
a
bunch
of
queries,
but
the
time
is
fixed
or
you
look
at.
Typically,
people
may
look
at
everything
coming
from
a
single
machine
or
one
given
metric
across
a
time
machine.
So
in
that
sense,
when
you
do
the
query,
there's
already
some
sort
of
locality
of
the
of
the
parameters
not
completely
completely
random.
A
And
other
kind
of
stuff
that
we
have
outbound,
which
I
touched
on
this
is
use
case.
Number
two
is
alerts
is
telling
you
when
something
goes
wrong.
So
it's
detecting
anomalies.
Latency
is
typically
60
seconds
because
you
need,
you
know,
there's
no
point
in
alerting
as
soon
as
a
metric
shoots
up
across
above
a
certain
level.
Usually
that's
not
what
you
want,
because
if
you
have
a
certain
spike
you'll
get
a
ton
of
false
positives,
say:
hey,
hey,
I'm
above
100,
hey
I'm
over
100!
You
don't
want
that!
You
want.
A
Usually
you
want
to
know,
hey,
I'm
above
100
and
that's
been
the
case
for
the
past
five
minutes
or
the
past
minute
or
past
hour,
so,
depending
on
the
on
the
case
on
the
use
case.
But
for
instance,
I
mean-
let's
say
I
mean
anything
around
like
cpu,
for
instance,
is
a
good
example
where
you
don't
want
to
you
know
the
spike
in
cpu
doesn't
really
matter.
You
want
to
know,
you
don't
want
to
know
it.
A
You
know
every
second
that
something's
happening
if
it's,
for
instance,
storage
or
some
kind
of
memory,
something
that's
constrained
where
you
don't
have
an
infinite
amount
of
it.
You
want
to
know
once
you're
getting
close
to
the
limit
because
there's
no
extending
it
like
you
know
if
you
run
a,
for
instance,
this
space
you're,
not
in
this
space,
there's
no,
you
want
to
know
as
soon
as
possible.
A
So
so
that's
alerts
really
and
the
spatial
aggregation.
If
you
it
makes
things
a
little
bit
more
complex
when
you
have
a
bunch
of
different
time
series
that
that
come
at
different
point
in
time
and
makes
things
a
little
bit
fuzzy,
especially
towards
sort
of
the
boundaries
of
the
interval,
and
the
last
thing
we
want
to
send
is
events.
A
So
so
that's
inbound
outbound
in
the
middle
there's
a
bunch
of
stuff.
So
I
it's
it's
all
the
stages
I'll
go
over
them
in
a
little
bit.
Basically,
things
that
will,
for
instance,
look
at
the
incoming
metrics
and
look
for
mlas
persist
them
do
roll
ups
to
aggregations
that
you
can.
You
can
have
sort
of
constant.
You
can
keep
your
latency
fairly
low
and
do
some
kind
of
reporting.
A
If
you
want
to
resolve
some
something
right
in
you
know
in
in
the
flow
you
want
to,
for
instance,
resolve
a
metric
name
from
from
some
from
a
name
to
an
actual
id
identifiers
to
10
milliseconds.
If
you
want
to
query
and
assemble
time
series
a
different
resolution,
for
instance,
if
you
want,
if
you
say
I
want
to
see
a
time
series-
that's
that
spans
a
month,
then
it
you
should
take
about
a
second.
I
know
more
and
more
detection.
I
talked
about
it
typically
60
second
latency
or
60.
A
Second
time
span
or
latency,
no,
no
less
than
that.
If
you
don't
want
to
do
hourly
aggregates,
so
anything
any
stage
in
the
in
the
processing
pipeline
that
rolls
up
hourly,
then
obviously
it's
every
60
minutes
that
it
needs
to
wake
up
and
just
do
its
stuff
to
compute
the
the
aggregates
for
the
past
hour,
same
thing
for
dailies
and
then,
if
you
do
bash
reports
every
week
and
then
it
takes,
you
know
it's
like
you
run
out
of
seven
days,
so
the
multiple
latency.
A
The
only
reason
why
I
wanted
to
have
have
it
there
is-
is
to
show
kind
of
the
different
time
scale
that
that
exists
throughout
the
system
and
how
and
as
you
can
imagine
so
something
that's
disk
based
10
milliseconds,
probably
not
a
good
idea,
that's
probably
going
to
mean
something
in
memory,
so
one
second,
that's
probably
something
in
memory
as
well.
60
seconds
could
be,
could
be
memory.
It
could
be
on
disk
depending
on
the
volume
and
so
on
so
forth.
So
that's
why
I
had
that
interesting.
A
The
interesting
observation
also
that
again
is
going
to
drive.
The
architecture
is
that
the
volume
of
data
inbound
volume
is
way
way
bigger
than
the
volume
of
data.
That's
outbound!
That's
because
you
have,
I
don't
know.
Let's
say
you
know:
100
000
machines
telling
datadog
hey!
This
is
how
I'm
doing
this
is.
These
are
my
metrics
and
then
you'll
have
that
1000
people
looking
in
front
of
1000
screens,
so
each
one
screen
each
let's
say
even
two
looking
at
10
graphs.
A
You
know
durable
queue,
basically
something
that
can
take
the
day.
That
has
a
predictable
intake
time
so,
where
you
want
to
write
data
persisted
so
it
can
be
processed
later
on
it
shouldn't.
Take
you
know
you
should
do
the
time
it
takes
to
to
get
the
data
should
not
vary
because
otherwise
you're
backing
up
at
the
very
entrance
or
the
very
start
of
the
pipeline,
and
that
means
your
customers
are
backing
up
as
well.
A
So
that
means
they're
not
going
to
get
the
data
on
time
and
it's
going
to
be
scalable
just
because
I
said
that
our
sort
of
intake
volume
is
just
ever
so,
you
know
grows
and
it's
going
to
be
scalable
in
the
sense
that
we
want
to
spend
less
money.
If
we
double
the
amount
of
customers,
for
instance,
we
don't
want
to
double
our
bill
to
ingest
the
data.
A
We
want
to
have
something
that's
lower
than
that,
so
that's
caliber
in
that
sense,
scalable
also
in
the
sense
that
it
has
to
consume
a
lot
of
data,
but
that's
that's
that
outbound
design
constraints
so
for
outbound
traffic.
You
want
to
have
fast
access
to
real-time
data,
because
people
want
to
see
what's
happening
now.
So
that's
going
to
be
the
main
use
case.
I
want
to
see
all
my
graphs
for
what's
happening
now,
so
you
want
to
have
something
that
gives
you
fast
access.
A
You
also
want
to
have
fast
access,
really
fast
access
to
persistent
data,
I.e
past
data,
for
instance,
where
you
want
to
see.
Oh,
I
have
you
know:
20
000
people
on
my
site
right
now.
How
much
did
I
have
a
week
ago?
At
the
same
time,
you
know
same
day
just
a
week
ago.
At
the
same
time,
so
that's
prob,
that's
gonna,
be
that's.
Gonna
live
somewhere,
I
mean.
If
you
have
a
lot
of
memory,
it
can
be
a
memory,
but
at
some
point
you're
gonna
have
to
persist.
A
Just
do
something
that
is
not
memory,
so
you
you
need
something
that
gives
us
access
to
it
and
internally.
The
kind
of
constraints
that
we
have
is
again
it's
all
cute,
it's
a
bunch
of
cues,
so
they've
got
to
be
pretty
fast.
We
want
to
have
minimal
coupling
between
the
various
aspects.
If
it's
a
pipeline
that
process
your
data,
you
don't
want
to
have
crazy
coupling,
because
if
something
happens
somewhere,
you
don't
want
the
problem
too.
A
Okay,
too
quickly,
and
then
we
need
a
bunch
of
data
stores
that
that
ranges
in
durability
and
access
sort
of
inversely.
So
you
can
have
low
durability
database,
that
is
to
say,
the
data
can
go
fairly
quickly
and
you
may
lose
it,
but
very
fast.
So
that's
typically,
you
know
stuff
in
memory
and
then
you
lose
the
box
and
then
the
memory
is
gone
and
then
that's
it.
I
mean
you
have
to
find
ways
to
fight
that
all
the
way
to
super
high
durability.
That
is
to
say,
I
don't
care
but
slower.
A
That
is
to
say,
I
don't
care
if
it
takes.
You
know
a
second
to
get
the
data,
but
that
data
has
got
to
be
there,
no
matter
what
so,
because
you
know
we
are
we,
we
store
historical
time
series
and
that's
kind
of
that's.
We
can't
say
oops,
we
lost
the
data.
Sorry,
you
know
we're
just
come
back
later:
okay,
so
from
a
so
adding
a
little
bit
a
few
pieces
to
the
to
the
equation.
A
So
really,
this
is
what
it
looks
like
you
get
data
in
either
from
the
typically
from
through
the
api
so
or
or
the
machine
descending
to
api
hey.
This
is
how
much
shipping
you're
consuming,
for
instance,
there's
a
bunch
of
there's
a
lot
of
machines
consumed
so
acting
as
intake
so
ingesting
the
data,
basically
storing
it
into
either
both
in
memory
and
then
chewing
it
on
disk.
A
So
the
in
memory
why
you
want
to
have
it
here
is
because,
if
you,
if
somebody's
asking
for
the
data
in
real
time
you
want
to
cut
through,
you,
don't
want
to
read
from
disk
what
you
just
what
you
what
you
just
wrote.
Rather,
if
it's
memory,
when
you
do
a
query,
you
can
the
web
tier,
that's
here
can
directly
hit
the
storage
and
say
hey.
You
know
I
want
the.
I
want
the
up
to
the
second
data,
just
give
it
to
me
and
it's
going
to
be
memory.
A
It's
going
to
be
just
much
faster
and
because
again
it's
a
it's
a
it's
going
to
be
a
small
fraction.
The
outbound
traffic
is
going
to
be
a
small
fraction
in
band
traffic.
You
know
you
can
you
can
play
it
in
a
few
games,
try
to
optimize
that
so
otherwise
the
path
is
you
go
to
cues
so
stuff
that
gets
queued
up.
A
It
gets
and
then
processed
and
then
written
to
some
durable
store
and
I'll
explain
which
which
store
we
picked
and
then
there's
a
bunch
of
cataloging
that
happens
as
well
at
the
bottom.
So
all
that
which
we
call
data
layer
is
something
that,
from
the
outside
from
the
web
from
the
web.
Application
looks
like
something
you
ask.
Give
me
this
metric
between
time,
a
and
time
b,
and
it's
going
to
return
the
time
series.
A
It
does
the
magic,
for
instance,
if
you're
asking
for
data
that
spans
the
last
six
hours
for
us,
some
of
it
will
be
served
from
memory.
Some
some
of
it
will
be
served
from
the
durable
one
of
our
data
stroller
cassandra
in
this
case
and
that's
completely
transparent.
So
so
the
delay
the
data
layer
does
exactly
that.
It
just
mixes
things
and
makes
them
seamless.
So
it's
an
abstraction
layer.
I
guess
so.
That's
really.
I
mean
the
high
level
architecture.
There's
a
there's,
not
a
ton.
A
A
So
graphite
has
it's
hard
to
scale
graphite,
definitely
not
to
the
scale
that
we're
at
it'd
be
pretty
hard
and
then
opencv
open,
tsb
it's
hbase
based
so
back,
then
we're
like
well
hbase.
I
don't
really
want
to
spend
too
much
time.
You
know
dealing
with
it,
so
we
ended
up
building
something
on
top
of
cassandra
since
there's
been
a
few
interesting
pro
open
source
projects,
sometimes
series
databases,
influx
db,
vena
from
ghost
circuit,
so
there's
a
number
of
them
that
are
that
are
out
there.
A
They
tend
to
be
based
on
like
level
db
for
storage
and
things
like
that.
So,
if
you're
interested,
I
can
definitely
get
into
details,
but
that's
that's
really
what
the
data
layer
is.
Is
it's
not
very
complicated?
It's
just
just
a
bunch
of
ways.
So
the
tools
we
used
for
that
is
so
redis
is,
as
you
can
guess,
all
the
stuff.
That's
in
memory
is
in
redis,
it's
nice
because
we
used
it
to
because
it
gives
you
very
regular
access
to
your
data.
A
So
if
you
have
time
series
kind
of
flow,
naturally
in
redis
we
use
so
that's
the
low
durability
super
fast
access.
You
have
cassandra
is
somewhere
in
the
middle,
where
it's
it's
much
more
durable
and
cassandra.
I
don't
know
if
you
guys
use
it
or
know
it's
a
distributed
kind
of
key
value
store.
Let's
play
it
that
way,
so
it's
relatively
simple
to
operate,
it'll
store
data
on
disk
and
it's
really
good
at
writing
at
cons
relatively
constant
speed
in
constant
time.
A
So
you
can
write
a
lot
of
data
and
it's
not
going
to
degrade
too
much.
Reading
is
a
different
story,
but
when
you're
ingesting
a
ton
of
data
in
it,
actually
it
works
pretty
well-
and
I
realized
that
I
forgot
one
thing:
the
cues
that
we
use
is
kafka,
so
this
is
from
linkedin.
I
don't
know
if
you
guys
know
what
it
is.
A
It's
basically
a
curing
system
that
is
much
simpler
than
most
queuing
systems
or
the
ones
you
may
be
used
to
like
mq
series
and
qp
and
so
on,
where
it's
a
thin
queuing
layer
on
top
of
the
jvm,
which
is
on
top
of
basically
the
the
kernel.
So
it's
really
it's
really
fast.
Your
file
system
faster
yeah,
exactly
it's!
It's
really
fast,
and
it's
really
good
for
sequential
rights
and
sequential
reads,
and
what
we
have
are
it's
a
first
and
first
out
sequential
rights.
A
You
get
the
data
you
write
it
to
disk
and
then
something's
going
to
pick
it
up
to
process
it,
and
it's
not
it's
not
good
at
all
for
random,
reads
and
random
rights.
But
that's
not
the
use
case
here
zookeeper.
This
is
zookeeper.
You
know
which
we
use
for
coordination
across
across
a
lot
of
services.
Elasticsearch,
that's
mostly
used
for
events
whenever
you
say
show
me
all
the
marketing
launches
in
the
past
week
and
overlay
them
on
a
graph
they're
going
to
come
from
elasticsearch
postgres.
A
We
use
it
for
basically
storing
the
application,
the
catalog,
you
know
who
the
users
are,
what
the
graph
definition
is
and
so
on
and
so
forth,
and
then
the
glue
really
is
in
python
itself.
So
there's
actually
one
of
my
teammates
gave
a
great
talk
at
pi
data.
I
think
it
was
10
days
ago
about
how
we
use
python.
So
I
don't
know
if
there
are
python
users
in
the
audience.
A
Yes,
if
you
do
anything,
I'm
sure
you
know
that,
but
we
had
great
experience
with
nunpai
python,
particularly
so
it's
great
for
us
python
is
great
because
it's
product
we're
very
productive,
reasonably
productive
with
it.
That's
great
library
support
it's
everywhere.
Every
machine
we
use
has
python
already,
it's,
unfortunately,
not
very
memory
friendly,
so
which
leads
us
to
some
interesting
optimization.
A
This
is
the
the
talk
you
want
to
you
want
to
watch.
If
you
want
to
get
get
into
the
sort
of
the
details,
I'm
happy
to
go
into
details
on
the
questions
as
well,
so
a
few
challenges
and
I
think
we're
getting
towards
the
end
of
the
talk.
So
in
terms
of
just
to
give
you
a
sense
of
scale,
so
it's
about
30
billion
data
points
every
day
that
we're
receiving.
So
it's
just
machines.
You
know
it's
any
number
of
machines
sending
in
total
30
billion
data
points.
A
It
runs
24
7.,
one
of
the
challenges
that
we
had
it's
hard
to
monitor.
So
we
are
monitoring
company,
but
it's
it's
interestingly
hard
to
monitor
our
stuff.
So
we
we
do
a
mix
of
dog,
fooding
plus
using
other
other
tools,
but
it's
yeah.
I
mean
it's
a.
I
guess
it's
reality
of
the
complex,
so
real-time
apps
these
days
is
like
the
monitoring
becomes
and
that's
why
we
have
the
business.
I
guess
is
non-trivial
long-term
operation.
A
One
of
the
challenges,
so
I
mentioned
a
few
times-
is
some
ever-increasing
amount
of
data
comes
in,
never
stops,
never
takes
a
break,
so
that's,
as
you
can
imagine,
both
on
a
technical
side,
but
also
on
the
operational
side
that
has
it's
not
easy
to
manage.
Memory
for
us
becomes
the
the
most
precious
resource,
and
that's
that's
just
you
know.
I
guess
that's
everybody
is
that,
but
because
latency
is
so
important,
even
though
it's
it's
bounded
at
you
know
a
few
seconds
to
tens
of
seconds,
it
can't
be.
A
It
can't
move
much
higher
than
that,
so
that
doesn't
give
us
a
lot
of
leeway,
and
so,
when
we're
running
when
we're
running
out
of
memory
somewhere,
you
can,
you
can
see
pretty
quickly
in
our
internal
graphs.
You'll
see
stuff
piles
up
somewhere
in
the
in
the
pipeline.
Stuff
starts
to
grow
or
grow.
It's
like.
Oh,
we
gotta
fix
this,
so
the
the
number
one
game
we
probably
play
at
datadog
is
on
top
of
building
a
new
product
is
basically
chase
latency
bubbles.
So
anything
that
slows
down.
A
So
it's
measure
and
the
way
we
do
it
is
very
simple,
is
like
we,
so
we
have
a
bunch
of
metrics.
Everything
is
instrumented,
so
we
find
oh
there's
a
jump
here.
Why
is
it
a
spike
in
time
it
takes
to
render
something
or
time
it
takes
to
query
one
of
the
data
stores.
Then
we
basically
profile
the
code
that
does
that
tune.
It
test
it
hotfix
and
so
on
so
forth.
That's
so
that
that's
a
very
constant
operation
or
game
that
we
play
pretty
much
every
single
day.
A
Some
things
happen,
something
like
like
that
is
happening,
that's
just
because
the
system
is
growing
growing
growing
and
the
interesting
thing
that
I
didn't
mention
is
when
we
started
in
2010.
Obviously
we
didn't
know
if
we
had
a
business,
I
mean
that's
a
startup
sorry,
so
we
wanted
to
have
something
out
of
the
door
as
quickly
as
possible.
A
So
we
started
with
a
very
naive
implementation
that
was
based
on
postgres
and
we
knew
it
was
naive,
but
it
was
it
took
about
maybe
six
hours
to
figure
out
that
we
had
to
change
it
after
after
we
started
to
ingest
some
data,
and
so
the
architecture
that
I
described
even
at
a
high
level
is
really
iterative.
It's
it's
just
basically
doing
this
all
the
time
so
replacing
you
know
replacing
this
replacing
that
we
had
queuing
based
on
cassandra
at
some
point,
which
is
a
terrible
idea
by
the
way.
A
A
Now
in
general,
one
thing
that
I
really
like
about
the
stuff
that
I'm
doing
and
monitoring
in
general
is
that
you
I
mean
in
a
technical
capacity
if
you're
engineers,
then
you
are
our
customers
and
that's
great,
because
that
means
that
I
you
know
we
interact
with
you
guys
and
then
we
want
to
make
something,
that's
useful.
So
so
that's
a
little.
A
The
last
thing
I
want
to
say
we
are
hiring,
and
so,
if
you
want
to
become
a
data
dog-
and
it
took
me
a
while
to
understand
the
joke,
so
if
you
don't
know
well
I'll
explain
because
it
took
me
at
least
a
month,
data
is
like
a
starter
guy
and
dogs.
Sleep
dog
anyway.
If
you
want
to
become
one
of
us
jobs
at
dinnerbeast.com.
A
A
I
will
repeat
the
question.
So
did
we
ever
have
a
customer
that
doesn't
have
mtp?
We
always
have
a
customer
who
doesn't
have
ntp,
and
so
what
you
see
is
there
they
call
you
or
they
call
you
a
day
on
our
c
and
say
hey.
I
can't
see
my
metrics
and
you
look
and
say:
yeah
there's
nothing
and
then
it
happens
that
either
their
graphs
are
basically
in
the
past,
because
the
machine
is
sending
data
thinking
it's
in
the
past
or
it's
in
the
future.
A
B
A
Okay,
so
the
question
is:
do
we
roll
up?
So
we
roll
up
past
data
and
I
said
we
roll
it
up.
That
is
to
say
we
summarize
it
so
that
whenever
you're
asking
for
data
to
graph
data
for
the
past
month,
we
give
you
200
pixels
200
data
points
back
or
600
data
points
back,
and
we
do
that
because
we
have
already
pre-computed
the
we
already
pre-computed
summaries,
the
time
summaries.
A
The
question
is,
we
save
everything,
so
we
save
the
highest
resolution,
data,
which
is
for
us
down
to
the
second
and
from
a
product
perspective.
What
happens
is
that
it's
like
almost
science
fiction,
you
look
at
the
month
of
data
and
you
zoom
in,
and
you
see
more
more
detail
you
zoom
in
and
you
see
more
detail
until
you
get
to
the
the
finance
resolution.
But
so
yes,
that's
that
yeah,
that's
available!
Yes,
absolutely!
Okay!
So
we'll
see.
A
A
So
if
you
ask
for
six
hours
that
started
six
hours
ago,
end
is
now
we'll
still
serve
the
last,
the
the
last
four
and
a
half
hours
out
of
memory,
partly
because
the
processes
down
the
pipeline
that
are
supposed
to
persist.
The
data
may
not
have
flushed
the
data.
Yet
so
you
go
to
cassandra
railway
store
sort
of
intermediate
intermediary
data.
Maybe
it's
not
there!
So
that's
why
we
stitch
the
data
on
the
fly
when
we
query.
A
What's
hard
about
it,
mostly
because
I
mean
the
main
thing
is
in
python,
for
instance,
you,
basically
you
have
to
use
numpy,
it's
not
particularly
hard,
but
you
just
have
to
align.
You
have
to
time
align
the
array
so
at
the
boundary,
when
you
stitch
it,
you
have
to
make
sure
that
there's
enough
overlap,
so
you
don't
have
a
gap,
for
instance
between
stuff.
That's
served
from
memory,
instead
of
so
from
this
that
kind
of
stuff.
That's
that's
a
fairly
well
understood
problem
for
us.
A
Yeah,
so
that
works,
that's
a
great
question.
I
need
to
be
able
to
answer
it.
So
the
question
is
how
we
nodded
about
the
cost
of
operations
while
we
double
number
customers,
so
the
in
what
I
showed
the
the
the
raw
the
raw
intake
does
not
necessarily
is
not
necessarily
the
vast
major
bulk
of
the
expense.
I
mean
the
bulk.
The
bulk
of
the
the
data
is
actually
going
to
live
in
in
storage
and
so
that
that
grows
reasonably
linearly,
but
we
can.
A
We
can
compress
it
the
bulk
of
your
processing
scales,
relatively
non-linearly,
because
you
can,
I
mean,
for
one
thing:
we
optimize
the
code
and
we
do
what
I
mentioned.
That
we
said
we're
doing
is
that
we
replace
pieces
as
they
get
too
slow.
So
that's,
basically
the
the
idea
to
keep
a
fairly
high
gross
margin
for
the
business.
A
A
Watch
out
for
the
size
of
the
indices,
that's
the
main
thing
we've
had,
so
we
it
elastic
search
as
such
works
great.
It's
like
postgres.
It
runs
out
of
the
box
and
you
don't
have
to
really
watch
it
until
materialize
that
there's
a
problem
and
then
you
didn't,
you
know
you
missed
something,
but
that's
the
because
you're
so
busy
with
the
rest
that
you
know
there
are
the
pieces
for
us
that
are
so
elastic
search,
redis
postgres.
A
These
are
run
almost
on
their
own,
because
sunrise
getting
there
s3
definitely
runs
runs
on
its
own
and
the
thing
I
wanted
to
say
about
s3.
Is
that
because
you
remember
that
said,
our
data
is
immutable
being
past
data,
then
s3
makes
a
perfect
case
for
that,
because
it's
you
know
we
never
need
to
like
randomly
seek
and
change
a
little
bit
here
and
there
all
the
way
behind
and
then
I'll
just
circle.
A
A
Actually,
I
may
have
had
a
I
blogged
about
it,
a
while
back,
but
if
you
read
the
fine
print,
they
give
you,
even
if
you
get
provision
ielts,
which
is
like
a
reserve
stuff
on
address,
the
fine
prints
are
fairly
interesting,
where
they'll
guarantee
performance
of
how
fast
you
can
read
and
write
only
only
in
x
percent
of
the
time,
and
only
if
you
read
and
write
within
certain
boundaries.
So
I
mean
it's
hard.
It's
definitely
a
hard
problem
and
I
think
they're
doing
a
great
job.
A
A
A
So
I
so
the
the
truth
is,
I
don't
even
know
because
this,
because
the
thing
I
obsess
about
and
is
how
much
data
we're
receiving
and
so
that
hasn't
really
changed.
I
mean
there
was
a
dip.
Actually
there
was
some
dip,
but
we
couldn't
see
much,
but
how
much
data
are
we
serving?
I
don't
really
care,
I
mean
I.
B
A
A
A
We
have
workload
metrics,
which
I
mentioned
the
amount
of
work
being
done,
so
our
processes
are
reading
from
the
queue
to
do
something
in
this
case
to
roll
up
or
to
do
anomaly,
detection
and
stuff
like
that,
so
the
metrics
that
I
will
watch
is
how
many
points
have
you
consumed
for
instance,
and
so
that
means
reading
a
bunch
of
messages
from
kafka,
opening
them
up.
Seeing
how
many
points
they
are
in
each
message
and
so
on
so
forth.
So
we
don't
have.
A
We
don't
really
have,
interestingly
enough,
a
great
sort
of
native
kafka
instrumentation,
it's
we
did
it
kind
of
above
above
that
layer,
just
we
care
about
if
we,
basically,
if
there's
a
drop
in
throughput,
so
in
the
number
of
messages
you're
reading,
assuming
the
messages
entering
the
system
is,
are
constant
and
it's
a
drop.
There
then,
usually
we'll
look
at
so
how
many
messages
are
consuming
and
also
what
what
the
machine
in
terms
of
I
o
is
doing
and
that'll
basically
tell
us
the
story.
A
So
so
they
themselves
feed
metrics
into
datadog,
and
so
we
have
also
alerts
that
will
tell,
and
the
other
thing
engineer
didn't
get
into
that.
The
learning
engine
runs
kind
of
you
know
on
a
on
a
given
schedule.
It
like
every
right
now,
it's
every
minute.
It
wakes
up
just
looks
at
everything
and
say:
okay,
everything's
fine
everything's
not
going
over.
Then
there
are
things
that
are
not
fine,
so
that
part
is
independent
from
queueing.
A
If
there's
no
data
I'm
making
to
it,
then,
for
instance,
if
a
in
front
of
it
or
some
things
that
are
feeding
it,
are
not
producing
anything,
then
they'll
tell
us,
hey,
you're,
not
producing
any.
You
know
the
producers
are
not
doing
anything.
So
then
we
could
say:
okay,
is
the
producers
not
doing
anything
because
they
crashed
and
we
monitor
all
of
them
at
once?
So
we
can
know
this
guy
crashed
or
restarted,
or
is
it
because
it's
something
else?
A
So
that's
that's
usually
just
the
starting
point
of
the
investigation
is
oh,
we
are
we're
expecting
to
receive
x
amount
of
data
points
per
second,
we
only
got
60.
What's
the
problem,
how
many
processors
we're
running
does
that
make
sense,
yeah,
okay,
cool,
actually,
all
the
way
to
the
back
and
I'll
come
back.
A
A
I
think
we're
on
zero,
seven
we're
deeply
thinking
about
zero,
eight,
but
yeah,
but
and
also
we
wanna
I'd
like
to
have
other
people
tested
in
production.
I
mean
beside
linkedin
and
not
not
find
out
that
so
the
failure
modes
of
the
so
the
queuing
system
has
some
replication
built
in
in
the
newer
versions,
and
the
question
is
so:
we
don't
use
that
yet
I'm
a
little
bit
cautious
about
what
the
failure
modes
are
going
to
be
like,
because
we
give
some
backdrop.
A
B
A
B
A
A
Or
is
it
just
in
case
so
that
at
this
point
it's
mostly
in
case
there's
there's
a
problem?
So
what
so?
That's
to
the
10
to
30
seconds?
It's!
It
has
happened.
That's
you
know
the
hit
the
fan
as
well
and
then
we
need
to
fail
over
and
then
we
didn't
automate
every
single
failover
scenario,
because
it's
really
damn
hard
to
conceptualize
ahead
of
time
what's
going
to
crash,
and
so
in
that
case
like
oh,
damn,
it's
going
down.
A
So
what
you
see
if,
if
that
happens,
you'll
see
the
graphs
exactly
what
I
was
talking
about
earlier
and
the
kind
of
trade-off.
I
don't
want
to
make
you
start
seeing
this
and
because
the
data
is
just
not
it's
arrived,
but
it's
not
being
read
consumed,
and
so
that's
when
this
is
like
major
major
crisis
mode
for
us
and
we
try
to
keep
those,
as
you
know,
as
singular
as
possible.
A
A
A
The
sharding
is
done
by
so
there's
a
key,
which
is
the
metric
name,
the
or
the
the
account
name,
and
also
the
the
general
time
frame
you're.
Looking
at
and
the
way
we
do
it
is
we
ttl
all
the
all
the
rows.
So
after
15
days,
the
data
is
going
to
disappear
from
cassandra,
no
matter
what,
because
otherwise
we
before
we
had
that
we
had
a
grow
ever
growing
cassandra
cluster
and
then
it
becomes
impossible
to
delete.
B
A
Yeah,
so
yes,
so
I
know
my
detection
so
yeah,
so
the
question
is:
do
we
automate
an
amount
of
detection
at
this
point?
We
don't
really,
and
the
reason
is
so
you
have,
if
you
have
using
the
you're
instrumenting
your
app
fairly
consistently.
You
have
thousands
and
thousands
of
metrics
and
while
a
lot
of
people
want
to,
you
know
want
to
buy
hey.
I
want
a
money
deduction.
You'll
tell
me
when,
whenever
there's
a
problem
somewhere
in
my
data,
you'll
hurt
me
in
reality
right
now.
A
A
A
A
A
So
how
do
we
manage
the
consistency
between
the
data
store?
So
we
have
downtowns
everywhere.
We
basically
inject
known
signals
inside
the
the
pipeline,
and
then
we
check
everywhere
that
that
we're
expecting
to
see
that.
So
we
don't
not
only
check
that
we
expected
to
see
them
that
they're
there
with
the
right
value,
for
instance
we
produce
we
have,
I
don't
have
it,
but
we
have
a
batman
series
that
sort
of
draws
the
top
half
of
batman
and
bottom.
A
You
know
like
the
sign
the
bat
sign,
so
we
checked
that
as
you're
injecting
the
system
and
you
should,
it
should
appear
somewhere
down
the
down
the
pipeline.
We
also
check
that
it
appears
it
appears
on
time
so
that
you
inject
the
signal
and
within
you
know,
top
60
seconds.
You
can
read
it
out.
If
not,
that
means
there's
something
that's
delayed
in
the
pipeline.
A
So
we're
doing
so.
The
question
is:
what
do
we
do
when
we,
when
we
trade
off
precision
for
for
latency,
we
do
it.
We
skip
points.
Basically,
we
skip,
for
instance,
every
you
know
we'll
skip
20,
10
or
20
percent
of
the
points.
So
that's
we
don't
do
that
normally,
but
sometimes
it's
it's
necessary.
So
that's
one
of
the
levers
we
have
to
always
try
to
stay.
There
is
to
skip
it
and
the
truth
is
because
you
know
again
it
has
a
fairly
high
signal
noise
ratio.
A
The
trend
is
still
there,
you're
losing
points.
It's
not
you
know
it's.
It
doesn't
really
affect
your
your
ability
to
decide
much
less
so
than
if
you're.
If
you
have
like
a
blind
graphing,
like
you
know,
what's
going
on,
you
know,
does
that
make
sense,
so
we
have
instrumented.
We
have.
Basically
you
run
a
command
line.
It
starts
for
a
given.
Account
starts
dropping
every.
You
know
every
tenth
point
of
what
we're
receiving
for
a
given
metric
or
something
like
that.
A
B
A
A
Yeah,
whatever
yeah,
whatever
agent,
we
we
have
yeah,
so
the
so
the
system
stuff.
So
if
you
install,
if
you
tried
it
out
dog
you'll
say
you
run
it
on
your
laptop
or
your
server.
You'll,
get
like
some
metrics
out
of
the
box
like
anything
around
or
consumption,
so
resource
consumption,
but
you
can
also
produce
your
own
metrics,
which
usually
the
more
interesting
stuff.
Is
you
send
your
metrics
about
your
application
because,
like
you
know,
cpu
it's
interesting,
but
not
that
exciting
and
not
actually
not
that
actionable?