►
From YouTube: C* Summit EU 2013: Blueflood: Simple Metrics Processing
Description
Speaker: Gary Dusbabek, Apache Cassandra Committer and Systems Architect at Rackspace Hosting
Slides: http://www.slideshare.net/gdusbabek/blueflood-open-source-metrics-processing-at-cassandraeu-2013
Rackspace needed a metrics system that could ingest 30 million signals generated from the Cloud Monitoring system. It had to offer custom data retention levels and still be able to offer graphs to customers in real-time. Gary and his team created a distributed system of shared-nothing nodes on top of Cassandra that split the responsibilities of: ingesting data, processing rollups, servicing data points for reads. Depending on the need, nodes can be easily reconfigured to support all or some of those functions. In this session you you will learn about techniques for scheduling rollups and still maintaining numerical accuracy, how to handled non-numerical data points, how to utilize open-source technology (Apache Cassandra, Scribe, Thrift, and Node.js) to deliver results relatively quickly and much more.
A
My
name
is
Gary
deuce
babik
and
I
work
at
rackspace
on
primarily
on
their
cloud
monitoring
team,
but
we
do
lots
of
other
different
things.
I
have
a
good
deal
of
experience
with
Cassandra
for
a
couple
years.
I
was
a
pretty
active,
committer
and
so
I
come
from
that
background.
I
know,
what's
going
on
inside
I
still
like
to
think
of
it
as
just
a
big
long
column
store,
which
is
different
than
then
a
that
probably
appears
today.
A
But
that's
the
that's
the
background
I
come
from
and
that's
that's
how
I,
how
I
use
Cassandra
I,
primarily
just
just
use
it
now
as
a
developer
I,
don't
get
to
have
a
lot
of
time
to
hack
on
it.
What
I'm
going
to
talk
about
today
is
a
is
a
some
software
that
we
created
at
Rackspace
to
help
us
monitor
our
time
series
data.
We
we
generate
a
lot
of
it
and
we
want
to
be
able
to
do
useful
and
interesting
things
with
it,
and
so
that's
what
this
presentation
is
about.
A
I'm
going
to
spend
some
time
talking
about
why
we
created
blue,
fled
in
the
first
place
and
then
spend
the
bulk
of
the
this
presentation
talking
about
the
guts
that
are
on
the
inside
the
design
decisions
that
we
made.
Why
we
did
things
this
way
or
that
way,
and
hopefully,
if
you
can,
you
know
if
you
understand
that,
then
you'll
understand
why
the
system
is
the
way
it
is
and
then
I'll
talk
about
some
of
the
things
that
we're
working
on
this
is
open
source
software
and
we're
we're
pushing
it
forward.
A
We're
doing
some
some
fairly
interesting
things
that
I
think
you'll
you'll
want
to
hear
about,
so
why
we
did
it.
The
most
there's
probably
is
33
important
things
that
that
blue
flood
does
or
that
we're
important
for
us.
The
the
most
important
thing
was
to
get
the
data
in
so
that
we
actually
have
it
somewhere.
We
don't
have
to
worry
about
it,
going
away
with
cloud
monitoring.
We
have
these
checks
and
they
generate.
A
You
know
sometimes
a
few
metrics,
sometimes
a
lot
of
metrics
apiece,
and
this
happens
in
multiple
data
centers,
and
so
you
know
the
example
of
there
is
each
check
generates
between
two
and
twenty
metrics,
but
then
each
check
gets
executed
in
up
to
up
to
five
data
centers,
usually
about
three.
We
generate
a
fair
amount
of
data
that
comes
in
and
for
the
longest
time
we
were
just
throwing
it
away.
We
knew
that
we
wanted
to
be
able
to
do
some
useful
and
interesting
things
with
that,
but
we're
a
pretty
small
team.
A
As
far
as
you
know,
Rackspace
development
goes,
and
so
we
had
to
put
this
off
for
later
and
that's
what
that's
what
blue
flight
is.
Is
that
that
later
part
that
we're
finally
getting
to
realize
some
examples
of
what
what
that
looks
like
is,
if
you
use
a
rackspace,
you
can
log
into
your
control
panel
and
if
you,
if
you
have
monitoring,
turned
on
the
grass
that
you
see
they're
actually
generated
from
data
collected
by
blue
flood.
A
Currently
we're
we're
ingesting
about
120
million
metrics
per
hour,
which,
by
cassandra
standards,
I
think,
is
still
pretty
small.
Our
production
cluster
is
only
32
nodes
and
we're
not
really
even
making
it
making
it
sweat
a
little
bit.
This
ends
up
being
about
40
million
aggregate
cassandra,
operation
or
writes
per
hour,
and
so
still
we're
not
really
stressing,
stressing
cassander
there.
We
have
quite
a
bit
of
a
room
to
grow
into
so
we're
we're
impressed
and
happy
with
what
we
see
so
far.
A
We
know
this
system
is
going
to
be
able
to
hold
a
lot
of
a
lot
of
data.
A
lot
of
metrics
for
us,
I,
don't
believe
I
brought.
My
I
didn't
include
a
slide
that
I
had
another
slide.
That
talked
about
the
the
right,
latency
and
read.
Layton
sees
that
we're
observing
right
now
and
they're,
both
within
very
acceptable
limits
for
us
and
so
we're
happy
with
the
way
the
system
the
system
is
working.
A
Another
one
of
the
important
things
that
that
motivated
us
is,
of
course
we
want
to
be
able
to
get
the
data
out.
It
had
to
be
pretty
simple
to
us
or
pretty
simple
to
to
consume
the
number
one
thing
as
far
as
that
goes,
though,
is
grass
I
showed
a
picture
of
the
graph
earlier
people.
It
turns
out
have
this
incredibly
low
tolerance
threshold
for
waiting
for
things
we're
just
by
our
nature.
We
just
we
don't
want
to
sit
around
and
wait
and
when
people
are
waiting
for
graphs,
they
just
they
lose
it.
A
You
know,
and
so
that
was
our
main
goal.
There
was
to
make
sure
that
we
can
get
data
points
out
quickly
and
that's
what
influence
our
decision
to
make
it
a
roll
up
system
rather
than
compute
the
roll-ups
on
the
fly
at
query
time.
We
get
a
much
better
service
level
and
things
just
in
generally
flow
a
lot
more
smoothly.
Graphs
are
fast
and
people
people
can
maintain
their
composure.
Another
goal
that
we
were
shooting
for
was
to
have
a
multi-tenant
system,
and
this
is
just
this
was
our
own
organizational
needs.
A
But
the
thing
is,
some
users
are
going
to
be
big
users,
they're,
going
to
demand
a
lot
of
resources
and
other
users
aren't
going
to
demand
a
lot
of
resources,
and
so,
whenever
you
start
dealing
with
this
kind
of
problem
in
multi-tenant
environments,
things
get
difficult
pretty
quick,
because
everyone
has
a
different
set
of
expectations
going
into
it.
It
actually
turns
out
to
be,
you
know,
pretty
pretty
hard,
I
think.
A
The
main
reason
of
this
is
because,
when
you
start
dealing
with
multiple
tenants,
multiple
expectations
you're
going
to
have
a
different
set
of
metadata
or
meta
requirements,
and
this
is
something
that
just
makes
its
mean.
It's
the
hard
part
about
multi
tennis
systems
is,
is
accommodating
everybody,
and
so
we've
had
to
do
a
couple
different
things
to
help
accommodate
that
and
at
the
at
the
far
ends
of
the
distribution.
A
There's
the
you
know,
they're,
probably
the
unhappy
group
but
I-
think
we've
managed
to
accommodate
a
pretty
pretty
big
section
in
the
middle.
Something
else
is
I
like
to
call
blue
flood
a
lipstick
system.
We
at
cloud
monitoring.
Our
primary
concern
is
making
sure
that
the
entities
servers
are
monitored
and
that
alerting
happens
when
that
monitor
or
when
those
servers
go
down
or
break.
Getting
pretty
graphs,
isn't
something
that's
going
to
be
mission-critical.
It's
it's
nice
to
have.
A
But
if
this
system
you
know
has
hiccups,
data
is
delayed
by
a
minute
or
something
like
that:
you're
not
going
to
have
customers,
while
your
customers
screaming
at
you,
you
might
be
screaming
at
the
at
the
people
running
your
graphing
system,
but
your
customers
aren't
going
to
be
upset.
If
you're,
not
your
data
points
as
quickly
as
so,
we
knew
going
into
it
that
it
wasn't
going
to
be
a
mission
critical
system
for
rackspace
business.
A
We
knew
it
was
just
going
to
be
something
that
was
that
was
nice
to
have,
and
so
with
that
in
mind,
we
wanted
it
to
be
a
pretty
cheap
system.
We
ended
up
co-opting
the
the
Cassandra
hardware
that
we
were
already
using.
We
realized
our
Cassandra
store
our
Cassandra
database
cluster
was
using
a
lot
of
disk,
but
we
really
weren't
using
a
lot
of
the
CPU
and
so
with
blue
flood.
We
realizable
this
is
going
to
be
a
pretty
cpu
intensive
application.
A
A
Another
thing
that
we
wanted
to
do
was
avoid
Hadoop
and
it's
not
because
it's
not
because
we
hate
it.
It's
just
it's
just
something.
That's
extremely
heavy
weight.
Hadoop
brings
operational
requirements
in
it,
brings
hardware
requirements
in
and
again
with
the
small
team
that
we
are.
It
wasn't
something
that
we
wanted
to
undertake.
We
like
Hadoop,
we
use
Hadoop
for
our
own
data,
warehousing
and
long
term,
crunching
type
things,
but
it's
not
something
that
we
wanted
to
make
part
of
our
metric
and
and
graphic
system,
and
so
those
were.
A
You
know
that
that's
what
motivated
us
and
drove
us
to
create
blue
flood.
This
is
what
we
ended
up
with
and
I'll
spend
the
rest
of
the
time
talking
about
the
different
components
here,
but
it
starts
at
the
bottom
and
it
works
its
way
up
at
the
bottom,
of
course,
is
Cassandra,
and
on
top
of
that,
there's
just
some
libraries
that
allow
us
to
get
data
in
and
out,
there's
a
little
bit
of
state
management.
This
is
all
one
unit
right
here
and
then
there's
some
tools
that
allow
you
to
do.
A
A
We
ship
you
that
this
stuff
here
you
have
to
come
up
with
on
your
own
I'm,
going
to
explain
why
we
did
that
later.
We
ship
some
some
reference
implementations
of
the
API
endpoints,
but
there
are
some
good
reasons
why
you
probably
want
to
do
things
your
own
way
there
and
I'll
explain
those
when
I
get
up
to
up
to
that
layer.
So,
let's
first
just
talk
about
the
database,
we've
been
using
Cassandra
at
Rackspace
for
for
quite
a
long
time.
A
We've
we've
made
sure
it's
compatible
if
the
new
versions
as
they've
come
along,
we
haven't
started
working
in
Cassandra
to
yet,
and
so,
although
I
imagine
blue
flood
will
work
fine
with
Cassandra
to
it's,
not
something
that
we've
had
a
lot
of
time
to
spend
testing
and
making
making
sure
we
also
started
using
cql
very
early
on
this
would
have
been
in
the
the
cql
10
timeframe
and
at
the
time,
cql
was
a
very
good
tool
for
getting
you
away
from
having
to
deal
with
rift.
You
know
thrift.
A
Everyone
complains
about
having
to
set
up
your
queries
and
actually
execute
them,
and
it
was
it's
something
that
was
that
was
involved.
You
had
to
do
a
lot
of
work
and
seek
you
out
the
time
made
that
made
that
pretty
easy
because
it
did
make
it
look
like
like
SQL
cql
is
a
you
know.
If
you
were
using
cqo
10
and
you
look
at
CQ
all
now,
you
realize.
Oh
no,
you
know
this
is
something
that's
a
that's
totally
different.
A
Cql
has
changed
more
into
something:
that's
really
good
for
data
modeling
which
for
us
you
know,
we
knew
it.
We
were
dealing
with
time
series
data.
We
knew
that
we
were
going
to
have
pretty
long
rows.
The
data
modeling
part
wasn't
so
important
for
us,
and
so
we
ended
up
moving
to
using
st
annex
there.
We
go
and
we've
been
pretty
pretty
happy
with
it.
I
still
think
it's
it's
pretty.
As
far
as
a
client
library
goes,
it's
pretty
low
on
the
the
maturity
level.
A
As
far
as
software,
we
notice
that
it
has
a
lot
of
bumps
when
your
network
isn't,
quite
so
predictable
and
at
the
connection,
pooling
especially
has
a
lot
of
tendency
to
to
start
going
through
the
connections
when
there's
when
there's
a
little
bit
of
network
hiccups
well
than
that,
we've
been
pretty
pretty
happy
with
it.
I
know
that
the
future
is
is
everything's
moving,
you
know
native
and
an
cql
driven,
and
we
plan
on
being
fine
with
that.
A
A
If
you,
if
you
want
to
use
this,
some
Cassandra
tickets
to
watch
out
for
is
first
that
could
change.
This
would
be
56
85
and
that's
what
the
one
that
sets
up.
Ttls
that
are
per
column,
family
TTL
end
up
being
pretty
pretty
important
in
blue
flood,
because
it's
just
a
nice
way
of
getting
data
out
of
the
system
without
having
to
go
back
and
reap
it
out
later,
every
customer
can
have
a
different
TTL
for
us,
though,
and
so
that
makes
per
column.
A
Another
one
is
cassandra
3974,
and
this
is
a
one
that
set
up
TTL
histograms,
so
that
compaction,
so
the
compaction
strategy
can
know
it.
Whether
or
not
an
SS
table
is
a
really
good
candidate
come
to
compact,
and
this
will
only
apply
if
you're,
using
Cyrus
tiered
compaction.
But
this
is.
This
is
obviously
something
that
that's
also
good.
A
Sandra
5228
is
one
that
it
tracks
the
maximum
TTL
/
SS
table
so
that
it
can
just
wipe
out
the
whole
thing
without
having
to
traverse
every
entry
in
it
again.
We
this
isn't
going
to
work
too
well.
For
us,
the
way
we
have
things
right
now,
because
within
a
single
SS
table,
we're
going
to
have
lots
of
different
details,
/
tenets,
but
we
could
make
this
work
I.
Think
if
we
wanted
to
bucket
our
column
families,
so
this
column
family
would
be
40
TLS
between
you
know
zero
seconds
and
12
hours.
A
A
We,
we
haven't
seen
a
lot
of
problems
with
compaction,
ruining
our
throughput,
ruining
our
read
latency
distribution,
and
so
we
were
actually
worried
about
that
at
first,
but
it
turns
out
not
to
not
to
be
a
problem
in
actual
practice
now,
like
I,
said
we're
just
ramping
up
this
system,
and
so
things
like
that
may
change
in
the
future,
and
we
may
need
to
start
thinking
about
how
compaction
affects
things.
But
right
now
we
we
don't
have
a
problem.
A
Things
are
working
good,
I'm
gonna
spend
some
time
now
talking
about
what
metrics
are,
how
they
get
into
blue
flood
and
and
some
some
aspects
of
that
a
metric
is
basic
at
its
most
basic
just
a
one
dimensional
signal,
it's
something
that
has
an
ID
and
a
value.
Now
in
this
there's
this
presentation,
I'll
say
ID
and
I'll
interchange
it
with
what
we
call
locator
a
locator
is
something
that's
used
in
the
in
the
blue
flood
code
base
as
a
concept
for
a
metric
ID.
A
It's
mostly
an
opaque
string
as
far
as
Cassandra
and
most
of
blue
flood
goes.
But
the
nice
thing
is
is
that
it's
a
it's
a
composited
key
that
starts
always
will
start
with
the
tenant
ID
of
whatever
who
owns
the
metric
and
anything
after
that.
It's
up
to
you.
You
can
stuff
as
many
things
in
there
as
you
want
I
included
an
example
up
here,
where
that's
just
going
to
end
up
being
the
whole
locator
and
the
primary
key
for
your
metric,
and
that's
how
you
you
find
it.
A
You
can
tell
I've
put
in
the
name
of
a
host
what
we
might
be
monitoring
and
then
the
units
of
what
we're,
what
we're
bringing
in
so,
but
whatever
you
want
into
it,
just
don't
change
it
becomes
it
because
it
becomes
very
permanent.
It's
it's
your
it's
your
roki
for
for
many
things,
a
metric
has
a
type
associated
with
it.
A
You
I
mean
it's,
it's
pretty
obvious,
you're
you're
going
to
be
dealing
with,
doubles
lungs
and
then,
in
our
cases
we
also
allow
string
and
boolean
metrics
and
those
are
a
little
bit
different
behave
a
little
bit
differently.
I'll
talk
about
those
in
a
bit
type
determines
how
the
metric
gets
serialized
on
disk.
We
do
use
a
version
serialization,
so
we
can
change
those
things
up
in
the
future
and
also
support
different,
more
complex
data
types
in
the
future.
A
A
Basically,
any
kind
of
TCP
check
it
can
do
HTTP,
it
checks,
mail,
headers
things
are
not
mail
headers,
it
checks,
mail,
banners
things
like
that:
sssh
banners,
anything
that
can
be
checked
externally
on
your
server
through
tcp
port
noi
DS
has
probably
got
a
plugin
plugin
to
do,
and
so
this
is
a
single,
a
single
check
that
got
executed.
It's
an
HTTP
check,
but
it's
got
five
metrics
in
there
one
that
talks.
That
explains
how
many
bytes
came
back.
In
this
case
it
was
a
zero
32-bit
integer
time
to
receive
the
first
byte
64-bit
integer.
A
It
took
two
hundred
fifty
seven
milliseconds
and
then
it
goes
on.
This
would
end
up
being
five
separate
metrics
in
blue
flood
it'll
get
split
out,
and
though
this
is
where
I
talked
about
or
earlier
where
I
referred
to.
Everyone's
justin
is
going
to
be
a
little
bit
differently,
so
we
let
those
ingestion
api's
be
pluggable
and
replaceable.
The
interface
says
it
turns
out.
It's
actually
pretty
simple
and
I'll
explain
more
about
that
at
the
end.
A
Metrics
sometimes
have
units,
as
you
can
see,
on
that
example
before
there
were
no
no
units
associated
with
us
with
those
metrics
by
looking
at
the
metric
name,
you
could
figure
out,
you
know.
Well,
this
is
bytes
or
this
is
milliseconds
or
this
is
whatever,
but
it
wasn't
something
that
was
that
was
explicitly
declared
as
part
of
the
metric
and
so
for
us,
and
again
this
is
part
of
our
ingestion
pipeline.
We
guess
on
this
for
lots
of
those
noite
metrics.
A
A
The
way
your
metrics
get
named,
you
can
you
can
you
can
guess
at
what
the
the
metric
units
are
going
to
be
as
far
as
the
way
the
column
families
were
built
for
for
blue
flood,
there
are
several
different
metrics
column,
families
where
the
metrics
are
actually
stored.
There's
one
full
resolution
table
that
as
the
raw
metrics
are
brought
in
there
just
written
straight
to
there
and
then,
as
they
age,
they
get
rolled
up
and
kind
of
conceptually
compacted
and
we
have
five
different
granularities
and
so
there's
roll-ups
that
happen
to
five
minutes.
A
20
minutes,
60
minutes
four
hours
and
then
one
day
this
is
this
is
how
we
keep
that
predictable
query
time
low
in
general.
The
less
data
points
you
query,
the
better,
the
more
predictable
you
know,
distribution
of
your
query,
latency
is
is,
is
going
to
be,
and
so
we
query
at
coarser
granularities
to
reduce
the
number
of
data
points
returned
we,
the
column
family
is
each
get
one
one
row
per
metric
and
the
locator
is
the
key.
A
There
is
there's
no
bucketing
one
of
the
constraints
that
we
went
into
if
this
system
was
that
the
average
frequency
of
a
metric
was
going
to
be
one
every
30
seconds.
Obviously,
if
your
frequency
is
a
lot
higher
than
that
you're
going
to
want
to
employ
some
kind
of
a
bucketing,
we
plan
on
doing
this
in
the
future
and
I,
don't
think
it's
going
to
be
pretty
hard.
A
It
just
changes
the
way
that
you
resolve,
you
know
which,
which
row
your
your
data
is
actually
in,
and
so
we'll
get
there
when
we
need
to
get
there,
it's
not
going
to
be
a
problem,
but
right
now
we
we
don't
bucket
the
rows
at
all.
We
just
have
one
single
long
row
per
metric,
there's
also
column
family,
that
stores
the
metadata,
and
this
ends
up
being
also
one
row
per
per
metric
and
it's
just
simple
key
value.
This
is
where
we
store
the
the
well
lots
of
different
things.
A
A
Chard
is
a
way
that
we
partition
the
roll-up
space
so
that
no
one
single
roll
up
note
is
responsible
for
for
rolling
up
too
much
of
the
of
the
data
set
and
we
have
to
keep
keep
track
of
what
has
been
rolled
up
and
what
has
it
been
rolled
up
and
that's
what
the
roll-up
states
for
we
also
liked
it.
We
also
have
need
to
have
a
way
of
keeping
track,
of
which
metrics
are
actually
active,
and
so
we
have
a
column
family.
A
That
does
that,
and
it's
got
a
detail
on
it
I
think
of
like
24
hours,
and
so
whenever
a
metric
comes
in,
we
will
it's
it's
it's
basically,
one
row
per
shard,
and
so
it's
limited
to
in
our
case,
128
rose
and
as
metric
comes
in,
will
append
its
locator
to
that,
and
so,
whenever
I
metric
stops
coming
in
well,
it'll
disappear
out
of
that
row
in
24
hours,
and
so
we
know.
Okay,
we
don't
have
to
roll
this
up
anymore
and
that
tends
to
work
pretty
well
in
practice.
A
A
There's
also
some
nice
things
in
there
to
where,
if
it's
a
little
bit
of
a
high
frequency
metric
that
it's
it's
kept
in
memory,
so
that
you
don't
have
to
always
go
back
to
the
database
to
compare
it
I
did
it
change?
Ok,
no,
don't
write
anything!
There's
some
some
good
good
things
in
there
for
that.
Moving
up
to
the
the
library
layers,
we
just
have
a
s,
I'll
start
I'll,
start
with
ingestion
and
then
go
to
roll
up
inquiry.
These
are
actually
just
pretty
simple.
I
want
to
explain
the
methods
that
we
have.
A
The
first
method
is
just
obviously
insert
a
list
of
metrics
metrics.
Are
they
generally
come
in
together
or
separate,
but
we
we
let
them
bundle
up
and
we
write
a
whole
a
list
of
them
that
ends
up
resulting
in
a
Cassandra
batch
operation.
There's
a
method
that
updates
the
state.
The
state
is
actually
a
kind
of
a
complicated
map
of
chard
to
granularity
to
slot
now
slots,
the
term
that
I
haven't
used
yet
and
basically,
what
a
slot
is
is
a
bucket
of
time
now
go
into
some
details.
A
If
I
have
time
at
the
end
about
what
that
is
rolling
up,
we
need
to
be
able
to
get
the
active
locators.
We
need
to
be
able
to
get
that
state
back,
that
we
wrote
at
ingestion
time,
and
then
we
also
need
to
get
the
metrics
so
that
we
can
run
the
computations
and
then
store
them
when
we're
done.
Of
course
we
update
stay
at
the
end,
so
we
know
the
roll-ups
actually
actually
happened.
A
A
So
when
they
come
in
at
full
resolution,
you
basically
have
one
data
point
and
in
in
five
minutes
at
thirty
second
frequency
you're
going
to
get
ten
samples.
We
store
that
number
you
know
of
samples,
and
then
we
also
keep
track
of
the
of
the
some
different
components,
serialize
that
into
one
structure
and
it
gets
ridden.
As
the
roll-up
as
I
mentioned
before,
serialization
is
version
querying.
We
just
have
to
simple
methods.
We
have
one
getdata
method
that
says
get
me
all
the
data
from
this
range
of
time.
A
For
this
granularity,
which,
if
you're
not
careful,
can
return
a
lot
of
data,
and
so
we've
also
had
at
another
method
that
says:
okay,
I
just
want
X
number
of
data
points
and
it
will
choose
the
right
granularity
for
you
and
then
give
you
back
the
data
points
and
that's
handy
if
you're
generating
a
graph.
If
you,
if
you
look
at
your
average
graph,
you
know
that's
300,
pixels
or
400
pixels
wide.
Well,
you
can't
really
fit
a
million
data
points
in
there
and
have
it
be
usefully
presented
anybody.
A
The
first
example
of
one
is
that
TTLs
they're
linked
to
tenants
and
well.
This
is
this-
is
this
is
specific
to
our
system.
Ttls
are
linked
to
tenants
when
they
come
in
when
metrics
come
into
the
system.
We
don't
know
what
the
what
the
tenants
TTL
is.
We
have
to
consult
a
different
API,
and
so
we
use
a
cache
to
to
to
keep
that
around.
So
we're
not
constantly
hitting
this
external
API
to
to
hold
that
data,
you
can
use.
You
use
the
same
kind
of
things
on
your
own.
The
external
API
requests.
A
We've
created
some
abstractions
there
to
make
these
kinds
of
things
pretty
simple.
Another
example
is
that
units
they're
usually
either
usually
only
very
valuable
at
query
time.
You
know
when
metrics
are
coming
into
the
system.
I,
don't
really
care
that
this
means
bytes
I,
don't
really
care
that
this
means
milliseconds,
and
so
we
we
don't
worry
about
it
too
much
at
query
time
other
than
to
make
that
little
association
save
it
to
the
metadata
base
for
blue
flood
so
that
it
can
be
used
at
at
query
time.
A
A
How
do
we
know
when
something
needs
to
be
rolled
up?
There's
two
different
ways:
first,
is
whenever
an
active
slot
hasn't
been
updated
in
a
certain
amount
of
time,
or
it
is
a
certain
second
old
you
know,
gets
scheduled
and
then
rolled
up.
There's
some
lodging
in
there
to
make
sure
that
we
don't
do
this
the
wrong
way,
roll
up
stuff
that
just
can
be
rerolled
again
later
and
things
like
that.
A
We
tolerate
late
data,
it's
going
to
be
dependent
on
your
ingestion
pipeline,
but
we
tolerate
data
up
to
24
hours
late
and
if
we
figure,
if
it's
24
hours,
late,
we've
probably
already
forgotten
about
and
don't
care
about
it
so
much
and
so
use
a
bulk
operation
or
use
something
else
to
get
that
in
ingestion.
Processors
are
where
you
can
do
quite
a
bit
of
customization.
A
This
is
the
this
is
where
you
can
augment
the
data
as
it
comes
into
your
system
and
typically,
what
you're
going
to
see
is
data
coming
in
from
several
different
places.
At
least
you
know,
that's
what
that's
what
we
see.
We
have
this
one
generic
computing
platform,
but
data
comes
in
from
from
many
different
places,
and
so
we
want
to
be
able
to
modify
this
data
depending
on
where
it
came
in
from,
and
so
we
use
the
ingestion
pipelines
for
that
yeah.
A
Well,
one
point
to
point
out
about
that
is
that
the
ingestion
processors
can
either
be
synchronous
or
asynchronous,
and
it
ends
up
working
out
pretty
well
for
us
now
the
API
endpoints.
This
is
what
is
its
it's
up
to
you.
We
found
that
as
we
started
building
this
and
we
started
accepting
data
for
more
places,
we
had
to
change
our
ingestion
quite
a
bit,
and
so
we
decided
okay,
we're
just
going
to
make
this
completely
pluggable.
You
put
your
own
pipeline
in
and
and
things
you
can.
You
can
respond
to
these.
A
The
scale
is
an
ingestion
and
then
rolling
up
ingestion
scales
linearly
by
just
adding
ingestion
nodes
until
Cassandra
becomes
your
bottleneck,
at
which
point
you
have.
You
know
other
kinds
of
problems.
We
have
to
ingestion,
notes
per
data
center
and
run
them
in
two
data
centers,
and
only
one
ingestion,
node
/
data
center
is
active,
but
we
do
double
ingest
and
so
we're.
A
You
know
we
can
tolerate
a
data
center
going
away
or
being
partitioned
and
still
we're
getting
our
metrics
into
into
blue
flub
role,
upscale
almost
almost
horizontally,
basically
by
spreading
out
the
shard
ownership.
Sorry
shards
were
capped
at
128
right
now,
but
it's
okay
to
have
multiple
nodes
processing
the
same
shards
there's
a
little
bit
of
coordination.
That
happens
right
now.
We
use
zookeeper
to
do
that,
but
that's
a
dependency
we'd
like
to
see
go
away.
I
think
we
have
some
interesting
ways
of
making
that
happen.
A
Some
things
that
we're
working
on
in
the
future
is
local
ingestion
durability.
Right
now
we
can't
wait.
When
data
comes
in,
we
you
know
as
it's
being
processed
it
gets.
It
may
not
get
saved.
You
know
immediately
we're
looking
at
some
different
ways
of
making
sure
that
that's
durable
locally
before
I
get
saving
into
Cassandra
that
helps
with
restart
scenarios
and
things
like
that.
We're
also
looking
at
having
a
richer
metadata
API.
So
when
metrics
come
in,
you
can
associate
tags
with
them
and
then
use
those
tags
as
some
kind
of
a
fast
set.
A
That
will
allow
you
to
query
for
metrics
in
more
interesting
ways,
rather
than
just
by
pertinent
and
we're
experiment
with,
and
also
looking
at
ways
that
we
can
do
our
own
bitmap
index
to
do
those
kinds
of
things
and
that's
something.
That's
that's
ongoing
kind
of
on
the
on
the
on
the
back
burner,
also
we're
looking
at
pre,
aggregated
metrics,
or
actually
we're
implementing
them
right
now,
if
you
use
stats
d
or
graphite,
you'll
know
what
I'm
talking
about
counters
timers
those
kinds
of
things
it
turns
out.
A
Deep
stats,
d
and
graphite
integration
is
is
ongoing,
but
stats
d,
it's
hard
because
you
they
have
this
concept
of
things,
get
flushed
and
the
data
go
kind
of
gets
reset
to
zero.
But
you
know
in
blue
flood
the
data
stays
around
until
it
times
out
your
counts,
just
continually
increase
in
most
the
time
you
you
end
up
wanting
that
graphite
is
just
involved
because
of
the
of
the
new
roll-up
types,
integrating
with
a
whisper
DB
whisper
DB
is
kind
of
a
involved.
A
Anyway,
that's
that's
what
we're
doing
to
to
get
as
far
as
the
the
future
work
goes.
If
you
want
to
get
involved
with
the
project,
that's
where
you
can
find
it
it's
on
github
we
have
an
IRC
channel.
We
have
about
three
four
developers
in
rackspace
that
are
that
are
very
active.
In
it,
and
then
some
that
are
coming
from
outside
places
who
are
starting
to
participate,
a
little
more
I'd
encourage
you
to
take
a
look
at
it.
We
have
a
minute
or
two
a
couple
minutes
for
questions.
A
Well,
do
the
math
for
us,
if
a
metrics
coming
in
once
every
30
seconds,
you
know
you
can
figure
out
after
how
you
know
after
a
year,
you're
going
to
have
two
hundred
thousand
rows
or
two
hundred
thousand
columns,
something
like
that,
which
is
still
very
manageable.
As
far
as
Cassandra
goes
now,
if
you're
having,
if
you're
having
hundred
metrics
come
in
a
second
well,
you've
got
a
different
kind
of
different
kind
of
problem,
you're
going
to
want
to
look
at
some
kind
of
bucketing,
and
we
do
plan
on
having
that
kind
of
thing.
Eventually,.
A
All
right,
sir,
how
do
we
manage
consistency
right
now?
I
believe
that
we
ingest
I
believe
that
we
ingest
and
read
both
at
quorum,
and
so
I
mean
it's
it's
something
that
is
it
configurable,
but
we
ought
to
make
it
configurable.
It's
just
that
the
hat
we
haven't
had
a
demand.
Yet
for
that,
and
so
we
haven't
done
it
yet,
but
I
can
see
different
enterprises
wanting
to
be
able
to
control
consistency.