►
Description
Speaker: Adrian Cockcroft, Cloud Architect at Netflix
Slides: http://www.slideshare.net/planetcassandra/c-summit-2013-netflix-open-source-tools-and-benchmarks-for-cassandra
Netflix has updated and added new tools and benchmarks for Cassandra in the last year. In this talk we will cover the latest additions and recipes for the Astyanax Java client, updates to Priam to support Cassandra 1.2 Vnodes, plus newly released and upcoming tools that are all part of the NetflixOSS platform. Following on from the Cassandra on SSD on AWS benchmark that was run live during the 2012 Summit, we've been benchmarking a large write intensive multi-region cluster to see how far we can push it. Cassandra is the data storage and global replication foundation for the Cloud Native architecture that runs Netflix streaming for 36 Million users. Netflix is also offering a Cloud Prize for open source contributions to NetflixOSS, and there are ten categories including Best Datastore Integration and Best Contribution to Performance Improvements, with $10K cash and $5K of AWS credits for each winner. We'd like to pay you to use our free software!
A
Oracle
racing
yacht
is
going
back
and
forth
out
here.
So
at
lunch,
we're
gonna
be
manning
up
some
row
boats
and
we're
gonna
go
take
it
okay,
who's
in
all.
Right
now
we
shouldn't
do
that.
Probably
we're
just
going
to
do
this
from
sure.
So
our
our
first
speaker
here,
he
probably
doesn't
need
much
of
an
introduction,
but
I'm
going
to
try
to
do
it
anyway.
Kind
of
a
personal
thing
for
me
because
of
adrian,
is
why
I
got
involved
with
cassandra
and
he
he
was
one
who
told
me
hey.
A
You
should
try
this
out
so
adrian
cockcroft
he's
the
is
the
director
of
cloud
architecture
at
netflix.
You've
probably
seen
a
few
of
his
talks,
but
today
he's
coming
to
you
as
a
netflix
open
source
guy
he's
been
doing
cassandra
for
a
really
long
time,
probably
one
of
the
one
of
the
foremost
speakers
on
just
how
to
deploy
this,
and
do
it
right.
So
thank
you
very
much
for
being
here
and
take
it
away.
Okay,
thank
you.
B
All
right,
slides,
hey,
I
said
I
should
stand
out
here:
okay,
so
I'm
going
to
talk
about
open
source
tools
and
benchmarks.
I've
got
a
new
benchmark
in
there
who
was
here
last
year,
who
was
at
the
cassandra
summit
last
year.
I
did
a
live
benchmark
that
year,
I'm
not
doing
a
live
benchmark
this
year.
It's
like
it
was
that
was
too
nerve-wracking
for
everyone
else,
but
we'd
have
done
some
interesting
work
since
then.
B
B
So
starting
off
looking
at
cloud
native
cloud
native
is
basically
an
application,
a
way
to
build
your
applications
that
run
in
the
cloud
where
we're
really
optimizing
for
speed.
We're
optimizing
for
time
to
market
because
netflix
is
making
a
global
land
grab
we're
trying
to
build
out
on
in
a
new
category
area
of
streaming
video
and
we're
also
trying
to
do
that
on
a
global
basis.
We're
trying
to
you
know
one
by
one
working
through
all
the
countries
in
the
world.
B
So
that's
that's
a
good
case
for
doing
something
where
you
have
a
very
elastic
need
for
compute
and
it's
growing
rapidly.
It
also
time
to
market
wins.
If
you're
trying
to
disrupt
your
competitors,
hopefully
and
you've
heard
of
the
uder
loop,
that
means
observe
orient
decide
act.
It
was
a
concept
that
came
out.
I
think
the
korean
war,
where
aircraft
in
dogfighting
were
trying
to
get
inside
their
competitors,
ability
to
respond.
B
So
if
you
can
get
inside
your
your
competitor's
ability
to
respond
they're,
looking
at
what
you're
doing
they
start
responding
to
it
by
the
time
they
figure
that
out,
you're
already
looped
around
and
you're
attacking
them
from
behind
or
something
so,
that's
disrupting
competitors,
a
big
piece
of
of
of
going
fast
and
basically
anything
delivered
as
web
services
lets.
You
do
this
if
you're
delivering
package
software,
it's
incredibly
hard
to
to
to
do
it
fast
enough.
B
B
B
If
you
can
do
that
in
seconds,
then
you
can
actually
do
lots
of
things
that
are
that
are
slightly
broken,
but
you
spot
them
really
quickly
when
when
it
matters-
and
you
can
fix
them
really
quickly,
so
I'll
talk
about
how
we
do
that,
so
this
is
really
tipping
the
balance.
I
did
a
talk
recently
called
utopia
dystopia
as
a
service.
So
instead
of
trying
to
make
everything
cheaper,
we
run
it
fairly
inefficiently
because
we're
trying
to
go
faster,
we
don't
try
and
make
stuff
better
now
make
our
code
better.
B
It's
slightly
broken,
but
it
doesn't
matter
because
we're
going
faster
and
we
don't
leave
everything
alone.
It's
there
isn't
a
lot
of
static
code,
a
lot
of
things
sitting
there
for
long
enough
for
us
to
tune
them
because
everything's
dynamic,
it's
everything's,
growing
and
shrinking
and
responding
to
the
market
and
responding
to
capacity
demands.
But
this
balance
of
doing
everything
sooner
actually
helps
us
be
more
competitive.
B
So
this
gives
us
a
a
new
engineering
challenge,
we're
trying
to
construct
this
highly
agile
and
highly
available
service
from
ephemeral
and
often
broken
components
and
the
com.
You
know
our
own
software
is
the
source
of
most
of
the
brokenness,
and
if
you
do
it
right,
you
can
go
at
incredible
speed.
We
can
roll
things
out
and
we
can
be
running
on
on
a
continuous
update
basis,
but
the
services
as
a
whole
keeps
working
so
with
some
inspiration
for
trying
to
do
this.
B
B
If
you,
if
you're
trying
to
keep
stuff
running
in
production,
you
should
go
read
that
thinking
in
systems
isn't
really
about
computers,
it's
more
about
economics
and
about
large-scale
systemic
behavior.
If
you're
interested
in
chaos,
theory
and
complexity
and
how
systems
behave,
it's
it's
a
very
interesting
book
to
go.
Look
at
anti-fragile
came
out
last
year.
We
use
a
lot
of
anti-fragile
patterns
that
the
concept
here
is
that
if
you
keep
trying
to
damage
something
eventually,
it
gets
stronger
a
little
bit
like
working
out.
B
B
Most
of
his
examples
consist
of
planes
falling
out
of
the
sky
and
people
dying
in
hospital,
because
systems
just
gradually
build
up
a
sort
of
technical
debt
in
a
sort
of
a
failure
way
and
everything's
working
fine
and
then
a
plane
falls
out
the
sky.
Oh,
it's
never
done
that
before
you
know,
but
you
know.
The
reason
why
it
fell
out
of
the
sky
is
a
series
of
perfectly
reasonable
decisions
that
were
made
that
in
aggregate
over
some
period
of
time
turn
into
it.
B
If
you're
trying
to
deal
with
the
human
side
of
operating
systems,
you
need
to
understand
what
people
you
know
what
people
see
as
obvious
things
that
are
obvious
in
hindsight,
if
you're
doing
incident
reviews,
it's
obvious
why
everything
died,
we're
stupid!
You
know
no,
it's
not
because
you're
stupid!
It's
because
things
are
only
obvious
in
hindsight,
so
understanding
how
to
make
things
obvious
before
the
the
before
it
breaks
is
an
important
aspect.
B
This
is
more
a
book
about
sociology
and
psychology
of
people
we're
doing
a
lot
of
rest
apis
right
now,
we're
consuming
a
lot
of
rest
apis
and
a
lot
of
them
are
really
nasty
a
lot
of
really
badly
designed
ones
and
george
reese.
If
you
follow
him
on
twitter,
is
sort
of
continuously
ranting
about
bad
apis
and
eventually
he
wrote
a
book.
It's
a
short
book,
definitely
worth
reading.
It's
full
of
all
the
anti-patterns
of
things
you
shouldn't
put
in
an
api
and
finally
cloudnomics
joe
wyman
has.
B
This
is
a
really
I
call
it
it's
an
exhaustive
book,
it's
really
quite
hard
to
read
to
get
all
the
way
to
the
end
of
it,
but
he
has
all
of
the
economics
of
cloud
computing
and
how
to
think
about
the
cost
model
and
all
the
all
of
the
behavior.
So
this
is
much
more
of
a
management
book
about
you
know
what
is
what
is
the
business
value
of
cloud
computing?
B
So
that's
the
inspiration,
but
genius
is
one
percent
inspiration
and
99
perspiration.
We
had
a
bit
of
thomas
edison
earlier
that
wasn't
coordinated.
So
this
is
our
perspiration.
We've
been
working
on
netflix
oss,
it's
cloud
native,
open
source
platform
for
for
some
years
we
about
a
year
ago.
We
wrapped
this
sort
of
brand
around
it
and
started
releasing
pieces
of
it
and
it's
all
on
github
and
it
has
evolved.
We
started
off
2009
to
10.
We
were
really
bleeding
it.
B
We
were
making
things
up
figuring
things
out,
trying
things
four
different
ways
working
out
which
one
worked.
There
wasn't
really.
There
were
a
few
guidelines
from
say:
google,
white
papers
and
amazon
white
papers,
but
we
didn't
have
much
to
follow
2011
2012
as
we
went
out
and
started
talking
about
it.
A
common
pattern
emerged.
There
are
other
companies
working
in
this
way.
We
shared
ideas
and
we've
evolved,
something
that
that
cloud
native
has
become
more
of
a
common
pattern.
B
Now,
what
we're
doing
this
year
and
going
forwards
is
sharing
that
pattern
by
releasing
open
source
the
pieces
that
make
up
that
and-
and
it's
because
we
don't
regard
this
as
being
a
a
competitive
advantage
for
netflix
having
a
cloud
native
application.
Our
competitive
advantage
comes
from
having
exclusive
rights
to
you
know
interesting
content.
B
So
our
goals
here
establish
our
solutions
as
best
practices
as
standards
test
them
in
the
marketplace
of
ideas.
So
we
we
release
something.
People
say
that's
stupid
with
a
better
way
of
doing
it.
There's
another
open
source
package
that
does
it
better.
We
start
cross-learning
we're
also
trying
to
hire,
retain
and
engage
top
engineers.
That's
been
working
out
quite
well.
We
have
several
people
that
have
joined
netflix
purely
because
of
our
open
source
program
that
works
really
well
as
a
as
a
a
motivating
thing
for
the
people.
B
B
So
that's
why
we're
doing
this,
but
what
we're
also
looking
for
is
some
of
your
perspiration
right.
We've
been
doing
all
this
work
and
we've
launched
the
cloud
prize,
so
this
is
a
prize
for
contributions
that
are
related
to
our
open
source
platform
and
a
lot
of
those
contributions
could
be
around
cassandra.
So
I'm
going
to
highlight
those
as
I
go
through.
B
You
enter
this
prize
by
going
to
github
looking
for
the
cloud
price
project
and
forking
it.
So
it's!
Basically,
if
you
don't
know
what
github
is
you
shouldn't
be
entering
this
prize?
That's
kind
of
one
of
one
of
the
filters.
Here
we
had
a
couple
of
confused
people
trying
to
give
us
ideas,
for
you
know
making
netflix.
B
You
know
movies
behavior
better.
This
is
about
the
platform,
so
we
have
some
categories
for
for
the
way
this
price
should
work
best
example:
application
mashup.
We
have
a
couple
of
a
couple
of
those
out
there
already
there's
one
that
netflix
did
and
there's
one
called
flux,
capacitor
that
a
guy
that
used
to
work
at
netflix
built
the
best
new
monkey
monkeys
are
the
automation
things
think
him
like
a
unix
demon
but
running
as
a
service
in
the
cloud
that
does
some
autumn
automation.
B
We
have
chaos,
monkey
and
things,
so
you
can
invent
a
new
monkey,
the
best
contribution
to
code
quality.
If
there's
some
qa
engineers
out
there,
you
can
come
in
or
anyone
working
with
our
our
tools.
If
you
contribute
bug
reports,
you
know
we'll
just
talk
up
who
has
the
biggest
number
of
bug
reports
and
made
the
best
biggest
difference
at
the
end
of
the
day,
we'll
give
them
a
prize
the
best
new
feature
added
to
an
existing
project,
best
contribution
to
operational
tools.
B
Best
data
store
integration,
yeah,
probably
the
wrong
audience
for
this,
but
I
have
a
suggestion:
why
doesn't
somebody
see
if
they
can
get
react?
Working
with
with
with
our
tools?
We
got
a
boo
from
anybody.
Okay,
good
and-
or
you
know,
one
of
the
other
ideas.
I've
got
is
well.
We
have
aston
x,
which
is
our
client
library.
You
could
make
that
work
with
amazon
dynamodb,
for
example,
and
just
have
that
as
a
compatible
interface
that
those
kinds
of
things
just
broadening
the
choices
that
we
have
best
usability
enhancement.
B
B
B
One
of
vogels
who's
kicked
in
some
additional
prizes,
I'll
talk
about
in
a
minute
joe
weinman,
who
I
mentioned
his
book
a
little
earlier
and
my
manager,
and
if
we
get
a
tie,
my
manager
will
sort
of
step
back
and
see
who
of
the
other
five
picks
who's
who
won.
So
these
are
the
six
people
you're
trying
to
impress.
B
So
what
do
you
win?
You've
got
ten
categories:
you're
going
to
get
to
go
to
re,
invent
in
las
vegas
in
november.
There's
a
trophy:
it's
a
cool
trophy.
I
don't
have
a
picture
of
it.
Yet
we're
still
working
on
that
it'll
be
more
than
just
a
boring.
Trophy
it'd
be
a
very
cool
trophy
ten
thousand
dollars
cash
and
five
thousand
in
amazon
credits
coming
from
aws,
so
aws
credits.
So
that's
that's
the
prize.
B
We
got
10
of
those
prizes
and
including
the
trip
to
vegas
and
everything
it's
roughly
sort
of
20k
worth
of
value
there.
So
that's
what
we've
got
we're
trying
to
encourage
people
to
add
a
bit
of
perspiration
to
what
we've
done
and
leverage
it
further.
So
just
trying
to
get
attention
get
your
attention
span
to
come
and
do
something,
and
if
you're
working
and
using
any
of
our
software
just
go
and
put
a
prize
entry
up
and
list
what
you've
been
doing,
we're
just
interested
to
know
keep
track
of
everything.
B
So
this
is
a
little
summary
of
that.
We
opened
registration
there's
a
six
month
period.
We
started
this
in
march.
It
runs
till
september,
so
I
know
everyone
procrastinates
and
waits
till
the
last
weekend
before
they
write
any
code.
But
there's
some.
You
get
extra
credits
if
you
do
something
by
july
17th,
because
we're
having
another
open
source
meetup
at
netflix,
where
we're
going
to
feature
contributions.
B
Okay,
so
let's
look
at
netflix
streaming
and
how
that
operates.
This
is
a
cloud
native
application
based
on
this
open
source
platform.
So
most
of
you
probably
have
netflix.
This
is
roughly
what
it
looks
like
you
see,
a
bunch
of
movies
picked
from
me
and
some
friends
favorites,
because
I'm
hooked
up
with
the
facebook
integration.
B
B
So
you
talk
to
the
website
or
discover
api
find
out
about
everything.
You've
got
user
data
coming
from
cassandra
personalization
data
coming
from
cassandra.
Then
we
have
the
streaming
api
after
you've
hit
play.
We
get
the
drm
data,
we
do
quality
of
service
logging,
which
gets
written
into
cassandra.
It
remembers
where
you
are
in
all
your
different
movies
for
bookmarks,
again
cassandra
again
and
then
behind
open
connect,
the
cdn
boxes,
the
health
of
all
of
our
cdn
boxes
is
also
managed
in
a
back
end,
that
does
cdn
management
and
steering
and
content
encoding.
B
B
B
Peak
period
applications
fixed
access,
north
america,
so
this
is
the
peak
sort
of
sunday
night,
the
dsl
lines
and
cable
modems
across
across
north
america
right,
not
mobile,
not
the
rest
of
the
world
they're
a
little
bit
different,
but
this
is
the
us
and
netflix
is
a
third
of
the
traffic
and
they
did
an
update,
and
in
this
update
they
mentioned
that
the
total
bandwidth
went
up
almost
40
percent
in
six
months.
So
that's
the
top
of
line,
number
went
up
and
we
were
still
roughly
a
third
of
that
traffic.
B
B
That's
80,
we're
18
times
the
the
bandwidth
of
amazon,
prime
and
six
months
later
we're
25
times
the
bandwidth
of
amazon,
prime.
I
had
to
actually
look
it
up.
They
weren't
even
in
the
top
10.,
so
not
too
concerned
about
that
as
a
as
a
major.
You
know
we're
concerned
about
them
as
a
competitor,
but
in
terms
of
traction,
I
like
the
way
these
numbers
are
moving
right
now
and
but
there's
a
good
reason
why
everyone
has
amazon.
B
Prime
anyway,
I
have
amazon
prime,
I
don't
watch
anything
it's
with
shipping
right,
okay,
so
going
back
to
what
netflix
looks
like
this
is
the
flow
of
traffic
behind
our
home
page?
The
start
here
button.
Where
is
it
this
thing
here
that
that
is
a
few
hundred
web
servers?
All
tomcat
servers
out
here
you've
got
it's
all
java
end
to
end
using
app
dynamics.
We
track
all
these,
so
there's
a
cassandra
cluster.
B
There's
a
memcache
there's
a
bunch
of
things,
so
this
is
from
a
little
while
ago,
but
this
shows
you
the
complexity
of
what
we
call
a
micro
service
architecture
and
by
microservice
we
mean
each
server.
Does
one
thing:
there's
a
single
function
service?
It's
not
so
much
that
it's
small!
Most
of
these
things
have
sort
of
20
gigabyte
heaps
java
apps,
but
that's
basically
what's
going
on
they're
pretty
big,
but
they
are
they're
single
function
so
and
you
you
replicate
them
and
the
the
that's
the
right
way
of
thinking
about
this.
B
So
if
that
service
breaks
the
site
keeps
working,
maybe
you
get
one
of
the
rows
on
the
on
the
home.
Page
no
longer
appears
and
is
filled
in
with
some
other
row.
So
that's
the
way
we
we
work
by
having
single
functions,
it's
very
easy
to
identify
what
went
wrong
it's
in
the
old
days.
We
used
to
have
lots
of
jar
files
that
would
be
integrated
every
few
weeks
into
this
massive
blob
of
code
and
when
it
broke,
nobody
could
figure
out
which
jar
file
was
responsible.
B
If
you,
this
is
a
simplified
view
of
all
of
the
services
talking
to
all
of
them.
So
this
is
not
just
the
home
page.
This
is
sort
of
everything.
In
fact,
this
is
a
very
simplified
version.
The
real
version
is,
we
call
it
the
death
star.
You
can't
actually
render
it
anymore,
but
this
one
was
one.
I
I
wanted
a
picture
that
just
showed
that
some
complexity-
and
we
test
this
by
you-
know
killing
individual
services
using
chaos,
monkey
and
using
latency
monkey
to
inject
errors
into
it.
B
So
this
is
part
of
the
sort
of
resilience,
anti-fragile
testing,
but
what
we
actually
do
is
we
run
three
copies.
So
that's
the
same
diagram
replicated
three
times
so
we
have
three
availability
zones,
three
buildings
or
data
centers
at
the
bottom.
We
have
cassandra
and
ev
cash,
which
is
our
memcached
wrapper
and
the
load
balancers
spread
traffic
over
all
of
them
and
once
you
land
in
one
of
these
zones,
mostly
you're
calling
services
within
that
zone,
we
have
zone
aware
routing.
So
you
don't
keep
calling
the
machines
in
the
other
zone.
B
B
So
we
test
this
with
chaos
gorilla.
We
do
this
every
quarter.
We
kill
a
third
of
our
infrastructure
in
production
just
to
see
if
it
survives-
and
we
did
this
a
few
months
ago
and
found
a
few
bugs,
but
it
didn't
take
the
site
down.
Basically,
the
response
to
losing
a
zone
is
to
turn
the
load
balancer
off,
so
we're
no
longer
sending
traffic
to
it,
and
everything
keeps
running
on
two
out
of
three
because
we're
using
quorum.
Two
out
of
three
works.
Everything
keeps
working
so
we've.
B
This
has
happened
multiple
times,
and
this
is
the
basic
chaos
gorilla
test
that
we
run
every
quarter
to
prove
that
we
can
still
do
this
if
you're
maintaining
cassandra
because
you're
updating
an
individual
instance,
the
replica
of
part
of
the
data
in
one
zone
will
disappear.
So
you
won't
be.
Those
machines
in
that
zone
won't
be
able
to
pick
up
the
copy
of
data.
That's
in
that
zone
which
they'd
preferentially
talk
to
because
of
the
lower
latency
to
talk
to
in
thin
zones.
B
So
the
replication
stops
working
because
that
one
cassandra
node
is
being
updated
to
a
new
version
of
cassandra
or
something,
and
so
basically,
in
that
case,
all
of
the
instances
in
that
one
zone
will
talk
across
zone
to
pick
up
copies
of
the
data.
So
that's
basically
what
happens
whenever
an
individual
in
cassandra
node
drops
out
of
gossip
and
is
is
unavailable
so
that
works
fine.
B
If
anything
goes
wrong
with
the
connection
between
the
two
everything
keeps
working.
This
is
the
key
thing
we
can
lose
all
of
the
us
yeah
or
all
of
the
europe
and
everything
the
other
the
other
continent
keeps
working.
So
this
is
really
important.
This
has
worked
very
well
in
in
in
production
for
now
like
a
year
and
a
half,
so
these
are
the
things
that
can
go
wrong
and
what
to
do
about
it,
application
failure,
we're
pushing
new
code
all
the
time.
We
expect
it's
a
fairly
large
proportion
of
those
failures
to
be
broken.
B
We
have
automatic
degraded
response
and
we
wrap
everything
around
with
circuit
breakers
and
bulkheads
and
we
can
handle
things
things
being
broken,
but
we
also
try
and
detect
it
within
seconds
and
respond
within
seconds
if
possible.
So
it
means
that
it's
a
very
you
there's
a
bias
to
action.
Basically
in
getting
your
code
out,
you
don't
have
to
put
it
through
qa
for
weeks.
You
can
just
say
well
this
most
of
the
time
it
works,
we'll,
try
it
oops.
We
back
figure
that
and
back
out.
B
So
it's
a
little
bit
sort
of
taking
a
bit
of
a
liberty,
but
it
does
work
region
failure.
We've
had
a
few
times
in
the
last
year.
It
doesn't
happen
that
often
what
we
want
to
get
to
is
be
able
to
switch
traffic
between
regions
and
I'll
talk
about
that.
Next.
We're
working
on
that
zone
failure
is
something
that
we
expect
to
happen
reasonably
often
and
to
be
able
to
deal
with
automatically
and
transparently.
So
the
customers
don't
see
it.
We've
had
data
center
failures
that
have
taken
our
services
out.
B
We
have
things
like
hardware,
security
modules
in
our
data
center
and
we've
migrated
those
to
the
cloud
now.
So
you
saw
aws
come
out
with
hsm
recently,
largely
because
we
beat
them
up
to
go
and
do
it
because
we
wanted
it
other
when
they
talk
to
other
people
turns
out
everyone
else
wanted
it
as
well.
So
that
was
that's
one
of
the
ways
that
we
cut
our
data
center
dependencies
and
that's
helped
us
be
more
more
available
data
store
failure.
B
So
if
anything
goes
wrong
there,
we
can
restore
to
some
point
in
time
to
get
get
back
to
a
working
cl,
working
data
store
and
if
s3
breaks,
we
actually
have
a
another
copy
of
that
data.
In
another
amazon
account
in
another
region.
We
do
a
date,
a
daily
sort
of
copy
of
s3
to
s3
from
a
remote
archive,
so
we're
pretty
well
covered
for
most
of
the
failure
modes
that
we
could
run
into.
B
So
this
is
based
largely
on
cassandra,
and
this
is
sort
of
generically.
What
one
of
our
services
looks
like
I?
This
is
another
app
dynamics,
diagram
showing
lots
and
lots
of
rest
clients.
The
thing
in
the
middle
isn't
cassandra:
this
is
our
rest
data
access
layer,
so
it's
a
rest
service
that
abstracts,
whatever
this
data
store,
is
into
a
single
function,
a
single
data
store.
So
we
have
over
50
cassandra
clusters
over
50
different
data
stores.
B
So
this
is
denormalized
to
the
point
where
every
table
or
every
materialized
view
or
whatever
you'd
normally
have
in
a
regular
sort
of
relational
database,
is
in
a
separate
cluster
managed
by
a
separate
team
with
its
own
update
rate,
its
own
capacity
problems,
its
own,
whatever
we're
doing
right.
So
that
means
that
you
there's,
you
can't
do
join
because
you're
not
talking
to
one
cluster
anymore.
B
B
Having
this
middle
data
tier
lets,
you
do
a
few
clever
things.
One
is
that
you
can
t
data
off
into
an
old
data
store
the
data
center.
So
I
wrote
it
at
the
bottom,
this
optional
data
center
here
the
way
you
transition
out
of
a
data
center
relational
model.
Is
you
build
a
data
access
layer
in
front
between
your
app
and
the
data
center
sql
table
or
whatever?
B
And
then
you
gradually
t
that
off
into
cassandra,
and
then
you
move
all
of
your
applications
to
use
your
data
access
layer
and
then
eventually
you
turn
off
the
access
to
the
to
the
old
data
store,
so
there's
kind
of
a
gradual
migration,
but
you
do
it
by
hiding
behind
this.
This
stateless
data
access
service
and
we've
done
that
for
all
of
our
different
data
sources.
B
Okay,
let's
see
more
to
say
on
that,
so
each
of
those
icons
on
this
show
is
over
over
three
zones
and
potentially
hundreds
of
actual
individual
servers
all
right.
So
each
of
those
stateless
micro
services
look
like
this.
There's
a
base
ami
we're
currently
in
the
transition
from
centos
to
ubuntu,
there's
a
copy
of
java
with
tomcat
that
consumes
most
of
the
memory
on
it.
That's
running
your
application
and
a
bunch
of
standard
packages
and
things
that
we
have
a
health
check.
We
have
status
jmx
all
that
kind
of
stuff.
B
So
this
is
our
stateless
service
architecture.
Most
of
our
our
services
are
running
this
way.
The
way
that
cassandra
looks
is
a
little
different.
If
you
take
that
wrapped
up
tomcat
server,
that
was
the
big,
the
main
box
here,
the
big
bluish
box,
and
we
put
that
over
in
the
top
left
corner.
That's
now
tomcat
running
our
pre-em
code,
which
is
one
of
our
opening.
How
many
people
here
use?
Pream
I've
got
a
few
yeah.
B
B
B
The
mirroring
is
between
cassandra
instances
across
zones,
so
everything
we
write
is
written
to
three
zones,
there's
three
copies
of
it
and
then
eventually,
it
finds
its
way
onto
the
two
terabytes
of
ssd
or
1.6
terabytes
of
of
local
hard
drive
on
the
instance
types
we
use.
Okay,
comment:
login
ss
tables
all
written
in
there.
B
That's
just
want
to
make
sure
people
understand
we're
not
using
attached
storage,
we're
using
the
local
ephemeral
storage
and
that's
been
working
very
well
for
us.
That
way.
When
you
talk
to
the
ssd
there's
no
network
in
the
getting
in
the
way,
you
can
get
a
gigabyte
per
second
for
ssd
around
a
tenth
of
a
millisecond
random
access.
You
can
do
a
hundred
thousand
iops
of
of
read
or
write
traffic
and
that
you
know
at
that
point
all
your
all
your
you
know,
dear
ending,
with
compaction
problems,
sort
of
go
away.
B
So
this
is
what
preem
does
we've
been
building
this
for
a
little
while
it's
one
of
the
early
projects
we
open
sourced
zero
touch
order,
configuration
it
automatically
replaces
broken
instances.
So
if
we
lose
an
instance
for
some
reason,
the
autoscaler
will
recreate
it
and
when
it
comes
up,
preem
will
figure
out
that
it's
replacing
a
node
in
the
existing
cluster
and
automatically
bootstrap
itself
with
the
data
and
do
repairs
to
get
itself
back
into
state
so
completely
hands
off.
B
B
You
can
also
use
it
to
grow
and
shrink
the
cassandra
ring.
So
there's
a
ring
doubling.
It
will
do
the
token
layout
for
managing
doubling
the
size
of
the
ring,
and
that
was
the
demo
I
did
last
year.
If
you,
if
you
pull
up
last
year,
cassandra
summit
presentation,
I
fired
up
a
ssd
and
a
non-ssd
12
node
cluster
live
on
stage
from
nothing
doubled
the
size
of
it.
We
ran
some
benchmark
low,
double
the
size
of
both
those
clusters
and
then
shut
them
down,
and
and
did
all
that
within
a
one
hour
talk.
B
So
what
are
we
doing
now?
Jason
brown's
the
somewhere
around
here
he's
the
committer
that
we
have
working
on
cassandra
and
he's
been
working
on
updating,
pre
and
for
1.2
and
for
v-notes.
There's
some
work's
been
done
here
for
restructuring
it,
but
we
have
to
think
about
a
lot
harder
about
how
we're
doing
backups.
The
way
we
do
backups
right
now
is
that
each
cassandra
node
has
one
token.
So
we
do
all
the
backups
relative.
To
that
token,
with
vnodes,
it's
it's
a
little
more
complex.
B
So
this
is
an
opportunity
to
make
a
cloud
prize
entry.
If
anyone
wants
to
work
on
this
or
work
with
us
on
this,
we're
going
to
have
to
figure
this
out
ourselves
at
some
point,
but
this
is
basically
going
to
block
us
from
using
v
nodes
until
we
figure
out
how
to
back
up
cassandra
in
a
new
way,
because
our
current
mode
is
based
on
you
know,
storing
the
data
for
one
token
in
s3
as
a
file,
so
you
know
I'm
going
to
put
that
logo
up
a
bunch
of
times.
B
So
anyone
working
in
this
area
talk
to
us,
put
a
cloud
prize
entry
in
and
we'll
work
together,
and
hopefully
you'll
win
something
okay.
So
we
do
big
data,
because
this
is
the
big
data
place
right.
So
when
I
think
about
big
data-
and
I
think
about
trying
to
do
it
in
a
data
center,
I
think
that
must
be
incredibly
frustrating
because
you
have
a
fixed
size,
cluster
and
everyone's
trying
to
fit
their
analytics
jobs
into
that
cluster.
B
We
turned
it.
We
turn
it
inside
out.
We
create
clusters
that
are
right
size
for
answering
whatever
question
we
have
right
now
and
we
never
have
to
wait
for
space
and
never
have
to
wait
for
answers
because
we
can
scale
dynamically
and
elastically
have
any
size,
big
data
analytics.
We
want
to
run
against
our
data
source,
and
this
is
what
it
looks
like
any
of
you
going
to
the
hadoop
summit
in
a
week
or
two
there'll
be
a
lot
more
on
this
and
we're
open
sourcing.
Two.
B
So
what
we've
got
here
is
data
coming
in
from
cloud
services.
We've
got
event,
data
about
100
billion
events
a
day
flowing
in,
and
then
we
have
because
from
cassandra
we're
pulling
in
dimension
data.
That's
like
the
universe
of
customer
ids.
That's
what
I
mean
by
dimension
dimension
data
we
bring
that
in
and
we
actually
put
it
in
s3.
So
s3
is
our
data
warehouse.
It
just
keeps
growing,
it's
elastic.
You
don't
have
to
worry
about.
You
know
I
need
another
four
terabytes
of
s3.
B
I
don't
have
to
provision
disks
to
do
that
at
the
bottom.
We
have
a
few
cassandra
clusters.
We
have
a
1300
node
cluster
that
does
just
continuously
processing
all
this
all
of
the
regular
stuff
we
do
to
clean
data
up.
We
have
an
800
node
cluster,
that's
used
as
the
analytics
cluster
for
most
analytic
jobs.
The
one
you
know
takes
the
sort
of
base
workload
of
people,
loading
up
data,
and
if
you
want
to
do
anything
special,
you
just
start
another
cluster
and
because
our
data
is
in
s3.
B
All
of
these
clusters
have
access
to
the
same
data.
They
can
all
read
from
s3
and
write
back
to
s3.
If
you
put
data
inside
hadoop,
it's
all
sitting
in
hdfs
and
and
then
you
have
to
send
your
request,
your
analytics
to
that
cluster,
to
get
at
that
data.
So
by
putting
the
data
in
s3
we're
able
to
do
lots
of
analytics
in
parallel,
we
can
do
upgrades
to
hadoop
versions
by
having
clusters
side
by
side
things
like
that.
B
Now,
every
night
we
have,
we
auto
scale
our
machines
down.
So
we
have
fewer
api
servers
running
and
we
don't.
We
have
leftover
reservations
machines.
We
reserve
that
we're
not
using,
and
we
fire
up
a
few
clusters,
150
node
clusters
for
doing
the
daily
sort
of
etl
roll
up.
You
know
there's
all
that
daily
business
intelligence
stuff.
What
happened
yesterday
so
past
midnight?
B
B
So
I
mentioned
on
here
it
says,
adjust
this.
This
is.
This
is
the
the
tool
we
did
for
that.
Our
data
is
spread
across
up
to
50
clusters.
We
need
to
renormalize
it
we're
trying
to
you
know
you're
trying
at
some
point
you
have
to
do
that
join
in
order
to
do
the
business
intelligence
work.
So
what
we
built
was
a
hadoop
cluster
that
has
an
input
formatter.
That
knows
how
to
read
ss
tables
cassandra,
backup
tables.
B
Just
this
is
the
person
that
killed
cassandra
in
greek
mythology.
So
this
works
on
a
dead
cassandra
cluster.
All
you
need
is
the
backups.
That's
why
we
call
it
justice,
well
the
team
that
they
were
able
to
kill
having
a
running
cluster.
By
doing
it
this
way,
and
then
we
put
it
in
the
teradata.
We
put
it
in
s3,
you
know
we
used
to.
We
still
have
a
little
teradata
running
on
the
side,
but
it's
gradually
being
replaced
by
a
combination
of
hadoop
and
amazon
redshift.
B
Okay.
So
let's
look
at
the
global
architecture
and
some
benchmarks,
if
you
think
about
what
happens
with
a
large
with
a
global
cassandra
cluster
there's
local
client
traffic
coming
in
from
the
top
logically
there's
synchronous
replication
going
to
the
other
replicas
in
in
that
zone.
You
know
within
zones
within
a
region
and
we
have
asynchronous
replication
going
across
regions
so
to
manage
that
we
built
our
own
client
library
who
uses
aston
x.
Here,
it's
okay,
more
people,
I
thought
so
that's
one
of
our
most
popular
projects.
B
So
a
few
things
here:
one
is
it
abstracts
the
connection
pool
from
the
protocol,
so
it
lets
us
manage
retries
and
failover
failure
modes
much
more
cleanly.
It's
got
a
fluent
api,
that's
quite
convenient
for
using.
We
don't
do
all
that
string,
chopping
and
slicing
that
you
typically
do
with
sql
or
cql
queries.
B
Here's
some
of
the
recipes
and
one
of
the
things
that
we'd
be
looking
for
as
cloud
prize
entries,
is
more
recipes,
so
there's
a
distributed,
roblox
that
doesn't
need
zookeeper
or
there's
a
version
that
you
can
use
zookeeper
integration.
If
you
want
to
as
well
multi-region
rollock
uniqueness
constraint,
a
multi-row
uniqueness
constraint,
large
file
store.
If
you
want
to
store
a
gigabyte-sized
object
into
cassandra,
you
soon
find
it's
a
really
bad
idea,
because
it
blows
everything
up
when
you
try
and
read
and
write
it.
B
So
the
point
about
these
recipes
is
this
is
all
the
stuff
that
our
developers
over
the
time
have
stumbled
into
built
a
bad
version
of
we've
debugged,
it
figured
it
out,
worked
out
how
to
make
it
work
right
and
then
publish
it
as
a
recipe
so
that
the
next
developer
just
picks
up
the
recipe
and
uses
it.
So
this
is
probably
the
main
value
of
aston
x
at
this
point.
These
are
really
high
level.
Well,
debugged
recipes
jonathan
talked
a
little
bit
earlier
about
durable
message,
queue
and
the
ways
not
to
do
it.
B
This
is
an
implementation
of
the
right
way
to
do
a
durable
message:
queue
without
with
the
proper
range
query,
so
that
you
don't
do
millions
of
you
don't
run
into
the
problem
of
having
lots
of
tombstones
in
your
query,
so
it's
code
that
implements
this
correctly,
so
you
don't
have
to
think
about
it
and
we
use
sqs
as
a
message
queue
for
our
most
of
our
systems,
but
it's
it
has
a
timeout,
so
the
data
doesn't
last
forever.
This
is
a
message
queue
where
you
can
leave
stuff
in
it
forever.
B
If
you
want
to,
we've
also
got
one
contributed
recipe
right
now,
which
is
a
high
cardinality
reverse
index,
and
that's
currently
in
the
review
process
not
been
posted
yet,
but
it's
a
pull
request
for
it.
So
it's
definitely
looking
for
more
recipes.
Anyone
using
aston
acts
if
you've
implemented
something
that
looks
a
bit
like
a
pattern,
throw
it
in
there.
B
So
futures
have
been
talking
about
what's
happening
next
and
particularly
with
the
new
driver.
Our
commitment
here
is
to
maintain
back
compatibility.
We
have
a
big
code
base
sitting
on
top
of
aston
x.
These
recipes
are
very
powerful.
All
of
that's
going
to
stay
there,
we're
not
going
to
walk
away
from
aston
acts,
we're
not
going
to
walk
away
from
these
recipes.
B
So
there's
a
will,
be
you
know,
seamless
forward
compatibility,
that's
that's
the
goal,
we're
working
on
it
and
we're
you
know
looking
for
contributions
if
people
want
to
help
us
work
on
it
with
there's
basic
cql
support
in
current
estimates,
we're
working
on
some
things
that
will
give
better
access
to
the
cql
level
queries,
so
you
can
mix
the
async
fluent
model
of
just
talking
in
basically
binary.
You
know
real
objects
and
you
can
also
mix
in
there.
B
Some
of
the
cql
more
text
based
queries,
select
queries
and
then
we're
looking
for
some
cloud
prize
ideas.
One
of
the
things.
One
of
the
thoughts
I
have
is
that
it's
kind
of
a
pain
to
run
a
really
small
cassandra
cluster.
It's
actually
somewhat
easier
to
just
fire
up
a
really
small
dynamo
db,
but
if
you
run
a
big
dynamodb
cluster,
it
gets
expensive
really
quickly
because
they
charge
you
per
request.
B
So
there's
a
cut
over
point
where
you
might
have
people
starting
off
on
dynamo
and
then
wanting
to
outgrowing
it
and
wanting
to
move
to
cassandra.
So
what
I'd
like
to
see
is
somebody
figure
out
how
to
take
the
the
aston
x
client
api
and
put
a
dynamodb
back
in
in
behind
it,
so
that
people
can
use
aston
x
and
just
migrate
back
and
forth
between
dynamo
and
and
cassandra
and
then
just
more
recipes?
B
Hector
isn't
token
aware
a
lot
of
the
clients
aren't
token
aware
they
just
access
a
they
pick
a
cassandra
instance
at
random
and
say
here's
some
data
or
I
want
to
read
something
and
the
node
that
they
connect
to
doesn't
have
that
data.
So
assinix
is
aware
of
the
tokens
in
the
cluster
and
this
again
needs
to
be
rethought
for
v
nodes,
but
this
is
the
way
we're
currently
working.
B
It
goes
in
there
and
it
says
I
know
that
this
is
one
of
the
three
nodes
that
has
a
replica,
so
I'm
gonna
go
straight
to
it.
That
gives
us
one
network
hop
instead
of
two,
so
it
halves
the
latency
how's
the
possibility
of
things
going
wrong
you
get
to
that
node
and
that
node
then
farms
out
the
other
two
replicas
for
you
as
a
coordinator
now,
when
you're
doing
that
in
multi-region
there's
an
additional
single
copy
that
that
coordinator
then
sends
across
to
the
other
region
where
it's
duplicated
and
that's.
B
This
is
the
basic
mechanism
we
have
for
multi-region
replication,
so
there's
an
asynchronous
replica,
it's
sort
of
100
plus
milliseconds,
to
go
to
europe
and
about
80
milliseconds
to
get
to
from
east
to
west
in
the
us.
So
that's
our
asynchronous
replication
mechanism,
and
this
has
been
working
very
reliably
for
us,
so
I
did
some
benchmarks
and
a
couple
of
years
ago.
This
is
when
I
posted
this
slide.
B
It
turns
up
in
all
the
data
stacks
presentations
and
I
kept
seeing
these
pop-up
adverts
everywhere
for
a
while,
which
turns
out
if
you
keep
clicking
through
it.
You
end
up
at
my
my
bench,
my
blog
post.
The
point
here
wasn't
that
we
were
doing
a
million
necessarily
it
was
that
it
was
linear
and
we
ever
we
just
increased
the
load,
double
the
number
of
machines
and
everything
just
scaled
up.
B
B
So
that
was
last
year
and
we
benchmarked
ssds
last
year,
so
this
year
we
have
a
new.
We
have.
We
have
a
new
use
case,
so
we've
been
doing
geographic
isolation,
euros
us
to
europe
replication
for
mostly
for
subscriber
data.
So
if
you
sign
up
in
the
us,
you'll
remember
everywhere
in
the
world
and
if
you
happen
to
be
traveling
in
europe,
your
netflix
just
works,
because
we
recognize
you,
that's
that's
the
model.
B
It's
read
intensive,
so
the
fact
that
you're
in
the
middle
of
watching
house
of
cards
in
the
us
is
not
known
about
in
europe.
We
don't
replicate
that
data.
What
we're
looking
at
now
is
redundant
regional
failover
redundancy
for
the
us.
We
want
to
go
from
u.s
east
to
west
and
we
want
it
to
if
you
happen
to
live
in
saint
louis
and
you
keep
crossing
the
mississippi
and
you,
you
know
we're
sort
of
flipping
you
from
east
coast
to
west
coast,
because
that's
roughly,
where
the
population
splits.
B
What
we
want
is
to
the
updates
to
to
your
bookmarks
to
appear
in
both
places
those
kinds
of
things
we
want.
If
you,
if
you
the
fact
that
you
like
a
certain
genre
to
appear
in
both
places,
so
what
that
means
is,
we
have
write
intensive
data
that
we're
going
to
have
to
replicate
both
sites
and
we've
been
testing
that
so
one
of
the
ways
we
wanted
to
test.
B
It
was
well,
let's
fire
it
up,
so
we
took
one
of
our
biggest
clusters:
16
ssd,
nodes
per
zone-
so
that's
a
48
node
cluster
and
that
was
running
in
virginia
in
us
east
and
we
set
up
an
additional
cluster
of
the
same
size
in
oregon
in
u.s
west,
and
this
is
fairly
it's
only
recently.
The
oregon
got
ssds
and
in
the
last
month
or
two
and
we
were
able
to
get
96
ssd
nodes.
B
So
this
is
almost
200
terabytes
of
ssd
raw
capacity,
but
the
data
set
we
had
was
we
loaded
from
s3,
it's
18
terabytes
of
data.
We
sucked
that
in
from
backup
that
took
a
few
hours
and
that
basically
the
next
thing
we
did
was
replicate
that
data
across
to
oregon.
So
we
pushed
that
18
terabytes
to
oregon
I'll
show
you
what
happened
when
we
did
that
in
a
minute,
and
then
we
so
that
caused
into
region
traffic,
pushing
the
data
over.
B
Regionally,
we
want
to
be
able
to
sort
of
get
through
the
christmas
peak,
with
more
capacity
turns
out
that
we
had
way
more
capacity
than
we
needed
doing
that,
so
this
generates
into
zone
traffic,
because
anything
I
write
into
one
one
region
is
now
being
replicated
within
region
and
across
and
into
region,
and
then
just
to
prove
that
this
worked,
we
set
up
a
validation
load
where
we
wrote
as
fast
as
we
could
a
million
rights
with
cl1.
So
that's
not
fully.
B
You
know
the
minimum
consistency,
we
wrote
them
to
one
region
and
then
we
read
back
those
million
rights
from
the
other
region
and
got
no
data
loss
and
measured
the
latency
and
made
sure
that
everything
worked.
So
this
was
kind
of
the
validation
we
did
to
basically
prove
to
ourselves
that
we
could
replicate.
You
know,
write
intensive
data
stores
from
east
to
west,
so
I
have
some
some
diagrams
that
show
what
happened
when
we
did
this
and
outside
there's
a
booth
for
a
boundary.
B
We
use
the
boundary
tool
to
monitor
this
at
the
network
level.
So
what
boundary
does
is
it's
monitoring
the
network
flows,
so
it
can
see
separately
the
flow
that's
going
to
clients
the
flow
that's
going
to
cross
region
and
the
flow
that's
going
across
zone.
We
aggregated
all
our
96
nodes
into
one
set
of
flows
that
we're
monitoring
here.
So
this
is
the
this
shows
us
what
happened
when
we
were
copying
the
18
terabytes
of
data,
so
it
ran
at
9.3
gigabytes
per
second
gigabits
per
second
into
region.
B
So
this
is
basically
the
data
being
sent.
Some
of
the
notes
had
more
data
than
others.
So
some
of
them
finished
here
and
then
there
were
a
few
more
that
finished.
So
you
can
see
the
overall
shape
here.
We
didn't
have
an
evenly
balanced
cluster
in
terms
of
the
num,
the
the
load
on
each
of
them
there's
more
data
on
some
than
others
so
went
through
copied
the
data
over.
I
was
a
little
disappointed.
B
This
system
has
480
gigabits
of
bisection
or
bandwidth,
and
I
really
thought
we'd
break
the
internet
with
this
one,
but
it
only
ran
at
eight
nine
gigabits,
so
yeah
that's
reasonable,
but
you
know
some
people
thought
we
wouldn't
even
be
able
to
do
that,
so
that
worked
pretty
well,
I
think
what
happens
here
is
that
the
bootstrap
operation
is
single
threaded,
so
we
effect
had
48
threads
of
traffic
going
across
with
ssds
at
either
end.
It
really
wasn't
disk
bound.
B
So
next
thing
is
the
the
load
where
we're
basically
saying
this
is
how
much
traffic
we
need.
We
want.
It
turns
out
that
meant
three
or
four
hundred
you
know
340
megabytes
per
second
of
continuous
traffic,
going
back
and
forth.
There's
another
measure
here,
which
is
the
boundary
measures,
the
latency,
the
tcp
round-trip
latency,
and
that
was
83
milliseconds.
So
what
we
have
now
is
in
order
to
sustain
our
this
is,
I
think,
a
5x
traffic
current
traffic
level
we're
sustaining
a
few
hundred
megabits
of
traffic
at
83
milliseconds.
B
While
running
this,
we
killed
nodes,
we
broke
it,
we
ran
repairs,
we
did
all
the
things
that
we'd
have
to
do
to
maintain
it
and
everything
actually
kept
working.
So
this
is
kind
of
cassandra
being
a
little
bit.
Indestructible
again
tried
to
break
it
and
it
worked
so
that's
kind
of
nice
and
then
it's
like
well,
you
have
to
ramp
it
up
until
it
breaks,
so
we
just
kept
going
and
we
ended
up.
B
I
think,
16
times
our
current
traffic,
which
had
spare
cpu
spare
iops
spare
network,
but
I
think
we
ran
out
of
some
buffer
in
the
hinted
handoff
area
or
something
like
that.
So
we
need
to
do
a
bit
of
tuning,
but,
as
you
can
see,
we're
running
almost
two
gigabits
of
traffic
continuously
again,
it
was
running
at
83,
milliseconds
and
didn't
seem
to
have
any
problems.
So
well
this
this
was
all
done.
B
It
was
a
pretty
big,
expensive
system
to
do,
but
this
is
a
confidence
builder
for
us
that
we
can
actually
do
right,
intensive
deploys
and
we
don't
have
to
invent
another
mechanism
for
doing
cross
cross
system
replica.
You
know
cross
region
replication,
but
at
this
point
we've
really
driven
it
a
bit
beyond
what
it
could
do
if
we
need
to
when
we
get
to
this
level,
we'll
do
more
tuning
and
we'll
have
more
nodes
or
something
okay.
B
So
this
is
what
it
looks
like
once.
We've
got
that
we
have
our
regional
load
balancers
on
either
side.
We
have
our
customer
and
our
customer
needs
to
connect
to
them
and
we're
using
dns
to
route
customers
to
either
place
and
dns
has
options
for
geographic
routing
if
you're,
using
ultra
or
dyne
aws
doesn't
currently
have
geographic
routing,
so
we're
using
those
two
right
now:
we've
those
are
three
radically
different:
apis
and
they're,
fairly
clunky
bad
apis.
B
In
some
cases,
so
we've
built
an
open
source
toolkit
called
denominator
and
if
you're
doing
anything
to
do
with
dns
management,
you
should
be
using
denominator.
It's
a.
It
was
written
by
adrian
cole,
who
wrote
j
clouds,
he's
working
one
of
the
talking
about
engineers,
we've
hired
because
of
our
open
source.
He
joined
us
in
december,
so
he's
been
building
this
dns
abstraction
layer
in
java
with
lots
of
features
and
we're
using
that
to
dynamically
switch
traffic
from
one
region
to
another.
B
Now,
if
anything
goes
wrong
with
a
zone,
we
just
keep
working.
If
anything
goes
wrong
with
a
region,
we
use
a
denominator
to
switch
all
those
endpoints
to
the
other
region.
If
anything
goes
wrong
with
a
load
balancer,
we
can
switch
the
traffic
to
the
other
region
and
if
anything
goes
wrong
with
one
of
our
dns
vendors,
we
can
use
denominator
to
switch.
The
traffic
switch
all
of
that
state
to
a
different
dns
vendor.
B
It
has
an
independent
view
of
what
the
state
that's
being
stored
in
dns
is
it
has
its
own
model,
because
these
these
apis
are
so
broken
that
you
can't
read
back
what
you
wrote
to
them,
and
things
like
that.
You
know.
So
you
have
to
keep
something
outside
that,
so
this
is
being
wrapped
up
in
a
a
distributed
service
again
with
cassandra
in
the
back
end,
so
that
even
if
a
region
is
down
the
state
that
was
stored
in
that
region
is
available
globally.
So
that's.
B
So
how
does
this
all
fit
together?
I've
got
a
few
minutes
to
talk
about
our
netflix
oss
platform.
In
total,
we
got
so
confused
trying
to
keep
track
of
all
the
names
of
these
things
that
we
built
a
an
example
application,
which
is
an
rss
reader,
and
you
can
see
all
the
different
components
here.
So
it's
got
a
front
end
using
a
this
is
a
base
server
there's
some
logging.
This
is
the
service
registry
for
talking
to
our
service.
B
There's
a
and
then
we
use
servo
and
aston
x,
servo
for
sending
metrics
to
aws
and
aston
x
for
talking
to
cassandra
so
ties
it
all
together,
a
little
bit
more
there's
a
an
an
even
more
sophisticated
example
application.
I
think
it's
at
fluxcapacitor.com,
but
look
for
chris
fragley
happened
to
own
flux,
capacitor
domain
and
decided
to
just
put
it
up
on
that,
not
sure
quite
what
it
does,
but
something
interesting.
B
So
if
we
look
at
all
the
different
projects,
we've
got
there's
basically
github
full
of
source
code,
maven
central
with
the
binaries
in
it,
and
we're
working
on
getting
some
base
amis
out.
This
is
currently
the
thing
that
is
really
unfortunate.
We
because
we're
in
this
transition
from
centos
to
ubuntu
internally
we've
got
a
lot
of
stuff
in
flight,
we're
trying
to
get
to
ubuntu
so
that
we
can
have
base
amis
based
on
the
new
new
code
rather
than
re,
implementing
it
on
centos
and
then
moving
on
to
yeah
chain.
B
Switching
out
all
that
data,
I'm
sorry
the
reformatting
didn't
work
quite
well,
but
we
use
cloudbees,
which
is
a
cloud-based
service
which
runs
jenkins.
We
use
that
to
run
our
own
bakery.
Well,
they
run
the
bakery
yeah
and
that
calls
dinosaur.
Well,
it
can
use
dinoslave
to
do
build
slaves
and
those
bake
amis
from
the
code.
So
that
gives
us
a
soda
baked
amis.
We
can
then
use
asgard,
which
is
our
console
to
deploy
those
to
an
amazon
account.
B
B
B
You
then
want
a
registry.
You
want
zookeeper
running.
We
typically
have
zookeeper
within
region,
but
not
cross
region
edda,
which
is
a
history
service,
simian
army,
all
of
those
all
those
monkeys
running
around,
and
we
have
a
traffic
manager
and
this
one's
going
to
be
released
later
this
week,
which
basically
we
use
for
routing
and
processing
the
incoming
traffic
streams
as
they
find
their
way
to
applications.
B
B
Then,
if
to
build
those
applications,
there
are
a
whole
set
of
libraries,
we've
produced
so
there's
governator
for
dependency,
injection
archives,
configuration
properties
and
eureka
for
service
registry.
So
that's
the
initialization
phase
of
an
application.
Then,
in
order
to
service
requests,
you
need
a
base
server.
We
have
a
reactive
pattern
based
on
java
rx
java
histrix,
which
is
the
circuit
break
up
library
and
turbine,
which
gives
us
one
second
monitoring
dashboard.
B
So
you
can
see
whether
your
circuit
breakers
have
flipped
ribbon
is
the
rest
client
for
outbound
calls,
which
also
understands
zones
and
zone
awareness
and
routing
traffic
correctly,
then
for
accessing
dataweb
astronax.
Obviously,
ev
cache
is
from
memcached.
Curator
is
a
zookeeper
pattern.
Library
zookeeper
has
recently
decided
they
like
this
enough
that
they're
sponsoring
it
as
a
apache
incubator
project,
so
curator's
the
first
piece
of
netflix
oss,
which
is
now
moving
up
to
apache
incubator,
and
it
will
eventually,
I
think,
become
part
of
zookeeper
itself
and
then
denominator.
I
mentioned
for
dns
routing.
B
B
So
one
of
the
more
recent
things
we
did
is
explorers.
We
have
internally,
we
have
a
cassandra
explorer
base
server,
explorer
and
a
whole
bunch
of
other
things.
This
is
a
d3
javascript
library,
it's
quite
a
powerful
system.
Just
for
building
libraries.
We
had
this
diagram,
we
showed
it
last
year.
A
lot
of
people
are
interested
in
it.
This
is
one
way
we
visualize
all
our
cassandra
clusters.
Unfortunately
open
sourcing,
it
we
cleaned
up
a
bunch
of
things,
and
this
is
currently
in
the
middle
of
being
completely
rewritten
by
iran.
B
So
you
have
to
kind
of
pigeonhole
him
over
when
he's
going
to
have
it
finished.
So
this
isn't
currently
out
but
it'll
be
out
as
an
update
to
the
explorers
library
and
with
these
you
mouse
over
it,
you
can
see
all
the
different
clusters
and,
what's
going
on
this-
is
the
amazon
usage
tool.
Cost
cost
reservation
aware
cost
monitoring.
Basically,
this
will
be
out
later
this
month,
so
what's
coming.
B
Next
is
more
portability,
better
availability,
making
it
easier
to
deploy
contributions
from
end
users,
particularly
price
winning
and
prize
entries,
and
contributions
from
vendors
like
eucalyptus
who've
already
got
this
ported.
There
are
people
working
on
this
for
cloud
stack
and
openstack
as
well,
and
even
potentially
the
google
cloud
people
are
poking
around
trying
to
figure
out
whether
they
can
run
this.
B
So
this
is
where
we
are:
we've
got
functionality
and
scale.
Now
we
have
portability
coming
moving
from
a
bunch
of
loose
parts
to
an
ohio
platform,
we're
building
an
ecosystem
sort
of
a
health
warning.
This
is
evolving
very
rapidly,
you're,
jumping
on
a
moving
train
here,
we're
iterating
very
quickly
with
all
this
ourselves
and
we're
trying
to
make
it
easier
for
everyone
to
become
cloud
native.
So
that's
that's
the
takeaway,
and
if
anyone
wants
more
details,
there's
a
bunch
of
links
here.
B
These
slides
will
be
up,
but
we
have
lightning
talks
from
our
open
days
that
we've
had
so
far
earlier
this
year.
There's
a
security
architecture
talk
and
if
you're
interested
in
the
cost,
analysis
and
optimization
there's
a
joint
talk.
I
did
with
janesh
varia
to
go.
Look
for
okay,
so
add
a
time
I
think
just
about
on
time
by
my
watch
anyway.
So
that's
it
thanks
very
much.