►
From YouTube: This Week In Cassandra 6/3/2016
Description
Link to blog: http://bit.ly/24lwZ03
A
Alright,
it
looks
like
we
are:
live
hello,
humans,
welcome
to
another
edition
of
this
week
in
Cassandra,
and
this
week
we
are
joined
as
usual
by
Technical
Evangelist,
John
Haddad
and
we
hide
own,
and
then
we
are
also
fortunate
to
have
Alto
be
former
technical
evangelists
used
to
be
on
the
evangelist
team
here
at
datastax
now
works
for
netflix
and
we're
super
excited
about
that.
Al
is
definitely
an
expert
when
it
comes
to
ops
and
Cassandra
and
I
like
to
say
that
he's
probably
forgotten
more
about
linux
than
I
ever
knew.
A
So
this
should
be
a
good
time.
So,
if
you
are
following
along
on
the
Cassandra
planet,
Cassandra
blog
post
we're
going
to
talk
about
a
couple
of
a
couple
of
those
posts
a
little
bit
here
and
the
first
one
we
kind
of
wanted
to
discuss
was
this
blog
post
about
securing
client
and
node
communications
in
Cassandra.
So
you
know
communications
from
your
app
to
the
Cassandra
cluster
and
juhi
hi.
You
were
saying
you
really
liked
this.
What
what
was
it
in
particular
that
you
kind
of
thought
was
good
about
this
post?
Oh.
B
B
Fine,
so
I
have
some
some
knowledge
about
security,
but
if,
even
though
you
don't
have
any,
you
can
just
follow,
all
the
step
is
really
really
really
great
and
I
need
to
stress
that
never
go
to
production
with
our
security.
Security
should
be
mandatory
on
your
checklist.
Whenever
you
deploy
Cassandra
into
production,
yeah.
A
C
There's
a
towing
problem
there
right
is
the
the
tooling
for
creating
and
distributing
certificates
to
servers
is,
are
still
kind
of
just
a
growing
industry,
I
guess
but
week,
there's
lemur
from
Netflix
I've
used
recently
that
we
have
internally
for
generating
certificates
and
it's
pretty
easy
to
use
and
it
automatically
deploy
to
Claudia
to
some
and
then
there's
Hoshi
core
vault,
which
does
some
similar
things
in
terms
of
managing
secrets
and
I.
Think
you
can
use
it
for
cert
district
distribution,
I
shouldn't
say
much
more
exciting.
C
D
Of
the
things
that
I'm
I've
always
kind
of
never
really
wrap
my
head
around
was
you
know,
it's
cool,
you
can,
let's
say,
generate
certificates
on
machine
and
you
know
pass
everything
around.
But
when
we
start
talking
about
like
these
certificates
doors,
then
my
mind
starts
to
wonder
like
what
happens,
if
that's
compromised
and
how
easy
is
that's
compromised,
and
how
do
you
secure
that,
like
that's
something
that
I
have
zero
experience
with
and
like
I,
don't
know
it's
one
of
those
like
okay?
Well,
I!
Don't
really
it's
not!
D
B
I
would
say
what
people
should
know
about
security
is
that
security
is
really
a
binary
thing
either
all
your
chain
is
secure
or
it
is
not.
Even
if
you
spend
a
lot
of
time
securing
parts
of
your
chain.
If
you
have
a
single
cut
into
your
chain,
which
is
broken
so
UK
it
is
your
chain
is
open
to
any
kind
of
attack.
That's
the
problem.
Yeah
I
agree.
C
With
that
I
think
the
important
thing,
especially
for
this
post
and
kind
of
Cassandra
security
and
client
security,
is
the
the
old
you
don't
hear
as
much
used
to.
But
it's
an
old
phrase
in
the
security
industry
called
defense
in
depth
and
yeah
sure
your
your
root,
CA,
your
your
self
sign,
CA,
might
get
owned
at
some
point.
C
But
that's
there's
a
lot
of
pieces
to
that
and
it's
complex
and
it
requires
a
lot
of
discipline
and
that's
why
a
lot
of
people
just
say
a
screw
it,
because
that
binary
approach
to
it
is
saying
you
know:
oh
well,
you
know
it's
hard
and
I'm
probably
going
to
get
it
wrong,
so
I
won't
do
anything.
But
the
reality
is,
you
know
like
you're
saying
earlier.
C
Is
you
need
to
start
looking
or
people
need
to
be
encrypting
their
connections
so
that
you
know
if
somebody
does
just
happen,
to
get
a
tap
on
their
network
they're
not
going
to
get
crap
right.
They've
still
got
to
own
the
machine
to
get
to
the
certificates.
They
still
got
to
own
the
ca
to
get
to
this
to
the
root
CA
and
there's
a
there's
a
longer
path
to
where
somebody
gets
arm.
Complete
control
of
your
environment.
A
All
right
so
I'm
kind
of
switching
gears
a
little
bit
and
we
had
a
second
blog
post
as
well
that
we
kind
of
wanted
to
talk
about-
and
this
was
a
super
in-depth
one
from
the
yelp
engineering
team
and
they're
talking
about
how
they
monitor
Cassandra
at
scale
and
some
of
the
sort
of
tools
they
use
and
whatnot.
You
know
the
one
thing
that
I
thought
was
interesting
before
I
kind
of
open
this
up
to
everybody.
C
Thought
it
was
a
pretty
clever
post
I've
looked
at
that
approach
before
of
doing
the
using
the
replication
factor
to
determine
you
know
if
the
if
the
cluster
is
healthy
and
it
was
a
lot
easier
in
the
bad
old
days
of
static.
Token
ranges
because
you
could
just
you
know,
do
a
little
bit
of
work
and
understand
where
how
many
tests
you
need
to
run
with
v
nodes.
It
gets
a
little
more
tricky
arm,
but
you
know
I'm
always
a
huge
profit,
always
in
favor
of
application
level.
C
Testing,
which
says,
run
an
actual
query
and
see
what
it
does.
You
know
there's
the
kind
of
the
black
box
monitoring
off
to
the
side.
You
know
where
we
come
in
over
jmx
or
in
this
case
jolokia,
which
is
another
nice
thing
to
see
out
there
arm.
But
you
know
that
gives
you
kind
of
a
synthetic
view
of
the
system,
whereas
if
you
come
in
through
over
cql,
you
get
kind
of
a
you
know
for
real
what
your
application
is
experiencing.
Yeah.
D
I
definitely
alum
with
you.
Whenever
I
built
a
system,
there
was
always
you
know,
monitoring
effectively
every
level.
It's
you
know
what
we
need
to
know
if
our
disks
are
getting
full
and
then
we
need
to
know
what
Cassandra
is
reporting
for
disk
usage,
but
we
also
need
to
build
things,
especially
if
we
have
api's
we're
exposing.
D
You
know
I
always
build
a
set
of
api's
that
were
used
to
determine
if
the
system
was
healthy,
then
it
was
nice
because
you
could
use
that
at
a
very
high
level
and
it
would
like
grab
real
data,
and
it
was
queries
that
we
know
needed
to
work,
and
so
you
could
execute
those,
but
you
could
also
list.
You
could
also
look
at
a
super
granule
level.
So
what
was
nice
was.
Those
api's
would
give
you
like.
Is
this
thing
working
like
that's
just
a
binary?
D
Yes,
no,
and
if
the
answer
is
no,
you
can
then
use
the
other
statistics
to
figure
out
what's
wrong,
because
a
full
disc
doesn't
especially
with
a
Cassandra
cluster.
Where
it's
like.
You
know
you
can.
You
could
lose
an
entire
rack
in
your
machine,
your
cluster
stuff
up
right.
The
thing
that
you
really
care
about
is:
are
my
users,
screwed,
right
and
I.
Think
that
that's
that's.
Why
I
like
doing
multi-layer
approach
to
operations.
A
B
Because
they
were
mentioning
Rokia
I
think
that
this
tool
is
not
known
enough
by
the
community.
This
is
a
really
nice
tool
because
you
can
expose
all
the
jmx
matrix
all
the
HTTP,
because
let
let
be
honnest
nowaday.
Very
few
people
like
working
with
Brody
amex
people
is
familiar
with.
You
know
a
REST
API
HTTP.
So
it's
a
nice
tool
to
add
in
you
to
box
when
you
are
nuts
yeah.
D
C
Know
and
the
thing
that
really
drives
me
crazy
about
jmx
is
you
know:
I've
worked
around
a
large
variety
of
ops,
people
of
varying
skill
levels
and
dbas,
and
the
thing
is,
is
almost
nobody
who
came
through
an
ops
background
and
is
not
a
Java
engineer,
can
get
their
head
wrapped
around?
How
Tara
around
that
thing?
It's
just!
C
D
Guy
I
actually
had
the
I
finally
got
merged
in
I,
think
it's
Peter.
He
went
into
34
or
36,
but
like
I,
think
before
I
even
worked
at
datastax
I
had
submitted
a
jira
to
get
a
CF
histograms
in
either
JSON
or
Amal
some
sort
of
structured
format,
and
that
finally
happened.
It's
like
I
asked
for
it
like
12,
and
now
it's
like
in
36
like
no
the
component.
C
Of
you
know,
if
you
look
at
dbas
at
almost
every
are
DBMS
that's
been
around
since
the
80s
that
they
all
use
just
their
natural
SQL
client.
You
know,
if
you're
my
skill,
you
can
hit
the
performance
tables
if
you're
an
Oracle,
you
have
the
performance
schema,
just
whichever
database,
it
is
almost
certain.
Almost
everybody
is
going
directly
through
their
sequel
interface
over
the
natural
protocol.
Cassandra
has
to
solve
some
different
problems
in
those
because
you
have
a
large
cluster
of
nodes
and
you
need
to
address
them
individually
for
monitoring.
C
But
that's
I
mean
that's
a
mere
matter
of
bits,
right
arm
it
that
that
I
think
just
leads
down
that
path
of
being
able
to
get
more
administrators
writing
more
tools
for
Cassandra
in
their
own
little
custom
monitoring
tools.
You
know
they
can
write
their
nachos
plugins
or
their
whatever
their
monitoring
system
is,
but
they
can
use
the
language
that
they're
best
at
as
well
yeah
and
that's
the
kind
of
outcomes
that
I
like
to
see
there.
B
C
D
C
D
C
D
A
C
A
C
My
my
exposure
to
Cassandra
is
kind
of
on
the
way,
because,
where
I
work
is
in
the
core
team
and
what
we
do
is
we
own
the
availability
at
the
end
of
the
day
of
the
netflix
service
so
arm
various
people
on
the
team
disagree
with
this,
but
I
kind
of
explain
it
as
the
last
line
of
defense.
Now
it's
not
kind
of
the
traditional
ops
last
line
of
defense,
I
don't
really
log
into
servers
very
often
to
fix
them.
C
What
it
is
is
is
if
one
of
the
engineering
teams
misses
an
alert
for
example,
or
they
don't
have
alerting
in
place
and
their
service
crashes
and
the
netflix
service
goes
offline.
Somebody
has
to
own
that,
at
the
end
of
the
day,
you
know
we're
debit,
we're
all
the
way
down
the
you
know
the
rabbit
hole
of
DevOps,
but
that
there's
still
a
gap.
Occasionally,
where
you
know
say
our
proxy
service
dies
arm
and
it
the
page
falls
through
the
engineers,
don't
hit
it
if
it
escalates
escalate.
C
Somebody
has
to
come
in
and
say
and
make
sure
that
that
incident
gets
managed
through.
So
we
get
the
service
back
online
and
get
our
users
back
watching
Netflix.
So
that's
what
my
team
does.
Is
we
own
the
incident
management
process
and
we
kind
of
catch
all
the
stuff
that
falls
through
the
cracks,
and
then
we
drive
that
through
and
go
back
and
work
with
the
engineering
teams
to
make
sure
that
they
address
it
in
the
future.
A
A
Interesting,
you
mentioned
some
con
testing
earlier.
We
were
chatting,
you
know
like
what
what
kind
of
interesting
stuff
are
you
guys
doing
on
on
your
team
or
inside
of
netflix
I?
Think
a
lot
of
people
like
Netflix
gets
held
up
as
sort
of
the
gold
standard
a
lot
of
times
for
for
DevOps
and
and
also
I
mean
in
the
Cassandra
community.
We
we
reference,
you
guys
a
ton,
because
yours
yeah
proponents,
so
you
know
what
kind
of
interesting
stuff
are
you
guys
doing.
C
Kinda
cool
what
my
team
does
is
you
know
a
lot
of
what
we
do
is
fairly
boring
right
is
incidents
and
responding
to
them
and
kind
of
some
of
the
tooling
around
that,
but
we
one
of
our
roles
is
consultative
through
the
rest
of
the
business
and
we
work
with
other
teams
in
terms
of
make
sure
that
our
availability
is
really
great.
How
do
we
measure,
but
how
do
we
talk
about
it?
C
And
one
of
the
teams
we
work
very
closely
with
is
our
chaos,
ninja
cats,
engineering
team
and
a
lot
of
people
in
the
tech
world
have
heard
about
this,
but
a
lot
of
people
netflix
users
don't
know.
Is
we
move
traffic
out
of
data
centers
all
the
time?
It's
probably
once
a
week
we
have
some
kind
of
chaos
of
exercise
and
I
think
about
once
a
month.
Well,
will
completely
evacuate
one
of
our
data
centers,
so
you
might,
and
users
don't
even
know
if
we've
gotten
so
good
at
it.
C
Now
that
it's
kind
of
a
non-event,
and
so
we
have
to
make
sure
that
we
keep
awareness
up.
You
know,
there's
actually
an
effort
that
we
have
to
make
to
say:
hey,
there's
a
core
exercise
going
on,
because
people
will
walk
by
our
desks
and
be
like
hey
is
something
going
on
in
u.s.
East
one
and
we'll
go
well.
There's
a
cog
exercise,
no
go
oh
okay
and
then
they'll
go
back
on
their
merry
way
because
they'll
just
see
their
stats
will
drop
off
and
they'll
go
oh
crap
and
there's
nothing
happening
in
u.s..
C
East
1,
I'm
not
getting
any
traffic
while
it's
all
in
europe-
and
you
know
us
too,
so
that
that's
kind
of
a
really
fun
thing
that
we
do-
and
you
know
that's
one
of
those
things
that
no
netflix
Jesus
leans
very
heavily
on
Cassandra.
For
is
we
don't
have
to
worry
about
kind
of
these
messes
around.
You
know,
I
moved
my
my
skill
master
from
us
east
one,
the
u.s.
C
D
C
It's
you
know,
it's
got
it.
It's
got
its
cost
right
in
terms
of
you
got
an
engineer
for
it.
You've
got
build
your
systems
around
it,
but
then,
once
you
get
there,
there's
there's
kind
of
a
easy
power
available
to
you.
In
that
you
know
we
look
at
a
call
and
somebody
to
say,
hey,
I,
think
something's
funny
in
you
know
us
East,
one.
Maybe
nems
owns
having
network
problems
and
we'll
go
okay
and
we'll
just
leave
right.
C
It's
you
know
you're
you're
at
the
bar,
and
you
know
you
start
to
see
a
fight
breaking
out.
You
go
I'm
out
of
here
and
that's
what
we
do
you
know
and
that
that's
just
kind
of
a
nice
easy
thing
that
we
get
in
the
clouds
that
you
know
a
lot
of.
If
you
don't
have
a
multi
data
center
design
out
of
the
box,
you
can't
do
yep.
A
So
is
there
anything
now
that
you
actually
work
for
netflix?
Is
there
anything
like?
I
don't
know,
give
us
some
some
insider
thing
that
you've
learned
now
that
you're
like
now
that
you're
inside
the
firewall,
as
opposed
to
like
you
know
before
when
you
were,
you
were
working
with
us
at
datastax
and
I'm
sure
you
worked
with
netflix
tons
when
you
were,
you
know
still
at
datastax,
but
is
anything
like
kind
of
like
surprised,
you
or
anything
cool
that
you
kind
of
yeah
learned.
Let.
A
C
When
I
was
interviewing
and
kind
of
the
thing,
I've
found
it,
this
is
going
to
sound
a
bit
fanboy
ish,
but
is
the
the
netflix
culture?
Deck
is
one
of
my
favorite
things
in
the
whole
world
and
it
was
before
I
started
there
and
the
thing
that
I
really
drilled
them
on
was.
Is
it
real?
Is
it
real?
Is
it
real
because
when
you
read
it,
it
sounds
like
a
fairy
tale
right.
C
You
know
freedom
and
responsibility,
and
you
know
if
you
decide
make
a
technical
decision,
you
just
go
for
it
and
it's
real
and
it's
sometimes
it
feels
like
you
know,
walking
a
tightrope
without
a
net
because
you're
you're
kind
of
like
okay,
so
you
mean
I,
can
just
go.
Do
this
and
go
and
I
can
use
Cassandra
or
whatever
service
I
want
they're
like
yeah,
you
just
go
ahead
and
do
whatever.
C
Meetings
and
design
meetings
and
stuff
like
that,
you
know
a
lot
of
times.
You
know
when
I'm
going
to
put
something
together,
I'll
talk
to
my
team
because
that's
just
being
a
decent
human
being,
which
I
can
pull
off
on
from
time
to
time.
But
you
know
there's
there's
it
there
is
that
freedom,
but
there's
also
the
responsibility,
and
I
think
that's
one
of
the
things
that
insight
being
on
the
inside.
It
just
is
taken
a
while
to
really
wrap
my
head
around.
A
I'm
curious
one
other
random
question
that
I've
seen
that
you
know
like
Netflix
I've,
saw
some
tweets
going
around.
You
know
this
this
past
week,
where
it
looks
like
you
open
source
to
another
part
of
the
dynamite
package.
So
that's
the
thing
that
basically
takes
Redis
and
makes
it
you
know
where
any
sounds
like
to
like
it
takes
any
sort
of
key
value,
store
and
kind
of
makes
it
distributed.
Have
you
had
any
interaction
with
with
that?
You
know
in
your
role
and
kind
of
you
know.
C
I
haven't
run
into
it
a
whole
lot.
It's
mostly
just
been
behind
the
scenes
and
has
it
blown
up
in
inspector,
so
I
only
really
find
out
about
things
if
it
blows
up
in
a
spectacular
way,
and
so
it
mean
it
has
to
be
like
all
of
our
s.
Geez
blew
up
write,
something
like
that
or
ultra
DNS
melts
down
globally
kind
of
thing
arm
that
that's
the
kind
of
stuff
that
bubbles
up
so
I
haven't
heard
a
lot
about
dynamite.
I
hang
out
with
that
team
every
once
in
a
while
arm.
C
C
So
there's
things
like
a
huge
memcache
d
farm,
the
EV
cash
farm
and
that's
used
for
almost
everything
that
you
hit
from
the
client
is
probably
hitting
a
bunch
of
keys
inside
EV
cash,
and
that
is
a
massive
cluster
and
the
other
thing
is
kind
of
the
joke
arm.
Oh,
that's
that's
different
thing,
but
it
gets
used
by
oh
pretty
much.
All
of
our
services
I
was
going
to
go
into
a
thing
about
atlas,
but
our
monitoring
system.
But
we
come
back
to
that.