►
From YouTube: ING: ExploitING Hotel Cassandra
Description
Speaker: Christopher Reedijk, Distributed Applications Engineer, and Gary Stewart, Sr IT Specialist
ING is truly becoming an Engineering Company. In the past 2 years ING is really adopting Cassandra fast and at the same time, ING is also focusing on the Customer Experience using a touch-point architecture based more and more on micro-services. For most API's, Cassandra is a perfect match whilst easing availability challenges by being active-active and having an always-on architecture.
However, at ING we have plenty of relatively small use cases which makes justifying dedicated clusters difficult due to avoiding SAN and hardware choices already made. Therefore a shared cluster is inevitable. We call this KaaS Keyspace as a Service and can be compared to a hotel with tenants. One crucial aspect is that we need to create a sandbox environment for the entire organization as implementing Cassandra is hard and requires experience and un-learning.
Our talk will share our experiences on how we are making the switch to NoSQL accessible for the whole organization. It is far from easy and we will also talk about how we are addressing containment and a cost model for the hotel rooms.
A
Good
afternoon
everybody
welcome
to
our
presentation.
It's
called
exploiting
Hotel
Cassandra.
Before
we
continue
with
the
presentation
itself,
we
get
the
question
a
lot.
How
did
you
came
up
with
the
name?
Exploiting
Hotel
Cassandra,
obviously
because
it
describes
our
presentation,
but
we
just
want
to
give
you
a
little
bit
of
insights
how
we
came
to
this
name.
Our
talk
is
going
to
be
about
a
multi-tenant
solution
within
an
enterprise
like
ing.
A
So
when
we
started
to
explain
the
multi-tenancy
Hotel
came
to
mind
a
lot
also
to
other
people,
so
Hotel
Cassandra
was
born
and
actually
we're
now
presenting
in
California.
So
we
just
heard
a
song
Hotel
California.
You
might
have
recognized
it
and
we
started
to
look
at
the
lyrics
and
the
lyrics
actually
explained
very
nicely
some
journey
that
you
can
go
through
in
life,
but
maybe
also
in
business,
and
in
this
guise
it
was
business.
A
We
actually
could
match
the
lyrics
of
the
song
quite
nicely
to
the
journey
that
we
went
through
at
ING
to
implement
Cassandra,
so
we're
using
the
lyrics
with
throughout
the
presentation
to
explain
how
how
our
journey
is
and
how
we
experienced
cassandra
woody
90
for
those
who
might
not
know
ing.
We
have
a
short
video
to
introduce
ing
which
I
want
to
show.
First.
B
See
possibilities
and
others
are
feeling
the
pressure
we're
starting
out
and
starting
over
making
changes
connecting
it's
quite
a
time
to
be
alive.
Our
customers
are
on
the
move,
taking
small
steps,
dreaming,
big
dreams
that
have
something
they're,
not
thinking
about
us,
they're
thinking
about
possibilities,
we're
not
gonna
change
that
we're
here
to
salute
them
to
help
them
forge
a
better
tomorrow
through
the
choices
they
make
today.
B
C
Just
a
short
recap
of
ing
we
are
based
in
over
40
countries,
so
we're
an
international
player
and
we
have
about
52,000,
plus
employees
were
predominantly
in
the
commercial
banking.
But
if
you
look
into
the
European
market,
we
then
have
more
of
a
challengers
and
retail
banking.
If
you
have
a
look
at
the
Netherlands,
we
are
the
largest
bank
and
we
are
a
full-service
Bank.
C
So
one
of
the
challenges
we
have
and
ideas
that
we
have
to
supply
all
products
and
services
for
customers,
and
that
actually
means
that
we
also
have
a
hundred
and
fifty
DevOps
teams
within
retail
banking
alone.
We
also
have
commercial,
banking
and
group
services,
so
as
an
organization
we're
really
dealing
with
a
lot
of
teams
a
little
bit
about
us.
My
name
is
Gary
Stewart
I'm,
an
engineer
and
an
architect.
I
gave
a
talk
last
year
and
I
announce
that
I
have
a
love-hate
relationship
with
Cassandra.
C
A
So
we
really
felt
like
we
were
on
a
dark
desert
highway.
We
didn't
see
the
end
of
the
roads,
but
we
did
know
that
the
road
around
us
or
the
scenery
around
us
was
changing.
The
customers
were
getting
new
expectations,
different
expectations
from
companies,
not
only
financial
institutions,
all
the
companies
around
us.
Why?
Because
we
are
more
and
more
becoming
a
always
own
environment.
A
Everybody
around
us
expects
everything
to
be
always
on,
so
they
expect
availability
always
and
not
only
availability
during
office
hours,
but
also
availability,
24/7,
we're
no
longer
only
sticking
to
our
own
countries,
we're
also
going
global
as
people.
So
that
means
that
we
also
saw
that
people
were
tripping
a
lot
and
they
also
wanted
to
have
access
to
our
data
when
they
are,
for
instance,
in
California
like
we
are.
A
We
also
want
to
see
what
our
balance
is
from
our
accounts,
even
though
we
are
here
in
a
California,
of
course,
so
24/7
was
becoming
more
and
more
important,
also
for
financial
institutions
and
consistency.
Consistency
was
always
important,
but
there
is
a
new
dynamics
in
there
to
say
it
like
that.
We
are
a
only
channel
bank
and
if
you
want
to
explain
omni-channel,
you
can
explain
it
partially
by
say
we
offer
multiple
channels
to
our
customers
and
what
are
those
multiple
channels?
Well,
I'll
give
you
three
examples
or
actually
for
the
first
one
is
mobile.
A
The
second
one
is
my
ing
banking.
So
that's
internet
banking
and
we
also
have
assisted
channels
called
the
call
centers
and
the
sales
offices,
so
the
branch
offices,
so
they
are
all
channels
to
our
customers
and
they
all
expect
the
same
information
coming
back
from
all
those
channels.
So
there's
no
longer
only
a
consistency
within
the
channel
expected
by
our
customers,
but
also
across
the
channels.
They
hop
the
channels
so
that
pretty
much
means
wellness.
A
They
have
opened
an
account
in
the
store
in
the
brain
surface
and
they
walk
out
and
they
log
on
to
the
mobile
device.
They
barely
had
see
that
account
and
they
better
yet
see
the
balance
and
they
want
to
start
bring
money.
So
that's
very
important.
That's
also
changing
that
was
changing
around
us
as
well
and
in
order
to
make
all
of
this
possible.
So
all
of
those
devices
that
we're
seeing
not
only
mobile
devices
tablets
even
watch
this
nowadays
they
connect
to
our
systems.
A
So
that
means
that
there's
a
lot
of
throughput
coming
on
our
systems
and
also
a
lot
of
demand
coming
on
our
systems
that
at
least
that
we
needed
to
be
more
scalable
than
we
ever
were
before.
We
need
to
be
fast
and
easy
scalable
as
a
bank
within
our
width
for
our
applications
as
well.
So
those
were
the
external
challenges
we
saw
so
the
world
around
us
was
changing,
but
also
the
world
inside
of
ing
was
changing
a
bit
and
what
was
changing,
or
it's
not
really
changing.
A
But
it
was
more
like
a
challenge
that
we
saw
when
we
started
with
a
with
a
technology
like
Cassandra
and
then
we're
looking
at
lots
of
small
use
cases.
Gary
already
introduced
the
one
of
the
50-plus
DevOps
teams
that
we
have
in
domestic
banking
alone.
So
that
means
that
they're
also
working
on
use
cases,
but
there
are
relatively
small
use
cases
if
you
compare
them
to
other
use
cases
for
Cassandra.
A
So
it's
low
TPS
low
data
footprint,
so
they
don't
really
deserve,
or
maybe
that's
the
wrong
word,
but
it
is
not
cost
efficient
to
have
a
cluster
were
small
use
case
in
this
sense,
so
we
needed
to
start
doing
something
about
that
one,
the
other
one
that
was
happening
is
we
were
going
to
son,
so
you
might
have
heard
it
a
couple
of
times
already
today.
I
guess
son
is
sort
of
an
anti-pattern
when
you
start
using
Cassandra.
A
If
you
didn't
hear
it
well,
then
this
is
the
first
time
you
hear
it
don't
use
son,
because
that
is
not
anywhere
recommended
when
you
start
using
Cassandra,
so
we
needed
ephemeral,
storage,
dedicated
storage
on
our
machines.
So
that
was
another
thing
that
was
changing
already
in
the
bank,
which
we
have
to
counter
measure.
In
that
sense,
and
the
last
one
is
most
actually
visible
when
we
started
the
journey
was
there
was
zero
to
none,
no
SQL
experience.
A
So
we
really
needed
to
do
something
about
that
one,
not
only
within
the
bank,
but
also
within
the
Netherlands
itself.
It
was
a
niche
expertise,
so
actually
I
liked.
What
Billy
Bosworth
was
saying
during
the
keynote
this
morning.
He
said:
if
I
look
two
years
back,
it
was
Cassandra
was
well
not
that
known,
but
it's
exponentially
growth
like
the
last
two
years,
and
we
saw
same
thing
happening
in
the
Netherlands.
The
gossamer
meet
up
in
the
Netherlands
had
one
of
the
15
or
something
users
active
users
within
the
Netherlands.
A
C
So
we
really
started
with
a
PLC
I
mean
we
did
a
lot
of
use
cases
and
we
did
testing
and
we
did
all
the
things
that
Patrick
McFadden
just
spoke
about
in
his
talk
previously
and
we
learnt
a
lot.
We
really
learned
a
lot.
We
stretched
our
comfort
zone
and
basically
we
came
to
the
conclusion
that
Cassandra
will
probably
be
the
solution.
Why,
if
you
look
at
Cassandra
compared
to
other
nodes,
eco-project
products
you
have
secured
CQ
well
is
familiar.
C
So
it
brings
that
familiarity
back
to
the
developers
they're
very
familiar
with
SQL,
so
they
don't
have
to
learn
a
new
language,
but
I
think
the
other
differentiating
factor
for
Cassandra
is
its
masterless
architecture.
When
you
have
a
massless
architecture.
In
this
start,
it's
actually
very
difficult,
and
by
being
sort
of
spending
the
time
on
the
attention
to
detail
at
that,
you
know
early
on,
you
get
the
benefits
later,
but
I
think
the
killer
feature
is
active,
active
over
data
centers
Chris
mentioned
earlier.
C
We
want
to
be
always
on
architecture,
so
we
can't
do
upgrades
without
having
an
active,
active
architecture.
So
we
started
realizing.
Cassandra
is
the
way
to
go
and
we
did
a
really
intensive
twofer
concept.
I
think
it
was
about
four
months,
I
think
we
walked
into
every
wall.
We
because
it
was
a
critical
process.
We
paid
attention
to
the
details
and
it
was
through
that
process
that
we
really
learned
how
to
use
the
sander
and
what
it
actually
means,
but
they
brought
us
into
a
sort
of
feeling
of.
C
Are
we
ing
ready
for
it
because
remember
we're
a
small
team
and
we've
got
a
large
organization
behind
us.
We've
had
the
period
to
embrace
and
adopt
the
technology,
but
everyone
else
needs
to
have
the
time
to
do
that,
because
it's
a
different
way
of
thinking.
Fortunately,
something
else
is
happening
in
parallel
and.
C
The
was
something
less
so
something
else
is
happening
in
parallel
and
we
realized
that
this
movement
restful
api
and
cassandra
fit
in
quite
nicely,
because
when
you
have
a
look
at
api
is
in
restful
services,
you
tend
to
have
a
resource,
and
in
this
example
we
have
a
customer
resource
and
you
tend
to
have
an
ID,
and
that
will
be
your
partition
key.
So
customer
could
be
your
table.
Your
ID
could
be
your
petition
key,
and
then
you
can
have
your
data
so
for
us
we
started
realizing.
C
Api
is
also
forced
you
to
think
inside
out
top
down.
So
you
have
to
know
your
queries
first
and
if
you
know
no
sequel
and
cassandra,
you
really
need
to
know
your
queries.
Not
your
entities,
not
your
relationships.
You
need
to
think
top-down.
So
with
this
movement
that
got
us,
how
can
you
put
it
an
accidental
success
that
is
happening
at
the
same
time?
The
opposite
is
not
true.
C
We
might
be
made
of
money,
but
we
also
have
to
be
careful
so
the
other
paradigm
shifters
that
we're
comfortable
with
sharing
in
the
hardware
level,
so
we're
all
going
to
cloud
or
going
to
San.
But
now
we're
asking
people
to
share
on
the
software
level,
and
this
is
again
something
that
people
get
nervous.
So
the
question
is:
do
we
dare
to
share?
And
we
do.
A
We
strongly
believe
we
do,
but
in
order
to
be
that
certain
of
our
case
we
add
the
visualization
session
with
the
team
and
within
the
the
output
of
the
visualization
session,
is
the
picture
that
you're
showing
here
and
it's
actually
sort
of
a
touch
point
now
the
architecture.
But
it's
a
touch
point
picture
of
what
our
vision
was
and
what
our
touch
points
were
that
we
needed
to
work
around
or
work
at
to
make
sure
that
we
get
the
customer
experience
that
we
want
from
a
key
space
as
a
service
solution
that
we
are
offering.
A
So
we
are
offering
a
luxury
hotel
to
our
customers
and
they
have
expectations.
They
have
high
expectations
of
this.
We
already
mentioned
in
the
beginning
of
this
presentation
that
the
experience
and
the
knowledge
on
Cassandra
was
very
niche
in
ing
and
then
and
in
the
Netherlands
itself.
So
training
is
key,
but
we
actually
it
was
sort
of
an
accidental
success.
We
also
learned
a
lot
from
giving
a
training
ourselves,
so
we
started
getting
the
training.
A
What
how
can
I
implement
my
use
case
and
I
really
liked
the
fact
that
we
were
actually
developers
as
well,
not
just
trainers
that
we
also
had
our
foot
in
the
mud
just
to
say
it
like
that,
not
sure
if
it's
a
proper
English
expression,
but
it
is
Dutch,
so
we
learned
a
lot
from
giving
a
training
and
the
feedback
moment
was
already
started
from
the
training
on
next
to
that
we
are
a
big
company.
We
are
a
corporate,
we
are
an
enterprise,
so
that
means
we
have
a
lot
of
procedures.
A
We
have
paperwork
that
we
need
to
go
through
before
we
can
put
something
in
production
right,
I.
Think
most
of
you
might
recognize
that
next
that
we're
also
a
financial
institution.
So
we
have
compliancy
rules
which
add
some
more
documentation
and
procedures,
so
that
is
tedious
work,
especially
for
developers
developers
like
the
code
they
like
to
build.
They
like
to
design
they
like
to
make
stuff
and
make
it
happen,
and
they
don't
want
to
spend
much
time
on
the
procedures
and
the
documentation
normally.
So
we
wanted
to
take
that
big
chunk
of
tedious
work.
A
That's
paperwork
away
that
there's
a
promise
that
we
gave
from
the
get-go.
If
you
take
a
key
space
from
us,
we
will
reduce
your
paperwork.
Oh,
that
could
be
great
right.
So
that's
what
we
did
and
in
order
to
do
that,
we
needed
to
take
big
chunk
of
their
responsibilities
away
or
the
ownership
way,
and
that
is
the
unexpected
load
and
the
infrastructure.
Those
come
together
very
10
intensity.
So
we
will
surface
the
infrastructure.
We
will
make
sure
it
is
there.
A
We
will
make
sure
that
the
strongest
rectus
on
our
stack
is
there
and
when
you
need
to
grow,
it
will
grow
automatically
for
you.
We
will
make
sure
it
happens.
So
that's
all
very
a
lot
of
paperwork
less.
For
them.
They
only
needed
to
do
the
little
paper
that
is
needed
for
the
key
space
itself
and
their
application.
What
she
already
needed
to
do,
anyways,
next,
that
we
also
had
something
called
functionality.
Well,
we
know
the
data
sex
driver.
It
got
introduced
a
couple
of
times
already
today
and
we
build
a
thin
layer
around
it.
A
Why?
Because
of
those
compliance
rules
again,
we
don't
want
every
team
to
build
those
compliancy
rules
over
and
over
and
over
again
for
connecting
to
Cassandra.
So
we
build
a
thin
layer
around
it
just
to
supply
this
to
them
and
make
sure
that
they
could
have
could
use
it
so
Wednesday
when
they
start
using
this
in
their
applications.
They
also
got
something
which
we
call
covenants,
so
the
driver
governance
and
it's
pretty
much
based
on
metrics,
but
Gary
will
touch
it
later
on
in
the
presentation,
and
that
gives
us
lifecycle
management.
A
C
So
we
basically
came
up
with
a
sandbox
environment
and
by
having
a
sandbox
environment,
active
active
our
data
centers
available
for
teams
to
test
their
use
cases
directly,
because
if
you
realize
it's
a
distributed
system,
so
you
can
do
your
unit
test.
You
can
test
everything
on
a
single
instance:
it'll
work,
it
almost
always
works,
but
it's
when
you
go
active
active
over
data
centers
and
then
you
start
testing
your
actual
use
case
in
the
behavior.
Then
you
start
learning,
and
that
was
what
we
needed
to
do
for
the
company.
C
So
by
having
a
sandbox,
we
enabled
teams
to
fail
fast
or
succeed
fast,
and
it's
that
real
sort
of
mentality
that
will.
How
can
you
put
it,
enable
a
smoother
feedback
cycle
because
remember
is
about
sharing
in
a
company
as
well.
We
don't
always
have
all
the
answers
and
we
don't
know
all
the
use
cases
that
are
there.
So
if
you
create
a
sandbox
active
active
with
your
infrastructure,
then
you
enable
teams
to
well
test
themselves
and
then
also
maybe
teach
you,
but
the
other
challenge
we
gave
was,
we
said
to
them.
C
A
Here
definitely
and
it
and
they
did,
that
is
a
very
successful
communication
from
our
sides.
So
we
learned
a
lot
on
the
sandbox
environment
already
we
saw
that
there
can
be
a
lot
of
containment
issues
and
we
definitely
didn't
want
to
go
that
one
on
to
the
production
environments
as
well.
So
we
wanted
to
go
and
see.
How
can
we
fix
these
containment
issues?
Can
we
fix
them
either
way?
So
that's
the
question.
Actually
we
know
we
can
do
it
with
the
hardware.
We
already
mentioned
it
before.
A
That's
what
a
lot
of
people
are
doing
or
a
lot
of
companies
are
doing,
but
that
was
not
cost-effective
for
us
as
enterprise
as
the
since
we
have
those
small
use
cases
that
I
was
talking
about
at
the
beginning
of
this
presentation.
So
hardware
was
not
the
option
for
us.
Then
we
looked
at
Cassandra
can
Cassandra
offer
containment
on
key
space
level
it
can
on
Cassandra
level
on
JVM
level
and
process
level.
A
It
can
definitely
give
you
containment,
but
on
the
key
space
level,
Cassandra
doesn't
give
you
containment
yet
might
in
the
future,
but
at
this
moment
of
time
on
key
space
level,
there's
no
containment,
so
we
needed
to
go
look
further
and
then
we
looked
into
new
nickel
other
technologies
like
next
to
Cassandra,
then
we're
thinking
about
container
like
technologies.
We
know
doctor,
we
know
measures
and
they
look
very
promising,
but
we
didn't
have
that
skill
in
house
yet
so
we
couldn't
actually
use
that
at
that
moment
of
time
to
make
it
work
for
ing.
A
At
that
moment,
we
are
still
looking
into
it
and
we
are
planning
to
start
incorporating
it
more
and
more
for
our
containment.
But
for
now
we
had
to
stick
to
what
we
have
and
with
that
is
we
have.
We
have
actually
got
our
software
and
we
have
our
teams,
our
people,
so
that
combination
that
strong
combination
is
needed
to
get
proper
containment
on
your
key
spaces.
But
in
order
to
make
sure
that
everybody
feels
the
ownership
you
need
to
make
some
stuff
visible
to
them.
You
need
to
give
them
insight
on
what
is
actually
happening.
A
So
we
adopted
a
spider
graph
and
a
spider
graph
is
pretty
much
what
you're
seeing
here.
So
we
defined
a
couple
of
points
on
the
XS
which
we
think
is
important
when
you
start
using
a
distributed
system
or
when
you
start
using
Cassandra.
So
some
examples
are
consistency,
read
part
right,
but
it
says
right:
heavy
should
read
heavy,
so
we
started
plastified
putting
patterns
on
our
clusters
based
on
what
we
want
to
do
with
the
cluster
itself.
A
This
is
a
guess
use
case,
so
that
is
very
important
to
have
consistency.
Wise
facilities
are
important
because
it's
a
cash
layer,
so
when
we
store
something
in
the
vacants,
the
system
of
records
it
better
yet
also
be
available
in
the
cash
layer
as
well.
So
consistency
is
very
important
for
this
use
case
as
firmly
getting
avoids
onwards.
A
Also,
the
the
the
the
the
true
sorry,
the
full
true.
So
we
need
to
be
sure
that
it
can
handle
fault
easily,
because
we
have
the
most
containment
issues
going
on
on
these
kind
of
clusters,
because
we
have
a
lot
of
case
use
cases
within
ing
itself.
So
those
are
just
some
examples
of
boys
that
we
can
stress
on
to
see
if
we
can
match
some
clusters
and
some
use
cases
on
top
of
each
other.
Just.
C
To
add
to
that,
the
point
is
when
you
are
on
the
zero
dennis
cheap
or
it's
less
complex
and
as
you
go
to
the
outside
of
the
spider
graph,
it
becomes
more
complex,
more
expensive
and
more
challenging.
So
this
is
the
way
that
we
were
bringing
it
to
the
teams
to
understand
how
complex
or
simple
each
of
the
aspects
were,
and
in
contrast
to
this
example,
we
have
another
example
and
this
one
is,
for
example,
a
for
detection
use
case.
So
what
is
this
particular
footprint
of
that
one?
C
Is
we
need
to
save
all
the
events
so,
every
time
the
customer
is
doing
something
we'll
save
all
that
information
and
when
we
suspect
something
going
wrong
or
an
incident,
then
we
need
to
read
so
you
can
imagine
very
heavy
right
and
very,
very
few
weeds.
So
that's
why
you
can
see
we
needed
to
be
tuned
for
the
rights
and
the
weeds
are
not
that
important.
C
C
We
started
a
project
called
the
Hubble
project
and
basically,
what
the
Hubble
project
is.
It
uses
the
confluence
API
if
you're,
not
the
middle
of
confluence
confluence
is
a
collaborative
documentation
platform.
Why
we
like
confluence
is
because
we
already
had
it
and
it's
integrated
into
LDAP,
so
we
have
all
the
security
and
access
on
it,
but
there's
two
really
nice
features
and
the
one
feature
is
versioning.
So
as
you
edit
your
page,
you
can
actually
see
in
time
what
the
previous
versions
were.
C
If
you
want-
and
you
can
compare
different
versions-
and
the
second
feature
is
notifications.
So
if
you
make
an
edit
on
a
page,
then
anyone
watching
that
page
can
get
a
notification
of
that
change.
So
if
you
use
confluence
and
then
you
say,
generate
your
data
models,
then
anyone
interested
in
that
schema
will
get
a
notification.
You
also
then
get
versioning
of
your
data
model
over
time.
So
we
wrote
the
program
in
Scala
and
we
basically
use
the
data
stacks
drivers
and
they
have
a
class
called
the
metadata.
C
And
if
you
use
the
metadata,
then
you
pretty
much
get
all
the
schema
information,
all
the
configuration
and
then
you
can
actually
use
that
to
say,
generate
your
documentation,
we'll
show
you
a
couple
of
screenshots
on
what
that
looks
like
so
now
you
can
see
a
sort
of
loop
where
developer
deploy
is
a
model
into
the
cluster
and
then
we
will
generate
documentation.
But
this
wasn't
good
enough
for
us.
We
needed
to
take
it
one
step
further
and
that
was
starting
to
add
operational
side
of
stuff.
C
So
if
you
take
OP
Center,
so
that's
a
tool
from
data
stacks
where
you
get
all
your
metrics
and
information
nice
thing,
but
off
center.
This
also
has
an
API.
So
all
the
information
that
you
see
there
is
exposed
to
api's.
If
you
configure
your
clusters
to
use
proper
naming
conventions-
and
you
push
all
your
metrics
to
graphite,
then
you
can
use
a
graphite
api's
to
extract
your
information.
But
this
is
only
half
the
story
because
we
don't
say
per
se
care
about
the
cluster.
C
In
the
end,
our
customers
are
the
application,
so
we
needed
to
know
how
the
applications
are
responding
to
or
we,
how
is
their
customer
experience
so
the
earlier
on
Chris
mentioned
about
the
DataFlex
driver
in
the
thin
layer
we
put
around
it.
That's
one
of
the
examples
is
naming
conventions
for
the
metrics
that
the
application
should
push
out.
Why?
Because
you
don't
want
to
overwrite
your
metrics
and
you
do
want
to
find
all
the
applications
that
are
running
on
that
cluster.
C
So
if
you
follow
good
naming
conventions,
then
you'll
be
able
to
use
the
API
to
find
your
information.
So
now
we
can
actually
use
the
graphite.
We've
got
off
center
and
we're
able
to
generate
interesting
web
pages
for
the
dev
ops
teams,
but
we're
a
bank.
So
we
have
one
extra
problem.
We
need
security
sort
of
again
a
bit
of
an
accident.
We
realized
that
we
are
almost
halfway
there.
C
So
if
we
create
a
continuous
delivery
flow,
where
we
get
teams
to
put
their
model
in
to
get
and
then
we
deploy
it
via
Jenkins,
then
the
dev
ops
teams
don't
need
access
to
the
cluster,
but
because
we
have
the
Hubble
project
and
because
we
generate
in
the
confluence
pages
and
because
they
get
notified
of
the
changes,
we've
completed
the
feedback
loop.
So
they
don't
feel
that
they're
blind
on
what's
happening
in
the
cluster
they're,
getting
the
information
they
need,
but
they
don't
need
to
be
in
the
cluster.
C
A
They
also
need
to
know
what
the
key
space
is
doing
across
all
those
environments
and
preferably
in
one
view,
so
we
are
in
this
sense
giving
them
the
overview
of
what
is
happening
with
the
key
space
across
all
those
environments.
Green
in
this
picture
means
there
is
a
key
space.
Your
key
space
is
in
that
environment
and
red
means
it's
not
there
pretty
much
so
the
other
one
we
have.
A
There
is
the
number
of
tables,
and
that's
there
just
to
show
you
if
there
is
any
difference
in
the
model
based
on
the
tables
that
is
there,
it
doesn't
necessarily
mean
if
you
have
a
different
amount
of
tables
per
environment
that
it's
wrong.
It
can
be
right
for
your
specific
use
case,
of
course,
but
it
can
already.
If
you
don't
expect
it
to
be
different.
You
get
all
the
insights
based
on
this
specific
page
that
we
are
offering
to
you.
A
So
let
us
the
key
spaces,
but
the
other
one
that
is
very
important
in
a
distributed
system
are
the
notes
themselves
and
the
cassandra'
llamo
is
most
likely
the
most
important
configuration
file.
There
is
in
Cassandra
and
that's
also
a
way
you
can
do
the
most
tuning
on
the
Cassandra
specific
level.
Let's
say
it
like
that,
so
we
want
to
show
them
the
actual
Cassandra
yamo
configuration
per
note.
A
This
is
your
the
suggest,
a
snippet,
of
course,
because
it's
a
much
larger
file
normally,
but
it
can
only
show
if
there
is
a
difference
in
configuration
per
note
per
environment,
so
that's
already
very
interesting
to
give
back
to
the
operators
and
developers
themselves
without
having
to
log
on
to
the
system
because
they
are
not
allowed
to.
So
that's
why
we
need
this.
We
need
to
show
this
to
them
so
that
we
can
have
zero
touch
on
our
systems
below
them.
That's
the
whole
idea
behind
it.
A
So
this
is
just
the
some
some
tuning
settings,
but
they
can
also
give
us
feedback
on
what
they're
seeing
so
if
they
think
we
tune
the
cluster
incorrectly,
we
can
change
it
and
most
likely
it
will
be
beneficial
for
all
the
other
key
spaces
over
the
cluster
as
well.
So
it's
really
going
to
distribute
its
support.
We
are
enabling
distributed
support
just
by
having
these
pages
as
well.
We
believe
so.
A
You
want
to
give
this
insight
because
it
might
be
that
your
data
model
is
using
the
wrong
petitions
and
in
this
specific
example,
we
see
that
the
min
size
of
a
note
in
the
max
size
of
a
note
have
a
relatively
large
difference
between
them,
so
that
could
mean
that
you're
using
a
wrong
partition
key
or
a
wrong
clustering
key
in
your
data
model,
so
that
is
already
giving
back
the
feedback.
Okay,
your
distribution
of
petitions
for
your
specific
key
space
is
not
really
working
out,
so
maybe
we
should
do
something
about
it.
A
Maybe
we
can
help
you
and
change
your
data
model
or
tune
your
data
model,
so
that
will
be
better
in
the
future.
So
this
is
also
giving
insights
for
both
the
DevOps
teams
and
also
us
itself
coming
back
to
those
cluster
patterns
that
we
touched
upon
before.
I
want
to
go
very
quickly
to
this
slide,
but
gary
already
mentioned
we
can
do
software
configuration
based
on
those
patterns,
but
we
can
also
do
hardware
configurations
and
for
this
one
we
already
explained
to
the
left.
A
One
and
the
only
thing
I
want
to
say
about
that
one
is
we
are
using
SSDs.
This
is
a
cluster
which
can
have
typical
containment
issues
and
last
year's
summit
we
actually
heard
during
a
training.
You
cannot
badly
configure
an
SSD
cluster,
so
go
Viktor.
We
could
use
SSDs
to
prevent
a
lot
of
containment
issues
already
from
the
get-go,
but
also
we
want
to
enable
solar
and
spark,
and
especially
for
solar.
You
cannot
go
with
spinning
disks.
A
You
need
SSDs,
so
we
need
to
be
prepared
for
the
solar
feature
that
we
want
to
enable
on
this
cluster
as
well,
going
back
to
the
fraud
detection,
one,
that's
an
interesting
one
and
we're
going
to
exceed
the
recommended
max
size
of
terabytes
per
node,
we're
going
to
10
to
15
terabytes
per
node,
where
the
next
recommended
is
5,
terabytes
per
nodes.
Why
cost-effective?
A
We
want
to
have
as
little
noise
as
possible,
but
with
the
right
amount
of
data
footprint
that
is
needed
for
the
huge
case
itself
and
we
believe
that
data
at
compaction
can
help.
Here
we
have
a
very,
very
nice
introduction
by
Jonathan
Ellis
this
morning
as
well,
but
we
also
need
to
do
some
tuning
on
our
data
model,
of
course,
so
we
have
to
block
it
off
as
a
stable
per
month,
for
instance,
so
that
we
only
have
compaction
going
on
for
one
active
table
at
a
time.
A
This,
the
middle
one,
is
about
high,
consistent
use
cases.
Let's
actually
need
to
talk
to
you
that
we
did
last
year
so
for
that
one
specifically,
we
introduced
the
third
data
center.
We
need
three
data
centers
for
high
consistent
use
cases.
So
that's
just
some
of
the
things
that
we
can
match
on
those
cluster
patterns
that
we
have
defined
every
9g.
Yes,.
C
And,
of
course,
you
know,
one
of
the
things
that
we
keep
trying
to
get
to
the
teams
is
that
is
a
total
stack
solution.
So
we've
now
been
talking
again
about
the
operations.
We
need
to
bring
the
developer
side
of
it
again.
So
earlier
on,
I
mentioned
that
we
document
the
data
model,
because
what
we
say
to
the
teams
is,
they
have
to
own
their
own
device.
They
know
their
data
models,
we
can
help
them,
but
they
know
their
own
data
model.
They
know
the
petition
size,
they
know
what
data's
going
to
be
there.
C
So
if
you
see
on
this
page
that
we're
generating
basically
was
generating
the
documentation
for
them,
because
one
thing
I
had
a
history
of
is
documenting
data
models
and
then
doing
the
data
model
and
they
get
out
of
sync.
So
we
decided
we're
not
gonna
document
data
models
anymore,
we're
just
gonna,
create
it
and
will
generate
the
documentation.
And
if
you
put
your
comments
in
to
the
secure
wall,
then
it
will
show
up
in
the
page.
C
So
if
you
have
a
look
at
the
queries
that
were
there,
we
also
generate
the
default
queries,
because
one
thing
we
noticed
is
that
teams
tend
to
say
use
other
queries.
So
we
say:
if
you
use
these
default
queries,
then,
generally
speaking,
you're
doing
something
right
and
as
the
song
goes,
they
stab
it
with
their
steely
knives.
So
we
realized
that
teams
are
making
the
same
mistakes
as
we
did
and
we
spend
a
lot
of
time
on
the
simple
mistakes.
C
So
if
we
can
get
rid
of
the
simple
mistakes,
then
we
can
prevent
a
lot
of
rework
so
part
of
the
project.
We
also
look
for
say,
overuse
of
indexes
or
say
incorrect
data
center
names
or
single
column
tables,
so
just
the
sort
of
simple
mistakes.
So
then
we
get
a
feeling.
Do
they
understand
Cassandra,
not
so
the
more
errors
we
have
per
se
team,
the
more
we
realize.
C
Okay,
we
need
to
spend
some
time
with
them
see
if
we
can
get
their
knowledge
up
figure
out
what
we
can
do
to
help
them,
because
we
want
to
spend
time
in
the
difficult
problems
we
need
to
free
up
all
the
time
to
not
do
all
the
simple
mistakes
and
then,
of
course,
we'll
aggregate
it
per
cluster
and
then
we'll
get
a
feeling
on
how
many
say
good
or
bad
tenants.
There
are
on
that
cluster.
Yes,.
A
And
these
are
just
some
features
that
we
showed
in
the
page
form.
We
have
some
more
features
in
there
and
we
also
want
to
have
at
it.
We
want
to
add
some
more
features
to
it
as
well.
Maybe
you
guys
can
help
us
because
it
is
open
source,
but
it
is
the
tracking
of
queries.
So
we
already
do
checks
on
the
data
model
itself,
but
you
can
also
make
a
lot
of
mistakes
in
your
queries
on
the
cluster
or
in
your
key
space.
So,
based
on
the
actual
queries
are
happening
on
your
key
space.
A
We
can
make
best
practices
on
them
as
well.
For
instance,
partition
scans
table
scans
the
basic
mistakes
everyone
makes
when
they
start
using
Cassandra
pretty
much
so
we
can
only
give
back.
Okay,
based
on
the
queries
that
we
see
happening,
you
should
be
changing
some
of
your
queries
because
we
think
you
are
experiencing
slow
queries
or
irregular
queries
based
on
what
you're
doing.
A
Next,
to
that,
we
also
want
to
show
and
visualize
the
actual
use
case
pattern,
so
we
defined
one
at
the
beginning
of
the
project,
but
once
it
is
per
in
production,
we
want
to
see
what
your
actual
use
Kies
pattern
is
and
if
it
still
matches
the
cluster
pattern
that
we
have
to
find
for
this
specific
cluster.
If
it
doesn't
match
you
might
need
to
move
to
another
cluster
or
we
need
to
create
the
whole
new
cluster
for
you
in
tune.
It
specifically
for
those
kind
of
use
cases.