►
Description
Speaker: Javed Roshan, Director of Data Services
As a leader in the financial industry, Capital One applications generate huge amounts of data that require fast and accurate handling, storage and analysis. We are transforming how we report operational data to our internal users so that they can make quick and precise business decisions to serve our customers. As part of this transformation, we are building a new Go-based data processing framework that will enable us to transfer data from multiple data stores (RDBMS, files, etc.) to a single NoSQL database - Cassandra. This new NoSQL store will act as a reporting database that will receive data on a near real-time basis and serve the data through scorecards and reports. We would like to share our experience in defining this fast data platform and the methodologies used to model financial data in Cassandra.
A
A
A
So
data
and
data
analysis
to
drive
a
business
strategy
is
the
founding
principle
on
which
Capital
One
has
been
built.
We
are
one
of
the
first
financial
institutions
to
have
created
the
product
strategy
based
on
analysis
of
the
data
back
in
late
eighties,
early
nineties,
fast
data
and
its
relevance
in
operational
reporting
is
the
use
case.
We
are
solving
the
fast
data
and
operation
Tapani.
What
I
mean
is
driving
intraday
business
decisions,
monitoring
process
quality,
an
example
is
ours.
A
The
time
takes
for
our
staff
to
proof
and
loan
application
end-to-end
right.
That's
one
of
the
ways
of
monitoring
the
process
quality.
So
where
do
we
get
the
data?
You
know
we
have
operational
support
systems
as
the
source
of
the
data.
We
have
business
support
systems
and
then
we
generate
lot
of
data
internally
by
our
marketing
sales
and
servicing
teams,
and
also
we
have
third
party
from
where
we
get
the
data.
So
for
this
use
case
a
simplistic
view
of
the
landscape.
A
Is
we
have
this
multiple
data
sources
that
I
just
mentioned,
and
then
we
perform
traditional
best
ETL
processing
on
this
data
right
and
restore
all
of
this
data
and
to
multiple
data
destinations,
predominantly
our
DBMS
kind
of
as
well,
and
then
we
run
a
slew
of
reporting
tools
on
top.
So
there
are
obvious
opportunity
areas
given
this
current
landscape
and
we
wanted
to
turn
our
operational
reports
into
near
real-time
and
meet
high
availability
residence
with
four
nines
rapid
time
to
market
delivery,
as
well
as
reduce
the
cost
of
developing
new
features
or
enhancements.
A
Reporting
we
kind
of
streamline
from
various
different
processes
that
we
had
to
on
to
the
new
platform.
So
the
three
key
components
we
changed.
You
know
we
have
to
transfer
the
data
and
also
make
it
available
for
distribution
to
various
other
systems.
So
we
choose
Kafka
for
this
and
the
data
transformation
that
is
needed
on
the
reports
they
are
written
in,
go
and
doc
raised
and
deployed
and
authorized
containers
and
all
of
the
raw.
A
The
two
key
driver
decisions
to
get
away
from
the
best
processing
on
the
new
technology
of
the
new
platform
is
moves.
The
data
when
it
is
available
and
then
process
the
data,
either
partially
or
completely.
Whenever
that
last
data
element
arrives
right,
I'm
going
to
talk
about
why
Cassandra
and
then
I'm
going
to
hand
over
to
Makram
to
talk
about
the
solution
stack.
So
what
are
the
some
of
the
design
decisions
that
we
have
to
go
through?
Let's
dig
into
Cassandra.
A
As
you
all
know,
Cassandra
is
the
AP
system
right,
availability
and
partition.
Tolerance
is
what
it
provides
and
we
have
to
work
to
tune
the
consistency
and
given
that,
as
a
as
a
feature
from
Cassandra
that
was
the
right
fit
for
our
follow
use
case.
We
wanted
to
get
away
from
the
joints
which
are
cost
in
our
DBM,
so
the
combination
of
partitions
denormalization
and
the
vibro
tables
for
the
name
no
plainer
first
adopt.
A
We
wanted
to
build
our
new
data
platform
incrementally
rather
than
a
big
bank
cut
over
so
casandra's
linear
scalability
was
was
important
for
us
as
anyone
we
get
more
data
on
Windows
platform.
We
wanted
the
system
to
scale
as
well
last,
but
not
the
least
out
of
the
boss.
Multi
datacenter
deployment
was
an
obvious
choice
for
us.
A
I'm
sure
you
all
know,
rights
are
fast
in
Cassandra.
You
have
to
model
the
data
for
the
reads.
Since
you
are
incrementally
building
the
data
platform,
we
constantly
looked
at
the
read
patters.
We
know
at
the
time
to
decide
how
do
how
we
should
store
the
data,
partition,
keys
and
size
I'm
sure
we
have
heard
this
many
times
here
in
this
in
this
conference
here.
A
This
is
where
we
had
the
push
and
pull
within
the
within
the
teams
to
decide
what
should
we
optimize
our
certain
partition
keys,
helped
us
satisfied
good
number
of
query
patterns,
but
then
you
also
have
to
consider
the
size
of
the
partition.
So
we
don't
skew
the
partitions
too
much
by
having
big
partition
storing
on
one
lower
cluster.
A
So
so
we
decided
that
we're
not
going
to
go
with
time
series
and
for
this
specific
use
case,
we
decided
to
use
an
application
idea.
The
partition
key
and
the
application.
Id
has
good
amount
of
data
size,
wise
and
then
the
partitions
are
going
to
be
many
number
of
partition
spread
across
our
cluster.
So
we
decided
to
stare
on
50
200
MB
in
the
partition
size.
A
There
are
a
number
of
questions
we
go
through
while
modeling
there
are,
some
of
them
are.
How
do
we
do?
Consistency
is
the
reading
and
writing
by
quorum?
Is
that
good
enough
right?
If
you
want
to
use
GSE
search
you'll,
have
to
think
about
the
same
thing
you
know.
Do
you
want
to
read
the
data?
If
you
read
the
data
using
DSC,
then
you
have
to
think
about
how
you
cannot
write
the
same
data
and
what
consistency
you
want
to
use
to
write
the
data,
so
a
bunch
of
options
available
through
indexes.
A
These
features
are
very
tempting
to
use
again,
the
question
is
going
to
be:
do
you
want
to
you
use
this
approach
to
query
the
data
and
if
you
are
and
you're
creating
the
data,
do
you
guarantee
that
each
time
you
created
the
data
you're
going
to
get
the
same
results
back
so
all
of
these
questions?
So
how
do
we
address
these
questions?
You
know
this
is
the
slide
or
the
table
that
we
have
spent
most
of
our
time
to
decide.
A
How
do
you
wanna
model,
our
data
and
I've
heard
like
some
of
these
questions
asked
in
the
previous
session?
What
is
the
Silver
Bullet
and
how
do
you
know
which
data
model
is
right
for
you,
I
would
like
to
find
out
the
consistency
formula
that
that
is
out
there.
I'm
sure
most
of
you
should
know,
but
let
me
know
you
know
is:
have
anyone
seen
this
or
know
about
this
consistency
formula
good.
So
let
me
briefly
explain
work
with
an
example.
So
data
stored,
let's
say
with
the
replicants
factor
of
three
right.
A
The
sum
of
the
read
and
the
write
consistently
should
be
at
least
four
for
you
to
get
the
consistent
data
back
right,
so
quorum
in
this
case,
for
a
three-day
finish
factor
would
be
two.
So
if
you
are
reading
by
two
and
writing
by
two
is
therefore
so
you're
guaranteed
that
if
you
do
that,
then
you're
gonna
get
consistent
data
back.
So
that
was
one
of
the
key
things
that
would
have
written
on
a
piece
on
a
board
that
would
always
go
back
to,
and
then
you
look
at
what
are
the
different
data
access
options?
A
We
have
that
we
want
to
model
or
data
against
right,
so
the
table.
If
you
look
at
it,
you
know
the
first
couple
of
rows
go
in
pair,
the
second,
the
third
and
the
four
corn
rows
goes
in
goes
another
pair,
and
then
you
have
DSS
search
in
the
bottom,
so
we
have
assessed
the
data
model
based
on
the
various
aspects
of
of
the
data
that
is
stored.
A
So,
let's
take
at
the
row
one
now,
you
can't
get
away
with
it
right,
because
this
is
the
basic
store
data
store
option
that
you
have
to
come
up
with
right,
so
the
colors
or
the
base
colors
that
you
see
on
the
table
is
basically
highlights.
What
are
the
pain
points
that
you're
going
to
feel?
You
cannot
go
with
that.
So
if
you
model
the
data,
you're
partition
the
data
you're
storing
the
data,
the
thing
is
that
the
limited
evaluation,
the
data.
So
let's
say
you
know
there
are
a
bunch
of
columns.
A
You
decided
to
be
your
partition
and
clustering
keys
right,
so
you
can
only
search
based
on
those
those
columns.
Now,
if
you
have
another
in
a
column
that
you
want
to
look
at
to
see,
if
you
can
kind
of
search
with
that
extra
column,
you
will
not
be
able
to
so
now.
You
have
to
have
think
about
options
of
how
do
you
wanna
optimize?
A
Your
data
store
right,
the
next
row
in
there
that
you
want
you
can
duplicate
the
data
right
duplicate
the
data
with
the
another
partitioning
scheme,
so
you
can
fulfill
the
search
criteria
so
how
you
want
to
read
the
data,
but
then
you
know
it
will
pay
a
price
for
that.
What
is
the
price
your
store
is
going
to
go
up
because
you
have
duplicating
the
data
you're,
storing
the
data
again,
depending
on
what
replication
factor
you
have
to
consider
that
as
well
and
then
you
know
as
an
application.
A
A
What
other
options
do
we
have
well
inverted
indexes
and
secondly,
indexes
probably
there's
a
trade-off.
Well
inverted
index
again.
Is
another
option
where,
as
an
application,
you
are
creating
the
data
as
an
index
and
you're
not
necessarily
duplicating
the
entire
data
right?
So
that's
another
option
again
the
penalty
that
you
would
pay
as
an
application.
You
have
to
manage
that
secondary
indexes
is
an
option.
I
think
you
know
I've
heard
people
ask
you
not
to
adopt
it
stay
away
from
it,
because
the
way
it
behaves
is
different
than
how
a
regular
partition
would
behave.
A
But
you
know
there
may
be
use
case
in
your
situation
that
you
might
use,
find
it
useful
right.
So
you
have
to
look
at
the
cardinality
of
the
data
that
you
are
trying
to
index
on
and
if
the
cardinality
is
small
or
low,
then
you
can
go
with
it
because
you
know
the
way.
The
second
indexes
work
is
you
do
get
the
parallelism
with
which
you
can
query
the
data
and
finally
DSE
search.
Dse
search
is
very
versatile
in
nature.
We
have
looked
at
the
option
in
the
previous
slide,
but
it
is
slower
relatively.
A
There
are
all
the
other
options.
The
consistency
is
one
other
thing
that
you
have
to
keep
in
mind
because
you're
always
going
to
read
by
one
consistency,
which
means
going
back
to
the
formula
up
there.
You
will
have
to
have
the
right
consistency
for
the
same
data
that
you
want
to
read
to
Soler.
You
have
to
do
it
by
replication
factor,
so
we
have
benchmarked
it
on
a
subset
of
data.
We
took
about
20
million
rows
of
data
on
a
six
node
Cassandra
cluster.
A
You
can
see,
you
can
hardly
see
the
numbers
here
for
the
primary
and
the
inverted
indexes
secondary
indexes.
You
know
came
third
and
in
the
solar,
so
some
of
the
performance
tuning
parameters.
You
know
we
looked
at
for
our
specific
use
case.
You
know
we
have
used
three
replication
factor
for
the
use
case
that
we
have
addressed.
We
are
a
right
heavy
database,
so
we
have.
A
These
are
Continental
concurrency
right,
so
64
and
will
decrease
our
concurrent
reads
appropriately
to
16
and
we
adopted
the
more
stable
sized
here
compacted
strategy,
Cassandra
cluster,
with
DSC
search,
enable
on
all
of
them
and
I.
Think
you
know
the
recommendation
is
that
if
you
go
with
that
virtual
nodes,
don't
even
enable
it
I
think
d4
is
256.
We
felt
that
16
optimization
is
worked
well
for
us
all
caches.
We
disabled,
Excel,
filter
cache,
and
since
we
are,
we
have
deployed
this
in
cloud.
We
used
ec2
snitch.
A
You
know
this
is
the
way
the
rack
awareness
cassandra
has
implemented.
We
use
three
availability
zones
and
using
easy
to
sneeze,
so
our
replication
factor
tree
worked
perfectly
well
for
us
and
DSL
all
software.
We
commit
we,
we
actually
bumped
it
the
time
to
attend
and
we
really
be
hard
to
use
that
with
this
I'm
gonna
ask
Mukarram
aziz
to
come
over
and
talk
about
solution.
Stack.
B
Hey
guys
a
Mukarram
mukarrama
zzz,
I'm
helping
build
capital
1
a
new
data
platform,
a
new
modern,
near
real-time,
distributed
data
platform
for
operational
reporting,
did
I
get
that
right,
Javed,
okay,
so
so
you've
already
this
I
know
that
Cassandra
is
our
final
data
store.
We
did
a
lot
of
good
work
in
actually
making
it
happen
for
us,
it
is
working
out
for
us,
but
there
were
other
components
also
that
we
had
to
build
when
we
are
trying
to
build
this
new
data
platform.
B
So
I'm
going
to
talk
about
a
little
bit
about
the
ecosystem
that
you
built
and
the
different
technologies
that
we
have
made
so
before
I
go
into
lot
of
details
and
explain
the
slides
I'd
like
to
point
out
two
things
at
the
baseline.
So
what
we
did
was
we
actually
had
I
would
say
an
independence
to
build
this
platform
from
ground
up.
So
we
have
a
wonderful
leadership
team
in
Capital
One
and
we
were
asked
to
build
it
from
ground
up.
We
did
not
have
to
leverage
any
existing
technology.
B
B
You
know
in
a
way
where
we
were
starting
this
platform
from
ground
up
and
second
thing
that
we
knew
very
early
when
we're
designing
it
that
we're
going
to
deploy
this
in
the
cloud
AWS
like
Javed
mentioned,
has
already
been
our
chosen
cloud
platform,
and
we
knew
that
the
services
that
we're
gonna
deploy
the
software
that
we're
going
to
build
all
would
have
to
be
run
in
the
cloud,
so
that
gave
us
two
distinct
challenges
and
advantages.
At
the
same
time
distributed
nosov,
the
cloud
will
have
to
build
software,
that
is,
that
is
actually
distributed.
B
We'd
have
to
choose
technology
that
is
distributed
because
cloud
can
now
give
you
that
freedom
to
actually
build
a
distributed
system,
and
also
we
had
to
build
resilient
systems
with
the
cloud.
We
knew
that
things
would
go
down.
Your
nodes
would
come
down
your
networks
break
down,
so
we'd
have
to
build
a
fault,
tolerance,
resilient
system
that
would
stand
up
any
breakdowns
in
the
network.
B
While
it
comes
into
the
platform
so
data,
when
we
actually
had
the
data
in
rest
or
in
motion,
we
would
actually
transform
the
data
aggregated
process,
it
do
some
monthly
roll-ups
and
that's
where
we
also
had
to
solve
for
the
data
transformation
part.
So
what
we
came
up
with
was
a
single
stack
of
a
framework.
We
looked
at
spark,
we
looked
at
Samsa,
we
looked
at
Genesis.
B
B
Okay,
so
we
are
a
big
fan
of
go
in
Capital
One.
We
actually
are
very
pleased
with
the
extremely
fast
build
times
and
run
times
it
gives
us.
It
also
brings
an
overall
overall
agility
to
the
development
team
and
we
are
happy
with
it
most
of
the
platforms
most
of
the
services
that
we
are
building,
at
least
for
this
particular
use
case,
is
being
done
and
go.
B
So
the
core
of
this
framework
is
basically
that
you
have
plugins.
You
have
developers
who
write
the
plugins
that
are
basically
run
through
a
runner.
All
this
is
basically
run
as
a
single
service,
and
then
you
would
have
an
input,
channel
and
output
channel,
in
which
data
is
coming
to
the
plug-in
and
basically
the
data
is
getting
processed
and
then,
finally,
it's
being
sent
out
to
the
output
plug-in,
which
is
actually
a
reference
to
the
next
plug-in,
to
create
a
service.
We
have
to
change
these
plugins
together
and
boostrap
it
with
the
configuration
file.
B
B
We
would
write
something
where
we
pull
lots
of
data
from
an
Oracle
database
process
it
as
it
goes
into
our
service
by
setting
up
lot
of
different
plugins
in
that
order,
and
then
finally
outputting
that
process
data
into
a
Kafka
topic.
So
because
we
had
to
reuse
lot
of
plugins,
we
wrote
a
lot
of
common
plugins
that
that
would
help
the
entire
development
team
and
by
the
way
we
would.
We
have
plans
to
open
sources
in
near
future.
It
would
be
available
in
our
open
source
community,
so
we
had
to
write
plugins.
B
B
So
all
our
services
and
most
of
our
infinite
items
that
we
are
in
for
components
that
we
build
are
actually
running
in
docker
containers.
Most
of
our
DevOps
happen
in
docker
containers.
The
message
within
that
team
is
very
clear:
any
code
that
you're
going
to
write
any
service
that
you
can.
I
deploy
any
infrastructure
item
that
you're
going
to
actually
use
would
have
to
run
in
a
docker
container
and
all
our
builds
all
our
services.
B
So
once
we
decided
with
the
deployment
model
the
framework
we
had
to
find
a
way
where
we
would
actually
have
message
distribution.
So
we
would
have
multiple
different
components:
processors
that
are
actually
playing
around
in
this
platform,
and
we
needed
to
find
a
solution
so
that
we
can
pass
messages
between
these
different
components.
So
huge
amounts
of
data
had
to
be
moved
between
two
processing
units.
Kafka
was
our
choice.
B
We
looked
at
other
technologies
out
there,
other
open-source
technologies,
and
we
decided
in
Kafka
because
it
gives
us
a
very
high
availability
as
well
as
high
throughput.
So
Kafka
serves
three
purposes
for
us:
it
acts
as
a
buffer,
so
we
have
two
different
components
that
are
actually
processing
at
a
different
rate.
So,
instead
of
creating
a
back
pressure,
we
use
Kafka
to
actually
act
as
a
buffer
between
these
two
components.
We
also
use
Kafka
for
distribution
of
our
messages
or
a
description
of
our
data.
B
So
while
we
get
this
huge
amount
of
data
into
our
platform
from
multiple
data
sources,
yet
other
clients
who
are
actually
also
interested
in
that
data,
they
could
actually
because
the
data
in
rest
is
going
to
be
in
Cassandra.
We
would
not
want
to
expose
Cassandra
as
as
a
discipline
Chang
and
for
our
data,
so
instead
we
would
make
this
data
available
through
Kafka
to
other
external
clients
within
Capital
One,
who
are
not
local
to
this
platform.
B
The
third
part
that
Kafka
serves
is
that
it
helps
us
to
do
the
replay
mechanism.
Things
go
wrong.
We
have
ingested
a
lot
of
data,
its
resides
in
Cassandra.
We
would
add
you
would
like
to
make
a
bug
fix.
So,
instead
of
pulling
in
the
data
again
from
the
data
source,
we
actually
just
replay
it
from
the
Kafka
topic
and
the
data
is
available
to
us.
So
once
we
have
the
Kafka
in
place,
we
had
to
other
think
about
other
components
as
well.
We
looked
at
an
API
server.
B
B
We
also
needed
a
mechanism
for
service
discovery,
so
we
looked
at
zookeeper.
We
looked
at
HDD,
we
will
tried
Redis,
but
we
decided
to
use
console
mainly
because
it
is
distributed,
and
it
is
highly
available.
So
consider
was
our
choice
because
all
our
services,
when
they
start
up,
they
actually
have
to
be
bootstrap,
so
a
service
when
it
comes
up
it
has
to
be.
It
goes
to
console,
discovers
who
it
is
and
then
also
discovers
other
services
that
it
needs
to
talk
to
in
the
platform.
B
B
It
also
provides
us
a
consolidated
view
that
our
DevOps
team
can
actually
look
as
to
what's
happening
in
the
entire
data
center
and
look
at
the
server
resources
like
disk
usage
for
the
entire
data
center
memory
usage
and
also
CPU,
and
then
finally,
we
use
marathon
to
actually
run
our
services
or
containers.
So
marathon
comes
with
miso
sphere
and
we
use
it
for
our
deployment
model
marathon
helps
us
to
starting
deploying
scaling
stopping
managing
our
docker
containers
that
are
highly
available
services.
B
So
again,
our
stack.
Basically,
it
looks
like
this.
This
is
a
new
data
platform.
We
have
Cassandra
as
our
data
storage,
we
have
using
our
in-house
100%,
develop
a
go
base,
plug-in
framework
for
all
of
our
data
ingestion
and
our
data
transformation.
We
use
docker
for
packaging
and
deployment.
We
have
mesosphere
and
marathon
for
our
cluster
and
our
instance
management,
and
we
use
Kafka
for
data
transfer
and
and
data
ingestion
and
distribution
channel.
We
use
console
for
service
discovery
and
configuration
management,
and
finally,
we
use
Jenkins
for
continuous
integration.
B
It's
a
couple
of
benchmarking,
slides
that
we
had
was
we
actually
tested
for
a
functional
group.
We
had
a
six
node
cluster
Cassandra
cluster,
a
six
node
Carrefour
cluster.
We
looked
at
pulling
one
year's
worth
of
data
that
ideally,
we
would
pull
in
production
in
a
day,
so
it
took
so.
It
took
us
about
six
hours
to
pull
this
data.
We
process
about
450
million
rows
from
Oracle
and
transform
them
denormalize
them
through
our
data
platform
into
Cassandra,
and
it
amounted
to
eleven
point.
B
Eight
million
rows
in
Cassandra
the
rate
of
processing
for
the
go
platform,
just
the
rate
of
processing.
How
we
were
talking
to
the
messages
was
around
300
thousand
messages,
transactions
per
second
and
from
an
overall
basis
from
a
platform
perspective,
we
were,
we
were
actually
handling
about
21,000
messages
per
second
with
this
platform
and,
as
you
see
most
of
the
time
that
we
spent
was
getting
the
data
into
the
platform,
but
once
it
came
in,
it
was
a
smooth
ride
and
it
was
processing
at
a
very
high
rate.
A
A
We
are
moving
from
a
real-time
to
the
past
to
a
near
real-time
platform,
so
we
had
to
always
go
back
to
our
business.
Talk
to
them.
Then,
whenever
we
talk
about
any
of
the
metrics
cast
nations,
the
way
they
still
think
is
more
bashed,
so
there's
a
challenge
that
the
businesses
are
going
through
to
adjust
the
entire
stack
that
you're
looking
at
is
new,
so
our
operation
staff,
great
guys
to
work
with
they
are
adjusting
to
that
backup/restore
is
I.
A
Think
it
is
one
of
the
questions
that
is
very
common
in
Cassandra,
you
take
a
snapshot
which
is
your
inconsistent
snapshot.
This
is
taken
at
a
node
level.
Can
you
take
a
consistent
snapshot
of
it?
You're
working
with
a
team
at
data
asks
to
work
on
that.
You
know
we
are
doing
a
pure
C
with
them
and
then
thank
you
guys
for
all
your
great
help,
and
you
know
finding
Talent
is
another
challenge.
You
know
we
are
hiring
so
come
talk
to
us.
You
visit
our
capital
on
website.
A
C
A
So
there
is
an
injection
framework
that
we
had
to
write
in
water
that
no
matter
what
so
we
went
with
a
specific
framework
for
that.
Then
we
realized
that
you
know
do
we
want
to
really
go
with
another
framework
spark
or
any
other
complex
event,
processing
engine.
Do
you
want
to
use
that?
Or
can
you
just
leverage
what
they
are
built,
because
what
we
really
see
is
a
replica
of
the
same
ingestion
framework
is
we
can
completely
leverage
to
do
our
transformations
as
well.
A
So
when
we
only
feel
find
a
need,
you
know
use
case
to
go
and
set
up
another
set
of
layer,
and
we
are
pretty
much
you
know
leveraging
whatever
we
have
done
up
here.
There
are
the
different
discussions
that
we
have
heard
in
the
summit
as
well,
that
there
is
localization
of
processing
that
spark
and
help
you
know
when
you
really
process
the
data
across
your
cluster.
A
D
B
So
we
actually
are
using
2.1
version
of
Cassandra
and
we
believe
that
materialized
views
are
going
to
come
in
the
next
version
that
the
P
dot
o.
So
we
are
waiting
for
that
once
it
comes
with
definitely
a
look
at
it,
we
are
interested
well,
while
we
were
actually
looking
at
yesterday.
We
are
talking
about
how
we
could
actually
relate
to
that
and
basically
look
into
it
into
more
detail.
Definitely
we're
going
to
look
at
materialized
views,
given.
E
A
E
A
The
way
we
are
processing,
as
I
said
very
as
we
incrementally
building
the
platform,
we
look
at
one
focus
area
or
functional
area
at
a
time.
So
from
a
Kafka
perspective,
we
didn't
really
find
too
much
of
time
that
we
need
to
really
spend
on
it.
You
know
once
we
define
our
structure
of
how
you
wanna
create
the
topics,
how
many
of
them,
and
how
do
you
want
to
partition
it
right
to
spread
across
work
like
a
four
cluster?
A
That's
pretty
much
it,
and
then
you
know
the
consumers
of
that
data,
and
the
producer
of
the
data
is
the
thing
contained
within
the
platform
itself
right.
So
our
consumer
groups
are
set
up
appropriately.
You
know
in
or
micro-services
on
their
on
the
consumption
side
and
they
would
simply
go
ahead
with
whatever
panel
ISM.
You
need
to
read
the
data
from
we
have.
What
we
have
not
yet
you
know
figured
out
is
when
we
use
Kafka
or
distribution
channel
for
the
rest
of
the
other
enterprise
organizations
to
come,
consume
it.
A
What
would
what
is
the
best
model
for
that?
You
know
we
looked
at
there
is
a
mirroring.
Kay
will
be
available
in
Kafka,
so
let's
say
there
are
two
different
graphic
clusters
can
mirror
data
across
I
think
you
know
one
of
the
feedback
that
we
got
from.
We
know
we've
been
working
with
confident.
Also
on
that
is
no.
Don't
do
that
you
know.
F
F
A
Existing
data,
no,
we
do
have
inserts
and
updates
what
we
don't
have
is
deletes.
Okay,
so
being
a
data
platform
for
reporting
operational
reporting,
we
have
constant
data
that
we
ingest
and
we
generally
either
insert
there's
new
data
or
we
we
do
an
update.
So
when
we
do
normalize,
we
looked
at
you
and
what
are
the
different
options
that
we
need
to
look
at.
You
know
sets
as
one
of
the
collection
columns
fit
some
use
cases,
Maps
fit
some
use
cases.
We
try
to
stay
away
from
rows
and
columns
because
now
I'm
dead.
B
In
cassandra
in
Kafka,
not
really
I
think
Kafka
served
us
I
mean
this
is
not
the
first
time
using
Kafka.
In
our
platform,
we
have
you've
had
experience
using
Kafka
in
some
other
programs
as
well
so
data
loss
as
such
I
don't
think
we
have
done
that.
We
have
other
issues
with
Kafka,
but
not
data
loss.
A
So
I'm
a
across
Cappy
when
there
are
multiple
kaffir
clusters
for
this
data
platform,
we
have
dedicated
cluster
that
you
have
created
and
we
are
using
it
for
data
ingestion
as
well.
As
you
know,
data
distribution,
and
so,
as
we
incrementally
believe,
the
the
entire
platform,
we
started
with
a
six
node
cluster
and
then
we're
probably
going
to
add
to
that.
If
need
be,
what.
B
Do
you?
What
are
you
right
over
here?
What
are
you
planning
on
using
for
reporting
tools
to
distribute
the
data
out
yeah,
so
we
actually
have
tableau
that
we
use
extensively
in
Capital
One,
so
our
data
model
like
what
job
as
mentioned
earlier,
you
had
rotate
and
you
have
process
data
so
process
data
when
we
process
it.
Finally,
tableau
server
is
going
to
come
and
pick
up
that
view
from
from
Cassandra.
G
You
said
that
it
was
a
one
of
the
key
problem
statement
was
to
move
from
the
batch
world
both
from
the
business
organization
perspective.
But
could
you
shed
some
light
on
from
an
operations
monitoring
perspective,
because
operations
also
would
be
very
much
involved
in
monitoring
and
instrumentation?
So
the
question
is:
what
was
the
level
of
instrumentation
that
you
had
to
do
and
how
did
you
set
up
your
operational
escalations
to
make
sure
everything
is
running.
A
Yeah,
so
you
know
we
are
still
working
on
it,
as
I
mentioned,
as
one
of
the
challenges
that
we
have
is
operational.
Support
adjusting
to
the
new
technology
stack
exactly
so.
You
know
the
entire
DevOps
base
model.
We
have
been
working
in
a
more
traditional
operational
support,
so
you
do
a
dev
QA.
You
have
it
over
through
mo
piece
and
whatnot,
and
then
you
go
to
a
deployment
and
operationally
support
it.
A
So
we
are
tending
that
we're
changing
that
shift
over
to
more
a
DevOps
based
kind
of
model,
so
there's
an
adjustment
there
and
also
it's
no
more.
You
do
develop
and
you
throw
it
over
the
wall
kind
of
philosophy
anymore.
Right
now
we
are
more
tightly
integrated
from
a
deaf
perspective.
Also,
and
the
way
we
want
to
solve
this
problem
is,
if
you
were,
our
selves
have
to
operate
and
support
it.
How
do
we
do
it?
A
A
The
goats,
if
you
look
really
look
at
it,
the
go
pie
pan
is
more
the
internals
of
one
micro-service
right.
So
it's
a
fun
process.
Really,
you
know,
so
you
would
not
necessarily
be
able
to
really
monitor
within
they
go
micro
service.
How
other
plugins
are
behaving
so
but,
as
Makram
has
mentioned,
the
bootstrap
the
service
right
and
we
deploy
the
service
thread
of
a
container.
A
H
B
Actually,
using
Shopify
sarama
for
for
Kafka,
but
that
is
that
is
not
the
actual
frame.
Exactly
are
built
and
I
said:
there's
a
plugin
that
actually
actually
would
go
and
produce
or
consume
from
a
cask
or
topic
that
plugin
would
actually
use
sarama,
but
the
plugin
internally
will
be
run
through
a
runner
and
a
builder,
and
we
have
that
platform
itself.
That's
hundred
percent
in
go
not
using
any
extra
libraries.
Okay.
B
A
Well,
okay,
awesome
question
so
that
the
two
things
that
we
have
done,
the
question
is:
how
long
did
it
took
us
to
to
take
it
from
end
to
end
into
a
production?
We
started
working
with
a
POC.
You
know
it
took
us
about
three
weeks
to
build
the
entire
POC
with
two
operation
reports
on
it
on
it,
on
AWS,
with
about
three
resources,
I
consider
myself
half
so
two
and
after
sources,
totally
right
and
then
I
think
we'd
spent.
So
right
now
we
are
in
a
pirate
production
mode.
A
We
have
stood
up
this
entire
cluster
on
areas
within
three
to
four
months
and
what
the
target
is.
As
we
are
incremented
building
the
platform
we
didn't
want
to
necessarily
know,
come
up
with
a
bunch
of
business
requirements
and
address
it,
but
rather
incremental
e.
The
Builder
platform,
so
only
one
of
the
operational
reports
is
what
we
have
implemented
on
this
platform.
So
far,
so.