►
Description
Speaker: John Gao, Manager - Product Development
Online analytical processing (OLAP) enables real-time processing of large multidimensional databases where the potential query is not known a priority. A more light-weight approach is preferred where queries are known and where the data is not hierarchical or not structured in relational set. The advantage of NOSQL database, such as Cassandra in this light-weight model is that it can be used for caching and pre-aggregation of data sets, against known query constructs. In this way, Apache Cassandra data layer can hide data complexity while decoupling the underlying data sources and consuming solution layers. Vertafore uses Apache Cassandra to create an enterprise platform data abstraction layer and thus provides an analytic solution to both on-premises and online users.
A
A
B
Guess
I
should
translate
the
title
for
it.
That's
basically
our
experience
implementing
a
adapt
form
you
utilize
it
to
Center
in
various
other
technologies.
I'm
robert
johnson,
I'm
a
senior
software
engineer
at
vertafore.
I've
been
there
for
five
years
and
I've
been
a
developer
for
about
ten.
The
project
we've
been
working
on.
We
have
been
working
on
for
about
nine
months
and
we
use
it
to
production
just
a
few
months
ago,
so
had
a
fairly
quick,
a
development
cycle
on
the
solution.
B
B
B
So
vertafore
is
a
growing
company
with
with
plenty
of
opportunities.
It's
a
company
of
about
1,200
employees
with
a
deep
domain
expertise.
Third
of
those
are
developers
providing
applications
for
our
insurance
companies.
We
provide
software
for
over
300,000
professionals
that
we
use
vertafore
software
on
a
daily
basis,
software
that
touches
all
the
areas
of
the
insurance
industry.
So
we've
got
soft
products
ranging
from
from
client
portals
providing
that
access
for
individuals
to
to
get
their
claims
and
policies.
B
We've
got
rating
systems,
but
our
primary
software
is
around
agency
management
systems
for
for
the
agencies
to
track
the
policies
and
invoices
and
all
the
daily
business
of
an
insurance
company.
We
do
have
multiple
agency
management
systems
in
our
vertafore
ecosystem
to
scale
from
a
large
agencies
and
small
agencies
and
most
things
in
between.
B
It's
off
of
whatever
the
configurations
of
that
individual
instance
were.
They
also
need
to
configure
how
these
systems
will
interact
with
each
other,
and
we
collect
all
that
data
and
presented
on
a
dashboard
where
we
need
to
support
very
strict
s.
La's
providing
the
data
in
less
than
250
milliseconds
was
our
stated
SLA
on
the
brightside
central
utilizing
cassandra
work,
we've
seen
most
of
our
requests
are
responding
with
intent,
Millicent,
so
we're
well
under
arrest
lates
there.
B
B
After
learning
some
things
from
from
after
we've
implemented
most
of
this,
there
could
be
some
other
possible
and
solutions.
Utilizing
spark
and
a
number
of
these
other
technologies
we've
seen
here,
but
rid
of
Horace
the
new
to
Cassandra.
So
we're
limiting
the
number
of
new
things
we
implement
at
the
same
time,
stick
with
the
mix
of
things
you
know
and
things
you're
working.
B
So
we
mentioned
protocol
buffer
protocol
buffers
there,
which
is
a
serialization
schema
from
google,
and
one
of
the
things
we
found
early
on
in
our
planning
phases.
Was
they
hand
it
as
12
gigabytes
of
xml
data
and
says
these
are
the
sorts
of
things
we
need
to
be
collecting
from
from
these
systems
and
as
we're
polling
from
on-premise
systems
into
a
hosted
solution.
B
Uploading
12
gigabytes
of
data
from
400
agencies
is
not
something
that
is
easy
to
look
at,
so
we
played
with
the
data
a
bit
utilizing
google
protocol
buffers
to
to
serialize
this
data
and
after
serializing
and
filtering
it
and
compressing-
and
we
got
down
to
about
a
third
megabytes
of
data
that
we
were
actually
interested
in
quality
of
events
of
data.
So.
A
B
We
have
in
the
bottom
right
agents
that
are
hosted
by
the
source
systems
that
will
collect
data
from
that
system
and
package
it
up
and
send
it
to
our
services
in,
in
our
hosted
environments,
that
can
do
to
be
an
initial
load
of
data
which
will
allow
the
historical
value
will
need
to
provide
dashboard
aggregations
as
well
as
after
they've
performed
that
initial
upload.
They
can
just
send
us
Delta's
of
what
changed.
B
Since
the
last
upload,
we
have
a
configuration
management
endpoint
that
lets
the
agency
administrators
configure
how
aggregation
of
their
data
works
as
I
mentioned.
There's
Possible's
situations
where
they'll
have
multiple
systems
in
a
single
agency,
and
we
need
to
be
able
to
aggregate
across
those
systems.
Even
if
those
systems
don't
know
about
each
other.
B
So
now
we're
utilizing
queuing
systems
as
are
most
of
our
processing
is
event-driven.
So
when
the
agency
uploads
data,
that'll
trigger
events
in
our
system,
that
will
end
up
processing
that
data
triggering
more
events,
and
we
have
a
state
machine,
we
can
transition
from
one
state
to
another
of
it
makes
the
maintenance.
A
A
B
So
the
primary
point
that
we'll
start
with
here
is
the
dashboard
itself,
where
we
needed
the
high
SLA
s
and
the
good
performance.
So
we
got
an
example
of
some
of
the
dashboard
data
that
we
present
to
our
customers
here.
The
primary
thing
here
at
least
some
of
the
quick
data
they
get
at
the
top
is
a
dashboard
the
show
comparisons
of
their
data
between
the
current
year
and
the
equivalent
point
in
time
for
the
previous
year.
So
I
can
see
at
a
glance
whether
they're,
doing
better
or
worse
along
those
metrics.
B
So
each
tile
has
a
corresponding
panel,
which
will
present
more
in-depth
data.
So,
as
we
see
here,
we've
got
a
net
agency
revenue
over
24
months
period
with
the
24
months
being
side
by
side.
So
they
can
look
at
this
june
compared
to
the
previous
june
other
metrics
like
new
and
lost
business,
to
give
them
indicators
of
how
their
sales
pipeline
is
doing
and
we've
got
total
transaction
totals
and
a
piggys
total,
so
they
can
see
at
a
high
level
which
carriers
are
driving
off
to
their
business.
B
B
So,
since
we're
providing
json.net
shop,
so
are
we
are
utilizing
a
asp.net
Web
API
references
more
specifically
were
also
utilizing
a
technology
called
Oh
data
to
help
provide
discoverability
on
top
of
a
restful
endpoint
we're
using
standard
patterns
of
dependency,
injection
and
repositories
to
abstract
out
our
data
calls
from
our
web
interfaces,
and
our
repositories
are
basically
wrappers
around
the
cassandra
dated
here's.
Here's
an
example
of
one
of
our
tables
where
we
partition
the
data
for
a
particular
data
set.
B
So
one
of
the
interesting
things
that
we
were
considering
when
developing
this
is,
we
didn't
want
to
change
the
dashboard
without
the
users
knowledge.
So
if
they
had
a
new
piece
of
data
come
in
and
they
were
in
the
middle
of
analyzing,
something
it's
not
a
good
idea
to
change
what
they're
analyzing
in
real
time.
Necessarily
we
want
to.
Let
them
be
aware
of
the
changes
there,
so
we
track
all
of
their
data
according
to
a
dataset
either.
B
B
A
B
B
B
Okay,
so
now
we
have
a
set
of
tables
that
were
provided
with
dashboard
to
give
these
high
SFA's
and
past
performance.
As
I
mentioned,
we're
utilizing
a
state
machine.
Here's
the
primary
states
of
our
state
machine
they
get
triggered
by
various
events,
so
uploading
data
triggers
processes
that
will
a
verifier
but
validate
and
store
that
data
and
then,
as
we're
aiming
towards
providing
data
for
a
dashboard.
B
We
have
a
few
aggregation
steps
where
we
aggregate
the
data
to
some
internal
forms,
which
takes
care
of
a
number
of
the
cross
system,
compatibility
issues
where
one
system
has
a
completely
different
structure
for
what
a
policy
is,
then
the
other
systems
in
them.
We
unify
all
the
data
schema
to
the
eternal
forms
here
and
then
during
our
finalization
phase,
where
we
r
d
normalizing
the
data
collected
into
each
of
those
dashboard
tables,
so
each
dashboard
graph
is
backed
by
a
special
table.
B
A
B
B
B
We
found
that
storing
this
data,
basically
as
flat
as
it
is
uploaded,
it
has
been
a
useful
way
of
savings
based
on
the
data
uploads,
so
we
distort
the
reference
to
what
the
data
that
was
uploaded
and
stored
on
disk
for
later
use.
As
I
mentioned,
we
do
have
the
intermediate
data.
So,
during
our
preparation
phase,
we
all
through
the
events
of
the
queue
and
the
state
machine
we.
A
B
B
A
B
Configurations
which
defines
the
rules
for
how
this
data
was
was
configured,
what
values
get
included
in
it,
and
it
also
corresponds
to
the
dashboard
data
the
batch
ID
will
rep.
It
is
representative
of
the
batch
of
data
that
this
instance
of
data
started
from
because,
as
mentioned,
should
they
reconfigure
change
the
rules
of
configuration.
We
need
to
go
and
reprocess
the
data
and
generate
it
for
for
a
new
configuration.
B
A
B
A
B
Daily
data,
in
a
structure
that
is
known
to
our
application,
some
of
the
other
pros
of
protocol
buffers
as
we
can
manage
the
schema
changes
and
change
the
serialization
data
with
relative
ease,
as
that's
some
of
the
built-in
things
to
how
protocol
buffers
see
realizes
its
data.
So
the
other
things
there
is
it
optimizes
the
network
transmissions.
So
we
know
that
we're
going
to
be
selecting
a
large
chunk
of
these
pre.
B
B
We
wanted
to
use
cassandra
is
the
ability
to
support
my
own
scale,
where
we
don't
have
to
worry
about
it,
I
when,
when
talking
to
other
developers,
they'll
often
ask
me
okay,
how
is
your
database
scale
when
choosing
another
solution
like
well?
I?
Don't
have
to
worry
about
that
because
I
chose
Cassandra
Brits
it
not
a
big
concern
so
as
to
some
of
the
things
that
we
would
like
to
see
better
improved
is
the
development
tools
around
Cassandra
are
are
fairly
minimal.
The
datastax
depth
dev
ops
center.
B
What
is
rather
limited
and
the
sort
of
devtools
you
you
can
use
it
for
and
perhaps
the
more
annoying
one
we
haven't
seen,
found
any
good
tools
for
schema
migration
and
deployment
management
for
for
managing
this
schemas.
That's
still
mostly
a
manual
process
at
this
point,
but
I
expect
we'll
get
to
some
sort
of
solution
to
that
sooner
or
later,.
B
Actually,
there
was
one
other
point
here
so
as
vertafore
is
newton
and
these
sorts
of
technologies
we
want
to
have
incremental
improvements
is
how
we
new
things.
So
it
was
suggested
from
from
upper
management
that
we
try
to
see
if
we
can
use
cassandra
here,
because,
as
with
many
businesses,
you
want
to
limit
the
risk
of
new
things,
and
so
doing,
as
I
mentioned
in
the
beginning.
Doing
one
new
thing
at
a
time
is,
is
a
good
way
to
slowly
build
up
your
your
domain,
expertise
in
new
technologies,
so
moving
forward.
B
Some
of
the
things
we
expect
to
support
and
future
versions
of
our
dashboard
is
additional
levels
of
detail
which
will
require
sorting
and
filtering
of
the
data
at
the
dashboard
level,
so
they
can
get
more
fine-grained
views
of
their
data.
We
anticipate
utilizing
solar
search
to
support
most
of
that
filtering.
That's
a
good
pairing
for
distributed,
filtering
and
search
further
down
the
road
when
we
added
add
additional
service
systems
to
our
aggregation
process.
That's
where
the
scalability
concerns
will
become
more
apparent,
where
we
need
to
support
our
other
agency.
B
A
B
B
There
has
been
look
into
I,
don't
remember
that
name
of
it,
but
there
there
was
a
third
party
aggregation,
visualization
tool
that
we've
been
investigating
at
vertafore
and
tying
what
we've
done
with
our
services
into.
That
is
something
that
we
are
looking
into.
As
to
the
second
question,
how
we
get
the
data
from
Cassandra
to
the.