►
Description
Speaker: Claudiu Barbura, Senior Director of Engineering at Atigeo
xPatterns is a big data analytics platform-as-a-service that enables rapid development of enterprise-grade analytical applications. It provides tools, API sets and a management console for building an ELT pipeline with data monitoring and quality gates, a data warehouse for ad-hoc and scheduled querying, analysis, model building and experimentation, tools for exporting data to Cassandra and solrCloud clusters for real-time access through low-latency/high-throughput (automatically generated) apis as well as dashboard and visualization api/tools leveraging the available data and models. In this talk I'll share some of the hard lessons we've learned in the past three years while leveraging Cassandra (and Hector) in large-scale enterprise-grade deployments. We will focus on three specific areas, in which we identified consistent best practices & design patterns: data model optimization as a result of exporting data from HDFS/Hive/Shark into Cassandra through Spark/Hadoop MR jobs under Mesos with throttling, instrumentation and resilience features, automatically publishing geo-replicated, instrumented and monitored REST API's on top of the exported Cassandra data, and lessons learned from running Cassandra at scale from 0.6 to 2.0.6, including performance tuning, and tips and tricks. You will see live demos of our Publish to NoSql tools (Spark/Shark, Mesos, Hive, Cassandra ), a dashboard application built on top of generated data apis (D3.js, Cassandra) and xPatterns' monitoring and instrumentation consoles (Graphite, Ganglia, Nagios).
A
A
Hi,
sir,
my
name
is
Claudia
I'm,
an
engineer
working
for
for
a
teacher
were
based
in
bellevue
washington
and
I
work
with
an
amazing
group
of
engineers
from
romania.
It
made
all
the
demos
he'll
be
singing
possible
special
thanks
to
tino
lega
for
working
hard
to
wrap
this
up,
and
I
want
to
ask
the
audience:
what
do
you
prefer?
Slides
or
live
demos
in
a
presentation
live
demos?
What
about
the
crashing
devil?
A
So
what
is
experience?
It's
a
big
data
analytics
platform
that
provides
tools
and
a
P
is
for
engineers
to
build
now.
End-To-End
big
data,
intelligent
applications.
So
now
we
express
that
in
three
powerful
verbs
connect
discover
an
act,
so
the
Kinect
is
the
infrastructure,
provided
the
big
data
infrastructure
with
all
the
tooling
from
provisioning
to
two
operations
allows
you
to
build
things
like
an
ETL
pipeline
or
their
data
warehouse.
Then
we
have
an
intelligence
layer.
A
The
discovery
part
it's
more
for
analyzing
data
scientist,
so
they
can
do
their
modeling
and
building
model
training
them
publishing
them,
and
then
we
have
the
Act
part.
The
application
layer
where
we
provide
you
know
ap,
is
that
allows
them
to
build
a
lot
of
application
engineers
to
build
user-facing
applications.
A
So
our
platform-
imagine
we
thought
infrastructure
allows
you
to
ingest
data
from
external
sources
into
a
compute
cluster,
where
we
build
a
data
warehouse
in
the
computer
lustre.
We
have
the
typical
Hadoop
in
hive
and
the
last
year
we
upgraded
that
sack
to
spark
and
sharp
sparks
equal
now,
Mesa
says
resource
manager
and
tachyon
as
a
new
memory
distributed
file
system.
A
So
we,
these
are
collocated
side
by
side,
is
also
Aurora.
That
we
use
is
another
framework
that
runs
under
mesos.
It's
not
mentioned
here.
All
the
data
is
being
ingested
in
this
in
the
data
warehouse,
HDFS
and
tachyon.
It's
also
geo
filled
over
through
s3
buckets
and
different
us
different
ec2
regions.
A
Then
all
the
machine
learning
all
the
data
warehouse
operations
are
performing
this
cluster.
Whenever
we
want
to
build
application,
exposes
applications
and
users,
we
separate
workloads
and
we
push
data
from
the
warehouse
into
Cassandra.
Mainly
it's
our
cassandra
is
our
main
real-time
database.
We
also
have
solar
cloud
and
sequel
on
the
side
talking.
The
demos
today
are
about
obviously
cassandra
and
in
the
process.
No,
not
only
we,
we
build
an
efficient
data
model
in
Cassandra
out
of
the
data
in
the
warehouse.
A
A
Aws
route,
53
dns
service,
which
really
directs
client
requests
to
the
closest
data
center.
Obviously,
here
on
the
west
coast,
will
be
sent
to
the
to
the
west
coast.
Data
centers
and
that's
part
of
the
demo
will
show
how
ingest
data
we
do.
Data
transformation
pipeline,
we
created
a
warehouse
and
then
we
publish
the
data
to
to
Cassandra
with
AP
eyes
and
thought
that
they're
ready
to
run
these
api
is
her
three
ingredients.
A
A
So
this
is
the
expanse
management
console
that
allows
all
this
will
happen.
You
see
the
three
verbs
connect
to
discover
act,
so
connect
allows
us
to
ingest
data
and
then
transform.
It
will
do
a
bit
of
discovery
to
discover
the
warehouse
that
we
build.
On
top
of
that
data
is
being
adjusted
and
then
ultimately
will
export
to
no
sequel.
I'll
briefly
go
through
to
the
first
set
of
tools,
the
ingestion
tools.
Now
they
all
use
park
behind
the
scene
connects
s3
or
SFTP.
A
You
can
make
a
run
your
job,
you
get
the
full
log
execution
laga.
What
happens
be
any
scene
which
means
decompression
the
backup,
the
Geo
backup
moving
data
over
to
the
other
data
center
so
forth.
At
the
end
of
it,
your
files
will
be
readily
available
or
our
data
store
right
now
is
HDFS
or
adding
support
to
to
directly
streaming
data
into
talking
as
well.
That's
work
in
progress.
So
that's
the
ingestion
piece.
A
Then,
once
you
have
the
data
in
HDFS,
you
can
build
your
data
pipeline
and
we
have
now
we
enhanced
Lucy
part
of
the
Hadoop
ecosystem
and
we
provided,
in
addition
to
their
noted
hive.
Shell
ssh
those
type
of
actions
that
we
provides
out
of
the
box.
We
added
spark
shark
and
email.
So
this
again,
it
runs
since
sends
all
the
logs
in
real
time
to
to
the
council
through
web
sockets
engineers
provide
the
creates
pork
jobs
or
equal
scripts.
A
A
A
We
we
built
a
Huell
Ike,
a
tool
that
allows
you
to
explore
your
data
warehouse,
your
tables
in
the
warehouse
with
liquor
structures.
You
can
quickly
build
queries.
You
have
query
editor,
you
have
you.
Have
you
have
the
results?
You
have
the
query
history
for
each
query.
You
get
the
results
and
executing
logs
behind
the
scene.
A
So
that's
available
with
this,
where
I
was
also
exposing
data,
that's
being
stored
in
the
Tachyon
right.
That's
a
remember.
The
stupid
file
system,
all
it
takes,
is
creating
a
table
and
so
fixing
that
table
with
underscore
tachyon
and
that
short
will
push
that
into
memory.
And
then
you
can
start
querying
it.
So
not
going
to
this
too
much
on
these
tools.
What's
behind
the
scene
is
an
open
source
project.
You
might
have
a
link
at
the
end
of
this.
A
At
the
end
of
my
slide,
we
call
it
jaws,
it's
our
HTTP
shark
server
or
spot
sequel
server,
depending
on
what's
the
execution
engine
behind
an
assist
more
on
the
on
the
export
to
knock
sequel
tool.
So
once
my
data
is
in
a
warehouse,
if
you
remember
the
I'll
go
back
to
my
2d
architecture,
architecture,
diagram.
A
What
this
tool
will
be
will
connect
to
any
data
in
the
warehouse
and
push
it
to
Cassandra
and
also
generate
these
ap
is
behind
the
scene.
It
does
all
the
Geo
replication
at
the
data
store,
the
data
store
level
and
the
publishing
of
the
instrumented
and
monitored
ap
is
what
we're
working
on
right
now
is
also
enhancing
this
tool
to
provide
support
for
connecting
two
data
sets
in
tachyon
they're
already
in
memory
index
stir
that
does
not
expose
the
app
through
these
through
these
tools.
So.
A
A
We're
not
like
I
said
we're
working
on
supporting
tachyon,
both
tables
created
through
shark
thats,
stolen
talkin
or
really
tacky
on
files
that
store
in
the
target
memory
would
manage
schema
on
the
side.
So
once
you
created
that,
then
all
you
have
to
do
is
drag
here
to
create
a
mapping.
So
we're
mapping
data
between
a
warehouse
and
no
sequel
looks.
We
have
a
bug
in
the
process.
We
we
were
supposed
to
pick
unique
keys
to
specify
what's
the
unique
key,
so
we
know
how
to
model
that
dating
Cassandra.
A
You
can
do
fuel
transformation,
change
types,
there's
a
few
filter
summation
operation
available,
and
let's
actually
do
that
and
say
that
pick
one
unique
key
and
you
can
drag
this
mapping
now
here
to
create
a
pis
and
these
api
is
another
value.
So
these
api
swill
will
need
the
read
access
pattern.
The
application
engineer
knows
best
what
data
to
read
and
how
to
be
it
so
now
using
the
mapping
they
they'll
they'll,
provide
as
a
bug
I'm
talking
about
that's.
A
Why
it's
the
drop
that
is
so
long
they'll
have
to
pick
a
lookup
field
and
they
can
do
sort
by
today
and
we're
working
on
adding
grew
by
and
filter
by
which,
with
no
sequel,
it's
not
it's
not
trivial.
At
the
end
of
this
job
run,
this
is
the
signature
of
the
API
there's
going
to
be
generated,
but
what
I'll
do
I'll
go
back
to
a
pre
created
job.
A
This
is
a
data
set
downloaded
from
the
national
provider.
This
database,
with
all
the
providers
in
the
country,
doctors,
hospitals
and
we
created
two
mappings
one:
that's
going
to
move
over
only
a
subset
of
the
fields
which
you'll
do
most
of
the
time.
If
you
want
to
create
some
fast
lookup
api's
and
providing
ideal
the
provider
and
getting
back
the
name,
I
also
created
a
mapping
that
allows
you
to
get
back.
No,
the
entire
data
set.
A
What
we
do
behind
is
c
media
normalizing.
We
create
no
views
for
each
each
of
the
data
set
so
for
starting
from
each
map,
and
we
created
to
api's.
One
is
now
look
up
by
name
and
sort
by
NPI
and
the
reverse
operation,
looking
up
looking
up
an
NPI
and
getting
back
the
names
and
the
other
is
for
the
for
the
partial
will
get
the
soft
of
the
field.
A
So,
just
by
doing
this
this
this
mapping
here
I'm
going
to
generate
for
AP
eyes
and
let's,
let's
run
this
job,
to
see
what
we
get
each
of
these
two
that
I
showed
always
they
use
part
behind
the
scene
and
they
allow
you
to
specify
the
level
of
parallelism.
How
many
spark
course
I
want
to
use
on
the
cluster,
and
in
this
case
I
know
I,
have
this
many
cores
in
a
cluster
I'll
use
everything
and
I'm.
My
job
starts.
A
So
what
I
get
is
again
an
execution
log
with
all
the
details
now
the
sparge
up
server
that
runs
that
context
we're
running
under
meses
I'll,
show
you,
the
missus
console
and
second,
the
the
execution
details
for
the
spark
job.
What
we
do
behind
now
is
now.
We
connect
it
to
this
database
table
demo
DB
table
NPI
all
we're
finding
the
the
underlying
HDFS
file
and
we're
sizing
it,
and
we
need
a
sizing
of
the
data
in
HDFS
so
that
we
know
to
build
that
efficient
data.
A
Modeling
Cassandra
will
never
imagine
you
have
1,000,000,000
record
on
one
table.
You
don't
want
to
just
transfer
that
into
1,000,000,000
cassandra
rose
now.
Skinnier
else
does
not
yield
the
data
model,
so
what
we
do
is
we
try
to
maintain
the
road
size
in
50
megabyte
range,
which
means
we're
we're
grouping
those
multiple
rows
from
the
warehouse
into
one
bigger
cassandra
row.
A
So
then
we
do
column
slicing
to
get
to
my
actual
entity
that
I'm
transferring
so
that's
done
behind
the
scene,
all
the
validation,
no
checking
for
note
fields
for
unique
values,
whatever
I
specifying
in
my
mapping
and
then
I'm
starting
my
export
right
into
a
column
family,
that's
pretty
created
behind
the
scene
in
the
case
entering
is
your
replicated
with
the
aps,
so
the
job
started
to
write.
Well,
what
I
also
get
is
is
an
instrumentation
dashboard
right.
A
I
have
four
slaves
in
a
spark
cluster
for
each
one
of
them
we
emit
metrics
can
be
captioned,
the
throughput
average
latency
expressed
in
microseconds,
so
we're
talking
about
two
seconds.
Spikes
now
four
seconds
here,
that's
the
that's
the
latency
reported
by
this
for
class.
The
most
of
the
time
is
spending
Network
IL
because
we're
moving
data
from
one
plus
a
TD
to
the
designer
hang
over
the
network
and
also
the
the
total
items
and
total
calls
they're
being
transferred.
A
There's
also
a
new
feature
here
called
a
retry
count,
which
is,
if
you
have
a
big
enough
cluster
Hadoop
spark
doesn't
matter.
You
have
the
same
problem
and
a
small
enough
Cassandre
rank
you
might
be
exercising
too
much
load
and
at
the
center
ring,
and
at
that
point,
Cassandra
will
start
dropping
those
requests
and
sending
back
exceptions.
So
what
we
do
in
the
sparks
I,
we
calculate
exceptions
and
we
omit
the
metric
we're
destroying
that
at
one
second
intervals,
but
we
also
exposing
that
in
this
dashboard.
A
So
since
it's
for
slave
vs,
3,
3
nodes
and
East
Coast,
that's
not
a
mismatch
in
capacity.
So
we'll
never
see
these
retries
here,
but
not
only
if
you
saw
that
means
you
would
be.
You
were
maybe
too
generous
with
that
initial
level
of
paterson
that
you
specify
when
you
start
the
job,
so
you
can
cancel
the
job
and
restart
with
less
and
in
the
future.
We
want
to
do
sort
of
throttling
mechanism
ourselves
behind
the
scene,
but
it
gives
you
that
kind
of
control.
A
So
at
the
end
of
the
at
the
end
of
the
job,
the
data
is
being
transferred
and
there's
four
AP.
Things
have
been
generated
and
they're
ready,
BTW
to
be
used
and
we
can
test
them
out
immediately.
We
can
look
at
any
of
them.
So
that's
my
api
definition.
I
have
a
look
up
api
and
I
have
a
list
right.
You
can
pass
in
a
value,
an
NPI
and
get
back
the
the
name
or
you
can.
A
You
can
iterate
through
the
entire
data
set
that
was
transferred
and
you
have
your
you-
have
the
full
signature
and
actually
natural
called
now
with
all
the
headers,
with
with
the
request
on
the
body
what
with
data
behind,
but
we
also
have
a
data
browser
here
just
to
quickly
test,
what's
being
transferred
yeah.
So
now
we
can.
You
can
page
through
that.
Let's
find,
let's
find
some
value
to
try
it
out
some
api
value.
A
A
A
A
A
So
what
you
see
I
have,
I
have
four
Tomcats.
Well,
the
ones
the
176
ones
are
from
the
west
coast.
The
1002
are
on
the
east
coast,
and
this
will
tell
me
now:
I
had
one
of
the
D
average.
Ladies
before
the
call
on
the
on
the
one
of
the
west
coast,
tomcat
152nd
milliseconds
and
you
have
throughputs
like
items
/
items
per
per
second
total
calls
and
global
agency.
A
Yeah,
so
it
just
showed
up
so
another.
The
not
199
60
items
from
the
east
coast,
API
was
were
achieved
to
this
API
called
and
only
two
from
the
from
the
west
coast.
So
now
the
API
you
just
created
it
and
just
published
is
now
instrumented,
although
we
also
do
is
also
monitor.
These
AP
is
so
we
use
nitrous
for
that,
which
you
monitor
the
entire
clusters,
all
the
classes.
Actually,
if
I
go
to
the
front,
ends
a
front-end
group
I
have
to
service
and
on
the
west
coast.
These
are
the
method.
A
Do
you
have
the
classic?
No
disk
I/o
network,
CPU
memory,
and
we
have
set
up
now
we
set
up
alerts
here
when
we
exceed
a
certain
thresholds,
were
being
a
lot
of
three
emails.
You
have
Hector
metrics.
These
are
your
Cassandra
clients
with
everything
that
they
they
they
expose
and
we
have
the
exponent
a
p.I
metrics.
The
symmetric
you
saw.
So
this
is
the
the
jmx
beam.
That's
really
instrument,
instrumenting,
that
that
API
call
read
data
and
obviously
we
can
set
up
a
large
here
for
various
thresholds
again.
A
So,
let's
talk
about
this
part
part.
So
I
said
we're
running
under
mesos.
We
have
a
cluster
of
four
nodes
for
slaves,
each
one
having
32
course.
It's
all
a
128
course,
and
this
much
allocated
memory
across
the
cluster.
So
we're
running
multiple
frameworks
here,
it's
Hadoop
running
side
by
side
with
or
two
restful
service,
on
top
of,
sparse,
equal
or
or
or
charcoal
jaws,
and
we
have
to
spur
job
servers.
So
everything
can
be
tracked
here.
A
Our
export
jobs
ran
here
in
coarse
grain
mode
and
it
finished
wrapped
up
after
the
job
you
can
get
more
logs
from
the
actual
execution
logs
in
sorry
in
MSO's,
I
can
look
at
now
the
things
that
happen
behind
the
scene,
any
exception
they're
caught
by
Hector,
and
there
are
some
yeah,
so
Hector,
there's
a
few
warnings
from
Hector.
Now
Hector
is
a
drift.
A
The
third
of
driver
rapper,
I
should
say
in
java
client,
which
does
it's
only
try
and
save
over,
so
those
will
want
to
capture
in
the
future
as
well
and
showing
that
instrumentation
dashboard.
So
you
have
full
full
logs
of
now
what
happens
behind
the
scene.
We
also
is
the
ganglia
for
monitoring
all
the
clusters,
including
the
Cassandra
east
and
Cassandra
west.
A
A
So
we
had
did
not
give
it
in
the
three
nodes
and
on
the
East
Coast
right,
we're
only
ready
to
the
east
coast,
but
that
data
is
being
replicated
a
real
time
to
the
west
coast.
Cluster
West
Coast
ring
and
we're
seeing
40
micro
seconds
in
terms
of
latency.
If
remember,
the
latest
yield
on
the
spark
side
were
were
to
four
milliseconds
at
the
most.
So
that's
that's
a
kind
of
legacy
we
see
in
this
transfer.
A
A
A
So,
let's,
let's
see
the
damn
of
that,
so
we're
going
to
pick
a
physician
and
a
provider
submit
by
the
way.
This
is
the
dashboard
it
just
uses
the
data
Explorer
exporting
Cassandra,
so
I
use
it
as
a
PA.
The
topic
is
on
or
nothing
else,
it's
it's
roughly
1.8
million,
provided
they
were
extracted
from
that
tendon
record
data
set
and
fifty
million
relationship.
So
now,
starting
from
that
provider,
we
have
a
grouping
by
specialty
in
internal
medicine,
jeriah,
I'm
not
going
to
pronounce
that.
A
Neither
that
oncology
I
can
pronounce
pullman
rd.
So
you
have
all
these
specialties
and
within
each
specialty
you
have
total
number
of
claims
patients,
three
providers
referred
and
a
total
charge
amount.
So
how
is
this
useful?
Now?
The
customer
asks
for
a
graphical
representation
of
all
the
relationship
of
other
provided
for
two
reasons.
One
may
be
to
detect
fraud.
A
A
To
good
use
cases,
all
these
stats
were
pre
computed
in
in
the
warehouse
through
to
shark.
A
lot
of
this
data
was
exported
through
these
api
is
in
the
cassandra.
So
all
it
does
is
just
extremely
fast
kilo
cups
and
if
you
have
no
good
JavaScript
engineers
with
DCJS
skills,
they
can
build
something
like
this.
So
that's
just
an
example
of
an
application
of
built
on
top
of
experiments,
so
starting
with
ingestion
transformation,
data,
warehousing
exporting
to
cassandra
published
api's
again.
These
are
geo
replicated
eps
and
the
actual
dashboard.
A
A
What
burners
kima
disagreement
so
whenever
we
transfer
these
and
we
create
these
column,
families
in
Cassandra
with
these
job
run
we're
creating
column
families
and
they
have
to
be
synced
across
the
two
regions
a
lot
of
times.
You
would
see
scheme
of
this
agreement,
which
would
prevent
us
from
performing
any
operations
on
those
column
families.
A
So
what
we
learned
is
that
before
we
create
and
delete
any
column
family,
we
create
a
lock-in
zookeeper
so
that
we
make
sure
that
there's
only
one
of
these
operations
create
a
performant
at
any
point
in
time
and
before
we
create
at
and
you
later
that
we
also
synchronize.
So
we
use
something
equivalent
to
note
to
reset
local
schema.
That's
going
to
give
to
every
node
then
make
sure
that
it's
synchronized
it's
schema
before
anything
and
before
we
change
make
photos
schema
changes
in
that
particular
node.
A
Today,
it's
not
even
recommended
to
create
a
large
number,
and
the
fact
that
there's
we
see
is
thrift
and
because
we
started
in
06
and
Hector,
essentially,
there's
no
schema,
so
you
have
the
freedom
to
store
any
rows
and
anything
in
those
columns,
and
it
makes
any
type
of
data
in
in
just
one
column,
family.
So
that's
what
we
learned,
how
to
mix
multiple.
No
data
sets
in
the
same
column
family,
to
avoid
growing
this
number,
something
that
you
should
be
aware.
A
It
happened
to
us
I'm,
not
dropping
a
column
family
before
emptying
it,
which
means
that
rank
a
turn
compaction
just
rendered
the
ring
useless.
It
happened
to
us
twice
and
now
we
learn
that
lesson
a
right
now
before
we
remove
something.
We
make
sure
that
be
truncate,
that
first
cassano
get
into
a
stable
is
trying
to
load
files
that
shouldn't
be
there
anymore
and
it
gets
into
a
deadlock,
monitor,
instrument,
everything
and
the
automatic
restart
beside
a
whiplash
very
often
that
we
have
scripts
to
to
restart
the
demons
automatically
consistent
level.
A
A
So
we
have
enough
time
to
enough
of
a
distance
between
these
two,
the
bass,
job,
the
transfer
and
the
actual
reads
that
really
we
can
write
and
read
because
it's
a
level
one
is
anything
happens
in
between
and
now
we
have
monitoring,
we
can
react
and
an
early
start
service
and
so
forth.
But
now
cl1
will
give
you
this
another
best
performance
use
the
key
cash
we
use
map
before
compression
and
v
notes.
A
One
more
about
data
modeling
see
what
we
learned
all
these
years
is
again.
We
have
high
throughput,
you
know
right
scenarios
and
really
intensive
sonars
as
well.
In
order
to
avoid
out
of
memory,
we
learned
that
we
should
keep
that
now.
Rosen
in
fifty
two
hundred
megabytes
size,
not
too
small,
not
too
skinny,
not
too
wide.
A
Now
pretty
much.
You
want
to
reduce
the
number
of
rows.
Like
I
said,
1
billion
rows
is
not
going
to
work
too
well,
like
a
sauna.
That
means
now
pressure
on
too
many
rows
too.
Many
not
call
the
network
to
grab
more
data
pressure
in
memory
for
the
bloom
filter
for
the
key
cash,
and
obviously
you
have
less
no
reduced
network
I
when
you
have
less
rose
on
the
other
side.
Having
rows
are
too
wide
can
create
no
inefficient
compaction
and
can
create
hotspot.
So
that's
why
we
picked
this
particular
size.
That's
all!
A
When
we
move
the
data
from
the
warehouse,
we
always
size
it.
If
we
look
at
the
data
in
agia
first
and
then
we
count
records,
we
look
at
types
and
then
we
look
at
the
day
up
the
actual
file
sizes
and
we
compute
that
so
they
can
then
do
the
1
billion
to
50,000
rosy
Cassandra
transformation,
and
what
we
learn
is
is
we're
doing
our
own
custom
civilization.
So,
instead
of
know
those
entities
that
we
create
now
provider
with
name
ID
title
address
what
we
do.
A
We
sterilize
them
on
the
client
side
into
a
byte
array
and
we
only
store
by
they
threw
in
Cassandra,
and
that
gave
us
now
almost
40%
performance
gains.
Even
though
only
applications
can
now
read
that
schema
right.
You
have,
you,
have
scheming
right,
like
you're,
reducing
the
interoperability
with
other
applications,
but
really
what
we
care
for
is
the
is
maximum
performance.
Since
we
manage
the
schema
through
our
API
definitions,.
A
So
the
question
is:
what's
the
danger
having
their
elves
to
skinny
skinny
rels
means
a
large
number
of
rows
right,
1
billion
rows
that
creates
all
sorts
of
pressure.
First,
no,
just
if
you
want
to
achieve
multiple
entities,
it's
very
likely
end
up
making
multiple
network
calls
now.
Maybe
your
date
is
going
to
be
spread
out
across
multiple
nodes.
If
it's,
if
the
rows
are
wider,
there's
a
high
chance
that
that
one
row
can
be
because
it's
in
one
why
the
role
now
your
entities
are
still
as
in
one
big
growing.
A
It
can
end
up
in
one
only
one
physical
note,
so
retrieval
might
mean
just
one
no
mental
meant
tables
can
or
one
this
can
on
one
just
one
know.
Instead
of
doing
multiple
network
calls
and
in
a
scattered
gather
pattern
and
also
know,
there's
there's
a
bloom
filter
which
are
stored
in
memory,
the
Morrow's,
the
higher
the
size
mm
table
metadata
associated
with
with
withdrawals,
especially
if
you
have
distinct
columns
in
use.
They
all
deal
with
no
storing
column,
name
in
column
values.
A
Sometimes
metadata
is
higher
in
footprint
than
the
actual
data
itself.
So
that's
what
we're
very
we're
very
stingy
with
that
footprint
like
we
store
columns,
we
stole
column
names,
annoying
one
bite,
and
we
do
that
ouran
serialization
and
by
the
race,
so
they
all
contribute
no
compaction.
You
help
compaction.
You
help
at
all
these
data
structures.
Memory
just
make
it
more
efficient.
A
A
A
A
So
we
have
our
own
restful,
sparse,
eagles
/
shark
server,
which
is
an
open
source
project.
The
link
is
over
there
now
github,
a
teacher
and
that
fuels
this
this
UI,
so
yes,
we're
running
sequel,
queries,
I
should
say:
hi,
p,
ql,
queries
through
this
interface
and
depending
on
the
engine
behind
sparse,
equal
or
shark,
that
works
on
data,
that's
in
store
by
hive
or
or
in
tachyon,
but
we
only
do
that
in
the
warehouse.
A
When
you
run
large
queries
in
and
in
the
warehouse
all
we,
it
would
spark
as
fast
as
it
is
that
if
you
want
now
really
Lolita
C
high
throughput,
now
we're
talking
sub-second
latency,
then
you
will
be
rough
exporting
to
Sandra.
But
when
we
do
that,
we
we
pick
exactly
how
many
rows
now
imagine
you
have
1
billion
records
with
100
fields
in
the
world
house.
It's
very
likely
that
no
dashboard
that
you're
building
is
going
to
need
to
expose
all
the
fields
and
all
the
one
beta
records
now.
A
So
then
you,
through
this
tool,
you're
allowed
to
you're,
allowed
to
pick
now
which
feels
in
the
mapping.
Now
you
can
hide.
For
example,
here
we
I
was
able
to
hide
tighter
than
a
dress,
so
I'm
only
transferring
these
two,
because
all
I
need
is
to
look
up
and
NP.
I
get
back
a
name,
that's
going
to
be
faster.
Api
then
retrieving
the
entire
not
set
of
100
fields,
for
example,
and
then
discarding
them
right.
So
that's
why
it's
what's
key.
Is
that
the
application
engineer
no
defines
the
rear
access
pattern
in
these
mappings.
C
Question
so
look
at
the
diagram
flowing
it
like
the
architecture
diagram.
Do
you
guys
do
stream
processing
for
input
I
saw
like
Kafka
over
there?
Is
that
the
way
you
guys
do
streaming
is
that
you
get
supports
streaming
input
for
for
data
for
external
data
such
as
like,
if
you
think
of
like
Twitter
feeds,
or
something
like
that
at
continuously
streaming
data
into
your
storage?
C
A
The
answer
is
yes,
we
are,
we
have
to
type
of
ingestion
the
poll
ingestion,
but
you
saw
the
tool
that
no
connects
to
external
data
sources
and
you
have
a
spark
job
that
is
triggered
in
it.
Schedule
then,
and
connects
and
pulls
a
day
in.
We
also
have
push
push
ingestion
ap
is
exposed,
they
are
storing
data
into
Kafka
and
then
there's
a
sports
streaming
job
triggered
that's
going
to
pulled
it
out
of
Kafka
that
sport
job
is
running
in
the
cluster.
A
Now,
alongside
other
analytical
workloads
could
be,
machinery,
algorithm
could
be
other
bad
jobs
all
made
and
all
managed
by
under
mesos.
So
we
do
use
that
we
have.
These
api's
are
published
it's
just
as
just
an
API.
That's
published,
there's
no
tool
there
too.
I
know
you
why
to
show
you
that
that's
a
very
good
question.
Yeah.
A
D
A
That's
a
very
good
question,
so
the
pole
pole
ingestion
model
out
of
the
box.
Today
we
only
have
a3
and
s3
energy
gfs
is
being
worked
on
what
we
were
doing
it
integrating
talent
talent,
which
now
has
an
advertised
450,
+
connectors,
and
then
we
want
to
be
able
to
know
configure
talent,
jars
really
applications
to
consume
data
from
external
sources.
So
we
rely
on
accounting
application
to
be
written,
but
then
push
to
us
and
then
we
managing
it
is
you
know
that
talent
job?
A
So
that's
how
we
going
to
support
multiple
data
sources,
but
most
of
our
you
know,
customer
needs
they
just
narrowed
down
to
files
in
SFTP
or
s3.
So
now,
more
recently,
a
GF
s,
okay,
another
HDFS
cluster.
So
we're
building
support
for
that
and
I
guess.
I
didn't
mention
this.
Now.
All
all
traffic
goes
to
these
two
servers
here:
jaws,
which
is
our
No
HTTP
layer
on
top
of
particular
sharp
and
our
own
spot.
A
A
job
server
and
both
of
these
are
open
source
project
and
our
github
you're
welcome
to
download
them
and
use
them
in
contribute
back,
but
what
we
did
with
the
spot
observer
now
we
took
the
yellows
job
server
and
we
saw
the
provenance
park.
It
hasn't
been
even
solved
today
in
this
park
or
which
is
running
multiple
jvn
to
the
multiple
context.
In
the
same
jvm,
like
you,
have
a
web
server
yield
you're
triggering
two
different
spark
applications.
They
will
not
run
at
some
point
if
both
of
them
are
trying
to
write
your
first.