►
Description
Presto, an open source distributed SQL engine, is widely recognized for its low-latency queries, high concurrency, and native ability to query multiple data sources. Proven at scale in a variety of use cases at Airbnb, Comcast, GrubHub, Facebook, FINRA, LinkedIn, Lyft, Netflix, Twitter, and Uber, in the last few years Presto experienced an unprecedented growth in popularity in both on-premises and cloud deployments over Object Stores, HDFS, NoSQL and RDBMS data stores. Delta Lake, a storage layer originally invented by Databricks and recently open sourced, brings ACID capabilities to big datasets held in Object Storage. While initially designed for Spark, Delta Lake now supports multiple query compute engines. In particular, Starburst, developed a native integration for Presto that leverages Delta-specific performance optimizations. In this talk we show how a combination of Presto, Spark Streaming, and Delta Lake into one architecture supports highly concurrent and interactive BI analytics. Furthermore Presto enables query-time correlations between S3-based IoT data, customer data in a legacy Oracle database, and web log data in Elasticsearch.
About:
Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unifie...
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Alright,
hello,
everyone-
this
is
Camille
CTO
and
co-founder
at
starburst
and
I
will
be
talking
about
powering
interactive
VI
analytics
with
Bristol,
and
that
data
like
today,
all
right
so
in
this
presentation.
I
first
want
to
introduce
presto
for
those
who
don't
know
what
that
is,
and
a
little
bit
of
starburst
and
what
we
do
here
to
help
enterprise
adoption
of
Westar.
A
You
know
the
main
topic
of
my
presentation
is
the
Delta
Lake
integration
that
we
done
for
Forrester
and
and
then
sort
of
show
how
we
can
combine
Forrester
and
data
breaks,
spark
and
Delta
together
in
one
data
path
from
architecture
and
how
to
efficiently
use
the
best
capabilities
of
both
technologies
and
then
finally
show
real
use
cases
where
that
combination
actually
delivers
best
results
for
for
your
team.
So
with
that,
it's
going
so
presto
and
starburst.
A
First
itself,
open-source
community
driven
project
and
its
core.
It's
a
high-performance,
NPP,
sequel
engine.
It
was
designed
specifically
to
be
geared
towards
interactive
analytics
and
perform
on
cc-cold
analytics
over
large
scale
data
ranging
from
gigabytes
on
the
low
end
to
to
petabytes
on
high
end,
and
the
design
goal
for
a
Presta
was
also
to
to
help
with
high
concurrency
queries
and
fast
performance.
A
Overall,
one
of
the
unique
capabilities
of
faster
is
that
it
provides
full
separation
of
computational
and
storage
layers,
so
that
allows
you
to
scale
those
two
independently,
add
more
storage
without
adding
compute
nodes
and
then
boosting
compute.
If
you
need
to
perform
lots
of
analytics
and
then
scaling
back
when
when
you
don't
need
them
at
that
capacity,
so
allows
you
to
manage
cost
and
performance
elastically
and
and
cost
effectively.
A
Also
crystal
itself
is
just
a
computer
layer
right.
It
accepts
a
sequel
statement,
does
all
the
eyebrow
execution
of
the
query
and
then
it's
done
right,
there's
nothing
in
fest
about
storing
the
data.
So
you
have
to
kind
of
bring
your
own
storage
and
storage
this
most
efficient,
effective
for
your
use
case
and
most
appropriate.
A
A
A
So
gives
you
a
lot
of
flexibility
and
you
have
to
be
tied
any
specific
platform
and
you
can
move
pressed
analytical
system
on
one
place
to
the
other
angle,
were
word,
your
center
of
gravity
for
de
Guise,
so
first
has
been
around
for
some
time.
It's
been
about
seven
years.
This
was
originally
developed
by
the
team
at
Facebook
and
got
initial.
Adoption
is
a
large
Silicon,
Valley
Internet
companies
and
then
progressively
expanding
more
and
more
companies
globally,
different
verticals
and
news
cases.
A
So
you
can
see
some
bloggers
here
on
the
screen
that
just
gives
you
a
sense
of
you
know,
presence
applied
pretty
much
anywhere
for
psychological
purposes
and
it's
been
the
fastest-growing
sequel
engine
among
any
anywhere
in
the
open
source
for
sure
you
can
run
at
a
massive
scale,
so
some
of
the
larger
deployments,
especially
its
Facebook
Links,
and
lived
talking
about
hundred
machines,
whirring
petabytes
of
data
and
running
thousands,
hundreds
of
thousands
of
queries
a
day.
So
it's
really
really
impressive
from
what
those
companies
are
doing
and
pushing
the
limits
and
scale
and
performance.
A
A
As
I
mentioned,
you
know,
ease
abuses,
super
4
into
enterprise
settings,
so
we
manage
configuration
of
a
scaling,
H,
a
monitoring
and
just
deployment
in
all
those
different
environments,
simplified
orchestrated
effectively,
and
all
this
is
obviously
wrapped
into
as
providing
support
for
presto
and
because
we
have
the
largest
team
of
experts
in
the
world.
We
have
first,
the
creatures
behind
company
right
now
and
provide
hotfixes
security
patches
24
by
7,
support
and
I
know
the
thing
that's
also
very
important,
and
it's
also
true
for
beta
breaks
like
we
are.
A
The
leading
contributor
and
committee
compressed
itself
the
open
source
project
all
right,
so
so
we
are
driving
the
roadmap
and
enhancing
quest
to
get
you
more
performance,
more
scale,
more
functionality
in
the
open
source
and
then,
if
you
choose
to
go
with
our
platform
and
managed
enterprise
offering
and
that's
where
you
get
additional
benefits
as
well.
Ok,
so
so
you've
I've
talked
about
how
priced
I
can
connect
to
many
different
sources
already.
A
So
why?
Why
did
data
like?
We
obviously
got
interested
in
in
the
Delta
like
when,
when
it
was
first
announced
by
little
bricks
a
couple
years
ago,
then
subsequently,
when
it
was
open
sourced
last
year
and
built
alike,
is
really
really
exciting
technology
for
many
reasons,
so
I
just
mentioned
several
of
those
here
which
I
think
are
fundamental.
A
First
of
all,
it
gives
you
the
acid
properties
over
the
data
Lake,
and
this
is
huge
right.
It's
massive,
because
you
can
now
go
and
and
delete
individual
rows
and
update
individual
rows
and
and
incidental
rows
effectively
over
your
data,
like
which
no
in
the
past
was
quite
cumbersome
and
and
complex
frameworks.
To
do
that.
That's
now
simplified,
everyone
can
treat
object,
storage
from
as
essentially
a
database
table,
which
is
which
is
amazing.
That's
how
we
want
to
work
and
that's
how
presto
here's
the
world
so
that's
great.
A
A
A
The
other
supports
object,
storage,
natively
on
on
as
free
adls
and
an
address
storage,
which
is
where
the
world
is
going
with
when
storing
large
amounts
of
bacon
these
days.
That's
that's
amazing.
On
top
of
all
those
basic
fundamental
benefits,
there
are
really
great
features
around
scheme,
evolution
and
even
time
travel
allowing
you
to
see
you.
What
would
be
the
answer
to
this
query?
A
If
you
are
in
fact
a
day
ago
and
I
think
that's
that's
very
interesting
for
for
in
many
workloads
and
especially
in
the
analytic
space,
and
then
you
know
just
to
further
show
the
benefits
of
Delta
having
the
dedicated
metadata
information.
That's
outside
of
highlighter
store,
having
statistics
on
the
data
that
helps
out
with
performance,
because
you
can
skip
the
data
and
you
know
if
you
have
data
arranged
in
a
special
order,
but
also
helps
to
speed
up
many
queries.
A
So
I
think
we
all
are
very,
very
thankful
to
data
breaks
for
inventing
Delta
and
making
that
popular
file
format
is
getting
lots
of
adoption.
Many
of
our
joint
customers
are
also
users
of
Delta,
so
it
really
made
make
a
lot
of
sense
for
for
us
at
starburst
to
enhance
presto
and
allow
it
to
to
also
equate
Delta
effectively.
A
As
I
mentioned,
stuber's
developed
a
native
purse,
Delta
Lake,
connector
and
I
would
like
to
acknowledge
great
help
from
engineers
at
data
bricks
that
assisted
us
with
that
integration
and
provided
further
details.
In
addition
to
the
official
specification
of
how
to
implement
this
effectively,
so
we
decided
to
build
a
native
connector
written
from
computer
from
scratch
right,
so
there
used
to
be
a
legacy
solution
that
I
was
manifest.
Files
and
hive
might
restore
integration
and
simulate
access
from
presto
to
Delta.
A
I
was
very,
very
inefficient
because
it
wasn't
taking
advantage
of
any
of
the
inherent
data
Delta
like
properties.
So
in
this
implementation
we
actually
natively
read
the
Delta
transaction
log.
We
perform
data
skipping
based
on
the
metadata
read
from
from
from
Delta,
and
we
are
able
to
fetch
the
Mystics
about
the
data
and
basically
leverage
those
as
an
input
to
the
optimizer
and
that
allows
us
to
effectively
perform
joins
among
both
the
tables
as
low
as
those
other
tables
and
other
other
team
was
coming
from
different
sources.
A
Ok,
so
once
we
build
the
very
first
version
of
that,
we
accuse
obviously
how
that
performs
to
this
legacy
solution
which
basically
treated
Delta
as
a
collection,
Alfredo
perky
files.
So
we
run
on
standard
age
benchmark
and
across
22
queries
in
this
benchmark.
We
so
on
average,
except
for
for
all
the
queries
and
obviously
those
queries
are
doing
much
more
than
just
real
data.
A
So
this
in
the
fastest
query-
and
do
we
observe
in
this
metric,
which
was
a
single
table
scan
doing
some
aggregation,
was
like
she's,
showing
six
acts
performance
boost
in
there
and
with
the
native
reader.
So
that's
that's
pretty
substantial
and
and
once
we
get
the
pollination
done,
efficient
implementation
of
the
Delta
reader.
A
We
also
share
this
with
our
preview
with
our
early
customers
and
the
feedback
we
got
was
actually
even
more
enthusiastic
because
they
were
reporting
speed.
Ups,
you
know
over
10
X
4
for
the
native
reader
versus
the
previous
solution.
So
this
is
dry.
Seductions
helps
house
for
the
performance
overall,
and
it
makes
users
more
happy
to
come
back
and
run
more
queries
and
do
more
analytics.
So
I
definitely
encourage
everyone
to
try
out
this
native
integration
of
Delta
with
starburst
Presta
and
the
link
and
documentation
link
here
on
the
screen.
So.
A
You
know
having
that
is
great,
obviously,
engineering
edition,
yet
another
connector
for
the
presto,
but
how
you
soft.I
all
those
things
together
right
to
work
together
effectively
in
one
environment
and
I
thought
I
would
spend
a
few
slides
on
this
topic:
how
to
effectively
leverage
data
reefs
and
starbursts
one
architect,
so
this
terrorist
platform
itself,
obviously
at
the
center
of
that,
is
Presta
as
this
fast
sequel.
Here
that's
talked
to
many
sources,
we've
done
in
addition
to
just
leveraging
open
source
press
or
we
build.
You
know
additional
connectors
to
more
sources.
A
We
enhance
support
for
additional
SP,
compatible
storage
engines
for
premises,
as
well
as
the
clouds
all
the
clouds,
obviously,
and
today
and
press.
It
can
also
talk
to
your
modern,
no
sequel
stores
such
as
manga
elastic
Cassandra,
etc.
I
can
and
so
having
a
tall
one
to
reach
to
all
those
different
sources,
both
legacy
and
and
modern
places
where
you
may
store
your
data
is
really
really
powerful.
However,
there's
always
always
challenge
well
with
all
the
diversity
of
sources.
A
How
to
do
it
effectively,
how
to
do
it
securely
and
how
to
manage
the
whole
experience
for
the
end
users
right.
So
this
called
as
part
of
our
platforms.
We
call
it
data,
consumption
layer
and
big
piece
of
that
was
building
a
global
security
mechanisms
that
will
govern
secure
access
to
all
the
sources
from
presto
and
and
also
from
the
users
that
are
occurring.
This
data
by
oppressed
and
of
managing
permissions
masking,
has
sensitive
information,
ensuring
data
encryption
both
for
reading
the
and
data
encrypted
at
rest,
as
well
as
encrypting
data
and
flight.
A
When
it's
moving
from
the
source
to
Preston
and
no,
we
Winterfest
a
cluster
and
then
between
the
client
tool
and
the
past
right.
You
can
all
these
all
the
queries.
You
can
have
very
fine-grained
access,
control
down
to
the
table
and
column,
and
actually
you
cannot
survive
role
or
all
filters
and
further
restrict
access
southern
things.
A
So
all
of
those
things
are
part
of
starburst
platform
edition,
and
on
top
of
that,
we
found
integrations
and
certifications
with
a
number
of
your
favorite
BI
tools,
there's
no
look
at
RBI
click,
cetera
and,
and
so
enabling
that
class.
All
the
modern
tools,
and
that
can
issue
sequel
starting
from
Jupiter
notebook,
superset,
read
ash,
which
I
think
right
now
is
part
of
data
breaks.
All
those
tools
can
talk
and
speak
to
restore
natively
and
gives
all
the
users
power
of
the
the
broad
set
of
connectors
and
security
features
of
starbursts.
A
So
that's
on
the
starboard
side,
and
now
this
question
aloud,
like
you
know,
I
have
spark
I
beta
breaks,
I
have
first
and
starbursts
like
how
do
these
things
work
together
was
the
best
way
to
effectively
leverage
both
right
and
what
we
see
we
all
live
deployments
with
our
customers.
Is
that
pretty
much
everyone
who
ever
just
arbors
and
impressed
they
also
leverage
spark
and
often
doubts,
data
bricks
spark
right
and
the
reason
is
that
I
think
those
technologies
excel
at
different
things
and
complement
each
other
very
well.
A
Yeah,
so
if
you
look
look
sort
of
holistically
at
the
data
ingestion
analytics
ecosystem,
this
is
all
coming
together.
By
having
you
know
your
raw
data
sources
all
being
ingested
by
a
data
breaks
and
spark
into
detail,
like
you
know
these
days,
like
specifically
right,
but
you
can
also
just
put
your
your
data
on
you
know
either
Amazon
s3
or
as
your
Els
and
and
you
can
run
all
the
machine,
learning
guy
and
all
flow
or
say
to
make
her
over
this
data
for
for
those
use
cases.
A
While
if,
if
you
want
sequel
over
this
data
plus,
you
know
correlate
this
information
with
with
lots
of
different
data
sources,
but
there
are
DBMS
new
sequel
and
all
over
the
place.
They're
like
Star,
Wars
and
presto
are,
is
the
perfect
answer
right
and
provides
so
higher,
more
responsive,
having
currency,
more
responsive
sequel
access
and
allows
you
to
leverage
or
the
di
tools
and
secure
editors
and
reader
super,
say:
Jupiter
notebook
and
your
favorite
tools
for
analytical
purposes
for
the
more
classical
unknowingly
purposes.
A
A
Also
in
practice,
so
we
work
closely
with
an
open
platform
for
the
Aqua
his
deployment,
and
we
see
people
playing
event
is
you
know
pretty
much
anywhere,
but
as
I
mentioned,
you
can
obviously
do
it
in
many
different
ways
as
well
at
your
company
and
and
so
if
we
simplify
all
that
configuration
deployment
management
sufficiently.
So
it's
really
very
very.
A
A
Okay,
so
now
bringing
all
this
together
right,
it's
all
great
architectural
technology
was
the
real
use
cases
that
we
can
show
here
right
so
they're
the
one
use
key
that
we
want
to
advocate
for,
and
we've
seen
this
already
being
leverages.
You
know
in
a
joint
architecture,
data
breaks
and
starbursts.
A
A
If
needed
right.
If
your
analyze
data
scientist
refers
to
look
at
the
very
raw
data,
that's
also
possible
from
compressed
and
now,
while
we
have
also
made
that
being
ingested
into
this
there's
data
like
it's
not
going
to
be
everything
you
have
potentially
in
your
ecosystem,
we've
seen
a
lot
of
companies
coming
with
data
like
started
this
and
having
the
center
of
gravity
there.
A
So
Adam,
as
I
mentioned,
will
be
doing
so.
The
real-time
ingestion
of
event,
data
and
I'll,
be
showing
this
in
a
demo
in
a
moment,
will
be
doing
also
the
the
hourly
jobs
of
hoping
that
the
more
traditional
enterprise
data
and
then
reaching
out
your
fukui
data
sources
as
well,
and
we
modify
this
and
refine
this
and-
and
we
prepare
this
data
for
further
outdoor
means,
as
appropriate
in
in
our
overall
architecture.
A
So
we're
you
know.
Obviously
the
thermosphere
provides
this.
This
single
access
later
right.
It's
no
longer
true
that
you
have
already
been
in
one
place,
so
there's
no
single
source
of
truth
for
day
care.
This
single
access
point
the
queried
is
date
right,
but
so
visualize
and
query
environment
is
where
the
power
of
questa
is
right.
A
If
you
give,
if
you're,
given
access
to
this
elasticsearch,
you
can
actually
push
down
some
process
from
there
and
find
appropriate
data
by
by
doing
that
and
external
it
takes
essentially
and
bringing
that
back
to
the
relational
world
and
unpressed
to
service
can
provide
so
far
a
global
access
control
for
sources.
So
analysts
are
appropriately
privileged
to
the
data.
That's
that's
meant
for
them
to
be
arised
so
now
in
the
in
this
demo,
know
and
users
will
be
leveraging
API
tools,
sequel,
edit.
A
First
look
at
a
blonde
power,
bi
leveraging
connectors
like
JDBC
drivers,
ODBC
and
wagons
for
other
languages
and
wizard
of
mouch
want
to
show
all
of
this.
You
know
in
a
demo
and
briefly
comment
how
you
put
all
those
things
together.
Ok,
so
we
are
in
a
database
notebook
here
and
we
have
obviously
as
we
bracket
mounted
and
we
are
setting
up
a
structured
streaming
ingestion
from
from
a
Kafka
topic
into
into
a
stream
that
will
then
save
into
a
Delta
table
so
sitting
up
here.
A
Making
sure
that
connectivity
is
is
live
stream,
checking
the
schema,
making
sure
it's
matching
what
to
expect
and
now
basically
issue
a
command
to
receive
this
data
and
save
into
a
Delta
like
calm
and
very
fine
that
the
stream
is
actually
live.
Okay,
so
we
have
ingestion
going
on
now
we
are
switching
to
a
DB
view,
which
is
a
secret
or
tool
in
which
we
have
mounted
all
different
data
sources
such
as
Delta
Lake
and
we'll
be
trying
out
the
query
Delta
from
Malaya
Prestone
from
from
DB
per
client.
A
That
came
up
very
quickly,
as
you
can
see
now,
showing
the
same
for
elasticsearch.
So
this
is
vastly
set
up
in
I
was
in
cloud.
We
can
query
that
very
quickly
as
well
individually,
so
that's
perfect
and
then
our
zone
RDS.
We
have
an
Oracle
system
with
some
customer
information
and
and
that's
together
now
bringing
all
this
together
into
one
sequel
statement.
We
can
sort
of
correlate
this
information
and
do
the
join
between
orders,
customers,
parts
and
execute
the
sequel,
and
here
in
the
first
web
UI
we
can
see
how
query
is
progressing.