►
Description
Delta Lake has quickly grown in usage across data lakes everywhere due to the growing use cases that require DML capabilities that Delta Lake brings. Outside of support for ACID transactions, users want the ability to interactively query the data in their data lake. This is where a query engine like Trino (formerly PrestoSQL) comes in. Starburst provides an enterprise version of the popular Trino MPP SQL query engine and has recently open sourced their Delta Lake connector.
In this talk, Tom and Claudius will talk about the connector, its features, and how their users are taking advantage of expanding the functionality of their data lakes with improved performance and the ability to handle colliding modifications. Get started with this feature-rich and open stack without the need of a multi-million dollar budget.
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/data...
Instagram: https://www.instagram.com/databricksinc/
A
A
A
Okay,
all
right,
we'll
we'll
talk
about
trino,
then
we're
going
to
talk
about
how
it
works
with
delta
lake
and
then
we'll
talk
a
little
bit
about
how
people
are
actually
using
this
in
the
real
world
to
to
solve
problems
and
then,
before
after
we
go
a
little
bit
into
what
is
coming
next
and
how
we're
going
to
build
this
out
further,
then
tom
is
going
to
actually
show
you
this
in
action.
So
you
can
you
don't
have
to
believe
what
I'm
saying
all
right.
A
So
what
is
streeno
trino
is
a
single
point
of
access
for
all
your
data
and
the
way
this
works
is
you
have
a
coordinator
and
a
number
of
worker
nodes,
the
largest
number
of
record
nodes.
We
have
do
you
know
what
that
is,
I
think
1200
I
think
1200
or
so
yeah
yeah,
okay
yeah.
So
let
me
give
you
a
little
aside
here.
You'll
see
hive
down
in
the
corner
there
right.
That
was
actually
the
original
use
case
when
our
developers
originally
built
trino.
A
They
were
working
at
facebook
and
they
have
sort
of
facebook
scale
data
on
hive,
and
the
issue
was
that,
on
a
good
day,
their
analysts
could
run
six
queries.
So
martine
dane
david
and
eric
got
together.
They
they
built
out
trino.
It
was
called
presto
at
the
time
and
promptly
crashed
facebook
because
they
saturated
the
network
links
and
the
interesting
thing
is.
The
response
was
not
to
turn
off
the
queries.
It
was
to
upgrade
the
network
infrastructure
because
and
there's
two
important
things
out
of
this
one.
A
Is
that
often
you
can
get
more
performance
out
of
your
data
sources
than
you
think
you
can
potentially
and
two.
If
there's
value
in
that
data,
then
you
should
pay
for
the
physical
infrastructure
to
be
able
to
get
that
value
out
and
to
be
able
to
use
it.
So,
let's
come
back
to
how
this
works,
you
take
any
sql,
capable
client
and
it
could
be
db,
beaver
or
command
line
tool
or
python
or
whatever
it
is.
A
Few
things
to
note
here,
one
is
that
we've
got
a
bunch
of
connectors
right,
there's
a
whole
lot
of
them
and
we
can
aggregate
this
information.
We
can
federate
across
them.
So
you
can
do
a
join
across
two
different
databases
that
are
physically
separated
and
number
two.
You
see
these
lines
for
hive
and
delta
lake
when
the
data
source
supports
it,
which
not
all
of
them
do
we'll
paralyze
on
them,
so
that
we
can
pull
the
data
in
faster.
A
So
how
does
this
look
like,
specifically
with
delta?
So
if
we
drill
in
on
this
lower
left
hand
corner,
we
can
see
a
little
bit
better.
What
this
looks
like
when
you're
actually
going
to
deploy
this-
and
this
is
much
closer
now
to
what
tom
is
going
to
show
you.
So
you
pick
object,
storage
and
we
don't
really
care
what
it
is.
A
On
top
of
that
not
falling
off,
you
have
a
sql
compute
engine,
so
this
can
obviously
be
sparked
with
delta.
Like
duh
right,
you've
all
seen
this
or
it
can
be
something
like
trino
or
starburst,
and
they
work
together
right
as
one
of
the
use
cases
that
they
were
trying
to
hit
when
they
built
delta
lake
was
that
you
can
have
multiple
things:
writing
to
the
same
delta
processes,
writing
to
the
same
delta
table
without
them
destroying
everything,
and
so
we
can
do
that
across.
A
So,
let's
move
on
now
who
actually
uses
this
emis
is
one
of
the
largest
healthcare
data
providers
in
the
uk
they
cover
10
000,
clinic
groups,
100
milli,
120
million
patient
records,
both
current
and
historical,
and
their
use
of
this
is
initially
around
privacy
and
security.
As
you
can
imagine,
this
is
all
very
sensitive
data,
it's
people's
health
status,
and
so
they
need
to
be
able
to
segregate
this
out.
A
There's
another
project
which
is
actually
even
more
interesting
clinical
researchers
need
to
target
their
patients
very
carefully,
and
this
is
very
important
for
two
reasons:
one
for
safety.
You
absolutely
need
to
make
sure
that
people,
you're
testing,
aren't
going
to
get
killed
in
your
trial
or
suffer
injury
and
number
two.
A
The
other
one
we'll
talk
about
today
is
comcast,
I'm
sure
many
of
you
have
had
some
interaction
with
comcast.
They
work
with
databricks
as
well,
and
they
obviously
have
a
ton
of
data.
They
maintain
data
on
what
is
being
shown
in
any
given
location,
so
in
san
francisco
and
in
new
york
they're
going
to
show
different
things,
they
maintain
records
on
what
the
individual
is
doing
right.
So
who
are
you?
Where
do
you
live?
How
many
people
in
your
house
and
so
on,
and
then
what
is
actually
being
watched
by
these
people?
A
A
So
where
are
we
going
from
here?
We
open
sourced
the
delta
lake
connector
early
in
march.
Was
it
I
think,
and
that's
been
very
well
received
since
then,
the
bulk
of
the
work
has
actually
been
on
testing,
it's
not
that
it
didn't
work
before
we
had
it
in
starburst
and
we
were
using
it
with
enterprise
customers.
A
We
absolutely
had
to
have
a
very
rich
set
of
automated
tests
that
were
not
just
in
our
enterprise
product,
but
in
the
community
where,
when
anyone
contributes
to
our
delta
late
connector
and
when
we
we're
working
with
the
databricks
team
on
their
standalone
reader,
so
that
we
can
incorporate
that
eventually
as
well,
all
that
needs
to
be
tested
so
that
when
people
actually
deploy
it,
it
works
the
way
we
need
to,
and
then
we
work
with
them
on
these
smaller
incremental
pieces.
That
may
not
be
obvious
right.
A
So
I'll
give
you
an
example,
the
second
one
here
you
can
mark
a
delta-like
table
to
be
append
only
so
nobody
can
delete
anything
and
nobody
can
update
any
rows
with
separation
of
storage
and
compute.
The
compute
engine
needs
to
know
about
that,
and
it
needs
to
honor
that
so
we've
been
able
to
add
that
coming
up
are
two
major
features
that
that
we're
going
to
be
wrapping
up
this
year.
Number
one
is
time,
travel,
so
does
everyone
here
know
what
time
travel
is
some
people
yeah?
There
are
great
movies,
several
great
movies
about
it.
A
Delta
lake
lets
you
because
of
how
delta
lake
stores
changes
to
the
table.
You
can
actually
go
back
and
say
what
was
the
state
of
the
table
at
some
arbitrary
point
in
the
time
in
the
past.
So
the
obvious
use
case
is:
oh,
my
god.
At
midnight
last
night
somebody
destroyed
my
table.
Let
me
roll
back
to
11
59
and
I'm
good
to
go.
That's
the
obvious
one.
We've
heard
some
people
say
no,
no.
We
actually
need
this
for
auditing
every
once
in
a
while.
Our
regulator
comes
to
us
and
says:
hey.
A
A
I
don't
know,
something's
wrong
with
this,
but
it
was
run
yesterday
and
since
then,
all
your
data's
changed
and
you're
not
really
sure
where
the
source
of
the
problem
is
so
people
are
saying:
well,
it
would
be
really
cool
if
I
could
just
roll
back
to
when
the
problem
occurred.
Have
exactly
that
data
set
I'll
c-task
that
into
some
other
table.
Now
I
can
work
on
it
now
I
can
debug
it
and
I'll
be
ready
to
go
as
soon
as
I'm
done.
A
The
second
one
is
materialized
views.
Now
everyone
is
probably
familiar
with
materialized
views
as
a
single
database
concept.
Right,
you
do
a
view
and
then
you
stick
it
in
a
table
and
it
sits
there
and
it
has
the
advantage
that
it
adds
some
syntax.
When
you
refresh
a
view,
it
remembers
how
you
created
it,
so
it's
not
just
a
simple
ctas
with
trino.
We
can
federate
that,
so
the
information
from
the
view
doesn't
have
to
come
from
the
data
source
that
you
eventually
put
it.
A
We
have
all
these
connectors,
so
somebody
may
say:
well
I
really
want
to
do
some
big
ml
project
right.
I
need
a
data
set
and
I
need
clean
data
sets
and
I
don't
want
to
deal
with
the
data
engineering
part
of
it.
You
can
use
materialized
views
to
join
your
data
from
teradata
and
sql
server
and
google
cloud
and
whatever
put
them
into
delta
lake.
A
Have
that
run
as
a
refresh.
We
can
actually
do
incremental
refreshes
so
that
when
there's
a
change
in
one
of
the
source
tables,
we
don't
have
to
kill
everything
and
redo
your
one
terabyte
a
materialized
view.
We
can
just
add
the
new
pieces
on
and
then
any
other
process
can
use
them.
We've
seen
this,
we
have
a
banking
customer
who
is
using
materialized
views
to
power,
a
dashboard
which
took
the
query
times
down
from
10
minutes
to
a
minute
and
a
half,
and
there
was
a
lot
of
data.
A
They
couldn't
get
it
down
below
that,
just
because
the
sheer
amount
that
was
going
in,
but
they
were
pretty
pleased
with
that,
so
we've
gone
over.
What
trino
is
how
it
works
with
delta
lake?
How
that
can
get
you
a
lake
house
on
completely
open
hardware,
completely
open,
sorry,
completely
open
software
and
how
people
use
this
in
the
real
world.
Before
I
hand
this
off
to
tom,
let
me
see
if
there's
any
questions
here
before
we
go
on
and
we
can
have
more
questions
at
the
end.
Yes,
please.
A
We
have
a
lot
and
we're
we're
starting
to
go
through
and
more
carefully
catalog
all
the
connectors.
So
you
know
exactly
what's
supported
in
the
various
ones
because
it
usually
what
we'll
do
is
we'll
build
reads
in
first
and
then
we
get
the
performance
up
to
there's,
usually
some
some
work
on
that
and
then
we'll
start
adding
rights.
Once
we
have
the
basic
crud
operations
in
then
we
start
adding
in
the
bells
and
whistles
of
a
particular
data
source.
A
B
Answer
your
question
yeah.
I
think
I
think
recently,
just
within
the
last
couple
months,
you've
done
a
lot
of
benchmarking
and
delta's
way
faster.
Now,
and
it's
mostly
because
the
s3
reads
so
you
have
to
do
an
s3
listing
on
everything
you
don't
have
to
do
that
with
delta.
So
that's
the
biggest
thing
is
when
you
have
a
whole
bunch
of
files,
10
20,
100,
000
files,
we're
seeing
huge
performance
reads.
So
that's
why?
Just
for
and
I'll
show
you
a
little
demo
of
it.
B
D
A
A
One
last
point:
I'm
going
to
hand
this
to
tom
here
the
a
lot
of
the
speed
improvements
on
delta
and
and
for
a
lot
of
for
the
the
set
of
connectors
where
we
separate
storage,
compute,
so
delta
and
hive,
and
things
like
that,
come
down
to
table
stats
because
once
you've
separated
these
things,
the
engine
really
needs
to
know
what's
going
on
in
the
table,
so
it
can
intelligently
create
these
query.
Plans
know
what
piece
to
send
from
where
know
which
files
to
pick
up.
A
B
All
right
sounds
good.
Let's
see
here,
I
have
I'm
using
a
product
that
we
have
called
galaxy,
which
is
basically
hosted
trina,
so
we're
going
to
play
like
I'm
sitting
on
adreno,
which
is
easier
for
me
to
deploy.
What
I'm
going
to
go
through
here
is
there's
actually
one
more
slide.
Isn't
there
there
we
go.
Let
me
go
through
this
real
quick
in
this
example.
This
is
typically
what
we
see
from
a
lot
of
our
customers.
B
We
have
data,
that's
coming
in
from
whatever
different
data
sources.
We
have,
let's
say
a
database
that
has
customer
information
in
and
then
some
kind
of
sales
data
source,
that's
just
populated,
json
files.
So
in
this
example,
I
have
landed
raw
data
files
that
are
coming
through
json
and
I
kind
of
switch
this
up
at
the
last
second,
because
I
kind
of
want
to
show
you
some
federation.
So
actually,
my
customer
information
is
going
to
come
from
a
postgres
database
and
then
the
sales
data
is
going
to
be
landed
on
s3,
just
with
json
files.
B
I'm
going
to
take
that
and
convert
that
over
to
delta
and
then
from
there
I'm
going
to
do
roll
up
tables
so
sales
by
week,
month
and
year,
and
then
obviously
all
on
aws
using
glue
and
then
I'm
just
going
to
show
you
through
a
quarter
editor
again,
you
can.
You
can
connect
to
this
through
through
a
bi
tool.
So
let
me
show
you
what
that
looks
like
from
the
s3
point
of
view.
So
on
s3.
Let
me
go
back
here.
B
There
we
go
so
s3.
I
have
this
data
being
landed
and
it's
it's
a
directory
per
day.
So
I
have
these
json
files
that
are
landing,
and
this
is
obviously
a
small
demo.
You'd
have
a
lot
more
information,
but
we
all
know
that
json
is
not
very
performant
so
from
here
I'm
going
to
create.
Let
me
go
to
my
query.
Editor.
B
The
first
thing
I'm
going
to
do
is
I'm
going
to
create
an
external
table
that
points
to
this
data
that
sit
out
sits
out
on
sales
land.
So
let
me
go
ahead
and
execute
that,
and
since
that's
just
an
external
table,
there's
not
much
going
on
there
and
usually
I
would
just
run
this
call
system,
because
I
need
to
go.
Tell
glue
that
there's
a
whole
bunch
of
directories
out
there
with
our
partitions,
so
I
I
wouldn't
cheated.
I
have
a
table
called
w,
so
I've
already
done
that.
B
So,
if
I
just
say
select
star
from
that,
so
this
is
the
json
files
that
I'm
reading
out
there.
It's
all
partition
by
day,
so
there's
like
2
000
or
something
like
that.
There's
a
whole
bunch
out
there
in
this
sample
data
set.
So
the
next
thing
I'm
going
to
do
is
I'm
going
to
create
a
delta
table
from
that
table.
So
I'm
going
to
say,
create
table.
B
That's
sitting
out
on
s3,
so
if
I
come
out
here
and
go
back
to
s3,
I
should
have
this
sales
base,
so
there's
nothing
there
yet
because
I
haven't
populated
it
yet
so
I'm
going
to
go
ahead
and
make
this
a
little
bit
faster
with
my
min
partition
and
I'm
going
to
insert
into
demo
sales
base,
that's
my
base
and
I'm
going
to
select
star
from
my
landing
table.
So
this
is
a
just
a
two
note
cluster.
B
This
takes
about
eight
or
nine
seconds
to
populate
so
once
this
goes
in
I'll,
show
you
what
it
looks
like
in
s3
so
get
97
and
that's
it
so
about
754
000
rows.
So
if
I
come
into
sales
base,
this
is
basically
the
same
thing
that
you
just
saw,
but
now
I
have
this
delta
log
table.
So
this
is
a
delta
table.
So
I
have
the
initial
json
file
in
there
when
I
created
the
table
and
then
I
have
that
last
insert.
I
did
with
all
that
data.
B
That's
sitting
out
there,
so
the
next
thing
I'm
going
to
do
here
is
come
down
here.
I'm
going
to
say
just
select
star
just
to
show
you
that
I
can
query
that
table
and
this
is
what
I
would
if,
if
my
end
users
want
to
go
directly
to
the
data,
I
would
direct
them
to
this.
I
want
to
have
them
query
the
json,
because
it's
just
too
slow.
This
is
all
in
part,
k
and
then
delta.
So
it's
going
to
be
much
quicker
for
them.
So
then
here's
my
customer
information.
B
I
have
it
in
a
postgres
database,
so
I
can
just
go
ahead
and
just
connect
and
just
say,
select
star
from
that
postgres
database,
and
this
is
my
customer
information,
so
two
different
options
here
that
I
can
give
my
end
users
based
upon
what
my
slas
are.
So
I
can
join
that
s3
data
that
sells
that
delta
lake
table
with
postgres.
So
if
I
just
join
these
two
tables
together,
this
is
going
to
give
me
the
real
time
information
from
postgres.
B
But
I
may
not
want
to
do
that
because
I
don't
want,
maybe
all
the
users
joining
a
lot
of
data
with
postgres,
because
that's
that's
the
live.
Let's
say
it's
a
crm
system.
So
what
I
want
to
do
is
I
want
to
create
a
delta
table
from
that
customer
table
so
create
customer
base.
The
type
is
delta
just
select
star
from
that,
and
I
could
create
some
another
job
in
here.
That
only
gets
you
know:
change
change,
customer
information,
new
customer.
B
If
I
wanted
to
so
now,
I
have
both
tables
that
are
sitting
out
on
s3,
if
I
say
select
star
now.
If
I
go
back
to
here,
I
have
my
my
customer
base
table,
which
is
now
delta,
so
I
have
two
tables
out
there
that
are
both
delta
and
then
I'm
going
to
create
some
roll-ups.
So
I'm
going
to
create
a
row
up
here
that
is
type
of
delta.
It's
going
to
be
ordered
by
the
the
date
format
it's
going
to
be
by
week.
B
So
this
gives
me
a
weekly
roll
up,
we'll
go
ahead
and
run
that,
and
these
are
all
delta
tables,
because
I
want
to
make
sure
that
I
have
the
like.
We
did
that
question
we
talked
about
before.
I
want
the
most
performance,
the
highest
performance
I
can
so
I
want
to
make
these
delta,
so
I'm
going
to
go
ahead
and
run
the
the
monthly
roll
up,
and
then
this
is
a
yearly
roll
there's
different
ways
you
can
do
ropes.
You
can
just
basically
drop
them
and
then
recreate
them.
You
could
use
views.
B
You
could
do
incremental
updates
to
to
roll
up
tables
whatever
you
want
to
do
so
I'm
going
to
go
ahead
and
select
from
my
weekly
reporting.
So
you
can
see
the
day
of
the
week
the
last
seven
days
of
my
sales
and
then
I've
got
a
monthly.
This
is
going
to
cover
a
year
of
data
of
my
sales
and
then
my
yearly.
So
I
think
I
just
have
2020
data
in
here.
So
that's
my
sales
for
the
entire
year.
So
the
next
thing
to
do
is
what
we
did
is
when
we
created
the
delta
connector.
B
Is
we
just
looked
at
the
spec?
We
worked
with
td
and
and
danny
and
and
the
databricks
folks
to
make
sure
that
we
followed
all
the
spec
correctly.
So
we
added
update
in
here
as
well,
so
we
have
full
dml
support,
so
I'm
going
to
say,
select
star
and
I'm
going
to
grab
this
customer
call
customer
blah
blah
blah.
I'm
going
to
update
that
customer
to
john
dodds,
and
if
I
go
look
out
in
here
and
I'm
going
to
go,
look
in
my
customer
base.
B
I've
got
two
jsons
in
here,
so
one
was
my
first
initial
insert
and
now
I
have
an
update
statement
in
here.
So
let
me
go
back
and
say,
select
and
now
that
customer
is
going
to
be
john
dodd,
so
just
to
point
out
that
we
have
full
dml
delete,
insert
update
as
well,
where
I
know
we're
going
to
be
adding
merge
pretty
soon
as
well,
so
merge
and
time
travel.
I
think
we
miss
out
on
the
roadmap
as
well
and
then
all
the
other
utilities
that
everybody
kind
of
knows
from
from
delta
lake.
B
So
I'm
going
to
do
an
optimize
here
and
this
is
going
to
be
against
the
sales
base
table.
So
we'll
go
ahead
and
optimize
that-
and
this
is
the
same
thing,
that
you
would
do
an
open
source,
delta
lake
or
databricks
as
well.
I
want
to
make
sure
those
small
files
are
hot.
I
have
make
the
combine
those
into
larger
files
as
well
and
then-
and
we
also
have
a
vacuum-
that
I'll
show
you
in
just
a
second
here.
B
B
And
then
my
sales
base,
so
if
I
come
into
my
sales
base,
I'm
going
to
go
pick
out
one
of
these
guys,
since
I
did
that
optimize.
I
have
two
files
in
here.
So
I'm
going
to
go
ahead
and
run
this
vacuum
command.
I
am
going
to
put
it
as
zero
seconds.
We
follow
basically
the
same
spec
that
databricks
did
with
delta
delta
lake,
so
it's
very
very
similar
for
if
you've
used
something
like
databricks
or
something
like
that
or
open
source
to
call
this.
B
So
this
is
gonna
go
through
and
just
vacuum
all
those
old
files
up
there.
Let's
see
if
that
caught
up.
Yet
there
we
go
so
now
it's
gone.
So
that's
finished
that
and
then
the
last
thing
I
want
to
do
is
have
something
that
we
added,
because
our
engine's
a
little
bit
different,
especially
when
it
comes
to
sql.
We
need
extra
extra
statistics.
One
of
them
is
like
ndvs,
which
is
a
number
of
distinct
values.
B
So
if
I'm
going
to
analyze
the
sales
base
table,
so
let
me
go
back
to
sales
base
and
then
under
delta
log.
I
just
have
my
checkpoint
files
that
I
have
now
so
if
I
run
this
analyze
command,
this
is
going
to
go
through
and
basically
count
all
the
different
number
of
distinct
values
that
are
in
that
table
and
we
create
an
additional
directory
called
trinometa
data.
B
So
we
need
this
ex
this
extra
stats
in
here
we
we
could
do
without
it,
but
we
we
find
out
when
we
have,
especially
when
we're
doing,
joins
against
tables
and
stuff.
Our
performance
greatly
increases
when
we
have
these
extra
statistics
that
we
that
we
need
for
our
engine
for
our
cost
based
optimizer.
B
So
that's
about
it.
We
have
optimized
vacuum,
went
through
all
the
the
dml
information,
so
this
is
kind
of
what
we're
seeing
a
lot
of
our
customers
use
in
delta,
for
is
that
either
writing
the
files
either
delta
and
then
getting
us
involved
with
high
concurrency,
especially
with
bi
tools.
Thousands
of
users
querying
the
data
at
the
same
time
and
then
also
creating
the
delta
files
as
well
using
trino,
and
I
think
that's
it.
A
So,
thank
you
tom,
so
our
what
what
tom
just
showed,
which
fortunately
matched
everything
I
said,
yay,
is
that
you
can
use
trino
and
delta
lake
to
have
your
fully
functional
lake
house.
It
works,
it
uses
off-the-shelf
components
and
it's
just
it's
a
very
simple
way
to
enhance
any
delta
lake,
whether
you
already
have
data
bricks
on
it
or
as
in
the
case
of
emis,
they
had
initially
only
used.
They
had
added
data
bricks
later
once
they
started
needing
more
ai
and
ml
capabilities
on
their
data.
A
But
whether
or
not
you
have
this,
it
makes
sense
to
put
reno
on
so
that
if
you
have
high
concurrency
workloads
that
you
need
federation
use
cases,
if
you
have
a
parallelization
that
you
need
or
moving
data
between
data
sources,
so
let's
take
a
little
bit
more
time.
I
think
we've
got
eight
and
a
half
minutes
hey.
What's
your
name.
A
E
B
B
So
I
mean
the
way
I
look
at
it
is
there's
there's
two
big
plays
that
for
for
trina
right
is
the
federation,
so
there's
over
40
connectors
now,
so
we
see
that
really
large
enterprises,
especially
they
have
legacy
source
systems
that
are
around.
They
want
to
do
exactly
what
I
just
showed
you
they
have
data,
but
they
don't
want
to
pull
that
constantly
into
s3.
They
want
that
data
right
now.
So
that's
a
huge
use
case
for
us
is
high
concurrency
federation
story
and
then
basically
data
lake.
I
mean
there
is
overlap
right.
B
Let's
not,
let's
all
be
serious
here
right.
It's
it's
one
thing
they
built.
We
replaced
hive
at
facebook.
That's
what
presto
did
that's
why
it
was
created
in
the
first
place,
and
we
still
continue
to
do
that.
I
still
think
we're
one
of
the
fastest
most
the
high
concurrency
sql
query
engine
on
a
data
lake.
That's
going
to
be
obviously
there's
photon
right,
so
we'll
see
how
that
works
out,
but
I
think
what
we've
really
invested
in
the
last
two
years
that
I've
seen
is
really
the
federation
story.
B
Sure
we
had
connectors,
but
actually
actually
look
in
the
statistics
and
figure
out.
What
do
we
want
to
push
down
to
the
source
system
and
then
have
that
database?
Do
the
join
only
pull
the
data
back
that
I
need
and
then
join
that
with
whatever
s3
or
something
like
that?
That's
when
I've
seen
the
biggest
amount
of
engineering
that
claudius
is
a
part
of
in
the
last
year
and
a
half
you
want
to
add
to
that.
A
Yeah,
so
that
okay,
thank
you.
I
misunderstood
the
question,
so
one
of
the
things
that
if
you
were
when
they
were
talking
about
the
the
sql
support,
they
were
adding
the
federation
support.
They
started
talking
about
the
query
planner.
So
when
we
built
all
these
connectors
initially
to
some
extent
when
we
were
federating,
we
had
to
take
a
very
naive
approach
right.
A
A
A
There
are
limits
to
what
you
can
push
down
there.
There
are
some
things
you
just
you
just
can't
right.
I'll
give
you
an
example,
one
of
the
awesome
things
that
that
we
got
with
with
data
lake.
Let's,
let's
give
data
lakes,
not
lake
house
and
data
lakes,
credit
where
it's
due,
they
separated
storage
and
compute.
So
we
can
scale
them
individually,
but
the
storage
layer
is
a
giant
glorified
hard
drive
right.
It's
it's
a
big
distributor,
hard
drive
and
it's
dumb.
A
B
And
then
you're
also
going
to
see
a
lot
of
the
databases
I
did.
I
worked
at
teradata
and
worked
on
teradata
a
lot
of
times
a
lot
of
years,
a
lot
of
times,
sometimes
it
actually
with
a
filter
to
be
able
to
pull
that
data
out
of
teradata
and
then
we're
an
mpp
engine
we
want
to
do,
joins
and
aggregations
in
our
engine,
because
we're
gonna
be
a
lot
faster
most
of
the
times
right,
there's
a
lot
of
times
when
you're
joining
teradata,
you,
everybody
fights
for
the
temp
space
right.
B
So
there's
a
lot
of
times
where,
like
comcast,
is
a
great
example
where
words
we
wanna,
we
wanna
treat
our
data
storage.
We
want
to
treat
these
some
of
these
data
sources
as
storage,
because
we
can
actually
pull
that
data
out
all
the
dynamic
filter,
all
the
really
cool
things
that
we
could
do.
We
could
do
that
much
quicker
than
a
over
work
database
can
so
that's
a
lot
of
the
work
that's
been
done
in
the
last
two
years
is
getting
down
to
instead
of
pulling
everything
out
of
the
tables.
B
Let's
just
get
intelligent
about
it.
That's
why
statistics
are
so
important
to
us
and
that's
why
we
had
to
add
that
analyze
command
because
to
really
get
that
proper
joins
and
the
performance.
We
need
extra
extra
statistics.
I
can't
say
that
word
out
of
delta
lake,
but
we
need
that
for
all
of
the
storage.
We
need
as
much
information
just
like
a
typical
database
right,
the
more
information
we
get,
the
better
joins
and
performance.
We're
going
to
give
you
so.
A
A
Yeah
I
mean
we've
had
much
higher
scale
than
that
it
can
be
challenging.
So,
let's
talk
afterwards,
we
can
try
to
figure
out
what
the
situation
is
with
you
and
we
can
maybe
get
you
connected
with
some
of
our
engineers
to
figure
out
what
the
what
the
issues,
the
specific
issues,
how
does
trino
handle.
A
B
Yeah
so
like
in
there
there's
something
called
file
based
access
control
in
trino
in
open
source
that
you
can
control
that,
based
upon
like
an
I
am
role
so
there's
I
am
role
mappings,
so
you
give
certain
im
roles
that
you
can
only
access
certain
tables,
so
that's
just
been
added
within
the
last
year.
So
if
you
go
to
reno
dot
io
and
go
to
the
security
part,
there's
a
couple
different
ways
to
do:
authentication
that
way
and
then
control
so
well.
We
could
talk
afterwards
as
well.
I
could
show
you
where
it's
at.