►
Description
Join this special D3L2 vidcast/podcast with Andy Grove who has been specializing in query engines and distributed systems. Among many of his accolades, he started the DataFusion and Ballista query engine projects and donated both to the Apache Software Foundation as part of the Apache Arrow project. He also donated the initial Rust implementation of Apache Arrow and recently created Ray-SQL, a distributed SQL query engine in Python using Ray.
Quick Links
Andy Grove: https://www.linkedin.com/in/andygrove/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
GitHub: https://github.com/delta-io
Join Google Groups: https://groups.google.com/forum/#!forum/delta-users
A
People,
even
though
the
two
vehicles
are
like
how
far
apart
are
they
really
sure
yeah
slag
Rocky?
No,
because
up
here,
Foresters
also
ran
out
fast,
okay.
So
the
only
reason
the
main
reason
I
have
an
Outback
is
because
that
was
the
one
I
was
actually
going
to
be
able
to
negotiate
on
because
they
actually
had
the
Outbacks
available
and
hi
everybody.
A
If
you're
wondering
no,
the
session
is
not
about
to
Subaru
Outback
versus
Subaru
Forester,
though,
if
you
want
to
get
in
the
debate
you're
more
than
welcome
to
join
us
and
chime
in
in
the
Q
a
and
chat.
That's
perfectly
fine
with
us
actually
in
all
seriousness,
but
I'm,
pretty
sure
you're
actually
here
for
a
very
different
reason.
In
this
case
it
would
be
the
d3l2
session,
so
Carly
you've
got
the
YouTube
the
LinkedIn
all
set
up
once
you
take
it
away.
Awesome.
B
B
So
just
so,
you
know,
Andy
Grove
started
the
data
fusion
and
ballista
Curie
engine
projects
and
donated
both
to
the
Apache
software
Foundation
as
part
of
the
Apache
Aero
project.
He
also
donated
the
initial
rust
implementation
of
Apache
arrow
and
recently
created
Rey
SQL
a
distributed.
Sql
query
engine
in
Python
using
Rey
I
also
want
to
give
a
friendly
reminder
that
data
and
AI
Summit
is
one
month
away:
June
26
to
29th.
B
A
A
So
I
want
to
give
a
shout
out
to
Jonathan
bird
for
for
chiming
in
already,
but
before
we
do
that,
let's
definitely
do
what
we
typically
do
in
these
d3l2
sessions,
which
is
let's
talk
to
our
guest
in
this
case
as
Carly
NoDa,
it's
Andy
Grove,
so
Andy.
Why
don't
you
start
off
and
tell
a
little
bit
about
yourself
like?
Where
are
you
currently
working
like
what
you're
currently
doing
and
then
I
want
to
actually
afterwards
go
back
in
time
a
little
bit
and
talk
about
like
how
did
you
even
get
here?
C
Awesome
so
my
date
up
right
now:
I
work
for
NVIDIA,
which
is
a
great
company
to
work
for
I
work
on
the
spark
records
project.
So
more
formally
the
records
accelerator
for
Apache
spark
and
basically
it
actually
run
your
spark
SQL
and
ecl
jobs
on
GPU
to
make
them
go
faster
and
you
know
Jupiter,
and
so
that's
pretty
cool
and
essentially
we're
mapping
Spark's
physical
plan
to
operations
in
the
qdf
library.
So
qdf
is
a
python,
C
plus
plus
library,
for
performing
today's
frame
operations
on
the
GPU.
A
A
Yeah
yeah-
that's
really
nice,
okay,
but
we're
not
here
to
talk
about
that,
though,
as
interesting
as
that
is
or
talk
about
super
Outback
versus
Forester.
So
that's
also
an
interesting
conversation.
No!
No!
What?
How
did
you
get
involved
in
this
in
the
first
place?
Andy,
let's,
let's
dive
in
go,
go
back
in
time
like
are
you
like,
like
we
were
laughing
in
the
background
of
how
we're
both
introverts
originally,
which
is
largely
funny
that
we're
both
on
a
podcast
right
now,
but
I
was
like
yeah?
A
C
No,
absolutely
I
have
to
go
back
kind
of
a
long
way
to
the
I
guess
1980s.
So
when
I
was
a
kid,
but
I
got
interested
in
programming
around
the
age
of
11
or
12.,
so
I
grew
up
in
England.
We,
my
parents,
sent
me
on
a
kind
of
mini
it's
kind
of
like
a
summer
camp.
It's
just
like
a
week-long
thing
at
my
local
library,
in
Little
Village
in
Essex
and
I
learned
about
computer
programming
and
I
thought
it
was
kind
of
neat.
C
C
C
It
would
work
great
since
it
went
viral
and
too
many
users
came
along
and
you
know
they
just
couldn't
scale
so
sharding
kind
of
emerged
there
as
one
of
the
solutions
so
instead
of
having
one
database
yeah,
why
not
partition
your
database
into
multiple
instances?
And
then
you
know,
send
your
queries
to
the
appropriate
database
based
on
where
you
will
use
the
data
is
that's
essentially
the
idea
so
I
joined
a
startup.
C
C
So
this
starts
up
as
work
before
kind
of
came
to
an
end
and
I
spent
all
this
time.
Learning
about
building
database
infrastructure
and
I
thought
it
was
really
cool
us
having
a
great
time
so
that
came
to
an
end
and
I
wanted
to
carry
on
learning
and
also
at
that
time.
I
just
started
learning
rust,
which
I
thought
was
like
a
really
interesting
language.
So
when
this
job
came
to
an
end,
I
figured
well
I'm
gonna
just
go
build
some
stuff
in
my
spare
time.
I
want
to
learn
rust.
C
A
No
I
I
love
that
so
you're.
Definitely
making
me
relive
some
of
my
past
with
the
database.
Sharding
Concepts,
like
I,
still
remember
trying
to
do
a
similar
concept
with
SQL
Server
itself,
like
Microsoft
SQL
Server,
so
that
was
that
was
interesting.
Let's
just
leave
it
at
that
and
then
but
then
I'm
curious,
okay.
So
the
why
rust
at
the
time
like
you
know,
because
rust
I
still
remember
right.
A
In
fact,
I
actually
had
old
debates
with
with
folks
about,
like
you
know,
should
we
be
using
Rush
versus
the
jvm
and
like
even
back
in,
like
when
just
rust
was
beginning
right
like
the
con,
the
idea
was
sound,
but
the
like
implementation
was
still
like
unknown,
so
I'm.
Curiously,
what
made
you
choose
rust
then.
C
C
After
you
know,
after
this
company
I
was
working
for
came
to
land
and
I
was
looking
for
a
new
job.
In
my
resume,
like
20
plus
years
in
Jehovah,
you
know,
I
started
to
get
kind
of
global
Vibes
and
I
figured.
You
know
it's
time
to
start
learning
something
new
kind
of
freshen
up.
My
skill
set
and
yeah,
and
it's
fun
going
back.
C
So
that's
part
of
the
motivation
and
like
the
thing
that's
great
about
Java
is,
is
because
it's
been
around
for
like
20
30
years.
The
ecosystem
is
just
incredible.
Any
Library
you
need
to
do
anything.
Is
there
and
you
know
that's
a
big
reason
why
so
many
like
data
platforms
have
been
built
on
the
jvm.
C
You
know
going
back,
it
was
often
iota
is
really
the
Baseline
act,
not
the
CPU.
So
the
fact
that
Java
isn't
the
most
efficient
language
didn't
really
matter
so
much
but
I
think
that's
changed
over
time
as,
like
you
know,
IO
has
got
faster,
so
yeah
so
like
having
the
native
clothes
and
not
having
a
garbage
collector.
Those
were
things
to
me.
C
That
seems
like
a
really
good
fit
for
data
processing
like
no
garbage
collection,
pauses
yeah,
so
that's
I,
guess
that's
kind
of
it
and
yeah
you're
right,
like
when
early
organizing,
around
students
crazy,
like
you'd,
have
your
code
written
in
Rust
like
using
rust,
nightly
and
then
like
a
week
later.
You
have
to
kind
of
rewrite
your
project
because,
like
the
language
changed,
it
was
yeah,
definitely
definitely
challenging,
and
then
the
compiler
is
not
a
smart
back
then
either.
So
the
whole
kind
of
borrow
checker
thing
was
like
much
more
primitive.
A
Yeah
and
I
found
in
general,
just
even
the
even
some
of
the
concepts
of
rust
itself
was
actually
confusing.
Right
I
mean
it
allows
for
memory
safety,
which
is
amazing,
but
like
especially
back
that
time,
unless
you
had
been
just
like,
as
you
noted
like
a
hardcore
programmer
people
didn't,
they
may
have
been
able
to
say
the
words,
but
they
didn't
really
understand
the
context
very
well
right
like
what
that
implied,
what
that
actually
meant.
A
A
So
no,
that's
that's,
obviously
a
very
ambitious
project
which
is
great
and
but
then
you
deviated
a
little
bit
right,
yeah
and
not
rig
deviate,
but
you
you've,
you,
you
just
Melissa,
still
actually
quite
active,
but
am
I,
but
then
you
jumped
on
and
did
a
bunch
of
other
projects
too.
So
let
which
one
came
first,
was
it
data
Fusion
first
I
think.
C
Data
fishing
was
first
and
data
Fusion.
My
goal
was
for
the
diffusion
to
be
distributed
and
that's
where
I
started
out,
and
so
it's
like
the
whole
story,
so
I
started
building
this
thing,
so
you
know
basically
started
building
I,
wasn't
really
sure
at
the
beginning
whether
it
was
going
to
be
a
database
or
quite
what
I
was
doing
but
okay
I
know.
C
C
Yeah
I
learned
so
much
through
these
projects,
but
if
I
were
starting
again,
I
would
start
off
by
building
like
a
physical
execution
plan.
Just
like
the
ability
to
actually
run
a
query
against
data,
then
I
would
work
my
way
backwards
to
psychological
plan
and,
like
the
query
front
ends,
but
anyway,
so
I
started
working
in
a
simple
password
and
then
yeah.
C
So
I
was
building
data
Fusion
as
a
distributed
thing
and
early
on
it
was
very
much
kind
of
vaporware
and
hacky
I
had
this
kind
of
demo
of
a
distributed
query
and
I
post.
You
know:
I
had
Michael
book,
I
posted
on
hacking
news
and
it
kind
of
you
know,
struck
a
chord.
There
are
clearly
a
lot
of
people
very
interested
in
something
like
this
and
that
kind
of
gave
me
the
motivation
to
keep
going
so
yeah.
C
It
gets
kind
of
working
at
this,
but
at
some
point,
I
realized
that
I
was
trying
to
do
so
much
having
a
distributed.
Query
engine
and
I
kind
of
yeah
kind
of
rescopes
the
project
just
to
be
an
in-memory
query
engine
and
that
obviously
made
it
like
an
order
of
magnitude.
Easier
and
also
easier
to
kind
of
attract
contributors
to
the
project
is
a
little
bit
more
manageable,
so
I
kind
of
put
the
distributed
bit
on
hold
for
some
time
and
then
eventually
use
the
data.
C
C
So-
and
all
of
this
was
kind
of
you
know
weekend
evenings
like
hobby
time
so
yeah,
it's
pretty
slow
going,
but
I
just
kept
chipping
away
that
over
quite
a
long
period
of
time
and
covert
helped
I
had
a
bit
more
time
on
my
hands
during
my
2020
2021.
So
that's
the
time
where
I
really
kind
of
got
the
story
over
the
edge
and
got
it
to
the
point
where
I
could
run
some
real,
like
queries,
distributed
and
get
some
kind
of
okay-ish
performance.
C
A
C
A
But
back
to
data
Fusion
a
little
bit
because
you
know
we've
been
like,
for
example,
even
folks,
like
from
my
team
I'm
at
databricks.
If
you
folks
don't
know
like
basically,
we've
been
playing
with
data
fusion
and
we're
just
noticing
the
speed
of
which
its
processing
is
just
amazing
and
again,
I
find
it
hilariously
funny
that
I
I
find
it
hilariously
funny
that
you're,
referring
to
the
concept
as
like
you,
wouldn't
you
went
backwards.
You
built
the
SQL
parser
first,
so
I'm
just
curious.
A
Like
from
your
perspective,
did
you
expect
to
see
data
Fusion
get
to
the
speeds
or
the
performance
that
you
actually
like
do
see
now
or
was?
Was
that
like
the
original
thought
process
or
was
more
like
as
you
progress,
you
realize?
Oh,
my
goodness,
I
can
actually
get
that
much
more
speed.
Out
of
this
sure.
C
Absolutely
so
I
wrote
a
blog
post
back
in
around
2018
when
I
started
this
project,
it's
called
rosters
for
Big
Data,
then
there's
another
one
that
was
kind
of
popular
news,
but
I
wrote
them
there.
So
I've
been
using
Apache
spark
and
I'm
a
big
fan
of
spark
I.
Think
it's
like
it's
an
amazing
product.
You
know,
but
it's
built
in
the
jvm,
and
there
are
a
bunch
of
things
in
there
to
kind
of
work
around
the
fact
that
it's
built
on
the
jvm.
C
C
It's
out
and
I
had
this
hypothesis
that
if
you
built
something
quite
simple
in
Rust,
it
would
probably
be
as
good
or
maybe
even
better
so
that's
kind
of
yeah,
where
our
start
is
and
then
there's
one
one
of
the
I
love
talking
about
my
mistakes,
but
one
of
the
mistakes
I
made
when
I
started.
This
I
was
actually
building
it.
C
C
Like
multiple
people
told
me
like,
don't
build
it
robust,
you
should
be
doing
condom
and
hey
take
a
look
at
Arrow,
so
yeah
I
refactored
the
whole
thing
to
use
like
I'm,
just
using
rest
of
it
because
of
represent
data
in
columns
instead
of
just
doing
row
based
and
yeah,
suddenly
like
at
least
twice
twice
as
fast
as
it
was
before.
That's
cool
and
then
I
started.
Looking
at
Arrow,
there
wasn't
a
rust
implementation,
so
I
kind
of
I
kind
of
migrated.
C
C
The
the
rust
implementation
of
areas
today
is
like
so
different
once
I
originally
contributed.
It's
like
way
better,
but
I
kind
of
got
a
bull
rolling
at
least
and
helps
you
know,
get
other
people
involved
so
that
works
out
yeah,
pretty
well
and
then
yeah
so
and
then,
with
other
people
contributing
to
Arrow.
We've
got
things
like
simile
support
and
yeah
over
time.
The
performance
just
kept.
You
know
getting
better
about
it.
A
Oh
absolutely
true,
so
you
you've
worked
with
QP.
How,
like
you
know
when
it
comes
to
some
of
the
rust
to
Arrow
stuff,
so
we
had
actually
a
past
d3l2
session
where
we
were
talking
about
this,
and
it
was
really
funny
because
when
he
and
I
were
having
this
conversation
about,
why
why
using
rust,
when
he
was
doing
it
was
just
like?
Well,
it
was
my
first.
You
know.
Delta
rust
in
this
case
happened
to
be
his
first
production.
A
Rust
project
he's
obviously
had
many
production
projects
before
just
specifically
the
first
rust-based
production
project.
But
what
was
interesting
about
his
perspective
is
which
is
sort
of
like
how
you're
talking
about
right
now
was
this
idea
that
oh
I
just
wanted?
You
know
these
features
or
I
just
wanted
memory.
Safety,
I
wasn't
actually
looking
for
Speed,
and
then
it
turned
out
what
they
were
doing
was
phenomenally
faster
to
the
point
where
they're
going.
A
We
actually
don't
need
the
distribution
capability
right
now,
because
a
single
core
or
single
machine
was
actually
able
to
go
ahead
and
pump
through
all
this
data
in
no
time
flat.
So
yeah
I
love
hearing
these
stories
like
this
and
then
from,
and
you
know
just
to
over
emphasize
your
point.
Yes,
I'm
I'm
a
long
time
spark
guy
myself
too
right
and
yes,
the
verbosity
of
the
error
messages
alone,
can
scare
The,
Living
Daylights
of
everybody,
so
I
I
understand
your
motivations.
A
Let's
just
say
to
to
do
this:
oh
there's
a
very
good
question,
which
I
actually
would
love
to
post
to
you
right
now
from
Richard
Katz
from
sorry
from
Zoom,
sorry
for
I'm
freaking,
which
platform
I'm
looking
at
right
now
so
on.
So
this
is
direct
it
to
you,
Andy,
which
is
like.
Did
you
ever
meet
lee
fesperin?
He
wrote
the
first
sequel
in
he
wrote
first
sequel
in
C
and
then
first
equal
in
Java
as
well.
A
A
The
idea
projects
on
the
idea
in
Hacker,
News
Post
the
project,
ideas
in
and
Reddit
and
then
what
it
seemed
to
blow
up
because
of
there
and
then
from
that
context,
it
made
sense
to
like
donate,
for
example,
to
the
arrow
to
the
arrow
Community,
just
because
you're
building
a
lot
of
the
stuff
around
rust,
rust
and
arrow,
for
example.
Within
that
context,
I'm
just
curious,
like
how
do
you
stumble
onto
this
idea
that
you
would
determine
a
lot
of
the
projects
that
you're
building
right
now
were
basically
based?
C
Sure
so
I
guess
there's
a
couple
of
things
there
like
I,
recognize
the
like
what
I
was
doing
essentially
like
rust.
There's
a
there's
like
a
wave
coming
along
this.
This
new
rust
thing:
it's
got
this
like
benefit.
It
can
apply
to
so
many
projects
that
we're
currently
doing
in
Java
and
other
languages.
So
I
wrote
like
if
I
hadn't
done
this.
C
Somebody
else
would
have
done
at
some
point,
but
I,
just
kind
of
recognized
early
on
that
this
would
be
like
such
a
great
fit
so
and
also
I
realized
that
I
can't
go
building
like
a
distributed.
Query
engine
on
my
own.
That's
not
going
to
happen
so
I
just
figured
I'd
start
building
a
community,
so
I
was
posting
like
like
this
week
in
data
Fusion
kind
of
newsletter,
just
read
it
each
week
or
posting
back
new
releases.
C
It's
okay
though,
and
it's
just
a
great
ways
to
promote
the
project,
get
feedback,
but
also
attract
contributors.
To
help
me
build
this
yeah.
Just
I,
don't
know
I,
just
I!
Think
that's
a
big
part
of
it
for
me,
like
I,
don't
just
be
sitting
here
in
Colorado,
just
coding
on
my
own
on
something
that
doesn't
have
you
know
that
I
mean
it's
kind
of
fun,
but
I
like
being
part
of
a
community
working
with
other
people.
C
A
Yeah
I
mean,
in
my
limited
involved,
limited
short
time
limit
involvement
with
the
our
community
is
it's
very
obvious
that
you
are
you're
shepherding
folks,
so
many
people
in
terms
of
being
able
to
build
better
projects,
better
products
and
things
of
that
nature,
and
that's
really
cool
and
I
I
guess
the
question
that
I
didn't
think
I
was
going
to
ask,
but
now,
but
now
that
we're
brought
up
let's,
let's
definitely
do
this
yeah.
What's
what
what's?
What
spotted
you
to
do
that?
C
That's
a
great
question
today
can
I
find
any
answers
to
that
myself
and
I
think
I
think
it's
different
I
think
being
extroverted.
It's
like
on
the
keyboard
is
a
little
bit
different
than
you
know,
being
extroverted
like
in
person
with
people,
it's
a
lot
easier
to
I.
Don't
know
it's
definitely
easier,
but
but
probably
this
journey
is
probably
I
think
it
has
helped
me
become
more
of
extrovert
generally
by
going
back
a
few
years.
C
The
idea
of
preparing
on
a
podcast
which
would
terrifying
me
or
you
know,
standing
up
for
a
meets
up
talk.
You
know
it's
something.
I
definitely
did
not
enjoy
in
the
past,
but
now
I
love
doing
it
so
yeah,
I,
guess
being
part
of
this
community
and
growing.
A
community
has
helped
me
on
that
front.
So
so
yeah
I
recommend
that
for
other
introverts
out
there.
A
The
only
reason
I
bring
that
up
is
because
that's
actually
my
introduction
to
social
was
people
were
tweeting
a
lot
right
and
but
I
didn't
want
to
go
to
the
events,
because
that
meant
I
actually
have
to
hang
out
with
people,
so
so
I
mean
I
would
go
to
like,
like
specifically
like
I'll,
ask
ask
questions
office
hours
because
you
know
I
didn't
mind
answering
when
it
was
very
specific
to
like
okay,
let's
go
into
the
internal
SQL
Server.
Okay,
no
problem
right!
A
Let
me
let
me
explain
that
stuff
to
you,
but
like
oh
how's,
your
day,
no
I,
don't
I,
don't
I!
Don't
want
to
answer
that
question
at
all
that
that
just
that's
just
really
weird,
and
so
what
I
ended
up
doing
was
I
end
up
not
going
to
sessions,
but
I
end
up
tweeting.
So
it
was
like
so
people,
basically
here's
the
secret
of
y'all
for
a
while
people
thought
I
was
there.
I
was
just
at
some
other
event.
All
they
did
was
tweet
for
the
cover
of
my
sofa.
A
So
that's
how
I
got
into
it
and
then
finally,
actually
because,
like
maybe
I,
should
go
see
these
people
supposed
to
just
tweeting
from
the
behind
the
scenes,
so
I
I,
so
I
get
it
like
from
from
a
one
fellow
introvert
to
the
other
introvert
which
again
especially
coming
from
maybe
people,
don't
believe
what
I
just
said.
But
the
context
is
yes.
A
I
definitely
understand
that
context,
and
so
you
spend
a
lot
of
time
a
small
chunk
of
your
time,
obviously
working
with
the
community
and
working
with
the
community
to
basically
Shepherd
various
projects.
Is
that
what
led
to
well,
ultimately,
why
you
wanted
to
create
race
sequel,
then?
Was
it
because
just
Internet
working
with
the
community
recognizing
these
other
projects,
because
there's
actually
a
good
question
about
python
versus
SQL,
which
is
what
sort
of
pythony's
versus
rust
excuse
me,
which
is
why
I
wanted
to
actually
jump
into
the
race
equal
discussion.
C
Oh
yeah
sure
so
so,
like
once,
I
haven't
been
that
as
involved
for
a
while
now
in
data
Fusion
in
the
community.
So
everybody
started
this
in
2018
that
yeah
I
was
working
on
this,
like
four
or
five
years.
More
recently,
I'm
doing
this
I'm
kind
of
still
doing
the
releases
Bob
kind
of
that's
Hands-On,
but
with
free
sequel,
I
was
looking
to
kind
of
do
a
little
bit
so
I
figured
I,
don't
have
like
so
much
time
to
contribute
there
and
the
project
is
so
active.
C
I
can't
even
keep
up
with
the
PRS.
You
know
it's
just
it's
going
too
big,
so
I
was
thinking
like
how
can
I.
How
can
I
help
now
and
I
figured
one
way
is
to
do
this
with
evangelism
and
just
kind
of
show
people
what
you
can
do
with
data
fusion
and
also
over
the
last
couple
of
years.
I
realized
how
important
python
is
it's
something
that
I
just
kind
of
missed
earlier.
C
C
Did
the
very
kind
of
hello
world
one
like
one
Saturday
morning
and
like
within
an
hour
like
I've
got
hello
world
working
distributed,
I
figured
well
I
can
I
can
make
this
into
a
district,
a
single
brick
engine,
so
just
kind
of
stole
some
store
some
code
from
ballista,
because
ballista
already
has
all
the
parts
you
can
serialize
a
query
plan
into
protobuf
and
Ray?
Would
let
me
just
send
that
parade
above
to
a
worker
node?
C
So
pretty
quick
and
data
Fusion
had
python
binding
So
within
a
weekend,
I
had
I
was
able
to
run
it
wasn't
really
building
a
new
query
engine.
It's
really
just
running
data
Fusion
on
Ray,
but
it's
quite
yeah,
pretty
cool
and
yeah.
So
I
built
this
thing
tweeted
about
it
and
that
yeah
that
kind
of
took
off
people
were
like
really
interested
in
that,
and
hopefully
that
attracts
more
kind
of
python
developers
to
data
Fusion.
Because
previously
it's
been
very
much,
you
know,
rust
developers
have
been
the
main
kind
of
audience.
A
A
But
then
so
what's
interesting
is
okay,
so,
basically
for
the
users
of
python,
because
Allah
Ray
you're,
just
like
you,
said
that
now
they're
able
to
leverage
data
fusion
and
they
have
distributed
queries
because
of
Ray.
So
this
is
pretty
cool,
pretty
powerful
and
it
has
got
this
nice
SQL
parser.
Hence
why
they're
called
race
equal
I?
Guess:
let's
go
backwards,
a
little
bit
because
now,
let's
directly
answer
so
this
is
Jonathan's
question
from
LinkedIn
reasons
to
use
rust
over
python
so
and
we'll
go
both
ways
by
the
way.
A
This
isn't,
let's
be
very
clear
to
everybody:
that's
okay!
Well,
this
is
not
a
Denny's
trying
to
do
a
gotcha
on
Andy
Grove
on
rust
is
better
than
python.
This
is
definitely
not
doing
this
we're
going
to
give
it
both
ways:
the
advantages
of
rust
over
python,
the
advantages
of
python
over
rest
like
so
that
way
it's
equal
right,
because
I
think
there's
it's
definitely
true
in
that
episode,
so
yeah
like
when
we
start
with
that,
let's
start
with
the
rust
over
python.
For
starters,.
C
Sure
I
mean
the
answer:
I
guess
it's
just
performance
like
if,
if
performance
is
absolutely
critical,
you
want
to
squeeze
like
every
last
piece
of
performance
out,
then
yeah
great
rust,
the
downside
of
that
you
know
you've
got
the
compilation
times
every
time
you
make
a
change
to
your
codes.
You've
got
to
recompile.
C
C
C
However,
if
you're
recording
rust
code
from
python
I
think
this
is
kind
of
a
superpower
like
the
yeah
sure
you've
got
like
I,
don't
know
if
nobody
overhead
is
according
into
a
Ross
method
from
python,
you
know,
I,
don't
know
if
it's
milliseconds
or
microseconds,
but
it's
a
pretty
small
amount.
So
if
you're
calling
a
piece
of
code,
it's
going
to
take
like
a
second
to
run
like
there's
already
an
overhead,
so
yeah,
so
that
so
to
me,
the
benefit
of
python
is,
is
where
you
have
this
kind
of
interactive
case.
C
You
just
want
to
try
running
code
and
see
what
happens
it
just
you
know
it's
just
incredibly
productive
and
you
can
start
off
having
a
lot
of
python
codes.
And
then
you
know,
if
you
get
to
a
point
where
there's
a
certain
piece,
that's
kind
of
the
performance
is
becoming
a
problem,
then
you
can
always
just
take
that
bit
of
code
and
rustify
it
and
then
continue
calling
it
from
python.
So
I'm
a
huge
fan
of
the
po3
projects
that
you
call
rust
from
python,
or
vice
versa.
C
A
Oh
absolutely
I
mean
when
you
look
at
the
Delta
Lake
Project
how
we
started
off,
that's
exactly
it.
It
was.
It
was
exactly.
She
called
out
the
we
started
with
rust.
I
still
remember
the
old
compilation
times
just
to
like,
like
just
the
hundreds
of
dependencies
or
crates
that
we
had
to
pull
in
right
and
so
the
Kapow
time
to
run
McCrory
like
the
query
itself,
like
with
data
Fusion,
was
like
in
seconds
or
or
less
like.
A
It
was
just
really
really
fast,
but
the
actual
compilation
time
it
was
like
I
think
this
thing's,
taking
like
almost
a
minute
right
now
for
me
now,
of
course,
this
is
early
on
so
we've
improved
on,
but
yeah,
but
yeah
like
yeah.
We
were
definitely
making
some
decisions
like
well,
especially
when
I
was
the
one
writing
like
not
writing
the
code.
No
I
I!
Don't
want
to
take
that
anyway,
from
anybody
they're
much
better
people
like
Tyler,
QP
Christian
that
were
writing
code.
No
I
I
mean
men.
A
A
C
Sure
so
I
mean
the
very
simple
project.
It
really
just
is
like
a
PSC
as
an
example
of
it's
like
a
division.
Example
like
here's,
how
you
could
run
the
diffusion
on
the
python
distributed
platform,
so
it
was
really
just
like
a
couple
of
weekends.
Creating
this
thing
and
I
had
some
help
got
a
contribution
from
Franklin.
C
He
did
some
great
work
to
make
it
more
performant.
He
was
familiar
with
Ray
and
then
they,
coincidentally
Ray,
has
the
way
race
and
data
around.
It
has
like
some
Arrow
features
in
there
and
the
fact
that
this
was
dealing
with
our
own
native
data.
There
were
some
optimizations
that
were
possible
that
I
didn't
really
get
into
so
yeah
I
was
working
this
over
like
two
or
three
weekends,
but
that
was
it.
That's
just
a
it's
really
just
a
proof
of
concept.
C
It's
like
yeah
like
this
is
a
cool
example
of
how
you
can
use
data
fusion
and
particularly
the
python
bindings,
the
so
data
fusions
python
bindings.
They
haven't
really
taken
off.
Yet
there
are
some
people
using
it,
but
I'm
trying
to
kind
of
promote
them
more
and
one
of
the
cool
features.
Now
you
can
use
data
Fusion
from
python
as
like
a
query
planner
and
you
can
then
get
the
logical,
Clan
back
and
access
it
all
natively
through
python,
so
you
can
kind
of
walk
through
the
the
operator
and
get
information.
C
So
if
you
wanted
to
build
a
new
like
SQL
platform,
query
engine
in
Python
data
Fusion
gets
you
all
the
way
so
getting
like
a
query
plan,
and
then
you
could
write
your
own
code
for
excusing
it
while
translating
it
to
some
other
Library
like
photos,
pandas
PDF,
whatever
so
I-
think
that's
kind
of
interesting.
So
yeah
thanks
this
podcast.
A
few
more
people
know
about
that
feature.
So
yeah,
maybe
maybe
awesome.
A
No,
no
again
I
mean
there's
this
notion
of
even
when
you
were
talking
forget
about
not
forget
about
it.
Excuse
me,
sorry,
even
when
we're
just
focusing
on
single
note,
processing
right,
there's
this
more
recent
Push
by
various
organizations
which
I
actually
by
the
way
agree
with,
maybe
just
not
the
extreme
that
they
do
it
is
this,
this
concept
of
medium
data
right
as
opposed
to
not
everything's
Big
Data.
A
Certainly
small
data
go
ahead
and
use
your
Excel
spreadsheet.
That's
fine,
I'm,
we're
not
trying
to
debate
on
that
one,
but
there
is
this
notion
of
medium
data
where
many
of
the
queries
we're
running
are,
in
fact
medium
data,
especially
if
you
can
filter
them
out
and
because
to
do
your
data
science
or
whatever
else
yeah
and
I,
guess
the
con.
The
reason
I'm
bringing
this
up
is
because,
even
just
on
a
single
node,
we've
we've
definitely
found
data
Fusion
to
be
extremely
fast
to
the
point
where
we're
going.
A
A
C
A
Cool
yeah
yeah,
it's
it's
I,
love
how
the
systems
are
growing.
I
think
you
and
I
had
chatted
a
little
bit
about
how,
when
you
look
at
the
Rustin
python
system,
there's
like,
for
example,
oh
my
gosh
I
just
did
a
brain,
fart
I
hate
when
I
when
I
do
that
they're
very
streaming:
databases
that
are
out
there
that
are
also
built
on
Rust
for
precisely
this
reason
for
Speed
and
Performance.
So
the
name
is
escaping
me
right
now.
C
A
You,
my
goodness,
it's
I,
think
I've
been
talking
about
materialized
views
too
long,
so
my
brain,
my
brains,
yeah
I
I.
The
only
reason
I
want
to
bring
that
little
story
up
is
because
I
still
back
to
your
jvm
portion
of
it
I
still
remember:
we
had
friendly
debates.
This
is
not
not
a
evil
debate.
We
were
both
actually
working
at
Microsoft
at
the
time
so
long
time
ago.
A
A
I
invariably
went
over
to
spark
and
because
our
Natures
right,
so
he
would
blog
these
amazing
things
how
he
could
just
get
this
one
core
to
do
all
these
calculations,
and
meanwhile
I'm
like
I'm,
going
Mr,
sparkboy,
I'm
gonna
distribute
my
way
out
of
every
single
problem.
I,
don't
care
about
the
what
the
single,
what
a
single
machine
needs
to
go,
do
and
so
I
thought
I
won
the
debate.
A
C
A
A
It's
the
the
the
C
plus
version
of
it
and
they've
they've
downgraded
the
for
the
quantize
for
for
the
LM
models
themselves,
but
it's
amazing
how
fast
my
little
M2
can
actually
go
ahead
and
pop
out
a
large
language
models
like
something
where
I'm
going
everybody's
been
talking
about
these
10
million
dollar,
like
you
know,
Azure
or
Cloud
instances
to
train
these
models
or
even
to
run
inferences
to
get
models.
I
mean
well
I'm
like
yeah.
My
laptop
can
actually
handle
this
no
problem
at
all.
A
C
A
A
So
I
guess
the
one
thing
I
did
want
to
sort
of
finish
things
up
per
se
is
just
any
like:
where
do
you
see
data
Fusion
itself
going
because
I
mean
I,
you've
called
out
the
fact
that
you've
seen
the
project
grow
it
it.
It's
basically
taken
the
life
of
its
own.
You've
got
your
day
job
too,
so
I'm
just
curious
to
like
produce.
Are
there
other
wishes
like
because
I
guess
this
is
me
pointing
to
folks
that
try
to
encourage
folks
to
actually
get
involved
with
the
data
Fusion
project
as
well?
C
So
I
think
I
think
where
data
diffusion
has
been
really
successful,
where
it's
really
taking
off
is
that
it's
so
extensible,
it's
not
so
much
something
to
use
as
an
end
user,
necessarily
because
you've
got
like
some
other
great
things
out
there
like
poders.tv.
You
know
if
you're
looking
just
to
run
a
query,
then
those
those
are
great
potato
Fusion
is
really
good
as
a
foundation
for
building
new
systems,
both
in
Rust
and
python.
C
Though,
and
that's
really
where
a
lot
of
the
contributors
are
coming
from,
there
are
I,
don't
know,
I
mean
at
least
a
dozen
companies
building
new
platforms
into
updated
fusion,
so
I
see
I
see
that
continuing
that
it
is
like
very
extensible,
so
you
can
plug
in
your
own.
Like
query,
optimization
rules,
your
own
execution,
you
know
whatever,
so
that's
great
I
guess
one
thing
in
my
wish
list
is
to
have
better.
So
we
see
all
these
benchmarks
online.
Comparing
these
different
query
engines.
C
Honestly,
like
television
Nation,
said
I
mean
it's
got
really
good
performance,
but
the
the
focus
hasn't
been
so
much
and
having
like
the
very
best
performance
in
the
industry.
That's
not
really!
The
the
main
value
of
it
but
I
think
it'd
be
good
to
do
a
little
bit
more
work
in
some
of
these
areas,
so
that
the
you
know
we
do
better
in
some
of
these
benchmarks
and
I'm
sure
that
will
happen
over
time.
C
A
No
definitely
definitely
so.
Okay,
perfect,
well
I
think
that's
a
good
way
for
us
to
end
today's
session
and
sorry.
So
we
on
Zoom,
just
chimed
in
when
will
rest
be
available
in
databricks.
Yes,
yes,
we're
working
on
it.
Okay,
there
is
a
there's.
A
contingent
of
people
like
myself,
who
are
very,
who
have
been
rustified,
I,
think
I'm
going
to
start
using
that
phrase
by
the
way
Andy.
So
we
are
there's
gonna,
be
more
of
us
stay
tuned,
not
nothing!
A
If
you
want
to
talk
to
him
more
honestly,
he's
online
with
the
datification
community
and
the
art
community
all
the
time,
so
you
can
find
them
there,
no
problem
at
all
and
if
you're
trying
to
talk
to
me
I'll
be
joining
that
more
often
or
you
can
still
find
me
hungry
down
at
the
Delta
Lake
community.
So
that's
pretty
much
it
so
yeah
any
last
words,
any
call
outs,
there's
a
good
call
about
your
book!
So
I
guess
we
probably
should
do
a
shout
out
to
your
book.