►
Description
The refurbished consumer electronics market is growing significantly as an alternative to purchasing new devices. With over 6 million customers, Back Market is the leading dedicated renewed tech marketplace bringing high-quality professionally refurbished electronic devices and appliances, including smartphones, laptops, gaming consoles, and more. The key to ensuring they meet the needs of each customer and seller is data, but as analytical workloads rose, so did the need to consume their data in a rapid, efficient and secure manner.
To enable this, Florian Valeye (a Delta Lake Committer) and the engineering team at Back Market have been important contributors to the creation of the Delta Rust API and associated Python bindings to enable low-latency queries of delta table without having to spin up a Spark cluster. They've also been actively involved in Delta Lake community office hours, contributors to the AWS Labs Athena Federation, reviewing code, and even our partnerships with Google BigQuery.
Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Denny Lee: https://www.linkedin.com/in/dennyglee/
Florian Valeye: https://www.linkedin.com/in/florianvaleye/
Join us on Slack: https://go.delta.io/slack
Delta Lake Releases: https://github.com/delta-io/delta/releases
A
Wait
a
couple
minutes:
this
is
the
next
D3
L2
session,
with
Florian
velai
from
the
from
back
Market
we're
just
waiting
a
couple
of
minutes
to
get
ourselves
all
set
up
on
the
live
streams.
We
just
clicked
it
on
YouTube,
which
is
great
we're
about
to
switch
over
to
LinkedIn.
So
in
the
meantime,
if
you
have
any
questions,
please
use
the
Q
a
panel
for
zoom
in
YouTube.
A
A
Fun
times
fun
times
all
right,
well,
just
give
it
a
couple
more
minutes
and
then
we're
gonna
talk
a
lot
about
some
good
stuff
about
Delta
rust's,
back
Market,
python,
rust,
and
you
know
if
we
have
time
we'll-
probably
talk
about
coffee
and
food
and
stuff
like
that
too,
like
you
know
how
to
how
to
properly
make
French
bakery
I
think
I,
think
pet
to
City
I
think
that's
actually
a
very
important
topic
as
well.
What
do
you
think.
A
A
All
right,
I
think
we're
we're
good
to
go
all
right.
So
hi
everybody
welcome
to
the
next
D3
L2
session
I'm
here
with
Florian
from
back
Market
we're
gonna
talk
about.
This
is
a
vidcast
and
podcast
series,
though,
if
you
do
have
any
questions,
please
chime
in
while
we're
doing
this
live,
please
chime
in
on
the
LinkedIn
comments
inside
the
YouTube
comments
or
in
Zoom
q
a
so
that
way
we
can
sort
of
track.
A
B
Yeah
sure
Danny,
so
yes,
hello,
everyone,
I'm
flying
ballet,
so
I'm
working
right
now
at
that
Market
and
my
background
is-
is
always
around
data
engineering
topics.
I
started
as
a
Consulting
team
working
with
different
company
size
on
many
different
topics
related
to
the
data
engineering,
mostly
with
data
science
and
data
analytics
Pipeline
and
Malaysian
T
I
joined
black
market
and
to
to
build
a
team
as
a
manager,
but
also
a
tech
leader
and
more
recently,
I
switched
this
position
from
the
tangling
manager
to
a
staff
data
engineer.
A
Perfect,
okay,
so
hey
before
we
go
dive
into
like,
for
example,
how
you
went
down
this
path,
about
building
data
engineering
pipelines
and
switching
from
a
manager
back
to
staff
data
staff,
engineer
which
actually
I'm
really
happy.
You
did
because
I'm
sort
of
like
you
there's
only
so
much
we
can
do
as
a
manager
like
what
got
you
interested
even
to
dive
into
the
data
space
in
the
first
place
like
or
are
you
like
a
traditional
computer
scientist?
Or
did
you
come
in
through
a
different
angle?
I'm
just
curious.
B
Yeah,
actually
it's
a
good
question
when
I
started,
I
was
really
a
software
Engineers
working
on
web
applications.
Front-End,
backend
and
I
saw
the
big
elephant
coming
with
Hadoop
and
some
strange
data
science
teams
working
on
kagger
and
it
was
the
good
old
times
to
say.
Okay,
maybe
something
is
really
interesting
and
to
deal
with
massive
data
and
I
switch
by
being
a
kind
of
data
Engineers.
A
Right
so
so
then,
so
in
your
case,
basically,
you
went
from
a
path
of
web
applications.
First,
that
in
terms
of
data,
you
saw
notice
the
data
science
with
Hadoop
and
then
basically,
that's
how
you
went
from
that
perspective
to
basically
into
the
realm
of
spark
and
then
basically,
just
like
you
called
out
spark
1.4.
So
we
actually
had
data
frames,
then,
before
I
think
we
announced
that
back
in
1.3,
if
I
recall
correctly,.
B
A
So
nevertheless,
okay,
so
that
got
you
into
the
realm
of
okay
data
pipelines,
data
processing
so,
like
you
said
before
traditionally
you're
doing
web
applications.
What
I'm
curious
about
is,
then,
how
do
you
go
from
basically
to
the
path
of
deciding
that
you
want
to
do
big
data
as
a
full-time
job,
especially
because
you
did
went
ahead
and
start
from
web
applications
to
back
Market,
and
for
that
matter
you
know,
provide
some
context
to
the
community
like
what
is
back
Market
just
in
case
some
folks,
don't
know
what
it
is:
yeah.
B
Our
mission
is
really
simple,
is
to
build
trust
and
desire,
for
you
know,
with
devices
and
in
the
meantime
reduce
the
electronic
waste
that
you
can
avoid
by
buying
a
new
tech.
So
you
can
find
all
these
products
online
and
we
are
more
than
700
right
now
at
back
market,
and
we
are
mostly
in
Europe
in
the
US,
and
we
started
to
have
some
Marketplace
in
Asia
as
well.
A
Oh
cool,
that's
awesome!
So
one
of
the
things
that
I
definitely
want
to
dive
into
is
then
okay,
so
you
did
use
spark.
But
then
how
did
you
like?
What
made
you
start
going
down
a
path
of
rust
and
python?
Then,
like
I'm,
just
curious,
because
this
for
anybody
who's
wondering
I'm,
definitely
segwaying
or
leading
to
the
conversation
about
Delta
rust.
A
B
So
I
was
really
interesting
to
get
the
metadata
information
the
schema
about
Delta,
but
without
having
to
span
a
spark
cluster
because
gvm
and
all
the
cost
considerations
that
you
can
take-
and
it
was
really
I-
started
to
actually
work
on
Delta
Lake,
python
Library,
but
I
found
really
interesting
projects,
Delta
iOS
1.
It
was
a
rust
library
to
access
to
to
Delta
tables
and
the
python
binding.
A
Oh
actually,
so
this
is
great.
So
we
talked
a
little
you
in
that.
In
what
you
just
mentioned,
you,
you
talked
a
little
bit
about
how
you
love
the
the
reliability
in
the
asset
transactions
from
Delta
Lake,
and
so
that,
actually
is
what
interesting
or
transitioned
you
over,
which
is
basically
looking
at
the
metadata
in
which
you're
like
I,
don't
want
to
spin
a
spark
cluster
for
it.
A
That's
completely
fair
I'm
going
to
go
back
to
that,
but
I
did
want
to
call
out
a
little
bit
about
airflow,
actually
because
I'm
just
curious
on
how
you
did
the
airflow
orchestration
and
by
the
way
part
of
the
reason.
Why
is
because
I'm
actually
doing
a
Shameless
plug
as
well,
that
we
have
a
D3
L2
session
in
I
want
to
say
early
December
was
Simon
spatty
from
everybody
specifically
to
talk
about
data.
B
About
we
switched
from
airflow
to
our
bike
actually
to
change
our
approach
when
dealing
to
get
data
from
outside,
mostly
in
Black
Market,
when
we
use
airflow,
is
to
fetch
data
from
external
apis
gotcha
or
for
Google
analytics,
for
instance,
and
I
have
a
way
to
integrate
this
into
a
Delta
Lake
for
the
Delta
IRS.
It
was
really
specific
to
have
more
optimization
on
the
data
storage
like
optimizing,
vacuuming
and
so
on,
for
the
other
party
was
to
fetch
data
coming
from
inside
and
to
plug
directly
into
the
Delta
lake.
B
When
we
switched
to
Albert,
it
was
to
to
not
having
all
the
data
Engineers
working
in
the
small,
the
same
code
base
with
a
lot
of
functional
knowledge.
No,
because
you
start
to
implement
one
dag
two
dags
a
lot
of
things
coming
into
the
same
code
base,
and
you
start
to
have
a
bottleneck
in
your
team,
from
airflow
to
air
byte,
to
have
something
directly
connected
to
the
and
to
make
decentralization
in
the
strategy
of
the
data
management
to
say.
B
A
No,
that
makes
all
the
sense
in
the
world.
Then:
okay,
let's
switch
back
a
little
bit
too.
So
you,
the
the
onus,
was
basically
not
needing
to
basically
spin
up
something
so
heavy
for
in
order
to
read
the
metadata
for
your
Delta
Lake
tables.
Basically,
you've
built
an
infrastructure,
and
at
this
time,
by
the
way,
if
I
recall
correctly,
you
had
built
everything
on
the
AWS
infrastructure.
Is
that
correct.
B
Notebook
will
do
right.
Now.
We
didn't
have
nothing
so
Delta
RS
was
a
meaningful
to
retrieve
some
data,
small
data
sets
or
to
explore
through
the
audit
phase
to
launch
vacuum.
It
was
really
depending
on
the
size
of
the
Delta
tables,
but
it
was
really
useful
for
us
at
the
beginning
to
have
this
kind
of
operations
and
also
a
lot
for
data
thanks
production.
A
B
Yeah
I
think
for
for
us.
The
first
thing
personally
is
I
love
rest
since
I
wrote
a
bit
of
the
code
base
because
it's
really
a
powerful
language
when
you
start
to
to
know
it
for
the
python
Rest
Library,
because
it
was
on
small
data
set,
but
with
the
performance
of
rest
and
safety.
All
about
the
concurrency
and
the
ways
is
managed
is
really
helpful.
When
you
are
dealing
with
python.
B
Python
is
just
a
binding
to
access
to
more
powerful
items
when
you
did
with
with
small
data
sets
but
yeah
the
difference
with
spark
and
Thai
spark
shops
on
massive
data,
and
when
we
deal
with
small
data
sets,
we
rely
on
the
rest,
because,
even
if
it's
more
it
depends.
Sometimes
you
need
to
learn
to
have
powerful
access
on
it
and
use
the
VR,
Python
and
and
I
think
that's
it.
And
for
us
we
can
talk
more
about
it.
B
But
it's
like
a
good
integration
tool
with
arrow
on
the
memory
management,
with
the
python
guidance
to
have
a
so
Best
of
Both
Worlds.
We
have
the
rest
and
a
strong
link
with
arrow
on
the
memory
and
the
Safety
Management.
The
dependency
management
that
is
really
helpful
with
rest
and
using
the
python
ecosystem.
So
if
you
need
to
have
more
contributors,
it's
pretty
useful
right
now,
because
at
that
market
everyone
is
a
python
developer
on
back
end
on
data
on
data
science,
so
it
was
with
the
fully
Workforce
to
have
this
binding
yeah.
A
See
there's
some
interesting
analogies
that
you're
actually
calling
out,
because
it's
almost
like
the
same,
the
the
python
bindings
with
rust
is
out
is
actually
similar
to
the
python
bitings
with
spark
right.
Basically,
you
get
a
lot
more
people
who
can
contribute
or
who
can
even
just
use
by
using
pi
spark.
A
There's
an
analogy
from
sampling,
rust,
Russ,
there's
a
few
people
who
know
how
to
do
it
really
really
well,
and
yes,
we
are
going
to
dive
into
that
in
a
second
and
then
for
the
most
people
who
are
either
not
interested
or
just
don't
want
to
do
it
right,
but
they
can
use
the
python
bindings
and
they
can
still
have
the
power
of
rust.
You
know
through
the
python
bindings
and
you're
good
to
go,
and
it's
really
small
and
it's
really
lightweight
sort
of
have
I
sort
of
sort
of
summarized
it
correctly.
A
B
Yeah
totally
I
think
it's.
You
have
a
learning
curve
that
can
be
add
when
you
start
to
work
on
it
and,
as
you
said,
it's
more
a
personal
Improvement
into
okay.
I
need
to
learn
less
and
to
implement
something
with
the
python.
Bitings
is
completely
abstracted.
It
means
that
if
you
want
to
play
with
python
it's
fine,
you
just
have
some
the
the
way
of
connecting
to
the
rust,
but
it's
pretty
abstracted
to
you.
A
Perfect,
okay,
so
let's
now
talk
a
little
bit
about
rest
and
by
the
way
we
actually
do
have
a
it's,
both
a
Shameless
plug
and
also
providing
context.
At
the
same
time
here
we
actually
do
have
a
d3l2
session
coming
up
in
early
December,
as
well
with
a
QP
Howe
who's.
One
of
the
main,
who
basically
I
could
probably
say,
is
the
creator
of
Delta
rust
API.
A
In
the
first
place,
the
you'll
hear
more
about
the
story
later,
but
from
him,
because
I'd
rather
have
him
talk
about
it,
but
basically
he
was
given
the
opportunity
to
say:
okay,
that's
it
I've
got
three,
you
got
three
months,
go
build
it
and
that's
basically
what
he
did,
but
let's,
let's
dive
into
some
providing
here
like
anybody,
who's
listing
either
on
the
vidcast
or
on
the
subsequent
podcast
like
what
underneath
the
covers
like,
let's
just
start
with
even
the
first
question.
Why
rust
like
this?
A
This
was
a
recent
blog
post
from
Simon
from
was
it
from
airbite
about,
like
the
Delta
rust
and
specifically
how
Russ
may
take
over
the
data
engineering
World.
There
was
also
the
same
another
post
from
Daniel
Beach
known
as
Confessions
of
a
data
guy
same
principle,
and
even
Wired
Magazine
went
ahead
and
recently
published
in
a
very
cool
article
talking
about
how
rust
may
take
over
as
data
engineering.
So
you
got
involved
before
the
hubbub,
though
so
like
yeah.
What?
What?
A
B
I
remember
this
time
it
was
since
I
discovered
the
Delta
Lake
and
Delta
arrest.
Livery
I
I
did
I
found
an
issue
the
first
issue
to
to
help
from
the
community
of
the
Delta
arrest
and
say:
okay
I
need
to
to
see.
What's
going
on,
I
need
to
change
something
because
I
was
working
on
the
python
biting,
but
I
need
to
touch
to
the
rust
core
to
make
The
Binding.
So
I
started
with
a
really
simple
features.
B
It
was
to
remove
the
last
slash
into
from
the
new
array,
so
it
was
okay,
it's
really
simple.
I
would
just
give
a
try
out
of
rest
and
I
work
on
it
during
two
days
and
I
face
medieval
to
say:
okay,
the
computer
say
no,
it's
not
working
like
this.
It
helps
me
to
Define
what
will
be
the
best
strategy
to
implement
and
to
test
again
and
again
and
again
and
I
re.
When
I
reached
the
point
of
knowing
the
ownerships
the
concept
of
ownership
interest.
B
I
was
like
okay,
it's
pretty
cool,
to
have
these
kind
of
features
to
retrieve
all
the
things
that
we
can
find
on
the
other
languages
and
I
started
to
discover
more
and
more
about
words.
First,
the
dependency
management,
which
was
great
pretty
simple.
When
you
look
with
python,
we
have
to
choose
one
between
poetry,
setup
tools,
clipboard
and
so
on.
So
it's
really
complicated
to
have
many
different
dependence
tools.
You
have
cargo
cargo.
Is
the
package
manager
already
pretty
simple,
to
use
everything
IP
to
check,
so
your
code
base
format?
B
B
The
last
point
is
about
the
ecosystem
I
found
the
rest
are,
is
for
developers
built
for
developers,
and
it
was
really
interesting
to
see
the
community
Beyond
it
with
a
lot
of
tools
with
polar
data,
Fusion
a
lot
of
rust
grade
that
really
Implement,
something
in
rest,
because
it's
it's
it's
really
useful
and
and
the
other
it's
a
full
power.
What
for
me,
the
last
point
was
about
integrating
a
new
language
visit,
ing,
Python
and
rust
is
really
simple.
I
remember
in
the
past,
dealing
with
Kaaba
or
DNE.
B
It
was
a
bit
of
nightmare
to
plug
all
in
the
interface
and
to
write
this
code
base
with
Rusty
was
really
simple.
With
the
bio
Free
Library
with
material,
it
was
really
simple
to
integrate
everything,
so
yeah
I
was
really
I,
felt
in
love
in
universe
and
started
to
work
more
and
more
around
all
the
projects,
because
it's
really
powerful
and
the
last
point
is
about
the
carbon
emission.
B
A
Oh
no,
no,
actually
now
this
will
be
a
little
bit
of
a
story
time
for
me.
I
hope
you
don't
mind,
but
I
still
remember
so,
there's
a
and
there's
a
gentleman
by
the
name
of
Frank
mcsherry
he's
actually
the
CTO
of
a
company
called
materialize
and
the
reason
I
bring
this
little
story
up
is
because
he
actually
has
built
some
pretty
interesting
stuff
for
like
a
streaming
database.
A
That
is
basically
built
entirely
on
Rust.
And
if
you
actually
look
at
his
past
blogs
directly
on
this
GitHub,
basically
that's
where
he
blogged
it.
He
would
talk
about
like
how
to
be
much
much
more
efficient
memory
management.
Things
of
that
nature,
which
you
and
I
are
going
to
switch
over
to
the
jvm
real
soon
by
the
way,
and
talk
about
that
for
a
second
for
jvm
and
GC,
and
all
those
other
things
without
rat
holding
otherwise
it'll
be
an
hour
and
a
half
of
just
that.
A
But
but
the
reason
I
bring.
This
up
is
because
I
still
remember
how
so
Frank
and
I
knew
each
other
back
in
what
2007
I
think.
So
it
tells
you
how
old
I
am
and
the
reason
I
bring.
That
up
is
because
he
and
I
were
on.
We
actually
were
working
on
a
completely
different
product
at
the
time
systems
at
the
time,
but
we're
actually
working
on
differential
privacy
at
the
time.
A
But
the
the
reason
I
bring
the
story
up
is
because
we
would
get
into
debates
friendly
ones
right,
which
is
like
basically,
oh
yeah,
like
a
distribution
versus
much
more
memory.
Efficient
and
I
definitely
went
down
the
decision,
distribution
path,
right
I'm
like
no.
No
we're
gonna
forget
about
Uber
efficiency
of
memory.
It
we
don't
need
it.
We
we
just
distribute
our
way
out
of
the
problem
and
it's
not
a
big
deal.
It's
of
course.
A
That
naturally
led
me
to
go
ahead
and
go
to
spark
because
I'm
like
no,
no
I,
don't
care
about
efficiency
anymore.
It's
fine!
It
just
just
costs
more
and
we
can
line
your
scale
and
we're
done,
and
so
now,
even
though
I've
been
working
on
Sparks
since
2011
or
so
so
like
near
the
Inception
of
his
product,
I
do
want
to
call
out
and
apologize
to
Frank
to
say
he
won.
The
debate
like
yeah
like
even
I'm
a
I'm.
A
B
Yeah
I
think
it
improves.
Your
programming
is
skills
because
the
way
that
you
will
write
code
and
by
it
will
be
more
efficient
because
the
concept
of
ownership
and
all
the
things
that
you
have
in
the
past
to
deal
with
is
Computing
remote,
and
you
have
a
lot
of
eye
abstraction
because
I
I,
like
that
in
the
past,
more
about
functional
programming
when
I'm
writing
with
python
is
the
same.
A
bit
of
this
character
world,
but
I
found
all
the
iterables,
the
closure
and
the
battle
matching.
B
Is
your
friend
you
know
it's
like
telling
you,
okay,
you
need
to
change
this
and
it
really
helps
you
to
to
be
more
efficient
and
with
couple
cargo
like
for
sure,
but
it
helps
you
to
Define
better
code
and
yeah
right
now,
when
I'm
writing
rust,
I
feel
like
I'm.
Writing
the
the
good
piece
of
software
that
I
want
and
just
to
to
to
jump
on
what
you
said
before
for
the
performance.
It
was
really
insane
because
I
remember
dealing
with
regex
and
regex.
A
Yeah,
and
and
actually
can
for
those
of
us
that
are
not
as
familiar
with
it.
What
is
your
you've
brought
this
topic
up
a
couple
times,
which
is
about
the
the
ownership
aspect
like?
Can
you
provide
a
little
context
from
your
perspective
like?
Why
is
that
important?
What
why?
What
what
did?
What
did
that
switch
a
flip
of
a
switch
help
for
you?
In
that
case,
I
guess.
B
I
think
when
you
are
dealing
with
distributed
system
or
a
lot
of
thread
or
concurrency
and
opinion
application,
you
have
to
make
sure
that
you
cannot
access
the
same
object
in
the
memory
and
it's
always
the
same
topic,
because
if
you
access,
you
can
have
a
segmentation
portal,
an
issue
with
with
your
application
and
sometimes
just
to
make
a
link
with
Delta
RS.
When
you
are
working
with
binding.
B
Sometimes
you
call,
for
instance,
the
library
in
C
or
C
plus
plus,
but
you
have
to
release
the
memory
because
before
going
out
because
you
can
introduce
memory
leaks
with
rust,
the
concept
of
ownership
is
to
make
sure
that
each
variable
as
an
owner.
So
it
makes
sure
that
it
cannot
be
the
case
of
having
to
to
thread,
for
instance,
actually
seeing
the
same
space.
So
when
you
write
the
code
and
the
computer
say
you're,
okay,
it's
fine!
It
means
that
you
can
really
work
in
a
way
that
you
you.
B
You
are
sure
that
every
piece
of
software
you
have
an
ownership
of
the
variables
and
it's
for
me.
A
New
Concept
I
work
with
really
a
lot
of
programming
languages
because
I
love
to
switch
from
time
to
time,
but
with
with
the
principal
functional
programming,
for
instance,
but
with
us
was
surprised
because
it
it
was
a
step
for
me
as
a
software
engineer
to
say.
A
That
actually
makes
all
sense
in
the
world
and,
frankly,
that's
actually
part
of
the
reason
why
I
asked
the
question,
because
you
had
called
out
also
the
same
thing
like
you
know,
when
you
compile
the
code,
you
know.
Basically
the
compiler
is
trying
to
tell
you
hey.
This,
isn't
good
what
you
just
did
here,
so
it
won't
work
or
it'll.
Give
you
enough
error,
slash
warning
so
you're
like
okay,
I,
guess,
I,
better
rewrite
that
and
I
know
for
folks
that
are
saying
like
hey,
but
I
want
to
write
something
faster
and
something
quick.
A
This
will
slow
me
down
I'm
like
yeah.
It
does
and
if
you
needed
something
fast
and
quick-
and
you
didn't
care
about,
you
know
how
effective
it
was.
You
could
always
just
write
in
SQL.
You
know
when
it
comes
to
data
processing,
and
this
is
not
a
knock-on
SQL.
It's
a
it's
a
great
language
for
that
purpose,
but
I'm
like,
but
if
you're
trying
to
manage
all
this
data
in
which
you
you're
trying
to
prevent
Corruptions
you're
trying
to
prevent
you
care
about
reliability,
all
these
other
Natures
right.
A
A
B
True
and
for
me
to
you
again
time
about
a
period
of
time,
meaning
if
you
build
something
in
rest,
you
can
go
to
the
production
without
having
a
strange
Corner
case
in
a
scenario,
if
I'm
taking
python
I
like
python
very
much
because
I
feel
it's
effective,
it's
ecosystem
is
huge.
It's
spreading
everywhere
on
all
the
technology.
It's
like
the
common
group
at
Black
Market
when
we
are
walking
because
it's
like
it's
a
language
from
everyone,
but
it
introduced
a
lot
of
typing.
B
For
instance,
you
use
my
buy
or
the
type
checking,
and
you
need
to
check
this
concurrency
issue
and
you
need
to
add
this
one,
and
you
add
you
you
add.
A
lot
of
things
on
top
of
python
with
rest
is
like.
Okay,
Simplicity
is
the
feedback.
Loop
is
a
really
really
fast
and
really
short
for
you.
But
when
you
go
to
prediction,
you
fail.
You
feel
safe.
Actually,
I
didn't
see
any
issue
on
the
rest
program
that
I
did
because
it
was
compiled,
but
everything
was
fine.
So
it's
okay
with
python,
sometimes
oh
yeah,.
A
So,
for
example,
you
already
called
the
South
Florian,
but
I
did
want
to
like.
Let's
go
through
this.
Let's
talk
about
Pi
Arrow.
Let's
talk
about
ballista,
let's
talk
about
polars
and
even
data
and
data
Fusion,
forsake
argument
right
and
anything
else
like
these
key
packages
that
we
really
want
to
call
out.
So,
whichever
one
you
want
to
go
first,
but
I
think
I
think
we
need
to
provide
the
users
here
or
so
the
attendees
here
listeners
here
this
context
of
why
like?
B
Yeah
sure,
and
for
Arrow,
if
I
take
some
the
perspective
from
python
developer,
it's
a
pi
Arrow
because
it
was
a
bit
the
same
arrow
is
the
rest
created
the
call
of
our
o,
and
you
can
multiply
a
row
that
are
binding,
I,
always
like
the
common
space.
When
you
deal
with
it's
many
different
things,
you
know
it's.
B
B
What
is
the
type
of
this
of
this
schema
by
arrow
is
a
common
structure,
and
when
you
are
dealing
with
a
lot
of
different
schema
metadata
and
types,
it
makes
sure
that
you
can
find
the
right
by
Arrow
one
and
be
optimized,
and
you
can
find
also
data
Vision
that
was
created
and
to
have
more
SQL,
planner,
optimizations
and
connected
to
to
Arrow.
So,
just
to
to
mention
that
I
rest
in
this
big
picture
that
I
arrest,
the
python
binding,
is
using
the
arrow
crate
through
the
Delta
Ray
crate.
A
A
He's
also
led
the
creation
of
of
Apache
era
right,
and
so
it's
basically,
this
idea
of
the
where
we're
saying
The
Best
of
Both
Worlds
between
rust
and
python
yeah
like
arrows
one
of
the
core
tenets
to
actually
allow
that
and
by
the
way
west
is
the
co-founder
and
CTO
of
Voltron
data.
So
they
actually
some
interesting
Concepts
like
flight
SQL
adbc,
but
just
wanted
to
call
that
out
like
this,
this
Arrow
being
a
important
component
that
allows
that
basically,
the
the
allows
for
the
data
management
of
things
in
essence.
A
Okay,
so
how
about
before
we
go
to
Kafka
Delta
ingest
just
because
and
not
to
mention?
I
can
probably
get
QP
to
dive
into
that
a
little
bit
too
much
anyway
wise,
but
like
how
about
like
projects
like
sorry
packages
like
data
Fusion
polars,
like
there's,
seems
to
be
a
lot
more
activity
around
these
things.
Can
you
provide
context
of
what
these
things
are
within
the
realm
of
the
the
python
world
or
Delta
rest
world?
For
that
matter?.
B
Yeah
I
think
it
was,
is
it's
in
the
name,
but
he's
bledging
fast
data
frame,
so
it
means
okay.
Maybe
we
we
have
Python
and,
for
instance,
if
we
use
Panda,
you
can
have
a
numpy
or
or
a
player
just
to
make
sure
that
you
can
be
really
fast.
Polar
is
a
naturally
fast
because
it's
implementing
implementing
interest
and
yeah
I
think
it's
it's
using
also
the
partial
colon
format,
the
memory
model.
B
For
me,
it
was
my
strategy
as
well,
because
I
heard
a
lot
of
about
rust
is
one
ranking
blanket
one
at
the
most
below
the
language
in
programming
survey,
but
how
I
can
Implement?
How
can
I
deal
about
scale
with
rust
and
the
community
is
really
large
and
you
can
Implement
because
it's
really
helpful
and
benevolent
when
I
joined
the
Terrace
QP
just
show
me
a
lot
of
things
to
how
to
improve
myself
to
to
be
efficient
with
rest.
So
it
was
really
nice.
B
The
committee
with
the
direct
as
well
and
the
script
and
all
the
teams
was
where
here
to
to
help
me
to
to
say:
okay,
maybe
you
have
to
to
learn
this
or
do
you
have
to
change
this?
So
I
think
rest
is,
is
something
that
the
developer
wants
to
play
with
and
find
an
opportunity
to
to
do
it's
like.
Okay,
maybe
we
need
to
rewrite
this
in
rest
to
see
how
it
goes
and
we
see
the
the
performance
gains.
They
say.
Oh
look
at
the
memory.
A
Yeah
and
so
by
the
way,
I
I
appreciate
a
particular
call
out
that
you
actually
sort
of
put
it
almost
as
a
secondary,
but
I
actually
want
to
sort
of
bring
it
up,
because
this
is
actually
very
helpful
for
the
person
who
asked
the
question,
like
you
know
who
somebody
who
came
primarily
from
PI
spark.
You
know
what
are
the
implementation
pipelines
Russ?
A
What
do
you
need
a
note
of
it's
the
far
part
that
the
Delta
rust
Community
went
out
of
its
way
to
basically
make
sure
anybody
that
wanted
to
learn
like
there
were
people
there
to
back
up
and
like
as
a
great
call
out
to
QP
by
the
way,
so
you
know
there's
a
but
also
like
we
have
a
bunch
of
other
folks.
You
know
in
the
Delta
rest
Community.
A
By
the
way,
so
because
that's
that's
a
natural
segue
here
right,
you've
returned
that
you've
went
ahead
and
also
I,
see
you
on
the
slacks
I
see
you,
you
know
on
the
GitHub,
often
you're,
basically
making
sure
everybody
else
can
learn
from
it
as
well,
and
so
the
fact
that
we're
all
helping
each
other
out
to
basically
learn
rust,
I
think
that's
an
important
thing
for
the
for
the
person
who
would
ask
about
implementing
pipelines.
Is
that
try
something
that
you
do
know
sure?
A
But,
like
hey,
there's
a
bunch
of
folks
here
that
actually
really
want
to
help
you
out
and-
and
please
do,
join
us
because
I
think
as
you
can,
you
can
almost
tell
there's
a
lot
of
us
that
are
all
super
excited
about
rust,
like
it's
become
a
very
geeky
thing
for
all
of
us
and
so
and
so
we're
and
note
the
fact
that,
like
you
know,
Florian's
a
back
Market
I'm
a
data
bricks
and
we're
talking
on
a
language.
Neither
of
us
are
actually
like,
technically
part
of
either
our
company.
A
Yet
we're
both
advocating
strongly
for
the
language
just
because
the
the
again
I'm
going
to
stop
because
you've,
probably
already
overheard
us
I'm
gonna
talk
about
type
safety
and
man
rematch.
But
the
point
is
that
it
just
it
allows
us
to
do
things
that
we
couldn't
do
and
then
one
final
tidbit
before
we
shift
gears
into
more
specifically
about
the
pipelines
and
how
you
implement
that
in
back
Market,
like
with
the
Kafka
Delta
ingest
project,
we
actually
had
a
video
from
last
year
with
Tyler
and
Christian
Amisha.
A
A
So
yeah
just
want
to
call
that
out
all
right.
We
only
have
about
like
I'd,
say
eight
to
ten
minutes
left.
So,
let's
segue
directly
to
the
person's
questions
like
what.
What
are
the
things
that
we
should
note
of
now
that
we've,
given
the
background
of
rust
and
given
the
background
of
python
and
how
back
Market
has
used
this,
you
know
in
order
to
be
able
to
implement
these
faster
pipelines
like
and,
and
you
came
from
a
pi
spark
background
too.
A
B
B
But
we
wanted
something
really
fast
and
cost
effective,
because
we
are
moving
data
from
one
proper
weather
to
another.
So
you
can
deal
with
a
lot
of
huge
aggress
cost
and
we
find
a
convenient
apis,
Snowflake
and
snow
pipe.
So
it's
basically
an
API
that
received
a
lot
of
bucket
files
and
making
things
available
inside
once
not
like
table.
So
it
was
pretty
useful
use
case.
B
We
deal
with
the
fast
ingestion
with
data
bricks,
to
make
things
really
really
fast
on
the
bronze
and
silver,
and
when
someone
requests
from
some
data
in
Snowflake,
we
have
a
dags
in
Python,
with
working
with
data
arrest.
Is
that
fetch
the
subscriptions?
Look
at
the
proper,
the
right,
the
right,
Delta
tables
in
the
bonds,
for
instance,
fetch
all
the
files
inside
the
partitions
or
the
description
that
the
data
was
requested
and
throw
it
into
snow
pipe,
so
snow
pipe
on?
B
So
it
means
that
when
it's
working
with
snowpack
with
this
kind
of
system,
you
can
switch
to
bigquery
or
other
storage
as
destinations
and
make
things
really
convenient
when
you
have
different
code
data
consumers
on
the
other
side.
So
the
switching
was
more
about
python.
The
rust
was
pretty
useful
to
fetch
the
the
files
to
the
partition,
the
schema,
all
the
the
metadata
audit
and
sometimes
Exploration
with
the
data
teams
to
work
with
small
data
sets,
but
not
a
writer
implementation.
It
was
an
except
for
that
IRS.
B
We
have
an
experimental
writer,
but
we
don't
use
it
because
we
use
more
about
Pi
spark
or
the
pipeline
with
data
like
at
the
moment,
but
it's
really
available,
but
we
are
just
fine-tuning
some
authentication
parameters
or
issues
sometimes
and
and
yeah.
It's
pretty.
It
is
depending
on
the
on
the
con
on
the
context
when
you
have
the
insulator
on
your
specific
use
case
right.
A
Gotcha
gotcha,
so
let
me
if
I
was
just
like
sort
of
in
this
current
state,
because
after
all,
we're
still
just
like
you
know
that
the
the
writer
is
still
very
much
experimental,
where
it
makes
sense
at
least
right
as
of
today,
November
17th
2022,
so
in
other
words,
if
you're
listening
to
this
podcast
or
watching
this
vidcast
at
a
year
from
now.
This
is
probably
not
the
case
anymore
and
we'll
probably
already
have
another
session
discussing
this.
A
The
context
is
like
when
you
get
switched
from
PI
sparked
over
to
Delta
rust.
No,
no,
which
situation
you're
applying
this
to
Delta
rust
is
very
reading,
is
great,
and
this
is
the
part
where,
like
we're
pro
you're
processing
a
fast
and
while
there
is
a
project
like
fat,
Delta
and
just
for
that's
basically
production
for
writing
Kafka
topics
into
Delta.
A
At
least
it's
been
used
in
production
by
multiple
companies
right
now,
in
the
case
of
the
Delta
rust
itself
portion
of
it,
that
part
is
still
very
much
experimental
because
you
know
it
it.
We
literally
what
two
months
ago
or
three
months
ago,
had
started
basically
putting
that
code
base
in
we've
stole
not
stole
but
took
some
of
it
from
Kafka,
Delta,
ingest
and
added
or
added
a
separate
set
of
code
based
in
order
to
get
up
and
running
by
the
way.
A
There
is
a
project
later
on
to
reconcile
the
two
so,
but
nevertheless
the
context
is
like
for
smaller
data.
The
you
know
this
it
all.
It
absolutely
makes
sense
for
larger
systems.
It
may
still
make
sense
to
use
Pi
spark,
though
over
time
we
suspect
that
that
even
that
might
start
changing,
so
did
I
sort
of
wrap
it
up
correctly.
I
guess
yeah.
B
B
And
I
I
forgot
to
mention
on
the
reading
on
the
Reading
part
of
the
the
data
arrest,
you
can
found
a
panda
integration,
so
you
can
just
easily
go
to
Panda
and
have
an
Explorations
for
the
data
scientists
or
data
analytics
to
say.
Okay,
maybe
I
need
I,
know.
Panda
I
know
how
to
use
it
so
plugging
to
data
arrest
is
really
convenient
and
you
can
have
the
time
travel
using
the
data
sets
and
so
on.
B
A
Yeah
I
don't
know,
actually
that
reminds
me.
I'm
gonna
have
to
Ping
afterwards
we're
gonna
have
to
actually
write
the
blog
about
exactly
about
that
by
the
way,
with
the
machine
learning
pandas
still
to
RS.
Just
because,
even
though
we're
seeing
a
lot
of
different
things,
it's
from
a
code
perspective
you're,
just
running
a
couple.
Imports.
B
A
Actually,
all
you're
really
doing
you're
you're
you're
calling
a
couple
of
imports
like
import,
Delta,
Delta
Lake
by
the
way
for
those
who
are
using
Delta,
the
spark
version,
It's
Delta,
slash,
I'm,
sorry,
Delta,
DASH,
Spark
versus
delta
Lake,
which
is
basically
it's
the
python
binding
for
Delta
rust.
So
it's
just
the
import,
Delta
Delta
Lake
and
import
pandas
and
then
okay
you're
Off
to
the
Races.
Basically,
so
it's
not
it
it
we're
explaining
it
horribly.
It's
actually
quite
simple:
to
run
yeah.
B
A
Actually,
as
a
final
call,
this
this
one
I
realized,
is
we're
it's
still
too
early.
Okay,
but
I
figured
why
not
just
for
the
fun
of
it
like
are.
Are
we
trying
to
figure
out
right
now
how
to
distribute
Delta
rest
better
with
ballista
or
or
is
that
our
current
approach?
Are
we
thinking
of
different
approaches
right
now,
I'm,
actually
not
sure
about
this.
One
myself
personally.
B
A
really
good
question,
I,
don't
know
actually
maybe
as
a
next
step.
I
don't
know,
but
for
now
I
think
it's
the
right
to
our
implementation
would
be
is
a
good
one
with
strong
ones
that
you
can
use
on.
Prediction
can
be
really
nice
steps
and
yeah
the
seven
one
would
be
maybe
another
more
multiple
data
is
working
together
or
also
returning
into
another
crate.
It
could
be
a
nice
depth.
A
Now,
no,
even
though
I
did
call
that
out
no
note
the
fact
that
in
many
of
the
UK's
back
market
and
actually
many
of
the
U
cases
in
production,
really
I've
not
needed
to
distribute
rust,
I've
just
been
able
to
go
ahead
and
have
the
one
thing
and
usually
we
have.
Our
problems
are
whole,
like
as
an
am
I
saturating
the
network
connection
with
that
one
system:
okay,
so
yeah.
So
honestly,
that's
I'm
asking
the
question
just
more
out
of
fun.
Yeah.
A
B
For
the
for
the
reading,
actually
it's
working
well!
It
has
been
two
years
now
that
he's
running
in
production,
to
fetch
all
the
metadata
schema
filters
and
so
on
and
no
issue
on
this
side.
So
it's
really
strong
for
the
reader
on
position
case.
We
are
dealing
it
with
a
lot
of
pipelines
inside
black
market
on
daily
usage.
So
it's
like
for
other
writers,
not
the
same.
That's
why
you
don't
want
to
use
it
at
the
moment.
Yeah.
A
Sure,
as
a
way
for
us
to
for
again
for
everybody
watching
this
vidcast
or
joining
the
podcast
later
yeah,
you
know
come
join
us
go.delta,
dot,
IO
slack,
we
actually
are
very
active
with
our
conversations
there
you
find
bugs
you
have
some
better
ideas.
Absolutely
come.
Come
talk
to
us,
I
think
that's
probably
a
good
end
point
for
us
today.
A
We
we
I
I,
think
if
I
we
keep
talking.
You
and
I
will
just
invariably
go
back
to
talking
more
about
rust
anyway,
since
that
then
we're
gonna
bore
the
audience.
So
any
last
tidbits
of
advice
for
anybody
like
again
going
into
the
space
going
into
like
we
talked
a
lot
about
it.
Just
you
know
if
the
if
we're
gonna
do
the
simpler
version,
any
last
tidbit
of
advice,
if
I
wrote
anybody.
B
I
would
say,
is
that
simple
join
and
it
can
be
an
issue
it
can
be
just
fixing
a
small
part
of
the
Rose
cut
base.
Definitely
we
will
help
you
to
learn
and
to
find
a
good
way
of
solving
the
issue
with
us.
So
join
us.
It's
a
really
good
cut
days
and
every
team
members
is
a
is
really
nice,
and
so
it's
a
good
opportunity
to
work
with
cool
members.
A
Excellent
call
out
I
do
want
to
I,
do
want
to
call
it
the
fact
that
these
folks
are
so
nice
they're.
Even
tolerating
my
questions,
so
hey
we're
doing
great
here
Florian.
Thank
you
very
much.
I
really
appreciate
your
time.
So
again,
this
is
the
d3l2
vidcast
podcast
I'm
here
with
Florian
from
back
Market
I'll
use
a
Delta
luck,
Delta
Delta,
Lake,
committer,
maintainer,
focusing
on
the
rust
and
the
python
bindings
again
Florian.
Thank
you
very
much.
I
really
appreciate
your
time
today.