►
Description
In this session we will introduce the delta-rs project which is helping bring the power of Delta Lake outside of the Spark ecosystem. By providing a foundational Delta Lake library in Rust, delta-rs can enable native bindings in Python, Ruby, Golang, and more.We will review what functionality delta-rs supports in its current Rust and Python APIs and the upcoming roadmap.
We will also give an overview of one of the first projects to use it in production: kafka-delta-ingest, which builds on delta-rs to provide a high throughput service to bring data from Kafka into Delta Lake.
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Hi,
I'm
r
tyler
croy
and
today
I'm
going
to
be
talking
about
what
we've
been
doing
to
grow
the
delta
lake
ecosystem
with
rust,
python
and
some
more
some
more
cool
things
before
I
get
into
that,
I
want
to
sort
of
share
a
little
bit
about
who
I
am
why
I'm
here,
I've
been
an
open
source
contributor
for
a
long
time.
Open
source
is
incredibly
important
to
me.
A
It's
been
a
huge
part
of
my
career
and
I've
been
in
the
jenkins
community,
the
ruby
community
python
community
and,
most
recently,
the
rust
community
and
I've
had
a
lot
of
fun
in
the
rust
community,
which
is
part
of
why?
Why
we're
here
to
talk
about
delta
rs
my
day,
job
isn't
to
work
on
open
source.
A
Unfortunately,
my
day
job
is
I'm
the
director
of
platform
engineering
at
scribd,
which
is
an
organization
composed
of
the
core
platform
team
data
engineering
and
data
operations,
and
together
these
three
teams
support
a
fairly
large
data
platform
infrastructure
for
scribd
to
really
accomplish
our
mission
of
changing
the
way
the
world
reads
before
I
get
too
far
into
talking
about
how
we're
growing
the
delta
lake
ecosystem.
I
want
to
make
sure
that
we
do
a
little
bit
of
a
rehash
on
what
delta,
like
actually
is,
which
is
important
to
know.
A
There's
a
lot
of
great
content
at
data
in
ai
summit,
around
different
aspects
of
delta
lake
getting
started
with
it,
but
at
sort
of
the
most
basic
level.
What
you
really
need
to
know
is
that
delta
lake
is
really
parquet.
Files
on
one
hand
married
to
a
transaction
log,
on
the
other
hand,
and
those
two
concepts
together
really
allow
quite
some
interesting
and
sophisticated
behaviors
on
top
of
data.
A
The
parquet
files
give
you,
obviously
higher
performance
reads
on
data,
and
then
the
transaction
logs
allow
you
to
perform
acid-like
transactions
almost
like
you're
working
with
postgres,
for
example,
that
well-defined
transaction
log
allows
us
to
do
some
really
interesting
things
in
terms
of
sequencing
transactions,
rights,
reads,
updates,
etc,
and
it's
very
very
cool.
But
it's
also
pretty
simple.
If
you
go
to
this
link
that
I've
got
at
the
the
top
of
the
the
deck
this
protocol
md,
it's
the
open
specification
for
what
delta
lake
actually
is.
A
A
The
delta
rs
library
was
something
that
was
started
in
mid-2020
and
there's
a
number
of
reasons
on
why
it
was
actually
needed
at
scribd.
We
originally
proposed
the
the
delta
rs
project,
because
not
everything
that
we
do
needs
a
spark
cluster
at
the
time,
and
this
has
since
changed,
but
at
the
time
you
needed
spark
in
any
form
whether
that's
a
single
node
cluster
or
a
hundred
node
cluster
to
read
delta
files.
A
A
I
should
also
point
out
that,
for
a
lot
of
people,
getting
data
into
delta
lake
is
an
ongoing
and
important
challenge
for
us.
We
ingest
a
tremendous
amount
of
data
from
kafka
from
my
sql
databases
from
external
sources,
where
the
full
power
of
spark
might
not
be
really
bringing
anything
to
the
table
in
some
uses.
Use
cases
excuse
me.
A
A
A
A
That's
not
everything
around
why
we
chose
rust
rust.
I
personally,
like
I,
I'm
certainly
biased,
but
the
design
of
rust
allows
for
the
implementation
of
very
fast
and
safe
libraries
and
when
I
say
safe,
I
don't
just
mean
type
safe.
I
mean
concurrency
safe.
It's
actually
fairly
easy
to
build
a
high
performance,
multi-threaded
multi-processor
daemon
in
rust
because
of
some
of
the
the
inherent
design
of
the
rust
language.
A
The
other
thing
I
don't
think
is
is
appreciated.
Well
enough
about
rust.
Is
that
it's
very
easy
to
embed
so
anything
where
you
can
do
sort
of
c
to
my
scripting
language
of
choice.
Interop
rust
can
fit
in
there
without
with
relative
ease.
That's
because
there's
not
a
run
time
in
rust.
There's
no
garbage
collector!
There's
none
of
that
stuff
that
you
would
find
in
in
you
know:
node
python,
golang
or
some
of
these
other
languages
that
have
a
runtime
to
help.
You
manage
your
memory
because
it's
so
easy
to
embed.
A
A
A
It's
helped
me
learn
rust
and
has
helped
a
couple
of
other
individuals
within
scribd
also
come
up
to
speed
on
rust,
which
I
can't
recommend
enough
so
delta
rs.
It's
a
lot
of
things,
there's
a
lot
of
of
pieces
into
it,
but
from
a
code
standpoint,
it's
actually
not
that
big.
At
the
base
layer,
we
have
some
rust
space
dependencies
like
our
underlying
asynchronous
async
await
provider.
A
I
would
say,
ready
in
in
mid
2020
when
we
started
to
work
on
this,
and
so
scribd
has
been
working
with
one
of
the
aero
committers.
A
guy
named
neville
who's
actually
brought
parquet
crate
writer
support
into
the
parquet
crate
and
has
done
countless
improvements
to
the
aerocrate
to
make
sure
that
it
would
support
delta
rs's
use
case
and
then,
of
course,
to
support
the
you
know.
A
The
object
stores
that
we
need
to
interact
with
we've
got
risotto
for
interacting
with
s3
and
then
the
azure
sdk
to
make
sure
that
we
can
interact
with
those
data
stores
as
well,
and
then
you
see
right
in
the
middle
there's
this
thick
line.
That's
the
delta
lake
crate.
You
can
do
a
cargo
install
delta
lake
of
that.
A
If
you
want
to,
we
provide
some
simple
tools
out
of
the
box,
or
you
can
just
add
that
into
your
cargo
tunnel,
but
that
layer
is
the
layer
upon
which
all
of
the
native
bindings
into
say,
python
or
ruby
or
grilling
can
be
implemented
and
there's
a
couple
more
little
bits
of
rust.
There's
this
really
cool
library
called
maturin
materian,
I'm
not
really
sure
how
to
say
it
and
then
pi
o
three
that
make
it
really
easy.
Like
really
really
easy.
I
can't
understand
that
enough.
A
A
In
this
example,
all
we're
doing
is
just
opening
the
table
and
then
printing
it
it's
fairly
simple
to
work
with
delta
tables
within
within
delta
rs,
where
things
get
a
little
bit
more
interesting
and
you
will
have
to
make
some
decisions
around
what
library
or
tools
you
want
to
do.
Things
is
when
it
comes
to
query
those
tables,
because
delta
lake
is
a
transaction
log
plus
a
parquet
file.
A
So
let's
go
look
at
python
because
I
think
python's
a
little
bit
more
easy
to
understand
and
it's
a
little
bit
I'd
say
clear
for
us
to
use
as
an
example
in
slide
formats.
So
this
this
is
a
very,
very
short
program
at
the
bottom
right.
This
program
is
just
opening
up
a
table
printing
some
metadata
and
then
generating
a
pandas
data
frame.
Pandas
is
a
data
processing
library
in
python.
It's
also
a
really
really
wonderful
library.
A
I
can't
recommend
it
highly
enough,
but
in
this
case
my
audit
logs
delta
table
just
has
one
record
in
it.
So
we've
got
one
file
which
you
can
see
listed
there.
This
you
know
blah
blah
blah,
parquet,
snappy
dot
or
snappy
dot,
parquet,
that's
one
of
the
parquet
files
that
was
saved
to
delta
table
and
then,
when
I
load
that
and
query
it,
which
is
just
the
the
printing
of
the
data
frame
you
see
there.
Then
it
just
dumps
this
one
record
and
which
is
11
columns
et
cetera.
A
A
A
A
So
now
I'm
going
to
look
at
the
files
and
there's
only
two
files
in
this
table.
It's
pretty
straightforward
and
because
I
have
pandas
installed,
I'm
going
to
go
ahead
and
create
a
pandas
data
frame,
and
this
takes
a
second
because
it's
got
to
load
those
two
files
into
memory
and
then
I'm
going
to
look
at
the
help,
just
to
make
sure
that
this
is
what
I
think
it
is.
This
is
a
data
frame
and
I
can
start
to
query
it,
which
is
really
cool.
So
let's
go
ahead
and
query.
A
I'm
going
to
query,
you
know
for
a
true
field
and
the
benefit
of
doing
this
in
your
in
your
terminal
is
you
can
find
these
query
errors
in
this
case?
Python
true
is
a
capital
t,
and
then
you
can
start
to
query
using
that
and
all
of
this
sort
of
comes
together
to
be
this
really
useful
means
of
experimenting
with
delta
delta
tables.
Excuse
me
right
there
from
your
local
machine
all
right,
so
that's
kind
of
in
a
nutshell,
using
delta
lake
from
python,
which
is
to
me
it's
super
cool.
A
I
really
enjoy
working
with
delta
tables
right
from
my
python
interpreter
and
doing
some
sort
of
local
data
science.
If
you
will
before
shipping
something
into
production,
what
you
can
do
right
now
with
delta
rs
and
the
the
delta
lake
python
package.
So,
as
I
mentioned
before,
you
can
access
delta
tables,
whether
they're
stored
in
aws
s3,
azure
data
like
storage,
gen,
2
or
your
local
file
system.
In
order
to
access
those,
you
will
need
to
make
sure
that
the
right
environment
variables
are
set.
A
You
know
the
the
aws
underscore
access
key,
etcetera,
etcetera,
but
once
you
once
you
have
a
delta
table
loaded,
you
can
actually
read
by
the
partitions.
You
can
look
at
checkpoints,
you
can
actually
with
the
rest
binding
right
now.
You
can
read
streaming
updates
to
the
table,
which
is
pretty
cool,
and
we
also
have
some
preliminary
support
for
actually
writing
to
the
transaction
log
that
writing
to
the
transaction.
A
Log
is
still
fairly
early,
and
so
what
that
means
is
you
can't
quite
write
parquet
directly
through
the
delta,
rs
library
or
the
delta
lake
python
binding
just
yet.
So,
if
you
have
a
parquet
file,
that's
been
written
somewhere
else
you
can,
or
by
written
by
something
else
or
by
another
piece
of
code.
You
can
then
make
the
transaction
log
entry
using
delta
rs
to
update
the
delta
table
itself
right
now.
Delta
rs
also
can't
really
create
checkpoints.
A
This
doesn't
prevent
the
table
from
being
red,
it
doesn't
create
invalid
delta-like
tables.
It
just
has
a
little
bit
of
a
performance
impact
and
if
you
take
a
look
at
that
protocol.md
that
I
referenced
earlier,
you
can
read
more
about
what
checkpoints
are
and
how
they
improve
performance
in
a
delta-like
table.
A
We
also
can't
optimize
right
now
and
optimize
typically
means
two
things:
there's
this
command
that
you
can
use
in
the
databricks
runtime
to
basically
bin
pack,
a
number
of
parquet
files.
So
what
that
means
is,
if
you
say
over
the
course
of
five
minutes,
I
created
10
file
or
not
10
files.
Let's
say
100
files
and
they're
all
fairly
small.
A
An
optimize
command
can
come
through
and
take
those
100
files
and
turn
those
into
one
or
two
files,
depending
on
what's
what
size
appropriate
so
that
your
query
performance
doesn't
suffer
because
your
query,
you
know
whatever's
querying
it
needs
to
access
100
different
files.
It
can
just
access
those
two
or
three
and
load
those
into
memory.
There's
another
part
of
optimize
that
we're
not
likely
going
to
be
supporting
anytime
soon,
because
it
is,
it
is
a
fairly
challenging
thing
to
do
and
does
require
some
compute
resources
and
that's
the
z
order.
A
So
all
of
that
said,
delta
rs.
Is
there
it's
ready
for
you
to
start
using
if
you're
in
python,
you
should
just
do
a
pip,
install
delta
lake
and
pandas
so
that
you
can
actually
work
with
the
data
like,
like
I
showed
in
my
demo,
and
I
also
wanted
to
recommend
this
book.
The
python
data
analysis
book,
which
should
give
you
a
good
introduction
and
and
overview
of
using
pandas
to
do
data
analysis
in
python.
A
If
you're
like
me,
using
rust,
you
can
just
add
the
delta
lake
dependency
to
your
cargo
tunnel
and,
of
course,
you
know,
I'd
highly
recommend
the
rust
book
if
you're
interested
in
getting
into
rust-
and
I
also
wanted
to
mention
that
both
of
these
titles
are
also
available
on
scribd.com
kind
of
what
we
do
is
is
make
really
interesting
content
like
this
available.
If
you
dig
around
on
scribd.com
you'll,
also
find
a
lot
of
really
interesting
documents
or
academic
papers
around
both
delta
lake,
but
also
data
analysis
and
data
processing.
A
With
that
said,
let's
talk
about
what's
coming
up
next,
there's
some
work,
that's
in
very
active
development
right
now
in
a
project
called
kafka
delta
ingest,
which
the
the
link
is
at
the
top
of
the
screen.
This
is
something
that
scribd
is
contributing
quite
a
bit
to,
of
course,
just
like
delta
lake.
It's
it's
open
source
and
the
goal
for
kafka
delta
ingest
is
to
make
it
show
that
we
can
as
quickly
and
effectively
bring
data
from
a
kafka
topic
into
into
delta,
like
tables
it's
still
a
little
bit
early.
A
We've
got
some
good
single
single
mode
support
working
right
now,
and
this
is
heavily
dependent
on
delta,
rs
and
so
to
a
certain
extent.
Delta,
rs
and
kafka
delta
ingest
have
been
developed
sort
of
side
by
side,
and
if
you
ever
find
yourself
developing
a
library,
I
can't
recommend
this
sort
of
setup
enough,
because
what
happens
is
we
find
new
and
interesting
things
or
or
use
cases
in
kafka
delta
ingest
that
we
then
need
to
or
want
to
bring
into
delta
rs
to
make
delta
rs
a
better
library
to
under
underpin?
A
You
know
multiple
different
use
cases,
not
just
kafka
delta,
ingest
kafka
delta
ingest
also,
I
should
be
very
clear-
is
not
doing
any
stream
transformations,
nor
is
nor
are
we
planning
to
it.
To
do
so.
Excuse
me,
spark
streaming
is
very
good
at
stream
transformation
and
processing,
heaps
and
heaps
of
data
on
the
fly
as
it
streams
through
in
in
batches.
A
What
kafka
delta
ingest
is
really
aiming
to
do
is
to
get
data
from
one
file
descriptor,
that's
kind
of
kafka
data
on
it
to
another
file.
Descriptor.
Writing
that
data
into
aws
s3,
for
example,
there's
a
number
of
different
problems
that
we're
trying
to
solve
with
with
kafka
delta
ingest.
And
if
you
go
to
the
github
link
above,
you
can
actually
look
at
the
design
document
there
as
well
to
get
a
better
sense
of
this.
A
But
I
wanted
to
highlight
the
the
design
here
spark
is,
I
would
say,
challenging
to
run
for
variable
throughput
topics
and
at
scribd
we
have
a
lot
of
variable
throughput
topics.
You
know
the
data
that's
coming
in
might
be
event:
data
from
scribd.com,
where
the
traffic
is
going
to
ebb
and
flow
throughout
the
day
and
what
that
means,
if
we're
to
allocate
spark
streaming
infrastructure
to
solve.
That
is
that
we
either
have
way
over
provisioned
infrastructure,
supporting
that
or
we
run
into
cases
where
you
know
infrastructure
might
be
underprovisioned.
A
You
know
the
cluster
might
not
be
big
enough
to
handle
the
load
coming
in
and
we
get
alerts
and
pages
from
that
kafka
delta.
Ingest,
in
contrast,
is
designed
up
front
to
be
auto
scalable,
which
has
made
some
interesting
challenges
for
making
sure
that
we
get
the
design
correct
to
allow
for
multiple
delta
rs
based
writers
to
be
working
in
concert
together
and
making
sure
that
we're
getting
data
flowing
correctly
and
properly
into
a
delta
table.
A
You
know,
actually
work.
So
I
encourage
you
to
check
out
delta
rs
the
github
link.
Is
there
pip,
install
delta
lake
or
cargo
install
delta
lake
either
will
work
fine
if
you're
interested
in
contributing,
I
would
love
to
see
golang,
bindings
or
node
bindings
so
that
we
can
grow
delta
lake
outside
of
the
jvm
ecosystem
outside
of
the
python
ecos
ecosystem
and
really
spread
it
as
far
as
we
can.
But
with
that
said,
thank
you
for
your
time.