►
From YouTube: Tech Talk | Diving into Delta-rs: kafka-delta-ingest
Description
Delta Lake committers Christian Williams and R. Tyler Croy from Scribd discuss with Denny Lee from Databricks the technical and business requirements around the Delta Rust API project: kafka-delta-ingest.
This project aims to build a highly efficient daemon for streaming data through Apache Kafka into Delta Lake and has been in production at Scribd for the last four weeks after six months of active development.
Come to learn about why they built it and how it's going.
Resource links:
https://github.com/delta-io/kafka-delta-ingest
https://kafka.apache.org/
https://delta.io/
Speakers:
R. Tyler Croy leads the Platform Engineering organization at Scribd and has been an open source developer for over 14 years. His open source work has been in the FreeBSD, Python, Ruby, Puppet, Jenkins, and now Delta Lake communities. The Platform Engineering team at Scribd has invested heavily in Delta and has been building new open source projects to expand the reach of Delta Lake across the organization. Tyler is also a Databricks Beacon.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University.
Christian Williams is a senior engineer on Scribd's Core Platform team. He has done application and data engineering for 15 years working with a wide range of languages and platforms, most recently working with Kafka, Delta Lake, Rust, and AWS to deliver streaming data ingestion. Before working in software Christian was also one of the fastest sandwich artists in the greater Jacksonville area. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
B
Thank
you
very
much.
Karen
really
appreciate
it
so
hey
before
we
start
going
into
it.
I
wanted
to
just
give
christian
and
tyler
just
a
couple
minutes
to
introduce
themselves
and
then
we'll
dive
right
into
kafka
delta
interest.
So
I'll
start
with
christian
christian,
why
don't
you
just
do
a
quick
introduction?
Let
the
audience
know
who
you
are
sure.
C
D
Yeah,
I'm
the
director
of
platform
engineering
here
at
scribd.
I
oversee
core
platform
data
engineering
and
data
operations
and
those
three
teams
are
responsible
for
the
data
platform
infrastructure
and
some
of
the
key
applications
and
services
on
top
of
them
like
kafka
delta
and
just
which
we're
going
to
talk
about
today.
B
Excellent
thanks
very
much
christian,
oh
tyler,
to
me
so
now,
quick
note
for
since
we
have
multiple
platforms
here
for
zoom.
Please
put
your
questions
in
the
q
a
section
for
linkedin.
Please
put
the
questions
directly
in
the
comments
same
as
youtube.
Now
there
is
a
slight
delay
with
linkedin
due
to
the
way
our
restream
io
works,
but
nevertheless
we
will
do
our
best
to
answer
the
questions.
B
C
A
D
Originally,
we
deployed
spark
streaming
applications
to
do
that
and
we
saw
some
flaws
which,
which
I'll
talk
about
a
little
bit
later
with
that,
and
so
we
sort
of
set
off
on
this
path
of
like
you
know
what,
if
we
did
a
rust
on
it
and
and
sort
of
that's
where
we
that's.
What
brought
us
to
here?
Basically,.
B
A
D
D
We
actually
started
with
our
syslog,
which
is
off
the
the
shelf
tool
and
ran
into
a
number
of
problems
with
it,
and
so
I
ended
up
building
this
little
daemon
and
put
it
out,
put
it
into
production
and
sort
of
didn't,
spend
a
lot
of
time,
thinking
about
optimization
or
or
making
it
run
fast,
and
it
just
ran
fast,
and
that
gave
at
least
me.
I
can't
speak
for
anybody
else.
D
That
gave
me
a
lot
of
confidence
that
we
could
use
rust
for
other
data
high
throughput
data
ingestion
problems
and
not
spend
a
lot
of
time
worrying
about
the
correctness
of
the
application
or
the
performance
of
the
application,
because
rust,
a
lot
of
the
characteristics
of
rust,
just
sort
of
in
court,
encourage
you
to
do
sort
of
the
right
things
from
the
get-go
that
lead
to
very
high
performance
daemons,
which
is
interesting
from
a
data
platform.
Standpoint.
D
He
started
like,
I
think
he
I
think
I
nerd
sniped
him
and
he
wrote
the
first
sort
of
delta
log
protocol
parsing
code
for
delta
rs
over
a
couple
of
weekends.
Just
because
it's
like
whoa,
I
wonder
if
we
could
do
this
in
rest.
I
don't
know
it's.
I
hadn't
looked
into
delta
lake
that
much
at
the
time,
and
so
he
sort
of
got
the
ball
rolling
and
then
late
2020.
We
started
to
see
really
like.
There
was
a
little
bit
more
pickup.
D
B
D
We
have,
we
have
kafka
delta
ingest
running
in
production,
so,
yes,
I
would
consider
a
production
ready
for
reader
and
writer
use
cases
in
rust.
I
there
are
production,
use
cases
for
delta
rs
in
python,
for
the
read
side
of
that,
so
one
of
our
one
of
the
other
committers
again
named
florian
from
backmarket,
I
believe,
he's
over
in
france.
They
have
they've,
had
delta
rs
the
python
bindings
and
that's
just
a
pip
install
of
delta
lake
they've
had
that
running
in
production.
B
Rock
on
so
which
I
think
this
needs
to
natural
segue
to
talk
about
kafka
delta
ingest,
which
is
when
we
you've
already
described
a
little
bit
about
why
you
wanted
to
do
that
so
christian
once
you
show
some
of
the
architecture,
diagram
of
it
for
starters
and
then
tyler.
If
you
want
to
maybe
start
off
with
the
rehashing
like
like
what
are
the
challenges,
what
are
the
issues?
What
what
led
you
to
go
down
this
route.
D
C
It
makes
it
easy
for
us
to
just
let
use
kafka
to
manage
partition
assignment
for
each
member
of
the
group
and
then
internally
we
manage
rebalance
events
which
happen.
If
you
know
new
consumers
are
added
to
the
same
group,
so
yeah
in
the
diagram
on
the
left.
You
see,
we've
got
one
kafka
topic
on
the
right.
We
have
one
delta
table.
We
could
have
one
or
many
kafka
delta
interest
workers
ingesting
from
that
topic
and
writing
to
the
same
delta
table.
D
I
want
to
mention
something
on
this.
If
you
don't
mind
christian
real,
quick,
I
was
talking
about
this
use
case
with
somebody
yesterday,
and
this
pattern
has
actually
been
replicated
a
number
of
times
where,
if
you
think
about
our
streaming,
data
is
coming
in
in
the
sort
of
like
top
line
of
kafka
kafka
delta
ingested
elderly
table.
That
means
we
get
data
to
customers
that
want
to
use
it
as
quickly
as
possible,
but
within
the
context
of
like
a
date,
stamp
partition.
D
It's
just
not
optimized
for
queries,
so
we're
sort
of
optimizing
in
a
day
for
freshness
and
then
afterwards,
we're
running
this
delta,
optimize
and
vacuum
to
make
sure
that,
for
the
long-term
query
needs
that
we
have,
you
know
optimizing
optimizing
for
performance,
and
so
by
combining
these
two,
we
basically
get
as
much
of
the
best
of
both
worlds
as
we
can,
for
you
know,
rapid
delivery
of
data
and
then
high
performance.
As
you
know,
the
table
gets
larger
or
we
look
back
in
historical
partitions.
B
B
Even
before
we
even
get
to
that,
but
we're
still
talking
about
the
overall
architecture,
one
of
the
things
that
I
I
did
want
to
talk-
and
maybe
it's
a
little
too
early
or
a
little
too
late.
But
it
was
the
the
issues
around.
For
example,
how
do
with
coffee
delta
ingest
you're
definitely
doing
concurrently
consistent
rights?
So
what
were
some
of
the
issues
that
or
that
you
were
running
into
when
it
came
to
that.
C
C
So
the
first
thing
we
do
is
we
we're
using
this
crate,
called
rd
kafka,
which
is
an
awesome,
rust
crate
for
kafka,
consumption
and
production.
First
thing
we
do
is
we
we
start
a
run
loop.
We
await
a
message
when
a
message
is
received,
we
deserialize
it
after
we
deserialize
it.
We
have
some
little
basic
transformations,
which
I'll
add
some
detail
to
later
to
provide
some
minor
enrichments
to
the
message
if
they
are
desired
and
then
we
buffer
it
the
reason
we
buffer
messages,
we'll
I'll
talk
about
this.
C
More
later
as
well-
but
you
know
just
in
general-
we're
we're
trying
to
create
like
a
tuning
dial
for
people
to
use
so
that
they
can
like
kind
of
choose
between
read
availability
versus
query
execution
time,
so
we
buffer
messages
for
a
while.
We
check
a
few
like
hard
parameters
to
decide
when
we
want
to
write
our
buffers
to
a
parquet
file.
C
C
I
mean
I'm
a
nerd
right.
I
count
myself
in
that
group,
but,
like
I-
and
I
agree
like
squeezing
every
last
inch
of
performance
out
of
over
the
wire
protocols,
is
it's
a
really
great
way
to
spend
your
day,
but
json
is
just
it's.
It's
such
a
nice
common
denominator
that
everybody
in
our
organization
is
very
familiar
with.
They
were
using
it
before
I
got
here.
Writing
it
to
kafka
christian.
D
C
D
A
C
C
C
D
You
did
sort
of,
and
I
I
want
to
make
sure
that
we
keep
the
appropriate
depth
here.
So
we
don't
give
like
I
I
guess,
go
to
like
you're,
confident,
delta,
ingest
expert.
You
misha
and
qp
know
this
stuff
in
and
out
yeah,
and
it
is
a
complex
project.
I
want
to
make
sure
that
we
don't
get
too
deep
into
the
like
how
kafka
delta
ingest
works,
gotcha.
D
Level
like
like
this
flow
is
actually
really
useful.
I
think,
because
we've
got
that
deserialize
and,
like
we
cue
up
these
things,
how
does
like
one
process
is
what
you
had,
I
think
on
the
previous
one.
How
does
this
work
when
you've
got
a
lot
of
data
coming
in?
Maybe
you
could
talk
a
little
bit
about
that
like
there's.
A
lot
of
data
coming
in
you've
got
this
sort
of
multiple
dotted
lines
here,
like.
C
C
So
we
hit
this
message
transform
step
and
this
this
hits
on
one
of
one
of
the
other
questions
that
you
mentioned.
The
the
reason
we
have
transformations
is
you
know
we.
We
want
to
provide
this
little
fuzzy
space
where
we
we
can
help
connect
the
dots
between
front-end
applications
that
don't
want
to
do.
Don't
want
to
know
too
much
about
the
data
warehouse
and
we
want
to
also
ease
the
downstream
burden
of
transforming
a
super
raw
ingestion
table
into
something
useful
for
downstream
jobs.
C
So
you
know
you
could
you
could
think
of
these?
As
you
use
cases
as
a
frontend
application
developer,
I
want
to
send
messages.
I'm
handling
straight
to
kafka
without
doing
extra
work
for
fields
that
are
required
by
the
warehouse
schema
so
that
I
can
stay
focused
on
front
end
and
not
have
to
learn
too
much
about
warehouse
from
a
downstream
data.
Engineer
perspective,
I
don't
want
to
have
to
include
the
same
calculated
query
and
every
query.
C
A
D
D
C
All
right,
let's,
let's
talk
about
this
one
a
bit,
so
this
sits
on
some
some
things
we
haven't
introduced
yet,
but
you
know
we're
going
from
one
kafka
topic
to
one
delta
table.
A
given
delta
table
does
have
multiple
partitions
that
we
could
need
to
write
to
when,
when
we're
running
kafka,
delta,
ingest
and
so
the
way
partitioning
works
in
kafka.
C
You
know
you
can
use
a
message
key
to
send
events
to
specific
kafka
partition
partitions.
But
most
often
we
do
not
do
that.
So
you
get
like
this
random
distribution
as
a
producer
when
you're
writing
messages
to
kafka-
and
you
know
so
the
diagram
we're
looking
at
right.
Now
we
have
one
topic:
we
have
two
kafka
delta
ingest
workers.
Our
topic
has
five
partitions:
three
of
them
are
going
to
one
of
the
workers
and
the
other
two
is
going
are
going
to
the
other
worker.
C
So
this
actually
hits
on
one
of
the
challenges
that
that
we
face
in
kafka
delta,
ingest.
We
we
need
to
be
able
to
consume
for
multiple,
multiple
partitions
and
we
need
to
be
able
to
write
to
multiple
delta,
partitions
and
delta
partition.
In
our
case
is
usually
a
date
stamp,
so
you
know
this
is
especially
useful
in
when
we're
crossing
time
boundaries
so
like.
C
At
the
same
time,
the
other
thing
this
diagram
shows
is
the
fact
that
each
worker
has
a
an
exclusive
subset
of
the
total
number
of
partitions
assigned
to
it,
and
so
this
has
an
impact
on
what
our
transaction
file
that
we
write
out
and
delta
ends
up
looking
like-
and
this
actually
leads
right
into
this.
This
next
slide.
B
This
is
perfect
before
we
get
into
this.
I
did
want
to
do
a
high
level
question,
if
that's
okay,
which
sort
of
leads
to
what
you're
talking
about,
which
is
one,
is
that,
if
you
run
into
this
is
a
question
from
the
linkedin
communities
like
how
do
you
actually
like,
like
because
people
are
so
used
to
the
idea
of
kafka
and
spark
where,
in
terms
of
managing
offset,
you
know
what
happens
when
the
failure
occurs?
B
C
C
C
He
had
a
conversation
with
some
databricks
folks
and
we
ended
up
doing
a
much
different
thing
that
really
simplified
our
job
in
kafka,
delta,
ingest
and
yeah.
It
all
boils
down
to
this
texan,
app
action
that
we
that
it
is
specified
by
the
delta
protocol
and
basically
when
we
do
a
write
to
delta.
We
include
multiple
texan
actions,
text,
I'm
pronouncing
txn
as
texan,
so.
B
D
C
C
C
So
we
in
this
example
here
we
have
a
topic
that
has
nine
partitions
being
handled
by
a
single
worker
and
we've
written
a
text
in
action
for
each
partition,
and
the
version
identifier
that
we
include
in
each
action
is
the
offset
that
we
left
off
from
from
kafka.
That
way,
if
a
process
terminates
and
when
it
comes
back
up,
it's
assigned
one
of
these
partitions.
Let's
say
it's
assigned
partition,
eight
six
and
five
it
it
can.
Go!
Look
in
the
delta
transaction,
long
to
see
all
right.
C
C
So
then
the
next
slide
here
is,
you
know
any
given
entry
in
the
delta
transaction
log
it
it
doesn't
only
conflict
include
text
and
actions.
It
also
includes
the
actual
files
that
were
written,
and
so
those
nine
texan
actions
correspond
to
this
one
ad.
So
this
worker
was
consuming
from
nine
partitions.
It
wrote
a
data
file
and
that
file
is
referenced
here
in
the
path.
C
B
Perfect,
okay,
so
I'm
actually
to
move
things
along.
I
do
want
to.
I
do
want
to
talk
to
tyler
a
little
bit
about
the
cost
structure
and
things
of
that
related
to
this,
but
before
I
do
that,
I
did
want
to
ask
one
question
from
linkedin.
Specifically:
is
that
well
in
in
your
case,
because
I'm
sure
it
happens
often,
how
do
you
deal
with
duplicates
like
what
of
coming
in
or
is
it
in
your
scenarios?
B
C
Use
offset
filtering
to
deal
with
duplicates
so
yeah.
I
I
put
it
this
way.
If
there
is
a
duplicate
in
kafka,
we
aren't
going
to
handle
it.
You
you
kafka,
does
provide
duplicate
prevention.
Settings
kafka
is,
you
know
like
the
original
kafka.
Spec
was
at
least
once
message
message
delivery,
but
they
now
have
settings
that
you
can
use
if
you
choose
to
to
do
exactly
once
message
delivery.
B
C
D
That's
that's
where
all
the
locking
is
going
on,
and
just
one
of
the
things
that
has
been
really
cool
to
watch
over
the
last
last
few
weeks.
As
we've
been
rolling
this
into
production
and
getting
things
sorted,
is
we've
actually
brought
some
of
that
dynamodb
locking
logic
into
our
into
item
number
two
here
where
we're
doing
the
delta
optimize
stuff
and
other
things
that
are
interacting
with
this
delta
light
table.
D
D
I
had
something
else
in
the
notes
about
the
locking
that
I
wanted
to
talk
about.
Oh
yeah,
let
me
mention
this.
If
you
use,
if
you're
running
on
s3
put,
if
absent,
is
the
missing
thing
so
you've
got
some
eventual
consistency,
concerns
and,
if
you're
running
on
data
bricks,
they
actually
sort
of
like
make
this
problem
disappear
because
they
have
the
s3
commit
service,
that's
sort
of
hiding
behind
the
scenes.
D
But
in
our
case
you
know,
we've
got
this
non-spark
writer,
we
don't.
You
know
we're
not
able
to
really
interact
with
that
s3
commit
service,
and
that's
why
we
added
this
dynamodb
locking
mechanism
to
ensure
that
we
still
had
consistency
when
we
were
doing
the
rights
to
the
delta
table,
and
I
want
to
take
while
I'm
just
sort
of
like
going
through
this
topic.
Lucas
on
zoom
actually
asked
I'm
going
to
answer
one
of
these
questions.
D
Can
you
go
to
slide
2
christian,
that
diagram?
Just?
Why
is
there
an
intermediate
step
to
flush
the
parquet
files
first
and
why
not
direct
right
directly
to
the
delta
table,
so
the
structure
of
a
delta
table
is
actually
parquet
files
sitting
on
object,
store
adjacent
to
a
json
transaction
log,
and
so
we
actually
have
to
write
the
parquet
files
first
in
order
to
make
the
write
to
that
delta
log
with
the
appropriate
like
this
is
the
file
that
actually
contains
the
data.
D
So
on
the
reader
side,
when
spark
or
something
else,
comes
in
and
reads
the
delta
table,
it's
actually
reading
that
json
file
and
then
looking
for
all
these,
like
ad
actions
that
christian
had
mentioned
to
say,
like
oh
here,
are
the
actual
park
k
files
that
make
up
this
table
now.
I
need
to
go
grab
those
parquet
files
and
load
those
into
memory
and
do
whatever
I've
got
to
do
with
them.
C
Yeah,
we
do
it
in
the
one
with
the
green
four
circle.
That's
where
we
obtain
the
the
dynamodb
lock
and
everything
you
said
before
is
right
on
the
money.
Basically,
we
we
can
write
parquet
files
to
the
same
directory
all
day
long
and
a
delta
query
will
not
read
them
until
it
sees
it
in
the
transaction,
yeah
so
yeah.
So
so
we
obtain
that
lock
when
we're
completing
the
transaction,
and
then
we
we
write
the
transaction
file
and
that
tells
it
basically
provides
the
appropriate
snapshot.
D
And
that
delta
table
by
the
way
it
basically
looks
like
an
append,
only
delta
table,
so
we
only
have
appends
going
into
it.
So
we
can.
We
can
upload
those
those
parquet
files.
You
know,
let's
say:
we've
got
a
100
megabyte
parquet
file
and
that
takes
a
second
or
two
to
upload.
That
can
happen
for
many
of
these
kafka
delta
ingest
processes
that
might
be
running
but
where
we
really
have
to
do
that.
Coordination
is
on
that
json
file.
D
D
Yeah
delta
rs
can
support
both
of
those.
Now
right.
I
think
I
remember
seeing
the
gcs
so
the
google
cloud
storage
support
come
in
for
the
delta
rs.
In
theory,
kafka
delta
ingest
can
work
on
another
cloud
provider,
but
we
just
haven't
done
it
yet.
So
the
kafka
delta
ingest
works
well
in
production
in
aws
like
join
us
in
slack.
If
you
want
to
talk
about
how
we
can
bring
it
to
another
cloud,
but
scribd
is
deployed
in
nato.
Yes,
so
that's
where
we're
focusing.
B
Yeah
yeah
and
one
thing
I
did
want
to
add,
basically
because
you
this
the
the
need
for
a
lock
store
here.
A
la
dino
db
is
specific
because
of
s3
in
the
case
of
the
other
cloud
providers,
at
least
in
terms
of
azure
and
google
cloud,
they
don't
have
they
they
actually
have,
because
this
is
transactional
consistency,
so
at
least
from
a
lock
store
perspective.
That's
not
needed
the.
A
B
C
B
And
one
thing
to
notice
by
the
we
are
working
with
the
community
for
open
sourcing,
the
ns3
commit
service,
basically,
and
so
there's
actually
both
the
work
that
these
gentlemen
have
been
doing
and
also
there's
the
work
that
has
been
run
written
by
some
of
the
folks
over
at
samoa
tv
to
actually
go
ahead
and
do
that
as
well.
So
we're
still
working
on
with
the
community
in
terms
of
open
sourcing
that
portion
of
it
as
well.
So
just
as
a
fyi.
D
B
Yeah,
okay,
so
I
actually
want
to
switch
a
little
bit
because-
and
this
is
more
directed
to
you-
tyler
like
basically
to
talk
about
the
operational
concerns
like
in
terms
of
an
apple
apple's
comparison
of,
like
you
know
of
sparks
to
rust,
about
how
why?
How
has
this
massively
benefited?
Scribd
your
environment
by
doing
building
this
the
system
here.
A
D
D
D
That's
going
to
stay
always
on
at
the
time,
and
I
believe
this
is
still
the
case
unless
databricks
has
announced
something
in
the
last
you
know
week
or
so
that
I
wasn't
told
about
there's
no
auto
scaling
as
well
with
spark
structure
streamings
at
least
deployed
on
the
database
platform,
but
I
don't
think
I've
seen
spark
structure
streaming,
auto
scale
anywhere
else.
So
when
we
look
at
data
streams
that
are
a
variable
throughput,
we
had
a
lot
of
operational
challenges
early
on
with
getting
clusters
to
the
right
size.
D
So,
as
our
data,
great
data
rates
would
grow,
we
would
reach
this
sort
of
tipping
point
where
applications
would
fall
behind
and
we'd
see.
Consumer
lag
spike
in
datadog
and
we'd
have
to
go
basically
redeploy
the
spark
streaming
application
with
a
newer
cluster
with
more
resources
in
order
to
keep
up
with
the
data
growth,
and
you
know
getting
alerted
on
a
weekend
or
in
the
evening
that
you
know
your
spark
streaming.
Application
can
no
longer
handle
the
throughput
of
data
that's
coming
at.
D
It
is
not
my
ideal
way
to
be
spending
my
time,
but
on
top
of
that,
it's
basically
required
a
little
bit
more
care
and
feeding
as
an
application.
As
far
as
an
application
goes,
as
opposed
to
some
of
the
things
that
we
would
deploy
into
say,
ecs,
where
we
sort
of
set
up
auto
scale
configurations,
we
wrap
things
up
in
a
container
of
some
form.
D
We
deploy
it
and
then
we
kind
of
forget
about
it
until
something
goes
wrong
so
sparks
like
for
an
online,
you
know
high
throughput
application
spark
streaming
that
does
no
transformations.
You
know
nothing.
Interesting
is
going
on
we're
not
using
any
of
the
power
of
spark
to
get
data
from
kafka
into
delta.
You
know
the
overhead
of
spark
was
just
not
right
for
the
problem
domain.
B
Yeah
and
one
thing
I
definitely
want
to
add
you're
going
to
allude
to
that
real
soon,
especially
because
I
will
even
help
to
peg
it,
which
is
like
it's
the
cost
right.
But
I
I
sort
of
want
to
overemphasize
that
point.
Like
yeah,
you
know
first
I'll
defend
dataworks
a
little
bit
just
a
little
time
bit,
which
is
to.
B
Do
it
a
little
bit
right,
which
is
basically
yes,
we
definitely
heard
your
concerns
and
you're,
not
the
only
one
so
in
terms
of
the
auto
scaling,
so
that
is
that
it
nothing's
been
announced
you're
absolutely
correct,
but
that
is
something
that's
being
actively
worked
on.
By
the
same
token,
even
if
we
had
this
service
available,
at
least
for
me
putting
you
know
putting
on
my
head,
my
engineering
hat
as
opposed
to
the
databricks,
had
right.
B
It
still
wouldn't
make
much
sense
right
the
because
of
exactly
what
you
said:
there's
no
transformations
going
on
so
you're
just
going
to
be
able
to
write
from
kafka
directly
to
delta
lake
and
which
really
leads
to
the
performance
and
the
cost.
So
why
don't
we
dive
into
that?
Well
like
what
were
the
massive
performance
and
cost
benefits
for
by
by
using
the
service
that
you've
built.
D
Yeah
on
the
on
the
spark
comment
like
we,
we
have
other
streaming
applications
in
production
that
are
doing
like
right
right
and
like
they're,
doing
real
work.
I
one
of
the
things
that
that
really
motivated
me
to
to
start
looking
at
this
as
a
possibility
is
we
sort
of
chose
spark
because
we
had
to
have
spark
to
write
into
delta
lake
if
it
weren't
for
that
requirement,
this
use
case
would
have
never
been
in
spark,
because
it's
just
not
a
spark
use
case.
D
You
know
doing
awesome,
query
things
is
where
spark
really
shines
so
we
have.
We
haven't
completed
our
migration
of
all
of
our
data
streams
over
into
kafka
delta,
ingest.
We've
got
a
a
number
that
are
over
there
and
chris
and
I
were
doing
math
yesterday,
which
was
really
funny
because
we're
both
terrible
at
it
like
wait.
This
can't
be
right
all
right.
Let's
run
this
again,
just
to
make
sure
that
we're
we
had
something
to
to
share
that
was
realistic.
D
But
when
we
look
at
the
actual
the
resource
utilization,
we
basically
can
fit
a
hell
of
a
lot
more
messages
per
second,
and
you
know
there
is
going
to
be
some
variability.
I
use
messages
per
second,
because
that's
the
the
useful
metric
that
we
have,
but
you
know,
different
width
fields
or
different
width.
Rows
will
will
sort
of
perform
a
little
bit
differently
or,
if
you've
got
very
painful
things
to
deserialize
and
json.
That
will
behave
differently,
but
we
you
know
we
can
fit.
You
know
thousands.
D
D
You
know
setting
up
ecs
with
auto
scaling,
and
then
you
know
this
is
just
a
daemon
that
looks
like
a
bunch
of
other
daemons
that
we
have
that
are
processing
data
and
bringing
it
into
the
data
platform
and
that's
operationally
as
far
as
the
team
goes
and
sort
of
the
skills
that
we've
got.
That's
so
much
easier
of
a
problem
like
so
many
more
people
on
the
planet
are
deploying
docker
containers
into
ecs
very
successfully
reliably.
D
You
know,
day
in
and
day
out,
as
opposed
to
spark
streaming
applications,
which
is
a
little
bit
more
of
a
niche
sort
of
domain
to
get
really
good
at,
and
so
I'm
expecting
that
we're
going
to
see
a
lot
less
people
time
that
goes
into
maintaining
these
these
these
daemons,
because
it's
just
gonna
like
once.
It's
up
and
running
we're
just
gonna
leave
it
alone,
and
it's
going
to
continue
ingesting
data
until
we're
done
with
it.
B
With
your
current
set
of
workloads
and
18
to
20
of
the
original
ingestion
cost
so
massively
easier
and
at
the
same
time,
auto
scaling
because
you're
just
using
ec2
with
auto
scale.
Yes,
oh
my
goodness.
Yes,
yes
for
auto
scale,
so
it's
easier
to
maintain
easier
for
people
to
get
it
up
and
running
significantly
cheaper
and
faster.
D
Yeah
the
yeah,
I
think
the
the
availability
of
ecs
with
fargate
is
one
of
the
reasons
why
that's
so
cheap.
So
this
is
in
fargate.
We
don't
use
the
sort
of
classic
ecs
sure
enough,
but
the
like
it
just
looks
like
an
application.
That's
to
me,
that's
the
biggest.
The
biggest
win
is
it's
so
simple.
You
know
slap
it
in
a
container
ship
it
to
production
and
call
it
a
day.
B
B
B
D
D
We
actually
like
kafka
delta
streams,
which
is
the
spark
version
we
ran
into
an
s3
bug,
which
is
like
unheard
of
s3
is
the
most
bulletproof
service
I've
ever
seen
on
the
planet,
but
we
ran
into
a
consistency
bug
with
aws
s3
that
has
taken
six
plus
months
to
sort
of
dive
into
and
and
really
understand
and
aws
hasn't
been
able
to
fix
it.
Yet
there's
no
data
corruption
like
if
you're
using
s3,
don't
worry
about
it.
D
If
you
haven't
seen
it
yet
you're
fine,
even
if
you
see
it
like
there's
no
problem
with
our
with
our
delta
tables,
but
what
that
would
mean
is
we
would
have
periodically
sparked
streaming.
Jobs
would
fail
and
go
offline,
and
that
means
an
engineer
has
to
go.
Look
at
what
went
wrong
and
make
sure.
Oh
it's
that
bug.
We
saw
it's
just
that
happening
again
and
not
something
else,
and
so
we've
had
to
babysit.
D
This
spark
streaming
version
of
this
infrastructure
a
lot
more
than
I
originally
anticipated,
and
so
the
fact
that
we're
you
know
we're
going
to
be
able
to
sort
of
set
it
and
forget
it
with
kafka.
Delta
ingest
is
to
me
the
biggest
the
biggest
success
as
part
of
this
project,
because
it
should
you
know,
there's
just
a
lot
fewer
moving
parts,
so
fewer
things
to
go
wrong.
All.
B
Right,
we
only
have
a
few
minutes
left,
so
I
wanted
at
least
to
to
ask
the
question
what
comes
next
and
then,
of
course,
call
out
that
everybody
should
be
joining
us
on
delta
uses
slack
anyways
to
ask
more
questions,
but
but
let
what
comes
next,
like
you
know
what
what's
going
on
with
delta
russ,
what's
going
on
with
kafka
delta,
why
don't
you
tell
us?
What's
what's
coming
up
down
the
pike.
C
C
We
have
several
new
streams
that
are
on
kafka
delta
ingest
already,
but
we
want
to
migrate
our
existing
kafka
delta
streams
that
are
using
spark
streaming
over
to
these.
So
this
is
this
is
kind
of
like
a
stop
gap
solution
so
that
we
can
just
terminate
our
existing
spark
streaming
jobs
and
then
start
kafka
delta
in
ingest
and
pick
up
where
it
left
off.
C
So
we
got
this
one,
the
next
another
one
we
have
is
you
know
I
personally
really
won't
want
the
ability
to
have
these,
like
kind
of
crown
trigger
once
style,
scheduled
jobs
for
kafka
delta
ingest,
because
we
do
have
a
lot
of
low
volume
topics
where
we've
just
got
like
you
know
a
a
small.
You
know
a
pretty
low
volume
of
messages
per
second
for
a
lot
of
jobs,
basically
tiny
ingestion
streams.
C
It
feels
really
wrong
to
have
an
ecs
task
running
24
hours
a
day.
Seven
days
a
week
and
just
very
gradually
incrementally
writing
these
messages
into
delta
it'd
be
really
great.
If,
if
we
could
schedule
these
jobs,
that
would
run
hourly
or
a
couple
times
a
day
and
just
suck
whatever's
in
kafka
from
the
latest
offset
into
delta,
and
we
could
save
a
good
bit
of
money
here.
C
D
Should
mention
that
these
are
also
not
things
that
scribbs
going
to
do
right
right
so
if
you're
interested
in
kafka
delta
ingest
without
the
kafka
part
like
if
you're
you're,
keen
on
kinesis,
for
example,
then
like
come,
join
us
like,
like
kafka
delta
ingest,
is
part
of
the
delta
project.
So
it's
not
scribs
baby,
but
our
our
sort
of
world
is
json
on
kafka.
So,
if
you're
interested
in
not
json
or
not
on
kafka,
then
sharpen
your
your
rust
tools
and
come
join
us.
D
I
should
mention
as
well
from
the
delta
rs
standpoint.
We
are
waiting
on
an
up
because
in
tight
we're
waiting
on
an
upstream
dependency
to
make
a
major
release,
because
we're
sort
of
pinning
some
things
to
do
the
next
delta
rs
release
and
then,
after
that,
we're
planning
on
tagging,
kafka,
delta,
ingest
1.0,
but
where
we
are
right
now
you
know
this
is
production
ready?
B
So
that's
perfect.
Thank
you
very
much.
That's
an
excellent
segue!
So
karen
we're
going
to
be
closing
up
in
the
next
few
seconds
here,
but
again,
I
will
over
emphasize
karen's,
already
sent
a
link
to
everybody
on
linkedin,
zoom
and
youtube
so
join
us
on
the
delta
users.
Slack.
If
you
have
more
questions
christian
tyler
myself,
I'm
not
very
useful,
but
nevertheless
we
are
online
to
go
ahead
and
answer
your
delta
russ
questions,
confident
delta
interest
questions.
B
A
Thanks
denny
thanks,
tyler
and
christian
again
great,
to
have
you
and
for
everyone,
just
a
reminder
that
the
recording
is
available
on
youtube,
so
check
it
out
there
and
yeah.
We
hope
you
all
will
join
the
delta
lake
slack
and
with
that
everyone.
I
hope
you
have
a
great
rest
of
your
day.
Take
care
bye,
take
care.