►
From YouTube: AMA | Growing the Delta Lake ecosystem
Description
Join us for this fun Delta Lake AMA session where we discuss with QP Hou, Christian Williams, and Alexander Kushnir from Scribd on growing the Delta Lake open-source ecosystem.
Scribd's current OSS projects aim to extend the power and flexibility of Delta Lake throughout the organization. Come ready with questions as this will be a fun interactive session hosted by R. Tyler Croy (Scribd) and Denny Lee (Databricks).
LINKS MENTIONED:
- https://github.com/delta-io/connectors
- https://github.com/delta-io/kafka-delta-ingest
- https://tech.scribd.com/blog/2021/introducing-sql-delta-import.html
- https://github.com/delta-io/delta-rs/discussions
Speakers:
QP Hou a Senior Engineer at Scribd and Airflow committer. QP manages batch processing pipeline at Scribd. Prior to that, he worked on Machine Learning and Monitoring Infrastructure at Floyd Labs and Linkedin respectively.
Alexander Kushnir is a Data Architect and a Tech Lead of Data Engineering Team at Scribd. Throughout his almost 20 year career has acquired experience in various Software Engineering domains: desktop applications, web development, mobile APIs, distributed computing, cloud architecture, big data. Designed and implemented solutions utilizing various Data Stores: relational databases, document databases, key/value stores, object stores. For the past 5 years focused on Distributed Computing in cloud environments utilizing various BigData tech stacks. He is a big fan of Spark.
Christian Williams is a Software Engineer at Scribd.
Hosts:
R Tyler Croy is the Director of Platform Engineering at Scribd, where he leads the efforts to empower data customers across the organization with higher quality and fresher data than had been previously possible. His background is in production data services, revolving largely around Apache Kafka and various stream processing tools. At Scribd, Tyler and his team work to bring data-driven insights closer to production applications with the “Real-time Data Platform”, built on Apache Kafka, Apache Spark, and Delta Lake.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Mode,
thank
you
very
much.
My
name
is
denny
lee.
I'm
a
developer
advocate
here
at
databricks,
but
I'm
just
a
lowly
little
host
for
today's
session,
because
we
really
wanted
to
feature
the
engineers
at
scribd
who
are
going
to
be
talking
about
growing
the
delta
lake
ecosystem.
So
just
in
case
you
forgot
what
session
we're
doing
so
without
further
ado,
let's
go
ahead
and
introduce
some
folks,
I'm
going
to
go
in
whatever
order.
My
system
shows
it,
as
so,
we'll
start
with
you
qp
once
you
introduce
yourself
buddy.
B
C
A
D
It's
good
that
y'all
can't
see
me
right
now,
so
the
director
of
platform
engineering
here
at
scribd,
and
so
I
lead
and
and
sort
of
oversee
the
core
platform
team
which
we
have
some
members
here
from
from
that
team.
Here
today,
the
data
engineering
team
and
the
data
operations
team
all
sit
under
platform
engineering.
So
I'm
usually
the
one.
That's
you
know
running
the
meetings.
A
You
know
your
place
all
right,
fair
enough,
and
I
was
about
to
introduce
chris
christian
williams,
but
I
think
we
just
lost
him
so
all
right
well,
whenever
he
comes
back
we'll
go
ahead
and
introduce
that
that
was
interesting
all
right.
So
in
that
case,
well,
hey
alexander,
we're
going
to
switch
over
to
karen.
Let's
switch
over
to
speaker
mode
alexander,
want
you
to
present
the
slides
and
then
that
way
we
can
go
ahead
and
at
least
start
the
session
for
everybody.
That's
listening
as
we're
setting
this
all
up.
A
The
idea
is
that
we're
going
to
present
only
about
a
few
slides,
and
then
we
want
you
to
chime.
In
with
your
questions,
please
put
your
questions
in
the
zoom
q
a
panel.
If
you're
actually
coming
here,
the
other
place
to
put
them,
we
will
aggregate
them
from
linkedin
and
also
from
youtube.
So
just
be
a
little
bit
patient
with
us.
A
I
might
we
might
not
be
able
to
answer
them
right
away
just
because
there
is
a
lag
between
youtube
and
linkedin
with
our
current
broadcast,
but
we'll
do
our
best
to
try
to
respond
all
right.
So
without
further
ado,
let's
switch
over
to
the
presentation,
qp
and
alexander.
Take
it
away.
Please
awesome.
B
So
I'm
very
happy
here
to
talk
about
one
of
the
open
source
projects
that
we're
working
on.
As
with
with
the
community,
it's
called
the
delta
rs
next
slide
piece,
so
delta
rs
is
a
native
implementation
of
delta
lake
in
pure
rust,
which
which
means
that
when
you're
using
this
library
you
can
access
delta
lake
read
write
with
outside
without
depending
on
the
jvm
ecosystem.
So
it
it
currently
supports
local
file
system,
s3,
remote
storage
and
azure
data
lake
storage
chain.
2.
B
so
and
read,
read
it's
already
well
supported
in
the
sense
that
we
have
tested
with
a
lot
of
production
tables
and
the
golden
test
data
set,
and
it's
read,
is
going
really
well
so
far,
and
thanks
to
christian,
we
have
a
really
early
and
native
support
for
writing
to
the
other
table
and
we
work.
Misha
from
script
is
also
working
on
adding
save
s3
multi-writer
support
to
to
the
library
and
the
the
right
interface
that
we
have
right
now.
B
It's
still
pretty
primitive
in
the
sense
that
it
interacts
directly
with
the
delta
transaction
log.
So
we're
also
in
the
process
of
designing
a
higher
level
api
where
users
of
data
delta
rs
can
write
to
the
table
with
you
know:
data
error
data
record
batch
directly,
so
they
don't
have
to
interact
with
the
transaction
transaction
log
at
the
low
level.
B
So
in
the
next
slide,
I'm
going
to
show
you
what
it
looks
like
from
a
high
level
architectural
design
of
the
library,
so
the
pink
and
dark,
pink
or
red
boxes
here
are
all
in
written
in
pure
rust.
So
at
the
very
bottom
we
can
see
that
there's
a
tokyo
library
which,
for
those
who
are
not
familiar
with
it,
it's
the
async
io
framework
that
most
that's
used
within
the
rust
community.
B
That's
what's
powering
the
async
io
to
the
remote
storage
backhands
like
s3
and
as
your
as
your
blob
storage,
and
on
top
of
that
we
have
parquet
and
arrow
crates.
He
said
to
pure
rust
implementation
of
arrow
that
that
we
heavily
depend
on
and
then
we
have,
you
know
s3
clients
and
azure
clients.
So
on
top
of
that
we
have
the
actual
implementation
of
delta
lake
logic.
B
That
does
the
read
and
write
to
the
delta
tables,
and
this
layer
has
the
has
the
stable,
well
end
user
api
that
that
for
rust
applications
and
then
on
top
of
that,
we
also
leverage
libraries
like
pi
03,
to
export
those
rust
apis
into
you
know
other
language
like
python,
ruby,
golan,
node.js.
So
the
one
of
the
nice
thing
about
this
setup.
B
Basically,
we
used
all
the
core
logic
in
the
rust
implementation,
which
is
really
nice
for
us
and
the
rust
the
python
binding
is
now
ready
to
rock.
If
you
just,
you
can
just
use
pip
install
to
get
start
working
on
it
and
there
are
some
documentations
in
the
repo
as
well
about
how
to
use
the
python
and
rust
apis.
And
one
thing
that
I
would
like
to
mention
is
because
the
rust
api
we
are
using
a
leading
edge,
unreleased
version
of
aerocrate.
B
B
D
Awesome,
I
was
kind
of
hoping
that
you
might
be
able
to
explain
like
you,
you
might
know
this
better
than
than
I
do
or
our
audience
does,
but
what
what
apache
arrow
does
in
the
con,
like,
I
don't
think
delta
the
spark
delta
implementation
uses
apache
arrow.
What
does
it
do
for
the
rust
implementation.
B
So
currently
a
it
it
it's
mostly
at
the
library
so
so
for
for
delta
table
itself.
We
mostly
write
two
product
files
to
store
the
actual
data,
so
the
parking
crates
that
we're
using
it's
built
on
top
of
the
arrow
crates,
which
that
it
also
lives
within
the
aero
project
and
so
for
the
for
the
the
delta
rs
implementation
itself.
A
Excellent,
so
that's
a
great
kick
start
question,
so
the
now
I'm
actually
going
to
go
a
little
bit
scroll
back
up
and
you
may
want
to
switch
back
to
the
slides
to
answer
this
question,
but
I
guess
the
first
things.
First,
why
did
you
want
to
build
delta
rs,
like
you
know,
after
all,
it's
awesome.
This
idea
that
you
want
to
you
know,
build
the
ability
to
not
use
spark.
So
could
you
provide
context
on
what
why
you
chose
to
do
that.
B
Yep,
so
I
think
that
the
next
presentation
or
the
next
project
that
we're
going
to
demo
will
be
a
good
example
of
how
this
library
can
be
used.
So
so
so
I'll.
Leave
that
the
question
on
that
and
then
the
second
reason
that
I
think
this
could
be
useful
is
for
local
development,
where
you
just
have
to
spin
up
a
python
environment,
and
you
want
to
load
some
data
from
that
data
table
into
you,
know
native
pandas
data
frame,
and
it
would.
This
will
work
really.
D
There's
also,
you
know
when
we
look
at
what
scribd
has
in
our
technology
stack,
there's
actually
a
fair
bit.
That
is
not
jvm
based.
You
know
our
data
platform
because
of
the
you
know
where,
where
spark
lives
and
the
you
know
importance
of
spark
to
most
data
infrastructures
like
our
own,
that's
all
heavily
spark
oriented,
but
when
we
look
to
who's
actually
producing
the
data,
you
know
that
that's
flowing
into
our
our
data
lakes,
it's
largely
things
that
are
not
running
on
the
jvm
and
so
from
a
you
know,
hand-wavy
strategic
standpoint.
D
You
know
the
the
hope
is
that
if
we
can
extend
writer
and
maybe
some
light
reads-
support
to
ruby
applications
or
or
rust
applications
or
golang
applications
that
we
can
start
to
bring
more
of
that
data
more
quickly
and
from
a
broader
set
of
sources
into
delta
lake.
That
we
can
then
do
the
really
fun
spark
things
with.
B
And
another
thing
that
I
I
would
like
to
mention
is:
it's
also
always
good
to
have
alternative
implementation
of
delta
spec,
because
you
know
we,
as
the
company
is
heavily
invested
in
delta
lake
and
during
the
you
know,
work
of
this
project.
We
actually,
you
know,
help
contributed
some
correct
corrections
or
into
the
data
spec
itself.
So
we
found
that
there's
some
ambiguous
spec
wording
in
the
spec
that
that
needs
to
be
corrected.
A
No,
that's
an
excellent
call
out
and
yes
for
those
who
are
interested
all
the
stuff
has
been
on
github.
Actually,
so
you
can
definitely
see
the
conversations
that
happened
right
in
github
about.
Oh
we've
completely
missed
this
part
of
the
spec.
We
should
call
that
out
so
yeah
qp
and
tyler,
and
everybody
else
in
this
group
have
done.
Those
call
outs
in
github
so
definitely
join
us
from
there.
So
this
actually
naturally
this
to
the
next
question
and
tyler,
I'm
going
to
ask
it
this
time,
but
you
can
ask
the
other
one.
B
A
A
D
D
B
Yep,
so
we
actually
share
a
same
set
of
golden
test
datasets
with
the
the
main
delta
connector
project
that
we
also
touch
on
a
little
bit
by
alex.
So
we
share
the
same.
You
know:
data
test
data
go
into
test
data
set
to
make
sure
that
every
commit
and
delta
rs
also
pass
those
tests.
And
of
course,
we
also
try
to
follow
this
back
as
strictly
as
we
can.
A
No,
that
makes
a
ton
of
sense.
Okay,
perfect,
let's
go
ahead,
the
other
quick
question.
Actually
there
is
a
quick
question
actually
from
the
linkedin
stream.
Actually
I
think
I
missed
it,
but
I
believe
it
was
about.
When
are
we
going
to
be
supporting
google
cloud
storage?
We
had.
We
did
mention
adls,
we
did
match
an
s3,
but
you
know
poor
google
cloud,
you
know
so
I'm
just
curious.
What's
our
plan
for
google
cloud
in
that
case.
A
A
D
No,
I
don't
want
to
support
gcp,
so
there's
actually
an
active
inactive
issue
in
the
in
the
project.
There's
an
issue
56.
If
everybody
anybody
wants
to
go,
look
it
up.
Basically,
you
know
scribd
as
a
company
we're
we're
in
aws
customer,
so
we're
investing
in
the
s3
support
and
qp
had
called
out.
Ben
ben
is
interested
as
an
azure
customer's
interest
in
azure
we're
kind
of
waiting
for
someone
to
step
up
to
say,
like
I
really
like
google
cloud,
I
can.
I
can
implement
this
support.
D
The
storage
back
ends
in
delta
rs
are
fairly
easy
to
swap
out
and
implement
a
new
one.
So
there's
already
there's
three
storage
back-ends
right
now:
there's
local
file
s3
and
then
the
azure
dsl,
whatever
that
acronym
is
that
supports
all
there.
So
someone
would
just
need
to
come
in
and
add
the
google
cloud,
storage,
backend.
D
I
I
I
don't.
I
don't
want
this
to
be
to
come
off
as
as,
like
a
flippant,
you
know,
patches,
welcome
sort
of
response,
but
one
of
the
things
that
I've
I've
really
enjoyed
about
the
rust
ecosystem
in
particular,
is
folks
that
are
new
to
rust.
If
you
express
an
interest
in-
and
this
goes
not
just
for
delta
rs,
but
a
lot
of
other
rust
projects.
D
If
you
express
an
interest-
and
you
want
to
try
to
figure
something
out-
there's
a
lot
of
very
helpful
people
in
the
rust
ecosystem
that
can
help
with
some
of
the
mentorship
and
pointers
to
sort
of
help
you
get
started.
So
we've
already
seen
this
with
a
couple
of
individuals
coming
through
the
pull
request
process
with
delta
rs,
where
you
know
just
going
through
code
reviews
with
someone
like
qp
who's,
a
very
talented,
russ,
developer
or
neville
who's.
Another
he's
an
apache,
aero
contributor,
you'll
learn
a
bunch
of
stuff,
and
it's
been.
D
A
Yes
and
in
fact,
not
to
to
go
with
tyler's
non-flippant
response,
if
you're
into
it
like
there's
a
bunch
of
us
that
can
also
very
very
help
you
not
just
on
the
rust
side,
but
also
the
google
cloud
storage
side
that
we
can
definitely
help
you
there
too.
So
absolutely
would
love
help.
Just
because
you
know
this
is
what
growing
the
delta
lake
ecosystem
is
all
about.
A
Okay,
nevertheless,
qp
will
be
joining
us
shortly.
In
fact,
I
think
he's
dialing
in
right
now.
Nevertheless,
I
believe
we
can
flip
back
to
slots,
and
the
next
on
the
daca
is
kafka
delta
ingest.
So
karen,
let's
switch
to
speaker
mode.
So
that
way
we
can
show
the
slide
slides
nicely
and
whoever's
next
on
the
docket
by
all
means,
go
for
it.
F
I
don't
do
any
of
the
fancy
ml
stuff.
That
qp
does
I'm
more
of
a
data
engineer
and
data
platform,
infrastructure
developer
and
I
happen
to
be
working
on
coffee
delta
ingest.
So
let's
talk
about
kafka
delta
ingest.
The
first
thing
to
note
here
is
kafka
delta
ingest.
It
you
know
if
delta
rs
is
early
on
in
development,
kafka
delta
ingest
is
very
very
early
on
in
its
development.
F
In
fact,
we
only
just
merged
our
first
major
pr
this
past
week.
Our
goal
with
this
project
project
is
really
to
ingest
kafka
data
quickly
and
efficiently
into
duplic,
so
that
downstream
spark
queries
running
on
data
breaks
either
batch
or
streaming.
Consumers
can
read
these
three
read.
These
data
sets
in
near
real
time
as
far
as
like
contribution
guidelines
go
if
you
wanted
to
contribute
to
kafka
delta,
ingest
delta.
Rs
is
by
far
and
away
the
most
critical
dependency
of
kafka
delta
ingest.
F
The
reason
these
particular
topics
are
interesting
for
kafka
delta
interests
ingest
is
that
streaming
trimming
jobs
tend
to
leave
small
files
behind
and
and
that
can
really
slow
down.
Ad
hoc
data
break
queries
because
you
know
those
queries
are
going
to
run
most
efficiently
when
there's
large
files
at
play-
and
you
know
specific
queries
that
target
large
files
and
we
want
our
data
breaks,
notebook
queries
to
run
fast,
so
building
functionality
into
delta
rs
that
allows
us
to
mitigate
the
small
files
problem
by
solving
the
problem
there.
F
F
If
you're
not
familiar
with
the
concept
of
trigger,
wants
trimming,
jobs,
they're,
especially
useful
for
low
volume,
kafka
topics
or
streaming
uk
use
cases
that
can
tolerate
a
little
extra
latency.
So
they
don't
have
to
be
like
ad
hoc
near
real
time
on
their
on
the
reader
side
rather
than
so.
Basically,
though,
rather
than
having
streaming
processes
up
and
running
24
hours
a
day,
seven
days
a
week,
these
trigger
once
style
jobs
will
be
launched
in
kind
of
a
cron
style.
A
Hey
christian,
actually,
let
me
dive
in
with
a
little
quick
question.
I
want
to
roll
us
back
up
a
little
bit
if
you
don't
mind,
okay,
sure,
because
I'm
just
curious
from
your
can
you
provide
just
at
least
from
from
for
everybody
that
may
not
be
familiar.
Why
do
you
have
we
have
you
built
kafka
delta,
just
like
what
is
the
use
case?
A
What's
the
scenario
you're
trying
to
solve,
because
I
think
for
some
folks
I
think
the
natural
question
would
be
like
well,
I
could
just
put
a
spark
job
and
do
it
and
that's
fair
right,
but
there's
obviously
very
good
reasons
for
having
kafka
dump
data
directly
into
delta,
using
kafka
delta
ingest.
So
can
we
provide
a
little
context
on
on
why
that.
F
D
A
D
A
D
At
a
high
level,
kafka
delta
ingest
for
us,
we
want
to
you,
know,
make
the
fastest
possible
route
from
basically
pulling
bytes
from
one
file
descriptor
out
of
kafka
and
writing
those
to
s3.
You
know
we
want
as
little
overhead
as
possible
because
we
want
things.
You
know
when
we
talk
about
real-time.
D
D
That,
like
just
did
a
deploy,
you
know,
or
they
just
flipped
an
a
b
test,
one
way
or
the
other.
We
want
to
get
that
data
to
them
as
quickly
as
possible.
So
for
us,
kafka,
delta
ingest
is
solving
the
problem
of
you
know.
We
don't
want
to
run
a
tremendous
amount
of
infrastructure
just
to
take
bytes
from
one
descriptor
and
put
them
to
another
one
and
that's
where
kafka
delta.
D
B
And
just
add
to
that,
for
real-time
use
cases
like
this.
We
also
it
also
works
really
well
to
handle
traffic
spikes
where
we
can
set
up
the
system
to
auto
scale
as
more
traffic
comes
in
because
inherently
it's
a
master
list
design,
there's
no
driver
node
right.
So
every
every
worker
will
just
acting
this
the
same
role,
and
so
it
makes
it
really
easy
to
auto
scale
based
on
traffic
load.
F
This
is
a
picture
of
the
deployment
architecture
we're
targeting
for
kafka
delta
ingest
to
run
on,
and
in
this
slide
you
can
see
producers
write
to
copic
topic
kafka
topics.
For
example,
we
have
topics
a
b
and
c
here
in
the
left
column,
a
specific
kafka
delta,
ingest
job
will
read
messages
from
one
topic
and
buffer
those
messages
for
a
while,
so
so
that
it
can
ultimately
write
a
big
chunk
of
data
to
a
delta
lake
table.
One
kafka
delta,
ingest
job.
A
Perfect
well,
actually,
I've
got
a
lot
of
questions
that
we
can
definitely
dive
into,
but
I'll
try
to
not
overdo
it,
because
we
only
have
about
20
minutes
left,
but
for
starters,
let's
go
with
even
the
more
basic
con
question.
How
does
that
json
even
become
a
row
in
delta,
like
for
starters,
like
like
what
what's
the
basic
principle,
how
you
even
do
that.
F
A
F
So,
on
the
on
the
producer
side,
you
know
you
might
have
a
web
application.
That's
writing
json
messages
to
a
content
topic
and
a
lot
of
people
like
to
write
those
as
in
a
json
data
format,
because
it's
really
easy.
You
don't
have
to
like
bootstrap
a
bunch
of
pretty
protocol
buffer,
avro
schema
type
of
stuff.
F
You
can
just
slap
a
string
on
a
kafka
topic,
so
what
kafka
delta
ingest
is
doing
internally
in
our
early
iterations
is
reading
those
json
messages
off
of
the
kafka
topic,
buffering
them
up
internally
and
then
using
a
an
api.
That's
already
provided
by
the
rust
aero
crate
to
decode
those
json
values
into
an
aero
record
batch,
and
so
basically,
what
kafka
delta
ingest
is
doing.
A
F
A
C
So,
some
time
ago
we
took
on
an
adventure
of
adopting
databricks
platform
here
and
in
order
to
get
there,
we
had
to
migrate
from
our
legacy:
hadoop
infrastructure
and
somewhere
around
20
of
our
nightly
workloads.
Our
batch
workloads
in
that
platform
for
scoop
jobs,
and
if
you
know
scoop,
is
basically
a
great
tool
that
comes
with
hadoop
and
that
can
move
data
into
hdfs
out
of
hdfs
like
it
supports
a
lot
of
different
use
cases,
and
I
want
to
highlight
that
sql
delta
import
does
not
support
all
of
them.
C
Sql
delta
import
is
used
for.
One
specific
purpose
is
bulk
loading
entire
table
from
a
jgbc
source
into
a
delta
lake
table.
So
why
didn't
we
use
scoop?
First
of
all,
scoop
doesn't
support
delta
directly.
We
can
write,
we
can
import
into
park
a
and
then
do
like
additional
conversion
steps,
but
we
want
it
to
not
do
that
in
order
to
run
scoop.
We
also
would
need
additional
infrastructure
right,
like
databricks
platform
does
not
come
with
scoop
either
and
also,
as
we
were,
using
scoop
in
script
in
prior
years.
C
So
if,
if
you
know
anything
about
scoop,
the
way
you
parallelize
your
workload
is
by
introducing
a
parameter
called
number
of
members,
so
scoop
rounds
mapreduce
under
the
hood
and
when
you
specify,
let's
say
10
mappers,
it
will
use
your
split
by
column.
It
will
go,
get
its
min
max
values
and
and
then
we'll
use
the
parameter
of
number
of
mappers
to
to
basically
split
the
source
table
into
chunks
that
it
wants
to
load.
C
So
when
you
do
that,
you
run
into
some
issues.
For
example,
if
you
have
a
like
a
very
large
table,
and
your
number
of
mappers
is
small,
you'll
have
a
very
long
running
scoop
job
and
like
really
long-lasting
connections
that,
as
we
know,
tend
to
go
down
once
in
a
while.
If
you
go
and
and
bump
up
your
number
of
members
too
high,
you
might
actually
overload
your
underlying
data
store
with
concurrency
right.
C
So
if
you
go
with
too
many
connections,
you'll
have
the
too
many
mappers
you'll
have
small
files,
not
enough
mappers
large
files.
I
guess
not
as
much
of
a
problem.
So
what
did
we
do
differently
in
sql
delta
import?
We
basically
decoupled
this
parallelism
with
the
load
on
the
source,
data
store
and
the
number
of
output
files.
C
We
still
do
the
same
basic
thing
by
introducing
by
using
split
split
by
column
to
split
our
workloads
by,
but
we
also
introduced
this
parameter
called
chunks
to
basically
tell
how
many
chunks
you
want
to
break
up
this
table
into
before
loading
it.
But
then
the
parallelism
is
actually
controlled
by
the
regular
spark
parameters
like
number
of
executors
in
executor
cores.
C
So
this
allows
you
to
break
up
your
import
into,
let's
say
2000
chunks,
but
only
use
at
most
100
executors,
which
will
translate
to
at
most
100
connections
to
the
sql
to
the
underlying
sql
data
store.
So
another
problem
we
were
able
to
solve
by
doing
this
is
the
data
skew
on
a
split
by
column,
and
I
think
I'll
demonstrate
that
in
the
next
slide,
so.
C
C
C
So,
if
you
use
your
standard
split
by
setup
right,
your
first
partition
will
have
actually
less
data
and
your
last
partition
will
have
the
most
of
the
data
when
you're
importing
it
in.
So
this
will
introduce
data
skew,
maybe
not
as
drastic
of
a
data
skew,
but
nevertheless,
another
problem
that
we
ran
into
when
dealing
with
importing
data
here
at
scribd
was
a
non-uniform
data
density,
because
at
some
point
in
time
somebody
will
just
go
and
delete
a
range
of
rows.
C
So
if
you
look
at
the
bottom
picture,
like
partition,
number
two
doesn't
do
any
work
almost
right
so
and
the
last
partition
is
doing
most
of
the
work.
So
if
you
do
this
with
scoop-
and
you
set
number
of
members
to
10
you'll,
first
of
all,
you'll
get
a
processing
time
skew
and
you
will
get
highly
varied
output
file
sizes
when
this
lands
on
disk.
C
C
C
Let
me
maybe
talk
a
little
bit
how
we
got
it
set
up
today,
it's
at
script.
When
we
moved
into
the
cloud.
This
actually
fits
pretty
nice
in
to
our
cloud
setup.
So
in
the
cloud
we
have
dedicated
replica
for
our
imports
and
during
the
day
when,
when
we're
basically
not
running
any
imports,
it's
a
very
small
sized,
scaled-down
aurora
cluster
when
we
start
our
import
process.
C
First
of
all,
we
pause
the
replication
to
make
sure
we
get
a
consistent
view
in
time
and
we
heavily
scale
out
our
aurora
clusters,
so
they
can
deal
with
the
massive
amount
of
connections
that
spark
is
about
to
throw
at
them
and
whenever
the
imports
are
done,
we
resume
the
replication
and
we
scale
our
aurora
clusters
back
down,
and
this
actually
works
out
pretty
good
for
us.
It's.
C
C
You
can
see
that
we
start
with
only
a
couple
nodes
and,
as
the
imports
pick
up,
we
auto
scale
and
bring
additional
nodes
in,
and
they
pick
up,
connections
and
yeah
and
get
work
done.
We
if
you
want
to
know,
learn
more
about
this.
Please
follow
our
blog.
We
just
actually
posted
a
blog
post,
which
goes
into
like
a
lot
of
details
of
on
how
to
use
it.
Why
use
it
and
all
things.
A
Say
alex
we've
actually
already
posted
the
blog
in
linkedin
youtube
zoom,
so
we
got
that.
Let's
go
back!
One
slide.
If
you
don't
mind,
this
is
pretty
sweet
because
you've
called
out
the
cpu
utilizations.
So
then
I'm
just
curious
in
your
environment.
Oh
actually,
I'm
sorry
one
slide
before,
because
it's
actually
about
the
aurora
sources
does
sql
delta
import
in
the
process,
because,
if
you're,
seeing
all
that
super
utilize
it
does
it
actually
scale.
C
So
we
have
aurora
set
up
to
auto
scale
and
basically,
we
had
to
play
with
configuration
on
both
sides.
We
are
using
connection
based,
auto
scaling
in
aurora,
so
once
a
certain
number
of
connections
hook
up
to
a
cluster,
it
starts
scaling
up
and
we
also
actually
had
to
tune
down
the
speed
at
which
the
spark
clusters
out
of
scale.
So
in
databricks
we
have
the
parameter
to
tune
how
how
quickly
spark
clusters
go
up
to
full
capacity.
C
A
Right
right
right:
okay,
basically,
it
allows
a
little
bit
of
delay
so
that
way
they
can
actually
kick
in
so
you're,
not
also
using
warm
pools
or
anything
like
that.
Got
it
okay,
fair
enough,
then,
is
right.
Now,
we've
got
a
question
actually
here
from
through
zoom
is
my
sequel.
The
only
database
that's
supported
for
now
and.
C
The
answer
is
no
mysql
is
what
we
use
in
production,
but
one
of
the
parameters
is,
I
guess
we
don't
call
it
out.
We
actually
you
can
connect
to
any
jgbc
data
source.
So
as
long
as
because
under
the
hood,
sql
delta
import
uses
spark
jdbc
to
do
all
of
its
magic
right.
So
jdbc
connection
is
actually
one
of
the
parameters
that
we
take.
So
anything
that's
jdbc.
A
D
I
know
like
I
remember,
watching,
watching
the
the
deployment
of
this
as
we
as
we
brought
this
into
production.
Could
you
maybe
talk
about
some
of
the
the
the
replication
coordination
that
we
do
or
any
of
the
other
like
pitfalls
that
we
ran
into
with
with
basically
bringing
this
to
production
with
aurora.
C
Replication
pitfalls
you'll
have
to
be
a
little
bit
more
specific
there
tyler
there's
there's
two
types
of
replications
that
basically
happening
here.
One
is
a
replication
from
our
primaries
into
aurora
cluster,
and
for
that
we
are
using
standard,
sql
replication
and
within
aurora
cluster
itself.
There
is
magic
aurora
under
the
hood
replication.
C
Well,
you
have
to
kind
of,
I
don't
want
to
get
into
the
weeds
of
how
aurora
internally
replicates
its
data
and
how
they
use
dedicated
storage
nodes.
But
basically
it's
super
fast.
It's
nothing
like
a
standard,
mysql
replication
and
it
allows
this.
This
is
why
we
can
actually
scale
out
so
quickly
when
the
no
additional
notes
come
online.
The
storage
nodes
are
already
available
for
them.
So
that's
that
works
out
great
for
us.
D
C
The
answer
is
we'll:
take
full
requests
right
so
or
we'll
write
it
right,
so
it
it
really.
The
way
it
works
is
basically
here
is
my
minimum
value
like
under
the
hood.
It
goes,
gets
min
value
max
value
and
then
splits
the
load
between
those
two
assuming
some
form
of
something
close
to
uniform
distribution.
The
mean
value
doesn't
have
to
be
the
absolute
minimum.
It
could
be
the
minimum
value
since
last
time
you
imported
right.
There
is
nothing
built
in,
but
it's
super
easy
to
add
and
yeah
I'll.
I
guess
I'll
stop.
C
A
No
yeah,
no,
this
is
perfect.
Okay,
we've
got
about
like
five
minutes
left
and
karen.
I
think
I'm
supposed
to
still
leave
two
minutes
for
you
to
close
it
up.
So
so
I
I
did
want
to
at
least
as
a
group
here
for
us
to
talk
a
little
about
just
a
tiny
bit
about
concurrency
and
the
transaction
log,
so
qp
chris
tyler
alex
whoever
wants
to
go
first,
let's
dive
right
into
it.
Yeah.
B
A
B
And
complex
problem
we
can
think
about
this
for
hours
if
you
want,
but
the
quick
overview
on
that
is
because
s3
is
not
a
strongly
consistent
data
store.
That
means
we
cannot.
It
does
not
support
cas
operations
so
for
those
who
don't
who
don't
know
about
it
is
compare
as
swap
so
so
in
order
to
do
safe,
concurrent
right
into
s3
from
multiple
writers,
we
have
to
also
leverage
dynamodb
and
there
are.
We
actually
have
a
really
lengthy
discussion
about
the
concurrent
right
design
in
the
open
source,
github
repo
discussion
section.
B
D
B
So,
if
you're
using
databricks,
if
you're
a
database
customer,
then
you're
fortunate
that
databricks
already
handled
handles
this
multi
s3
writer
for
you
and
if
you
using
open
source
spark
right
into
s3,
you
can
only
write
from
a
single
writer
and
for
delta
rs,
specifically
because
we're
using
it
in
delta
kafka
delta
interest
projects.
Where,
like
I
said
earlier,
every
process
within
the
the
deployment
just
has
the
same
role
where
they
will
just
write
to
the
data
table
concurrently
right.
So
there's
no
much
coordination
between
any
any
of
those.
A
Perfect
one
of
the
things
I
did
want
to
call
out
like
this
is
actually
specific
to
s3,
because
azure
data,
lake
storage,
adls
and
also
google
cloud
storage
actually
does
have
that
capability.
So
we
we
actually
it's
actually
not
required
for
those
two
sets
of
cloud
object
stores,
but
this
is
specific
to
s3.
B
A
Yeah,
okay,
hey
there's!
We
could
go
on
for
hours,
but
this
is
only
a
one-hour
show,
and
so
please
do
join
us
as
per
the
links
that
we've
posted
into
zoom.
Slash
youtube,
slash,
linkedin,
because
we'd
love
for
you
to
dive
in
with
us,
because
there's
a
lot
of
deep,
deep
dive,
geeky
stuff
here,
which
is,
if
you're
into
that,
that's
exactly
where
you
should
join
us,
but
saying
this
because
we're
almo
it's
almost
time
for
us
to
wrap
it
up
karen.
A
E
Thanks
denny
yeah.
I
just
want
to
echo
that
thanks
everyone
for
joining
us
for
this
session
and
great
job,
everyone
on
the
script
team,
we're
always
happy
to
have
you
and
I
just
dropped
the
link
to
the
video
recording
in
the
the
chat
spaces
so
join
us.
There
I'll
also
drop
all
of
the
links
that
were
referenced
throughout
the
throughout
the
session
in
the
video
description.
So
with
that
I'll
just
say,
thank
you
again,
and
I
hope
everyone
has
a
great
rest
of
your
day.
Take
care.