►
From YouTube: Tech Talk Series Part Four: Continuous Integration and Continuous Delivery with Delta Lake
Description
Join us for the final in a four part series with Salesforce Engineering.
Link to complete series recording: https://delta.io/news/salesforce-engineering-delta-lake-tech-talk-series/
Abstract:
Abstract: As we build our Engagement Delta Lake on Databricks Workspace, one of the challenges is how to automate the integration testing of our Spark jobs in the CI/CD pipeline. We came up with two designs to tackle the challenge : Namespace Deployment and Scenario Based Testing. In this talk, we will discuss the rationale and implementations of the two designs.
Part 1: Engagement Activity Delta Lake - https://youtu.be/a7_I1Qi1LoU
Part 2: Boost Delta Lake Performance with Data Skipping and Z-Order - https://youtu.be/CwJeKANlSLo
Part 3: Global Synchronization and Ordering in Delta Lake - https://youtu.be/OtYXc6ud2bQ
-----------------
Speakers
-----------------
Zhidong Ke, Software Engineer PMTS, Salesforce
Zhidong is passionate in designing distributed systems, real-time/batch data processing and building applications.
Yifeng Liu, Software Engineer LMTS, Salesforce
Yifeng is a software engineer who has extensive experience in big data processing and distributed system, and interested in high volume, high complexity, low latency data pipeline and framework building.
Aaron Zhang, Software Engineering PMTS, Salesforce
Aaron is an experienced software engineering leader with interests and areas of focus in engineering secure, fault-tolerant, high volume systems built on micro services.
Heng Zhang, Software Engineering PMTS, Salesforce
Heng is a software engineer who is interested and specialized in micro services, distributed systems and big data. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
C
D
A
A
B
C
F
F
A
G
Awesome
yeah
thanks
for
joining
us,
yeah
phil,
I've
done
a
few
polls
and
I
I
think
we
usually
have
a
lot
of
new
folks.
So
I'm
curious
if
you
want
to
chime
in
the
chat,
I'd
love
to
hear
if
this
is
your
first
session
or
if
you've
joined
us
before
this
is
the
fourth
part
in
this
four
part
series
so
like
to
see
who's
joined
us
previously,
so
we're
live
on
linkedin
and
youtube
so
hi
everyone
thanks
for
joining
us,
we're
excited
to
be
here.
G
G
All
right,
so
thanks
everyone
for
joining
us.
This
is
the
data
and
ai
online
meetup,
a
tech
talk
series
with
salesforce
engineering,
and
this
is
the
fourth
and
final
part
of
this
four
part
series.
Today
the
team
is
going
to
be
talking
about
continuous
integration
and
continuous
delivery
with
delta
lake.
G
So
I
wanted
to
do
a
quick
call
out
for
our
data
and
ai
online
meetup.
There's
the
link
there
and
I'll
drop
that
in
the
chat
spaces,
but
I'd
love
for
you
to
join
us
there.
We
announce
all
of
our
upcoming
tech
talks,
they're
all
live
and
recorded
and
hosted
on
our
youtube
channel.
So
we
have
a
lot
of
awesome
sessions.
G
We
have
summit
coming
up.
We
have
a
bunch
of
great
meetups
in
in
conjunction
with
our
summit,
so
we
announced
all
of
those
on
our
meetup
page,
so
we
have
all
of
our
details
and
you
can
check
that
out
and
join
us
within
zoom.
If
you'd
like
for
future
events
subscribe
to
our
youtube
channel
again,
we
have
tons
of
content
in
there
all
the
video
recordings
from
our
live
tech
talks
are
in
there.
G
G
G
Excuse
me
over
200
sessions,
lots
of
awesome
content,
three,
maybe
four
meetups
live
meetups
and
we
have
lots
of
lightning
talks
from
the
community.
We
have
some
amazing
keynote
speakers,
including
mate
and
ali
and
malala
and
renault,
like
lots
of
great
speakers,
and
we
have
two
days
of
training
as
well.
So
I
know
as
part
of
the
meetup
community.
We
have
a
few
training
discounts,
so
maybe
reach
out
to
me
and
I
can
pass
one
of
those
along
for
for
our
viewers.
So
I
hope
you
register.
G
For
today
the
conference
is
free
and
the
training
is
a
is
a
small
feat,
but
the
all
the
content
is
free.
So
I
hope
you
you
will
join
us
there
and
with
that,
just
a
quick
reminder
that
this
session
is
recorded.
I'll
drop
the
link
to
the
recording
in
all
the
chat
spaces.
So
you
can
visit
us
at
a
later
date,
if
you'd
like
to
and
with
that
I'll
pass
it
to
our
moderators
craig
and
team.
C
Awesome
thanks
karen
cool,
so
to
introduce
myself
briefly.
My
name
is
craig:
I'm
a
solutions
architect
at
databricks
and
I'll
be
moderating
this.
This
talk
along
with
denny
and
chris,
so
I
think,
to
start
off
we'd
like
to
do
some
introductions
with
the
salesforce
team
and
so
I'll
have
them
introduce
themselves
in
turn.
E
D
D
B
C
Awesome
thanks
so
before
I
hang
it
over
hand
it
over
to
hang
and
aaron.
If
you
have
any
questions,
regardless
of
where
you're
watching
linkedin
youtube
or
zoom,
please
feel
free
to
put
questions
in
the
q
a
we'll,
be
either
answering
them
directly
or
collating
them
to
to
answer
after
we
have
the
presentation.
E
Okay,
hello,
everyone
and
welcome
to
our
final
session
of
engagement
activity,
delta
lake
talk
series
and
aaron,
and
I
will
co-present
today
about
our
continuous
integration
and
continuous
delivery
with
the
other
lake
on
data
breaks.
As
usual,
we
want
to
introduce
ourselves
who
we
are
and
what
we
do
at
salesforce.
We
build
and
maintain
a
platform
to
capture
custom
activities
like
emails
meetings
and
videos.
E
These
events
are
either
consumed
by
downstream
products
in
real
time
or
stored
in
our
data
lake
using
data
lake.
So
this
data
lake
suppose
supports
the
creation
of
up-to-date
dashboards
by
downstream
components
like
einstein
analytics
and
the
training
of
machine
learning
models
for
our
customers,
who
are
using
sales
cloud
einstein.
E
In
our
last
three
talks,
we
share
and
show
how
we
build
a
high
performance,
buildable
data
lake
using
delta
lake.
Specifically,
we
use
a
multiple,
independent
stream
process
to
support
engagement,
act,
data
lifecycle
to
satisfy
the
soa,
and
we
use
the
notification
table
to
support
incremental,
read
for
our
downstreams.
E
We
also
want
to
maintain
a
high
data,
consistency
and
the
integrity.
Then
we
create,
and
then
we
come
up
with
two
designs.
One
is
the
exact
ones,
exact
ones
right
across
tables,
and
the
second
is
the
global
synchronization
and
ordering
so
with
the
other
design
in
places,
we
have
successfully
built
the
data
lake
for
multi-type
of
our
sales
activities,
such
as
engagements
voice,
insights,
email
and
meetings.
E
So
in
today's
agenda
we
will
start
with
the
assumptions
and
requirements
that
our
csv
build
with,
and
then
we
will
touch
a
bit
a
little
about
the
databricks
integration
about
the
databricks
api,
cli
and
the
jobs.
Then
we
will
focus
on
the
continuous
deployments
from
simple
deployment
to
namespace
deployment.
E
So
it
can
be
run
by
developers
locally
on
a
laptop
or
can
be
hooked
into
a
traditional
automation
tools
like
sims
team
city
on
jenkins,
or
can
be
integrated
into
the
more
cloud
native
deployment
tools
like
a
spinnaker.
As
we
run,
our
data
delta
lag
on
data
breaks,
so
we
have
to
integrate
with
it.
Databricks
provide
a
rich
set
of
apis
for
us
to
do
it
so
in
greenbox
those
apis
are
commonly
used
by
us.
E
The
cluster
api
allows
us
to
do
crude
operations
on
clusters
and
the
dbfs
api
allowed
us
to
upload
files
copy
delete
files
in
dbfs.
So
by
dbfs
we
mean
the
databricks
file
system,
which
is
an
abstraction
on
top
of
a
object:
storage
like
s3
or
hdbs,
and
with
jobs
api.
We
can
do
the
crude
operations
on
jobs
when
you
work
with
databricks
in
a
platform
you
have
to,
everything
is
a
job,
so
a
job
is
a
run
unit
and
a
job
has
metadata
like
a
name
id
or
timestamp.
E
E
E
E
So
in
the
in
this,
in
the
configure
cluster
page,
you
can
provide
very
detailed
cluster
configurations
such
as
the
databricks
runtime,
with
the
version
of
scala
and
spark,
and
you
can
set
the
cluster
size
and
the
worker
type
and
the
driver
type
this
may
be.
This
may
vary
across
different
cloud
providers,
but
will
look
similar,
and
you
can
also
further
provide
a
more
customized
configure
configurations
like
availability,
zoom,
the
ebs
volumes
and
spark
configuration
tags,
ssh,
log
logging
and
initial
scripts,
and
fortunately
you
can
put
all
the
configurations
in
one
json
file.
E
E
So
this
is
example
of
the
job
definitions
and
we
put
into
the
engagement
projects
for
the
simple
deployments.
There
are
primary
two
steps.
One
is
to
create
the
job
entry
on
databricks
using
create
job
api
and
upload
the
sample
jar
to
the
dbfs
using
dbfs
put
api,
and
we
encapsulate
all
the
dependencies
and
complexities
in
a
docker
image.
E
So
when
we
want
to
do
a
single
deployment,
we
just
create
a
container
out
of
it
by
providing
a
couple
of
parameters.
So
the
app
path
volume
tells
the
ccd
well
to
find
the
sample
jars
and
the
databricks
jobs
tells
well
to
find
those
job.
Definitions
and
databricks
notebooks
tells
well
to
find
those
notebooks
files
and
you
provide
the
deploying
environment,
could
be
a
dav.
Could
it
be
a
station
or
production
and
provide
the
databricks
host
and
token
to
talk
to
the
actual
databrick
service
and
the
specified
project
names.
E
This
docker
container
can
be
run
locally
by
developers
if
they
want
to
do
depart
dual
deployment
from
a
local
laptop
or
can
be
hooked
into
a
the
existing
ci
cd
pipelines.
For
us,
we
use
the
team
city
but
or
but
both
will
show.
Where
shall
the
signal
steps?
So
in
the
team
city
we
first
approve
the
code
from
the
github
and
use
the
sbt
to
create
assembled
jars
and
put
images
to
do
the
deployments
so
inside
the
containers.
E
We
quickly
see
the
conflicts,
and
one
quick
solution
for
us
is
to
we
use
the
data
level
separations,
so
each
developers
use
his
own
org
data
for
testing
and
we
only
run
at
that
point
of
time.
We
only
have
one
environment
and
one
pipeline,
so
every
everyone
relies
onto
that
pipelines,
so
we
used
at
the
beginning.
We
used
provided
org
data
for
each
of
the
developers
for
their
testings,
and
we
really
we
quickly
realized
that
this
is
very
inefficient
ways,
because
developers
ended
up
in
the
waiting
lines.
E
E
E
Then
we
borrowed
the
concept
of
the
namespace
from
kubernetes
and
applied
into
our
delta
cicd
pipeline,
so
the
first,
the
application
runtime
contacts
consist
of
the
developer
application
or
jobs
themselves
and
all
the
dependent
services.
So
our
delta
applications
rely
on
this
following
services.
First,
we
use
the
kafka
as
a
for
the
structured
streamings.
We
use
zookeeper
to
store
our
offsets
and
and
also
global
synchronizations,
and
we
use
aws
three
for
read
and
write
raw
files.
We
use
a
delta
table
and
we
use
the
thermologic
for
logging
and
data
log
for
matrix.
E
A
namespace
is
just
a
given
stream
provided
provided
by
data
by
developers
who
to
define
a
scope
of
application,
runtime
context,
so
with
application
development,
it
decorates
all
the
artifacts
created
in
those
dependent
services
and
the
application
itself
to
create
an
isolated
along
time
context
with
a
certain
declaration
rules.
So,
for
example,
for
kafka
topics.
We
we
decorate
the
namespace
as
a
prefix
of
all
those
topics
and
so
to
the
database
name
and
for
the
data
database
brackets.
E
We
also
add
these
namespaces
path
as
part
of
the
prefix
s3
prefix,
and
we
have
this
namespace
a
part
of
the
application
name
as
well.
The
namespace
can
be
passed
through
our
environment
variable
or
in
our
program
arguments
and
is
enforced.
With
our
in-house
developed
spark
framework.
We
call
it
ap
spark
framework
activity
platform,
spark
framework,
animation
context
is
consumed
by
all,
specifically
the
application
contacts
normally
read
the
properties
from
local
property
files,
which
has
all
the
configurations
and
with
namespace.
E
It
will
also
read
the
namespace:
try
to
read
the
namespace
from
environment
or
arguments
and
create
a
scoped
properties
with
the
declaration
rules
that
will
be
consumed
by
our
dependent
service
modules
and
create
a
scope
objects
so
to
enable
namespace
deployments
developers
only
need
to
specify
a
namespace
as
a
environment,
variable
and
optionally.
He
can
pass
a
cluster
id
because
the
namespace
deployment
is
usually
used
for
testings.
So
if
the
developers
have
already
have
a
cluster
enhanced,
you
can
directly
reuse
it.
Otherwise
we
will
create
the
foreign.
E
E
All
the
tables
are
private
to
the
developers,
and
the
data
file
backed
by
the
data
table,
also
has
a
private
sv
path
so
that
you
can
do
whatever
you
have
full
control
on
the
on
your
features,
and
even
it
is
very
disruptive
with
a
new
with
a
new
schema
and
there
is
no
impact
to
others,
and
also
you
have
a
private
kafka
topic
for
you
to
push
the
testing
data
to
your
kafka
topics
and
the
offsets
path
in
in
zookeeper,
for
that
topic
is
also
very
private.
It's
perfect!
E
E
In
our
second
talk,
we
share
our
bench:
how
we
do
our
performance
benchmarking
for
id
mutation
use
case
in
order
to
find
the
answer
for
the
question
of
what
will
be
the
best
value
for
the
max
maximum
file
size.
So
we
need
to
a
multiple
set
of
our
pipeline
with
the
duplicate
set
of
jobs
instead
of
data
instead
of
the
table,
schemas
and
tables,
but
with
different
configurations
of
max
max
file
size.
E
So
initially
we
only
have
one
job
for
ingestion
and
one
table
for
data
table
and
later
on,
we
add
a
second
table
notification
table
to
support
the
incremental
rate
and
later
we
add
engagement,
id
mutation,
job
to
address
multiple
id
mutation,
cascading
cases
and
handle
the
retry
for
these
orderings
and
so
on.
And
then
we
add
the
data
retention
for
ttl
requests
and
we
hooked
the
downstream
jobs
into
the
whole
pipelines.
E
So
as
we
build
more
data,
lake
delta
lake
spark
applications
to
our
pipelines
on
data
bricks,
we
come
up
with
a
concept
called
scenario:
to
effectively
organize
the
applications
and
test
cases.
So
the
scenario
is
a
group
of
relevant
applications
that
collectively
fulfill
certain
kind
of
functionalities
and
the
tests
are
test.
Cases
are
developed
per
scenario,
and
this
is
a
very
simple
scenario
specification
and
we
give
each
of
the
scenario
a
specification
in
json.
E
So
the
the
scenario
has
a
simple
metadata,
a
name
and
a
description
and
has
a
run
list.
So
the
round
list
indicates
the
the
test
logic
you
want
to
do
in
this
scenario
so
which
follows
the
normal
test
logic.
So
in
this
example,
in
a
round
list
in
the
first
three,
the
first
three
jobs
create
engagement,
database
and
tables,
cleanup
tables
sas
welcome,
is
to
prepare
the
testbed
and
the
create
engagement
test.
E
Data
is
to
generate
the
engagement
test,
data
engagement
data
and
the
mutation
request
for
testing,
and
then
we
run
the
engagement
ingestion.
This
is
our
actual
job
for
ingestion,
which
read
those
test
data
to
our
data
table
and
the
mutation
record
tables,
and
then
also
the
mutation
record.
Notification
ingestion
is
to
ingest
the
mutation
requests.
E
So
this
after
these
two
jobs,
we
will
run
the
mutation
require
record
ingestion
test.
So
this
is
a
notebook
to
verify
the
results
of
the
ingestions
and
then,
after
the
ingestion
is
in
place,
we
will
run
the
engagement
id
mutation,
which
has
the
actual
our
business
logic
for
the
mutation
cases.
Then,
after
that
we
run
the
id
mutation
integration
test,
which
is
a
notebook
to
verify
the
actual
mutation
results.
E
E
E
E
So
in
this
case
the
first
run
id
mutation
scenario
and
the
second
hand
will
run
the
engagement
data
retention
process
scenario
and
followed
by
a
name,
mutation
scenario
and
so
on,
and
this
is
another
example
for
the
voice
in
size,
catalytic
data
lake.
So
similarly,
if
we
come
up
with
a
group
of
scenarios
for
our
sanity
on
tests,
so
you
notice
that
some
of
the
scenarios
is
small
with
a
couple
with
just
a
few
jobs
and
some
of
scenario
are
big
with
more
jobs.
D
D
So
keep
this
in
mind
while
I
go
through
so
the
goals
here
is
we
want
to
be
able
to
have
a
mechanism?
That's
low
friction.
You
know.
Obviously
nobody
likes
going
into
production
and
you
know
going
and
have
to
go
through
tons
of
hoops
in
order
to
you
know,
do
your
debugging,
as
I
mentioned,
we
want
to
prevent
data
tampering.
We
want
the
operations
to
be
pre-defined,
meaning
we
don't
want
users
to
be
able
to
go
in
and
randomly
do
some
ad
hoc
queries
like
drop
table.
D
That's
only
specifically
used
for
the
notebooks
and
the
access
for
most
users
on
those
clusters
is
limited
to
cameron.
They
can't,
like
you,
know,
stop
start
or
what
I
mean
they
can't
yeah.
They
can
do
other
stuff
on
the
clusters
and
to
help
us
enforce
the
predefined
access
requirement,
meaning
no
add-on
queries.
We
actually
use
the
databricks
widgets
to
pass
in
parameters.
D
So
here
is
an
example,
so
we
have
a
dedicated
interactive
cluster
called
foundation
debugging
over
here,
it's
created
as
part
of
the
sre
infra
provisioning
and
all
users
into
the
system
will
only
be
able
to.
You
know
run
the
cluster,
not
manage
it
and
we
act
at
the
top
over.
Here
we
have
the
widgets
and
in
order
to
set
up
the
widgets,
we
actually,
you
know
just
create
a
command.
The
api
and
stuff
is
listed
in
the
previous
slide.
It's
all
that's
available
online.
D
So
when
users
come
in
over
here,
they'll
just
you
know
select
whatever
database.
They
want
the
format
of
any
output.
You
know
we
use
evra
over
here
as
part
of
the
final
transformation
step
or
whatever,
and
then
the
s3
path,
where
the
data
is
stored.
What
fuels
in
the
select
on
the
delta
table
the
table
that
you
want
to
select
the
time
if
you
want
where
we
can
actually
filter
and
any
additional,
where
clauses.
D
Obviously
it's
you
know
re-owned
as
usual,
and
this
is
one
example
over
here
so
as
part
of
this
predefined
notebook
there's
a
read-only
command
which
allows
us
to
retrieve
the
the
number
of
files
in
a
certain
bucket
so
over
here
is
an
example
where
you
know
the
breakdown
of
all
the
files.
Obviously,
there's
most
of
the
files
fall
into
the
under
500k.
D
Bucket,
but
I
mean
this
is
all
good:
it's
extremely
user
friendly
and
it's
very
low
friction.
It
kind
of
meets
all
our
criteria
except
the
last
one,
which
is
the
security
approval,
and
there
are
three
issues.
The
first
is
that
the
out-of-box
audit
logging,
that's
provided
by
databricks,
it's
really
rich,
but
it
lacks
one
key
feature
which
is
as
part
of
the
notebook
widget.
D
It
doesn't
log.
It
doesn't
lock
the
input
parameters
that
the
user
you
know
actually
uses.
So
that's
something
that
didn't
sit
quite
well
with
security.
The
second
issue
is
that
users
actually
have
to
physically
log
in
to
the
database
portal
and
use
the
ui
and
everything,
and
that
requires
usability.
D
D
So
as
part
of
the
api
we
have,
you
know,
database
provides
methods
to
start
the
clusters
and
actually
invoke
commands
the
invoke
the
command
api
is
part
of
their
older
api.
It's
the
1.2
version,
but
they
still
support
it
and
it
after
analyzing
and
doing
some
you
know
spiking.
We
actually
found
that
it
would
meet
our
need
and
we
are
able
to
actually
build
it.
So
we're
actually
able
to
build
a
wrapper
api
service
within
our
own
salesforce
infrastructure,
which
would
meet
with
you
know,
take
all
the
boxes.
D
D
An
internal
job
which
then
calls
a
service
internal
service
which
basically
just
wraps
the
data
bricks
api,
so
that
we
can
invoke
all
the
databricks
commands
now
invoking
the
databricks
api
requires
an
access
token
that
you
know
was
also
mentioned
by
hanging
out
earlier.
In
this
case,
we
actually
have
a
single
user
provision
on
databricks,
specifically
for
this
debugging
purpose,
and
as
part
of
that,
we
generate
the
access
token
and
store
it
in
actually
called
fault.
D
So
the
api
would
retrieve
the
token
from
vault
and
you
know
invoke
whatever
apis
that
are
needed
for
the
production
development
to
work
so
yeah.
This
is
pretty
much
what
I
talked
about.
So
we
have
a
secure
access
portal
which
requires
both
sso
and
ubc.
It
requests
an
open
customer
case
and
it
provides
full
audit
trail
logging.
D
The
token
is
rotated
and
everything
here
is
fully
pre-defined
and
there's
no
available.
There's
no
way,
someone
can
actually
run
ad-hoc
queries
and
you
know
do
like
sql
injection
or
whatever,
so
the
basic
workflow
is
a
user
will
go
in
start.
The
cluster
they'll
check
whether
the
cluster
started
and
then
they'll
create
an
execution
context.
D
D
A
Awesome
thanks
very
much.
This
was
super
interesting,
hang
and
aaron.
I
really
appreciate
you
taking
the
time
to
provide
this
awesome
information.
We've
got
about
10
minutes
or
so
left
so
and
there's
a
bunch
of
questions.
So
let
me
try
to
tackle
some
of
the
most
recent
ones
that
just
came
through
on
the
pike.
If
you
do
have
some
more
questions
by
all
means,
please
go
ahead
and
continue
chiming
in
I.
A
C
Well,
it
sounds
good,
let's
see
so
we
did
have
a
question
from
evan
asking
about.
You
know
whether
you
can
use
an
interactive
azure,
databricks
cluster
for
batch
ingestion,
that's
results
like
and
which,
which
the
ideal
cluster
setup
be
for
that.
So
in
this
case
it
it
doesn't
matter
if
you're
using
like
which
cloud
provider
you're
using
databricks
on,
but
in
this
case
you
can
absolutely
use
an
interactive
cluster
for
batch
ingestion.
C
Now
the
second
part
of
the
question
talking
about
the
ideal
cluster
setup.
I
think
that's
a
little
more.
It's
not
nuanced,
but
I
think
there
are
a
couple
different
dimensions
to
consider.
So
first
would
be
the
actual
cluster
configuration
itself
right.
So
by
cluster
configuration
I
mean
what
node
types,
how
many
nodes
and
that's
specifically
going
to
be
highly
dependent
on
your
own
workload.
C
So
that's
something
you'll
need
to
test,
but
the
the
other
dimension
that
you
might
you
want
to
consider
in
this
case
is:
what's
called
the
cluster
execution
model
on
databricks,
so
in
databricks
generally,
there
are
two
types
of
clusters
that
you
can
create.
One
is
an
interactive
cluster,
where
you're
creating
it
to
be
always
on
to
do
interactive
work
on
whether
it's
development
or
running
queries,
things
of
that
nature
and
the
other
would
be
a
a
jobs
cluster,
which
is
a
cluster
that
a
databricks
job
creates
for
for
each
specific
runs.
C
C
So
you
know,
let's
say
that
you
you
need
this
job
absolutely
to
succeed
every
hour,
then
you
might
want
to
select
on-demand
nodes.
Only
so
you'll
want
to
configure
that
you'll
want
isolation
for
those
jobs
so
that
there's
a
resource
contention
to
potentially
slow
it
down.
With
other
queries
running
on
that
same
cluster
and
the
last
and
probably
most
important
consideration
is
job.
C
Job
clusters
are
much
cheaper
and
so
once
once
you're
in
production,
it
really
behooves
you
to
move
from
interactive
to
jobs
so,
and
I
think
to
what
aaron
was
sharing
if
you're
debugging
the
job
or
you're
needing
to
investigate.
That's
where
you'll
want
to
potentially
spin
up
an
interactive
cluster
again,
so
you
know
it's
that
there's
a
lot
of
information,
but
that's
the
way
that
we
want.
A
Cool
excellent
context,
craig
by
the
way
before
I
aaron
I'm
going
to
ask
you
the
next
question
that
it's
actually
a
pullback
from
the
the
first
sessions
about
in
the
engagement
activity
delta
lake,
but
the
context
I
I
did
want
to
call
out.
There
was
a
question
from
youtube:
considering
are
the
slides
and
presentations
going
to
be
available?
A
Yes,
they're
going
to
be
propped
right
here
into
youtube
since
you're
on
youtube,
then
they're
going
to
be
populated
there
in
addition
to
that,
we're
also
putting
them
into
the
delta
delta.ios
link
that
I'm
actually
going
to
send
to
you,
everybody
on
linkedin
and
youtube
as
well
and
zoom.
So
that
way
you
have
the
the
series.
Actually
all
there
saying
that
back
to
you
aaron
again,
this
is
referring
to
the
engagement
activity
delta,
like
the
one.
A
D
Exactly
micro
batch,
it's
not
really
real
time,
so
the
downstream
consumers
actually
pull
from
the
deltoid
once
currently,
once
every
four
hours,
there's
obviously
talks
to
increase
or
decrease
that
time,
and
you
know
so,
it
runs
triggers
once
once
every
four
hours
and
but
switching
to
real
time,
we've
actually
tested
it.
It
looks
fine,
our
pipeline
totally
scaled
yeah.
A
C
C
Maybe
you
can
just
address
it
real,
quick,
so,
okay,
so
we
have
a
question
about
utilizing
cloud
files
with
adls
gen
2.
I
think
maybe
just
to
well
to
address
it
briefly.
Yes,
like
cloud
files
is
what
you
want
to
use.
If
you
have
a
file
source
rather
than
using
the
default
file
source
in
structured
streaming,
we
we
do
highly
recommend
that
you
move
to
cloud
files
so
that
you're
not
having
to
spend
a
lot
of
time
listing
the
object
storage.
C
A
Sure,
and
actually
one
thing
I'll
add
to
crick's
call
out
so
specifically
it's
about.
I
believe
the
question
was
from
rada,
I
believe
and
from
linkedin,
and
he
was
asking
the
context
of
two
million
json
files
a
day
to
insert
or
update
a
delta
table
is
autoloader
a
good
idea,
so
underneath
the
cover
is
basically
the
way
autoloader
works
is
that
there
is
a
roxdb
frame
that
actually
keeps
track
of
all
of
the
files
that
are
being
processed,
there's
actually
multiple
threads.
That
will
go
ahead
and
fetch
and
delete.
A
So
that
way,
you
don't
actually
have
to
do
constant
cloud
file
listings.
The
cloud
file
listings
basically
makes
things
really
really
significantly
slow,
okay.
So
every
time
you
hit
a
cloud
object
store
and
you
ask
for
all
the
data
that
becomes
a
very
slow
process
right,
and
so
nobody
really
wants
to
deal
with
it,
especially
if
you're
constantly
rereading
that
full
list,
even
if
that
list
was
fast,
which
it
is
not
just
to
over
emphasize
that
point
again
right,
you
you're
not
going
to
know
necessarily
which
files
were
processed
which
files
weren't.
A
So
when
you
talk
about
millions
of
json
files,
that's
exactly
what
you're
worried
about.
So
what
all
loader
does
is
basically
takes
the
instead
of
actually
taking
the
list
of
files.
They'll
actually
make
use
of
cloud
file,
storage's
event,
notifications,
so
basically
without
zero
b,
as
your
q
stores,
I
believe,
and
with
aws
it's
event,
notifications
and
the
sk
that
goes
into
the
sqs
queue
okay.
A
So
what
ends
up
happening
is
that
information
that
file
anytime,
a
new
file
gets
placed
in
the
cloud
object,
storage
that
information
shows
up
in
the
queue
so
from
that
queue.
Any
process
that
we're
taking
on
from
with
autoloader
can
go
ahead
and
say:
okay,
wait,
there's
a
new
file
based
on
the
notification
that
I
just
received,
we
chuck
into
the
rocksdb.
So
that
way
we
can
figure
out
if
we're
processing
it
or
not.
A
If
we
need
to
process
that,
because
we've
already
loaded
that
information
into
a
very
high
performance
low,
latency
store,
this
allows
us
to
figure
out
hey
if
you
have
a
duplication,
because
don't
forget
those
cues
have
exactly
at
least
once
semantics,
in
other
words,
you'll
get
notified
of
the
file
at
least
once,
but
how
about?
If
you
get
notified
of
the
file
about
five
times
right?
Well,
that's
the
whole
point.
A
You
don't
want
to
do
that,
so
you
want
to
ensure
you're
not
be
running
into
duplications
of
the
files,
so
having
that
rocksdb
store
actually
allows
us
to
figure
that
out,
and
so
we
can
figure
out.
Yes,
we
processed
it.
No,
we
have
not
okay,
grab
the
file
and
continue
processing,
it
chuck
it
to
structure
theming
and
it's
on
its
merry
way,
and
so
that's
the
context
underneath
the
cover.
A
So
that's
why
we're
saying
yeah
yeah,
actually
autoloader
is
a
very
powerful
mechanism
to
process
that
data,
so
hopefully
that
provides
some
additional
context
around
that
cool
all
right.
Let's
go
back
to
the
fun
stuff,
though,
on
on
this
particular
session.
Okay,
there's
actually
a
pretty
a
good
all
question.
Referring
to
actually
boost
the
second
session,
boosting
delta
lake
performance
right
and
for
specifically,
there
was
a
question
around
why
I
believe
this
is
for
either
aaron
or
hank.
Why
do
your
ids
in
this
case
mutate?
A
Why
not
just
having
a
mapping
table
to
coordinate
all
that
information,
because
for
some
folks
and
the
reason
I
think
they
ask,
the
question
is
because
the
idea
is
that
you
know
like
you-
could
always
just
create
a
large
mapping
table
that
to
track
all
the
ids
and
the
new
ids
like
a
primary
key
circuit
key,
if
you
want
to
almost
think
of
it
that
way.
So,
instead,
why
do
you
have
the
mutate
instead
in
time
in
terms
of
your
design.
E
When
the
ownership
of
the
engagement
changes,
then
it
will
trigger
the
ids
of
ownership
to
change
and
will
also
trigger
all
the
engagement
ids
to
trick
to
trigger
that
belongs
to
this
belongs
to
this
owners.
So
this
is
which
to
simply
appeal:
that
is,
this
is
out
of
control
and
this
is
controlled
by
upstream
and
we
have
photo.
We
have
to
follow
that
rules
and
so
that
there's
no
way
for
us
to
use
a
mapping
table
to
map
the
engagement
and
the
owner
ids.
B
D
C
C
F
B
F
B
We
are
going
to
like,
because
the
data
that
the
actual
injection
there
is
no
in
table,
we
are
going
to
detect
it,
and
then
we
know
okay.
This
thing
this
is
the
like
early
arrival
mutation
data,
so
we
are
going
to
keep
those
data
and
then
in
the
retry
table,
and
then
we
are
going
to
do
like
rechoice
until
like
we
reach
like
the
the
maximum
number
of
rechores
yeah,
that's
how
we
handle
the
the
layer
of
data
in
in
our
case.
A
Perfect,
thank
you
very
much.
This
has
been
a
great
session.
I
realize
we
have
a
lot
more
other
questions,
but
we
do
need
to
close
it
up.
So
so
here's
my
ask
for
everybody
basically
number
one.
I
want
to
say
thank
you
very
much
to
hang
aaron
and
iphone
for
this
wonderful
session,
also
to
zone
he's
not
here
right
now,
but
we
definitely
want
to.
Thank
you.
I
want
to
thank
craig
and
chris
fish,
who
was
here
for
a
while
for
answering
questions.
A
Karen
of
course,
you've
been
leading
us
through
this
entire
thing.
Thank
you
very
much.
I'm
sure
you
have
other
questions.
So
here's
what
I
ask
we'll
ask
all
of
you:
go
to
delta.io
join
us
on
our
delta
user
slack.
We
are
very,
very
active
there
and
so
we're
constantly
answering
questions
they're,
whether
it's
your
delta
oss
questions
or
they're,
your
delta
database
questions.
Actually,
so
please
join
us
there
because
we're
very
intere
we're
very
reactive
there
and
we
try
to
provide
context
as
much
as
we
can.
So
I
think
that's
about
it.
G
Sure,
thanks
so
much
denny,
yeah
we'd
love
for
everyone
to
to
get
involved.
Denny
put
the
delta.io
slack
link
or
sorry
delta.io
link
to
the
tech
talk
series.
So
that's
what
I
was
going
to
point
to
is
all
the
four
talks
of
the
series
are
available
online,
so
we
hope
you'll,
hope,
you'll
check
those
out,
and
so
thanks
thanks
everyone
again
from
salesforce
and
craig
and
chris
for
this
awesome
session
and
thanks
everyone
for
joining
us.
So
hope
you
all
have
a
great
rest
of
your
day.
Thanks
so
much
take
care.