►
From YouTube: Tech Talk | Using Delta as a Change Data Capture Source
Description
Join us for an online tech talk on Delta Lake presented by Denny Lee and Paul Roome. Tech talks include a technical presentation with slides and a demo, with time for Q&A at the end.
While it is common to use Delta Lake as a sink for change data captured from traditional data sources; customers are increasingly asking how to use Delta tables as a source for a change data capture (CDC) process. To state a different way, how can we read a stream of changes from a Delta table, so that they can be propagated downstream.
Some example use cases include (but are not limited to):
- After cleaning the data following the Delta Architecture (bronze, silver, and gold tables), propagate this data to multiple downstream systems.
- An e-commerce company is using a Delta table to store features related to each of their customers sourced from multiple upstream sources. Upon any customer data change, this is propagated to update downstream ML models to provide the latest product recommendations to the customer.
- A large software company is using a Delta table to process and store 100s of TBs of customer telemetry data. Changes in this table need to be sent to a downstream consumer for updating a range of dashboards and analytics.
In each of these cases, we want to capture a change stream from a Delta table and send it somewhere for further processing. In this session, we will discuss the architecture, use cases, and solutions.
Agenda: 10AM PDT - 11AM PDT (GMT-8)
9:00AM - 9:50AM - Tech Talk
9:50AM - 10:00AM - Q&A
Link to Github for the notebooks: https://github.com/databricks/tech-talks Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Okay,
and
that
the
topic
at
hand
is
how
do
we
capture
change
data
from
a
delta
table
before
we
get
too
far
into
the
weeds
on
that
topic?
Lots
to
talk
about
on
that
today.
I
wanted
to
also
welcome
denny
lee
who's.
My
colleague
at
david
to
bricks
he's
the
he's
a
developer
advocate
here
at
databricks
he's
run
about
26
of
these
online
meetups
in
the
last
week.
A
So
what
are
we
speaking
about
today?
So
the
topic
of
the
talk
is:
how
do
we
capture
change
data
from
a
delta
table
and
we're
first
going
to
begin
with
motivation
behind
this?
Okay?
Why
are
we
even
here
today,
then
we're
going
to
look
at
three
different
architectural
patterns
of
how
we
can
actually
achieve
this
using
delta
and
in
these
patterns?
A
We'll
look
at
some
slide
based
architectures,
but
we'll
also
have
live
demos
if
the
demo
gods
smile
on
us
favorably
today,
we'll
wrap
that
up
with
a
summary
and
have
lots
of
time
for
questions
at
the
end.
Please
get
your
questions
in
throughout
the
presentation.
We
may
find
good
points
to
break
and
answer
some
of
those
throughout
the
presentation
as
well.
A
Okay,
so
capturing
change
data
from
a
delta
table
like
why
is
this
an
important,
an
important
subject?
And
I
think
it's
really
well
summed
up
by
a
conversation
that
I
had
with
my
colleague
yesterday.
I
told
her
that
I
was
gonna.
Do
an
online
delta
tech
talk
on
this
subject
and
she
said
paul,
that's
a
really
confusing
topic.
He
said
you're
mixing
concepts
from
two
different
worlds
and
I
completely
understand
what
she
means.
A
So,
on
one
hand,
change
data
capture
is
a
topic
that
enterprise
architects
have
been
speaking
around
about
for
the
past
30
years
and
generally
it's
in
relation
to
large
sql
stores
and
perhaps
in
this
data
warehouse
paradigm
and
every
technology
has
a
way
to
do
it,
whether
it's
scraping
bin
logs
or
stored
procedures
or
triggers,
and
you
might
be
familiar
with
a
schema
like
this.
That
mysql
gives
you,
where
you
get.
These
cdc
underscore
tables
that
give
you
this
change.
Data
from
each
of
these
tables,
but
delta
lake
operates
up
in
the
cloud
right.
A
It's
in
the
cloud
data
lake
paradigm.
So
we're
really
talking
about
change
data
capture
again
and
the
counter
to
my
colleague
was
it's
kind
of
like
what
I
really
like
about
delta
and
part
of
the
appeal
behind
it.
Many
of
the
things
that
we
know
and
love
about
data
warehouses
like
acid
transactions
and
scheme,
reinforcement
and
fine-grained
updates
and
deletes.
A
We
are
re-establishing
in
this
paradigm
of
the
cloud
data
lake
and
personally,
I
think
the
change
data
capture
will
be
one
that
we
look
to
to
capture
in
this
setting
as
well,
and
this
is
just
a
really
natural
cycle
of
innovation
and
disruption
right.
We
were
forced
to
innovate
from
the
data
warehouse
because
data
volumes
are
exploding
and
data
variety
was
exploding
and
you
know
the
data
no
longer
fit
into
these
nice
tabular
structures.
A
So
we
arrived
at
the
cloud
data
lake
where
we're
using
object,
storage,
which
is
super
cheap,
we're
using
managed
cloud
services
that
allows
us
to
be
really
agile,
and
you
know
we
have
cloud
elasticity,
but
that's
the
innovation,
the
disruption
is,
we
lose
some
of
the
things
that
we
know
and
love
right.
So
the
part
of
the
appeal
behind
delta
for
me
is
re-establishing
some
of
these
good
practices
that
we
had
in
previous
generations.
A
All
with
asset
guarantees
right,
so
you
love
that
you're
there
and
then
you
get
that
call
from
your
business
lead
and
they
say
you
know
that
delta
lake
table
you
created
every
time
a
record
is
inserted
or
changes
or
is
deleted
from
that
table.
Can
you
send
that
downstream
for
me,
because
we've
got
a
business
process
that
relies
on
this
and
you're
left
very
much
like
the
dog
in
the
photo
here
with
the
rubber
gloves
on
you're
like
now
what
okay?
So
that's
really
what
we're
here
to
talk
about
today?
A
A
So
your
bronze
area
is
an
area
of
your
lake
where
you
collect
the
raw
data
coming
in
from
its
source.
So
perhaps
you've
got
events
coming
into
kafka
and
you
might
write
them
into
a
delta
lake
table
and
just
keep
on
appending
onto
the
bottom
of
that
table,
and
you
don't
want
to
perform
any
cleaning.
You
want
to
capture
data
in
its
purest
form
so
that
no
mistakes
can
creep
into
this
data
whatsoever.
A
But
then
you
want
to
actually
apply
some
cleaning
on
to
it
and
perhaps
apply
a
schema
and
do
some
filtering
and
create
a
silver
table
right,
and
this
might
be
a
table
that
data
scientists
are
consuming.
So
they
get
something
close
to
raw,
but
don't
have
to
do
all
the
boring
stuff
and
then
finally,
you
might
perform
some
aggregates
this
day
to
produce
these
business
level
aggregates
that
multiple
consumers
will
be
really
interested
in,
perhaps
that's
the
gold
area,
and
so
in
each
area
of
this
lake
we
are
propagating
data
through
it.
A
Okay
and
actually
this
is
a
really
a
really
good
place
to
start
talking
about
change
data
capture
right,
because
if
we've
got
events
that
are
being
pumped
into
our
bronze
delta
lake
table
here,
then
we
want
a
really
efficient
way
to
propagate
them
into
our
silver
delta
lake
table,
okay,
having
cleaned
them.
So
specifically,
if
we
insert
a
thousand
records
into
our
bronze
table,
then
we
want
to
be
able
to
you
know
just
read
those
thousand
records
clean
them
and
write
them
to
our
silver
table.
A
So
we
can
actually
use
an
api
which,
possibly
you
all
are
familiar
with.
We
know
and
learn
from
spark
strictest
streaming
it's
spark.readstream
and
that
fully
supports
the
delta
format
here
and
we
can
just
simply
point
spark.readstream
format,
delta
and
point
it
at
our
underlying
delta
store.
So
here
this
is
a
customer
bronze
delta
table
that's
stored
in
s3
right.
So
the
first
point
here
you
can,
if,
if
this
table
is
append
only
you
can
simply
read
this
as
a
stream
using
spock.readstream.
A
And
the
second
point
is:
you
should
totally
use
this
pattern
because
it's
something
that
we
really
encourage
and
lots
of
customers
do
and
I'll
give
some
of
the
reason
behind
that
there
was
previous
methods
where
you
could
par.
You
could
point
re,
you
know,
spark
read
stream
at
an
s3
directory
and
use
wild
cards,
and
it
would,
you
know,
basically
read
it
as
a
stream.
A
However,
anyone
who's
actually
tried
to
do
that
in
production,
realizes
that
soon,
you're
you're
performing
list
book
operations
on
millions
and
millions
of
files,
which
is
very
costly
and
just
degrades
in
performance
over
time,
so
delta
doesn't
have
that
penalty.
Because
if
you
subscribe
to
the
last
delta
tech
talk,
we
keep
this
transaction
log
that
essentially
tells
delta
exactly
which
files
make
up
this
delta
table.
So
we
don't
have
to
go
back
to
s3
to
do
the
list
bucket.
We
already
have
that
information.
A
So
we've
read
our
bronze
table
as
a
stream.
We
can
perform
our
cleaning
using
all
of
our
structured
streaming
apis,
which
I'll
show
you
in
a
moment
and
then
all
we
need
to
do
now
is
write
that
stream
as
format
delta
and
write
it
into
our
customer
silver
delta
table.
Okay,
so
we're
propagating
from
bronze
to
silver
here
and
the
missing
piece
of
the
puzzle
that
we
haven't
talked
about
yet
is
how
do
we
ensure
that
every
time
you
know
that
thousand
records
is
inserted
into
bronze?
A
I
only
read
that
thousand
records
versus
the
full
petabyte
worth
of
data.
That's
in
that
table-
and
here
we
borrow
a
concept
from
structured
streaming
called
checkpointing
right,
takes
care
of
this,
so
you'll
notice.
The
second
line
here
we
are
using
this
checkpoint
location
again
we're
just
pointing
to
a
location
on
s3
which
is
safe
to
do
so
on
on
databricks,
and
this
is
going
to
take
care
of
all
of
the
semantics
to
do
with
managing
offsets
and
just
reading
the
data
exactly
once
okay.
So
this
this
guarantees
exactly
once
processing.
A
And
we're
going
to
have
a
look
at
exactly
this
pattern
of
that
we
just
described
in
the
slide,
which
is
how
can
we
automatically
propagate
data
from
our
bronze
delta
table
into
our
silver
delta
table
after
applying
some
cleaning
to
it?
Okay,
so
the
data
that
we're
going
to
be
looking
at
today
and
we'll
share
all
these
notebooks
there's
going
to
be
a
companion
blog
post
to
this
webinar.
So
all
of
these
will
be
shared
as
well.
A
So
we're
going
to
begin-
and
let
me
just
make
this
writing
a
bit
bigger
for
you-
we're
going
to
begin
by
just
generating
some
fake
data
here.
Okay,
so
there's
there's
two
million
records
that
we're
going
to
put
into
this
table,
half
of
them
say
seinfeld,
jerry
and
half
of
them
say
david
larry,
okay,
so
it
looks
something
like
this:
this
very
duplicated
data-
this
is
our
dell.
This
is
our
delta
table
here.
So
we're
just
going
to
begin
by
inserting
this
data
into
our
bronze
table.
A
Then
what
we
need
to
do
is
to
define
the
pipeline,
which
will
automatically
scan
this
table
and
propagate
any
of
the
data.
The
new
data
to
our
customer
silver
table
having
cleaned
it.
So
I'm
just
going
to
start
my
my
streaming
job
here
while
I,
while
I
talk
through
it,
so
this
is
very
similar
to
the
slides
that
we
just
saw
so.
Firstly,
we're
just
going
to
use
our
helpful,
read
stream,
api
and
point
it
at
our
delta
table.
A
We're
then
going
to
perform
our
actual
name,
our
cleaning
of
the
data.
So
here
we
are
just
simply
splitting
seinfeld
jerry
into
two
columns,
jerry
and
seinfeld,
and
then
we're
going
to
write
that
data
to
our
customer
silver
delta
table
and
we
should
be
able
to
see
that
we've
processed
some
records
here
yep,
so
some
records
are
initially
processed
and
we'll
be
able
to
see
the
results
if
we
run
this
next
cell
here.
So
this
is
a
this
is
a
visualization
of
the
the
customer
silver
data
set.
A
So
you
can
see
that
we've
split
that
one
column
of
seinfeld
jerry
into
two
okay,
so
we've
propagated
data
from
bronze
to
silver.
Here:
okay,
let's
run
some
quick
diagnostics
on
this
table,
so
let's
have
a
look
at
this
silver
delta
table
and
we
can
see
that
in
fact,
yes,
we
have
a
million
records
corresponding
to
jerry
seinfeld
and
a
million
larry
david
records,
perfect
okay,
but
let's
really
put
ourselves
to
the
test.
A
Nothing
like
a
live
demo
of
a
test
here.
So
since
we
have
defined
this
streaming
pipeline
and
it's
still
running
now
that
is
automatically
scanning
our
bronze
table
and
propagating
it
to
silver.
We
should
be
able
to
insert
more
records
into
our
bronze
table
and
it
be
automatically
picked
up
and
propagated.
A
So
let's
do
exactly
that.
Let's
generate
a
million
more
records
that
now
say
elaine
bennis
in
them
and
write
it
into
our
customer
bronze
delta
table
and
we
should
be
able
to
see
a
spike
appear
in
our
chart.
There
we
go.
So
if
we
look
back
at
our
streaming
pipeline,
we
can
see
that
we've
processed
a
bunch
of
data.
This
spike
is
the
data
that
I
just
inserted
and
we
should
be
able
to
see
them
automatically
propagated
both
in
well.
They
should
obviously
be
in
the
bronze
table
here.
So
now
we
see
in
bronze.
A
A
A
And-
and
we
could
just
insert
those
records
here-
we
should
be
able
to
see
a
spike
in
our
chart
here:
okay,
corresponding
to
the
records
that
have
been
processed,
and
then
they
will
be
automatically
propagated
into
our
silver
delta
lake
table
here,
fantastic.
So
now
we
have
four
different
records
here
we
have
cosmo,
we
have
seinfeld,
venice
and
david,
so
it's
really
cool
right,
because
we
could
just
be
pumping
in
records
into
our
bronze
table
via
kafka
and
we've
got
this
automated
pipeline.
A
A
So
this
is
our.
This
is
our
first
pattern
of
reading
change
data
from
from
a
delta
table.
I
wanted
to
mention
one
date
that
you
can
make
to
this
pipeline
that
many
of
our
customers
choose
to
do
so
in
the
example
that
we
just
saw,
the
data
was
really
duplicated
and
you
might
be
saying
paul.
That's
not
a
clean
data
set
right
like
I
don't
want
my
users
consuming
that
data.
So
what
you
might
want
to
do
is
add
a
deduplication
step
there
as
well.
A
B
Oh
no
problem,
we
answered
a
bunch
of
the
questions
right
away.
Basically,
on
the
q
a
panel
considering,
like
you
know,
the
street
stream
only
works
with
the
delta
format
right
away.
One
thing
I
didn't
want
to
clarify
right
away
is
that
which
you
alluded
to,
but
I'll
just
I
want
to
call
it
out
specifically.
Was
that
yeah?
B
A
A
So
let's
introduce
this
concept.
So
now,
let's
assume
that
we
have
a.
We
have
a
a
source
of
updates
which
is
perhaps
coming
from
kafka
right
and
now
we're
streaming
insert
and
updates
into
a
delta
table,
and
we
want
to
be
able
to
capture
any
time.
We
have
a
change
and
propagate
it
downstream
somewhere.
So,
for
example,
we
might
insert
a
record.
A
We
definitely
want
to
capture
that,
but
we've
already
discussed
that
we
might
update
a
record
that
was
inserted
30
days
ago,
and
we
definitely
want
to
be
able
to
capture
that
and
push
it
downstream.
So
this
is
kind
of
a
classic
change
data
capture
use
case
we're
not
considering
deletes
yet
just
fyi.
A
So
I
actually
came
across
a
really
interesting
example
of
this
in
the
field
which
I
think
will
illuminate
this
use
case.
So
I
was
working
with
an
e-commerce
website
company
here
in
the
bay
area
and
they
were
using
delta
to
store
a
big
feature
store
for
their
customers.
So
you
can
imagine
each
one
of
the
records
in
this
feature.
A
A
So
we
can
still
still
do
that
using
our
favorite
read
stream
api.
However,
this
time
we
need
to
use
what's
called
the
ignore
changes,
flag,
okay,
and
I
will
explain
exactly
what
that
means.
So
using
the
ignore
changes,
flag
will
emit
any
file,
rewrite
operation
to
the
stream
and
that
might
sound
critical.
So
I
have
a
diagram
for
you
here.
A
So
if
we
just
go
back
to
my
previous
slide
here,
this
might
make
a
little
bit
more
sense.
Now.
The
the
changes
that
we
get
in
our
stream
will
be
a
file
level
cdc,
so
they're
guaranteed
to
con
to
include
all
the
data.
That's
changed,
but
it
might
be
a
superset
of
that
meaning
there
might
be
extra
records
that
weren't,
the
one
customer
we've
changed
in
our
case,
I'm
completely
fine.
A
With
this
trade-off,
it's
a
really
easy
way
to
get
change
data
out
of
delta,
and
in
this
case
you
know
we
just
want
to
rescore
customers
who
have
a
changed
feature
and
make
sure
they
get
a
new
recommendation.
But
if
we're
rescoring
some
other
customers
as
well-
I'm
not
too
concerned
about
that
because
their
recommendation
won't
change
okay.
So
this
is
a
really
easy
way
to
do
it
if
you
understand
the
trade-offs.
A
So
the
scenario
that
we're
going
to
try
and
replicate
here
is
we're
going
to
build
our
feature,
store
delta
table,
we're
going
to
throw
some
updates
at
it
and
see
whether
we
can
automatically
propagate
them
downstream,
having
scored
them
okay.
So
let's
have
a
look
at
this
architecture
here,
so
we're
going
to
use
our
favorite
characters
from
seinfeld
here,
or
at
least
two
of
them,
jerry,
seinfeld
and
larry
david
right
here.
So
this
time
the
the
data
frame
or
the
delta
table
is
going
to
look
like
this
right.
A
It
says
it's
got
a
few
extra
columns
here
last
purchase
day
and
a
customer
id
and
we're
going
to
begin
by
again.
Writing
this
into
our
feature,
store
delta
table
using
our
data
frame
right
api
and
we're
just
writing
it
into
s3.
Here,
as
our
feature
store
delta
table,
we're
then
going
to
define
a
model
to
actually
score
these
customers.
I'm
not
going
to
go
too
much
into
this
model.
A
A
Apply
our
model
udf,
which
is
the
model
that's
going
to
score
the
customer
and
we're
going
to
create
a
new
column
with
that,
and
then
we're
simply
just
going
to
write
this
as
delta
format
again
to
our
customers
scored
table
okay,
so
we
should
be
able
to
see
yep.
Some
records
have
been
processed
through
our
streaming
pipeline
already.
So
let's
have
a
look
at
the
output
here.
So
here
is
the
customer
scored
data
delta
table
and
we
can
see
that
it
looks
very
similar
to
what
we
saw
before.
A
We've
got
two
extra
columns,
which
is
the
score
of
the
customer
and
the
score
timestamp.
Okay,
and
in
this
live
demo
session,
we
like
to
again
put
ourselves
to
the
test
here.
So
let's
throw
an
update
at
this
feature,
store
table
and
see
whether
it
gets
automatically
propagated
into
our
customer
score
delta
table.
Okay.
So
I'm
going
to
run
this
command
right
here
and
those
you're
familiar
with
delta
apis.
This
is
the
scholar
api
that
allows
you
to
update
a
record
or
a
series
of
records
in
a
delta
table.
A
So
what
I'm
saying
here
is
find
customer
zero
and
update
the
last
purchase
date
right
in
our
feature,
store
delta
table
here,
and
when
I
do
that,
I
should
be
able
to
see
if
we
scroll
back
up
to
our
chart
here,
look
yeah,
so
we
already
have
a
spike
in
our
stream,
which
means
some
data
was
propagated
into
our
pipeline
right.
So
our
we
have
detected
this
change
and
we
should
be
able
to
see
our
customer
has
been
re-scored
in
our
customer
score
data
set.
A
A
The
one
caveat
that
you
know
we've
already
discussed,
but
I
just
wanted
to
make
sure
that
you're
very
clear
on
this
point.
So
let's
read
this
customer
score
data
set
again
and
if
you
see
what
actually
happened
here,
there
are
31
000
records
that
were
rescored
as
part
of
the
last
update
okay.
So
we
made
one
change
and
that
triggered
31
rescorings
took
a
fraction
of
the
second,
so
it
didn't
really
matter
so.
A
A
Table
fantastic,
so
this
is
this
is
what
I
would
call
a
really
good
pattern
for
really
easily
reading
change
data
from
a
delta
table
when
you've
got
updates
being
applied
to
that
that
delta
table,
and
I
just
wanted
to
take
a
break
and
a
check
there
danny
any
questions
that
I
can
answer
or
any
comments.
I
should
be
aware
of
in
the
in
the
chat
there.
B
Cool,
actually,
the
the
quick
call
out
here
right
now
actually
had
a
little
less
to
do
with
the
pattern
that
you
designed,
because
I
think
a
lot
of
the
the
a
lot
of
people
are
completely
grocking.
The
questions
around
that,
but
more
about
the
internals
and
so
to
help
clarify
some
of
these
things.
Just
because
it
would
take
a
long
time
right
to
clarify
all
those
things.
What
I
did
is
I
went
ahead
and
put
three
youtube
links
inside
the
chat
for
everybody
that
on
unpacking
the
transaction
log.
B
If
you
want
to
know
more
about
how
the
delta
lake
transaction
log
works,
also
about
schema
evolution
enforcement
that
by
default
delta
lake,
enforces
the
schema
and
then
just
as
paul
has
been
talking
about.
If
you
want
to
evolve
the
schema
you
can
and
then
how
that
works
and
we've
put
a
link
inside
there
as
well
and
finally,
also
about
we're
talking
about
updates
and
a
precursor
to
delete
how
the
internals
of
that
work
in
delta
lake
as
well,
and
we've
included
that
link
as
well
inside.
B
A
Perfect,
thank
you
very
much
danny,
of
course,
all
right.
So,
let's
crack
on
with
the
third
and
final
pattern
here
and
probably
the
title
now
makes
sense.
Can
we
do
better
so
last
time
we
saw
how
we
can
produce
file
level
change
streams
by
using
the
ignore
changes
flag.
But
can
we
actually
get
record
level
change
streams
and
can
we
deal
with
deletes?
So
that's
pattern.
Number
three
and
the
first
part
of
pattern.
Number
three:
is
you
might
call
this
a
you
know
a
cheat
pattern
for
the
for
the
really
simple
use
cases?
A
Okay,
so
let
me
describe
this
so
let's
say
we
have
a
source
of
updates
inserts
and
deletes
perhaps
get
this
is
coming
from
kafka
again.
The
first
of
the
way
that
we
can
achieve
this
record
level.
Change
stream
from
the
delta
table
is
to
first
write
this
into
a
bronze
table
as
a
pen.
So,
even
if
a
record
is
saying,
you
know
update
a
particular
record,
we're
going
to
write
that
as
an
append
into
our
del
into
our
bronze
layer.
A
We'll
then
subscribe
to
that
table
as
a
stream
like
in
pattern
one
and
we
will
apply
those
updates
and
deletes
and
inserts
how
they're
meant
to
be
applied
right,
so
we'll
apply
them
as
those
things
and
produce
our
silver
table
and
the
silver
table
will
be
the
one
that
people
are
consuming
now.
The
question
we
have
here
is:
how
do
we
detect
the
changes
that
are
happening
in
our
silver
table?
A
We
don't
actually
have
to
detect
the
changes
right,
there's
only
one
pipe
coming
into
our
silver
table,
so
we
can
simply
redirect
that
pipe
elsewhere
as
well.
So
we
can
send
that
data.
We
can
do
a
second
read
stream
on
this
bronze
table
and
they're
completely
independent
of
each
other
using
different
checkpoint
directories,
and
we
can
send
that
downstream.
A
So
this
will
work
in
really
simple
use
cases
where
we
have
one
source
of
of
updates
and
deletes
and
inserts,
and
we
have
very
simple
update
logic
happening
in
our
delta
layer.
Right
if
there's
very
complicated
logic,
when
we
apply
our
updates,
then
we
perhaps
don't
want
to
have
to
replicate
that
downstream
right.
But
if
it's
very
simple,
then
that's
less
of
a
risk.
A
But
how
do
we
do
this
in
a
more
general
pattern,
when
we
don't
have
just
this
one
input,
source
of
of
updates
and
deletes,
and
we
can
actually
borrow
a
trick
from
the
data
warehousing
world
and
we
can
introduce
a
change
log
table.
Many
of
you
might
be
familiar
with
this,
so
what
this
allows
us
to
do.
Let
me
speak
through
the
architecture
diagram
here,
so
we
have
our
source
of
updates
inserts
and
deletes
commits
streaming
in
and
we're
going
to
perform
a
transactional
right
to
two
different
delta
tables
here.
A
So,
firstly,
we're
going
to
apply
the
upstates
updates
inserts
and
deletes
to
our
main
delta
table
the
table
that
people
will
be
consuming
for
and
using
for
analytics,
but
then
we'll
also
perform
a
second
right
into
our
change
log
table
and
we'll
write
everything
as
an
append
right.
So
you
can
imagine
over
time
this
change
log
is
gonna.
It's
very
self-descriptive.
It's
gonna
be
a
record
of
all
the
changes
that
have
been
happening
in
our
main
delta
table
and
because
we're
writing
them
as
appends.
A
We
can
actually
read
this
as
a
stream
like
we've,
been
talking
about
and
perform
a
join
against
the
main
delta
table
to
retrieve
to
retrieve
the
full
record.
So
let's
have
a
look
at
how
we
do
that.
So
how
can
we
begin
by
reading
our
streaming
source
and
writing
it
to
two
different
delta
tables
in
this
manner,
so
the
magic
we
use
here
for
those
of
you
familiar
with
the
structured
streaming
world?
Is
we
use
this
concept
of
a
for
each
batch?
A
Okay,
so
spark
structured
streaming
operates
in
this
micro
butch
batch
format,
and
we
can
use
this
for
each
batch
function
or
api
to
implement
a
custom
function
with
what
we
want
to
do
with
that
batch.
Okay.
So
this
upset
to
delta
capture
cdc
function
is
the
thing
that's
going
to
tell
delta
or
structured
streaming
rather
to
write
to
two
tables.
A
So,
let's
begin
with
the
following
the
following
diagram:
so
here's
our
main
delta
table
that
has
three
columns
id
some
string
and
some
other
string.
Okay-
and
this
is
the
table
that
all
of
our
users
are
consuming.
Then
we
have
this
source
of
updates,
that's
coming
in
and
essentially
it's
just
gonna.
It's
gonna
change,
one
of
the
columns
in
our
main
delta
table
to
oranges.
So
you
can
imagine
if
we
apply
this
to
this
table,
we're
going
to
have
apples
and
oranges
sprinkled
through
our
main
delta
table-
and
these
are
all
type
updates
here.
A
So
what
we're
going
to
do
in
our
for
4-h
back
function
is
the
first
thing
we're
going
to
do
is
actually
just
drain
our
records
out
of
our
streaming
source
and
we're
going
to
apply
them
as
updates
to
our
main
delta
table.
So
here
we
have
apples
and
oranges
sprinkled
through
our
table,
but
then
we're
also
going
to
write
them
as
inserts
into
our
change
log
and
record
the
batch
they
happened
in.
So
we
can
see
that
we
are
recording
what
is
happening
in
our
main
delta
table
in
a
separate
table
called
changelog.
A
A
The
static
side
will
actually
automatically
refresh
with
changing
data
under
the
hood.
So
this
main
table
is
actually
changing
under
the
hood
right.
We're
playing
up
we're,
applying
updates
to
it,
and
we
don't
need
to
do
anything
every
time
this
this
executes
in
each
batch
of
the
structured
streaming.
A
So
here's
what
we're
going
to
try
and
do
so.
This
is
the
architecture
that
we
had
on
the
slide
before
we
have
a
source
of
updates
here,
it's
being
captured
in
a
delta
table.
This
could
be
kafka
or
kinesis
or
event
hubs,
or
anything
we're
going
to
read
from
here
and
we're
going
to
automatically
apply
our
updates
to
our
main
delta
table
and
simultaneously
write
to
our
change
log.
A
Then
we're
going
to
read
our
change
log
as
a
stream
and
join
it
back
to
our
main
delta
table
and
send
it
downstream
somewhere.
Okay.
So,
let's
begin
by
making
some
fake
data
and
writing
it
into
our
delta
table,
okay,
so
our
main
delta
table
is
gonna.
Look
like
this
okay,
so
it's
as
you
saw
in
the
slides.
We
have
three
columns
here
and
everything
says
apples
and
cakes.
It's
all
a
happy
world.
A
Now
what
we're
going
to
do
is
define
this
part
of
the
streaming
pipeline
that
performs
the
simultaneous
write.
So
here
is
my
my
alluded
to
function
that
will
perfor,
which
will
be
operated
on
as
part
of
the
for
each
batch.
You
can
check
this
out
when
we
provide
the
links,
but
what
I
wanted
to
do
was
just
to
start
my
streaming
pipeline
here.
A
Now
stream
is
just
initializing
right
now
it
should
be
processing
its
data
finishing
up
any
second
now
so
here
we
go
so
we
can
see
that
our
data
has
actually
been
processed
through
this
part
of
the
pipeline,
and
we
should
be
able
to
see
to
start
with
that.
Our
main
delta
table
is
being
updated.
Okay,
so
we
should
be
able
to
see
apples
and
oranges
sprinkled
through
this
data
set.
A
A
Okay,
now
what
we
need
to
do
is
assemble
the
second
part
of
the
pipeline,
which
is
read
the
change
log
as
a
stream
and
join
it
to
our
main
delta
table.
So
let
me
start
that
part
of
the
processing.
While
I
talk
you
through
the
code
here,
so
we
are
reading
as
a
stream
here
we're
reading
this
change
log
as
a
stream.
We
are
simply
joining
it
onto
our
main
delta
table.
This
is
a
stream
static
join
and
then
we
are
writing
it
out
to
our
destination
table
a
cdc
consumer.
A
If
you
like
the
change
stream,
full
records-
okay
and
we've
already
sucked
some
data
into
this
pipeline
and
processed
it.
So
we
should
be
able
to
query
this
table
the
change
stream
for
records
and
see
what
the
cdc
consumer
will
be
getting.
So
indeed,
we
are
sending
every
second
id
to
the
downstream
consumer
and
we're
sending
the
full
record
that
was
changed
in
the
delta
table.
A
So
let's
just
throw
some
data
in
there
and
the
data
that
we're
going
to
throw
in
there
is
every
third
id
we're
going
to
change
to
pairs
now
right.
So
let's
have
a
look
at
that
update
set
here
so
every
third
id
we
want
to
change
some
string
to
to
pairs
okay,
so
we've
written
that
into
our
update
table.
Since
my
streaming
pipelines
are
running,
it
should
be
getting
propagated
through
my
pipeline
here.
So
let's
just
have
a
quick
look
up
at
my
pipeline,
so
here's
my
streaming
pipeline.
A
We
see
that
we
have
a
a
spike
in
my
stream.
That
means
I
read
some
data
and
some
data
was
processed,
so
that's
the
first
part
of
the
pipeline,
so
that
looks
good
and
then
the
second
part
of
the
pipeline.
I
have
a
second
spike,
which
means
that
data's
being
propagated
through
there.
So
if
I
go
down
here,
we
should
be
able
to
see
that
some
more
data
has
arrived
in
our
downstream
location,
and
here
we
see
it.
A
So
we
see
that
we
have
data
corresponding
to
batch
id1
and
some
string
has
now
been
changed
to
pairs
okay.
So
this
is
what
we
can
send
downstream,
so
we
were
actually
able
to
implement
a
pipeline
which
captures
changes
that
are
happening
in
our
main
delta
table
and
sends
them
downstream
and
we're
doing
that
at
a
at
a
record
level,
which
is
fantastic.
A
So
the
last
comment
I'll
make
on
this
is:
we
haven't
talked
about
deletes
right,
but
this
can
really
easily
be
extended
to
capture
deletes
as
well.
So
let's
say
that
one
of
our
records
in
our
streaming
source
was
a
delete.
It
says,
delete
record
10.,
so
in
this
pattern,
what
we
would
do
is
go
and
go
ahead
and
apply
that
delete
to
our
delta
table.
But
we
would
then
also
log
that
in
our
change
log
the
delete
happened
and
then
we
would
just
modify
our
join
a
little
bit
to
do
a
left
join
here.
B
B
B
The
other
question,
basically,
is
that,
should
you
what's
the
best
way
to
do
this
as
a
nightly
batch
rather
than
a
stream,
and
so
the
quick
call
that
I
would
say
is
that
you
absolutely
can
run
this
as
a
batch.
If
you
wanted
to
the
in
terms
of
like
just
processing
as
a
change
set,
and
in
fact,
what's
great
about
paul's
example.
B
Anyways
is
because
of
the
fact
that
if
you
need
to
orchestrate
many
many
different
batch
jobs,
instead
of
actually
having
one
back
like
instead
of
having
50
or
70
different
batch
jobs,
you
could
potentially
just
run
this
as
a
stream
instead.
So
in
other
words,
basically
a
bunch
of
mini
bash
jobs,
and
so,
for
example,
customers.
B
Like
comcast,
which
you'll
which
talked
about
how
they
did
this
back
in
spark
summit
san
francisco
2019
in
the
keynote
they
went
from
84
different
jobs
down
to
three
and
because
they
were
able
to
using
delta
lick
underneath
the
covers
they
were
able
to
reduce
their
batch
jobs
from
84
backdrops
down
to
three
streaming
jobs.
And
so
it
was
operationally
much
simpler
to
be
able
to
maintain
and
operate
this
type
of
design.
B
It
is
also
about
managing
or
simplifying
the
management
of
many
batch
jobs
as
well,
and
so
that's
an
important
call
out
that
I
figured
that
should
be
called
out
here
that,
as
of
why,
when
you're
looking
at
a
session
like
this
one
in
terms
of
cdc,
even
if
you
initially
want
to
start
as
a
batch
and
again
just
switch
from
restream
to
read
right,
you
may
want
to
seriously
consider
the
context
of
staying
as
a
stream
job
anyways.
Just
because
of
the
potential
operational
benefits
as
well.
A
Great
point,
and
just
one
thing
I'd
add
to
that
another
really
common
pattern
here
is
using
a
trigger
once
philosophy
that
basically
turns
a
read
stream
into
a
batch
execution
and
allows
you
to
kind
of
then
go
backwards
so
trigger
once
will
essentially
every
time
it
goes
to
the
streaming
source.
It
will
just
read
all
the
records
that
are
in
there
and
process
them
in
batch
right.
So
this
is
a
really
common
pattern,
as
well.
A
Fantastic,
so
coming
up
on
the
last
10
minutes
says
I
just
want
to
wrap
this
up
in
a
summary.
What
have
we
seen
today?
So
we've
looked
at
three
different
patterns
of
reading
change:
data
from
a
delta
table.
The
first
one
of
these
is
what
I
called
an
append:
only
pipeline
really
commonly
used
in
this
bronze
silver
gold
propagation
across
your
lake.
A
The
second
was
what
I
like
to
call
the
easy
button
for
the
update
pipelines
using
the
ignore
changes
flag
and
remember
what
happens
here.
We
get
this
file
level
cdc
stream,
but
it's
really
really
easy
to
implement
and
then
the
third
pattern
here
is
this
change.
Log
paradigm,
borrowing
a
concept
from
data
to
warehousing,
and
this
actually
allows
us
to
capture
this
record
level.
Change
stream,
like
you,
would
perhaps
get
in
in
a
data
warehouse.
A
So
I'd
love
to
hear
your
feedback
on
the
session,
as
I've
said
throughout
there's
going
to
be
a
companion
blog
post
with
links
to
all
of
the
the
notebooks
and
code
here
and
all
the
all
the
data
which
is
generated
in
those
notebooks
as
well
so
you'll
be
able
to
check
this
out
on
your
own.
So
really
appreciate
you
spending
the
time
with
us
this
morning
and
danny.
If
there's
any
more
questions
happy
to
take
it
or
we
can
go
from
there.
B
So
I'll
answer
the
first
one,
okay,
the
first
question
is:
if
we
have
any
query
questions
about
in
terms
of
from
the
webinar
per
se,
from
this
from
this
webinar
and
now,
how
can
we
follow
up
with
you
after
the
fact?
And
so
that's
why
we've
been
talking
about
the
youtube,
our
youtube
channel,
so
youtube.com
databricks,
this
video
actually
is
propped
there,
and
so
we
actually
do
go
out
of
our
way
to
answer
questions
also
on
the
youtube
channel
itself.
B
So
pop
right
into
the
comments
and
we'll
do
our
best
to
answer
questions.
It
usually
takes
us
about
48
hours
to
answer
them,
because
we
have
a
lot
of
questions
with
a
lot
of
videos
these
days.
So
we
don't
want
to
be
able
to
promise
you
immediate
answers,
but
we
do
review
them
on
a
pretty
regular
basis.
Okay,
and
so
hopefully
that
will
help
you
jump
start
this
process.
Okay,
all
right!
The
next
question
is
basically
it.
Oh
here,
here's
a
good
one
and
paul.
B
You
can
answer
it
or
I
can
answer
if
orchestration
steps
are
dependent
on
each
other,
in
other
words,
step
a
has
to
be
complete
before
state
b
will
have
to
start.
Basically,
it's
like
you
have
these
dependencies
on
your
jobs
between,
like
your
goal,
table
versus
your
silver
table.
What
do
you
think
is
the
best
approach,
and
I
realize
there
are
multiple
answers,
but
you
know
so
why
don't
we
just
start
that
discussion
right
now.
A
Yeah
sure
yeah
really
common
question
we
get
and
if
you
want
sort
of
a
system
to
be
able
to
run
arbitrarily
complex,
dags
right
of
dependencies,
which
is
what
you're
describing
there.
We
really
recommend
the
integrations
that
spark
and
databricks
has
with
technologies
like
apache,
airflow
or
azure
data
factory.
Those
are
really
good
options
for
for
that
which
provide
that
kind
of
arbitrarily
complex,
dag
manipulation
and
and
execution.
B
Exactly
yeah
I
was
just
about
so
that,
hence
the
reason
for
me
also
calling
out
apache
airflow,
that's
a
very
common
one
for
folks
to
use
in
the
open
source
land,
but
the
the
context
is
that
it's
as
simple
or
as
complicated
as
you
want
it
to
be.
When
it
comes
to
tracking
those
dependencies,
because,
for
example,
you
could
always
just
create
a
shell
script
that
places
in
like
basically
case
statements
in
your
cron
job.
It
could
always
start
with
that
right
and
it
could
go
into.
B
For
example,
when
you
talk
about
databricks
notebooks,
you
could
potentially
just
go
ahead
and
actually
build
workflows
within
your
notebooks
that
have
that
stacks,
but
then
exactly
as
pole
called
out
right,
whether
it's
an
azure
data
factory
style
or
if
it's
a
apache,
airflow
or
whatever
else.
The
point
is
that
those
dags
can
get
more
and
more
complicated
and
usually
when
we
involve
those
latter
ones.
It's
not
just
because
of
the
dependencies
between
silver
and
gold,
but
because
of
dependencies
with
other
systems
that
you're
orchestrating
way
beyond,
like
in
terms
of
your
own
system.
B
So
partners
like
informatica
or
partners
like
stream,
sets,
which
also
talk
to
delta
lake.
They
also
typically
will
get
involved
as
well.
So
again,
there's
a
lot
of
really
good
ways
to
do
it.
I
don't
think
there's
any
wrong
way
per
se.
It's
just
more
a
matter
of
understanding.
What
is
the
level
of
complexity?
B
You
actually
need
and
then
basically
designing,
therefore
deciding
for
that,
because
in
the
end,
when
it
comes
to
this
type
of
orchestration,
if
you
were
able
to
do
it
with
spark
you're
able
to
be
able
to
do
a
delta
lake,
because
delta
lake
is
an
underlying
storage
layer.
So
whatever
you
were
trying
to
do
with
before
with
spark
it's
pretty
much
applicable
here
as
well.
Okay,
all
right,
oh
and
then
there
is.
We
got
some
quick
calls
outs
to
say
they
really
appreciate
the
session
paul.
B
I
think
they
love
you
so
good,
good,
good
yeah.
I
I
I
can
now
quit
now.
No
I'm
joking,
but
yes,
the
powerpoints
and
the
code
will
be
available.
So
two
quick
call
notes.
We
actually
put
all
the
code
and
the
notebooks
and
the
powerpoints
directly
in
the
databricks
tech
talks,
github,
which
is
also
linked
in
the
youtube
channel.
Okay,
and
so
we
put
all
there
and
then
just
as
paul
alluded
to
multiple
times.
We
are
actually
currently
writing
a
blog.
B
That
provides
a
lot
more
of
the
details
that
we
actually
want
to
go
for
much
deeper
than
just
what
we're
able
to
do
in
a
you
know:
50
55,
minute
tech
talk,
so
we
will
be
following
with
that,
with
additional
information
linked
to
this
video,
but
also
link
to
the
tech
talks,
github,
repo
and
link,
oh,
which
contains
the
code
and
the
notebooks
okay,
all
right
cool.
I
I
think
the
final
question
I
want
to
tackle
here
is
basically
before
we
head
back
to
karen.
B
A
Yeah,
so
that's
if
you
use
the
just
the
read
method,
then
you
don't
you
don't
get
to
use
the
checkpoints,
so
you
have
to
track
the
watermark
yourself.
That's
the
real
benefit
of
using
the
trigger.
Once
is
that
you,
you
can
execute
in
like
a
batch
mode,
but
still
take
advantage
of
all
of
those
nice
things
in
stricted
streaming
like
the
checkpointing,
and
the
other
thing
to
say
is
that
it
allows
you
to
evolve
in
the
future.
B
Excellent
and
that's
exactly
by
the
way,
just
to
add
to
paul's
point,
that's
precisely
what
what
I
meant
by
the
orc
simplifying
your
orchestration
of
batch
jobs
to
streaming,
because
the
idea
is
that
you
have
things
like
checkpoints
and
watermarks
that
are
included
in
streaming
that
are
not
included
in
batch
that
you
basically
can
take
advantage
of
for
the
purpose
of
orchestration,
okay
and
cool
with
that,
I
did
want
to
finish
it
up.
Thank
you
very
much
for
the
great
questions.