►
Description
Online Tech Talk with Denny Lee, Developer Advocate @ Databricks
One must take a holistic view of the entire data analytics realm when it comes to planning for data science initiatives. Data engineering is a key enabler of data science helping furnish reliable, quality data in a timely fashion. Delta Lake, an open-source storage layer that brings reliability to data lakes can help take your data reliability to the next level.
In this session you will learn about:
* The data science lifecycle
* The importance of data engineering to successful data science
* Key tenets of modern data engineering
* How Delta Lake can help make reliable data ready for analytics
* The ease of adopting Delta Lake for powering your data lake
* How to incorporate Delta Lake within your data infrastructure to enable Data Science
QUICK LINKS:
Download the notebook: https://dbricks.co/dlw-01
https://delta.io
https://mlflow.org
Spark + AI Summit Discount Code: DennySAI020
Register for Spark + AI Summit: https://databricks.com/sparkaisummit/north-america Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Okay,
thanks
very
much
hi
everybody.
My
name
is
Danny
Lee
I'm,
a
developer
advocate
here
at
data
bricks.
We
are
having
a
fun
session
called
getting
data
ready
for
data
science
with
Delta
Lake.
The
session
will
be
slides,
of
course,
but
we'll
definitely
do
demos,
and
on
top
of
that,
we'll
also
go
ahead
and
have
a
live
Q&A.
So
you
have
Q&A.
Please
go
ahead
and
click
on
the
panel
for
Q&A
and
actually
add
your
questions
there.
Just
because
that's
the
best.
A
So
my
apologies
for
those
who
are
not
currently
in
the
Pacific
time
zone
then
good
evening
for
the
you
folks,
as
well
so
or
good
afternoon,
for
that
matter,
so
Who
am
I.
Let's
start
off
with
that.
Well,
as
I
said
I'm
a
developer
advocate
here
at
data
bricks,
I'm
an
I'm,
a
hands-on,
distributed
engineering
systems,
engineer
with
data
science
background
I,
basically
help
build
internet
scale,
infrastructure
I
was
at
Microsoft
part
of
being
and
also
part
of
the
sequel.
Server.
A
I
was
on
the
team
that
helped
build
what
is
now
known
as
or
hdinsight
on
the
project
isotope
and
worked
with
very
large
implementations
of
sequel
server
and
as
your
implementations
and
I've
been
working
with
the
practice
parks
since
0.5.
So
at
least
I
can
have
a
good
sense
of
impostor
syndrome
to
be
able
to
talk
a
little
bit
about
spark
and
data
engineering.
A
So,
before
I
go
into
succeeding
into
the
session
a
little
bad
data
breaks.
We
are
the
company
that
accelerates
innovation
by
unifying
data,
science,
engineering
and
business
all
together
with
our
unified
data
analytics
platform.
So
we,
their
companies
or
the
founders
of
our
company,
were
the
original
creators
of
Apache
spark.
They
we've
also
created
Delta
Lake,
which
is
part
of
the
Linux
Foundation
and
I'll
flow,
and
as
well
Qualis.
These
are
all
really
cool,
open
source
projects.
A
A
These
are
things
that
Apache
spark
incorporates
and
it
can
ping
a
lot
of
different
data
sources,
whether
you're
talking
about
s3
as
your
Hadoop
kafka,
park'
and
other
ecosystem
of
source
ecosystems,
and
so
that's
part
of
the
reason
why
you
see
apache
spark
as
the
de
facto
unified
analytics
engine.
Apologies
for
that.
A
So
let's
talk
about
what
the
theme
of
today's
session
is,
which
is
to
get
data
ready
for
data
science.
Ok,
we're
not
gonna
go
cover
all
of
the
different
aspects
here,
just
because
it's
a
lot
to
put
a
rather
lightly
right.
But
basically
what
you
have
here
is
the
data
science
lifecycle.
It
typically
starts
with
data
preparation
and
there's
a
whole
set
of
tools
that
are
involved
with
that
oK
you've
got
of
course
spark,
but
then
you
also
in
the
ecosystem.
A
And
if
you're
you
happen
to
be
using
a
technology
that
I
didn't
mentioned,
it
was
not
so
much
a
it's
not
meant
as
a
call
out
against
your
technology.
Quite
the
opposite.
It's
just
that
the
ecosystem
for
training
and
all
these
other
technologies
is
quite
large
and
you're.
Gonna
have
to,
and
you're
gonna
have
scaled
that
because,
after
all,
when
you're
training
there's
also
a
horse
of
hyper
parameters
that
you're
gonna
have
to
go
ahead
and
deal
with
right
and
you're
gonna
have
to
tweak
and
change
them,
optimize
them
right.
A
So
the
technologies
that
you
do-
and
that
goes
right
back
to
the
raw
data,
because
afro
once
you
deploy
it,
there's
gonna
be
raw
data
that
goes
into
it.
You're
gonna
have
to
have
governance
and
sure
that
the
data
itself,
actually
it
will
follow
GD
P,
R
or
C
CPA
compliance
rules.
We're
gonna
go
into
that
a
little
bit
during
today's
session,
because
a
important
aspect
of
that
getting
data
ready
for
data
science
is
exactly
that.
The
ability
to
ensure
you
have
GRC
isin
governance
risk
compliance
covered.
A
Those
trends
are
those
changes
because
there's
more
red,
dongles
purchased
or
more
warmer
temperatures
or
whatever
else
invariably
not
just
affect
the
bi
reports
that
you
will
report
out,
but
just
as
important,
if
not
more
important.
In
this
particular
case,
the
data
science
you're
gonna
do
against
it
the
machine
learning
models
that
you're
training.
It's
there's.
Basically,
this
concept
the
model
drift
right
and
it
begins
with
data
drift
right.
The
changes
in
data
over
time-
and
this
is
normal
right-
it's
an
expected
aspect
of
how
things
work,
and
so
this
is
a
continuous
cycle
right.
A
It's
not
like
it
ends
at
one
point:
if
you're
successful
and
you're
able
to
build
excellent
awesome,
machine
learning
models,
deep
learning
laws
or
whatever
maybe
you're
gonna
continue
doing
this
right.
You're
gonna
continue,
training,
you're
gonna
continue,
deploy,
continue
getting
new
data,
so
in
other
words,
not
just
the
data
that
you
have
but
adding
new
additional
data
sources
that
maybe
you
originally
never
processed.
Then
you
join
that
data
and
then
you
have
to
prep
it
again,
and
then
you
have
to
train
again
and
deploy
and
so
forth.
A
So
it's
a
vicious
vicious
cycle
and
an
important
aspect
of
that
also
is
that
now
that
I
just
covered
some
of
the
technologies
they're
very
common
in
that
data
science
lifecycle,
if
I
was
to
ask
for
a
raise
of
hands.
Obviously
we're
on
a
YouTube
live
slash,
zoom
setup,
so
it's
gonna
be
a
little
bit
difficult.
First,
pull
it
off,
but
if
I
was
ask
you
to
raise
your
hand,
how
many
of
you
know
all
of
the
technologies
that
I've
listed
here
and
for
that
matter,
all
the
technologies
that
actually
are
used
in
your
environment?
A
A
Well,
the
first
things
we
often
need
to
do
when
we
operate
the
first
thing
we
often
need
to
do
when
it
comes
to
the
dais
lifecycle
is
to
focus
on
this
concept
of
the
role
of
data
engineering.
Okay
in
engineering
actually
ultimately
becomes
a
very
crucial,
crucial
aspect
of
how
you
actually
deliver
data
science,
and
even
today,
this
far
too
many
teams
that
are
actually
built
up
incorrectly.
So
what
ends
up
happening
is
that
data
science
is
the
pool
du
jour
or
the
word
du
jour,
the
job
Kurdish
or,
and
it
deservingly.
A
So
so
don't
get
me
wrong.
I'm
not
trying
to
insult
anybody
who's
a
data
scientist
here,
I've,
quite
the
opposite,
I
think
it's
an
awesome
job.
The
context
I'm
coming
from
is
more
of
the
fact
that
we
hire
all
these
data
scientists,
but
we
don't
hire
any
data
engineers,
okay,
and
so
this
is
actually
becomes
really
problematic
right,
because
what
ends
up
happening
is
that
the
data
scientists
themselves.
She
will
actually
have
to
do
the
data
engineering
right.
A
It's
not
about
the
fact
that
they
actually
have
to,
because
it's
not
like
the
data
will
magically
appear
in
tabular
format,
cleansed
partitioned
in
parquet,
with
no
data
problems
whatsoever,
I
mean,
if
you
do
that's
great
and
I'm
really
happy
for
you,
but
in
reality
that's
not
what
happens
right?
What
actually
happens
is
that
you've
got
a
lot
of
data.
You
have
to
filter
it
out,
it's
dirty.
You
have
to
aggregate
or
augment
that
data
all
right.
A
A
If
you
want
to
that's
awesome
and
in
fact
I,
we
can
probably
have
another
discussion
about
rigger
software
principle
rigor
with
in
data
science
itself,
but
before
we
can
even
talk
about
that,
we
actually
need
to
talk
about
the
rigor
for
data
engineering
and
so,
for
example,
in
one
of
my
past
lives,
I
was
at
concur
now
purchased
by
ASAP.
I
had
helped
build
up
the
data
science
engineering
team.
What
hemmed
up
happening
there
was
that
we
had.
A
The
company
had
hired
a
lot
of
data
scientists
first
and
so
really
smart
people,
but
they're
all
using
a
disparate
set
of
tools
right
and
they
some
were
standards.
Some
were
not
and
because
they
were
using
disparate
set
of
tools
and
they
were
really
working
in
isolation
from
each
other.
What
resulted
is
that
we
had
different
copies
of
the
same
data
all
over
the
place.
A
So
you
have
these
very
reliable
quality
data
for
quickly,
efficiently
and
securely
right
that
that's
what
you
need.
You
need
reliable
data
lakes.
We're
gonna
talk
about
data
like
shortly,
but
this
concept
is
that
whatever
data
source
that
your
data
scientists
are
going
to
work
with,
it
actually
needs
to
be
reliable,
and
this
ends
up
becoming
a
paramount
of
paramount
importance
in
order
for
you
to
have
successful
data
science
projects.
A
So
when
you
look
at
these,
like
big
data,
architectures
right,
it
starts
typically
and
I've
oversimplified
it
just
because
that
way,
I
don't
have
to
give
you
a
gigantic
Burger
style,
all
the
different
batch
sources
or
all
the
difference,
treatments
or
you've
got
input
sources.
Typically,
it's
actually
really
broken
down
by
latency
there,
either
it's
a
backyard
stream
that
can
be
all
sorts
of
things.
A
A
file
source
clouds
source
database
streaming
could
be
your
your
streaming
of
pinging
of
some
form
of
REST
API,
whatever
else
Kafka
Kinesis
as
their
event
hubs
things
of
that
nature,
all
right.
Ultimately,
you're
gonna
want
to
put
this
stuff
in
a
data
Lake,
okay,
this
concept
of
a
centralized
single
source
of
truth,
but
it's
not
really
a
single
source
of
truth.
A
From
the
stand
point
then,
oh,
it's
completely
other
trustworthy,
at
least
not
yet
right
now,
up
to
this
point,
it's
actually
a
store
right,
basically
I
just
dumped
any
and
all
data
that
we
have
and
this
when
we
brought
in
the
Hadoop
stage
of
the
lifecycle
career
right.
That
was
it.
The
great
thing
about
to
do
was
that
we
could
just
take
all
the
data
store
it
into
our
file
system.
It
automatically
would
replicate
it,
so
I
could
make
sure
it's
reliable.
A
What
I
wouldn't
lose
the
data
I
had
to
do
what's
concept
of
virtual
consistency,
but
nevertheless
the
Dena
was
there
yeah
I'm
good
to
go
problems
that,
oh
sorry
and
then
we
would
do
merge,
schema
and
read
where
basically,
at
the
point
in
time,
I
was
querying,
the
data
I
would
actually
get
the
scheme
out
of
it.
But
guess
what
like
that's
a
lot
of
data
and
we
weren't
sure
what
we're
keeping?
Maybe
we
need
to
filter
that
out?
Maybe
we
need
to
cleanse
it.
A
Maybe
we
need
to
go
ahead
and
actually
remove
it,
because
us
actually
corrupt,
but
nevertheless,
now
we
have
a
data
like
and
so
and
there's
ultimately
ad
to
consumer.
Now
we
talked
about
AI
we're
talking,
wishing
learning
deep
learning
bi
and
from
where
I'm
sitting
from
at
least
from
a
standpoint
of
reliable
data
likes,
there's
actually
no
difference
right.
It
is
very
much
about
this
concept
that
you're
going
to
have
consumers
that
look
at
this
data
that
want
to
understand
how
this
data
works.
A
A
These
are
the
basic
concepts
with
these
input
sources
or
this
your
data
Lake
you've
got
a
store
for
structured
and
semi-structured
data.
Sorry,
you
pull
data
in
from
various
input
sources.
There
there's
a
single
central
location
to
access
all
this
data,
basically
breaking
this
data
silos.
That's
what's
great
about
this,
that
everybody
can
come
in
whether
you
were
all
the
different
teams,
instead
of
actually
working
separately
from
each
other.
We
could
just
all
go
to
this
data
like
and
grab
that
data.
That's
awesome
all
right.
A
It's
open,
accessible
format,
there's
no
vendor
lock-in
and
sequel
machine
learning
against
that
single
source
of
truth.
So
big
data
architecture
like
this,
where
you
have
a
table,
that's
super
promising
super
awesome,
but
there
are
always
issues
with
that.
Of
course,
and
these
are
the
data
reliability
challenges
with
the
data
likes?
Okay,
if
you
have
failed
production
jobs,
it
leaves
data
in
a
corrupt
state
requiring
tedious
recovery.
What
that
means
is
a,
for
example,
are
you're
running
a
spark
job
or
a
Hadoop
job
or
whatever
right,
it's
multi,
multiple
tasks.
A
The
task
fails,
the
entire
job
fails.
Well,
it's
written
something
to
disk
right
and
if
it's
written,
something
the
disk
that
means
at
that
point
in
time
when
somebody
else
goes
ahead
and
reads
from
disk
and
disk
in
this
case
can
be
cloud
storage,
so
I'm
not
trying
to
specify
one
of
the
other
there's
a
bunch
of
remnant
data
that
actually
could
be
read.
That's
actually
not
trustworthy,
it's
not
reliable.
It's
not
something!
A
You
want
to
actually
be
reading
all
right,
so
that's
actually
a
big
problem,
especially
when
the
more
data
you're
working
with
the
more
likely
jobs
will
fail
and
if
the
jobs
will
fail.
That
means
your
data.
Scientists
are
writing
algorithms
that
actually
are
reading
data.
That's
actually
incorrect,
which
is
really
really
bad.
A
Okay,
so
there's
a
lack
of
quality
enforcement,
which
means
there's,
inconsistent
and
unusable
data
that
called
enforcement
could
be
something
as
simple
as
your
schema.
In
other
words,
there's
a
additional
column.
That's
noted
inside
the
source
data
except
the
source
data
had
an
extra
comma,
not
because
the
mainly
because
there
was
a
corruption
or
because
there
was
actually
a
text
with
comma
in
it,
but
because
of
the
comma,
it
was
a
common
living
with
the
file.
This,
your
your
the
codebase,
never
understood
it's.
A
Also,
it's
actually
adding
an
additional
column
to
the
data
there
by
putting
incorrect
data
in
the
wrong
format.
So
when
you
read
it,
you're
actually
reading
the
wrong
thing,
alright
and
the
final
thing,
but
the
most
important
thing
is
lack
of
transactions.
The
this
lack
of
transactions
itself
basically
prevents
you
from
ensuring
that
trustworthy
data
set
the
that
corrupted
data
when
you,
for
example,
have
a
failed
production
job
right.
If
you
actually
had
a
transaction
protecting
it.
What
would
happen
is
that
the
job
fails
anything
even
wrote
to
disk
it.
He
actually
reversed.
A
It
would
remove
the
files
that
were
written
okay,
because
that
that's
actually
what
a
transaction
would
do
right.
This
is
what
traditional
are:
DMS
is
relational
database
systems
actually
able
to
do,
and
this
lack
of
transactions
makes
it
nearly
impossible
to
actually
combine
batch
and
streaming
together
right
because
either
you
have
streams
of
data
running
in
at
the
same
time,
when
apps
at
what
point?
Can
you
trust
the
data
as
it's
coming
in?
A
Well,
if
you
have
a
transaction
that'll
protect
it
then
sure,
then
you've
got
something
to
work
with,
but
if
you
don't
have
a
transaction
protecting
that
data,
you
don't
know
if
the
data
that's
being
written
in
here
actually
is
the
final
state.
And
so
how
can
you
then
combine
that
with
your
batch
data
right?
A
So
now,
let
me
look
at
the
the
pipelines,
okay
and
so
with
those
pipelines
this
guy
she
gets
really
funky
really
fast
right.
So,
for
example,
I'm
gonna
just
happen
to
use
this
particular
design.
You
can
use
something
else,
whether
it's
Kinesis
or
Yzerman
have
hubs
versus
cosmic,
but
I'm.
Just
gonna
use
Kafka,
for
example,
I
said
so.
When
you
have
these
complex
pipelines,
you
you're
increasing
the
engineering
overhead,
so
the
events
go
to
Kafka
Kafka,
because
you're
you're
doing
streaming
so
you're,
gonna
you'll
run
apache
spark
structure.
A
Dreaming,
yes,
cough
it'll,
get
written
to
the
table,
the
table
gets
continuously
written
and
then,
for
example,
there'll
be
a
stream
that
goes
from
the
table
to
for
your
AR
reporting
right.
But
these
spark
jobs
get
exponentially
slower
as
there's
as
they're,
more
and
more
of
these
small
files,
because
that's
what
happens
with
streaming
jobs
right,
you
create
lots
of
really
small
files.
The
more
files
there
are,
the
more
overhead
there
is
for
spark
or
for
that
matter,
any
system
to
go
ahead
and
read
those
files
and
that
overhead
basically
slowed
that
performance.
A
Alright,
then,
let's
add
the
fact
that
okay
well,
how
about
there's
late-arriving
data
right?
In
other
words
the
data
that
was
supposed
to
be
processed
every
five
minutes
or
every
10
minutes,
in
fact,
there's
data
from
two
days
ago
that
for
that
they
forgot
the
process,
I
need
to
go
back
in
and
process
it.
A
So
that
means,
in
that
case,
I,
have
to
potentially
delay
processing
because
of
the
fact
that
there's
the
delay
in
that
data
and
then
okay,
maybe
I,
can
change
that
where
I
have
two
new
stream,
but
then
what
I
do
is
that
I
have
the
left
table
that
we're
table
where
it's
receiving
data
that's
constantly
streamed
in,
but
then
I
have
a
batch
process
that
goes
ahead
and
compact
sit
and
organize
it.
So
that
way
it
takes
care
of
the
late
processing
delays.
A
Okay,
so
now
it's
getting
increasingly
more
difficult,
okay,
but
then
more
times
or
not,
this
latency
doesn't
satisfy
business
needs,
not
to
mention
it's
getting
more
complicated
right.
So
then
what
do
you
typically?
Do?
You
typically
switch
to
oh
well,
then
let
me
do
a
lambda
architecture.
This
I'm
doctor,
where
I'm
streaming
and
I
have
that
so
the
upper
portion
is
where
streaming
and
then
I
have
this
cost
of
the
stream
stream
to
a
unified
view
of
the
streaming
and
batch
data.
So
you're
AI
reporting
to
go
off
that
meanwhile
I'm
batching
data.
A
They
were
wherever
another
streaming
against
coffee.
At
the
exact
same
time,
don't
get
writes
the
data
to
the
table
at
the
bottom
left.
I
have
a
batch
process
that
goes,
they
hadn't
processed
the
data
into
the
table.
It
process
it
in
batch.
That
way,
we
can
take
an
account
of
the
delays,
data
and
the
batch
process
into
the
unified
view.
A
Now
the
you
know,
my
view
has
both
streaming
a
batch
okay
and
by
the
way,
I'm
going
this
fast,
not
because
I'm
trying
to
emphasize
that
you're
supposed
to
listen
this
fast
and
going
this
fast
because
I'm
trying
to
explain
to
you
this
gets
more
and
more
complicated
as
I
go
there
every
slide.
Ok!
So
now
the
problem
with
that
particular
option
is
that
you're
gonna
actually
have
to
have
a
validation
check
that
checks
the
data
that
goes
through
the
upper
streaming
to
the
top
left
streaming.
A
So
that
way
we
can
make
sure
the
batch
data
and
the
streaming
data
are
actually
have
the
same
values.
Otherwise,
your
unified
view
election
result
in
reconciliation
problems,
ok
and
then,
if
you
need
to
fix
mistakes
well,
the
validation
takes
care
of
or
the
streaming
data,
but
you
need
to
go
ahead
and
reproach
the
data
that
gets
written
to
the
batch
on
the
bottom
left
and
so
forth
and
so
forth.
Ok,
so
this
becomes
really
really
complicated
and
then
oh,
not
to
mention
updates,
emerges
like
again
there's.
This
is
really
complex
thing
to
do.
A
Ok,
now,
I'll
stop
and
breathe.
The
question
then
is:
can
this
be
simplified
because
in
the
end
I
know
many
people
who
saying
hey?
You
know
I'm
a
data
scientist
I
want
to
go
ahead
and
talk
about
algorithms,
but
this
is
the
reason
why
we
did
talk
about
getting
data
ready
for
data
science.
Well,
don't
look
at
mo
flow
because
we're
trying
to
get
across
the
point
that
there's
all
these
data
engineering
pipeline
steps
that
you
have
to
do
before
you
get
to
do.
A
The
fun
stuff
will
actually
have
sessions
later
on
which
dive
into
all
the
different
machine
learning
algorithms
are
available
in
SPARC,
M
11,
so
definitely
chime
in
some
comments.
So
we
can
know
how
to
customize
these
Tech
Talks.
So
we
can
talk
about
stuff
that
you
guys
are
interested
in,
but
or
which
albums
you
like
us
to
tackle.
First,
like
XG
boost
or
PI
torch
or
whatever
else.
A
But
the
point
is
that
before
I
can
even
go
to
that
area,
I
need
to
make
sure
I
can
handle
the
data
in
production
right
because
in
production
this
is
the
scenario
we
have
and,
of
course
the
question
I'm
gonna
ask
is:
can
dismiss
be
simplified,
even
if
I
were
had
a
dedicated
data,
they
designs
engineering
team
that
was
focusing
on
building
these
pipelines.
This
is
a
lot
to
maintain
right.
So
how
can
I
simplify
this,
and
so
the
key
thing
to
simplify
this
is
that
we
want
to
introduce
acid
transactions
back
into
the
fold.
A
Okay
and
so
acid.
Basically,
transactions
were
proper
lives
because
of
relational
databases.
Okay,
they.
What
they
did
is
that
they
introduced
its
concepts
of
atomicity
consistency,
isolation
and
durability.
We'll
have
we
have
actually
separate
videos.
I
got
don't
go
into
the
details
of
what
those
mean,
but
the
context
basically,
is
that
if
you
write
something
to
disk
cloud
or
on
Prem,
it
means
it's
got
written.
If
there's
an
error,
it
will
fail
out
any
other
clients
that
are
concurrently.
A
Trying
to
read
from
it
can
reliably
know
that
the
data
is
either
there
and
for
them
to
read
it
or
it
isn't
and
will
get
that
information
right
from
the
get-go.
If
you're
concurrent
writes
to
the
system,
we
can
actually
those
with
asset
transactions,
have
the
ability
to
coordinate
the
fact
that
there
are
concurrent
transactions
and
prevent
corruptions
overwrites
and
any
other
issues
once
you
have
that
you
can
actually
handle
back
streaming
updates
deletes
easily.
A
For
example,
if
I
go
back
to
my
days
at
sequel,
server
right
I
could
handle
streaming
and
I
could
handle
batch
at
the
same
time.
In
my
one
single
database,
because
I
had
asset
transactions,
we're
allowed
me
to
do
superfast
OLTP,
where
I
was
like
you
know.
Financial
Services
was
writing
data
to
the
DES
super
fast.
It
was
no
problem.
I
could
handle
because
that
asked
the
transactions.
At
the
same
time,
I
was
going
ahead
and
doing
batch
processing
of
aggregate
data
or
of
other
sources
data
to
that
I'll
eventually
join
with
the
financial
data.
A
So
again,
I
could
go
do
that.
The
problem,
of
course,
is
that
I
couldn't
scale
it
at
least
not
to
Big
Data
internet
scales,
type
sizes
right
if
I
could
get
a
database
to
do
that,
I'd
still
be
a
sequel
server,
and
this
is
not
a
knock
of
sequencer.
It's
a
great
product.
It's
just
simply
saying
that
when
I
have
to
work
on
big
data
problems
where
I
have
to
handle
streaming
and
I
have
to
handle
batch
and
machine
learning
data
signs
all
these
other
things
all
together.
A
This
is
where
why
we
started
looking
at
things
like
Apache
spark
right
and
then
in
order
for
a
data
scientist
for
you
to
even
do
bi
reporting
reliably.
We
needed
to
reintroduce
as
the
transactions
back
into
big
data
systems,
and
that's
actually
why
Delta
Lakes
their
Delta
Lake
is
still.
It
allows
us
to
bring
acid
transactions
back
to
Apache
spark
and
Big
B
the
workloads.
So,
ultimately,
you
get
consistent
reads
and
you
get
roll
backs,
so
in
other
words,
if
there's
an
error,
I
can
roll
back
on
that
which
is
pretty
sweet.
A
So
don't
do
like,
as
we
say
here
right,
it
is
a
open
format
based
on
park'
which
transactions
utilizing
the
Apache
spark
api's.
Okay,
it
allows
us
to
address
the
data
practitioners,
dreams
process,
data
continuously
and
incrementally,
as
new
data
arrives
in
an
efficient
way
without
choosing
between
bacter
streaming
right.
It's
the
same
difference
to
us
right,
so
you
so
that
previous
screen
that
you
had
you
simplified
it.
So
it
just
becomes
this
one
line
for
cigar
that
you're
looking
at,
which
is
sort
of
nice
all
right.
A
A
Included
with
this
concept
is,
is
a
concept
that
called
Delta
architecture,
so
in
other
words,
when
you
design
your
systems,
you
what
you
want
is
you
want
to
make
sure
that
there
actually
is
a
continuous
flow
model
that
allows
you
to
go
ahead
and
unify
the
batch
of
stream.
So
how
you
do
that
is
that
you
start
with
your
source
and
you
build
in
essence
what
we
deem
as
a
broadens
table
right.
This
is
Rajesh,
and
this
is
similar
to
how
you
traditionally
use
your
original
data
data
lake.
A
A
So
what
ends
up
happening
here
is
that
you
have
that.
Okay,
then
you
have
silver
all
right.
The
silver
basically
is
filtered
cleaning,
augmented
data,
okay
and
then
so
what
that
means
is
that
when
you
process
the
data,
excuse
me
when
you
process
the
data
you
can
filter
out
for
sake
of
argument,
all
the
logins
that
you
don't
need,
or
whatever
else
that
you
want
to
do.
Okay,
you
can
clean
the
data
and
augment
it
with
additional
information
from
other
sources,
and
you
can
combine
that
together.
A
Okay
and
then
there's
gold,
okay,
so
these
this
is
where
you
can
then
run
your
business
level
aggregates
this,
where
you
can
do
your
AI,
your
streaming,
your
assuming
your
AI,
your
data
science,
your
machine
learning,
deep
learning
things
like
that.
This
is
where
then
often
you
will
do
your
merges
and
your
overrides
okay,
this
isn't
to
say
you
can't
do
inserts
and
gold
or
you
can't
do
deletes
in
bronze.
It's
just
a
simple
saying
that
from
a
particular
dish,
'''l
data
flow
perspective.
A
Basically
our
data
pipeline
perspective.
It
is
more
or
less
like
this.
You
know
you
inserts
and
bronze,
deletes
and
silver
merge
and
overwrites
and
gold.
Alright
and,
like
I
said
it
alternately
goes
ahead
and
allows
you
to
do
your
streaming,
Analects
and
AI
reporting.
It
also
allows
you,
then,
to
reprocess
the
data
as
if
there's
corruptions
issues
or
with
the
data
or
changes
in
business
requirements
that
require
the
reprocess,
because
you
have
the
original
raw
ingestion.
You
can
go
back
to
the
beginning
and
reprocess
it.
A
So
you
have
the
silver
or
you
have
the
silver
and
gold
date
again.
So
this
is
the
concept
of
a
delta
lake
architecture.
Okay
and
basically,
each
step
that
you
see
here
is
a
progression
of
how
you
can
make
things
cleaner.
Okay,
how
you
gonna
the
data
quality
basically
goes
up
all
right.
This
architecture
then
introduces
the
end.
End
pipeline
SLA,
reduces
pipeline
maintenance
burden
and
eliminates
the
lambda
architecture
for
minute.
A
Latency
use
cases
okay,
and
so
we
make
this
architecture
real,
because
you've
got
the
open
up,
knives
file,
source
asset
transactions,
scaleable
metadata
handling
for
high
throughput,
insert,
updates
and
deletes
okay
data
versioning
and
schema
right
options
and
force
data
quality.
Now
I
skim
through
this
really
quickly,
because
I'm
gonna
show
you
how
that
all
this
works.
So
so
you
know
what
I'm
actually
to
stop
presenting
now
and
I'm.
Actually,
gonna
go
right
to
the
demo,
because
I'll
end
with
the
how
do
I
use
Delta?
A
Okay,
so
give
me
a
second
all
right
perfect,
so
so
here's
a
notebook
by
the
way
and
by
the
way
this
notebook
will
be
available
for
you
to
go
ahead
and
download
and
use
yourself.
Okay.
So
this
is,
you
can
use
it
on
dan
bricks,
Community
Edition.
So
don't
you
don't
need
to
purchase
data
bricks?
It's
a
it
is
a
date
Brooks
notebook,
but
you
can
use
on
the
free,
DBC
I
even
call
that
out.
A
A
Okay.
So
let's
think
here
now
in
order
for
us
to
do
this,
what
I
first
did
his
eyes
easily
converted
this
from
parquet
to
Delta
Lake
format,
so
what
I
did
is
I
basically,
in
this
particular
case,
simply
ran
a
create
table
by
statement
so
I
basically
converted.
What
will
you
be
part
k
into
the
into
delta
lake
there's?
Actually
a
delta
like
one
step,
migration
API
as
well.
That
allows
you
to
do
that,
but
I
just
happen
to
do
it
this
way.
A
Nevertheless,
now
I
have
a
Delta
table
and
this
Delta
table
basically
you'll
notice.
This
here,
Delta
DB
of
S,
is
the
neighbors
file
system.
Little
much
T,
Delta
I've
called
it.
The
partition,
columns,
things
of
that
nature
and
you're
good
to
go
okay,
so,
in
other
words,
basically
lets
you
know
the
schema
all
right.
So
I've
just
done
this
real
quick
set
up
now.
What
I
will
do
is
I
will
now
run
it
live
from
this
point
onwards.
A
Okay,
so,
if
I
look
at
the
file
system,
what
you'll
notice,
basically
as
its
as
its
showing
its
stuff
okay,
is,
is
these
are
all
the
different
part,
K
files,
and
so
this
looks
pretty
similar
to
just
pretty
much
like
any
other
Parkay
see,
it
said
Edina
that
you
actually
are
used
to.
Okay,
you
have
a
table
low
by
state
Delta.
It
has
a
bunch
of
park'
files,
yeah
except
the
one
big
difference
is
this
Delta
log
folder,
okay
and
the
Delta
log,
basically
is
exactly
what
it
sort
of
implies.
A
It's
a
transaction
log,
every
single
transaction.
Every
single
change
modification
to
the
data
is
recorded,
as
recorded
in
this
format
and
you'll
see
the
space
at
least
JSON
files,
okay,
and
so
the
underscore
delta
underscore
log
contains
those
JSON
file.
So
you
only
see
one
of
them
the
zero
file,
mainly
because
that's
the
creation
of
the
table,
alright,
so
let's
go
ahead
and
actually
run
a
streaming
snare
where
I'm
gonna
go
ahead
and
do
a
unified
batch
and
streaming
source
and
sync.
A
So
what
you
notice
here
is
that
I've,
just
Delta
like
solar
path
but
I
initially
had
a
setup.
Alright,
I'm
gonna
do
a
read
stream,
ie
I'm
gonna
do
a
run
a
streaming
job
and
I'm
gonna
read
whatever
data
is
coming
out
from
that
Delta
Lake
silver
path.
That's
this
path
here
by
the
way
that
I
had
called
out
before
alright.
So
now,
I'm
gonna
go
read
that
stream
in
this
case
I'm
just
gonna
output,
the
data
by
stage,
so
my
apologies
for
them
for
you,
non-americans,
I.
A
The
data
set
I
have
is
currently
only
for
the
United
States.
Ok.
So
the
the
the
explanation
of
this
particular
data
set
by
the
way,
is
that
these
are
loans.
Okay,
so
these
are
loans
by
state
restream,
ok,
so
in
other
words
the
the
number
loans
for
each
state
that
we
have
okay.
So
let's,
let's
let
that
sucker
run
you.
A
A
So
as
I
wait
for
the
initial
streaming
jobs
to
kick
in,
basically,
the
quick
call-out
is
this
world.
This
recording
of
the
zoom
webinar
be
available
as
well.
Yes,
this
just
as
a
quick
note,
this
webinar,
this
Tech
Talk,
is
going
to
be
available
for
you
to
go
ahead
and
work
with
no-profit,
all
okay.
So
all
right!
So
for
some
reason
this
one
is
running
a
little
slow
today,
so
I'm
just
gonna
use
this
one,
all
right
cool.
So
right
now,
poor,
Iowa's
null.
Okay,
there's
no
values
for
Iowa.
A
So
let's
go
ahead
and
run
that
right
now.
This
is
the
great
thing
about
live
demos,
so
I'm
just
gonna
insert
data
into
it.
Now
note
the
fact
that
I'm
inserting
data
into
low
by
state
Delta
what
that
means
is
I'm
inserting
data
into
the
batch
table,
not
the
read
stream,
not
the
right
stream,
but
The
Bachelor,
but
it
ultimately
going
to
the
same
file
system.
A
But
it's
read
a
balance
eart
as
opposed
to
doing
a
right
stream,
and
what
you
notice
is
Iowa
is
slowly
steadily
building
up
more
data,
as
you
can
see
that
it's
progressing
from
450
900,
because
I'm
just
doing
an
old-school
insert
here
right
now,
but
the
context
and
by
the
way
in
production,
you
don't
want
to
really
do
it
this
way,
it's
just
I'm
just
doing
it.
So
that
way
you
can
see
I'm
doing
a
batch
right
to
a
read
stream.
Concurrently.
That's
that's
actually
all
I'm
trying
to
do
here.
A
Okay,
so
that's
just
a
quick
call
out
here.
Okay,
so
all
right,
so
I
believe
most
of
the
data
has
been
processed
already,
the
six
loops
there
you
go
and
for
that
matter,
I
will
go
ahead
and
take
a
look
at
the
data
here.
So
if
I
go
ahead
and
look
at
the
log,
oh
sorry,
I
forgot
to
stop.
There
go
you'll
notice
that
now
I
have
multiple
files
that
we
have
the
original
zero,
but
we
have
one
two
three
right
up
to
six
all
here.
A
These
are
the
six
inserts
associated
with
what
use
of
the
six
insert
statements
that
you
have
here.
Okay
and
we'll
show
you
actually
how
that
looks
in
shortly.
I
just
want
to
show
you
the
FASTA
first
and
then
I
can
go
back
to
the
low
and
wide
State
Delta
right,
so,
in
other
words,
I
had
the
reach
theme,
which
was
showing
it
all
here.
Okay,
but
I
can
also
go
ahead
and
see
it
back
in
the
batch
and
sure
enough
this
same
value.
A
It
doesn't
seem
that
interesting,
but
the
the
important
aspect
here
is
that
now
I
can
read
and
right
whether
it's
streaming
or
batch
to
the
exact
same
file
system
and
I,
don't
and
I
can
do
so
in
a
reliable
fashion,
and
that's
in
itself
it's
a
very
powerful
thing
that
I
can
do.
Okay,
and
so
what's
also
great
about
Delta.
A
Lake
is
that
now,
if
you
are
thinking
yourself
like
I,
am
somebody
who
is
used
to
using
sequel
syntax,
because
you
forsake
are
going
to
come
from
a
sequel
background,
or
for
that
matter
you
just
like
the
syntax,
which
is
understandable.
What's
great
is
that
you
can
now
run
with
Delta
Lake
with
spark,
you
can
actually
run
delete,
so
you
can
run,
updates
and
merges
okay.
So,
for
example,
I
just
created
the
same
version
in
the
same
table
like
something
I'm
craving
a
PQ
one
ie
a
park,
a
version
of
that
so
for
sake,
argument.
A
If
I
was
a
run,
a
delete,
okay,
it'll
fail
right.
The
delete
only
works
when
it
because,
when
supported
by
Delta,
so
not
a
big
deal,
it's
because
I
realized
wait.
No
I
wasn't
supposed
to
put
the
data
into
Iowa.
I
was
supposed
to
put
that
date
into
Washington,
so
no
problem
I
can
go
ahead
and
run
the
delete
right
there,
okay
and
so
boom.
Now
that
deletes
done,
I
can
just
go
run.
A
So
now
I
want
to
run
an
update
statement.
So
this
is
like
our
we
are.
Some
command
is
basically
doing
the
update.
Sorry,
the
delete.
The
eighth
command
is
doing
this
update
where
I'm
going
to
try
to
update
it,
because
it
was
those
2,700
value
that
I
saw
there.
I
was
actually
supposed
to
give
it
to
Washington,
not
the
Iowa,
okay,
so
of
course
it
fails
with
Park
a
so
the
again.
These
failures
are
actually
designed
to
happen.
A
So
by
the
way,
when
you
get
the
notebook
and
you
upload
it
and
run
it,
you'll
see
three
errors
right
away.
That's
where
the
three
are
two
of
the
three
errors
that
you
just
saw.
Okay,
you'll
see
the
third
error
pretty
soon,
and
so
then
we
have
the
update
statement
again,
seeing
an
idea
with
Delta
Lake
I
can
go
ahead
and
do
that
right
away,
not
a
big
deal,
we're
good
to
go
okay
and
then
again,
let's
review
the
data.
A
Sure
enough
here
you
go
okay,
so
Washington
State
has
2,700
okay
and
then
same
idea
if
I
want
to
do
a
merge
statement.
So
the
reason
I'm
just
going
to
run
that
and
obviously
I'm
just
going
to
run
that
in
in
Delta
at
this
point,
because
I
think
I've
nailed
that
point
across
I
want
to
update
Iowa
to
ten
I
want
to
change.
A
California
2500
I
want
to
make
Oregon
null
for
some
reason,
so
nothing
against
Oregon
I'm
actually
born
there,
so
nothing
against
them,
but
normally
a
merge
statement
is
actually
the
seven
steps
identify
new
rows
that
to
be
inserted
the
same
row
of
new
rows
that
have
to
be
replaced
or
updated.
We
rows
that
are
not
impact
by
an
inserted
update,
create
a
new
tape
that
temp
table
for
where
all
three
inserts
have
to
occur,
delete
the
original
table.
We
need
the
temp
table
back
to
original
table
and
drop
the
temp.
A
Okay,
that's
what
emerge
is
usually
okay.
If
you
were
to
run
this
yourself
in
a
standard
distribution-
or
you
can
just
run
this
one
merge
statement
yeah
in
Delta
like
so
again,
we
try
to
make
things
a
little
bit
simpler.
So
again
we
meet
California
2500,
we
nullified
Oregon
and
sure
enough.
Oregon's.
B
A
California
2500
Iowa
stead,
okay,
so
this
is
really
cool
stuff
all
right,
but
I
also
want
to
evolve
the
schema
so,
for
example,
the
counts
that
I
have
here
right
now.
That's
just
simply
the
count
right.
That's
all
this
is.
It
simply
tells
me
the
count
alone
I
over
time.
The
real
business
value
should
be
actually
the
dollar
amount,
so
I've
generated
this
by
the
way.
Okay,
this
is
a
obviously
fake
data.
A
Okay,
so
I've
got
the
state
I'm
going
to
count,
but
now
I've
got
the
amount,
and
so
now
I
want
to
just
go
ahead
and
say
yeah.
Let
me
include
that
amount
information,
because
that's
super
important.
Well,
that's
the
fourth
over
here,
because
we're
actually
want
to
enforce
the
schema.
So
this
is
actually
a
desired
error,
in
other
words,
as
you're
putting
data
into
your
data
Lake,
you
know
with
Delta
Lake.
A
You
basically
ensure
that
you're
not
gonna
overwrite,
the
data
that
you
have
there
or
you're,
not
gonna,
go
ahead
and
put
potentially
corrupt
data,
because
maybe
it
was
human
error.
We
weren't
supposed
to
put
the
some
amount
data
inside
that
we
just
just
our
Burt
was
just
it
was
by
accident.
Okay.
Well,
that's
fine!
So
with
schema
enforcement,
we're
actually
preventing
that
from
happening,
but
if
I
was
able
to
add
this
option,
option
merge
scheme
is
true.
Well,
what's
great
about
this
particular
options
and
then
I
can
do
exactly
that.
A
I
am
now
saying
to
myself.
No
I
want
to
evolve
the
schema,
because
it's
what
we
have
there
is
correct.
In
fact.
So
let
me
go
do
that
right
now
and
sure
enough
now,
I
can
query
this
exact
same
data
from
the
exact
same
table
below
and
by
State
Delta
and
in
this
case,
I'm
looking
at
by
amount,
as
opposed
to
by
counts,
all
right
and
then
we're
gonna
finish
this
off
with
travel
time,
travel
and
ml
flow.
A
Okay,
so
I
wanted
to
describe
the
history
of
my
lone
state,
doesn't
remember
how
I
had
shown
all
the
files?
Well,
here's
a
better
view
of
that
stuff
right.
So
every
single
change
I
did
the
initial
creation
the
six
inserts
that
we
did
the
delete,
the
update,
the
merge
and
the
appending
that
we
just
did
they're
all
recorded
right
here.
So
now
we
know
what
transactions
occurred,
but
what's
really
cool
about
this.
Is
that
because
I
have
those
transactions
and
I
also
have
the
original
files?
A
If
I
was
to
simply
go
loan
by
state
delta
version?
As
of
0,
which
is
the
initial?
What
I
originally
create
I'm
literally
going
back
in
time,
and
this
is
what
my
table
looked
like
initially,
ok
would
where
I
always
know,
Washington
was
336,
not
2700.
Oregon
was
still
182,
not
null
California
had
a
different
value
of
2006.
Ok
versus
the
ninth
version
of
the
table,
where
basically
I
was
able
to
go
ahead
and
include
everything,
but
the
inclusion
of
the
amount
and
sure
enough.
A
Here's
what
the
table
looks
like
now,
so
I
was
10
Oregon's
null
Washington's
2700
California's
25,
so
all
of
that
is
actually
retained.
So
you
can
go
back
in
time
and
we
recall
what
version
of
the
data
you
had
is
assigned
to
that
transaction
and
actually
that's
pretty
powerful,
as
you
think,
as
you
can
guess.
Ok!
Well,
why
is
that
power
from
a
data
science
perspective?
A
So
here's
what
I'm
doing
I'm
just
simply
downloading
some
US
census
data,
because
I'm
building
a
really
boring
model:
ok,
a
really
boring
model
where
I'm
actually
combining
the
the
Delta
data,
the
lone
state
data
for
each
and
every
single
different
version
of
that
data
by
the
the
population
of
that
state.
Okay,
and
so
that's
what
this
particular
set
of
queries
are
for,
but
now
you
what
I'm
doing
is
I'm
doing
for
the
data
version,
0,
6
and
9
0
is
the
original
version.
A
The
real
version
of
the
data
6
is
the
version
of
data
that
got
modified
where
I
went
ahead
and
nullified
Iowa
and
increased
Washington
2700
and
then
the
ninth
value
of
it.
It
was
basically
one
where
I
really
messed
it
up.
We're
also
nullified
Oregon
added
10
to
Iowa
and
also
messed
up
messed
around
with
California
from
the
fun
of
it.
Okay,.
B
A
What
what
I'm
doing
here
is
actually
just
running
a
standard
I'm,
actually
running,
yellow
brick,
so
I
can
do
some
residual,
so
you
can
just
visualize
it,
but
I'm,
actually,
just
simply
doing
assuming
I
predict
a
loan
count.
That's
all
I'm
doing
so,
I'm
doing
a
standard,
linear
regression
I'm
using
the
exact
same
metrics
for
each
and
every
one.
It's
basically
like
if
I
was
to
deploy
the
model,
so
I'm
just
gonna
run
that
right
now.
Oh
sorry,
my
goodness
simply
run
the
code.
A
Okay
and
then
now
I'm,
actually
gonna
run
the
code,
we're
actually
gonna
run
the
the
predict
loan
counts
for
the
three
different
versions,
where
0
the
original
6
version
and
that's
version.
The
reason
I'm
calling
this
out
is
that
I'm
sort
of
faking
this
concept
of
data
drift.
You
know
and,
as
you
are
running
your
machine
learning
models,
the
data
can
change
over
time.
Now.
Maybe
it's
because
you
screwed
up
on
the
model
or
I
screwed
up
the
model
for
that
matter,
or
maybe
it's
because
the
data
actually
did
change.
A
A
That's
all
this
is
so
the
only
thing
I
really
did
to
make
this
somewhat
interesting
is
that
I
added
these
statements
with
ml
flow
start
run
and
I
also
included
the
data
version
and
the
I
log,
the
metric
and
I'll
log,
the
model
itself.
Okay,
I
also
happen
to
log
the
the
yellow-brick
residuals
view
as
well,
just
because
it's
sort
of
next
okay,
so
that's
it.
That's
all
I've
really
done
all
right.
A
So
what
what
happens
that
if
I
go
ahead
and
click
on
the
runs
folder
here,
all
right,
all
three
runs
are
here
and
yes,
you
can
progress.
Okay,
these
are
the
older
versions.
I
did
last
night,
but
this
is
the
ones
that
I
did
live.
As
you
can
tell
here
version
zero.
We
have
an
ARMA
C
of
65-
okay,
not
great,
but
not
bad,
but
as
you
can
tell
using
the
exact
same
hyper
parameters
I'm
again,
this
is
linear
aggression,
so
not
really
that
interesting,
but
still
it
gets
worse
with
version
six.
A
This
is
the
one
where
I
nullified,
assuming
I
went
ahead
and
knocked
up
Washington
State's,
okay,
the
our
mystique
got
worse
and
now
I
went
ahead
and
changed
California
no
added
data
to
Iowa,
but
should
have
been
added
nullified,
Oregon,
it's
as
you
can
tell
it's
getting
worse,
worse
and
worse
right,
and
so
let
me
go
ahead
and
click
on
mo
flow
and
open
up
the
the
window
here,
okay
and
by
the
way,
even
though
I'm
running
this
all
on
Dana
bricks.
This
is
actually
all
open
source
technologies.
A
I'm
talking
about
I
just
happen
to
be
showing
this
on
data
breaks,
but
this
is
all
the
source
stuff.
Okay,
so
you
know
if
I
want
to
do
a
comparison.
I
can
just
compare
the
three.
Those
three
different
runs
version:
nine,
six
and
zero,
and
you
could
again,
you
can
tell
that.
Basically,
the
rmse
is
much
better
when
I
go
with
go
here
right
with
V
0
versus
V
6
3
9,
you
can
also
visualize
it
here.
A
Right
right
here
are
maseeh
to
see
the
different
runs,
but
what
I
also
like
about
when
I'm
working
with
mo
flow?
Is
that
actually,
for
example,
I'm
looking
at
this
one
version,
9
right
so
I
can
tell
the
RSC
is
horrible
right,
but
also
I've
also
went
ahead
and,
like
I
said,
I
went
ahead
and
did
yellow
break
I,
look
to
the
residuals
and
right
away.
I
can
tell
okay,
but
a
train
makes
it
lame
means.
A
So
we
have
the
ml
model
right
here.
It
tells
you
exactly
which
version
that
you're,
using
what
python
version
smart
ml,
the
conga
that
goes
with
it.
So
these
are
the
dependencies
so
ml
flow.
This
version
of
Pi
sparks
this
version
of
Python
and
for
that
matter
all
the
metadata
and
the
stages
for
your
animal
model
are
now
stored
directly
in
ml
flow.
So
I
can
go
back
and
redeploy
it
or
go
back
and
change
it.
So
because
there's
always
invariably
the
issue
that
maybe
the
data
that
came
in.
A
Actually,
there
were
problems,
but
then
later
on
they
got
cleaned
up
and
fixed,
so
you
can
go
back
and
use
this
mod.
Ok,
so,
as
you
can
tell
there's
a
lot
of
power
by
combining
these
two
open
source
technologies
together
to
help
you
go
ahead
and
get
yourself
theta
ready
for
data
science.
Ok,
so
I'm
gonna
finish
up
my
presentation
here
and
then
I'll
still
leave
a
little
bit
of
time
for
questions.
A
So
hey
how
do
I
use
Delta
Lake,
ok,
so
to
use
it
obviously
I'm
gonna,
we're
gonna,
give
you
this
notebook
than
where
you're
gonna
be
a
download
stuff,
that's
great!
But
if
you
want
to
run
this
yourself
and
you
don't
directly
write
from
terminal
or
from
a
Jupiter
notebook,
that's
fine.
Add
the
spark
package
packages.
Io
Delta,
Delta,
Corps
I'm,
using
Scala
2001
to
the
current
latest
version,
is
Delta
like
0.5
0,
so
there
you
go
and
then
I'll
keep
on
updating
the
slide
deck.
A
A
As
of
this
recording
is
0.5
well
you're
right
instead
of
using
park',
in
other
words,
data
frame,
dot,
write
down,
perform
at
Parque,
just
simply
change
it
to
data
frame
down,
write
that
format,
dot
them,
and
that's
it
now,
just
like
that,
you're
using
Delta
like
ok-
and
we
already
talked
about
the
update
merges.
We
already
talked
about
the
time-travel
and
yes,
because
I
have
that
time-travel
capability
for
10
a
bunch
of
those
steps.
Six
seven,
eight
nine
were
wrong:
I
can
now
roll
back
that's.
A
So
this
is
small
but
fundamental
component
of
that
data
science,
lifecycle
that
allows
you
to
go
ahead
and
have
reliable
data
that
by
basically
adding
Delta
like
a
normal
flow
that
you
can
ensure
that
no
matter
what
tools
you're
using
to
do
your
analyses,
no
matter
what
tools
you're
using
to
do.
Your
processing,
you're,
actually
gonna,
have
a
reliable
data
link
and
you
can
track
both
the
data
version
itself
and
the
model
versions
that
you're
working
with
okay.
So
if
you
want
to
build
your
own
Delta
Lake,
please
go
to
H
TSS
Delta
o.
A
A
A
We're
gonna
send
you
this
email
out
with
all
this
information
anyways,
but
the
just
which
includes
these
slides
by
the
way,
but
yeah
like
it's
right
there
for
you
to
download
and
play
try
out.
Do
you
want
to
call
a
couple
things
before
I
answer?
Questions?
Hey
you
got
more
questions
you
want
to
dive
into
deeper,
want
to
join
us
a
spark
in
the
eye,
someone,
okay,
a
spark,
an
area,
some
it's
June
22nd
to
25th
of
San
Francisco.
A
We
actually
two
training
days
now,
just
because
of
the
popularity
of
them
it's
organized
by
data
bricks,
but
you
know
listen
to
the
open
source
technology
so
come
on
down.
It's
in
San
Francisco
use
the
code
denne
my
name
Denny
SS
s,
AI
0,
20,
okay,
that's
the
code!
You
get
20%
off
so
go
ahead
and
give
that
a
shout
would
love.
A
You
would
love
to
see
you
down
there
love
to
answer
your
questions
both
online
and
also
physics
to
Mase,
okay,
and
also,
if
you
have
questions,
come
to
Delta,
dial
or
ml
flow
org,
we
actually
have
a
ton
of
questions
where
we
can
actually
we're
deactive
under
slack
channels.
We
reactive
on
the
distribution
list
so
definitely
go
ahead
and
and
join
us
there.
Okay,
so
saying
that
I'm
gonna
leave
you
at
this
I'm
going
to
leave
a
few
minutes
for
questions
and
we'll
go
from
there.
Okay,.
B
A
Okay,
I'm
gonna
go
through
looking
the
questions
here,
so
I
apologize,
I'm
just
trying
to
scroll
through
them
right
now.
There
is
a
question
say,
oh
by
the
way,
as
a
small
joke
here,
the
when
I
did
say
morning,
everyone
and
it's
a
good
morning
for
Cisco-
that's
from
Karen
I'm,
actually
Seattle
bass.
So
just
as
a
quick
collapse,
there's
a
question
about.
Is
there
difference
between
data
books
Delta,
as
in
the
database
version
versus
versus
the
open
source,
one,
the
the
eight?
A
There
is
no
difference
between
the
api's
v
version,
1.0
most
of
the
API
changes.
Differences
are
actually
are
all
resolved
now,
but
I
do
want
to
call
out
that
you
know
since
we're
a
pre
100
project,
we're
still
trying
to
get
make
sure
that
all
of
the
functionality
that
exists
in
data
books,
data
bricks,
Delta
from
a
reliability
perspective,
are
included
the
in
the
open
source
version.
Okay,
oh
by
the
way,
if
you're
asking
questions,
please
don't
use
the
Q&A
panel.
A
Please
put
it
on
to
the
the
chat
panel
for
some
reason
by
QA
and
paddles,
not
opening
right
now
and
so
by
version
1.0.
If
not
earlier
will
actually
mean
sure
that
every
single
feature
that
is
that
exists
in
data
box
Delta,
is
also
available
in
open
source
Delta
like
now.
What
is
the
difference,
then
the
difference
right
now,
primarily,
for
example,
some
of
the
update
delete
statements
that
you
saw,
those
are
not
gonna,
be
ready.
A
You
can
run
updates,
you
can
run
delete,
but
you
can
only
run
them
from
the
Python
and
Scala
API
is
in
the
open
source,
one
we're
actually
waiting
for
a
spark
3.0
to
release
with
data
source
v2
that
allow
us
to
actually
make
some
of
the
changes
that
allow
us
to
go
ahead
and
put
the
sequel
statements
in
as
well.
So
that
way,
you
can
use
spark
sequel
for
deletes
and
updates.
Just
like
I've
shown
you
on
the
screen
here,
as
opposed
to
using
the
API
but
Sophie.
A
But
if
you
want
to
use
the
API,
obviously
you
can
use
the
API
right
now
to
do
the
leaks,
okay
and
updates
emerges,
and
so
the
the
main
difference
post
1.0
is
that
we'll
all
of
the
api's
will
stay
exactly
the
same.
It's
just
that
data
bricks
itself
is
a
managed
service
so
because
we're
a
managed
service,
we're
gonna
go
ahead
and
run
Delta
Lake
for
you
we're
gonna.
Imagine
we're
gonna
do
performance,
it
has
those
things
of
that
nature,
but
it
is
a
open
source
project.
A
A
So
there's
a
question:
can
I
change
the
Delta
Store
to
blob
Lake
store
dB,
not
two
dB,
at
least
not
certainly
not
yet,
but
you
can
certainly
make
the
changes
for
Delta
Lake
to
be
pointing
to
whatever
a
file
store
that
you
want.
So
in
other
words,
you
want
to
run
this
on
the
Hadoop
file
system
by
all
means.
If
you
want
to
run
this
on
the.
A
Data
like
Gen,
2
or
s3,
or
Google
Cloud
Storage.
These
are
fine.
There's
actually
nothing
stopping
you
from
doing
that,
because
if
you
recall
in
the
end,
the
the
primary
thing
we
do
is
we
add
an
additional
folder
underscore
delta,
underscore
log
where
we
put
the
transaction
log
into
it,
that
that's
the
primary
thing
difference
that
we
make
right
outside
of
that,
it's
the
still
technically
the
same
park'
files
that
you
would
see
in
a
park
a
table
on
that
file
system
right.
So
we're
still
doing
the
same
thing
there.
A
Certainly
some
optimizations
done
underneath
the
covers,
but
for
all
intents
of
still
parquet
Files,
the
the
most
notable
change
is
that
you,
when
you're
trying
to
look
at
the
data,
there's
a
lot
more
data
in
there
right.
So,
for
example,
when
I
do
a
delete
technically
I'm
not
doing
a
delete
right,
technically
I'm,
building
up
a
tombstone,
because
what
I've
done
is
I've
created
a
new
version
of
the
data
that
doesn't
contain
the
the
data
that
I
delete
it.
A
This
is
a
reason
why
I
can
go
back
and
use
time
travel
to
basically
build
up
the
different
versions
right.
I
can
go
back
to
version
0
and
have
no
have
the
version
of
the
data
that
says
iowa's
null
and
Washington
is
336,
because
the
data
still
actually
exists
right.
It's
still
there.
It's
just
that,
basically,
because
the
transaction
log
allows
me
to
keep
track
of
that
all
right.
So
that's
the
main
difference.
A
So
if
you
try
to
force
it
argument,
have
multiple
transactions
on
your
Delta
like
table,
but
then
you
try
to
just
read
the
parking
logs
without
actually
reading
the
Delta
log.
There
could
be
multiple
versions
of
this.
They
just
have
been.
Presumably
you
could
see
duplicate,
certificates
or
so
forth.
These
are
actually
seeing
like
the
previous
versions
of
the
data
as
well,
so
because
it's
an
open
source
project.
A
Actually,
we
recently
released
a
blog,
Delta,
Lake
0.50,
where
we
actually
talked
about
manifests-
and
this
is
where
presto
and
Athena
can
actually
go
ahead
and
take
a
look
at
the
Delta
Lake
data.
They
can
look
at
the
parkade
either,
because
what
you
do
is
you
generate
a
manifest
file.
The
manifest
file
basically
simply
contains
here
are
all
the
files
that
you
should
be
looking
at.
For
the
most
current
version
of
data,
that's
it
and
so
look
for,
say
karma.
A
There
are
20
files
inside
there,
but
if
you
want
to
look
at
the
most
current
versions,
you
only
need
to
look
at
the
six,
so
the
manifest
space
that
contains
the
file
names
of
those
six
files
and
then
press,
don't
Pina
can
read
that
manifest,
which
is
invariably
tell
them
to
go.
Look
at
those
six
files
and
from
there
you're
pretty
much
good
to
go
okay.
So
hopefully
that
answers
that
question.
A
So
actually,
in
this
case
I
apologize
but
I
think
we
are
we.
Our
time
is
up
now,
so
I
will
go
ahead
and
probably
end
this
session
now.
By
the
same
token,
if
you
do
have
questions,
I
highly
would
advise
you
to
join
us
at
Delta
that
IO
or
join
us
at
ml
flow
org
and
as
well.
We
will
be
sending
an
email
out
will
actually
not
so
much
an
email
out
but
will
be
sending
out.