►
From YouTube: Delta from a Data Engineer's Perspective
Description
Take a walk through the daily struggles of a data engineer in this presentation as we cover what is truly needed to create robust end to end Big Data solutions.
About:
Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business.
Read more here: https://databricks.com/product/unifie...
Connect with us:
Website: https://databricks.com
Facebook: https://www.facebook.com/databricksinc
Twitter: https://twitter.com/databricks
LinkedIn: https://www.linkedin.com/company/databricks
Instagram: https://www.instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
Hello,
everybody
and
welcome
to
this
Delta
tutorial
in
an
and
intro.
My
name
is
Paul
Ince
and
I'm.
An
associate
RSA
right
after
my
slides
I
will
pass
it
on.
As
a
theory,
honest
data
breaks
customer
success,
engineer
for
a
delta
demo.
What
is
it
that
all
data
engineers
want?
We
want
to
push
play
on
a
pipeline
and
have
it
flow
beautifully
and
flawlessly
as
it
continues
to
process
incremental
data
in
the
most
cost
efficient
way?
Oh
and
don't
forget
it
be
a
dream
if
we
could
have
both
streaming
and
batch
processing
work
seamlessly
together.
A
So
the
data
engineer
begins
their
journey
to
make
this
dream
a
reality,
but
there
are
a
lot
of
hills
to
climb
on
the
way.
First,
we
start
with
a
stream
from
Kafka
into
our
table,
which
we
have
a
spark
job
running
in
order
to
create
meaningful
reports
from
our
data.
However,
after
some
time
our
data
volume
may
grow
and
our
spark
job
may
simply
not
be
able
to
keep
up
with
the
number
of
files
every
single
time
we
write
to
the
table.
A
So
we
add
a
batch
process
which
compacts
our
table
problem
solved.
But
what
about
this
late
arriving
data
that
the
data
engineer
didn't
plan
for
in
their
original
architecture?
How
do
we
update
our
reporting
table
with
everything
that
might
be
going
on
with
our
incoming
data?
This
causes
delays
in
our
pipeline,
based
on
the
user's
SLA
s.
They
might
not
be
able
to
handle
or
accept
hours
worth
of
latency
for
data
being
unavailable.
A
So
then
the
data
engineer
tries
to
add
streaming
to
the
mix.
Now
it's
a
lambda
architecture.
This
seems
pretty
great
in
theory,
but
the
data
engineer
quickly
realizes
that
operational
overhead.
They
posed
themselves
now
they
have
to
maintain
this
pipeline
in
two
different
places,
not
to
mention
how
can
the
data
engineer
be
certain
that
the
data
that
is
being
processed
through
the
stream
is
validated
against
the
batch
process?
Now
there
is
a
need
for
validation
checks,
which
adds
complexity
and
latency
to
our
pipeline
even
worse.
What?
A
If
something
went
wrong
and
the
data
engineer
needs
to
reprocess
an
entire
batch
of
data?
There
will
need
to
be
cleanups
in
multiple
places,
and
this
is
becoming
harder
and
harder
to
manage
more
than
it's
really
worth
without
a
large
team
focused
solely
on
this
pipeline
alone
updates
and
merges
become
overly
complicated
and
costly,
as
the
data
engineered
devises
a
way
to
make
sure
that
no
data
is
missing
and
no
data
is
duplicated
throughout
the
pipeline.
A
A
The
answer
to
our
problems
is
Delta.
With
Delta
we
can
stream
and
perform
batch
merges
into
the
same
Delta
table.
At
the
same
time,
this
unifies
the
streaming
and
batch
and
a
continuous
flow
with
Delta.
We
can
also
historically
query
snapshots
of
the
data
with
the
capability
to
retain
the
historical
data
indefinitely
and,
lastly,
Delta
architecture
allows
for
independent
elastic,
compute
and
storage
to
scale.
A
A
This
transaction
log
is
then
reconciled
with
the
park'
files
in
order
to
build
an
accurate
and
current
picture
of
the
data
due
to
the
fact
that
the
10th,
the
transactions
have
a
timestamp,
the
users
are
able
to
query
back
in
time
and
what
the
table
would
have
looked
like
at
any
given
period.
Data
within
a
Delta
table
can
still
be
partitioned.
However,
there
are
even
more
optimizations
you
can
do
on
a
Delta
table
that
I
will
get
to
in
a
moment.
You'll
notice
that
there
are
two
table
versions
saved
for
this
table
at
any.
A
In
other
words,
adults
a
table
is
a
set
of
actions.
Those
actions
can
be
reconciled
to
the
data
itself
to
build
a
picture
of
the
data
for
the
user.
The
actions
of
the
transaction
log
consists
of
our
changes
to
the
metadata
entire
files
that
are
added
or
appended
to
the
table
and
removal
of
files
from
that
table.
The
result
of
these
actions
is
the
version
of
the
table
itself.
A
A
Changes
to
the
table
are
stored,
as
ordered
atomic
units,
which
we
call
commits.
You
can
see
to
the
right
side
that
we
first
added
to
parquet
files
in
one
job
and
then
followed
that
with
the
removal
of
those
two
files
and
an
addition
of
a
file
called
three
parquet,
since
this
is
the
order
in
which
these
tasks
were
performed.
This
is
the
order
in
which
they
are
stored
within
the
transaction
log,
but
let's
say
that
two
users
try
to
do
something.
A
So
let's
say
that
I
was
trying
to
write
an
update
record
that
someone
changed
milliseconds
before
me.
Their
action
to
the
data
would
win
and
my
process
would
be
reattempt
'add
so
that
I
get
the
latest
accurate
version
of
the
data
when
I'm
reading
it
in
if
I'm,
if
I'm
user.
Two
in
this
diagram,
my
task
would
have
to
wait
for
user
once
commit
to
go
through
and
then
my
commit
would
be
the
next
one.
A
Large
tables
can
have
a
significant
amount
of
files
in
them,
especially
if
we
have
yet
to
run
an
optimization
command,
such
as
the
keyword
optimize,
which
we
will
cover
later
in
the
demo.
How
can
we
scale
the
metadata?
Of
course
we
can
use
bark.
We
can
use
check
pointing
to
our
advantage
to
choose
a
time
frame
for
how
often
we
want
a
check
point,
for
example
every
10
seconds
every
minute
and
so
on.
A
So
let's
dive
into
the
Delta
architecture
a
little
bit
more.
If
you
recall
this,
is
our
data
engineer's
dream
pipeline,
the
one
they
originally
architected
but
could
not
bring
into
reality
with
Delta?
We
brought
in
the
ability
to
read
consistent
data.
We
talked
about
the
functionality
of
the
optimistic
conflict
resolution
transaction
log.
A
We
also
discussed
how
we
can
read
incrementally
from
a
large
table
with
good
throughput
what
Delta's
ability
to
optimize
the
underlying
files
without
disturbing
the
process
running
on
the
table
and
its
ability
to
scale
to
handle
the
metadata
Delta
also
gives
us
the
ability
to
roll
back
in
case
of
bad
rights
with
Delta.
We
can
roll
back
to
a
certain
place
in
time
and
continue
as
if
those
bad
changes
never
happened.
A
Similarly,
with
Delta,
we
have
the
ability
to
replay
historical
data
or
compare
versions
of
data
without
having
to
save
hefty
snapshots
of
the
entire
data
itself.
We
can
query
an
older
version
of
the
table
without
touching
our
current
version
of
the
table
and
even
compare
them
side-by-side
with
Delta.
We
have
the
ability
to
handle
late-arriving
data
without
a
delaying
downstream
process
due
to
Delta's
capability,
to
have
many
processes
writing
to
the
table
at
a
single
point
in
time.
A
Since
my
time,
working
at
data
bricks,
I
have
seen
Delta
helped
many
customers
wrangle
their
most
complicated
pipelines
into
simple
and
easy
to
maintain.
Workflows.
Delta
can
also
be
used
directly
for
reporting
tables
to
provide
the
end-user
with
the
fastest
up-to-date
data
that
the
data
engineer
could
possibly
supply.
B
My
name
is
Jake
Terry,
honest
I
am
a
customer
success
engineer
here
at
data
brakes,
I
have
been
with
the
company
for
about
a
year
now,
before
this
I
was
in
the
tech.
Consulting
industry.
I
came
over
because,
as
a
data
engineer,
you
know
mostly
working
with
Oracle
databases
and
doop
implementations.
I
wanted
the
opportunity
to
start
working
with
some
of
these
newer
awesome,
Big
Data
technologies
like
spark
and
Delta
like
so
today,
I'll
be
using
this
notebook
to
demo.
Some
of
these
data,
like
architecture
concepts
that
Paula
just
explained
to
us.
B
This
notebook
has
a
lot
of
good
content.
It
gives
us
some
nice
code.
Examples
of
these
features
here
listed
in
bold
I'll.
Try
to
get
through
as
many
of
these
as
possible
time
permitting
we'll
start
by
showing
Delta's
ability
to
unify
fashion
streaming
processes
with
its
asset
transactions.
Delta
table
is
both
a
Bosch
table
as
well
as
a
streaming
source
and
sync
or
multiple
concurrent,
reads
and
writes
so.
This
ability,
along
with
schema
enforcement
and
evolution,
provide
tremendous
advantages
over
your
typical
data
link
architecture.
B
B
So
in
case
you
wanted
to
run
this
notebook
on
the
data
breaks,
Community
Edition.
You
can
absolutely
do
so,
and
we've
already
configured
it
for
you
to
do
that
using
our
Community
Edition
is
free
and
a
great
way
to
get
some
practice
on
dated
breaks
and
learn,
learn
some
spark
so
coming
down
here.
The
first
thing
we're
doing
is
creating
a
park,
a
version
of
the
table
we're
doing
this
because
we
don't
want
to
dive
into
Delta
right
away.
B
B
B
B
So
when
I
kick
off
the
stream
and
what
we'll
notice
here
is
the
input
rate
in
the
processing
rate
or
both
is
zero,
so
no
data
is
actually
being
written
in
to
my
table
from
the
second
screen.
If
we
come
back
up
here
to
our
first
stream,
you
can
see
our
input
rate
of
five
seconds
which
we
defined
earlier.
So
that
makes
sense
and
we
have
a
processing
race.
We
have
data
going
into
the
table
if
we
rerun
our
count
function.
B
So
I
do
have
data
going
in,
but
I
can't
add
a
second
stream
here.
So
basically,
what
this
means
is,
if
I'm
building
these
big
data
pipelines
or
I
need
to
scale
and
process
multiple
threads
from
multiple
sources
and
load,
these
all
into
my
central
target
table
I'm
not
able
to
do
that.
Another
way
to
think
of
this
is
that
by
opening
up
my
first
stream
I've,
basically
locked
in
my
table
to
all
other
sources
writing
to
that
table,
while
the
stream
is
active.
So
there's
one
problem
now:
the
next
problem
that
we
discovered
earlier.
B
So
here
we
can
see
the
actual
data
and
all
my
columns
sitting
here
and
you
can
see
if
our
initial
four
columns,
then
we
also
have
these
additional
two
columns.
This
is
not
what
we
were
expecting
so
what
happened
here
is
our
initial
function
to
generate
our
streaming.
Data
up
here
is
actually
using
this.
This
rate
stream
source
function,
this
dot
format
rate
option
rows.
Four.
Second,
five:
to
generate
our
new
records,
which
is
really
used
mostly
for
development
and
testing
purposes.
It
actually
suits
our
needs.
B
B
B
But
basically,
what
it
comes
down
to
is
that
because
I'm
using
a
part,
a
part
K
table
I,
actually
can't
run
multiple
streams
against
my
table.
At
the
same
time,
and
once
there's
a
schema,
change
I
end
up
corrupting
my
table
either.
These
are
very
fun
or
easy
issues
to
work
with.
However,
they
are
pretty
common
problems
that
most
of
us
do
see
as
data
engineers
or
when
working
with
pretty
much
any
day
to
pipeline.
B
So
there
are
my
four
columns
just
like
we
expected
to
see
so
write
a
good
starting
point
here.
So
we're
back
in
action
ready
to
go
using
the
Delta
table
now,
instead
of
our
parkade
table,
but
can
I
do
some
of
these
Delta
architecture,
things
that
Paula
just
mentioned?
So,
let's
start
getting
into
schema
enforcement
and
we're
just
going
to
cover
this
at
a
high
level.
B
B
B
And
I'll,
let
that
kick
in
okay
cool!
So
that's
going!
We
can
see
our
count
of
fourteen
thousand
seven
hundred
and
five,
which
makes
sense,
and
so
now
we
can
try
running
our
function
to
append
our
table,
but
this
time
we're
trying
to
use
a
delta
format
and
giving
it
the
delta
path,
as
you
can
see
here
so
run
it
we're
on
thinkable
or
if
you
remember
back.
This
is
generate
generating
this
two
additional
columns
that
broke
our
table
last
time
and
we
were
using
park'
format,
so
go
ahead
and
run
this.
B
B
And
it
tells
me
that
if
I
want
to
go
ahead
and
merge
these
schemas
and
go
ahead
and
turn
on
this
option
here,
merged
schema
we'll
talk
more
about
this
in
a
sec
down
here.
It
explains
the
issue
on
what's
actually
going
on,
so
you
can
see.
I
have
my
table
schema
with
four
columns
and
then
my
data
schema,
which
has
six
doodle
fields.
B
B
B
B
B
So
now
you
on
all
might
be
asking
hey
Jake
what
happens
if
you
actually
have
to
change
the
scheme
of
the
table.
The
data
at
my
company
slowly
evolves
over
time,
or
we
have
end-users
in
my
department
that
can't
make
up
their
mind
on
the
data
that
they're
interested
in
seeing.
Well,
don't
worry
because
we've
got
you
covered
with
that
backup
option
of
merge
schema.
B
So
what
we're
gonna
do
here
is
actually
create
some
dummy
data
with
an
extra
column
and
do
a
simple
right
to
append
that
to
our
table.
So
you
can
see
here
we
have
our
columns,
define
with
this
extra
column
called
closed
and
and
then
boolean
values
and
then
we're
creating
a
data
frame.
We're
actually
appending
that
data
frame
and
we're
using
this
option.
Merge
schema
true,
so
I
run
that.
B
So,
to
recap,
what
we
just
walked
through
around
some
of
the
advantages
that
Delta
Lake
will
give
you
over
for
a
parquet.
It
has
full
support
for
simultaneous
batch
and
streaming
workloads
and
any
number
of
reads
and
writes
to
infirm
adults
table
it
gives
you
schema
enforcement
and
schema
evolution
capabilities
to
ensure
you
maintain
data
quality
that
will
always
be
usable
in
queryable,
and
it
gives
you
asset
transactions
that
will
guarantee
data
integrity
even
during
several
concurrent.
We're
closing
queries.
B
B
So
the
first
one
I
want
to
cover
is
what
most
of
you
know
as
an
absurd,
otherwise
gotta
merge
so
scrolling
down
here
to
up
surd.
So
an
absurd
is
essentially
a
logical
atomic
combination
of
a
typical
sequel
insert
and
update
ends
the
name
absurd.
So
the
logic
of
an
upstart
basically
does
when
trying
to
insert
new
records
into
a
table
check
to
see
if
the
record
already
exists,
based
on
a
matched
predicate
key.
If
so,
then
update
the
existing
record
and
if
not
then
insert
the
net
new
record.
B
So
Delta
gives
us
this
absurd
capability
with
the
merge
command,
so
just
real,
quick
before
we
dive
into
that.
Let's
walk
through
what
an
operation
like
this
would
look
like
using
normal
arcade
tables.
So
here's
the
steps
so
first
we
identify
the
new
rows
to
be
inserted.
Then
we'd
identify
our
rows
that
will
be
replaced.
Ie
updated
next,
we'll
identify
all
of
the
rows
that
are
not
impacted
by
the
insert
or
update
will
create
a
new
tape.
Temp
table
based
on
all
three
of
those
insert
statements
will
delete
the
original
table.
B
So
here
we
have
a
little
moving
diagram
to
kind
of
show.
What
that
kind
of
looks
like
and
Barket
now
Delta
like
that
same
operation
can
be
done
in
one
simple
atomic
step,
so
here
I'm
showing
you
the
spark
Python
API
syntax,
but
almost
all
Delta
commands
support
original
sequel,
syntax
that
most
analysts
and
DBAs
are
used
to
so
right
here,
I'm,
just
creating
a
subset
of
my
initial
loans,
Delta
Theta
Delta
Theta,
using
a
filter
right
here
on
the
loan
ID
and
then
making
a
temp.
B
B
B
The
first
record
is
an
existing
loan
ID
of
two,
which
has
been
fully
repaid.
The
corresponding
road
needs
to
be
updated
here
and
then
the
next
new
record
is
a
new
loan
ID
of
three,
which
has
been
fully
funded
in
California
and
misro
needs
to
be
up.
It
sorry
inserted
as
a
new
row,
so
I'm
creating
this.
B
This
data
frame
that
we
just
just
ran
through
and
then
now
down
here
I
can
perform
my
actual
merge
of
this
data
on
my
small
Delta
table.
So
this
actually
shows
the
syntax
for
sparks
equal
right
here
and
then
my
actual
command
is
using
the
PI
spark
data
frame,
API
syntax.
Both
things
are
doing
the
same
thing
at
the
end
of
the
day,
so
both
of
them
are
using.
This
merge,
command
and
you're.
A
B
B
So,
as
you
can
see
up,
cert
is
a
super
powerful
feature
and
it
acts
a
lot
like
the
sequel.
Dml
merge
command,
but
it
provides
even
more
flexibility
to
assist
with
more
advanced
use
cases,
including
streaming
and
complex
conditionality
and
deduplication,
and
this
notebook
actually
does
Kummer
cover
some
of
those
I
won't
get
into
those
right
now.
Just
so,
I
have
time
to
hit
on
a
few
other
important
features
so
scrolling
back
up
to
this
delete
command,
so
delete
from
Delta
lake
table
similar
table
start
there's
delete.
B
In
the
same
way,
we
were
mashing
on
a
predicate
to
perform
an
absurd
before
Delta
lets,
you
specify
a
predicate
to
determine
which
individual
rows
need
to
be
deleted.
This
is
just
another
standard
sequel
like
command
the
Delta
supports
and,
of
course,
just
like
absurd.
We
provide
the
standard
sequel
syntax
as
well
as
the
Python
API
syntax.
So
if
we
were
using
sequel,
you
know
we
could
write
delete
from
table
where
predicate
and
I
can
actually
show
you
that
real
quick,
so.
B
A
B
B
B
And
then
I
can
go
ahead
and
run
my
cap
command
again
and
wallah.
Zero
say
all
of
those
fully
fully
paid.
Loans
have
not
been
deleted
out
of
my
table,
so
it's
really
important
when
working
with
Delta
and
using
these
super
handy
commands
to
really
understand
how
it's
all
working
under
the
covers,
then
you
can
really
start
building
efficient
data
pipelines
and
unlocking
the
power
of
Delta's
optimization
potential.
So
we've
talked
about
Delta's
transaction
log
briefly,
and
that's
a
really
key
concept.
B
If
you
want
to
start
opening
up
the
hood
and
understanding
how
this
all
works
well
go
over
the
transaction
log
in
a
few
minutes,
but
the
key
thing
to
understand
with
this
delete
command
so
that
Delta
is
not
actually
deleting
any
physical
data
or
files.
It's
simply
intelligently,
understanding
which
files
contain
the
records
that
need
to
be
deleted,
marking
them
has
removed
in
the
transaction
log
and
then
writing
additional
duplicates
of
those
files.
Just
without
the
rows
that
you
deleted.
Now,
that's
not.
A
B
Say
that
if
there
are
five
files
containing
rows
that
match
our
predicate,
the
Delta
will
be
rewriting
exactly
five
files.
Instead
of
can
combine
the
data
that
it
needs
to,
and
it
may
be
just
one
file
ready
to
disk
and
then
add
that
file
to
the
transaction
log
and
of
course
this
gets
more
complex
when
you
start
working
with
things
like
partitions.
B
Okay,
so
moving
on
now.
Another
feature
here
is
audit
table
history.
This
records
all
commits
or
operations
that
happen
to
our
Delta
table
based
off
the
tables
transaction
log.
So
we
can
use
this
tape.
Sorry,
we
can
use
this
history
command
to
view
the
history
of
operations
made
on
the
table
version
by
version
with
tons
of
Twp
information.
So
if
I
go
ahead
and
run
this
history
command.
B
I
get
this
nice
pretty
transaction
log,
it's
easy
to
read
and
I
can
go
down
here
and
see
all
of
my
operations
on
the
table.
So
here's
my
initial
write
that
first,
you
know
initialize
my
table
and
I've
got
all
of
my
stringing
updates
and
then
I've
got
that
right.
That
I
showed
you
earlier
where
we've
made
the
append.
B
So
if
we
go
over
here,
you
can
see
all
the
super
detailed
information
where
it
shows
us
our
number
of
files
removed,
number
deleted,
rows,
number
of
added
files
and
number
of
copied
rows
scrolling
back
over.
Another
really
important
thing
to
note
here
is
that
each
operation
comes
with
a
version
and
a
timestamp.
So
this
actually
brings
me
into
the
next
feature.
I
want
to
talk
about,
which
is
time
travel.
B
So,
with
time
travel
were
actually
able
to
query
old
versions
of
a
table
based
on
either
the
version
number
or
the
timestamp.
I'll
show
you
examples
of
both
when
thinking
about
time
travel.
It's
important
to
remember
what
we
just
discussed.
The
concept
of
the
transaction
log
and
that
Delta
doesn't
actually
delete
files.
B
Instead,
it
just
records
the
state
of
the
table
in
the
Delta
log,
which
is
actually
right
here
inside
of
the
tables
main
path
which
I
will
show
you
now,
so
here's
the
tables
main
path
that
we
defined,
and
so
here's
all
the
data
and
within
that
main
path
we
also
have
this
folder
called
Delta
log.
So
if
we
go
in
here,
you.
B
Can
actually
see
that
each
transaction
produces
a
JSON
file,
so
each
transaction
is
alternate
all
details
about
that
transaction
are
contained
in
each
of
these
JSON
files
and
they're
numbered
for
us
in
every
tenth
transaction.
We
checkpoint
it
by
adding
this
part.
A
file
was
big
which
basically
aggregates
all
of
our
previous
transaction
data
and
that's
kind
of
how
we
deal
with
large-scale
metadata
management
in
Delta.
B
So
you
can
see
all
this
really
detailed
information,
and
this
is
actually
where
the
history
command
pulls
from
when
it
creates
that
really
nice
audit
history
table.
So
you
can
see
all
that
similar
information
timestamp,
the
user
ID
the
operation
delete,
and
you
can
see
the
operation
metrics
that
we
went
through
before
we
remove
two
files.
Number
of
rows
deleted
the
number
of
files
added
the
number
of
rows
copied,
and
then
you
can
actually
see
the
detailed
transaction
history.
B
So,
by
maintaining
these
transaction
details,
Delta
can
intelligently
interpret
what
any
given
version
of
the
table
should
look
like
to
query
the
appropriate
files.
So
if
I
come
down
here,
I
can
run
my
query
using
this
option
for
time
travel
and
here
I'm,
just
using
version
as
of
to
specify
that
I
want
to
see
a
specific
version
number
which
I'm
passing
through
here
with
this
previous
version
variable,
and
that
just
contains
the
version
number
of
my
table
as
it
was
right
before
at
our
in
that
delete
command.
B
We
can
also
do
the
same
thing
with
the
time
stamp.
This
works
similarly,
but
the
big
thing
to
note
here
is
that
Delta
will
always
go
back
to
the
closest
time
stamp
that
is
prior
to
the
date
given
so
I'll
show
you
what
I
mean
by
that.
If
I
come
up
to
the
transaction,
log
and
I
take
my
most
current
timestamp,
so
time
send
up
the
current
version
and.
B
I
pass
that
through
as
my
argument
and
run
it
and
I'll
get
the
state
of
the
table,
as
it
is
right
now
so
zero
records.
However,
let's
say
I
pass
in
a
time
that
is,
you
know
just
one
second,
before
my
current
timestamp.
So
this
is
a
time
that
is
inbetweening
in
my
previous
version,
in
my
current
version,
obviously
much
closer
to
my
current
version,
but
it's
still
you
know
in
between
both
of
them
and
let's
see
what
I
get
here.
B
B
So
while
it's
nice
to
be
able
to
try
and
travel
to
any
previous
version,
sometimes
you
want
to
actually
delete
the
data
from
storage
completely
or
reducing
storage
bus
or
for
compliance
reasons
such
as
gdpr.
So
the
vacuum
operation
deletes
data
files
that
have
been
removed
from
the
table
for
a
certain
amount
of
time
by
default.
B
It
will
only
delete
files
that
are
older
than
seven
days
so
for
the
purposes
of
this
example,
since
you
know
we
just
created
our
table
and
we
don't
have
seven
days,
we
can
actually
go
and
change
that
default
setting,
so
it
will
set
a
retention,
duration
check,
enabled
to
false,
and
then
we
can
actually
pass
through
a
retention
hour
or
a
retention
time
of
zero
and
well,
and
that
will
just
go
ahead
and
delete
all
files
that
are
no
longer
used
in
the
transaction
log.
So
we
go
ahead
and
run
this.
B
B
And
so
that's
gonna
go
ahead
and
give
us
this
error
here.
That's
gonna
fail
because
those
files
no
longer
exist,
and
so
vacuum
is
actually
really
really
powerful
for
GDP
our
use
cases
when
usual
used,
along
with
with
you
know,
like
delete
and
update
commands,
because
you
can
actually
go
ahead
and
delete
out
the
physical
data.