►
Description
Join us for an online tech talk on Delta Lake. Tech talks include a technical presentation with slides and a demo, with time for Q&A at the end.
Abstract:
The General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) both aim to guarantee strong protection for individuals regarding their personal data and apply to businesses that collect, use, or share consumer data, whether the information was obtained online or offline. This remains one of the top priorities for the companies to be compliant and they are spending a lot of time and resources on being GDPR and CCPA compliant.
Your organization may manage hundreds of terabytes worth of personal information in your cloud. Bringing these datasets into GDPR and CCPA compliance is of paramount importance, but this can be a big challenge, especially for larger datasets stored in data lakes.
Learn how you can use Delta Lake which is created by Databricks and powered by Apache Spark™ to manage GDPR and CCPA compliance for your data lake. Because Delta Lake adds a transactional layer that provides structured data management on top of your data lake, it can dramatically simplify and accelerate your ability to locate and remove personal information (also known as “personal data”) in response to consumer GDPR or CCPA requests without disrupting your data pipelines.
Join our Tech Talk to learn:
- The compliance challenges big data and data lakes create for organizations.
- How Delta Lake improves data lake management and makes it possible to quickly find and surgically remove or modify individual records.
- Best practices for GDPR and CCPA Compliance using Delta Lake.
- Use of “Pseudonymization” (https://en.wikipedia.org/wiki/Pseudonymization) and structuring pipelines to locate and remove the identifier to destroy the linkage between the pseudonyms and identifiers.
- Demo on how to easily fulfill data requests with Delta Lake and Databricks.
Agenda: 10AM PDT - 11AM PDT (GMT-8)
10:00AM - 10:50AM - Tech Talk
10:50AM - 11:00AM - Q&A
To join the live chat, check out the meetup page: https://www.meetup.com/data-ai-online/events/270370715/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
So
thanks
everyone
for
joining
us,
we're
here
for
a
online
Delta
like
Tech
Talk,
so
addressing
gdpr
and
CCPA
scenarios
with
Delta,
Lake
and
Apache
spark,
and
we
have
some
really
awesome
speakers
for
you
today
from
data
bricks.
If
I
could
just
have
each
of
them
do
a
quick
introduction,
let's
start
with
Justin
right.
B
Thanks
everyone
for
joining
today
as
the
bio
notes,
I'm
David
Brooks
is
associate
general
counsel
of
product
and
privacy
I
essentially
work
with
our
engineering
and
product
teams
to
determine
what
impact,
if
any,
various
privacy
and
other
types
of
laws
have
on
the
features
we
have,
or
you
know,
internal
compliance
things
that
we
have
to
do
like
current,
for
instance,
the
work
we
did
to
enable
customers
to
be
able
to
handle
their
own
data.
Subject.
Requests
within
our
platform
do
a
lot
of
other
stuff
too,
but
that's
the
most
relevant
for
today.
D
Hi
I'm
Vinnie
well
customer
success,
engineer
at
theta
bricks,
Base
in
San,
Francisco
I
work
closely
with
customers
every
day
to
drive
their
to
drive
their
implementation
of
cloud
and
big
data
and
I'm
very
passionate
about
strategizing,
AI,
machine
learning
and
analytics
use
case
with
them
and
I
have
been
working
with
various
industries
financed
geospatial,
gaming,
retail,
etc.
So
thank
you
for
joining
us.
D
And
so
welcome
to
our
Tech
Talk
for
the
agenda
today.
Justin
is
gonna,
speak
about
GBP,
R
and
C
CPA.
What
that
means
for
data
organizations.
Darshan
is
gonna
cover
the
overview
of
challenges
to
comply
with
these
laws,
how
to
build
an
ideal
data
leak
and
how
data
data
breaks,
Delta,
R,
powerful
to
build
by
creators
of
Apache
spark
and
data
breaks,
addresses
those
challenges
and
I
will
conclude
it
with
demo.
B
That
would
be
a
processor
there's
a
30-day
clock
that
starts
from
the
moment.
The
request
is
received
and
you
have
to
be
able
to
satisfy
that
request
within
that
time
period.
There
are
certain
circumstances
in
which
that
deadline
can
be
extended,
but
on
a
general,
ongoing
basis,
you've
got
a
30-day
window
and
then,
as
a
sort
of,
why
do
we
care
as
a
reminder
of
the
gdpr,
has
the
ability
for
the
data
protection
authorities
in
Europe
to
levy
substantial
fines
against
companies
potentially
is
highest?
B
Four
percent
of
your
global
turnover,
also
those
annual
revenues
or
20
million
euros
next,
so
CCPA
in
many
ways
is
talked
to
talked
about
as
kind
of
the
GDP
our
white
California
version
in
the
gdpr.
There
are
a
lot
of
ways
in
which
that
is
it
true,
but
one
of
the
ways
in
which
it
basically
is
true
is
the
the
consumer
rights
requests
concept,
there's
no
formal
name
for
it
under
the
CCPA,
but
essentially
consumers
which
are
people
in
California,
have
the
right
to
know
what
data
is
being
collected.
B
It's
called
personal
information
rather
than
personal
data,
have
what
the
right
to
have
certain
that
information
delete
it
both
by
a
so
called
business
as
well
as
the
business's
service
providers.
Business
is
roughly
equivalent
to
the
concept
of
controller
under
the
GDP,
our
service
provider,
roughly
equivalent
to
the
concept
of
processor,
and
also
it
gives
companies
the
the
requirement
that
they
allow
their
consumers
to
opt
out
of
the
so-called
sale
of
personal
information.
B
If
you
choose
to
exercise
any
of
these
rights
and
then
the
businesses
themselves-
and
this
is
sort
of
what's
most
relevant
for
the
folks
on
this
call-
essentially
you
have
to
create
processes
like
with
the
gdpr
that
enable
them
to
respond
to
these
consumer
rights.
Requests
within
various
timelines
leave
timeline
to
actually
satisfy
the
request
is
a
little
bit
more
lenient
in
California,
it's
a
45-day
period.
B
However,
there
is
also
a
ten
day
window,
ten
business
day
window
where,
after
receiving
the
consumer
rights
request,
you've
actually
got
to
respond
to
it
and
say:
hey:
we've
received
your
request
and
we
will
begin
taking
action
against
it
and
from
the
perspective
of
fines,
the
CCPA
in
some
ways
is
a
little
bit.
Let
potentially
less
onerous
but
in
other
ways
could
actually
be
far
worse.
B
There's
the
ability
for
the
California
Attorney
General
to
let
me
find
on
the
basis
of
violations
as
a
CCPA,
which
can
be
up
to
$2,500
per
violation
if
it's
not
found
to
be
intentional
or
as
much
as
seventy
five
hundred
dollars.
If
intentional,
a
violation
is
not
really
tightly
defined
and
so
there's
a
decent
chance
that
this
ends
up,
meaning
every
single
consumer
who
you
have
violated
the
law
in
favor
of
would
essentially
be
a
multiplier
here.
B
You
do
so.
The
CCPA
applies
essentially
to
for-profit
companies
that
do
business
in
California
and
also
meet
one
of
the
three
prongs
listed.
They
either
have
to
collect
personal
information
on
50,000
or
more
California.
Consumers
have
annual
revenues
of
greater
than
20
five
million
dollars
or
earn
50%
of
their
annual
revenue
from
the
sale
of
California
residents
data
and
then
the
other
thing
to
note
from
like
a
data
science
perspective
is
what
constitutes
personal
data
on
the
GDP.
B
Our
side
or
personal
information
on
the
California
side
mostly,
is
similar,
but
one
way
in
which
the
CCPA
is
actually
expanded.
It
is
this
concept
of
household,
so
there
was
at
least
of
theoretical
arguments
we
made
under
the
GDP
are
that
an
IP
address
might
not
actually
constitute
personal
data,
if
you
weren't,
really
sure
which
person
it
related
to
whether
it
was
because
it
was
behind
an
added
firewall,
it
was
a
company
or
whether
it
was
you
were
trying
to
take
the
position
that
hey.
D
B
Seen
two
different
user
agent
strings
associated
with
this
IP
address.
That
must
be
two
different
people,
I,
don't
know
which
one
it
is,
and
so
it's
not
personal
data
under
the
GDP
are
the
CCPA
sort
of
gets
rid
of
that
wiggle
room
in
states
that
if
it
relates
to
a
household
which
it
helpfully
doesn't
define,
but
is
probably
any
number
of
people
living
within
the
same
home.
That
will
mean
that
it's
personal
information,
even
if
you
can't
say
it's
John
versus
Jane
next
and
I-
will
turn
it
over
to
darshan.
Oh.
C
C
C
We
almost
have
the
power
to
process
that
much
amount
of
data
as
well,
but
what,
with
the
compliance
factor
there,
we
don't
know
if
we
will
be
able
to
spend
the
amount
of
time
from
a
human
or
a
person
perspective
or
the
time
perspective,
to
remain
in
compliance
from
these
laws.
Considering
we
have
terabytes
of
data
being
collected
across
our
organizations
and
considering
we
have
a
budget
in
mind
to
keep
to
remain
in
compliance
with
these
things.
C
So
how
do
we
go
around
this
problem
to
ensure,
as
a
compliance
to
ensure
we
are
able
to
do?
Fine-Grained
deletes?
Do
not
comb
through
petabytes
of
data,
so
all
of
these
address
challenges
are
addressed
with
parts
of
the
technologies
that
are
put
together
in
the
form
of
Delta.
The
other
thing
that
also
comes
to
mind
is
that,
with
so
much
of
information
being
collected,
how
do
you
ensure
that
you've
deleted
all
of
the
information
for
a
particular
subject?
C
So
from
those
perspectives,
how
can
you
efficiently
update
information
requests
or
you
sort
of
need
some
technologies
like
indexing
or
caching,
or
faster
access
to
all
of
that
petabytes
of
information,
so
we'll
go
over
all
of
those
steps
that
Delta
like
actually
addresses
and
how
we
can
use
them
to
our
benefit,
to
reduce
and
do
these
things
in
a
much
more
efficient
fashion
in
a
time
bound
fashion,
and
you
know
bring
down
the
person
and
even
meant
that
you
have
to
put
in
to
remain
in
compliant.
Thank
you
next,
one,
all
right.
C
So
let's
talk
about
each
of
those
challenges
that
we
talked
about
in
the
previous
slide
and
how
we
address
them
into
Delta
Lake,
and
you
know,
while
I'm
talking
through
it.
Some
of
these
parts
are
specific
to
data
breaks,
Delta
and
those
have
been
highlighted
with
a
black
background.
So
we'll
talk
about
those
as
we
go
ahead,
but
just
keep
that
in
mind.
C
So
with
the
the
most
important
thing
that
you
have
to
care
about.
When
you
want
to
do
transaction
like
operations,
you
have
to
go
transaction
on
top
of
something
like
s3
or
a
TLS.
On
my
on
the
Mac
software
or
on
the
Amazon
Cloud
right,
so
in
that
sense,
Delta
has
a
layer
of
transaction
wrapped
to
basically
make
your
data
Lake
asset
compliant
italic
that
solves
a
bunch
of
problems.
C
You
don't
have
to
worry
about
eventual
consistency
of
these
dialects
and
you
don't
have
to
worry
about
if
my
data
was
deleted
or
not
deleted
with
my
transaction
went.
Fine
Delta
takes
care
of
that.
The
next
part
is
being
able
to
efficiently
figure
out
where
my
data
is
in
the
data
Lake
for
a
dataset
that
involves
features
of
elderly,
including
compaction,
indexing
and,
in
some
respects
caching
as
well.
C
We
also
want
to
talk
about
a
framework,
or
we
want
to
promote
a
particular
idea
of
suriname
ization,
which
is
particularly
useful
for
remaining
gdpr
compliant
and
we'll
talk
about
that
in
detail.
The
whole
idea
is,
you
can
separate
out
your
personal
information
from
your
non
personal
information.
So
if
there's
a
request
to
delete
stuff,
you
can
remove
or
delete
the
personal
information
and
let
alone
the
non
personal
information
which
can
still
be
used
for
making
sure
aggregate
values
or
aggregate
tables
and
user.
Accurate
and.
B
Just
to
have
a
little
bit
more
color
there
darshan,
when,
when
you
do
the
pseudonymous
ation
process,
it's
about
taking
the
elements
that
are
themselves
identifiable
because,
under
both
the
gdpr
and
CCPA,
if
their
information
stored
against
and
I,
don't
that
in
itself
is
personal
data,
so
you
might
have
an
equal
of
an
email
address.
It's
kind
of
your
object,
identifier
and
you
might
have,
for
instance,
city,
state
country,
height,
color
of
eyes
or
whatever.
B
If
you
had
that
tied
against
an
email
address
that
in
from
all
of
that,
information
is
personal
data
or
personal
information.
What
pseudonymous
ation
allows
you
to
do
is,
instead
of
tying
it
to
an
email
address.
You
tie
it
to
an
internally
understandable
identifier,
which
say
abc123
and
when
you
that
the
information
at
that
point
is
still
personal
data
or
personal
information.
However,
once
you
remove
the
link
between
the
email
address
and
the
actual
pseudonymous
identifier,
then,
as
long
as
you
aren't
able
to
recreate
that
information
yourself,
you.
B
The
remaining
information
as
non
personal
data
or
non
personal
information,
which
means
when
you
get
a
data,
subject,
request
to
delete
something.
All
you
have
to
do
is
take
the
step
to
delete
that
link,
and
it's
also
worth
noting
at
this
point
and
we'll
get
into
a
little
bit
more
detail
shortly.
But
the
GPR
doesn't
allow
you
to
do
any
additional
processing
once
you
receive
that
initial
request.
B
So
if
you
don't
have
a
legal
basis
to
continue
processing
someone's
data
and
like
even
though
they've
provided
the
the
data
subjects
request,
you
would
not
be
able
at
that
point
to
do
the
pseudonym
ization
process
and
then
do
the
delete.
That
would
be
a
violation
of
the
gdpr.
But
if
you're
already
keeping
your
data
in
this
form,
you
actually
get
the
ability
to
just
delete
that
linkage,
keep
all
the
data.
And
then
it
is
an
added
cherry.
B
On
top,
the
GPR
explicitly
notes
that
if
you
are
properly
seen
on
amaizing
your
data,
if
there
ends
up
being
a
data
breach
of
the
pseudonymous
data
on,
you
will
end
up
getting
either
a
slap
on
the
wrist
or
fined.
Far
less
than
you
would,
if
the
the
data
that
was
leaked
was
directly
and
immediately
title
to
an
actual
person.
C
Yep
absolutely
so
pseudonymous
issue
is
extremely
important
for
many
of
you
may
be
starting
within
the
but
GDP
or
some
of
you
have
being
you
know
some
many
companies
come
to
us
and
say
we
want
to
be
model
to
dpr
citizen.
Somebody
want
to
be
model.
Citizens
they've
already
started
converting
the
text
into
sadhana
mines,
data
formats.
C
So,
as
justin
was
mentioning,
you
have
a
link
between
your
personal
information
and
a
non
personal
information,
for
example,
personal
information
like
a
name
email
address,
I
color
and
so
on,
and
your
purchase
history
for
that
matter,
which
is
like
more
factual
and
you
can
replace
and
say
this
is
person's
personal
information
or
you
can
see
this
is
person
access,
purchase,
history.
It
shouldn't
matter
in
terms
of
calculating
how
much
I
spend
right.
C
C
All
right
so
I'm
just
gonna
go
over
high-level
steps
and
some
notions
of
what
we
generally
observe
while
doing
to
dpr
deletes
and
so
on.
So
the
picture
over
here
shows
s3
bucket,
but
technically
this
is
for
any
object.
Oh
really,
so
you
have
an
object
store
and
you
have
some
data
there.
The
first
step
would
be
to
convert
that
data
into
Delta.
Now
you
may
have
two
things
you
may
have
data
in
a
parka
format,
which
is
the
format
on
which
Delta
Blake
is
built
on,
on
top
of
it.
C
So
converting
parka
data
to
Delta
is
reasonably
simple
and
it
happens
in
places.
Should
you
have
data
in
any
other
format?
You
will
have
to
convert
that
data
not
in
place
or
you
have
to
read
converter
rewrite
the
data
into
elta
format.
Just
that
and
then,
once
you
have
the
data
in
that
format,
you
just
have
to
issue
a
sequined
elite
command.
C
So
another
important
step
is
to
make
sure
that
whatever
you
rewrite
as
a
part
of
it
deletes,
is
optimized
enough
to
not
break
the
SLS
of
downstream
queries
on
that
same
data
set.
That's
extremely
important
and
the
next
step
is
always
to
vacuum
it
out
what
vacuum
is
nothing
but
a
process
of
removing
fully
removing
invalidated
files
which
may
arise
from
your
delete,
commands?
Okay,
those
are
the
high-level
steps.
Now,
let's
look
at
the
delete
logic
and
let's
the
previous
slide
on
the
delete
logic.
What
happens
is
based
on
this
logic?
C
We
will
see
what
performance
of
them
as
we
can
do.
So.
What's
the
polite
logic,
the
logic
says:
figure
out
the
files
that
are
impacted
because
of
the
delete
and
then
rewrite
those
files
without
the
records
that
are
to
be
deleted.
It's
very
simple,
as
episode
based
on
this,
you
have
to
optimize
in
two
steps.
First,
optimization
will
be
in
figuring
out
the
files
which
need
to
be
deleted
or
which
are
impacted
because
of
the
date.
C
The
easiest
way
to
do
this
is
to
ensure
you
can
do
a
broadcast
of
your
smaller
data
frame
when
you
do
the
daily.
Basically,
what
this
means
is
figuring
out.
The
files
which
are
impacted
are
done
in
a
broadcast
fashion,
and
your
void
have
sought
much
joint
visible,
and
that
is
much
faster,
so
I
would
definitely
promote
the
thought
of
distributing
your
deletes
over
partitions
in
an
effort
to
broadcast
a
smaller
data
frame.
To
have
this
phase
of
the
delete.
Logic
go
much
faster
and
the
next
phase
is
about
rewriting
files.
C
So
if
you
have
some
lesser
files,
you'd
have
less
of
things
to
rewrite,
and
that
should
naturally
go
faster.
One
of
the
thing
that
we've
come
across
now
and
then
is
that
when
you
have
these
data
sets,
you
always
have
invalid
values
and
those
invalid
values
can
result
in
to
skew
so
make
sure
that
you
filter
out
the
invalid
values.
You
care
about
values.
You
need
to
delete
and
not
about
the
values
like
nulls
or
empty
space,
and
so
on.
So
you
want
to
filter
for
those.
C
Other
two
things
which
are
specific
to
inner
breaks,
which
include
some
people,
as
the
ordering
is
a
method
of
organizing
editors,
specifically
in
a
format
which
enables
faster
retrieval
of
the
information
which
is
particularly
available
in
data
breaks,
and
it's
going
to
basically
enable
figuring
out
the
files
faster
and
rewriting
those
files
faster
as
well
and
compaction.
You
know,
like
I,
said
before,
once
you
start
mucking
with
a
file
in
order
to
do
a
delete
or
absurd.
C
C
C
You
know
official
deletes.
We
spoke
about
this
one
on
the
overview
slide.
What
what
I'm
trying
to
promote
here
is,
when
you
have
to
figure
out
which
files
to
rewrite
you
can
structure
your
deletes
in
a
way
that
you
distribute
them
over
partitions
and
you
send
those
delete,
commands
out
specific
to
different
partitions
separately,
so
each
partitions
delete
operation
could
leverage
broadcast.
What
could
that
possibly
mean
for
us
is
that
you
could,
because
of
the
broadcasting
error
you
could
enable
partition
problem
or
you
put
a
nipple
file
pruning.
C
If
you
wanted
efforts
like
I,
said,
remove
and
valid
values
try
to
see
if
you
can
use
data
breaks
and
if
your
own
reader
breaks
make
sure
you
use
your
drink
for
the
right
key
and
use
compaction
for
downstream
Optima.
Well,
I
think
we
spoke
about
this
in
detail.
If
there
are
questions
in
the
end,
we
can
address
the
next
one
all
right.
So
this
is
an
important
step
to
discuss
vacuum
and
retention.
C
If
you
are
on
the
verge
of
being
out
of
compliance
in
terms
of
time,
you
can
do
a
vacuum
on
the
table
with
retain
zero
hours,
which
basically
says
don't
worry
clean
up
everything
up
until
now,
every
garbage
in
my
data
set,
which
are
invalid,
just
cleaned
up
right
away
and
go
from
there.
So
what,
by
default,
the
value
is
7
days
and
it
will
automatically
get
cleaned
up.
But
if
you
are
on
the
words
of
a
boundary
condition
make
sure
you
use
rating
0
hours.
C
So
Hank
said
that
what
we
also
thought
about
is
when
people
actually
land
your
data
directly
into
datasets
on
into
did
Alex.
They
may
be
landing
them
in
the
format
in
which
they
receive
it.
So,
as
a
first
step,
you
receive
raw
data
and
you
take
that
raw
data
and
you've
kind
of
promoted
over
into
you
know
different
layers
reaching
a
goal
layer
of
data
which
becomes
a
consumable
data's
layer
right
so
in
that
sense,
always
have
retention
policies
on
your
object
stores
to
delete
that
or
data.
C
So
you
don't
have
to
worry
about
GDP
or
compliance
there
at
all
in
the
first
place.
So
if
you
have,
let's
say
s,
the
retention
policy
of
saying
I
am
going
to
delete
this
data,
which
is
older
than
30
days.
It
automatically
deletes
your
PII
data
as
well.
Now.
The
only
thing
that
then
you
have
to
worry
about
is
data
which
is
managed
by
Delta
in
your
Delta
lake,
and
that
becomes
easy
next
slide,
all
right,
so
Justin
also
kind
of
promoted.
This
idea
and
I
think
that's
just
the
GDP.
B
C
Yep,
so
let's
go
dive
in
your
details
and
what
pseudo
anonymize
would
look
like
if
you
were
to
connect
meant
out.
So
what
does
basically
means
is
that
let,
for
example,
I
have
my
information
about
my
name,
my
last
name,
my
social
security
number,
my
email
address
and
then
I
have
a
bunch
of
information
about
the
things
I
buy
on
an
e-commerce
website
and
how
much
do
I
spend
on
those
e-commerce
websites.
C
So
there
is
some
factual
information
about
my
purchase
history
and
the
things
that
I've
purchased
and
those
quantities
and
those
colors
and
those
clothings
and
all
of
those
things
and
there's
information
about
just
me
which
says
this
is
the
guy
who
bought
it.
So
we
separate
out
the
two
types
of
information
and
we
call
information
about
me
as
personal
information
and
information
about
what
I
did
in
terms
of
a
transaction
on
the
edge'
commerce
website
as
factual
information.
C
And
what
you
want
to
do
is
separate
the
personal
information
from
factual
information,
and
you
have
a
link
in
between
the
two
which
is
an
artificial
link
to
should
you
have
the
need
to
figure
out
who
actually
who's
a
real
person
who
actually
bought
this
particular
item?
You
can
use
that
link
to
connect
the
artificial
key
or
a
surrogate
key
to
an
actual
customer
ID
and
look
up
details
of
that
customer
now.
C
All
linkages,
I
have
for
version,
for
instance,
and
then
what
happens
is
all
of
my
purchase
history,
which
was
there
in
this
one
on
my
status,
a
belongs
to
a
cryptographic,
hash
of
an
email
address,
for
instance,
a
one-way
cryptographic
hash,
so
I
have
no
way
of
saying
that
version
had
actually
made
this
purchase.
But
my
numbers,
which
are
dependent
and
on
that
particular
of
just
his
shift
on
those
dollars
for
monthly
reporting
for
data
science,
analysis
or
anything
else-
are
you
reporting
on
may
not
have
to
change,
but
are
not
impacted.
They
still.
C
They
still
are
true
to
purchases
made
on
the
platform.
One
more
important
thing
that
we
also
spoke
about
in
the
previous
slide
is
that
you
have
data
learning
in
the
raw
zone
and
that
may
not
be
controlled
by
Delta.
So
you
want
to
ensure
that
you
have
policies
to
gonna
expire,
that
data
and
delete
that
data
on
retention
intervals.
Thank.
B
You
just
two
more
quick
points
to
add
in
here
the
best
part
about
Sudan
amaizing.
Your
data
is
it
actually
allows
you
to
do
need
to
know
based
access
permissions
so,
as
darshan
is
pointing
out,
if
you've
got
a
data
science
team,
that's
trying
to
run
numbers
even
against
single
users
on
sort
of
how
their
purchase
pattern
might
change
or
what
might
happen
if
they
were
presented
with
a
particular
advertisement
before
making
a
purchase?
B
C
Perfect
and
you
know
to
think
of
it
visually,
what's
what
we're
trying
to
use?
It's
not
just
one
table,
think
of
it
as
a
person's
transaction
history.
A
person's
clickstream
is
yours,
behavior
being
captured
on
a
website.
It's
tremendous
amounts
of
data
when
you
actually
blow
it
up
into
the
number
of
people
actually
transacting
on
that
website.
It
curves
into
petabytes-
and
you
know,
a
large
volume
of
amount
of
data,
but
the
amount
of
data
that
you
have
the
lookup
table
is
fairly
small.
C
C
All
of
the
others
features
to
quickly
do
a
small
little
delete
operation
instead
of
a
cup
table
and
everything
else
boom,
no
other
table
who
needs
to
be
touched,
but
as
he
was
mentioning,
you
could
have
multiple
strategies
of
having
multiple,
artificial
keys
or
pseudonyms
in
this
case
for
multiple
different
purposes.
Next
time,
all
right.
C
So
the
reason
I
put
this
right
over
here
is
to
gonna
say
that
if
you
are
in
the
process
of
making
your
data
lake
or
pseudonymous
titillate
in
that
sense,
what
you
would
want
to
do
is
you
want
to
use
that
ability
to
observe
the
mores
and
change
those
values
in
existing
tables
from
actual
identifiers
to
pseudonyms?
And
you
know,
Delta
enables
you
to
do
that
using
it's
absurd
capability
or
you
know,
updating
individual
values
or
even
merging
into
it.
So
all
of
it
can
be
done
in
one
case.
C
The
examples
I've
shown
over
here
is
that
I'm
replacing
a
customer's
name
with
a
pseudonym
for
it,
but
it
can
be
extended
to
a
lawyer.
You
know
beta
cell
next,
one
alright,
so
we
spoke
about
dating.
We
spoke
about
Delta
enabling
faster
deletes,
efficient,
deletes
less
people
in
volumes
of
data
pipeline
ease
of
operations
or
beta
pipelines.
You
don't
have
to
know.
I
have
had
customers
who
have
had
multiple
copies
of
the
data
to
reconstruct
how
the
data
should
look
like
and
so
on.
None
of
that
is
required
know
what
copies
are
required.
C
All
you
need
to
do
is
make
sure
you
have
a
transactional
Delta
delete
on
top
of
your
data
and
most
you
know
you
reduce
your
operations
or
time
so
landed
it
and
didn't
like
have
a
policy
there.
Convert
it
to
Delta,
do
GDP
or
deletes
on
top
of
Delta
make
sure
you
have
sternum
eyes
data
like
at
entry
and
then
nothing
on
the
aggregation
has
to
change
so.
D
D
Yeah,
so
we
learned
that
Delta
Lake
is
very
effective
for
addressing
GTP
RNC
CPI
compliance
requirements.
Now,
let's
see
it
in
action
with
the
demo,
so
in
this
demonstration,
I
will
be
reviewing
how
to
convert
your
existing
data
to
Delta.
I
will
also
show
you
different
approaches
of
deltas
capabilities
to
speed
up
delegates,
including
broadcasting
and
zra.
D
We
will
discuss
audit
history,
vacuum
and
retention,
and
at
last
I
will
talk
about
recommendations
and
best
practices
on
setting
up
efficient
pipelines,
so
I
will
cover
the
use
of
pseudonymous,
ation
and
compaction
and
optimize
that
duction
already
talked
about.
So
to
begin
with,
this
I
am
setting
up
our
environment,
so
I'm
using
data
bricks
environment
here
for
the
demonstration
purposes,
but
you
can
use
it
with
notebook
of
your
choice.
D
To
run
this
notebook,
we
have
to
create
a
cluster,
so
the
cluster
that
I
have
is
running
on
data
bricks,
runtime
version,
six
point:
five:
three
worker
nodes
and
a
driver,
each
of
a
three
large
X,
large
30,
lead
set
each
and
for
coarse.
So
why
that
matters
is
because
I'm
going
to
show
you
some
performance
metrics
or
later
on.
So,
let's
talk
about
our
data
set
in
my
demo,
I
am
referencing
our
database
GDP
R,
which
contains
a
sample
customer
table
with
65
million
rows,
which
amounts
to
about
three
gigs
of
data.
D
Let's
review
the
schema,
so
I
have
a
customer
table
here
which
contains
customer
personal
information
like
address
first
name
last
name,
salutation,
birthdate,
email,
address,
etc,
and
some
transactional
records
the
keys
to
be
deleted
as
a
part
of
customer
request
is
located
under
the
table.
Customer
delete
keys-
and
these
are
roughly
about
10%
of
the
original
data
that
we
have
in
the
customer
table
about
and
the
size
of
my
the
keys
table
is
roughly
about
337
megabytes
in
this
table.
D
I
have
customer
SKU
and
customer
ID
ID
information,
so
customer
ID
is
what
identifies
the
customers
that
have
requested
to
be
forgotten
to
get
started
with
data
leak.
First,
we
need
to
convert
and
convert
our
existing
data
into
manage
to
other
tables,
so
here
I
am
ingesting
our
raw
data,
which
is
already
in
parquet.
So
if
the
data
is
already
in
parquet
format,
we
can
just
use
convert
to
Delta
command
without
rewriting
the
data
just
for
the
interest
of
time.
D
I
have
already
run
this
notebook
before
the
demo,
so
I'm
just
gonna
talk
about
the
results
as
well.
So
as
you
see
here,
because
Delta
uses
parquet
format,
which
is
an
open
file
format,
your
converted
data
won't
be
locked
in,
so
you
can
quickly
and
easily
convert
your
data
back
to
another
format.
If
you
need
to
now,
if
the
data
is
in
other
formats,
like
CSV,
JSON,
etc,
you
can
use
our
Apache
spark.
Api
is
to
rewrite
the
format
of
data.
D
Step
two
is
performing
deletes,
so
after
we
ingest
our
data,
convert
the
tables
into
totally.
We
can
now
delete
the
personal
information
of
users
who
have
requested
to
be
forgotten,
so
I'm
gonna
show
you
deletes
with
three
approaches,
as
I
mentioned
before.
The
first
approach
is
a
straightforward
delete
of
customer
personal
data
from
the
customers
table.
D
So
for
this
purpose,
I
am
gonna
return
broadcasting
off
for
my
for
state
now
the
record
that
needs
to
be
forgotten
are
located
in
customer
delete
keys
in
this
code,
I'm,
exploring
how
much
of
the
requested
records
I
present
in
my
main
customer
table
so
observe
the
timing.
Here
it
took
22
point
one
nine
seconds.
D
This
is
just
a
sample
query
to
recover
the
original
state
of
data,
just
because
I
want
to
recover
two
more
delete
approaches
in
this
step
in
the
next
step.
So
in
the
second
approach,
let
me
recover.
Let
me
explain
like
what
I'm
gonna
use
here
so
here
I
have
used
broadcast
earlier
when
I
did
negative,
one
that
means
I,
guess
I
forgot
casting,
but
here
I'm
setting
up
our
broadcast
value.
What
this
does
is
broadcast
helps
to
reduce
the
time
to
find
relevant
files.
So
once
I
run,
this
I'm
gonna
run
this
command
again.
D
To
find
my
matching
records
so
observe,
observe
that
so
real
quickly
on
the
hint
here.
This
instructs
part
to
broadcast
each
specific
table,
then
joining
it
with
another
table
or
view
by
default.
We
have
ten
megabytes
of
the
threshold,
so
here
finding
the
matching
records
took
12
seconds,
which
is
almost
half
of
what
it
took
without
the
broadcast.
D
D
Now,
let's
look
at
our
third
approach,
which
is
delete
with
broadcasting
NZ
ordering
no
diet,
Z
ordering
is
topics
feature
and
not
available
in
open
source
part,
but
I
just
wanted
to
you
know,
share
this
best
practice
as
well
in
case
if
you
are
running
it
on
data
breaks.
So,
while
deleting
records
for
GG
GD,
P,
R
and
C
CPA,
it
is
important
to
reason
about
the
amount
of
data
from
the
target
table
that
you
are
deleting.
D
For
example,
you
may
find
that
the
data
you
have
to
delete
is
spread
across
90%
of
the
files
and
therefore
you
may
end
up
rewriting
more
than
90%
of
the
data.
To
avoid
situations
like
this,
it
will
be
crucial
that
you
see
order
your
data
by
relevant
key
columns,
so
here
I
am
using
customer
ID
to
the
order
by
before
running
delete.
D
So
this
is
going
to
reduce
the
number
of
files
touched
and
it
can
make
my
V
writes
much
more
efficient.
So
let's
find
the
matching
records
now
and
observe
that
it
took
twelve
point
three
seconds
and
then
deleting
actually
quickly
deleting
actually
took
one
point,
eight
minutes.
But
what
is
important
here
to
note
is
when
you
run
Z
ordering
it
actually
reduces
the
number
of
partitions
as
well,
so
that
matters
because
it's
helpful
to
have
like
less
partitions
for
performance
and
optimization
purposes.
D
So
hopefully,
this
gives
good
idea
on
techniques
for
performance
improvements
when
you
are
handling
delete
requests.
So
those
are
three
of
the
approaches.
Now.
Let's
talk
about
audit
and
transaction
logs,
so
darshan
and
Justin
already
covered
a
little
bit
of
this.
But
many
organization
requires
auditing
capability
again
for
compliance
reasons
to
understand
who
performed
what
and
what
the
transactional
history
of
data
is.
So
to
do
this
in
Delta.
D
You
can
use
history
command
to
view
tables
history,
so
I'm
running
this
history
command
on
my
customer
table
and
you
can
see
that
all
the
changes
to
the
Delta
people
are
recorded
as
US
transactions.
So
here
there
are
granular
logs
for
each
transaction,
like
timestamp
user
information,
operational
parameters.
D
Notebook
which
was
used
to
compute
this
and
then
other
than
for
me,
our
isolation
level,
which
is
really
important
and
an
operational
matrix.
So
using
this
operation
that
matrix,
you
can
compare
the
benchmarking
and
performance
matrix
between
different
methods,
so
really
good,
really
good
matrix.
What
Delta
does
is
it
just
records
commits
in
transaction
locks
so
as
we
right
into
Delta
table
or
directory,
every
operation
is
automatically
version,
so
this
not
only
helps
with
audit
but
also
gathering
metrics
on
Delta
T
per
operations
for
performance,
benchmarking.
D
Now,
let's
talk
about
vacuum,
as
darshan
mentioned
before
vacuum
is
useful
to
delete
the
data
for
from
your
storage
completely
for
compliance
reasons.
So
this
operation
will
let
you
delete
data
that
delete
the
files
that
have
already
been
removed
from
our
table
for
a
certain
amount
of
time
by
default.
Vacuum
retains
all
the
data
needed
for
last
seven
days,
but
you
can
set
the
retention
policy
by
changing
the
value
here.
So
here
is
the
retention
hours
and,
as
you
see
like
here,
this
is
keeping
the
transaction
history
and
then
I
as
I
perform
vacuum.
D
It's
going
to
delete
my
files
from
the
from
the
table
I
mentioned
about,
and
then
I
want
to
make
sure
that
you
know
vacuum
is
actually
deleting
my
files,
so
this
this
command
is
already
run
before,
but
now
it
is
failing
because
I
already
performed
vacuum,
and
this
will
actually
tell
you
that
this
failed
because
it
couldn't
find
the
records
in
the
in
the
existing
table.
So
Delta
provides
a
very
good
log
log
capability
as
well,
so
for
troubleshooting.
You
can
see
like
what
the
errors
are.
D
So
this
is
particularly
because
it
cannot
recover
the
records
anymore
cool
on
so
one
more
thing
which
darshan
and
Justin
already
covered
is
setting
up
the
retention
policy
with
your
cloud
provider.
So
if
you
created
Delta
tapirs
using
spark
api's
to
rewrite
non-party
files
to
data
you're
in
draw
data
may
still
be
somewhere
containing
personal
information
that
you
have
deleted
or
anonymized.
D
D
So
the
Shin
kind
of
already
gave
you
an
overview
of
you
know
if
you
have
like
full.
If
you
have
original
tableware
personal
identifiable
information
about
the
custom
is
located.
A
best
practice
is
to
separate
out
columns
into
PII
and
non
ti-84.
Let's
see
it
by
example,
so
this
is
my
original
table
and
it
contains
personal
identifiable
information.
So
there
is
first
name
last
name
email
address.
D
These
are
some
of
the
identifiable
information.
Now
there
are
several
functions
that
can
be
used
for
one
Baker
flow.
Graphic,
hashing,
shavon
and
shadow
are
two
such
functions,
which
are
readily
available
in
spark
here.
I
am
using
shadow
function
to
anonymize
to
pseudonymous
my
email
address
column,
I'm
pseudonym
izing
only
one
column,
because
I
don't
need
first
name
and
last
name
pseudonym
eyes
and
I'm
gonna
separate
that
in
a
customer
lookup
table
so
once
I
run
that
look
at
look
at
the
records
that
we
are
getting.
D
Let's
check
both
the
tables
now
just
to
make
sure
that
we
don't
those
columns.
So
as
you
see
after
after
that
data,
munging
I
already
removed
the
identifiable
information
from
my
customer
sudo
table,
and
it
only
contains
the
email
address
pseudonym
and
in
my
customer
lookup
table
there
is
email
pseudonym
which
links
to
the
original
table
and
then
email
address
first
name
and
last
name
of
the
customers.
D
Let's
say
you
get
a
delete
request
so
when
there
is
a
delete
request
to
forget
information
about
a
particular
customer,
I'm
gonna
I'm
gonna
find
it
from
my
database
like
what,
if
it
exists
in
my
custom,
a
lookup
table
so
I
ran
that
query
and
then
I
found
that
this
is
the
record
which
came
up.
I'm
gonna
delete
that
record
from
my
customer
lookup
table.
So
once
I
delete
that
I'm
gonna
query
it
once
I
believe
that
it's
gone
forever.
D
You
won't
be
able
to
recover
it
back,
but
let's
see
if
in
my
pseudo
table
there
was
still
a
link
of
email
address,
pseudonym,
so
I
see
that
now
I
I
can
I
can
find
that
email
address
pseudonym.
But
if
I
queried
this
table
again,
it's
gone
forever
from
here.
So
in
case,
if
you
have
a
request
where
all
the
records
needs
to
be
forgotten,
you
can
just
delete,
run
the
delete
on
the
main
table
as
well,
so
it
won't
even
have
the
non-identifiable
data.
D
D
All
right,
so
we
saw
one
recommendation
now.
Another
recommendation
is
to
improve
your
query
performance.
In
addition
to
increasing
the
broadcast
threshold
and
co-authoring
capabilities,
there
are
several
other
methods
that
you
can
improve.
Your
query
performance
with.
So
first
one
is
the
compaction
for
improving,
read
performance.
D
Let's
see,
if
you
are
continuously
writing
data
to
a
Delta
table,
it
will
over
time
accumulate
a
large
number
of
files,
especially
if
you
add
data
in
small
batches.
This
can
have
an
adverse
effect
on
efficiency
of
table
reads
and
it
can
also
affect
the
performance
of
your
file
system.
So
you
can
compact
a
file
by
ABI
partitioning
to
a
smaller
number
of
files.
This
would
ensure
that
concurrent
operations
aren't
minimally
affected
due
to
the
compaction
operation.
In
addition
to
this,
you
can
also
add
data
change
option.
What
this
data
change
option
do
false.
D
What
this
indicates
is
that
the
operation
does
not
change
the
data
it
only
rearranges.
The
data
layout
second
method
is
auto
optimize,
I'm
gonna,
so
this
is
only
offered
in
tiga
bricks
and
also
like
runs
on
version,
I,
think
5.5
or
a
pow.
So
I,
don't
optimize
consist
of
two
complementary
features.
First,
one
is
optimized
writes
and
second
one
is
auto
compaction.
D
So
with
optimized
writes
here,
data
bricks
can
actually
dynamically
optimize
a
party
spot
partition
sizes
based
on
actual
data,
and
it
will
read
it
will
attempt
to
write
out
128
megabytes
files
for
each
table,
partition.
That
is
like
approximate
sizing,
but
it
can
be
on
your
data
set
characteristics
to
ensure
that
all
the
delta
T
burs
have
features
enabled
already
by
default.
D
You
can
set
those
parameters,
but
if
you
want
to
run
it
like
session-based,
you
can
specify
the
table
properties
like
here
cool,
and
so
once
you
get
once
you
once
you
complete
the
Auto
optimize.
There
is
also
this
factor
called
Auto
compaction.
So
after
an
individual
right
data
breaks
checks
if
the
files
can
be
further
compacted
and
runs
a
quick
optimized
job
that
you
know,
128
megabytes
file
sizes
to
further
compact
files
for
partition.
So
you
so
you
receive
smaller
number
of
files
yeah.
So.
B
D
I
think
that's
all
so,
once
you
delete
it,
you
can
see
that
it
will
be
deleted.
It
will
be.
You
can
retain
the
record
in
12
seconds
and
similarly
you
can
run
the
delete
operation
as
well
so
on
yeah.
So
that's
about.
Hopefully
that
gave
a
good
overview
of
the
different
methods
and
you
know
how
you
can
improve
the
efficiency
of
Delta
and
ticket
operations.
So
thank
you
so
much
for
joining
our
Tech,
Talk
and
I
think
we
can
open
for
questions.
Okay,.
B
Great,
so
it
looks
like
a
number
of
the
questions
is
coming
in
or
questions
for
me.
Just
a
disclaimer
X
and
the
lawyer
at
the
top,
not
actually
providing
legal
advice.
It
may
sound
like
legal
advice,
but
you'll
want
to
obviously
check
stuff
with
your
own
lawyers.
The
first
question
that
was
asked,
though
it's
not
in
the
Q&A
section,
was
whether
or
not
the
GD
P
ours
enforce
in
the
United.
B
So
the
next
question
that
was
asked
is,
if
we're
encrypting
data,
is
it
a
type
of
suit
autumn
ization,
essentially
a
suit
autumn
is
a
ssin.
Is
any
identifier?
That's
not
externally
understandable
without
some
additional
info.
So
it's
personal
data
or
personal
information.
If
you
can
just
look
at
it
and
know
who
the
person
is
it's
pseudonymous
if
you've
converted
it
to
any
kind
of
ID
that
you
internally
still
know?
Who
is
because
you
have
the
ability
to
reverse
it
to
an
actual
person,
so
the
short
answer
is
yes.
B
B
The
next
question
is,
let
me
just
go
through
the
the
legal
stuff
and
you
guys
can
do
the
other
ones.
So
what
hash
algorithm
is
recommended
for
gdpr
select
CCPA,
it's
gonna
depend
on
your
actual
use
case,
though
I
would
caution
you
to
not
use
things
like
md5
or
sha-1,
because
they're
no
longer
considered
safe,
hashing
algorithms
and
then
there's
another
question
about
vacuuming.
Vacuuming
only
clears
that
data,
that's
been
expired
and
the
pieces
of
data
them
and,
oh
sorry,
that's
not
the
one
I
wanted
to
answer
the
last
one
is
when
we
delete
data.
B
The
deleted
records
are
still
available
in
the
transaction
logs
I'm,
assuming
it's
a
violation
of
CC
TGA
are
gdpr
until
we've
vacuumed,
that's
exactly
right.
So
the
reason
you
need
to
go
through
and
do
the
vacuum
command
is
because
otherwise
you
would
still
have
that
that
personal
data
and
you
wouldn't
have
actually
deleted
it.
B
Yeah,
that's
correct,
so
the
reason
that
you
want
to
do
it
in
some
timescale
is
generally
speaking.
You
don't
want
to
have
to
worry
about
necessarily
doing
these
things
manually
each
time
they
come
in,
if
you
have
it
30
days
or
less
you're
likely
to
always
be
in
compliance
as
a
consequence
of
that,
that's.
C
All
right,
I
have
some
technical
questions.
I
can
take
them
on
the
first
one
is:
do
you
have
sources
other
than
s3,
like
my
sequel?
Right
will
DB,
etc.
There
are
multiple
ways
in
which
we
can
use
plumbing
to
Cana,
have
disparate
sources,
syncing
data
into
Delta,
lake
or
and
in
general,
on
data
breaks.
We
support
multi
cluster
writes
on
Delta
tables,
so
that
even
actually
simplifies
the
ulceration
simply.
Obviously
this
is
a
lot
more
to
it.
You
know
feel
free
to
come.
I'll
leave
me
a
note.
C
If
there
are
questions
on
that
one,
the
next
one
was
wouldn't
more
number
of
partitions,
be
better,
as
it
gives
the
ability
to
paralyze
and
compute
with
less
memory
resources.
So
there
will
be
more
number
of
steps
involved.
Well,
technically
speaking,
yes,
the
more
the
distribution,
the
better
for
you.
C
If
you
are
able
to
broadcast,
we
know
the
thought
is
that
if
you
have
lots
of
table
data
that
you
can't
broadcast
and
you
can
spread
it
over
multiple
partitions,
you
are
basically
avoiding
the
shortwave
join.
So
that
was
the
third
thing.
Oh,
there
was
a
question
about
vacuuming.
Yes,
vacuum
cleans
up
expired
data
or
invalidated
data
files,
so,
depending
on
where
your
data
is
in
terms
of
compliance
use
vacuum,
what's
the
threshold
for
the
size
of
it,
I
should
prefer
for
the
pseudo
group
of
table
to
simple
simply
deleting
IDs,
using
the
delete
command.
C
C
C
Think
of
streaming
as
if
you
think
of
batch
there
is,
we
don't
make
a
disturbin
solutioning
out
for
streaming
solutions
or
bad
solutions,
I
think
of
a
stream
as
a
batch
and
I
think
of
a
batch
as
a
stream.
So
from
that
perspective,
if
you
are
streaming
from
an
external
source,
you're
sinking
into
a
Delta
table,
deleting
all
that
Delta
table
is
fine
too,
unless
you
want
to
take
it
on
the
fly
in
the
stream.
C
C
B
You
and
then
there
was
one
more
question
that
was
just
in
the
webinar
chat.
Will
the
audit
logs
also
capture
who
query
what
the
audit
logs
don't
actually
contain
the
command
contents
themselves,
so
you'll
have
to
sort
of
speak
with
our
technical
folks
to
get
a
little
bit
deeper
on
that,
but
it's
out
of
the
scope
for
the
purposes
of
this
session.
B
D
A
Yeah
and
I'll
make
sure
to
drop
that
github
link
in
the
YouTube
description,
I
believe
it's
already
there,
but
I'm
gonna
double
check
after
to
make
sure
it
is.
You
know
the
resources
and
then
everyone
that
attended
the
zoo,
meetup
will
get
an
email
or
follow-up
email
with
all
the
links
as
well,
and
then
just
a
quick
call
out
I
wanted
to
let
everyone
know.