►
From YouTube: Multi-Table Transactions with LakeFS and Delta Lake
Description
This is a recording of the live tech talk hosted on August 31, 2021.
Join us for a tech talk about LakeFS and Delta Lake. Paul Singman from Treeverse and Denny Lee from Databricks discuss multi-table transactions with LakeFS and Delta Lake.
LakeFS enables you to manage your data lake the way you manage your code, allowing for a collaborative development environment and CI/CD deployment of data. Delta Lake is an open-source project that enables building a Lakehouse architecture on top of existing storage systems such as S3, ADLS, GCS, and HDFS. In this session, Paul and Denny will discuss LakeFS and Delta Lake and the integration of LakeFS and Delta Lake simplifies your multi-table pipelines.
Speakers:
Paul leads DevRel for lakeFS, after several years as a ML engineer at Equinox Fitness. He enjoys contextualizing the latest data trends and technologies in blog posts and talks, instead of getting caught up in the hype surrounding specific tools.
Denny Lee is a developer advocate at Databricks, where he works on Delta Lake, Apache Spark, Data Sciences, and Healthcare Life Sciences. He has previously built enterprise DW/BI and big data systems at Microsoft including Azure Cosmos DB, Project Isotope (HDInsight), and SQL Server as well as the Senior Director of Data Sciences Engineering at SAP Concur. Denny holds a Masters in Biomedical Informatics from Oregon Health Sciences University. Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. https://databricks.com/databricks-named-leader-by-gartner
A
A
So
I
wanted
to
mention
that
this
is
part
of
the
data
plus
ai
online
meetup
group.
So
if
you're
joining
us
on
youtube
or
linkedin
I'll,
send
the
link
to
our
meetup
group
in
both
of
those
chat
spaces,
I'd
love
you
for
you
to
join
the
group,
that's
where
you'll
be
updated
on
all
the
upcoming
sessions
and
you
can
join
the
zoom
broadcast
if
you'd
like
and
interact
with
us
there.
A
So
also
too,
if
you're
joining
us
on
zoom,
I
invite
you
to
check
out
our
youtube
channel,
so
that
has
all
of
the
recordings
of
all
of
our
tech
talk
sessions.
So
I
do.
I
have
a
playlist
there
that
has
specifically
you
know
the
events
that
we
run
on
the
meetup
group
and
then
we
also
have
like
recorded
sessions
from
the
data
plus
ai
summit
and
also
a
lot
of
other
awesome
videos
and
and
technical
sessions
there
too.
A
So
I
invite
you
to
check
that
out
and
then
join
us
on
linkedin,
if
you'd
like
to
so
I
wanted
to
give
a
quick
shout
out.
We
have
a
podcast
series
and
denny's.
Actually,
one
of
is
one
of
the
co-hosts,
along
with
brooke
wennig
from
databricks
they're
on
their
second
season.
So
if
you're
into
podcasts-
and
you
want
to
learn
about
machine
learning,
I
invite
you
to
check
that
out.
A
So
it's
available
wherever
you
like
to
get
your
podcast
and
same
thing
when
I'm
done
with
my
intro
slides
I'll,
put
the
link
in
the
chat,
so
you
guys
can
check
it
out
and
then
we
have
a
guest
blog
program.
So
I
invite
you
to
check
out
this
page
we're
looking.
We
want
to
see
what
you're
working
on
and,
if
you're
interested
in
writing
you
know,
even
if
you're
a
beginner,
you
haven't
written
any
blog
posts,
but
you
have
something
you'd
like
to
share.
A
I
invite
you
to
reach
out
to
me,
so
you
all
can
reach
out
to
me
directly
on
linkedin.
I'm
help
manage
this
program
or
you're
also
welcome
to
submit
your
idea
through
through
that
webpage.
So
there's
more
information
there
if
you'd
like
to
kind
of
check
it
out
and
think
about
writing
I'd
love
to
hear
from
you
and
then
one
other
program
I
wanted
to
share
and
we
just
launched
it
a
few
weeks
and
we're
really
excited
about
it
is
called
the
databricks
beacons.
A
These
are
all
individuals
in
the
community
and
it's
our
way
of
saying
thank
you
and
sharing
the
awesome
things
that
they're
doing
their
contributions.
A
lot
of
them
write
blog
posts.
A
lot
of
them
do
videos
they
share.
They
write
books,
they
teach
some
of
them
are
in
your
university
alliance
program.
But
really
these
are
members
of
the
community
and
it's
it's
our
way
of
recognizing
the
awesome
work
that
they're
doing
related
to
data
bricks
and
the
open
source
projects,
including
spark
delta
lake
mlflow.
A
So
I
invite
you
to
check
out
the
page
that
we
have
on
our
website
and
we
have
profile
pages
of
all
of
them,
so
you
can
see
their
websites
their
books,
lots
of
awesome
content,
lots
of
technical
materials
to
help
you
learn
about
databricks
in
our
open
source
projects
and
also
too
we're
looking
for
community
nominations.
So
if
you
see
somebody
or
you
know
of
somebody
in
the
community-
that's
doing
great
work,
you
know
it
could
be
organizing
meetups,
giving
tech
talks,
mentoring,
answering
questions
on
stack,
overflow
or
forums.
A
Please
reach
out.
I
manage
this
program
so
again
you
can
I'm
you.
I
welcome
messages
on
linkedin
or
again.
You
can
visit
the
page
and
kind
of
check
out
more
about
the
program
and
submit
a
nomination
if
you'd
like
to
all
right
and
with
that
my
intro
slides
are
done
and
just
a
quick
reminder
that
this
session
will
be
recorded,
so
it'll
be
available
on
youtube
right
after
we're
we're
finished
so
I'll
drop
that
link
in
the
chat
spaces
as
well.
A
B
Thank
you
very
much
all
right.
Let's
see
if
I
can
present
my
screen.
That
would
always
be
helpful
in
the
presentation.
Last
time
I
checked.
So
let's
do
that
alright,
so
we
have
slides,
go
figure,
hi
everybody
just
in
case
you
forgot
what
session
here,
it's
multi-table
transactions
with
lakefest
at
delta
lake
paul
and
myself
had
already
chatted
a
little
bit,
but
by
the
same
token,
we
want
to
know
who,
if
you
want
to
know
who
we
are,
my
name
is
denny.
B
I'm
a
developer
advocate
at
databricks
a
long
time
geek
at
databricks
and
before
that
was
at
microsoft,
building
really
large
scale
systems
that
was
part
of
sql
server
and
and
other
groups
there.
So
hopefully
I
can
be
provide
some
technical
context,
or
at
least
play
really
well
really
well
on
tv
here.
If
you
want
to
know
more
about
delta
lake
by
all
means,
please
check
out
o'reilly's
delta
lake,
the
definitive
guide,
early
release.
B
C
Hey
everyone,
my
name
is
paul
very
excited
to
be
here.
I
am
also
a
developer
advocate
at
the,
but
at
lakefest
I
previously
led
a
platform
engineering
team
at
equinox
fitness,
where
I
worked
on
a
number
of
what
I
would
describe
as
operational
and
user-facing
analytic
projects
or
services,
and
a
little
about
me
outside
of
work.
I
like
to
run
golf
play
ping
pong
and
back
emin.
B
Perfect,
thank
you,
so
all
right,
we're
gonna
dive
right
into
it.
So
there's
a
bunch
of
like
fun,
slides,
providing
some
context
in
terms
of
what
is
the
promise
of
a
data
lake,
okay
and
the
context
here,
basically,
is
that
you
know
when
we,
especially
because
I
was
actually
part
of
the
team
that
was
actually
had
created.
What
is
now
known
as
azure
hdinsight.
I
was
where
we're
saying
hey.
You
know
we
were
pitching
the
idea
of
the
promise
of
the
date.
B
I
just
come
from
the
sql
server
team
and
we're
saying
like
yeah
yeah.
We
need
to
put
everything
in
dave
like
we
can't
put
everything
in
a
structured,
sql
database.
It
just
can't
work
anymore.
We
can't
distribute
it.
You
know
none
of
that
funky
stuff
right.
So,
let's
that's
it.
Let's
collect
everything,
let's
store
it
all
and
then
because
we
store
it
all
bam,
we
can
do
data
science,
machine
learning
and
we're
done
right.
Everything
is
solved
by
putting
everything
on
data
lake
and
everything's
texting
everything's
cool
right.
B
So
if
you
I'm
presuming
because
I'm
presenting,
I
can't
tell
if
anybody's,
raising
their
hands,
but
if
I
could
see
you,
I
hopefully
you're
grinning
and
wincing
in
pain.
With
that
statement,
I
just
made
you're
going
like
what
no
dude
that's
not
even
remotely
close,
because
the
problem
is
garbage
in
garbage
out
right.
That's
ultimately
the
problem
right.
You
collect
everything,
that's
great,
but
you're,
also
potentially
collecting
garbage,
and
in
fact
not
even
potentially,
you
usually
are
okay.
So
what
does
a
typical
data
lake
project?
Look
like
right.
B
If
you're
you're
you
we,
we
dream
that
dream
of
the
data
lake
right,
okay!
Well,
the
evolution
of
cutting
edge
like
it's
like
okay,
I've
got
events
that
are
coming
into,
I'm
just
using
kafka
as
an
example.
This
is
not
a
knock
on
anybody
else.
Just
happen
to
use
kafka
right,
so
I'm
going
to
use
spark
to
do
streaming
analytics,
I'm
going
to
go
ahead
and
use
spark
to
go
ahead
and
store
the
data
for
your
data
like
because
I
need
to
be
able
to
differentiate
between
my
streaming
data
and
my
historical
date.
B
Well
then,
that
means
boom.
I
gotta
create
a
lambda
architecture.
Okay,
the
whole
premise
of
that
is
basically
I
need
to
go
ahead
and
have
two
streams
to
process
the
same
set
of
data,
one
for
historical
purposes
and
one
for
streaming
purposes:
okay
and
then
I'll
run
yet
another
spark
job
to
take
the
data
from
the
data
lake
and
then
for
ai
reporting
yay.
B
That's
easy
right,
again,
sarcasm,
okay!
So,
but
then,
because
you
have
this
right,
you
need
to
do
data
validation,
because
you
want
to
make
sure
any
of
the
processing
that's
done
within
your
land
architecture.
The
streaming
actually
matches
the
data,
that's
being
processed
in
your
batch
right.
You
want
to
make
sure
these
two
are
matching
otherwise
you're
going
to
be
reporting
different
data,
which
would
be
really
really
bad,
okay,
but
then
how
about
if
there
are
mistakes
and
failures
to
this
process?
Right?
Okay,
so,
okay?
B
Well
I'm
going
to
partition
my
data
in
my
data
lake,
I'm
going
to
reprocess
the
data,
my
data
lake
and
then
no
problem
at
all
I'll,
just
reprocess
the
data
and
everything's
solved.
So
no
problem,
okay,
everything's
fixed
right,
but
then
I
need
to
do.
Updates,
okay!
Well,
that
means
I
need
to
run,
updates
and
merge
against
my
data
link
and,
if
you're,
using
traditional
distributed
systems,
whether
it's
spark
or
anything
else
updates
or
mergers
typically
mean
running
massively
large
select
statements,
insert
statements
and
basically
yourself
manually,
merging
all
this
stuff
together
right.
B
Okay,
so
this
seems
a
little
complicated
all
right
and
because
it
is
right
and
you
basically
the
context
you're
wasting
time
and
money
you're
solving
systems,
problems
you're,
actually
not
solving
your
data
problems
now
right
and
you're,
not
extracting
the
value
from
your
data.
That's
you're
not
doing
these
things,
and
so
what
the
big
problems
when
it
comes
to
data
lakes
in
general.
B
Is
that
there's
no
atomicity,
I.e,
there's
no
when
you-
and
this
is
coming
the
reason
I
brought
up
the
database
thing
wasn't
just
because
I
was
trying
to
say,
like
ooh,
I
refused
to
work
for
sql
server.
The
reason
I
called
it
up
is
because
when
we
went
in
and
shifted
from
databases
to
data
lakes,
we
thought
the
data
lakes
would
solve
some
of
our
problems
and
we
summarily
forgot
all
about
the
fact
that
this
is
concept
of
asset
transactions
that
we
got
used
to
working
with
databases.
B
B
You've
got
a
lot
of
orphan
files
and
you
have
to
clean
those
up
somehow
right
and
you
have
to
know
which
files
to
clean
up
and
then
there's
no
quality
enforcement,
because
there's
actually
no
mechanism
of
saying
okay,
this
table
schema
is
supposed
to
be
these
columns
with
those
strings
and
insert
whatever
else.
None
of
that's
actually
in
play
here
right
and
there's
no
consistent
isolation.
So
the
key
call
here:
no
acid
transactions.
Okay,
so
that's
what
oss
delta
lake
does
okay
and
provides
you
that
asset
transaction
protection
for
your
data
lake?
B
B
So
sometimes
it
is
appropriate
to
let
your
data
like
say
table
x
does
need
to
actually
add
an
additional
two
columns,
but
you
get
to
control
that
process
as
opposed
to
the
system
arbitrarily
running
over
it.
Okay
and
then
remember
how
I
talked
about
update,
merge
and
then
for
that
matter,
deletes
right.
Dml
operations,
data
modification,
language,
mod
operations,
the
ability
for
is
delta
lake.
It
gives
you
that
ability
to
do
that,
so
basically
extending
the
scala,
sql
and
python
apis
of
spark.
It
allows
you
to
go.
B
B
That's
great,
but
I've
been
also
very
spark:
centric,
okay
and
that's
a
really
key
important
aspect
of
delta
lake.
Yes,
delta
lake
came
from
sparks
so
we're
very
proud
of
that
heritage,
but
that
isn't
where
we're
going
we're
we're
not
limit
we're,
not
at
all
limited
there.
Okay,
the
the
the
as
part
of
a
road
up,
we
just
just
published
two
weeks
ago
right,
there's
the
delta
standalone
reader
delta
style
writer.
B
But
the
idea
is
that
you
have
multiple
languages
that
can
go
talk
to
it
right,
and
so
we
also
have
flink
delta
source,
I.e.
Flink
will
be
able
to
read
from
delta,
but
that's
currently
targeted
for
q4
calendar
year
q4
and
the
flink,
the
delta
sync,
which
means
flinking
right
to
delta,
we're
currently
targeting
q1
of
of
early
next
year
right
pulsar,
the
same
thing:
apache
pulsar
spark
3.2
support
hive
three
connector.
B
It's
super
important
and
so
that
that
we
are
able
to
get
the
delta
ecosystem
to
work
with
every
system
out
there.
That
isn't
spark,
and
that's
precisely
why
we're
talking
in
these
sessions
like
this
and
by
the
way,
if
you're
interested
join
the
delta
slack
users,
a
group
delta
user,
slack
group.
Excuse
me:
we
actually
have
community
meetings.
Every
two
weeks
I've
called
out
trino,
flink,
nessie
lake
fs,
pulsar,
rust
and
core
right
now,
oh
yeah.
I
did
forget
to
put
presto
inside
here.
B
So
yes,
there's
actually
presto
actually
has
the
same
timeline
as
here
too,
by
the
way,
so
my
bad
on
for
getting
that
slide.
But
that's
the
call
out.
We
have
a
lot
of
cool
things,
and
so
now,
obviously
I
want
to
go
ahead
and
talk
more
about
the
lake
lake
fest.
This
ability
for
us
to
do
multi-table
transactions
now
fir
we're
going
to
talk
initially
about
single
table
consistency,
first,
okay,
and
so
that's
what
the
purpose
of
this
demo
that
we're
going
to
showcase
now
super
quick,
because
I
want.
I
definitely
want
to
showcase.
B
That
would
have
been
really
bad
okay,
so
this
notebook
is
available
by
the
way
I
got
a
ping
from
some
folks.
I
apologize
haven't
been
an
answer
but
currently
presenting,
but
for
both
the
slides
as
well
as
this
notebook
are
going
to
be
publicly
available,
we're
going
to
attach
it
to
both
the
linkedin
and
to
the
youtube
and
also
there's
a
database
tech
talks,
github
repo,
which
we
put
all
of
the
stuff
in
so
you
can
absolutely
get
used
to
all
this
okay.
B
So
all
right,
so
this
notebook
I'm
just
going
to
skip
past
this
part.
We
just
simply
had
just
to
give
some
context.
We
had
a
set
of
data,
it's
a
loan
by
state
data,
it's
from
kaggle,
it's
a
data
breakday
set,
so
you
don't
actually
have
to
load
it.
So
I
just
ran
a
bunch
of
stuff
where,
basically,
I
created
a
parquet
version
of
the
table
and
I
created
a
delta
version
table.
B
Okay,
that's
it
and
by
the
way,
if
you
have
existing
part
k
tables,
you
can
just
literally
run
the
command
convert
to
delta,
and
you
can
convert
that.
Okay,
I
ran
into
about
you,
know
30
minutes
ago,
so
it's
not
relatively
new
so
far,
okay,
so
I
also
have
these
streaming
functions.
The
whole
purpose
is
I'm
going
to
be
streaming
data
into
the
parquet
table
and
into
the
delta
table.
This
is
the
part
of
the
demo
saying
that
why
asset
transactions
are
important
and
how
delta
is
a
streaming
of
batch?
Sync,
okay.
B
So
now-
and
this
is
just
to
show
the
fact
that
the
par
k,
data
and
the
delta
data
are
exactly
the
same
in
this
part
of
demo-
I'm
just
showcasing
the
fact
that
I'm
using
parque
as
a
header
and
delta,
as
a
header
just
to
say
you
can
literally
query
by
the
file
string
the
file
path.
Excuse
me,
by
the
same
token,
if
you
merge
with
a
catalog
of
some
type,
yes,
you
can
still
query
your
delta
tables
or
parquet
tables
by
just
using
the
catalog
as
opposed
to
the
file
path.
B
You
have
both
options,
okay
and
so
all
right
now.
Let
me
go
ahead
and
run
this,
so
this
is
the
function
where
I'm
just
going
to
go
ahead
and
run
load
data
into
a
parquet
table.
Okay,
so
now
streaming
initializing,
it's
important.
If
you're
going
to
run
this
demo
yourself,
you
want
to
wait
a
couple
seconds
just
for
the
streaming
to
initialize,
and
so
you
can
definitely
tell
by
looking
at
the
ui
here
basically
that
okay,
now
data
is
going
in.
That's
why
that
is
73.2
records
per
second
and
the
batch
duration's
about
1.2
seconds.
B
Okay,
so
far
so
good,
and
if
I
run
this
query
next
you'll
notice
that
basically
data
is
going
in
190
rows
so
far,
so
good
all
right.
But
how
about?
If
I
try
to
run
a
second
stream
into
that
data
all
right,
so
I'm
going
to
run
the
exact
same
thing,
I'm
just
simply
specifying
stream
number
two,
but
there
should
be
no
difference
right.
So
I'll,
take
a
look
at
the
count
and.
A
B
Steadily
going
up,
but
that's
probably
from
the
first
one
here
as
you
can
tell
from
the
input
processing
rate
and
the
batch
duration,
but
I
look
at
this
one.
You
notice
that
while
there
is
a
batch
duration,
it's
only
380
milliseconds
and
there's
no
data
going
in
and
that's
actually
an
issue.
That's
correla!
That's
specific
to
the
fact
that
when
you're
dealing
with
things
like
parquet
or
any
other
quote-unquote
regular
storage
system
file
system,
you
actually
you're
going
to
run
into
issues.
B
Basically
where,
because
you
don't
have
asset
transactions
to
protect
the
data,
you
actually
can't
run
multiple
streams
to
write
to
the
same
table.
It's
not
possible
and
there's
also
the
issue
of
potentially
running
into
corruptions
of
that
data.
Okay,
so
I'm
just
going
to
let
it
run
a
couple
more
seconds
just
to
so
you
can
see
what's
going
on,
but
yes,
rows
are
still
going
in,
but
there's
nothing
happening
on
the
second
stream.
It's
all
happening
on
the
first
string.
Okay.
So
now
let
me
stop
all
the
streams
and
continue
the
process.
B
Now
I'm
going
to
do
the
exact
same
exercise,
except
I'm
going
to
use
delta
lake
to
do
this.
Okay,
so
so
we'll
start
this
process
again
all
right
same
function.
I
just
simply
specified
the
table
formats
delta,
as
opposed
to
parquet,
and
I
specified
a
basically
a
different
path
for
where
I
want
to
store
this
information.
That's
pretty
much
it
so
again.
We're
processing
data
data
is
going
in
same
context.
B
Okay,
and
as
this
opens
up
right
what
you'll
notice
already
there's
records
going
in
you'll
notice,
here's
stream
three
stream
two
is
still
trying
to
run.
I
think,
but
it
happens.
Well
there.
Oh
there
you
go
44.9
seconds
records
per
second
okay,
and
if
you
recall,
if
we
go
back
here,
this
is
what
the
stream
two
for
per
k
was
no
no
new
rows
inside
here,
but
stream,
two
one
two
and
three.
Hopefully
I
didn't
just
give
you
guys
vertigo
for
spinning
up
and
down
the
notebook
so
quickly.
B
Basically,
there's
als
rows
constantly
going
in,
and
so
for
that
matter.
I
can
do
a
quick
count
and
I'll
see
the
row
count
and
it'll
be
significantly
larger,
okay,
even
for
the
less
same
amount
of
time.
Okay,
let
me
stop
all
streams
demos
almost
over
by
the
way,
ladies
and
gentlemen,
okay
and
so
the
when
we
review
the
streams
and
we
look
at
parquet.
What
you'll
notice
is
that
there's,
basically
one
bar
chart?
Okay
and
the
one
bar
here
basically
will
showcase
I'll
go
figure.
B
This
prayer
would
take
the
longest
okay
there's
about
almost
a
thousand
rows
and
there's
only
one
stream
in
there.
What
you'll
find
interesting
is
that
when
I
run
the
delta
table,
not
only
are
there
three
bars
because
each
to
represent
the
three
streams
zero.
You
know
one
two
and
three,
but
there's
also
a
zeroth
stream.
B
Now
this
happened
because
the
initial
data
wasn't
actually
it
wasn't.
Actually
a
stream
this
day
was
50
rows
that
I
initially
had
put
into
the
table.
If
you
query
back
to
the
top
of
it
that
that's
actually
how
we
started
well,
there
were
50
rows
of
data,
and
then
what
happened
is
that
okay?
Well,
when
there's
a
new
stream
there's
new
data
coming
in
okay?
No
problem!
So
that's
why
you
see
one
two
three,
but
why
didn't
parquet
knows
that?
B
Well,
what
happened
is
that
there's
actually
technically
corruption
in
data
it
actually
overwrote
the
initial
50
rows
of
data?
That's
why
you
don't
see
a
stream
zero
here
stream,
one
actually
overwrote
stream
zero.
So
that's
bad
right,
and
so
this
is
what
we're.
This
is
just
a
small
example
of
that
idea
of
schema
enforcement
and
evolution
over
time.
B
If
you
need
evolution
as
well,
and
so
this
is
just
a
relatively
simplistic
example,
but
this
is
that
concept
of
single
table
transactional
consistency
that
delta,
like
provides
now
I'm
gonna
before
I
shift
it
over
to
you
paul.
So
just
you
know
10
more
seconds.
This
notebook
actually
has
all
other
examples.
Like
audit
audit
history,
I'm
not
going
to
run
all
these
because
I
really
wanted
to
go
focus
on
the
lake
fs
part
of
things.
But
the
point
is
that
there's
all
this
this
demo-
notebook
like
I
said
I'll-
give
it
out.
B
You
can
use
it.
It
actually
has
all
the
other
features
of
delta
lake
all
packed
in
there
and
if
you're
wondering
yes,
you
can
run
this.
I
have
a
the
notebook
is
running
in
database.
You
could
run
a
database
community
edition
which
is
free,
but
you
can
take
all
the
code
base
and
run
it
in
delta
oss
without
any
problem
as
well.
Okay,
so
saying
that
I'm
going
to
stop
sharing
paul,
why
don't
we
switch
over
to
you
and
we
can
talk
about
multi-table
transaction
consistency.
C
C
Well,
if
you
install
a
cfs
create
a
repo
as
we
call
it,
you'll
you'll
see
that
we,
we
borrow
essentially
as
many
get
terms
as
we
can,
so
you
install
a
cfs,
create
a
repository.
I
called
mine,
my
repo
in
this
diagram
and
add
your
delta
tables
into
it.
You
can
now
take
commits
over
the
repository,
which
saves
a
snapshot
of
all
your
tables
for
a
specific
point
in
time
that
you
can
reference
with
a
generated
commit
id.
C
From
multiple
tables,
so
the
way
this
works
is
first,
you
create
a
branch
which
is
a
it's
like
a
copy
of
your
main
data
set,
but
it's
a
purely
metadata
operation
that
doesn't
at
least
initially
copy
the
data
on
this
branch.
You
can
add
data,
know,
write
new
data
or
add
data
to
as
many
delta
tables
as
you
want
and
then
perform
a
merge
operation
to
save
the
changes
back
to
the
main
branch.
C
So
the
answer
is,
it's
all
metadata
and
you
know
knowing
this
audience.
I
think
this
metaphor
will
make
sense
so
very
similar
to
the
way
that
delta
lake
has
a
transaction
log,
which
is
a
small
piece
of
metadata
that
is
stored.
Alongside
your
data,
lakefest
similarly
manages
what
we
call
a
thin
layer
of
metadata.
C
This
is
a
two
layer
b
tree
that
keeps
track
of
which
files
are
contained
in
each
commit
essentially,
and
the
details
are
not
really
important.
Unless
you
really
want
to.
You
know
dive
into
the
details
which
you
can
check
out
our
blog
and
get
as
low
level
as
you
want,
but
this
thin
layer
of
metadata
wraps
your
object,
storage
to
provide
the
additional
data,
versioning
functionality
like
branching,
committing
and
merging
reverting,
etc.
C
So
the
way
this
works
is
there's
really
two
things
to
configure
or
change.
If
you
want
to
integrate
lakefs
and
delta,
this
is
kind
of
from
a
like
databricks
specific
configuration,
although
it
might
be
the
same
on
just
regular
spark.
So
first
are
a
couple
spark
configs
to
set
here's
the
really
like
the
the
main
ones
they
are
set
on
the
repo
level.
I
want
to
call
out
and
the
most
important
one
is
this
s3
api
endpoint.
C
C
C
C
We
published
it
like
a
couple
months
ago.
It
goes
into
a
bit
more
of
the
details
that
I'm
going
to
cover
of
how
to
set
up
the
like
validation
hook,
that'll,
succeed
or
fail.
The
merge,
so
actually
first
we
want
to
do-
is
go
into
our
delta
demo
repository
and
right
now,
there's
only
one
branch,
which
is
the
main
branch:
we're
going
to
create
a
new
branch
to
write
the
new
data
to
we'll
call
it
ingest.
C
C
I
didn't
confirm
that,
but
essentially
you
know
you
can
see
that
you
might
have
been
able
to
see
the
changes
we
just
saved.
So
let's
create
the
second
table.
This
is
the
fact
table
of
all
the
loans
we're
going
to
set
the
addresses
to
a
mostly
to
a
good
address
or
state
address
of
new
jersey,
but
we're
going
to
create
one
that
has
a
bad
state
address.
C
So
we
created
this
delta
table
loan
details
and
we're
going
to
insert
a
bad
record
that
has
nj
and
j
as
the
state
address
and
all
right.
This
has
been
added,
and
basically
we
want
to
do
now,
so
we
should
see
okay
great.
So
we
do
have
some
uncommitted
changes
now
on
the
ingestion
branch
of
this
repo.
Let's
go
ahead
and
commit
these.
C
I
will
note
that
I
am
using
the
lakefs
ui
for
the
commit
and
merge,
but
you
can.
Similarly,
you
know
we
have
a
cli.
You
can
run
all
these
sorts
of
operations
and
similar
to
what
denny
mentioned.
We
also
have
like,
like
rust,
bindings
that
has
a
python.
You
know
wrapper
to
create
like
a
a
python
api
as
well.
All
right,
so
we
we
committed
our
loan
tables
now,
let's
try
merging
them
from
the
main
branch
and
start
from
the
yeah
from
the
ingest
branch
into
the
main
branch.
C
C
So
on
the
actions
tab,
we
can
see
the
results
of
these
pre-merge
validation
checks.
The
one
we
just
ran
is
just
finished,
so
you
can
see
it
failed.
This
means
that
the
merge
operation
did
not
happen.
There
was
bad
data.
Any
data
that
the
you
know
that's
under
most
workflows
would
have
gotten
into
production
and
potentially
caused
all
these
errors.
C
Now
it
fails
and
isn't
exposed
on
the
main
branch
which
most
production
consumers
should
read
from,
and
what
we
can
do
is
very
simply
in
this
case,
just
go
back
into
our
notebook,
delete
the
bad
record
and
deleted
a
few
rows,
and
we
can,
let's,
let's
retry
the
merge,
so
do
the
merge
from
ingest
to
main
and
yes
so,
while
this
does
run
for
a
couple
seconds,
the
way
this
is
working
to
give
you
some
sense
is
we
have
this
separate
notebook,
which
is
the
validation
notebook
and
it
runs?
I
call
it
an
arbitrary.
C
You
know:
data
validation
or
quality
check,
so
you
could
put
any
type
of
query
you
want
here
here,
I'm
you
know
left
joining
the
two
tables
and
seeing
that
there's
no,
you
know,
there's
no
record
where
the
address
state
in
one
of
the
tables
is
null
which
would
indicate
that
there
was
a
state
that
doesn't
exist
in
the
other
table
and
there's
an
assertion
here.
So
you
know
that's
what
determines
it's
succeeding
or
failing,
so
we
can
go
back
into
like
if
sui
it
finished
and
it
passed
a
little
green
check
mark.
C
B
Oh,
that's
great
paul.
Thank
you
very
much.
This
is
super
helpful.
We
do
have
a
couple
questions
and
they're
actually
more
for
specifically
for
lake
fest,
so
yeah.
Why
don't
I
just
direct
them
to
you
and
the
the
original
question
is
I
assume
lake
fest
works
with
azure
storage
blobs
like
adl's
gen
2?
Is
that
can
you
please
confirm
that.
C
Yes,
I
confirm
it
and
yeah
so
the
the
object
store
underneath
lake
fest
can
be
azure.
It
can
be,
google,
obviously
s3
and
even
you
know
other
we
have
some
users
having
mineo
under
the
hood.
You
know
other
tools,
reading
and
writing
will
think
that
they're
reading
and
writing
from
s3-
I
guess,
based
on
the
api
but
lakefest
behind
the
scenes,
is
then
using
in
your
case
azure
to
actually
store.
B
Yep,
perfect
excellent
and
then
another
question
is:
do
we
have?
Are
there
apis
to
integrate
the
git
operations
within
the
notebooks
development
environment?
And
so
I
believe
that
you
had
already
answered
it,
but
I
figured
I'd
just
go
ahead
and
re-clarify.
The
question
which
is
basically
there
should
be.
There
are
excuse
me
that
should
be
there
are
apis
and
cli
calls
that
you
can
do
as
opposed
to
necessarily
doing
all
of
this
stuff
directly
within
the
ui
itself,
with
a
when
it
comes
to
lakefs.
Is
that
correct.
C
C
I
think
it
could
very
quickly
just
give
you
know,
people
the
I
the
the
picture
of
what
it
looks
like
so,
okay
yeah.
So
this
is
a
python
notebook.
We
have
a
pip
installable
lakefest
client.
These
are
not
real
aws
keys.
I
was
aware
of
that.
So,
once
you
can
set
up
this
little
bit
of
configuration,
you
can
do.
I
think
the
operations
that
you're
asking
like
branches
creep
branch,
for
example,
the
make
a
commit.
So
that's
that
is
certainly
something
you
can
do.
B
C
B
But
what
happens
when
you
have
two
concurrent
operations
working
on
the
same
set
of
records
like
your
read,
write,
read:
write
situations
through
lake
fest
in
this
case
right
for
forsake
argument
where
you've
got
like
doing
read,
write
across
multiple
tables,
so
each
individual
table
has
their
own
asset
transactions
that
have
been
protected
by
delta
lake,
but
then
now
you're
doing
read
and
write
for
two
concurrent
ones
through
like
fs.
How
does
that?
Actually?
A
C
You
know
that
what
technically
goes
on
behind
the
scenes-
I
maybe
don't
have
a
great
answer
to
right
now,
but
we
definitely
have
users
that
that
do
a
pretty
high
level
of
throughput.
You
know
concurrent
operations
happening
and
you
know
I
think
the
answer
is
they're,
just
they're,
ordered
by
by
the
lakefs
server
and.
C
B
Perfect
thanks
paul,
so
just
just
to
provide
a
little
context,
just
because
I
I
happen
to
know
a
little
bit
about
how
league
office
works.
The
don't
forget
also
the
the
fact
is
it
underneath
the
covers
lake
effect
is
using
a
b
tree
right.
So
basically,
it's
a
self-balancing
tree
very
common
within
database
systems,
so
in
other
words,
so
it
can
actually
figure
out
which
transaction
is
associated
with
the
transaction
at
that
higher
level,
because
at
the
git
branch
level,
and
so
you've
got
basically
the
gift
branch
level.
B
B
So
obviously,
at
some
point
you
we
can
figure
out
if
somebody's
going
to
be
able
to
push
the
system
both
systems
down
but
yeah,
but
by
the
same
token
you
know
we
feel
comfortable,
at
least
at
least
I
can
say
from
a
delta
lake
perspective,
we
feel
comfortable,
saying
that
yeah
yeah
we've,
we
have
pretty
large
customers
and
with
massively
large
petabytes
of
data
that
tried
to
push
it
down
and
we're
okay.
So.
B
Well,
I
think
that's
it
for
today
I
apologize
for
not
getting
to
everybody's
questions,
but
we
do
actually
have
to
jet
right
now
so,
but
if
you
do
have
more
questions,
definitely
join
the
delta
lake
user
slack
actually,
because
both
paul
and
myself
are
actually
on
it
and
so
we'll
actually
answer
questions.
There
is
actually
a
lake
fest
channel
too,
so
you
can
definitely
chime
questions
there
so
feel
so
we
apologize
for
not
answering
all
of
your
questions
right
from
the
get-go,
but
yes,
we'll
be
there.
B
A
A
So
sorry
about
the
confusion
between
the
two
two
events,
but
I
think
I
think
we're
all
set
the
recording's
available
and
you
should
have
the
links
and
yeah
feel
free
to
ping
either
of
them
in
the
github
or
they
get
up
in
the
slack
tell
to
like
slack
all
right.
Well,
thank
you
so
much
everyone
for
joining.
I
hope
you
all
have
a
great
rest
of
your
day.
Take
care.