►
Description
Airbyte is an open-source data integration platform that syncs data from applications, APIs & databases to data warehouses, lakes, and other destinations. Delta Lake is an open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python. Together they can help address many data integration issues.
In this session, Simon Späti from Airbyte and Denny Lee, Delta Lake maintainer from Databricks discuss data integration with Airbyte and Delta Lake. From ELT vs. ETL to normalization of data to orchestration, we will discuss the complexities and potential solutions to simplify your data integration.
Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Simon Späti: https://www.linkedin.com/in/sspaeti/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
Delta Lake Releases: https://github.com/delta-io/delta/releases
A
Data
integration
with
air
byte
and
Delta
Lake
I
am
honored
to
have
Simon
on
board.
Now
we're
going
to
wait
a
couple
minutes
before
we
actually
start
the
show
in
reality,
because
we're
currently
restreaming
live
to
both
the
LinkedIn
and
YouTube
platforms.
So
meanwhile,
as
we're
waiting
for
folks
to
Jump,
On,
In
and
and
discuss
with
us
love
to
go
ahead
and
actually
start
with
the
fun
stuff,
which
is
where
are
you
based
out
of.
A
A
Capital,
birth,
okay,
okay,
oh
excellent,
okay,
yeah,
yeah
I
know
so.
Okay,
are
you
okay?
Now
here's
the
real
debates,
because,
even
though
I
know
this
is
supposed
to
be
about
everybody
in
Delta
Lake,
which
one
do
you
prefer
snowboarding
or
skiing,
because
I
know
that
that
itself
becomes
a
bit
of
a
you
know.
B
A
B
A
So
yeah
we
might.
This
might
turn
into
a
snowboarder
versus
skiing
debate
here,
no,
no
I'm,
actually
all
for
cross
country,
skiing
and
because
I
love
doing
that
and
I
actually
do
a
lot
of
snowshoeing
as
well
or
so.
I
should
rephrase
it
I
used
to
do
a
lot
of
snow
showing
as
well
all
right
but
but
I
actually
started
off
as
a
skier
and
then
especially
in
East,
Coast,
Canada
and
then
now
live
in
Seattle,
so
now
west
coast,
so
that
so
that
has
plenty
of
powder.
A
A
A
Yeah
absolutely
yeah,
yeah,
yeah,
yeah,
yeah,
yeah,
absolutely
all
right,
so
I
think
we
are
live
on
both
YouTube
and
on
LinkedIn.
So
I
think
that
and
we're
been
live
on
Zoom
this
entire
time.
So
this
is
perfect,
really
happy
to
have
all
three
platforms
working
so
perfect.
Well,
okay!
So
now
that
everybody's
live,
let's
go
ahead
and
reintroduce
ourselves
to
explain
what
session
this
is
is
again.
This
is
a
d
D3
L2
session,
discussing
data
integration
with
air
byte
and
Delta
lake.
B
B
Then
I
had
a
stage
in
Denmark
like
two
three
years,
where
I
got
into
so-called
big
data
and
yeah
I
learned
more
before
it
was
all
like:
a
PL,
SQL,
T,
SQL
and
they're
a
lot
more
like
Python
and
coding
and
yeah
when
I
switched
back
to
Switzerland.
I
also
got
more
into
the
real
data
engineering,
with
like
also
setting
up
with
kubernetes
and
like
open
source
stack
and
stuff
like
this
and
yeah
as
a
side.
Hustle
I
also
started
to
block.
B
So
that's
where
I
also
got
a
little
bit
more
into
all
the
open
source
stuff.
So
I
like
to
spread
the
word
for
open
source
and
yeah
also
kind
of
interest
in
any
kind
of
new
stuff,
just
to
get
my
hands
on
and
try
it
out
so
yeah
now,
mainly
a
dirt
bike,
yeah
and
doing
data
engineering
and
technical
writing
yeah.
That's
my
current
job,
perfect.
B
A
A
So
back
in
2000,
I
won't
say
2011.
That
was
when
Dave
Mariani
from
Yahoo.
He
was
at
Yahoo
at
the
time
he's
right
now,
the
co-founder
and
CPO
of
at
scale,
but
at
the
time
back
in
2011
he
and
I
had
went
to
Hadoop
world.
That
tells
you
how
old
we
were
talking
we're
referring
to
a
new
world.
Okay,
to
talk
about
the
largest
analysis
analysis,
Services
Cube,
on
the
planet.
At
that
time
the
cube
was
24
terabytes
on
top
of
a
five
petabyte,
two,
no
two
petabyte
I'm,
sorry
5000.
A
A
We
were
like
we
had
to
understand
the
infrastructure
super
well,
so
that
way
we
could
like
clone
like
physical
clones
of
the
data
like
actually
at
the
hardware
level,
physically
clone
the
data
in
order
to
be
able
to
go
ahead
and
keep
it
up
running
and
knowing
full
well
that
this
thing
could
crash
at
any
moment,
and
then
we
have
to
have
physical
clothes,
and
so
we
have
both
the
physical
clones
from
the
Clarion
machines
like
EMC,
clarions
and
also
the
logical
clone.
So
we
were
messing
around
with
all
of
these
things.
This
is.
B
B
A
Gotcha,
oh,
but
by
the
way,
I
just
forgot
to
do
some
important
Logistics,
but
then
we'll
switch
back
back
to
that
exactly
which
is,
if
you
have
questions,
go
ahead,
chime
Us
in
on
the
LinkedIn,
if
linked,
if
we're
not
actually
right
now,
I
haven't
seen
any
questions
LinkedIn,
but
we've
been
noticing
some
issues
with
LinkedIn
comments.
If
you
are
running
into
issues,
go
to
go
dot,
Delta,
dot,
IO,
slash
events
and
then
just
click
on
the
event
and
join
us
on
the
zoom
or
you
can
just
ask
us
questions
directly
there.
A
Okay,
so
just
as
an
FY
or
you
can
go
ahead
and
just
join
us
at
YouTube.
You
can
also
ask
questions
if
you
want
to
do
YouTube
instead
of
LinkedIn
and
that's
go.delta
dot,
IO
YouTube,
It's,
the
streaming
event
so
yeah.
So
hopefully,
if
you're,
not
if
we
don't
see
right
now
again,
I'm
not
seeing
any
questions
through
Linkedin,
that's
one
of
the
more
popular
platforms
for
us.
A
So
if
I'm
not-
but
you
do
want
to
ask
questions-
please
go
to
either
the
zoom,
which
is
like
I,
said,
go
dot,
Delta,
dot
events
or
the
YouTube,
and
you
can
just
join
us
there
and
then
ask
your
questions
away.
Okay
and
so,
for
example,
manyo!
Actually,
just
chimed
in
ssas
will
stay
in
companies
until
the
people
that
deployed
it,
retired
and
yeah
I
actually
absolutely
agree
that
statement
when
I.
A
So
to
give
you
some
context
on
an
assignment
I'm,
pretty
sure
you
suffer
from
the
same
thing
here
right,
I'm
sure
you
have
your
own
story
of
this
one.
Where
what
happened
is
that
I,
because
I
was
working
with
Yahoo
on
that
Uber
large
cube
right?
What
ended
up
happening
is
afterwards
is
that
we
would
continue
having
these
discussions
and
then
I
realized
about
six
months
later
that
we
were
having
these
weekly
discussions.
We
didn't
talk
about
a
single
Microsoft
Technology.
A
We
were
talking
about
Hadoop
and
we're
talking
about
spark.
So
this
is
during
the
transition
where
the
Yahoo
team
actually
was
starting
to
switch
over
to
spark
themselves.
So
literally
I
was
going
like
wait.
I've
been
talking
to
Yahoo
for
six
months
as
an
employee
of
Microsoft
and
I
haven't
talked
about.
I
haven't
talked
about
a
single
Maxwell
technology
for
six
months,
maybe
I.
B
A
A
All
right,
so
all
right!
So
now
you
you!
This
is
sort
of
cool
that
both
Simon
and
I
can
talk
about
our
analysis,
services
or
bi
past.
So,
just
like
you
were
noting,
like
as
you
transitioned
over
to
joining
like
when
you
move
between
Denmark
back
to
Switzerland,
okay,
that
you
were
saying
that
how
hey
you
wanted.
You
had
like
basically
started
working
on
big
data
and
we'll
talk
about
the
bi
stuff
and
all
back
into
Big
Data
space,
but
I'm
just
curious.
B
Something
else
together,
sorry
at
first
it
was
only
like
Python
and
I
worked
at
like
a
subsidiary
of
airbots
and
there
we
used
like
a
palantir.
So
that
was
like
all
built-in
tools.
So
basically
you
could
just
use
SQL
or
python
already
back
then
and
build
your
pipelines
and
stuff.
So
that
was
quite
cool,
but
fortunately
I
didn't
have
to
work
with
Hadoop.
So
when
I
transitioned
yeah,
the
first
thing
was
spark
that
I
used
for
like
massive
amounts
of
data.
A
Perfect,
so
actually
we
got
a
question
about
the
analysis,
servers
that
just
popped
up
on
LinkedIn,
but
I
went
or
let's.
Let's
continue
this
conversation
I'll
bring
it
we'll
bring
it
back
to
as
afterwards,
so
yes,
saurabh
and
anybody
else
on
LinkedIn.
If
you
guys
want
to
continue
asking
questions
on
analysis
Services,
as
you
can
tell
it's
close
to
both
of
our
hearts.
So
yes,
we'll
we'll
go
back
to
it,
but
we've
already
transitioned
to
Sparkle
we'll
make
sure
to
leave
time
to
to
go
back
and
talk
about
Now
sources.
So
don't
worry.
A
A
Was
before
3.0,
so
probably
two
done
something
got
it
got
it.
Okay,
oh
no
I
was
just
the
only
reason.
I'm
asking
is
because
I
just
want
to
remind
myself
to
see
if
he
how
much
pain
you
were
going
through.
But
it
looks
like
you
were,
looks
like
you
like.
You
did
at
least
didn't
suffer
from
some
of
the
earlier
pains
that
we
had
so.
B
Yeah
I
suffered
from
the
long
error
message
in
Python
and
and
where
you
could
still
do
stuff
in
Scala,
but
not
in
Python,
but
afterwards
it
got
Ultra
like
transfer
to
python
as
well
right.
But
when
I
started
python
was
not
yet
like
I
would
say:
I
said
I
thought
I
think
most
people
use
Python,
but
it
was
not
like
the
main
API
From
databricks
perspective.
A
No,
no,
no
you're,
absolutely
right,
I
mean
the
python
API
like
basically
is
like
I
still
remember
rental.
Talking
about
like
when
we
originally
built
the
pi.
The
python
API
was
originally
built,
was
like
2
000
lines
of
code,
which
was
just
an
extraction
layer
and
because
we
always
love
the
fact
that
python,
on
top
of
you
know,
which
is
built
on
top
of
C,
basically
is
doing
an
abstraction
labor
over
the
jvm,
because
that
makes
a
ton
of
sense
I'm
by
the
way
for
anybody,
who's,
pythonic
and
not
knowing
who
I
am
we?
A
A
This
is
sarcasm
here,
but
yeah
you're,
absolutely
right,
like
basically
the
error
messages
that
we
would
just
literally
spit
out
the
Scala
based
error
messages
that
which
themselves
were
already
confusing
enough
as
it
was,
and
then
anybody
who's
an
expert
in
python
or
just
used
to
using
python
like
you're.
Looking
at
these
scholar
messages
girl
like
okay,
wait.
What
is
this.
A
Well,
especially
because,
during
those
early
years
we
often
were
talking
about
like
pie,
Funk
right.
So
basically
it
was
like
the
idea
that
you,
each
of
the
workers,
actually
had
to
go
ahead
and
spin
off
of
their
own
like
python
process
within
the
worker
itself,
and
then
you
had
to
go
Pi
4J,
so
it
literally
was
like
the
jvm
back
to
python,
so
python
to
jvm
jvm
back
to
python.
Okay,.
B
A
Then
later
on,
we
started
going
in
and
saying:
okay,
no
we'll
put
more
of
the
stuff
in
the
driver,
which,
of
course
all
we
did
was
overload.
The
driver
I'm
like
oh,
that's
great,
so
yeah
it
like,
as
you
can
probably
tell
we
can
go
on
for
hours,
so
we'll
probably
switch
gears
out
of
this,
because
I
can
rat
hold
on
this
one
particular
topic,
as
you
can
tell
there's
a
little
bit
trauma
for
me
in
terms
of
basically
like
when
we
started
doing
vectorized
udfs.
A
So
that
way
it
can
make
our
lives
a
little
bit
easier,
but
but
saying
that,
let's,
let's
talk
a
little
bit
about
your
story
into
Arabic
like
so.
How
did
you
get
involved
with
everybody?
Was
it
because,
like
you
know,
invariably
like
the
alt
versus
ETL
side
of
the
house,
or
just
interesting,
like
you
mentioned,
you
were
Airbus
before
with
python
yeah?
How
did
that
transition
over
to
everybody?
I'm
curious.
B
Yeah
this
is
a
mainly
because
of
Open
Source
and
because
I'm
a
bit
believer
in
the
like
in
the
product
of
everybody
itself,
I
I
built
like
like
ETL
or
elt
pipelines
at
every
company
that
I
have
been
with
so
I.
Basically,
you
start
work
and
then
either
there
is
already
something
like
a
template.
Mostly
custom
built
and
I
just
found
the
idea
that
you
can
reuse
and
build
it
in
the
open
so
appealing,
and
it
was
also
more
also
soft
reason
that
I
would
wanted
to
work
remote.
B
Everybody
had
these
opportunities
that
you
can
work
remote
and
also
it's
yeah
and
I,
really
I
really
like
coding,
but
I
also
like
writing-
and
this
was
actually
the
only
job
I
found
like
where
I
can
work
as
a
coder
or
data
engineer.
And
writing.
So
that's
how
it
got
started.
But
it's
also
yeah,
because
I
write
about
the
open
source
data
stack
tools
and
yeah.
Everybody
was
one
of
them
that
I
really
like,
and
so
that's
how
I
yeah
I
found
the
job
percentage.
Essentially
at
the
end,
yeah.
Oh.
A
No
that's
great
and
actually
makes
a
lot
of
sense.
I
mean
just
to
give
some
context
of
folks,
like
you
know,
developer
relations
itself
and
now
mind
you
of
course,
we're
honestly
we're
doing
Shameless
I'm
doing
a
Shameless
plug
on
both
of
our
behalfs
of
our
actual
roles,
but
in
a
lot
of
companies.
This
is
starting
to
pick
up
the
the
need
for
exactly
what
Simon
just
described,
which
is
this
idea
that
you
can
still
code
but
also
be
able
to.
A
You
know,
speak
or
to
be
able
to
write
to
be
able
to
communicate
that
effectively.
This
and
there's
that
interesting
balance
and
Dev
relations
actually
allows
us
to
do
that
now,
let's,
let's,
let's
focus
a
little
bit
specifically
on
airbite,
though
okay,
so
you
talked
about
okay,
you
started
using
yourself,
you
started
using
for
coding
purposes.
A
suite
that
makes
a
lot
open
source
again
makes
a
ton
of
sense.
Why
don't
we
go
backwards?
A
little
bit,
though
right?
Why
everybody?
What
what
is
it
about
air
bike?
A
Okay,
so
for
this
audience
here
since
this
is
the
Delta
Lake
audience,
they
may
or
may
not
be
familiar
with
it.
So
why
don't
we
start
just
even
with
that
I
mean
you
know
like
why?
Why
do
we
care?
Why
is
airbite
such
a
cool
thing
in
the
first
place,
and
yes,
it
is
cool,
so
don't
get
me
wrong
if
you
want
to
just
use
everybody
just
for
that
reason,
that's
a
valid
reason
honestly,
but
but
no
no
I'm
sure
there's
actually
like
valid
technical
reasons.
So
so
why
don't
we
talk
about
that
like?
B
Think
for
Delta
uses
is
especially
cool
because
you
before
you
use
Delta,
you
need
to
have
some
data
right
that
you
need
to
load.
So
basically
you
create
your
Delta
or
data
Lake
and
you.
The
first
challenge
will
be
okay.
How
do
I
get
in
this
data?
So
is
it
an
API?
Is
it
a
database?
Is
it
like
a
Excel
sheet?
Is
it
like
ssas
Cube,
so
whatever
it
is,
you
need
to
have
to
integrate
that
and
that's
basically
what
the
airbag
does.
B
So
we
have
like
330,
connectors,
source
and
destination
so
like
the
destination
will
be
the
Delta
lake
so
that
you
can
basically
yeah.
You
might
have
some
Google
analytics
data.
You
might
have
some
Facebook
ads.
You
might
have
some
Oracle
databases,
you
might
have
some
Microsoft
whatever
you
have.
You
need
to
integrate
that
into
your
Delta
lake
or
your
data
link
to
make
sense
out
of
it
and
that's
essentially
what
everybody
does
and
the
best
thing
is
it's
open
source
right.
So
it's
you
you.
B
Basically,
you
don't
lock
yourself
in
so
whenever
you
have
a
problem,
you
can
just
look
at
the
code
or
the
connector
and
actually
fix
it
or
if
you're
lucky,
if
it's
a
well-established
connector,
maybe
someone
else
already
fixed
it.
So
it's
because
the
hard
part
of
the
connector
is
not
like
building
it.
The
hard
part
is
maintaining
it
because
this
apis,
this
they
change
all
the
time.
So
it's
really
hard
to
maintain
it
and,
in
my
opinion,
I'm
a
big
believer
in
open
source.
B
The
best
thing
is
to
with
open
source
right,
because
then
you
can
work
together
on
that
and
yeah.
That's
there's
more
reasons.
We
have
like
a
ctk,
so
like
a
connected
developer
kit
that
you
basically
don't
start
from
scratch,
you
you
can.
You
can
just
yes
start
up
with
existing
code.
We
scaffold
some
stuff,
so
you
don't
so
you're
very
fast.
Just
a
couple
of
weeks
back
we
introduced
a
low
code
connector.
So
that
means
you.
You
can
just
yes,
I'm
love
it
some
hate
it
so
with
yaml.
B
You
can
just
like
declaratively,
like
design
your
your
connector,
which
is
not
good
for
everything,
but
when
you
have
an
API,
you
don't
need
more
right.
You
just
have
an
endpoint.
You
have
like,
maybe
the
patient,
how
many
data
you
want
to
do
in
one
gold,
maybe
have
different
tables
and
the
apis
that
there's
only
a
couple
things
that
you
do
with
an
API,
and
that
is
what
we
wanted
to
tackle
with
that
low
code
connected
that.
Basically,
everyone
can
build
a
connector
within
a
couple
of
minutes
hours.
B
A
B
It's
code,
it's
always
harder
because
it's
obviously
bigger
and
you
need
to
do
more
and
yeah,
but
we
also
have
we
support
different
languages.
We
have
python
Java,
C,
C,
sharp,
JavaScript
and
so
on,
but
because
we
run
in
Docker
you
can
essentially
build
any
kind
of
connect
this.
So
that's
that's
also
a
nice
thing
that
I
think
we
have
is
that
we
are
language
agnostic
and
you
build
the
connector
in
the
language
you
feel
either
most
comfortable
with
or
even
better.
That
is
most
performance
right,
so
that
it's.
A
No,
that
makes
a
ton
of
sense
so,
where
I'm
coming
from
then
is,
then
you
sort
of
implied
the
the
answer
here.
But
let's,
let's
dive
into
this
a
little
bit
you
with
airbite,
basically
you're
able
to
connect
to
all
of
these
different
sources,
and
so
that's
and
it
takes
care
of
the
key
point
of
the
ingestion
point,
which
is
basically
how
do
I
connect
to
a
source
and
how
do
I
get
to
a
Target
because
you're
following
open
source.
A
The
idea
is
that
you've
simplified
the
processes
massively
and
so
from
your
perspective,
especially
like
like
how
like
what
are
the
primary
complexities
that
your,
like,
you
say,
the
air
byte
customer
base
is
typically
going
to.
Is
it
just
that
connection
point?
Is
it
just
a
low
code
I'm
just
curious
like
what
like,
because
ultimately
it's
about
getting
to
that
point
of
data
integration
right
like
so
that's,
ultimately,
what
you're
trying
to
get
into
so
so
I'm
just
curious
is.
Is
that
from
your
perspective,
from
everybody's
perspective?
B
Maintaining
a
lighthouse
I
think
you
have
other
challenges,
but
when
you
don't
have
any
data,
I
think
it
doesn't
make
sense
to
build
one.
So
the
I
think
you
wouldn't
need
airbight.
If
you
have
like
one
huge
source
that
you
have
everything
from,
but
as
soon
as
you
already
have
like
Salesforce,
and
maybe
some
sap
or
just
like
two
three
others
I
would
definitely
go
for
air
byte
already.
B
You
can
also
use
air
byte
for
whatever
connector
is
there
and
still
build
your
own
right,
it's
even
easier
with
Airbus,
because
you
have
the
customer
developer
kit
that
supports
you
along
the
side,
but
I
think
yeah
I
would
that
is
probably
the
yeah
the
most
and
then
the
challenges
that
that's.
What
I
said
before
right
that
they're
just
changing
all
the
time
and
there
is
like
new
SAS
products
every
day.
So
that's
another
challenge
that
right
right.
B
Almost
every
product
that
gets
like
built
these
days,
the
creating
data
as
well
right,
so
it's
essentially
a
new
source
but-
and
you
cannot
really
tackle
that
problem
with
a
closed
Source
approach.
In
my
opinion,
because
you,
you
will
just
be
too
slow
and
the
people
they
want,
also
to
like
fix
the
stuff
fast
right.
If
they
have
a
highly
demanding
connector,
they
cannot
wait
for
someone
else
to
fix
it.
So
not
everyone
has
the
skills
or
the
engineers
to
fix
it
themselves.
B
But
at
least
you
have
the
the
code
and
you
can
really
fix
it
and
yeah.
Then
the
next
step
is
obviously
like
integration,
so
you
would
then
integrate
with
like
orchestrator
or
if
yeah,
Delta
lake
or
that's
then
the
next
thing,
but
I
think
it
mostly
data
driven
or
data
architect
to
start
with
integration
and
transitions
to
like,
like,
of
course,
you
need
a
store
so
the
where
do
you
store
it,
and
then
you
have
like
okay,
you
have
like
I,
always
like
to
call
it
three
levels
right.
B
So
they
best
thing
is
when
it's
like
optimized
for
like
storage.
So
it's
like
compromised
and
maybe
even
a
columnar
format
that
you
can
like
do
analytics
queries
and
then
the
next
level
is
like
the
table.
Format
right,
which
data
lake
is,
is
obviously
very
good
where
you
get
like
the
I,
always
like
to
call
it
like
a
database
for
fi
distributed
files,
because
essentially
you
get
all
the
features
that
the
database
has
right.
Yeah
like
assets
transaction.
You
have
time
to
even
get
more,
but
I.
B
Always
it's
like
a
table,
but
under
this
table
is
like
huge.
It
can
be
massively
massive
files
right.
So
it's
for
me
that
was
quite
a
big
eye
opener
when
I,
when
you
can
do
like
database
stuff
based
on
files
with
and
yeah
that
that's
also
where
I
yeah
I
used
the
Delta
Lake
in
my
previous
work,
also
heavily
because
of
that
fact,
because
you
can,
because
it's
just
so
it's
open
source-
and
it's
also
so
nice
to
use
and
have
all
the
features
that
you
basically
know
from
databases.
Yeah,
oh.
B
It's
a
good
thing
and
a
bad
thing
right,
because
you,
you
can
store
things
without
thinking
and
it's
fast,
but
yeah
you're,
just
causing,
like
mostly
data
silos
and
yeah
after
value
the
data
catalog,
just
to
know
what
data
you
have
exactly
so
you
start
into
adding
on
top
of
it.
Although
yeah,
you
should
probably
know
your
data
and
think
about
yeah,
it's
always
I
I
had
both
worlds.
So
with
the
data
warehouse,
your
I
was
also
frustrated
because
you're
so
slow
right.
Whenever
someone
said
oh,
can
you
get
me
this
new
table?
B
Oh
okay,
that
takes
three
months.
Okay
right,
so
it's
like
because
you
need
to
deal
much
more
right.
You
need
to
think
about
how
you
want
it
watch
granularity.
Do
you
want
like
every
event
or
you
want?
Can
you
like
aggregate
it,
and
so
there's
much
more
thoughts?
So
it's
also
nice
to
just
store
it,
but
yeah,
obviously
they're
both
as
their
pros
and
cons.
Oh.
A
Absolutely
so
this
actually
is
a
nice
segue
to
the
other
topic.
I
want
to
discuss
a
little
bit,
which
is
like
ETL
versus
elt
right
and
so
the
idea,
and
so
for
those
who
are
listening
right
now,
they're
thinking
hey
what
the
heck
are
you
guys
talking
about
we're
talking
about
extract,
transform
than
load
which
is
prototypical
of
something
system
where
you
are
outside
of
the
database,
where
you're
doing
the
transformation
there
per
se
or
outside
the
storage,
I
should
probably
put
versus
elt,
which
is
extract.
A
Load
then
transform,
in
other
words,
which
is
prototypical
of
a
lot
of
like
a
lot
of
databases,
data
warehouses,
which
is
saying
no
no
extract
the
data
from
multiple
sources
loaded
into
this
one
thing
and
then
transform
it
in
that
one
thing,
so
I'm,
just
curious
from
your
perspective,
like
you
know,
pros
and
cons.
Does
everybody
allow
for
both
like?
Is
there
a
preference
I'm
just
curious
from
you
know
your
perspective,
whether
it's
yourself,
personally
and
or
from
a
nearby
perspective,
I'm,
just
curious.
B
Yeah,
this
is
actually
a
also
a
feature
that
not
many
people
know
about
airbag
to
be.
We
sometimes
call
it
like
e
small
t,
l
t
so
there's
a
t
inside
the
ETL
and
load,
because
under
the
hood
we
also
use
DPT
for
transformation,
so
you
can
actually
within
air
byte.
You
can
do
Transformations,
which
obviously
should
be
like
small
things,
because
otherwise
you
break
stuff,
but
we
actually
have
the
ability
to
do
transformation
in
front
of
the
the
the
yeah
loading
it
right.
So
you
have
right
transformation
twice.
B
So
this
is
very
nice
actually
for,
like
gdpr
stuff
or
like
security
stuff,
because
you
can
actually
take
away
pii
data
that
is
sensitive
before
you
load
it
to
a
data
warehouse
or
a
data
Lake
right.
So
this
is
right,
a
very
nice
feature,
but
obviously
we
are
doing
the
elt
and
we
we
load
it
and
then
transform
it.
So
I'm
a
I'm,
not
so
much
of
liking,
one
than
the
other,
because
at
the
end
of
the
day
you
just
load
data
somewhere
and
you
transform
it.
B
So,
in
my
opinion,
you
should
do
it
the
most
effective
way
and
if,
if
let's
say,
if
you
have
an
API,
it's
not
so
effective
to
transform
somewhere
in
between
so
I
guess
it's
easier
to
just
load
it
and
then
do
it
with
with
some
compute.
That
is
much
faster
and
better.
But
if
you
have
like,
if
the
source
is
a
database,
they
have
the
compute
right.
So
you
could
do
it
on
the
way
like
the
El,
etl's
approach,
so
yeah
and
we
yeah
support
both.
B
It's
not
so
well
known
that
we
do
the
transformation
and
probably
I'm
a
bit
skeptical.
If
you
do
too
much
of
of
the
transformation
between,
of
course,
you
have
more
errors.
You
because
I
think
airbikes
also
used
for
you
want
to
load
the
data
correct
and
on
time
right.
So
it's
a
the
more
you
do,
transformation
there,
the
more
you
actually
yeah
can
shoot
yourself
in
defeat.
If
something,
if
you
do
something
too
complex
or
so
yeah
I
think
it's
always
a
balance,
as
always,
but
yeah.
A
You
put
the
information
in
by
the
same
token,
if
you're
dealing
with
tons
of
data
honestly,
you
probably
need
to
just
dump
it
somewhere
and
so
sorry
called
it
as
like.
A
raw
immutable.
Zone
I
was
like
yeah
yeah
and
so,
for
example,
in
Delta,
like
we
often
talk
about
like
The
Medallion
architecture,
the
bronze
silver
gold,
and
the
idea
is
that
you
have
this
bronze,
which
is
the
whole
reason
you
have.
This
bronze
zone
is
because,
because
you're
asking
for
so
much
data
like
Kafka
topics
or
whatever
it
is,
your
system
is
right.
A
You,
you
can't
keep
state
right,
for
example
like
just
to
pick
on
Kafka
and
not
because
I,
don't
like
coffee
Court,
the
offset
I'm
a
huge
fan
of
Kafka,
it's
just
like,
but
you
usually
configure
Kafka
to
only
have
like,
like,
for
example,
I
was
just
talking
to
Tyler
Crow
yesterday
and-
and
he
was
telling
me
like
yeah.
Typically,
you
want
to
set
up
at
least
four
days
right.
A
A
Yeah,
but
you
you
understand,
we're
trying
to
get
at
like,
but
the
idea
is
like.
Typically,
you
don't
want
to
actually
have
the
Kafka
topics
stored
for
that
long
right.
You
like
fine.
You
set
up
for
four
days
so
that
way,
you
don't
have
to
be
alerted
over
the
weekend
if
something
breaks
right.
But
the
point
is
like
in
general
what
you
still
have
to
keep
it
into
some
solid
state
right,
which
is
like
a
data
Lake.
A
You
know
obviously
I'm
gonna
I'm,
biased
toward
Delta
Lake,
but,
let's
be
clear,
that's
actually
applicable
to
any
data
lake
or
lake
Lakehouse
storage
format.
That's
the
whole
reason
for
them,
because
you
just
need
to
get
that
data
down.
So
that
way
you
know
like
that
data
is
there
that
state
is
complete
and
then
you
can
transform
it
accordingly.
Now,
which
leads
me
to
the
to
the
question
which
actually
was
asked
from
LinkedIn,
and
let
me
go
back
where
it
is.
There's
a
question
from
but
I
apologize.
A
B
Right
now
we're
focusing
on
batch
batching,
but
you
can
set
like
a
chrome
shop
that
does
every
minute
or
so
with
small
datas.
Of
course,
it
just
need,
like
the
sync
itself,
need
to
be
fasted
and,
like
the
frequency
you
set
right,
you
cannot
set
a
job
that
takes
one
hour
every
minute,
obviously,
but
so,
but
you
can
schedule.
So
it's
like
a
what
is
it
the
name,
the
real
batch
real
time
or
it's
like.
A
Normal
time
but
yeah
exactly
yeah
micro,
batching,
basically
yeah
yeah,
which
is
the
traditional
context
of
the
structure
streaming
by
the
way
same
same
principle.
Basically
you
just
you
you,
basically
the
time
it
takes
to
process.
The
data
is
better,
be
shorter
by
usually
by
by
like
half
or
less
right
than
the
time
that
you're
actually
indicating
for
the
job
to
run
so
that
otherwise,
all
you're
going
to
do
is
just
queue
up
forever
and
you'll.
B
A
Perfect,
so
so,
but
this
also
tells
me
a
little
bit
Segways
right
nicely
into
the
topic
of
orchestration.
So
from
the
standpoint
of
air
bite,
like
you
know
any
preferences
or
context
about
how
do
we?
Typically,
how
do
you
typically
see
folks
in
terms
of
or
orchestrating
their
pipelines
or
their
ingestion
or
their
processing
within
the
context
of
erabite
I'm,
just
curious.
B
So
either
you
use
our
Cloud
solution.
Then
there
is
the
the
you.
Can
everything
works
out
of
the
box
and
you
can
schedule
everything
if
you
basically
when
you
open
airbag,
you
have
all
your
connections.
So
connection
is
basically
you
just
stitch
together
a
source
in
the
destination.
That's
also
one
thing
that
everybody
is
quite
nice
at
we.
B
You
can
actually
connect
every
source
with
every
destination,
so
there's
no
dangling
between
okay,
you
can
only
Oracle
only
that
works
with
Delta
lake
or
so
that
is
like
one
feature,
so
the
source
and
the
destination
you
you
put
together
as
a
connection
and
the
connection,
you
see
all
the
connection
in
air
byte
and
then
each
of
them
have
like,
like
a
schedule
interval.
So
then
you
can,
you
can
also
do
it
manually
and
and
we
also
provide
a
an
API.
B
B
Also
like
use
these
kind
of
tools
to
yeah
orchestrate,
because
normally
you
don't
only
have
airbag.
Maybe
you
have
a
DPT
model
and
then
you
have
a
like
stuff
that
you
need
to
orchestrate
on
top,
maybe
have
a
machine
learning
model.
So
then,
of
course,
it
makes
sense
to
use
like
a
like
a
true
orchestrator
and
everybody's
nicely
integrated
in
these
tools
as
well.
Oh.
A
That's
really
good
to
hear
actually
as
a
call
anybody
who's
actually
listening
right
now,
if
you
want
us
to
include
some
of
the
I
have
another
session
with
our
friends
from
over
a
prefect
or
friends
over
at
dagster.
Let
us
know
so
we
can
go
ahead
and
set
that
up
as
well.
So
it's
a
even
though
it's
it
seems
like
it's
a
wide
Community.
We
all
work
together
really
closely
anyways,
at
least
at
least
the
open
source
above
arms
of
all
of
us.
A
B
I
think
also
the
the
it's
also
something
we
need
to
solve
right.
We
need
to
have
more
Integrations,
because
it's
sometimes
a
bit
overwhelming.
If
you
for
each
small
thing,
you
need
a
different
tool
and
then
you
you
would
like
to
build
a
end-to-end
data
pipeline
or
an
analytic
system,
and
then
you
need
to
integrate
if
you
want
to
be
an
open
source,
integrate
like
five
tools
or
so
right.
So
it's
right,
I
think
if
we
want
to
succeed
more,
it's
also
about
integration
and
to
like
support
each
other
exactly
like.
A
B
To
you
can
then
I
I
just
writing
about
this,
so
the
like
the
modern
data
stack
and
how
to
fit
the
tools
together.
So
it's
also
I
think
it's
nice
then
to
extend
right.
You
can
always
you
choose
the
tools
you
want
and
then
you
can
extend.
So
you
don't
need
to
start
with
everything
from
scratch,
but
it's
still
important
that
you
can
start
up
fast
and
don't
you're
not
overwhelmed
with
all
these
names
and
tools,
which
obviously
happens
quite
fast.
When
you
start
out.
Oh.
A
No
no
I
agree
because,
if
you
like
I'm
sure,
just
because
of
the
nature
of
earbud,
you've
got
300
different,
connectors
right
and
so
the
fact
is
great.
We
can
all
use
air
byte
to
basically
go
from
source
to
Target,
but
the
reality
is
that's
just
one
of
the
pie
right
then.
Delta
Lake,
for
example,
is
another
part
of
the
pie.
But
then
she
said
Daxter
prefix,
that's
another
part
of
the
pie
and
the
reality
is
all
of
us
are
taking.
B
A
One
specific
area
we
have
a
little
bit
of
bleed
over
at
times
I'm
sure,
but
yeah,
like
you
know,
like
exactly
Victor,
just
says
sure
it's
like
an
array
extended
like
basically
we're
extending
over
each
other,
but
the
reality.
It's
it's
a
it's
a
it's
a
huge
thing
that
we
are
all
trying
to
work
other,
so
the
fat,
the
faster
that
we
actually
are
integrating
with
each
other
and
and
I
hate
to
use
the
word
standardizing,
because
that
sounds
so.
A
Thank
you.
Let's
just
use
the
phrasing
open
standard,
so
we
have
to
use
open
standard
so
that
way,
it's
possible
for
us
to
interact
with
each
other.
So
that
way
we
actually
will
ultimately
make
things
easier
for
everybody
right.
So
yeah
and
oh
by
the
way
sarama
has
been
asking
about
like,
and
he
asked
the
question
about.
Apache
beam
and
air
bite
right
and
so
I
did
want
to
call
out
that
the
reason
sorry
number
one
I
apologize
for
not
answering
that
question.
A
A
Airbite
really
is
about
going
ahead
and
talking
to
any
source
and
in
essence,
to
any
Target
to
make
your
life
a
lot
easier
which
we're
actually
going
to
talk
up
a
little
bit,
because
this
you're
referencing
about
writing
about
the
modern
data
stack
is
going
to
invariably
lead
me
to
talk
about
rust,
of
course,
but
specifically
about
the
question
about
beam
and
air
bite
beam.
Is
that
abstraction
layer
that
that
also
you
hear
in
Google
data
flow
and
things
of
that
nature?
A
Can
I
just
use
one
abstraction
layer
over
both
it's
interesting
but
again
not
to
knock
any
of
the
folks
who
work
on
beam
just
because
I
actually
have
friends
who
are
there,
but
I
personally
feel
like
if
you're
going
to
do
it
just
use
Flink
or
just
use
Mark
directly,
because
you'll
have
more
control
over
what
you're
trying
to
do.
The
whole
purpose
of
airbite
is
to
go
ahead
and
say:
no,
no,
like
I,
just
need
to
figure
out
what
my
source
says:
I
need
to
figure
out
what
my
target
is.
A
Let
me
go
into
that
yeah.
So,
hopefully,
I've
answered
your
question.
There
so
say
saying
that
let's
go
back
to
the
modern
data
stack
because
you've
been
blogging
about
it.
Simon
you've
been
doing
some
great
blogs
on
the
stuff
which
we'll
include
in
the
follow-up
to
this
old
texture
to
this
video
as
well
to
some
of
your
blog
posts
but
yeah.
Let's
talk
a
little
bit
about
rust,
because
I
did
eventually
want
us
to
get
back
to
Delta,
Lake
too,
which
is
so
yeah
yeah.
A
B
Yeah
I
I
was
just
hearing
like
a
lot
of
good
things
about
Ross
right.
It
was
like
seven
years
in
a
row.
The
most
loved
language
on
stack
overflows,
I
was
like
really
curious
and
I
was
always
thinking.
Okay,
I
want
to
learn
a
new
language
that
was
either
a
you
know
like
one
of
the
functional
programming
languages
or
like
Scala
or
like
like
there's
so
many
nowadays,
Tulia
and
all
the
others
and
then
but
now,
I
think
brost
is
really
gaining
popularity
and
actually
I
blog
about
that
right.
B
What
is
a
rust
versus
python,
so
I
X
I
just
put
myself
in
there
and
tried
out
rust
and
I
think
it's
just
a
good
fit
for
data
Engineers,
although
it's
actually
systems
language
that
is
maybe
on
First
Look.
It's
not
made
for
data
engineering,
but
one
disadvantage
of
python
is
obviously
you.
You
only
know
when
you
run
things,
so
you
cannot
really
testing
or
you
need
to
test
things
heavily.
So.
A
B
B
You
need
to
Define
your
memory
and
stuff
like
this,
which,
but
you
get
a
lot
of
stuff
or
you
get
like
the
Clarity,
maybe
that
it
you
will
not
run
into
that
many
errors
in
in
the
production,
so
that
that's
one
big
thing
and
also
I
think
just
using
Ross
I'm,
not
a
any
expert
on
it,
but
just
using
it,
it
was
so
nice
with
the
compiler.
That
tells
me
everything
while
writing.
Okay,
you
should
do
this.
You
should
do
their
messages
are
good
and-
and
obviously
it's
also
blazingly
fast
right.
B
So
it's
a
if
we
want
to
do
data
stuff.
We
have
lots
of
data,
so
it
must
be
fast.
I
I
still
think
there
will
always
be
Python
and
you
have
like
an
interface
to
to
rust,
but
if
developers
liking
Ross
more,
they
will
also
start
trying
to
make
sense
and
trying
to
use
it
right.
So
they
will,
while
so
I,
think
it's
really
getting
more
popularity
and
yeah.
That's
also
a
nice
thing
that
I
saw
in
Delta
Lake
that
you
support
that
and
active
actively
having
this
this
repo
and
yeah.
B
It
can
also
be
one
thing
that
we
had
to
look
at
is
also
using
spark
with
air
bite
but,
like
we
are
all
dockerized,
so
it's
quite
hard
to
just
spin
up
like
a
spark
cluster
within
our
open
source.
We
already
have
quite
some
containers
that
for
the
UI
for
the
server
for
the
database
and
stuff
like
this,
so
then
there
will
be
one
more
and
that
will
be
a
bit.
I
always
say:
spark
is
a
bit
of
Beast
right.
B
A
Oh
perfect,
yeah,
and
by
the
way,
so
there
is
a
spark
Docker
by
the
way,
an
official
one.
Now,
but
even
after
saying
that,
like
which
I
do
talk
about
yeah
I'm
with
you
100
right,
you
know
rust
we
by
the
way,
just
as
a
caller
I'm,
because
I'm
gonna
time
box
it
because
I
do
want
to
answer
sort
of
the
original
question
back
to
analysis
Services
again,
but
we
do
actually
have
a
D3
L2
session
that
happened
last
month.
A
Oh
right
now,
it's
December
by
the
way,
so
so
just
in
case
you're,
watching
this
later
last,
one
with
Florian
who
actually
built
who's
a
maintainer,
Delta,
rust
maintainer
and
he
focuses
on
the
python
apis.
Exactly
to
your
point,
I
think
the
python
API
is
beyond
here
to
stay.
It
is
the
standard
that
we
all
work
off
of
right.
A
A
The
reason
I
brought
this
up
is
because
I
really
enjoy
the
fact
that,
even
as
somebody
like
yourself,
myself
like
who
are
come
from
database
backgrounds,
we're
we're
actually
talking
about
rust
and
so
what's
interesting
and
I
I've
used
this
phrase
from
Dave
Mariani
I
want
to
attribute
it
to
the
right
person
person
by
the
way.
Is
that
we're
over
the
last
10
years,
or
so
we've
seen
this
convergence
of
data
engineering
and
software
engineering
like
it's
not
never
going
to
be
always
the
same.
A
But
the
point
is
that
they're
that
we're
starting
to
apply
software
engineering
principles
directly
into
Data
engineering
right,
so
that
we
actually
care
about
types.
We
care
about
how
memory
management
things
of
that
nature
and
to
your
point,
exactly
rust,
which
is
a
no
is
seen
as
a
systems,
programming
language.
Yet
exactly
to
your
point,
it's
being
applied
to
software
engineering
in
general,
like
from
a
functional
operational
perspective
and
it's
being
applied
to
data
and
Engineering,
because
we're
seeing
this
convergence
of
software
and
data
engineer,
which
is
really
cool
right.
A
A
So
that's
exactly
what
we're
working
on
right
now,
so
there's
actually
two
or
three
GitHub
issues
open
from
the
Delta
rust
side
that
we
got
to
go
fix,
so
we
just
released
Delta,
0.5.0
and
Delta
rust
0.6.4,
so
we're
pretty
close
to
getting
ourselves
there.
And
so,
if
you're
interested
come,
join
us
in
the
Delta
user
slack,
that's
go.io,
slack,
okay,
Delta
RS
channel
and
you
can
join
us
there
and
to
talk
about
how
you
can
go
ahead
and
basically
build
it.
A
Once
we
do
that,
I'm
hoping
Simon
and
I
can
actually
and
whoever
helps
us
build.
It
can
actually
have
another
session
in
the
near
future,
specifically
on
this
right
so
and
anything
that
you'd
think
that
we
should
add
as
a
call
up,
because
I'm
pretty
sure,
I
called
out
everything,
but
I
just
wanted
to
call
anything
that
you
thought
I
might
have
missed
here.
A
Right
all
right,
perfect.
Well,
then,
let's
go,
let's
see:
okay,
we've
got
most.
Questions
are
pretty
much
been
answered
vessel.
Let
me
just
at
least
end
with
siroval
call
out
it's
like
so
he's
wondering
so
just
because
you
know
Simon
and
I,
both
from
analysis
or
so
back
back
to
that
little
tidbit
he's
wondering
he's
a
he's
like
for
that
Cube
that
24
terabyte
Cube
on
analysis
surface
is
against
the
five
petabyte
or
two
stupid,
two
petaby
five
thousand
Hadoop
cluster.
A
A
We
got
this
on
one
server
now
we
did
replicate
this
over
three
times
because
we
did
clone
it.
So
here's
actually
how
we
did
it.
We
had
physical
clone
of
one
of
them,
and
so
that
was
our
backup.
It
was
a
backup
cluster.
So
the
idea
is
just
if
it's
process
is
going
to
blew
up
or
if
it's
processing
and
taking
too
long
that
people
can
query
that
back
up.
We
also
did
two
replicates.
We
also
do
two
logical
clones,
which
are
basically
aimed
at
the
same
Clarion
or
whatever
Hardware.
A
We
were
using
at
the
time
which,
basically
it
could
go
ahead.
In
essence,
you
had
the
analysis,
service,
Quarry
Engine
in
essence
running
in
the
server,
but
the
storage
engine
and
I'm
being
very
facetious
when
I
say
that,
because
in
analysis
of
those
actually
wasn't
a
split
between
the
two,
but
basically
the
storage
of
the
indexes,
the
maps
and
all
these
other
things
that
were
in
analysis
were
actually
stored,
basically
on
and
where
multiple
systems
could
go
hit
it
at
the
exact
same
time.
A
The
reason
we
could
afford
to
do
is
because,
typically,
when
we're
processing
the
data,
we
process
it
on
a
completely
different
server,
a
completely
different
setup,
so
and
and
also
replicate
the
data
fully.
So
that
way
we
weren't
trying
to
overwrite
during
because,
after
all,
it's
running
a
day
and
a
half
to
three
days
right,
so
we
weren't
over
trying
to
overwrite
but
in
essence
yeah.
It
was
the
one.
A
B
A
Yeah
yeah
yeah
yeah
exactly
you
had
to
buy
a
whole
server
because
this
was
wasn't
a
cloud
environment.
So,
basically
like
and
don't
forget,
remember
I
I
know
the
fact.
There's
a
rack
Oracle
rack,
sir,
behind
all
that
you
can't
scale
that
down
either
we're
talking
like
a
bunch
of
Hardware,
not
cheap
at
all.
That
was
basically
hitting
the
Hadoop
cluster
and,
of
course,
the
history
cluster.
With
all
thousands
of
notes,
I'm,
like
yeah
yeah,
we
basically
beating
the
living
data.
Less
of
this
thing,
so
yeah
fun
time,
cool,
so
I
think
that's
it.
B
Is,
oh,
please,
one
last
block
just
absolutely
just
we
have
like
the
like
a
date
engineering
conference
tomorrow
and
the
day
after.
So
in
the
case,
you
are
interested.
We
have
like
also
big
speakers
from
also
from
databricks.
We
have
one,
so
we
have
all
across
the
modern
data
stack
two
days
only
two
hours,
so
it's
like
only
lightning
talks,
so
in
case
anybody
is
in
this
interested.
It's
like
move
airbyte.io.
A
All
right
correct,
so
you
know
what
just
to
help
everybody
out.
I
am
posting
that
into
all
three
channels,
if
I
can
find
all
three
channels.
Okay,
here
we
go
sorry
there
we
go
all
right,
so
I've
posted,
yes,
everybody
I
think
it'll
be
a
great
session
for
you
all
to
join.
So
there's
the
move
data
conference,
the
seventh
and
the
eighth,
so
you
can
join
that
I
I'll,
make
sure
to
log
in
if
I
haven't
done
it
myself.
A
So
this
is
great
and
then
just
as
a
plug,
because
both
as
you
can
tell
both
Simon
and
I,
are
very
much
in
the
rust
QP.
Is
this?
What
is
this
Thursday?
He
and
I
are
going
to
talk
about
the
Genesis
of
Delta.
Rest
and
it'll
be
a
fun
conversation
there
too,
so
so
Simon
I.
Thank
you
for
your
time.
I
really
appreciate
it.
We're
you
and
I
are
hopefully
going
to
have
another
session.
Okay,
yeah.
A
Later
on,
in
which
we'll
be
able
to
talk
about
the
the
the
Delta
Lake
and
air
bike,
connector
the
open
source,
one
that
works
well
together,
so
because
we're
pretty
close
but
we're
not
quite
there
yet
so,
like
I
said,
but
hopefully
we'll
have
a
on
that.
But
again,
thank
you
very
much
for
your
time
and
for
the
audience.
Thank
you
very
much
your
time
for
listening
to
us
and
enjoying
our
complete
and
random
offshoot
into
the
world
of
analysis
services.
So
thank.