►
From YouTube: Delta Lake Community Office Hours: January 12, 2023
Description
Looking for inspiration or new ideas to kick of the new year? Who better than to learn from someone who has taken a data engineering journey from inception to scaling the platform at a big fin-tech investment firm! Join us on Thursday, Jan 12th, 2023 at 9AM for the next live Delta Lake Community Office Hours where we will be joined by the co-head of data engineering at Yipitdata, Anup Segu. And also get to ask your Delta Lake questions live.
Ask us your #DeltaLake questions. These sessions allow our community to ask questions about Delta Lake OSS and get to learn what we are building, planning to build and know about recently released features. These sessions are live and the recordings are available on the Delta Lake YouTube channel.
Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Watch previous Community Office Hour sessions on YouTube: https://www.youtube.com/c/deltalake
Join us on Slack: https://go.delta.io/slack
Contributing Guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
Join the Google Group: https://groups.google.com/forum/#!forum/delta-users
A
Please
tune
in
to
our
feed
on
the
Delta
Lake
channels.
We
have
YouTube,
we
have
LinkedIn
live.
We
have
Linux
Foundation
webinar
as
well.
So
if
you
can
tune
in
there
and
ask
your
questions,
we
will
take
the
questions
throughout
a
quick
reminder
for
those
who
are
new
to
the
session.
Delta
Lake
office
hours
happen
every
two
weeks
and
they
are
live
and
if
you
miss
out
any
session,
we
always
record
it.
A
So
you
will
have
like
recordings
available
afterwards
as
well
that
you
can
share
with
your
friends
your
teams
Etc.
We
keep
it
to
like
9
A.M
Pacific
time,
and
then
we
bring
a
lot
of
contributors
and
champions
of
Delta
Lake.
To
answer
your
questions
and
also
share
some
of
the
important
content
that
we
get
from
the
community.
A
So
if
you
missed
any
previous
amas
I,
definitely
encourage
you
to
check
them
out
on
our
YouTube
or
LinkedIn
and
in
the
meantime,
I
would
love
to
see
where
you
are
rolling
in
from.
If
you
are,
if
you
are
attending
our
LinkedIn
live,
please
feel
free
to
Ping.
There
love
to
see
participation
from
all
over.
A
That
was
like
the
last
session
of
2022
and
we
had
an
engaging
discussion
on
the
Delta
Lake
projects.
So
we
covered
three
different
projects
within
Delta
Lake,
which
was
Delta
core
Delta,
rust
and
Delta.
Sharing
three
of
the
Delta
Lake
contributors
were
they're,
covering
the
features
and
Community
momentum
florianville
and
myself,
and
today
we
have
an
awesome
guest
speaker
Anup.
A
My
journey
with
Anup
goes
back
more
than
four
years
where
we
were
just
getting
started
with
data
breaks,
so
it
has
been
an
absolute
pleasure
to
work
with
him
and
today,
I
have
like
definitely
a
very
lucky
day
because
I
get
to
host
him
today.
So
we
are
going
to
bring
lessons
learned
throughout
these
many
years
of
Journey,
and
you
know
I'm
super
excited
to
have
a
loop
on
this
community
office
hours.
So
I
know
why
don't
we
kick
off
with
introductions?
You
know
why?
Don't
you
tell
us?
B
B
Over
those
years,
we've
been
a
long
time,
Delta
and
databricks
user
I
think
we've
been
using
Delta
Lake
since
around
2020
so
definitely
had
a
couple
of
years
of
experience
working
with
it,
and
it's
been
fantastic
so
far
in
terms
of
just
a
quick
recap
of
what
yet
the
data
does.
We
are
a
Data
Insights
company.
We
help
investors
and
companies
make
better
decisions
by
leveraging
alternative
data,
and
so
our
data
teams,
storage
data
sets
from
all
across
the
world
from
a
variety
of
public
data
sources.
B
So
our
entire
data
platform
is
built
on
top
of
Delta,
Lake
spark
and
databricks,
and
so
every
single
one
of
our
product
teams
will
be
ingesting
these
data
sources
trying
to
transform
them
kind
of
making
a
bronze
silver
gold
ETL
pipeline,
so
that
they
can
produce
clear
and
Rich
Data
Insights.
That
can
then
go
to
the
right
side,
which
is
our
data
product.
B
So
our
company
typically
provides
their
services
in
the
form
of
research
reports,
but
we
also
have,
over
the
years
more
technical
ways
to
consume
our
product,
so
that
could
come
in
the
form
of
interactive
dashboards,
a
client
portal
programmatic.
You
know
data
deliveries,
even
exports
to
different
data
where
houses,
and
so
we've
really
tried
to
cater
to
our
clients
and
give
them
as
much
ability
to
work
with
the
refined
data
that
we
can
produce,
but
that
all
kind
of
happens
by
being
able
to
kind
of
pull
the
data
in
from
different
sources.
B
A
That's
really
valuable,
I
know
what
you
shared.
You
know
a
quick
overview
of
like
how
a
data
engineering
works
at
Deep,
with
data
and
kind
of
sharing,
like
some
of
the
journey
that
you
took
throughout.
You
know
this
architecture
setup
of
this
architecture.
So
my
question
to
you
is,
like
you
know
you
have
like
this
complex.
Of
course
we
shared
in
one
slide,
but
it
it's
like
a
very
complex
merge
of
different
teams,
different
tools.
So
definitely
you
had
to
put
a
lot
of
thought
into.
A
B
Absolutely
I
know
just
like
even
looking
at
diagram,
it
feels
complex
and
the
reality
is
it's
even
more
complex
than
than
just
that,
so
just
just
to
kind
of
put
in
perspective,
our
data
teams
around
250
to
300
people,
and
today
you
know
the
price
of
the
day,
like
the
Delta,
like
that
we
have
around
like
14,
teen
petabytes
and
like
150
000
tables,
so
we're
working
with
a
lot
of
data.
A
lot
of
data
is
in
motion
at
any
given
point.
B
Our
teams
are
kind
of
organized
and
we're
trying
to
take
more
of
like
a
data
mesh
approach
where
we
have
different
data
teams,
be
experts
in
their
domain.
Try
and
produce
refined
data
sets
and
either
distribute
those
data
sets
internally
to
other
teams
or
go
to
clients,
depending
on
kind
of
where
they
fit
within
the
organization,
and
that
process
has
worked
well
in
terms
of
just
being
able
to
orient
people.
B
A
key
part
of
our
business
is
that
our
ability
to
really
answer
tough
questions,
that
our
clients
can
produce
answers
to
their
own
data
teams
and
so
to
be
able
to
do
that.
We
need
these
data
teams.
Internally,
to
be
as
autonomous
as
possible
to
be
able
to
kind
of
work
with
data
as
far
up
in
the
chain
as
possible
and
be
able
to
pull
as
much
insights
and
extract
as
much
value
out
of
it.
And
so
we
don't
necessarily
see
you
know,
data
engineering
as
kind
of
a
important
step
in
that
chain.
B
In
fact,
we
try
to
remove
ourselves
from
that
feedback
loop
and
try
and
give
these
data
teams
as
much
kind
of
independent
capabilities
to
work
with
the
data,
because
we
don't
want
to
necessarily
slow
down
their
process,
a
big
part
of
why
our
products
valuable
is.
We
can
acquire
the
most
valuable
data
sources.
B
We
have
Technologies
available
to
our
data
team
so
that
they
can
work
and
reshape
and
restructure
that
data
to
make
sense
of
it
and
they
can
produce
all
of
that
kind
of
data
Insight
on
a
really
Dependable
time
frame
where
it's
basically
actionable
for
our
clients.
So
there's
like
a
Time
component,
there
is
a
quality
component
and
so
for
all
of
those
reasons,
we've
built
this
data
platform
so
that
the
data
teams
have
standard
set
of
tools
to
be
able
to
work
with
the
data,
no
matter
what
kind
of
source
it
might
be
coming
from.
B
You
know
one.
We
need
to
make
sure
that
they're
able
to
do
their
job,
but
two
it
needs
to
be
able
to
be
done
on
whatever
data
source.
We
have
there's
so
much
variety,
where
we
don't
want
each
team
to
kind
of
go
out
into
the
wild
and
come
up
with
their
own
way
to
work
with
data,
because
that
could
create
a
lot
of
complexity
internally,
we
want
everyone
to
kind
of
work
with
data.
B
The
same
way,
it's
kind
of
bringing
it
back
a
little
bit
to
kind
of
Delta,
that's
really
where
we
wanted
to
orient
and
have
all
of
our
data
stored
in
Delta
Lake,
because
that
was
a
good
way
to
make
sure
that
people
work
with
data
the
same
way
and
they
have
enough
kind
of
flexibility
to
work
with
data
to
do
the
job
that
needs
to
be
done.
So
a
lot
of
our
philosophy
has
been
pushed
kind
of
the
ability
to
do
data
operations
to
our
data
teams.
B
A
Yeah
so
I
know
just
let
me
dig
in
a
little
bit
on
the
side,
so
you
mentioned
that
you
have
tons
of
data.
I
can
imagine,
like
you
know,
to
build
the
financial
reports.
The
data
is
coming
in
from
variety
of
different
sources.
You
know
you
do
like
a
lot
of
web
scraping
and
all
that
stuff.
So
what
kind
of
things
goes
into
like
you
know
what
kind
of
data
do
you
deal
with
and
what?
What?
B
Yeah,
so
on
the
ingest
side,
a
big
challenge
is
kind
of
variety
of
data
sets
that
we
call
that
we're
encountering.
So,
as
you
had
mentioned,
we
do
a
lot
of
work
with
public
web
data
that
was
kind
of
historically
been
our
bread
and
butter.
So
we
collect
petabytes
of
data
from
the
internet
and
use
that
for
those
kind
of
public
data
sources
and
ingest
them
into
our
platform.
B
We
also
work
with
proprietary
data
sets
internally,
so
we
have
kind
of
an
e-receipt
data
set
that
we've
been
producing
and
growing,
and
we
have
licensed
data
sets
that
come
from
a
variety
of
data
vendors
and
that
can
Encompass
a
number
of
different
Industries.
Some
are
more
specific
to
an
industry.
Some
are
more
about
just
kind
of
General
consumer
Trends,
General
market
overview
type
information,
and
so
the
thing
is
with
each
of
these
channels.
They
all
kind
of
work
with
data
in
different
way.
Public
web
data
is
completely
unstructured.
B
B
How
do
we
get
this
data
into
the
Delta
Lake
and
in
Delta
format,
and
not
worry
too
much
about
processing
the
data
at
the
interest
layer?
So
we
don't
do
too
many
Transformations
there.
We
just
want
to
take
that
data
wholesale
and
get
it
into
Delta
format,
depending
on
like
the
type
of
channels.
So
there
are
tools
for
kind
of
working
with
CSV
and
parquet
and
kind
of
just
native
file
formats
and
Sparks.
They
will
build
tooling
to
kind
of
just
make
it
really
easy
to
ingest
those
files.
B
B
The
more
high
volume
file
deliveries
that
we
have
there's
also
some
kind
of
solutions
where,
if
people
are
vendors,
that
we're
working
with
work
in
a
certain
data
warehouse
like
snowflake
or
as
well
as
just
kind
of
like
working
a
relational
database,
we'll
use
kind
of
third-party
Integrations
to
be
able
to
read
that
data
in
from
Snowflake
or
from
the
relational
database
and
kind
of
put
that
into
our
Delta
lake.
So
5tran
is
one
that
we
commonly
use
for
those
kind
of
patterns
as
well.
B
So
what
we're
really
trying
to
do
is
identify
just
really
Simple
maintainable,
Solutions
and
extensible
solutions
that
can
work
no
matter
what
the
data
is
to
just
bring
that
data
into
Delta
Lake
and
then
kind
of
work
with
the
data.
You
know
Delta
onwards
and
that's
been
kind
of
the.
The
main
strategy
is
like
how
we've
been
handling
ingests.
A
B
Yeah
great
question:
I
think
this
is
a
philosophy
that
we've
really
taken
to
heart
for
our
data
teams
and
they
they've
really
connected
the
dots
as
to
the
value
of
it
when
we're
working
with
data
sources
on
the
ingest
side,
one
of
the
things
that
we've
realized
is
something
can
go
wrong
on
the
ingest
pad
and
when
it
does
go
wrong,
what
do
you
do?
If
you
do
a
lot
of
Transformations
on
the
ingest
side,
then
you
start
introducing
another
variable
of
the
equation.
It
could
be.
B
B
This
kind
of
fidelity
and
reliability
is
really
important
to
them
because
they
need
to
be
sure
that,
like
if
they're
sending
something
to
our
clients,
so
they
can
go
back
and
see
the
source
data
and
make
sure
it
is
correct
or
they
can
do
follow-up
questions
or
anything
of
that
nature.
So
we
really
try
and
emphasize.
We
don't
want
to
do
any
Transformations
at
the
ingest
side,
bring
it
into
Delta
the
nice
thing
with
Delta.
Is
you
keep
like
a
transaction
log
of
all
the
data
that
you're
onboarding
into
a
Delta
table?
B
So
it
helps
a
lot
because
it
just
gives
people
more
confidence
internally,
that
the
data
that
they're
bringing
in
looks
like
something
at
a
certain
part
point
in
time,
and
then
they
can
kind
of
depend
on
that
protocol,
giving
them
consistency.
Moving
forward
from
there,
then,
like
there's
a
lot
of
transformation
that
happens
at
the
silver
and
gold,
Pipeline
and
I.
Think
that's
really,
where
we
kind
of
push
the
data
teams
to
kind
of
build,
whatever
pipelines
that
you
want
continually
work
in
the
Delta
format.
B
But
you
know:
if
you
mess
up
and
just
deliver
a
goal
layer,
that's
fine!
You
still
have
the
bronze
layer
to
go
back
to
and
re.
You
know,
redo
your
pipeline,
fix
your
mistakes
and
that's
been
a
great
way
to
kind
of
really
protect
our
data
assets,
protect
kind
of
the
pipelines
that
are
going
out
to
clients.
Things
like
that.
A
You
brought
a
very
good
point
here:
Anup
is
data
access,
I,
think
a
lot
of
organizations
are
still
thinking
about
how
they
structure.
You
know
access
levels
for
each
each
table
for
each
table
so
that
both
their
use
cases
are
done
properly.
They
have
the
data
that
they
need
and
also
like
put
some
guard
rails
together
right.
So
you
did
talk
about
like
you
know
how
you,
how
you
have
the
control
for
Downstream
users
in
the
silver
and
gold
there.
A
B
Yeah,
it's
kind
of
evolved,
I,
think
that
was
definitely
our
underarching
idea
like.
Yes,
we
want
to
protect
the
bronze
data
and
make
it
very
difficult
to
be
able
to
delete
or
corrupt
that
data,
and
so
for
a
lot
of
our
bronze
tables.
They
will
typically
be
read
only
for
many
teams
and
one
thing
about
the
bronze
silver
gold
that
I
should
mention.
B
Is
we
see
it
as
a
philosophy,
but
because
of
our
kind
of
data
mesh
type,
our
approach,
one
team's
kind
of
gold
table-
might
be
another
team's
bronze
table,
and
so
it's
kind
of
like
this
self-fulfilling
cycle,
so
even
for
the
gold
tables
that
one
team
produces,
we
make
them
read
only
for
another
team
that
might
be
consuming
it
just
so
that
there
isn't
a
way
for
them
to
corrupt
that
data
flow.
So
short
answer.
B
Yes,
we
want
to
make
the
bronze
level
as
read-only
as
possible
for
the
appropriate
teams,
and
so
we
see
that
at
kind
of
multiple
steps
throughout
our
organization.
But
that's
because
you
know
we
have
one
team
producing
data
for
another
team
and
then
that
team's
releasing
data
that
goes
out
to
our
clients
and
and
that's
kind
of
been
the
approach
that
we
take.
A
God,
that's
pretty
helpful,
I
know
so.
I
think
you
mentioned
a
very
fair
amount
of,
like
you
know,
complexity
of
your
architecture
as
well
as
like
some
of
that
architecture
decisions
you
made
I
wanted
to
ask
what
were
some
of
the
challenges
before
before
you
know
you
had
the
scalable
architecture.
What
were
some
of
the
challenges
you
are
dealing
with,
and
you
know
I
think
you'd
already
mentioned
about
some
choices.
You
did
some
choices
you
made
for
Delta
lake,
so
I
just
want
to
double
down
on
that.
B
Yeah
so
I
mean
kind
of
winding
back
the
clock
before
we
were
using
Delta.
Our
data
Lake
was
much
smaller
in
size,
we're
also
much
smaller
company,
so
it
was
still
quite
large.
I
think
we
had
like
around
four
or
five
petabytes
of
data
at
that
time
and
we
were
storing
all
of
that
data,
typically
as
external
parquet
tables.
So
in
the
absence
of
Delta,
like
our
gold
standard
of
light
hottest
or
data
was
parquet,
and
that
was
like
already
like
before
the
time
like
it
was
pretty
great
I.
B
Think
there's
a
lot
of
advantages
to
parquet
I
mean
Delta
uses
parquet
under
the
hood,
but
there
were
still
operational
challenges,
so
I
think
a
few
of
the
key
things
that
we
were
seeing.
Data
retention
was
a
real
challenge
because
we
would
have
pipelines
running
every
day
like
we
do
now,
they're
overriding
tables,
and
we
because
we
don't
really
have
a
protocol
like
Delta
like
a
transactional
kind
of
system.
We
don't
really
know
how.
If
we
can
guarantee
that
old
versions
of
tables
are
available
to
restore
in
case
people
make
mistakes.
B
We
did
have
custom
tooling
that
we
built
inside
to
kind
of
give
people
the
illusion
of
a
versioning
system,
but
it
wasn't
necessarily
Ironclad
I
think
it
was
very
hard
for
us
internally
to
guarantee
that
those
versions
of
those
tables
were
available
or
if
they
weren't
the
reason
for
why
that
retention
process
failed,
and
so
a
lot
of
that
internal
tooling,
which
is
having
a
hard
time
scaling
and
as
our
data
Lake
get
larger
and
larger.
Those
are
kind
of
custom
processes.
B
You
know
operations
and
Spark
pipelines
that
were
maintaining
in
data
engineering
that
just
couldn't
keep
up
with
the
volume
of
data
that
we
have.
So
that
was
one
large
challenge.
We
were
also
doing
a
lot
more
work
with
raw
file
formats
that
people
would
be
creating
external
parquet
tables,
sometimes
to
be
creating
tables
with
CSV
format.
There
wasn't
any
unification
of
how
people
should
be
organizing
their
data
and
that
could
produce
a
lot
of
challenges
in
performance
and
kind
of
the
quality
of
the
data
itself,
so
things
that
could
happen
in
raw
file
formats.
B
That
would
often
provide
us
data.
Corruption
schema
drift
difficulties
like
if
we
wanted
to
use
streamings
using
streaming
in
the
absence
of
Delta
is
much
more
difficult.
So
a
lot
of
these
kind
of
options
to
be
more
performant
and
be
you
know
more
reliable
with
their
data
pipelines
were
off
the
table
because
we
just
didn't
have
a
solid
kind
of
like
storage
layer,
and
so
when
we
were
looking
at
Delta
I
think
at
that
time.
B
One
of
the
key
things
you're
looking
at
is
like
how
to
use
streaming,
and
so
that
was
kind
of
one
of
the
big
momentum
shifts
like.
We
knew
that,
like
our
parquet
kind
of
data,
Lake
wasn't
scaling
but
wanted
to
use
streaming
in
some
situations
and
can
we
first
kind
of
switched
to
Delta
and
then
that
kind
of
made
the
streaming
and
the
data
retention
much
easier
to
manage.
A
Got
it
I
think
you
brought
a
very
good
point
on
streaming
I.
You
know
throughout
my
conversations
with
different
data
engineers
and
data
scientists.
There
is
always
this
myth
that
streaming
has
to
be
us.
They
always
associate
streaming
with
real-time
processing,
but
in
reality
it
depends
on
your
use
case
and
how
you
process
it
right.
So
can
you
share
your
perspective
on
that
like
how
to
keep
it?
Data
thinks
about
streaming,
yeah.
B
Absolutely
great
question:
I
think
we
we've
from
kind
of
the
Inception.
We
never
really
looked
at
streaming
as
a
real-time
solution,
but
what
we
were
looking
at
streaming
for
is.
We
have
large
volume
data
deliveries
from
some
of
our
vendors
that
are
coming
and
even
though
they
are
large
volume.
Historically,
on
like
a
daily
interval,
it
might
be
like
a
few
gigabytes
that
they're
delivering
every
day
the
entire
table
might
be
hundreds
of
terabytes.
B
So
we
ran
into
a
problem
because,
if
you're
not
using
streaming
using
batch,
where,
if
you
want
to
do
Transformations,
you
have
to
read
in
terabytes
of
data
only
to
process,
you
know
on
each
day
you
know
netting
you
a
few
gigabytes
of
data.
That's
been
added
that
was
really
inefficient.
You
know
expensive
from
a
compute
perspective,
would
break
down.
If
you
don't
have
enough
compute
in
your
spark
clusters
to
handle
into
streaming
just
made
like
much
more
sense,
conceptually
as
to
like
yeah.
B
We
should
only
process
the
data
that
we
just
received,
but
it
can
be
difficult
because
I
think
a
lot
of
the
documentation
and
kind
of
guidelines
out
there
were
designed
around
a
real-time
kind
of
data
flow,
so
something
that
we
learned
early
on
that
in
spark
streaming,
that's
pretty
cool.
Is
you
have
this
option
of
using
triggers
like
trigger
once
or
I
think
more
recently
trigger
available?
Now
that.
B
Yeah
that
can
help
control
a
stream
so
that
you
only
process
the
data
that
has
been
available
been
made
available
to
that
stream,
and
then
it
stops
the
stream
and
shuts
down
your
compute
process,
and
so
what
we
thought
with
that
was
like
really
substantial
cost
savings,
because
you
don't
need
to
keep
the
streaming
cluster
up.
24
7.,
you
only
work
with
just
a
few
gigabytes
of
data
that
you
care
about,
but
then
from
there
I
think
what
we
we've
really
seen
over
the
long
period.
B
You
don't
really
need
to
think
about
it
and
you're,
not
necessarily
having
to,
like
you
know,
run
the
entire
kind
of
data
process
on
the
entire
data
set,
which
is
going
to
be
really
risky
prone
to
failures
on
your
spark
cluster,
so
the
pipelines
that
we've
moved
to
streaming
and
we've
done
it
well,
we've
seen
that
they're
just
much
more
reliable
and
fault,
tolerant,
I.
Think
today,
like
that's,
how
we
see
streaming
and
internally
is:
do
we
have
like
a
large
volume
data
set?
Do
we
want
it
to
be
highly
reliable?
A
Got
it
got
it?
That's
really
helpful.
I
know
I.
Think
on
that
note
I,
you
know
I
want
to
understand
how
you
you
know.
What
does
the
frequency
of
data
processing
look
like?
Is
it
dependent
on
the
application
or
you
know
like?
Do
you
have
three
hours
window
when
you
process
the
data?
So
how?
How?
How
is
that
frequency?
How
does
that
frequency
look
like.
B
Yeah
it
really
it's
a
spectrum
I,
don't
think
we're
necessarily
taking
for
each
data
source.
The
same
approach
and
part
of
that
is
ultimately
like.
When
is
that
data
made
available
for
our
vendors?
That
might
be,
you
know
a
daily
delivery
or
weekly
delivery,
monthly
delivery,
and
so
that
will
dictate
the
kids.
If,
when
we
run
our
ETL
and
process
our
data,
some
of
them
are
high
volume
data
sets.
You
know
we'll
process
every
day
and
then
occasionally
we
might
be
processing
a
few
times
every
day.
B
I
think
that's
more
for
like
internal
data
sets
like
internal
logs
and
things
like
that,
where
we
need
to
keep
like
more
closer
to
real
time,
but
not
quite
real
time.
I,
don't
think.
We've
necessarily
quite
reached
a
point
where
real
time
is
necessary.
It's
kind
of
nice
to
have,
but
that's
been
kind
of
like
where
we've
seen
a
good
balance
of
cost
control.
Slash.
You
know,
recency
of
your
data.
A
Got
it
got
it?
So
you
know
some
of
the
blogs
that
you
know
you
wrote
unope
and
you
know
Steve
and
Andrew
wrote
I,
think
some
of
the
things
that
are
really
valuable
then
I
am
giving
references
is
like
you
know,
cost
savings,
and
you
did
mention
that
quite
quite
a
lot
during
our
conversation.
So
how
do
you
can
you
give
some
tips
to
our
viewers?
How
do
you
manage
and
keep
your
cost
efficiently
while
running
at
scale.
B
Yeah
costs
are
always
I,
think
top
of
mind
for
for
any
organization
right
now,
I
I,
think
for
if
you're,
using
Delta
and
I'll
try
and
focus
on
just
kind
of
like
constant
relationship,
cost
optimization
for
Delta,
Lake,
I,
think
there's
two
variables
or
two
drivers
of
cost:
one
is
storage,
overhead
and
one's
compute
overhead.
The
good
thing
is
I
think
there
are
techniques
that
Delta
offers
to
provide
solutions
to
both
I'll
start
with
storage.
B
I
think
the
biggest
you
know,
surprise
for
people
that
are
just
using
Delta,
like
might
be
like
versioning,
is
awesome
that
it's
baked
into
the
tables,
but
that
can
really
start
adding
up
and
accruing
on
your
storage
layer,
because
each
version
is
just
a
new
set
of
files
for
that
table.
So
if
you
don't
really
have
like
a
data
retention
policy
in
mind
or
think
about
that
actively,
that
can
be
really
surprising
and
we've
even
seen
situations
internally
where
we
had
a
Delta
table.
B
That
was
updating
and
someone
forgot
to
vacuum
it
and
all
of
a
sudden.
It's
like
massive
in
our
S3
bucket,
but
really
the
table
is
small
overall.
So
some
things
that
Delta
offers
you
know
use
vacuuming.
We
basically
have
written
kind
of
tooling
internally
like
when
someone
creates
a
Delta
table
or
overwrites
it.
We
just
vacuum
it
automatically,
based
on
a
retention
window,
we're
comfortable
other
things
that
are
that
are
on
the
storage
layer.
That's
interesting!
B
B
That
kind
of
has
like
a
knock-on
effect
which
I'll
get
into
compute.
Now
it's
just
your
compute
processing
on
Delta
tables
will
typically
be
better
because
your
files
are
more
reasonably
organized.
You
don't
have
to
worry
about
kind
of
keeping
things
like
repartitioning.
Your
files
before
you
write
it
out
or
anything
like
that,
like
Delta
just
takes
care
of
it.
But
what
were
you
see
on
the
compute
side?
Beyond
just
kind
of
Auto
optimize?
B
Is
you
can
use
things
like
Z
ordering
so
we've
seen
like
in
certain
situations
where
you
have
very
large
tables,
but
people
are
querying
those
tables
on
certain
sets
of
columns.
The
ordering
can
be
super
effective
in
bringing
down
the
compute
cost
of
those
Downstream
workflows,
because
you're
really
just
paying
for
scan
time
on
the
Delta
table,
and
it's
de-ordering
you
just
kind
of
erase
that
and
make
it
very
efficient.
B
So
we've
seen
workloads
go
from
like
hours
to
like
minutes,
because
we've
de-ordered
on
the
right
columns
wow,
but
but
you
have
to
like
be
very
tactical
and
when
you
use
the
ordering,
so
it
is
a
compute
intensive
operation,
but
it's
kind
of
like
an
investment
up
front.
You
pay
that
computer
ahead
of
time
to
organize
your
data.
So
as
long
as
you're
querying
on
those
kind
of
z-ordered
columns,
you
get
a
lot
of
benefits
and
that
can
have
a
lot
of
kind
of
knock-on
benefits.
B
So
I
feel
like
those
are
kind
of
some
of
the
key
things.
There's
also
streaming,
as
we
were
talking
about
that
can
get
path
to
cost
savings
to
using
things
like
trigger
ones.
Trigger
available
now
can
really
make
working
with
large
Delta
tables
really
efficient,
and
the
nice
thing
is
it's
kind
of
all
the
same
spark
data
frame
API.
So
it
doesn't
feel
that
different
from
batch,
if
you're
starting
to
use
it
with
Delta.
A
B
Yeah,
it's
something
that
we're
still
trying
to
get
like
a
better
handle
on
so
I
would
say
one
thing
that
we
look
at
is
you
know
we
first
don't
really
think
about
starting
unless
we
need
to
so.
If,
unless
someone's
really
staying
or
we
see
in
kind
of
our
databricks
usage,
these
workloads
are
really
slow.
B
We're
not
really
thinking
about
de-ordering
actively,
but
where
we
do
start
looking
at
is
like.
Are
there
certain
tables
that
multiple
Downstream
teams
are
reading
from
so
like
when
we
talk
about
that
data,
mesh
diagram,
there's
sometimes
Central
data
teams
and
multiple
sector
teams
that
might
be
reading
from
those
tables?
And
there
might
be
just
one
set
of
tables
that
everyone
is
reading
from.
So
those
are
great
candidates,
because
there's
a
lot
of
read
operations
going
on
and
then
we
look
at
how
are
people
reading
it
oftentimes?
It's
like.
B
So
if
you
have
like
a
streaming
based
or
incremental
workflow,
like
the
columns
that
you're
merging
on,
if
you
see
the
order
on
we've,
seen
like
a
nice
performance
bump,
but
I'd
say
like
it's
very
situated
additional
and
I
think
you
don't
need
to
over
optimize
unless
you
need
to
so.
You
know,
I'd
say
like
always
measure
first
understand
where
your
overhead
is
or
where
the
bottlenecks
are
before.
I
use.
Some
of
these,
like
techniques.
A
Wow,
that's
really
helpful,
yeah!
It's
it's
interesting.
How
long
we
have
come
along!
I
know
you
know
back
then
even
at
databricks,
you
know
we
couldn't
understand
the
scale
of
our
customers
were
running
on
and
I
I
still
remember.
You
know
one
of
our
support
person
reaches
out
to
me
and
says
like.
Oh,
you
put.
Data
is
running
like
this
huge
other
chatbots
or
something
because
you
were
running
like
quite
some
experimentation
on,
like
you
know,
growing
volumes
of
data,
so
I
I
do
remember
our
like
scaling
process.
A
B
I
think
we've
definitely
grown
together
with
data
bricks,
on
kind
of
understanding
how
to
use
Delta.
A
lot
of
these
lessons
that
we've
been
talking
about
just
in
this
talk
like
have
been
coming
from,
like
guidance
from
data
breaks
like
Solutions,
Architects
PMS.
You
know
people
like
yourself
so
I
think
it's
been
definitely
a
growing
domain
area
and
I
think
we're
still
trying
to
learn
more
about
like
how
to
make
the.
B
A
B
Yeah
I
think
I'll
just
kind
of
end
on
this.
You
know:
we've
been
very
successful
at
using
Delta
I
think
if
you're
kind
of
an
organization
trying
to
think
about
your
data
Lake
and
how
to
organize
it,
I
definitely
Advocate
looking
at
Delta,
and
it
really
provides
a
lot
of
futility.
I
think
for
us,
we've
noticed
is
because
it
just
kind
of
standardizes
a
lot
of
operations
of
how
you
work
with
the
storage
layer.
There's
just
a
lot
of
operational
efficiencies.
B
You
get
compute
and
storage
benefits,
so
we
were
talking
about
just
the
real
Tailwind
to
your
organization,
so
I
would
say
like
definitely
strongly
consider
it.
If
you
are
thinking
about
looking
at
your
storage
layer,
because
there's
a
lot
of
benefits,
I
think
it's
been
surprising
how
much
it's
even
grown
from
like
when
we
first
started
using
it
towards
now
and
every
time
there's
like
a
new
feature,
that's
rolled
out
to
Delta
tables,
we're
like
so
excited,
because
every
one
of
those
like
150
000
tables
can
start
benefiting
from
it.
A
That's
absolutely
amazing,
I
know
yeah.
Thank
you
very
much,
yeah
that
those
are
some
of
the
amazing
lessons
and
I
hope.
Our
community
found
this
and
site
useful.
Just
a
quick
reminder:
our
office
hours
happens
every
two
weeks
and
you
know
for
this
month.
I.
We
had
this
amazing
session
with
Anup
and
you
definitely
should
attend
next
one
as
well,
because
we
will
have
Michael
ambers.