►
From YouTube: 2021-03-30 delta-rs open development meeting
Description
Tentative agenda :
* Neville to share the 2.6.0 parquet writer updates
* Misha sharing the dynamodb lock work * Review rustdoc progress* Granting committer privileges to Christian * Surfacing writer stats through arrow/delta-rs
A
All
right
welcome
everybody
to
the
regular,
as
in
every
other
week,
regular
delta,
rs,
open
development
meeting
today
for
the
tentative
agenda.
We
had
a
whole
bunch
of
topics
that
we
had
suggested.
So
I
tried
to
to
cut
down
a
little
bit
and
if
we
get
through
these,
then
we'll
go
through
everything
else
that
we
had
in
the
slack
thread.
But
at
a
high
level
I
would
enable
to
share
the
the
2.6
parquet
writer
updates.
A
A
I
also
wanted
to
make
sure
that
we
discussed
granting
chris
committer
access
and
then
hopefully
we'll
have
time
to
talk
about
surfacing,
writer's
stats
through
arrow
or
sorry
through
the
parquet
crate
or
up
through
delta
rs,
which
is
a
topic
that
christian
will
leave.
But
let's
go
ahead
and
get
started
with
you.
Nibble.
B
C
On
my
side
of
the
update,
so
I've
materially
completed
the
package,
2.6.0
right
support,
there's
only
one
item
pending
in
terms
of
data
types
that
need
to
be
written
and
that's
the
decimal
128
data
type.
It's
actually
dependent
on
a
an
open
pr
in
the
aero
site,
where
we
still
whistle
going
through
it.
So
it's
still
in
a
draft
state.
But
with
that
said
it's
it's
just
really
the
actual
type,
but
the
underlying
you
know
fixed,
fixed
length,
binary
type.
C
We
can
already
write,
so
it's
just
semantics
waiting
for
it
to
sort
of
be
ticked
off,
and
then
we
can
sort
of
say
we
completely
done
with
that.
So
I've
just
attached
the
umbrella
jira
on
the
on
the
chat
now
I'll
also
put
it
on
on
slack,
but
it
just
talks
about
the
the
final
work
that
I've
been
doing
on
the
on
the
right
support.
You
know
supporting
nanosecond
timestamps.
C
I
think
that
was
the
big
one
for
a
couple
of
people
and
then
what
I've
also
done
in
this
in
the
past
week
is
I've
so
when,
when,
when
trying
to
write
tests
or
all
or
benchmark
or
even
you
know,
check
check,
what's
working,
what's
not
working,
it's
been
very
difficult
because
you
sort
of
have
to
write
the
schema
by
hand
and
then
generate
the
data
by
hand.
So
I
opened
the
pr
which
I
mentioned
today
to
allow
us
to
generate
arbitrary,
random
data.
C
So
you
just
provide
the
schema
that
you
want
very
useful
if
you've
got
like
deeply
nested
structures,
you
know
with
a
struct,
struct
list,
etc,
and
then
you
supply
the
number
of
records
you
want,
and
then
it
generates
that
for
you.
So
with
that,
I'm
also
just
creating
some
benchmarks
to
look
at
a
couple
of
to-do's
that
I'm
left
with
in
terms
of
cleaning
up.
C
You
know
the
stuff
that
I,
where
where
I
would
write
a
vector
of
booleans
instead
of
a
bitmap
just
so
that
I
could
be
able
to
diagnose
easily
without
having
to
worry
about
bit,
manipulation
logic.
So
there's
a
bit
of
those
things
that
I
need
to
clean
up,
I'll,
run
them
up
and
just
put
them
in
a
in
a
single
gyro
so
that
I
can
check
them.
But
that's
it
from
my
side.
A
A
C
So
bulletproof
is
still
in
draft
mode,
but
I
just
need
I
just
need
to
figure
out
integration
tests
and
then
yeah
and
then
the
creation
client
within
s3,
but
other
than
that.
The
the
data
log
workflow
is
this:
it's
it's
copied
from
java
info,
meaning
that
it
supports
all
of
the
major
features,
such
as
acquire,
lock,
updating
and
releasing,
and
with
a
with
a
simple
addition
from
my
site.
C
Is
that
there's
the
specific
scenario
where
java
impul
is
lacking-
and
I
addressed
that
in
this
implementation-
is
where
the
orchid
that
holds
a
lock
dies
and
then
there's
more
than
one
workers
trash
to
acquire
this
lock
and
in
java
impo
when
when
they
try
to
acquire
a
lock,
there's
least
duration
for
which
they
wait
like
by
default,
is
22nd
and
only
then
it
is
able
to
expire
lock.
But
the
issue
is
that
they
are
not.
C
If
the
lock
is
changed,
they
are
not
extended,
at
least
at
least
duration,
so
only
for
example,
if
you
have
like
more
than
one
workers
and
both
of
them
like
example,
if
both
of
them
wait
for
20
seconds
and
only
one
which
is
the
the
faster
one
will
get
a
lock,
another
will
will
fail
with
a
with
a
timeout.
So
that's
that
should
not
be
our
case.
C
I
extended
that
logic
with,
with
the
check
that,
if
the
second
work,
which
is
the
which
does
not
acquire
a
lock,
it
says
that
the
actual
lock
has
changed
like
that
record
version
number
has
changed.
It
will
still.
It
will
then
increase
eight
amount
duration
that
may
cause
to,
for
example,
a
waiting
move
for
more
than
20
seconds.
But
if
the
worker
that's
faster,
which
require
lock
releases,
then
then
we'll
get
the
normal
workflow
anyway.
A
I
ask:
can
I
ask
you
and
I
think
qp
you've
been
collaborating
on
this
a
little
bit.
I
wouldn't
ask.
I
see
in
this
draftpool
request
that
there's
a
new
feature
like
a
new
cargo
feature
for
dynamodb,
that
we'd
be
adding.
I
was
hoping
that
maybe
one
of
you
could
sort
of
share
the
reasoning
behind
adding
another
another
feature
flag
here.
C
Yeah,
I
can
answer
that,
so
the
I
we
have
not
yet
integrated
the
911db
into
a
street
storage,
but
the
idea
is
that
not
the
every
use
case
will
need
it
in
the
dynamodb
lock,
for
example,
if
that
simple
worker
central
worker
be
a
behavior
right
and
they
might
not
use
the
aws
s3
something
like
this,
so
they
they
won't
need
a
dynamo
db,
lock
within
it.
So
they
might
they
just
so.
C
And
also
recently,
when
I
was
not,
I
I
had
a
meeting
with
chris
on
kafka
delta
ingest
and
we
were
talking
on
how
we
can
benefit
from
a
delta,
rs
dynamodb,
lock
that
there's
there's
additional
meta
field
data
where
we
can
store,
for
example,
for
kafka
delta
ingest.
We
might
store
a
kafka
offset
and
for
delta
arrest.
We
might
store
a
latest
latest
delta
version.
A
C
It's
it's
also
in
java
import,
but
in
lack
of
simplicity,
I
was
not
a.
I
was
not
using
a.
I
was
not
including
that
in
delta
arrests,
but
which
recently
figured
that
it
will
might
be
beneficial
for
us
for
kafka
delta
ingest
to
store
the
latest
offsets
and
for
a
delta
rs.
You
might
use
that
for
latest
version.
So,
instead
of
relying
on
optimistic
currency,
where,
where
worker
tries
to
create
a
new
delta
version
in
loop
right
with
atomic
rename,
it
just
might
start
from
the
latest
version.
B
C
Also,
just
to
add,
in
this
dynamo
db
lock,
I
have
to
try
to
copy
every
a
every
like
a
dynamo
schema,
a
another
structure
from
a
and
and
and
a
and
the
final
code
to
be
similar
to
java
impulse.
So
I
in
case
there
will
be
an
actual
part
of
that
library.
In
us.
We
will
be
able
to
easily
immigrate
and
yeah.
That's
it
for
me,.
D
A
E
Yep,
this
part
is
easy,
so
we
have
a
work
in
progress.
Pr
open
for
adding
rust
stocks
for
all
our
rust
coolbase
florian
has
already
added
docs
string
for
all
the
python
binding
already,
and
the
goal
of
that
is
to
once.
We
have
all
the
docstring
added
to
public
interfaces.
We
will
enable
missing
dogs
linkedin
rule
so
that
all
future
code
change
require
doctrine
in
any
new
public
interfaces
so
that
we
could
make
sure
that
our
documents
are
documented.
D
A
What
I'd
recommend
for
anybody
that's
working
on
this,
and
this
goes
for
anybody.
That's
watching!
The
video
is
to
also
maybe
announce
what
area
you're
working
for
working
on
for
adding
rustoc
in
the
slack
channel.
To
make
sure
that
you
know
two
folks,
don't
have.
You
know,
spend
a
lot
of
time,
adding
the
same
restocks
qp,
I'm
assuming
that.
So
this
is
pull
request.
156,
I'm
assuming
once
this
goes
green.
We
merge
it
and
then
that's
that
right,
yep.
A
A
I
believe
the
next
thing
that
I
wanted
to
to
have
us
discuss
is
qp
suggested
that
christian,
who
is
the
big
letter
c
in
this
video
christian,
be
granted
commit
privileges
to
the
delta
rs
repository.
I
think
florian
qp
myself,
I
don't
know
if
miesha
or
neville
are
committers
at
the
moment.
Like
a
look.
E
A
All
right,
I'm
also
I'm
also
cool
with
this.
So
christian,
I'm
clicking
the
buttons
now
hooray
great
power,
great
responsibility,
yada
yada,
yeah,
yep
yep,
so
christian's
been
great
at
rate
access
welcome
board.
Thank
you
for
your
contributions,
etc.
I
guess
we
should
go
into
a
next
topic.
That
was
actually.
This
is
the
last
topic
that
I
had
said
that
we
definitely
should
get
through
and
maybe
neville
and
christian.
This
is
y'all
that
stats
support
bubbling
up
through
delta
rs,
something
or
other.
D
Yep,
mostly
just
going
to
defer
to
neville
on
this,
but
basically
you
know
the
the
decision.
We're
trying
to
reach
here
is
for
deriving
the
stats
to
include
in
each
add
action
for
a
delta
transaction.
Where,
where
do
we
want
that
code
to
live?
Do
we
want
to
live
it
want
it
to
live
in
the
parquet
crate,
the
aero
crate
or
in
kraka
delta
ingest
running
compute
kernels
from
arrow,
so
neville's
done
some
research
on
this
so
I'll
hand
it
off
to
you.
C
Cool,
thank
you.
So
broadly,
there's
two
options.
The
first
option
is
to
compute
the
stats
using
the
aerocompute
functionality,
so
we've
got
the
minimum
maximum
kernel.
I
think
we
only
calculate
minimum
maximum.
The
null
count
is
already
provided
by
the
record
badge
and
then
there's
distinct
columns.
I
think
that
that's
that's
only
the
other
one,
but
it's
more
optional
than
the
others.
C
Now
with
the
parquet
one,
there's
a
slight
challenge,
but
I've
been
exploring
it
in
the
background
because
I've
mob
I've
I've
been
thinking
about
it,
but
not
actually
trying
it
out,
but
the
challenge
there
is
that
chris
needs
to
well
is
writing
data
to
an
in-memory
buffer
and
then
writing
that
in
memory
buffer,
then
to
to
s3.
So
after
you
write
the
file
and
you
close
the
file,
we
currently
don't
have
a
way
of
returning
the
statistics
of
the
file
without
actually
reading
the
file
again
now
to
read
the
file
again.
C
If
it's
an
if
it
was
a
file
on
disk,
it
would
be
fine,
you'd,
sort
of
instantiate
the
file
writer
and
then
just
retest
the
the
metadata
and
then
get
the
statistics.
But
with
an
in-memory
in
memory
file
that
has
just
been
written
to
of
a
bunch
of
data,
we
actually
don't
have
any
functionality
to
read
data
from
an
in-memory
source.
C
I
added
some
text
on
this
on
the
chat
on
the
select
channel
just
before
the
talk,
so
that's
option
two
there
on
and
there
also
option
one
I'm
looking
at.
You
know
the
writer.
The
close
function
returns
an
empty
result,
but
I'm
looking
at
whether
we
could
retain
like
the
file
metadata
and
I
can
potentially
share
my
screen.
Let
me
see
if
I
can
do
that.
A
D
Here
yeah,
it's
mostly
for
optimization
of
reads
when
running
queries
from
databricks,
so
they
spark
sql
queries
or
or
spark
queries
that
you
run
from
a
databricks
notebook
against
a
table
that
we
would
write
from
delta
rs.
Can
leverage
leverage
these
statistics
to
optimize
optimize
the
query
and
give
better
better
read
performance.
C
C
What
you
have
with
these
statistics
is
that
you'd
have
the
minimum
of
one
year
and
then
the
maximum
of
90.,
so
in
the
next
vowel,
if
you're
looking
for
somebody
who's,
let's
say
92
years
old,
then
in
this
case
you
won't
even
touch
this
file
at
all.
So
we
need
those
statistics.
Otherwise,
if
you
don't
have
them
here,
you're
forced
to
read
the
right,
the
whatever
readable
one
will
be
written,
we'll
be
forced
to
go
here
and
then
find
what
it
means.
Yeah
great.
B
C
So
this
this
guy
here
in
this
trade,
the
foul
writer
trait
before
this
change-
that
I'm
exploring
I'm
just
letting
the
compiler
guide
me
to
see
what
else
I
need
to
change.
If
I
get
stuck,
I
get
stuck
sort
of.
If
I
perish,
I
perish
so
this
guy
here,
I'm
when
you
close
the
file,
it
returned
an
empty
empty
result.
So
I'm
just
exploring
what
happens
if
we
return,
you
know
the
actual
metadata
the
file
metadata,
because
with
the
file
metadata
you'll
then
have
with
the
file
metadata
you'll
then
be
able
to
see
this
one.
C
Yes,
it's
this
one!
Sorry
I
keep
creating
like
random
ripples
just
to
try
stuff
out
with
the
file
metadata.
What
you'll
then
be
able
to
do
I'm
here.
So
here's
an
example
where
I
was
talking
about
the
random
data
generator
thing.
I
generate
a
large
enough
record
batch.
I
think
here
we've
got
like
20
million
records.
I
write
it
to
the
packet
file
and
then
to
get
the
to
get
the
statistics
I
need
to.
After
closing
it
I
need
to
read
back
the
file,
but
christian
wouldn't
be
able
to
do
this
currently.
C
C
No,
we
we're
not
using
error.
It's
it's
sorry,
it's
it's
actually
a
bit
inefficient
because
we
we
sort
of
doing
it
on
a
row-by-row
basis.
Okay,
no.
I
want
to
look
for
the
code
we
sort
of
doing
it
on
a
row-by-row
basis.
It's
one
of
the
things
that
I
wanna
look
at
improving,
because
what
happens
is
when
you
write
the
file.
You
can
provide
this,
that's
if
you've
pre-computed
them
or
if
you
you
can
provide
like
it's
it's
an
option.
C
If
you
provide
nothing
it'll,
then
compute,
the
stats
for
you
on
a
row
by
row
basis
so
where
we
could
use
arrow
is
actually
to
compute
the
stats
for
the
the
chunks
or
the
or
the
batch
or
whatever,
and
then
you
know
pass
them
to
the
to
the
writer
so
that
it
doesn't
compute
the
stats,
but
now
the
problem
with
that
is
that
sometimes
there's
a
there's
a
bit
of
a
disconnect
between
the
parquet
record.
Sorry,
the
error
record
badge
size
and
the
pacquay
row
groups,
so
you
would
have
a
in
this
instance.
C
It's!
It's
10
million
records
that
I'm
writing
here,
but
the
because
the
rust
implementation
actually
doesn't
split
the
the
batch
into
you
know
smaller
chunks.
It's
writing
the
whole
thing
in
one.
Go
sorry
I'll
stop
scrolling
in
a
bit
once
I
reach
I'm
going
here,
we
are
so
I've
written
a
481
megabyte
file,
which
is
one
row
group.
So
that's
that's
a
problem
from
a
parallelism
for
a
reader
perspective.
C
C
But
now
going
back
to
the
discussion
at
hand
of
the
of
of
whether
you
could
compute
the
statistics
from
arrow
the
problem,
there
is,
if
you
compute
the
statistics
at
the
at
the
parquet
right
level,
if
you
compute
the
statistics
for
the
whole
record
badge
and
the
record
batch
has
10
million
records.
What
happens
if
parquet
writes
a
chunk
of,
let's
say
a
hundred
thousand
records.
You
still
have
to
compute
the
stats
for
just
that
hundred
thousand,
so
I'm
still
exploring
that,
but
it
it
just
specifically
answers
your
question.
Quepita.
E
A
Yes,
unless,
from
a
from
a
performance
standpoint,
I
kind
of
I
don't
understand
why
one
would
pre-compute
the
stats
at
the
arrow
level,
because
it's
not
like
you
would
pre-compute
on
one
node
and
write
on
another
like
it's
just
you'd,
basically
be
just
deciding
where
you
want
the
cpu
overhead
to
be
in
a
single
process.
Either
way.
Wouldn't
you.
C
The
rust
library
computes
row
by
row,
so
it's
it's
part
of
the
problem
actually
with
well
the
performance
issue,
the
main
performance
issue
with
with
the
rust
parquet
writer
and
reader.
So,
unlike
the
the
c
plus
plus
implementation,
for
example,
where
they
sort
of
rewrote
the
parque
write
and
read,
support
you
know.
From
from
from
the
very
low
level
we
took,
we
took,
we
took
a
a
a
more
a
more
convenient
route
of
saying.
Well,
you've
got
the
low
level
where's
that
thing
you've
got
the
low
level
column.
C
Writer
you've
got
the
low
level
column
right.
So
this
is
where
the
stats
actually
compute
computed.
So
you
see
you
sort
of
say
the
minimum
is
the
minimum
of
of
oh.
No,
no
sorry!
This
is
not
it.
It's
it's
oh
yeah!
Here
it
is
so
this
is
where
you
sort
of
computing,
your
your
stats,
you
we
do
it
row
by
row
and
the
reason
why
we
do
this
is
that
we
we
use
this
function
of
writing
the
batch.
C
The
internal
batch
instead
of
what
the
c
plus
plus
implementation
did,
where
they
sort
of
re-re-worked
the
whole
support
from
scratch.
That's
why
it
took
them
like
over
over
a
year
or
so
to
to
eventually
get
the
the
the
right
support
completed.
But
what
we
did
is
we
sort
of
using
the
low
level
column
functionality.
C
C
We
would
iterate
through
the
you
know,
through
the
through
the
error,
column
and
compute
these
things
as
you
go
along
for
primitive
values,
it's
easier,
but
once
you
once
you
get
to
lists
and
structs
where
it's
it's
deepliness,
that
it
becomes
a
bit
tricky.
But
this
is
the
the
sort
of
direction
that
we'd
like
to
take
and
when
we
take
that
direction,
even
the
computing
of
stats
will
probably
be
a
bit
easier
because
we
would
sort
of
then
be
able
to
say
well
for
this.
C
For
this
column,
chunk
of
yeah
for
this
column
chunk
that
we're
on
the
right.
We
only
want
to
write
like
ten
thousand
of
the
hundred
thousand
records
in
this
in
this
error
column.
So,
let's
slice
the
let's
do
a
zero
copy
slice
of
the
arrow
column
compute
the
stats
for
that
quickly
and
then
pass
them
here
as
an
option.
Instead
of
then
having
to
calculate
them
sort
of
on
a
row
by
row
here.