►
From YouTube: 2021-03-16 delta-rs open development meeting
Description
Tentative agenda :
* @Neville Dipale to share the 2.6.0 parquet writer updates
* @QP to review the recent Python and Rust bug fixes
* Try to come to a decision on a path forward for atomic rename support (led by @misha and @Christian Williams)
* ???
A
A
I
saw
somebody
knock
at
the
lobby,
but
I
can't
actually
see
it
on
my
screen.
Let
me
go
to
a
different
view:
okay,
sorry
anyways.
So
the
delta
rs
slack
channel
details
are
in
the
readme
on
github
and
the
agenda
for
today.
I
wanted
to
start
off
with
neville
for
a
change
so
that
you
can
sort
of
give
you
your
parquet
update
and
then
we'll
move
on
from
there.
B
B
After
that,
I'm
going
to
be
able
to
work
on
the
timestamp
nanosecond
support,
and
I
can't
remember
what
else
it's
just
two
types
of
arrays
that
we
can't.
We
can't
write
as
a
limitation
because
we
didn't
really
support
2.2.6
yet
and
then,
after
that,
I
think
I'll
be
done
with
the
actual
schema
support
of
2.2.6,
so
we'll
be
able
to
start
reading
a
lot
more,
a
lot
more
types
of
files
that
are
that
are
written
with
2006.
B
A
A
A
And
for
for
anybody,
who's
not
already
aware
that
the
reason
like
the
the
rust
crate
and
the
python
binaries
those
depend
on
get
versions
of
the
aero
underlying
crate.
The
big
motivation
to
depend
on
a
released
version
of
the
aerocrate
is
so
that
we
can
publish
a
new
delta
light
crate
to
creates.I
o
it's
not
blocking
any
of
the
python
work
or
any
of
the
work
that
the
kafka
delta
ingest
work,
which
is
downstream
of
delta
rs.
A
B
So
luckily,
and
another
really
good
news
is
misha-
has
added
a
oh
on
that
list,
be
sure
I
said
added
a
single
writer
support
for
our
s3
backhand.
So
we
are
now
at
least
feature
comparative
with
the
the
reference
scholar
implementation,
which
also
only
has
single
s3
support
and
we're
now
working
on
multi
rider,
s3
support.
C
A
B
So
there
was
some
discussion
about
adding
tests
for
that.
For
that
particular
case,
it's
not
it.
We
so
the
test
data
we
have
today
it's
enough
to
to
cover
that.
It's
just
that
we
need
to
add
some
more
functionalities
in
delta
rs
to
expose
some
of
the
operational
metrics
for
us
to
actually
write
the
unit
test.
A
B
B
A
A
A
D
D
They
they
cover
only
very
basic
operations
such
as
create
a
acquire,
a
lock
release
that
and
move
on.
But
if
something
happens,
wrong
like
the
lock
has
has
died,
there's
no
way
to
recover
this
unless,
unless
you
manually
delete
a
delete,
a
number
dbo
which
will
require
like
calling
the
number
db
elsewhere
without
like
using
dynalog
api.
D
So
in
order
to
use
that
we'll
still
need
to
write
some
dynamodb
related
code
and
also
there's
there's
another
amazon,
dynamodb
lock,
client,
which
is
created
by
the
aws
lab
labs,
and
it
also
there's
a
lot
of
ports
like
to
nodejs
and
go
which
which
actually
like
pours,
and
then
I
tested
that
and
it's
actually
fully
functional
for
us.
It
supports
both
like
acquiring
logs,
updating
and
also
fail.
D
So
the
main
issue
is
that
if
you
start
worker,
for
example,
worker
a
on
amazon,
the
number
to
be
a
like
java,
client
and
if
it
dies,
then
the
other
worker
reads
the
previous
lock
item.
It
says
when
lock
is
created
and
it
has
like,
for
example,
lease
duration
for
30
seconds,
it's
very
30
seconds
and
then
deletes
previous
logs
and
creates
new
one
and
that
functionality
of
waiting
and
reupdating
the
lock
is
missing
in
dynalog.
A
C
In
the
reference
implementation,
there
is
no
pessimistic
clocking
happening
like
this.
It's
basically
there's
a
there's
a
file.
I
can't
quite
remember
the
name
of
it,
something
like
single
driver,
s3,
storage,
backend,
or
something
like
that.
So,
basically,
it
seems
like
they're
they're
funneling,
all
of
the
s3
interactions
through
a
single
thread
on
the
driver.
B
E
Sorry
I
meant
to
chime
in,
but
I
completely
forgot
that
I
muted
myself,
that
is
correct
because
in
essence,
in
order
to
ensure,
when
you're
they're
doing
the
renames
or
the
rewrites
for
the
json
that
basically
it
actually
at
that
point
in
order
to
be
the
most
safe
in
and
easiest
implementation.
The
I
that
was
the
idea
that
we
would
only
have
a
single
writer.
A
I'm
wondering
this
is
this
is
not
something
I
think
we
necessarily
need
to
decide
now
and
there's
a
fairly
robust
discussion,
which
I'll
link
in
the
slack
channel
that
mostly,
I
think,
actually
entirely
christian
and
qp,
have
been
having
around
the
the
create
not
of
this
exist
guard,
which,
I
believe
is
related.
Is
this
issue?
It's
the
same
thing
I
think.
A
But
what
what
comes
to
mind
is
that
whatever
functionality
we
add
here,
because
if
we
are
talking
about,
you
know
dynamodb
being
another
dependency
to
have
these
safe,
multiple
writers
that
I
think
we'll
need
to
provide
a
flag
so
that
you
know
if
you
want
to
use
the
python
binding
or
the
rust
crate
directly
you
won't
have
to
like.
If
you
want
to
shoot
from
the
hip
and
be
you
know
cowboy
coder
you
can
you
can
disable
it
and
not
have
the
requirement
for
dynamic
tv
and
aws.
B
Yeah,
I
think
so
that
our
design,
so
we
we
had
two
designs
in
that
discussion.
One
of
the
designers
moved
the
transaction
log
entirely
to
dynamodb
that
designed
it
more
performance
and
safe,
safer
from
a
lifeless
point
of
view,
but
it
has
the
downside
that
all
the
readers
has
to
use
dynamodb
as
well.
So
we
went
with
this
locking
design
where
only
the
writer
needs
to
use
dynamodb
and
readers
can
just
use
s3
without
accessing
dynamodb
at
all
and
and
so
on.
B
D
B
A
A
B
D
Ideally,
it
would
be
nice
to
have
a
full
port
like
go
library
and
not
gs
did,
but
it
will
be
time
consuming
to
port
everything
from
a
java
client,
since
this
is
really
rich,
it
supports
like
background
threads,
which
which
which
sends
heartbeats
it's
each
second,
so
they
fully
support
those
errors,
but
I
I'm
pretty
sure
for
mvp.
We
won't
need
all
all
of
that.
So.
B
A
Well,
like
what
I'm
hearing
is
that
you
know
there's
two
two
poles
here:
one
is
dynalock
which
doesn't
have
everything
that
we
would
need,
and
the
other
is
like.
The
the
aws
labs,
dynamodb,
lock,
client,
some
implementation
of
that
which
has
a
lot
of
extra
functionality
and
what
we
actually
need
is
somewhere
between
those
two
points.
What
I'm
asking
for
is
for
somebody
to
actually
write
down
what
it
is
we
actually
need
so
that
we
only
implement
what
we
need
in
delta,
rs,
yeah
and
not
implement
amazon
db.
Their
tenant
would
be
lock.