►
From YouTube: 2021-06-08 delta-rs open development meeting
Description
Agenda not yet determined
A
A
The
meeting
you're
welcome
to
join.
If
you've
got
questions,
you
can
join
the
double
rs
slack
channel
and
we
can
discuss
them
the
agenda
today
that
I
suggested
and,
of
course
we
we
can.
We
can
modify
this
as
necessary.
I
wanted
to
really
go
over
three
things.
First,
I
wanted
to
to
talk
about
the
arrow
changes
that
christian
and
neville
y'all
were
chatting
about
in
slack,
if
possible,
to
see
if
there's
there's
anything,
I
can
do
to
jump
in
and
help
oh
and
welcome
kp,
and
then
I
wanted
to.
A
A
B
I
I
could
share
my
screen.
I
I
don't
know
if
it's
it's
necessary.
Basically
the
the
crux
of
the
issue.
It
is
in
order
to
write
delta
checkpoints
from
delta
rs.
Let's
see
what
I
can
do,
yeah
we
we,
I
you
know.
If
you
want
me
to
share
my
screen,
I
can
I
I
I
got
it.
I
have
to
go
find
the
code,
but
so
the
crux
of
the
issue
is
delta.
A
C
B
B
D
Yeah,
so
the
parquet
writer
gives
you
like
a
low
level
right
where
you
can
pass
a
what's
this:
a
vector
of
of
values
and
then
definitions
and
repetitions.
So
it
assumes
that
you've
already
done
the
calculation
for
them
and
that
you've
checked
that
everything
is
fine.
And
then
you
you
write
that
data.
So
all
data
types
are
supported
there.
Because
of
you
writing
a
bunch
of
you
know
it
could
be
primitives
in
3264,
float
or
bytes.
D
You
write
that
using
using
that
low
level
interface.
So
what
the
arrow
writer
does
is
it
gives
you
the
ability
to
take
error
data
and
then
it
sort
of
does
all
the
work
in
between
for
you.
So
you
don't
have
to
know
how
to
calculate
the
definition,
repetitions,
etc.
And
then
you
write
that
data
to
parquet.
D
So
that's
that's
the
work
that
I
was
doing
for
you
know
supporting
all
the
ik
2.6
data
types
map
map
type
was
not
a
priority
when,
when
I
was
doing
the
work,
so
we
didn't
focus
on
it,
partly
also
because
the
that
the
currently
isn't
map
type
support
in
the
rust
aero
crate.
But
it
is
there
in
the
specification-
and
it
looks
quite
similar
to
the
parquet
one.
D
D
I
think
I've
listed
those
four
things
there
on
the
issue
on
the
yeah
on
the
issue,
so
so
putting
it
in
array
which
includes
you
know,
quality
comparisons
etc,
supporting
the
data
type,
so
adding
an
enum
for
map
onto
the
you
know,
error
data
types
and
then
wiring
through
all
the
way,
I
think
supporting
supporting
it
from
a
json
perspective.
Christian.
You
can
confirm
here.
D
Okay
and
then
ipc
is
optional
for
our
case.
It's
normally
just
to
make
sure
that
the
integration
tests
with
the
rest
of
the
error
implementation,
so
looking
at
c,
plus
plus
as
a
defect
one
pass
and
then
once
we've
done
that,
then
we
will
then
be
able
to
wire
up
the
the
the
right
from
arrow
two
to
per
k.
So
there's
this
issue
395
and
then
there's
also
396.
I'll
open
them
one
after
the
other
396
becomes
a
bit
easy
once
395
is
implemented
because
effectively
what
a?
What?
What
a
map
is?
D
C
A
B
Yeah,
so
I
do
have
a
pr
up
for
delta
rs,
which
basically
what
I
what
I'd
like
to
do
is-
and
this
is
pr
number
280-
that's
on
the
screen.
Okay,
I
see
it
now,
so
I
feel
like
I'd
like
to
merge
this
before
the
map.
Sport
goes
in
it
and
and
basically
what
we'll
be
doing
here
is
we're
gonna,
be
writing
incomplete
checkpoints.
They
they're
missing
some
fields
that
that
are
important,
but
it
at
least
includes
some
testable
code
so
that
you
know
we.
A
B
A
B
B
A
I
like,
from
my
perspective
christian,
I
actually
have
no
problem
with
merging
code
that
is
not
quite
ready
into
maine,
but
because
we
have,
you
know
the
delta
lake
python
package
and
potentially
other
other
folks,
depending
on
what
goes
into
maine
or
what
we
release
to
delta
or
not
to
delta
sorry
to
create
io.
I
just
want
to
make
sure
that
we
we
guard
those
things
with
feature
flights
appropriately.
B
A
A
B
B
A
B
A
C
B
B
B
B
B
And
I
had
proposed
a
design
where
we
used
like
a
channel
and
like
a
checkpoint
writer
that
runs
on
a
separate
thread.
Qp
raised
a
really
neat
what'd.
You
call
it
idea,
I
guess
in
delta
rs
about
what,
if
we
could
run
these
checkpoints
as
lambdas
and
looking
into
that,
I
feel
like
that's.
Actually
the
best
way
to
do
it.
B
So
I,
rather
than
implementing
the
channel
structure,
I'm
starting
to
work
on
testing
running
a
lambda
function,
triggered
by
a
s3
event,
notification
of
a
when
any
file
lands
in
the
delta
log
that
ends
in
zero.
Basically,
and
so
the
idea
here
is
that
the
checkpoint
process
would
be
fully
decoupled
from
the
actual
application.
B
That
is,
writing
data
and
initial
transaction
logs,
and
this
thing
basically,
is
just
going
to
get
trigger
triggered
every
time
a
10th
commit
is
committed
to
the
transaction
log
and
create
a
checkpoint
from
that
the
pr
actually
references,
my
branch,
I
got
a
comment
in
there.
Let's
see
here
note.
I
also
have
this
dirty
thing
right
yeah,
so
this
shows
the
diff
where
I'm
I'm
working
on
a
lambda
checkpoint.
B
I've
I've
got
the
initial
code
written
and
I'm
thinking
thinking
this
would
make
sense
as
a
binary
to
include
in
delta
rs
welcome
for
feedback
on
that
on
that
thought.
If,
if,
if
anybody
think
that
thinks
that's
a
bad
idea,
but
the
idea
is,
we
would
build
this
binary
and
and
make
it
available
for
anybody
to
kind
of
host
in
aws
man.
A
B
D
C
Delta
files,
and
so
in
the
threat
in
the
snack
channel,
we-
and
so
that's
what
started
a
bunch
of
discussions
on
what
other
ideas
we
can
do
with
lambda
within
delta,
and
then
this
is
checkpointing
is
one
of
the
ideas
and
then
there
are
a
couple
more.
We
think
pretty
cool
things
we
can
do
with
them.
Those
in
the
for
delta
tables
as
well-
and
another
thing
I
would
like
to
point
out-
is
lambda
checkpointing.
It's
optional
like
this
is
not
enforced
in
the
delta
rs
design.
C
A
Yep
speaking
of
hacking
on
things
like
lambdas-
let's
talk
about,
I
just
want
to
mention
the
delta
hack
stuff
that
we
have
for
later
in
the
week.
There's
this
official
official,
very
official
event
that
denny
and
I've
been
putting
together
delta
hack,
2021
I'll
drop
this
in
the
in
the
slack
channel
as
well.
We
actually
have
a
mystery
judge,
so
denny
ryan
and
I
are
going
to
be
a
judge.
A
I've
got
the
mystery
judge,
I'm
not
going
to
spoil
the
mystery
just
yet
for
some
of
the
projects,
but
some
of
the
things
that
we
wanted
to
do
along
with
this
were
schedule.
Some
some
like
some
live
streams,
and
things
like
that.
So
on
the
youtube
channel
already
there's
the
introduction
which
danny
is
going
to
do
and
then
I'm
going
to
do
a
delta
rs
live
coding
session,
just
creating
creating
a
pull
request
for
delta
rs.
B
A
A
A
D
I'm
actually
in
the
deeper
thing,
so
I'm
still
working
on
the
array.
I've
done
the
data
type,
I'm
working
on
the
array
bit,
I
I
can.
I
can
drop
like
an
estimate.
I
I
don't
ever
guess
this
yet
because
I'm
still
going
into
the
deeper
thing,
so
the
other
thing
I
was
doing
last
night
was
looking
at
the
the
formal
specification
to
see
how
that
you
know
the
conversion
conversion
to
json
looks
like
so.
I
let
me
say
I
can
give
you
an
estimate
by
tomorrow.
C
D
A
C
So,
since
flooring
challenge,
I
just
want
to
mention
one
quick
thing,
which
is
foreign
found
that,
during
the
python
release
our
documentation,
release
pipeline
is
broken,
probably
due
to
default
permission
change
from
github
action.
We
have
since
fixed
the
pipeline,
but
the
documentation
for
python
will
still
be
out
of
date
until
we
do
the
next
python
release
just
fyi.