Delta Lake delta-rs Development Meetings, 13 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2021-04-13 delta-rs open development meeting

Description

Tentative agenda :
* Demo use-case of Python in production * Discussing Data and AI Summit plans * ???

A

All right, oh qp, is very yellow welcome to the uh delta rs open development meeting um qp. I'm really not sure why you're yellow, oh now, you're not yelling for me anymore.

A

um I don't believe florian will be joining us, uh so we unfortunately won't have uh the use case that he was gonna share last time that we didn't have time for so I wanted to um discuss the data in ai summit coming up and what we might be able to do from the delta rs perspective or talk about the delta like python binding. Or what have you but um danny? Could you maybe give us a high level like here's, here's, the gist of what's happening with this event,.

B

Yeah, so in terms, I wanted to actually work with you guys to figure out how we want to do any particular press releases or any launches or of thing of that nature. The current plan basically is to announce delta lake 1.0 uh prior to um uh summit. So that way uh we can get the press releases out.

B

um We want to definitely include have multiple press releases, I'm just trying to get figure out how to work with linux foundation on it. They they have a particular set of template and rules that they would like us to follow, and so they're actually looking for what the rules are right now.

B

So as soon as the lf tells me, what we're allowed doing then we'll go ahead and then um usually how it works, at least in the past, is that will be our respective companies that will basically craft the messaging and all that stuff like basically well us and our respective, like companies, basically will do that.

B

But then we go work with the lux foundation to push it out, and so that's the idea at least and so we're trying to figure out what the timing uh we did plan on, uh calling out the delta rs project by the way uh during one of the keynotes. So I wanted to call that out. Whoo, yes, so.

A

uh When, when you say delta lake 1.0, uh we're of course talking about the reference implementation there does that mean, uh like is delta lake, going to adopt semantic versioning around the protocol from then moving forward?.

B

I don't know yet.

A

B

Yeah, I think we're still trying to figure that one out, and so I think, if you can give me- I literally just returned uh from yesterday- so I'm still catching up- I freely admit so. The fact that I can give you even this and even remember that there was this meeting honestly.

B

This is sort of a pro at this point honestly.

C

It looks to me, like um the current political version, that's tied to the um the java, the scala implementation version, so that that seems to be the case right now.

D

C

A

I keep you what I I don't. I don't follow the development of the the reference implementation enough to know if there is an intent to actually follow semantic versioning there. That's more the question to me.

C

Yeah, I see yeah.

A

E

Yeah, even mentioning that um I um I remember seeing on a pr from from florian uh some disagreement between the protocol version and the current state of the reference implementation. I I can look that up, but.

A

I know I can I can. I can share that with denny. I know exactly which pull requests you're talking about okay,.

B

Cool yeah so definitely definitely share the pr with me, so I can go ahead and uh start those discussions more like start more like continue them, because I I did start them actually already based on some previous conversations. We had so.

A

um So doing press releases in some form around the delta project for data in ai summit is there? um Is there any other? I know I should say uh some of you know this, but maybe not all of you knows. I submitted a talk and it got accepted to give an overview of delta rs and the python bindings. And what we've been doing here, I kind of still have to create some slides there. So we'll have that and then we'll have the um the press releases.

A

Denny was there anything else in mind that you had for the delta project in general.

B

I mean honestly was basically promoting all those sessions right. So basically I figured whatever press releases we did and then whatever social we did, we want to be able to promote, and oh, I did want to call out since we're voting for committers by the way that I did want to actually call that out as well, uh whether that goes in I'm not sure it's supposed to. I still need to again follow what lf is asking us to do, but we definitely in our blogs.

B

We always go ahead and call out who are all the new uh committers. So we want to make sure that that's done, but I'm checking if lf is going to allow us to do it also in the press releases again, uh it's honestly, it's more semantics anything else, but still I got to verify so.

B

A

It's too bad. This isn't gonna be a physical event. You could have qp. I could be up on stage as mr delta rs I'm.

C

Not sure you want that.

B

It's okay, we're we're almost done by the time by the time. It's europe, we might actually be okay to do physical events. Yeah assuming things don't go.

A

So you're inviting qp to come keynote in europe. That's what I'm hearing hell! Yeah.

C

You're gonna regret saying that.

A

So uh some of some of the reason I wanted to discuss this is, uh I was brainstorming, some ideas with denny a couple of weeks ago around this and the python and rust's uh uh clients seem to be far enough along to where I was kind of hoping that that florian would be able to join us today. But I'm curious qp from your perspective. Are there any? Is there a milestone that we can announce or you know, plant a flag and say you know we did it.

A

uh You know in in time for uh data in a summit.

C

Yeah, I think the python binding right now for at least for read, it's feature complete, so um I've been following on a lot of discussions around reading delta table from using panda, pandas ud up. Sorry, not beauty, just pandas python package and um I've been commenting in all those threads um with delta rs right now and looks like delta rs fits all of the use cases. So um there are a lot.

D

C

Asking like how can I read delta lake from using pandas and um yeah? I've been answering all those questions online and I also have added a link to the artificial pandas documentation that links to delta rs as the recommended way to read delta lake using pandas and florian's also published a official python, documentation, user doc and api doc for delta rs python binding. So I like from a read perspective, I think we're pretty um future complete at this point.

A

I was looking at those docs yesterday actually because I was pointing them out to somebody. um We don't actually mention anything around pandas in our own docs. Should we.

C

uh We are not there's a example code block that uses panda to demonstrate the the feature, but we don't that's correct. Okay,.

A

Maybe I'll file a ticket about that. I think one of the things when I went in at the person I was showing this to I was like yeah. You can totally use python to easily query and you know outside of spark easily query delta lake and then I pulled up the dachshund like.

C

Yeah yeah there's only.

A

C

There's an example block that you see.

A

Yeah, I know I see what you're talking about when I'm looking at this now, um it's just the that two pandas um call which is helpful, but so there's your full reader support. You say um I I get the impression that florian's use case doesn't require writing to delta. Like is, is there a ticket to like pull that writer support up um into the python bindings, or is it there and just not used.

C

It's not there. We don't. We haven't implemented that yet gotcha, it's not exposed yeah. So there is another ask from another community member about um exposing the metadata api um and.

A

C

Took up on that one.

A

Okay, so we can announce full writer compatibility there um danny do you do we lose.

C

Any yeah we lost it.

A

F

All right, it's coming back.

E

Tell you what, on that python, binding support and the pandas use cases I feel like once we get pd from delta. Oh my god! That's that's great.

A

From delta, what do you mean.

E

uh Pd is like penta, it's a common, alias for painting.

A

E

A

It's like five letters: why do you need to abbreviate them.

B

It's just what everybody does dude dude come on. Yeah come on numpy as np right.

G

Pencil plus t out all right come on, you know we do. We we're we're we're that.

C

We're so lazy exactly.

A

It's like the military like: let's not, let's not use words. Let's instead talk in acronyms.

B

Hey I'm up for it, I'm pretty sure we can figure out how to do an entire conversation. This way.

A

Okay, so danny I'm glad you I'm glad you returned um for for the verbal posting, but also.

A

Once upon a once upon a while, I had asked you about how we would how we, as like the delta, rs, committers and contributors, would be able to create blog content for delta.io, and I think, uh if, if we go through the trouble of writing documentation for, like writing a tutorial like here's, an easy problem that you can use pandas to to read and deal with delta lake, that's something that I would love to put up on the delta.

A

I o blog, but I'm not sure if we would like the way that I'm thinking about this is in the lead up to uh data in ai summit, which is at the end of may. For for the rest of you who don't just have this in your head already um would be to have at least one or two blog posts go up somewhere like they could be. On personal blogs. I know.

A

Qp's personal blog is a little bit uh under appreciated, but every post there goes straight to number one on the news apparently, um but like we can write some stuff that doesn't necessarily fit in a press release, but would be a good intro to here's the thing: here's how you do the thing with the stuff.

B

Okay, so quick call outs right now: the delta dot io site has only it has posts which basically shows up on the right side um and so like. If you look at the main page of delta.oh,.

A

There I'm going to.

B

Share this screen. Oh sorry, yes.

A

Application window.

B

Boom. Okay, so these are the latest news like so here on the right side here like so those are basically posts but they're, not very long-lived. You can see them, but they they're not very well done. You can also click on resources. Oh yeah, they got blogs, yeah, that's fine, too yeah, and then this gives you the blogs, which I need to go up. Wait.

A

This is a different list of things.

B

Yes, because that references external blogs, as opposed to the yeah, so now we are planning to update delta dot io site to actually have its own vlogs as well. Okay, technically under latest news, when you click on it, just like click on the salesforce there you go right, so you can. Was it scroll all the way down and you can get uh you can get the listing of all of them? Then? um Actually it's a scroll down! I'm sorry! No! It's not! um Actually. Sorry click on resources, news, I'm sorry! There you go.

A

This looks like yeah.

B

So we can definitely put it here. I'm not sure why the images aren't working anymore. They were working fine before, but I'll go back and figure out. What's going on that one, um but.

A

Like talk to your webmaster.

B

Yeah yeah. Well, that was the thing we're gonna do right, so long story short the dang thing which is beyond irritating, and so yes, I'm saying this publicly uh is an oxygen template on top of wordpress like oh they're gone yeah, so that is going to be changed. Obviously, so I we're it's one: thing's post summit, we're gonna, be I'll, be able to focus on that. Getting that up and running now, part of the reason. Why is because we're actually hiring some additional people that so they can take? Take it from me.

B

Yeah, so the idea is, and so we're what like like to do, is actually put the website on the github page. So we can actually basically also make the submissions we can.

C

B

And then we basically have a process to push, uh take the the take the markdown from github and basically push it directly up to the site. So that way that we don't need you basically right.

A

Now I follow you almost.

B

Administrator access in order to be able to do this, that's really stupid. So I.

A

Follow you, I just tried to think of like what. What does this mean? We can do as delta rs peeps ahead of data nai summit.

B

So we can create either external blogs and I'll link it to the blog port, and I will also create a news post and that will point to both of the most of the blogs as well. So prior.

A

B

Prior to summit, that's what I would do and then post submit when I can finally get a actual github page. Then, basically it won't just be me: it'll, be multiple people that'll be have uh basic emitters will have rights to push to the site it. Just I haven't had a chance to get it off and running.

A

Okay, so uh we can, I guess, we'd, discuss like what type of content we might want to create. It sounds like uh I'll ask florian if I think back channel is his company if they would like to do a blog or if they have a blog, where they would like to talk about the python stuff, and maybe we can, it can help with that and then I think qp and christian misha. From the script side, we can just use the scribd tech blog for at least some of the delta rs and kafka delta engine.

C

Yeah, we have a of things. We can write about yeah, okay,.

A

Okay, so we will do we'll. Do some blogging be super hip, we'll do some vlogging and then we can talk about uh reader support for the rust, binding, qp or christian.

A

Is there anything you know big and notable that you would want to to promote at the end of may for the rest, binding.

E

E

Potentially the multi-rider support if it gets there since we have disrupted locking now, uh but we have more stuff to work through um to kind of surface that better, but uh other than that. I I can't think of anything.

C

Yeah, it's sort of similar to python where the read support it's complete and you can use data fusion to do any kind of queries like you would do with pandas and right it. I think at that point we'll probably only still have a pet. Only right.

E

Yeah yeah, I feel like we'll we'll be append only for a good long, while.

D

E

That's our primary use.

D

Case it also doesn't support pointing so I need to work on that too.

C

But I guess the delta interest demo will be a good thing to print up that that uses the right functionality.

E

Yeah sure, like a specific use case for the append, only okay, that makes sense.

B

Yeah, we should definitely call that out and then, uh if you're for especially for stuff like this, if you're up for it, I would still definitely blog it with you guys first, but then we can. I could potentially work on also um cross-posting it at a later time uh on the databricks blog as well, uh specifically, obviously on open source community stuff, but because we definitely talk about that in on the databricks blog as well. We definitely do that in addition to delta, obviously yep. That sounds.

A

Cool I've got a lot more, a lot more thoughts than we have time for data ni assignment danny, and I have to talk about what our what our marketing plans are going to be this week, anyways so I'll, let y'all know I I've got some. I've got some ideas and I think I'll just write down sort of what I think we can do um some of this just for for denny and anybody else. That's watching.

A

Some of this does come down to a question of time like just how much, how much time and effort can we put into making a big splash that did it in any summit.

A

um Any other thoughts from anybody before I change topics to uh new committers.

B

Nope not probably plus one misha, plus one meter, plus one.

C

A

All right, well with that, uh I think I didn't even have to ask so let me go ahead and give github my security key miesha.

A

I can never remember your github username.

D

Yeah, that's a shorthand from my first name: sort of.

A

All right, let's invite muscop.

A

Or mozy p, if you will.

D

Yeah, my full ukrainian name is so hard to pronounce for english speakers.

A

Can could you, could you say it with full accent and everything for us? Okay,.

D

Can I try first again, I actually, I actually taught christian yesterday how to pronounce that.

C

Let's do a test.

E

Come on chris, let's see it.

D

E

Nervous but I'm going to try, I think it's.

D

D

I feel like learning more languages, uh that's because, like uh there's the the sound, the first small, it's, it's not common for english speaker to use that with m.

A

Yeah there's a lot the the I'd say, the further east you go from great britain there's some mouth sounds that are just very difficult for english speakers to to create yeah. um So there's a pending invite. You should have that in your inbox. Welcome to the delta rs committer group. uh Thank you for the contributions that you've created already the dynamodb stuff is pretty sweet, um and with that that's actually the last thing I had on my very hastily put together agenda for today do y'all have anything else that we need to discuss.

B

No, I think, we're good. This is awesome.

A

Well, thank you. Everybody for joining, uh welcome, aboard mikhailo and we'll see you back here in a couple weeks. Yep.