Delta Lake Community Office Hours, 9 Nov 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Flink Delta Connector Community Sync 2021 11 09 at 08 05 GMT 8

Description

This is the Flink/Delta Connector Community meeting from November 11th, 2021

A

Like a uh peanut and apache hoodie contribution, so I was just looking at like a delta and you know like willing to contribute over here. So, like that's the reason.

B

The trigger is job well, I mean welcome aboard and so right now we're recording, because we're going to be posting this video on to the delta lake youtube later on. I I did want to not stream it just that way, especially because my internet issue might happen anyways, but uh nevertheless uh uh this is for publicly uh welcome everybody. This is today uh november 9th at 8 a.m. Pacific uh for the flick delta, connector sync uh community meeting um had just introduced himself. um Welcome aboard.

B

uh Why don't we just go around the horn just to do like a quick, 30-second introduction, scott! I want to do it yourself.

C

Hey, I'm scott, I'm a software engineer on the uh delta ecosystem team at databricks, and I've been working on the delta standalone writer, which is a single jvm library, which is a dependency for the flink connector to write into delta.

B

Thanks scott fabian.

D

Hey I'm fabian, I'm based in berlin and I'm with vivarica, so basically, the original founders of apache flink and I'm guiding like the whole delta, like contribution from like the fling side and assisting in whatever question comes up and also working behind the scenes and to improve the think that yeah, we may get better experiences in the future and ease future development.

E

Excellent excellent. Thank you pavel uh hi. I'm pablo. I would say that I'm the main developer for this spring connector itself uh and to do that. I'm using called helpful from the delta and vertica guys.

B

Thank you excellent. Thank you very much. Paul uh td.

F

Hey everyone- I am a software engineer on the databricks side and been involved with uh everything delta since the beginning, so I'm this is I'm guiding this whole initiative from the delta side of things.

B

Awesome, thank you very much. Okay and my name is denny. I'm a developer advocate here at databricks um yeah, let's rock on so um jagadesh. You said you were interested. Do you have before we dive into the details, because we are going to dive into some details real fast. Did you have any questions that you want to ask any of us here from the flink side or from the delta side? For that matter?.

A

No! No, as of now I'm good yeah,.

B

Yeah, okay, perfect! Well, no again! Well, awesome don't hesitate to ping us any questions. So uh td. I want to let you take the uh take the mantle here, but I believe where we're currently at is where we, where are we currently on the pr's right? There was the umbrella pr that you originally created uh the the public created. I believe that was the last check. So basically, I figured we'd want us to probably start from there since the design document's already out and published and everything's good to go from that front.

B

So I figured we'd just dive into the code, so.

F

Yeah sure wait did you want me to do that uh or powell.

B

Oh, no, I'm sorry uh paddle to dive into the code, but you just from sample the questions that you normally ask. That's all I mean sorry, okay,.

F

Okay, so um from my side paul, I see that you, just in the last half an hour, updated the first pr uh which sets up all the skeleton code for the delta sync. uh Thank you very much for working on that. um I can see a lot of the scholar.

D

Class say it's not.

F

Scholar, the class and method docs have been updated. uh Thank you.

F

I will start going through it in more detail uh now that you're updated uh just one question I had at a high level is that very able to get the javascript stuff working.

E

Yes, uh okay, yes, it's! It should be, I'm not sure about the pr. I I just run it before, pushing it to the uh to github. I run the styles on my local setup and he does okay.

E

So I'm not I'm not sure about the pr but yeah. If not, I will align it. Okay, good! I can.

E

Is on so, I think it passed okay, but.

G

You're sure that the java style checks are running on the flink, so.

E

Yeah, because I, on the main pr that I left for reference, I had like a lot of checks. So it's for sure it's working, okay,.

C

um In our build.sbt configuration you can see that every compile that occurs or compilation occurs is dependent on um java style checks passing so, okay and his is being compiled and then tested so yeah. Okay,.

F

So the style check is in the common tables common settings.

C

Yeah we've just made that anytime, you want to compile and then test something you need to pass style checks as well, but does the style.

F

Checks apply on all the projects.

C

Any project when you compile any project, it will also run it yeah, good, awesome, okay, good.

F

Thank you just for controlling okay, good. This. This definitely looks even the cursory grants that I'm doing right now is definitely a much better shape than before. So, yes, we have our work cut out on our site to start reviewing this stuff, but in the meantime, paul. What's your next step like what can you start working on something like adding more tests and stuff while we are reviewing whatever is out there.

E

uh Yeah, so uh not sure if you notice, but the current, the first, the couple pr a little different differs from the one from the previous week, because I strapped all the implementation codes. Now, it's really just a uh pure implement uh pure interfaces and classes with very little implementation in it.

E

Yeah, I I thought that it would be easier for you uh just to review no implementation. I mean almost no implementation at all. I will add the implement as asi plans. So now is just uh interfaces and classes. Then I will add the functionalities for the uh writing uh files to the file system. So basically, what's the fire sync is doing so. I will uh implement this pr for only this behavior. Then I will start adding stuff just connected to the delta implementation itself, hi. So, okay, that's the plan.

E

F

No, this is great. This is a much like. I just realized. This is a much smaller pr, which is so then we should focus on getting this in as quickly as possible so that you can make the subsequent years. Actually, why don't you? It's you make the subsequent prs without this massive diff and focus on just the individual implementation with that is going to be the real pr of interest.

E

Yeah, I will create the prs anyway, but I will keep them in my repo and I will link them in slack on slack.

E

I just wanted to add that for next couple of days we will be having some bank holiday. So I want to prepare all of those pr's that I mentioned for you that so because during the next thing you you you can keep yes.

F

Yes, absolutely sounds good, so yeah, okay, slack us the list of pr stacked on top of each other. Now I want to get some of the baseline straight here, uh so this first pr is on the standalone writer feature branch. Scott, do you have a sense of when the standalone writer feature branch is going to be merged to muster.

C

Most likely today or tomorrow, as far as I know, all we're working on is like uh wiki or sorry later. Tomorrow, still, we should do wiki and like examples like documentation, but that's unrelated to code, pretty much.

F

Yeah but but those doesn't really need to block the merging right exactly so, yeah, okay, okay, so so, then, basically, today, tomorrow we merge the standalone writer feature branch to the master. Then I think the next step would be to um redo this pr on top of master oil, which should not take any much more effort. It will be just just a rebase as far as I expect, um then.

F

Basically, we don't want all of this flink stuff to be yet merged into the master, because we want to make a delta standalone release uh the release of that entire repo. uh Before all this flink work is complete, so we were going to maintain the flink as a feature branch and all subsequent pr's will continue in to be merged in that feature, branch, which is exactly what we did for the standalone writer as well, does.

G

It make sense, paul and.

E

Scott yeah sure, uh so are you going to create a separate branch for me that I can create a pr or how it's going to work.

F

Good question: scott: what did you do for your standalone writer? How did you was the first pr automatically the base branch.

B

F

C

F

C

You'll just branch off with master, and that will be your future branch and then you can branch off of that and your incoming prs can be stacked on top okay. So.

F

Then do you so basically are suggesting explicitly fork off and create a branch called flink feature? What's yours called now, I'm sure the name's fine.

C

Flingsing feature one uh right now: it's called slingstone feature which is perfect and right now so right now it's branched off of this, this kind of writer, so you can just rebase it off of master and change the target branch to be into master and then you're fine.

C

Does that make sense.

E

um Yeah but now uh I'm merging into your branch, but to create a separate branch or too much. uh I don't have the the rights yeah. I think.

C

You can't change, you can't change the uh the target branch. You can't change the base branch because I can, if you'd like me to.

E

uh I'm not sure, but you.

G

E

Branches, so I cannot create the branch in the connector.

E

Oh, but you're.

C

In the connectors we go, I see okay, then then that means I will probably have to do this for you. So.

E

C

Just change I once you've rebased it with master. I can change the target to be into master and then it'll be there.

E

Okay and you want p2 to rebase right now or wait uh until the uh that is writer will be merged, yeah waiting thoughts, merged, yeah, okay, so just let me know- and we do rebase then.

G

F

uh So the consent, the going forward, the feature branch in the main repo- would be called flink sync feature: link, underscore synchronous, feature, okay, which will be different from powell's own fling sync feature branch, which is the all-inclusive of all the changes which you are splitting from.

F

Because powell has all the changes in his flingsing feature branch in his repo, we don't want to merge exactly that, because we are splitting off those changes and making separate stacked, pr's right.

C

Sure, okay yeah, so he could just rename his current branch and then we can still have links future sure yeah. Yes,.

F

Okay, just making sure that there is no confusion in the future. Okay, good good. We have established uh what is expected good. Okay, then scott. I think you let paul know when you have merged it and then we can take it forward. So paul tell us about your bank holidays. When are you out.

G

uh From thursday, basically it's it's it's today, it's day and friday, thursday and friday. So uh we are.

F

Not meeting on thursday in our internal sink, okay, cool, okay and then until then, let's hope that this gets merged and you can do your refactory basing to the master or whatever the new gun structure is before you leave, so that we can continue working on this sounds good good, okay, great.

E

uh So maybe a question from my site about the next steps, because yeah, given that we'll be having this first version of flink delta, connector uh fling delta sync, uh what would you say will be the next step focusing on the source or maybe focusing on upgrading to higher click versions and providing that table api.

F

That's a good question. I think we need to figure that out. That's a that's, a very good question.

F

What I definitely want is make the flink sync feature complete and ready for release with documentation and stuff before we start moving our focus on source or table apis and stuff like that. So I think what we should do is have a sort of a mini project plan to track all the tasks that we need to do. uh Maybe we should set up as like a spreadsheet or a google sheets kind of thing where we track.

F

These are all the tasks we need to do and so that we can track them and so that, if they're, like small tasks here and there, like fix that tiny thing fix that anything drops up, we can keep adding it over there so that we don't drop any small tasks. Maybe that's what we should do and that will give us a lot more clarity on what are the next steps left to be done and then, accordingly, we can start thinking about more next steps beyond the sync.

C

Td, what about just using the design dock and adding a project plan at the bottom of that sure.

F

Sure that works too yeah, let's have just one dock as long as we continuously keep it updated, and if there are new items that come up that are surprises, additional testing needed or whatever we keep adding to that uh tip.

C

And one question that reminds me of pabll is: we haven't talked about tests in a while and like what the current uh integration or unit tests are. uh Could you update your customer status of that.

E

um Yeah sure so most of them are there. You can look at the reference pr and if you are missing something just just let me know uh what for sure, I'm missing some edge cases for a global commuter, but I was planning to add them with the pr implementing the global computer it. It has some parameterized tests for global commuter, but this is just a general behavior for partition, map partition and so on and yeah edge cases will be the next one.

E

So that would be the main missing test and besides that, most of them should be there. So I would say that it's mostly like one or two classes that needs to be tested. Additionally, okay. So if, if you please look at the reference pr and if you will be missing something off of a greater importance, just let me know.

F

Makes sense? um Okay, so how exactly are you testing the what has been written out? Are you using basically the delta standalone, to read the log and get the list of rk files and then read it using the standalone itself? It's inbuilt rk reader to see whether the contents of the table matches what is expected.

E

Yeah, so I'm testing in it on different levels, for instance, I I inspired my pr by the think sync, and so I get some of the ideas how to test the behavior. So, for instance, we have this file writer bucket uh and in case of delta. It's the delta writer bucket that implements the actual writing to the file system.

E

So I'm testing it just by uh writing the uh writing the data, the records rolling, some files along the way as well and reading it back so just testing if it can be burst back and if all the records was all the records was written to device and then the structure is okay and so on, and uh I'm doing the same for the delta global committer. So but out there I'm not writing the actual files, I'm only committing the metadata to the delta log.

E

So it's just stripped off of the writing capabilities and in case of the general id testing cases I'm doing all of it. So I'm running the job, I'm writing the files. I'm reading back.

G

And checking the delta log as well. So when you're reading back.

F

You are you're reading back only the rk files written. How are you reading back.

E

I'm passing the data, I'm passing every records to raw data, that this is a structure from flink, so I'm checking every record that was that can be passed back. Okay, counting them.

F

Okay, so okay man, some some food for thought, for you- is that in the in a if we had a flink reader for delta, then the nice, the right way, the cleanest way we would have tested is to use a flink reader to read the delta table after the streaming sync has written out pipeline is stopped, use a flink batch reader to do a batch lead and verify whether the data is correct or not.

F

It could be, and eventually we are going to work on the fling source to make that happen right. So eventually we will have that so for such integration tests, it might be a good idea to construct the framework of the test in such a way such that right now we are doing like a relatively hacky job of testing.

F

Only the parquet files testing the log details with the logs like separate pieces and stuff in the absence of the fling source, but in future, when we do have the flink source, we should be able to just change one function, the implementation of one function and just replace it with the flink source test and make it much cleaner. So think of making the integer this integration test framework such that you encapsulate the checking of the final delta table into a single function, or something like that, so that it can be replaced and made cleaner.

F

A lot less verbose in the future.

E

I think it's it's pretty much it in the integration test right now, but yeah. I will take it into the consideration and because I encapsulated the testing functions anyway, uh but yeah I will look again and make sure it is good. Awesome, awesome.

G

Awesome. Thank you. Thank you. This is great. Okay,.

F

I think I'm out of questions from my side.

D

I think one recommendation I have regarding the maybe next steps, um so we usually when we like get new contributions for like connectors things or sauce.

B

D

Usually treat it less like it needs to be. There needs to be also a table. Api integration, because otherwise users who are basically only writing table api queries or sql, cannot use the connector otherwise.

D

So this is currently a limiting factor. So basically the connector you're, currently designing is only for data stream and there needs to be some syntactic sugar around it. Some wrapper to make it available in the table api.

D

Usually it's not difficult, but it's just. It needs to be done. Unfortunately, so we have no generic way yet to plug them into other apis and yeah, and I would probably.

A

Also hope that.

D

Python integration is probably off the table currently because it's yeah a bit more tricky. I would say.

F

Okay, so I think, as far as based on it, I think it's covered in the design dock as well, that I think the general plan was that we're going to make a release on flink 1.1.2 without the table api automatic integration- I guess python stuff as well and then we'll keep improving and releasing it on newer versions of as those api changes does that.

D

Make sense makes sense totally. I I just want to make clear not that your customers, at the end only use table api and can't use the sync at all yeah.

F

Yeah yeah, no, no, no, absolutely make sense. I think the only reason we're doing this that we have certain and then you can actually elaborate more on that. We have some customers who are in 1.1.2 and that's why we're going to target that as the minimum version, at least for the first release, so that we have some people unblocked with the basic implementation and then always improve things in the future. Sounds good.

F

Isn't that right that some of our people.

B

Don't want to yeah yeah, so long story short. We actually chatted with them about their willingness to go to 1.15 right away. Oh also, 1.14, right away. At the time when we were talking, I was talking about 1.4 um and they were like, maybe but really to get on block 1.12.

B

Now saying that if you think it's worth while, I can certainly go ahead and reach out again and see if, like, if they're willingness to upgrade to the latest version, if you think that's actually worth the effort, if you do think it is, I actually have no problem doing it.

D

I mean if their data stream uses, they can definitely use the sync right away when once it's merged, just maybe make sure that they're not want to use it from another api, because this won't work in 112..

E

But yeah, as I said it still will be, it will be impossible yeah with some syntactic sugar and bridging back from typo api to data stream. Maybe it would not be the best way to do it, but they they could use it yeah. If they have table apis based jobs, they can bridge back from table api to data streams and use the sync.

D

Yeah so basically flink supports like some sort of bridging between the apis, at least for table api. So if users write table api programs, they can to like unlock the full capabilities, go to the data stream api and then use everything which is available there. So this should be possible yeah.

E

But yeah it's not.

A

The most natural.

D

Way for like people who are used.

A

D

Table api, where you have like most like sql statements or select squares and aggregates um to more like a functional programming style.

F

Well, I think, using the bridge basically like do all your table operations using the usual sql style, select uh project and all of that, and only at the final step when you're specifying the output. Do you need to do that bridge to write to the to invoke the sync api begins in that setup.

D

Yes, basically, yes, but conceptually the apis are, are completely different. So that's why the bridging is not without the cost.

F

Okay, are there any known limitations of what, when the bridge cannot be used.

D

I don't know specific gotchas of 112.

D

um I would hope it works in all the cases, and I know we have currently open bug which has worked on that timestamps might not be probably propagated, but as far as I understood, the delta sync doesn't use any timestamps.

D

So this shouldn't be a problem.

F

Okay, cool, okay, cool! Thank you. I think that's good clarity to have that. That's why what we documented in the design, doc yeah this.

D

Is already shown.

F

Yeah exactly but just good to double check. Anyways.

B

Cool cool, then I think we have all of our action items and things that way to target so questions answered. So if there's nothing else, um we can certainly call it a day.

B

All right well, then, thank you very.

D

Much everybody.

B

uh And I'll post this up to the delta light community shortly and uh that's it. Thank you very much. Everybody awesome. Okay,.

D

Thank you, stay safe, everyone, bye-bye.

C

Cheers. Thank you.

C