Delta Lake Community Office Hours, 16 Sep 2021

Previous Meeting

⏯

youtube image

►

From YouTube: Delta Lake Community Office Hours 2021-09-16

Description

Join the Delta Lake Community Office Hours to ask your questions on all things Delta Lake! You can also join us in the Delta Lake Users Slack at: https://dbricks.co/delta-users-slack. Thanks!

A

Okay, I think I'm live on youtube now, so so hey folks on youtube. I think you can hopefully.

B

You can see us.

A

I'll start with that, I was messing around with the live stream, and- uh and here it is so, okay, perfect.

A

Qp and ryan, you guys are joining awesome, we'll see if anybody else joins. I'm actually currently live streaming as well to see uh if anybody has asked us questions, excellent misha, I see you on youtube. If you have any questions, why don't you go ahead and chime in and ask us questions um and just type them in and we'll do our best to answer we'll do some introductions in a second, we were just working through some of the technical issues. Like usual.

A

And qp and ryan, if you can just unmute yourself and uh so that way you can talk whenever you're ready.

A

Thanks qp how's it going buddy.

C

Pretty good, pretty good all right really excited for the first office hour.

A

Yeah yeah well, we'll see what happens. I'm you know I did. We did promote it, but I'm not sure if anybody's going to join us, it's okay. I have five questions.

C

A

C

A

Worst case scenario, we'll just ask our questions exactly each other for this session. It's not that big of a deal.

A

All right, let's see yeah excuse me sorry, um yeah and I'll fix these zoom links uh shortly uh to everybody for the attendee. So I can uh uh give you guys panelists. It was a at a mad dash because I mistakenly forgot that a zoom meeting and a zoom webinar are different things, so I couldn't broadcast it to youtube. That's also hey and misha. Did you want to ask any questions by the way if you do you're more than welcome to just chime in and type in here, if you want.

A

Okay. Well, meanwhile, in that case, uh let's just jump start. It uh welcome to the first of what will be bi-weekly every two weeks, delta lake community office hours we're starting off with a little light today, but fortunately we actually have multiple people that are attending and we're probably just going to ask each other questions.

A

um We also talk a little bit about the delta lake road map, so for any of the folks who are watching this post session, not a big deal, uh you can go ahead and actually dive in and learn a little bit about the roadmap as we speak. So um so, for starters, uh I'm going to introduce myself. My name is danny lee. I'm a developer advocate here at databricks, long time, brickster a long time, patchy, spark and delta guy as well.

A

So um I'm going to be here to answer some questions and also host a little bit. um I want to also include um next scott want to introduce yourself.

D

Morning or afternoon everyone- uh I don't think you can see me, but I'm scott, I'm a software engineer on the delta ecosystem team at databricks. I'm really excited to be here.

A

Excellent thanks and um uh next is qp. I think.

C

Yes, um hi, I'm qp, I'm one of the engineers um from script and um I'm one of the maintainers of the delta lake rust findings and the native python bindings.

A

Perfect, thank you very much qb and by the way, uh uh qp, scott and ryan. You guys should be able to turn your cameras on now, because I made you all uh panelists, so you can you know we can should be able to turn cameras on if we want to um and last, but certainly not least, ryan. Why don't you introduce yourself.

E

Yeah by the way, I cannot start a video because it said the host has stopped it.

B

Oh interesting, okay,.

E

Yeah yeah, I will introduce myself and while you are doing it, anyways.

A

I'll fix that, and you mean well yeah exactly.

E

I'm ryan from datapress I'm a software engineer and I have been like working on delta lager for like many years and uh and basically since the beginning, I end up before. Like the other leg, I also like to work in spark, which is mainly focused on the structure streaming project yeah pretty excited to be here.

E

Oh excellent, yeah, thanks danny for yeah.

A

Cool, thank you very much. My apologies. Yes, I would you would figure. I would know how to work with the zone by now, but, okay, it is what it is all right, perfect. Well, um misha uh has chimed in on youtube saying he is not. He doesn't have any questions yet so not a big deal, um but then I'll I'll call someone like qp go for it. You've got some questions, so, let's.

C

Start with you first question uh I was going to ask: are you going to show the awesome roadmap in this office hour.

A

That is a great question. Let me go ahead and actually pull that up right now and speak. So give me one second, uh and I will show the roadmap. Sorry. I've got too many tabs open again, so I'm fighting with my tabs.

A

You can tell that how we're a very professional organization right now.

A

uh Let's see okay, here we go uh presenting the roadmap as we speak. Oh sorry, there.

B

A

uh Give me one: second: it is not showing the roadmap all right.

B

Okay, how's that yep, we.

A

B

A

It perfect, so do you have any questions any call-outs, I mean I've got a couple myself, but I want to figure if you know any, uh whether it's scott or qp or ryan, if you want to call anything particular out for starters, I'll, just zoom in a little bit just so, it's easy to read.

E

Yeah, I think one big project is the delta standard writer, which is we are building to support like writing without spark which we will try to use. This build a lot of characters and uh kindly scott is like leading this project and working on, and probably we expect. This can be like a release on like a q4 uh next quarter. So yeah, I'm pretty excited about this project.

A

Excellent call scott, would you like to add anything to it since you've been obviously, you know neck deep with the delta standalone writer.

D

Yeah, I know it's just a really exciting feature and project, and it's coming along well and uh looking forward to releasing it next quarter.

A

Cool and so for anybody who actually is interested more on it in addition to chiming us with the delta user, slack channel and or the delta lake email distribution list and or directly on github, where you can just chime in like you know, we've already had some uh uh in addition to the official road map which calls out flink and presto right uh and pulsar to me the uh those three we we recently have pings from beam as well so and there are actually some other folks as well.

A

So if apache here on and b excuse me, those are the two that immediately come to mind. So if there are additional projects that would like to basically utilize uh a jvm based writer in this case, which is the delta standalone writer, but not you spark at all, then absolutely please chime in and ask us questions and talk to us about this because we're actually, I can probably say this for scott, like we're, we're actively working on it right now, but we're still early enough where we can get uh new requirements.

A

So yeah I mean anything you want to add ryan or scott to that.

D

Yeah just uh leave feedback because, like like danny just mentioned, we're early stages of development, so things can change if users want any different feature or any different exposed api, whatever we can do to help people build connectors.

A

Excellent perfect that that's uh the big one delta standalone writer um qp any what callouts you want from the the rust api, especially because of the fact that, for example, the cockpit built into this project, which is related to delta rust, actually is in production. So anything that you particularly want to call out there.

C

um Yeah, I just I'll just repeat that um it so we've been working on a kafka to delta late connector, but um not in jbm native connected, that's um fully written in rust, and it's been in production and running really well so far, and um so we don't have any major plans from script site. I know there are some um new features that the community is planning to add to the rust findings.

C

um But um I I just call out that um the delta rust api for creating a new tables- that's already completed from one of the community members, I'm connor and that one's already completed- and so I'm repeat, really happy with the uh the delta and just performance. So far.

A

Excellent and then one thing, I'd love to add is also the fact that we're actually planning, I think in early october to actually have a session specifically devoted to the kafka delta, ingest and uh the delta rust associate delta rust api. So if you want to dive deep from that, obviously you can talk to us at the the delta user slack channel, but at the same time, yeah we are planning a session, so you can dive deeper there as well.

A

So, but of course, if anybody chimes in right now and has any questions, that's what these office hours are for cool anything else that uh you'd like to add in. uh I know that there's work from like, for example, uh nessie and lakefest. We also are having some great work with the uh folks over stream, with uh uh with pulsar uh gerard, has been doing some amazing work with the power bi connector, so that that's been extended uh even much.

A

So um anything we want to add about the hive connector that you can think of for uh ryan or right now. I don't think, there's any. I don't recall any updates yet per se.

E

uh Have connector yeah? We are planning to like uh support the hype three uh in this quarter, and so we are like working on. uh Have three support right now,.

A

Excellent okay, perfect and then uh for any of you folks who are delta inc delta lake inclined, uh but also not so inclined to write any of the code. We actually have a lot of updates that we're planning to do with both the website itself and the documentation so feel free to go ahead and chime into those github issues uh and and uh and call and basically chime in that you're interested in helping or you think you want to see xyz and that's a great way to get involved as well.

A

um One other thing I did want to call out is um that we will be sending out a survey, a delta lake user survey later today, in both uh slack and email we'd love to get your uh feedback about the delta lake project. So these are I I want to call it out. This is the initial proposed roadmap right, yes, middle enough.

A

We are in september now, so we there's only four months left, so I'll blame myself for initially being one month first, one month late and then we've been going through various discussions together with community for the last month or so, but and so I'm a little late on the survey.

A

So my I will call myself out on that one, but by the same token, you know we're still earning enough where, if we're seeing other prioritizations like from the community, we're glad to change them accordingly and in fact anybody that chimes in and provides feedback, you're automatically will get a delta lake t-shirt as well. Okay. So this is me broadcasting, the brandeis t-shirt, so we'd love your feedback, we'll be like. I said, we'll be sending that out later today and I'll and connect it attach it to this youtube. Video as well.

A

So just want to call that out so again um love to get your feedback. uh So that's it for that point, uh any other questions from either the panel or from anybody. That's currently on youtube.

A

Qp, did you have any more questions or.

C

uh I have a question um shoot of course: okay, so this is for timestamp handheld right. So for the context in the delta lake spec, we defined the timestamp to be of microsecond precision and but when we write a parquet in spark by default, it's using it 96, which is nanosecond position right. So I'm curious: what's uh what's the expected behavior when as a alternative implementation? How should we handle this um when there is a mismatch between the spec position and, what's physically, being written out in 4k.

E

Yeah, so basically, I think that for this case, we usually get tranquil micro seconds or milliseconds, and I think most in most of the time. I think this should be fine. Actually, for example, in most of the system you won't be able to get like eight kills, like microseconds yep,.

C

Yeah, so so so so the implementation is expected to do the conversion right yeah when reached from the party okay about it. Thanks for the clarification.

C

All right, that's all the questions. I have perfect.

A

All right: well, I think, let's see if there's I'm checking youtube one more time um doesn't look. Oh here we go. We do have some questions coming in now perfect. uh Can you tell a good example from delta rs python to adls gen2, so any samples available? So I I'm going to direct that to you lqp. If that's okay,.

C

So there is a discussion uh and that really lengthy github discussion in the delta rs project, I'll I'll link I'll send danny the link and maybe danny, can link to the issue um long story short. There are some example code there, but it won't work out of the box because, due to some limitations in the adls python file system library- but um there is work around for that.

A

Perfect, thank you very much. Yes, I remember reading it some of the issues from before, but I didn't recall the details any longer so perfect. Let me take that.

B

B

A

um Oh sorry, did you slack me or with that or the link, I'm still trying to find a link? Oh okay, no problem! Okay, sorry I heard I heard a slack and I wasn't sure: okay, okay, um open source techie, says I'm joining the delta lake community after two after almost two years, that's great open source techie. uh um If you can go ahead and chime us with your question or you can also, of course you can always ping us on slack but yeah.

A

um So, oh, I think you're coming with a question so yeah yeah I'll. Let you finish your question um that you're trying to ask.

B

A

Okay, um let's think here, okay, um the open source techie is asking the question: I'm using a toonie replicate. The issue is that records are coming in late. What is the best practice to use delta lake in a cdc system? Okay, so specifically the ideas that you want to change data capture, so um I'm going to start with a quick call out um which is uh paul room and myself actually had done a tech talk.

A

um I have to look I'll fight off to find the link and post it here in which we talk about how using delta as a cdc source and it basically sort of calls out some of the best practices to do that. There was an ask in the open source community to potentially open source cdf change data feed.

A

um There have been discussions on whether we can go. Do that or not. um I don't I'm not in pervy to what the current discussions are right now. So it's not, I don't think uh we're against the idea of doing it per se. I think it's more of a matter if we're trying to figure out what the priorities are and right now, the priorities unequivocally, even though there's a lot of interest with the cdc.

A

uh The priorities, at least so far from the community, seems to be much more about the integration points, which is why we've been focusing primarily on that uh saying that I will find the cdc link uh shortly and I'll post it here into this youtube channel um and uh young luca. You said: can someone shed some light on why we need to strip the user info here? Could you clarify what user information you're saying that is being stripped?

A

Could you clarify that please, in your youtube channel, please chat meanwhile I'll go look for the link so.

A

So uh yawn yeah, if you can go ahead and uh potentially get back to us like about what could you clarify what your question is? Please, meanwhile, oops sorry. Meanwhile, I will I've just shared the cdc best practices, um video into.

A

Directly into the youtube channel, so hopefully that helps that, hopefully to open source techie, that's a good! uh That's a good starting point! Okay for us, um oh uh remy kinshan! I I hope I said your name correctly. um You've asked the question uh why the parallelism is def. Oh great question actually, and so I'm gonna probably direct this to you to you ryan.

A

Why is parallelism disabled by default for vacuum operation? um Under what circumstances enabling parallels for s3 can take a performance hit.

E

Let's one is mostly because if the layer s3 has some red limit of like object distance, if you like a star a lot of like a thread to a lot of node, to try to delete the object concurrently, it's pretty easy to hit a red limit, which will let us know how your request.

E

So we decide to know if enable is by default, because it's harder to check the condition.

E

For example, if we like to enable it may depend on how many like a node you have and what the parallelism you have in the cluster and also, what's the rather limited you are like setting in like a s3. So this so you may need to figure out all these questions and try this by yourself. Instead,.

A

Thanks very much ryan that that's actually super helpful uh uh ram kishan. If you have any more questions, definitely chime in here and john, I I did go ahead and respond to your point. Your question concerning username information. I believe you're referring to the s3, commit service. But could you please clarify that so that way we can sort of understand a little better? What's what you're? Referring to so.

A

A

So uh you haven't got hasn't chimed in yet, but if you're referring to the s3 commit service- okay, so okay, gotcha gotcha, all right, so, okay, so yawn I'll, probably direct this to ryan or scott there's a method in the s3 single driver log store a live 63 where it actually strips the user information from the s3 paths.

A

Okay, I believe- and your guys are gonna correct me now. I believe that actually has to do with the fact that we're trying to remove any personal pi information from being logged into the system uh for basically for security purposes, but I I I do want to verify that so ryan or scott. If you can chime in then.

E

Yeah, let's basically just try to prevent lego s3 to lock your like a usual quotation. For example, sometimes you can just put a key svt on the path and then, if you try to query list- and this file doesn't exist, it may slow, like a final fun exception with the four parts we just try to avoid. This.

A

Thanks, ryan, okay, so here's what I thought so, thank goodness, I'm not I'm not completely off, so the context basically is very much we're trying to ensure that, especially from a gdpr or you know, um basically from a grc compliance perspective that we're basically not logging your credentials directly into s3. That's what we're trying to avoid. So um I can understand. Maybe why you're going like hey I'd like to know who the user information is, but from the standpoint of us ensuring that we meet compliance rules. That's the reason why that that line's there.

A

So that's that, hopefully that answers your question so.

A

Cool well, we've got a couple minutes left before we're gonna uh sign off. So if you have any other questions, please chime in here on the youtube channel, uh we'd love to, like I said, we're we're try our best to answer all your questions and then, if the the the question's gonna be a little bit longer, like, I said before, definitely join us in the delta user's slack channel, uh as that will be the best way for us to get give more long-winded answers.

A

Cool um all right. Well then, you know what going once going twice any other questions, anybody all right, you know what, then I will call it for today, um qp ryan scott. Thank you very much for uh helping for today's session community office hours. We will be doing the community office hours um every two weeks, so this week we're having at 9 00 a.m, pacific and two weeks from now we're going to have at 4 p.m, pacific and then uh to answer agent, diro's question when's the golang biting y'all gonna happen.

A

I did want to chime out that you know what, whenever you go ahead and write, it agendero and that's a running joke for everybody who may not realize that that's that's tyler who's decided to chime in from youtube, as opposed to being on this panel today. So it is what it is so again. Thank you very much. um I appreciate your time and we'll see you in a couple weeks.

C

All right see you all bye.