Delta Lake Community Office Hours, 14 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Delta Lake Community Office Hours

Description

Join the Delta Lake Community Office Hours to ask your questions on all things Delta Lake! You can also join us in the Delta Lake Users Slack at: https://dbricks.co/delta-users-slack. Thanks!

A

I forgot to click the go, live button.

B

It says, live for me in the top left.

A

Yeah yeah, so I'm what I'm doing is I'm checking if uh it's showing up on youtube itself. Yeah.

A

All right all right, we are actually live now. So, okay, perfect ryan. Thanks for joining, let's see, okay, interesting! I'm sorry, I'm in the middle of trying to get christian involved too. uh He should have been able to join, but.

A

All right well, right now, uh oh perfect! There we go.

A

Say um chris, can you christian, can you hear us now? Can you meet yourself when you do.

C

Yep, I can hear you, I don't know why my video's not working I've got my camera set up today.

A

Go figure all right well as we're dealing with all the logistics. It took us a little bit while to figure this out, but thank you very much welcome to the um uh to the delta lake community office hours for today uh today being october 14th at 9am, pacific um right now. We're broadcasting live to youtube. So if you have any questions or any uh um thing that you'd like to know more about, please go ahead and chime in directly through the youtube link and just ask us your questions there.

A

um If you don't have anything, that's fine, then we will make this a very short session, uh but I figured we also want to provide a few quick updates. um uh So hence we've invited chris christian, uh scott and ryan aboard today and I'll also give a quick, quick update. Well, actually, scott. You can probably give a quick update on the connector as well too so um so right now it looks like there aren't any questions yet so that's perfectly cool for thus so um christian. Why don't? We start with you?

A

um You you and you and tyler had joined me with the last week of the week before on the kafka delta ingest. uh So that was a very popular fun session that everybody enjoyed. I was just curious if there was any any updates uh that you thought would be interesting for folks to know about uh whether it was the delta rust api or of the kafka delta just itself, for example,.

C

Sure yeah I mean we, we we've had so we're running. uh I I want to think I want to say about 24 streams in production right now, uh with kafka delta ingest and having a pretty great success. um We have a couple of interesting bits coming up. um uh You know, I can't remember I I think I'm yeah. Definitely at the end of our tech talk last week I mentioned that we were adding a starting offsets feature so that you can start from explicit kafka offsets um if you're starting a new kafka delta, ingest stream.

C

um We are very near completion of the pr that adds that functionality, uh and it includes a you- know, fairly invasive refactor as well. um You know, as we were kind of building out kafka delta, and just um you know you always get into that- that mode where uh you know you're, you're, adding new features and eventually you need to do a cleanup, and you know it's always hard to kind of spot.

C

The point like, where should we actually do the cleanup and when, when I was adding the starting offset feature, I I felt the need to do a big cleanup, so we have a big uh big pr um in flight right now. That includes uh a pretty significant refactor of some code uh and also adds the starting offset feature.

A

Oh sorry helps if I I mute myself awesome, that's really good to know. uh I'm really glad to hear that um cool. That's that's! A great update! um I'm gonna go next with you ryan! uh Welcome back by the way from vacation a much deserved one as well. um Any quick updates that you could think of, uh especially from the delta sharing side. I did notice some, for example, some questions on delta sharing, for example. uh Actually I do have a question now that I think about it. Does delta sharing actually work with python 3.9?

A

I know it works with 3.8, but I don't recall if it works for 3.9.

D

Yeah, that's a good question. Actually, we haven't tested with lattice python version. This is probably something we we should do before. We do the naturally yeah.

A

Okay, that's good to know: okay, perfect, okay, yeah, because there's a couple of questions on the slack. So I just figured I'd. uh Ask you that, because I noticed uh uh somebody was trying to run the delta sharing demo and what it worked well in three, eight and three seven, but I didn't see it working on three nine. So well, that's what they're doing, admittedly enough, I haven't upgraded myself yet so cool all right. Well, um any any other updates. I I like that, for example, like maybe like spark 3.2 all right.

A

We noticed that I noticed that it had just passed um the voting. I think yesterday, the day before, I think.

D

Yeah, actually, it's part 3.2 0.0, it's already on like a main century, so we are currently working on upgrading their dialect to use the lattice spark version right now and for connectors. We already have hive three support ready and we were. We also like working on the release for data connector, so that people can get like the have three support, and also some enhancement of the del respinal reader.

A

Excellent awesome thanks right now. That's super helpful, perfect. Well, then, hey we'll shift you to scott uh for a quick update. um The any updates you'd like to give us um both from the delta standalone writer and the upcoming flink and delta sync.

B

Thanks denny hi everyone uh yeah, uh both projects are coming along uh very well. The standalone writer is approaching uh code completion, so we've definitely now focused most of our efforts towards like qa and tests and performance tests. So that's coming along really well, uh and you know the the code is open source. If anyone ever wants to check out the pr or the design dock and leave some comments or feedback, you know people can feel free to go ahead and do that and the flink connectors also come along out really well.

B

We are also beginning to finalize that design and we are beginning to prepare to to uh to share more broadly the design, dock and the actual open source pr for that to start getting some community feedback on that, and even some community help as well with testing. Perhaps so, yeah come along great.

A

Perfect perfect thanks very much scott okay! Well, that's it! I think, for today's session, we're going to cut it a little bit short just because looks like there's not any, uh not any uh new questions for right now. So thanks everybody for the quick updates, uh appreciate your time and we'll all see you in about a couple of weeks, uh we're gonna be scheduling um on the 28th at 4 p.m, pacific.

A

So we can trade time zones for everybody uh and again, if you have any questions in the interim, please do ask us on the delta user slack channel. That's it for today. Thank you very much.