Delta Lake Community Office Hours, 20 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Delta Lake Community Office Hours (2022-10-20)

Description

Join us on October 20, 2022 at 9:00 AM PDT for the Delta Lake Community Office Hours! Ask your Delta Lake questions live and join our guest speakers, alongside Matthew Powers from Delta Lake!

Ask us your #DeltaLake questions. These sessions allow our community to ask questions about Delta Lake OSS and get to learn what we are building, planning to build and know about recently released features. These sessions are live and the recordings are available on the Delta Lake YouTube channel.

Learn more about Delta Lake
Read Our Newest Blog Post: https://delta.io/blog
Watch previous Community Office Hour sessions on YouTube: https://www.youtube.com/c/deltalake
Join us on Slack: https://go.delta.io/slack
Contributing Guidelines: https://github.com/delta-io/delta/blob/master/CONTRIBUTING.md
Delta Lake Releases: https://github.com/delta-io/delta/releases

A

I'm here with Ryan and Scott, and we're going to take everybody's questions, talk about the exciting Delta Lake updates. Okay. So let's get started with uh some exciting uh news. So from what I understand the oh I got a pause that so um from what I understand the 2.1.1 and 2.0.1 Patch releases are coming up so Scott. uh Let me know uh what are the exciting updates coming in there.

B

uh Thanks Matthew so hi, everyone, uh I'm Scott um I'm, an engineer at databricks I'm working on uh Delta Lake for the past year and a half um I had a lot of cool features in the past and excited to talk about like these upcoming patch releases. We have um Patrick releases are just a variety of bug, fixes a couple performance improvements as well.

B

um One really cool bug fix, that's coming in is uh tweak in our algorithm for our dynamodb based um S3 multi-cluster right solution, um so just a bit of History, so Delta Lake requires some some Cloud API guarantees. In order to be able to write uh to these cloud-based Stores um and S3 is missing one of those Mutual exclusion guarantees. So what we've done is we've created um a certain algorithm and solution that uses data would be as basically like a lock store to provide that Mutual exclusion when you write to S3.

B

So this is an update to that algorithm. We're really happy with it. um A Community member was actually able to point out a bug and that's great, and they actually helped us contribute the fix.

A

Yeah sweet yeah- let's, let's actually give some like higher level context there, because that's actually something that's really good to dig into so when you're uh performing operations uh with the Delta Lake, you want something: that's called transactional guarantees, which means you want to make sure your transaction happens.

A

um So I think our Delta Lake customers are using a variety of different clouds, they're using AWS and gcp and Azure, and with AWS there's certain apis available, and we want to make sure that to give them transactional guarantees and part of that is with the dynamodb solution. So yeah, maybe uh Ryan. You could talk a little bit about transactional uh guarantees at a high level and uh why that's so important in our space.

C

Yeah, so in general, transaction guarantine means, whenever you run a query on query or multiple queries like for example. At the same time, you don't need to worry about the whether that we are like a for example, if you have first query right, like 10 files. Well, the second query will see like five files only in the middle or you can always see either zero or of the term files. Basically just guarantee you get a Lego snapshot.

C

Every time you read the table, you will never see any middle to each other in the middle or any like a partial result or cropped. For example, if you have a query, maybe, like you are running some updated, but it's failed, Delta transaction will make sure you either change nothing or you successfully basically change the table. This is like how we provide the current key and the weather in the water.

C

How we do this is basically we're trying to generate like a a transaction log for each operation and then, whenever two operation trying to basically writing the same file return transitional, one of them will fail and then for S3 this they don't provide such apis. So we need less why we need to build electron, multiple solution for like R2 and gcp. They actually provide API to guarantee, like you, will enable like overwriter a file lessb Reason by another one.

A

uh Thank you very much for that uh explanation, Ryan and I think it makes sense that it's something that it's very important from a user usability perspective right when you're running a query when you're running a transaction you either want it to fail or you want it to complete, but you don't want it to be in that middle ground where it it did some stuff, but it broke and it corrupted and then it errored out and now your data lake is is unreadable. uh So that's that's possibility.

A

We really don't like that um and it's so it's kind of interesting. So from what I understand in order to provide that transactional guarantee, we need to write different code, have different approaches for the different clouds um so Scott. Maybe you could talk a little bit about that and, if that's a correct assumption.

B

Yeah well, as Ryan said, most clouds do provide this guarantee and of the of the main production, log stores or or Cloud vendors that we support. S3 is the only one that doesn't so. What we had to do here is think of some sort of algorithm um that uses some external store. That does provide neutral exclusion and how can we use the combination of those two to actually get the type of transactionality that we want?

B

um So in this case, what we do is you know, competing writers will first try and grab the lock using dynamodb and we're guaranteed that when there are competing concurrent writers, only one of them will actually win and grab in that lock, and then they now have the permission to move ahead with trying to write to S3 and, of course, there's all sorts of failure, scenarios and retries that we have to handle that's all encapsulated in the algorithm.

A

Yeah it sounds like a difficult engineering problem. To be honest, uh all right, so, let's, let's shift our focus a little bit to the connectors.

A

um So from what I understand, uh there's a flint connector, so maybe Scott you could go at a high level like what is what is the Flint connector and what's that, what's that for.

B

Yeah for sure, um when talking about like these sorts of connectors I, like giving an analogy of of Delta to spark so spark is a unified computation engine. We want it to be able to work with any sort of um data, type file, format, Etc CSV, um parquet Etc, and we want Delta to also to be a an open file format. We want it to work with any sort of compute engine, so Delta Lake um is our largest.

B

You know: Delta implementation, that's heavily optimized to work uh work on spark, but we want other engines to be able to write, read and work with Delta and Flink is one of them. The Flink is basically a streaming first engine, um it's really optimized for streaming use cases, and in the past year we've been developing um a Delta Flink connector, and what this enables you to do is we have a sync and a source, so you can read to and um read from and write to, Delta tables and there's a lot of really.

B

You know great use cases of why you might want to do this. You might have, um for example, some sort of pipeline that has really low latency requirements and using full link is one way to achieve that. But you also want you know, really really optimized REITs.

B

So a common scenario that we typically might see from our customer user to do is have their their right streaming and using Flink to a Delta table and then using Delta Lake in the background using spark which is optimized for these massive queries performing an optimized in the background, so you're able to use and mix and match different compute engines all with the same Delta table in the same file format. To achieve you know your different use cases and needs.

A

Yeah interesting so from what I understand, there's uh some updates to the Flint connector. So tell me a little bit about those.

B

Yeah so, um like I said, we've uh We've released the fling source and Flink sync um in the past two Delta connectors releases. That's the Delta connectors is our repository that hosts a variety of our connectors that work with Delta Lake that aren't just spark um The Next Step that we're working on is SQL and catalog support.

B

So we want full uh SQL support when you, when you write, uh SQL and execute it with Flink to be able to integrate with Delta using a Delta catalog um and so we're making a lot of Headway there that's coming along well, but another exciting thing we've done actually um is: we've actually found some some bugs during this process um uh with actual what we think is Flink itself, which is exciting because we love making making compute engines better. We love we're here to help them make Flink better.

B

um So we actually found like some data loss issues and some issues with one of flink's versions, with its support of an older version and we've actually submitted PRS for this um most of them have been merged. They just need to be back ported, so the next various patch releases, a Flink, there's actually going to be some really useful bug fixes that will enable our connector to work a little bit better.

A

Interesting so you're saying that we in building our Delta Lake connector, we've exposed some Flink bugs and we fixed them. Yes, very, very altruistic uh I like that: okay, Ryan Let's. uh Let's move on to one of our other connectors that I've been curious about, so we have a hive connector. uh Can you tell me a little bit about um what functionality The Hive connector offers users.

C

Yes, so basically, if you, uh you have some all the like systems or like a queries, you need to maintain like learning and hives, you can still reading like from the data Lake tables. So this means you can like use your spark. Maybe some latest technology to write the generator your data lake table, but your Downstream queries or you need to maintain they can still use list. Latency like has enable Bluetooth engine to read the Delta lake table.

C

If, in this case, you can slowly like maybe migrate, your jobs, like you, maybe have a very long ETL pipeline, you can microwave the lighter long, regular up to downstreams like a slowly flown later for tomorrow. I can have two other engines.

A

I see what you're saying so you're saying like if somebody has let's say a mapreduce job so which is like kind of a legacy uh framework and then like.

C

Not really it's basically, it's high five yeah yeah height, basically it's just with, like my Bluetooth jobs. So it's no, like you write the membership job manually. It's like you, use a hive. Basically, it's a single engine right to kind of you know background you have with you know later. Maybe this job is to run okay.

A

So hives you're using the hive interface to run mapreduce under the hood and this connector lets you do that with a double leg table.

C

A

Got it yeah I've never used Mac I haven't had the privilege of using mapreddies, so I don't know much about it.

A

um Okay cool, so those are some kind of the the connectors I wanted to like uh shift gears a little bit and talk about some of the just kind of the basics of Delta Lake and what we're working with so uh Scott. Maybe, let's talk a little bit about just the transaction log. uh How data is stored in parquet files and at a high level kind of like what are the limitations of parquet files like what? uh How does that work?.

B

Ryan is the pro here: uh I think it should be a little bit better for this one.

A

Let's start with just like the immutable nature of parquet files, what does what does that mean that a files immutable so.

C

Basically, yeah.

B

Okay, um great question and thanks for clarifying that uh Matthew, um so parquet files are immutable um and what this means for Delta lake is the fact that we have a Delta log, which is a set of ordered, commits and changes to your table.

B

um What this inevitably means is that a it allows us to time travel back in time. If the latest version is version 100, we still have Commit set version 10. Are you able to travel back in time and see the parquet files that are in your table at that version? So it enables really powerful features like time travel, and you know that those files haven't been changed and that's one really powerful um benefit of that.

B

The other, which isn't necessarily a benefit right away, is the fact that when you make updates to your Delta table- and you want to update rows in your files, we can't mutate those files. So what we have to do is rewrite the files.

B

So, even if you are just updating a single Row in your parquet file, we need to rewrite that file with that one changed row and obviously like this seems inefficient right and there's there's ways to get around this there's ways to structure, structure your data or try and make sure you partition so you're only updating specific files, um Z ordering can help with this as well, but we're also really excited because um Delta is working on uh releasing a new feature called deletion vectors which are trying to solve exactly this problem when you have updates or deletes, or even merges that are touching very few rows in your parquet files.

B

Instead of immediately doing this expensive rewrite during your write, we're actually pushing this problem later to make you kind of have to solve this during reads, so it enables much faster rights with a slight read overhead and it's up to customers and users to decide when they want to do that. Trade-Off.

C

Yeah and one like uh I went, one point I want to add is for most of like a cloud storage. They actually don't allow you to just modify a few bytes in the file uh most of the apis, the provided you just you need to override the entire file.

C

So that's why, basically, why we have a immutable table and in multiple partifier and write a new file on top of this one. Basically, if you just want to change a few bites, you it's still the same as like writing enough on a color storage.

A

Yeah, that's kind of interesting, so it's kind of like you could just be. You could have a huge data set and you could just need to delete like let's say a thousand rows and it could end up being a big rewrite operation.

A

um So Scott you were alluding to deletion, vectors uh and I am still not entirely sure how they help with that. So can you maybe I know you just answered it, but you maybe answer it again. Sorry.

B

Yeah I'd love to um so this all comes as trade-offs right between readers and writers. You can maybe make one a little bit slower in order to make one a lot faster. So right now with our rewrites during your right operation, um though that's an expensive operation to do to completely rewrite thousands or tons of files um when very few rows are being updated.

B

So what we're instead doing is essentially saying hey. We actually don't need to rewrite this file, but we're going to include some metadata and tell you exactly which rows were changed and therefore the right operation is very quick. It's essentially just a metadata only operation which, which is extremely lightweight compared to the massive you know, gigabytes terabytes that can be done with parquet files, and now what we've done is we've heard.

B

We've pushed the burden of Performing this computation to the reader and now what the reader will do is they will notice that hey I have to read this parquet file, but here's an extra bit of information that tells me about the rows or that were updated or changed or deleted in this file, and now I have to do a little bit of a little bit more magic to to read the correct latest state of the table.

B

But you know it it's a much much faster right with a very, very small, read overhead, which is a trade-off that most people would be willing to take.

C

Yeah and you know decent like, for example, we will also provide some optimize commands. You basically can compare this like a deletion Vector with the practice file, basically doing the file rewriting offline. So you don't need to impact your little right. Queries basically.

A

All right, so thank you very much I'm I'm learning here so so I'm, just Deborah I'm, not an engineer. So, uh okay, so we've got some great questions for the audience here. uh So uh one of our uh audience, members is asking uh if we're gonna release some Docker images to create some a better uh spark and Delta workflow. So maybe we can talk about the development with Docker images to create a more streamlined installation process for users.

B

I think I think you've had a couple discussions uh with some open source community members, cool.

A

I'll answer my own question is that how this is supposed to go I'm just like talking to myself I'm into this, so uh so we uh we're working on developing some Delta, like Docker images. uh We're definitely prioritizing those um I'm kind of like still learning about Docker. This.

A

This user, in particular, is talking about Docker images for uh Windows I'm, not sure if we need to create separate docket, Docker images for Windows users, but basically we do we're well aware that we need a smoother get started installation instructions, especially for folks that don't want to install Java- and you know, conda and all that stuff on their look machine. So it's definitely on the radar thanks for the great question. uh Another question is: should we take the import stars out of our um code Snippets, because those are break code, lenders um uh I?

A

Think the answer is yes, so we'll we'll get on that? uh Okay, another question from the audience is: do we have any utility to validate the streaming data? Let's say the data comes from Kafka as a source and Target as Delta Lake, uh so yeah Ryan? What do you do you know of any uh data validation that can be done from a streaming data source like Kafka,.

C

I'm not sure I get the real question, so what the Delta does it provider like a schema of education, for example, you can add like a check constraint to your Delta table, and then this will prevent you writing the any data that may break this constraint. For example, if you require one column is no now, then you will enable right now to list Delta table, or you can also write a complicated expression that I want to only data.

C

Like maybe question in 2022, then you can write the search, expression and edit the constraint, and then, if you want to write like last year, like data into the query, will fail and tell you like you, try to write something that will break your table. Okay, no sure I'm, not sure whether at least answer the question or not, but.

A

Yeah I think I think that that answers the question: let's let's kind of roll it back a little bit and think about it at a high level. So we have like a streaming data source in Kafka and there's a couple different ways to go from a streaming task of data source to a Delta Lake.

A

um Now one way is to use this Kafka Delta ingest Library, which is part of our that's, actually a rust project. So if you're interested in deploying rust in production, you could do that now. Another way of doing it would be ingesting the Kafka data stream with something like spark structured streaming and then writing it out to a Delta lake. So Scott. Maybe you could talk about that workflow at a high level and how um you know in in reading the streaming data into a spark structured streaming data frame.

A

They could potentially do some uh data cleaning and Transformations before writing it out to the Delta Lake.

B

um Yeah uh great question, Matthew I think this would be a great time to bring up uh The Medallion architecture that Delta is kind of like famous for for really building upon and encouraging, um and this is where you have different tiers of tables that lets you safely ingest um and Aggregate and transform your data at different qualities until it's ready for production.

B

uh So a common thing to do is to just ingest your data into a bronze table and actually don't perform any transformations to it as soon as you start writing Transformations, let's use as soon as you start writing code, you introduce the possibility of writing bad code and writing in a bug and messing up with your data.

B

So it's often good just to ingest your data and have it stored uh permanently, um and then you can slowly begin to transform it methodically often there, for example, from a branch table to a silver table where you actually start to clean up your data start to develop some some business like Aggregates and then start to structure your data a little bit more ready for for being ready for production inquiries.

B

So and just the bronze start to transform it to Silver, and then you can develop your gold level queries on top of that at the top of your silver tables. I'm sure Brian might have something to add here.

C

Yeah I think that's a great answer, so I hope a lag at least I look great.

A

So tldr just uh make sure to ingest all the trash data into the bronze table and then validate later. You might want that trash data later all right. So we have another uh question: how can we ensure that concurrent, merge and optimize operations never operate on the same partition uh avoid the concurrent append exception without restructuring the underlying partition schema.

A

um So concurrent I'm just going to look at this question. Concurrent, merge and optimize operations never operate on the same partition. Okay sounds complicated, I'll, throw that one to you Ryan. Let's see what you can do.

C

With this thing, what's the like, uh I I saw. You also mentioned that you put your table probably partitioned by date. Right in this case, you can try to use optimize to optimize at least load load. The data like maybe your merge, will never touch like, for example, you can optimize like a data like a one day ago, and then you might only imagine data like a to test data, something like this I know which lists like automatic provider like aware condition you can specify which partition you want to optimize.

C

In this case, you can avoid optimal, merging catch, the same partition and uh in. uh In addition, we are in the Delta Lake Community. We are also working on a future, a feature called the red serializable which you can actually solve. Like optimizing, merge conflicts for the user and the user. Don't need to worry about it anymore in the future, once this feature gets ready.

A

Here repeat the name of that feature: Ryan, please.

C

What is serious, people I think it's in our load map.

A

Cool so you're saying: there's a roadmap feature: that's going to help this user. With this particular issue, yeah.

C

C

A

So, okay, so uh Scott I'm gonna pose the same question to you. uh But can you just give me, like the high level context, answer so like what is what what's happening here? That's causing this user problems with this concurrent merge and optimize operations operate operating on the same partition like why? Why is that problematic? At a high level,.

B

So the thing to to remember here is that what is the merge doing uh merge is going to be editing some existing files because you're updating some rows and it could very well be inserting new files from a source to a Target and what this means at the transaction log level is your you're removing metadata files and you're adding metadata files.

B

So that's how you need to view it and then what is an optimized job doing optimize is when you compact um many many small files into larger files, and this helps speed up future um future, read queries because there's less of a cloud overhead of Performing lists and optimize is also going to be removing the small files like the metadata files and then inserting new ones to represent the larger compacted ones. So you have these two um operations performing similar things, they're, both adding and removing entries in the Delta log, and these can conflict.

B

You could be removing something that you can reading something that another operation has then changed, and the Order of these operations can then like conflict and matter, and that's how you get like the concurrent depend exception, um or the delete read exception that the Delta log will throw so at a high level. You have these two operations that are reading and writing and messing up with what the other operation was trying to read or write, um and so this is where Ryan mentioned.

B

That right, serializability will then be able to come to the play and just solve this without the user having to worry.

A

Give me some good high level context all right, so we are for three minutes uh away from our end time here. So I'm gonna wrap this up, um wanted to give the community some good, um concrete action items in case they'd like to get involved. So we have a Delta Lake slack, uh we're pretty active there so feel free to jump in there with your questions, yeah there's, actually a variety of channels, depending on how you want to interact so feel free to uh interact with us on slack we're all in there.

A

um Another way you can uh reach us is in the Delta Lake LinkedIn Group, which now has like 11 000 members or something we're having some pretty good growths there. So that's been pretty fun, uh so those are great for users now, if you're feeling super ambitious and would like to contribute, we'd love your help. um Now. As a reminder, Delta lake is not just a scholar project, there's a variety of different initiatives we have going on. So we have the Delta.

A

You know I O, slash Delta project, which is Scala, there's a bunch of good first issues in there. So if you'd like to contribute like, if you want to write code with Ryan and Scott, that's where that's, where you do it highly recommend it uh very fun. As you can see, they're they're, very smart guys, but you know, just to recap: for the community like Delta lake is a protocol and Scala is not the only implementation of Delta Lake. We also have a rust implementation of Delta Lake.

A

um So if you want to write a rust code and help build out the rust implementation of Delta Lake, that's the Delta RS project um and another cool kind of update for the community is that there's python bindings for the rest, implementation, which is allowing for pandas access to Delta, Lake I, just wrote a blog post on that. If you go to delta.io blog um talking about how you conversion a pandas data set and also time travel appendix- and you might be wondering well Delta rust, how's that related to pandas.

A

Well, we have these python bindings so and then another project Delta Lake Project we were talking about- was the Kafka Delta ingest, which is a way to that's another rust project to ingest data. From a Kafka data stream into a Delta lake, so uh Delta lake is a programming language, agnostic protocol we're working on a bunch of different connectors. uh If you feel it's so inclined, you know, let me know we can build the Julia connector, but uh that's what Delta Lake's all about we'd love to have you contribute?

A

But if you don't want to contribute, you want to be a user. That's cool too! We'll answer your questions and uh you know we got some great questions from the audience today we're pumped to build out those Docker images. Those are long overdue. Thanks for the Ping on that and uh I think that's about it. uh It says Scott and Ryan. Thank you. So much for your your wisdom. I always learn a lot from chatting with you.

B

Thank you thank.

C

B