Delta Lake Delta Lake Tutorials, 31 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Module 2: Delta Lake 1.2 Tutorial with Jacek Laskowski

Description

Join us for Module 2: DML and Schema - Tuesday, May 31

-Create, Insert, Update, Delete, Merge
-Schema Enforcement and Evolution

This 3-part workshop is intended to teach you what Delta Lake is and how to use Apache Spark and Delta Lake in your data architectures for reliable large-scale distributed data pipelines. This course will show the features of Delta Lake that, alongside Spark SQL and Spark Structured Streaming, introduce ACID transactions and time travel (data versioning) to your ETL batch and streaming workloads. Slides, demos, exercises, and Q&A sessions should all together help you understand the concepts of the modern data lakehouse architecture.

Quick links:
https://delta.io/
https://github.com/jaceklaskowski/spark-delta-lake-workshop/blob/main/notebooks/Delta_Lake.py
https://github.com/badal-io/datastream-deltalake-connector
https://groups.google.com/g/delta-users

A

All right, perfect.

B

Webinar is starting and we have participants joining so welcome board everybody we're just going to wait a couple minutes to go ahead and uh um let other attendees attend as well as joining us on both the linkedin and youtube. So the live streams on youtube, so welcome aboard to all the folks on youtube, so woohoo.

B

uh Meanwhile, while you're waiting for while we're waiting, I'd love to have you chime in on where you're based out of so, for example, myself, I'm based out of the emerald city or seattle, washington. How about yourself yazak? Where are you based out of.

A

Warsaw, warsaw, poland, nice weather outside sunny, so I can't wait to have it done and run into the sun.

B

That rhymes right into the sun, love it so hey for the rest of you that are chiming in. Why don't you go ahead and chime in the chat uh on where you're based out of love to always know where you're all coming from.

B

Meanwhile, I looks like I'm connected to youtube now, so I think that's working now. So that's good! Oh that's cool one of our first uh chime is uh somebody from nigeria. That's pretty cool.

A

B

Awesome awesome and there's a quick reminder by the way, if you're wondering what session this is hey.

A

We got matt.

B

From uh connecticut woohoo, um uh let's see we are live in youtube, live in linkedin, all right, perfect yasuk. In that case, um why don't you take it away and because we are live in all the places we need to be live for today's delta lake 1.2 uh module part 2 tutorial.

A

Thank you denny, so you know I'm on stage and presenting delta lake. Please you with all uh you know, news and latest stuff, that's available in delta lake one two. Just today I finished exploring um one of the latest commands that was added in delta lake, one two, which is a restore.

A

uh You know this is a very fresh uh in my mind, so I'm I I I might cover it soon, perhaps not today, because you know I need to let this new knowledge sink into my head settle down, and so so I can get prepared better anyway, okay, so, ladies and greatest of delta, like delta lake 1 2 in specifically and again, this is open source storage, layer with asset transactions for apache spark and other processing data processing engines.

A

My name is diaz glastowski, and this is the last day of may I'm not mistaken correct it.

B

Is by the way, are you changing slides because the slides aren't changing by the way? I don't know if you.

A

Yeah, no, no, no, no! No! No! No! No! No! No! No! No! No! Okay! There! You go perfect for very first time. Thank you. Thank you. Demi yeah, so I'm miata kalaskovsky it freelancer uh working closely with benny to uh offer you the best of uh delta lake 1-2 and today is second module of our three module uh series about delta lake and I'm streaming live from. Warsaw. Poland, if I'm not mistaken danny, is from seattle or sitting in seattle.

A

Today, okay and I'm best known by this internals of online books, that's uh available on this book, sephila pl- and this is the the website that you should be visiting. Often and just today, as I mentioned, I updated uh brain internals about the latest addition to delta lake. uh As part of this one two release which is restored, and there is another command that we may cover uh next time, which is optimize uh yeah.

A

Just just keep keep in mind that you know new releases of delta lag bring, may bring new commands, so whatever we cover today may be outdated. You know the other release. Okay, so friendly reminder: we've got this. Url go delta, il delta users that you can click and to be redirected to slack channel where you can leave all your questions. Doubts concerns about this presentation and how you feel about delta lake.

A

What you would like to see in the upcoming third part of this three-part series, as well as uh what cool feature you'd like to see in delta lake, in the upcoming release, okay, so delta lake- we cover it already, and you know that this is open source storage layer. You know, there's something about asset transactions. There is time travel.

A

Today's topic is scalable metadata handling with schema enforcement and schema evolution, and I briefly covered that delta lake tries to unify, uh unify, um trex tries to unify streaming and batch data processing, atop, um apache, smart again, um delta lake is just yet another um file format um based on already supported parquet file format in spark, slash, spark, sql and so yeah. If you are curious, you know what it. What delta lake is internally, just think about parquet with some extra features, and you should be good to go.

A

Okay, so demo, one okay. So this is the moment where I'm gonna switch to a notebook that I'm that I prepare for for this presentation for this. um For today's presentation, where I'm gonna show you a few commands uh that should give you enough to get going with delta like so um we skip this part of installing delta lag, because you know I'm gonna use the lake delta lake as part of databricks, so it's already pre-installed.

A

So what else could you expect from people who created um apache, spark and delta lake and then eventually um created this this created or established this company? They called data bricks, so they they offer you the unified um experience and where you can just um immerse yourself into delta lake as well as other features.

A

Okay, so let me switch to to the notebook and okay so module two. Okay- and this is this- is module two, where we are going to talk about how to create a data delta table, and if I confuse you with saying data table, um I really mean saying delta table. You know it's like me, you know saying delta, while saying data, or vice versa, okay, so creating delta table. So what it takes to create delta table?

A

First of all, you can just start from uh very beginning having a data set. That's uh just parket a parked data set. Then the the difference which you may notice. While we are you know doing all the operations atop our um newly created delta table is that we are going to do uh some modifications. Some updates some deletes some mergers, so things that change that are gonna change. The data. That's part of a table, okay and it's not possible um without these features that the light provides.

A

So um if you started using, you know, update or delete or merge, well, yeah merge um a top uh pocket table or pocket tables. Well, you'd run into issues, if not exceptions telling you that it's not supported for file formats by default by by whatever spark sql version, you are using.

A

That's where you may start thinking or considering delta delta lake or delta table. So uh first thing to start working with delta delta lake um is to convert your existing parkette tables packet table into delta own format, which, as I mentioned, is basically a parquet with some extra metadata and json code.

A

Okay, so, ladies and gentlemen, um my our our server is up and running. Our cluster is up and running demo with the latest and greatest of dbr databricks runtime 11. it's in beta. So it's a better release so yeah I we may run into issues if we do well. I'm gonna say that this is part of this better release that they'll eventually fix so not to blame the delta lake right away, okay, so convention.

A

So let me start afresh, so I'm gonna drop my table if it exists and what what I'm gonna use is just uh the name. The name of the table is a delta demo table and and by the way, if you are hearing that my voice is breaking up, it's because I try to be as fast as possible, providing you with all the you know: cool features, because the last time I did this presentation, you know I run out of time very quickly. You know I'm I'm talkative.

A

So so I don't wanna ruin your yet another session so that that's why I'm trying to speed speed myself up a little bit? That's that's perhaps also.

B

That's like you're doing josek you're doing fine. Just if we just work on this you'll be good to go. So, let's, let's go through it because we've got lots, you got lots of material, but if you go out a study piece, I think we're fine buddy.

A

Okay got it. Thank you. Thank you, so so uh dirt table done. Okay and now you know, I'm scout left. So, let's just for fun, let's use some item to create a pocket table pragmatically. So this is just a you know. Demo table and it's gonna be arcade table, and if you are in this databricks uh environment, you have to specify your format explicitly for packet, which is unlike what you do in vanilla spark, and the reason is that, by default in um data breaks, the default for file format is delta, not parquet.

A

Okay, so I specified the format as packet. I was explicit about it and I've got my table, so this table show tables like demo table yeah. I've got one which is okay and I'm gonna, you know select just basic stuff, nothing really out of ordinary.

A

You know this is just a party table: okay, nice now, I'm gonna describe this table and with this extended um version, so now you see that this is just pocket table. Okay, fine! You can also see that there is this directory with all the tables, with all the data sets and with all the files that were created as part of this creating pocket table.

A

Now now we are ready to use this convert delta, sql command, okay and again, since I'm scala dev by day, I'm adventurous and I'm gonna use sql commands as often as possible with some uh python, followed by scala. If I won't be able to you know finish all I wanted with sql or python so be with me. If there are any mistakes, they are definitely part of the show, so I'm gonna convert to delta. This newly created pocket table demo table okay and you see compute snapshot. There are some com entering uh commands uh about.

A

You know, what's going on uh under the hood eventually, you see this okay, um prompt, okay, let's describe this table again with this extended uh variant or option, and you see now that what what was previously a parquet table now, it's delta, you see the provider is delta.

A

So from now on what you previously, um what you previously, um what what previously previously was uh parquet now it's, oh, I didn't know. I didn't know it okay, so it's spark 3, 3.. Okay, that's very first time I see spark 3 3 in action. ah Yeah, that's one of the many reasons why they release databricks 11.. Okay, so parquet now you've got delta, nice. Okay, so we've got our brand new delta table okay. So what was the change to this underlying uh directory? Slash file, structure? Okay?

A

Well, there is this backtrack called underscore delta, underscore lock, that's transaction lock, that's the directory with all the um all the um all the underlying infrastructure that let your command delta specific command work. Fine for you, nice all other files are intact there. They haven't been changed at all. Okay, then uh you can just query this delta log directory because well it's just uh yet another directory, and you see that there is this crc json files for this long, long-ish, zero version. Okay. So this is the version that well.

A

This is this is the way uh delta lake tracks versions of your delta tables. Okay, so every delta table has its own um delta log directory with these long, not a series of number numbers files and these files describe what happened to create this particular version uh from the previous versions. Okay, nice. So we see that there are just five rows: five files, fine.

A

Now um you could also use another way to create if you are into programming another way to create a brand new delta table out of a packet table. It's just delta table object with this utilities like convert to delta, so delta table is a programmatic api or it's a it's, a utility with utility objects in scala with commands like convert delta.

A

So, depending on your needs, whether you are sql a developer or engineer or python or scala, you may use this convert to delta sql command or rely on a delta table api among them, among among the the methods, is just this convert to delta nice, so um yeah, you can just do this uh right right away. Let's see what's going to happen, if I do this demo table again, if I use it, uh this demo table again it's already uh that table. So what were there? Any new uh change to this table.

A

Well, let's see the skype extended demo table, it's still delta, which is very nice and, let's see all the versions. So well, let's, let's go here and let's see if there are any new files in this directory previously there was only zero zero. Now we've got just zero zero and that's it okay, nothing, nothing changed!

A

So um that's the information that or the takeaway from from this presentation is that you can convert your delta table um um again and again, and this this operation is just quite important, uh so so uh converting existing delta table uh changes nothing. Okay, nice! So we've got our delta table uh out of existing pocket table. Now. The question you may be asking yourself is: is there any way to create a delta table from scratch?

A

And the answer is yes, there is, and this is just basically create table specifying the file format you want to use. So you know already that there is this file format for delta, like which is called delta. So just by using um this using a clause, you can just do it failing.

A

So, let's drop this table, so we can start afresh. Let's see if there are any tables that meet this pattern, no, no, no tables and let's create table delta table and the reason I commented it out.

A

I I didn't use I'm not going to use this using clause is that is because data breaks environment and considers delta file format as a default file format for your operations. So unless you specify your file format um delta is the default. It's unlike what you may have seen in vanilla spark installations.

A

Okay, so let me create this table just by using sql again and it returned no results. So let's have a look. What we can get from this describe extended delta demo, and you see it's just delta, okay, so nothing out of ordinary. It's just same results.

A

Except that there are other data files beneath this directory, because previously we we used existing packet table. While now we just use this values, table creation, uh format, okay, nice, so yeah describe uh okay, so another command that you might be um interested in, while working with delta lake is describe detail so describe detail, shows you extra information, extra information that is not available using a pure spark, sql commands, so delta lake offers you a brand new described detail where you could just um describe your delta tables.

A

So you know that there is this format: delta, identifier, internally, fire there's. This name description is now. Whenever I see mouse it's it's like. Why did they do this? Couldn't they just skip it anyway? I'm just you know drifting, this conversation away from the main topic, so um location and some other information.

A

Now you can use a spark sql on way to access tables, and this is just spark.table and whether you are um if you are sql dev, you might be using select star from a table name while um scala, slash, java, slash, python devs are using spark the table and something that that I found really interesting. Is that this display function is only available in databricks, so uh yeah in your vanilla spark installations, uh you'd use show but show is not, as you know, interesting as display in this environment.

A

So I stick to display in case you are wondering how it's possible that I can execute display on this spark table, which is just yet another data frame. Okay, so I listed uh the content of this delta table delta table called delta demo with this three rows: nice.

A

Okay, so we touched base uh um a bit on transaction log, which is basically directory called delta log. So again, just showing you the content, it's just uh three rows uh for this for this table and so hold on. I just got a little bit: ah okay, yeah three, three uh yeah, that's okay!

A

um While I was- uh um and while I was stopping my conversational, my my uh presentation is because I started to think where, where are the uh you know, data files and where the data files are, and but since we are looking at transaction log, they are not here. They are just here. So.

A

Yeah- and these are the the data files, so the difference between these directories is that this is transaction log delta like and specific directory, while the directory that owns this delta log is reserved for data files, forked files, nice. So let's have a look at you know this zero and json file that describes how xero version was created. Okay, what it took to create this version, so we know that there is a protocol.

A

I think we need to make it a little bit bigger okay. So there is this protocol. um You know that like evolves and offers new features, so protocol specifies what readers and writers are able to read or write particular uh delta table. So uh it's similar to what you may have noticed in kafka. For example, when there is this uh protocol um establish establishing process, I might put it this way and there is. There are some other information again.

A

This json file is just regular json json um supported well a spark sql supported json file and by saying a spark, sql json supported file. I mean that you know every line is just yet another json encoded line. Okay, so do not expect this json file to be proper, json, vanilla, json file, it's just this specific format that spark uses whenever you are working with json file format.

A

Okay, so so now, up to up to this moment, you heard about three file formats that spark the delta like uses to offer, offer you to to offer you all the features that you can find in delta, like which was delta like own um file format and by the way in version 1 2. They did create their own um extension of parquet format. This is just to provide this a brand new feature called.

A

uh Renaming, if I'm mistaken, I might be mistaken, because it's a it's just a brand new feature that I started exploring so um so they did yes.

B

That's correct.

A

B

A

Thank you, so they did extend parquet. Just to uh add this extra feature, atop something that's already provided by parquet 5 format, so we've got two file formats already covered and the third one is json for uh all the you know: transaction log, specific information, okay, so protocol is one action, they call it action and there is a metadata change or or update, and since we started afresh, there was no metadata change it. Just um how this delta, like table, was created with this metadata.

A

So you've you see this id and format provided as paquette, some options, schema string and all the you know, information that delta like needs to offer you all the commands, especially for schema evolution and enforcement.

A

Then you will see um from time to time, if not often, and more often a series of add and remove actions, and whenever you see add you should be thinking about a new file being added as part of this version, while remove, which is not the case for this version, because we started crash would be to remove a particular file. There is some extra common info that you may turn off.

A

If you don't like, and some of them yeah that that's basically what it is in this file, okay, so, ladies and gentlemen, we already covered how to create a brand new delta like table from scratch or out of existing packet table. Now, once we have it, we can just play around with this newly created created that table. So what you could expect from this environment called delta lake.

A

Well, there is insert support something that you probably you face problems with, while working with a non-delta lake file formats, because by default inserts well, you cannot insert into existing tables. You have to you know, do it um by creating a brand new directory and some other tricks anyway.

A

So in delta lake, just by doing modification to your data, you basically create a brand new version of this table. So if you, if you stop for a moment and think about all changes that you could expect to be applied to your delta table, just think about new version, that's created and described in transaction lock, that's basically it!

A

So! Whenever you insert or delete or update or merge into a data table, you basically create brand new, uh a pair of um transaction log on files, uh crc and json that that describe what it took to create a particular version by insert delete, update, merge into uh or any other command okay.

A

So, um let's start with insert so, let's add new rows to this delta label. So it's as simple as simple as insert into specifying your table.

A

So let's let me do this and yep you got this extra matrix number of rows affected, inserted and yeah. You can just use it for uh for monetary purposes and looking at transaction log, you realize that you change the shape the data of your existing delta table, which means which implies that you created brand new version, which implies uh in turn that you've got a new pair of um these two files, crc and json and yeah that that's what happened. So we started with zero zero.

A

Now you've got zero, zero, zero one, okay and oh, and on so every transaction. Every command that changes.

A

How your data looks like inside a delta table will basically create another entry in this transaction log and by saying entry I mean another pair of uh these two files: okay, so uh yeah, if you, if you wanna, have a look at this um at one of these files. Well, uh you can just do this because it's a json file, it's human, readable- and you see that there is this art, add commit and there's not remove. I can see remove okay anyway, so you can just have a look at this.

A

uh Just just just um inspect it closely and from time to time you may see art and removes, uh while for other versions you might see on the ads or removes depending on what you did, what you changed in your existing build table.

A

So let me read this table. Okay, just just you know uh what you would do in using just vanilla, spark, sql installation, just spark table and table name and given that we have this delta table delta lake package installed in this environment, and given that this delta demo is a delta table, we get all the delta features for free meaning. We will always be reading the latest and greatest version of this that table so bam.

A

Okay and you've got this 0 1, two three, four, okay, so five rows, and because we inserted these uh extra two rows, we ended up with five rows and all is always cool and fine. So, okay, so let me do this insert over right now. Okay, so you might be wondering okay, we we inserted um two new rows. Is it possible to overwrite existing cross?

A

Well yeah? So the answer is yes, it is supported by delta, like and given that we are modifying underlying the data files or data in this delta table. You basically create yet another version right.

A

So if you list your directory again, let me copy this so next time you are going to use this notebook by the way this notebook is on my github. So so after this talk, um we are going together with danny. We are going to share it with you, where you can find this notebook and you can just play around yourself and you know, develop your understanding better, so yeah. So let me run this command up and yeah. You see that now we've got two verses.

A

Okay, so you've got three pairs of these crc and json files. So, just by looking at this directory, you can determine the the latest version of your belka table nice, so yeah. We we can have a look at this transaction log, something that we did already and yeah. So, uh oh for this particular um for this particular version for version two, um given that we um have overwritten uh two records, you can expect that some files that um held these two rows uh had to be well rewritten.

A

So by looking at this two json encoded description of version, two of this delta table. You quickly realize that one two two files were added while one two three four five files were removed.

A

Okay, and if you are asking yourself or could ask me, you know why why it took two arts and five deletes or removes well, I may find answer, but I'm just hoping that you won't be asking these questions.

A

Okay, uh don't want to show how little I know about that like not today, but just just keep that in mind that it depends on. You know how many files were used for a particular to create a particular version. It may be that you will see five removes while next time you run similar commands on existing table. You may end up with one remove or just 10 or 100 moves. Okay.

A

So again, just by looking at this json encoded file, you will know exactly what happened between versions, okay, so uh yeah. So so you you, may you might be wondering uh what what happens with existing files existing packet files, so the the good news? Well, we get to it whether it's a good news or bad news. So the good news is that all the all the previous files, even though they were marked as removed in this transaction log for version two- they are still there.

A

Okay, and when I, when I say the I mean in this data directory, so over time, you will see more and more files being created, and the reason for this is that you want to keep these files, so you can restore.

A

This is a brand new um brand new command in delta lake, one two or time travel back to your um a previous version.

A

So to do this to support time, travel delta lake keeps all the previous files intact and until you vacuum them and physically removed from removed from your um storage.

A

So yes, this is expected to see this delta demo directory to grow over time yeah. Not only will you see new uh files under this transaction log directory, they are pretty small to be honest, but you will also find new packet files uh created for data for data that you may have considered.

A

A part of you know existing packet files, and this is just to support time travel. Okay, so let me select uh just by using sql. So now you see that we've got two rows in this delta demo table. Previously we had five and three rows now, it's um just. There are only two rows, nice. So let's have a look at this history of this delta demo.

A

You see that we've got three versions. Why we first created this table, then we inserted new rows. So we ended up with five row.

A

Data set table delta table and eventually we overwrote some rows and ended up with version 2., and this is just you know, described by this operation, which does not necessarily correspond to the command that you executed, not the one that you as a user executed, but this is just uh internal representation of the commands that well right so um insert into uh is part of this right, family of commands and that's what appears here under this operation come on sorry, nice, and there are also other you know, operation parameters that you can use for monitoring what happened to create a particular version of your data label.

A

Nice cluster id read version, isolation level and this blind pant, so lots of cool stuff is given here in this describe history, and I'm yet to you know, master all this, so uh you know hats off to um to all the people who can explain all the columns. uh You know without any doubts, I'm I'm I'm not yet there anyway. So, um let's see okay. So let me show you this.

A

I said that all the files, let me go up, I said that all the files, all the data files parked encoded data files are still available even for version for previous versions, so we ended up with eight rows. Okay, so the obvious question or the question you might be asking yourself is: is it possible to consider these files as part of one huge, parkit encoded table?

A

And the answer is yes now be careful, you can read all the parquet files as if they constituted just one single pocket table, but you know already that they are not part of one single that delta table, so you cannot consider them um a part of one single pocket table so, but just just knowing you know uh how delta like uh works under the covers can uh show you questions like this: okay, okay, so, uh and just by uh doing uh you know this display.

A

You will know that, even though you try to read this as a part table well, databricks will prevent doing so, and you should see exception telling you incompatible format detected because there is transaction lock and that's probably something uh delta related. So you should be rather use delta and delta phi format rather than parquet. You can just disable it, but don't do this.

A

Okay, uh nice, okay, so hold on ah okay, so, um okay! So let's move on and let's uh let's discuss, it is basically a simple as simple as insert it's. This good old delete command that you may have been using in the past, while working with relational databases uh like postgres or mysql or whatever, okay, so good old, do it come on, so let me run it so delete from delta delta demo where id equals one number of affected rows is one.

A

So you know that we modified the underlying data, which means that you created brand new uh version. You don't believe me: well that's proper behavior. So let's have a look. So what I'm gonna do is just list.

A

A

Fs ls hop or even better, so you've got two and three or even better, just switch to sql and describe okay. Let me show you this describe a history.

A

I I wanted to cover it later: okay, that that's okay, yeah, show history describe history; okay, let me skip it for now, because I see that I'm gonna cover it later. So so I'm gonna do this right right now, so, just by looking at this transaction lock, you see that there is this third version or fourth version of your delta table. Now update, update, just like insert delete commands, as is updating or is, is about to update your underlying data files of exist of your existing delta table.

A

So, basically, you are creating a brand new version of your uh table and it's as simple as saying update, whatever you want to update so for delta demo for row with id equals two. You wanna change this name to do a french french variant of two or in polish and yeah select uh from and you've got this dude okay, nice.

A

There is this merge into so yeah again a new new version, so you can just do this and you should see fourth, oh, which is fourth so pairs of a pair of um these four base files for fifth yeah. um Fifth version of this demo table nice. There is very advanced merch into command. I, to be honest, it took me some time to sink into it. It's very advanced in a sense that I haven't worked with a similar command in the past, so that was my first encounter.

A

When I started exploring delta lake, I noticed merch into is supported and touted. As uh you know, one of the coolest features of delta lake zero- eight. If I'm mistaken, it was introduced in zero eight. So but I've got demo, so you should feel covered and again you will have. You will have access to this notebook, it's already open source, but after this presentation I'm going to update it with all the newly added cells. So you should have everything that I use during this presentation.

A

So merge merge operation so um again, just like insert, update or insert update or delete. You've got to emerge and if you wanted to run it in vanilla, spark, sql installation- and let's say you have three two one- you would get information that matched into is not supported and this support comes with delta lake. So only after you installed delta lake or some delta lake, like uh file format, you get support for merchant, otherwise you will run into or you will face exception much into it's not supported.

A

So this demo shows delta table merge operation with this merge into command, which is which sits behind the scenes of this uh table match operation nice. So uh you can yeah. We already created this delta table, so we know that we've got our table our data table and you can just say query for uh how many versions of a delta table or this delta table we have, and I for, for the sake of the demo. I just assume that this is a newly created um table. It makes things a lot, a lot easier.

A

B

um So sorry, joseph just uh interrupt since we've got about uh I'd, say about 10 minutes left when we wrap up this section, um and so that way we can leave a little bit of time for some of the questions as well, because we actually had some questions. I did answer them online, but I figured they'll be still useful for everybody to listen to.

A

Nice, okay got it got it. So how many, how many minutes, how much? How much time do I have before we.

B

I would say honestly, let's give give it like two more minutes just to wrap up. Okay, thank.

A

You yeah sure yep, and so so uh again uh you know see some uh scalab. You know for this demo, I've been using scala, but you can just quickly convert it to a sql based demo. um Let me know if you want to see it, okay, so, um and so what I'm gonna do here or what I'm doing here is just creating uh this target delta table api or I use I'm using delta table api to access delta table programmatically, and this target is just description of my uh delta table.

A

There are, there are two views of your existing delta table, something that might not be obvious. um You know, reading documentation and having very little experience with delta lake. One is the history, the changes that led you to a particular delta lake table and the other view is a view on your data.

A

So whatever so, you might have this distinction in your mind, while working with delta tables, given that delta, like supports uh you know, changing underlying data and leading to a new version of a delta table, uh just know that you might be wondering what led to a particular version, or you might be wondering what data is in a particular version, nice, so um yeah. So I created this uh target table and uh okay. So, let's uh dive in into this merge with schema evolution right away.

A

So this merge into is sophisticated uh to the point that you can just specify what you want to merge your existing belt table and based on what condition. So all this leads you to um again very sophisticated command merged into that allows you to do multiple updates overwrites in changes removals uh just by using one single match into a command. So just just just know that um again, I'm using here scala api for delta like, but the same is also available using sql.

A

Okay, since we run our time- and I don't feel uh you know- we will have enough time for questions. I stop here, and I start our next presentation with just this part with merge into and quickly cover schema evolution and enforcement uh during our last uh module. Okay, so danny stage is yours.

B

Perfect, thank you very much. uh So when we switch over you, you can stop presenting the slides by the way I'm gonna close up by the way by presenting a slide. But meanwhile the first question. Actually uh that was asked by bernholt is that does the convert to delta make a copy of the parquet files? Or is this an in-place update.

A

um Very good question, as they say, and the answer is well: there are no updates to existing files. So, as I mentioned already, a transaction log holds all the previous packets encoded files for previous versions. Just so delta lake uh lets. You time travel back to the version that you just changed. So no over no in place overrides no in-place modifications.

A

uh Just a new file creation uh takes place.

B

Perfect perfect and then, uh let's see oh, we will be sharing this notebook right. Oh yeah,.

A

Definitely yes, yes, by the way.

B

Yes, so we'll we will share this notebook shortly, so we'll be updating the youtube link to include uh this notebook as well. Okay. So this is a quick call out I'll answer. The next question is, which is: is there uh possible for aws athena and or redshift spectrum to query a delta table? And yes, in fact, oh uh check out delta delta dot, io slash integrations, uh which includes integrations for all of the other systems, which include presto, trino, um uh flink and other systems?

B

Okay, that actually will work with delta lake uh no problem now. The key call out for athena and redshift is that they are currently using uh the manifest file to query, there's actually being work done by the community to fix that by the way, so that there's more native integration, but the manifest file has been there for a while, and it does allow athena and redshift to actually query delta light directly. So hopefully that helps that question. um Let's see anything else, oh.

A

B

This one, I think, you'll enjoy uh um uh yasek like there there's a great question. That's like there's a cumin seems to be a lot of logs being accumulated when you're running all that process like so if you're performing thousands of data changes in the delta table is like, there's, probably a lot of stuff going in there. Is there a mechanism to clean up those logs.

A

Oh yeah, there is vacuum command that you can just mark files to be deleted eventually, so, yes, it's not executed automatically. You have to use it regularly. As part of your. You know: administration of your delta lake installation or delta lake tables, uh so yeah there is vacuum and I make a note to to cover it during our last session, so vacuum and that's basically it so yeah.

B

Yeah no great question so I'll take the next one, which is basically oh sorry, I just had it and then, of course I lost it. um Okay, I I apologize. I don't remember the actual can't seem to find the text of the question, but basically you're wondering if you can actually do multi-cluster writes to uh for delta tables. In other words, you have multiple spark clusters or multiple, whatever clusters writing to delta. At the exact same time.

B

um The answer is yes, yes, but there's always a but okay yeah the, but is that it doesn't work for s3 for s3.

B

It can be multiple writes by a single cluster or single drive or single jvm, but how you work around that is, there is a feature as part of delta lake 1.2, which is includes the s3 multi-cluster writes, so this allows multiple spark clusters or for them or multiple clusters like, for example, flight clusters to write to the that are not using the same jvm, but not using the same driver to write to the same delta table at the exact same time. Now this is specific to s3, because s3 lacks what they call put.

B

If absent consistency guarantees. These guarantees actually already exist within uh gcs and as well in azure data as your data lake adl's gen 2, I can.

A

Never say the whole thing anymore,.

B

All right, so those those guarantees are already there, but they're not included within s3. This is the reason why there's something special about s3, specifically, okay, so uh definitely check out the blog. We actually wrote it on delta. I o, I think, last week or the week before that specifically calls out. uh It was um written by um marius for osama tv, myself and scott sander, where we explain exactly how that works, and so all right, so we're actually covering coming up.

B

So let me actually do one little quick presentation before I uh uh before we um uh call the day and thank of course uh yasik for this awesome presentation.

B

um If you have any more questions about delta like between now and the next session, that we're doing uh don't go to go.delta io, the three dashes are for, if you want to join our slack, just go dot delta, dot, io slack or go to the uh our google groups, where you can ask tons of questions, go to our linkedin or go to youtube. So by all means. Please go ahead and chime us with questions there.

B

The most common place is the slack one, so just go dot delta, dot, io, slash, slack, okay, there's also delta dash users by the way, so there's actually two slacks. I just realized afterwards it'd be easier if we called it slack from now on.

B

So that's why, with that okay and then um a small little caller, if you want to learn more, including from our esteemed gentleman yasuk, who will be joining our amas and all these other things that we have a ton of sessions for in uh at data and ai summit, go to go dot delta, dot, io, slash, dais data and ai summit 22 and use the code. Dais.

B

Excuse me uh to get 25 off for the conference talks and for uh the training so so you'll be he see both josek and myself there and we'll be able to answer questions live so that's it for today's session again. I want to thank yasik very much for all the great work that he's done so again, we'll update the youtube channel, with uh both the with the notebooks, so you'll be able to download and then we'll be seeing you uh in two weeks for module.

B

Three, where we, uh where yasa, will continue with uh this awesome tutorial.

A

Thank you, then. Thank you.

B

A

It for today I appreciate.

B

Your time, thank you very much.

A

Thank you, ciao bye,.