Delta Lake Delta Lake Tutorials, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Module 1: Delta Lake 1.2 Tutorial with Jacek Laskowski (2022-05-19)

Description

Join us for Module 1: Introduction to Delta Lake - Thursday, May 19

-Bringing Reliability to Data Lakes (Concepts)
-Convert existing tables to Delta Lake [SQL]
-Unified Batch and Streaming [Python, SQL]

This 3-part workshop is intended to teach you what Delta Lake is and how to use Apache Spark and Delta Lake in your data architectures for reliable large-scale distributed data pipelines. This course will show the features of Delta Lake that, alongside Spark SQL and Spark Structured Streaming, introduce ACID transactions and time travel (data versioning) to your ETL batch and streaming workloads. Slides, demos, exercises, and Q&A sessions should all together help you understand the concepts of the modern data lakehouse architecture.

Quick links:
https://delta.io/
https://github.com/badal-io/datastream-deltalake-connector
https://groups.google.com/g/delta-users

A

My friend, ah thank you, thank you. Thank you.

B

No problem so full show that full screen and meanwhile for anybody, that's currently joining right now. This is a session with my buddy yasik. uh Thank you very much for staying up late at night to join us uh since you're, based in warsaw, poland, um and so and me being nice and comfortable here at the pacific northwest.

B

So we are currently live on youtube, so we're waiting for that to go so we're waiting for linkedin and once we go ahead and uh show it on the the delta lake linkedin, then we're good to go so now. um Just in case again, you want to know what session you are you're with yousek and myself denny. uh We are currently talking to the latest and greatest of delta, like we have a delta lake 1.2 tutorial, so give us just a couple more minutes to get ourselves up and running.

B

Meanwhile, uh whether you're on zoom or on youtube or up in linkedin. Let us know where you're coming from which city uh we're uh yasek already admitted that we already called it yasik's from uh from poland, I'm from seattle, washington, hey! Why don't you chime in in the chat and tell us where you're based out of exactly.

B

Sorry things going my friend.

A

Yeah man, you know, I'm so stressed.

B

You, you always say you're stressed, but you're not really.

A

Sorry, I don't know.

B

Yeah we got some great people. We got from ecuador, india, uh the bay area from pune, india, bangalore, india. This is awesome. Oh, we got somebody from turin, italy, that's sweet, oh yeah, oh yeah, how's that in turn, by the way rosario it's got to be gorgeous there right now. Portugal woohoo, oh, always glad to see somebody from portugal. We got some folks from lithuania. That's pretty happy! That's pretty sweet!

B

A

Did you just mention? Did you just mention italy, because I know next time and next week I'm gonna fly to uh bari? If I'm not mistaken, you know my ladies handle it all so so I'll be in italy. So if uh icardo or you know whoever is in italy the time you just ping me, we can just shop in person. There.

B

You go so yasuk's gonna be italy rosario, and they noticed that they're celebrating eurovision, that's pretty sweet, yeah yeah and then the the ukrainian team won right. If I recall correcting your vision right.

A

Yeah, I thought I'm mistaken, you know I didn't watch it but yeah. It was a ukraine that one that the team, but I don't know exactly.

B

All right: well, let's get started right now, just because I figured the people were not here to discuss eurovision, I mean as much as that's enjoyable but exactly yeah, so hi everybody. uh My name is denny. I'm a developer advocate here at databricks, I've been with the delta lake project for quite some time, but instead of listening to my horrible voice, I want to introduce and bring my buddy yasek here. um Great apache spark contributor um delta lake contributor. So hey uh he's going to go ahead and do start.

B

This three-part series on the delta lake 1.2 tutorial now some quick uh uh house cleaning notes, if you have any questions by all means chime in the q, a section on zoom or in linkedin. uh If and also on youtube, I will do my best to go ahead and actually answer some of those questions during the um as the assets presenting. But please a couple things to know number one.

A

B

We're going to be running this on databricks community edition, the notebooks will be available for you to download and number two during the first. You know 30 40 minutes. Please try to keep the questions to the tutorial itself.

B

If you have questions that are outside the tutorial, we still will leave 20 minutes for uh after the the initial uh portion of the tour to answer those questions as well, but we did want to call that out, okay and so now now that I got all that taken care of further without further ado, my buddy yazak, please go ahead.

A

Thank you, danny. Okay, thank you danny. Thank you all for joining me today to share my passion and love to the lake towards the lake to this open source version uh with some flavor of data breaks.

A

Whatever they offer these days in community innovation, I'm gonna use azure data bricks, for the sake of you know, setting up the stage and making sure that if you are trying to follow me exactly what I, what I'm going to do just know that that's going to be azure database, but I don't expect any differences if you are on a different cloud provider anyway. So this is latest and greatest of the lake, and so I'm gonna talk about that.

A

Like I've got three hours to talk about it and from one perspective on this is that you know it seemed like uh three hours should be just enough to to give you a a nice intro to delta, like, on the other hand, I've been with delta lake for so long that it's definitely way longer than three hours so so for for one three hours should be enough to to give you a nice intro to this. You know a solution to this project.

A

uh On the other hand, it might be not enough anyway, just keep asking questions. um You know during this three hours. uh You've got me and danny and other folks who can just chime in- and you know, help you get going with with the lake and what is delta lag just just just by looking at this interest. Intro slide it just open source storage layer with acid transactions for apache spark, okay and other other processing data processing platforms. We'll get to all these details uh in a sec.

A

Just know that if you need just one sentence to describe delta lake, I think that this fits the bo very well. Okay, nice, okay, so um just a little bit about myself yeah. I know that I I can ruin my presentations just by spending hours. Speaking about myself, even though this is not the main topic of my of my presentation. Sorry for this. If it takes too long, so my name is jesse laskowski, I'm I.t, freelancers, specializing in apache spark delta lake and apache sp apache kafka.

A

So whenever you need me, my first question is: is it something about spark delta lake or kafka? If your answer is no, it's probably the end of our conversation. Sorry about it! That's you know how my life goes these days. Well, you might be wondering why is that? Okay, am I not afraid of losing market? You know, because it might be that you know my. My job is no longer necessary for others. For my customers.

A

No, I can see that this is quite the opposite, even though I've been with sparkful for over 10 years, if I'm mistaken, oh 10-ish is to say maybe five, okay, five, let's let's stick to five, it looks like you know, I I keep finding people who are so new to spark and belt lake and apache kafka, but I wonder you know: how is it possible? That's you know they they've been missing it uh for so long. These people did this project so anyway.

A

So I also uh sometimes look at trino and k sequel db. You know the these sql-ish data processing platforms and you know contrary to my beliefs. Back then, when I was younger, you know, sql is not that and it's you know the more I'm with uh sql the more I see um fit.

A

For you know, data processing platforms like like spark and delta lake, and even though I am and still believe, that I'm I'm gonna be hardcore scalab, I see potential for uh sequel and looks like you know: sequel is again cool kids on the block, okay, so contact me at yacht, prpl or dm me on twitter.

A

So this is the twitter handle. So if you've got any questions about, you know spark delta lake and kafka. Just just drop me. An email shoot me a message and we can start conversation. I usually start this conversation with. Did you ask this question or have you perhaps asked this question on stack overflow? So so, because I believe that this way is the best way to reach out to wider community of uh you know: spark delta lake and kafka, so yeah, so don't treat it too personal.

A

If I, if I respond like this, just feel free to to spark my my curiosity about your problems, okay, so friendly, reminding just like, as danny mentioned, we'll have a q a session after me talking and presenting delta lake today during this one-hour session- uh and there is this link: go delta delta, io delta users.

A

This is the link to slack channel the channel where, where you can leave your questions, you know have a nice conversation or conversations with you know: delta users, delta lovers and anybody who who subscribe to delta users channel just just feel free to to be welcomed and and and join the channel and and start conversation about what you know or what you miss about delta lake, okay, so delta lake. So these are the slides and yeah.

A

We will have slides and we'll have some practical demos I used to or I I I tend to present uh what I love um spending my time on uh for this presentations. It's it's delta lake, um using a live coding session format. So, but from time to time I find slides very helpful because they they keep me focused on some some kind of agenda, so so yeah.

A

So so you will see some slides some notebooks I'll, be you know, switching back and forth between different resources, but don't worry if you feel that you got lost um during me. Switching between you know different windows doing different stuff. At the same time.

A

Fear not you will get all the slides all the notebooks, if not just drop us an email uh to danny myself or or leave a comment. um You know on linkedin and we'll respond with all the resources you need, but um whatever I am gonna say about delta lake is my take on what I could what I managed to see through the source code of delta lake. So um I apologize for not using all the buzzwords that people love like you know: data lakes or lake house architecture, blah blah blah, I'm not good at it.

A

I I leave it to. I leave it to danny and others who are well versed with all this. You know nice cool works, I'm catching up. Okay, in other words, all my data, engineering and knowledge and wisdom, if I may put it this way, is coming directly from the sources of apache spark delta lake and kafka and some other systems but or projects open source projects. That's one of the many reasons why I love spending hanging out with these open source projects because well I can learn from them.

A

Read these these sources as a book, a kind of a book and- and these sources teach me you know how to think about delta like and others so and all this information that the nice looking documentation introduction is on home page of delta lake, which is https delta lake io delta io.

A

So so for me, you know whenever I I hear this word delta, the very first time I I I learned about delta lake, uh but then you know three four years ago and ish uh it was like delta. Is it are they talking about this a-line airlines? Or is this you know some new term so, but but now delta is only about delta lake.

A

So whenever I say delta just know that I'm speaking about delta, like from time to time, I might get mistaken by saying data lake whenever I make this mistake, it's, uh but this is just a mistake and I'm thinking about delta lake, okay, so delta, like is an open source, transactional storage layer over over object stores and believe me, it took me some time to sometime to understand that the value proposition of delta lake- I remember watching burak td and others michael.

A

You know these guys, who created belt lake. You know the big brains, uh you know the the the smart guys behind the lake project. Sorry, if I, if I didn't include other others who contributed to this delta lake, but the point is that they put you know um this description, why they needed delta lake- and I remember michael understood he- he mentioned something that they they struggled with, uh write using spark or sparks as streaming.

A

I think it was about spark streaming uh using file, uh file, format, um data source and writing to to object, stores and making sure that you know these rights are consistent and there was this story about you know um all these cloud providers with this with cloud object, stores being not non-consistent, not perhaps they were atomic, but not you know all these acid properties. So all this brought them to this concept of hey.

A

Let's create this idea: let's create delta lake okay, so they just extended apache spark with a brand new format to support to bring asset transactions, time, travel and scalable metadata handling to apache spark. That was there that that's my understanding, what they, what they needed or what they did with delta lake and what's the what's the intention or what's the goal of delta. So all these three asset transactions, time travel and scalable metadata handling features were not and still are not available in pure vanilla, apache spark.

A

So we needed them for interacting with this inconsistent object, stores in cloud providers like azure, aws and gcp, so we needed it to transactionally.

A

Write data to this large inconsistent object source. If I'm going to put it this way again, if I put something using words that mean something something different for you I apologize, it might be because you know I didn't study all these topics. I just read: um read about them about these topics from from the source code itself or in articles by uh the wise men from you know, databricks and out and other projects, okay and speaking about apache spark and delta lake.

A

um The the nice cool feature of delta lake is that it supports streaming and badge data processing. In other words, for me, from from my perspective, when I started exploring delta lake, the the very first discovery of the day was hey. What's the fast? What's the what's the what's cool about delta lake, since it's just yet another data source of apache spark so for for a long time, I couldn't understand the value proposition of delta lake, given that it looked like something that I had in apache spark already.

A

Yet I missed this three main features of delta lake, something that I think made delta lake, so cool, um so asset transactions. Again time travel and scalable metadata handling.

A

You could do this by yourself without the leg, but it definitely lets you to create something that would resemble that like in the end, so it's fully compatible with purchase park apis. So again, this is just yet another data source data format in apache spark. So if you know apache spark by heart or you, you worked with phuket um with csv, json or other formats, uh you should be good to go and just just use delta lake.

A

So it is it's especially very important to to understand that if you always work with parquet um tables or data sets, uh deltalike is just parked. Data sets with json metadata.

A

So as I, as I put it, as I described in this uh here, think about delta lake as an open source, transactional storage layer that uses parquet data sets.

A

But that's this extra metadata kind of like hive metastore ish metadata uh in in json code format in in json format, so so so data lake using apache sport infrastructure can give you this nice properties like acid transactions and others. Okay.

A

Okay, so let me let me wrap it up, wrap it up in this a nice demo that will show you what exactly it means to um use delta lake and show you exactly what I meant by saying that this is just sparking data sets with extra json encoded metadata.

A

So if you are old enough- and you worked with well old enough- might not be a good intro to this.

A

So uh if you've ever worked with a hyphen with with hive enabled environments with hype, metastore or some kind of a metastar where tables were described externally by by some some um external properties uh delta lake, with these with json encoded files may look very similar.

A

So since I haven't spent enough time with hive, you know I came to this big data and data processing platforms after hadoop was considered, especially my produce was considered a big no, no because, while spark was already available on the market in the I joined spark when it was released as 1-0. If I'm mistaken, I may have joined it when it was zero eight, but they can't remember it exactly now.

A

So so I had no pleasure or pain working with you know, tools before spark, so I might confuse you if I use something hyper related here. Doing dispensation apologize for this okay, so what it takes to so this demo is supposed to show you what it takes to install delta lake in your spark application. Again we are speaking spark here. We are spark uh lovers, spark users at least we've been using spark for data processing, and now we want to extend the vocabulary of spark with these features of build alec.

A

Okay, so we need to install deltoid. So this demo is going to show you what it takes to to install that lag in your spark application. Then we are going to create a very first delta table, review its internals and then just see what it takes to read and write to a delta table. It's going to be this table or another one. We can, you know create as many tables as we want, so we can end up with couple and we are going to use spark sql and purchase a stream.

A

So if this session stops here well, if it, if the session stopped here well by saying here, I mean after this demo I'd be very happy, because that should show you that using delta lake cost you nothing, and especially in environments like the databricks, where they pre-installed delta lake.

A

More more importantly, they made delta lake the default format in spark contrary to, unlike what you may have found in vanilla, apache spark installation. So you may get confused or happier just by finding out that databricks comes with delta lake as pre as a pre-installed package and being the default format in your databricks notebooks.

A

Okay, so I'm gonna switch to a databricks platform. So let me switch to my um notebook and this notebook delta. Like it comes from this repo, and by saying this repo I mean when you go to and again I'm going to provide you with all the links, if you need them um so so no worries, if you miss this or that uh that's gonna be also, this presentation is gonna, be uh recorded, is being recorded. So so uh you should miss no information that I'm showing during this presentation.

A

So um at this url github.com, so you can find this spark delta, like workshop repo, where you can find notebooks a few notebooks that I've been using for my uh for my demos among them is delta like and you might get scared seeing pi y as the default uh as the extension of this notebook, but feel note, uh even though I used pi y for as an extension for this notebook and yes, python is the default format for this notebook. um I um used a sequel.

A

I I used mixture of sql, uh um scala and python to present the uh the features of delta lake in this notebook, and uh just if you are curious, uh why did you? Why didn't you stick to sql, uh then, since sequel um seemed to be, you know highlighted as the as the as the vehicle or tool uh to um to intro you to um data lake?

A

Well, um I can't do this in the default format, but this extension is not going to be sequel that that's at least my understanding anyway uh again uh just to uh highlight the fact that I'm um uh scala def, it's been a while you know it's. It took me a while and before I converted myself my feelings to be more friendly towards this platform, this notebook platform, you know, because you know I I tend to stay away from uis- that hide the um the real uh features of uh what I'm using through this curtain.

A

Through this you know a ui uh ui based um platform that that's that's why I hang out with a tools via the source code. Okay, so delta lag, so a notebook is in my repo is freely available. You can use it for for your own uh for your own learning purposes or you can just use them, uh use this this or other notebooks to teach others what you learned in this presentation, okay, so, uh okay! So there are some links to some other learning.

A

Resources uh among them is my book, uh something that I um I've been updating. um You know every every delta lake release so now I'm focusing on delta lake one to one. So um unless you use a one two one version of delta lake, you may find that some features are not available in your version and that's because well, you are using that the older version- that's uh um well. You should uh in essence you should upgrade to the lake one to one immediately or as us as soon as possible.

A

Why? Because there are some features only available in delta lake. One two like um among them are features like optimize command- that's not available in in delta lake. Before this one two release, as well as some filtering optimization that I'm gonna discuss later on.

A

Okay, during during this three hour, uh the tutorial session, uh serious okay, so uh we'll get to internals in a sec. Okay, so um does it this session? This section is to guide you to the slides that I have here so in case you are wondering where you can find these slides. Well, you can just find them in the notebook. So all you need to get going with this. Intro is just open this notebook and you should be good to go. Okay, so intro overview.

A

So let me start with overview. So what are the features that that make delta lake uh so cool, uh at least for me and some other people, so uh delta lake is open source? That's one of the uh cool features that I found in delta lake um other is that it's apache spark based.

A

um There are features like asset transactions and time travel that I already mentioned. Okay- and I found this quote uh very um heart- touching- if I put this way a nice intro to delta lake, another intro to delta lake, so delta is a storage format, while spark is an execution engine.

A

So combining this, these two together will uh give you storage and compute platforms uh in in in um yeah platforms, so uh think about delta as a storage format, while spark uh as this a compute platform that enables all the features that the delta lake provides.

A

No the delta lake one two one supports apache spark three, two zero. The latest and greatest of spark is three to one, but no worries uh three one, two three three two one uh um should work fine too, because it's just a minor release uh just just know that, um although it's not officially supported via this built-in that I provide link to uh uh I've been using delta late, one two one uh with apache spark three, two one, uh just fine in my projects.

A

Okay, so what are the other uh cool features in delta lake? um There are tables, so delta tables um are basically parket tables, just good old parking tables with transactional with transactional or transaction transactional, log or transaction log, and this is the extra directory that you can find in directory where you store your delta encoded, if I may put it this way or delta owned files.

A

This transactional log makes all the goodies that are available in delta lake two um pocket tables. So we all know that parket parquet format is a nice columnar format. Nice for apache spark analytical workloads. Now we know that we've got something. That's even uh cooler, so consider delta, like um something like parkit plus plus. If this analogy to c and c plus plus, is something that you consider good, so all the changes to the state of a delta table are reflected as actions that are persisted to this transactional log in json format.

A

We'll get to it, see what it means. Okay, again, I am introducing you to delta like via very low level, um very low level information so, and they all come from uh from from directly from the source code. So if you find it too low level, let me know: okay keep asking questions.

A

uh There is support for optimizing optimistic transactions.

A

So there is this a nice cool feature where, whatever you do with delta lake, it's usually transactional, meaning that uh delta lake will um read table at a given transaction, meaning you know what version your read is from or is based on so next time you try to commit your optimistic transaction. You basically check what changed since you read this table and if there are modifications that do not conflict uh with yours, you should be good to go to commit your transaction.

A

If there are any conflicts, delta lake itself will handle them, will retry this commit and will keep retrying until this commit goes in. That's that's, basically what happens under the covers and that's what we all love um in in that lake, because it's just you know free of charge. It's just part of the correct.

A

So there are these transactional writers, um while exploring some some code in apache spark. I found this very informational, very influential to my understanding of delta. Like so all these commands are basically transactional writers, in other words whatever they do. They do it under this transaction. So in other words, you can execute these commands from different clients at the same time and be assured that these commands, even if they conflict they will these these conflicts are going to be resolved by delta lake.

A

If possible or delta lake will report that well, your transaction could not be committed, and you will get this information. Why that's part of delta lake? Okay, there's there there are pragmatic, pragmatic pokemon. They they probably pragmatic too, but there are programmatic apis, something that's you developer, scala or python can use to work with delta lake, so there are apis to work with built table or delta tables.

A

Via this delta table api, you can interact with commands. So if you wanna know what exactly is part of this delta delta table api? Well, you've got this management interface and you can just do something like convert delta, create create, if not exist, create or replace full name. All these sql-ish or sql-like commands uh that now are available for you, scala or python developer. So you see a lot of similarities between the names of the corresponding sql commands and that's the beauty of delta, like working with delta.

A

Like I, I found myself very often confused by uh not being able to pick the language I should be using for my uh delta lake development, because for some for some use cases sql is very declarative, very concise and expressive, and exactly to my needs while false for some cases, I find um I I I find a sequel, a bit constrain, constraining or limiting my my my my hopes to develop this. All that so I just switched to um to python or scala and I'm good to go again.

A

I can just mix and match languages in one single apache spark based application using delta lake sql or python or scala apis. That's the beauty of this combo and that's what I always highlight when I'm presenting spark or build like this variety of apis and languages. Supported um for for people who are new to this environment may be overwhelming and maybe very frustrating what to pick to get uh to get the best out of these platforms spark and delta lake.

A

Well, for me for more advanced people, if they, if you, if you feel adventurous, you just pick sql for for one one parts of your application while for others you just start using sql, okay, so yeah, so that that's what you will see in notebooks, uh because notebooks, how to say, invites you to mix and match uh the best out of these. These platforms spark delta lake and and databricks okay, so program programmatic and pragmatic apis for working with commands against the table, creating delta tables modifying them.

A

Then there is this delta table builder to create.

A

Some more advanced applications like generated columns.

A

There is more to it, okay, but we'll get to them, uh especially after I hear from you your feedback, how you felt during this presentation during this day, one what you miss what you like based on your feedback, I can just adjust pace and agenda and just cover what you feel might be most beneficial for you, okay, so I mentioned it already that there are there's support.

A

Delta lake supports structured queries by started queries I mean whatever spark provides for devs for data data users, data processing, user.

A

They people who need to process large amounts of the data, and we've got support for batch queries, as well as fostering queries, so delta lake supports uh watch, reads and batch rights as well as streaming reads and submit rights exactly what you could expect from something that was developed by the very same people who created spark success streaming so td and others td, michael and bruno, joined the streaming team later, but he also contributed to to sparks streaming anyway. These two people, michael and tds, perhaps others they created sparks of swimming.

A

Now they are the core team. Behind delta lake, so whatever else could you expect from from these people and if they've got so much experience with spark? Well, they made delta lake so much spark away.

A

Okay and there's this support for different, lock stores. I just keep it um there's this information, excellent information, how you can find that your logical query plans or your queries, work with delta tables, so from a spark perspective. Delta tables are nothing else like logical relations over hadoop fs relations.

A

Okay, so if you are not with this whole file index, this is this is crucial. So if you are looking at query plans- and you want to immediately know whether you are working with delta or not with all the you know- cool features just search for just just collect all the logical relations in your query. Plans then find all the uh logical relations over hadoop facilitation and then see if these hadoop fs relations uh use the whole file index for for curious for people who are curious about this.

A

The whole name included in this file index name just now that the the internal name for delta lake was initially tahoe. This is the if I'm not mistaken, the biggest lake around okay.

A

Let me skip this part, but this is just a lake, so at some point they decided to open south this project say after you know, uh checking all the cool features internally, so they they decided to to rename it from tahoe to delta lake. So you can just find tahoe in many places uh reviewing uh source code, okay, so um yeah. Whenever you see tahoe file index, just know that you are working with delta delta, like in your spark applications.

A

uh There is the support for concurrent blind transactions that cannot conflict with other transactions. uh Let's keep it because it's very internal, very, very sophisticated, and I'm I'm not very convinced you might need this information just right today. There there's support for generated columns, something that people found very interesting. uh um One of the you know cool features that the delta lake provides, and I skip it again as well, um because I don't feel that you know you need it just today.

A

There are there's support for table constraints um and yeah, and now this intro leads you to the um nice two features in delta lake, one two.

A

So, among all among these uh features that came as part of delta lake one two, I managed to explore and learn learn a bit about um and they are compacting small small files, which is yet another sql command called optimize.

A

If I'm not mistaken, there is no support for optimize commands using this delta table, and while I'm saying this I always say I always promise myself to write a ticket to this open source project and ask them to provide this optimized command as part of this delta table api. um I I couldn't find this uh part of this um api. So well you need so, regardless of what you think about sql. If you need to optimize your delta table, you've got no choice but to stick to this optimized sql command and you've got this.

A

Also nice cool feature um something that that's not necessarily user user facing. uh It's called data skipping and data skipping uses metrics that are collected during writes, and that's why I needed all these all these transactional writers, because they will trigger writing extra extra metrics. That spark will use spark via delta lake or, shall I say, delta lake using spark infrastructure to optimize your um your queries and by optim.

A

What I mean by that is that you can skip scanning some files uh if you are using filters that do not match data, that's uh part of the this or that file, and that's something that they added in one two and they call it data skipping, something: that's not that's not user facing, but except you will get better performance yet another reason why you should upgrade to delta lake one two: okay, that's enough about overview of the features that I may or may not touch base during this three hour. uh Technical presentation.

A

Well, 45, minutes past, oh man! Where did this time go by? I can't believe you know, I'm so stressed, and every time I'm presenting the leg time flies by so quickly, okay. So what about installation? So installation is basically very simple if you are using databricks.

A

So if you are using databricks platform, delta is pre-installed and you've got it as the default format. I was part of I've been part of a project recently and um they didn't know that without specifying what format they need they've been using delta lake, something they requested.

A

Can you imagine so this project, the you know, teammates in this project? Just asked me hey, yes, can you con convince our our our leaders to to just start using delta lake because we've been hearing so many? You know good stories about delta lake and I said, hey show me the code. You know I was independent consultant show me the code and they showed me- and I said, and what's the execution environment, it's databricks, it's azure databricks and I said well so you're lucky because you you you already.

A

You already have been using delta lake just just without asking anybody for this. Just by sticking to this default file format. Well, on databricks, you are using delta lake on a vanilla, open source, apache spark installation. You need to do some extra steps, something that that well requires a couple of things.

A

First, you need to use this spark sql extensions property and point it to io delta, sql delta sparks session extension, and I doubt you are going to remember it. uh You shouldn't, to be honest, just say: go to this page and just copy it so um I'll promise I'm gonna, keep it updated. If not, let me know I'll. Do it right away? So uh if you are scala developer, just like myself, um you are going to instantiate your spark session just like this accession builder, and then you are going to instantiate instantiate this spark session.

A

uh Adding these two new properties, one is extension, another is catalogs or catalog spark catalog spark underscore catalog is that is the name of the default catalog in spark. So just by rewrote the um changing configuration of this for catalog, you are making delta to be responsible for all the queries again in four tables lookups, you know renames, all all the lookups, all the table lookups are going to go through this delta catalog infrastructure.

A

So it's gonna check. If this this table is perhaps a delta table. If so, you will get all the cool features, uh if not well. Your applications should just work like it did before without delta lake, okay and in spark shell, and it's just that basically, uh two um conf properties uh again via this minus minus gone.

A

So even though I I I showed you how to instantiate accession via these two configs. To be honest, you shouldn't be doing it programmatically. You should just use spark submit with these two configs and you should be good to go. The reason for this is that sometimes you may need this extension. Sometimes you might not, but definitely whatever you can just specify externally to your code, will require a no recompilation code, recompilation and that will make uh devops slash.

A

Admin, live or user's life easier, okay, and something that I I found quite recently is how to know the version of your delta table that delta package installed. I think they added in version one two, one one, two one yeah one, two one and or one two zero.

A

So if you do this sprinkle an io delta version, that's what I managed to find as the way to know about well version installed in your environment, given that I'm in python, I'm gonna, switch to scala and I'll, make it a little bit bigger and yeah, okay and just execute, and I'm hoping that I'm get this. Oh that's interesting!

A

Oh that's! A very interesting discovery! Honestly. I didn't expect it why, because this server uses the very recent dbr 105, which is not, which is the latest and greatest of betaflix runtime. So what I did expect was dealt alike one two one at the very least, if not something they provide, you know internally, so sync one one, hey danny: we need to talk.

A

Definitely okay,.

B

So, okay, it just just to provide some context. Basically, you probably need to use the um the the beta release of delta in order for you to have the one two one um used use the official release, so uh this is just showcasing how the oss delta lake oss is actually running on a very different, much faster pace than even databricks can keep up with that now, which is.

A

Sort of cool- oh yeah. Definitely thank you. Thank you for this nice explanation.

A

Okay, so speaking about you know about this environment, uh we know that we are working with one one, but now that all these cool features that are part of delta lake oss, open source version of uh of delta lake were already part of data data bricks. So just just know that, even though you are looking at you know, features of data, bricks, phone or internal delta lake.

A

You know all the features that are part of uh open source version um are and and have always been- not perhaps not always, but have a long been part of this environment database. Okay, speaking about delta lake, okay, so the default flight formats I mentioned that this is uh um this is just uh delta.

A

So if you wanna check, what's that, what's the you know, format just execute this in scala or, if you are, um you know, sql user just set spark sql data sources default, and you will see that okay, nice um creating delta table so uh yeah. So, first of all, you might start very simple, very um easy. um You may have a very simple way to del the lake via your existing pocket tables. There is this command called sql command, not also not only a sql. There is also a scala and python support too.

A

So a convert to delta is command that will convert existing park tables. So if you have existing park tables, you can easily convert them to delta and again, if you are wondering, what's the matching behind this convert delta a command, it's basically to add this extra directory, underscore delta, underscore log to your directory with your pocket data sets, and that's it that's all it is it's it. It's all there is to to this converted command. So what I'm gonna do here is to drop this table uh if exists.

A

Okay, then yeah. There was another presentation where I was adventurous and I wanted to generate a brand new demo table and you see that now, I'm generating it using python again. Python is the default. So just so I'm explicit about this. This is fine, okay, so importing functions and I'm creating five row delta packets parquet table by using this data, five rows with extra column.

A

Random number and I'm just writing the format again. I know nothing about the audience. So if you are feeling a little uh a bit uncomfortable hearing, something that's very basic for you, I apologize, but given that I know nothing about the audience, I I need to say that this is basically spark and this is spark way to create parkette, encoded or parquet table.

A

Okay, so I'm doing data. This data set is written down or written out us in parquet format with overwrite, but we know that this table doesn't exist. Also, this override basically does nothing and we save it as table so and so this table is going to be registered to hive like metastore, where we can just see what this table represents. So let me execute it.

A

It's run. It runs successfully the last time I I executed it, so I'm hoping that nothing bad will happen here and honestly um for all this sorry to say it sorry for for exaggerating a little bit for all the hate and love I have towards this data bridge platform. Again, I'm not ui user. I love you know shell, linux and sticking to come online and then scala.

A

I really like using notebooks for presentations, because I've got this version thanks to databricks. I can version it to via this repos feature in my github repo and I've got all the cells all the columns, something that I can just import right away and have this spark environment spark aware, spark enabled environment of data breaks available and happily ex available and ready to happily execute my my you know, sales, my my code, something that I found really hard in other environments.

A

Even uh when I was you when I was using intellij idea for presentations, it was not as simple teach me. You know how uh um teach me how to do this outside databricks. If you feel that you know database does not give features that you found interesting in other platforms, for example intellij idea, okay, so I executed so now I'm gonna use sql. So you see this is the very the very best feature of beta breaks. I can switch between languages whenever I feel more comfortable with one language like sql or python.

A

I simply mark a cell just by using this magic and I'm I'm ready to go so sql, this time short tables like double table, and you can see that I just created this double table. It's not temporary and again. This is just a pure apache spark come on well spark sql come on nothing to do with delta lake.

A

As a matter of fact, uh it's it's it's not even a delta table, then what I'm gonna do I'm gonna show you what seats or what's inside this table so select start from the table again, if you feel uncomfortable, seeing something- that's very obvious for you. I apologize again know nothing about the audience. So I need to do this and do this assumption that my audience knows knows nothing about. You know sql data breaks and all the stuff that you consider basics. Sorry, okay, so this is the table that I saved.

A

If you are wondering why these ids are not sorted well, we are in distributed environment where sorting is pretty heavy. So uh unless you need it badly, uh you shouldn't be doing ordering it's it's often uh overkill for for what you what you need.

A

So can we do uh that's extended yeah? Why not, and, and what is this extended describe extended is sql way to ask for a table metadata and it's supported by spark sql again and again, it has nothing to do with parquet with sorry we have delta. I was thinking about parquet because I wanted to mention parquet, and I said smart parquet.

B

Sorry, uh sorry to uh chime in jessica, we've got about two minutes left, so why don't we just make sure we wrap it up so that way where we can.

A

Yeah, okay, so you can just yeah yeah. Thank you. Thank you for reminding me that time flies by so quickly and while presenting something something so cool like like okay, so you can just see you know this table. This is just basically a direct string on your file system and you see success committed all the directories as well as files. Okay and these files are working files.

A

Now, why am I showing you this, because I need to show you that the only difference between these nine rows that represent a bracket table will see extra row for directory.

A

That's the only part that make your pocket table delta lake away, so I'm gonna use, convert delta command and I'm going to do this on this demo table.

A

So the only change I am introducing via this convert to delta sql command is that, rather than seeing nine rows after I release files under this directory, you will see 10 or 11. You will see we'll see in a moment. So it's okay. It took just 15 seconds and let's describe extended first, so we now learn that this is delta table you see previously. It was parquet and now, ladies and gentlemen, I'm gonna see how many files I end up with for this previously parquet now delta lake table, so 10 rows.

A

So what's the difference, if you start comparing these directories, this listings sorry this one, this one against what I just listed. You will see this directory.

A

This is the distinction, the main the the the game game changer, as you used to say um what makes this at actually a delta lake table, and I think we can stop here and I could just talk and talk. You know about you know what's inside, but we got two more hours, so so I'll stop here and let danny uh close the show.

A

Thank you. Denny.

B

A

B

No thank you very much uh hey. Thank you very much for the time I did my best to answer all the questions uh that that came on in so please join us and the delta user slack between now and the next session, which is on the 31st of may, uh where yasik is going to go ahead and do the second session, and so, like I said before, join us until they'll use the slack in the interim yazak. I want to thank you very much for this fun session.

A

Thank you, pleasure is all mine. Definitely all right.

B

Looking forward to seeing you on the 31st there, we go eventually yeah.

A

Everybody thank you. Ciao, bye,.