Delta Lake Last Week in a Byte, 30 May 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Last Week in a Byte (2023-05-30)

Description

We have a #rust takeover in Delta Lake this week! Plus, we'll hear how SQLFluff can make your Spark SQL code on Delta Lake look pretty. And, of course, we can't forget to mention the upcoming Data + AI Summit this year from June 26 - 29 in sunny San Francisco, CA.

A

A

Back to another edition of last week in a bite, your latest news and events for all things- Delta Lake, my name- is Will guertin and I'll, be your host today, I'm filling in for The Talented Denny Lee. Again this week, Lauren Dawson recently published an article on medium that outlines how SQL fluff can be used to enforce consistency and reliability in spark. Sql code within the article Lauren explains how SQL fluff can be incorporated into a CI CD pipeline to automatically lint and format code changes you push to your team's code repo.

A

Lastly, Lauren explores how SQL fluff is the perfect companion for Delta Lake complementing features such as schema enforcement scheme Evolution and check constraints found in Delta Lake yosri Muhammad, lead databricks engineer at kosu published an article that explains how to enable concurrent writers to Delta Lake tables using rust on AWS within the article yosari explains a shortcoming of S3 and how this challenge can be overcome to allow concurrent rights to a Delta table.

A

I really like how all the code samples are easy to read and follow along next shingo Okawa published an article on medium that covers how to deploy a Delta sharing server on AWS shingo even includes sample terraform scripts for deploying the sharing server. He explains which variables to populate in terraform and even how to install and start the code of zero sharing server.

A

Also recently published is an excellent article by Tommy dang on how to build a data pipeline using Delta Lake within the article Tommy provides great code examples of reading and writing to a Delta table, retrieving historical snapshots using time travel and even building a sample data pipeline using mage and I can't forget to mention the upcoming data and AI Summit, hosted by databricks this year between June 26 and 29th. In San Francisco California, there will be a ton of featured guest speakers, special events and even on-site training. You don't want to miss this year's event.

A

That's all we have for this week. Folks, thanks for tuning in and don't forget to go out to go.delta dot IO forward, slash last week to subscribe to the Weekly Newsletter full sound GitHub for latest news and feature releases and ping us out in slack or LinkedIn with any of your questions or comments. See you next week.

A