Delta Lake Last Week in a Byte, 18 Apr 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Last Week in a Byte (2023-04-18)

Description

Learn about the latest #deltalake news a week late! This quick byte includes:
- delta-rs 0.9.0 release: https://github.com/delta-io/delta-rs/releases/tag/rust-v0.9.0

- delta-spark 2.3 release slides cc #apachespark https://www.linkedin.com/posts/willgirten_delta-lake-230-was-released-last-week-and-activity-7051596577588551680-cN5M/?lipi=urn%3Ali%3Apage%3Ad_flagship3_publishing_post_edit%3Bt9SxCaqKRzedAIzhUgh%2BQw%3D%3D

- Kotosiro is a minimalistic #rustlang implementation of a #deltasharing server that currently supports both #AWS and #GCP environments. https://github.com/kotosiro/sharing

- DataLakeIO connects #ApacheBeam and data lakes such as Apache Hudi, Apache Iceberg, and of course, Delta Lake https://github.com/nanhu-lab/beam-datalake

Great publications including:
- GeekCoders: How I use MACK Library in Delta Lake using Databricks/PySpark https://youtu.be/qRR5n6T2N_8

- Lakehouse by the sea: Migrating Seafowl storage layer to delta-rs by Marko Grujic https://www.splitgraph.com/blog/seafowl-delta-storage-layer

- Matthew Powers, CFA published two great Delta Lake blogs - How to use Delta Lake generated columns (https://delta.io/blog/2023-04-12-delta-lake-generated-columns/) and How to create and append to Delta Lake tables with pandas (https://delta.io/blog/2023-04-01-create-append-delta-lake-table-pandas/) #pandas #generatedcolumns

- Khuyen Tran posted How Delta Lake simplifies pandas DataFrame versioning and allows access to prior versions for auditing and debugging using delta-rs cc #pandas #dataframes https://www.linkedin.com/posts/khuyen-tran-1401_python-datascience-deltalake-activity-7050102630802460672-zfIV/?lipi=urn%3Ali%3Apage%3Ad_flagship3_publishing_post_edit%3Bt9SxCaqKRzedAIzhUgh%2BQw%3D%3D

- Long time Apache Spark contributor and Databricks Beacon Bartosz Konieczny recently published Table file-formats - Z-Order compaction: Delta Lake. https://www.waitingforcode.com/delta-lake/table-file-formats-z-order-compaction-delta-lake/read

A

A

Welcome to last week in a bite, your latest news and events for Delta Lake a week late, I'm really happy to announce the release of Delta rust to API 0.9.0 with various enhancements, including hdf support, upgrade data fusion and write support for additional Arrow data types, there's more to come and we're already working on the Zero tot 10 release.

A

Do you want to know more about Delta Lake follow, will glorton and get the latest Delta Lake information, including these handy, slides, showcasing? What is new in the Delta spark 2.3 release?

A

Well, he love seeing the growth of the Delta sharing community and the rust community and with kodociro we have both kotocera is a minimalistic Delta of rust implementation of the Delta sharing server that supports both AWS and gcp.

A

It's really cool, because they're call outs to the original rust, sharing Delta, sharing implementation, known as Riverbank by Delta maintainer r, Tyler Kroy as follow callouts to row API, which allows you to create fully fledged apis for slowly moving data sets without running a single line of Code by Delta Delta maintainer, QP Howe. So thanks very much shingo akawa.

A

For your awesome contribution, we're also happy to announce another Apache beam-based project called beam data Lake, which has an I o connector called uh Delta Data Lake IO, which connects beam and data Lakes such as Apache, hoodie, Iceberg and, of course, Delta Lake, I'm, really happy for the shout out for Geek coders as uh where they show the case. The Mac library in Delta Lake, the Mac library, is spearheaded by Matthew.

A

Patterson provides a variety of helper methods that make it easy for you to perform common operations for Delta, like from split graph, recently published a great blog lake house by the Sea migrating C file, storage layer to Delta RS. What's really cool is positive, call outs for the Delta IRS project, including its maturity, and how open and welcome and active the the community of maintainers are so thanks, Marco and welcome to the Delta RS community.

A

In addition to the great Mac project, Matthew Powers also published two very informative blogs on how to use Delta Lake generated columns and how to create an append, Delta Lake tables with pandas, and we want to give a shout out to a win Tran for posting on LinkedIn. How Delta link simplifies Panda's data frames, versioning and allows access to Prior versions for auditing and debugging and, of course, she's using Delta, RS and last, but certainly not least, long time.

A

Apache spark contributor and databricks Beacon Bartos kunichik recently published table file format Z order, compaction, Delta Lake, so that's it for this week. So go follow us at go.com last week for the latest newsletters and don't forget us to Ping us on GitHub, slack or LinkedIn. See you next week.

A