Delta Lake Last Week in a Byte, 20 Dec 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Last Week in a Byte (2022-12-20)

Description

Learn about the latest #deltalake news a week late! This quick byte includes:
- aws-pandas-sdk (aka aws-wrangler) which includes an optional dependency to Delta Lake to simplify integration with AWS services: https://github.com/aws/aws-sdk-pandas/pull/1834

- polars, blazingly fast DataFrames in Rust, Python, and node.js has support for reading Delta Lake tables: https://github.com/pola-rs/polars/pull/5761

- It is also included in the Python Polars 0.15.3 release: https://github.com/pola-rs/polars/releases/tag/py-0.15.3

- Christina Taylor who leads the Next Generation Communications Platform at Carvana recently published the blog “Open Format in an Omni-cloud World: From EDW targets to Structured Streaming on Delta Lake” which showcases her team’s migration from a data warehouse (Big Query) to Delta Lake using structured streaming: https://medium.com/@christinataylor0926/open-format-in-an-omni-cloud-world-from-edw-targets-to-structured-streaming-on-delta-lake-7913523a868e

- Simon Whiteley and I had our monthly Ask Us Anything answering your most pressing Delta Lake questions https://youtu.be/2HxXM150TnA

- Vini Jaiswal, Florian Valeye, and Will Girten did a 2022 retrospective in the last community office hours for the year on Delta Spark, Delta Rust, and Delta Sharing respectively. https://youtu.be/4OK7jpzj5yM

- At Linux Foundation’s Open Source in Finance Forum, Antoine Amend and Ashley Trainer discuss the FINOS Legend project as part of the session Modernize Regulatory Reporting: Get Ready for T+1 Settlement https://youtu.be/wDxm-zAnlno where they include a call out for Delta Lake to harmonize with open data standards

Delta Sharing connectors:
- Java connector: https://github.com/databrickslabs/delta-sharing-java-connector
- MLflow: https://github.com/databrickslabs/arcuate
- Node.js: https://github.com/goodwillpunning/nodejs-sharing-client
- Power BI: https://learn.microsoft.com/en-us/power-query/connectors/deltasharing
- GoLang: https://github.com/magpierre/delta-sharing/tree/golangdev/golang/delta_sharing_go
- C++: https://github.com/magpierre/cpp_delta_sharing_client
- Terminal: https://github.com/magpierre/dsmb
- Airflow: https://github.com/apache/airflow/pull/22692
- R: https://github.com/zacdav-db/delta-sharing-r
- Excel: https://www.exponam.com/exponam-launches-delta-sharing-excel-add-in/
- Rust: https://github.com/r3stl355/delta-sharing-rust-client

A

A

Welcome to last week in a bite on Delta Lake for the latest Delta Lake news and events a week late. Let's start off with the latest integration, which includes AWS, pandas, SDK, also known as AWS Wrangler, which includes an optional dependency to Delta Lake, to simplify integration with AWS services such as Athena glue, Neptune, EMR, MySQL, S3, and more as well. There's Integrations with polars blazing fast data frames in Rust, Python and node.js, which has support for reading Delta Lake tables. This is including the previous support for data Fusion as well.

A

It's also included in the python polars 0.15.3 release Christina Taylor, who leads the Next Generation communications platform at carvana, recently published the blog open format in Omni, Cloud World from edw targets to structure streaming on Delta Lake, which showcases her team's migration from a data warehouse in this case bigquery to Delta Lake using structured streaming. We had three great sessions at the end of 2022., Simon, Whiteley and I.

A

Had our monthly ask us anything where we answered your most pressing Delta Lake questions, Vinny jazzwell, Florian, velai and will guertin did a 2022 retrospective for the Delta Lake Community office hours for this year on Delta spark Delta, rust and Delta sharing, respectively and atletic foundation's open source Forum uh in finance form, Antoine, amend and Ashley. Traynor discusses Finos Legend project as part of the session. Modernizing regulatory reporting get ready for t, plus one settlement where they include a call-out for Delta Lake to harmonize with open standard data standards.

A

Last but certainly not least, a shout out to will guertin for this great graphic, showcasing all the Integrations with Delta sharing, including Java power, bi terminal ml flow, node.js, C plus go airflow, Excel R and my favorite rust I hope you enjoyed this week's quick bite on Delta Lake. We will take time off for the new year and the winter holiday. So please enjoy this break and we'll see you in the new year.