Delta Lake / Delta Lake Discussions with Denny Lee (D3L2)

Join Robert Pack, Sr. Digital Expert Cloud Native Machine Learning Platform and Technology Principal at BASF as he discusses the relationship between process engineering and data engineering. In a connected world, there are very interesting graph-like relationships when we’re talking about processing chemicals in a sustainable way to efficiently fast querying of data to training and running machine learning models at large scale, high variety and low cost.

Quick Links
Robert Pack: https://www.linkedin.com/in/robert-pack/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
GitHub: https://github.com/delta-io
Join Google Groups: https://groups.google.com/forum/#!forum/delta-users

Join this special D3L2 vidcast/podcast with Andy Grove who has been specializing in query engines and distributed systems. Among many of his accolades, he started the DataFusion and Ballista query engine projects and donated both to the Apache Software Foundation as part of the Apache Arrow project. He also donated the initial Rust implementation of Apache Arrow and recently created Ray-SQL, a distributed SQL query engine in Python using Ray.

Quick Links
Andy Grove: https://www.linkedin.com/in/andygrove/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
GitHub: https://github.com/delta-io
Join Google Groups: https://groups.google.com/forum/#!forum/delta-users

Combining SageMaker Studio and Delta Lake brings state-of-the-art machine learning to your data lake. In this session, we show how you can train ML models and how you can take advantage of the capabilities offered by Delta Lake using Amazon SageMaker Studio.

Quick Links
Vedant Jain: https://www.linkedin.com/in/vedantjain/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
GitHub: https://github.com/delta-io
Join Google Groups: https://groups.google.com/forum/#!forum/delta-users

As a follow up to our session "Why did we migrate to a Data Lakehouse on Delta Lake for T-Mobile Data Science and Analytics Team", Robert Thompson and Geoff Freeman, Members of Technical Staff at T-Mobile continue their in-person discussion with Denny Lee on how their data lakehouse improves their data science and data analytics efforts.

Quick Links
Blog: https://delta.io/blog/2022-09-14-why-migrate-lakehouse-delta-lake-tmo-dsna/
Join us on Slack: https://go.delta.io/slack
Join the Google Group: https://groups.google.com/forum/#!forum/delta-users

In this session, Yeshwanth Vijaykumar, Senior Engineering Manager and Architect at Adobe and our host Denny Lee will discuss how the data lake house architecture at Adobe Experience Platform combines with the Real-time Customer Profile architecture to increase our Apache Spark Batch workload throughputs and reduce costs while maintaining functionality with Delta Lake.

Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Yeshwanth Vijaykumar: https://www.linkedin.com/in/yeshwanth-vijayakumar-75599431/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
Join the Google Group: https://groups.google.com/forum/#!forum/delta-users

In this D3L2 episode, we chat with Robert Kossendey, Tech Lead at Claimsforce on their journey from unifying data lake and data warehouse. As Robert’s team builds and expands, they chose Delta Lake and AWS Athena as the foundation for their lakehouse.

Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Robert Kossendey: https://www.linkedin.com/in/robert-kossendey-303b0019a/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
Join the Google Group: https://groups.google.com/forum/#!forum/delta-users

In this D3L2 episode, Denny Lee sits down in person with R. Tyler Croy, Delta Lake maintainer and Director Of Platform Engineering at Scribd, about the creation and inception of Delta Rust and how Rust and Python have become the backbone and frontend of data engineering, data pipelines, and data science.

Learn more about Delta Lake: https://delta.io/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
Delta Rust on GitHub: https://github.com/delta-io/delta-rs

In this D3L2 episode, we sit down with Christina Taylor, data engineer at Carvana, Bread Finance, and Walt Disney Company to discuss her path from data warehousing to the lakehouse. In the process, she led her teams to an open data lake that unifies batch and streaming workload with Delta Lake that decouples data storage from proprietary formats, dramatically reducing data extraction costs.

For this next session of D3L2, we are happy to have a conversation with QP Hou who led the genesis of the Delta Rust project. How did the Delta Rust project start? Why build an open-source data engineering project using Rust and Delta Lake? Learn more about this popular Delta project with QP Hou.

Learn more about Delta Lake: https://delta.io/
QP Hou: https://www.linkedin.com/in/qingpinghou/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
Delta Rust on GitHub: https://github.com/delta-io/delta-rs

Airbyte is an open-source data integration platform that syncs data from applications, APIs & databases to data warehouses, lakes, and other destinations. Delta Lake is an open-source storage framework that enables building a Lakehouse architecture with compute engines including Spark, PrestoDB, Flink, Trino, and Hive and APIs for Scala, Java, Rust, Ruby, and Python. Together they can help address many data integration issues.

In this session, Simon Späti from Airbyte and Denny Lee, Delta Lake maintainer from Databricks discuss data integration with Airbyte and Delta Lake. From ELT vs. ETL to normalization of data to orchestration, we will discuss the complexities and potential solutions to simplify your data integration.

Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Simon Späti: https://www.linkedin.com/in/sspaeti/
Denny Lee: https://www.linkedin.com/in/dennyglee/
Join us on Slack: https://go.delta.io/slack
Delta Lake Releases: https://github.com/delta-io/delta/releases

The refurbished consumer electronics market is growing significantly as an alternative to purchasing new devices. With over 6 million customers, Back Market is the leading dedicated renewed tech marketplace bringing high-quality professionally refurbished electronic devices and appliances, including smartphones, laptops, gaming consoles, and more. The key to ensuring they meet the needs of each customer and seller is data, but as analytical workloads rose, so did the need to consume their data in a rapid, efficient and secure manner.

To enable this, Florian Valeye (a Delta Lake Committer) and the engineering team at Back Market have been important contributors to the creation of the Delta Rust API and associated Python bindings to enable low-latency queries of delta table without having to spin up a Spark cluster. They've also been actively involved in Delta Lake community office hours, contributors to the AWS Labs Athena Federation, reviewing code, and even our partnerships with Google BigQuery.

Quick Links
Read Our Newest Blog Post: https://delta.io/blog
Denny Lee: https://www.linkedin.com/in/dennyglee/
Florian Valeye: https://www.linkedin.com/in/florianvaleye/
Join us on Slack: https://go.delta.io/slack
Delta Lake Releases: https://github.com/delta-io/delta/releases

T-Mobile’s mission to build the nation’s best 5G network drastically increased the number of monthly network projects planned and directly impacted the enterprise procurement and supply chain organizations. Like many enterprises, data at T-Mobile was spread between disparate, unintegrated and complex systems.

In this session, we will discuss the how and why we migrated from databases and data lakes to a data lakehouse on Delta Lake. Our lakehouse architecture allows reading and writing of data without blocking and scales out linearly. Business partners can easily adopt advanced analytics and derive new insights. These new insights promote innovation across disparate workstreams and solidify the decentralized approach to analytics taken by T-Mobile.

Quick links:
https://delta.io/
https://go.delta.io/slack
https://github.com/delta-io/delta/releases
https://groups.google.com/g/delta-users

In this session on August 30th, 2022, Ryan Harris, Principal Cybersecurity Engineer at HSBC, follows up on his Data+AI Summit 2022 session Accidentally Building a Petabyte-Scale Cybersecurity Data Mesh in Azure With Delta Lake at HSBC with Denny Lee for this fun ask-us-anything technical session.

We can dive into the infrastructure and architecture employed, ranging from the landing zone concepts, secure access workstations, data lake structure, and isolated data ingestion, to the enterprise integration layer. Ask your questions on how to build a flexible, secure, self-service environment that is unlocking your team’s data capabilities.

Quick links:
https://delta.io/
https://go.delta.io/slack
https://github.com/delta-io/delta/releases/tag/v2.0.0
https://groups.google.com/g/delta-users