Databricks, a diversified data infrastructure and AI company, launched Databricks LakeFlow, a new service to integrate and simplify the various aspects of data engineering, including data ingestion, transformation, and orchestration, at the Data + AI Summit 2024.

Databricks LakeFlow includes key features, such as LakeFlow Connect, which facilitates simple and scalable data ingestion from diverse sources; LakeFlow Pipelines, which streamlines and automates real-time data pipelines seamlessly; and LakeFlow Jobs, which orchestrates workflows across the entire Data Intelligence Platform.

The primary advantage of employing the LakeFlow service is the integration and simplification of all aspects of data engineering through a single platform. It also enhances data governance and enables highly efficient execution, addressing concerns over low data quality, reliability issues, high costs, and work backlogs.

Furthermore, Databricks revealed that it is open-sourcing Unity Catalog, a unified governance solution for managing data assets within the Databricks ecosystem. Unity Catalog is a central repository for various data assets, such as files, tables, views, and machine-learning models, and offers tools for data governance, access control, auditing, and lineage tracking.

Analyst QuickTake: Databricks's announcement closely follows its main competitor, Snowflake's summit , which occurred earlier this month. During the summit, Snowflake announced collaborations with multiple AI companies to enhance its capabilities and remain competitive in the data and AI infrastructure space.