Depending on a single vendor for all data needs creates significant bottlenecks: vendor lock-in, unoptimized compute engines for specific workloads, and slow developer feedback loops. The next phase of data architecture demands a shift toward decoupled layers and flexible compute.
In this session, we will explore the future of data transformation built around open table formats and multi-engine processing. We will discuss how open formats like Apache Iceberg can turn Data Mesh theories into reality, while addressing current catalog challenges and where the ecosystem is heading next.
Finally, we will focus on enhancing the developer experience through local compute. We will run a practical demonstration where a Databricks/BigQuery dbt project is executed locally on DuckDB using automated SQL dialect translation. The result: zero cloud compute costs during development and lightning-fast iteration cycles for your data engineering teams.
