Learn how Databricks Predictive Optimization, AWS S3 Tables, and Iceberg native actions automate compaction and snapshot management before small files degrade performance.

Automating Table Maintenance Before Small Files Accumulate

Data clean rooms enable secure multi-party analytics without sharing raw data. Learn how Databricks Clean Rooms, AWS Clean Rooms, and BigQuery differential privacy work.

Clean Rooms for Privacy-Preserving Analytics

Apache DataFusion, Velox, and Substrait form the foundation of modern composable query engine stacks. Learn how these components fit together and when to use each.

Building Composable Query Engines with Rust Runtimes

dbt Fusion entered public beta in May 2025 with a Rust-powered runtime that changes how analytics engineers develop, validate, and deploy SQL models. Here's what changed.

How dbt Fusion Reshapes Analytics Engineering

The FOCUS 1.3 specification and native warehouse cost views make real-time cost attribution practical. Learn how to build a FinOps pipeline for Snowflake, BigQuery, and multi-cloud environments.

FinOps for Data Warehouses with Open Billing Data

Kafka 4.0 removes ZooKeeper and ships KRaft and KIP-848 by default. Learn what those changes mean for platform operations, upgrades, and client configurations.

Kafka 4.0 Changes Streaming Platform Operations

OpenLineage provides a standard API for collecting pipeline lineage across Airflow, Spark, Flink, and dbt. Learn how it powers blast radius analysis and incident triage.

OpenLineage as the Spine of Data Observability

Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn when to transition to Dremio MPP.

Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail

Apache Iceberg 1.11.0 delivers manifest list encryption, the new pluggable File Format API, credential lifecycle refreshes, and Spark/Flink improvements.

An In-Depth Overview of the Apache Iceberg 1.11.0 Release

What Are Table Formats and Why Were They Needed?

The Metadata Structure of Modern Table Formats

Performance and Apache Iceberg's Metadata

Partition Evolution: Change Your Partitioning Without Rewriting Data

Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans

Writing to an Apache Iceberg Table: How Commits and ACID Actually Work

What Are Lakehouse Catalogs? The Role of Catalogs in Apache Iceberg

When Catalogs Are Embedded in Storage

How Data Lake Table Storage Degrades Over Time

Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup

Apache Iceberg Metadata Tables: Querying the Internals

Using Apache Iceberg with Python and MPP Query Engines

Approaches to Streaming Data into Apache Iceberg Tables

Hands-On with Apache Iceberg Using Dremio Cloud

Migrating to Apache Iceberg: Strategies for Every Source System

How Query Engines Think: The Tradeoffs Behind Every Data System

Row vs. Column: How Storage Layout Shapes Everything

How Databases Organize Data on Disk: Pages, Blocks, and File Formats

B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum

Inside the Query Optimizer: How Engines Pick a Plan

Volcano, Vectorized, Compiled: How Engines Execute Your Query

Buffer Pools, Caches, and the Memory Hierarchy

Partitioning, Sharding, and Data Distribution Strategies

Hash, Sort-Merge, Broadcast: How Distributed Joins Work

Concurrency, Isolation, and MVCC: How Engines Handle Contention

Learn about the Apache Software Foundation's history, governance model, and why it matters for open source data lakehouse technologies like Iceberg, Parquet, and Arrow.

Apache Software Foundation: History, Purpose, and Process

A deep dive into Apache Parquet's columnar storage format, encoding strategies, and why it's the backbone of modern data lakehouse architectures.

What is Apache Parquet? Columns, Encoding, and Performance

Understand how Apache Iceberg brings ACID transactions, schema evolution, and time travel to object storage, enabling the modern open data lakehouse.

What is Apache Iceberg? The Table Format Revolution

Apache Polaris is the open-source Iceberg REST catalog that enables multi-engine interoperability across your entire lakehouse ecosystem.

What is Apache Polaris? Unifying the Iceberg Ecosystem

Apache Arrow provides a standardized in-memory columnar format that eliminates serialization overhead between query engines, eliminating the cost of data movement.

What is Apache Arrow? Erasing the Serialization Tax

Learn how to combine Apache Iceberg, Apache Parquet, Apache Arrow, and Apache Polaris into a complete, vendor-neutral open data lakehouse.

Assembling the Apache Lakehouse: The Modular Architecture

Explore how the Apache Lakehouse stack enables agentic AI systems to autonomously query, reason over, and act on large-scale structured data.

Agentic Analytics on the Apache Lakehouse

How to Think Like a Data Engineer

How to Design Reliable Data Pipelines

Data Quality Is a Pipeline Problem, Not a Dashboard Problem

Idempotent Pipelines: Build Once, Run Safely Forever

Schema Evolution Without Breaking Consumers

Batch vs. Streaming: Choose the Right Processing Model

Partition and Organize Data for Performance

Testing Data Pipelines: What to Validate and When

Pipeline Observability: Know When Things Break

Data Engineering Best Practices: The Complete Checklist

An updated introduction to Apache Iceberg

A 2026 Introduction to Apache Iceberg

A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.

2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

Discover DremioFrame and IceFrame, two new Python libraries that simplify working with Dremio and Apache Iceberg. Learn how these tools streamline data management and enhance productivity for modern data teams.

dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg

Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform. Learn how to leverage its intuitive API to streamline your data engineering workflows.

Introducing dremioframe - A Pythonic DataFrame Interface for Dremio

Walkthrough with the new trial of the Dremio Cloud Platform

Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.

2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.

An Exploration of the Commercial Iceberg Catalog Ecosystem

Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.

Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.

Intro to Apache Iceberg with Apache Polaris and Apache Spark

What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats

The State of Apache Iceberg v4 - October 2025 Edition

Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.

The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

Unlocking the Power of Agentic AI with Apache Iceberg and Dremio

Guide on How to Be Part of the Lakehouse Community

How to Discover or Organize Lakehouse & Apache Iceberg Meetups

Introduction to the terms in data engineering

Introduction to Data Engineering Concepts | What is Data Engineering?

Introduction to Data Engineering Concepts | Understanding Data Sources and Ingestion

Introduction to Data Engineering Concepts | ETL vs ELT – Understanding Data Pipelines

Introduction to Data Engineering Concepts | Batch Processing Fundamentals

Introduction to Data Engineering Concepts | Streaming Data Fundamentals

Introduction to Data Engineering Concepts | Data Modeling Basics

Introduction to Data Engineering Concepts | Data Warehousing Fundamentals

Introduction to Data Engineering Concepts | Data Lakes Explained

Introduction to Data Engineering Concepts | Storage Formats and Compression

Introduction to Data Engineering Concepts | Data Quality and Validation

Introduction to Data Engineering Concepts | Metadata, Lineage, and Governance

Introduction to Data Engineering Concepts | Scheduling and Workflow Orchestration

Introduction to Data Engineering Concepts | Building Scalable Pipelines

Introduction to Data Engineering Concepts | DevOps for Data Engineering

Introduction to Data Engineering Concepts | Cloud Data Platforms and the Modern Stack

Introduction to Data Engineering Concepts | Data Lakehouse Architecture Explained

Introduction to Data Engineering Concepts | Apache Iceberg, Arrow, and Polaris

Introduction to Data Engineering Concepts | The Power of Dremio in the Modern Lakehouse

Why retrieval-augmented generation systems fail in enterprises—and what to do about it.

RAG Isn’t a Modeling Problem. It’s a Data Engineering Problem.

A personal story of how I built Pangolin Catalog over a holiday break using an AI-powered IDE.

Building Pangolin - My Holiday Break, an AI IDE, and a Lakehouse Catalog for the Curious

Introduction to ANSI SQL - Understanding the Syntax and Concepts

Understanding the Data Vault Style of Data Warehouse Modeling

Introduction to Data Vault Modeling

Convert XLS/CSV Files without having to write python

No Code - Convert XLS/CSV files into Parquet with Dremio

An Approach to Architecting a Lower Cost, Fast and Self-Service Data Lakehouse

Handling Cross-Origin Cookies with ExpressJS

Creating a Local Data Lakehouse using Spark/Minio/Dremio/Nessie

Project Nessie: A Look in the Depths

Overview of File Encryption Algorithms for Everyone

Parquet File Compression for Everyone (zstd, brotli, lz4, gzip, snappy)

Dremio and Modern Data Architecture: Data Lakes, Data Lakehouses and Data Mesh

What is Nessie and Why as a Data Engineer or Architect you should care?

Resources for Learning more about Catalog level versioning with Project Nessie & Dremio Arctic (Rollbacks, Branching, Tagging and Multi-Table Txns)

Understanding the Cutting Edge of Data Engineering

Post Search