A background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths.

Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking Queries

DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.

DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg

ALP and FSST target Parquet's worst cases: floats and high-cardinality strings. How they work and what they change for Iceberg tables.

FSST and ALP: The Two Encodings Fixing Parquet's Weakest Compression Cases

High-throughput Iceberg branch merges need conflict detection and automation. How to reconcile concurrent writes without stalling pipelines.

High-Throughput Branch Merging: Automating Concurrency and Conflict Resolution in Multi-Branch Iceberg Pipelines

Polaris can catalog Iceberg tables across AWS, Azure, and GCP. Four topologies, credential vending, and the tradeoffs of each design.

Multi-Cloud REST Catalog Topologies: Running Apache Polaris Across AWS, Azure, and GCP

Iceberg v4 is moving manifests from Avro to Parquet so planners can read only the stats they need. Why the metadata layer is going columnar.

Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar

Land small Iceberg micro-batches with PyIceberg and DuckDB in a serverless function. Commits, concurrency, and why Spark is the wrong default.

Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster

Move a legacy warehouse to Iceberg without downtime by virtualizing first. Consumer cutover, parity checks, and background copy without double-ETL.

Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime

Trace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives.

The Hidden Cost of Tiny Iceberg Commits

Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which.

Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026

Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.

Iceberg Is Becoming a Library, Not Just a Table Format

Rust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine.

Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations Matter

One suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it.

The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should Pass

Remote scan planning moves Iceberg file selection into the catalog. What that changes for engines, governance, and operational cost beyond query speed.

Iceberg REST Remote Scan Planning Changes More Than Query Performance

Iceberg row lineage gives rows a durable identity across rewrites. Why CDC pipelines and AI workloads will eventually depend on it.

Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On

Iceberg v4's adaptive metadata tree, explained from first principles: why commits rewrite too much today and how the tree makes change cheaper.

Iceberg v4's Adaptive Metadata Tree, Explained From First Principles

Iceberg v4 is really about making the cost of a change proportional to the change. The principle, the current tax, and what the redesign pays down.

Why Iceberg v4 Is Really About Making the Cost of Change Proportional to the Change

A mechanics walkthrough of Iceberg REST scan planning: client-side planning, remote endpoints, pagination, and where engine support stands in 2026.

The Catalog Can Now Plan Your Iceberg Query: Inside REST Scan Planning

Can seven Iceberg REST catalogs run the same DuckDB script? What the protocol makes portable, what still differs, and a test matrix you can rerun.

Can Seven Different Iceberg REST Catalogs Really Run the Same DuckDB Code?

Iceberg Variant stores JSON as navigable binary with shredding for columnar filters. Why flattening wide tables is no longer the only performance path.

Stop Flattening Your JSON: How Iceberg Variant Changes Semi-Structured Analytics

What happens when two engines write the same Iceberg table at once: snapshot isolation, optimistic commits, conflict detection, and when retries fail.

What Actually Happens When Two Engines Write the Same Iceberg Table at Once?

Variant shredding turns messy JSON into Parquet columns with statistics. How the layout works, how readers reassemble values, and why some queries prune.

Variant Shredding Explained: How Iceberg Gets Columnar Performance From Messy JSON

Managed and external Iceberg tables mean different things on every platform. Five ownership dimensions and a translation method for vendor vocabulary.

Who Actually Owns an Iceberg Table? Managed, External, and the New Vocabulary of Lakehouse Control

Apache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay.

Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming Ingest

The decoupled data lakehouse: multi-engine freedom with open REST catalogs, credential vending, and an estate that outlives its tools.

The Decoupled Data Lakehouse: Multi-Engine Freedom with Open REST Catalogs

Build lightweight serverless ingestion to Apache Iceberg with PyIceberg and DuckDB, running small feeds in functions that bill for seconds.

Building Lightweight Serverless Ingestion to Apache Iceberg with PyIceberg and DuckDB

Write-Audit-Publish with Apache Iceberg branches brings CI/CD to your data: stage, audit, then fast-forward to main so consumers never see unvalidated bytes.

Write-Audit-Publish with Apache Iceberg Branches: CI/CD for Your Data

A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then materializing what earns a migration.

Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime

Autonomous table optimization when query workloads stop being predictable: observing file layout and query patterns, scoring compaction work, adaptive sort order, and cost discipline.

Autonomous Table Optimization When Your Query Workload Stops Being Predictable

Surviving optimistic commit collisions when hundreds of agents write to Iceberg: which conflicts are real, commit buffers, partitioning, and the patterns that prevent commit storms.

Surviving Optimistic Commit Collisions When Hundreds of Agents Write to Iceberg

How Iceberg V3 deletion vectors replaced accumulating positional delete files and made merge-on-read viable for streaming and CDC tables.

How Iceberg V3 Deletion Vectors Fixed Merge-on-Read for Streaming Tables

How Iceberg V3's Variant type and Parquet shredding turn JSON columns into prunable typed columns, with real benchmark tradeoffs and a migration path.

How Iceberg V3 Variant Shredding Changed Semi-Structured Data on S3 Tables

A field guide to the Apache Iceberg V4 proposals: adaptive metadata trees, single-file commits, typed statistics, column families, and what is safe to build on today.

Reading the Apache Iceberg V4 Proposals Before They Land

How the Iceberg REST catalog became the lakehouse control plane: multi-table atomic commits, credential vending, capability negotiation, and what still breaks.

How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane

What zero-copy data sharing actually does across Salesforce, Snowflake, and Databricks: query federation, file federation, catalog federation, and when copying still wins.

What Zero-Copy Data Sharing Actually Does Between Salesforce, Snowflake, and Databricks

Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.

Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.

Designing Batch Pipelines That Write Well Into Apache Iceberg

How AWS, Google Cloud, and Microsoft Azure actually support Apache Iceberg: storage, catalogs, maintenance, governance, and interoperability, layer by layer.

Apache Iceberg Support Across the Major Hyperscalers

A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud, and the operational realities that surprise teams.

Running an Apache Iceberg Lakehouse With No Internet Connection

Autonomous materialized view management replaces quarterly review meetings with workload-driven scoring, and it's essential when AI agents generate unpredictable query patterns.

When the Query Optimizer Starts Managing Its Own Materializations

Equality deletes made streaming upserts into Iceberg practical at the cost of read performance. V4 proposes retiring them in favor of deletion vectors with an async conversion path.

Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It

Fluss puts a columnar, indexed hot tier between Kafka and Iceberg. Here's what it changes structurally, what Kafka still does better, and how to benchmark the comparison yourself.

Apache Fluss and Kafka Solve Different Problems in an Iceberg Pipeline

A lakehouse cannot serve sub-second queries over seconds-old data. A hot tier in front solves it, with consequences for consistency, governance, and operational surface.

Serving Sub-Second Queries Over an Iceberg Lakehouse With a Hot Tier

Commit conflicts multiply with writer count, and AI agents introduce unpredictable write patterns. Here's how to diagnose, tune, and architect around Iceberg's optimistic concurrency.

Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table

Jackson 3 changes everything: package names, unchecked exceptions, flipped defaults. Here's what breaks, why the engines are fine and your service isn't, and how to migrate safely.

The Jackson 3 Problem in Apache Iceberg, and What It Means for Your Code

The catalog is the right attachment point for AI agents working against a lakehouse. Here's how to wire the official Polaris MCP Server and add the read path it deliberately leaves out.

Wiring an AI Agent to Apache Polaris with the Model Context Protocol

Catalog federation gives you one authorization model and one audit point across regions. Here's what it solves, what it doesn't, and how to build a topology you can actually govern.

Governing Iceberg Tables Across Regions Without Three Sets of Permissions

Federate first so analytics work now, migrate what benefits from migrating, and leave the rest where it is indefinitely. Here's how pushdown and view layers make it work.

Federating Oracle With an Open Lakehouse Instead of Migrating It

Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's format version model.

The Parquet Versioning Problem, and Why Iceberg Cares About It

A large share of production transformations fit comfortably on one machine. PyIceberg, DuckDB, and branch isolation give you a production path that debugs in an IDE.

Building Iceberg Pipelines in Python Without Standing Up Spark

An in-depth exploration of iceberg v3 deletion vectors and faster dml

Iceberg v3 Deletion Vectors and Faster DML

An in-depth exploration of iceberg variant type for ai json data

Iceberg Variant Type for AI JSON Data

An in-depth exploration of iceberg concurrency for ai agent writes

Iceberg Concurrency for AI Agent Writes

An in-depth exploration of rest catalog credential vending for iceberg

REST Catalog Credential Vending for Iceberg

An in-depth exploration of rest catalog v2: fixing iceberg protocol debt

REST Catalog V2: Fixing Iceberg Protocol Debt

High-Performance Columnar Transfers: Combining Apache Arrow Flight and Iceberg REST Catalogs

Conversational AI on Managed Iceberg: Exposing Amazon S3 Tables through MCP

Decoupled Catalogs vs. Managed Tables: Architectural Freedom in the Age of Table Format Convergence

Implementing Positional Deletes in Iceberg v3: Streamlining Merge-on-Read for Fast-Inbound Event Lakes

Mapping the Variant Type in Iceberg v3: Standardizing Semi-Structured AI JSON Payloads

What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026.

The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter

Multi-Engine Catalog Federation with Apache Polaris: Syncing Google Cloud, AWS, and Azure Metadata

Designing Private, Air-Gapped Data Lakehouses: Scaling Iceberg in Highly Secure, On-Premises Clouds

Every path for streaming data into Iceberg in 2026 — Flink, Spark, Kafka Connect, broker-native, managed pipelines — with honest latency numbers and sub-second hybrid architectures.

The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

Lakehouse Table Formats in 2026: Iceberg, Delta Lake, Hudi, Paimon, and DuckLake, How They Work, Where They Stand, and Where They're Going

How Iceberg optimistic concurrency control, partition-level isolation, and idempotency keys enable safe concurrent writes from multiple AI agents to the same lakehouse tables.

Agentic Lakehouse Concurrency and Isolation

The four-layer architecture of the agentic lakehouse: object storage, Apache Iceberg table format, Apache Polaris catalog, and the semantic/agent layer. How each layer provides guarantees for AI agent access.

Anatomy of an Agentic Lakehouse

A deep technical breakdown of Apache Iceberg v4's proposed architecture: adaptive metadata trees, one-file commits, relative paths, column families, and what the Delta 5.0 convergence actually means for your data platform.

Apache Iceberg v4 Roadmap and Metadata Trees

Lakehouse context layers bridge the gap between raw Iceberg tables and AI agents that need business meaning. Atlan and Snowflake Horizon each take different approaches to the same problem.

Lakehouse Context Layers with Atlan and Iceberg v3

How goal-directed analytics agents decompose business questions into sub-tasks, execute action loops over Apache Iceberg tables, and use the lakehouse as both a data source and a state store for agent action logs.

Goal-Directed Analytics Agents on Apache Iceberg

Iceberg REST catalog remote signing provides per-file pre-signed URL access for regulated datasets. How it differs from credential vending, audit trail capabilities, and Snowflake implementation for PII/compliance workloads.

Iceberg Remote Signing for Regulated Datasets

Apache Iceberg v3 deletion vectors replace positional delete files with binary bitmaps in Puffin files, delivering up to 10x faster DML on Snowflake. Deep dive into architecture, benchmarks, and migration.

Apache Iceberg v3 Deletion Vectors on Snowflake

Iceberg v3 row lineage adds _row_id and _last_updated_sequence_number to every table, enabling native change data capture without Debezium or Kafka. Technical deep dive with Snowflake and Databricks examples.

CDC Without Complexity Using Iceberg v3 Row Lineage

Iceberg views standardize SQL view definitions across engines, enabling view federation across Polaris, Nessie, and Gravitino catalogs. How Snowflake Horizon, Databricks Unity Catalog, and open source catalogs handle portable SQL.

The 2026 Guide to Iceberg View Federation

Iceberg REST catalog credential vending issues short-lived, table-scoped storage tokens for S3, ADLS, and GCS. How Polaris, Snowflake Horizon, Databricks Unity Catalog, and Dremio implement credential vending for secure lakehouse storage access.

REST Catalog Credential Vending for Lakehouse Security

How OAuth 2.0 token exchange, OAuth 2.1 device authorization grant, credential vending, and fine-grained access control secure AI agent identities in the Iceberg lakehouse.

Securing Agent Identities in the Lakehouse

Snowflake Horizon Catalog enables bidirectional Iceberg writes from external engines like Spark and Trino, powered by Apache Polaris. Deep dive into REST Scan Plan API, governance, and Snowflake Summit 2026 announcements.

Bidirectional Iceberg Writes with Horizon Catalog

Zero-copy mirroring gives teams a safer migration path because they can expose a lakehouse surface before they duplicate every byte or rewrite every workload.

Zero-Copy Mirroring for Modern Lakehouse Migration

Apache Iceberg v3 adds deletion vectors, VARIANT type, row lineage, and table encryption. Here's what changed and how to plan your upgrade.

Mastering Apache Iceberg v3: What's New and How to Plan Your Upgrade

Apache Iceberg gives regulated enterprises data sovereignty with hybrid-cloud deployments. Learn how open catalogs and Iceberg handle compliance requirements.

How Apache Iceberg Resolves the Hybrid-Cloud Challenge in Heavily Regulated Markets

Apache Iceberg doesn't enforce row and column security on its own. Learn how catalog-level governance with Apache Polaris and Dremio delivers enterprise FGAC.

Securing Apache Iceberg Tables with Fine-Grained Row and Column Level Access Control

Iceberg snapshot bloat silently kills query performance. Learn the best practices for snapshot expiration, manifest cleanup, and automated table maintenance.

Designing an Immutable Data Lakehouse: Best Practices for Iceberg Snapshot Expiration

Learn how Apache Iceberg decouples storage from compute to cut data platform costs, enable multi-engine routing, and eliminate proprietary lock-in.

Decoupling Storage and Compute in Apache Iceberg: A Deep Dive into Cost Optimization

Choosing an Apache Iceberg catalog? Compare open-source Apache Polaris, open Unity Catalog, and managed cloud REST control planes to unify your lakehouse.

Choosing the Right Iceberg Control Plane: Polaris vs. Unity Catalog vs. Cloud REST

DuckDB 1.4 LTS and Polars streaming engine now both support reading and writing Apache Iceberg tables. Learn how to use them for lakehouse analytics in 2025.

Using DuckDB and Polars to Query Iceberg Tables

Apache Iceberg V3 brings deletion vectors and row lineage that reshape CDC pipeline design. Learn what these features mean for your streaming data architecture.

What Iceberg V3 Advances Mean for CDC Pipelines

LanceDB and Apache Iceberg serve complementary roles in a multimodal AI lakehouse. Learn when to use Lance for embeddings and random access, and Iceberg for structured metadata and SQL analytics.

Lance and Iceberg for Multimodal AI Data

Apache Paimon uses LSM-Tree storage for native CDC upserts without restart. Learn when Paimon outperforms Iceberg for high-churn mutable streaming workloads.

When Paimon Beats Iceberg for Mutable Streams

Learn how to build a real-time lakehouse with Apache Flink 2.1 and the Dynamic Iceberg Sink, covering schema evolution, exactly-once delivery, and compaction.

Real-Time Lakehouse Patterns with Apache Flink and Iceberg

Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.

The Endgame  – Building an Autonomous Optimization Pipeline for Apache Iceberg

Learn how to scale Apache Iceberg table optimizations across large datasets using parallelism, checkpointing, and fail recovery to ensure reliability and performance.

Managing Large-Scale Optimizations  – Parallelism, Checkpointing, and Fail Recovery

Partition evolution in Apache Iceberg is a powerful feature, but if not managed carefully, it can introduce fragmentation and impact compaction performance. Learn how to handle it effectively.

Hidden Pitfalls  – Compaction and Partition Evolution in Apache Iceberg

Discover how to use Apache Iceberg's metadata tables to proactively detect small files, bloated manifests, and table fragmentation - so you can trigger compaction only when it's needed.

Using Iceberg Metadata Tables to Determine When Compaction Is Needed

Learn how to design an effective schedule for compaction and snapshot expiration in Apache Iceberg to balance cost, performance, and data freshness.

Designing the Ideal Cadence for Compaction and Snapshot Expiration

Learn how to prevent and clean up metadata bloat in Apache Iceberg by expiring snapshots and rewriting manifests for better performance and manageability.

Avoiding Metadata Bloat with Snapshot Expiration and Rewriting Manifests

Improve query performance in Apache Iceberg by organizing your data layout with sorting and Z-order clustering. Learn how to reduce scan cost and improve filter effectiveness.

Smarter Data Layout  – Sorting and Clustering Iceberg Tables

Learn how to design fast, incremental compaction strategies in Apache Iceberg to support high-throughput streaming pipelines without disrupting freshness or performance.

Optimizing Compaction for Streaming Workloads in Apache Iceberg

Learn how standard compaction works in Apache Iceberg and why bin packing your data files is essential for maintaining query performance and cost efficiency.

The Basics of Compaction  – Bin Packing Your Data for Efficiency

Learn how Apache Iceberg tables can degrade over time without optimization and what issues this causes for performance, cost, and governance.

Post Search