What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not.

Will AI Replace Data Engineers?

Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.

Iceberg Is Becoming a Library, Not Just a Table Format

A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then materializing what earns a migration.

Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime

Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.

Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.

Designing Batch Pipelines That Write Well Into Apache Iceberg

A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud, and the operational realities that surprise teams.

Running an Apache Iceberg Lakehouse With No Internet Connection

Equality deletes made streaming upserts into Iceberg practical at the cost of read performance. V4 proposes retiring them in favor of deletion vectors with an async conversion path.

Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It

Commit conflicts multiply with writer count, and AI agents introduce unpredictable write patterns. Here's how to diagnose, tune, and architect around Iceberg's optimistic concurrency.

Surviving Commit Conflicts When Dozens of Writers Hit the Same Iceberg Table

Parquet files have a version field that doesn't reliably signal feature requirements. A new versioning discipline is coming, borrowing from Iceberg's format version model.

The Parquet Versioning Problem, and Why Iceberg Cares About It

A survey of the Apache Iceberg market in July 2026: the state of the specification, platform support, the acquisition wave, the catalog contest, and how to evaluate real Iceberg support.

The Apache Iceberg Market in the Middle of 2026

Databricks, Snowflake, and SAP are closing the gap between operational databases and analytical platforms through acquisition, betting on different layers of the same five-part architecture.

Three Vendors Are Rebuilding the Path From Transaction to Agent

An in-depth exploration of active analytics loops for ai action agents

Active Analytics Loops for AI Action Agents

An in-depth exploration of hybrid lakehouse design for regulated markets

Hybrid Lakehouse Design for Regulated Markets

An in-depth exploration of iceberg concurrency for ai agent writes

Iceberg Concurrency for AI Agent Writes

An in-depth exploration of rest catalog v2: fixing iceberg protocol debt

REST Catalog V2: Fixing Iceberg Protocol Debt

How to use AI agent harnesses for data engineering without losing determinism, reproducibility, and trust in your data pipelines and analytics.

Deterministic Data Engineering With AI Harnesses: Using Claude Code, Codex, Antigravity, and OpenCode for Data Work You Can Actually Trust

What the Iceberg v4 dev list tells us about adaptive metadata trees, single-file commits, column updates, and the format's next chapter in mid-2026.

The State of Apache Iceberg v4 in July 2026: What the Dev List Tells Us About the Format's Next Chapter

The agentic AI protocol stack is solidifying in 2026 — MCP for tools, A2A for agents, WebMCP for the web, OSI for semantics, payments, identity, and security.

The State of Agentic AI Standards in 2026: MCP, A2A, WebMCP, OSI, and the Protocol Stack Taking Shape

Apache Arrow at 10 — ADBC, Flight SQL, nanoarrow, the AI reinterpretation, and how an in-memory standard eliminated the copy tax across the data stack.

The State of Apache Arrow in 2026: Ten Years In, the Invisible Standard Is Everywhere

Apache Parquet in 2026 — variant types, geospatial, ALP encoding, footer redesign, the versioning debate, and how the decade-old format is renovating for AI workloads.

The State of Apache Parquet in 2026: The Quiet Format Enters Its Loudest Decade

Apache Polaris as a TLP — federation, credential vending, semantic layers, lineage, and how the open catalog became the governance plane of the multi-engine lakehouse.

The State of Apache Polaris in July 2026: From Incubating Catalog to the Governance Layer of the Open Lakehouse

Every path for streaming data into Iceberg in 2026 — Flink, Spark, Kafka Connect, broker-native, managed pipelines — with honest latency numbers and sub-second hybrid architectures.

The State of Streaming to Apache Iceberg in July 2026: Every Path, Its Latency, and What to Do When Seconds Are Not Fast Enough

A deep dive into the four categories of agentic coding tools dominating 2026: CLI agents, desktop IDEs, 24/7 autonomous agents, and model routers. 40+ tools compared.

The Complete Guide to Agentic Coding Tools in 2026

How Iceberg optimistic concurrency control, partition-level isolation, and idempotency keys enable safe concurrent writes from multiple AI agents to the same lakehouse tables.

Agentic Lakehouse Concurrency and Isolation

The four-layer architecture of the agentic lakehouse: object storage, Apache Iceberg table format, Apache Polaris catalog, and the semantic/agent layer. How each layer provides guarantees for AI agent access.

Anatomy of an Agentic Lakehouse

A deep technical breakdown of Apache Iceberg v4's proposed architecture: adaptive metadata trees, one-file commits, relative paths, column families, and what the Delta 5.0 convergence actually means for your data platform.

Apache Iceberg v4 Roadmap and Metadata Trees

Lakehouse context layers bridge the gap between raw Iceberg tables and AI agents that need business meaning. Atlan and Snowflake Horizon each take different approaches to the same problem.

Lakehouse Context Layers with Atlan and Iceberg v3

ClickHouse has become the leading real-time analytics engine for AI agent workloads, with event-loop architectures that let agents query and act at sub-second speeds.

Real-Time Agentic Analytics with ClickHouse

Metric catalogs define what terms mean. Composable analytics defines how terms combine, transform, and relate. For AI agents, composability is what turns definitions into reasoning.

Composable Analytics Beats Metric Catalogs

How goal-directed analytics agents decompose business questions into sub-tasks, execute action loops over Apache Iceberg tables, and use the lakehouse as both a data source and a state store for agent action logs.

Goal-Directed Analytics Agents on Apache Iceberg

Iceberg REST catalog remote signing provides per-file pre-signed URL access for regulated datasets. How it differs from credential vending, audit trail capabilities, and Snowflake implementation for PII/compliance workloads.

Iceberg Remote Signing for Regulated Datasets

Apache Iceberg v3 deletion vectors replace positional delete files with binary bitmaps in Puffin files, delivering up to 10x faster DML on Snowflake. Deep dive into architecture, benchmarks, and migration.

Apache Iceberg v3 Deletion Vectors on Snowflake

Iceberg v3 row lineage adds _row_id and _last_updated_sequence_number to every table, enabling native change data capture without Debezium or Kafka. Technical deep dive with Snowflake and Databricks examples.

CDC Without Complexity Using Iceberg v3 Row Lineage

Iceberg views standardize SQL view definitions across engines, enabling view federation across Polaris, Nessie, and Gravitino catalogs. How Snowflake Horizon, Databricks Unity Catalog, and open source catalogs handle portable SQL.

The 2026 Guide to Iceberg View Federation

How to build a Model Context Protocol (MCP) server that exposes lakehouse tables and semantic views as AI-accessible tools, with Python implementation patterns and authentication.

Implementing MCP in the Lakehouse

Microsoft Build 2026 revealed an agentic analytics stack built on Fabric IQ, OneLake Iceberg support, and semantic models. The architecture shows how Microsoft competes with open lakehouse platforms.

Microsoft Fabric Build 2026 Agentic Analytics Stack

PyIceberg, IceFrame, and the Iceberg CLI form a complete Python toolchain for Iceberg table management. Each tool targets a different workflow from metadata inspection to data engineering.

Modern Python Tooling for Apache Iceberg

Iceberg REST catalog credential vending issues short-lived, table-scoped storage tokens for S3, ADLS, and GCS. How Polaris, Snowflake Horizon, Databricks Unity Catalog, and Dremio implement credential vending for secure lakehouse storage access.

REST Catalog Credential Vending for Lakehouse Security

Enterprise SaaS procurement in 2026 evaluates how platforms expose data to AI agents. Semantic layers have become a decision criterion alongside dashboards and APIs.

SaaS Buyers Now Inspect Your Semantic Layer

How OAuth 2.0 token exchange, OAuth 2.1 device authorization grant, credential vending, and fine-grained access control secure AI agent identities in the Iceberg lakehouse.

Securing Agent Identities in the Lakehouse

Snowflake Horizon Catalog enables bidirectional Iceberg writes from external engines like Spark and Trino, powered by Apache Polaris. Deep dive into REST Scan Plan API, governance, and Snowflake Summit 2026 announcements.

Bidirectional Iceberg Writes with Horizon Catalog

Zero-copy mirroring gives teams a safer migration path because they can expose a lakehouse surface before they duplicate every byte or rewrite every workload.

Zero-Copy Mirroring for Modern Lakehouse Migration

A comprehensive comparison of AI agent tooling across Dremio, Snowflake, Databricks, Microsoft Fabric, AWS, Google Cloud, ClickHouse, VeloDB, SpiceAI, Bauplan, and Qlik.

Data Platform Native AI Agent Tooling in 2026

Optimize single-node data engineering with DuckDB, DataFusion, Polars, and LakeSail. Compare architectures and learn when to transition to Dremio MPP.

Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail

Apache Iceberg 1.11.0 delivers manifest list encryption, the new pluggable File Format API, credential lifecycle refreshes, and Spark/Flink improvements.

An In-Depth Overview of the Apache Iceberg 1.11.0 Release

What Are Table Formats and Why Were They Needed?

The Metadata Structure of Modern Table Formats

Performance and Apache Iceberg's Metadata

Partition Evolution: Change Your Partitioning Without Rewriting Data

Hidden Partitioning: How Iceberg Eliminates Accidental Full Table Scans

Writing to an Apache Iceberg Table: How Commits and ACID Actually Work

What Are Lakehouse Catalogs? The Role of Catalogs in Apache Iceberg

When Catalogs Are Embedded in Storage

How Data Lake Table Storage Degrades Over Time

Maintaining Apache Iceberg Tables: Compaction, Expiry, and Cleanup

Apache Iceberg Metadata Tables: Querying the Internals

Using Apache Iceberg with Python and MPP Query Engines

Approaches to Streaming Data into Apache Iceberg Tables

Hands-On with Apache Iceberg Using Dremio Cloud

Migrating to Apache Iceberg: Strategies for Every Source System

How Query Engines Think: The Tradeoffs Behind Every Data System

Row vs. Column: How Storage Layout Shapes Everything

How Databases Organize Data on Disk: Pages, Blocks, and File Formats

B-Trees, LSM Trees, and the Indexing Tradeoff Spectrum

Inside the Query Optimizer: How Engines Pick a Plan

Volcano, Vectorized, Compiled: How Engines Execute Your Query

Buffer Pools, Caches, and the Memory Hierarchy

Partitioning, Sharding, and Data Distribution Strategies

Hash, Sort-Merge, Broadcast: How Distributed Joins Work

Concurrency, Isolation, and MVCC: How Engines Handle Contention

Learn about the Apache Software Foundation's history, governance model, and why it matters for open source data lakehouse technologies like Iceberg, Parquet, and Arrow.

Apache Software Foundation: History, Purpose, and Process

A deep dive into Apache Parquet's columnar storage format, encoding strategies, and why it's the backbone of modern data lakehouse architectures.

What is Apache Parquet? Columns, Encoding, and Performance

Understand how Apache Iceberg brings ACID transactions, schema evolution, and time travel to object storage, enabling the modern open data lakehouse.

What is Apache Iceberg? The Table Format Revolution

Apache Polaris is the open-source Iceberg REST catalog that enables multi-engine interoperability across your entire lakehouse ecosystem.

What is Apache Polaris? Unifying the Iceberg Ecosystem

Apache Arrow provides a standardized in-memory columnar format that eliminates serialization overhead between query engines, eliminating the cost of data movement.

What is Apache Arrow? Erasing the Serialization Tax

Learn how to combine Apache Iceberg, Apache Parquet, Apache Arrow, and Apache Polaris into a complete, vendor-neutral open data lakehouse.

Assembling the Apache Lakehouse: The Modular Architecture

Explore how the Apache Lakehouse stack enables agentic AI systems to autonomously query, reason over, and act on large-scale structured data.

Agentic Analytics on the Apache Lakehouse

How to Think Like a Data Engineer

How to Design Reliable Data Pipelines

Data Quality Is a Pipeline Problem, Not a Dashboard Problem

Idempotent Pipelines: Build Once, Run Safely Forever

Schema Evolution Without Breaking Consumers

Batch vs. Streaming: Choose the Right Processing Model

Partition and Organize Data for Performance

Testing Data Pipelines: What to Validate and When

Pipeline Observability: Know When Things Break

Data Engineering Best Practices: The Complete Checklist

An updated introduction to Apache Iceberg

A 2026 Introduction to Apache Iceberg

A look back at key developments in Apache Iceberg, Polaris, Parquet, and Arrow in 2025.

2025 Year in Review Apache Iceberg, Polaris, Parquet, and Arrow

Discover DremioFrame and IceFrame, two new Python libraries that simplify working with Dremio and Apache Iceberg. Learn how these tools streamline data management and enhance productivity for modern data teams.

dremioframe & iceberg - Pythonic interfaces for Dremio and Apache Iceberg

Discover dremioframe, a new Python library that offers a DataFrame-like experience for interacting with Dremio's data lakehouse platform. Learn how to leverage its intuitive API to streamline your data engineering workflows.

Introducing dremioframe - A Pythonic DataFrame Interface for Dremio

Walkthrough with the new trial of the Dremio Cloud Platform

Comprehensive Hands-on Walk Through of Dremio Cloud Next Gen (Hands-on with Free Trial)

A curated guide to mastering Apache Iceberg, data lakehouse architectures, and the emerging field of Agentic AI for data professionals.

2025-2026 Guide to Learning about Apache Iceberg, Data Lakehouse & Agentic AI

Dive into the world of commercial Iceberg catalogs and discover how they enhance data lakehouse architectures for modern data engineering.

An Exploration of the Commercial Iceberg Catalog Ecosystem

Exploring paths to a universal lakehouse catalog that supports multiple data formats and engines, building on Apache Iceberg's success.

Building a Universal Lakehouse Catalog - Beyond Iceberg Tables

Learn how to leverage Apache Iceberg with Apache Polaris and Apache Spark to build scalable and efficient data lakehouses.

Intro to Apache Iceberg with Apache Polaris and Apache Spark

What's Coming in Apache Iceberg v4: A Deep Dive into the Future of Open Table Formats

The State of Apache Iceberg v4 - October 2025 Edition

Understanding Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

The Ultimate Guide to Open Table Formats - Iceberg, Delta Lake, Hudi, Paimon, and DuckLake

What is the Data Lakehouse and the Data Lakehouse Ecosystem? This comprehensive guide covers everything you need to know about the Data Lakehouse architecture, open table formats like Apache Iceberg, Delta Lake, Apache Hudi, and Apache Paimon, and the modern data ecosystem that supports them.

The 2025 & 2026 Ultimate Guide to the Data Lakehouse and the Data Lakehouse Ecosystem

Learn how to automate compaction, snapshot expiration, and layout optimization in Apache Iceberg using metadata-driven triggers and orchestration tools for a self-healing lakehouse.

Post Search