A comparison of Great Expectations, Soda, dbt tests, and anomaly detection, and a layered design that uses each where it fits.

Data Quality Tooling Compared: Great Expectations, Soda, dbt Tests, and Anomaly Detection

The case for generalists owning end-to-end data workflows with agents, the counterargument, and how to make the transition work.

The Data Team of the Agentic Era: Generalists Owning End-to-End Workflows

How dbt incremental materializations map to Iceberg operations, and the configuration, predicates, and maintenance that keep them healthy.

dbt on Iceberg: Incremental Models on Open Tables

Disaster recovery for Iceberg across four tiers: snapshots, object versioning, catalog backup, and cross-region replication.

Disaster Recovery for Iceberg Tables: Replication, Backup, and Restore

How to turn a logical delete on immutable Iceberg into a physical erasure across snapshots, versions, replicas, and downstream copies.

Deleting User Data From an Immutable Lakehouse: GDPR Hard Deletes on Iceberg

How Iceberg v3 geometry and geography types, bounding boxes, and native Parquet types give spatial data first-class standing.

Geospatial Data in Apache Iceberg: Geometry, Geography, and GeoParquet

How field IDs and initial and write defaults let Iceberg change schemas on large tables without rewriting data, at the spec level.

Default Column Values and Field IDs: How Iceberg Schema Evolution Works at the Spec Level

The Iceberg table properties that decide file count, pruning, write amplification, retention, and metadata growth, by workload.

The Iceberg Table Properties That Actually Matter

The Puffin file format inside out, byte by byte, covering Theta sketches for distinct values and deletion vectors.

Inside the Puffin File Format

How Fivetran, Airbyte, dlt, and CDC and streaming tools land well-behaved Apache Iceberg tables, and how to choose and maintain them.

The Lakehouse Ingestion Tool Landscape: Fivetran, Airbyte, dlt, and CDC vs Batch

Local Iceberg development environments: in-process catalogs, a Docker Compose stack with MinIO, and CI configurations that run either.

Local Iceberg Development Environments: Docker, MinIO, and In-Memory Catalogs for CI

How the technical catalog and the metadata platform are converging in 2026, and how to arrange the two layers for a lakehouse.

Metadata Platforms in 2026: DataHub, OpenMetadata, Atlan, and Catalog Convergence

Why moving Iceberg tables between catalogs is a pointer copy, and the protocol that makes a cutover safe for one table or thousands.

Moving Iceberg Tables Between Catalogs Without Rewriting Data

Building an OpenTelemetry data lake on Iceberg: schemas for spans, logs, and metrics, ingestion, query patterns, and retention.

Logs, Traces, and Metrics as Tables: Building an OpenTelemetry Data Lake on Iceberg

Where Airflow 3, Dagster, Prefect, and event-driven triggering stand for lakehouse pipelines in 2026, after the Prefect acquisition of Dagster.

Orchestration in 2026: Airflow 3 vs Dagster vs Prefect vs Event-Driven

What pg_lake, pg_duckdb, and pg_mooncake do at the Iceberg level, and honest thresholds for when Postgres is enough.

Postgres Meets the Lakehouse: pg_lake, pg_duckdb, and When Postgres Is Enough

How Avro, Protobuf, and JSON Schema evolve through a registry, and how that maps to the schema evolution rules of Iceberg.

Schema Registries and Event Schemas: Avro, Protobuf, and JSON Schema on the Way Into the Lakehouse

How the spec-defined bucket transform lets engines skip the shuffle in joins, and how to set it up and keep it engaged in Spark.

Storage-Partitioned Joins and the Bucket Transform

What synthetic data in a lakehouse is for, the generation methods, how to preserve fidelity, and where synthetic tables belong.

Synthetic Data in the Lakehouse: Generation, Governance, and Testing

What the evidence shows about whether AI replaces data engineers, which parts of the job compress, and which parts do not.

Will AI Replace Data Engineers?

A background agent can move cold Iceberg partitions to cheaper tiers without breaking live queries. Heatmaps, path-safe moves, and restore paths.

Agent-Driven Storage Tiering for Apache Iceberg: Moving Cold Data Without Breaking Queries

Agentic lakehouse gateways face prompt injection and exfiltration through query results. A threat model and defenses for the layer in front of data.

Securing the Agentic Lakehouse Gateway: Preventing Prompt Injection and Data Exfiltration

Apache Ossie and Polaris put metric definitions in the open catalog. What the spec covers, what Polaris stores, and what is still unfinished.

Apache Ossie and Apache Polaris: Putting Semantic Models in the Open Catalog

JDBC and ODBC often dominate large-result time. Flight SQL and ADBC keep data columnar from server to client, with Python, Go, and Rust examples.

Arrow Flight SQL and ADBC: Why the Database Driver Is the Slowest Part of Your Query

DataFusion Comet 1.0 replaces Spark Iceberg scans with native Rust. What speeds up, what still falls back to the JVM, and how to deploy it.

DataFusion Comet 1.0 and What Native Rust Scans Change for Spark on Iceberg

ALP and FSST target Parquet's worst cases: floats and high-cardinality strings. How they work and what they change for Iceberg tables.

FSST and ALP: The Two Encodings Fixing Parquet's Weakest Compression Cases

Put REST catalog RBAC and masking in Git. How to review grants, apply them safely, and keep lakehouse access from drifting.

Governance-as-Code for the Lakehouse: Managing REST Catalog RBAC and Masking in Git

Metric contracts in code let teams test, version, and serve business logic to multi-agent systems without each agent inventing its own SQL.

Metric Contracts in Code: Testing, Versioning, and Serving Business Logic to Multi-Agent Systems

High-throughput Iceberg branch merges need conflict detection and automation. How to reconcile concurrent writes without stalling pipelines.

High-Throughput Branch Merging: Automating Concurrency and Conflict Resolution in Multi-Branch Iceberg Pipelines

Polaris can catalog Iceberg tables across AWS, Azure, and GCP. Four topologies, credential vending, and the tradeoffs of each design.

Multi-Cloud REST Catalog Topologies: Running Apache Polaris Across AWS, Azure, and GCP

Iceberg v4 is moving manifests from Avro to Parquet so planners can read only the stats they need. Why the metadata layer is going columnar.

Parquet-Only Manifests in Iceberg v4: Why the Metadata Layer Is Going Columnar

Route each lakehouse query by shape, not by sender. Signals, rules, and how to keep dashboards, batch jobs, and agents from sharing one engine.

Query Routing at Machine Scale: Dynamic Workload Distribution Across Lakehouse Engines

One logical model over Iceberg and databases on three clouds. Pushdown, egress, Reflections, and where semantic federation still breaks.

Semantic Layer Federation: One Logical Model Over Data on Three Clouds

Land small Iceberg micro-batches with PyIceberg and DuckDB in a serverless function. Commits, concurrency, and why Spark is the wrong default.

Serverless Iceberg Ingestion with PyIceberg and DuckDB: Micro-Batches Without a Spark Cluster

Move a legacy warehouse to Iceberg without downtime by virtualizing first. Consumer cutover, parity checks, and background copy without double-ETL.

Zero-Copy Warehouse Modernization: Migrating Legacy Databases to Apache Iceberg Without Downtime

Named, persistent agents need a file format. Open Agent Profile, Buzz, Grok Bot, and Hermes Bot Mode show why a portable agent identity matters.

The Agent Is Now a Named Coworker, and It Needs a File Format

A field guide to AI gateways on Slack, Discord, Telegram, Signal, and Teams: architecture, auth, cost, and the failure modes that matter.

Your Agent Should Answer the Phone: A Field Guide to AI Gateways on Slack, Discord, Telegram, Signal, and Teams

Knowledge graphs, GraphRAG, and LangGraph solved three problems. The fourth is the work itself: a reviewable graph of bounded agentic loops.

Graphs in AI Engineering Have Solved Three Problems. The Fourth Is the Plan.

Trace what one tiny Iceberg commit writes, then model hourly, per-minute, and per-second cadences so streaming costs become arithmetic, not adjectives.

The Hidden Cost of Tiny Iceberg Commits

Position deletes, equality deletes, and deletion vectors compared from the Iceberg spec: what each writes, how readers apply it, and when to use which.

Deletion Vectors vs Position Deletes vs Equality Deletes: The Iceberg Delete Story in 2026

Iceberg is turning from a JVM table format into a library other systems embed. What that shift changes for engines, catalogs, and the spec itself.

Iceberg Is Becoming a Library, Not Just a Table Format

Rust, Go, Python, and C++ Iceberg implementations change who can write the format. Why multi-language clients matter more than another JVM engine.

Iceberg Is Escaping the JVM: Why Rust, Go, Python and C++ Implementations Matter

One suite of REST catalog operations every Iceberg platform should pass. What sameness means, where implementations diverge, and how to test it.

The Iceberg REST Catalog Compatibility Test: One Suite of Operations Every Platform Should Pass

Remote scan planning moves Iceberg file selection into the catalog. What that changes for engines, governance, and operational cost beyond query speed.

Iceberg REST Remote Scan Planning Changes More Than Query Performance

Iceberg row lineage gives rows a durable identity across rewrites. Why CDC pipelines and AI workloads will eventually depend on it.

Iceberg Row Lineage: The Feature AI and CDC Workloads Will Eventually Depend On

Iceberg v4's adaptive metadata tree, explained from first principles: why commits rewrite too much today and how the tree makes change cheaper.

Iceberg v4's Adaptive Metadata Tree, Explained From First Principles

Iceberg v4 is really about making the cost of a change proportional to the change. The principle, the current tax, and what the redesign pays down.

Why Iceberg v4 Is Really About Making the Cost of Change Proportional to the Change

A mechanics walkthrough of Iceberg REST scan planning: client-side planning, remote endpoints, pagination, and where engine support stands in 2026.

The Catalog Can Now Plan Your Iceberg Query: Inside REST Scan Planning

Can seven Iceberg REST catalogs run the same DuckDB script? What the protocol makes portable, what still differs, and a test matrix you can rerun.

Can Seven Different Iceberg REST Catalogs Really Run the Same DuckDB Code?

Iceberg Variant stores JSON as navigable binary with shredding for columnar filters. Why flattening wide tables is no longer the only performance path.

Stop Flattening Your JSON: How Iceberg Variant Changes Semi-Structured Analytics

What happens when two engines write the same Iceberg table at once: snapshot isolation, optimistic commits, conflict detection, and when retries fail.

What Actually Happens When Two Engines Write the Same Iceberg Table at Once?

Variant shredding turns messy JSON into Parquet columns with statistics. How the layout works, how readers reassemble values, and why some queries prune.

Variant Shredding Explained: How Iceberg Gets Columnar Performance From Messy JSON

Managed and external Iceberg tables mean different things on every platform. Five ownership dimensions and a translation method for vendor vocabulary.

Who Actually Owns an Iceberg Table? Managed, External, and the New Vocabulary of Lakehouse Control

Apache Iceberg v3 deletion vectors for high-throughput streaming ingest: how bitmaps and Puffin files fix CDC write amplification and read decay.

Mastering Apache Iceberg v3 Deletion Vectors for High-Throughput Streaming Ingest

The decoupled data lakehouse: multi-engine freedom with open REST catalogs, credential vending, and an estate that outlives its tools.

The Decoupled Data Lakehouse: Multi-Engine Freedom with Open REST Catalogs

The five layers of an agentic lakehouse: Storage, Catalog, Semantic, Gateway, and Agent Surface, and how one question travels through all of them.

The Five Layers of an Agentic Lakehouse

Goal-directed data quality agents that watch Apache Iceberg tables, detect anomalies, and quarantine suspect data safely with snapshot isolation and branches.

Goal-Directed Data Quality Agents: Anomaly Quarantine on Apache Iceberg

Managing the total cost of agentic analytics: token budgets, query throttles, unit economics, and the FinOps discipline that keeps AI spend under control.

Managing the TCO of Agentic Analytics: Token Budgets, Query Throttles, and the Economics of Autonomy

Metric contracts in 2026: versioned, testable definitions of business logic that let multi-agent frameworks compute revenue identically, with OSI interchange.

Metric Contracts in 2026: Standardizing Business Logic Across Multi-Agent Frameworks

Policy-aware lakehouse telemetry for the EU AI Act era: build auditable, immutable, governed AI decision records on Apache Iceberg.

Policy-Aware Lakehouse Telemetry: Building Auditable AI Records on Apache Iceberg for the EU AI Act Era

Query routing at machine scale for the agentic lakehouse: engine selection, acceleration substitution, admission control, and placement across multi-engine estates.

Query Routing at Machine Scale: Multi-Engine Workload Distribution for the Agentic Lakehouse

A threat model for the agentic lakehouse gateway covering prompt injection, exfiltration, and the firewall that reads sentences.

Securing the Agentic Lakehouse Gateway: A Threat Model for Prompt Injection, Exfiltration, and the Firewall That Reads Sentences

Build a federated semantic layer across multi-cloud data so one set of governed metric definitions serves BI tools, dashboards, and AI agents identically.

Semantic Layer Federation: One Meaning for Data That Lives Everywhere

Build lightweight serverless ingestion to Apache Iceberg with PyIceberg and DuckDB, running small feeds in functions that bill for seconds.

Building Lightweight Serverless Ingestion to Apache Iceberg with PyIceberg and DuckDB

How to build stateless AI tool gateways with FastMCP, the 2026 MCP specification, and Kubernetes, and why statelessness finally makes MCP scale.

Building Stateless AI Tool Gateways with FastMCP, the 2026 MCP Spec, and Kubernetes

Write-Audit-Publish with Apache Iceberg branches brings CI/CD to your data: stage, audit, then fast-forward to main so consumers never see unvalidated bytes.

Write-Audit-Publish with Apache Iceberg Branches: CI/CD for Your Data

A practical guide to modernizing a data warehouse to Apache Iceberg without downtime, using federation first, then redirecting new data, then materializing what earns a migration.

Zero-Copy Warehouse Modernization: Moving to Apache Iceberg Without Downtime

Two open specifications, the Agentic Graph Specification and the Open Agent Profile, turn agent plans and agent identity into portable, reviewable files.

The Plan and the Worker: Two Open Specifications for Agent Harnesses

Arrow Flight and ADBC move database results as columnar data, ending the row-oriented bottleneck between engines and applications. Here's how.

Apache Arrow Flight and ADBC, and Why Database Connectivity Finally Went Columnar

Budgeting for agentic analytics when every question costs something different: token economics, query economics, instrumentation, and the cost controls that actually return.

Budgeting for Agentic Analytics When Every Question Costs Something Different

The five layers of an agentic lakehouse and where the MCP server sits: storage, catalog, semantic layer, MCP gateway, and agent surface, plus identity, session isolation, and budgets.

The Five Layers of an Agentic Lakehouse and Where the MCP Server Sits

Autonomous table optimization when query workloads stop being predictable: observing file layout and query patterns, scoring compaction work, adaptive sort order, and cost discipline.

Autonomous Table Optimization When Your Query Workload Stops Being Predictable

How to build an Apache Iceberg lakehouse that runs fully offline: storage, catalog, compute, cross-zone transfer, compliance, and the failure modes that bite.

Building Apache Iceberg Lakehouses That Run Without an Internet Connection

Wiring analytical queries to transactional APIs in closed-loop decision agents: conditional writes, sagas with compensations, decision records, and blast radius controls.

Wiring Analytical Queries to Transactional APIs in Closed-Loop Decision Agents

Surviving optimistic commit collisions when hundreds of agents write to Iceberg: which conflicts are real, commit buffers, partitioning, and the patterns that prevent commit storms.

Surviving Optimistic Commit Collisions When Hundreds of Agents Write to Iceberg

How Iceberg V3 deletion vectors replaced accumulating positional delete files and made merge-on-read viable for streaming and CDC tables.

How Iceberg V3 Deletion Vectors Fixed Merge-on-Read for Streaming Tables

How Iceberg V3's Variant type and Parquet shredding turn JSON columns into prunable typed columns, with real benchmark tradeoffs and a migration path.

How Iceberg V3 Variant Shredding Changed Semi-Structured Data on S3 Tables

A field guide to the Apache Iceberg V4 proposals: adaptive metadata trees, single-file commits, typed statistics, column families, and what is safe to build on today.

Reading the Apache Iceberg V4 Proposals Before They Land

An honest TCO framework for open lakehouses versus proprietary warehouses: five cost categories, measured numbers, sensitivity analysis, and where each side still wins.

Building an Honest TCO Model for Open Lakehouses and Proprietary Warehouses

Why agentic AI needs a governed semantic layer behind the Model Context Protocol: metric consistency, access control, Apache Ossie for portable definitions, and Apache Polaris for enforcement.

Why Agentic AI Needs a Governed Semantic Layer Behind the Model Context Protocol

Moving from supply chain dashboards to decision loops with MCP: sense, decide, act, and verify, with typed action tools, idempotency keys, and graduated human approval.

Moving From Supply Chain Dashboards to Decision Loops With the Model Context Protocol

Metric contracts as the interface AI agents need: calculation, inclusion rules, grain, temporal semantics, ownership, semantic versioning, and testing metrics in CI.

Metric Contracts as the Interface AI Agents Actually Need

How Apache Polaris vends short-lived, prefix-scoped storage credentials across AWS, Azure, and GCP, and how to retire permanent storage keys for good.

Cross-Cloud Credential Vending in Apache Polaris and the End of Permanent Storage Keys

Designing policy-aware AI telemetry tables in Apache Iceberg: what to log, tamper evidence, retention against conflicting deletion requirements, and tracing agent decisions.

Designing Policy-Aware Telemetry Tables for AI Systems in Apache Iceberg

Defending the lakehouse gateway against prompt injection and data exfiltration: per-user identity, no-SQL tool surfaces, volume bounds, and detection in query behavior.

Defending the Lakehouse Gateway Against Prompt Injection and Data Exfiltration

How the Iceberg REST catalog became the lakehouse control plane: multi-table atomic commits, credential vending, capability negotiation, and what still breaks.

How the Iceberg REST Catalog Turned Into the Lakehouse Control Plane

A dependency-first playbook for migrating legacy warehouses onto Apache Iceberg: snapshot vs migrate vs add_files, four-level parity validation, and federation-based cutover.

A Migration Playbook for Moving Legacy Warehouses onto Apache Iceberg

What zero-copy data sharing actually does across Salesforce, Snowflake, and Databricks: query federation, file federation, catalog federation, and when copying still wins.

What Zero-Copy Data Sharing Actually Does Between Salesforce, Snowflake, and Databricks

Apache Polaris 1.7.0 deep dive: idempotent writes, semantic models, stricter credential vending, orphan cleanup, and what the upgrade asks of you.

Apache Polaris 1.7.0 and the Quiet Work of Making a Catalog Trustworthy

How to design batch pipelines that write well into Apache Iceberg: commit strategy, partitioning, sort order, write-audit-publish, and maintenance done right.

Designing Batch Pipelines That Write Well Into Apache Iceberg

How AWS, Google Cloud, and Microsoft Azure actually support Apache Iceberg: storage, catalogs, maintenance, governance, and interoperability, layer by layer.

Apache Iceberg Support Across the Major Hyperscalers

The risk isn't agents going rogue, it's agents acting correctly on bad input at machine speed. Here's how to classify actions by consequence, gate capability, and design approval steps people actually use.

Guardrails for Analytics Agents That Do More Than Answer Questions

A practical guide to building agent decision traces in Apache Iceberg that support audit reconstruction, governance review, and cost attribution across sessions.

Building Agent Telemetry Tables in Iceberg That Survive an Audit

Agent analytics generates two cost streams that scale on different variables. Here's the arithmetic, the levers that actually move the number, and how to build attribution before you need it.

What Agentic Analytics Actually Costs, and How to Keep It Bounded

A practical guide to deploying an Iceberg lakehouse in air-gapped environments: component choices, artifact pipelines, identity without a cloud, and the operational realities that surprise teams.

Running an Apache Iceberg Lakehouse With No Internet Connection

Autonomous materialized view management replaces quarterly review meetings with workload-driven scoring, and it's essential when AI agents generate unpredictable query patterns.

When the Query Optimizer Starts Managing Its Own Materializations

Agents fail on raw lake data because business rules live in people's heads. Data products with semantic contracts fix this at the source.

Why AI Agents Fail on Raw Data, and What to Give Them Instead

Equality deletes made streaming upserts into Iceberg practical at the cost of read performance. V4 proposes retiring them in favor of deletion vectors with an async conversion path.

Why Iceberg V4 Wants to Retire Equality Deletes, and What Streaming Teams Should Do About It

Agent reliability is a property of the stack the model sits on. Five layers with distinct owners and failure modes turn the agent is unreliable into a specific diagnosis.

The Five Layers Between Your Lakehouse and a Trustworthy Agent

Fluss puts a columnar, indexed hot tier between Kafka and Iceberg. Here's what it changes structurally, what Kafka still does better, and how to benchmark the comparison yourself.

Post Search