Open table formats such as Apache Iceberg and Delta Lake gave data teams warehouse-like features on cheap object storage: ACID transactions, time travel, schema evolution and engine independence. They also handed those teams a job that a traditional warehouse quietly did for them: housekeeping. Every commit writes new data files and new metadata. Every streaming micro-batch adds small files. Every update or delete can leave delete files or rewritten copies behind. Left alone, a healthy table slowly turns into a slow, expensive one.
In 2026, maintenance is no longer an afterthought. Lakehouses are feeding AI agents and near-real-time dashboards, streaming ingestion into Iceberg is mainstream, and storage and compute bills are under FinOps scrutiny. At the same time, managed options such as Amazon S3 Tables and Databricks predictive optimization now run maintenance automatically. This playbook explains what maintenance actually does, how to design a policy, and when to let a managed service do it for you.
Why Lakehouse Tables Degrade Over Time
A lakehouse table is a set of immutable data files plus a tree of metadata that says which files belong to which version of the table. That design is what makes time travel and concurrent writers possible, but it creates four kinds of debris.
- Small files. Streaming writers and frequent small batches produce many small Parquet files. Query engines must open and plan each one, so a table with hundreds of thousands of tiny files is slower and more expensive to scan than the same data in a few thousand well-sized files.
- Accumulated snapshots. Each commit creates a new snapshot. Old snapshots keep references to data files that have been replaced or deleted, so storage keeps growing even when the logical table does not.
- Metadata bloat. Manifests and metadata files multiply with commits. Large or poorly organized manifests slow down query planning, which hurts interactive and agent-driven queries most.
- Orphan files. Failed jobs and aborted writes can leave files in the table location that no metadata references. They cost money and never get read.
Row-level operations add a fifth concern. Merge-on-read updates and deletes write delete files that readers must apply at query time. Until compaction folds them into rewritten data files, every query pays that cost.
The Core Maintenance Operations Explained
The Apache Iceberg documentation groups maintenance into recommended operations (expiring snapshots, removing old metadata files and deleting orphan files) and optional ones (compacting data files and rewriting manifests). Delta Lake has close equivalents. Here is what each one does and what to watch for.
1. Snapshot expiry
Expiring snapshots removes old table versions from metadata and allows data files referenced only by those versions to be deleted. In Iceberg this is the expireSnapshots operation; in Delta Lake, history retention combined with VACUUM plays the same role. The Iceberg documentation is explicit about the trade-off: once a snapshot is expired, you can no longer time travel to it or roll back to it. Your retention window is therefore a business decision about auditability and recovery, not just a storage setting.
2. Old metadata file cleanup
Iceberg writes a new metadata file on every commit. The table property write.metadata.delete-after-commit.enabled tells Iceberg to delete the oldest metadata files after each commit, and write.metadata.previous-versions-max controls how many to keep. The Iceberg docs note this is off by default and is especially useful for tables with frequent commits, such as those written by streaming jobs.
3. Orphan file deletion
Orphan file removal scans the table location and deletes files that no metadata references. It is the most dangerous operation in the toolkit. The Iceberg documentation warns that using a retention interval shorter than the time any write might take can delete files that an in-progress job is about to commit, corrupting the table; the default interval is three days. It also warns that because Iceberg tracks files by path string, a change in how paths are written (for example, a storage authority change) can make valid files look orphaned. Run it periodically, conservatively and after verifying path consistency.
4. Data file compaction
Compaction rewrites many small files into fewer larger ones and, for merge-on-read tables, applies pending deletes. Iceberg exposes this as rewriteDataFiles; Delta Lake as OPTIMIZE. Common strategies are bin-packing (simply combining files), sorting by frequently filtered columns, and multi-dimensional clustering such as Z-ordering. Sorting and clustering make filtered queries faster by improving file skipping, but they cost more compute to run.
5. Manifest rewriting
Iceberg’s rewriteManifests regroups data file entries into manifests that align with how the table is queried. The Iceberg docs recommend it when write patterns do not match read patterns, for example when data arrives by ingestion time but is queried by business date. Better-organized manifests mean faster planning.
6. Statistics collection
Query optimizers depend on table and column statistics. Delta Lake’s ANALYZE and Iceberg’s statistics files (such as Puffin-based column statistics) keep cost-based optimization effective as data changes.
Maintenance is not optional overhead. In an open lakehouse, compaction, snapshot expiry and orphan cleanup are part of the table’s design, and the retention window you choose is a governance decision as much as a cost decision.
Managed Maintenance: What the Platforms Now Do for You
The biggest change since 2024 is that major platforms now run maintenance automatically for tables they manage. Understanding exactly what they cover is essential, because the gaps are where problems appear.
Amazon S3 Tables
According to the AWS documentation, maintenance is enabled by default for tables in S3 table buckets. Compaction targets a default file size of 512 MB (configurable from 64 MB to 512 MB) and supports an automatic strategy that uses sort if the table defines a sort order and bin-packing otherwise, plus explicit binpack, sort and Z-order strategies; AWS notes that sort and Z-order may cost more. Snapshot management defaults to keeping at least one snapshot and expiring snapshots older than 120 hours, configurable per table. AWS also warns that snapshot management fails for a table if user-defined tags or branches exist or if certain Iceberg expiry properties are set, and that objects removed after expiry and the unreferenced-file policy are permanently deleted.
Databricks predictive optimization
Databricks documentation describes predictive optimization as enabled by default for Unity Catalog managed tables in Delta Lake or Iceberg format, with account admins able to change the setting and catalogs, schemas and tables able to override it. It runs OPTIMIZE (without Z-order), VACUUM and ANALYZE on serverless jobs compute, and logs operations, costs and impact to the system table system.storage.predictive_optimization_operations_history. It does not run on external tables.
Other engines and catalogs
Snowflake-managed Iceberg tables, Microsoft Fabric’s OneLake tables, Google BigQuery tables for Apache Iceberg and several independent catalogs also offer forms of automatic optimization. Capabilities differ, so the same questions apply everywhere: which operations run, on which tables, with what defaults, at whose cost, and how you can see what happened.
Designing a Maintenance Policy: Step by Step
- Inventory and classify tables. List every table with its format, catalog, writer type (streaming, batch, CDC merge), size, commit frequency, and whether it is managed by a platform or external. Classification drives everything else.
- Decide retention by business need. Agree with data owners and compliance how far back time travel and rollback must reach for each class of table. Seven days might suit a staging table; a regulated reporting table may need longer, or a separate archival mechanism such as tagged snapshots.
- Choose who runs maintenance. For platform-managed tables, prefer the managed service and disable overlapping custom jobs. For external tables or self-managed catalogs, schedule your own jobs with Spark, Flink or your engine’s procedures.
- Set compaction strategy per table. Use bin-packing for append-heavy tables with no dominant filter. Use sort or clustering where queries consistently filter on a few columns. Set target file sizes that suit your engines and data volumes.
- Schedule by writer pattern. Streaming tables need frequent, incremental compaction of recent partitions. Batch tables can be compacted after each load. CDC merge tables with heavy deletes need compaction often enough to keep delete files under control.
- Make orphan cleanup conservative. Use retention intervals comfortably longer than your longest-running write job, verify storage paths, and run it weekly or monthly rather than daily.
- Avoid conflicts. Compaction commits can conflict with concurrent writers. Target partitions that are no longer receiving writes, use engine features that support concurrent rewrites where available, and coordinate with streaming jobs.
- Monitor outcomes, not just job success. Track file counts and average file size per table, snapshot counts, planning time, scan bytes and storage growth. A job that succeeds but does not change these metrics is not working.
- Attribute cost. Maintenance consumes compute. Record its cost per table or domain so you can see when sort-based compaction is paying for itself and when it is not.
Maintenance Cadence Cheat Sheet
The right cadence depends on your workloads, but these starting points are a reasonable baseline to tune from.
- Streaming ingestion tables: compact recent partitions hourly to daily; enable metadata file cleanup; expire snapshots daily with a short retention window agreed with owners.
- Daily batch tables: compact after each load or nightly; expire snapshots weekly; rewrite manifests when planning time rises.
- CDC merge tables: compact frequently enough to keep delete files low; monitor merge duration as an early warning.
- Large, rarely changing dimension or reference tables: compact and sort occasionally; keep longer snapshot history if audits need it.
- All tables: orphan cleanup weekly or monthly with a conservative interval; statistics refresh after significant data changes.
Worked Example (Hypothetical)
Imagine a hypothetical online retailer that streams clickstream events from Kafka into an Iceberg table every minute and merges order updates from its commerce database via CDC every five minutes. After three months, analysts complain that dashboards are slow and the finance team notices storage costs growing faster than event volume.
An assessment finds the clickstream table has several hundred thousand small files, thousands of retained snapshots because no expiry was configured, and old metadata files accumulating with every commit. The orders table has a large backlog of delete files from merge-on-read updates. Nobody ever scheduled orphan cleanup, and failed jobs during an outage left unreferenced files behind.
The team agrees retention with data owners: seven days of time travel for clickstream, thirty for orders, with month-end snapshots tagged for audit. They enable metadata cleanup on both tables, schedule hourly bin-pack compaction of the previous hour’s clickstream partitions, and nightly sort-based compaction of orders by customer and order date, the columns most dashboards filter on. Snapshot expiry runs daily, orphan cleanup weekly with a generous interval. They add a dashboard tracking file counts, average file size, snapshot counts and query planning time. Because the tables live in a self-managed catalog, they run these jobs themselves; if they later moved to a managed table service, they would retire the custom jobs and keep only the monitoring.
Common Pitfalls
- Double maintenance. Running custom compaction on tables a platform already optimizes wastes compute and can cause commit conflicts.
- Aggressive orphan cleanup. Short intervals can delete files from in-flight writes. This is the fastest way to corrupt a table.
- Expiring snapshots auditors need. Set retention with compliance input, and use tags or separate archives for long-term needs rather than keeping everything forever.
- Ignoring managed-service caveats. For example, AWS documents that snapshot management on S3 Tables fails if branches or tags exist on a table. Read the limitations before relying on defaults.
- Sorting everything. Sort and Z-order compaction help selective queries but cost more; apply them where query patterns justify it.
- No ownership. Maintenance falls between platform and domain teams. Make one team accountable for policy and another, or the same, for execution.
Outlook: Toward Self-Driving Tables
Through 2027 and beyond, expect maintenance to become more autonomous and more catalog-centric. Managed table services and catalogs are absorbing compaction, expiry and cleanup, and they increasingly decide when to run them based on observed query patterns. Iceberg v3 features such as deletion vectors reduce the cost of row-level changes, which changes compaction economics. The data team’s role shifts from writing maintenance jobs to setting policy, watching outcomes and managing cost. Teams that classify tables and define retention now will be able to hand execution to automation with confidence later.
Conclusion
The open lakehouse delivers flexibility only if its tables stay healthy. Compaction keeps queries fast, snapshot expiry and metadata cleanup keep storage and planning under control, and careful orphan cleanup removes waste without risking corruption. Use managed maintenance wherever your platform offers it, understand its defaults and gaps, and run disciplined, monitored jobs for everything else. Treated as part of table design rather than an afterthought, maintenance is what turns a working lakehouse into a dependable one.
How Hanoi Technology Can Help
Hanoi Technology designs and runs lakehouse maintenance policies, monitoring and managed operations for Iceberg and Delta tables through projects, dedicated teams or managed services.