For most of the last decade, “big data” meant very large tables of structured events: clicks, transactions, sensor readings and logs. The tools that grew up around it, from Parquet files to table formats such as Apache Iceberg and Delta Lake, were optimized for scanning many rows and aggregating a few columns. That model still powers analytics and business intelligence. But AI has changed what organizations want to store and how they want to read it.
Today’s AI workloads need images, video frames, audio, PDFs, code and the embeddings computed from all of them. Training jobs shuffle and sample individual records. Retrieval-augmented generation (RAG) and AI agents fetch a handful of relevant items from billions. These patterns strain storage designed for sequential scans, and teams have responded by bolting on separate vector databases, search engines and blob stores. The multimodal lakehouse is the emerging answer: a single, open storage layer on object storage that holds structured metadata, raw media and embeddings together, and serves analytics, training and retrieval from the same data.
This article explains why the shift is happening, how new formats such as Lance relate to Iceberg, what a reference architecture looks like, and how to decide whether your organization needs one.
Why Classic Lakehouse Formats Struggle With Multimodal Data
Parquet and the table formats built on it are excellent at what they were designed for. Problems appear when the workload changes in three ways.
- Rows got much bigger. A row that contains an image, an audio clip or a long document with its embedding can be megabytes rather than bytes. Columnar files with large row groups make it expensive to read a single record.
- Access became random. Training shuffles, evaluation sampling and retrieval all need to fetch specific rows quickly. As Gradient Flow’s analysis of multimodal lakehouses notes, Parquet is optimized for sequential scans, which creates input/output bottlenecks for training and RAG.
- Data changes in new ways. AI teams frequently add a new column, such as a fresh embedding from a newer model or a new label, to an existing dataset. In classic formats this often means rewriting large amounts of data.
The common workaround has been fragmentation: metadata in a warehouse or Iceberg tables, media in object storage, embeddings in a vector database, and text in a search engine. Each system has its own security model, copies of data and synchronization jobs. Keeping them consistent is a constant source of bugs, and governance across them is difficult.
What Lance Brings to the Table
Lance is an open source format designed for AI and multimodal workloads. According to the Apache Polaris project’s January 2026 description, Lance covers three layers: a file format, a table format and a catalog specification called Lance Namespace, which lets Lance tables be registered in existing catalogs. Its key capabilities include:
- Fast random access to individual rows without sacrificing scan performance. Vendor and project materials report large speedups over Parquet for random reads; treat such figures as claims to validate on your own data.
- Blob handling for images, video, audio and text, with lazy loading so large media is fetched only when needed.
- Built-in indexes for vector similarity search and full-text search, enabling vector, keyword and hybrid queries on the same dataset alongside SQL filters.
- Schema evolution with backfill, so new columns such as fresh embeddings can be added without full rewrites.
- Versioning and time travel, with ACID transactions, which supports reproducible training and evaluation.
Lance is also becoming part of the open lakehouse ecosystem rather than a silo. Apache Polaris can register Lance tables as generic tables alongside Iceberg tables in the same namespace hierarchy, and the Polaris project lists engines such as Spark, Ray, Trino and LanceDB working with Polaris-managed Lance tables. Integrations with Arrow, pandas, Polars and DuckDB are also documented. Lance has adopted a formal community governance structure, which matters for organizations wary of single-vendor formats.
The multimodal lakehouse is not a replacement for Iceberg. It is the recognition that AI data needs a storage layer built for random access and search, governed in the same catalog as your analytical tables.
Iceberg and Lance: Complementary, Not Competing
A pragmatic pattern, described in a May 2026 analysis by Data Lakehouse Hub, uses both formats as layers over the same object storage:
- Iceberg holds structured metadata about each asset, such as source, labels, dataset splits, timestamps and quality scores, and serves SQL analytics and governance.
- Lance holds embeddings and other AI-facing columns and serves random-access retrieval and vector search. Records reference media stored in object storage or, in simpler designs, store the bytes directly.
- Object storage holds raw images, audio and video.
- A shared catalog such as Polaris, Unity Catalog or Gravitino provides discovery and access control across both formats.
- Query engines such as DuckDB, Spark or Trino join Lance results with Iceberg metadata when a workload needs both.
The same analysis warns against defaulting to “Iceberg for everything” for AI retrieval, but it also notes trade-offs: running two formats adds operational complexity; embedded deployments can hit concurrency limits when many jobs read at once; and for production search with very low latency requirements, a dedicated vector database may still be appropriate alongside the lakehouse.
A Reference Architecture for 2027
The following architecture is a starting point for organizations with substantial unstructured data and AI ambitions. Adapt it to your cloud, tools and scale.
- Landing zone. Raw files arrive in object storage from applications, devices, document systems and partners, with immutable storage and content hashes for provenance.
- Metadata extraction. Pipelines extract technical metadata (format, size, duration, resolution), business metadata (customer, product, case) and quality signals, and write them to Iceberg tables.
- Enrichment. GPU and CPU workers, often orchestrated with Ray or Spark, compute embeddings, captions, transcripts, OCR text and classifications. Results land in Lance tables keyed by the same content identifier.
- Indexing. Vector and full-text indexes are built on Lance tables, with refresh schedules matched to how fast data changes.
- Governance. All tables are registered in one catalog with role-based access, tags for sensitive data and lineage captured from pipelines.
- Consumption. Analysts query Iceberg with SQL; data scientists sample training sets with random access; RAG services and agents retrieve through hybrid search; and experiment tracking records the table versions used for each model.
Reproducibility by design
Record the Iceberg snapshot ID and the Lance table version used for each training run, evaluation or index build. When a model behaves unexpectedly, you can reconstruct exactly what it saw. This is also useful evidence for AI governance and regulatory requirements on training data documentation.
Governance and Security Considerations
Unstructured data carries risks that structured tables often do not. Images contain faces, documents contain personal and confidential information, and audio can contain anything said in a call. A multimodal lakehouse should treat governance as a first-class design concern.
- Classification at ingestion. Detect personal data, sensitive categories and confidential content as files land, and tag records accordingly.
- Access control at the right grain. Apply catalog-level permissions to tables and columns, and consider row-level filters for sensitive subsets. Remember that embeddings derived from sensitive content can themselves leak information.
- Retention and deletion. Implement deletion workflows that remove raw files, metadata, embeddings and index entries together, so right-to-erasure requests are honored everywhere.
- Licensing and provenance. Track the source and license of every asset, particularly third-party and scraped content used for training.
- Credential vending. Prefer catalogs that issue short-lived, scoped storage credentials to engines rather than giving every engine broad bucket access.
When You Need a Multimodal Lakehouse, and When You Do Not
Not every organization needs a new storage format. Use the following questions to decide.
- Do you have, or expect to have, large volumes of images, video, audio or documents that feed AI models or retrieval?
- Do multiple teams need the same unstructured data for analytics, training and serving, and are they currently maintaining separate copies?
- Are you running separate vector databases, search engines and blob catalogs that are hard to keep in sync and govern?
- Do training jobs or retrieval services suffer from slow random reads on your current storage?
- Do you need reproducible snapshots of training and evaluation data for audit or regulatory reasons?
If most answers are yes, a multimodal lakehouse is worth piloting. If your unstructured data is small, or your AI use is limited to a single RAG application over a modest document set, a managed vector database or the vector features of your existing database may be simpler and entirely sufficient.
A 90-Day Pilot Plan
A focused pilot is the best way to test whether a multimodal lakehouse fits your organization. Keep the scope narrow and the success criteria explicit.
- Days 1–15: choose the use case and baseline. Pick one workload that genuinely combines structured metadata, unstructured content and embeddings, such as a document assistant or an image training set. Measure the current state: how many copies of the data exist, how long retrieval and training data loading take, and how access is governed.
- Days 16–45: build the foundation. Stand up or extend your catalog, create the Iceberg metadata tables and Lance tables for one dataset, and build the ingestion and enrichment pipeline with consistent content identifiers. Apply classification and access policies from the start rather than retrofitting them.
- Days 46–75: connect consumers. Point one analytics dashboard, one training or evaluation job and one retrieval service at the new tables. Benchmark random-access reads, vector and hybrid query latency, and pipeline cost against the baseline.
- Days 76–90: decide. Review results with data engineering, data science, security and finance. Decide whether to expand, adjust the design, or stay with the existing approach for this class of workload.
Document what you learn about operations, such as compaction, index refresh, concurrency limits and deletion workflows. Those lessons matter more for a production decision than headline query speeds.
Pitfalls to Avoid
- Adopting a format without a catalog strategy. Tables scattered across buckets without a shared catalog recreate the governance problems you were trying to solve.
- Trusting benchmarks you did not run. Random-access and vector query speedups depend heavily on data shape, index settings and hardware. Benchmark with your own data.
- Confusing format with platform. Lance the open format and commercial platforms built on it are different things. Be clear which features are portable.
- Ignoring embedding lifecycle. Embedding models change. Plan for versioned embedding columns and re-indexing rather than overwriting in place.
- Underestimating media pipelines. Decoding video, extracting frames and running OCR are expensive. Size compute and queues realistically.
A Hypothetical Worked Example
A hypothetical industrial equipment manufacturer holds millions of inspection photos, maintenance videos, PDF manuals and technician notes. Its data science team wants to train defect-detection models; its service organization wants an AI assistant that answers technician questions with references to manuals and similar past cases.
- Inspection photos and videos land in object storage. Metadata such as machine model, site, inspection date and defect labels goes into Iceberg tables used for quality dashboards.
- An enrichment pipeline computes image embeddings, extracts frames from videos, runs OCR on manuals and chunks documents. Embeddings and text chunks are stored in Lance tables with vector and full-text indexes.
- Both table types are registered in one catalog. Images containing people are tagged, and access is restricted to approved roles.
- Data scientists sample balanced training sets using random access, recording table versions with each experiment. The assistant uses hybrid search to retrieve relevant manual sections and similar past inspection cases.
- When a new embedding model is adopted, a new embedding column is added and indexed alongside the old one, and the assistant is switched over after evaluation.
The example is illustrative, but it shows the core value: one governed copy of the data, serving analytics, training and retrieval.
Outlook
Through 2027 and beyond, expect open catalogs to treat multimodal formats as first-class citizens alongside Iceberg, query engines to add native support for vector and full-text search over lakehouse tables, and AI agents to become major consumers of multimodal data through retrieval interfaces. The boundary between “data lake,” “vector database” and “search engine” is likely to keep blurring. Organizations that standardize on open formats and a shared catalog now will be able to adopt those capabilities without another round of copying data between silos.
Conclusion
Big data is no longer only about rows of numbers. AI needs images, video, documents and embeddings, accessed randomly and searched semantically. The multimodal lakehouse brings these together on open formats and object storage, with Iceberg continuing to serve structured analytics and AI-oriented formats such as Lance serving retrieval and training. Start with a clear catalog and governance strategy, benchmark on your own data, and pilot with a use case that genuinely needs unified unstructured data.
How Hanoi Technology Can Help
Hanoi Technology designs and builds lakehouse platforms that bring structured, unstructured and embedding data together under one governed catalog for analytics and AI.