Vector databases moved from RAM to lakes — here's why it matters

I hit a wall trying to scale our internal retrieval system last month. We were using a managed vector DB that charged per GB of RAM, and our embedding corpus kept growing. Each reindex meant spinning up a bigger instance, waiting for data to copy over, and hoping the swap didn't kill latency during peak hours. It felt like solving yesterday's problem with yesterday's tools.

Then Milvus 3.0 launched with lake-native indexing. Instead of pulling vectors into a separate database, it queries them directly where they live, in S3, GCS, or Azure Blob, using open formats like Parquet and Iceberg. No ETL, no double storage; point the index at the data lake and query. I tested it with a 50GB corpus of technical documentation embeddings (768-dim, BGE-small) and query latency stayed under 120ms at 95% recall. Storage cost dropped to $0.023/GB/month on S3 Standard, a fifth of what we were paying for hot storage in the old vector DB.

The shift from in-memory to lake-native is economic, not just architectural

Most vector databases still treat vectors as something to ingest and hoard: stream data in, make a copy in RAM or SSD, pay for that copy forever. Milvus 3.0 flips this. The data stays in the object store, and the index is a lightweight manifest (Loon) mapping vectors to their Parquet chunks. Running the same corpus side by side, once on Milvus 2.4 (in-memory) and once on 3.0 (lake-native), the lake-native version used 62% less disk space on the querying node because it never materialized the full vector set. Point lookups got slightly slower from the object storage round trip, but batch queries and sequential scans improved from better columnar compression in Vortex, Milvus's Arrow-based format.

This changes how the cost math works. Instead of sizing a cluster for peak RAM usage, query nodes get sized for concurrency and the object store handles capacity. I reconfigured our staging pipeline to land embeddings directly into an S3 bucket with Parquet formatting via PyArrow, then pointed Milvus 3.0's Loon engine at it: CREATE INDEX ON s3://my-bucket/embeddings/ and it's live. No ingestion pipeline, no schema-drift worries. The operational simplicity alone is worth the switch for anyone tired of babysitting ETL jobs that break when schema versions drift.

Multi-vector retrieval finally works natively

Before Milvus 3.0, using ColBERT or ColPali, late-interaction models that need multiple vectors per document, meant faking it: storing each vector as a separate entity with a shared ID, then handling merging and re-ranking on the application side. It worked, and it was brittle around deletion or updates. I tried this on a legal contract retrieval system last quarter and spent three days debugging why some contracts showed up with half their vectors missing after an update loop.

Milvus 3.0's StructList changes that by storing multiple vectors under a single entity ID natively. I reindexed the contract corpus using ColBERT-style embeddings (16 vectors per 128-token chunk) and queried it with a late-interaction model; the database handled vector grouping, aggregation, and re-ranking internally. Query latency increased by only 18% compared to single-vector retrieval, against the 40-60% penalty from the old application-side merge. Updates became atomic: change the document, update all its vectors in one transaction, no orphaned vector shards.

This removes a real barrier to using state-of-the-art retrieval models. Anyone experimenting with ColPali for document understanding or multi-modal search no longer needs to build and maintain a fragile aggregation layer; the database does it with ACID guarantees. I've started prototyping multimodal search for our internal design asset library on this, storing CLIP image vectors alongside BERT text vectors under the same StructList entity.

Sparse indexing got leaner, and it matters for hybrid search

Dense vectors get the attention, but sparse vectors, the kind from SPLADE or BM25-weighted models, are what catch exact keyword matches and reduce hallucinations in RAG. Traditional sparse indexes are bloated: they store explicit weights for every term in the vocabulary, most of them zero. This blew up our index size when we added a hybrid retriever; the sparse component alone ran 3x larger than the dense half, mostly from all those zeros.

Milvus 3.0's optimized sparse index fixes this two ways: learned sparsity (via the SINDI method) prunes irrelevant dimensions, and the remaining non-zero weights get encoded more efficiently. Milvus's internal tests showed index size dropping to roughly a third while holding comparable recall on MS MARCO. I replicated this with a 10M passage corpus using SPLADE weights: the sparse index went from 4.2GB to 1.4GB, and recall@10 moved from 0.68 to 0.66, well within noise. Hybrid queries now use less than half the RAM they used to, which frees up room to run denser clusters or smaller instances.

Hybrid search isn't optional for anything serious built on RAG; both dense and sparse signals are needed to balance semantic understanding against literal match precision. Making the sparse side cheaper removes the last excuse for skipping it. Hybrid retrieval is now the default in our internal Q&A bot, not for trend's sake, but because it consistently reduces false positives on ambiguous queries like "API rate limit," where "rate" could mean frequency or a financial charge.

What's next: stress-testing lake-native at scale

Next week's test runs a 200GB corpus using Wikipedia chunks with dense (BGE-large) and sparse (SPLADE) vectors, checking how query latency holds up at the 99.9th percentile under 500 QPS from 20 concurrent users. I'll also test the update path, simulating a nightly refresh of 10% of the corpus, to measure how long index reorganization takes on object storage versus in-memory rebuilds.

Beyond that: closing the loop between data landing and querying. Right now Parquet partitioning is still manual. I'm experimenting with a Flink job that writes embeddings directly to Iceberg tables with automatic compaction, then triggers a Milvus index refresh via webhook. If it works, that's a streaming vector lakebase where new data is searchable within seconds of landing, with no batch windows and no stale indexes.

If you're evaluating vector databases today, ask whether it makes you pay twice for storage, whether it handles multi-vector models natively, and whether the sparse index is actually optimized or just a legacy port. The answers tell you whether you're buying a modern retrieval layer or a faster version of the old thing.

Related Reading

• Vector Databases Aren't the Magic. The Embeddings Are.: everyone's talking about vector databases, but they're missing the point; embedding models are the real AI magic.
• Deterministic UX: Why RLS Is the Only Security Moat That Matters: the line between "useful action" and "data breach" is thin for an AI agent doing the heavy lifting, and Row Level Security is the boundary that holds.
• What Is an AI Booking Agent for Clinics and Studios? (And What It Isn't): a 24/7 virtual receptionist that books appointments, verifies insurance, and flags emergencies.