I hit a wall trying to scale our internal retrieval system last month. We were using a managed vector DB that charged per GB of RAM, and our embedding corpus kept growing. Each reindex meant spinning up a bigger instance, waiting for data to copy over, and hoping the swap didn’t kill latency during peak hours. It felt like we were solving yesterday’s problem with yesterday’s tools.
Then I saw Milvus 3.0 launch with lake-native indexing. Instead of pulling vectors into a separate database, it queries them directly where they live, in S3, GCS, or Azure Blob, using open formats like Parquet and Iceberg. No ETL. No double storage. Just point the index at your data lake and query. I tested it with a 50GB corpus of technical documentation embeddings (768-dim, BGE-small) and the query latency stayed under 120ms at 95% recall. The kicker? Storage cost dropped to $0.023/GB/month on S3 Standard, a fifth of what we were paying for hot storage in our old vector DB.
The shift from in-memory to lake-native isn’t just architectural, it’s economic
Most vector databases still treat vectors as something to ingest and hoard. You stream data in, they make a copy in RAM or SSD, and you pay for that copy forever. Milvus 3.0 flips this: your data stays in the object store, and the index is just a lightweight manifest (Loon) that maps vectors to their Parquet chunks. When I ran a side-by-side test with the same corpus, one using Milvus 2.4 (in-memory) and one using 3.0 (lake-native), the lake-native version used 62% less disk space on the querying node because it never materialized the full vector set. The trade-off? Slightly higher latency for random point lookups due to object storage roundtrips, but batch queries and sequential scans actually improved thanks to better columnar compression in Vortex (their Arrow-based format).
This changes how you think about cost. Instead of sizing your cluster for peak RAM usage, you size your query nodes for concurrency and let the object store handle capacity. I reconfigured our staging pipeline to land embeddings directly into an S3 bucket with Parquet formatting via PyArrow, then pointed Milvus 3.0’s Loon engine at it. No ingestion pipeline. No schema drift worries. Just CREATE INDEX ON s3://my-bucket/embeddings/ and go. The operational simplicity alone is worth the switch if you’re tired of babysitting ETL jobs that break when schema versions drift.
Multi-vector retrieval finally works natively, no more fragile workarounds
Before Milvus 3.0, if you wanted to use ColBERT or ColPali (late-interaction models that need multiple vectors per document), you had to fake it. Common hacks: store each vector as a separate entity with a shared ID, then do application-side merging and re-ranking. It worked, but it was brittle, especially when dealing with deletion or updates. I tried this with a legal contract retrieval system last quarter and spent three days debugging why some contracts showed up with half their vectors missing after an update loop.
Milvus 3.0’s StructList changes that. It lets you store multiple vectors under a single entity ID natively. I reindexed our contract corpus using ColBERT-style embeddings (16 vectors per 128-token chunk) and queried it with a late-interaction model. The database handled the vector grouping, aggregation, and re-ranking internally. Query latency increased by only 18% compared to single-vector retrieval, versus the 40-60% penalty we saw with the application-side merge approach. More importantly, updates became atomic, change the document, update all its vectors in one transaction. No more orphaned vector shards.
This isn’t just a convenience feature. It removes a major barrier to adopting state-of-the-art retrieval models. If you’re experimenting with ColPali for document understanding or multi-modal search, you no longer need to build and maintain a fragile aggregation layer. The database does it for you, with ACID guarantees. I’ve already started prototyping a multimodal search for our internal design asset library using this, storing CLIP image vectors alongside BERT text vectors under the same StructList entity.
Sparse indexing got leaner, and it actually matters for hybrid search
Dense vectors get all the attention, but sparse vectors (like those from SPLADE or BM25-weighted models) are where you catch exact keyword matches and reduce hallucinations in RAG. The problem? Traditional sparse indexes are bloated. They store explicit weights for every term in the vocabulary, even if most are zero. I saw this blow up our index size when we added a hybrid retriever, the sparse component alone was 3x larger than the dense half, mostly due to zeros.
Milvus 3.0’s optimized sparse index fixes this with two changes: first, it uses learned sparsity (via the SINDI method) to prune irrelevant dimensions; second, it encodes the remaining non-zero weights more efficiently. In their internal tests, the index size dropped to ~1/3rd while maintaining comparable recall on MS MARCO. I replicated this with a 10M passage corpus using SPLADE weights: the sparse index went from 4.2GB to 1.4GB, and recall@10 changed from 0.68 to 0.66, well within noise. The real win? Hybrid queries now use less than half the RAM they used to, letting us run denser clusters or lower instance sizes.
This matters because hybrid search isn’t optional anymore. If you’re building anything serious with RAG, you need both dense and sparse signals to balance semantic understanding with literal match precision. Making the sparse side cheaper removes a key excuse for skipping it. I’ve since made hybrid retrieval the default in our internal Q&A bot, not because it’s trendy, but because it consistently reduces false positives on ambiguous queries like “API rate limit” (where “rate” could mean frequency or financial charge).
What’s next: stress-testing lake-native at scale and closing the loop
I’m setting up a 200GB corpus test next week using Wikipedia chunks with dense (BGE-large) and sparse (SPLADE) vectors. Goal: see how query latency holds up at 99.9th percentile when we hammer it with 500 QPS from 20 concurrent users. I’ll also test the update path, simulating a nightly refresh of 10% of the corpus, to measure how long index reorganization takes on object storage versus in-memory rebuilds.
Beyond that, I want to close the loop between data landing and querying. Right now, we still rely on manual Parquet partitioning. I’m experimenting with a Flink job that writes embeddings directly to Iceberg tables with automatic compaction, then triggers a Milvus index refresh via webhook. If this works, we could have a truly streaming vector lakebase where new data is searchable within seconds of landing, no batch windows, no stale indexes.
If you’re evaluating vector databases today, ask: does it make you pay twice for storage? Can it handle multi-vector models natively? Is the sparse index actually optimized, or just a legacy port? The answers will tell you whether you’re buying a modern retrieval layer or just a faster version of the old thing.
Related Reading
- Vector Databases Aren't the Magic. The Embeddings Are. — Everyone's talking about vector databases, but they're missing the point. Discover why embedding models are the real AI magic and learn how they power...
- Deterministic UX: Why RLS is the Only Security Moat That Matters — In an era where AI agents are doing the heavy lifting, the line between "useful action" and "data breach" is paper thin. Row Level Security (RLS) is no longe...
- What Is an AI Booking Agent for Clinics & Studios? (And What It Isn’t) — An AI booking agent isn’t just a fancy chatbot—it’s a 24/7 virtual receptionist that books appointments, verifies insurance, and flags emergencies. Here’s wh...
