Namespace branching for A/B testing embedding models

Switching embedding models is one of the riskiest operations in a vector search system. A new model changes the geometry of the embedding space — vectors from two different models are not comparable, so a partial migration will corrupt your index.

ManyVector’s copy-on-write namespace branching solves this. Fork your production namespace, re-embed into the fork, benchmark recall, then promote atomically — without downtime or data duplication.

Why forking matters

Traditional approaches to model migration involve:

1. Provisioning a second cluster (expensive, slow)
2. Dual-writing to both clusters during migration (error-prone)
3. Big-bang cutover with a maintenance window

With ManyVector namespace branching:

1. Fork creates a pointer in milliseconds — zero bytes copied
2. Writes to the fork are isolated (copy-on-write)
3. Promotion is an atomic metadata swap — no data movement
4. Rollback is free — delete the fork, parent is unchanged

Step 1: Understand the current state

from manyvector import ManyVector

client = ManyVector(api_key="mv-...")

# Get production namespace stats
prod = client.namespace("knowledge-base-prod")
stats = prod.stats()
print(f"Production: {stats.vector_count:,} vectors, model=text-embedding-3-small")

Step 2: Fork production

# Fork takes milliseconds regardless of namespace size
fork = client.fork_namespace(
    source="knowledge-base-prod",
    name="knowledge-base-text3-large-eval",
)
print(f"Forked to: {fork.name} in {fork.fork_duration_ms}ms")
# Forked to: knowledge-base-text3-large-eval in 47ms

The fork shares all objects with the parent. No data is copied. Only a metadata record is written.

Step 3: Re-embed into the fork

Clear the fork and re-embed with the new model. Writes go only to fork-specific objects:

import openai

oai = openai.OpenAI(api_key="sk-...")

def re_embed_namespace(source_ns, target_ns, new_model: str, batch_size: int = 100):
    """Re-embed all vectors from source into target using a new model."""
    page_token = None
    total = 0

    while True:
        # Fetch a page of records from the source
        page = source_ns.list(limit=batch_size, page_token=page_token)
        if not page.records:
            break

        # Re-embed the text attributes with the new model
        texts = [r.attributes["text"] for r in page.records]
        response = oai.embeddings.create(input=texts, model=new_model)
        new_embeddings = [item.embedding for item in response.data]

        # Upsert into the fork with the new embeddings
        target_ns.upsert([
            {**r.attributes, "id": r.id, "vector": new_emb}
            for r, new_emb in zip(page.records, new_embeddings)
        ])

        total += len(page.records)
        print(f"Re-embedded {total} records...")
        page_token = page.next_page_token
        if not page_token:
            break

    print(f"Done. Re-embedded {total} total records.")

prod_ns = client.namespace("knowledge-base-prod")
fork_ns = client.namespace("knowledge-base-text3-large-eval")

# Re-embed into fork using text-embedding-3-large (3072 dimensions)
re_embed_namespace(prod_ns, fork_ns, new_model="text-embedding-3-large")

Step 4: Define your eval set

Build a ground-truth evaluation set before benchmarking:

# eval_set.jsonl format: {"query": "...", "relevant_doc_ids": ["doc-1", "doc-2"]}
eval_queries = [
    {"query": "How does HNSW graph traversal work?", "relevant_doc_ids": ["doc-12", "doc-45"]},
    {"query": "What is the cost of object storage?",  "relevant_doc_ids": ["doc-7", "doc-23"]},
    # ... 100+ queries for a reliable benchmark
]

Step 5: Benchmark both namespaces

def recall_at_k(namespace, query_text: str, relevant_ids: list[str], k: int = 10) -> float:
    embedding = oai.embeddings.create(
        input=[query_text],
        model="text-embedding-3-small",  # use same model for both to compare retrieval
    ).data[0].embedding

    results = namespace.query(vector=embedding, query_text=query_text, search_mode="hybrid", top_k=k)
    retrieved_ids = {r.id for r in results}
    relevant_set = set(relevant_ids)
    return len(retrieved_ids & relevant_set) / len(relevant_set)

def evaluate(namespace, eval_set, k=10):
    scores = [
        recall_at_k(namespace, q["query"], q["relevant_doc_ids"], k)
        for q in eval_set
    ]
    return sum(scores) / len(scores)

prod_recall  = evaluate(prod_ns, eval_queries)
fork_recall  = evaluate(fork_ns, eval_queries)

print(f"Production recall@10:  {prod_recall:.3f}")
print(f"Fork recall@10:        {fork_recall:.3f}")
print(f"Delta:                 {fork_recall - prod_recall:+.3f}")

Step 6: Route a percentage of live traffic to the fork

Before committing, route a small percentage of real queries to the fork to measure production quality:

import random

def query_with_ab(user_id: str, query_vector, query_text: str, top_k: int = 10):
    # Route 10% of users to the fork
    use_fork = (hash(user_id) % 100) < 10

    namespace = fork_ns if use_fork else prod_ns
    results = namespace.query(vector=query_vector, query_text=query_text, top_k=top_k)

    # Log which variant was used for analysis
    log_variant(user_id, "fork" if use_fork else "prod", results)
    return results

Step 7: Promote or rollback

If the fork wins, promote it atomically:

# Promotion is a metadata swap — no data movement, instant
client.promote_fork(fork_name="knowledge-base-text3-large-eval")
print("Fork promoted to production.")

If the fork doesn’t win, rollback by deleting it:

# Rollback: delete the fork. Production namespace is unchanged.
client.delete_namespace("knowledge-base-text3-large-eval")
print("Fork deleted. Production is unchanged.")

Checklist for safe model migrations

• Fork production before any re-indexing work starts
• Use a representative eval set with at least 100 queries
• Compare recall@10 and MRR (mean reciprocal rank)
• Run a live A/B experiment with real traffic before promoting
• Monitor query latency after promotion (new model dimensions may affect HNSW performance)
• Keep the fork for 7 days post-promotion as a rollback option

Cost of branching

The fork costs nothing for storage beyond the new objects written to it. If you’re re-indexing 10M vectors, you’re paying for 10M new embeddings (embedding API cost) and ~5-6GB of new objects (storage cost) — not for duplicating the original 10M vectors.