How We Made RAG Indexing Faster With an Adaptive Embedding Endpoint Pool

Imagine you are building a coding assistant that can understand an entire repository.

It needs to scan files, split code into chunks, turn those chunks into embeddings, and save them into a vector database. That sounds chill until your repo has thousands of files and your embedding model is running locally on a couple of machines.

Then one machine is fast.

The other one is... emotionally unavailable.

In our case, one embedding endpoint worked normally:

http://192.168.0.73:11434

The second endpoint sometimes took around 101 seconds for one batch:

http://192.168.0.87:11434

The funny part? The system had already completed a lot of embedding batches, but the database was still behind. It was not because the database was slow. It was because the indexing code was waiting for futures in the wrong order.

This article explains the problem, the fix, and the performance effect in plain English.

The Setup: Local Embeddings for a Codebase

In a RAG system, embeddings are the bridge between text and search.

You take something like:

public List<User> findActiveUsers() {
    return userRepository.findByStatus("ACTIVE");
}

And turn it into a vector:

[0.013, -0.221, 0.884, ..., 0.031]

Later, when someone asks:

Where do we load active users?

The system can search by meaning instead of exact words.

For code repositories, this embedding step can be one of the slowest parts of indexing. So instead of using one endpoint, we use an embedding endpoint pool:

bf:
  embedding:
    multi-endpoint:
      endpoints:
        - url: http://192.168.0.73:11434
          model: qwen3-embedding:0.6b
        - url: http://192.168.0.87:11434
          model: qwen3-embedding:0.6b

Both endpoints run the same embedding model. The goal is simple:

Use both machines, finish indexing faster.

But real life always brings snacks and chaos.

The Problem: One Slow Endpoint Can Freeze the Whole Line

The first version looked reasonable:

1. Split code into embedding batches.
2. Submit batches to workers.
3. Wait for futures.
4. Insert embeddings into the database.

The trap was step 3.

If you wait for futures in the same order you submitted them, one slow batch near the beginning blocks everything behind it.

Think of it like a coffee shop:

• Order #1 is a complicated drink that takes 10 minutes.
• Orders #2 to #20 are simple black coffees and are already done.
• But the cashier refuses to hand out #2 to #20 until #1 is finished.

That is called head-of-line blocking.

In our real indexing run:

• The actuator already showed about 40 completed embedding batches.
• The database had only about 700 embeddings inserted.
• The pipeline was waiting for an earlier slow future.
• One endpoint had tail latency up to about 101 seconds per batch.

The system was doing work, but the results were stuck in traffic.

The First Fix: Consume Futures by Completion Order

The first big improvement was simple:

Don't wait for batch #1 if batch #2 is already done.

Instead of this idea:

for (Future<EmbeddingBatchResult> future : futures) {
    EmbeddingBatchResult result = future.get(); // waits in submit order
    insertEmbeddings(result);
}

Use a completion service:

ExecutorCompletionService<EmbeddingBatchResult> completionService =
        new ExecutorCompletionService<>(executor);

for (EmbeddingBatch batch : batches) {
    completionService.submit(() -> embedBatch(batch));
}

for (int i = 0; i < batches.size(); i++) {
    Future<EmbeddingBatchResult> completed = completionService.take();
    EmbeddingBatchResult result = completed.get();
    insertEmbeddings(result);
}

Now the database gets results as soon as they are ready.

If batch #17 finishes first, batch #17 gets inserted first. That is totally fine because embeddings are independent rows. The database does not care about the original batch order.

Result:

• Fast endpoint results no longer wait behind slow endpoint results.
• Database inserts start earlier.
• Progress metrics become more honest.
• One slow tail no longer freezes the visible indexing progress.

This alone removes a huge source of wasted time.

The Second Fix: Don't Treat All Endpoints Equally

Round-robin sounds fair.

But "fair" is not always fast.

If one endpoint usually returns in a few seconds and another sometimes takes 100 seconds, sending them equal work is like putting a bicycle and a sports car into the same delivery race and saying:

You both get exactly 50% of the packages. Good luck.

That is cute. It is also slow.

The endpoint pool now uses adaptive routing. Every endpoint gets a score, and the lowest score wins the next batch.

Conceptually:

score =
  latency_ewma_ms
  * slow_success_penalty
  * (1 + load_ratio)
  / priority

Let's unpack that.

In Java-like code:

double score() {
    double load = adaptiveLimit > 0
            ? (double) inflight / adaptiveLimit
            : 1.0;

    double slowSuccessPenalty = 1.0 + slowSuccesses;

    return (ewmaLatencyMillis * slowSuccessPenalty * (1.0 + load)) / priority;
}

Lower score means:

This endpoint is probably the best place for the next batch.

Config Order Is the First Hint

When the system starts, it does not yet know which endpoint is faster.

So we let the configuration order act as a hint:

endpoints:
  - url: http://192.168.0.73:11434
    model: qwen3-embedding:0.6b
  - url: http://192.168.0.87:11434
    model: qwen3-embedding:0.6b

The first endpoint gets a higher starting priority:

So if you already know one machine is faster, put it first.

You can also set priority manually:

endpoints:
  - url: http://192.168.0.73:11434
    model: qwen3-embedding:0.6b
    priority: 10
  - url: http://192.168.0.87:11434
    model: qwen3-embedding:0.6b
    priority: 1

But priority is only a starting bias.

If the "fast" machine becomes overloaded, the scoring model will notice. If the "slow" machine starts behaving well, it can earn more work again.

The Important Part: Slow Does Not Mean Dead

This was the key design decision.

At first, it is tempting to quarantine a slow endpoint:

You took 101 seconds? Go sit in the corner for 10 minutes.

But that is not always smart.

A slow endpoint can still be useful when the fast endpoint is busy. If the fast endpoint is already full, sending one extra batch to the slow endpoint might still improve total throughput.

So the pool separates two cases:

The code idea:

void recordSuccess(long durationMillis) {
    failures = 0;
    updateLatencyEwma(durationMillis);

    if (durationMillis >= slowLatencyThresholdMillis) {
        slowSuccesses++;
        decreaseLimit();
        log.info("Endpoint kept as overflow after slow success");
        return;
    }

    slowSuccesses = 0;

    if (durationMillis <= targetLatencyMillis) {
        increaseLimit();
    }
}

And failure handling stays separate:

void recordFailure() {
    failures++;
    slowSuccesses = 0;
    decreaseLimit();

    if (failures >= failuresBeforeSkip) {
        skipFor(skipDurationMillis, "failures");
    }
}

This is the difference between:

Slow but useful

and:

Broken and wasting time

Those are not the same thing.

Adaptive Concurrency: Give More Work to the Endpoint That Proves Itself

Each endpoint also has its own concurrency limit.

If an endpoint is fast, it can get more parallel requests:

if (durationMillis <= targetLatencyMillis
        && adaptiveLimit < maxConcurrencyPerEndpoint) {
    adaptiveLimit++;
}

If it becomes slow, the limit goes down:

if (durationMillis > targetLatencyMillis * 2
        && adaptiveLimit > minConcurrencyPerEndpoint) {
    adaptiveLimit = Math.max(minConcurrencyPerEndpoint, adaptiveLimit / 2);
}

Simple idea:

• Fast machine: more lanes.
• Slow machine: fewer lanes.
• Failed machine: temporary timeout.

This keeps the system self-balancing without needing a human to babysit every indexing run.

What the Full Flow Looks Like

Here is the new mental model:

Repository
   ↓
Scan files
   ↓
Parse code entities
   ↓
Split into embedding batches
   ↓
Adaptive endpoint pool
   ↓
Fast endpoint gets most work
Slow endpoint gets overflow
   ↓
Completed batches return in any order
   ↓
Insert immediately into vector database

The pool does not try to be "nice" to endpoints.

It tries to be useful to the indexing pipeline.

The Performance Effect

The original bottleneck was not just raw embedding speed.

It was the combination of:

1. Uneven endpoint latency.
2. Submission-order future joining.
3. Database inserts waiting behind slow earlier batches.

The observed bad case:

Fast endpoint: normal
Slow endpoint: tail batches up to ~101s
Actuator: ~40 completed embedding batches
Database: only ~700 embeddings inserted

That means the system had useful completed work, but the insert stage could not consume it yet.

After the change:

• Completed fast batches are inserted immediately.
• Slow endpoint no longer blocks later completed batches.
• The fast endpoint gets preferred because of lower latency and higher config priority.
• The slow endpoint stays active as overflow instead of being thrown away.
• Repeated failures still trigger temporary quarantine.

In practical terms, this removes the worst kind of waiting:

"We have completed data, but we refuse to insert it because an older batch is still running."

That is the kind of performance bug that does not always show up in simple benchmarks, but hurts badly in real indexing.

Why This Pattern Matters

This is not only about one project or two local Ollama servers.

The same pattern appears everywhere:

• Local AI servers with different GPUs.
• Cloud endpoints in different regions.
• Mixed CPU/GPU inference workers.
• One model replica under load while another is idle.
• RAG indexing jobs with thousands of chunks.

The general lesson:

Do not assume all AI endpoints are equal. Measure them, score them, and route work based on reality.

Also:

Do not wait for async work in the wrong order.

If tasks are independent, consume them as they complete.

A Small Checklist for Your Own RAG Pipeline

If you are building something similar, check these things:

• Do you have per-endpoint latency metrics?
• Do you know p95 and p99 latency, not just averages?
• Can one slow request block later completed work?
• Do you insert completed embedding batches immediately?
• Do you separate "slow success" from "failure"?
• Can a fast endpoint automatically get more work?
• Can a slow endpoint still be used as overflow?
• Can a broken endpoint be skipped temporarily?

If the answer to most of these is "no", your RAG indexing pipeline probably has free speed waiting to be unlocked.

Final Takeaway

The best optimization here was not some fancy GPU trick.

It was better traffic control.

We stopped treating every endpoint as equal, stopped waiting for results in the wrong order, and stopped punishing slow-but-successful workers like they were broken.

That gave us a pipeline that behaves more like a good delivery app:

• Send most orders to the fastest driver.
• Use slower drivers when demand is high.
• Stop sending orders to drivers who keep failing.
• Deliver completed orders immediately.

For RAG systems, that kind of boring engineering is often where the real speed comes from.

Published on 4/11/2026