Private retrieval

Embedding and reranking models for private RAG

Compare models that encode content for search or rerank retrieved passages before generation. These are often smaller than the answering model, but index volume, candidate count and latency targets determine the useful system.

Selection approach

Choose for the workload, then size the complete deployment

Treat embedding and reranking as separate jobs. Benchmark retrieval recall first, then measure whether a reranker improves answer evidence enough to justify the extra inference stage.

01

What to compare

  • Language and domain coverage
  • Embedding dimension and index size
  • Instruction support and passage length
  • Need for a second-stage reranker

02

What changes the hardware

  • High-throughput document ingestion
  • Candidate count and reranker sequence length
  • Vector database RAM and storage outside GPU memory
  • Separate online and batch workers where needed

03

What to test before purchase

  • Recall on known-answer queries
  • Ranking quality for difficult near-duplicates
  • Index-build and refresh throughput
  • End-to-end answer citation accuracy

3 current models

Compare models by workload and GPU memory

Showing 3 models

Qwen

Qwen3-Embedding 8B

The largest Qwen3 embedding checkpoint, designed for multilingual dense retrieval, classification, clustering and text matching.

  • Embeddings & RAG
Minimum GPU memory
20GB GPU planning floor
Recommended starting system
Team 32
Licence
Apache License 2.0
See specifications and all 11 systems

BAAI

BGE-M3

A multilingual embedding model that supports dense, sparse and multi-vector retrieval with 1,024 dimensions and inputs up to 8,192 tokens.

  • Embeddings & RAG
Minimum GPU memory
4GB GPU planning floor, or CPU for light use
Recommended starting system
Team 32
Licence
MIT License
See specifications and all 11 systems

Qwen

Qwen3-Reranker 8B

An 8-billion-parameter multilingual reranker for scoring retrieved passages across more than 100 natural and programming languages.

  • Embeddings & RAG
Minimum GPU memory
24GB on one GPU
Recommended starting system
Team 32
Licence
Apache License 2.0
See specifications and all 11 systems

Model-to-hardware fit

Memory fit is the first gate, not the final recommendation

  1. 01Exact model version

    Use the precise model version, numerical format and complete software pipeline intended for production.

  2. 02Minimum memory

    Check whether it can load on one GPU or requires supported multi-GPU loading.

  3. 03Working headroom

    Allow for context, cache, batch, media encoders and concurrent users.

  4. 04Workload test

    Measure quality, latency and stability on representative work.

What a complete embeddings & rag deployment needs

Model weights are only one part of the system. Data access, runtime software, storage, monitoring and administrator handover also affect a reliable deployment.

Diagram showing an approved request, a local service, an approved store and a policy-controlled data path
Diagram showing an approved request, a local service, an approved store and a policy-controlled data path
A private deployment starts with the permitted data path, access policy and logging boundary. GPU Servers technical illustration.
A private deployment starts with the permitted data path, access policy and logging boundary. GPU Servers technical illustration.
GPU server remote management dashboard with system status, access logs and sensor monitoring panels
GPU server remote management dashboard with system status, access logs and sensor monitoring panels
Supplier screenshot of the platform management interface. The final management features and access policy depend on the ordered system. OEM supplier reference image.
Supplier screenshot of the platform management interface. The final management features and access policy depend on the ordered system. OEM supplier reference image.
Diagram showing approved documents moving through a searchable index to an answer with a source citation
Diagram showing approved documents moving through a searchable index to an answer with a source citation
A retrieval workflow should connect each useful answer to approved source material and defined refusal behaviour. GPU Servers technical illustration.
A retrieval workflow should connect each useful answer to approved source material and defined refusal behaviour. GPU Servers technical illustration.
Diagram of an evidence pack containing an asset schedule, burn-in record, health readings, workload test and admin guide
Diagram of an evidence pack containing an asset schedule, burn-in record, health readings, workload test and admin guide
A complete handover includes the supplied assets, test results, operating instructions and agreed follow-up work. GPU Servers technical illustration.
A complete handover includes the supplied assets, test results, operating instructions and agreed follow-up work. GPU Servers technical illustration.