Domain Embedding Fine-Tuning — Adapting BGE to Finance and Risk Documents
This post addresses that gap. Not by switching models, but by teaching the one we have.
The IOS architecture blueprint I published earlier ended with an honest admission: the embedding model hasn’t been domain-fine-tuned yet. We’re running BGE-base-en-v1.5 out of the box on regulatory filings, Basel III disclosures, and capital adequacy reports — texts it was never trained to distinguish.
This post addresses that gap. Not by switching models, but by teaching the one we have.
Embedding Space: Before and After Domain Fine-Tuning
The Intuition: Why Semantics Break Down at Domain Boundaries
General-purpose embedding models are trained on the internet — Wikipedia, Common Crawl, web text, books. They learn that “bank” and “financial institution” are semantically close. They learn that “capital” relates to “funding” and “investment.”
But in finance and risk, “capital” almost exclusively means regulatory capital — Tier 1, CET1, Additional Tier 1. “Leverage” doesn’t mean using your position; it means the leverage ratio under Basel III. “Exposure” is a specific risk quantity, not a general concept. “Liquidity” has a defined meaning under LCR and NSFR frameworks that is narrower and more precise than everyday usage.
When a general encoder sees a query like “What is the CET1 ratio for Q3 2024?”, it encodes it near finance-adjacent vectors. But so does “What was the capital raised in Series B?”, “How much equity does the company hold?”, and “What is the net worth of the organization?” — none of which retrieve the right chunks from a regulatory filing.
The result is retrieval that is semantically adjacent but factually wrong. You pull chunks about equity positions instead of risk-weighted asset calculations. Your analysis engine gets the right document type, the wrong passage, and produces hallucinated or misattributed numbers.
The fix is not prompt engineering. It’s not a better reranker. It’s teaching the encoder what these concepts actually mean in context. That’s what domain fine-tuning does.
The Mechanics: How Contrastive Fine-Tuning Works
Embedding models are fine-tuned using contrastive learning: you show the model pairs of texts and train it to pull semantically related pairs closer together in embedding space, and push unrelated pairs apart.
The training signal comes from triplets: an anchor (a query or text), a positive (a passage that answers the query), and one or more negatives (passages that look plausible but don’t answer it).
The loss function most commonly used for this — and the one we use for IOS — is MultipleNegativesRankingLoss (MNRL). Given a batch of (anchor, positive) pairs, MNRL treats every other positive in the batch as an in-batch negative. The model learns to score the correct positive higher than all the others.
The gradient update is:
L = -log( exp(sim(a, p+) / τ) / Σ exp(sim(a, pj) / τ) )
Where a is the anchor, p+ is the positive, pj are all other positives in the batch used as negatives, and τ is a temperature parameter (typically 0.05). Larger batches give better in-batch negatives and faster convergence — use as large a batch as memory allows.
For IOS, we add a second technique: Matryoshka Representation Learning (MRL). The standard BGE-base outputs 768-dimensional vectors. MRL trains the model to produce good embeddings at multiple truncated dimensions simultaneously — 768, 512, 384, 256, 128. The key insight: with MRL, you can truncate to 256 dimensions at inference and lose only ~3% of retrieval quality versus full 768-dim, while cutting storage and compute roughly in half.
For a system that will index hundreds of thousands of regulatory filings, that tradeoff is material.
The Data Problem: What You Train On Determines What You Learn
Fine-tuning is only as good as the training data. For IOS, we need pairs that distinguish finance-specific meaning from general language.
We build the dataset in three ways:
1. Hard positive mining from regulatory corpora. Take Basel III documents, SEC filings, IFRS standards. Extract question-answer pairs using the structure of the documents: table headers become queries, corresponding rows become positives. Section titles become queries, section bodies become positives. This produces thousands of (query, positive) pairs grounded in actual regulatory text.
2. Synthetic pair generation using an LLM. For each document chunk, generate 3-5 queries that the chunk specifically answers. This is the fastest path to volume. The risk is synthetic queries that don’t reflect real analyst language — mitigate by reviewing a sample and iterating on the generation prompt.
3. Triplet mining with hard negatives. Use the general BGE model to find the top-k nearest neighbors for each anchor. Any neighbor that is not the correct positive is a hard negative — it looks similar in general embedding space but shouldn’t be retrieved for this query. Hard negatives make the model learn exactly the distinctions that matter for finance.
from sentence_transformers import SentenceTransformer, InputExample, losses
from sentence_transformers.evaluation import InformationRetrievalEvaluator
from torch.utils.data import DataLoader
from typing import List, Dict, Tuple
import json
def build_mnrl_dataset(
regulatory_pairs: List[Tuple[str, str]], # (query, positive_passage)
synthetic_pairs: List[Tuple[str, str]],
) -> List[InputExample]:
"""
Combine regulatory and synthetic pairs into InputExample format.
MNRL only needs (anchor, positive) — negatives are in-batch.
"""
examples = []
for query, positive in regulatory_pairs + synthetic_pairs:
examples.append(InputExample(texts=[query, positive]))
return examples
def fine_tune_bge(
base_model_name: str = "BAAI/bge-base-en-v1.5",
train_examples: List[InputExample] = None,
eval_queries: Dict[str, str] = None, # qid → query text
eval_corpus: Dict[str, str] = None, # doc_id → passage text
eval_relevant: Dict[str, set] = None, # qid → {relevant doc_ids}
output_path: str = "models/bge-finance-v1",
epochs: int = 3,
batch_size: int = 64,
warmup_ratio: float = 0.1,
) -> SentenceTransformer:
model = SentenceTransformer(base_model_name)
# MultipleNegativesRankingLoss with in-batch negatives
train_loss = losses.MultipleNegativesRankingLoss(model)
# Optionally wrap with MatryoshkaLoss for multi-dim training
matryoshka_dims = [768, 512, 256, 128]
train_loss = losses.MatryoshkaLoss(
model, train_loss, matryoshka_dims=matryoshka_dims
)
train_dataloader = DataLoader(
train_examples, shuffle=True, batch_size=batch_size
)
evaluator = InformationRetrievalEvaluator(
queries=eval_queries,
corpus=eval_corpus,
relevant_docs=eval_relevant,
name="finance-retrieval",
show_progress_bar=False,
)
warmup_steps = int(len(train_dataloader) * epochs * warmup_ratio)
model.fit(
train_objectives=[(train_dataloader, train_loss)],
evaluator=evaluator,
epochs=epochs,
warmup_steps=warmup_steps,
evaluation_steps=200,
output_path=output_path,
show_progress_bar=True,
save_best_model=True,
)
return model
A few important details in that code:
MatryoshkaLoss wraps MNRL: the outer loss handles the multi-dimension training; the inner loss handles the contrastive objective at each dimension.
save_best_model=True: evaluation runs every 200 steps; only the checkpoint with the best nDCG@10 on the finance eval set is saved.batch_size=64: larger is better for MNRL. On a single A100 80GB, you can run batch_size=256 with BGE-base. On a T4 16GB, 64 is the practical ceiling.
Generating Synthetic Training Pairs for Regulatory Documents
The bottleneck for domain fine-tuning is almost always data, not compute. Here’s the synthetic generation loop we use:
from openai import OpenAI
from typing import List, Tuple
client = OpenAI()
QUERY_GENERATION_PROMPT = """You are a financial analyst reviewing regulatory documents.
Given the following passage from a regulatory filing, generate {n_queries} questions that:
1. Can ONLY be answered using this specific passage
2. Use the technical language a risk analyst or compliance officer would use
3. Are specific enough that a general finance article would NOT answer them
4. Include at least one question about specific numerical thresholds, ratios, or requirements
Passage:
{passage}
Return ONLY a JSON array of question strings. No other text."""
def generate_query_pairs(
passages: List[str],
n_queries_per_passage: int = 4,
model: str = "gpt-4o-mini",
) -> List[Tuple[str, str]]:
pairs = []
for passage in passages:
response = client.chat.completions.create(
model=model,
messages=[{
"role": "user",
"content": QUERY_GENERATION_PROMPT.format(
n_queries=n_queries_per_passage,
passage=passage[:1500] # truncate very long passages
)
}],
temperature=0.7,
response_format={"type": "json_object"},
)
import json
try:
queries = json.loads(response.choices[0].message.content)
if isinstance(queries, list):
for q in queries:
if isinstance(q, str) and len(q) > 10:
pairs.append((q, passage))
except (json.JSONDecodeError, KeyError):
continue # skip malformed outputs
return pairs
For IOS, we generated 12,400 pairs from 3,100 passages across Basel III, IFRS 9, several hundred SEC filings, and the EBA stress test framework documents. With 400 hard-mined triplets added, the dataset took 3 hours to produce and the fine-tuning run took 47 minutes on a single A100.
Evaluating the Fine-Tuned Model
Before deploying, you need a held-out evaluation set that reflects real analyst queries — not the same distribution as the training data.
For IOS, the eval set was assembled separately: 150 queries written by a compliance consultant against a set of 2,800 document chunks the model had never seen. Evaluation metrics: nDCG@10, Recall@5, MRR.
from sentence_transformers.evaluation import InformationRetrievalEvaluator
def run_comparative_eval(
models: dict, # {"name": SentenceTransformer}
eval_queries: dict,
eval_corpus: dict,
eval_relevant: dict,
):
results = {}
for name, model in models.items():
evaluator = InformationRetrievalEvaluator(
queries=eval_queries,
corpus=eval_corpus,
relevant_docs=eval_relevant,
name=name,
score_functions={"cos_sim": lambda a, b: (a * b).sum(dim=-1)},
)
score = evaluator(model, output_path="./eval_results")
results[name] = score
print(f"{name}: nDCG@10={score:.4f}")
return results
The results on our eval set:
Model nDCG@10 Recall@5 MRR BGE-base-en-v1.5 (general) 0.57 0.61 0.53 BGE-base + BM25 hybrid 0.62 0.66 0.59 BGE-finance-v1 (768-dim) 0.81 0.84 0.77 BGE-finance-v1 (256-dim MRL) 0.78 0.81 0.74
The jump from general to domain-fine-tuned is 24 percentage points on nDCG@10. The 256-dim Matryoshka truncation costs 3 points and roughly halves the vector storage requirement. For production, 256-dim is the right choice.
Fine-Tuning Pipeline, Retrieval Quality, and Matryoshka Tradeoff
Production Considerations
Re-indexing. When you switch embedding models, every document in the vector database must be re-encoded. For IOS at current scale (circa 40,000 chunks), this takes about 12 minutes on a single GPU. Plan for downtime or run dual-index with a cutover.
Embedding drift. As you continue fine-tuning on new data — new regulatory frameworks, new filing formats — the embedding space shifts. Documents indexed with v1 won’t be directly comparable to documents encoded with v2. Either re-encode the whole corpus on model updates, or maintain model version tags on every vector and run separate indices per model version.
Query-document asymmetry. BGE was pre-trained with separate instruction prefixes for queries versus documents: "Represent this sentence for searching relevant passages:" for queries, no prefix for documents. Keep this asymmetry when fine-tuning — it’s part of the model’s architecture.
When fine-tuning doesn’t help. If your training pairs don’t reflect the actual distribution of queries your system receives, fine-tuning will overfit to the synthetic distribution. The signal here is a large gap between fine-tuned eval performance and production retrieval quality. The fix is to capture real analyst queries and use those as the primary training source.
Monitoring in production. Add a retrieval quality signal to your eval pipeline: a set of 50 “golden” (query, expected_chunk_id) pairs that run on every deployment. If nDCG@10 drops more than 2 points versus the previous model version, block the deployment.
What This Means for IOS
The IOS architecture described yesterday had hybrid retrieval (BM25 + semantic) with a cross-encoder reranker. With general BGE, the semantic leg of hybrid search was underperforming — pulling plausible but incorrect chunks, leaving the reranker to compensate. The reranker was doing work the encoder should have done.
With BGE-finance-v1, the encoder pulls the right chunks. The reranker’s job becomes refinement rather than correction. The retrieval quality signal in our continuous eval pipeline goes from 0.62 to 0.81. That delta propagates upward: better retrieval → more accurate extractions → fewer low-confidence outputs → lower human review rate.
The embedding model is now the strongest link in the retrieval chain. Which means the constraint has moved.
The next bottleneck — as of today — is the agent layer. The architecture describes the shape of the analysis engine, but the implementation is early. The routing logic is simple, the tool set is incomplete, and the error recovery is nonexistent. That’s what we’re building next.
Action for today: Take your most common failed retrieval cases and ask whether they’re a query representation problem or a document representation problem. That distinction determines whether you need to fine-tune on the query side (harder, needs real query logs), the document side (easier, can use synthetic), or both.
I am publishing lessons and my thoughts as I’m constructing IntelligenceOS — a domain AI platform, for Finance Ops and Risk — built on a family of fine-tuned models — in public, one project at a time. You’ll get the real account: what I built, what broke, what I learned, and how it compounds into a production system.