Hybrid Search System for Research Papers
A search engine for research papers that combines lexical and semantic retrieval. It searches arXiv paper titles and abstracts, and runs as a public demo on Hugging Face Spaces.
The problem with single-approach retrieval
Pure dense retrieval missed exact-match queries like paper titles, author names and technical acronyms. Pure BM25 missed semantic matches, where the wording differed but the meaning was the same. Combining the two handles both kinds of query.
The retrieval pipeline. BM25 and dense retrieval each produce a ranked list; RRF merges them and a cross-encoder reorders the result.
How it works
- Lexical: BM25 (Okapi) over the paper corpus.
- Semantic: abstracts embedded with
bge-large-en-v1.5, normalized, and searched in a FAISS index by inner product, which equals cosine similarity for normalized vectors. - Fusion: each retriever returns five times more candidates than requested, and the two lists are merged with reciprocal rank fusion (). RRF only uses ranks, so the very different BM25 and cosine score scales never need to be calibrated against each other.
- Reranking: a cross-encoder (
bge-reranker-base) reads each query–paper pair together and reorders the fused results.
The demo lets you switch between BM25, dense and hybrid modes, which makes it easy to see the difference on the same query.
Key trade-off
Cross-encoder reranking improves precision but adds latency, because every candidate needs its own forward pass. Running it only on the fused top results (at most 20) keeps the cost manageable while still improving on first-stage retrieval.
Stack
Python · FAISS · rank-bm25 · Sentence Transformers · Cross-Encoders · Gradio · Hugging Face Spaces