opensearch-project/opensearch-neural-sparse-encoding-v1
The model should be selected considering search relevance, model inference and retrieval efficiency(FLOPS). We benchmark models' zero-shot performance on a subset of BEIR benchmark: TrecCovid,NFCorpus,NQ,HotpotQA,FiQA,ArguAna,Touche,DBPedia,SCIDOCS,FEVER,Climate FEVER,SciFact,Quora.
Overview
Benchmarks
CQADupstackPhysicsRetrieval
Duplicate question retrieval from StackExchange Physics
Corpus: 38,314 Queries: 1,039
Quality
ndcg at 10 0.3652
map at 10 0.3090
mrr at 10 0.3574
Performance L4 b1 c16
Corpus 28.5K tok/s
Corpus p50 69.1ms
Query 2.4K tok/s
Query p50 66.5ms
CosQA
Code search with natural language queries
Corpus: 6,267 Queries: 500
Quality
ndcg at 10 0.2435
map at 10 0.1868
mrr at 10 0.1985
Performance L4 b1 c16
Corpus 15.2K tok/s
Corpus p50 52.0ms
Query 2.0K tok/s
Query p50 46.6ms
FiQA2018
Financial opinion mining and question answering
Corpus: 57,599 Queries: 648
Quality
ndcg at 10 0.2632
map at 10 0.2004
mrr at 10 0.3273
Performance L4 b1 c16
Corpus 40.2K tok/s
Corpus p50 59.4ms
Query 3.6K tok/s
Query p50 50.0ms
LegalBenchConsumerContractsQA
Question answering on consumer contracts
Corpus: 153 Queries: 396
Quality
ndcg at 10 0.7272
map at 10 0.6626
mrr at 10 0.6635
Performance L4 b1 c16
Corpus 89.1K tok/s
Corpus p50 86.7ms
Query 5.1K tok/s
Query p50 50.1ms
NFCorpus
Biomedical literature search from NutritionFacts.org
Corpus: 3,593 Queries: 323
Quality
ndcg at 10 0.3600
map at 10 0.1403
mrr at 10 0.5690
Performance L4 b1 c16
Corpus 58.7K tok/s
Corpus p50 76.0ms
Query 1.5K tok/s
Query p50 51.0ms
SCIDOCS
Citation prediction, document classification, and recommendation for scientific papers
Corpus: 25,656 Queries: 1,000
Quality
ndcg at 10 0.1537
map at 10 0.0877
mrr at 10 0.2688
Performance L4 b1 c16
Corpus 44.8K tok/s
Corpus p50 63.2ms
Query 3.5K tok/s
Query p50 49.5ms
SciFact
Scientific claim verification using research literature
Corpus: 5,183 Queries: 300
Quality
ndcg at 10 0.6998
map at 10 0.6522
mrr at 10 0.6684
Performance L4 b1 c16
Corpus 54.1K tok/s
Corpus p50 75.7ms
Query 5.2K tok/s
Query p50 50.0ms
StackOverflowQA
Programming question answering from Stack Overflow
Corpus: 19,931 Queries: 1,994
Quality
ndcg at 10 0.7703
map at 10 0.7392
mrr at 10 0.7392
Performance L4 b1 c16
Corpus 52.6K tok/s
Corpus p50 68.8ms
Query 65.6K tok/s
Query p50 71.9ms