colbert-ir/colbertv2.0
Primitive: /score · Score ·
BERT
[](https://colab.research.google.com/github/stanford-futuredata/ColBERT/blob/main/docs/intro2new.ipynb)
Overview
Hardware: — drives latency, throughput & cost
| Size | 110M params |
|---|---|
| Tasks | /encode · /score |
| License | mit |
| Languages | en |
| Latency | 51 ms |
| Throughput | 3.8K tok/s |
| Cost | $0.058 /1M tok |
Cost is approximate — computed from list GPU prices; your actual price depends on the provider you deploy SIE with.
Scoring
| Inputs | text |
|---|---|
| Max sequence length | 512 |
Benchmarks
AskUbuntuDupQuestions
Duplicate question detection from AskUbuntu
Corpus: 6,743 Queries: 360
muvera
Quality
ndcg at 10 0.5987
map at 10 0.4373
mrr at 10 0.7052
default
Quality
ndcg at 10 0.5910
map at 10 0.4330
mrr at 10 0.6673
CMedQAv1-reranking
Chinese medical question answering reranking (v1)
Corpus: 100,000 Queries: 2,000
default
Quality
map at 10 0.1403
mrr at 10 0.1954
muvera
Quality
ndcg at 10 0.1494
map at 10 0.1051
mrr at 10 0.1518
CMedQAv2-reranking
Chinese medical question answering reranking (v2)
Corpus: 108,000 Queries: 4,000
default
Quality
map at 10 0.1534
mrr at 10 0.2081
muvera
Quality
ndcg at 10 0.1458
map at 10 0.1047
mrr at 10 0.1448
CQADupstackPhysicsRetrieval
Duplicate question retrieval from StackExchange Physics
Corpus: 38,314 Queries: 1,039
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.3402
mrr at 10 0.3958
ndcg at 10 0.3939
Performance L4 b1 c16
Corpus 33.2K tok/s
Corpus p50 58.6ms
Query 2.9K tok/s
Query p50 58.1ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.2352
mrr at 10 0.2745
ndcg at 10 0.2798
CosQA
Code search with natural language queries
Corpus: 6,267 Queries: 500
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1213
mrr at 10 0.1189
ndcg at 10 0.1611
Performance L4 b1 c16
Corpus 14.7K tok/s
Corpus p50 56.1ms
Query 1.8K tok/s
Query p50 51.8ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.0882
mrr at 10 0.0907
ndcg at 10 0.1161
FiQA2018
Financial opinion mining and question answering
Corpus: 57,599 Queries: 648
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.2511
mrr at 10 0.3879
ndcg at 10 0.3200
Performance L4 b1 c16
Corpus 38.0K tok/s
Corpus p50 59.5ms
Query 3.9K tok/s
Query p50 47.3ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1349
mrr at 10 0.2354
ndcg at 10 0.1867
LegalBenchConsumerContractsQA
Question answering on consumer contracts
Corpus: 153 Queries: 396
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.6988
mrr at 10 0.6996
ndcg at 10 0.7501
Performance L4 b1 c16
Corpus 77.2K tok/s
Corpus p50 96.9ms
Query 5.4K tok/s
Query p50 49.0ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.5618
mrr at 10 0.5618
ndcg at 10 0.6224
MMarcoReranking
Multilingual MARCO passage reranking (Chinese)
muvera
Quality
ndcg at 10 0.0590
map at 10 0.0464
mrr at 10 0.0464
default
Quality
map at 10 0.0843
mrr at 10 0.0899
NFCorpus
Biomedical literature search from NutritionFacts.org
Corpus: 3,593 Queries: 323
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1086
mrr at 10 0.4783
ndcg at 10 0.2922
Performance L4 b1 c16
Corpus 42.8K tok/s
Corpus p50 89.2ms
Query 1.4K tok/s
Query p50 57.1ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.0406
mrr at 10 0.2862
ndcg at 10 0.1476
SCIDOCS
Citation prediction, document classification, and recommendation for scientific papers
Corpus: 25,656 Queries: 1,000
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.0829
mrr at 10 0.2649
ndcg at 10 0.1433
Performance L4 b1 c16
Corpus 41.3K tok/s
Corpus p50 66.2ms
Query 3.7K tok/s
Query p50 48.2ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.0641
mrr at 10 0.2139
ndcg at 10 0.1137
SciFact
Scientific claim verification using research literature
Corpus: 5,183 Queries: 300
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.6358
mrr at 10 0.6484
ndcg at 10 0.6796
Performance L4 b1 c16
Corpus 53.2K tok/s
Corpus p50 72.8ms
Query 5.1K tok/s
Query p50 51.0ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.4665
mrr at 10 0.4793
ndcg at 10 0.5118
StackOverflowQA
Programming question answering from Stack Overflow
Corpus: 19,931 Queries: 1,994
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.4172
mrr at 10 0.4172
ndcg at 10 0.4480
Performance L4 b1 c16
Corpus 40.5K tok/s
Corpus p50 74.0ms
Query 62.4K tok/s
Query p50 75.6ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.2842
mrr at 10 0.2842
ndcg at 10 0.3207
T2Reranking
Chinese passage ranking benchmark
default
Quality
map at 10 0.4985
mrr at 10 0.7208
muvera
Quality
ndcg at 10 0.6191
map at 10 0.4351
mrr at 10 0.6426
Compare (0)Compare →