answerdotai/answerai-colbert-small-v1
Primitive: /score · Score ·
BERT
answerai-colbert-small-v1 is a new, proof-of-concept model by Answer.AI, showing the strong performance multi-vector models with the new JaColBERTv2.5 training recipe and some extra tweaks can reach, even with just 33 million parameters.
Overview
Hardware: — drives latency, throughput & cost
| Size | 33M params |
|---|---|
| Tasks | /encode · /score |
| License | apache-2.0 |
| Languages | en |
| Latency | 122 ms |
| Throughput | 1.7K tok/s |
| Cost | $0.128 /1M tok |
Cost is approximate — computed from list GPU prices; your actual price depends on the provider you deploy SIE with.
Scoring
| Inputs | text |
|---|---|
| Max sequence length | 512 |
Benchmarks
AskUbuntuDupQuestions
Duplicate question detection from AskUbuntu
Corpus: 6,743 Queries: 360
muvera
Quality
ndcg at 10 0.6331
map at 10 0.4738
mrr at 10 0.7266
default
Quality
ndcg at 10 0.6259
map at 10 0.4680
mrr at 10 0.7165
CMedQAv1-reranking
Chinese medical question answering reranking (v1)
Corpus: 100,000 Queries: 2,000
muvera
Quality
ndcg at 10 0.2192
map at 10 0.1670
mrr at 10 0.2283
default
Quality
ndcg at 10 0.2088
map at 10 0.1602
mrr at 10 0.2210
CMedQAv2-reranking
Chinese medical question answering reranking (v2)
Corpus: 108,000 Queries: 4,000
muvera
Quality
ndcg at 10 0.2356
map at 10 0.1835
mrr at 10 0.2397
default
Quality
ndcg at 10 0.2311
map at 10 0.1838
mrr at 10 0.2409
CQADupstackPhysicsRetrieval
Duplicate question retrieval from StackExchange Physics
Corpus: 38,314 Queries: 1,039
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.3646
mrr at 10 0.4214
ndcg at 10 0.4155
Performance L4-SPOT b1 c16
Corpus 3.9K tok/s
Corpus p50 203.2ms
Query 186 tok/s
Query p50 300.7ms
Performance L4 b1 c16
Corpus 37.3K tok/s
Corpus p50 54.7ms
Query 3.6K tok/s
Query p50 47.1ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.3163
mrr at 10 0.3688
ndcg at 10 0.3702
CosQA
Code search with natural language queries
Corpus: 6,267 Queries: 500
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.2256
mrr at 10 0.2148
ndcg at 10 0.2897
Performance L4-SPOT b1 c16
Corpus 1.1K tok/s
Corpus p50 345.4ms
Query 102 tok/s
Query p50 466.2ms
Performance L4 b1 c16
Corpus 15.7K tok/s
Corpus p50 53.9ms
Query 2.0K tok/s
Query p50 47.4ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1953
mrr at 10 0.2087
ndcg at 10 0.2536
FiQA2018
Financial opinion mining and question answering
Corpus: 57,599 Queries: 648
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.3337
mrr at 10 0.4965
ndcg at 10 0.4103
Performance L4-SPOT b1 c16
Corpus 3.4K tok/s
Corpus p50 384.8ms
Query 174 tok/s
Query p50 547.3ms
Performance L4 b1 c16
Corpus 43.1K tok/s
Corpus p50 59.1ms
Query 3.7K tok/s
Query p50 50.0ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.2223
mrr at 10 0.3540
ndcg at 10 0.2900
LegalBenchConsumerContractsQA
Question answering on consumer contracts
Corpus: 153 Queries: 396
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.7315
mrr at 10 0.7315
ndcg at 10 0.7840
Performance L4-SPOT b1 c16
Corpus 11.2K tok/s
Corpus p50 286.1ms
Query 254 tok/s
Query p50 500.2ms
Performance L4 b1 c16
Corpus 83.1K tok/s
Corpus p50 83.9ms
Query 4.8K tok/s
Query p50 52.2ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.6346
mrr at 10 0.6346
ndcg at 10 0.6991
MMarcoReranking
Multilingual MARCO passage reranking (Chinese)
muvera
Quality
ndcg at 10 0.0725
map at 10 0.0502
mrr at 10 0.0507
default
Quality
map at 10 0.0602
mrr at 10 0.0602
NFCorpus
Biomedical literature search from NutritionFacts.org
Corpus: 3,593 Queries: 323
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1437
mrr at 10 0.5860
ndcg at 10 0.3706
Performance L4-SPOT b1 c16
Corpus 5.7K tok/s
Corpus p50 300.1ms
Query 210 tok/s
Query p50 178.7ms
Performance L4 b1 c16
Corpus 60.7K tok/s
Corpus p50 69.4ms
Query 1.4K tok/s
Query p50 52.3ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1186
mrr at 10 0.5151
ndcg at 10 0.3192
SCIDOCS
Citation prediction, document classification, and recommendation for scientific papers
Corpus: 25,656 Queries: 1,000
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.1045
mrr at 10 0.3076
ndcg at 10 0.1777
Performance L4-SPOT b1 c16
Corpus 2.9K tok/s
Corpus p50 501.5ms
Query 222 tok/s
Query p50 353.0ms
Performance L4 b1 c16
Corpus 48.4K tok/s
Corpus p50 57.9ms
Query 3.8K tok/s
Query p50 46.7ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.0973
mrr at 10 0.2854
ndcg at 10 0.1681
SciFact
Scientific claim verification using research literature
Corpus: 5,183 Queries: 300
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.7014
mrr at 10 0.7120
ndcg at 10 0.7404
Performance L4-SPOT b1 c16
Corpus 4.9K tok/s
Corpus p50 443.1ms
Query 332 tok/s
Query p50 430.3ms
Performance L4 b1 c16
Corpus 61.5K tok/s
Corpus p50 63.3ms
Query 5.2K tok/s
Query p50 50.5ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.6522
mrr at 10 0.6645
ndcg at 10 0.6933
StackOverflowQA
Programming question answering from Stack Overflow
Corpus: 19,931 Queries: 1,994
default_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.5125
mrr at 10 0.5125
ndcg at 10 0.5457
Performance L4-SPOT b1 c16
Corpus 4.4K tok/s
Corpus p50 379.7ms
Query 5.2K tok/s
Query p50 432.3ms
Performance L4 b1 c16
Corpus 55.3K tok/s
Corpus p50 60.2ms
Query 73.2K tok/s
Query p50 64.7ms
muvera_candidates-k-50_candidates-model-Alibaba-NLP__gte-multilingual-base
Quality
map at 10 0.4498
mrr at 10 0.4498
ndcg at 10 0.4880
T2Reranking
Chinese passage ranking benchmark
muvera
Quality
ndcg at 10 0.6591
map at 10 0.4793
mrr at 10 0.6847
default
Quality
ndcg at 10 0.6821
map at 10 0.5041
mrr at 10 0.7178
Compare (0)Compare →