RWKV-Evals

RWKV Evaluation Data presents RWKV's performance across various large language model benchmarks, including Uncheatable Eval, MMLU, RULER, and LongBench.

Uncheatable Eval Test

::: tip Uncheatable Eval is an "uncheatable evaluation" that uses real-time data, such as the latest academic papers and news articles, to assess the true modeling capabilities and generalization abilities of open-source large language models.

The result of the Uncheatable Eval test is the compression ratio; therefore, a lower score indicates better model performance. :::

Below is a comparison of Uncheatable Eval scores between RWKV and other models:

14B Parameter Models

Name	Params (B)	Average (lower=better)	ao3 english	bbc news	wikipedia english	arxiv computer science	arxiv physics	github cpp	github python
RWKV7-g0b-13.3b	13.269	6.843	9.848	8.202	7.636	7.108	7.380	4.026	3.892
Qwen3-14B-Base	14.768	6.845	10.569	8.445	7.942	7.001	7.210	3.439	3.312
gemma-3-12b-pt	12.187	6.945	10.540	7.914	7.607	7.286	7.387	3.883	3.997
Qwen2.5-14B	14.770	6.951	10.558	8.317	7.944	7.224	7.392	3.625	3.599
Mistral-Nemo-Base-2407	12.248	6.970	10.165	8.118	7.642	7.287	7.455	4.079	4.042
Motif-2-12.7B-Base	12.704	7.099	10.628	8.328	7.897	7.134	7.404	4.189	4.114
Llama-2-13b-hf	13.016	7.540	10.655	8.307	7.901	7.993	8.122	4.795	5.009

7B Parameter Models

Name	Params (B)	Average (lower=better)	ao3 english	bbc news	wikipedia english	arxiv computer science	arxiv physics	github cpp	github python
Qwen3-8B-Base	8.191	7.091	10.890	8.718	8.255	7.207	7.465	3.617	3.482
Meta-Llama-3-8B	8.030	7.162	10.619	8.295	7.785	7.536	7.541	4.174	4.181
RWKV7-g0a3-7.2b-20251029-ctx8192	7.199	7.222	10.164	8.480	7.996	7.440	7.747	4.378	4.347
Qwen2.5-7B	7.616	7.323	11.079	8.729	8.449	7.539	7.792	3.868	3.806
Falcon-H1-7B-Base	7.586	7.339	10.958	8.576	8.225	7.403	7.569	4.251	4.392
Mistral-7B-v0.1	7.242	7.406	10.662	8.306	7.976	7.745	7.903	4.612	4.635
Hunyuan-7B-Pretrain	7.505	7.541	11.509	8.987	8.499	7.653	8.108	4.201	3.829
falcon-mamba-7b	7.273	7.548	10.760	8.958	8.589	7.674	7.737	4.437	4.680
Zamba2-7B	7.357	7.582	10.702	8.627	8.074	7.843	8.124	4.833	4.869
Minitron-8B-Base	8.272	7.582	10.835	8.654	8.284	7.856	8.230	4.508	4.708
Olmo-3-1025-7B	7.298	7.595	11.101	8.784	8.522	7.490	7.947	4.930	4.394
RWKV-x060-World-7B-v3-20241112-ctx4096	7.636	7.633	10.629	8.753	8.288	7.936	8.109	4.786	4.929

3B Parameter Models

Name	Params (B)	Average (lower=better)	ao3 english	bbc news	wikipedia english	arxiv computer science	arxiv physics	github cpp	github python
RWKV7-g1a4-2.9b-20251118-ctx8192	2.948	7.486	10.481	8.800	8.310	7.712	8.072	4.553	4.474
Llama-3.2-3B	3.213	7.643	11.219	8.701	8.365	7.928	8.065	4.661	4.562
Qwen2.5-3B	3.086	7.722	11.575	9.139	8.895	7.911	8.220	4.203	4.113
SmolLM3-3B-Base	3.075	7.784	11.187	8.905	8.611	8.097	8.631	4.513	4.546
RWKV-x070-World-2.9B-v3-20250211-ctx4096	2.948	7.800	10.812	8.909	8.501	8.049	8.307	4.955	5.066
stablelm-3b-4e1t	2.795	7.907	11.211	8.815	8.434	8.299	8.476	4.906	5.207
Falcon-H1-3B-Base	3.149	7.936	11.685	9.158	8.910	7.891	8.161	4.832	4.917
recurrentgemma-2b	2.683	8.052	11.632	8.951	8.835	8.401	8.488	4.897	5.157
RWKV-x060-World-3B-v2.1-20240417-ctx4096	3.100	8.147	11.005	9.161	8.815	8.451	8.559	5.479	5.561
mamba2attn-2.7b	2.698	8.204	11.436	9.246	8.947	8.474	8.236	5.336	5.751

1.5B Parameter Models

Name	Params (B)	Average (lower=better)	ao3 english	bbc news	wikipedia english	arxiv computer science	arxiv physics	github cpp	github python
Qwen3-1.7B-Base	1.721	7.965	12.016	9.743	9.352	7.936	8.350	4.260	4.095
RWKV7-g1b-1.5b-20251015-ctx8192	1.527	7.969	10.972	9.250	8.843	8.110	8.537	5.041	5.027
Qwen2.5-1.5B	1.544	8.124	12.114	9.562	9.393	8.270	8.646	4.502	4.384
RWKV-x070-World-1.5B-v3-20250127-ctx4096	1.527	8.231	11.273	9.320	8.965	8.431	8.758	5.385	5.483
SmolLM2-1.7B	1.711	8.298	11.536	9.373	9.351	8.547	9.047	5.080	5.152
Llama-3.2-1B	1.236	8.306	12.036	9.331	9.097	8.556	8.755	5.267	5.101
Index-1.9B	2.173	8.340	11.831	9.493	9.069	8.497	8.561	5.380	5.547
stablelm-2-1_6b	1.645	8.396	11.761	9.237	8.943	8.762	9.088	5.558	5.425
Falcon-H1-1.5B-Deep-Base	1.555	8.505	12.144	9.666	9.482	8.407	8.968	5.497	5.368
RWKV-x060-World-1B6-v2.1-20240328-ctx4096	1.600	8.564	11.434	9.555	9.276	8.822	8.990	5.906	5.968
Falcon-H1-1.5B-Base	1.555	8.639	12.287	9.796	9.645	8.507	9.089	5.635	5.514
mamba2-1.3b	1.344	8.699	11.944	9.710	9.463	8.925	8.714	5.851	6.286
RWKV-5-World-1B5-v2-20231025-ctx4096	1.578	8.715	11.595	9.731	9.451	8.977	9.103	6.039	6.110
mamba-1.4b-hf	1.372	8.806	12.026	9.783	9.552	9.081	8.836	5.958	6.408

MMLU Test

::: tip MMLU (Massive Multitask Language Understanding) is a widely recognized benchmark of general knowledge attained by AI models. It covers a broad range of topics from 57 different categories, covering elementary-level knowledge up to advanced professional subjects like law, physics, history, and computer science. :::

Model	MMLU	MMLU COT
RWKV7-g0b-13.3b	0.765	0.827
RWKV7-g0a3-7.2b	0.651	0.723
RWKV7-g1a4-2.9b	0.613	0.675
RWKV7-g1b-1.5b	0.505	0.542

MMLU Pro Test

::: tip MMLU-Pro dataset is a more robust and challenging massive multi-task understanding dataset tailored to more rigorously benchmark large language models' capabilities. This dataset contains 12K complex questions across various disciplines. :::

Model	MMLU-PRO	MMLU-PRO COT
RWKV7-g0b-13.3b	0.502	0.612
RWKV7-g0a3-7.2b	0.359	0.521
RWKV7-g1a4-2.9b	0.324	0.43
RWKV7-g1b-1.5b	0.222	0.292

GSM8K Test

::: tip GSM8K (Grade School Math 8K) is a dataset of 8.5K high quality linguistically diverse grade school math word problems. The dataset was created to support the task of question answering on basic mathematical problems that require multi-step reasoning. :::

Model	GSM8K
RWKV7-g0b-13.3b	0.923
RWKV7-g0a3-7.2b	0.839
RWKV7-g1a4-2.9b	0.773
RWKV7-g1b-1.5b	0.585

MATH500 Test

::: tip MATH-500 is an authoritative benchmark for measuring the mathematical reasoning capabilities of AI models. It contains 500 challenging mathematical problems covering algebra, geometry, calculus, probability, statistics, and other fields. :::

Model	MATH500
RWKV7-g0b-13.3b	0.768
RWKV7-g0a3-7.2b	0.678
RWKV7-g1a4-2.9b	0.482
RWKV7-g1b-1.5b	0.298

IFEval

::: tip IFEval (Instruction-Following Evaluation) is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". :::

Model	IFEval (strict prompt-level)
RWKV7-g0b-13.3b	0.689
RWKV7-g0a3-7.2b	0.58
RWKV7-g1a3-2.9b	0.51
RWKV7-g1b-1.5b	0.421

RULER Test

::: tip RULER is a new LLM testing method that optimizes and extends the NIAH (Needle In A Haystack) test. It includes four types of tasks: Enhanced Retrieval (Extended NIAH), Multi-hop Tracing, Information Aggregation (CWE, FWE), and QA with interference. :::

Enhanced Needle In A Haystack (NIAH)

RULER includes the Enhanced Needle In A Haystack (NIAH) test, divided into four sub-tasks to evaluate the model's retrieval capabilities:

Sub-task	Brief Description
Single NIAH (S-NIAH)	Tests the model's ability to handle a single input and a single target output.
Multi-keys NIAH (MK-NIAH)	Tests the model's ability to handle multiple key-value pairs, where each key is associated with a single output.
Multi-values NIAH (MV-NIAH)	Tests the model's ability to handle multiple key-value pairs, where each key is associated with multiple values or outputs.
Multi-queries NIAH (MQ-NIAH)	Tests the model's ability to synthesize and generate corresponding results under multiple query conditions.

Single NIAH test results are as follows:

Model	NIAH_single_1	NIAH_single_2	NIAH_single_3
RWKV-6-7B-v2.1-4k	100	98.67	95
Llama2-7B-4k	97.6	100	96.8
Mamba-2.8B-4k	100	19	1
Mamba-1.4B-4k	94	21	5
RWKV-6-3B-v2.1-4k	100	88	79
RWKV-6-1.6B-v2.1-4k	98	53	55

NIAH-Multi-keys test results are as follows:

Model	NIAH_multikey_1	NIAH_multikey_2	NIAH_multikey_3
RWKV-6-7B-v2.1-4k	48.33	7.67	1.33
Llama2-7B-4k	100	84.4	60
Mamba-2.8B-4k	7	0	1
Mamba-1.4B-4k	8	0	0
RWKV-6-3B-v2.1-4k	36	1	0
RWKV-6-1.6B-v2.1-4k	25	1	0

Multi-values and Multi-queries NIAH test results are as follows:

Model	NIAH_multivalue	NIAH_multiquery
RWKV-6-7B-v2.1-4k	80.42	83.67
Llama2-7B-4k	94	96.7
Mamba-2.8B-4k	0.75	1.25
Mamba-1.4B-4k	5.25	4.75
RWKV-6-3B-v2.1-4k	38.5	40.75
RWKV-6-1.6B-v2.1-4k	25	20.75

Variable Tracking (VT)

::: tip Multi-hop Tracing - Variable Tracking: This task mainly checks whether the model can successfully identify and track entities (variables) and reference relationships with multi-hop connections within a long context. For example, given the assignment $X_1 = V$, then $X_2 = X_1$, $X_3 = X_2$ ... finally returning all variable names pointing to the value $V$. :::

Model	Multi-hop Tracing
RWKV-6-7B-v2.1-4k	7.53
Llama2-7B-4k	63.12
Mamba-2.8B-4k	45
Mamba-1.4B-4k	23.4
RWKV-6-3B-v2.1-4k	11.8
RWKV-6-1.6B-v2.1-4k	1.4

Information Aggregation (CWE, FWE)

::: tip Information Aggregation (CWE, FWE): This task involves Common Words Extraction (CWE) and Frequent Words Extraction (FWE), used to test the model's ability to aggregate common information across long contexts. :::

Model	Common Words Extraction (CWE)	Frequent Words Extraction (FWE)
RWKV-6-7B-v2.1-4k	38.6	78.33
Llama2-7B-4k	73.04	78.8
Mamba-2.8B-4k	2	53
Mamba-1.4B-4k	15.5	57.33
RWKV-6-3B-v2.1-4k	30.3	62.67
RWKV-6-1.6B-v2.1-4k	11	46.33

Question Answering (QA)

::: tip Question Answering (QA): This task adds distracting information to the input of existing short-context QA datasets to evaluate QA capabilities under various context sizes. :::

Model	qa_1	qa_2
RWKV-6-7B-v2.1-4k	45	37
Llama2-7B-4k	59.2	42
Mamba-2.8B-4k	23	18
Mamba-1.4B-4k	24	23
RWKV-6-3B-v2.1-4k	35	25
RWKV-6-1.6B-v2.1-4k	35	28

Data Source

RULER Data Source: https://github.com/Ojiyumm/RULER_RWKV

LongBench Test

::: tip LongBench is a benchmark for evaluating the long-text understanding capabilities of large language models.

LongBench consists of six major categories and twenty-one different bilingual (Chinese-English) tasks, covering critical long-text application scenarios such as Single-Document QA, Multi-Document QA, Summarization, Few-shot Learning, Synthetic Tasks, and Code Completion. :::

Below is a comparison of LongBench scores for RWKV and other models, with data tables presented by the six categories:

Single-Document QA

Single-Document QA includes the following four test tasks:

Task	Task Description
NarrativeQA	QA based on stories or scripts, including understanding of characters, plots, themes, etc.
Qasper	QA based on single papers; questions are asked by NLP readers and answered by NLP practitioners.
MultiFieldQA-en	Answering English questions based on a single document from relatively diverse fields.
MultiFieldQA-zh	Answering Chinese questions based on a single document from relatively diverse fields.

Single-Document QA Test Results:

Model	NarrativeQA	Qasper	MultiFieldQA-en	MultiFieldQA-zh
GPT-3.5-Turbo-16k	23.6	43.3	52.3	61.2
Llama2-7B-chat-4k	18.7	19.2	36.8	11.9
LongChat-v1.5-7B-32k	16.9	27.7	41.4	29.1
XGen-7B-8k	18.0	18.1	37.7	14.8
InternLM-7B-8k	12.1	16.7	23.4	33.6
ChatGLM2-6B-32k	21.1	31.5	46.2	51.6
Vicuna-v1.5-7B-16k	19.4	26.1	38.5	43.0
ChatGLM3-6B-32k	26.0	43.3	51.7	62.3
Mamba_1B4	2.23	4.44	11.33	13.03
Mamba_2B8	2.32	4.89	8.15	6.83
Llama2-7B	18.7	19.2	11.90	36.8
Mistral-7B	12.79	8.9	30.55	17.91
RWKV-6-World-1B6-v2.1	4.53	19.79	22.99	18.57
RWKV-6-World-3B-v2.1	2.87	14.2	18.78	21.49
RWKV-6-World-7b-v2.1-4k	20.75	40.2	36.01	50.19

Multi-Document QA

Multi-Document QA includes the following four test tasks:

Task	Task Description
HotpotQA	Answering questions based on HotpotQA documents; involves many 2-hop questions written by native speakers based on two related paragraphs.
2WikiMultihopQA	Answering questions based on 2WikiMultihopQA data; composed of up to 5-hop questions synthesized via manually designed templates.
MuSiQue	Answering questions based on MuSiQue data; composed of simple questions requiring up to 4-hop reasoning.
DuReader	Answering questions based on the Chinese DuReader dataset, containing 200k questions and 1M documents from Baidu Search and Baidu Zhidao.

Multi-Document QA Test Results:

Model	HotpotQA	2WikiMQA	Musique	DuReader (zh)
GPT-3.5-Turbo-16k	51.6	37.7	26.9	28.7
Llama2-7B-chat-4k	25.4	32.8	9.4	5.2
LongChat-v1.5-7B-32k	31.5	20.6	9.7	19.5
XGen-7B-8k	29.7	21.1	10.3	11.0
InternLM-7B-8k	28.7	22.8	9.0	11.1
ChatGLM2-6B-32k	45.1	34.0	21.9	37.6
Vicuna-v1.5-7B-16k	25.3	20.8	9.8	19.3
ChatGLM3-6B-32k	54.4	44.9	40.4	44.78
Mamba_1B4	5.73	8.77	3.3	11.95
Mamba_2B8	5.49	8.45	3.45	13.96
Llama2-7B	25.4	32.8	9.4	5.2
Mistral-7B	9.39	11.17	4.58	11.68
RWKV-6-World-1B6-v2.1	8.72	11.86	3.96	11.40
RWKV-6-World-3B-v2.1	6.79	9.64	4.13	17.41
RWKV-6-World-7b-v2.1-4k	22.74	16.3	10.49	28.01

Summarization

The Summarization category involves the following four test tasks:

Task	Task Description
GovReport	Summarization task requiring summaries of government work reports.
QMSum	Summarization task requiring summaries of meeting minutes based on user queries.
MultiNews	Multi-document summarization task requiring summaries based on multiple news articles.
VCSUM	Summarization task requiring summaries of Chinese meeting minutes.

Summarization Test Results:

Model	GovReport	QMSum	MultiNews	VCSUM (zh)
GPT-3.5-Turbo-16k	29.5	23.4	26.7	16.0
Llama2-7B-chat-4k	27.3	20.8	25.8	0.2
LongChat-v1.5-7B-32k	30.8	22.7	26.4	9.9
XGen-7B-8k	27.3	20.5	26.2	2.2
InternLM-7B-8k	9.7	15.9	22.8	12.4
ChatGLM2-6B-32k	32.4	24.0	26.5	16.2
Vicuna-v1.5-7B-16k	27.9	22.8	27.2	15.1
ChatGLM3-6B-32k	36.8	23.9	27.9	17.8
Mamba_1B4	9.34	10.85	15.86	6.33
Mamba_2B8	10.41	11.42	18.94	6.1
Llama2-7B	27.3	20.8	25.8	0.2
Mistral-7B	28.84	20.32	22.79	4.1
RWKV-6-World-1B6-v2.1	17.51	20.36	21.52	10.71
RWKV-6-World-3B-v2.1	19.21	21	21.76	10.18
RWKV-6-World-7b-v2.1-4k	31.64	21.31	26.06	15.19

Few-shot Learning

Few-shot Learning includes the following four test tasks:

Task	Task Description
TREC	Classification task requiring question classification, containing 50 categories in total.
TriviaQA	Single-document QA task, providing several Few-shot examples.
SAMSum	Dialogue summarization task, providing several Few-shot examples.
LSHT	Chinese classification task requiring news classification, containing 24 categories in total.

Few-shot Learning Test Results:

Model	TREC	TriviaQA	SAMSum	LSHT (zh)
GPT-3.5-Turbo-16k	68.0	91.4	41.7	29.2
Llama2-7B-chat-4k	61.5	77.8	40.7	19.8
LongChat-v1.5-7B-32k	63.5	82.3	34.2	23.2
XGen-7B-8k	65.5	77.8	25.3	20.5
InternLM-7B-8k	52.0	77.8	21.2	15.2
ChatGLM2-6B-32k	62.5	78.7	36.3	27.7
Vicuna-v1.5-7B-16k	71.5	86.2	40.8	28.8
ChatGLM3-6B-32k	79.0	87.1	38.2	42.0
Mamba_1B4	45.5	37.33	12.56	8.5
Mamba_2B8	21.5	34.62	9.3	5
Llama2-7B	61.5	77.8	40.7	19.8
Mistral-7B	70.0	89.26	43.74	25.5
RWKV-6-World-1B6-v2.1	39.5	47.64	13.58	18.8
RWKV-6-World-3B-v2.1	51.5	57.15	17.95	15.2
RWKV-6-World-7b-v2.1-4k	55.5	86.89	44.25	30.2

Synthetic Tasks

Synthetic Tasks include the following three test tasks:

Task	Task Description
PassageCount	Determine the total number of unique paragraphs among the given paragraphs.
PassageRetrieval-en	Given 30 English Wikipedia paragraphs, determine which paragraph the given summary belongs to.
PassageRetrieval-zh	Given several Chinese paragraphs from the C4 dataset, determine which paragraph the given summary belongs to.

Synthetic Tasks Test Results:

Model	Passage Count	PassageRetrieval-en	PassageRetrieval-zh
GPT-3.5-Turbo-16k	4.5	71.0	77.5
Llama2-7B-chat-4k	2.1	9.8	0.5
LongChat-v1.5-7B-32k	1.0	30.5	7.6
XGen-7B-8k	2.1	8.5	3.5
InternLM-7B-8k	3.0	6.0	0.9
ChatGLM2-6B-32k	1.5	77.0	64.5
Vicuna-v1.5-7B-16k	6.5	4.5	5.0
ChatGLM3-6B-32k	2.0	99.0	94.0
Mamba_1B4	0.45	3.32	3.81
Mamba_2B8	0.74	1.83	3.37
Llama2-7B	2.1	9.8	0.5
Mistral-7B	1.05	12.5	16.75
RWKV-6-World-1B6-v2.1	0	4.25	4.16
RWKV-6-World-3B-v2.1	0	3.83	4.12
RWKV-6-World-7b-v2.1-4k	5	34.5	54.22

Code Completion

Code Completion includes the following two test tasks:

Task	Task Description
LCC	Given a long piece of code, predict the next line of code.
RepoBench-P	Given code from multiple files in a GitHub repository (including inter-file dependencies), predict the next line of code.

Code Completion Test Results:

Model	LCC	RepoBench-P
GPT-3.5-Turbo-16k	54.7	53.6
Llama2-7B-chat-4k	52.4	43.8
LongChat-v1.5-7B-32k	53.0	55.3
XGen-7B-8k	38.6	38.6
InternLM-7B-8k	44.1	28.8
ChatGLM2-6B-32k	55.6	49.9
Vicuna-v1.5-7B-16k	51.0	43.5
ChatGLM3-6B-32k	57.66	54.76
Mamba_1B4	44.33	41.86
Mamba_2B8	39.53	24.38
Llama2-7B	52.4	43.8
Mistral-7B	70.64	59.7
RWKV-6-World-1B6-v2.1	39.5	40.44
RWKV-6-World-3B-v2.1	40.01	41.35
RWKV-6-World-7b-v2.1-4k	73.84	54.1

Comprehensive Score Comparison of RWKV, Mamba, and Llama2

Model	Single DocQ	Few-shc	Summarization	Multi Doc QA	Code Completion	Syntetic
RWKV-6-World-1B6-v2.1	16.470	29.868	17.525	8.985	39.970	2.803
RWKV-6-World-3B-v2.1	14.335	35.443	18.038	9.493	40.680	2.650
RWKV-6-World-7b-v2.1-4k	36.788	54.203	23.550	19.385	63.970	31.240
Mamba_1B4	7.758	25.973	10.595	7.438	43.095	2.527
Mamba_2B8	5.548	17.605	11.718	7.838	31.955	1.980
Llama2-7B	21.650	49.950	18.525	18.200	48.100	4.133
Mistral-7B	17.538	52.833	19.013	9.205	65.17	10.100

Data Source

Evaluation Data Source: https://github.com/Ojiyumm/LongBench_RWKV