55 lines
2.8 KiB
Markdown
55 lines
2.8 KiB
Markdown
# Evaluation data
|
|
|
|
`ds4-eval` contains small, curated subsets of external benchmarks. They are
|
|
integration tests for DwarfStar, not official benchmark distributions or
|
|
leaderboard scores. Case IDs are retained so each prompt can be checked against
|
|
its source.
|
|
|
|
## Hard suite
|
|
|
|
The 50-case `hard` suite was assembled on 2026-09-03:
|
|
|
|
| Source | Cases | Source revision | License |
|
|
|---|---:|---|---|
|
|
| [MMLU-Pro](https://github.com/TIGER-AI-Lab/MMLU-Pro) | 30 | `f418b116db00b065c2aea046518d8fcf74d39872` | Apache-2.0 |
|
|
| [OlympiadBench](https://huggingface.co/datasets/Hothan/OlympiadBench) | 10 | `91184b52131e7fc9455fef848035173aea8cc01a` | Apache-2.0 |
|
|
| [LiveBench](https://huggingface.co/datasets/livebench/reasoning) | 5 | `6fc6498a5dfba553f69f4413feabade1f1a2d384` | Apache-2.0 |
|
|
| [NIST Juliet 1.3](https://samate.nist.gov/SARD/test-suites/112) | 5 | archive SHA-256 `ada9d7e1c323d283446df3f55bdee0d00bda1fed786785fe98764d58688f38eb` | CC0-1.0 / public domain |
|
|
|
|
Selected source IDs:
|
|
|
|
- MMLU-Pro: `86`, `671`, `899`, `1467`, `1995`, `2398`, `2826`, `3024`,
|
|
`3546`, `3696`, `4674`, `4699`, `5078`, `5848`, `6034`, `6500`, `6844`,
|
|
`6955`, `7738`, `7754`, `8771`, `8863`, `9055`, `9487`, `10360`, `10567`,
|
|
`10775`, `11120`, `11292`, `11346`.
|
|
- OlympiadBench: `1606`, `1613`, `1614`, `1618`, `1697`, `1709`, `1716`,
|
|
`1736`, `1750`, `1766`.
|
|
- LiveBench: `c833574b0ff6e8609bee19b36fabc48501cdcd9a64b99cd0422f54ada1acc53d`,
|
|
`b7b7e331df4614775210ef5e524ed7f20c5fec6655fb0dd179c94e4d71170862`,
|
|
`0daa7ca38beec4441b9d5c04d0b98912322926f0a3ac28a5097889d4ed83506f`,
|
|
`9ee37b9a04ab050936c86b2c5bb7abbaa0bc0e737d59a7bff9ba11e9b4069c1d`,
|
|
`d7071c9ff5d9779e7ab955366d0ae8db40f785aadfe7ff0b5a7ede98c05c44ea`.
|
|
- NIST Juliet reductions use the local IDs beginning with `juliet-` in
|
|
`ds4_eval_cases.c`.
|
|
|
|
The MMLU-Pro choices and keys are unchanged. OlympiadBench cases are English,
|
|
text-only rows; only their published final answers are embedded. LiveBench's
|
|
benchmark-specific output wrapper was removed because `ds4-eval` supplies one
|
|
uniform final-answer instruction. The NIST cases are short, locally written
|
|
reductions of Juliet weakness patterns rather than copies of complete Juliet
|
|
test cases.
|
|
|
|
Multiple-choice order is retained for MMLU-Pro. Open answers declare their
|
|
grading type and, where needed, a small list of equivalent surface forms. The
|
|
12-case `hard-smoke` suite is a fixed subset of `hard`.
|
|
|
|
Apache-2.0 terms for redistributed benchmark material are included in
|
|
[`licenses/Apache-2.0.txt`](licenses/Apache-2.0.txt). DwarfStar's own source
|
|
remains under the repository's top-level MIT license.
|
|
|
|
## Core suite
|
|
|
|
The original 92-case `core` suite contains 25 GPQA Diamond, 25 SuperGPQA, 25
|
|
AIME 2025, and 17 reduced defensive code-review cases. Its existing source IDs,
|
|
ordering, prompts, and answer keys are unchanged by the hard-suite addition.
|