TCSExam

Project website

TCSExam evaluates model–agent systems on verifiable theoretical computer science research tasks. This repository is for researchers inspecting tasks, running evaluations, and comparing verified results.

Choose a reading path from the documentation home. For your first evaluation, follow Run TCSExam from scratch. For an existing setup, use Run configuration and parameters.

Tasks

Full contains 100 tasks. Lite selects 20 from Full, with five from each task category. The table shows the number of tasks in each suite; every Solver task provides separate Online and Offline packages.

Suite	Theorem proving	Solver engineering	Constructions	Open conjectures	Total
Full	20	30	20	30	100
Lite	5	5	5	5	20

Theorem tasks ask for a Lean proof of a published result. Solver tasks ask for an algorithm implementation evaluated on hidden instances. Construction tasks ask for a verified construction or bound. Open-conjecture tasks accept a complete Lean proof or refutation of the stated conjecture. See the task catalogue for task-level statements and sources.

Full defaults to Online Solver, which provides judging feedback during model development. Lite defaults to Offline Solver, which provides public self-tests during development and hidden judging afterward. Both suites support both protocols through --solver-protocol; report their results separately. Solver protocols defines the feedback and network rules.

Run an evaluation

Use a Linux x86-64 host, Python 3.12+, Docker Engine with Compose and Buildx, and a model API account. The runtime dependency installs Harbor 0.22.0, which runs the agents and collects their outputs. Solver judging additionally requires Landlock ABI 3 or later and the required container syscall permissions. Online Solver uses a judging backend; Offline Solver runs its controller on the Docker host. See runtime requirements for host sizing.

The getting-started guide takes you through cloning, installation, a single discrete-bidding-games task, Lite, Full, and reading the actual result files. It uses Kimi Code by default. Claude Code, Codex and Grok Build are also supported; agent configuration provides their settings and commands.

Use Run configuration and parameters for task selection, budgets, credentials and saved jobs; resource preparation for inputs and image builds; and the Docker backend guide for Online Solver service deployment. Source-package contents and build entry points are described in source packages.

Scores

text

TCSExam Index = 30 × theorem success
             + 30 × fixed-K3-relative Solver score
             + 30 × original Construction score
             + 10 × complete Open-conjecture success

Average adopted attempts within each task, then average tasks equally within each track. Report all four track scores alongside the Index. Construction scores may exceed one, so the Index is measured in points and may exceed 100. Failed or missing judging remains distinct from a completed zero score. Scoring defines the formulas, fixed Solver reference and input formats for tcsexam solver-score and tcsexam score.

Licensing and third-party attribution

Project-authored code, checkers, scripts and build configuration use Apache-2.0. Original task statements, documentation and website content use CC BY 4.0. Third-party materials retain their own terms; preserve their task-level sources and notices. Consult the license guide before redistributing inputs or images.

For attribution, licensing or material-use concerns, open a repository Issue; the corrections process describes the process. Development and source-export instructions are in CONTRIBUTING.md.

Cite TCSExam

If you use TCSExam in your research, please cite the paper:

bibtex

@misc{tcsexam2026,
  title = {{TCSExam}: Evaluating frontier {LLMs} on the full range of {TCS} research},
  author = {Luo, Yifan and Liu, Zhengying and Yuan, Yang and Yang, Zhilin and Yao, Andrew Chi-Chih},
  year = {2026},
  month = nov,
  url = {https://github.com/MoonshotAI/TCSExam}
}

The experiment reporting guide explains how to identify the software version, model configuration and scoring reference alongside your results.

TCSExam ​

Tasks ​

Run an evaluation ​

Scores ​

Licensing and third-party attribution ​

Cite TCSExam ​