Skip to content
Crossing HurdlesCrossing HurdlesExpertsTerminal-Bench Research
ApplyExpert login
Methodology · Open to Inspection

Good tasks deserve a rigorous, transparent process.

Inspect the live Terminal-Bench authoring standard, all 14 sequential evaluation gates, witness replay rules, and runtime versions. Every recorded verdict stays bound to its exact Git commit SHA.

Take Live AssessmentRead Expert Playbook
14 Sequential CI StagesOracle = 1.0 · NOP = 0.0Separate No-Network EvaluationStage 15 Reviewer Abomination
ISOLATION & LIVE PR TELEMETRY
  1. Schema 1.4 container split

    Non-root rwe runtime with separate no-network test handoff.

    ISOLATED
  2. Oracle & NOP calibration

    Reference solver hits 1.0 while empty baseline stays at 0.0.

    1.0 / 0.0
  3. Automated PR stage tags

    Check-runs sync CI: 14/14 Passed and Abomination: Passed live.

    WEBHOOK
Every verdict, label, and settlement binds to a pinned SHA

Live policy register

A standard you can read. A result you can trace.

The register below streams directly from the platform’s public methodology endpoint. Project evaluation and canonical certification each carry their own recorded profiles.

Loading the published methodology…

Package architecture

Anatomy of a self-contained task package.

Every submitted task separates solver-visible inputs from isolated verification assets so autonomous agents earn credit strictly through valid engineering outputs.

task.toml & instruction.md

Declares Schema 1.4 resource bounds, network_mode = "no-network", environment_mode = "separate", and explicit /app/... artifacts matching the candidate prompt.

Contract & Prompt Alignment

environment/ & solution/

Provides the reproducible non-root container, input datasets, and an unprivileged reference implementation that solves the task without hidden generator constants.

Oracle = 1.0 Calibration

tests/grade.py & README.md

Executes offline inside a clean container receiving only declared artifacts, paired with maintainer documentation stating difficulty, success criteria, and plain limitations.

Anti-Tamper Verification

Evidence interpretation

Read every check in context.

Completed stages measure a specific submitted revision. Unrun stages stay pending rather than masquerading as failures, and infrastructure errors remain separated from task defects.

Oracle and NOP controls

A working reference confirms that the objective is achievable inside the declared limits. An unchanged starting container confirms that doing nothing earns zero reward.

Repeated model attempts

Trial outcomes belong to the named model, harness configuration, and source SHA. Pass@k reports whether at least one of k independent attempts succeeds alongside individual trial traces.

Independent human review

Reviewers audit difficulty, execute Stage 15 Abomination, and verify project rules before escalating to Super Reviewers for final squash-merge delivery.

Put the standard into practice

Author a Terminal-Bench 4.0 task in the live qualification studio.

See how the 5-gate rubric evaluates task originality, Schema 1.4 isolation, verifier rigor, and prompt alignment.

Take Live AssessmentWorkflow Guide