Separated capture and scoring
Runner and scorer queues separate endpoint IO from the evaluation logic. Raw payload storage and per-dimension rationales make individual results inspectable.
01 / THE IDEA
A closer look.
An adversarial evaluation platform with a FastAPI control plane, Celery workers, and a Next.js results interface. It captures agent responses and decomposes their behavior into explicit scoring dimensions with per-prompt rationales.
A single leaderboard number hides the reason an agent failed. Separating raw capture, scoring, and aggregation creates a trail from a surprising result back to the response that produced it.
Explicit score composition
Weight snapshots travel with a run. Missing dimensions are excluded and remaining weights are normalized, rather than silently counting missing values as zero.
Multiple inspection surfaces
The repository contains a public leaderboard, run results and trace views, paginated per-prompt results, NDJSON export, and completion webhooks.
02 / UNDER THE SURFACE
From endpoint to evidence
Follow a prompt through capture, independent scorers, and a weighted result. Recovery is a separate scorer, not wired into the single-turn worker.
DRAG TO PAN · SELECT A NODE · + / − TO ZOOM
Read the architecture as text
- Submit a run — The API selects active, non-quarantined prompts for a corpus version and creates pending result rows. The run records the supplied endpoint and a snapshot of the scoring weights.
- Dispatch prompts — dispatch_run marks the run running, then enqueues a run_prompt task for each result placeholder. Tasks carry run and result identifiers.
- Agent endpoint — run_prompt builds a system/user conversation and sends it to the supplied HTTP endpoint. A perf_counter interval records elapsed time at the runner boundary.
- Capture response — The worker extracts input/output token usage, estimates cost from a rate table, attempts an S3 upload, and commits captured fields before requesting scoring.
- Raw payload store — Raw endpoint JSON is written beneath runs/{run_id}/results/{result_id}.json. Upload failures are logged; the worker continues with a null S3 key.
- Scoring task — score_result extracts response text, calls the single-turn scorers, and attaches their rationales. Recovery is explicitly passed as None in this worker path.
- Resistance — AdversarialScorer compares the requested behavior and response against refusal and compliance signals. These rules are a transparent heuristic, not an exhaustive attack detector.
- Tool misuse — ToolMisuseScorer inspects structured tool calls and returns a score, rationale, and optional call graph that are persisted alongside the result.
- Claim grounding — The scorer heuristically extracts claims, excludes temporal statements, and checks overlap with a small fixed fact set. Ungrounded claims receive partial credit.
- Latency + cost — Latency and cost are normalized with explicit thresholds in their source modules. Unknown cost receives a neutral 0.5; missing latency receives 0.0.
- Recovery rubric — RecoveryScorer implements a multi-turn rubric. The inspected single-turn worker does not invoke it, so its weight must be excluded from that composite.
- Weighted composite — CompositeScorer filters missing dimensions, sums only available weights, and divides weighted scores by that active total. It also returns per-dimension contributions.
- Result records — Dimension scores, rationales, tool-call graphs, and timestamps are persisted. Finalization waits for all result rows to be scored and then computes run-level means.
- Public results — The repository includes a public leaderboard API and a Next.js leaderboard screen. Scores are only interpretable together with corpus version, weights, and underlying traces. The snippet is route notation, not a literal function body.
- Completion event — Finalization enqueues completion webhooks. Delivery includes run status and dimension scores, with retries on HTTP failure.
03 / INTERACTIVE STUDY
Build the composite
Adjust an illustrative resistance score and endpoint measurements. Tool misuse and grounding are held fixed; recovery is absent, following the inspected single-turn worker.
CHANGE THE INPUTS
Illustrative inputs, using source scoring bands. No agent endpoint is called and no real agent is being ranked.
ILLUSTRATIVE MODELLIVE
04 / ENGINEERING CHOICES
Why it works this way.
Keep measurements interpretable
Latency is measured outside the provider at the runner boundary. It includes HTTP work rather than claiming to isolate model inference time.
Preserve partial observability
A missing dimension is represented as None and removed from the active denominator. The single-turn worker deliberately leaves recovery unscored.
Expose heuristic limits
The grounding implementation uses a small fixed fact corpus and word overlap; this is an inspectable prototype, not general-purpose factual verification.
05 / OPEN THE SOURCE
Trace it back.
Implementation details, examples, and project documentation.
Runner and persistence
Inspected task dispatch, HTTP capture, raw payload storage, score persistence, and finalization.
Composite scoring
Inspected exclusion of missing dimensions and active-weight normalization.
Grounding implementation
Inspected temporal exclusions, fixed facts, and partial credit logic.
Scope & limitations
- Recovery has a scorer module but is not executed by the inspected single-turn worker.
- No production deployment, detection accuracy, or comparative leaderboard results are asserted here. The classifier and grounding rules have limited coverage.
Architecture and descriptions reflect the linked repository snapshot. The playground explains a mechanism; it does not execute the repository or report measured performance.