Don't take our word for it. Here are the numbers.
We run every employee against a fixed set of held-out tasks and grade the results with tests the employee never sees. This page shows what those runs found — including where the sample is still too small to publish, and where a run needs credentials we don't have in the test environment.
Snapshot as of 2026-09-04, generated from gate run 33916352182. Numbers below are read directly from that snapshot.
How we measure
Each employee runs its real execute_task against a fixed corpus of held-out tasks. Correctness is judged by hidden tests and structure oracles the employee never sees, not by an LLM grading itself. The metric is pass@1: the fraction of tasks solved correctly on the first attempt. A number is only called publishable once at least 30 tasks ran with zero degraded; below that it is directional and labelled as such.
Each task calls the employee's actual execute_task path — the same code a customer would hit.
Correctness is decided by hidden tests and structure oracles the employee never sees — not by a model grading itself.
The score is the share of tasks solved correctly on the first attempt. A corpus is only “publishable” at 30+ tasks.
22 employees with a proven number
Each of these ran its full corpus of 30+ held-out tasks with zero degraded runs. These are the numbers we stand behind.
A score belongs here only after the whole corpus runs cleanly. Everything below is early signal we are still growing.
22 employees with early signal
These corpora did not fully run — most need customer credentials the test environment doesn't hold, so only part of each corpus could be scored. A partial score is an early read, not a claim, and where one runs low we show it rather than hide it. Wiring the remaining credentials is in progress.
For context only: the mean pass@1 across all 44 scored corpora is 98.7%. Because that average is dominated by these small samples, we treat it as directional context, not a headline. 22 corpora meet the publishable bar in gate run 33916352182. The 0.987 mean is over all scored corpora including sub-floor samples, so it is directional context, not a headline claim.
0 employees we couldn't score
These need credentials or a database that the test environment doesn't hold, so they were skipped rather than guessed. We would rather show a gap than invent a number.
What “publishable” means to us: a corpus of at least 30 held-out tasks, run against real execution with zero degraded runs, graded by hidden tests. Today that bar is met by 22 employees. Reporting it that plainly — instead of averaging small samples into a flattering headline — is the point of this page. As more corpora reach the floor, they appear here under the same rules.
Evaluating LiquidCortex and want the raw per-task detail? Email [email protected].