DataAnnotation

DataAnnotation trains and evaluates frontier models using human specialists, and this is its top tier coding track. My work splits three ways. First, preference data: running the same multi file task through competing coding agents and writing up which one was actually right, judged on correctness, instruction adherence, tool use efficiency and how each one failed. Second, authoring the environments themselves on Harbor and Terminal-Bench 2, where every task ships with a containerised config, an Oracle solution and a deterministic verifier, so a run passes for a real reason or does not pass at all. Third, grading whole trajectories against rubrics that score planning, retrieval, tool selection, error recovery and diff correctness, rather than only looking at the final patch. A good part of the job is reproducing a failing build to work out whether the model hallucinated an API, edited half of what it promised, drifted off the spec, or quietly routed around the test.

What that meant day to day

• Ran 60+ side by side comparisons of frontier coding agents across 12 multi file repositories, producing preference labels on correctness, instruction adherence, tool use efficiency and failure modes.
• Authored 25+ agentic tasks and RL environments on Harbor and Terminal-Bench 2: containerised task configs, Oracle solutions, deterministic verifiers, Pytest harnesses and solution certificates.
• Designed multi axis grading rubrics for agent trajectories covering planning, retrieval, tool selection, error recovery and diff correctness, applied across 50+ graded runs.
• Reproduced builds and test suites to root cause model failures, classifying each one as a hallucinated API, an incomplete edit, spec drift or a silent test bypass, and red teamed agents for reward hacking.
• Held top tier quality scores across 6 project cohorts, earning the highest pay band and invites to specialised projects. The output feeds SFT and reward model training.