Mercor
Applied Engineer, Evaluations Role
- Pay
- $100–120/hr
- Type
- Part-time
- Location
- Remote
- Posted
- Jun 2, 2025
Tip: In our experience, people who apply through a referral link have a better chance of being selected, as I’m also working on Mercor.
Free to apply. Our referral link costs you nothing and doesn’t change your pay. If you’re hired, Mercor may pay us a bonus, which keeps this site free. Never pay anyone to get AI training work.
About this role
Mercor is seeking software engineers to build and refine evaluations. You’ll turn completed pull requests into engineering tasks and use our in-house evaluation framework to multiple frontier coding agents, including Claude Code, Codex and more. We work alongside an in-house research team and have produced industry leading benchmarks to compare the performance of frontier large language models. In this role, you will: Identify repositories with enough substantive work to support challenging evaluations, and select suitable completed PRs. Investigate each problem, its reference solution, and the repository’s architecture, tests, and conventions. Write task prompts and build evaluations using automated tests, shell commands, and LLM grading prompts. Validate evaluations against reference solutions and deliberately flawed implementations. Find missing checks, incorrect grades, and criteria…
Categories
Similar jobs
-
AI Jailbreak & Prompt-Injection Security Expert New
$50–90/hr
-
Senior Software Engineer, Backend (Node.js, TypeScript, Rust) New
$50–80/hr
-
Senior Software Engineering Experts
$120–180/hr
-
Senior Data Scientist and Data Engineering Experts
$120–180/hr
-
Certified Inpatient Medical Coder
$50–70/hr
-
Rust Architect / Backend Developer