About the role#
As a Research Engineer, you will work at the intersection of engineering and applied AI research. You will own benchmarking pipelines, evaluation systems, and failure analysis workflows that inform how we train and improve frontier language models. Your work defines how we measure tool use, agentic behavior, and real-world reasoning. You will design and run evaluations, build rubrics, and turn failure analysis into improvements for post-training and data pipelines.
What you'll do#
- Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning.
- Ensure benchmarks scale with training and stay aligned with research goals.
- Build and operate LLM evaluation systems, including scoring and reporting.
- Run systematic failure analysis on model outputs to categorize failure modes and quantify prevalence.
- Feed findings into reward design, data curation, and benchmark design.
- Create and refine rubrics, automated evaluators, and scoring frameworks.
- Quantify data usability, quality, and impact on key benchmarks.
- Collaborate with AI researchers and data producers to align evaluations with training objectives.
- Operate in a high-iteration research setting with ownership of benchmarks and workflows.
What you'll need#
- Strong applied research background with a focus on model evaluation, benchmarking, or failure analysis.
- Strong coding skills and hands-on experience with ML models and evaluation code.
- Solid grasp of data structures, algorithms, and backend systems.
- Comfort with APIs, SQL/NoSQL, and cloud platforms.
- Ability to reason about model behavior, experimental results, and data quality.
- Excitement to work in person in San Francisco five days a week.
- Pursuing a degree in Computer Science, Data Science, Mathematics, Computer Engineering, or Electrical Engineering.
- Industry experience on a post-training or evaluation team is preferred.
- Publications at top-tier venues like NeurIPS, ICML, or ACL are a plus.
- Experience with synthetic data generation, rubric design, or RL-style workflows is helpful.
- Work samples or code demonstrating relevant skills are encouraged.
Location & details#
- Location: San Francisco, California.
- Modality: On-site, five days a week.
- Type: Full-time, paid internship.
- Term: Rolling.
About Mercor
Mercor is a software development company founded in 2023. Based in San Francisco, the firm employs between 201 and 500 people. It focuses on training frontier models by using professional domain experts. The company also deploys custom AI agents and helps organizations integrate their internal knowledge into AI systems.
How to get in at Mercor
Securing an internship at Mercor requires speed because early applicants often get seen before the pile grows. Intern Insider sends an instant alert the moment a role matching your target is published anywhere, ensuring you apply among the first. This gives you a significant advantage over those who wait for standard job boards to update. You can also improve your chances by reaching out to the right people. Intern Insider surfaces the recruiters behind the company roles, allowing you to contact them directly to ask about the position or a referral, which materially improves your response rates.
