About the role#
As a Research Engineer, you will work at the intersection of engineering and applied AI research. You will own benchmarking pipelines, evaluation systems, and failure analysis workflows that inform how we train and improve frontier language models. Your work defines how we measure tool use, agentic behavior, and real-world reasoning. You will design and run evaluations, build rubrics, and turn failure analysis into improvements for post-training and data pipelines.
What you'll need#
- Strong applied research background with a focus on model evaluation, benchmarking, or failure analysis.
- Strong coding skills and hands-on experience with ML models and evaluation code.
- Solid grasp of data structures, algorithms, and backend systems.
- Comfort with APIs, SQL/NoSQL, and cloud platforms.
- Ability to reason about model behavior, experimental results, and data quality.
- Excitement to work in person in San Francisco five days a week.
- Pursuing a degree in Computer Science, Data Science, Mathematics, Computer Engineering, or Electrical Engineering.
- Industry experience on a post-training or evaluation team is preferred.
- Publications at top-tier venues like NeurIPS, ICML, or ACL are a plus.
- Experience with synthetic data generation, rubric design, or RL-style workflows is helpful.
- Work samples or code demonstrating relevant skills are encouraged.
What you'll do#
- Design, implement, and maintain benchmarks and metrics for tool use, agentic behavior, and real-world reasoning.
- Ensure benchmarks scale with training and stay aligned with research goals.
- Build and operate LLM evaluation systems, including scoring and reporting.
- Run systematic failure analysis on model outputs to categorize failure modes and quantify prevalence.
- Feed findings into reward design, data curation, and benchmark design.
- Create and refine rubrics, automated evaluators, and scoring frameworks.
- Quantify data usability, quality, and impact on key benchmarks.
- Collaborate with AI researchers and data producers to align evaluations with training objectives.
- Operate in a high-iteration research setting with ownership of benchmarks and workflows.
Location & details#
- Location: San Francisco, California.
- Modality: On-site, five days a week.
- Type: Full-time, paid internship.
- Term: Rolling.


