Part-Time Student Worker – AI Validation and Benchmarking Engineer

About the role#

This position is part of the student worker program, designed to integrate students into engineering teams to contribute to real-world projects. You will support the end-to-end validation pipeline for AI tools by maintaining test datasets, running benchmarks, and measuring agent accuracy across routing decisions, classification labels, and structured output fields.

What you'll do#

• Run and maintain the benchmark pipeline, analyzing results to identify routing errors and regressions across agent variants.
• Build and expand ground truth datasets to evaluate agent outputs against known-correct answers.
• Identify and address gaps in benchmark validation to build a more comprehensive evaluation infrastructure.
• Develop new evaluation dimensions, including label accuracy and structured output correctness.
• Investigate failure modes in agent outputs and assist engineers in surfacing actionable improvements.
• Write scripts and tooling to automate data collection, result parsing, and metric reporting.
• Document findings, track benchmark trends over time, and present results to the team.

What you'll need#

• Current enrollment in a B.S. or M.S. program in Computer Science, Data Science, Engineering, or a related field.
• Familiarity with Python, Cursor, or Claude.
• Understanding of evaluation concepts such as precision, recall, F1 score, and confusion matrices.
• Ability to work with structured data formats including CSV and JSON.
• Experience modifying or writing reproducible analysis scripts.
• Ability to commit to at least 20 hours per week for a minimum of three months.
• Prior exposure to LLM-based systems, prompt engineering, or AI agent evaluation is a plus.

Location & details#

• Location: Foster City, California.
• Modality: On-site.
• Compensation: $30/hour.
• Employment: This is a contract position through a third-party vendor.