AI Evaluation Rubric Auditor
The work
We are testing AI agents on realistic shopping and delivery tasks. Your role as a Project Auditor is to audit the work done by our Project Reviewers. You will audit complete tasks (prompt and grading rubric) and determine if their rubric failures are fair:
\- Was the information available?
\- Does the rubric accurately grade what the prompt asked for?
\- Would a valid answer have been marked wrong?
\- Is the rubric structure correct per the guidelines?
This role includes providing feedback to reviewers regarding prompt and rubric issues.
Must-haves
\- Substantial experience writing or reviewing rubrics for LLM evals. Please list the projects in your application.
\- Experience providing feedback to writers/reviewers
\- Consistent availability
\- Can read a long agent transcript and find where it went wrong.
\- Precise writing, and you check the data instead of trusting the reference answer.
Location: Remote
Eligible residence: Canada, United States, ARG, BOL, BRA, CHL, COL, ECU, GUY, PRY, PER, URY, United Kingdom
Shared by Remote Opportunities | Independent Referrer. I may receive a referral fee for qualifying referrals. Mercor handles applications and hiring decisions. Candidates are never charged.