C

Expert Talent Pool, Evals and Benchmarks (Associate)

Clera

Remote (Worldwide) Remote Data Science & AI Posted 13h ago

About this job

About the Role Join a specialist talent pool supporting human-data projects that evaluate AI models. You will help design and run assessments that measure model capabilities and contribute to AI safety and capability research. What You'll Do

  • Write questions and tasks that isolate and test specific model capabilities.
  • Develop rubrics that define strong, partial, and poor answers.
  • Create reference answers and edge cases to challenge AI models.
  • Grade model outputs against rubrics and explain scoring decisions.
  • Analyze failed answers, group error patterns, and document findings.
  • Compare multiple models using identical prompts.
  • Validate benchmarks and flag compromised or ineffective questions.
  • Participate in small user studies evaluating ratings or interactions with model outputs. What We're Looking For
  • Zero to five years of professional experience.
  • Experience in QA or test engineering, assessment design, psychometrics, UX research, data analysis, ML evaluation, research support, or relevant subject-matter work involving question writing.
  • Experience writing test cases, exam questions, rubrics, or evaluation tasks, and reviewing work consistently against written criteria.
  • Comfort using spreadsheets or Python to organize and analyze results, and identifying and documenting error patterns.
  • Experience with evaluation tools such as Inspect, lm-eval-harness, HELM, or promptfoo is useful.
  • Clear written communication and the ability to compare AI model outputs or participate in user and rating studies. Compensation & Benefits Compensation is $30 to $60 per hour, depending on project, field, and experience. Payment is for hours worked. Location Fully remote. Projects typically last from one week to several months. Compensation Range: $62.4K - $124.8K Originally posted on Himalayas

This job was collected from Himalayas. You apply on the original posting; TechShilpo is not the employer.

Keep looking

Similar Data Science & AI jobs

All →

AI Engineer

UPSTRA Communications Ltd.

New
Dhaka, Bangladesh On-site 🇧🇩 Bangladesh

Position Overview: We are looking for a talented and motivated AI Engineer to join our growing technology team. The ideal candida…

Salary: negotiable 13h ago · via LinkedIn
Remote (APAC) Remote

Who We Are At OKX, we believe that the future will be reshaped by crypto, and ultimately contribute to every individual's freedom…

Salary: negotiable 13h ago · via Jobicy
New
Remote (Anywhere) Remote

About Supabase Supabase is the Postgres development platform, built by developers for developers. We provide a complete backend s…

Salary: negotiable 13h ago · via Jobicy
Remote (Worldwide) Remote

About the Role Join an expert network supporting AI human-data projects focused on agents and reinforcement learning. You will de…

USD 80 – 200 / hourly 13h ago · via Himalayas
Remote (Worldwide) Remote

About the Role Join a team supporting physical AI projects by collecting high-quality data from real-world human movement and man…

USD 20 – 40 / hourly 13h ago · via Himalayas
Remote (Worldwide) Remote

About the Role Join a network of data operations specialists supporting human-data projects for AI research. You will source, cur…

USD 20 – 40 / hourly 13h ago · via Himalayas