Where you come to do the best work of your life. Follow @WeAreNetflix on Twitter, IG, Facebook, & Youtube for more

Human Evaluation Program Manager

Program ManagerProgram ManagerFull TimeRemoteTeam 10,001+Since 1997H1B SponsorCompany Site LinkedIn

Location

United States

Posted

45 days ago

Salary

$230K - $340K / year

Bachelor Degree4 yrs expEnglish

Job Description

• Lead end-to-end execution of human evaluation and data operations initiatives—from intake and scoping to delivery • Develop and operationalize frameworks for evaluating GenAI and ML outputs • Collaborate across research, product, UX, and engineering to embed evaluation into model development cycles • Build and maintain project timelines, proactively manage blockers, and ensure timely execution • Develop clear, scalable guidelines and scoring rubrics to ensure consistent rater judgment • Oversee rater onboarding, calibration, and QA workflows • Define and monitor success metrics such as speed to IRR, throughput, and task effectiveness • Pilot and refine evaluation tasks to improve clarity, inter-rater reliability, and feedback quality • Build foundational documentation and drive adoption of best practices across teams • Track evaluation health and proactively communicate progress to stakeholders clearly and proactively • Anticipate and proactively resolve bottlenecks and blockers • Act as the connective tissue across multiple partners to ensure alignment and effective execution of evaluations at scale

Job Requirements

4+ years of experience working in human evaluations, data collection, labeling, or annotation operations in GenAI/ML environments
Track record of implementing process improvements or quality control systems for data collection needs
Prior experience managing human annotation vendors, raters, or data labeling teams
Strong understanding of evaluation design, including guidelines, rubrics, and scoring protocols
Proven ability in end-to-end management of complex, cross-functional programs, demonstrating strong Program Management skills and clear accountability for successful delivery
Experience with human labeling platforms
Excellent written and verbal communication skills
Ability to synthesize feedback into clear recommendations and process improvements
Familiarity with responsible AI principles and how to embed them into evaluation design
Strong organizational skills and executional focus; ability to track details while seeing the bigger picture.