Herkese, her yerde, dünya standartlarında, ücretsiz eğitim... #HerŞeyiÖğrenebilirsin www.khanacademy.org.tr

Senior Platform Engineer, AI Evaluation

Platform EngineerPlatform EngineerFull TimeRemoteTeam 11-50Since 2012H1B No SponsorCompany Site LinkedIn

Location

California

Posted

98 days ago

Salary

$137.9K - $172.3K / year

Bachelor Degree5 yrs expEnglishAirflowPythonSQLGo

Job Description

• Evolve and extend our internal evaluation framework for assessing the quality of our AI-driven experiences at Khan Academy • Work closely with ML data engineers and platform developers to help internal teams adopt an eval-driven development process incorporating offline benchmark tests and online experiments • Gather internal requirements, get buy-in for changes, and develop documentation and training materials • Be fluent in the range of offline and online evaluation strategies, and when to apply the techniques over the lifecycle of development • Have intuitions about how to specify eval pipelines succinctly using declarative syntax • Understand the role of stratified datasets and ground truth labeling • Appreciate the range of eval scoring schemes from human raters to automated LLMs-as-judge

Job Requirements

Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field
5 years of Software Engineering experience with 2+ of those years working on the evaluation of generative AI systems
Strong programming skills in Go, Python, SQL, and at least one data pipeline framework (e.g., Airflow, Dagster, Prefect)
Familiarity with the architecture of large language models and their industry-standard APIs
Experience with labeling platforms (e.g., Label Studio, Scale AI, Toloka) and human-in-the-loop concerns such as rubric development and inter-rater agreement
Exposure to MLOps practices such as model registry, feature store, or continuous evaluation
Background in education technology or other human-centered AI applications

Benefits

Competitive salaries
Ample paid time off as needed – Your well-being is a priority
8 pre-scheduled Wellness Days in 2026 occurring on a Monday or a Friday for a 3-day weekend boost
Remote-first culture - that caters to your time zone, with open flexibility as needed, at times
Generous parental leave
An exceptional team that trusts you and gives you the freedom to do your best
The chance to put your talents towards a deeply meaningful mission and the opportunity to work on high-impact products that are already defining the future of education
Opportunities to connect through affinity, ally, and social groups
And we offer all those other typical benefits as well: 401(k) + 4% matching & comprehensive insurance, including medical, dental, vision, and life