• Direct annotation by managing annotation budgets, sourcing data, and providing statistical analysis of annotations • Maintain and expand metrics used to evaluate agent performance and fairness • Create evaluation packages for Data Scientist and Machine Learning Engineers to use when running experiments • Ensure accurate, regular reporting on evaluation metrics to stakeholders • Manage periodic re-evaluation of production agents using user data and annotation