• Define and measure conversation quality KPIs that align with product, support, and business goals. • Conduct systematic quality audits of AI conversations using both quantitative metrics and qualitative assessment, flagging conversations that went well and those that need improvement. • Analyze conversation data to identify failure patterns, drop-off points, conversion blockers, and untapped sales opportunities. • Partner with product and engineering to translate analytical insights into specific improvements: prompt refinements, conversation flow redesigns or updates to logic for how our AI responds to queries. • Establish quality assurance processes and frameworks that scale with conversation volume. • Support the integration of conversational quality tools and frameworks into existing workflows and tooling. • Present findings and recommendations to leadership with clear narratives connecting AI performance to business outcomes. • Lead cross-functional initiatives. • Own the AI conversation quality testing strategy, define standards, performance benchmarks, and a long-term improvement roadmap.