• Analyze conversation transcripts and user feedback to identify areas of confusion, failure, and prompt leakage. • Work with the Customer Impact Team Product Lead to define and track conversational KPIs (e.g., resolution rate, containment rate, user satisfaction). • Optimize prompts and model selection for cost efficiency, response latency, and scalability in production environments. • Collaborate with the engineers to improve conversation-specific evaluation criteria (e.g., NLU accuracy, intent recognition). • Design and maintain evaluation frameworks to measure prompt performance using golden datasets and automated scoring (e.g., LLM-as-judge, rubric-based scoring, precision/recall of intent routing). • Implement guardrails to reduce hallucinations, prevent prompt injection, and ensure compliant, safe responses. • Collaborate on design, map, and implement complex conversation flows, including error recovery and contextual handoffs (escalation to human support). • Own the continuous optimization of system prompts and instructions for LLMs (Gemini, OpenAI) to ensure Velo's response is accurate, tone is consistent, and on-brand. • Design and optimize structured outputs, function calling, and tool-routing logic to ensure accurate data capture and downstream system integrations.