Know ATS Score
CV/Résumé Score
  • Expertini Resume Scoring: Our Semantic Matching Algorithm evaluates your CV/Résumé before you apply for this job role: AI Quality Analyst (Conversation Intelligence & Evaluation).
South Africa Jobs Expertini

Urgent! AI Quality Analyst (Conversation Intelligence & Evaluation) Job Opening In Randburg – Now Hiring DASH BPO

AI Quality Analyst (Conversation Intelligence & Evaluation)



Job description

AI Quality Analyst (Conversation Intelligence & Evaluation)

We are seeking a detail-oriented and insight-driven AI Quality Analyst to evaluate the performance of AI-powered customer interactions across voice and chat channels.

This role focuses on ensuring that AI agents deliver accurate, empathetic, and business-aligned responses while continuously surfacing opportunities for model and workflow improvement.

The ideal candidate will combine analytical precision, QA discipline, and data storytelling to transform large volumes of conversational data into actionable insights.

You will help bridge the gap between raw AI performance metrics and meaningful business outcomes—guiding both internal stakeholders and clients on where AI is adding value and where it needs refinement.

This is not a call review position—it’s an intelligence role that requires curiosity, pattern recognition, and the ability to see beyond surface-level metrics.

Key Responsibilities

AI Interaction Evaluation

  • Evaluate AI-handled customer and sales interactions for accuracy, intent recognition, tone, compliance, and effectiveness.

  • Identify gaps, hallucinations, misclassifications, or missed intents in AI responses.

  • Provide structured and qualitative feedback on conversational quality and system performance.

  • Maintain consistent scoring rubrics and calibration across evaluation sets.

Data Analytics & Insight Generation

  • Analyze AI interaction data to uncover trends, recurring issues, and optimization opportunities.

  • Build and maintain dashboards, scorecards, and performance summaries for internal and client stakeholders.

  • Generate structured datasets from evaluations for retraining and continuous improvement loops.

  • Translate quantitative findings into clear narratives and recommendations that influence roadmap decisions.

Quality Assurance & Tooling

  • Operate within existing QA and conversational intelligence tools (e.g., Observe.AI, Level.AI, Tethr, custom dashboards).

  • Contribute to the evolution of internal evaluation frameworks, metrics, and annotation guidelines.

  • Ensure that QA processes align with business goals, compliance standards, and customer satisfaction KPIs.
  • Support release sign-offs and regression checks for new conversational flows or AI model updates.

  • Collaborate with AI engineers, data scientists, and client teams to communicate findings and improvement priorities.

  • Present analytical insights and performance reviews in clear, client-friendly formats.

  • Contribute to ongoing refinement of QA workflows, evaluation standards, and reporting templates.

  • Stay current on conversational AI trends, evaluation methodologies, and tooling innovations.

  • Share learnings across teams and contribute to a culture of measurement-driven improvement.

  • Develop domain expertise across multiple client verticals (e.g., support, sales, finance, healthcare).

Collaboration & Reporting

  • Collaborate with AI engineers, data scientists, and client teams to communicate findings and improvement priorities.

  • Present analytical insights and performance reviews in clear, client-friendly formats.

  • Contribute to ongoing refinement of QA workflows, evaluation standards, and reporting templates.

Continuous Learning & Domain Growth

  • Stay current on conversational AI trends, evaluation methodologies, and tooling innovations.

  • Share learnings across teams and contribute to a culture of measurement-driven improvement.

  • Develop domain expertise across multiple client verticals (e.g., support, sales, finance, healthcare).

Requirements

  • Bachelor’s degree in Data Analytics, Linguistics, Computer Science, or a related field (or equivalent experience).

  • Strong analytical and QA background with demonstrated experience in data evaluation, quality frameworks, or conversational review.

  • Familiarity with conversational AI platforms or QA tools (e.g., Observe.AI, Level.AI, Tethr, CallMiner, or custom LLM dashboards).

  • Proficiency in data analysis and visualization tools (Excel, or BI platforms such as Power BI or Tableau).

  • Exceptional attention to detail and ability to balance qualitative and quantitative perspectives.

  • Excellent communication skills — able to synthesize complex data into clear business insights.

  • Fluency in English (C1+).

Nice to Have

  • Experience working with conversational AI models (OpenAI, Anthropic, Google Dialogflow, etc.).

  • Exposure to LLM evaluation frameworks (e.g., prompt testing, response accuracy scoring).

  • Background in contact center analytics, speech analytics, or customer experience optimization.

  • Knowledge of compliance and quality standards in customer operations (PCI, HIPAA, GDPR).

Benefits

  • Evaluate and shape the next generation of AI-driven customer experiences.

  • Turn conversation data into measurable business impact.

  • Work across analytics, QA, and client collaboration to ensure that AI agents deliver real value.

  • Be at the intersection of AI performance, customer satisfaction, and business intelligence.

#J-18808-Ljbffr


Required Skill Profession

Business Intelligence & Analysis



Your Complete Job Search Toolkit

✨ Smart • Intelligent • Private • Secure

Start Using Our Tools

Join thousands of professionals who've advanced their careers with our platform

Rate or Report This Job
If you feel this job is inaccurate or spam kindly report to us using below form.
Please Note: This is NOT a job application form.


    Unlock Your AI Quality Potential: Insight & Career Growth Guide