Senior Quality Assurance Engineer
Pearson · LK
Job Title: Senior Quality Assurance Engineer
Description: Senior Quality Assurance Engineer role focused on testing and improving an LLM-powered AI Support Assistant, with emphasis on conversational AI quality, RAG validation, integration testing, automation, and scalable quality monitoring across EdTech platforms. This role aligns to industry level titles such as Lead Quality Engineer, Senior Software Test Engineer, Senior AI Test Engineer
About the Role
We’re looking for a Senior Quality Engineer to own the quality strategy for our AI Support Assistant, a context-aware, LLM-powered support tool embedded across our digital learning platform, including admin consoles and student-facing dashboards. This assistant uses runtime context (user role, product, browser, error metadata) to deliver accurate, personalized troubleshooting to millions of students and instructors annually. This is a hands-on QE role centered on AI and conversational testing — you’ll spend most of your time evaluating LLM response quality, not traditional UI testing.
You’ll be responsible for ensuring the assistant is accurate, reliable, and delivers a seamless experience across a complex integration landscape spanning multiple LMS platforms, e-commerce flows, and entitlement systems.
What You’ll Do
AI & Conversational Testing
- Design and execute test strategies for LLM-powered conversational flows, including response accuracy, relevance, assumption detection, and hallucination prevention
- Build and maintain evaluation datasets to measure assistant performance against target metrics (95% accuracy, 75% CSAT)
- Validate context injection — ensure the assistant correctly receives and uses runtime context (role, product, LMS type, browser, workflow state, structured error/diagnostic codes) to tailor responses
Integration & End-to-End Testing
- Test the context-sharing contract between front-end applications and the AI Support Assistant — verifying session storage writes, schema compliance, and read interfaces
- Validate cross-platform behavior across major LMS integrations and multiple product lines across the courseware portfolio
- Test entitlement and enrollment and third-party content-provider provisioning flows
Quality Infrastructure
- Build automated regression suites for assistant response quality, context propagation, and UI behavior (chat widget placement, discoverability, mobile responsiveness)
- Develop performance and load testing strategies for projected scale (millions of interactions annually, growing year over year)
- Establish monitoring and alerting for production assistant accuracy, escalation rates, and context-pass-through failures
Collaboration
- Partner with ML/AI engineers to define acceptance criteria for model updates and prompt changes
- Work closely with product management to translate user research findings (e.g., accuracy as the #1 trust driver, step-based answers over article links) into testable requirements
- Coordinate with backend platform and Customer Success teams on dependency validation
What You Bring
Required
- 5+ years in software quality engineering, with at least 2 years testing AI/ML systems, conversational AI, chatbots, or NLP-driven products
- Hands-on experience testing LLM-based applications — prompt evaluation, response grading, factual accuracy measurement, and regression testing for non-deterministic outputs
- Hands-on experience validating Retrieval-Augmented Generation (RAG) pipelines, including retrieval accuracy, context grounding, and relevance validation
- Experience testing semantic search and vector-based retrieval systems (embeddings, similarity scoring, ranking relevance)
- Experience with LLM Evaluator frameworks and LLM-as-judge methodologies for automated, scalable scoring of model outputs
- Experience building evaluation harnesses for LLM outputs, including automated scoring and human-in-the-loop review pipelines
- Strong experience with API testing (REST) and integration testing across distributed systems
- Proficiency with test automation frameworks (Selenium, Playwright, Cypress, or similar) and CI/CD pipelines
- Strong scripting/programming ability (e.g., Python, JavaScript, or Java) to build custom test tooling, evaluation harnesses, and data pipelines
- Experience testing across multiple browsers, devices, and platforms — including mobile web
- Solid understanding of session/local storage, client-side state management, and front-end data contracts
- Familiarity with e-commerce or EdTech platforms — checkout flows, entitlements, user provisioning
Preferred
- Experience with LMS platforms (e.g., Canvas, Blackboard, Moodle) and LTI integrations
- Familiarity with MCP (Model Context Protocol) or similar AI tool-integration patterns
- Knowledge of accessibility testing (WCAG 2.1 AA) for embedded chat interfaces
- Experience with performance/load testing tools (k6, Locust, JMeter)
- Familiarity with Jira, Confluence, and Agile workflows
Why This Role Matters
The AI Support Assistant is Pearson’s first LLM-powered, context-aware support experience — and quality is the make-or-break factor. User research is clear: accuracy is the #1 driver of trust. If the assistant gives wrong answers, users won’t come back. Your work directly determines whether millions of students get instant, reliable help or hit a dead end.
Our Stack
Jira · Confluence · LMS Platforms (Canvas, Blackboard, Moodle) · E-commerce Platform · LTI 1.3 · REST APIs · Session Storage · LLM/AI Services
What We Offer
- Competitive salary and performance-based bonus.
- Comprehensive health, dental, and vision benefits.
- Generous PTO, holidays, and flexible working arrangements.
- Annual learning and development budget.
- Access to Pearson's full catalog of learning products and certifications.
- Opportunity to shape quality practices across a globally recognized education technology organization.
About Pearson
Pearson is the world's leading learning company, delivering education to over 160 countries. Our digital-first strategy powers millions of learners through a growing portfolio of integrated learning tools, assessments, and LMS-connected experiences. We are committed to building inclusive, equitable, and effective digital learning—and quality engineering is at the heart of that mission.
#LI-PJ1
#LI-Hybrid
More jobs at Pearson · Software Engineering jobs in Sri Lanka