EY - GDS Consulting - AIA - Gen AI - Senior
EY · Pune
At EY, you’ll have the chance to build a career as unique as you are, with the global scale, support, inclusive culture and technology to become the best version of you. And we’re counting on your unique voice and perspective to help EY become even better, too. Join us and build an exceptional experience for yourself, and a better working world for all.
AI QA – Senior
Educational qualification: BTech/Masters/PhD
The opportunity
We are seeking an experienced AI QA Engineer with 5–9+ years of professional experience and deep expertise in AI testing, quality engineering, validation frameworks, and enterprise AI solution assurance. The ideal candidate should possess strong hands-on experience validating AI systems, Agentic AI workflows, RAG applications, conversational AI solutions, and enterprise-scale AI platforms.
The individual will play a key role in ensuring the quality, reliability, security, safety, and performance of AI systems while collaborating with AI engineers, architects, developers, and business stakeholders to establish robust quality assurance practices across AI and Generative AI implementations.
Important note: This role requires professional implementation experience in relevant AI technologies. Personal portfolio projects, cloned demos, certifications, or tutorial-based builds are helpful, but are not sufficient on their own.
Application guidance: Candidates should be able to clearly explain at least two relevant AI implementations, including the business use case, their direct contribution, the technical approach, and the outcome delivered.
Your key responsibilities
- Design, develop, and execute comprehensive testing strategies for AI, GenAI, Agentic AI, and Agentic RAG systems.
- Validate Large Language Model outputs for accuracy, consistency, relevance, safety, and compliance requirements.
- Develop test frameworks and automation suites for AI services, copilots, AI assistants, and enterprise AI applications.
- Design and execute evaluation frameworks for RAG systems, retrieval pipelines, vector search solutions, and knowledge-based AI applications.
- Perform functional, integration, regression, performance, reliability, and scalability testing of AI-powered applications.
- Develop automated testing frameworks using Python to validate AI workflows, APIs, models, and orchestration pipelines.
- Validate prompt engineering approaches, agent workflows, tool usage patterns, and reasoning capabilities in Agentic AI systems.
- Execute AI model testing across conversational AI, multimodal AI, and intelligent automation solutions.
- Perform AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and security validation activities.
- Assess AI solutions for bias, hallucinations, content quality, safety risks, and responsible AI compliance requirements.
- Validate integrations between AI systems, enterprise applications, APIs, databases, and cloud platforms.
- Utilize RAG evaluation frameworks such as Ragas to measure retrieval quality, answer faithfulness, relevance, and response effectiveness.
- Develop testing dashboards, quality metrics, defect reporting processes, and continuous quality improvement practices.
- Collaborate closely with AI engineers and architects to identify quality risks and improve operational reliability.
- Contribute to quality standards, governance controls, testing accelerators, reusable frameworks, and best practices.
- Stay current with advancements in AI testing, evaluation methodologies, Agentic AI, AI security, and emerging quality assurance technologies.
Skills and Attributes:
Professional Experience
- 5–9+ years of total experience, including 3+ years of directly relevant hands-on experience in AI/ML, applied AI, or GenAI systems.
- Proven experience building production, pre-production, or enterprise pilot AI applications.
- Strong hands-on depth in at least 3 of the following areas, with meaningful ownership in at least 2:
- LLM application development.
- Retrieval pipelines.
- Agent or tool orchestration.
- Evaluation, guardrails, or quality systems.
- Model adaptation or fine-tuning.
- Cloud deployment or optimization of AI applications.
- Conversational AI.
- Multimodal AI.
- Backend engineering for AI systems.
- Strong software engineering foundation with ability to build reliable, maintainable AI services.
- Able to clearly explain direct contributions to delivered AI implementations.
Educational Background
- Bachelor’s/master’s degree in computer science, Data Science, Engineering, or related field.
Technical Skills
- Strong understanding of Large Language Models (LLMs), Generative AI, Agentic AI, and Agentic RAG systems.
- Hands-on experience validating and testing AI-powered applications and enterprise AI platforms.
- Experience developing AI testing strategies, validation frameworks, and automated test suites.
- Strong expertise in AI Model Testing, functional testing, regression testing, integration testing, and end-to-end quality validation.
- Experience performing AI penetration testing, adversarial testing, prompt injection testing, jailbreak testing, and vulnerability assessments.
- Strong understanding of prompt engineering validation, reasoning evaluation, and response quality assessment techniques.
- Experience implementing evaluation frameworks and quality measurements for Agentic AI workflows and AI agents.
- Hands-on experience with Ragas and similar evaluation frameworks for assessing retrieval quality, answer correctness, faithfulness, relevance, and grounding.
- Experience validating Agentic RAG systems, retrieval pipelines, and knowledge-based AI applications.
- Experience testing AI integrations across APIs, enterprise applications, databases, and cloud services.
- Strong understanding of AI observability, evaluation metrics, monitoring, and performance benchmarking.
- Experience working with Microsoft Azure AI Platform and Azure AI services.
- Hands-on experience developing automated test frameworks and validation utilities using Python.
- Familiarity with FastAPI-based services and API testing frameworks.
- Experience designing and executing automated API testing strategies.
- Knowledge of Responsible AI principles, AI safety controls, governance requirements, and regulatory compliance considerations.
- Experience evaluating hallucinations, bias, toxicity, safety violations, and model robustness.
- Strong understanding of software quality engineering, secure development practices, test automation frameworks, and CI/CD testing integration.
- Experience working with performance testing, scalability testing, reliability testing, and production-readiness validation.
- Familiarity with retrieval augmentation architectures, vector databases, embeddings, and enterprise search solutions.
- Strong analytical, troubleshooting, and root-cause analysis skills.
Soft Skills
- Excellent problem-solving skills and ability to connect AI capabilities to business value.
- Strong communication and presentation skills.
Other Responsibilities:
- Contribute to end-to-end quality assurance activities across AI and GenAI engagements.
- Collaborate with AI engineers, architects, data scientists, and product teams to establish robust testing and validation practices.
- Participate in AI design reviews, risk assessments, model evaluations, and quality governance initiatives.
- Ensure AI solutions meet functional, security, compliance, governance, and business requirements before production deployment.
- Support development of reusable testing assets, automation frameworks, quality accelerators, and evaluation methodologies.
- Contribute to AI governance, Responsible AI, risk management, and compliance initiatives.
- Support presales, innovation initiatives, technical demonstrations, and client workshops where AI testing expertise is required.
- Provide mentorship to junior QA engineers and help establish AI quality engineering best practices.
Why Join Us
- Be at the forefront of AI-driven innovation across multiple client sectors.
- Work with global clients to drive real business impact.
- Collaborate with a team of AI experts, analytics leaders, and industry specialists in a highly entrepreneurial environment.
What we offer
EY Global Delivery Services (GDS) is a dynamic and truly global delivery network. We work across six locations – Argentina, China, India, the Philippines, Poland and the UK – and with teams from all EY service lines, geographies and sectors, playing a vital role in the delivery of the EY growth strategy. From accountants to coders to advisory consultants, we offer a wide variety of fulfilling career opportunities that span all business disciplines. In GDS, you will collaborate with EY teams on exciting projects and work with well-known brands from across the globe. We’ll introduce you to an ever-expanding ecosystem of people, learning, skills and insights that will stay with you throughout your career.
- Continuous learning: You’ll develop the mindset and skills to navigate whatever comes next.
- Success as defined by you: We’ll provide the tools and flexibility, so you can make a meaningful impact, your way.
- Transformative leadership: We’ll give you the insights, coaching and confidence to be the leader the world needs.
- Diverse and inclusive culture: You’ll be embraced for who you are and empowered to use your voice to help others find theirs.
EY | Building a better working world
EY exists to build a better working world, helping to create long-term value for clients, people and society and build trust in the capital markets.
Enabled by data and technology, diverse EY teams in over 150 countries provide trust through assurance and help clients grow, transform and operate.
Working across assurance, consulting, law, strategy, tax and transactions, EY teams ask better questions to find new answers for the complex issues facing our world today.