Bengaluru QA AI Evaluator 560103

Job description

The QA AI Evaluator will be responsible for assuring the quality, reliability, governance, and business effectiveness of Agentic AI solutions. The role focuses on evaluating AI agents, prompts, LLM outputs, and end-to-end workflows to ensure accurate, consistent, and trusted outcomes. • Define AI evaluation frameworks, standards, KPIs, and success criteria. • Validate agent accuracy, reasoning, reliability, workflow quality, and business outcomes. • Assess prompts, LLM responses, hallucination risks, and AI-generated content quality. • Benchmark models, tools, prompts, and agent configurations to improve performance. • Ensure Responsible AI, security, privacy, governance, and compliance requirements are met. • Provide clear insights, risks, recommendations, and scorecards to leadership and stakeholders. What you’ll be doing •Strong experience in AI Testing, Quality Engineering, AI/LLM Evaluation, or Agentic AI Assurance. •Deep understanding of LLMs, Prompt Engineering, RAG, AI Agents, and Multi-Agent Architectures. •Expertise in defining evaluation frameworks, quality metrics, KPIs, scorecards, and benchmarking approaches. •Experience validating AI accuracy, reasoning, reliability, hallucinations, and business outcomes. •Knowledge of Responsible AI, Security, Privacy, Governance, Risk, and Compliance practices. •Strong analytical and problem-solving skills with experience in root cause analysis and continuous improvement. •Familiarity with AI evaluation tools, test automation, and data-driven quality assessment techniques. •Ability to collaborate effectively with Product, Engineering, Architecture, QA, and Business stakeholders. •Excellent communication and stakeholder management skills, including presenting insights to leadership. •Experience driving AI adoption, change management, and business transformation initiatives. Essential Skills / Experience •4-6 Years of experience with strong experience in AI testing, quality engineering, AI/LLM evaluation, or Agentic AI assurance. •Good understanding of LLMs, prompt engineering, RAG, AI agents, and multi-agent workflows. •Ability to define evaluation frameworks, quality metrics, KPIs, scorecards, and success criteria. •Hands-on experience validating AI accuracy, reasoning, reliability, hallucination risk, relevance, completeness, and business outcomes. •Strong analytical, problem-solving, stakeholder management, and communication skills. Desirable Skills / Experience •Minimum 3-4 years of experience with AI evaluation tools, automation frameworks, and data-driven quality assessment techniques. •Exposure to benchmarking models, prompts, tools, and agent configurations to improve performance. •Experience supporting AI adoption, user acceptance, change management, or wider business transformation initiatives. BT is the UK’s leading communications group and the holding company behind some of the country’s most recognised brands – including BT, EE, Openreach and Plusnet. Our purpose is as simple as it is ambitious: we connect for good. Our customers include consumers, small, medium and large businesses, public sector organisations and other communications providers. Having come through the most capital-intensive phase of our fibre investment, our focus now is on what comes next – simplifying how we operate, using technology and AI to work smarter, and organising ourselves to serve customers better and grow sustainably. We have a singular culture that unites all our people: we are customer-first challengers, who are committed, clear and connected. These behaviours unite us as one team to deliver for our colleagues, our customers, our stakeholders and the country. Joining BT means working at the heart of a business that matters to the UK, with the opportunity to shape decisions, influence outcomes and help set the future course of one of the country’s most important companies. Apply Now Start applying with LinkedIn Please wait... Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. ©BT Group 2026 Cookie Consent Manager When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. Because we respect your right to privacy, you can choose not to allow some types of cookies. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer. Required Cookies These cookies are required to use this website and can't be turned off. Required Cookies Required Cookies Provider Description Enabled SAP as service provider We use the following session cookies, which are all required to enable the website to function: "route" is used for session stickiness "careerSiteCompanyId" is used to send the request to the correct data centre "JSESSIONID" is placed on the visitor's device during the session so the server can identify the visitor "Load balancer cookie" (actual cookie name may vary) prevents a visitor from bouncing from one instance to another Cookies from provider SAPasserviceprovider are required and cannot be turned off Functional Cookies These cookies provide a better customer experience on this site, such as by remembering your login details, optimising video performance or providing us with information about how our site is used. You may freely choose to accept or decline these cookies at any time. Note that certain functionalities that these third-parties make available may be impacted if you do not accept these cookies. Consent to all Functional Cookies Functional Cookies Provider Description Enabled Vimeo Vimeo is a video hosting, sharing and services platform focused on the delivery of video. Opting out of Vimeo cookies will disable your ability to watch or interact with Vimeo videos. Cookie Policy Privacy Policy Terms and Conditions Consent to cookies from provider Vimeo YouTube YouTube is a video-sharing service where users can create their own profile, upload videos, watch, like and comment on videos. Opting out of YouTube cookies will disable your ability to watch or interact with YouTube videos. Cookie Policy Privacy Policy Terms and Conditions Consent to cookies from provider YouTube Advertising Cookies These cookies serve ads that are relevant to your interests. You may freely choose to accept or decline these cookies at any time. Note that certain functionality that these third parties make available may be impacted if you do not accept these cookies. Consent to all Advertising Cookies Advertising Cookies Provider Description Enabled Google Tag Manager Google Tag Manager is a tag management system for conversion tracking, site analytics, remarketing and more. Privacy Policy Terms and Conditions Consent to cookies from provider GoogleTagManager LinkedIn LinkedIn is an employment-oriented social networking service. We use the Apply with LinkedIn feature to allow you to apply for jobs using your LinkedIn profile. Opting out of LinkedIn cookies will disable your ability to use Apply with LinkedIn. Cookie Policy Cookie Table Privacy Policy Terms and Conditions Consent to cookies from provider LinkedIn