Senior Technical Lead

Job description

AI QE Role Summary The AI Quality Engineer (AI QE) is responsible for validating the quality, reliability, safety, and user experience of AI and Generative AI solutions. The role combines software quality engineering with AI evaluation to test Large Language Model (LLM) applications, copilots, chatbots, Retrieval-Augmented Generation (RAG) systems, and agentic workflows across real-world user scenarios. The AI QE establishes repeatable evaluation methods, ground-truth datasets, scoring rubrics, regression suites, and quality reporting to ensure AI-powered features are accurate, grounded, relevant, complete, and production-ready. Key Responsibilities Key Responsibilities · Validate AI-powered features, LLM applications, copilots, chatbots, agents, and conversational experiences across functional and non-functional requirements. · Design test strategies for deterministic application behavior and probabilistic AI outputs, covering positive, negative, adversarial, boundary, and edge-case scenarios. · Test agent behavior across user personas, multi-step workflows, tool usage, function calling, memory, routing, fallback, and error-recovery paths. · Evaluate AI-generated responses across single-turn and multi-turn conversations for factual correctness, grounding, relevance, completeness, coherence, and adherence to instructions. · Create and maintain ground-truth, golden, and benchmark datasets for evaluation and regression testing. · Define evaluation rubrics, metrics, scoring scales, acceptance thresholds, and detailed rationale for qualitative and quantitative assessments. · Perform Human-in-the-Loop reviews and analyze agreement, variance, and recurring quality gaps across evaluators. · Validate RAG pipelines, including retrieval relevance, context utilization, citation accuracy, faithfulness, and hallucination behavior. · Test prompt templates and prompt changes for robustness, consistency, injection resistance, and unintended regressions. · Build or execute automated AI evaluation and regression suites, integrating quality gates into CI/CD pipelines where applicable. · Log defects with reproducible evidence, expected behavior, evaluation scores, rationale, and business impact. · Produce quality dashboards, evaluation reports, trend analysis, release recommendations, and improvement feedback for engineering and product teams. · Collaborate with AI Engineers, Fullstack Engineers, LLMOps, Observability, Product, Security, and Responsible AI teams. AI Evaluation & Quality Scope · LLM and conversational evaluation: response correctness, relevance, completeness, consistency, tone, instruction following, context retention, and multi-turn coherence. · Grounding and RAG validation: retrieval precision, context relevance, faithfulness, citation validation, unsupported claims, and hallucination detection. · Agentic workflow testing: planning, routing, tool selection, tool-call arguments, execution sequence, memory, guardrails, retries, and termination behavior. · Safety and Responsible AI testing: prompt injection, jailbreak attempts, harmful or restricted outputs, sensitive-data leakage, bias checks, and policy compliance. · Operational quality validation: latency, availability, throughput, cost, failure handling, observability signals, and production-readiness criteria. Skill Requirements Required Skills & Experience · Hands-on experience in software quality engineering, test design, defect management, and test reporting. · Practical exposure to Generative AI, LLMs, copilots, chatbots, RAG systems, and AI agents. · Experience defining evaluation rubrics, metrics, scoring frameworks, acceptance thresholds, and evaluation rationale. · Experience creating ground-truth or golden datasets and conducting Human-in-the-Loop reviews. · Understanding of prompt engineering and testing prompt behavior across varied user inputs and scenarios. · Ability to assess grounding, factual correctness, relevance, completeness, hallucinations, and conversational quality. · Experience with API testing and automation using Python and tools such as PyTest, REST clients, or equivalent frameworks. · Familiarity with AI evaluation or observability frameworks such as RAGAS, DeepEval, promptfoo, LangSmith, Azure AI evaluation capabilities, or equivalent. · Working knowledge of CI/CD, test automation, version control, and cloud-based application environments. · Strong analytical, communication, documentation, and stakeholder-reporting skills. Preferred / Nice-to-Have · Experience with LLM-as-a-judge approaches, pairwise comparison, model-based evaluation, and correlation with human evaluation. · Knowledge of statistical sampling, inter-rater agreement, confusion matrices, precision/recall, and quality trend analysis. · Exposure to performance, security, accessibility, localization, or usability testing for AI-enabled applications. · Familiarity with Responsible AI controls, privacy requirements, auditability, and regulated-domain quality practices. · Experience testing enterprise AI solutions deployed on Azure, AWS, or GCP. Experience Level · 4–8 years of overall software testing, quality engineering, automation, or AI/ML validation experience. · 1–3+ years of hands-on exposure to AI, ML, Generative AI, conversational AI, or data/model evaluation is preferred. · Proven experience validating complex digital products or production-grade AI-enabled applications. Success Measures · Repeatable and auditable AI evaluation frameworks established for key use cases. · Higher accuracy, grounding, relevance, completeness, and consistency of AI-generated outputs. · Reduced hallucinations, unsafe responses, escaped defects, and regressions across releases. · Effective automated and Human-in-the-Loop quality gates supporting confident production releases. · Clear quality reports and trend insights that drive measurable product and model improvements Other Requirements Good Communication Skils Apply now Information at a Glance Why HCLTech? At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first. HCLTech is a global technology company, home to more than 230,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending September 2026 totaled $14.8 billion. Disclaimer Privacy Statement Terms of use Sitemap Raise a Grievance We use cookies on our site. Please read more about cookies policy here. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Copyright © 2026 HCL Technologies Limited Cookie Consent Manager When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. Because we respect your right to privacy, you can choose not to allow some types of cookies. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer. Required Cookies These cookies are required to use this website and can't be turned off. Required Cookies Required Cookies Provider Description Enabled SAP as service provider We use the following session cookies, which are all required to enable the website to function: "route" is used for session stickiness "careerSiteCompanyId" is used to send the request to the correct data center "JSESSIONID" is placed on the visitor's device during the session so the server can identify the visitor "Load balancer cookie" (actual cookie name may vary) prevents a visitor from bouncing from one i