Track Manager AWS IAC, Terraform,Python

Job description

Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails Key Responsibilities Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails Skill Requirements Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails Other Requirements Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard. Key Responsibilities • Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring • Build drift detection against each agent\'s golden-set baseline • Wire thumbs-down/negative-feedback signals into a candidate eval-set queue • Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost) • Respect data-classification gating on sampled prompt/completion content (capture default OFF) Required Skills • 6+ years engineering experience with observability/monitoring pipeline background • Experience with OpenTelemetry collectors/pipelines and a dashboard tool (Splunk, New Relic, Grafana, CloudWatch) • Comfortable with async/sidecar architecture patterns • Understanding of data-classification-aware logging Nice to Have • Experience building LLM-judge-based drift/regression detection specifically • Familiarity with Bedrock Guardrails or similar content-safety signals • Exposure to FS/regulated-industry monitoring-control requirements Core Tools / Stack: OTel Collector, Splunk/New Relic/CloudWatch, Python, AWS Lambda/ECS, Bedrock Guardrails Apply now Information at a Glance Why HCLTech? At HCLTech, you'll supercharge your potential. You'll find your career. And you'll find your spark. All at a place that knows that helping its customers stay on top starts by putting its people first. HCLTech is a global technology company, home to more than 223,000 people across 60 countries, delivering industry-leading capabilities centered around digital, engineering, cloud and AI, powered by a broad portfolio of technology services and products. We work with clients across all major verticals, providing industry solutions for Financial Services, Manufacturing, Life Sciences and Healthcare, Technology and Services, Telecom and Media, Retail and CPG, and Public Services. Consolidated revenues as of 12 months ending June 2026 totaled $14.8 billion. Disclaimer Privacy Statement Terms of use Sitemap Raise a Grievance We use cookies on our site. Please read more about cookies policy here. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Opens in a new tab. Copyright © 2026 HCL Technologies Limited Cookie Consent Manager When you visit any website, it may store or retrieve information on your browser, mostly in the form of cookies. Because we respect your right to privacy, you can choose not to allow some types of cookies. However, blocking some types of cookies may impact your experience of the site and the services we are able to offer. Required Cookies These cookies are required to use this website and can't be turned off. Required Cookies Required Cookies Provider Description Enabled SAP as service provider We use the following session cookies, which are all required to enable the website to function: "route" is used for session stickiness "careerSiteCompanyId" is used to send the request to the correct data center "JSESSIONID" is placed on the visitor's device during the session so the server can identify the visitor "Load balancer cookie" (actual cookie name may vary) prevents a visitor from bouncing from one instance to another Cookies from provider SAPasserviceprovider are required and cannot be turned off Functional Cookies These cookies provide a better customer experience on this site, such as by remembering your login details, optimizing video performance, or providing us with information about how our site is used. You ma