Senior Data Scientist - in Ireland / remote

Job description

BMC empowers nearly 80% of the Forbes Global 100 to accelerate business value, faster than humanly possible. Our industry-leading portfolio unlocks human and machine potential to drive business growth, innovation, and sustainable success. BMC does this in a simple and optimized way by connecting people, systems, and data that power the world’s largest organizations so they can seize a competitive advantage. We're looking for a Senior Data Scientist to join our AMI group. This hands-on role combines Machine Learning, statistical modeling, Generative AI, RAG, model evaluation, and fine-tuning for enterprise AI applications. AMI supports commercial AI providers and on-premises models, with agents, MCP tools, and A2A interoperability delivered in partnership with AI Engineering. About the Role: You will independently design, develop, evaluate, and operationalize complex AI/ML solutions. You will contribute hands-on technical expertise, apply rigorous experimentation and MLOps/LLMOps practices, improve modeling and evaluation approaches within assigned initiatives, and mentor other Data Scientists. You will be a part of the AI Migration team , responsible for building tools that help customers transition from competitor products to BMC solutions. You'll work at the intersection of software engineering and AI, combining code-based conversion with AI-assisted processing to tackle migration challenges. Your work will help reduce manual effort and enable services to deliver accurate, timely customer migrations Key Responsibilities: Machine Learning & Applied Data Science • Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases. • Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques; compare learned approaches with rules, embeddings, and LLM-based baselines. • Use statistical modeling to estimate execution latency, token demand, inference costs, workload trends, and resource requirements. • Apply clustering and anomaly detection to identify recurring intents, capability gaps, unusual agent behavior, and execution regressions. • Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments. Foundation Models, Prompting & Context Engineering • Evaluate commercial API and self-hosted open-weight models for accuracy, reasoning, instruction adherence, structured outputs, latency, throughput, and total cost. • Design and optimize system instructions, prompt templates, few-shot examples, tool-calling prompts, and context assembly strategies. • Benchmark model alternatives for specific enterprise workloads and document quality/cost/latency tradeoffs. • Collaborate with AI Engineering and Platform Engineering on inference performance, quantization tradeoffs, and GPU/VRAM capacity assumptions. Retrieval-Augmented Generation (RAG) • Design and optimize document parsing, chunking, metadata, embedding, retrieval, reranking, and context selection strategies. • Evaluate vector, keyword/BM25, hybrid, and semantic retrieval, including access-aware retrieval where relevant. • Diagnose retrieval versus generation failures; measure Recall@K, nDCG, groundedness, faithfulness, and end-to-end answer quality. • Compare embedding and reranking models using representative enterprise datasets and reproducible benchmarks. Model Fine-Tuning & Adaptation • Assess when prompting, RAG, supervised instruction fine-tuning, or parameter-efficient adaptation is appropriate. • Curate instruction datasets and holdout evaluations; apply SFT, LoRA, QLoRA, or comparable methods. • Measure gains over base-model and prompting/RAG baselines, including generalization, regressions, data leakage, and operating costs. • Collaborate on the packaging and validation of fine-tuned model artifacts and adapters. Agentic AI, MCP & A2A • Apply modeling and evaluation expertise to intent routing, agent/skill discovery, MCP tool selection, workflow selection, and planning effectiveness. • Evaluate task completion, tool-call correctness, argument extraction, delegation quality, and recovery from failures. • Work with AI Engineering on agentic frameworks and understand MCP tool schemas and A2A Agent Cards, skills, and task delegation. • Contribute decision models and quality signals without owning agent runtime, MCP server, or A2A protocol implementation. AI Evaluation & Experimentation • Design golden datasets, holdout sets, human review protocols, LLM-as-judge calibration, and statistical comparison methods. • Automate evaluations and regression checks for prompts, foundation models, RAG, fine-tuned models, routing, and agents. • Define measurable quality thresholds for accuracy, relevance, groundedness, instruction following, task completion, and cost/performance. • Investigate regressions and use evidence to guide model, prompt, retrieval, and routing decisions. • Implement and maintain reproducible experimentation, model evaluation, and lifecycle workflows for assigned AI/ML initiatives, including automated validation and production readiness checks. MLOps, LLMOps & Model Lifecycle • Apply reproducible experiment tracking across datasets, code, model versions, hyperparameters, prompts, retrieval configurations, metrics, and artifacts. • Develop and maintain automated training, fine-tuning, benchmarking, and evaluation pipelines, with model registry and artifact traceability. • Define and integrate quality gates, regression tests, and model/prompt/RAG release-readiness criteria with engineering teams. • Analyze production quality and drift signals, agent routing accuracy, latency, throughput, and token usage; recommend remediation. • Collaborate with engineering on packaging, deployment handoffs, observability, rollback criteria, and ongoing model lifecycle management. Technical Leadership & Collaboration • Independently lead complex Data Science initiatives from problem formulation through model development, evaluation, and production handoff. • Make technical decisions within assigned initiatives, balancing model quality, performance, cost, and delivery requirements. • Improve reusable modeling, benchmarking, experimentation, and MLOps/LLMOps practices within assigned initiatives. • Collaborate with AI Engineering, AI Quality, Platform Engineering, Product Management, and domain experts. • Translate business requirements into measurable AI and ML objectives. • Mentor junior and mid-level Data Scientists and contribute to technical reviews. • Communicate complex AI and ML concepts clearly to technical and non-technical stakeholders. Must-Have Skills & Experience • 8+ years of relevant Data Science, Machine Learning, or Applied AI experience. • Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience. • Advanced Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills. • Strong statistics, experimental design, hypothesis testing, ML validation, and applied classification, regression, ranking, clustering, forecasting, and anomaly detection knowledge. • Hands-on LLM prompting, embeddings, model comparison, and RAG design/optimization, including chunking, retrieval, and reranking. • Practical instruction fine-tuning or parameter-efficient model adaptation experience, with measurable baseline comparisons. • Experience designing evaluation datasets, automated quality metrics, and statistical model/LLM/RAG experiments. • Hands-on MLOps and LLMOps: experiment tracking, model/dataset/prompt versioning, reproducible pipelines, registries/artifacts, evaluation automation, quality gates, and monitoring. • Working knowledge of agent planning, routing, tool calling, MCP, A2A, and multi-agent patterns. • Unde