Applied AI Analyst II

Job description

At Bristol Myers Squibb, our employees often ask, “Who are you working for?”—a question that fuels collaboration, accountability, and urgency in our work. Our purpose-driven culture inspires us to discover, develop, and deliver innovative medicines to prevail over serious diseases. We offer uniquely interesting and meaningful work, opportunities for growth, and a supportive environment that values inclusion, wellbeing, flexibility, and comprehensive benefits. This is work that transforms the lives of patients, and the careers of those who do it. The Applied AI Analyst II, Enabling Functions will help evolve Enabling Functions Analytics from static reporting toward intelligent, conversational, and reusable analytics products. The role combines practical AI research, context engineering, semantic modelling, and hands-on prototyping to connect governed enterprise data with large language models and modern analytics platforms. The individual will investigate how LLMs process context and generate responses, translate those learnings into reliable solution patterns, and build AI-enabled experiences using Claude, Databricks, Power BI, and related technologies. The role includes building text-to-SQL workflows, tool-using agents, and multi-agent systems that answer business questions reliably and transparently Roles & ResponsibilitiesApply working knowledge of concepts and procedures to a variety of analytics and AI assignments, using judgement to solve problems with a moderate level of guidance. Own well-defined research, experimentation, and prototyping workstreams, documenting assumptions, decisions, results, and limitations. Research and test LLM behaviours, reasoning patterns, context-window usage, tool use, structured outputs, and failure modes, and convert findings into practical design guidance and reusable patterns for analytics use cases. Design and optimise system instructions, business context, examples, metadata, retrieval strategies, and tool interfaces so AI responses are grounded, relevant, and consistent. Manage context quality, freshness, and token efficiency. Build text-to-SQL workflows that translate natural-language questions into governed queries, grounded in certified semantic models and metric definitions, with validation steps that check generated SQL before execution. Design tool-using agents that call approved functions and data services (such as SQL execution, metric lookup, and visualization generation) with clearly defined tool schemas, permissions, and fallback behaviour. Design and prototype multi-agent workflows in which specialist agents (for example, query planning, SQL generation, validation, and insight narration) coordinate under an orchestrator, with clear handoffs, state management, and loop and cost limits. Prototype chat-with-data and chat-with-document experiences that let users explore governed business information in natural language, while preserving traceability to approved data and metric definitions. Design and improve business-ready semantic models covering grain, dimensions, measures, relationships, hierarchies, synonyms, descriptions, and certified definitions, ensuring the same logic supports Power BI, Databricks, and AI experiences. Evaluate when to use prompting, retrieval-augmented generation, tool calling, multi-agent workflows, or traditional analytics, and define fit-for-purpose architectures rather than applying LLMs where deterministic methods are more appropriate. Create test cases and evaluation datasets for SQL accuracy, tool-call correctness, agent task completion, groundedness, latency, and cost. Investigate hallucinations, retrieval or semantic failures, and agent loops, and recommend corrective actions. Build proof-of-value solutions using Python, SQL, APIs, Claude, and Databricks, integrating AI capabilities with existing analytics assets and, where appropriate, Power BI or lightweight application experiences. Break ambiguous business questions into clear problem statements, hypotheses, data needs, and solution options. Rapidly learn unfamiliar domains and communicate trade-offs to technical and non-technical stakeholders. Document patterns, reusable components, guardrails, and lessons learned, and contribute to internal standards for context engineering, semantic models, LLM evaluation, and responsible AI adoption. Partner with business analysts, BI developers, data engineers, AI/ML specialists, architects, and governance teams to move successful prototypes toward sustainable products. Balance innovation with responsible AI, data governance, security, privacy, quality, and production-readiness expectations, including human-in-the-loop controls for consequential actions. Skills and competenciesStrong analytical thinking, curiosity, and structured problem-solving, including comfort with ambiguous questions and incomplete information. Practical understanding of how large language models work, including tokenisation, embeddings, attention, context windows, inference, common failure modes, and the difference between model knowledge and supplied context. Hands-on expertise in prompt and context engineering, including instruction design, few-shot examples, grounding, retrieval-augmented generation, structured outputs, tool use, and context optimisation. Strong SQL workflow skills: writing and reviewing complex queries, designing text-to-SQL pipelines, enforcing read-only access and query guardrails, validating generated SQL against schema and metric definitions, and diagnosing incorrect joins, grain mismatches, and aggregation errors. Experience designing tool-using agents, including function and tool schema design, argument validation, error handling, retries, and permission boundaries. Working knowledge of multi-agent architectures such as orchestrator-worker, planner-executor, and specialist-agent patterns, including state management, handoffs between agents, termination conditions, and observability across agent steps. Hands-on experience with Anthropic Claude or comparable enterprise LLM platforms, including API-based integration, prompt orchestration, tool calling, and solution evaluation. Hands-on experience with Databricks, including notebooks, SQL Warehouses, governed data assets, and lakehouse-based analytics workflows. Exposure to Unity Catalog, AI/BI Genie, MLflow, or Mosaic AI is valuable. Experience designing and integrating REST APIs or equivalent service interfaces to connect LLMs, enterprise data platforms, and application experiences. Proficiency in Python and SQL for data exploration, AI prototyping, integration, evaluation, and automation. Experience with Git-based source control, branching, and pull-request practices, together with CI/CD pipelines for automated build, test, and deployment Hands-on understanding of AWS deployment patterns for AI and analytics solutions, including environment configuration, security, logging, monitoring, and release processes is a plus Experience designing or consuming semantic models, with strong understanding of business metrics, dimensional modelling, relationships, grain, hierarchies, synonyms, and reusable business definitions. Experience with vector search, agent frameworks, Model Context Protocol, LLM observability, or automated evaluation methods. Ability to design experiments and evaluation datasets and assess groundedness, relevance, consistency, completeness, latency, and cost. Ability to present complex information to non-technical stakeholders in a clear and concise manner, and to create relevant and engaging PowerPoint presentations. Awareness of responsible AI, privacy, cybersecurity, data governance, access control, and human-in-the-loop controls. Clear written and verbal communication, with the ability to translate technical AI concepts into practical business implications and document reusable engineering patterns. Ability to work collaboratively in cross-functional teams and manage multiple projects simultaneously. Good