Staff AI Engineer

About the Opportunity

We’re seeking a Staff AI Engineer to lead the technical development of an agentic AI platform designed to investigate production incidents, analyze infrastructure behavior, and help engineering teams resolve complex operational problems.

This is a senior individual contributor position combining machine learning systems, large language models (LLMs), distributed systems, and production software engineering. You’ll shape the architecture behind autonomous investigation workflows, develop reliable agent execution systems, and establish measurable standards for AI quality.

The role requires hands-on expertise in both production ML/LLM systems and agentic AI engineering. You’ll have significant influence over technical decisions while remaining directly involved in software design, implementation, and troubleshooting.

Key Responsibilities

Agent Architecture and Platform Engineering

• Design the architecture supporting autonomous AI agents that investigate production infrastructure and application incidents.
• Develop multistep agent workflows involving tool execution, retrieval, reasoning, and external system integrations.
• Build persistent context and memory capabilities that allow agents to retain useful information across investigations.
• Develop knowledge representations, including service maps and knowledge graphs, that capture relationships between infrastructure components.
• Implement secure data handling and tenant isolation for enterprise environments.
• Improve how agents combine information from multiple sources to produce accurate, actionable findings.

Agent Reliability and Distributed Execution

• Build reliable execution frameworks for AI workflows operating under strict time and resource constraints.
• Implement checkpointing, recovery mechanisms, and fault-tolerant task execution.
• Improve tool-calling reliability and error handling across external integrations.
• Design graceful degradation strategies for unavailable services or incomplete information.
• Manage latency, execution deadlines, and computational cost across agent workflows.
• Develop telemetry that helps engineers understand agent behavior, failures, and performance.

AI Evaluation and Technical Leadership

• Design evaluation frameworks that measure agent accuracy, reliability, and task completion.
• Build evaluation datasets using realistic infrastructure incidents and operational scenarios.
• Establish regression testing and quality thresholds for AI platform releases.
• Improve evidence grounding so agent recommendations are supported by relevant system data.
• Lead architecture reviews, establish engineering standards, and guide technical implementation.
• Mentor AI engineers through design discussions, code reviews, and collaborative problem-solving.

Required Skills and Qualifications

Production AI and Software Engineering

• Approximately 5 to 15 years of experience in software engineering involving machine learning systems, AI applications, or agentic products.
• Deep experience building and operating production ML or LLM systems, including deployment and evaluation.
• Hands-on expertise with agent orchestration, persistent context, or agent memory architectures.
• Experience developing multistep workflows involving tool use, retrieval, and autonomous task execution.
• Strong software engineering fundamentals and the ability to debug complex production systems.
• Demonstrated experience using measurable evaluation results to improve AI application quality.
• Ability to make architectural decisions and guide technical work without direct management responsibility.

Agentic AI and ML Technologies

Relevant technical experience includes:

• Agent orchestration: Multistep execution, parallel agents, delegated tasks, and tool integration.
• Agent memory: Persistent context, cross-session state, and retrieval-backed knowledge.
• LLM systems: Production model integration, evaluation, and application reliability.
• Retrieval-Augmented Generation (RAG): Grounding model responses in relevant external information.
• Knowledge graphs: Representing relationships between services, infrastructure, and operational events.
• Evaluation-driven development: Benchmark design, regression testing, and release quality gates.

Strong candidates should bring substantial experience in both ML systems and agentic AI, with particular depth in either orchestration or persistent memory and working knowledge of the other.

Additional Technical Experience

• Distributed systems, checkpointing, recovery, and fault-tolerant execution.
• Latency optimization, execution deadlines, and computational cost management.
• Observability technologies involving logs, metrics, traces, and infrastructure events.
• Site Reliability Engineering (SRE), AIOps, or production incident management.
• Secure multi-tenant architectures and customer data isolation.
• AI grounding techniques and methods for reducing unsupported model outputs.

Engineering Environment and Technical Ownership

You’ll work on an established AI platform that supports real production investigations. The engineering challenges involve improving existing agent capabilities while developing the next generation of platform architecture.

What to Expect

• Architecture ownership: Influence the technical direction of the agent platform and its underlying execution systems.
• Production-focused AI: Build systems that must operate reliably against real infrastructure data and operational deadlines.
• Evaluation-led development: Use measurable quality improvements to guide technical decisions and software releases.
• Complex data environments: Work with telemetry, service relationships, retrieval systems, and infrastructure knowledge.
• Hands-on technical leadership: Guide other engineers while remaining responsible for implementation and production reliability.
• Individual contributor scope: Exercise senior technical authority without a direct-report management requirement.

Compensation and Position Details

• Position: Staff AI Engineer, Agent Platform
• Location: Redwood City, California
• Employment: Full-time
• Work arrangement: Fully on-site, five days per week
• Annual base salary: $210,000 to $280,000
• Equity: Additional equity compensation, with the amount determined during the offer process
• Visa sponsorship: Sponsorship and eligible visa transfers may be considered, including H-1B and TN arrangements

Why Consider This Opportunity?

This Staff AI Engineer position offers the chance to lead important architectural decisions for a production agentic AI platform while continuing to work directly on challenging engineering problems.

• Develop AI agents that investigate real production incidents and infrastructure failures.
• Advance agent memory, orchestration, knowledge representation, and retrieval capabilities.
• Build evaluation systems that directly influence production release decisions.
• Apply distributed systems and ML engineering expertise to demanding reliability requirements.
• Help establish technical standards and mentor engineers without moving into a traditional management role.

This opportunity is particularly suited to an experienced AI engineer who has operated machine learning systems in production, understands the challenges of autonomous agent execution, and wants substantial responsibility for the architecture and reliability of a sophisticated AI platform.