Senior Software Engineer (LLM Ops & Evals)

Join our team as a Senior Software Engineer specializing in LLM Ops & Evals. In this role, you will own the LLM gateway, deploy and manage models across multiple clouds and environments, build a shared evaluation platform, and oversee infrastructure for AI workloads. You will also implement security and compliance controls, define integration contracts, and collaborate with product and ML engineers. The ideal candidate has 5+ years of experience in backend or platform engineering, strong production Python skills, and experience running a shared service in production.

• Own the LLM gateway, including routing, provider failover, rate limits, retries, and cost attribution across multiple model providers.
• Deploy, version, and deprecate models across clouds, regions, and environments, including quota and capacity planning, managed as infrastructure as code.
• Build the shared evaluation platform, including versioned datasets, experiment tracking, run and result schemas, reporting, and trace linkage back to the run.

• Comfort with privacy and compliance work: PII handling, anonymisation, retention, access control
• Experience with cloud at the infrastructure level and infrastructure as code (we run across Azure and GCP with Terraform)
• Experience building platform or shared-service capabilities consumed by multiple internal teams
• 5+ years in backend or platform engineering, with strong production Python
• Experience running a shared service in production: on-call, incidents, postmortems, SLOs
• Experience building or running LLM inference infrastructure: a gateway or routing layer with multiple providers, failover, rate limiting and cost attribution
• Hands-on experience with observability tools (OpenTelemetry, Grafana), including instrumenting services and designing dashboards and alerts
• Judgment: You exercise sound judgment in ambiguous situations, balance speed and accuracy, and adjust priorities proactively
• Ownership: You own work end-to-end, anticipate issues, and ensure high-quality delivery without close supervision
• Adaptability: You navigate ambiguity calmly, model positive behavior, and help peers adjust through clear communication
• Collaboration: You build strong cross-functional relationships and influence peers through expertise, data, and empathy
• Growth Mindset: You encourage open feedback exchange and provide clear, balanced feedback that helps others grow
• Synthetic data or document anonymisation pipelines
• Document AI: VLMs, OCR, structured extraction and the metrics that go with it
• Domain experience in audit, accounting or fintech
• Temporal or another durable workflow engine
• Self-hosted inference, capacity planning, load testing
• Experience with LLM or agent evaluation