Sr. Site Reliability Engineer, Vehicle Software, Build Infrastructure

Description:
Tesla’s continued success depends on engineers being able to develop, debug, and deploy software quickly. Our build, tools, and developer experience infrastructure directly impact over 1000 vehicle, energy, and Autopilot software engineers. You'll be joining a small team located at the center of the firmware organization called Engineering Productivity, Build and Internal Infrastructure. This unique position exposes us to a wide array of interesting technical challenges and enables us to be the defenders of best practices such as code hygiene, reuse, and maintainability. You will ensure high availability of tools, services, and computational clusters by implementing SRE best practices and methodologies, while also managing over 350k cores worth of compute dedicated to building and compiling software.

Responsibilities:
Ensure the availability of new and existing developer tools
Own management and administration of large scale HPC clusters dedicated to distributed build computing
Capacity planning and analysis, and infrastructure change management (including tuning, reshaping, resizing, and migrating infrastructure), for services and their immediate downstreams
Work with SWE counterparts to identify and mitigate production issues; validate, document and exercise failover/disaster recovery plans and graceful degradation mechanisms policies and standard methodologies
Actively participate and contribute to code reviews and technical design documents, with an eye toward identifying performance and reliability bottlenecks
Provide SRE expertise and implementing standard methodologies in the areas of CI/CD, dashboard integrity improvements, identifying and evaluating for the right set of alerts, SLOs and error budgets to use for services on an ongoing basis
Participate in team support rotation

Requirements:
Bachelor’s Degree in Computer Science, proof of exceptional skills in related fields with practical software engineering experience, or equivalent experience
Expert knowledge of Linux operating system internals, filesystems, disk/storage technologies, and storage protocols, and networking stack
Troubleshooting and full-cycle incident response (mitigation, correction, prevention)
2+ years of handling services in a large-scale distributed systems environment, preferably bare metal
2+ years of experience with containerization software such as: Kubernetes, Docker
Expert knowledge of systems programming (bash and shell tools) and practical, validated knowledge of at least one higher-level language (Python, Go)
Expert knowledge of CI/CD platforms such as: Jenkins, TeamCity, Github Actions

Compensation & Benefits:
Benefits

Along with competitive pay, as a full-time Tesla employee, you are eligible for the following benefits at day 1 of hire:
Medical plans > plan options with $0 payroll deduction
Family-building, fertility, adoption and surrogacy benefits
Dental (including orthodontic coverage) and vision plans, both have options with a $0 paycheck contribution
Company Paid (Health Savings Accounts) HSA Contribution when enrolled in the High-Deductible medical plan with HSA
Healthcare and Dependent Care Flexible Spending Accounts (FSA)
401(k) with employer match, Employee Stock Purchase Plans, and other financial benefits
Company paid Basic Life, AD&D
Short-term and long-term disability insurance (90 day waiting period)
Employee Assistance Program
Sick and Vacation time (Flex time for salary positions, Accrued hours for Hourly positions), and Paid Holidays
Back-up childcare and parenting support resources
Voluntary benefits to include: critical illness, hospital indemnity, accident insurance, theft & legal services, and pet insurance
Weight Loss and Tobacco Cessation Programs
Tesla Babies program
Commuter benefits
Employee discounts and perks program

Expected Compensation
$140,000 - $312,000/annual salary + cash and stock awards + benefitsPay offered may vary depending on multiple individualized factors, including market location, job-related knowledge, skills, and experience. The total compensation package for this position may also include other elements dependent on the position offered. Details of participation in these benefit plans will be provided if an employee receives an offer of employment.