Senior / Principal Infrastructure Engineer - ML Platform

ML Platform @ Roblox today supports hundreds of ML use cases and billions of inferences per day across Discovery, Safety, Engine, and much more. As an Infrastructure Engineer on the ML Platform team, you will design, scale, and maintain the foundational infrastructure powering our entire machine learning ecosystem. We are looking for accomplished engineers to spearhead the development of our next-generation ML tooling and platform capabilities.

You will:

• Bootstrap and maintain Kubernetes and Cloud infrastructure for ML Platform components--Serving Layer, Metadata Store, Model Registry, and Pipeline Orchestrator.
• Set technical strategy and oversee development of high scale and reliable infrastructure systems.
• Propose and implement new platform tooling to improve time to production for MLEs and Data Scientists across the full ML lifecycle.
• Work on infrastructure projects such as GPU fleet management, hybrid-cloud orchestration, and writing custom Kubernetes controllers and resources.
• Stay abreast of industry trends in machine learning and infrastructure to ensure the adoption of leading-edge technologies and practices.
• Partner across organizations to build tooling, interfaces, and visualizations that make the ML@Roblox a delight to use.

You have:

• 6+ years of professional experience and a tool chest of system design experience upon which to draw to build scalable, reliable platforms.
• Deep experience with Kubernetes (K8s) and cluster management at scale — e.g., managing 100s–1000s of nodes, serving 100k+ QPS, and ideally having experience writing custom Kubernetes controllers.
• Strong proficiency in Infrastructure as Code (IaC), specifically using Terraform to bootstrap, manage, and automate cloud infrastructure across AWS, GCP, or similar environments.
• Bachelor's degree in Computer Science, Computer Engineering, Data Science, or a similar technical field or equivalent practical experience.

You are: 

• Proficient in DevOps tooling such as Docker, Kubernetes, CI/CD systems, and bootstrapping cloud infrastructure (AWS, GCP, etc.)
• Experienced with the end-to-end ML model lifecycle such as model serving, training, model CI/CD, and GPU resources management, and have built ML platform features that are delightful to use.
• An automation advocate: you're passionate about infrastructure-as-code and automating painful manual processes.
• A reliability nut: you love digging into tricky postmortems and identifying weaknesses in complicated systems.
• Passionate about supporting internal partners (data scientists and ML Engineers) to meet and understand their needs.