AI Compute Engineer
Breakmark · Colombia
AI Compute Architect
📍 Medellín, Colombia
About the Role
We're building a next-generation AI infrastructure platform that delivers high-performance bare-metal GPU compute for large-scale AI training and inference.
As our AI Compute Architect, you'll lead the architecture, qualification, and operational readiness of our GPU infrastructure across rapidly scaling data centers. You'll make the platform work as an integrated production system that connects GPU servers, firmware, OS, NVIDIA software, high-performance fabrics, storage, provisioning, scheduling, observability, and on-site operations.
You won't replace our networking, storage, facilities, DevOps, or site-ops specialists, and you won't build every automation pipeline yourself. Your job is to provide the technical model that connects these domains. You'll define requirements, interpret the evidence, find where failures originate, and decide when infrastructure is ready for production. This is a hands-on architecture role, close to the hardware but with a platform-wide view.
What You'll Do
Define the architecture: Set lifecycle standards for NVIDIA HGX platforms and validated baselines for firmware, BIOS, BMC, OS, drivers, CUDA, and NCCL. Own compatibility requirements and the model of GPU/CPU/memory/PCIe/NVLink/NVSwitch/NIC/storage relationships. Decide what testing each platform change requires before production.
Lead qualification and burn-in: Design the strategy for new clusters, new GPU platforms, repaired nodes, and major changes. That includes load profiles, duration, coverage, and pass criteria. Analyze fleet-wide distributions and outliers to separate isolated failures from systemic issues in hardware, firmware, software, topology, power, thermals, or test configuration.
Diagnose health and performance: Interpret NVIDIA diagnostics and telemetry, covering GPU, NVLink, NVSwitch, PCIe, memory, network, thermal, and power signals. Lead root-cause analysis for intermittent, multi-node, and fleet-wide issues, including collective and point-to-point communication problems.
Validate with real workloads: Define representative training and inference workloads, including dense, MoE, and different parallelism strategies. Set reproducible baselines for throughput, scaling, latency, and numerical behavior, and pinpoint whether degradation comes from compute, communication, storage, scheduling, or configuration.
Integrate across domains: Work with Network (topology, GPU-to-NIC locality, RDMA), Storage (access, checkpointing, recovery), Platform/DevOps (scheduler, containers, provisioning), and Facilities (power, thermal correlation). Route issues to the right specialists with solid evidence.
Own operational readiness and acceptance: Define health, readiness, quarantine, and return-to-service gates. Guide automation teams in building reproducible workflows and interpret their results. Translate customer requirements into validation strategies, rule platforms ready, degraded, conditional, or unsuitable, define remediation and re-test scope, and communicate risks to engineering, vendors, customers, and leadership.
Required Qualifications
Strong hands-on experience with NVIDIA GPU infrastructure, including HGX or comparable large GPU server platforms.
Demonstrated experience designing, executing, or leading burn-in and qualification across multi-node GPU clusters on H100, H200, B200, B300, or a comparable platform (direct B300 experience not required).
Experience analyzing fleet-level qualification results, performance distributions, and hardware/software failures.
Strong Linux systems administration and troubleshooting.
Experience with NVIDIA drivers, CUDA, NCCL, DCGM, and GPU platform tooling.
Experience diagnosing GPU, NVLink, NVSwitch, PCIe, thermal, power, or firmware problems.
Experience validating intra-node and multi-node GPU communication, plus a working understanding of RDMA, GPUDirect RDMA, and high-performance AI fabrics.
Experience deploying, provisioning, or operationalizing GPU clusters.
Experience defining firmware, BIOS, OS, and software baselines, as well as health, quarantine, readiness, and return-to-service criteria.
Experience interpreting distributed training or inference workload behavior.
Ability to troubleshoot performance across GPU, host, network, storage, and software boundaries.
Ability to define requirements for automation teams and interpret the outputs of automated validation systems.
Strong root-cause analysis and problem-solving skills.
Clear communication of technical findings, risk, and remediation requirements.
Comfortable in fast-moving deployment, commissioning, and production environments.
Preferred Qualifications
NVIDIA B200 or B300 infrastructure.
ConnectX adapters, NVIDIA Quantum, InfiniBand, UFM, RoCE, or comparable networking.
Large-scale NCCL and distributed workload qualification.
PyTorch, NeMo, Megatron, vLLM, or TensorRT-LLM.
GPU schedulers such as Slurm, Kubernetes, Run:ai, Base Command, or equivalent.
Validating shared storage and distributed checkpointing on high-performance platforms such as WEKA, VAST, or equivalent.
Infrastructure automation, observability, telemetry, or fleet-management platforms.
Formal customer acceptance testing.
Operating GPU infrastructure across multiple data centers.
Python, Bash, or equivalent scripting for troubleshooting infrastructure workflows.
What We Value
Engineers who have qualified and operated real GPU clusters and who understand what a test reveals, not just how to run it. We value evidence-based judgment, strong ownership of platform integrity, cross-domain leadership without overreach, disciplined and reproducible operations, clear communication of risk and uncertainty, and curiosity across GPU generations.
More jobs at Breakmark · Software Engineering jobs in Colombia, Colombia