ai-hardware-engineer-roadmap

The AI Hardware Engineer Roadmap

Master AI on hardware. Build the physical AI chip.

Design a custom system-on-chip for advanced vision-language-action models: dedicated AI acceleration, your own AI compiler, embedded Linux, and integrated wireless. Learn the full stack from models to silicon to power the next generation of robots, drones, and autonomous machines.

The Goal

Master AI on hardware, then design a custom physical-AI inference system-on-chip built around advanced VLA models—with a dedicated accelerator, an AI compiler, embedded Linux, and integrated wireless.

The destination is a complete computing platform for robots, drones, and other autonomous machines. Start by understanding how a model sees, reasons, remembers, and produces actions. Use that workload to design the compute, memory, compiler, runtime, operating system, sensor interfaces, and wireless subsystem together. The chip and its software should serve the machine’s behavior: useful autonomy within measured latency, energy, memory, and reliability budgets.

Native VLA support means the model’s actual execution path shapes the architecture: visual encoding, attention, temporal state, action generation, numerical formats, data movement, and control timing. The compiler must map that workload to the accelerator, and the runtime must carry observations through to usable actions. Design for a defined set of frontier workloads and evaluate new model revisions as they emerge, with programmable components and explicit operator coverage.

GPU and Jetson experiments establish workload evidence; FPGA and board prototypes validate architecture and interfaces. These feed the custom SoC capstone, then implementation and silicon bring-up. Each stage produces evidence for the next.

Who This Is For

• AI/ML engineers who want to stop treating inference as a black box and design the chip that runs it.
• Inference engineers who want to co-design VLA execution, AI compilers, and custom accelerators.
• Embedded/firmware engineers who want to connect Linux, wireless, sensors, and AI acceleration on a custom SoC.
• Hardware/RTL/FPGA engineers who want real model and compiler requirements to guide accelerator design.
• Robotics engineers and students who want to build the computing platform behind intelligent robots, drones, and autonomous devices.

Baseline: comfortable with Python or C++, and basic Linux command-line use. No prior hardware background required.

Already past the basics? Phase 1 and Phase 2 exist to fill specific gaps — digital logic, embedded Linux — not to gate you. If you already know CUDA/C++, go straight to Phase 3 and pick up Phase 1–2 material only where a later module says you need it.

You can enter through software, robotics, or hardware. Choose one working artifact first and add lower-level material as your project needs it.

Learn by building and testing

Follow the Physical AI Design Studio to connect the courses through one project: task contract → VLA reference → robot simulation → measured inference → compiler → accelerator → Linux → wireless → system evidence. Each gate names the courses to study, the artifact to build, and the check needed before proceeding. Includes original diagrams with downloadable PNGs.

The October 18–19 curriculum review links the revised lessons and their execution evidence. For a connected AI/robotics experiment, train a small CPU policy, then run it in a timed robot loop.

Start with How to learn this roadmap: entry diagnostics, a repeatable lesson cycle, hardware access options, and a shared completion rubric. Each course’s practice section asks you to predict a result, build a baseline, test a variation, and explain the evidence.

New applied sequences: DeepSeek V4.1 Flash NVFP4 inference engineering and Isaac Sim / Isaac Lab engineering labs. Both have CPU exercises and separate milestones for work requiring compatible GPU hardware.

New professional course: VLA Foundations to Frontier 2026 — 32 lectures, covering history, architectures, training, evaluation, Isaac integration, and verified research through October 13, 2026. Includes runnable CPU labs, source dossiers, and two capstones.

Continue into Programmable VLA Inference Accelerator — 10 lectures: turn policy traces into tensor/vector tiles, memory plans, a compiler/runtime ABI, scheduling, and a prototype design defense. Includes CPU labs, worked architecture studies, and primary-source research checked October 14, 2026.

Four Ways In

Choose your first subsystem. The full path connects these entry points into the same custom physical-AI SoC.

Each entry point produces a useful subsystem. The Path brings them together: VLA workload → compiler and runtime → accelerator and SoC → embedded Linux and wireless → validated physical behavior. The roadmap’s unifying goal is the complete chip and software platform.

The Three Pillars

The model defines the work. The compiler and runtime map that work onto the machine. The SoC supplies the compute, memory, operating system, connectivity, and physical interfaces. Learn all three together so a decision in one layer can be tested against its consequences in the others.

1. AI Models and VLA Inference

Understand the workload before choosing the accelerator. Learn visual and language representations, attention, action tokens, diffusion and flow action heads, memory, adaptation, and evaluation. Trace tensor shapes, precision, operator cost, and memory traffic. Use GPU kernels and existing runtimes to measure a reference implementation, then connect those measurements to sensor-to-action timing and task success.

Output: a pinned VLA workload, operator and memory profiles, a numerical reference, closed-loop evaluation, and compute/energy budgets for the proposed chip. The professional VLA course supplies the model-to-action foundation.

2. AI Compilers and Physical-AI Runtimes

Build the software that makes custom silicon usable. Learn graph capture and import, operator semantics, intermediate representations, quantization, fusion, tiling, buffer allocation, instruction selection, and code generation. Implement a compiler backend and the runtime/driver interface that submits work, moves data, synchronizes execution, and exposes timing and errors.

Connect model execution to robot state, sensor timestamps, action chunks, controllers, and updates. Agent harnesses contribute planning, tool use, memory, and observability to this stack; their requests must respect the device and control interfaces. Validate the compiled model against its reference and measure the complete application path.

Output: a working compiler/backend, runtime and driver contracts, correctness checks, and execution traces from model input to controller-ready output. Start with the compiler and inference track and connect it to the agent systems course.

3. Custom Accelerators, SoCs, and Embedded Platforms

Design the hardware around those measured requirements: compute arrays, vector operations, SRAM, DMA, interconnect, CPU integration, sensor interfaces, and wireless. Learn RTL and verification, FPGA prototyping, timing and power analysis, physical implementation, board bring-up, firmware, bootloaders, device trees, Linux drivers, and reproducible system images.

Prototype Linux and wireless behavior on available boards, then carry the verified interfaces into the custom SoC architecture. Define what runs on the application CPU, accelerator, and controller domains, and where external memory, RF components, antennas, and sensors connect. The integration objective is a chip that supports an entire physical-AI application.

Output: tested RTL, a prototype platform, compiler-visible hardware interfaces, a bootable Linux BSP, wireless integration evidence, and an implementation-ready SoC specification. Continue into the AI chip design track.

What You’ll Have at the End

These milestones connect the courses to the chip you are designing:

• A VLA workload contract: selected model revisions, input/action schemas, operator coverage, memory traffic, control cadence, and task-quality targets.
• A measured inference baseline: reference kernels/runtime, numerical checks, full sensor-to-action latency, sustained power, and closed-loop results.
• Your own AI compiler path: model import, IR transformations, memory planning, scheduling, and code generation for a simulated or FPGA accelerator backend.
• A dedicated accelerator prototype: verified RTL or HLS, a software-visible interface, timing constraints, resource usage, and model-subgraph parity.
• An embedded Linux platform: boot flow, BSP, drivers, device tree, reproducible image, model runtime, and updates on the chosen prototype.
• Wireless and physical I/O: connectivity, sensor acquisition, synchronized timestamps, controller interfaces, and tested communication-loss behavior.
• A robotics or drone integration: a bounded application with logged outcomes, deadline misses, energy use, and recovery; simulation, hardware-in-the-loop, and physical results labeled separately.
• A complete physical-AI SoC design: CPU, custom accelerator, memory/interconnect, wireless subsystem, I/O, firmware, compiler, and Linux architecture, with traceable verification and power/performance/area assumptions.

The full hardware path continues through physical implementation, tape-out preparation, fabrication, and silicon bring-up. Keep milestones explicit: a simulator result, an FPGA prototype, a signed-off design, and a measured fabricated chip are different deliverables. The SoC capstone ties the evidence together.

The AI Chip Stack

Everything in this roadmap maps onto an 8-layer stack. The point isn’t to memorize layers — it’s to understand how decisions in one layer ripple through the others.

When you’re designing a chip, every layer is a constraint and a degree of freedom. The roadmap teaches you to read the whole column.

The Path

Six phases. The first four build the model, software, and hardware foundations; Phase 5 combines them into the physical-AI SoC; Phase 6 prepares you to explain and defend your engineering work.

Phase 1 — Digital Foundations (Hardware pillar)

The language of hardware. Logic gates → GPU code.

Phase 2 — Embedded Systems (Hardware pillar)

Get hands-on with real hardware. MCUs, sensors, embedded Linux.

Phase 3 — AI Workloads (AI models and runtime foundations)

Understand the workloads your chip must serve. Core + two tracks.

Core (everyone):

• Neural Networks — backprop, CNNs, transformers from first principles
• Transformer Fundamentals — the prerequisite for every inference lecture downstream
• Deep Learning Frameworks — micrograd → PyTorch → tinygrad

Track A — Hardware & Edge AI: Computer vision, sensor fusion, Voice AI, Edge AI & optimization. Feeds Phase 4B and Phase 5C.

Track B — Agentic AI & ML Engineering: 42 lectures on agent harnesses, LangGraph, multi-agent systems, RAG, evaluation, production runtime discipline, OpenClaw, OpenAI Agents SDK, security, plus a Qwen3.5-4B-Base Unsloth fine-tuning course. This supplies the planning and agent-runtime layer of the physical-AI platform; connect its interfaces to model execution, Linux services, and robot control.

Phase 4 — Deployment & Compilation (All three pillars co-exist here)

Take AI to real silicon. Three specialized tracks.

For the complete chip path, combine A for RTL and FPGA validation, B for embedded Linux and device integration, and C for the compiler and runtime that target your accelerator.

Phase 5 — Specialization & Convergence

The three pillars converge here. Specialization tracks plus the chip-design endpoint.

The signature physical-AI chip path: Phase 1 → Phase 2 (Linux + wireless) → Phase 3 (models + perception + agents) → Phase 4A/B/C (FPGA + Jetson + compilers) → Phase 5D (VLA + Isaac robotics) and Phase 5C/G (edge inference + systems) → Phase 5F (custom accelerator + SoC).

The MLSys path: Phase 1 §3/§4 → Phase 3 Core → Phase 4B/4C → Phase 5A/B/C → Phase 5G.

Featured Inference Lectures

The deepest, most current technical content lives in these Phase 5 lectures — read them as a single arc:

Read them in order if you’re new. Skip to whichever solves your current problem if you’re not.

How to Use This Roadmap

Don’t read this like a book. Treat it like a build-and-measure curriculum.

For every block:

1. Read the theory.
2. Build the subsystem or implement the technique.
3. Measure something — latency, throughput, occupancy, bandwidth, power, accuracy, area, perplexity.
4. Ship one reusable artifact (benchmark, kernel, board, dashboard, RTL block, eval report).

Each artifact is a brick in the chip you’re going to design.

Before you start, decide three things:

1. Where you’re entering the stack. (See “Who This Is For” above.)
2. What hardware you can actually use. Start with the CPU exercises. For device work, choose hardware supported by the specific lab: a Jetson board for edge deployment, a compatible GPU for profiling or simulation, a Zynq development board for FPGA integration, or an ESP32 plus sensor for embedded interfaces. Check the lab’s memory, software and equipment requirements before buying or renting anything.
3. How you’ll track outputs. A notebook, a benchmarks repo, a project log — any system you actually use.

Course Quality Bar

Every serious module in this roadmap should end with evidence, not vibes.

Use this standard for each course block:

Weak completion:

I read about CUDA, TensorRT, and FPGAs.

Strong completion:

I built a TensorRT INT8 benchmark on Orin Nano, captured latency/RAM/power,
compared it to FP16, and explained why one layer stayed memory-bound.

The roadmap is intentionally broad, but the completion standard is narrow: build something real, measure it, and explain the tradeoff.

Reference Projects

These projects exist for you to study, not just read about:

Target Roles This Enables

The roadmap develops skills used in several specialist roles. Use the projects to demonstrate the responsibilities relevant to your target role:

→ See Roles & Market Analysis for role responsibilities, prerequisite paths, portfolio evidence, and ways to check current openings.

Why This Roadmap Exists

The aim is to give engineers the skills to build the computing foundation for future robotics, drones, and autonomous machines. That requires understanding AI deeply enough to decide what hardware it needs, and understanding hardware deeply enough to build the compiler, runtime, Linux platform, and connectivity that make the models useful.

Each experiment should answer a design question for the eventual SoC: which computation to accelerate, which data to keep on chip, how to schedule inference around fresh observations, how to preserve action quality at lower precision, or how to keep the machine operating when workloads or network conditions change.

The result is a coherent design effort from model behavior to physical implementation: advanced VLA models → custom AI compiler and runtime → dedicated accelerator → Linux and wireless SoC → robots, drones, and autonomous machines.

Star History