What networking infrastructure do AI workloads require?

Expensive accelerators are only as fast as the network that feeds them, and what that network must look like depends on the workload. Training a model across many servers needs a dedicated, very fast, low-latency network between them, because the servers spend much of their time exchanging results. Running a trained model to answer requests, known as inference, is often much closer to ordinary data-centre networking. Preparing data needs high throughput to storage. Working out which of these you are doing, and the traffic each creates, is the starting point; the choice of technology follows from it.

Nor does a high link speed on a datasheet prove that the complete system will perform. Topology, congestion behaviour, storage throughput and physical layout all play their part.

Keeping network functions apart

One published reference design, NVIDIA's DGX SuperPOD architecture for the DGX H100 generation, uses four separate networks. Other vendors' designs differ in detail, but the division of labour is instructive.

Not every deployment needs this structure, but the separation of concerns is a model to copy. The guide explains that the out-of-band network sits on its own fabric because there is no case in which users need access to those ports. A management network that stays available when the production fabric is congested or failing is what lets operators recover the system.

Build for the full unit, not today's purchase

NVIDIA's reference architecture recommends that if fewer servers are bought than a full scalable unit, the fabric should still be built for the full unit, leaf switches and leaf-to-spine cables included, with the unused positions left empty, so that traffic routing and performance stay consistent. Growing the network piecemeal around whatever has been purchased can create uneven performance that is hard to diagnose.

Any deviation from a supplier's validated architecture needs engineering review. Components that look interchangeable on paper may behave differently under load.

Cabling and layout are part of the network

High-performance fabrics impose physical limits. Cluster designs can cap the total cable path of each fabric link, and a power constraint that forces racks to spread out can lengthen cables and affect latency. Supported link distances, transceiver types, tray capacity, patching and maintenance access therefore need fixing before rack positions are final, with space reserved for the next stage.

Seeing across the boundaries

When a job runs slowly, the cause could be the compute fabric, storage, the servers, the software or the facility itself; a cooling constraint can throttle processors. Monitoring should let the team tell these apart, and the contract should say who investigates a problem that spans several suppliers.

Accepting on useful performance

Link-level test results are necessary but not sufficient. The acceptance method, agreed with the IT and application teams, should measure what the business actually cares about: job completion time or throughput on representative workloads, behaviour when a link or switch fails, and recovery time.