Several GPUs can provide independent workers, replicated services or one distributed model. Those patterns use the same hardware inventory in very different ways.
Aggregate memory is not automatic pooling
Adding the memory printed on each GPU gives an aggregate figure. Software still needs an explicit partitioning or sharding strategy to use several cards for one model.
If one stage or tensor shard exceeds the memory of its assigned GPU, a large aggregate total does not solve the placement problem.
Independent workers
One model or job per GPU reduces communication and can provide predictable isolation. It suits embedding, transcription, rendering, image generation and replicated smaller language-model endpoints.
The scheduler still needs to understand memory, queue priority and failure handling.
Sharded inference
Tensor and pipeline parallelism divide a model or its execution across devices. Transfers then sit on the critical path. The runtime, checkpoint architecture, GPU count and topology all affect the result.
A quoted “supports eight GPUs” chassis does not establish useful eight-GPU scaling.
PCIe lanes and paths
The physical slot can be wider than its electrical link. A path may cross a switch, host bridge or CPU-level interconnect. On dual-socket systems, CPU and memory affinity can add another boundary.
NVIDIA DCGM can report GPU-to-GPU paths and CPU affinity. That describes topology; it does not measure application bandwidth or prove performance.
NVLink and NVSwitch
NVLink provides supported GPUs with direct high-speed GPU communication. NVSwitch connects many NVLink endpoints into a fabric on suitable platforms.
These technologies do not make all software scale automatically. Link presence, link state, supported collectives and the application’s communication pattern still matter.
Consumer RTX 5090 and the controlled RTX PRO 6000 workstation references in this catalogue are not presented as NVLink systems. Their combined VRAM must not be marketed as a universal pooled-memory promise.
NUMA and host resources
Balance GPUs across CPU sockets and root ports, then place CPU threads, system memory, storage and network with the workload in mind. Model loading, CPU offload and network serving can expose an otherwise hidden host bottleneck.
Record system RAM population and storage path as part of the test, not as decorative specifications.
Multi-node is a different design
Once a model or service spans servers, network bandwidth, latency, topology, orchestration and failure behaviour become first-class requirements. A single-node benchmark does not predict a cluster.
What to request before purchase
Ask for:
- a board and PCIe topology diagram;
- negotiated link width and generation for every GPU;
- CPU-socket and NUMA relationship;
- exact GPU edition and interconnect support;
- power and cooling limits at simultaneous load;
- runtime topology output;
- an application benchmark using the selected parallelism.
The correct design is the smallest topology that passes the accepted workload with operating headroom.
Technical context
See the physical and operating boundary
Use these views to connect the guide to the machine, its airflow and its operating environment. Captions state the limits of what each image shows.
Primary sources
- NVIDIA DCGM topology and NVLink documentation
- NVIDIA-Certified Systems configuration guide
- NVIDIA CUDA programming guide
Sources are checked at the review date. Platform terms, prices and public guidance can change; verify them at the point of decision.