AMD Instinct MI300X system optimization — ROCm Documentation (2025)

AMD EPYC 9004-based systems#

For maximum MI300X GPU performance on systems with AMD EPYC™ 9004-seriesprocessors and AMI System BIOS, the following configurationof system BIOS settings has been validated. These settings must be used for thequalification process and should be set as default values in the system BIOS.Analogous settings for other non-AMI System BIOS providers could be setsimilarly. For systems with Intel processors, some settings may not apply or beavailable as listed in the following table.

Each row in the table details a setting but the specific location within theBIOS setup menus may be different, or the option may not be present.

Appending strings via Linux command line#

It is recommended to append the following strings in GRUB_CMDLINE_LINUX.

pci=realloc=off

With this setting Linux is able to unambiguously detect all GPUs of theMI300X-based system because this setting disables the automatic reallocationof PCI resources. It’s used when Single Root I/O Virtualization (SR-IOV) BaseAddress Registers (BARs) have not been allocated by the BIOS. This can helpavoid potential issues with certain hardware configurations.

iommu=pt

The iommu=pt setting enables IOMMU pass-through mode. When in pass-throughmode, the adapter does not need to use DMA translation to the memory, which canimprove performance.

IOMMU is a system specific IO mapping mechanism and can be used for DMA mappingand isolation. This can be beneficial for virtualization and device assignmentto virtual machines. It is recommended to enable IOMMU support.

For a system that has AMD host CPUs add this to GRUB_CMDLINE_LINUX:

iommu=pt

Otherwise, if the system has Intel host CPUs add this instead toGRUB_CMDLINE_LINUX:

intel_iommu=on iommu=pt

CPU core states (C-states)#

There are several core states (C-states) that an AMD EPYC CPU can idle within:

• C0: active. This is the active state while running an application.

• C1: idle. This state consumes less power compared to C0, but can quicklyreturn to the active state (C0) with minimal latency.

• C2: idle and power-gated. This is a deeper sleep state and will have greaterlatency when moving back to the active (C0) state as compared to when the CPUis coming out of C1.

Disabling C2 is important for running with a high performance, low-latencynetwork. To disable the C2 state, install the cpupower tool using your Linuxdistribution’s package manager. cpupower is not a base package in most Linuxdistributions. The specific package to be installed varies per Linuxdistribution.

sudo apt install linux-tools-common

sudo yum install cpupowerutils

sudo zypper install cpupower

Now, to disable power-gating on all cores run the following on Linuxsystems, run the following command.

cpupower idle-set -d 2

/proc and /sys file system settings#

sudo crontab -e

Environment variables#

HIP provides an environment variable export HIP_FORCE_DEV_KERNARG=1 thatcan put arguments of HIP kernels directly to device memory to reduce thelatency of accessing those kernel arguments. It can improve performance by 2 to3 µs for some kernels.

It is recommended to set the following environment variable:

export HIP_FORCE_DEV_KERNARG=1

Change affinity of ROCm helper threads#

This change prevents internal ROCm threads from having their CPU core affinity maskset to all CPU cores available. With this setting, the threads inherit their parent’sCPU core affinity mask. If you have any questions regarding this setting,contact your MI300A platform vendor. To enable this setting, enter the following command:

export HSA_OVERRIDE_CPU_AFFINITY_DEBUG=0

IOMMU configuration – systems with 256 CPU threads#

For systems that have 256 logical CPU cores or more, setting the input-outputmemory management unit (IOMMU) configuration to disabled can limit thenumber of available logical cores to 255. The reason is that the Linux kerneldisables X2APIC in this case and falls back to Advanced Programmable InterruptController (APIC), which can only enumerate a maximum of 255 (logical) cores.

If SMT is enabled by setting CCD/Core/Thread Enablement > SMT Control toenable, you can apply the following steps to the system to enable all(logical) cores of the system:

1. In the server BIOS, set IOMMU to Enabled.

2. When configuring the GRUB boot loader, add the following argument for the Linux kernel: iommu=pt.

3. Update GRUB.

4. Reboot the system.

5. Verify IOMMU passthrough mode by inspecting the kernel log via dmesg:

   dmesg | grep iommu

[...][ 0.000000] Kernel command line: [...] iommu=pt[...]

Once the system is properly configured, ROCm software can beinstalled.

ROCm SMI#

To query your GPU hardware, use the rocm-smi command. ROCm SMI listsGPUs available to your system – with their device ID and their respectivefirmware (or VBIOS) versions.

The following screenshot shows that all 8 GPUs of MI300X are recognized by ROCm.Performance of an application could be otherwise suboptimal if, for example, outof the 8 GPUs only 5 of them are recognized.

To see the system structure, the localization of the GPUs in the system, and thefabric connections between the system components, use the commandrocm-smi --showtopo.

The first block of the output shows the distance between the GPUs similar towhat the numactl command outputs for the NUMA domains of a system. Theweight is a qualitative measure for the “distance” data must travel to reach oneGPU from another one. While the values do not carry a special, or “physical”meaning, the higher the value the more hops are needed to reach the destinationfrom the source GPU. This information has performance implication for aGPU-based application that moves data among GPUs. You can choose a minimumdistance among GPUs to be used to make the application more performant.

The second block has a matrix named Hops between two GPUs, where:

• 1 means the two GPUs are directly connected with xGMI,

• 2 means both GPUs are linked to the same CPU socket and GPU communicationswill go through the CPU, and

• 3 means both GPUs are linked to different CPU sockets so communications willgo through both CPU sockets. This number is one for all GPUs in this casesince they are all connected to each other through the Infinity Fabric links.

The third block outputs the link types between the GPUs. This can either beXGMI for AMD Infinity Fabric links or PCIE for PCIe Gen5 links.

The fourth block reveals the localization of a GPU with respect to the NUMAorganization of the shared memory of the AMD EPYC processors.

To query the compute capabilities of the GPU devices, use rocminfo command. Itlists specific details about the GPU devices, including but not limited to thenumber of compute units, width of the SIMD pipelines, memory information, andinstruction set architecture (ISA). The following is the truncated output of thecommand:

For a complete list of architecture (such as CDNA3) and LLVM target names(such gfx942 for MI300X), refer to theSupported GPUs section of the System requirements for Linux page.

ROCm Bandwidth Test#

The section Hardware verification with ROCm showed how the commandrocm-smi --showtopo can be used to view the system structure and how theGPUs are connected. For more details on the link bandwidth,rocm-bandwidth-test can run benchmarks to show the effective link bandwidthbetween the components of the system.

You can install ROCm Bandwidth Test, which can test inter-device bandwidth,using the following package manager commands:

sudo apt install rocm-bandwidth-test

sudo yum install rocm-bandwidth-test

sudo zypper install rocm-bandwidth-test

Alternatively, you can download the source code fromROCm/rocm_bandwidth_test and build from source.

The output will list the available compute devices (CPUs and GPUs), includingtheir device ID and PCIe ID. The following screenshot is an example of thebeginning part of the output of running rocm-bandwidth-test. It shows thedevices present in the system.

The output will also show a matrix that contains a 1 if a device cancommunicate to another device (CPU and GPU) of the system and it will show theNUMA distance – similar to rocm-smi.

Inter-device distance:

Inter-device NUMA distance:

The output also contains the measured bandwidth for unidirectional andbidirectional transfers between the devices (CPU and GPU):

Unidirectional bandwidth:

Bidirectional bandwidth