A VM configured to use a single NIC can still have its assignment blocked by an NVMe controller on another downstream port if Linux places both functions in the same IOMMU group. Rebinding the NIC leaves the storage controller doing DMA on behalf of the host while VFIO needs to claim ownership for the guest.
If we leave the NVMe in place and move the same NIC to a slot behind another root port while keeping the kernel and firmware fixed across both boots, what could make the new assignment viable? I'd start with the DMA alias walk because the version of pci_device_group() examined here can return an existing group before it checks ACS on a single upstream bridge.
Both devices can stay bound to their usual drivers while we collect the PCI paths and group membership on either side of the move.
Requester identity and fabric routing
For the NIC the question is whether the write ever reaches the remapper because the permissions in its DMA domain only apply to traffic that takes that path. If the destination falls inside a peer BAR and the switch routes the TLP locally then the write can reach the peer without the remapper ever consulting the page tables.
Two functions can enumerate at different BDFs and still issue DMA under the same effective requester ID so even if software assigns them separate domains the remapper has no way to choose between those domains from the identity it receives.
Change the path
Before moving the card I'd check where both slots sit in the PCI tree because two connectors on the same board can lead back to the same switch. Keeping the NIC and kernel build fixed across both boots and leaving the firmware settings alone makes it easier to trace a change in group membership back to the PCI path.
Compare the upstream path from the NIC to its root port in each of the two slot arrangements shown below:
case a / shared switch
0000:00:01.0 root port
0000:01:00.0 switch upstream port
0000:02:01.0 switch downstream port
0000:03:00.0 nic
0000:02:02.0 switch downstream port
0000:04:00.0 nvme
case b / nic moved to a separate root port
0000:00:02.0 root port
0000:05:00.0 nic
For case A assume the switch path lacks the ACS isolation Linux requires and the NIC shares a group with the NVMe. Case B moves the NIC onto a path that passes the ACS check and assumes no DMA alias ties it to a peer. When repeating the comparison on hardware use the BDFs from each boot and compare the devices in each group across boots because moving the card can change its address as well as its group number.
The grouping path
When an IOMMU driver uses pci_device_group() the helper calls pci_for_each_dma_alias() with a callback that remembers the last device visited and terminates the search as soon as it finds an existing group. If the alias walk finishes without finding one then the helper starts the upstream bridge walk from the last device the callback recorded.
For each upstream bus with a bridge the code calls pci_acs_path_enabled(bus->self, NULL, REQ_ACS_FLAGS) with a mask of PCI_ACS_SV | PCI_ACS_RR | PCI_ACS_CR | PCI_ACS_UF and stops ascending if the path passes. If it fails then the bridge becomes the candidate and any existing group on that bridge can terminate the search. The test covers the path towards the root so the immediate parent's ACSCap line is only the start of the evidence and has to be read alongside ACSCtl and the PCI core's device specific handling.
If the bridge walk finds no group then get_pci_alias_group() searches DMA alias relationships in both directions and uses a devfn bitmap to terminate revisits while following relationships between devices on the same bus. The subsequent get_pci_function_alias_group() check handles functions in the same slot that lack the required ACS isolation before the code falls back to allocating a group. The PCI tree can explain the bridge walk while the sibling and alias checks explain dependencies that the parent links alone can't show.
Although sysfs tells you which group the NIC ended up in after enumeration the group number alone gives you no way to tell which of those branches returned it. If the NVMe follows the NIC into the same group after the move I'd check the aliases before trying to explain the result from the switch registers.
Legacy VFIO checks viability through the group file while the cdev interface lets you open a device file and claim DMA ownership through VFIO_DEVICE_BIND_IOMMUFD before getting full access to the function. The file descriptor names one device while the ownership constraint still applies to every member of its group.
Check an assignment plan
The OCaml check takes the endpoints in one group and prints the BDFs whose proposed owners conflict with the assignment. The first run should name the NVMe because the plan keeps it on the host while handing the NIC to the guest. For the second run the NIC is the only endpoint in the group so the conflict list is empty and the plan passes this check while leaving driver binding and the actual ownership claim to VFIO.
type owner = Host | Guest of string
type endpoint = {
bdf : string;
owner : owner;
}
let endpoint owner bdf = { bdf; owner }
let conflicts requested_owner members =
List.filter (fun dev -> dev.owner <> requested_owner) members
let report group requested_owner members =
match conflicts requested_owner members with
| [] ->
Printf.printf "group %d: one owner in this plan\n" group
| peers ->
let ids = List.map (fun dev -> dev.bdf) peers in
Printf.printf "group %d: conflicting owners at %s\n"
group (String.concat " " ids)
let () =
let guest = Guest "guest0" in
let nic_before = endpoint guest "0000:03:00.0" in
let nic_after = endpoint guest "0000:05:00.0" in
let nvme = endpoint Host "0000:04:00.0" in
assert (conflicts guest [nic_before; nvme] = [nvme]);
assert (conflicts guest [nic_after] = []);
report 12 guest [nic_before; nvme];
report 19 guest [nic_after]
With the NVMe assigned to the host in the first run and the NIC alone in the second group the program prints:
group 12: conflicting owners at 0000:04:00.0
group 19: one owner in this plan
Capture the group and the path
I keep both sysfs paths with each capture because resolving the PCI device symlink gives me the upstream bridges in order while following iommu_group gives me every member of the group Linux assigned. Save the script as inspect.sh and pass it a domain qualified BDF to collect both paths and the bindings:
set -eu
bdf=${1:?usage: sh inspect.sh PCI_ADDRESS}
sysfs=${SYSFS_ROOT:-/sys}
dev="$sysfs/bus/pci/devices/$bdf"
if ! group=$(readlink -e "$dev/iommu_group"); then
printf 'no iommu group visible for %s\n' "$bdf" >&2
exit 1
fi
printf 'path: %s\n' "$(readlink -e "$dev")"
printf 'group: %s\n' "${group##*/}"
for member in "$group"/devices/*; do
[ -e "$member" ] || continue
driver=$(readlink "$member/driver" 2>/dev/null || :)
printf '%s %s\n' "${member##*/}" "${driver##*/}"
done
Record the kernel build and boot arguments alongside the topology so you can look up the grouping code that ran on that boot. Run the following for the first bridge and repeat the verbose dump for each remaining hop so the capture includes the config space registers along the entire upstream path:
uname -r
cat /proc/cmdline
lspci -D -t
sh inspect.sh 0000:03:00.0
lspci -D -vv -s 0000:02:01.0
Attribute the split
When the group changes across boots I'd first check where the alias walk left the kernel and then follow the NIC upstream to see whether the first changed bridge accounts for the split. If the new group is smaller then check where the upstream ACS predicate starts passing and whether an alias or a same slot relationship would have caused group reuse regardless.
If the group remains shared then trace the first candidate that can supply an existing group under the new topology while keeping the config space dump next to the exact PCI core implementation for that kernel. An apparent disagreement between the raw ACS fields and the grouping result is a reason to inspect device specific handling before attributing the result to a single downstream port.
Check who else joined the group even if the NVMe disappears from it because moving the NIC can trade a storage controller for a different host owned peer and leave the assignment just as unusable. Feed the OCaml check every endpoint in the new group or that conflict drops out of the input.
A raw diff of two sysfs listings counts the NIC moving from bus 03 to bus 05 as a removal and an addition. I'd match the physical functions across boots first and keep the original BDF beside each label in the capture notes. Identical adapters need separate labels even when the vendor and device IDs agree.
The comparison takes two files with one matched function per line. Each file must cover the entire group containing the target. A repeated label would silently collapse two rows into one set member so the reader rejects it. Checking for the target in both files also catches the case where a capture belongs to another group. Under those assumptions set subtraction gives the peers lost or gained across boots.
module Members = Set.Make (String)
let read_members path =
let channel = open_in path in
let rec read seen =
match input_line channel with
| line ->
let label = String.trim line in
if label = "" || Members.mem label seen then
failwith ("empty or repeated label in " ^ path);
read (Members.add label seen)
| exception End_of_file -> seen
in
Fun.protect ~finally:(fun () -> close_in_noerr channel)
(fun () -> read Members.empty)
let () =
if Array.length Sys.argv <> 4 then
failwith "usage: ocaml compare.ml TARGET BEFORE AFTER";
let target = Sys.argv.(1) in
let before = read_members Sys.argv.(2) in
let after = read_members Sys.argv.(3) in
if not (Members.mem target before && Members.mem target after) then
failwith "target missing from a capture";
let report change members =
Members.iter (Printf.printf "%s %s\n" change) members
in
report "left" (Members.diff before after);
report "joined" (Members.diff after before)
Each label names the same physical function across boots. These inputs reproduce the slot comparison:
printf '%s\n' nic nvme > before.members
printf '%s\n' nic > after.members
ocaml compare.ml nic before.members after.members
The run printed left nvme. Adding a replacement peer labelled usb to the second file produced another line reading joined usb while reordering an identical membership produced no output. The comparison can't tell whether two labels refer to the same physical function. That part depends on the capture notes.
On an Arm64 host with SMMUv3 I'd keep the firmware mapping with the capture because a device tree system describes the translation from PCI Requester ID to stream ID through iommu-map and any iommu-map-mask on the PCI host bridge. That mapping gives you the identity presented to the remapper for upstream traffic while the switch can still route a peer transaction locally.
Scope of the result
Put the resolved device path beside the group members for each boot and see how far that evidence takes you. If two branches could have returned the same group then identifying the one that ran requires observing the helper during enumeration. A pair of sysfs captures can tell us that the NVMe left the group when the NIC moved slots. Working out where a particular TLP went would take a transaction trace and we would still need separate evidence for reset scope and interrupt remapping.
References
VFIO in the Linux kernel documentation explains IOMMU groups and DMA ownership for both VFIO interfaces.
PCI grouping in the pinned Linux IOMMU source contains the group lookup, bridge walk and alias checks discussed above.
PCI requester ID to IOMMU mapping specifies the device tree properties that translate a requester ID into an IOMMU specifier.
PCI peer-to-peer DMA in the Linux kernel documentation describes when the fabric can route a transaction between peers.
ACS handling in the pinned Linux PCI core defines the path predicate and the device-specific handling used by the grouping code.
Set.S in the OCaml 4.13 standard library documents the set operations used to compare group membership across boots.