Driver reads of NIC-written memory are sound on x86-64 with cache-coherent DMA

Scope

The subject is a kernel-bypassed userspace driver for the Intel 82599 10 GbE controller, operating without an IOMMU, in which the device performs DMA to and from host memory the driver owns. Concurrent mutation of that memory by the device is not modelled by the Rust abstract machine, so the ordering properties the driver depends on are established here from the governing specifications.

Three primary sources are used: the 82599 datasheet, revision 3.5, for device ordering behaviour; the PCI Express Base Specification for transaction ordering on the link; and the Intel Software Developer’s Manual, volume 3A, for the x86-64 memory-ordering model.1,2,3 Reference implementations were consulted for structure but are not cited as authority for any ordering claim.

The account is not an unconditional soundness claim. It holds for x86-64 hosts with cache-coherent DMA and for this controller over PCI Express. It is established by reading the driver against the cited specifications; no execution result is offered as evidence, because a path that works does not demonstrate that a crossing is sound.

The obligations

The device writes receive descriptors and packet payloads into host memory, and reads transmit descriptors and packet buffers out of it, concurrently with the CPU. Two obligations follow.

Read: a set descriptor-done bit implies a complete payload

Device half. The datasheet states, under descriptor and data ordering, that the controller “insures that a Rx descriptor is written back on PCIe only after the data that the descriptor relates to is written to the PCIe link.”1 Both the payload and the descriptor writeback are PCI Express posted writes, and the same section binds the controller to the PCI Express ordering rules, under which a posted request may not pass another posted request in the same direction.2 The two writes therefore reach host memory in issue order: payload first, descriptor second.

Relaxed ordering does not weaken this. The datasheet states that the controller cannot relax ordering for descriptor writes, and permits receive data writes to bypass one another only “because software does not process this data until their associated descriptor writes are done.”1 The permitted relaxation is predicated on the same contract it might appear to threaten, so the guarantee holds whether or not relaxed ordering is enabled.

CPU half. The driver reads the descriptor-done bit with an acquire-ordered atomic load rather than a volatile read, because acquire ordering is defined for atomic operations and not for volatile accesses. The acquire orders every subsequent load in the single-threaded reactor after it, including the payload reads that follow in program order. On x86-64 the manual states that reads are not reordered with other reads, so load-load ordering holds on this platform regardless;3 the acquire states the requirement explicitly and portably, and supplies the necessary barrier on weaker architectures. Cache-coherent DMA makes the device’s writes visible to the CPU load without an explicit cache maintenance operation.

Tearing. The descriptor-done bit sits in a naturally aligned 32-bit word, and a single-word aligned read is atomic on x86-64,3 so the observation itself cannot tear. The multi-byte payload is read only after that observation, and the device completed the payload write before it, so no device write to the buffer is in flight during the read. Non-tearing of the payload follows from the ordering established above rather than from any atomicity of the payload read.

Write: a published descriptor is fully written

CPU half. The driver writes the packet buffer bytes and the descriptor fields as ordinary stores to write-back cacheable memory. The manual states that writes are not reordered with other writes, so these retire in program order.3 A release fence precedes the doorbell store. The doorbell is a write to the uncacheable memory-mapped register region: an uncacheable write is strongly ordered and drains the store buffer, so all prior write-back stores are globally visible before the doorbell completes. On x86-64 the release fence is a compiler barrier only, since the uncacheable write already supplies the processor ordering; it is retained as the explicit and portable form.

Device half. The controller advances its ring only when the tail register moves, and fetches descriptors it did not previously own after observing that write.1 Its DMA read of the descriptor and buffer is coherent, and because the tail write is ordered after the descriptor stores, the descriptor it reads is complete.

Transmit buffer reuse

A complementary obligation applies in the transmit direction: the driver must not overwrite a transmit buffer while the device is still reading it for an in-flight transmission. This is closed by control flow rather than by a memory barrier. A buffer is returned to the free pool only after the device reports the descriptor complete, and that report is itself subject to the read obligation established above.

What the account does not establish

Four limits are stated explicitly.

The account is platform-scoped. It rests on x86-64 memory ordering and on cache-coherent DMA. On a platform without coherent DMA, explicit cache maintenance would be required at both obligations, and the acquire and release operations would carry the processor ordering rather than merely stating it.

It is device-scoped. The device-half properties are cited from one controller’s datasheet at one revision. A different controller requires its own citation, and the same manufacturer’s other parts are not covered by inheritance.

It is established by reading, not by execution. No test result is offered in support of any ordering claim. A run that produces correct output demonstrates that the path functioned on that occasion under that scheduling; it does not distinguish a sound crossing from an unsound one that has not yet been observed to fail.

It does not address the absence of an IOMMU. The driver operates without IOMMU containment, so the device is capable of DMA to physical memory the driver did not allocate. That is a containment property rather than an ordering property, and it is out of scope here.