Home:ALL Converter>Precise details of writing a byte into PCIe address space from CPU

Precise details of writing a byte into PCIe address space from CPU

Ask Time:2019-10-18T11:24:29         Author:Marco Merlini

Json Formatter

I am extremely confused about the exact series of steps involved in having the CPU write a value into a PCIe card's memory. It's very difficult to understand the precise meaning of stuff you read on the internet, so I'm hoping someone can read my theory of what's happening and point out any mistakes.

Setup

Suppose I have a PCIe card with some memory on it. For the sake of discussion, assume the following concrete setup:

  • It has 4 MB to be accessed via base address register 0 (whatever that means)
  • It is the only PCIe card in the whole system
  • It is plugged into a PCIe slot, which connects to the root complex via copper wires in the motherboard
  • There is a root complex connected directly to the CPU's bus (is this the normal way things are connected?)
  • The PCIe card is somehow configured to be device number 0 (how is this done?)
  • We're using Linux.

Let's also settle on terminology:

  • The system bus is the CPU's own bus.
  • The PCIe bus refers the literal wires on the motherboard between the CPU and PCIe slot
  • A driver is a Linux kernel module
  • A device is a literal physical object
  • A device struct is the pci_dev structure filled by the kernel
  • A BAR (base address register) is the field inside a PCIe device's configuration space
  • A BAR space is the address space which is indicated (?) by the value in a BAR.

My theory of what's going on

  1. At boot time, Linux starts probing different addresses to see if there's anything there.

    • Since the PCIe bus wires are only connected to the system bus wires via a bridge (i.e. the PCIe controller on the system bus) Linux must know how to interact with the PCIe controller.
    • Linux sends special commands to the PCIe controller (through memory-mapped IO?) that end up triggering the correct series of voltages on the PCIe wires
    • If it gets a response, it will fill in a pci_dev struct with other information in the configuration space
    • At some point in the future (when?) the kernel will iterate through the list of PCI drivers to try and match them to devices
  2. When Linux detects a device, it will map its BAR space into the system bus. (How is this done?) Suppose it maps the BAR0 space to address 0x55500000 to 0x5550FFFF on the system bus.
    • Linux must have to tell the root complex to listen to these addresses, and that they correspond to the PCIe card it detected.
    • By the way, Linux will set base address register 0 on our PCIe card to have 0x55500000 in its base address field (why bother?)
  3. Subsequent writes on the system bus between 0x55500000 and 0x5550FFFF are "caught" by the root complex, and issued out to the PCIe card
    • The root complex will essentially build a packet will all the headers and checksums and the like and blast them out over the motherboard's copper wires to the PCIe slot
  4. Supposing the CPU wrote 0xDEADBEEF to address 0x55501230 on the system bus, and that the root complex sent out the packet to the PCIe card, the card receives the packet and writes 0xDEADBEEF to 0x01230 in its local 4 MB memory

So: what parts of this are right and which are wrong?

Author:Marco Merlini,eproduced under the CC 4.0 BY-SA copyright license with a link to the original source and this disclaimer.
Link to original article:https://stackoverflow.com/questions/58443506/precise-details-of-writing-a-byte-into-pcie-address-space-from-cpu
yy