10.1. OVERVIEW OF THE SYSTEM BRINGUP PROCESS
This section presents a high-level overview of the system bringup process. There are several components to this process. We will look at this process from a high level first and then dive into the details of each component of the process.
The host processor fetches the initial boot code (if necessary). If two processors are present, both can fetch the initial boot code.
The system exploration and enumeration algorithm is started.
All devices have been enumerated and stored in the device database, and routes have been set up between the host device and all end point devices. The enumeration process may optionally choose to do the following:
Compute and configure optimal routes between the host device and end point devices, and between different end point devices.
Configure the switch devices with the optimal route information.
Store the optimal route and alternate route information in the device database.
The address space is mapped.
10.1.1. Boot Requirements
The following system state is required for proper system bringup.
After the system is powered on, the state necessary for system enumeration to occur by multiple host processors is automatically initialized as follows:
System devices are initialized with the following base device IDs:
Non-boot-code and non-host device IDs are set to 0×FF (0×FFFF for 16-bit deviceID systems)
Boot code device IDs are set to 0×FE (0×00FE for 16-bit deviceID systems)
Host device IDs are set to 0×00 (0×0000 for 16-bit deviceID systems)
After the system is initialized, the device IDs will all be set to unique values. Before system initialization, the IDs are preconfigured to one of the three values just described.
The RapidIO physical layer links will initialize themselves. This process was described in Chapters 7 and 8 for the serial and physical layers respectively.
The default routing state of all switches between the boot code device and the host devices is set to route all requests for device ID 0×FE (0×00FE for 16-bit device ID systems) to the appropriate boot code device. All response packets are routed back to the requesting host from the boot code device. This will allow the host device to retrieve boot code from the boot device, which will most likely be a flash ROM memory device. RapidIO systems should have only one boot code device and all switches in the system should be configured to forward requests addressed to device ID 0×FE towards the boot code device.
10.1.2. Enumeration
Enumeration is the process of finding all the RapidIO connected devices in a system and assigning them unique device IDs. Although RapidIO networks are quite flexible and provide many features and capabilities, there are a few assumptions and restrictions that can significantly simplify the enumeration process. These assumptions and restrictions are as follows. Only two hosts may simultaneously enumerate a network. Two hosts may be required on a network for fault tolerance purposes. System integrators must determine which hosts should perform this function. Only one host actually completes the network enumeration (this is referred to as the winning host). The second host must retreat and wait for the enumeration to complete or, assuming the winning host has failed, for enumeration to time out. If a time-out occurs, the second host re-enumerates the network.
The enumeration algorithm sets priority on the basis of the value of the power-on device ID. The winning host is the device with the higher power-on host device ID. The losing host has the lower power-on host device ID. The losing host enters a wait state until the winning host completes enumeration or until the wait state times out.
The prioritization mechanism never results in a deadlock if the priorities of both host processors are unique. The enumeration process is initially performed in parallel by both hosts until they meet at a device. When a meeting occurs, prioritization guarantees one winning host—the other host retreats (enters a wait state).
After enumeration, other hosts in the system can passively discover the network to gather topology information such as routing tables and memory maps that may be useful to them.
Any host that participates in the discovery process must change its own device ID to a unique ID value before starting the system initialization process. This value is used by a device's host base device ID lock CSR to ensure only one host can manipulate a device at a time. The allowed ID values for a discovering host are 0×00 (0×0000) and 0×01 (0×0001). A host with an ID of 0×00 (0×0000) has a lower priority than a host with an ID of 0×01 (0×0001).
All host devices, participating in enumeration, should have their master enable bit (port general control CSR) set to 1. Switch devices do not have a master enable bit.
Enumeration is complete when the winning host releases the lock on the losing host. It is the losing host's responsibility to detect that it has been locked by the winning host and to later detect that the lock has been released by the winning host.
As mentioned previously, two hosts can be used to enumerate the RapidIO network. The host with the higher power-on host device ID is generally regarded to have priority over the other host. Because of this pre-defined priority, only one host (the one with higher priority) can win the enumeration task. In this case, the losing host enters a wait state.
If the winning host fails to enumerate the entire network, the losing host's wait state times out. When this occurs, the losing host attempts to enumerate the network. In an open 8-bit device ID system, the losing host must wait 15 Seconds before timing out and restarting the enumeration task. The length of the time-out period in a closed or a 16-bit deviceID system may differ from that of an open system.
10.1.3. Address Space Mapping
Address space mapping would typically occur after device enumeration. Device enumeration builds a database of devices and connectivity in the system. After enumeration it is possible to proceed with address space mapping in the system.
RapidIO devices may offer visibility to zero or more addressable memory regions to other RapidIO devices in a system. Instead of creating a single global address space that all devices must share and map their own regions into, the RapidIO model expects that all devices may have their own independent address regions. Devices can offer a single address space with up to 64 bits of double-word addressability. To make packets more efficient, the address used for any given device may be 34, 50 or 66 bits in length. The target device determines the choice of address length. The address field lengths supported by a target device can be determined by querying the processing element features CAR. The actual length used is established by programming the processing element logical layer control CSR. The CAR tells the system what memory sizes the endpoint can support, the CSR is used to set the size that will be used. A full description of these registers can be found in Chapter 18. Once configured, all memory-addressed transactions to a given end point must use the chosen memory size.
Rather than interact directly with these registers, a set of device access routines (DAR) have been developed. For example the rioDarGetMemorySize() routine may be used to determine the visible memory regions associated with any given RapidIO end point. These routines are described in detail in Section 10.2.4.
The address range that is made visible by a RapidIO end point, in conjunction with the device ID of the end point comprise a unique address in the system. This unique address can be mapped through address translation logic to the local address space of other devices in the system. This means that for any given physical address, composed of a device address and device ID combination, there may be many virtual representations of this address mapped to other devices in the system. Transactions to these apparently local addresses, would be transported across the RapidIO fabric to the target devices physical address.
The RapidIO architecture also anticipates and supports the mapping of PCI-based address ranges into the RapidIO network. PCI devices enter a RapidIO network through a RapidIO to PCI bridge. The address space mapping across this bridge must be done when devices are enumerated and stored in the device database. This allows the address of a found device to be retrieved later and presented to the device access routines during the operating system (OS) initialization. The pseudocode for this process is as follows:
FOR every device in the database
IF the component is a RapidIO device
ACQUIRE the device's address-space requirement
MAP the address space into a new host address partition
EXPAND the partition window to cover the new partition
UPDATE the device database with the new host address
ELSE IF the component is a PCI bridge
ACQUIRE the bridge's PCI bus ID
ACQUIRE the bridge's address-space requirement
// All devices that appear behind this PCI bridge must have their address spaces
mapped within the region specified for this bridge.
MAP the address space into a new host address partition
EXPAND the partition window to cover the new partition
UPDATE the device database with the new host address
ENDIF
ENDFORAfter discovery has been concluded, it is expected that the processing devices in the system will then attempt to load in a software image from a boot device.