14.3. SUPPORTING FUNCTIONS
14.3.1. RapidIO Controller Configuration
Most systems require both 'out-of-band' and 'In-band' configuration capabilities. Out-of-band configuration implies that the devices can be configured using other signaling or protocols beyond the mechanism provided within the RapidIO protocol. Conversely, in-band configuration relies solely on the mechanisms provided by the RapidIO protocol. In either case, it is desirable that all device configuration registers be accessible by either out-of-band or in-band signaling.
There are many standard approaches that can be used for out-of-band configuration of RapidIO devices. Two examples are:
I2C Interface: the I2C Interface is commonly used on many device types and can be used for initializing internal end point registers after reset. The I2C interface can also be used to provide a method for performing random access master read and write operations to EEPROMs or to other I2C compatible slave devices. I2C is an acronym for intelligent I/O controller. It is an industry standard for a low-pin-count serial interface for device initialization and control.
JTAG Interface: a RapidIO end point can also include a JTAG interface as an out-of-band configuration method. The JTAG interface would have the ability to access and configure the internal end point registers. Register access through JTAG can also be used during normal operation to perform extensive read and write accesses on the performance registers, without slowing down the normal traffic in the device. JTAG is an acronym for joint test access group. JTAG is an industry standard for a low-pin-count serial access to internal registers and other state bits of devices.
For in-band configuration, RapidIO maintenance request packets are used. Maintenance packets are the only packet types that access the registers internal to a device, for both read and write operations. An end point can be designed with an internal register bus that enables access to the registers while the device is operating. The register bus would be accessed through RapidIO maintenance packets.
14.3.2. Link Training
RapidIO training is a process that configures two adjacent parallel RapidIO ports for data transfer. The RapidIO training process occurs automatically during system initialization; user intervention is not required. This training process enables a RapidIO inbound module to align its input signals to properly sample data and frame signals transmitted from an adjacent device's outbound module. In a system where an 8-bit parallel RapidIO interface is connected to a 16-bit interface, training is also used to detect the usable width of the 16-bit interface.
The RapidIO training process requires that a pre-defined signal pattern, or training pattern, be applied to the RapidIO interface during initialization. The training pattern is sent after system power-up, system reset, error recovery, or (during normal operation) when the RapidIO inbound module loses the previously established alignment. For example, the alignment can be lost owing to excessive system noise or to a power supply fluctuation.
The training pattern is aligned to the 32-bit boundary, and it is easily distinguishable from control symbols and packet headers. The training pattern is a 64-bit pattern for 8-bit ports with the framing signal switching at the same time as the data bits. The format provides four beats of logic 1 that alternate with four beats of logic 0 for both the data bits and the framing signal of the interface widths. The frame signal does not have to transition high-to-low or low-to-high in phase with the data bits.
The following describes the training (or alignment) sequence:
The first port begins the alignment sequence by transmitting the link-request/send-training control symbol followed by transmitting the training pattern 256 times to the attached port.
At the same time, the first port tries to detect and align its input sampling window to the training pattern that is sent from the attached port. If the first port has completed transmitting the 256 iterations of the training pattern, but has not yet successfully adjusted its input sampling window, it again sends a link-request/send-training control symbol and another 256 iterations of the training pattern.
When the attached port is operating, the first port finishes adjusting its input sampling window to the training pattern coming from the attached port. At this point, the first port sends one idle control symbol (instead of sending a link-request/send-training control symbol) between sending 256 iterations of the training pattern.
The first port continues to send one idle control symbol after sending 256 iterations of its training pattern until it has received an idle control symbol from the attached port.
The training pattern cannot be embedded in a packet or used to terminate a packet. All activity on the inbound module must stop gracefully before training patterns are issued.
Serial RapidIO end points have similar training protocols that must be supported.
14.3.3. Device Maintenance
A RapidIO end point supports the following Maintenance transaction types.
Maintenance read or write request: 4-byte access to configuration registers. The request is directed to the internal register bus. If more than one RapidIO ports are present on the end point device, the register bus may be common for all ports. If this is the case, an arbitration scheme is required to service requests from different ports. Since each port can independently accept read or write request to the register bus, the designer needs to determine how many outstanding maintenance transactions are allowed. Once the maximum number of outstanding transaction is reached, the RapidIO port retries all subsequent transactions.
Maintenance response: read response with 4-byte payload or write completion response. Responses are generated by the register block logic and inserted into the outbound queue (for CRC calculation) of the same port where the request was generated.
Maintenance outbound port-write request: the packet generated by the error handler logic. The port-write packet is forwarded to the port where an error condition had occurred. The end point device should also generate an interrupt to the UDI at the same time.
Maintenance inbound port-write request: the request generated by a remote RapidIO device to report an error condition. The end point device should load the inbound port-write payload into the status registers and generate an interrupt to the UDI.
Maintenance debug packet: a user-defined packet length that is generated on a by-port basis (under user register access) for debug purposes only.
A RapidIO switch claims maintenance packets when the hop count in the packet header is zero. The packet is claimed, de-packetized and the data is sent to the internal register bus. Upon receipt of a maintenance request packet, a maintenance response packet must be assembled in the inbound module and a CRC field generated. The source ID and the destination ID are reversed from the maintenance request packet; the source ID becomes the destination ID. When the packet is fully assembled, the inbound module makes a request to the switching fabric to transfer the response packet to the outbound module on the same RapidIO interface. When the outbound module receives the maintenance response packet, it sends the packet to the RapidIO source.
When a RapidIO switch inbound module receives a maintenance packet with a hop count not equal to zero, the hop count value is decremented and the packet is sent to the internal switching fabric for transmission to an outbound module. Because the hop count field is decremented, the inbound module must recalculate the CRC value prior to sending off the packet. This is the only instance of CRC recalculation within switches for RapidIO packets.
14.3.4. Performance Monitoring
Performance monitoring is a very useful RapidIO switch feature, as it allows observation of device and system status. Since RapidIO traffic can be initiated by unrelated sources and can experience congestion in any destination interfaces, it is desirable to implement performance monitoring on each port of a RapidIO switch.
System software can use the data gathered through performance monitoring to identify and correct situations that impact system performance. For instance, system software can be designed to routinely read the performance monitoring registers, analyze the traffic flow patterns, and re-route accordingly to avoid congestion.
The following are potential parameters that can be gathered as part of a device's performance monitoring:
number of retries on CRC error
number of CRC errors
number of 32-bit words
number of transactions
number of reads/writes for each packet priority (0, 1, 2 and 3)
queue depth for inbound and outbound buffer
number of reordering commands
From these parameters the following system performance metrics can be calculated:
14.3.4.1. Traffic Efficiency
packet rate (number of transactions/time)
average packet size (number of 32-bit words/number of transactions)
utilization ((packet rate × packet size)/max capacity)
The calculations of the packet rate, packet size, and utilization are typically done external to the switch. Registers within the switch could store the count for the number of transactions and the number of 32-bit words for each interface.
14.3.4.2. Bit Error Rate
number of retries on CRC error
number of CRC errors
In each switch interface, counters could be implemented to track the number of times a particular error occurs and store the data in internal switch registers. One of the registers could also contain a programmable threshold field, which when reached, would trigger an interrupt.
14.3.4.3. Throughput
The number of read and write transactions in each interface can be very important parameters when debugging RapidIO systems. This information can also be valuable when used by system software to dynamically re-route traffic around congested interfaces. The following parameters could be used to monitor the throughput on each RapidIO interface:
number of reads for each priority level (0, 1, 2 and 3)
number of writes for each priority level (0, 1, 2)
Each interface could have four counters for reads and four counters for writes (one for each priority level). The value of each counter would be reflected in the internal switch registers.
14.3.4.4. Bottleneck Detection
Monitoring the queue depth of the inbound and outbound modules can detect bottlenecked traffic in a RapidIO switch port. The number of packets waiting to be processed can be counted, and this count placed into internal registers. A queue depth watermark for the number of packets in the outbound and inbound buffers can be programmable. After the value in a programmable threshold is reached, an interrupt can be triggered to notify the system host of a potential bottleneck situation.
14.3.4.5. Congestion Detection
Packets need to be reordered when forward progress through the internal switching fabric is impeded. Packet reordering can be a sign of congestion in a RapidIO interface. A count of the number of times packets are reordered in each interface can be monitored and stored in internal registers. After the value in a programmable threshold is reached, an interrupt can be triggered indicating congestion on a particular interface.
14.3.5. Error Handling
Hardware-based error detection and recovery is a very important aspect of RapidIO systems. Compliance with the RapidIO specification requires that interfaces be designed with extensive error detection and recovery circuitry. Each packet deploys cyclic redundancy check (CRC) to detect bit error conditions. Because RapidIO does not allow dropped packets, every device maintains a copy of the sent packet until it has received a positive acknowledgement from the receiving device.
RapidIO errors that may occur in a system include packet errors, control symbol errors, and time-out scenarios. RapidIO has embedded provisions for error recovery in the case of recoverable errors, but non-recoverable errors require a higher level of resolution. In order to manage errors so they have a minimum effect on system performance, devices must implement a RapidIO error handler. If the error is recoverable, the handler needs to log data about the error and retry the transaction without external intervention. If the error is non-recoverable, the handler logs data about the error and communicates the information to another device, such as a host processor, that is capable of resolving the error condition.
14.3.5.1. Recoverable Errors
The RapidIO interfaces on a device are designed to detect two basic types of recoverable error: an error on a packet (inbound or outbound), and an error on a control symbol. Recoverable errors have provisions within the RapidIO protocol for being retried at the link without any system host intervention. In addition, resource conflicts such as an internal lack of queue space or other local problem can be retried and resolved with no system intervention. A switch can track recoverable errors in one of its registers. These registers would typically not report errors to the local processor until a threshold is exceeded.
Recoverable packet errors (inbound and outbound packets) and recoverable control symbol errors are described in Sections 14.3.5.2 and 14.3.5.3.
14.3.5.2. Inbound and Outbound Packet Errors
There are four basic types of RapidIO packet error:
Unexpected AckID: a new packet has an expected value for the AckID field at the receiver, so bit errors on this field are easily detected and the packet is not accepted.
Bad CRC: CRC protects the Prio, TT, and Ftype fields and two of the reserved bits, as well as the remainder of the transport layer and logical layer fields in a packet.
S-bit Parity: an error on the S-bit is detected with the redundant inverted S-bit parity. The S-bit is duplicated as in the control symbols to protect the packet from being interpreted as a control symbol.
Packet Exceeds Maximum Size: a packet that over-runs some defined boundary such as the maximum data payload (maximum data payload is 256 bytes, maximum total packet size is 276 bytes) or a transactional boundary.
14.3.5.3. Control Symbol Errors
There are three basic types of RapidIO control symbol error:
Corrupt Control Symbol:
True and Inverted 16-bit Halves Do Not Match: the entire aligned control symbol is protected by the bit-wise inversion of the control symbol used to align it to a 32-bit boundary. A corrupt control symbol is detected as a mismatch between the true and complement 16-bit halves of the aligned control symbol.
S-bit parity error: The S-bit, distinguishing a control symbol from a packet header, has an odd parity bit to protect a control symbol from being interpreted as a packet. An error on the S-bit is detected with the redundant inverted S parity bit.
Note: an indeterminate error is an S-bit parity error in which it is unclear whether the information being received is for a packet or a control symbol. These errors are handled as corrupt control symbols.
Uncorrupted Protocol Violating Control Symbols: a packet-accepted, packet-retry, or packet-not-accepted control symbol that is either:
Unsolicited
Has an unexpected AckID value
Time-out: counters that expire when the allotted time has elapsed without receiving the expected response from the system.
Note: a time-out on an acknowledge control symbol for a packet is treated like an acknowledge control symbol with an unexpected AckID value.
14.3.5.4. Non-recoverable Errors
Non-recoverable errors initiate re-synchronization with the adjacent RapidIO device. Once the devices negotiate the bus width, timing, and other necessary considerations, the system can transfer data across the local links and out across the fabric. Non-recoverable errors should be tracked in internal registers and the switch should be capable of generating interrupts for non-recoverable errors as required by the system. There are five basic types of RapidIO non-recoverable error:
Link Response Timeout: a link response is not received within the specified time-out interval.
Link Response Nonsensical AckID: AckID received with unexpected link response.
Error Recovery Threshold Error: error recovery threshold exceeded.
Retry Recovery Threshold Error: consecutive acknowledge retry threshold count exceeded.
Missing Idle After Training: idle not received after a requested training sequence completes.
14.3.5.5. Outbound Port-write
Port-write packets are generated by error conditions and need to be sent to a user programmable destination, generally a system host. Port Write request generation reports error and status information. The payload of the maintenance port-write packet contains the contents of several CSRs, the port on the device that encountered the error condition, and some optional implementation specific information.
The following port-write requirements must be supported in a switch design:
Maintenance port-write requests are sent to a predetermined system host defined in the registers.
The 16-byte data payload of the maintenance port-write packet contains the contents of several registers, the port on the device that encountered the error condition, and some optional implementation-specific information
The port that received the packet detects physical layer errors. The error reporting logic of this port generates a port-write out request and submits it to the predetermined system host.
The sending device sets the port-write pending status bit in the registers. Software indicates that it has seen the port-write operation by clearing the port-write pending status bit.
The port-write generation feature can be disabled per device.
The port-write out request is a packet that does not have guaranteed delivery and does not have an associated response. However, the serial RapidIO interface will continue to send port-write out requests until software clears the status registers. The programmable time out counter is defined in the registers.
A write access in the registers stops the time-out counter of the physical layer's port-write out generation logic.
14.3.5.6. Inbound Port-write
End points should be designed to allow for inbound port-write requests. The number of outstanding requests a device can support is implementation specific. When only one outstanding request is supported and the device receives the maintenance port-write request packet with correct CRC and DestID (optional); it locks the payload in internal registers and generates an interrupt to the local processor. All subsequent inbound port-write requests are discarded until software unlocks the internal registers.
14.3.6. Interrupts
The RapidIO Specification dictates that interrupt programming models are beyond the scope of the RapidIO architecture. Interrupt programming models are typically associated with processing devices and not peripheral devices. It is likely that RapidIO-equipped devices will need to generate interrupts to signal exceptional events to processing devices in a system.
RapidIO devices may participate in the transmission of interrupts within a system. For example, in order to support PCI hardware interrupts between multiple bridges in a system, RapidIO mailbox messages can be used to transfer hardware interrupt data across the RapidIO fabric. A RapidIO bridge configured to receive PCI inbound hardware interrupts would generate special RapidIO outbound mailbox messages when it detects the assertion of a PCI interrupt signal on its PCI bus. A bridge configured to generate PCI outbound hardware interrupts would do so based on receiving these special RapidIO mailbox messages. In this way PCI hardware interrupts may be tunneled through a RapidIO fabric.
The remainder of this section provides information for consideration when implementing interrupts in a RapidIO device.
14.3.6.1. Sample Controller Interrupts
Some sample interrupts generated by a RapidIO controller could be:
Inbound buffer queue threshold interrupt: Two registers would be used to track this interrupt. The first register would set the inbound buffer queue watermark. A second register would be used to monitor the state of the inbound buffer. New packets accumulate in the inbound buffers, destined for the switching fabric. When the number of buffers in use equals or exceeds the watermark, a bit in this register would be incremented by one. A threshold bit defines the number of times the count would be incremented before an error interrupt would be generated. This interrupt indicates there may be a problem with the throughput of data on this port.
Inbound CRC error threshold interrupt: This register would contain a count of the CRC errors. It would define the number of CRC errors that can occur before an error interrupt would be generated.
Inbound reordering threshold interrupt: When a packet cannot make forward progress in the inbound module, the switching fabric should reorder all the packets, based on packet priority. Each time the packets are reordered, the inbound reordering count could be incremented by one. The inbound reordering threshold would define the number of times the inbound reordering count would be incremented before an interrupt is generated. This interrupt would indicate there might be a problem with the throughput of data on this port.
Inbound destination ID not mapped interrupt: This interrupt would be used to signal to the system host that an inbound packet was received with a destination ID that did not have a corresponding mapping in the routing table. The destination ID and switch port the packet was received on should be stored in internal registers.
Outbound buffer queue threshold interrupt: Two registers could track this interrupt event. The first register would set the outbound buffer queue watermark for the number of packets in the outbound buffers. The second register would be used to monitor data congestion in the outbound buffer. New packets come from the switching fabric and accumulate in the outbound buffers. When the number of packets in the outbound buffers equals or exceeds the watermark set in the first register, the outbound buffer queue count would be incremented by one. If this count equals or exceeds the threshold, an interrupt would be generated.
Outbound retry threshold interrupt: A value would be set in a register to dictate the maximum number of attempts to send a packet with a CRC error detected at destination. If the number of attempts to send the packet equals the value set, the packet would be discarded and an error reported to the system host.
14.3.6.2. Switching Fabric Interrupts
It may be desirable to generate an interrupt from within the switching fabric when a time-out is detected, because the requested destination is blocked. When this signal is asserted, it could be used to indicate that the requested transaction could not be completed and will be removed from the request queue. The following registers would be needed to control the interrupts:
A register to control the transaction error signal. This enables or disables the transaction error interrupt, enables or disables the transaction error timer, and sets the number of clock cycles a request will wait for an acknowledge before a transaction error acknowledge.
A register to contain a status bit for every port on the fabric. The status bits indicate on which port(s) a transaction error has occurred.
14.3.6.3. Terminating End Point
In a terminating end point an interrupt generation block would be designed into the device. Error interrupts would be generated simultaneously with outbound port-write generation. The error interrupts provide error reporting to a local processor, whereas the outbound port-write request packets provide error reporting to a remote host, if necessary. The error interrupts are generated under error detection conditions by the receiving port (as described previously). A reset interrupt would be generated when four RapidIO reset-device commands are received in a row without any other intervening packets or control symbols, except status control symbols. Status and enable registers should be included in the end point design to define the reset and interrupt functionality. Similarly, the end point device could be designed to generate interrupts based on receiving a multicast event control symbol or an inbound port-write packet at its inbound RapidIO port.
14.3.6.4. Bridging Device Interrupts
In our example of bridging RapidIO to PCI, an interrupt controller could be used to translate hardware interrupts to and from RapidIO mailbox messages. This translation ability allows multiple bridges to communicate interrupt data across a RapidIO fabric. In addition, it allows bridge devices to share interrupt information with other RapidIO devices in a RapidIO fabric.
Potential features of a RapidIO end point interrupt controller:
Translates hardware interrupts to, and from, RapidIO mailbox messages.
Can be designed to support multiple physical hardware interrupt lines.
Each interrupt line could be independently programmed for direction (inbound or outbound).
Each interrupt could generate, or respond to, a RapidIO mailbox message.
Each interrupt could be routed to any other interrupt and/or RapidIO endpoint attached to a RapidIO fabric.
Support outbound message signaled interrupt addresses (PCI terminology) and/or message translated interrupt addresses (RapidIO terminology).
Support multiple pending hardware requests per outbound hardware interrupt.
Handle hardware interrupts and message translated interrupts simultaneously from PCI to RapidIO.
Handle PCI SERR/PME forwarding functions across a RapidIO fabric.
14.3.6.5. Mapping to Interrupt Output Pins
A RapidIO device or switch can be designed with any number of its own output interrupt pins. Interrupts that occur within a device can be routed to one of the device interrupt output pins.
The following registers would be needed to control the interrupts:
A register to indicate which port has generated an interrupt. It could show which part of the device has generated the interrupt when the routed interrupt output pin is signaled.
A register to route the interrupt's sources (such as inbound and output modules or the internal switching fabric) to a particular interrupt output pin.
14.3.7. Reset
The RapidIO specification dictates that system reset is beyond the scope of the RapidIO architecture. However, the RapidIO specifications describe mechanisms for communicating reset information within maintenance packets. RapidIO controllers will need to respond appropriately to these packets. This section describes some considerations for resetting RapidIO devices.
Hardware reset: this event would reset all of a RapidIO device's facilities.
Inbound RapidIO maintenance reset: an external linked RapidIO device uses RapidIO link maintenance request packets containing reset commands to initiate the inbound RapidIO maintenance reset. According to the RapidIO specification, reset needs to be triggered when a switch receives at least four link maintenance request packets containing reset commands in a row, without any other intervening packets. After the fourth packet, the switch would generate an interrupt to signify the RapidIO software reset request. No reset action is taken by the RapidIO ports of the switch in response to the request. The system must provide a reset controller device on the board to determine how to process the reset request. The interrupt pin would be asserted when the software reset request is validated.
Outbound RapidIO maintenance reset: a switch could reset an external linked RapidIO device using RapidIO link maintenance reset request packets it would generate from one of its RapidIO interfaces. The system would instruct the switch to send the reset packets to the external linked RapidIO device. Once the device receives the reset packets, it would determine how to process the reset request. According to the RapidIO specification, a device receiving a reset command in a link-request control symbol should not perform the reset function unless it has received at least four reset commands in a row without any other intervening packets or control symbols, except idle control symbols. Based on a system host setting appropriate values in the switch control registers, the switch would begin the link maintenance reset. The switch then issues the RapidIO maintenance reset sequence of at least four consecutive link maintenance control symbols containing reset commands to an external linked RapidIO device.
14.3.8. Clocking
14.3.8.1. Parallel Switch Clocking
A parallel RapidIO interface supports clock input and output options. The clock inputs are received by the switch's RapidIO high-speed inbound module from an adjacent RapidIO device. The clock outputs are transmitted by the switch's outbound module to an adjacent RapidIO device. The outbound module needs to drive the clock signals (differential signals TCLK_p and TCLK_n) to ensure it operates in an exact phase relationship with the adjacent device.
14.3.8.2. End Point Clocking
A parallel RapidIO Interface needs to support clock input and output options (serial RapidIO devices have 8b/10b embedded clocks in the serial links). The RapidIO inbound module receives the clock inputs from an adjacent RapidIO device. The inbound module receives the clock signals and would generate an internal clock that has an exact phase relationship with the adjacent device output clock. The outbound module transmits the clock outputs to an adjacent RapidIO device. The outbound module should drive the clock output such that it operates in an exact phase relationship with the adjacent device.
14.3.8.3. RapidIO Interface PLL Configuration
A parallel RapidIO interface can be designed with a clock generation phase-locked loop (PLL) configurable to clock the transmitter interface at the appropriate speed. The speed must be selected before the de-assertion of reset using a clock select signal. Once reset is de-asserted the speed would be latched internal to the switch. In our sample switch we would use an internal PLL to create clocking options for the core clock and the RapidIO interfaces.
14.3.8.4. Sample PLL Bypass
To provide greater system design flexibility, it may be desirable to bypass the PLL-based clock within the switch. The PLL bypass clock signals would need to be differential inputs of a high-speed LVDS-compatible clock input interface. The transmit clock, which is created from the bypass clock, is half the frequency of the bypass clock. This division by two would be performed within the switch to ensure the correct duty cycle of the transmit clock. As such, the frequency used for the PLL bypass clock must create a supported RapidIO transmit frequency.
14.3.8.5. Serial Switch Clocking
Using SERDES in a serial switch, it may be desirable for each SERDES channel to have its own receive and transmit clocks, so that individual ports could run at different frequencies. Data to and from each SERDES would need to be written to an elastic buffer that is clocked on the system clock domain (recovered from the 8b/10b link). The elastic buffers are used for crossing the clock boundary, lane de-skew and clock compensation. Control symbols must be decoded at the system clock speed in order to keep up with the link in 4x mode. Packet data would need to be aggregated to an internal parallel bus and passed, through a FIFO, into the internal fabric clock domain. The use of a parallel bus permits the internal fabric to run on a clock that is slower than the SERDES clock.
A serial switch can be designed with multiple ports that may operate at different link speeds. Our sample RapidIO Serial switch would support the following clock requirements:
independent link speed for each port
no dependency to system clocks of each port
all gated clocks and a general clock must be aligned
50% duty cycle to all clocks