RC/UC Connection Establishment
Figure 10-1 on page 193 illustrates the process of establishing a connection between two RC or UC QPs. It should be noted that, although the dialogue shown is 100% correct, it does not include all of the information that is exchanged in the REQ and REP messages.
Figure 10-1. RC/UC Connection Establishment
Local QP Initial Creation and Setup
Refer to Figure 3-4 on page 58.
CQ(s) and PD Created Before Creating the Local QP
CQ Creation
Before creating the local QP, software must first cause the HCA to create the CQ(s) to be associated with the SQ and RQ of the QP about to be created. The QP's SQ and RQ can have separate CQs or may share a CQ. To create one or two CQs, software executes the Create CQ verb call once or twice. When calling this verb, software provides the following input parameters:
- HCA identifier (referred to as the HCA Handle).
- The desired size of the CQ to be created.
The verb layer commands the HCA to create a CQ of the requested size. The verb then returns the following output parameters:
- Handle that identifies the newly created CQ.
- The actual size of the CQ (the HCA may or may not support a CQ of the requested size; if it doesn't, it creates a CQ of a smaller size and the verb returns the actual size to the caller).
PD Creation
In addition to the CQ(s) to be associated with the QP about to be created, the PD (Protection Domain) that the QP will belong to must also have been created. A PD is created by executing the Allocate PD verb. The only input parameter is the HCA handle, and the only output parameter is the PD value.
Create the Local QP
See step one in Figure 10-1 on page 193. Software causes a QP to be created in a HCA by executing a Create QP verb call. In response, the verb layer commands the HCA Interface to create a QP of the desired type. In this case, a RC or UC QP is created.
Create QP Input Parameters
When calling the Create QP verb, the following information is provided as input parameters (note that this a partial list):
- The HCA Handle.
- The type of QP (in this case, RC or UC).
- Handles of the CQs to be associated with the QP's SQ and RQ.
- The maximum number of WRs software expects to post to the QP's SQ and RQ.
- The maximum number of buffers that will be specified in the Gather or Scatter Buffer Lists of WRs posted to the QP's SQ and RQ.
Create QP Output Parameters
The HCA creates the local QP and the verb returns the following output parameters to software:
- The handle of the newly created local QP.
- The QPN associated with the local QP.
- The actual number of WRs that may posted to the QP's SQ and RQ (the HCA may or may not support a SQ and RQ of the requested size; if it doesn't, it creates a queue of a smaller size and the verb returns the actual size to the caller).
- The actual maximum number of buffers that the HCA supports in the Gather or Scatter Buffer Lists of WRs posted to the QP's SQ and RQ.
Assign HCA Port To QP
See step one in Figure 10-1 on page 193. Immediately after creation, the QP is by no means ready to send and receive messages. Using the Modify QP verb call, software now provides additional information to the newly created local QP. One of these items is the local HCA port (specified as the port number) that the QP will use to send and receive messages.
Software “Remembers” Local QP's Handle
After creation of the local QP in the HCA, software remembers the QP's assigned handle (see “Create the Local QP” on page 187). The handle must be used when:
Posting WRs to the QP's SQ or RQ.
Modifying the QP's operational characteristics.
Checking the QP's current operational characteristics.
Destroying the QP.
Send REQ GMP to Remote CA's CM
Software Issues Connection Request to HCA's CM
See step two in Figure 10-1 on page 193. Also refer to Figure 10-2 on page 194. The next step is to cause the HCA's CM to send a communications management REQ (Request for connection establishment) message to the remote CA's CM (Communications Manager). To do this, software issues a connection establishment request to the CM (Communications Manager) associated with this HCA. the CM is typically a software application running on the host processor. The API that the software application desiring the connection establishment uses to call the CM is outside the scope of the specification.
Figure 10-2. Connection Establishment Message Exchange
Each Port on a CA Implements QP1 (Its GSI)
Upon receiving the request for the establishment of a private communications channel between the HCA's RC or UC QP and a QP of the same type on the remote CA, the CM must cause the HCA to issue a REQ message to the CM within (or behind) the remote CA. Each port on a CA implements its own, dedicated QP1. It is referred to as the port's General Services Interface (GSI) and is used to send and receive General Services Management Packets (GMPs) on behalf of the various General Services Managers (GSMs). The CM is a GSM and it uses the GSI associated with one of the HCA ports to send and receive GSMs.
CM Builds 256-byte REQ Message in Memory
The CM builds (i.e., writes) the 256-byte REQ message in main memory. This message specifies the following (see step two in Figure 10-1 on page 193):
Class = ComMgt. The GSI that receives the packet on the remote CA will deliver the packet to the CA's CM for processing.
Method = ComMgtSend. Informs the remote CA's CM that this is a connection establishment message type.
Attribute = ConnectRequest. Specifically, it is a REQ message.
The Service ID. Specifies which IOC service to establish a connection with. See “Discovering Services a CA Provides” on page 185.
The QP type (RC or UC) to be created on the remote CA.
The QPN of the HCA's QP (so the remote QP's SQ Logic can send messages to it).
The HCA QP's SQ Start PSN. The QP to be created on the remote CA will assign this as the ePSN used by its RQ Logic.
The address of the local port used by the HCA QP (so the remote QP can send packets to the HCA QP through the proper port).
The port address of the remote CA port that the new remote QP is to use to send and receive messages.
CM Posts WR To SQ of Any HCA Port's GSI
The CM then executes a Post Send Request verb call to post a WR to the SQ of any HCA port's GSI. The WR specifies the following (this is a partial list):
The start memory address of the REQ message.
The operation type (in this case, a message Send).
The LID address of the port on the remote CA that the REQ message will be delivered to (this can be any port on the remote CA). This will be placed in the GMP's LRH:DLID field.
The destination QPN is set to QP1 (the destination port's GSI).
HCA Port's GSI SQ Logic Reads REQ Message from Memory
On executing the posted WQE, the GSI's SQ Logic reads the 256-byte REQ message from memory in preparation for sending it. It is placed in the outgoing GMP's data payload field.
Message Transmitted and Received
The REQ message is wrapped in a packet and is transmitted by the port the CM is using to send the packet. The packet is addressed to any port on the remote CA (via its LRH:DLID field). In addition, the BTH:DestQP field is addressing the packet to the destination port's GSI (i.e., its QP1). The packet traverses one or more links, arrives at the destination port, and is delivered to that port's GSI RQ Logic.
GSI Uses Message's Class to Route to CM
The GSI's RQ Logic looks at the Class field in the packet and routes it to the CA's CM for processing.
CM Determines It's a REQ Message and Creates QP
See step three in Figure 10-1 on page 193. Upon receipt of the REQ message, the remote CA's CM takes the following actions:
Checks the supplied ServiceID to see if the requested service is supplied by one of the IOCs within the IOU.
It creates a QP of the requested type (RC or UC).
It stores the remote QPN and its port address in the newly created QP's Context.
It assigns a QPN to the new QP.
It assigns a Start PSN to the new QP's SQ Logic.
It sets the new QP's RQ ePSN = the Start PSN assigned to the HCA QP's SQ.
It associates the new QP with the local CA port defined in the REQ message.
At this point, the new QP is ready to receive messages, but it cannot send messages until the HCA QP's setup has been completed.
Remote CA's CM Returns REP Message
See step four in Figure 10-1 on page 193. The remote CA's CM returns a REP (communications reply) message containing the QPN and SQ Start PSN assigned to the new QP. The message is sent back to the HCA by the GSI on the CA port that received the REQ message. It is sent to the GSI on the HCA port that originally sent the REQ message.
REP Message Routed to HCA's CM
The GSI's RQ Logic looks at the Class field in the packet and routes it to the CA's CM for processing.
HCA CM Supplies REP Message to Software Application
The CM, in turn, passes the information contained in the REP message back to the software application that originated the connection establishment request.
Software Completes Setup of HCA QP
See step five in Figure 10-1 on page 193. Using the information contained in the REP message, the software application takes the following actions (by executing the Modify QP verb call) to complete the setup of the HCA QP:
It sets the HCA QP's RQ ePSN = the Start PSN assigned to the remote QP's SQ.
It stores the remote QP's QPN in the HCA QP's Context.
When its setup has been completed, the HCA QP is fully enabled for message send and receive.
Software Application Tells CM to Send RTU Message
See step six in Figure 10-1 on page 193. The software application then issues a request to the HCA CM to send a Ready To Use (RTU) message to the remote CA's CM. The CM has its CA port's GSI send the message to the GSI of a port on the remote CA.
On Receipt of RTU, Remote CM Activates QP
See step seven in Figure 10-1 on page 193. When the port on the remote CA receives the packet, it sends it to its GSI's RQ Logic. The RQ Logic determines from the packet's Class field that the packet is to be sent to the CA's CM. The CM receives the RTU message and activates its local QP (i.e., it transitions it to the RTS state). It is now ready to send and receive messages with the HCA QP.
It should be noted that the remote QP will automatically make the transition from the RTR state to the RTS state (even if its CM has not yet received the RTU message) upon receipt of the first inbound request packet generated by the remote QP's SQ Logic.
RC/UC WR Doesn't Specify Target Port or QPN
A WR posted to the SQ of a RC or UC QP does not specify the target remote port or QPN. There is no need because the local QP “knows” the target port and QPN (because this information is stored in the QP Context of the local QP).