Queue Key (Q_Key)

The Problem

An unauthorized RD or UD QP may attempt to deliver a message to a remote QP.

A Little Background

Why This Isn't a Problem for RC and UC

When using the RC or UC service types, a local QP of either of these types can send messages only to the remote QP that it was initially connected to. By definition, any message transfer request posted to its SQ will be sent to the remote QP identified in the local QP's Context. The target QP is therefore not supplied as a parameter in WRs posted to the local QP's SQ. The Q_Key protection mechanism prevents a local QP from sending messages to various remote QPs that it isn't authorized to access. Since RC and UC QPs cannot exhibit this type of behavior, the Q_Key protection mechanism doesn't apply to the RC and UC service types.

Why This Is a Problem for RD and UD

When using the RD or UD service types, however, the local QP was not connected to a specific remote QP when it was initially created and setup. Rather, each WR posted to the local QP's SQ specifies the target remote QP. This means that each message transfer request can target a different remote QP of the same service type. Therein lies the problem, and the Q_Key protection mechanism prevents local QPs from sending messages to remote QPs that they aren't authorized to access.

Basic Q_Key Description

The following is intended as a basic description of how Q_Keys are created and used. For a detailed description, refer to “RD Q_Key Usage” on page 336 and “UD Q_Key Usage” on page 339.

Local QP Assigned a Q_Key at Creation Time

After a local RD or UD QP is created, it is assigned a 32-bit Q_Key by software (assuming that the QP is in an HCA). Software uses the Modify QP verb to provide the newly created QP with this Q_Key and it is stored in the QP Context.

QP Creation and Setup Is Privileged Operation

It should be noted that only kernel mode (i.e., privileged) software is permitted to execute the Create QP and Modify QP verb calls. Therefore, only kernel mode software can assign a Q_Key to the newly created QP.

Some Q_Keys Are Privileged

A Q_Key with its high-order bit (bit 31) set to one is considered to be a privileged Q_Key (referred to as a controlled Q_Key) kernel mode software assigns to QPs that provide special services. A classic (and, as of this writing, the only) example would be the Q_Key reserved for QP1 which acts as the GSI (General Services Interface) for sending and receiving GMPs. It has a special, controlled Q_Key of 80010000h.

Remote QP Is Then Created and Q_Keys Are Exchanged

Software local to the HCA causes an exchange of special communications management messages (for an introduction to the communication establishment process, refer to the chapter entitled “Intro to Connection Establishment” on page 183) with a remote CA, causing a QP of the desired type to be created in the remote CA. During this message exchange, the following actions are taken:

• In the communications request message (REQ) sent to the remote CA's CM by the local CA's CM, the local CM provides the local QP's QPN and Q_Key to the remote CA's CM.

• The remote CA's CM creates a new QP and a locally assigned 32-bit Q_Key is stored in its context.

• In the communications reply message (REP), the remote CA's CM provides the QPN and Q_Key of the newly created QP back to the local CA's CM.

Software Remembers the Remote QP's Q_Key

When it receives the reply message from the local CA's CM, software stores the QPN and Q_Key of the remote QP so it can use them when posting message transfer requests to the HCA QP's SQ.

Remote QP's Q_Key Placed in All Message Packets Sent to It

Once communications has been established between the two QPs, when software associated with the HCA QP wishes to send a message to the QP in the remote CA, it uses the Post Send Request verb to post a WR to the SQ of the local QP. The QPN of the destination QP is specified along with its Q_Key. When the local QP's SQ executes the request, it embeds the destination QPN and its Q_Key (see the next section) in each request packet of the message it transmits.

On Message Sends

As mentioned in the previous section, when software posts the WR to the local QP's SQ, it specifies the destination QPN and Q_Key of the remote QP to whom the message is to be sent. It is possible that, in an illegal attempt to communicate with a remote QP that provides special services, the software posting the request might specify one of the special, controlled Q_Keys (with its high-order bit set to one). The following two subsections describe the two cases: the first wherein the WR specifies a normal (i.e., not a controlled) Q_Key, while in the second case, the WR specifies a controlled Q_Key.

If Specified Q_Key Is Not a Controlled Q_Key

If the high-order bit of the 32-bit Q_Key supplied in the WR is cleared to zero, this instructs the local QP to embed the Q_Key supplied in the WR (rather than the one from QP's context) in each of the outbound packets it generates for this message. The Q_Key in the WR is the Q_Key that was assigned to the remote QP when it was created and it is not a controlled Q_Key. The target QPN is supplied in the WR.

If a Controlled Q_Key Is Specified

If the high-order bit of 32-bit Q_Key supplied in the WR is set (i.e., a controlled Q_Key is specified), this instructs the local QP to embed its own Q_Key (contained in its QP context), rather than the one supplied in the WR in each of the outbound packets it generates for this message. In other words, the message is to be sent to the target QP specified in the WR using the sender's Q_Key. Upon packet receipt, unless the destination QP was assigned the same Q_Key, it will have a miscompare between its own Q_Key and the one in the packet and will therefore discard the packet.

It should be noted, however, that the Q_Key of the local QP (i.e., the one in its context) could have been set to a controlled Q_Key by kernel mode software when the QP was initially set up. In this case, software would be successful in sending a message with a controlled Q_Key in its packets.

On Message Receipt

On receipt of a message request packet from a remote QP, the Q_Key in the inbound packet is checked against the receiving QP's Q_Key (i.e., the Q_Key stored in the receiving QP's Context). The packet is accepted if the Q_Key matches and is dropped if it does not.

RD Q_Key Usage

Introduction

The process of initially creating a local and a remote RD QP (and, possibly, additional RD QPs in the two CAs) that can exchange messages with each other (through a RD Channel, or RDC) is a multi-step process. The following sections explain each step in the process. The discussion assumes that the RDD to which the local EEC and QP(s) will belong has already been created using the Allocate RDD verb. For a detailed description of the RD service, refer to “RD Transport Service” on page 461.

Step One: Create HCA RD QP(s) and Assign Q_Key(s)

The first step is to create one or more local RD QPs in the HCA by executing the Create QP verb call one or more times. The verb returns the QPN assigned to the newly created local RD QP. A subsequent Modify QP verb call is used to assign a Q_Key to the newly created QP. Note that the 32-bit Q_Key is generated by the software making the call and is provided as one of the input parameters to the Modify QP verb. It is not generated by the verb itself.

Step Two: Local EEC Creation

The next step is to create a local EEC in the HCA by performing a Create EEC verb call. The verb returns the EEC Number (EECN) that identifies the newly created EEC. This EEC acts as the local end of the pipeline (i.e., RDC) through which multiple local and remote RD QPs can communicate with each other. Software remembers the EECN of the local EEC so it can use it when posting message transfer requests (i.e., WRs) to a local RD QP's SQ. Note that additional local EECs can be created in the HCA by making additional Create EEC verb calls.

Step Three: Create Remote EEC and Initial Remote RD QP

The next step is to cause the creation of a corresponding EEC and RD QP in the remote CA. Software accomplishes this by requesting that the local HCA's CM send a communications request (REQ) message to the remote CA's CM. The REQ message contains the following information:

• The EECN of the local EEC that will act as the HCA's end of the RDC (i.e., the EEC-to-EEC connection).

• HCA's RD QPN.

• Q_Key assigned to the HCA RD QP.

• HCA port address (GID and/or LID) through which the HCA EEC will send and receive messages.

• The port address that the remote CA should use as its end of the RDC.

In response to receipt of the REQ message, the remote CA's CM takes the following actions:

Upon receipt of the REP message, the HCA's CM passes the REP message to software that saves the QPN and Q_Key of the newly created remote QP, so it can use them when posting message transfer requests to the HCA QP's SQ.

Step Four (optional): Create Additional Remote RD QP(s)

If so desired, the previously created EEC-to-EEC pipeline (i.e., RDC) can be used by multiple local and remote RD QPs (in the two example CAs) to send messages to each other. To create additional RD QPs in the remote CA, the following actions are taken:

Validating an Inbound Request Packet

On receipt of a message request packet from a remote QP, two checks are performed:

• The receiving EEC validates that its RDD and the RDD that the destination RD QP (identified by the packet's BTH:DestQP field) belongs to are the same. If not, the packet is dropped and an Invalid RD Request Nak is returned.

• The Q_Key in the inbound packet is checked against the receiving QP's Q_Key (i.e., the Q_Key stored in the receiving QP's Context). The packet is accepted if the Q_Key matches and dropped if it doesn't. In addition to dropping the packet, the QP's RQ Logic returns an Invalid RD Request Nak to the QP that sent the packet.

UD Q_Key Usage

Background

An IOC Provides Device-Specific Services

An IOU (IO Unit) consists of one or more IOCs (IO Controllers) interfaced to the IBA fabric via a TCA. The purpose of a specific IOU and the IOC(s) it contains is to provide device-specific services to remote entities (such as software modules residing on a processor node and interfaced to the IBA fabric via an HCA).

IOC May Provide Service(s) Through UD QP(s)

The manner in which a remote entity may request that an IOU perform a specific service may be by sending it messages through an UD QP within the IOU's TCA. Likewise, the specific service within the IOU may send messages to an UD QP within the HCA. In other words, a software module associated with the HCA may send requests to and receive responses from the remote IOU via a pair (one local and one remote) of UD QPs within the HCA and the IOU's TCA.

Discovering If an IOC Provides a Specific Service

To discover if a remote IOC provides a specific service through a UD QP, a software application must cause the HCA's CM to exchange a pair of special communications messages with the IOU's CM. The messages are exchanged between a GSI (QP1) within the HCA and a GSI within the IOU's TCA. A detailed discussion of how this is accomplished can be found in “UD Connection Issues” on page 202.

Step One: Create HCA UD QP(s) and Assign Q_Key(s)

The local HCA UD QP is created by software using the Create QP verb call. Software then uses the Modify QP verb call to assign a 32-bit Q_Key to the newly created UD QP.

Step Two: Ask Remote CA If It Supports Desired Service Type

As described in “UD Connection Issues” on page 202, the software application then causes the HCA's CM and the remote CA's CM to exchange the SIDR_REQ and SIDR_REP messages. Assuming that the SIDR_REP message indicates that the requested service is supported by an IOC within the IOU, the software application saves the QPN and Q_Key of the IOU's UD QP to use when posting message transfer requests to the HCA's UD QP.

Validating an Inbound Request Packet

On receipt of a message from a remote UD QP, the Q_Key in the inbound packet is checked against the receiving UD QP's Q_Key (i.e., the Q_Key stored in the receiving QP's Context). The packet is accepted if the Q_Key matches and is dropped if it doesn't.