WRs
WRs Posted to SQ
Purpose of a SQ WR
Software posts a WR to a QP's SQ in order to initiate a message transfer with a QP of the same type in a remote CA.
Posting a WR to the SQ
A WR is posted to the SQ by executing the Post Send Request verb call. Refer to “Content of SQ WRs” on page 262 for a description of the WR's content.
Types of SQ Operations
The types of WRs that can be posted to a QP's SQ depends on the type of QP:
Message Send operation. Supported on RC, RD, UC, UD, and Raw QPs.
Memory Window Bind operation. Supported on RC, UC, and RD QPs.
Resync operation is atypical. Supported on RD QPs. It should be noted that this operation is automatically initiated by the EEC hardware in the event of an error. It is not initiated by software posting a WR to the SQ of a QP. When a Resync is required, the EEC's Send Logic automatically initiates it. For a detailed description, refer to “Resync Operation” on page 499.
Note that only the Send operation is supported on Raw QPs.
WRs Posted to RQ
Purpose of a RQ WR
Software posts WRs to a QP's RQ in order to handle two types of inbound message transfer requests issued by the remote QP's SQ Logic (see “Inbound Message Types Handled by RQ WQEs” on this page).
Posting a WR to the RQ
A WR is posted to the RQ by executing the Post Receive Request verb. Refer to “Content of RQ WRs” on page 276 for a description of the WR's content.
Only One Type of RQ Operation
Only one type of operation may be specified in a WR posted to the RQ. It is the Receive operation.
Inbound Message Types Handled by RQ WQEs
Software posts WRs to a QP's RQ (once posted, it's referred to as a WQE) to handle the receipt of one of two types of messages sent by the remote QP's SQ Logic:
An inbound message Send operation. In this case, the top RQ WQE contains the Scatter Buffer List defining the series of one or more buffers in the CA's local memory to which the message will be written. When the message's final request packet data payload has been written to the CA's local memory, the top entry on the RQ is retired and a CQE is created on the RCQ. If the final packet includes the ImmDtETH containing a 32-bit immediate data value, the 32-bit immediate data value is stored in the CQE. This data value can be used to inform software associated with the destination QP regarding the nature of the message just sent to it.
An inbound RDMA Write With Immediate message transfer operation. In this case, the WR is really a dummy WR. The final request packet of the RDMA Write operation includes the ImmDtETH containing the 32-bit immediate data value. When the data payload of the final request packet has been written to the CA's local memory, the top entry on the RQ is retired and a CQE is created on the RCQ. The 32-bit immediate data value is stored in the CQE. This data value can be used to inform software associated with the destination QP regarding the nature of the message just sent to it.
Inbound Message Types Not Handled by RQ WQEs
RQ WQEs do not handle the following types of inbound operations sent by the remote QP's SQ Logic:
RDMA Write Without Immediate message transfer operation. Although this operation is handled by the destination QP's RQ Logic, a RQ WQE is not used to define where the inbound message is written in the CA's local memory. Rather, the start memory address, transfer length, and a remote access key (R_Key) are supplied by the remote QP's SQ Logic in the first request packet of the message transfer.
RDMA Read message transfer operation. This type of inbound operation is posted in a special queue (within the RQ Logic) reserved to handle inbound RDMA Read and Atomic operations.
Atomic operations (see the previous bullet item):
- Atomic Fetch and Add operation.
- Atomic Compare and Swap If Equal operation.
WR Content
Content of SQ WRs
Send Operation WR
The message Send operation is supported on all QP types. The input parameters supplied when executing the Post Send Request verb are defined in Table 13-1 on page 263.
RDMA Write Operation WR
The RDMA Write operation is supported on UC, RC, and RD. The input parameters supplied when executing the Post Send Request verb are defined in Table 13-2 on page 266.
RDMA Read Operation WR
The RDMA Read operation is supported on RC and RD. The input parameters supplied when executing the Post Send Request verb are defined in Table 13-3 on page 268.
Atomic Fetch and Add WR
The Atomic Fetch and Add operation is supported on RC and RD. The input parameters supplied when executing the Post Send Request verb are defined in Table 13-4 on page 270.
Atomic Compare and Swap If Equal WR
The Atomic Compare and Swap If Equal operation is supported on RC and RD. The input parameters supplied when executing the Post Send Request verb are defined in Table 13-5 on page 272.
Memory Window Bind WR
The Memory Window Bind operation is supported on RC, UC, and RD. Unlike the other SQ operations, it is not posted to the SQ using the Post Send Request verb. Rather, the Bind Memory Window verb is used. The input parameters supplied when executing the Bind Memory Window verb are defined in Table 13-6 on page 274. A detailed description of memory windows can be found in “Memory Windows” on page 308.
| WR Element | QP Type Applicability | Required? | Description |
|---|---|---|---|
| HCA Handle | All | Yes | Identifies the target HCA. Returned by the Open HCA verb. |
| QP Handle | All | Yes | Identifies the target HCA QP. Returned by the Create QP verb. |
| WR ID | All | Yes | 64-bit, software-assigned unique identifier for this WR. It is stored in the CQE created at the end of the operation. |
| Operation type | All | Yes | Operation type = Send. |
| CQE generation indicator | All | Yes, if... | ...QP was set up to only generate SQ CQE when requested in WR. |
| Gather Buffer List | All | Yes | List of the CA's local memory buffers that contain the message to be sent. The list consists of:
|
| Number of buffers | All | Yes | The number of Gather buffers in the list (note that 0 is valid). |
| Immediate Data Indicator | RC, UC, UD, RD. NA for Raw. | Yes | Indicates if 32-bit immediate data value is to be delivered in final request packet's ImmDtETH. |
| Immediate Data | RC, UC, UD, RD. NA for Raw. | No | 32-bit immediate data value to be delivered in final request packet's ImmDtETH. Only required if the Immediate Data Indicator is set. |
| Fence Indicator | RC, RD | No | If set, all RDMA Read and Atomic WRs posted to SQ before this WR must completed before processing this WR. |
| Address Handle | UD | Yes | Identifies destination port address (DLID), SL, Maximum Static Rate, Source Path Bits. If the destination port is not in the same subnet, also includes DGID, FlowLabel, TClass, HopLimit, and SGID index. |
| DLID | Raw | Yes | 16-bits. Destination port in this subnet. |
| Source Path Bits | Raw | Yes | On transmission, Link Layer receives packet from Transport Layer and uses this value to insert correct port LID address into packet's LRH:SLID field. |
| SL | Raw | Yes | 4 bits. Service Level (QoS) to be used in this subnet. |
| Destination QPN | RD, UD | Yes | 24 bits. Self-explanatory. |
| Q_Key | RD, UD | Yes | 32 bits. When WR posted to RD/UD QP's SQ, software supplies Q_Key assigned to destination QP. Q_Key inserted in outbound request packets:
|
| EECN of local EEC | RD | Yes | 24 bits. This is the EECN of the local EEC through which the message will be sent. In other words, it's the RDC ID. |
| Solicited Event indicator | All | Yes | Tells the QP's SQ Logic whether or not to set the BTH:SE bit in the message's last request packet. |
| Ethertype | Raw EtherType | Yes | 16 bits. Tells SQ Logic what protocol type to indicate in the request packet's RWH:EtherType field. Must conform to definition in IEEE Type Field Registrar (see standards IEEE 802.3, 1998 Clause 3.2.6 Length/Type Field specifications and IEEE 802.1H-1995 for use of the Type Field.) |
| Maximum Static Rate | Raw | Yes | See “Packet Injection Delay” on page 43. |
| WR Element | QP Type Applicability | Required? | Description |
|---|---|---|---|
| HCA Handle | UC, RC, RD | Yes | Identifies the target HCA. Returned by the Open HCA verb. |
| QP Handle | UC, RC, RD | Yes | Identifies the target HCA QP. Returned by the Create QP verb. |
| WR ID | UC, RC, RD | Yes | 64-bit, software-assigned unique identifier for this WR. It is stored in the CQE created at the end of the operation. |
| Operation type | UC, RC, RD | Yes | Operation type = RDMA Write. |
| CQE generation indicator | UC, RC, RD | Yes, if... | ...QP was set up to only generate SQ CQE when requested in WR. |
| Gather Buffer List | UC, RC, RD | Yes | List of the CA's local memory buffers that contain the message to be sent. The list consists of:
|
| Number of buffers | UC, RC, RD | Yes | The number of Gather buffers in the list (note that 0 is valid). |
| Immediate Data Indicator | UC, RC, RD | Yes | Indicates if a 32-bit immediate data value is to be delivered in the final request packet's ImmDtETH. |
| Immediate Data | UC, RC, RD | No | 32-bit immediate data value to be delivered in final request packet's ImmDtETH. Only required if the Immediate Data Indicator bit is set. |
| Fence Indicator | RC, RD | No | If set, all RDMA Read and Atomic WRs posted to SQ before this WR must complete before processing this WR. |
| Destination QPN | RD | Yes | 24 bits. Self-explanatory. |
| Q_Key | RD | Yes | 32 bits. When WR posted to RD QP's SQ, software supplies Q_Key assigned to destination QP. Q_Key inserted in outbound request packets:
|
| EECN of local EEC | RD | Yes | 24 bits. This is the EECN of the local EEC through which the message will be sent. In other words, it's the RDC ID. |
| Virtual Address (VA) | UC, RC, RD | Yes | 64-bit virtual start memory address where data is to be written in remote CA's memory. Placed in first request packet's RETH. |
| R_Key | UC, RC, RD | Yes | 32-bit remote access key. Identifies the sender's access rights within the target memory area. Placed in first request packet's RETH. |
| Transfer length | UC, RC, RD | Yes | 32 bits. Up to 2GB. Placed in first request packet's RETH. |
| Solicited Event indicator | UC, RC, RD | Yes | Only necessary if CQE to be generated (see CQE Generation Indicator in this table). Tells QP's SQ Logic whether or not to set BTH:SE in message's last request packet. |
| WR Element | QP Type Applicability | Required? | Description |
|---|---|---|---|
| HCA Handle | RD, RC | Yes | Identifies the target HCA. Returned by the Open HCA verb. |
| QP Handle | RD, RC | Yes | Identifies the target HCA QP. Returned by the Create QP verb. |
| WR ID | RD, RC | Yes | 64-bit, software-assigned unique identifier for this WR. It is stored in the CQE created at the end of the operation. |
| Operation type | RD, RC | Yes | Operation type = RDMA Read. |
| CQE generation indicator | RD, RC | Yes, if... | ...QP was set up to only generate SQ CQE when requested in WR. |
| Scatter Buffer List | RD, RC | Yes | List of the CA's local memory buffers that message data will be written to when returned by remote QP. The list consists of:
|
| Number of buffers | RD, RC | Yes | The number of Gather buffers in the list (note that 0 is valid). |
| Fence Indicator | RD, RC | No | If set, all RDMA Read and Atomic WRs posted to SQ before this WR must complete before processing this WR. |
| Destination QPN | RD | Yes | 24 bits. Self-explanatory. |
| Q_Key | RD | Yes | 32 bits. When WR posted to RD QP's SQ, software supplies Q_Key assigned to destination QP. Q_Key inserted in outbound request packets:
|
| EECN of local EEC | RD | Yes | 24 bits. This is the EECN of the local EEC through which the RDMA Read request packet is sent and the returned read data is received. In other words, it's the RDC ID. |
| Virtual Address (VA) | RD, RC | Yes | 64-bit virtual start memory address where data is to be read from in remote CA's memory. Placed in first request packet's RETH. |
| R_Key | RD, RC | Yes | 32-bit remote access key. Identifies the sender's access rights within the target memory area. Placed in the request packet's RETH. |
| Transfer length | RD, RC | Yes | 32 bits. Up to 2GB. Placed in first request packet's RETH. |
| Solicited Event indicator | RD, RC | Yes | Only necessary if CQE to be generated (see CQE Generation Indicator in this table). Tells the QP's SQ Logic whether or not to set the BTH:SE bit in the request packet. |
| WR Element | QP Type Applicability | Required? | Description |
|---|---|---|---|
| HCA Handle | RD, RC | Yes | Identifies the target HCA. Returned by the Open HCA verb. |
| QP Handle | RD, RC | Yes | Identifies the target HCA QP. Returned by the Create QP verb. |
| WR ID | RD, RC | Yes | 64-bit, software-assigned unique identifier for this WR. It is stored in the CQE created at the end of the operation. |
| Operation type | RD, RC | Yes | Operation type = Atomic Fetch and Add. |
| CQE generation indicator | RD, RC | Yes, if... | ...QP was set up to only generate SQ CQE when requested in WR. |
| Fence Indicator | RD, RC | No | If set, all RDMA Read and Atomic WRs posted to SQ before this WR must complete before processing this WR. |
| Destination QPN | RD | Yes | 24 bits. Self-explanatory. |
| Q_Key | RD | Yes | 32 bits. When WR posted to RD QP's SQ, software supplies Q_Key assigned to destination QP. Q_Key inserted in outbound request packets:
|
| EECN of local EEC | RD | Yes | 24 bits. This is the EECN of the local EEC through which the request packet is sent and the return read data is received. In other words, it's the RDC ID. |
| Virtual Address (VA) | RD, RC | Yes | 64 bits. Quadword-aligned virtual start memory address of the target memory semaphore. Placed in the request packet's AtomicETH. |
| R_Key | RD, RC | Yes | 32 bits. Remote access key. Identifies the sender's access rights within the destination memory area. Placed in request packet's AtomicETH. |
| Add Data | RD, RC | Yes | 64-bit Add Data. Target QP's RQ Logic reads target location, adds Add Data, and writes result back to memory. Placed in request packet's AtomicETH. |
| Scatter Buffer List | RD, RC | Yes | Specifies a single 8-byte local memory buffer where returned read data will be written. The list consists of a single entry specifying:
|
| Number of Buffers | RD, RC | Yes | Number of buffers in the Scatter Buffer List. Must be one. |
| WR Element | QP Type Applicability | Required? | Description |
|---|---|---|---|
| HCA Handle | RD, RC | Yes | Identifies the target HCA. Returned by the Open HCA verb. |
| QP Handle | RD, RC | Yes | Identifies the target HCA QP. Returned by the Create QP verb. |
| WR ID | RD, RC | Yes | 64-bit, software-assigned unique identifier for this WR. It is stored in the CQE created at the end of the operation. |
| Operation type | RD, RC | Yes | Operation type = Atomic Compare and Swap If Equal. |
| CQE generation indicator | RD, RC | Yes, if... | ...QP was set up to only generate SQ CQE when requested in WR. |
| Fence Indicator | RD, RC | No | If set, all RDMA Read and Atomic WRs posted to SQ before this WR must complete before processing this WR. |
| Destination QPN | RD | Yes | 24 bits. Self-explanatory. |
| Q_Key | RD | Yes | 32 bits. When WR posted to RD QP's SQ, software supplies Q_Key assigned to destination QP. Q_Key inserted in outbound request packets:
|
| EECN of local EEC | RD | Yes | 24 bits. This is the EECN of the local EEC through which the request packet is sent and the return read data is received. In other words, it's the RDC ID. |
| Virtual Address (VA) | RD, RC | Yes | 64-bit quadword-aligned virtual start memory address of the destination memory semaphore. Placed in the request packet's AtomicETH. |
| R_Key | RD, RC | Yes | 32-bit remote access key. Identifies the sender's access rights within the destination memory area. Placed in the request packet's AtomicETH. |
| Swap Data | RD, RC | Yes | 64-bit Swap Data. Placed in request packet's AtomicETH. Target QP's RQ Logic reads target location, compares its contents to Compare Data (see next row). If equal, writes Swap Data back to memory. Original data is returned in the response packet. |
| Compare Data | RD, RC | Yes | 64-bit Compare Data. See previous row. |
| Scatter Buffer List | RD, RC | Yes | Specifies a single 8-byte local memory buffer where returned read data is written. The list consists of a single entry specifying:
|
| Number of Buffers | RD, RC | Yes | Number of buffers in the Scatter Buffer List. Must be one. |
| WR Element | QP Type Applicability | Required? | Description |
|---|---|---|---|
| HCA Handle | RC, UC, RD | Yes | Identifies the target HCA. Returned by the Open HCA verb. |
| QP Handle | RC, UC, RD | Yes | Identifies the target HCA QP. Returned by the Create QP verb. |
| WR ID | RC, UC, RD | Yes | 64-bit, software-assigned unique identifier for this WR. It is stored in the CQE created at the end of the operation. |
| Memory Window Handle | RC, UC, RD | Yes | Returned by Allocate Memory Window verb. |
| R_Key | RC, UC, RD | Yes | The R_Key (Remote access Key) currently associated with the Memory Window. Returned by Allocate Memory Window verb. |
| Memory Region Handle | RC, UC, RD | Yes | Identifies the region that the window will be bound to. Returned by Register Memory Region, Register Physical Memory Region, Reregister Memory Region, Reregister Physical Memory Region, and Register Shared Memory Region verbs. |
| L_Key | RC, UC, RD | Yes | Local Access Key for the memory region that the Window is associated with. Returned by Register Memory Region, Register Physical Memory Region, Reregister Memory Region, Reregister Physical Memory Region, and Register Shared Memory Region verbs. |
| Virtual Address (VA) | RC, UC, RD | Yes | Address of the first byte of the bound range. Maximum width of the Virtual Address is 64 bits. |
| Length | RC, UC, RD | Yes | Length of range to be bound in bytes. |
| Access Control | RC, UC, RD | Yes | Following may be selected in any combination except as noted:
|
| CQE generation indicator | RC, UC, RD | Yes, if... | ...QP was set up to only generate SQ CQE when requested in WR. |
| Fence Indicator | RC and RD. | No | If set, all RDMA Read and Atomic WRs posted to SQ before this WR must complete before processing this WR. |
Content of RQ WRs
As mentioned earlier, a WR is posted to a QP's RQ using the Post Receive Request verb. The only type of operation that may be specified is a Receive operation. The input parameters supplied when executing the Post Receive Request verb are defined in Table 13-7 on this page.