WQE Execution and Completion Order
SQ WQE Execution and Completion Order
SQ WQE Execution Order
Irrespective of the QP type, message transfer requests (WQEs) posted on a QP's SQ are always executed one at a time in the order that they were posted. They are never executed out of order.
SQ WQE Completion Order
Likewise, SQ WQEs always complete in the same order that they were posted.
RQ WQE Execution and Completion Order
RQ WQE Execution Order
For all of the transport service types, RQ WQEs always begin execution in the order that they were posted by software.
As described earlier, WRs are posted to a QP's RQ to handle two types of inbound message transfers:
An inbound message Send operation.
An inbound RDMA Write With Immediate message transfer operation.
Other types of inbound messages (RDMA Write Without Immediate, RDMA Read, or an Atomic operation) are not handled by RQ WQEs.
As each inbound Send or RDMA Write With Immediate arrives at the QP's RQ Logic, the RQ Logic accesses the WQE at the head of the RQ:
If it's a Send operation, the WQE's Scatter Buffer List defines where the message is to be written in local memory. As each of the message's request packets are received, the data payload of each is written to memory. Upon receipt of the last request packet of the message, the final data payload is written to memory. The RQ WQE is then retired and a CQE is created on the RQ's CQ. If the final packet includes the ImmDtETH, the 32-bit immediate data value contained in the ImmDtETH is stored in the CQE.
If it's an RDMA Write With Immediate, the data payload of each request packet is written to memory using the address information supplied in the first request packet's RETH. Upon receipt of the last request packet of the message, the final data payload is written to memory. The RQ WQE is then retired and a CQE is created on the RQ's CQ. The 32-bit immediate data value contained in the last packet's ImmDtETH is stored in the CQE.
When the entire Send or RDMA Write With Immediate has been received and processed, the RQ Logic advances to the next WQE to handle the next inbound Send or RDMA Write With Immediate.
In all but one case, the QP's RQ Logic will not move on to the next RQ WQE until it has completed handling the current Send or RDMA Write With Immediate operation. This is true for RC, UC, and UD. The exception to the rule is RD (see the next section).
RQ WQE Completion Order
RC, UC, UD RQ WQE Completion Order
As just demonstrated, when using the RC, UC, or UD transport type, the QP's RQ Logic will not move on to the next RQ WQE until it has completed handling the current Send or RDMA Write With Immediate operation. This being the case, when using the RC, UC, or UD transport type, RQ WQEs always complete execution in the same order as they were posted.
RD RQ WQE Completion Order
RD RQ WQEs can complete out of order. Refer to Figure 13-1 on page 280 and consider the following scenario:
Use any RD QP in either CA to act as the target QP in the example. Also notice that in both of the example CAs, two separate RDCs exists (i.e., two local EECs).
A remote RD QP starts a Send operation to Send a 100-packet message to the example RD QP's RQ Logic using the RDC consisting of the upper EEC in each of the two CAs.
When the first request packet of the Send is delivered to the upper EEC's Receive Logic in the destination CA, the EEC's Receive Logic reads the top entry from the destination RD QP's RQ and saves it within the EEC. This contains the Scatter Buffer List defining where the incoming message is to be written in the CA's local memory. Copying the WQE to the EEC will allow the RD QP's RQ to advance its RQ pointer to the next WQE should another inbound message show up through another RDC.
Using the information in its copy of the RD QP's RQ WQE, the EEC's Receive Logic stores the data payload of the Send message's first request packet in the CA's local memory.
Another remote RD QP has also started a Send operation to send a 25- packet message to the same target RD QP's RQ Logic, but using the RDC consisting of the bottom EEC in each of the two CAs.
When the first request packet of this Send is delivered to the lower EEC's Receive Logic in the destination CA, the EEC's Receive Logic reads the next WQE from the target RD QP's RQ and saves it within the lower EEC. This contains the Scatter Buffer List defining where the incoming message is to be written in the CA's local memory.
Using the information in its copy of the RD QP's RQ WQE, the EEC's Receive Logic stores the data payload of this Send message's first request packet in the CA's local memory.
The Receive Logic in each of the two EECs will use their copies of their respective RQ WQEs to store inbound message packets as they arrive from the remote EECs.
In the example, the second RQ WQE would complete execution before the first one does because it only has to handle 25 incoming packets while the first WQE has to handle 100 packets.
Figure 13-1. RD RDCs
This is a classic example of why RD RQ WQEs can complete out of order.