SQ Logic Can Use MSN to Complete WQEs
Initial State of MSN and SSN
When a RC QP is first created, its MSN and SSN are initialized as follows:
The RQ Logic sets its MSN = 000000h.
The SQ Logic sets its SSN = 000000h. The first WQE posted to the SQ has an SSN of one. As each WR is posted to the SQ, the SQ Logic assigns it the next sequential SSN.
MSN Change Indicates One or More Messages Completed
The responder QP's RQ Logic returns its current credit count and MSN value in each Ack packet's AETH (see Figure 17-6 on page 369). In Figure 17-21 on page 430 the first WQE posted to the requester QP's SQ Logic causes it to start a two-packet Send operation. The two packets have PSNs of 15 and 16, respectively:
The responder QP's RQ Logic receives the first request packet, writes its data payload to memory, and returns an Ack packet. The Ack packet has a PSN of 15 and MSN of zero. The RQ Logic has not updated the MSN because it has not yet completed the receipt of the message.
The SQ Logic receives the first Ack packet. Since the MSN is still zero, the SQ Logic cannot retire any WQEs.
The responder QP's RQ Logic receives the second request packet, writes its data payload to memory, and returns an Ack packet. This Ack packet has a PSN of 16 and the MSN has been updated to one, reflecting the completion of the message with an SSN of one.
Upon receipt of the second Ack packet, the SQ Logic sees that the MSN has been updated to one, indicating that it can retire the SQ WQE with an SSN of one.
Figure 17-21. MSN Usage on First Message Transmit
Multiple SQ WQE Retirement
Figure 17-22 on page 431 illustrates an example wherein the responder QP's RQ Logic is coalescing its responses. In this case, the RQ Logic waits until it has completed the receipt of three Send messages before it sends a coalesced response back to the SQ Logic. When the SQ Logic receives the coalesced Ack packet and determines that the MSN has been updated to three, it retires the three requests with SSNs of one, two, and three.
Figure 17-22. MSN Used To Retire Multiple SQ WQEs
RDMA Read or Atomic Requests Make It More Interesting
During this discussion, remember the rule regarding completion of SQ WQEs: SQ WQEs must always complete in the order that they were posted.
Refer to Figure 17-23 on page 433. When there are any outstanding RDMA Read or Atomic requests on the requester QP's SQ, the situation is a little more interesting. Upon receiving an Ack packet with an updated MSN, the requester's QP's SQ can only retire outstanding WQEs up to:
the first outstanding RDMA Read request, or
the first outstanding Atomic request, or
the SQ WQE whose SSN matches the MSN in the response packet,
Figure 17-23. MSN and RDMA Read and Atomic Operations
whichever comes first.
The requester cannot complete and retire an RDMA Read operation before it has received all of the requested read data and written it to local memory. Likewise, it cannot complete and retire an Atomic operation before it has received the data read from the target remote location and written it to local memory.
In Figure 17-23 on page 433, the startup conditions are as follows:
The first four WRs posted to the SQ are: two single-packet Send operations, a three packet RDMA Read operation, and a single-packet Send operation.
These WQEs are assigned SSNs 1 through 4.
The startup value of the remote QP's RQ Logic MSN is zero.
The Start PSN assigned to the SQ Logic when setting up the QP is one.
The following actions then take place:
The first message Send operation consists of a single request packet with a PSN of one.
The RQ Logic receives it, writes its data payload into memory, and increments its MSN to one. It does not send back an Ack packet yet, however, because it is coalescing its responses.
The second message Send operation consists of a single request packet with a PSN of two.
The RQ Logic receives it, writes its data payload into memory, and increments its MSN to two. It does not send back an Ack packet yet, however, because it is coalescing its responses.
The third request packet transmitted is the RDMA Read request packet requesting three packets worth of read data. The request packet has a PSN of three, and the SQ Logic expects to receive back three RDMA Read response packets with PSNs 3 through 5.
The RQ Logic receives the read request and verifies that it contains a valid VA, R_Key, and length. The ordering rules for the responder QP's RQ Logic state that, after verifying the header information supplied in an RDMA Read request packet, the RQ Logic is permitted to perform subsequently received Send or RDMA Write operations before it starts returning the requested read data (for more information, refer to “RDMA Read Relaxed Ordering Rules” on page 280).
While preparing to perform the read from its local memory, the RQ Logic receives another single-packet Send request. In this example, the RQ Logic decides to take advantage of the RDMA Read relaxed ordering rules. It writes the data payload to its local memory and advances its MSN to four (signifying that it has completed the SQ message transfer operation with an SSN of four).
Having read the first payload of read data from local memory, the RQ Logic then issues the first RDMA Read response packet back to the SQ Logic. The first response packet of a multi-packet RDMA read contains an Ack (i.e., an AETH). It Acks all message transfers posted before the RDMA Read WQE.
Upon receipt of the first RDMA Read response packet, the SQ Logic retires the SQ WQEs with SSNs one and two (they were posted before the read). Though the MSN in the first RDMA Read response packet is four, the SQ Logic cannot retire the Send with an SSN of four, even though the responder QP Logic is indicating that it has completed the Send on its end. If it did, this would result in out-of-order completion of SQ WQEs (which is forbidden).
The RQ Logic returns the middle RDMA Read response packet with read data.
The SQ Logic writes the returned read data into local memory.
The RQ Logic returns the final RDMA Read response packet. The final response packet of an RDMA Read contains an Ack (i.e., the AETH). This Ack confirms that the RQ Logic has completed the read operation. The SQ Logic therefore retires the Read from its SQ.
The RQ Logic now sends the Ack for the Send operation with an SSN of four.
Upon receipt of this Ack, the SQ Logic retires the Send WQE with an SSN of four from the SQ.