Additional Reference Information
Ack Coalescing Detail
Coalescing Can Correct for Missing Ack
A requester may not receive the request packet Ack for one of three reasons:
The request packet was lost in the fabric and was not received by the responder.
The responder received the request packet but failed to generate an Ack packet.
The responder generated an Ack packet, but it was lost in the fabric.
This type of error is often automatically corrected by Ack coalescing. There are some cases, however, where Ack coalescing does not automatically make up for missing Ack packet(s). As an example, a lost Ack packet containing a Nak isn't resolved by Ack coalescing because the responder cannot have more than one Nak outstanding at a given time.
Retry and PSN
When a requester retries a request, the PSN of the reissued request packet is probably > the highest PSN transmitted to date. To properly handle the expected response, the requester must maintain three checkpoints:
PSN of the oldest unAck'd request.
PSN of the most recently issued request packet.
PSN of the retried request.
Example scenario:
The requester issues a request packet.
The requester continues emitting additional request packets without awaiting the corresponding response packets.
The responder receives the request packet issued in step 1, but doesn't issue a response packet (because it is coalescing responses).
The responder receives subsequent request packets and continues coalescing their responses.
At some point, the responder issues a coalesced response covering a range of PSNs (including that of the request packet issued in step 1 as well as a number of subsequent request packets).
Before the coalesced response arrives at the requester, the requester's Transport Timer times out waiting for a response.
The requester retries the request issued in step 1 (using its original PSN).
When the responder receives the retried request, it treats it as a duplicate request. The rules dictate that the responder is obligated to return the PSN of its furthest point of advance.
The PSN in the response to the retried request is therefore > the PSN of the request.
Ack Packet Scheduling
Treatment of Request Packet with AckReq Set
Any request packet with the BTH:AckReq bit set to one requires the responder QP's RQ Logic to stop coalescing its responses and return a response:
- The responder continues to accept new request packets while it prepares to return the required response.
- As a result, the responder may issue a response packet with a PSN > the PSN of the request packet that had the AckReq bit set.
- In other words, the responder can coalesce multiple responses into just one response packet.
Example Case Where AckReq Is Useful
When the requester QP's SQ Logic sends the last request packet associated with the last WQE posted on the SQ, it is advisable for the requester to set the BTH:AckReq bit or to use some other mechanism that will force the responder to return a response. If the requester doesn't do so, there is a possibility that the responder could coalesce responses indefinitely.
“Other” Mechanisms to Force a Response
“Other” mechanisms that the requester can use to force the responder to return a response are:
- Always set the AckReq bit in the last (or only) request packet of every message.
- Follow a given request packet with a NOP request packet (BTH:Opcode = “NOP) with the AckReq bit set.
- If a response is not received for a specific request packet, retry the request for which a response is desired with the AckReq bit set.
Situations Where AckReq Has No Effect
There are some situations wherein the state of the AckReq bit in a request packet has no effect on the responder QP's RQ Logic behavior:
- A zero in the AckReq bit does not stop the responder from generating a response (even for Send or RDMA Write).
- RDMA Read and Atomic requests always require explicit responses, so the state of the AckReq has no effect.
Ack Scheduling for Send or RDMA Write
For Send or RDMA Write requests, the corresponding Ack may be scheduled before the data payload has actually been written into the local memory. The requester SQ Logic's receipt of the Ack indicates that the write data has successfully reached the “fault domain” of responder (this is the phrase used in the specification):
The data was received by the CA and the CA will write the data to its local memory, or
Software associated with the responder QP's RQ Logic will at least be informed in the event of a failure.
SQ Logic's Response Validation
Figure 17-24 on page 437 illustrates the checks that the SQ Logic performs when validating a response packet. This process is performed for both the RC and RD transport services. For a RC QP, the RD QP's check of the QPN is not performed by the SQ Logic. For the step labeled “Validate packet headers,” refer to Figure 17-25 on page 438.
Figure 17-24. SQ Logic's Validation of a Response
RQ Logic's Handling of Duplicate Request Packets
General
The responder QP's RQ Logic does not increment the MSN when it executes a duplicate request packet. For a detailed description of the MSN, refer to “SQ Logic Can Use MSN to Complete WQEs” on page 429.
Handling a Duplicate Request Packet
The actions taken by the RQ Logic when responding to a duplicate request packet are determined by the request type:
If it's a duplicate Send or RDMA Write:
- The RQ Logic schedules an Ack packet to be transmitted. The Ack packet's PSN is not the one contained in the original request, however. Rather, it must be the PSN of the request most recently completed by the RQ Logic.
- The RQ Logic does NOT perform the write to its local memory again.
- The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
- After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN.
- If the RQ Logic detects an error while generating a response to a duplicate Send or RDMA Write request, it silently drops the duplicate request (to avoid confusion with any possible outstanding Naks to new requests).
If it's a duplicate RDMA Read:
- The RQ Logic re-executes the read from its local memory.
- It sends back the requested read data in a series of one or more RDMA Read response packets.
- The first response packet uses the PSN of the original request packet, and each subsequent response packet increments the PSN by one.
- The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
- After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN.
- The PSN in the duplicate RDMA Read request packet must fall within the PSN range of the read response packets returned in response to the original request.
- The amount of read data requested must be a subset (or all) of the data requested in the original request.
- After returning the read data for the duplicate RDMA Read request packet, the RQ Logic may Ack any subsequently received duplicate Send or RDMA Write request packets using the PSN of the most recently completed request. This higher PSN acts as a coalesced Ack for all previously issued request packets.
If it's a duplicate Atomic operation:
- The RQ Logic must NOT re-execute the Atomic RMW operation.
- It returns an Atomic response packet using the saved results from the original execution of the Atomic operation.
- The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
- After issuing the response to the duplicate request, the RQ Logic resumes waiting for the request packet with the ePSN.
- If the duplicate Atomic request packet's PSN doesn't match that of any recently executed Atomic operation, the request packet is invalid and is silently dropped. This should never happen because the requester QP's SQ Logic should not send more Atomic requests than the responder QP's RQ Logic is capable of handling.
- If a local error within the CA prevents the RQ Logic from reproducing the original Atomic operation's read data, the RQ Logic must silently drop the duplicate Atomic request packet.
Rules for Handling Multiple Duplicate Request Packets
The responder QP's RQ Logic must respond to all duplicate requests in PSN order. The rules below are numbered for ease of reference.
If another duplicate request packet is received while awaiting the next expected request packet (with a PSN = ePSN), it is handled exactly like the earlier one.
It is highly possible that the RQ Logic may receive multiple duplicate request packets that may not be in sequential PSN order (relative to the first duplicate request packet received). The RQ Logic uses the same ordering rules when transmitting responses to duplicate requests as those used for expected new request packets. Duplicate RDMA Read or Atomic request packets must be responded to with the RDMA Read Response packet(s) or Atomic Response packet prior to sending the Acks for subsequently received duplicate Send or RDMA Write requests.
If, while reexecuting a duplicate RDMA Read request, the RQ Logic detects an error before returning the first response packet, the RQ Logic silently drops the duplicate request.
If, while reexecuting a duplicate RDMA Read request, the RQ Logic detects an error after returning one or more RDMA Read response packets, the RDMA Read response operation is aborted (i.e., no more response packets are returned).
If the starting PSN and length of a duplicate RDMA Read request is not a subset of the original request, then the request is invalid and the RQ Logic may silently drop it.
If, while responding to a new or duplicate RDMA Read request, another duplicate RDMA Read request is received with an earlier PSN, the RQ Logic ceases to respond to the original request and begins responding to the newly received duplicate request that has an earlier PSN.
If the RQ Logic's RDMA Read response is interrupted by another duplicate RDMA Read request, the RQ Logic is not required to resume sending the response packets for the first read. It is the requester QP's responsibility to resend any unAck'd requests.
RQ Logic Validation of Request Packet
Packet Header Validation
On receipt of an inbound request packet, the RQ Logic performs validation of the packet headers as shown in Figure 17-25 on page 438.
Opcode Sequence Rules
The rules below are numbered for ease of reference.
If the request packet just received is the first packet received immediately following connection establishment, the BTH:Opcode must indicate that it is either a “first” or “only” request packet.
If the previous valid request packet received was a “first,” the current request packet's Opcode must be either a “middle” or a “last.” The operation type must also match the operation type specified in the previous valid request packet (i.e., a Send or an RDMA operation). It is an error if the current request packet's Opcode is a “first” or “only,” since this implies that the last packet of the previous message was missed.
If the previous valid request packet received was a “middle,” then the current request packet's Opcode must be a “middle” or a “last.” The operation type must also match the operation type specified in the previous valid request packet (i.e., a Send or an RDMA operation). It is an error if the current request packet's Opcode is a “first” or an “only” packet, since this implies that the last packet of the previous message was missed.
If the previous valid request packet is a “last,” then the current request packet's Opcode must be a “first” or an “only.” It is an error if the current request packet's Opcode is a “middle” or a “last,” since this implies that the first packet of the current message was missed.
If the previous valid request packet received was an “only”, then the current request packet's Opcode must be a “first” or an “only.” It is an error if the current request packet's Opcode is a “middle” or a “last,” since this implies that the first request packet of the current message was missed.
Miscellaneous Request Packet Checks
The rules below are numbered for ease of reference.
If the responder QP's RQ Logic detects a request packet with an invalid length, it issues an Invalid Request Nak.
For an RDMA Read request, the responder QP's RQ Logic must verify that the transfer length specified in the RETH does not exceed 2GB. It issues an Invalid Request Nak if it does. It may optionally also perform this check on an RDMA Write request.
When executed, a validated request can still fail due to a local error within the responder QP (e.g., local virtual/physical memory address translation error).