CM MADs

Use of the Terms “Local” and “Remote”

This section provides a detailed description of the MADs sent between the CMs associated with two CA's to setup, modify, or tear down a communications channel between either:

  • A QP in one CA and a QP in the other CA (if RC or UC).

  • An EEC in one CA and an EEC in the other CA (if RD).

In the description of each field in a message, the words “local” and “remote” are frequently used (e.g., the local CM and the remote CM). The use of these two terms can be confusing. They are defined as follows:

  • Local. Refers to one of the following:

    - The CM sending the message.

    - Logic (e.g., a QP) within the CA associated with the CM that is sending the message.

  • Remote. Refers to one of the following:

    - The CM receiving the message.

    - Logic (e.g., a QP) within the CA associated with the CM receiving the message.

REQ (Request) MAD

When a software application local to a CA wishes to establish a communications channel with a service provided by a remote CA, it issues a request to the local CA's CM. In turn, the CM initiates a communications establishment message exchange with the remote CA's CM, starting with the sending of the REQ message (see Table 36-4 on page 1075) and Figure 36-1 on page 1075.

Figure 36-1. Communications Establishment Message Exchange


Table 36-4. REQ MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from REQ sender's point of view. REQ sender:

  • Must use same ID in all CM MADs associated with establishment and release of this communications channel.

  • Must not reuse this ID (for another connection) for the life of this connection or while any messages related to this connection could still be in the fabric. ID allows remote CM to determine whether message is duplicate of an old message or represents a new connection request.

Reserved32-bit field.
ServiceIDRC/UC/RD. 64-bit value. Sent by the REQ sender to indicate the type of service it wishes to establish a connection with.

CM receiving REQ uses the supplied ServiceID to determine if the desired service exists within its associated subsystem.

If the service exists, the receiving CM either:

  • Creates a QP of the desired type (as indicated by the Transport Service Type field in the REQ message) in its associated CA.

  • Locates a preexistent QP of the desired type in its associated CA.

This QP will act as the interface between the service and the REQ sender's QP.
Local CA GUIDRC/UC/RD. 64-bit value. EUI-64 GUID of the REQ sender's CA. This is provided in case both CM's simultaneously send REQ messages to each other. In that case, the two CM's must compare the GUIDs of their associated CA's to determine which will take the active role and which the passive role for the remainder of the communications establishment message exchange. For more information, refer to “Active Client to Active Client” on page 1112.
Local CM Q_KeyRC/UC/RD. 32-bit value. This is the Q_Key of the GSI (QP1) the CM issuing the REQ message is using to send and receive CM messages. The CM receiving the REQ message must use this Q_Key in the DETH:Q_Key field of all message MADs it sends to the CM that issued the REQ message.
Local Q_KeyRD. 32-bit value. The CM sending the REQ MAD must supply the other CA's CM with the QPN (supplied in the REQ message's Local QPN field) and Q_Key (supplied in the REQ message's Local Q_Key field) of the local RD QP that will serve as its end of the communications channel being established. The receiving CM supplies this QPN and Q_Key to the software associated with the requested service on its end. That service-related software then uses that Q_Key and QPN in any WRs that it posts to the SQ of its QP.
Local QPNRC/UC/RD. 24-bit value. QPN of QP at REQ sender's end of connection.

  • If the connection being established is RC or UC, the CM receiving the REQ message creates a QP of the desired type (as indicated in the REQ message's Transport Service Type field) in its local CA and stores this QPN in the QP Context of the newly created QP.

  • If the connection being established is RD, the CM receiving the REQ message supplies this QPN and the Q_Key received in the Local Q_Key field to the software associated with the requested service on its end. That service-related software then uses that Q_Key and QPN in any WRs that it posts to the SQ of its QP.

Offered Responder ResourcesRC/UC/RD. 8-bit value. Depth of the special queue that the REQ sender's newly created CA QP uses to hold and process incoming RDMA Read/Atomic requests. The CM sending the REQ message obtains this value using Query HCA verb (if its associated CA is an HCA).

If = 0, this indicates that the REQ sender's CA QP doesn't support inbound RDMA Read or Atomic requests.

If the queue depth indicated by this value proves insufficient for the CM receiving the REQ message, it responds to the REQ with a REJ. Conversely, if the REQ recipient's CA queue depth (indicated in the REP message) is insufficient for the REQ sender, the REQ sender responds to the REP with a REJ. Receipt of a REJ message discontinues the connection establishment process.
Local EECNRD. 24-bit value. This is the EECN that identifies the EEC on the REQ sender's end of the RDC. When a new RDC is being created (i.e., RDC Exists bit in this REQ message = 0), the receiving CM creates a new EEC in its CA and programs this EECN into the context of the newly created EEC.
Offered Initiator DepthRC/UC/RD. 8-bit value. Depth of the special queue that the REQ sender's CA QP uses to issue outgoing RDMA Read and Atomic requests to the receiving CA's QP. The CM sending the REQ message obtains this value using the Query HCA verb (if its associated CA is an HCA). The depth chosen must not exceed the depth of the corresponding queue on the receiving CA's QP or EEC. Note that the depth of the receiving CA's QP or EEC special receive queue is returned in the REP message.
Remote EECNRD. 24-bit value. EECN of EEC on REQ recipient's end of RDC:

  • If a new RDC is just being created, = 0. In this case, REQ:RDC Exists must be set to 0. The receiving CA's CM then creates a new EEC, assigns it an EECN and returns the EECN of the newly created remote EECN in the REP message.

  • If REQ:RDC Exists = 1, this indicates to the receiving CM that it is not to create a new EEC. Rather, only a new RD QP is to be created by the receiving CM on its end, and the QP's Context must be programmed with the EECN supplied in this REQ field. In other words, this EECN identifies the EEC through which the newly created RD QP will send and receive messages. The receiving CM must also program the newly created RD QP's context with the same RDD that it assigned to this EEC when it was created earlier in time.

Remote CM Response TimeoutRC/UC/RD. 5-bit unsigned value. This value tells the CM receiving the REQ message how quickly it must generate a response whenever it receives a CM message from the REQ sender. The response time is defined as

4.096us X 2Remote CM Response Timeout.

If the REQ sender cannot respond to a subsequent message received from the other CM within this time period, it must send an MRA (Message Receipt Acknowledgement); see “MRA (Message Receipt Acknowledgment) MAD” on page 1096) to prevent a timeout on the message sender's part.
Transport Service TypeRC/UC/RD. 2-bit value. Indicates type of QP (and possibly EEC) to be created on CA receiving REQ:

  • 0: RC QP.

  • 1: UC QP.

  • 2: RD QP and, possibly, an EEC.

  • 3: Reserved

End-to-End Flow ControlRD. 1-bit value. Indicates whether REQ sender's newly created QP RQ Logic:

  • 1 = supplies End-to-End Flow Control credits, or

  • 0 = always advertises infinite credits.

It should be noted that Table 85 in Volume 1 of the specification indicates this applies to both RC and RD, but it only applies to RC (RD does not support End-to-End Flow Control).

For more information on End-to-End Flow Control, refer to “End-to-End Flow Control” on page 417.
Starting PSNRC/UC/RD. 24-bit value. This is the start PSN the REQ sender has assigned to the Send Logic of its newly created QP or EEC. The receiving CM creates a QP or EEC in its CA and assigns this value as the ePSN of the newly created QP's or EEC's Receive Logic. The value should be chosen to minimize the chance that a packet from a previous connection could fall within the valid PSN window. For more information, refer to “Select a Start PSN Value That Precludes Stale Packets” on page 207.
Local CM Response TimeoutRC/UC/RD. 5-bit unsigned value. This value tells the CM receiving the REQ message how long it should wait for a response whenever it sends a subsequent communications message (e.g., a REP message) to the CM that sent the REQ message. The response time is defined as

4.096us X 2Remote CM Response Timeout

and includes the round-trip packet flight time plus the message processing time on the other end. If the REQ sender cannot respond to a subsequent message received from the other CM within this time period, it must send an MRA to prevent a timeout on the message sender's part.
Retry CountRC/UC/RD. 3-bit value. Indicates the maximum number of times the QP or EEC to be created in the receiving CM's CA should retry a request packet for which:

  • It doesn't receive a response.

  • It receives a PSN Sequence Error Nak.

  • It detects a missing RDMA Read response or Atomic response packet.

After the maximum retry count has been exhausted, the QP's or EEC's Send Logic will either:

  • Retire the SQ WQE and create an error CQE (if APM is disabled) or

  • If APM is enabled, it will migrate to the new path, reinitialize its Retry Count, and try again.

P_KeyRC/UC/RD. 16-bit value. P_Key to be used for channel being established. When the receiving CM creates the new QP or EEC, it must assign a Primary P_Key Index value that selects this P_Key in the P_KeyTable attribute of the port that the QP or EEC uses to send and receive messages.
Path Packet Payload MTURC/UC/RD. 4-bit value. Specifies the maximum packet payload size (i.e., PMTU), in bytes, for the channel being established. Applies to both primary and alternate paths.

  • 1: 256

  • 2: 512

  • 3: 1024

  • 4: 2048

  • 5: 4096

  • 0, 6–15: reserved

When the receiving CM creates the new QP or EEC, it must program this PMTU into the context of the QP or EEC.
RDC ExistsRD. 1-bit value.

  • 0 = new RDC is being created. The receiving CM must create a new EEC and RD QP in its CA and return the EECN, QPN, and the QP's Q_Key in the REP message. The same RDD is to be assigned to the newly created EEC and RD QP.

  • 1 = RDC already exists. This REQ message contains the EECN (in REQ:Remote EECN field) of an EEC created earlier (when the RDC was created). The CM sending the REQ wishes the receiving CM to create a new RD QP and program its context with:

    - The same RDD as the EEC specified in the REQ:Remote EECN field.

    - The EECN specified in the REQ:Remote EECN field. This is the EEC that the newly created RD QP will use to send and receive messages.

RNR Retry CountRC/UC/RD. 3-bit value. Indicates the total number of times the newly created QP or EEC in the receiving CM's CA is to retry request packets for which it receives RNR Naks. After this count is exhausted, the QP's or EEC's Send Logic will post an error CQE on the Send Logic's CQ.
Max CM RetriesRC/UC/RD. 4-bit value. Maximum times either CA's CM can resend a REQ, REP, or DREQ message. After resending the maximum times without a response, the sending party terminates the connection establishment sequence by sending a REJ message indicating it timed out.
Reserved4-bit field.
Primary Local Port LIDRC/UC/RD. 16-bit value. The LID of a port on the CA sending the REQ that the new QP or EEC on the sender's end will use to send and receive messages. The receiving CM programs this LID address into the context of the QP or EEC about-to-be-created.
Primary Remote Port LIDRC/UC/RD. 16-bit value. The CM sending the REQ message indicates a LID address of the port on the receiving CA. The receiving CM uses this value to identify which port number on its CA is to be programmed into the context of the about-to-be-created QP or EEC. The QP or EEC will use that port to send and receive messages.

In addition, the receiving CM programs the QP or EEC Context with the Source Path Bits value (refer to “Source Port's LID Address” on page 45) that will select the specified port LID address to be inserted in the LRH:SLID field of all outbound packets generated by the QP or EEC.

The receiving CM may send a REJ message to reject this port, and may optionally suggest an acceptable port [see “REJ (Reject) MAD” on page 1098]. The CM sending the REQ message is responsible for ensuring that the Primary Remote Port LID and the Primary Remote Port GID field of REQ refer to same port.
Primary Local Port GIDRC/UC/RD. 128-bit value. GID address of the port that will be used by the newly created QP or EEC in the REQ sender's CA to send and receive messages. The receiving CM programs this address into the context of the QP or EEC about-to-be-created as the DGID address. If both CAs reside in the same subnet (as indicated by the REQ:Primary Subnet Local field), this field is 0.
Primary Remote Port GIDRC/UC/RD. 128-bit value. If the two CAs reside in the same subnet (as indicated by the REQ:Primary Subnet Local field), this value is 0. If they are in different subnets, this is the GID address of the port on the receiving CM's CA that the about-to-be-created QP or EEC will use to send and receive messages. The receiving CM uses this value to identify the Source GID Index value that will be programmed into the context of the about-to-be-created QP or EEC. The index value selects the specified GID address to be inserted in the GRH:SGID field of all outbound packets generated by the QP or EEC. Also see the description of the Primary Remote Port LID field in this table.
Primary Flow LabelRC/UC/RD. 20-bit value. Only necessary if the two CAs reside in different subnets (as indicated by the REQ:Primary Subnet Local field). If they do, this value is programmed into the context of the about-to-be-created QP or EEC and is inserted in the GRH:FlowLabel field of all outbound packets generated by the QP or EEC.
Reserved4-bit field.
Primary Packet Rate (IPD)RC/UC/RD. 8-bit value. Tells the receiving CM what IPD value to program into the context of the about-to-be-created QP or EEC. It defines the maximum rate at which the QP or EEC may transmit packets. See “Static Rate Control” on page 589 for more information.
Primary TClassRC/UC/RD. 8-bit value. Only necessary if the two CAs reside in different subnets (as indicated by the REQ:Primary Subnet Local field). If they do, this value is programmed into the context of the about-to-be-created QP or EEC and is inserted in the GRH:TClass field of all outbound packets generated by the QP or EEC.
Primary Hop LimitRC/UC/RD. 8-bit value. Only necessary if the two CAs reside in different subnets (as indicated by the REQ:Primary Subnet Local field). If they do, this value is programmed into the context of the about-to-be-created QP or EEC and is inserted in the GRH:HopLmt field of all outbound packets generated by the QP or EEC.
Primary SLRC/UC/RD. 4-bit value. The receiving CM programs this value into the context of the about-to-be-created QP or EEC. It defines the value placed in the LRH:SL field of all outbound packets generated by the QP or EEC.
Primary Subnet LocalRC/UC/RD. 1-bit value. Tells the receiving CM whether the two CAs reside in the same or different subnets:

  • 0: CAs reside in different subnets.

  • 1: CAs reside in same subnet.

If they reside in different subnets, then the REQ sender also supplied the following parameters to the receiving CM:

  • Primary Local Port GID.

  • Primary Remote Port GID.

  • Primary Flow Label.

  • Primary TClass.

  • Primary Hop Limit.

Reserved3-bit field.
Primary Local Ack TimeoutRC/UC/RD. 5-bit unsigned value. Transport (Ack) timeout for use by the receiving CA's QP or EEC

= 4.096us X 2Local ACK Timeout.

Calculated by the REQ sender based on (2 X Packet LifeTime + Local CA's response generation delay). The receiving CM may program this value into the context of the about-to-be-created QP or EEC as the Transport Timer timeout. The receiving CM is not required to use this value, but is strongly encouraged to do so. PacketLifeTime represents the maximum expected flight time for a packet to traverse the path between the two CAs. PacketLifeTime is contained in the SA's PathRecord.

  • If the value used is too small, the number of packet transmission timeouts reported by the CA may increase, which may increase the amount of work required to successfully send a packet.

  • If the value used is too large, the time it takes to notice that a packet was not successfully transmitted (e.g., due to a CRC error on the wire) is increased, which may increase the amount of time to recover from or to report such errors.

Reserved3-bit field.
Alternate Path InfoOptional (because APM is optional). State of supplied Alternate Local Port LID field indicates whether or not an alternate path is supplied.

  • A 0 indicates no alternate path has been supplied, therefore APM activation is not being requested by the REQ sender.

  • Non-zero. Alternate path supplied, therefore APM activation requested by REQ sender.

If valid, alternate path information consists of:

  • Alternate Local Port LID (16-bit field).

  • Alternate Remote Port LID (16-bit field).

  • Alternate Local Port GID (128-bit field).

  • Alternate Remote Port GID (128-bit field).

  • Alternate Flow Label (20-bit field).

  • Reserved (4-bit field).

  • Alternate Packet Rate (IPD) (8-bit field).

  • Alternate Traffic Class (8-bit field).

  • Alternate Hop Limit (8-bit field).

  • Alternate SL (4-bit field).

  • Alternate Subnet Local (1-bit field).

  • Reserved (3-bit field).

  • Alternate Local ACK Timeout (5-bit field).

  • Reserved (3-bit field).

If the failover information is accepted, each CM causes their respective, newly created QP or EEC to be placed in the ReArm Migration state so that they are ready to perform a path migration. Receiving CM may accept connection request, but reject supplied alternate path information. It does this by sending REP message with one of the following values in the 2-bit Failover Accepted bit field:

  • 0: Failover accepted.

  • 1: Failover port rejected because failover not supported. Alternate Path was not checked.

  • 2: Failover supported and Alternate Path valid, but failover port rejected for some other reason.

If the failover information is rejected, each CM causes its respective QP or EEC to be placed in the Migrated state upon transition to the RTR state.
Reserved3-bit field.
PrivateDataRC/UC/RD. 736-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a request based on the private data. The format and meaning of the PrivateData field is specific to the ServiceID and the message type, and is not specified within the CM specification.

REP (Reply) MAD

Refer to Figure 36-1 on page 1075. Upon receipt of a REQ MAD [refer to “REQ (Request) MAD” on page 1074] from a remote CA's CM, the local CA's CM determines whether the requested service is available on the this CA. If it isn't, the CM sends a REJ MAD [see “REJ (Reject) MAD” on page 1098] back to the other CM stating the reason for the rejection.

If the requested service is supported by the local CA, the local CM creates an EEC and/or a QP to handle its end of the communications channel, programs that QP's or EEC's context with information supplied in the REQ MAD, transitions the newly created QP or EEC to the RTR state, and then sends a REP MAD (see Table 36-5 on page 1089) back to the requester with information about the newly created QP of EEC and QP.

Table 36-5. REP MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from REP sender's point of view. REP sender:

  • Must use same ID in all CM MADs associated with establishment and release of this communications channel.

  • Must not reuse this ID (for another connection) for life of this connection or while any messages related to this connection could still be in the fabric. ID allows the remote CM to determine whether the message is a duplicate of an old message or represents a new connection request.

Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the REP message. This would be the same Communication ID received in the REQ message. Values in Local and Remote Communication ID fields in CM MADs are exchanged in requests and replies. The pair of IDs is used to reference a specific connection during establishment, failover management, and release. CM messages with invalid IDs are not processed and are rejected.
Local Q_KeyRD. 32-bit value. In response to the receipt of the REQ message, the CM creates a RD QP and assigns it a Q_Key. This field contains that Q_Key. Upon receipt of the REP message, the other CM passes this Q_KEY and the QPN returned in the REP:Local QPN field to the applications software that requested the establishment of a communications channel with a service in the other CA. Whenever the applications software then wishes to pass a message to the remote RD QP associated with the service, it supplies the following information in a WR posted to the local RD QP's SQ:

  • This QPN and its Q_Key.

  • The EECN of the EEC that the local RD QP is to use to send the message (returned in the REP:Local EECN field).

Local QPNRC/UC/RD. 24-bit value. This is the QPN of the QP created by the CM upon receipt of the REQ message.

  • If RC/UC, upon receipt of the REP message the other CM programs the QP context of the QP in its CA with the QPN of this CA's newly created QP.

  • If RD, upon receipt of the REP message the other CM passes this QPN to the applications software that requested the establishment of a communications channel with a service in the other CA. See the description of the Local Q_Key field in this table.

Reserved8-bit field.
Local EECNRD. 24-bit value. This field only has meaning if the CM sending the REP message had created a new EEC in its CA at the behest of the REQ message sender (i.e., the REQ:RDC Exists field = 0). It then creates the new EEC and returns its EECN in this field of the REP message. The other CM then programs this EECN into the context of the EEC in its CA that will act as the other end of the newly created RDC.
Reserved8-bit field.
Starting PSNRC/UC/RD. 24-bit value. This is the start PSN that the CM sending the REP message has assigned to its newly created QP's or EEC's Send Logic. Upon receipt of the REP message, the other CM programs this value into the ePSN in the context of its newly created QP or EEC.
Reserved8-bit field.
Responder ResourcesRC/UC/RD. 8-bit value. Depth of the special queue that the REP sender's newly created CA QP or EEC uses to hold and process incoming RDMA Read/Atomic requests. The CM sending the REP message obtains this value using Query HCA verb (if its associated CA is an HCA).

If = 0, this indicates that the REP sender's newly created CA QP doesn't support inbound RDMA Read or Atomic requests.

If the queue depth indicated by this value proves insufficient for the CM receiving the REP message, it responds to the REP with a REJ. Receipt of a REJ message discontinues the connection establishment process.
Initiator DepthRC/UC/RD. 8-bit value. Depth of the special queue that the REP sender's newly created CA QP or EEC uses to issue outgoing RDMA Read and Atomic requests to the receiving CA's QP. The CM sending the REP message obtains this value using the Query HCA verb (if its associated CA is an HCA). The depth chosen must not exceed the depth of the corresponding queue on the receiving CA's QP or EEC. Note that the depth of the receiving CA's QP or EEC special receive queue was supplied in the REQ message.
Target ACK DelayRC/UC/RD. 5-bit unsigned value.

4.096us X 2Target ACK Delay

It represents the maximum expected time interval between REP sender's reception of a message and its transmission of the associated Ack or Nak. REQ sender uses this value along with its best estimate of network propagation delay (i.e., packet flight time) to determine how long to wait before timing out a packet transmission to REP sender.
Failover AcceptedRC/UC/RD. 2-bit value. Indicates whether REQ recipient accepted or rejected the alternate path information contained in the REQ message. By sending the REP message, the REQ recipient accepts the connection request, but it may still reject the proposed failover port.

  • If failover accepted, each CM causes its newly created QP or EEC to be placed in the ReArm migration state.

  • If failover rejected, each CM causes its newly created QP or EEC to be placed in the Migrated state upon transition to the RTR state.

Possible Failover Accepted values:

  • 0 = Failover accepted.

  • 1 = Failover port rejected because failover not supported. Alternate path parameters not checked.

  • 2: Failover supported and all alternate path parameters valid, but failover port rejected for some other reason.

End-To-End Flow ControlRC. 1-bit value. Indicates whether REP sender's newly created QP RQ Logic:

  • 1 = Supplies End-to-End Flow Control credits, or

  • 0 = Always advertises infinite credits.

It should be noted that Table 89 in Volume 1 of the specification indicates this applies to both RC and RD, but it only applies to RC (RD does not support End-to-End Flow Control).

For more information on End-to-End Flow Control, refer to “End-to-End Flow Control” on page 417.
RNR Retry CountRC/UC/RD. 3-bit value. Indicates the total number of times the newly created QP or EEC in the receiving CM's CA is to retry request packets for which it receives RNR Naks. After this count is exhausted, the QP's or EEC's Send Logic will post an error CQE on the Send Logic's CQ.
Reserved8-bit field.
PrivateDataRC/UC/RD. 1632-bit field. Data opaque to the CM protocol, passed from the REP sender to the REP recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

RTU (Ready to Use) MAD

Refer to Figure 36-1 on page 1075. When the CM requesting the establishment of a communications channel with a remote CA's CM receives a REP MAD from the remote CM, it uses the information in the REP MAD to finish programming the context of the newly created local QP or EEC and transitions the local QP or EEC to the RTS state. At this point, the newly created QP or EEC in the remote CA is in the RTR state.

The remote QP or EEC can be transitioned to the RTS state in one of two ways:

  • The local CM can send an RTU MAD to the remote CM. Upon receipt of the RTU MAD, the remote CA's CM transitions the newly created QP or EEC to the RTS, fully operational state.

  • Alternatively, the CM can forego sending the RTU MAD. Instead, the local applications software that requested the establishment of the communications channel can post a WR to the SQ of the newly created local QP, causing the QP's SQ Logic to transmit a message to the remote QP's RQ Logic. Upon receipt of the first request packet, the remote QP, or QP and EEC automatically transition from the RTR to the RTS state.

Table 36-6. RTU MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the RTU message. This CM must use the same ID in all CM MADs associated with the establishment and release of this communications channel. This CM must not reuse this ID (for another connection) for the life of this connection or while any messages related to this connection could still be in the fabric.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the RTU. This is the same as the Local Communication ID received in the earlier REP message. Values in the Local and Remote Communication ID fields in CM MADs are exchanged in requests and replies. The pair of IDs is used to reference a specific connection during establishment, failover management, and release. CM messages with invalid IDs are not processed and are rejected.
PrivateDataRC/UC/RD. 1792-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

DREQ (Disconnect Request) MAD

The DREQ MAD (see Table 36-7 on page 1095) can be sent by the CM at either end of the connection to tear down (i.e., destroy) the connection. Upon receipt of the DREQ MAD, the other CM validates the Communications IDs associated with the connection, destroys the local QP or EEC, and responds by sending a DREP MAD—see “DREP (Disconnect Reply) MAD” on page 1095.

Table 36-7. DREQ MAD Content
FieldDescription
Local Communications IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the DREQ message.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the DREQ message.
Remote QPN or Remote EECNThis is the QPN or EECN of the remote CA's QP or EEC. Provides additional validation that the Local and Remote Communication ID pair reference is correct.
Reserved8-bit field.
PrivateDataData opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

DREP (Disconnect Reply) MAD

Refer to Table 36-8 on this page and to “DREQ (Disconnect Request) MAD” on page 1094.

Table 36-8. DREP MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the DREP message.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the DREP message.
PrivateDataData opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

MRA (Message Receipt Acknowledgment) MAD

When a CM receives a MAD message for which the appropriate response is a REP, RTU, APR, or REJ MAD, it may be some time before it is ready to send that response. The CM that had sent the MAD had triggered its CM Response Timeout while it awaits a response MAD. If the CM that had received the MAD takes too much time before sending a response MAD, the other CM's CM Response Timeout will expire, causing it to retry the transmit of its original MAD. To prevent this, the CM should send an MRA MAD as a placeholder for its eventual response. When the MRA is received by the CM awaiting the response, the MRA's ServiceTimeout field (see Table 36-9 on this page) tells the CM the maximum time required before the MRA sender will be ready to send its real response.

Table 36-9. MRA MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the DREP message.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the MRA message.
Message MRAedRC/UC/RD. 2-bit value. Identifies the message type that the MRA issuer needs more time to process before it send its real response:

  • 00b - REQ.

  • 01b - REP.

  • 10b - LAP.

  • 11b - Reserved.

Reserved6-bit field.
ServiceTimeoutRC/UC/RD. 5-bit unsigned value. The maximum time required before the MRA sender will be ready to send a REP, RTU, APR, or REJ message (as appropriate). This value is expressed as

4.096us X 2Service Timeout

from the time the MRA is posted to the CM's GSI SQ. The recipient of the MRA must wait the specified amount of time plus the PacketLifetime (i.e., the packet flight time) after receiving this message before timing out.
Reserved3-bit field.
PrivateDataRC/UC/RD. 1776-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

REJ (Reject) MAD

Upon receipt of a communications establishment MAD, a CM may choose to issue a REJ response MAD (see Table 36-10 on this page) indicating that it intends to break off the communications establishment message exchange. The reason for the rejection is indicated in the REJ MAD's Reason field. The possible reasons for rejection are listed in Table 36-11 on page 1099.

Table 36-10. REJ MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the REJ message.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the REJ message.
Message rejectedRC/UC/RD. 2-bit value indicating the type of message type being rejected:

  • 00b - REQ

  • 01b - REP

  • 10b - Unknown/No message.

  • 11b - Reserved.

Reserved6-bit field.
Reject Info LengthRC/UC/RD. 7-bit field. If ≠ 0,this is the length in bytes of the Additional Reject Information (ARI) field (defined in this table).
Reserved1-bit field.
ReasonRC/UC/RD. 16-bit field. Error code indicating the reason for the REJ sender's termination of the communication establishment process. See Table 36-11 on page 1099.
Additional Reject Information (ARI)RC/UC/RD. 576-bit field. Contents defined by the error code delivered in the Reason field. See Table 36-11 on this page.
PrivateDataRC/UC/RD. 1184-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

Table 36-11. Reject Message's Reason Field Encoding
CodeReasonDescriptionMeaning of ARI Field
1No QP availableThe REQ message required the recipient to allocate a QP and none were available.NA
2No EEC availableThe REQ message required the recipient to allocate an EEC and none were available.NA
3No resources availableThe REQ message required the recipient to allocate resources other than a QP or an EEC and none were available.NA
4TimeoutThe CM protocol timed out waiting for a message.NA
5Unsupported requestReceiving CM does not support this request.NA
6Invalid Communication IDThe recipient received a CM message in which the Local Communication ID, Remote Communication ID, or both were invalid.NA
7Invalid Communication InstanceThe communications tuple (i.e., the combination of the Local Communication ID, Remote Communication ID, and QPN or EECN) does not refer to any valid connection.NA
8Invalid ServiceIDThe recipient of the REQ message does not recognize or does not support the service associated with the specified ServiceID.NA
9Invalid Transport Service TypeThe recipient of the REQ message did not recognize the requested Transport Service Type.NA
10Stale connectionThe recipient of the REQ determined that it already had a connection with the “Local QPN” or “Local EECN” specified in the REQ. Upon receiving a REJ for this reason, the REJ recipient places the QP or EEC to be placed into the TimeWait state. The TimeWait timer is set to twice the PacketLifeTime value (i.e., the source-to-destinaation round-trip flight time) plus the remote's Ack Delay. The CM is responsible for placing a QP or EEC in the TimeWait state, for maintaining it in that state for a period not less than the TimeWait period, and for removing it afterward. Receipt of a DREQ while in the TimeWait state does not affect the TimeWait timer.NA
11RDC does not existUpon receipt of a REQ message requesting the creation of a new RD QP using a preexistent EEC, the CM that received the REQ determined that the specified EEC does not exit.NA
12Primary Remote Port GID rejectedThe recipient of the REQ message could not (or would not) accept the Primary Remote Port GID for its end of the connection. See the description of Primary Remote Port GID in Table 36-4 on page 1075.GID of acceptable port.
13Primary Remote Port LID rejectedThe recipient of the REQ message could not (or would not) accept the Primary Remote Port LID for its end of the connection. See the description of Primary Remote Port LID in Table 36-4 on page 1075.LID of acceptable port
14Invalid Primary SLThe recipient of the REQ message does not support the requested Primary SL.Acceptable SL
15Invalid Primary Traffic ClassThe recipient of the REQ message does not support the requested Primary Traffic Class.Acceptable Traffic Class
16Invalid Primary Hop LimitThe recipient of the REQ message could not (or would not) accept the Primary Hop Limit.Acceptable Hop Limit
17Invalid Primary Packet RateThe recipient of the REQ message could not adjust its transmitter to send as slowly as would be required to comply with the requested Primary Inter-Packet Delay (IPD).Minimum acceptable IPD
18Alternate Remote Port GID rejectedThe recipient of the REQ message could not (or would not) accept the Alternate Remote Port GID for its end of the connection.GID of acceptable port
19Alternate Remote Port LID rejectedThe recipient of the REQ message could not (or would not) accept the Alternate Remote Port LID for its end of the connection.LID of acceptable port
20Invalid Alternate SLThe recipient of the REQ message does not support the requested Alternate SL.Acceptable SL
21Invalid Alternate Traffic ClassThe recipient of the REQ message does not support the requested Alternate Traffic Class.Acceptable Traffic Class
22Invalid Alternate Hop LimitThe recipient of the REQ message could not (or would not) accept the Alternate Hop Limit.Acceptable Hop Limit
23Invalid Alternate Packet RateThe recipient of the REQ message could not adjust its transmitter to send as slowly as would be required to comply with the requested Alternate Inter-Packet Delay (IPD).Minimum acceptable IPD
24Port and CM RedirectionThe recipient of the REQ message supports the requested ServiceID, but at the endpoint specified by the ARI. Any further CM messages should be sent to that CA port as well.ARI contains ClassPortInfo data structure (see Table 28-7 on page 794) describing where to send subsequent CM messages, and also describing the GID of the port to propose in the new REQ
25Port RedirectionThe recipient of the REQ message supports the requested ServiceID, but at the port specified by the ARI. Further CM messages must be sent to the port to which the original REQ was sent.GID of port to propose in new REQ
26Invalid Path MTUThe recipient of the REQ message cannot support the PMTU specified.Maximum acceptable PMTU
27Insufficient Responder ResourcesThe size of the special queue (Responder Resources) in the REP message for RDMA Read and Atomic requests was insufficient.NA
28Consumer RejectThe consumer (i.e., software) decided to reject the communication establishment attempt for reasons other than those listed in the other entries of this table. Typically, this happens based on rejection of information in the PrivateData field of a message.Defined by the consumer software
29RNR Retry Count RejectThe recipient of the message rejects the RNR NAK Retry Count value.NA

LAP (Load Alternate Path) MAD

A CM would need to send new alternate path information to the remote CM under two circumstances:

  • APM was not initially enabled when the communications channel was created, but software now wishes to program the remote QP or EEC with alternate path information and transition it to the ReArm state. Software can program the local QP or EEC with alternate path information directly using the Modify QP or Modify EEC verb (if the local CA is an HCA).

  • APM was previously enabled on both ends of the connection, and the path was subsequently migrated to the alternate path (either due a hardware problem or because software triggered the migration). When software is informed that the connection between the two QPs or EECs has migrated to the alternate path (via a call to the Asynchonous Event Hander), it may wish to program new alternate path information into both the local and remote QPs or EECs.

The CM sends a LAP MAD (see Table 36-12 on this page) to the remote CM to load new alternate path information into the context of the remote QP or EEC. Upon receipt of the LAP MAD, the remote CM verifies that new alternate path information is different than the alternate path information currently contained in the remote QP's or EEC's context.

Assuming that it is different, the CM verifies that all fields in the LAP are good values. It then loads the new alternate path information into the context of the remote QP or EEC and changes the migration state of the QP or EEC from the Migrated state to the ReArm state. The remote CM then sends an APR MAD [see “APR (Alternate Path Response) MAD” on page 1107] to the CM that sent the LAP MAD. The APR MAD tells the LAP sender the status of the operation.

For a detailed description of APM, refer to “Automatic Path Migration” on page 575.

Table 36-12. LAP MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the LAP message.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the LAP message.
Local CM Q_KEYRC/UC/RD. 32-bit value. Q_Key of the GSI (QP1) used by the CM sending the LAP message.
Remote QPN or Remote EECNThis is the QPN or EECN of the QP or EEC in the CA receiving the LAP message. Provides additional validation that the Local and Remote Communication ID pair reference is correct.
Remote CM Response TimeoutRC/UC/RD. 5-bit unsigned value. Time, expressed as:

4.096us X 2Remote CM Response Timeout

within which remote CM (the LAP recipient) must transmit a response to the LAP message. If the remote CM cannot respond to the LAP (with an APR message) within this time period, it must send an MRA to prevent a timeout and a retry on the LAP sender's part.
Reserved3-bit field.
Reserved32-bit field.
Alternate Path InfoNew alternate path information to be programmed into the context of the remote QP or EEC. It consists of:

  • Alternate Local Port LID

  • Alternate Remote Port LID

  • Alternate Local Port GID

  • Alternate Remote Port GID

  • Alternate Flow Label

  • Reserved. 4-bit field.

  • Alternate Traffic Class

  • Alternate Hop Limit

  • Alternate Inter-Packet Delay (IPD)

  • Alternate SL

  • Alternate Subnet Local

  • Reserved. 3-bit field.

  • Alternate Local ACK Timeout

Reserved3-bit field.
PrivateDataRC/UC/RD. 1,344-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

APR (Alternate Path Response) MAD

Refer to Table 36-13 on this page and to “Loading a New Path or a Tertiary Path” on page 586.

Table 36-13. APR MAD Content
FieldDescription
Local Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM sending the APR message.
Remote Communication IDRC/UC/RD. 32-bit handle uniquely identifying this connection from the point of view of the CM receiving the APR message.
Additional Information576-bit field.
AP StatusAlternate Path Status. 4-bit value. Valid values are:

  • 0 = Alternate path information accepted and loaded into the QP's or EEC's context.

  • 1 = Invalid Communication Instance tuple (i.e., the combination of the Local Communication ID, Remote Communication ID, and QPN or EECN).

  • 2 = Alternate paths (i.e., APM) not supported. Alternate path parameters were not checked.

  • 3 = Alternate paths (i.e., APM) supported and all Alternate Path parameters valid, but failover port rejected for some other reason.

  • 4 = Alternate path information rejected; redirect.

  • 5 = Proposed alternate path matches current primary path.

If AP Status = 4, the message's Additional Information field contains a ClassPortInfo data structure (see Table 28-7 on page 794). The LAP sender may send a new LAP proposing the alternate path indicated in the ClassPortInfo data structure.
Reserved4-bit field.
PrivateDataRC/UC/RD. 1208-bit field. Data opaque to the CM protocol, passed from the sender to the recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

SIDR_REQ (ServiceID Resolution Request) MAD

Refer to Table 36-14 on this page and “UD Connection Issues” on page 202.

Table 36-14. SIDR_REQ MAD Content
FieldDescription
RequestIDUD. 32-bit value identifying request. Recipient of request message returns this value unchanged in SIDR_REP. If the SIDR_REQ message is resent (due to a timeout awaiting a response), the sender must use same RequestID.
Reserved32-bit field.
ServiceIDUD. 64-bit value. ID specifying service being requested. These include, but are not limited to, service numbers defined for typical TCP services.
PrivateDataUD. 1728-bit field. Data opaque to the CM protocol, passed from the SIDR_REQ sender to the SIDR_REQ recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

SIDR_REP (ServiceID Resolution Reply) MAD

Refer to Table 36-15 on this page and “UD Connection Issues” on page 202.

Table 36-15. SIDR_REP MAD Content
FieldDescription
RequestIDUD. 32-bit value identifying the SIDR_REQ message that this a reply to.
QPNUD. 24-bit value. The CM sending the message places the QPN and Q_Key of the QP on which the requested service (identified by the ServiceID) is supported. Valid only if so indicated by this MAD's Status field. Also see the description of the Q_Key field.
StatusUD. 8-bit value. Indicates whether the QPN field is valid. If the QPN is invalid, the Status field indicates the reason that the QPN has not been provided.

  • 0 = QPN and Q_Key in the message are valid.

  • 1 = ServiceID not supported.

  • 2 = Rejected by Service Provider.

  • 3 = No QP available.

  • 4 = Request must be redirected to the port specified by the ClassPortInfo field in this message (see the later entry in this table). The QPN and Q_Key in the message are not valid.

  • 5–255 = Reserved.

ServiceIDUD. 64-bit ID value that was contained in request.
Q_KeySee the description of the QPN field.
ClassPortInfoUD. 576-bit field. ClassPortinfo data structure (see Table 28-7 on page 794) describing where to redirect SIDR_REQ. Only valid if the Status field indicates that the request must be redirected.
PrivateDataUD. 1120-bit field. Data opaque to the CM protocol, passed from the SIDR_REP sender to the SIDR_REP recipient. The recipient may choose to accept or reject a connection setup request based on the content of the private data. The format and meaning of the PrivateData is specific to the ServiceID and message type, and is not specified within the CM portion of the specification.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.222.35.225