Memory Windows
Definition of a Memory Window
As described earlier, the following access control attributes can be defined for an entire region:
Local read access is always automatically granted.
Enable/disable local write access.
Enable/disable remote write access (requires enabling of local write access as well).
Enable/disable remote read access.
Enable/disable remote atomic operation access (requires enabling of local write access as well).
Enable/disable memory window binding (window binding is described in “Binding a Window To a Region” on page 310).
Assuming that multiple QPs in remote CAs will be accessing various areas of a memory region local to an HCA, it may be desirable that not all of the remote QPs have the same access rights within the region. Refer to Figure 15-3 on page 309. This can be accomplished by overlaying the region with multiple windows and assigning different remote access rights within each window.
Figure 15-3. Window Concept
This permits software to:
Grant or revoke remote access rights to a subset of a registered region in a dynamic fashion with less performance penalty than experienced when using the region deregistration/registration/reregistration verbs.
Grant different remote access rights to different remote QPs and/or grant those rights over different ranges within a registered region.
Windows Only Supported for RC, UC, and RD
Windows are only supported on the RC, UC, and RD transport service types because these are the only QP types that support RDMA or atomic operations.
Window Management Verbs
The window management verbs are:
Allocate Memory Window. Creates a window and assigns it to a PD (for a detailed description of PDs, refer to “Protection Domains” on page 313). A Bind Memory Window verb call must then be performed to bind the window to a previously created region.
Bind Memory Window. Associates a window with a region, assigns it a start VA (and length) within the region, and assigns the remote access rights for accesses by remote QPs.
Binding a Window To a Region
General
Windows cannot be bound to a region unless the region was enabled for window binding at the time that the region was created (see “Region Windowing May Be Enabled When Region Is Created” on page 301). Assuming that the region supports window binding, the next step is to create the window using the Allocate Memory Window verb. The window is not yet associated with any memory region when it is allocated. The following are the input parameters software supplies in the Allocate Memory Window verb call:
HCA Handle.
Protection Domain to be assigned to the Memory Window.
The output parameters are:
Window Handle. It must be specified when making any subsequent calls to manipulate or query the window.
R_Key. The remote access key returned by this verb is really a dummy R_Key (the specification refers to it as an “unbound” R_Key) that has no access rights defined. It is used as one of the input parameters in the subsequent Bind Memory Window verb call.
After it has been created, a window is bound to a region by executing the Bind Memory Window verb. The memory region is required to have local write access enabled. The memory window, the region, and the QP used to perform the bind operation (see “Binding Doesn't Take Place Immediately” on page 312) must all belong to the same PD.
Bind Memory Window Input Parameters
The HCA Handle.
The QP Handle.
The WR containing the information required to perform the request. The WR contains the following elements:
- A user-defined, 64-bit WR ID.
- Memory window handle. Returned by the Allocate Memory Window verb.
- The R_Key currently associated with the Memory Window. This was returned when the Allocate Memory Window verb was executed.
- Memory region handle. Returned by the Register Memory Region or Register Physical Memory Region verb.
- The L_Key for the memory region that the memory window will bound to.
- The virtual start address of the bound range. The maximum width of a Virtual Address is 64 bits.
- The length of the range to be bound (in bytes).
- Access rights. The following may be selected in any combination (except as noted):
- Enable Remote Write Access (requires that the region have local write access enabled).
- Enable Remote Read Access.
- Enable Remote Atomic operation access (if Atomic operations are supported). Requires that the region have local write access enabled.
- Completion notification type. This must be specified if the SQ was set up for selectable signaling.
Bind Memory Window Output Parameters
The only output parameter returned is the R_Key associated with the new binding (its value must be different from that of the R_Key supplied as an input parameter).
Binding Doesn't Take Place Immediately
At the time that the Bind Memory Window verb is executed, a Bind Memory Window operation is posted to the SQ of the specified local QP [that the remote QP(s) will use to access local memory]. The window is not actually bound to the specified region, however, until the WQE is executed by the QP's SQ Logic.
It is a rule that any WQE posted to a SQ subsequent to a previously posted Bind Memory Window operation cannot begin execution until the Bind operation completes.
Window Characteristics
The following is a list of window characteristics:
The creation or destruction of a window necessitates a user mode-to-kernel mode transition (a high-overhead operation), but changing its binding does not (and is therefore low overhead).
To use a window:
Create the new window using the Allocate Memory Window verb call and assign it to a PD.
Using a Bind Memory Window verb call, bind the window to a preexistent region that is enabled for usage with windows.
Binding (or rebinding) a window is accomplished by posting a WR to a QP's SQ (note, however, that execution of the WQE does not cause any packets to be sent over the link).
The window and region must be members of the same PD.
The virtual address range assigned to a window can cover all or a subset of the range assigned to the region to which it is bound.
Common window usage:
- Allocate the window once.
- Then use it for multiple bindings.
A window being bound and any QP that attempts to access locations within the window both must be members of the same PD.
Each execution of the Bind Memory Window verb must cause the previously assigned R_Key to be invalidated and a new R_Key to be returned to the caller.
An HCA enforces the window access rights with byte-granularity.
A window with remote access rights can be bound to a region with no remote access rights. Remote access within the window is then permitted.
Submit a bind request with a window length of zero to invalidate a window's bindings without deallocating the window or enabling remote access to new areas.
It is improper to deallocate or rebind a window while it is being accessed by a remote QP. If this occurs, however, any transfer requests by remote QPs that are in process and actively using the window must fail with a protection violation.
It is an error to deregister or reregister a region while it still has any windows bound to it. Such windows are referred to as “orphans”.
Bind requests are only supported for the RC and UC (and, if the HCA supports RD, the RD) service types.