Packed Floating-Point Arithmetic

Listing 9-1 begins with the declaration of a C++ structure named YmmVal that’s declared in the header file YmmVal.h. This structure is similar to the XmmVal structure that you saw in Chapter 6. YmmVal contains a publicly-accessible anonymous union that facilitates packed operand data exchange between functions written in C++ and x86 assembly language. The members of this union correspond to the packed data types that can be used with a YMM register. The structure YmmVal also includes several formatting and display functions (the source code for these member functions is not shown).

The C++ code for example Ch09_01 starts with declarations for the assembly language functions AvxPackedMathF32_ and AvxPackedMathF64_. These functions carry out various packed single-precision and double-precision floating-point arithmetic operations using the supplied YmmVal arguments. Following the assembly language function declarations is the function AvxPackedMathF32. This function starts by initializing YmmVal variables a and b. Note that the C++ specifier alignas(32) is used with each YmmVal declaration. This specifier instructs the C++ compiler to align each YmmVal variable on a 32-byte boundary. Following YmmVal variable initialization, AvxPackedMathF32 calls the assembly language function AvxPackedMathF32_ to perform the required arithmetic. The results are then streamed to cout. The function AvxPackedMathF64 is the double-precision floating-point counterpart of AvxPackedMathF32.

Near the top of the assembly language code in Listing 9-1 is a .const section that defines packed constant values for calculating floating-point absolute values . The text dup is a MASM operator that allocates and optionally initializes multiple data values. In the current example, the statement AbsMaskF32 dword 8 dup(7fffffffh) allocates storage space for eight doubleword values and each value is initialized to 0x7fffffff. The following statement, AbsMaskF64 qword 4 dup(7fffffffffffffffh), allocates four quadwords of 0x7fffffffffffffff. Note that neither of these 256-bit wide operands is preceded by an align statement, which means that they may not be properly aligned in memory. The reason for this is that the MASM align directive does not support 32-byte alignment within a .const, .data, or .code section. Later in this chapter, you learn how to define a custom segment of constant values that supports 32-byte alignment.

Following the .const section, the first instruction of AvxPackedMathF32_, vmovaps ymm0,ymmword ptr [rcx], loads argument a (i.e., the eight floating-point values of YmmVal a) into register YMM0. The vmovaps can be used here since YmmVal a was defined using the alignas(32) specifier in the C++ code. The operator ymmword ptr directs the assembler to treat the memory location pointed to by RCX as a 256-bit wide operand. Use of the ymmword ptr operator is optional in this instance and employed to improve code readability. The ensuing vmovaps ymm1,ymmword ptr [rdx] instruction loads b into register YMM1. The vaddps ymm2,ymm0,ymm1 instruction that follows sums the packed single-precision floating-point values in YMM0 and YMM1; it then saves the result to YMM2. The vmovaps ymmword ptr [r8],ymm2 instruction saves the packed sums to c[0].

The ensuing vsubps, vmulps, and vdivps instructions carry out packed single-precision floating-point subtraction, multiplication, and division. This is followed by a vandps ymm2,ymm1,ymmword ptr [AbsMaskF32] instruction that calculates packed absolute values using argument b. The remaining instructions in AvxPackedMathF32_ calculate packed single-precision floating-point square roots, minimums, and maximums.

Prior to its ret instruction, the function AvxPackedMath32_ uses a vzeroupper instruction, which zeros the high-order 128 bits of each YMM register. As explained in Chapter 4, the vzeroupper instruction is needed here to avoid potential performance delays that can occur whenever the processor transitions from executing x86-AVX instructions that use 256-bit wide operands to executing x86-SSE instructions. Any assembly language function that uses one or more YMM registers and is callable from code that potentially uses x86-SSE instructions should always ensure that a vzeroupper instruction is executed before program control is transferred back to the calling function. You’ll see additional examples of vzeroupper instruction use in this and subsequent chapters.

Packed Floating-Point Arrays

In previous chapters, you learned how to carry out integer and floating-point array calculations using the general-purpose and XMM register sets. In this section, you learn how to perform floating-point array operations using the YMM register set.

Simple Calculations

Listing 9-2 shows the source code for example Ch09_02. This example illustrates how to perform simple array calculations using 256-bit wide packed floating-point operands. It also demonstrates how to detect and exclude invalid array elements from packed calculations. Source code example Ch09_02 is an array implementation of example Ch05_02 from Chapter 5, which calculated sphere surface areas and volumes. In that example, the assembly language function CalcSphereAreaVolume_ computed the surface area and volume of a single sphere. In this example, the sphere radii are passed via an array to calculating functions coded using C++ and assembly language. To make the example a little more interesting, both the C++ and assembly language calculating functions test for radii less than zero. If an invalid radius is detected, the calculating functions set the corresponding elements in the surface area and volume arrays to QNaN.

//------------------------------------------------

//        Ch09_02.cpp

//------------------------------------------------

#include "stdafx.h"

#include <iostream>

#include <iomanip>

#include <random>

#include <limits>

#define _USE_MATH_DEFINES

#include <math.h>

using namespace std;

extern "C" void AvxCalcSphereAreaVolume_(float* sa, float* vol, const float* r, size_t n);

extern "C" float c_PI_F32 = (float)M_PI;

extern "C" float c_QNaN_F32 = numeric_limits<float>::quiet_NaN();

void Init(float* r, size_t n, unsigned int seed)

{

  uniform_int_distribution<> ui_dist {1, 100};

  default_random_engine rng {seed};

  for (size_t i = 0; i < n; i++)

    r[i] = (float)ui_dist(rng) / 10.0f;

  // Set invalid radii for test purposes

  if (n > 2)

  {

    r[2] = -r[2];

    r[n / 4] = -r[n / 4];

    r[n / 2] = -r[n / 2];

    r[n / 4 * 3] = -r[n / 4 * 3];

    r[n - 2] = -r[n - 2];

  }

}

void AvxCalcSphereAreaVolumeCpp(float* sa, float* vol, const float* r, size_t n)

{

  for (size_t i = 0; i < n; i++)

  {

    if (r[i] < 0.0f)

      sa[i] = vol[i] = c_QNaN_F32;

    else

    {

      sa[i] = r[i] * r[i] * 4.0f * c_PI_F32;

      vol[i] = sa[i] * r[i] / 3.0f;

    }

  }

}

void AvxCalcSphereAreaVolume(void)

{

  const size_t n = 21;

  alignas(32) float r[n];

  alignas(32) float sa1[n];

  alignas(32) float vol1[n];

  alignas(32) float sa2[n];

  alignas(32) float vol2[n];

  Init(r, n, 93);

  AvxCalcSphereAreaVolumeCpp(sa1, vol1, r, n);

  AvxCalcSphereAreaVolume_(sa2, vol2, r, n);

  cout << " Results for AvxCalcSphereAreaVolume ";

  cout << fixed;

  const float eps = 1.0e-6f;

  for (size_t i = 0; i < n; i++)

  {

    cout << setw(2) << i << ": ";

    cout << setprecision(2);

    cout << setw(5) << r[i] << " | ";

    cout << setprecision(6);

    cout << setw(12) << sa1[i] << "  ";

    cout << setw(12) << sa2[i] << " | ";

    cout << setw(12) << vol1[i] << "  ";

    cout << setw(12) << vol2[i];

    bool b0 = (fabs(sa1[i] - sa2[i]) > eps);

    bool b1 = (fabs(vol1[i] - vol2[i]) > eps);

    if (b0 || b1)

      cout << " Compare discrepancy";

    cout << ' ';

  }

}

int main()

{

  AvxCalcSphereAreaVolume();

  return 0;

}

;-------------------------------------------------

;        Ch09_02.asm

;-------------------------------------------------

    include <cmpequ.asmh>

    include <MacrosX86-64-AVX.asmh>

    .const

r4_3p0 real4 3.0

r4_4p0 real4 4.0

    extern c_PI_F32:real4

    extern c_QNaN_F32:real4

; extern "C" void AvxCalcSphereAreaVolume_(float* sa, float* vol, const float* r, size_t n);

    .code

AvxCalcSphereAreaVolume_ proc frame

    _CreateFrame CC_,0,64

    _SaveXmmRegs xmm6,xmm7,xmm8,xmm9

    _EndProlog

; Initialize

    vbroadcastss ymm0,real4 ptr [r4_4p0]    ;packed 4.0

    vbroadcastss ymm1,real4 ptr [c_PI_F32]   ;packed PI

    vmulps ymm6,ymm0,ymm1            ;packed 4.0 * PI

    vbroadcastss ymm7,real4 ptr [r4_3p0]    ;packed 3.0

    vbroadcastss ymm8,real4 ptr [c_QNaN_F32]  ;packed QNaN

    vxorps ymm9,ymm9,ymm9            ;packed 0.0

    xor eax,eax               ;common offset for arrays

    cmp r9,8

    jb FinalR                ;skip main loop if n < 8

; Calculate surface area and volume values using packed arithmetic

@@:   vmovdqa ymm0,ymmword ptr [r8+rax]    ;load next 8 radii

    vmulps ymm2,ymm6,ymm0          ;4.0 * PI * r

    vmulps ymm3,ymm2,ymm0          ;4.0 * PI * r * r

    vcmpps ymm1,ymm0,ymm9,CMP_LT      ;ymm1 = mask of radii < 0.0

    vandps ymm4,ymm1,ymm8          ;set surface area to QNaN for radii < 0.0

    vandnps ymm5,ymm1,ymm3         ;keep surface area for radii >= 0.0

    vorps ymm5,ymm4,ymm5          ;final packed surface area

    vmovaps ymmword ptr[rcx+rax],ymm5    ;save packed surface area

    vmulps ymm2,ymm3,ymm0          ;4.0 * PI * r * r * r

    vdivps ymm3,ymm2,ymm7          ;4.0 * PI * r * r * r / 3.0

    vandps ymm4,ymm1,ymm8          ;set volume to QNaN for radii < 0.0

    vandnps ymm5,ymm1,ymm3         ;keep volume for radii >= 0.0

    vorps ymm5,ymm4,ymm5          ;final packed volume

    vmovaps ymmword ptr[rdx+rax],ymm5    ;save packed volume

    add rax,32               ;rax = offset to next set of radii

    sub r9,8

    cmp r9,8

    jae @B                 ;repeat until n < 8

; Perform final calculations using scalar arithmetic

FinalR: test r9,r9

    jz Done                 ;skip loop of no more elements

@@:   vmovss xmm0,real4 ptr [r8+rax]

    vmulss xmm2,xmm6,xmm0          ;4.0 * PI * r

    vmulss xmm3,xmm2,xmm0          ;4.0 * PI * r * r

    vcmpss xmm1,xmm0,xmm9,CMP_LT

    vandps xmm4,xmm1,xmm8

    vandnps xmm5,xmm1,xmm3

    vorps xmm5,xmm4,xmm5

    vmovss real4 ptr[rcx+rax],xmm5     ;save surface area

    vmulss xmm2,xmm3,xmm0          ;4.0 * PI * r * r * r

    vdivss xmm3,xmm2,xmm7          ;4.0 * PI * r * r * r / 3.0

    vandps xmm4,xmm1,xmm8

    vandnps xmm5,xmm1,xmm3

    vorps xmm5,xmm4,xmm5

    vmovss real4 ptr[rdx+rax],xmm5     ;save volume

    add rax,4

    dec r9

    jnz @B                 ;repeat until done

Done:  vzeroupper

    _RestoreXmmRegs xmm6,xmm7,xmm8,xmm9

    _DeleteFrame

    ret

AvxCalcSphereAreaVolume_ endp

    end

Listing 9-2.

Example Ch09_02

The C++ code in Listing 9-2 includes a function named AvxCalcSphereAreaVolumeCpp. This function calculates sphere surface areas and volumes. The sphere radii are passed to AvxCalcSphereAreaVolumeCpp via an array. Prior to calculating a surface area or volume, the sphere’s radius (r[i]) is tested to verify that it’s not negative. If the radius is negative, the corresponding elements in the surface area and volume arrays (sa[i] and vol[i]) are set to c_QNaN_F32. The remaining C++ code performs the necessary initializations, exercises the C++ and assembly language calculating functions, and displays the results. Note that the function AvxCalcSphereAreaVolume employs the alignas(32) specifier with each array declaration.

The assembly language function AvxCalcSphereAreaVolume_ performs the same calculations as its C++ counterpart. Following its prolog, AvxCalcSphereAreaVolume_ uses a series of vbroadcastss instructions to initialize packed versions of the required constants. Prior to the start of the processing loop, a cmp r9,8 instruction checks the value of n. The reason for this check is that the processing loop carries out eight surface area and volume calculations simultaneously using 256-bit wide operands. The jb FinalR conditional jump instruction skips the processing loop if there are fewer than eight radii to process.

Each processing loop iteration begins with a vmovdqa ymm0,ymmword ptr [r8+rax] instruction that loads eight single-precision floating-point radii into register YMM0. The ensuing vmulps instructions calculate the sphere surface areas. The next instruction, vcmpps ymm1,ymm0,ymm9,CMP_LT, tests each sphere radii for a value less than 0.0 (register YMM9 contains packed 0.0). Recall that the vcmpps instruction signifies its results by setting elements in the destination operand to either 0x00000000 (false compare predicate) or 0xffffffff (true compare predicate). The vandps, vandnps, and vorps instructions that follow set the surface area of each sphere whose radius is less than 0.0 to c_QNaN_F32. Figure 9-1 illustrates this operation in greater detail. A vmovaps ymmword ptr[rcx+rax],ymm5 instruction saves the eight sphere surface area values to the array sa.

Following the calculation of the surface areas, the vmulps ymm2,ymm3,ymm0 and vdivps ymm3,ymm2,ymm7 instructions compute the sphere volumes. The processing loop uses another vandps, vandnps, and vorps instruction sequence to set the volume of any negative-radius sphere to c_QNaN_F32. These values are then saved to the array vol. The processing loop repeats until there are fewer than eight remaining radii.

The next block of code computes sphere surface areas and volumes for the remaining (1–7) radii. Note that AvxCalcSphereAreaVolume_ carries out these calculations using scalar single-precision floating-point arithmetic. The scalar processing loop performs the same arithmetic and Boolean operations as the packed processing loop. Similar to the previous example, AvxCalcSphereAreaVolume_ uses a vzeroupper instruction immediately after the scalar processing loop. This instruction is needed since AvxCalcSphereAreaVolume_ carried out its calculations using the YMM register set. When a vzeroupper instruction is required, it should always be positioned before any function epilog macros (e.g., _RestoreXmmRegs and _DeleteFrame) and the ret instruction. Here are the results for source code example Ch09_02:

Results for AvxCalcSphereAreaVolume

 0:  3.80 |  181.458389   181.458389 |  229.847290   229.847290

 1: 10.00 | 1256.637085  1256.637085 | 4188.790527  4188.790527

 2: -6.10 |     nan      nan |     nan      nan

 3:  3.70 |  172.033630   172.033630 |  212.174805   212.174805

 4:  9.60 | 1158.116821  1158.116821 | 3705.973877  3705.973877

 5: -6.60 |     nan      nan |     nan      nan

 6:  2.60 |  84.948662   84.948654 |  73.622169   73.622162 Compare discrepancy

 7:  9.30 | 1086.865479  1086.865479 | 3369.283203  3369.283203

 8:  9.00 | 1017.876038  1017.876038 | 3053.628174  3053.628174

 9:  5.80 |  422.732758   422.732758 |  817.283386   817.283386

10: -2.90 |     nan      nan |     nan      nan

11:  8.10 |  824.479675   824.479675 | 2226.095215  2226.095215

12:  3.00 |  113.097336   113.097336 |  113.097328   113.097328

13:  8.00 |  804.247742   804.247742 | 2144.660645  2144.660645

14:  1.40 |  24.630087   24.630085 |  11.494040   11.494039 Compare discrepancy

15: -1.80 |     nan      nan |     nan      nan

16:  4.30 |  232.352219   232.352219 |  333.038177   333.038177

17:  6.60 |  547.391113   547.391113 | 1204.260376  1204.260376

18:  4.50 |  254.469009   254.469009 |  381.703522   381.703522

19: -1.20 |     nan      nan |     nan      nan

20:  4.50 |  254.469009   254.469009 |  381.703522   381.703522

The output for source code example Ch09_02 includes a couple of lines with the text “compare discrepancy.” This text was generated by the compare code in AvxCalcSphereAreaVolume to exemplify the non-associativity of floating-point arithmetic. In this example, the functions AvxCalcSphereAreaVolumeCpp and AvxCalcSphereAreaVolume_ carried out their respective floating-point calculations using different operands orderings. For each sphere surface area, the C++ code calculates sa[i] = r[i] * r[i] * 4.0 * c_PI_F32, while the assembly language code calculates sa[i] = 4.0 * c_PI_F32 * r[i] * r[i]. Tiny numerical discrepancies like this are not unusual when comparing floating-point values that are calculated using different operand orderings irrespective of the programming language. This is something that you should keep in mind if you’re developing production code that includes multiple versions of the same calculating function (e.g., one coded using C++ and an AVX/AVX2 accelerated version that’s implemented using x86 assembly language).

Finally, you may have noticed that the function AvxCalcSphereAreaVolume_ handled invalid radii sans any x86 conditional jump instructions. Minimizing the number of conditional jump instructions in a function, especially data-dependent ones, often results in faster executing code. You’ll learn more about jump instruction optimization techniques in Chapter 15.

Column Means

Listing 9-3 shows the source code for example Ch09_03. This example illustrates how to calculate the arithmetic mean of each column in a two-dimensional array of double-precision floating-point values.

//------------------------------------------------

//        Ch09_03.cpp

//------------------------------------------------

#include "stdafx.h"

#include <iostream>

#include <iomanip>

#include <random>

#include <memory>

using namespace std;

extern "C" size_t c_NumRowsMax = 1024 * 1024;

extern "C" size_t c_NumColsMax = 1024 * 1024;

extern "C" bool AvxCalcColumnMeans_(const double* x, size_t nrows, size_t ncols, double* col_means);

void Init(double* x, size_t n, unsigned int seed)

{

  uniform_int_distribution<> ui_dist {1, 2000};

  default_random_engine rng {seed};

  for (size_t i = 0; i < n; i++)

    x[i] = (double)ui_dist(rng) / 10.0;

}

bool AvxCalcColumnMeansCpp(const double* x, size_t nrows, size_t ncols, double* col_means)

{

  // Make sure nrows and ncols are valid

  if (nrows == 0 || nrows > c_NumRowsMax)

    return false;

  if (ncols == 0 || ncols > c_NumColsMax)

    return false;

  // Set initial column means to zero

  for (size_t i = 0; i < ncols; i++)

    col_means[i] = 0.0;

  // Calculate column means

  for (size_t i = 0; i < nrows; i++)

  {

    for (size_t j = 0; j < ncols; j++)

      col_means[j] += x[i * ncols + j];

  }

  for (size_t j = 0; j < ncols; j++)

    col_means[j] /= nrows;

  return true;

}

void AvxCalcColumnMeans(void)

{

  const size_t nrows = 20;

  const size_t ncols = 11;

  unique_ptr<double[]> x {new double[nrows * ncols]};

  unique_ptr<double[]> col_means1 {new double[ncols]};

  unique_ptr<double[]> col_means2 {new double[ncols]};

  Init(x.get(), nrows * ncols, 47);

  bool rc1 = AvxCalcColumnMeansCpp(x.get(), nrows, ncols, col_means1.get());

  bool rc2 = AvxCalcColumnMeans_(x.get(), nrows, ncols, col_means2.get());

  cout << "Results for AvxCalcColumnMeans ";

  if (!rc1 || !rc2)

  {

    cout << "Invalid return code: ";

    cout << "rc1 = " << boolalpha << rc1 << ", ";

    cout << "rc2 = " << boolalpha << rc2 << ' ';

    return;

  }

  cout << " Test Matrix ";

  cout << fixed << setprecision(1);

  for (size_t i = 0; i < nrows; i++)

  {

    cout << "row " << setw(2) << i;

    for (size_t j = 0; j < ncols; j++)

      cout << setw(7) << x[i * ncols + j];

    cout << ' ';

  }

  cout << " Column Means ";

  cout << setprecision(2);

  for (size_t j = 0; j < ncols; j++)

  {

    cout << "col_means1[" << setw(2) << j << "] =";

    cout << setw(10) << col_means1[j] << "  ";

    cout << "col_means2[" << setw(2) << j << "] =";

    cout << setw(10) << col_means2[j] << ' ';

  }

}

int main()

{

  AvxCalcColumnMeans();

  return 0;

}

;-------------------------------------------------

;        Ch09_03.asm

;-------------------------------------------------

; extern "C" bool AvxCalcColMeans_(const double* x, size_t nrows, size_t ncols, double* col_means)

    extern c_NumRowsMax:qword

    extern c_NumColsMax:qword

    .code

AvxCalcColumnMeans_ proc

; Validate nrows and ncols

    xor eax,eax             ;error return code (also col_mean index)

    test rdx,rdx

    jz Done               ;jump if nrows is zero

    cmp rdx,[c_NumRowsMax]

    ja Done               ;jump if nrows is too large

    test r8,r8

    jz Done               ;jump if ncols is zero

    cmp r8,[c_NumColsMax]

    ja Done               ;jump if ncols is too large

; Initialize elements of col_means to zero

    vxorpd xmm0,xmm0,xmm0        ;xmm0[63:0] = 0.0

@@:   vmovsd real8 ptr[r9+rax*8],xmm0   ;col_means[i] = 0.0

    inc rax

    cmp rax,r8

    jb @B                ;repeat until done

    vcvtsi2sd xmm2,xmm2,rdx       ;convert nrows for later use

; Compute the sum of each column in x

LP1:  mov r11,r9             ;r11 = ptr to col_means

    xor r10,r10             ;r10 = col_index

LP2:  mov rax,r10             ;rax = col_index

    add rax,4

    cmp rax,r8             ;4 or more columns remaining?

    ja @F                ;jump if no (col_index + 4 > ncols)

; Update col_means using next four columns

    vmovupd ymm0,ymmword ptr [rcx]   ;load next 4 columns of current row

    vaddpd ymm1,ymm0,ymmword ptr [r11] ;add to col_means

    vmovupd ymmword ptr [r11],ymm1   ;save updated col_means

    add r10,4              ;col_index += 4

    add rcx,32             ;update x ptr

    add r11,32             ;update col_means ptr

    jmp NextColSet

@@:   sub rax,2

    cmp rax,r8             ;2 or more columns remaining?

    ja @F                ;jump if no (col_index + 2 > ncols)

; Update col_means using next two columns

    vmovupd xmm0,xmmword ptr [rcx]   ;load next 2 columns of current row

    vaddpd xmm1,xmm0,xmmword ptr [r11] ;add to col_means

    vmovupd xmmword ptr [r11],xmm1   ;save updated col_means

    add r10,2              ;col_index += 2

    add rcx,16             ;update x ptr

    add r11,16             ;update col_means ptr

    jmp NextColSet

; Update col_means using next column (or last column in the current row)

@@:   vmovsd xmm0,real8 ptr [rcx]     ;load x from last column

    vaddsd xmm1,xmm0,real8 ptr [r11]  ;add to col_means

    vmovsd real8 ptr [r11],xmm1     ;save updated col_means

    inc r10               ;col_index += 1

    add rcx,8              ;update x ptr

NextColSet:

    cmp r10,r8             ;more columns in current row?

    jb LP2               ;jump if yes

    dec rdx               ;nrows -= 1

    jnz LP1               ;jump if more rows

; Compute the final col_means

@@:   vmovsd xmm0,real8 ptr [r9]     ;xmm0 = col_means[i]

    vdivsd xmm1,xmm0,xmm2        ;compute final mean

    vmovsd real8 ptr [r9],xmm1     ;save col_mean[i]

    add r9,8              ;update col_means ptr

    dec r8               ;ncols -= 1

    jnz @B               ;repeat until done

    mov eax,1              ;set success return code

Done:  vzeroupper

    ret

AvxCalcColumnMeans_ endp

    end

Listing 9-3.

Example Ch09_03

Toward the top of the C++ code is a function named AvxCalcColumnMeansCpp. This function calculates the column means of a two-dimensional array using a straightforward set of nested for loops and some simple arithmetic. The function AvxCalcColumnMeans contains code that uses the C++ smart pointer class unique_ptr<> to help manage its dynamically-allocated arrays. Note that storage space for the test array x is allocated using the C++ new operator, which means that the array may not be aligned on a 16- or 32-byte boundary. In this particular example, aligning the start of array x to a specific boundary would be of little benefit since it’s not possible to align the individual rows or columns of a standard C++ two-dimensional array (recall that the elements of a two-dimensional C++ array are stored in a contiguous block of memory using row-major ordering as described in Chapter 2).

The function AvxCalcColumnMeans also uses class unique_ptr<> and the new operator for the one-dimensional arrays col_means1 and col_means2. Using unique_ptr<> in this example simplifies the C++ code somewhat since its destructor automatically invokes the delete[] operator to release the storage space that was allocated by the new operator. If you’re interested in learning more about the smart pointer class unique_ptr<>, Appendix A contains a list of C++ references that you can consult. The remaining code in AvxCalcColumnMeans invokes the C++ and assembly language column-mean calculating functions and streams the results to cout.

Following argument validation, the assembly language function AvxCalcColMeans_ initializes each element in col_means to 0.0. These elements will maintain the intermediate column sums. In order to maximize throughput, the column summation code uses slightly different instruction sequences depending on the current column and the total number of columns in the array. For example, assume that array x contains seven columns. For each row, the elements of the first four columns in x can be added to col_means using 256-bit wide packed addition; the elements of the next two columns can be added to col_means using 128-bit wide packed addition; and the final column element must be added to col_means using scalar addition. Figure 9-2 illustrates this technique in greater detail.

The mov r11,r9 instruction next to the label LP1 is the starting point for adding elements in the current row of x to col_means. This instruction initializes R11 to first entry in col_means. The col_index counter in register R10 is then set to zero. The instruction group near the label LP2 determines the number of columns remaining to be processed in the current row. If four or more columns remain, the next four elements from the current row are added to the column sums in col_means. A vmovupd ymm0,ymmword ptr [rcx] instruction loads four double-precision floating-point values from x into YMM0 (a vmovapd instruction is not used here since alignment of the elements is unknown). The ensuing vaddpd ymm1,ymm0,ymmword ptr [r11] instruction sums the current array elements with the corresponding elements in col_means, and the vmovupd ymmword ptr [r11],ymm1 instruction saves the updated results back to col_means. The function’s various pointers and counters are then updated in preparation for the next set of elements from the current row of x.

The summation code repeats the steps described in the previous paragraph until the number of array elements that remain in the current row is less than four. As soon as this condition is met, the elements in the remaining columns (if any) must be processed using 128-bit wide or 64-bit wide operands. This is the reason for the distinct blocks of code in AvxCalcColumnMeans_ that process four elements, two elements, or a single element per row. Following computation of the column sums, each element in col_means is divided by n, which yields the final column mean. Here are the results for source code example Ch09_03:

Results for AvxCalcColumnMeans

Test Matrix

row 0 125.6  59.9 100.0 170.5 140.1 197.2  73.7  15.2  92.4 155.3 159.2

row 1  77.6 105.4  45.0 176.8  65.9  12.3 189.1 102.0  56.2 112.8  17.2

row 2 198.9 199.3  74.6 137.9  65.0 125.0  19.8  32.1  58.6  94.1 123.5

row 3  1.7  29.1  99.1 200.0 109.0 123.7 130.0 125.3 146.2  90.6  52.2

row 4  8.7  88.7  84.8 174.6 164.4 106.2 114.0 151.8 130.8 101.9 116.2

row 5  42.7 130.5 180.4 199.4 196.6  99.7 163.6  34.2  5.5 146.1 108.5

row 6 120.0 159.5  26.0  83.4  58.7  10.1 170.1  20.5  10.8  48.3 121.9

row 7 148.9 148.4 142.0 106.6 198.4  60.3  72.1 137.8  74.5  75.7  44.8

row 8  25.7 192.0  12.1  23.4  98.7 145.3 196.8  43.9 143.1  25.1 122.6

row 9  5.4 134.7 165.1  61.8  46.7 183.3 173.7 146.9  76.5 186.2  24.9

row 10 174.5 158.9 127.8  58.9  42.9 182.9  7.8  50.3  68.0  62.0  66.1

row 11  47.3 166.2  8.2  71.2  98.5  12.4 179.0 100.2  29.7 167.4 155.2

row 12  23.9 196.6 148.7  7.1 128.2 128.8  66.3 153.7  60.7 115.4  71.6

row 13 103.4 184.3 161.5  57.9 199.2  79.3  28.1  73.1  12.5  71.3 100.4

row 14 130.3 154.2 127.5  29.7 198.2 170.3 121.9  80.4 159.8  70.0  82.6

row 15  26.7  45.6  67.7 109.7  5.1  96.2 188.7 100.7  48.3 164.2  75.4

row 16 115.4  25.5  58.8 148.5  80.7 149.1 156.7 153.8  42.0 103.7  4.2

row 17  67.9 161.5  16.9 102.1  77.3  3.9 104.7  97.2 181.8 182.0 155.1

row 18 169.5 122.4 102.2  5.5  14.5 105.1 181.5  83.3 117.6  52.1 111.2

row 19  47.1 146.9  21.0  8.6 130.3  24.7  95.7  6.7 159.9  38.8  82.6

Column Means

col_means1[ 0] =   83.06  col_means2[ 0] =   83.06

col_means1[ 1] =  130.48  col_means2[ 1] =  130.48

col_means1[ 2] =   88.47  col_means2[ 2] =   88.47

col_means1[ 3] =   96.68  col_means2[ 3] =   96.68

col_means1[ 4] =  105.92  col_means2[ 4] =  105.92

col_means1[ 5] =  100.79  col_means2[ 5] =  100.79

col_means1[ 6] =  121.66  col_means2[ 6] =  121.66

col_means1[ 7] =   85.46  col_means2[ 7] =   85.46

col_means1[ 8] =   83.75  col_means2[ 8] =   83.75

col_means1[ 9] =  103.15  col_means2[ 9] =  103.15

col_means1[10] =   89.77  col_means2[10] =   89.77

Correlation Coefficient

The next source code example illustrates how to calculate a correlation coefficient using packed double-precision floating-point arithmetic. This example also demonstrates how to perform a few common auxiliary operations with packed floating-point operands, including 128-bit wide extractions and horizontal addition. Listing 9-4 shows the source code for example Ch09_04.

//------------------------------------------------

//        Ch09_04.cpp

//------------------------------------------------

#include "stdafx.h"

#include <iostream>

#include <iomanip>

#include <string>

#include <random>

#include "AlignedMem.h"

using namespace std;

extern "C" bool AvxCalcCorrCoef_(const double* x, const double* y, size_t n, double sums[5], double epsilon, double* rho);

void Init(double* x, double* y, size_t n, unsigned int seed)

{

  uniform_int_distribution<> ui_dist {1, 999};

  default_random_engine rng {seed};

  for (size_t i = 0; i < n; i++)

  {

    x[i] = (double)ui_dist(rng);

    y[i] = x[i] + (ui_dist(rng) % 6000) - 3000;

  }

}

bool AvxCalcCorrCoefCpp(const double* x, const double* y, size_t n, double sums[5], double epsilon, double* rho)

{

  const size_t alignment = 32;

  // Make sure n is valid

  if (n == 0)

    return false;

  // Make sure x and y are properly aligned

  if (!AlignedMem::IsAligned(x, alignment))

    return false;

  if (!AlignedMem::IsAligned(y, alignment))

    return false;

  // Calculate and save sum variables

  double sum_x = 0, sum_y = 0, sum_xx = 0, sum_yy = 0, sum_xy = 0;

  for (size_t i = 0; i < n; i++)

  {

    sum_x += x[i];

    sum_y += y[i];

    sum_xx += x[i] * x[i];

    sum_yy += y[i] * y[i];

    sum_xy += x[i] * y[i];

  }

  sums[0] = sum_x;

  sums[1] = sum_y;

  sums[2] = sum_xx;

  sums[3] = sum_yy;

  sums[4] = sum_xy;

  // Calculate rho

  double rho_num = n * sum_xy - sum_x * sum_y;

  double rho_den = sqrt(n * sum_xx - sum_x * sum_x) * sqrt(n * sum_yy - sum_y * sum_y);

  if (rho_den >= epsilon)

  {

    *rho = rho_num / rho_den;

    return true;

  }

  else

  {

    *rho = 0;

    return false;

  }

}

int main()

{

  const size_t n = 103;

  const size_t alignment = 32;

  AlignedArray<double> x_aa(n, alignment);

  AlignedArray<double> y_aa(n, alignment);

  double sums1[5], sums2[5];

  double rho1, rho2;

  double epsilon = 1.0e-12;

  double* x = x_aa.Data();

  double* y = y_aa.Data();

  Init(x, y, n, 71);

  bool rc1 = AvxCalcCorrCoefCpp(x, y, n, sums1, epsilon, &rho1);

  bool rc2 = AvxCalcCorrCoef_(x, y, n, sums2, epsilon, &rho2);

  cout << "Results for AvxCalcCorrCoef ";

  if (!rc1 || !rc2)

  {

    cout << "Invalid return code ";

    cout << "rc1 = " << boolalpha << rc1 << ", ";

    cout << "rc2 = " << boolalpha << rc2 << ' ';

    return 1;

  }

  int w = 14;

  string sep(w * 3, '-');

  cout << fixed << setprecision(8);

  cout << "Value  " << setw(w) << "C++" << " " << setw(w) << "x86-AVX" << ' ';

  cout << sep << ' ';

  cout << "rho:   " << setw(w) << rho1 << " " << setw(w) << rho2 << " ";

  cout << setprecision(1);

  cout << "sum_x:  " << setw(w) << sums1[0] << " " << setw(w) << sums2[0] << ' ';

  cout << "sum_y:  " << setw(w) << sums1[1] << " " << setw(w) << sums2[1] << ' ';

  cout << "sum_xx: " << setw(w) << sums1[2] << " " << setw(w) << sums2[2] << ' ';

  cout << "sum_yy: " << setw(w) << sums1[3] << " " << setw(w) << sums2[3] << ' ';

  cout << "sum_xy: " << setw(w) << sums1[4] << " " << setw(w) << sums2[4] << ' ';

  return 0;

}

;-------------------------------------------------

;        Ch09_04.asm

;-------------------------------------------------

    include <MacrosX86-64-AVX.asmh>

; extern "C" bool AvxCalcCorrCoef_(const double* x, const double* y, size_t n, double sums[5], double epsilon, double* rho)

;

; Returns    0 = error, 1 = success

    .code

AvxCalcCorrCoef_ proc frame

    _CreateFrame CC_,0,32

    _SaveXmmRegs xmm6,xmm7

    _EndProlog

; Validate arguments

    or r8,r8

    jz BadArg              ;jump if n == 0

    test rcx,1fh

    jnz BadArg             ;jump if x is not aligned

    test rdx,1fh

    jnz BadArg             ;jump if y is not aligned

; Initialize sum variables to zero

    vxorpd ymm3,ymm3,ymm3        ;ymm3 = packed sum_x

    vxorpd ymm4,ymm4,ymm4        ;ymm4 = packed sum_y

    vxorpd ymm5,ymm5,ymm5        ;ymm5 = packed sum_xx

    vxorpd ymm6,ymm6,ymm6        ;ymm6 = packed sum_yy

    vxorpd ymm7,ymm7,ymm7        ;ymm7 = packed sum_xy

    mov r10,r8             ;r10 = n

    cmp r8,4

    jb LP2               ;jump if n >= 1 && n <= 3

; Calculate intermediate packed sum variables

LP1:  vmovapd ymm0,ymmword ptr [rcx]   ;ymm0 = packed x values

    vmovapd ymm1,ymmword ptr [rdx]   ;ymm1 = packed y values

    vaddpd ymm3,ymm3,ymm0        ;update packed sum_x

    vaddpd ymm4,ymm4,ymm1        ;update packed sum_y

    vmulpd ymm2,ymm0,ymm1        ;ymm2 = packed xy values

    vaddpd ymm7,ymm7,ymm2        ;update packed sum_xy

    vmulpd ymm0,ymm0,ymm0        ;ymm0 = packed xx values

    vmulpd ymm1,ymm1,ymm1        ;ymm1 = packed yy values

    vaddpd ymm5,ymm5,ymm0        ;update packed sum_xx

    vaddpd ymm6,ymm6,ymm1        ;update packed sum_yy

    add rcx,32             ;update x ptr

    add rdx,32             ;update y ptr

    sub r8,4              ;n -= 4

    cmp r8,4              ;is n >= 4?

    jae LP1               ;jump if yes

    or r8,r8              ;is n == 0?

    jz FSV               ;jump if yes

; Update sum variables with final x & y values

LP2:  vmovsd xmm0,real8 ptr [rcx]     ;xmm0[63:0] = x[i], ymm0[255:64] = 0

    vmovsd xmm1,real8 ptr [rdx]     ;xmm1[63:0] = y[i], ymm1[255:64] = 0

    vaddpd ymm3,ymm3,ymm0        ;update packed sum_x

    vaddpd ymm4,ymm4,ymm1        ;update packed sum_y

    vmulpd ymm2,ymm0,ymm1        ;ymm2 = packed xy values

    vaddpd ymm7,ymm7,ymm2        ;update packed sum_xy

    vmulpd ymm0,ymm0,ymm0        ;ymm0 = packed xx values

    vmulpd ymm1,ymm1,ymm1        ;ymm1 = packed yy values

    vaddpd ymm5,ymm5,ymm0        ;update packed sum_xx

    vaddpd ymm6,ymm6,ymm1        ;update packed sum_yy

    add rcx,8              ;update x ptr

    add rdx,8              ;update y ptr

    sub r8,1              ;n -= 1

    jnz LP2               ;repeat until done

; Calculate final sum variables

FSV:  vextractf128 xmm0,ymm3,1

    vaddpd xmm1,xmm0,xmm3

    vhaddpd xmm3,xmm1,xmm1       ;xmm3[63:0] = sum_x

    vextractf128 xmm0,ymm4,1

    vaddpd xmm1,xmm0,xmm4

    vhaddpd xmm4,xmm1,xmm1       ;xmm4[63:0] = sum_y

    vextractf128 xmm0,ymm5,1

    vaddpd xmm1,xmm0,xmm5

    vhaddpd xmm5,xmm1,xmm1       ;xmm5[63:0] = sum_xx

    vextractf128 xmm0,ymm6,1

    vaddpd xmm1,xmm0,xmm6

    vhaddpd xmm6,xmm1,xmm1       ;xmm6[63:0] = sum_yy

    vextractf128 xmm0,ymm7,1

    vaddpd xmm1,xmm0,xmm7

    vhaddpd xmm7,xmm1,xmm1       ;xmm7[63:0] = sum_xy

; Save final sum variables

    vmovsd real8 ptr [r9],xmm3     ;save sum_x

    vmovsd real8 ptr [r9+8],xmm4    ;save sum_y

    vmovsd real8 ptr [r9+16],xmm5    ;save sum_xx

    vmovsd real8 ptr [r9+24],xmm6    ;save sum_yy

    vmovsd real8 ptr [r9+32],xmm7    ;save sum_xy

; Calculate rho numerator

; rho_num = n * sum_xy - sum_x * sum_y;

    vcvtsi2sd xmm2,xmm2,r10       ;xmm2 = n

    vmulsd xmm0,xmm2,xmm7        ;xmm0 = = n * sum_xy

    vmulsd xmm1,xmm3,xmm4        ;xmm1 = sum_x * sum_y

    vsubsd xmm7,xmm0,xmm1        ;xmm7 = rho_num

; Calculate rho denominator

; t1 = sqrt(n * sum_xx - sum_x * sum_x)

; t2 = sqrt(n * sum_yy - sum_y * sum_y)

; rho_den = t1 * t2

    vmulsd xmm0,xmm2,xmm5    ;xmm0 = n * sum_xx

    vmulsd xmm3,xmm3,xmm3    ;xmm3 = sum_x * sum_x

    vsubsd xmm3,xmm0,xmm3    ;xmm3 = n * sum_xx - sum_x * sum_x

    vsqrtsd xmm3,xmm3,xmm3   ;xmm3 = t1

    vmulsd xmm0,xmm2,xmm6    ;xmm0 = n * sum_yy

    vmulsd xmm4,xmm4,xmm4    ;xmm4 = sum_y * sum_y

    vsubsd xmm4,xmm0,xmm4    ;xmm4 = n * sum_yy - sum_y * sum_y

    vsqrtsd xmm4,xmm4,xmm4   ;xmm4 = t2

    vmulsd xmm0,xmm3,xmm4    ;xmm0 = rho_den

; Calculate and save final rho

    xor eax,eax

    vcomisd xmm0,real8 ptr [rbp+CC_OffsetStackArgs] ;rho_den < epsilon?

    setae al                ;set return code

    jb BadRho                ;jump if rho_den < epsilon

    vdivsd xmm1,xmm7,xmm0          ;xmm1 = rho

SavRho: mov rdx,[rbp+CC_OffsetStackArgs+8]   ;rdx = ptr to rho

    vmovsd real8 ptr [rdx],xmm1       ;save rho

Done:  vzeroupper

    _RestoreXmmRegs xmm6,xmm7

    _DeleteFrame

    ret

; Error handling code

BadRho: vxorpd xmm1,xmm1,xmm1        ;rho = 0

    jmp SavRho

BadArg: xor eax,eax             ;eax = invalid arg ret code

    jmp Done

AvxCalcCorrCoef_ endp

    end

Listing 9-4.

Example Ch09_04

A correlation coefficient measures the strength of association between two variables. Correlation coefficients can range in value from -1.0 to +1.0, signifying a perfect negative or positive relationship between the two variables. Real-world correlation coefficients are rarely equal to these theoretical limits. A correlation coefficient of 0.0 indicates that the data variables are not associated. The C++ and assembly language code in this example calculate the well-known Pearson correlation coefficient using the following equation:

In order to calculate a correlation coefficient using this formula, a function must compute the following five sum variables:

The C++ function AvxCalcCorrCoefCpp shows how to calculate a correlation coefficient. This function begins by checking the value of n to make sure it’s greater than zero. It also validates the two data arrays x and y for proper alignment. The aforementioned sum variables are then calculated using a simple for loop. Following completion of the for loop, the function AvxCalcCorrCoefCpp saves the sum variables to the array sums for comparison and display purposes. It then computes the intermediate values rho_num and rho_den. Before computing the final correlation coefficient rho, rho_den is tested to confirm that it’s greater than or equal to epsilon.

Following its prolog, the assembly language function AvxCalcCorrCoef_ performs the same size and alignment checks as its C++ counterpart. It then initializes packed versions of sum_x, sum_y, sum_xx, sum_yy, and sum_xy to zero in registers YMM3–YMM7. During each iteration, the loop labeled LP1 processes four elements from arrays x and y using packed double-precision floating-point arithmetic. This means that registers YMM3–YMM7 maintain four distinct intermediate values for each sum variable. Execution of loop LP1 continues until there are fewer than four elements remaining to process.

Following completion of loop LP1, the loop labeled LP2 processes the final (1–3) entries in arrays x and y. The vmovsd xmm0,real8 ptr [rcx] and vmovsd xmm1,real8 ptr [rdx] instructions load x[i] and y[i] into registers XMM0 and XMM1, respectively. Note that these vmovsd instructions also zero out bits YMM0[255:64] and YMM1[255:64], which means that the same chain of vaddpd and vmulpd instructions used in loop LP1 to update the intermediate sum variables can also be used in loop LP2 (the scalar instructions vaddsd and vmulsd cannot be used here to update the sum variables without extra code since these instructions set bits 255:128 of their destination operand register to zero). Following completion of loop LP2, each packed sum variable is reduced to a single value using a vextractf128, vaddpd, and vhaddpd instruction, as illustrated in Figure 9-3. The final sum values are then saved to the sums array .

Function AvxCalcCorrCoef_ uses simple scalar arithmetic to compute the intermediate values rho_num and rho_den. Like the corresponding C++ function, AvxCalcCorrCoef_ compares rho_den to see if it’s less than epsilon (a value below epsilon is likely a rounding error and considered too close to zero to be valid). If rho_den is valid, the correlation coefficient rho is calculated and saved. Here are the results for source code example Ch09_04:

Results for AvxCalcCorrCoef

Value        C++    x86-AVX

------------------------------------------

rho:     0.70128193   0.70128193

sum_x:     53081.0    53081.0

sum_y:    -199158.0   -199158.0

sum_xx:   35732585.0   35732585.0

sum_yy:   401708868.0  401708868.0

sum_xy:   -94360528.0  -94360528.0

Matrix Multiplication and Transposition

The C++ source code that’s shown in Listing 9-5 is very similar to what you saw in Chapter 6. It begins with a function named AvxMat4x4TransposeF64 that exercises both the C++ and assembly language matrix transposition calculating routines and displays the results. The function that follows, AvxMat4x4MulF64, implements the same tasks for matrix multiplication. Similar to the source code examples in Chapter 6, the C++ versions of matrix transposition and multiplication are implemented by the template functions Matrix<>::Transpose and Matrix<>::Mul, respectively. Chapter 6 contains additional details regarding these template functions.

Near the top of the assembly language code is a macro named _Mat4x4TransposeF64. This macro contains instructions that transpose a 4 × 4 matrix of double-precision floating-point values. The four rows of the source double-precision floating-point matrix must be loaded in registers YMM0–YMM3 prior to its use. Macro _Mat4x4TransposeF64 uses the vperm2f128 instruction to permute the 128-bit wide floating-point fields of its two source operands. This instruction uses an immediate 8-bit control mask to select which fields are copied from the source operands to the destination operand, as outlined in Table 9-1. Figure 9-4 shows the entire 4 × 4 matrix transposition operation in greater detail. The assembly language function AvxMat4x4TransposeF64_ uses the macro _Mat4x4TransposeF64 to transpose a 4 × 4 matrix of double-precision floating-point values.

Table 9-1.

Field Selection for vperm2f128 ymm0,ymm1,ymm2,imm8 Instruction

Destination Field	Source Field	imm8[1:0]	imm8[4:3]
ymm0[127:0]	ymm1[127:0]	0
	ymm1[255:128]	1
	ymm2[127:0]	2
	ymm2[255:128]	3
ymm0[255:128]	ymm1[127:0]		0
	ymm1[255:128]		1
	ymm2[127:0]		2
	ymm2[255:128]		3

Matrix Inversion

The multiplicative inverse of a matrix is defined as follows: Let A and X represent n × n matrices. Matrix X is an inverse of A if AX = XA = I, where I denotes an n × n identity matrix (i.e., a matrix of all zeros except for the diagonal elements, which are equal to one). Figure 9-5 shows an example of an inverse matrix. It is important to note that inverses do not exist for all n × n matrices. A matrix without an inverse is called a singular matrix.

The inverse of a 4 × 4 matrix can be calculated using a variety of mathematical techniques. Source code example Ch09_06 uses a computational method based on the Cayley-Hamilton theorem , which employs common matrix operations that are relatively easy to carry out using SIMD arithmetic. Here are the required equations:

Toward the top of the C++ code is a function named Avx2Mat4x4InvF64Cpp. This function calculates the inverse of a 4 × 4 matrix of double-precision floating-point values using the aforementioned equations. Function Avx2Mat4x4InvF64Cpp uses the C++ class Matrix<> to perform many of the required intermediate computations, including matrix addition, multiplication, and trace. The source code for class Matrix<> is not shown but included with the chapter download package. Note that the intermediate matrices are declared using the static qualifier in order to avoid constructor overhead when performing benchmark timing measurements. The drawback of using the static qualifier here means that the function is not thread-safe (a thread-safe function can be simultaneously used by multiple threads). Following calculation of the trace values t1 - t4, Avx2Mat4x4InvF64Cpp computes c1–c4 using simple scalar arithmetic. It then checks to make sure the source matrix m is not singular by comparing c4 against epsilon. If matrix m is not singular, the final inverse is calculated. The remaining C++ code performs test case initialization and exercises both the C++ and assembly language matrix inversion functions.

The assembly language code in Listing 9-6 begins with a custom segment that contains definitions of the constant values needed by the assembly language matrix inversion functions. The statement ConstVals segment readonly align(32) 'const' marks the start of a segment that begins on a 32-byte boundary and contains read-only data. The reason for using a custom segment here is that the MASM align directive does not support aligning data items on a 32-byte boundary. In this example, proper alignment of the packed constants is essential in order to maximize performance. Note that the scalar double-precision floating-point constants are defined after the 256-bit wide packed constants and are aligned on an 8-byte boundary. The MASM statement ConstVals ends terminates the custom segment.

Following the custom constant segment is the macro _Max4x4TraceF64. This macro contains instructions that calculate the trace of a 4 × 4 matrix of double-precision floating-point values. Macro _Max4x4TraceF64 requires the four rows of the source matrix to be loaded in registers YMM0–YMM3 and uses the vblendpd, vperm2f128, and vhaddpd instructions to calculate the matrix trace, as shown in Figure 9-6. The vblendpd (Blend Packed Double-Precision Floating-Point Values) instruction merges values from its two source operands according to an immediate control mask. If bit 0 of the control mask equals 0, element 0 (i.e., bits 63:0) from the first source operand is copied to the corresponding element position in the destination operand; otherwise, element 0 from the second source operand is copied to the destination operand. Bits 1–3 of the control mask are used in a similar manner for the other three elements. Register XMM0[63:0] contains the trace value following execution of the vhaddpd instruction.

Blend and Permute Instructions

The C++ code in Listing 9-7 begins with a function named AvxBlendF32 that initializes YmmVal variables src1 and src2 using single-precision floating-point values. It also initializes a third YmmVal variable named src3 for use as a blend control mask. The high-order bit of each doubleword element in src3 specifies whether the corresponding element from src1 (high-order bit = 0) or src2 (high-order bit = 1) is copied to the destination operand. These three source operands are used by the vblendvps (Variable Blend Packed Single- Precision Floating-Point Values) instruction, which is located in the assembly language function AvxBlendF32_. Following execution of this function, the results are streamed to cout.

The C++ code in Listing 9-7 also includes a function named Avx2PermuteF32. This function initializes several YmmVal variables that demonstrate use of the vpermps and vpermips instructions. Both of these instructions require a set of indices that specify which source operand elements are copied to the destination operand. For example, the statement idx1.m_I32[0] = 3 is used to direct the vpermps instruction in Avx2PermuteF32_ to perform des1.m_F32[0] = src1.m_F32[3]. The vpermps instruction requires each index in idx1 to be between zero and seven. An index can be used more than once in idx1 in order to copy an element from src1 to multiple locations in des1. The vpermilps instruction requires its indices to be between zero and three.

The assembly language function AvxBlendF32_ begins by loading the source data operands into registers YMM0 and YMM1 using two vmovaps instructions. The vmovdqa instruction that follows loads the blend control mask into register YMM2. The ensuing vblendvps ymm3,ymm0,ymm1,ymm2 instruction blends elements from registers YMM0 and YMM1 into YMM3 according to the control values in YMM2. The high-order bit of each doubleword element in YMM2 specifies whether the corresponding element from YMM0 (high-order bit = 0) or YMM1 (high-order bit = 1) is copied to YMM3. Figure 9-7 illustrates the execution of this instruction in greater detail. The vblendvps instruction and its double-precision counterpart vblendvpd are examples of AVX instructions that require three source operands. Floating-point blend operations using an immediate control mask are also possible with the vblendp[d|s] instructions.

Following AvxBlendF32_ in Listing 9-7 is the function Avx2PermuteF32_, which demonstrates use of the vpermps and vpermilps instructions. The vpermps instruction permutes (or rearranges) the elements of its first source operand (which is 256 bits wide and contains eight single-precision floating-point values) according to the indices in the second source operand. The vpermilps (In-Lane Permute of Single-Precision Floating-Point Values) instruction performs its permutations using two independent 128-bit wide lanes (i.e., bits [255:128] and bits [127:0]). The control indices for an in-lane permutation must range between zero and three, and each lane uses its own distinct set of indices. Figure 9-8 illustrates the execution of these instructions in greater detail. AVX and AVX2 also include the double-precision floating-point permute instructions vpermilpd and vpermpd.

Data Gather Instructions

The C++ source code in example Ch09_08 includes four functions that initialize test cases to perform single-precision and double-precision floating-point gather operations using signed doubleword or quadword indices . The function Avx2Gather8xF32_I32 begins by initializing the elements of array x (the source array) with test values. Note that this function uses the STL class array<> instead of a raw C++ array to demonstrate use of the former with an assembly language function. Appendix A contains a list of C++ references that you can consult if you’re interested in learning more about this class. Next, each element in array y (the destination array) is set to -1.0 in order to illustrate the effects of conditional merging. The arrays indices and merge are also primed with the required gather instruction indices and merge control mask values, respectively. The assembly language function Avx2Gather8xF32_I32_ is then called to carry out the gather operation. Note that raw pointers for the various STL arrays are obtained using template function array<>.data. The other C++ functions in this source example—Avx2Gather8xF32_I64, Avx2Gather8xF64_I32, and Avx2Gather8xF64_I64—are similarly structured.

The assembly language function Avx2Gather8xF32_I32_ begins by loading registers YMM0, YMM1, and YMM2 with the test arrays y, indices, and merge, respectively. Register RDX contains a pointer to the source array x. The vpslld ymm2,ymm2,31 instruction shifts the merge control mask values (each value in this mask is zero or one) to the high-order bit of each doubleword element. The ensuing vgatherdps ymm0,[rdx+ymm1*4],ymm2 instruction loads eight single-precision floating-point values from array x into register YMM0. The merge control mask in YMM2 dictates which array elements are actually copied into the destination operand YMM0. If the high-order bit of a merge control mask doubleword element is set to 1, the corresponding element in YMM0 is updated; otherwise, it is not changed. Subsequent to the successful load of an array element, the vgatherdps instruction sets the corresponding doubleword element in the merge control mask to zero. The vmovups ymmword ptr [rcx],ymm0 then saves the gather result to y.

Summary