So far in this book, we have been taking the term thread for granted. Let's step back for a moment and see exactly what this means—a thread is a sequence of instructions that is executed on a single core of the GPU—cores and threads should not be thought of as synonymous! In fact, it is possible to launch kernels that use many more threads than there are cores on the GPU. This is because, similar to how an Intel chip may only have four cores and yet be running hundreds of processes and thousands of threads within Linux or Windows, the operating system's scheduler can switch between these tasks rapidly, giving the appearance that they are running simultaneously. The GPU handles threads in a similar way, allowing for seamless computation over tens of thousands of threads.

Multiple threads are executed on the GPU in abstract units known as blocks. You should recall how we got the thread ID from threadIdx.x in our scalar multiplication kernel; there is an x at the end because there is also threadIdx.y and threadIdx.z. This is because you can index blocks over three dimensions, rather than just one dimension. Why do we do this? Let's recall the example regarding the computation of the Mandelbrot set from Chapter 1, Why GPU Programming? and Chapter 3, Getting Started with PyCUDA. This is calculated point-by-point over a two-dimensional plane. It may therefore make more sense for us to index the threads over two dimensions for algorithms like this. Similarly, it may make sense to use three dimensions in some cases—in a physics simulation, we may have to calculate the positions of moving particles within a 3D grid.

Blocks are further executed in abstract batches known as grids, which are best thought of as blocks of blocks. As with threads in a block, we can index each block in the grid in up to three dimensions with the constant values that are given by blockIdx.x , blockIdx.y, and blockIdx.z. Let's look at an example to help us make sense of these concepts; we'll only use two dimensions here for simplicity.