Of course, we may seek to gain concurrency on the host side by using multiple processes or threads on the host's CPU. Let's make the distinction right now between a host-side operating system process and thread with a quick overview.

Every host-side program that exists outside the operating system kernel is executed as a process, and can also exist in multiple processes. A process has its own address space, as it runs concurrently with, and independently of, all other processes. A process is, generally speaking, blind to the actions of other processes, although multiple processes can communicate through sockets or pipes. In Linux and Unix, new processes are spawned with the fork system call.

In contrast, a host-side thread exists within a single process, and multiple threads can also exist within a single process. Multiple threads in a single process run concurrently. All threads in the same process share the same address space within the process and have access to the same shared variables and data. Generally, resource locks are used for accessing data among multiple threads, so as to avoid race conditions. In compiled languages such as C, C++, or Fortran, multiple process threads are usually managed with the Pthreads or OpenMP APIs.

Threads are much more lightweight than processes, and it is far faster for an operating system kernel to switch tasks between multiple threads in a single process, than to switch tasks between multiple processes. Normally, an operating system kernel will automatically execute different threads and processes on different CPU cores to establish true concurrency.

A peculiarity of Python is that while it supports multi-threading through the threading module, all threads will execute on the same CPU core. This is due to technicalities of Python being an interpreted scripting language, and is related to Python's Global Identifier Lock (GIL). To achieve true multi-core concurrency on the host through Python, we, unfortunately, must spawn multiple processes with the multiprocessing module. (Unfortunately, the multiprocessing module is currently not fully functional under Windows, due to how Windows handles processes. Windows users will sadly have to stick to single-core multithreading here if they want to have any form of host-side concurrency.)

We will now see how to use both threads in Python to use GPU based operations; Linux users should note that this can be easily extended to processes by switching references of threading to multiprocessing, and references to Thread to Process, as both modules look and act similarly. By the nature of PyCUDA, however, we will have to create a new CUDA context for every thread or process that we will use that will make use of the GPU. Let's see how to do this right now.