A CUDA context is usually described as being analogous to a process in an operating system. Let's review what this means—a process is an instance of a single program running on a computer; all programs outside of the operating system kernel run in a process. Each process has its own set of instructions, variables, and allocated memory, and is, generally speaking, blind to the actions and memory of other processes. When a process ends, the operating system kernel performs a cleanup, ensuring that all memory that the process allocated has been de-allocated, and closing any files, network connections, or other resources the process has made use of. (Curious Linux users can view the processes running on their computer with the command-line top command, while Windows users can view them with the Windows Task Manager).
Similar to a process, a context is associated with a single host program that is using the GPU. A context holds in memory all CUDA kernels and allocated memory that is making use of and is blind to the kernels and memory of other currently existing contexts. When a context is destroyed (at the end of a GPU based program, for example), the GPU performs a cleanup of all code and allocated memory within the context, freeing resources up for other current and future contexts. The programs that we have been writing so far have all existed within a single context, so these operations and concepts have been invisible to us.
Let's also remember that a single program starts as a single process, but it can fork itself to run across multiple processes or threads. Analogously, a single CUDA host program can generate and use multiple CUDA contexts on the GPU. Usually, we will create a new context when we want to gain host-side concurrency when we fork new processes or threads of a host process. (It should be emphasized, however, that there is no exact one-to-one relation between host processes and CUDA contexts).
As in many other areas of life, we will start with a simple example. We will first see how to access a program's default context and synchronize across it.