When you start your computer, the program that starts is the operating system (OS). The OS controls the machine hardware and the programs that you can run on the machine. When you start a program, the OS creates a new process. It means that the OS allocates a new entry in a table (array) where it administers the processes and fills in the parameters that it knows, and needs to know, about the process. For example, it registers what memory segment the process is allowed to use, what the ID of the process is, and which user started from which other process. You cannot start a process just out of thin air. When you double-click on an EXE file, you actually tell the file explorer, which is a program running as a process, to start the EXE file as a separate process. The explorer calls the system via some API and kindly asks the OS to do that. The OS will register the explorer process as the parent of the new process. The OS does not actually start the process, but creates all the data that it needs to start it and, when there is some free CPU resource, then the process gets started, and then it gets paused very soon. You will not notice it because the OS will start it again and again and is always pausing the process repeatedly. It needs to do it to provide run possibilities to all processes. That way, we experience all processes running at the same time. In reality, processes do not run at the same time on a single processor, but they get time slots to run often.
If you have more than one CPU in the machine, then processes can actually run at the same time, as many CPUs as there are. As the integration gets more advanced today, desktop computers have CPUs that contain multiple cores that function almost like separate CPUs. On my machine, I have four cores, each capable of executing two threads simultaneously; so, my Mac is almost like an 8 CPU machine.
Processes have separate memories. They are allowed to use one part of the memory and if a process tries to use another part that does not belong to it, the processor will stop doing so. The OS will kill the process.
The memory handling by the operating system is very complex in addition to separating the processes from each other. When there is not enough memory, the OS writes part of the memory to disk freeing up the memory and reloading that part when it is needed again. This is a very complex, low-level implemented and highly optimized algorithm that is the responsibility of the OS.