The number of threads you create really does depend on the nature of the application. For our discussion here, we will consider which the application tends to be – CPU versus IO bound.

Earlier in this chapter (specifically within the sections on Design Motivation and Overlapping CPU with I/O), we mentioned the fact that a thread, in terms of its execution behavior, falls somewhere on a continuum, somewhere between two extremes: one extreme being a completely CPU-bound task and the other extreme being a completely I/O-bound task. The continuum may be visualized like this:

A thread that is a 100% CPU-bound will be continually hammering away on the CPU; a 100% I/O-bound thread is one that is always in a blocking (or wait) state, never executing on CPU. Both extremes are unrealistic in real applications; however, it's quite easy to visualize the domains where they tend to have one of these. For example, domains that involve heavy mathematical processing (scientific models, vector graphics such as flash animations in a web browser, matrix multiplication, and so on), (un)compression utilities, multimedia codecs, and so on will certainly tend to be more CPU-bound. On the other hand, many (but not all) applications that us humans interact with on a daily basis (think of your email client, web browser, word processing, and so on) tend to wait for the human to do something; in effect, they tend to be I/O-bound.

Therefore—a bit simplistically, but nevertheless—this serves as a useful design rule of thumb: if the application being designed is I/O-bound in nature, then creating even a large-ish number of threads that just wait for work is all right; this is because they will be asleep the majority of the time, thus not placing any strain on the CPU(s) (of course, create too many threads and they  do strain memory.)

On the other hand, if the application is determined to be highly CPU-bound, then creating a large number of threads will stress the system (and end up causing thrashing – a phenomenon wherein the meta-work takes longer than the actual work!). Thus, for CPU-bound workloads, the thumb rule is this:

max number of threads = number of CPU cores * factor;
 where factor = 1.5 or 2.

Actually, our discussion has been quite simplistic: many real-world applications (think of powerful web servers such as Apache and NGINX) will dynamically create and adjust the number of threads required based on the exact circumstances, configuration presets, and present workload. Nevertheless, the preceding discussion serves as a starting point so that you can start thinking about design for multithreaded applications.