Macroarchitecture vs. Microarchitecture

The term “architecture” can be used in semiconductor engineering to mean one of two things – System Architecture or Microarchitecture.

System Level, or macro-level silicon architectures that are used to define entire chip families, are technically described by different kinds of Instruction Set Architectures (ISA). ISAs describe the way in which instructions are delivered from the programmer to the computer (Thornton, 2018). Because software serves as a conduit between the programmer and the physical hardware it lies on top of, the type of architecture used to build the hardware can have a measurable impact on the type of programs that a processor can run and how the overall system will perform.

Microarchitecture, on the other hand, describes the way an ISA is actually implemented into the hardware design itself. A CPU, GPU, and PMU may all be designed using a single ISA that governs the design of the whole SoC, but still have unique microarchitectures.

Historically, there are two main types of machroarchitectures.

Common Chip Architectures

Von Neumann Architecture vs. Harvard Architecture

The Von Neumann Architecture is a macroarchitecture developed by the physicist and mathematician John Von Neumann in the 1940s and is what most modern computers are based on to this day. The Von Neumann framework relies on three components – the CPU, input/output interfaces, and memory. Inside the CPU, there exist (1) registers, where data and instructions are delivered by and given to the memory; (2) a control unit, which determines which instructions should be executed; and (3) the arithmetic and logic unit, where instructions are carried out and information is actually processed (BBC, n.d.).

The Harvard Architecture has most of the same system components – CPU, memory, and I/O interfaces. The two are different primarily in how they access (input) and distribute (output) information. In a Von Neumann device, the CPU both receives instructions and data from memory and distributes output using the same I/O interfaces (armDeveloper, n.d.). A device with a Harvard Architecture, on the other hand, parses instructions and data into two separate memory banks, with a unique bus for each type of input (Khillar, 2018). By keeping data and instructions separate from one another, Harvard devices can theoretically access both memory and instruction sets at the same time, reducing the clock cycles necessary to perform a single instruction. The difficulty comes in properly timing, or pipelining, instructions and data to get to the CPU at the same time (Khillar, 2018). Because the Von Neumann framework has only a single bus, design is less complex and thus less costly (Khillar, 2018). We can see the differences between Von Neumann and Harvard Architecture visualized in Figure 8-1 and summarized Table 8-1.

Two flow charts. Harvard has 3 blocks named data memory, C P U, and core program memory interlinked with address and data bus. Von-Neumann has 2 blocks named ROM plus RAM and a central processing unit.

Table 8-1

Von Neumann vs. Harvard Architecture

A table has three columns named memory, frequency and clock cycles, data transfer and instruction delivery, chip design, processing type, and real-world applications according to von Neumann and Harvard.

Von Neumann devices account for the vast majority of complex devices, though Harvard Architecture is used in limited cases for microcontrollers and digital signal processors. To be clear, Von Neumann and Harvard Architectures are more theoretical macroarchitectural categories than specific macroarchitecture options chosen from in practice. The vast majority of IC’s use a Von Neumann “type” of machroarchitecture – system designers must weigh the pros and cons to decide which kind Von Neumann macroarchitecture is best for the device they are building.

Instruction Set Architecture (ISA) vs. Microarchitecture

Instruction Set Architectures (ISA) determine the set of instructions that a given processor can support. Microarchitecture, on the other hand, determines how the processor receives and executes those instructions at an implementation level. Another way to think about this is that ISAs provide higher-level design requirements that define which types of instructions a given system must be able to perform, while microarchitecture provides specific, lower-level design guidelines for how to build a system that supports these instructions (Maity, 2022). To be clear, instructions here are lines of code written by a programmer that a computer follows, and not the instructions that came with the bookshelf you bought from Ikea. The goal of these instructions is to get the computer to execute a specific task, or function, and perform the directed operation.

We can better visualize these differences by drawing back to a more detailed version of the macro system stack from Chapter 7 that we will call the universal architecture stack (see Figure 8-2). In our Universal Architecture Stack, we start at the bottom with transistors and move up with increasing levels of abstraction until we reach the application layer containing the user interface that you or I might interact with. This abstraction can be clearly seen at each layer of the stack. At the second level in our stack, transistors from the bottom layer are fit together to form the logic gate building blocks used to enable key computing functions and the registers that serve as an interface between the CPU and peripheral circuitry. Above that, microarchitecture determines how these logic gate building blocks fit together to form the functional modules that constitute the system. The modules shaped by the microarchitecture of individual design teams must abide by the macro-level instruction set architecture in layer 4 that determines which types of instructions the system must be able to perform. Above the bottom four hardware layers, the platform layer containing the operating system is connected to the hardware via firmware. Finally, the middleware layer containing the core back-end software programming logic and application layer containing the user interface run on the operating system from the layer below it.

Each level is dependent on the collection of the levels below it and must be designed to support the levels above it. Whether you’re a student saving a school essay on your laptop or a firmware engineer programming the interface between hardware and operating system, your input ultimately must be carried out by the transistors and functional components that comprise an integrated circuit. The translation process that makes this possible is difficult and requires engineering skill spanning numerous areas and disciplines. Bridging this gap is why abstraction layers are so vital to building high performance systems.

The model of universal architecture stack. Software includes platform, middleware, and application layer. Hardware includes microarchitecture, registers, and transistors is classified into a level of abstraction.

Beyond the challenges faced within systems, another reason these abstraction layers are so important is that they enable hardware designers to maintain compatibility across systems, so that software programs can run on different devices that share the same set of basic instructions. Without standardization of basic instructions, software developers would be unable to build applications that run on devices designed by companies supporting separate instruction sets (ISAs). Imagine the headache if every software company had to create a dozen separate versions of their software to run on each kind of laptop their customers might be using!

To further illustrate this difference, we can think about the distinct ISAs developed by separate companies. The popular licensed ISA ARM Architecture developed by ARM Holdings may be used by multiple companies at once designing the same type of device but with completely different microarchitectures. Samsung and Qualcomm, for example, might both use ARM Architecture to build a custom processor for the next generation smartphone devices, but each has separate implementations, or microarchitectures, of their licensed ARM ISA, each with unique features, levels of performance, etc.

In addition to determining the type of instruction a process will support, ISAs also determine the format and maximum length of each instruction a system can process (Maity, 2022). These underlie the differences between the two main types of ISAs – CISC and RISC.

Instruction Pipelining and Processor Performance

Before delving into differences of CISC and RISC, it’s important that we review the key factors that determine processor performance. These differences are useful to understand in a variety of contexts, but are especially useful in understanding the trade-offs of CISC vs. RISC. From an architectural perspective, there are really three main things we can do to increase the speed of a given processor – we can (1) increase clock frequency; (2) synchronize task performance; and (3) pipeline operations (Engheim, 2021). We’ll work through each in the following paragraphs.

Looking to boost our processor’s performance, we first look for ways to increase clock frequency. As we covered in previous chapters, clock frequency is the number of clock cycles per second and is a common measure of processor performance. A clock cycle measures the amount of time between two pulses of a processor oscillator (device that releases electrical signals) (Intel, n.d.). We can increase clock frequency and boost processor speed by either making our signal move faster or by reducing the distance it must travel to complete the circuit. But how would this work? To help explain, let’s pretend our circuit is the assembly line in a school cafeteria where student lunches are made (see Figure 8-3). For simplicity’s sake, we’ll assume that it’s Taco Tuesday and we start with a single cook on staff who works on each taco plate along a four-step assembly line one lunch at a time.

Four illustrations of architecture of cooking and clock cycles. The people are seated at some distance in four-factor named boost clock frequency, synchronous task execution, pipeline, and regular sitting.

If a clock cycle is the time it takes for a signal to move all the way through a processor and complete an operation, then a “cooking cycle” is the time it takes for a plate to move all the way through the food production line as a completed student meal. To speed up the rate of meal production in our school cafeteria, the first thing we can do is shorten the length of the “cooking cycle.” In the same way that our clock frequency is reduced and processing speed is increased when we reduce a clock cycle, if we can get our school cook to move faster, then our “cooking frequency” will be reduced and our “meal processing” speed will increase. Unfortunately for us, electrons and school cooks can only move so fast, so there are limited ways to boost performance through processing speed alone. Alternatively, we could try shrinking the size of the processor or shortening the length of the assembly line, which would shorten the distance our signal or lunch must travel, reduce our clock or cooking cycle, and increase our processing speed. We can only shrink this distance so far though, after all, our cook must have a place to put the ingredients and there are physical limits to how much we can shrink our processor. Between reducing our clock and cooking cycle and shortening the distance of our critical path, we’ve been able to boost performance by a significant margin, but there’s still more we can do.

The next thing we could do to boost processor performance is incorporate synchronous task performance. To do this, we would need to add another processing core(s) to our processor, sometimes called a co-processor. Each core can now handle separate tasks in sync with one another, reducing the net amount of time required to complete a given task and boosting overall performance. GPUs are particularly well suited for this kind of parallel processing, demonstrating strong performance for applications requiring high volumes of repeatable calculations like machine learning and visual processing.

In our cooking analogy, we could accomplish synchronous task performance by hiring a second cook. Like a pseudo-independent processor core executing a function in parallel to other cores, our new cook could work on a separate, but identical meal line. The combined effort of these two cooks should be able to complete more lunches in a shorter period. Theoretically, it follows that if we wanted to increase the throughput of school lunches, we could simply keep making new cooking lines until our kitchen was completely full. However, there is only one place we can collect money from students and one counter we can deliver the food to them – at some point we will be making taco plates quicker than students are taking them and they will start to pile up. More cooking lines means more costs as well, which may not be possible on a slim school budget. This is similar to the cost and throughput constraints faced by microprocessors, which have only so much “electronic counter space” (registers) available to accept inputs and deliver outputs (to the cache memory). If only there was another a way to speed up processing time that didn’t depend on more cores, while accounting for throughput bottlenecks! Lucky for us, there exists one such method – our third and final option for boosting processor performance, pipelining.

Pipelining is a method of speeding up tasks by breaking up a lengthy workflow into smaller constituent parallel tasks. Unlike synchronous processing that relies on tasks being performed in parallel along separate paths (multicore processors), pipelining breaks up a process into smaller constituent parts performed in parallel along a given path. Pipelining methodology is by no means restricted to microelectronics and is used across a variety of application areas (like cooking!).

Returning to the cafeteria, what if instead of having our school cook make one full meal at a time, we had four cooks, side by side, getting the meal one fourth of the way done, before passing to the next cook standing beside them? It’s taco Tuesday, so the first cook warms the tortillas and arranges them on the plate in the first cook cycle. During the second cook cycle, the second cook throws some chicken on the tortillas, while the first cook warms some new ones. For cook cycle number three, the third cook adds lettuce while cooks one and two repeat their steps. Finally, after cook cycle number four, the fourth cook completes the meal adding some salsa and is ready for the next plate in the following cycle. We may not be able to increase the frequency of a single cycle end-to-end, but we can increase the volume of deliveries in a given period of time.

Turning back to our processor, we can break up our longer, “complex” instructions into four equally sized, “reduced” instructions that each take about a quarter of the time to complete (25% of the clock cycle). After clock cycle one, the first instruction is one quarter of the way there. After clock cycle two, the first instruction is half the way complete, while the second instruction is one quarter done. By clock cycle four, the first instruction is completed in roughly the same time as the lengthier “complex” instruction. However, with a pipeline full of incoming instructions, instead of having one instruction finished every full “complex” clock cycle, one pipelined “reduced” instruction is fully completed every quarter cycle. Herein lies a key advantage of RISC over CISC – its reduced instructions are far better suited for pipelining, providing performance advantages over its CISC counterparts.

Whether we’re talking about tacos or electrons, pipelining can significantly improve our processing throughput, enabling us to finish more lunches or instructions per cycle than before (Engheim, 2021). This can prove to be a substantial advantage, giving one ISA an edge over another depending on the application.

CISC vs. RISC

The two main types of ISAs are CISC (Complex Instruction Set Computing) and RISC (Reduced Instruction Set Computing) (Thornton, 2018). Common processors that used CISC architectures include Motorola 68k, PDP-11, and several generations of Intel x86, while notable processors that use RISC ISA’s include ARM, RISC-V, MIPS, PowerPC, and Atmel’s AVR (McGregor, 2018). There are numerous differences between RISC and CISC, though the most notable is the way in which they process instructions.

Each processor has a clock cycle that limits the number of instructions that it can process over a given period of time. Conceptually, you can think of clock cycles like a metronome or a heartbeat – each beat is a window of time for the processor to perform a given task. To understand how this works, it is important to remember what is actually happening when a computer executes an instruction – it is converting software “input” into physical signals (patterns of electrons) and running these signals through the circuit, where they are manipulated and processed before being converted back into some sort of useful software “output.” A clock cycle is a single pulse of “instruction signal” coursing through the processor circuitry. The number of clock cycles a processor can run per second (clock speed) determines how fast it can process instructions. The greater the “clock cycle frequency” (clock frequency), the faster the processor. CPUs can run millions of clock cycles per second, notching processing speeds of millions or even billions of hertz (cycles per second) (Intel, n.d.).

The key difference between CISC and RISC processors is that RISC processors break down instructions into smaller segments that can run one at a time, while CISC processors run instructions that take more than one clock cycle to execute, hence their names Reduced Instruction Set Computing vs. Complex Instruction Set Computing (Engheim, 2020). At first glance, our intuition might tell us that CISC processors must be better since they can run more units of instruction per unit clock cycle. However, for many of the processor performance factors outlined in the previous section, RISC is often seen as a more effective option for many applications. Unlike CISC processors which use lengthy, multi-clock cycle instruction sets, RISC processors break up instructions into smaller, standardized instruction sets that are better optimized for instruction pipelining and more easily digested by compilers. Pipelining is a complicated scheduling operation that must ensure both the data and the instruction necessary to execute a given task or operation arrive at the CPU at the proper time. RISC processors aim to maintain an instructions-to-clock cycle ratio of 1-1, ensuring a steady and predictable clock cycle, which significantly improves processing speed (Engheim, 2020).

The reason for these differences is rooted in semiconductor history. CISC came about in 1970, followed by the first RISC prototypes about a decade later (IBM, n.d.). This early on in the industry’s development, compilers were unreliable and programmers often wrote directly in assembly code. Compilers translate high-level programming languages like Java and C# to machine-level language a computer can understand, while assembly languages are much closer to the hardware and have many disadvantages to compiled languages, including increased complexity, difficulty of use, and reduced portability (Pedamkar, n.d.). As compilers improved, RISC architecture was more widely adopted by design teams across the industry.

Because they process simplified instructions, RISC processors tend to consume less power, making them ideal for applications where power is particularly limited, such as mobile phones or other battery-powered devices. This advantage is reduced in higher-performance applications, like servers or personal laptops, which tend to use CISC processors more often.

Though more efficient in many ways, RISC processors do require more RAM (to access additional code) and greater programming efficiency (shorter instructions means more lines of code) (Bisht, 2022). This used to be a significant disadvantage when memory chips were a lot more expensive in previous decades when RISC architecture was just gaining traction, but has become much less so as memories have become smaller and less expensive (Teach Computer Science, 2021). The differences between CISC and RISC are summarized in Table 8-2.

Table 8-2

CISC vs. RISC

A table has three columns named clock cycle, instruction length, instruction volume, pipelining, memory usage, and design orientation according to C I S C and R I S C.

Choosing an ISA

Choosing an ISA is a difficult and consequential decision. Many ISAs are licensable, like those offered by MIPS and ARM Holdings, though some are proprietary or open source, like RISC-V. Typically, a licensed ISA will come with a pre-designed processing core, while an open-sourced ISA does not (McGregor, 2018). Licensing fees and royalties are key factors in deciding whether to license, build, or borrow.

Often more important than direct costs are the risks that come with each ISA. Depending on which is chosen for a particular design, each will come with advantages and disadvantages. To start, engineering leadership must consider the time and costs of physically developing a core processor in-house. Most ISA licensing companies license processors that serve as the core of any customized end-product. Even if a unique core processor design is completed within a feasible time frame, there are always manufacturing risks that can cause a new processor and architecture to fail.

Perhaps even more important are the downstream software implications. ISAs like ARM and x86 have mature software “ecosystems” with fully developed software development stacks. Building a new processor with a proprietary architecture requires the development of new firmware, operating systems, and development tools (Hill et al., 2016). Even if hardware and software execution is seamless, time-to-market is likely to be an issue. While your company was spending its time developing a core processor and proprietary architecture, the competition was releasing new products.

One question to ask yourself is this: are customers buying the processor, or are they buying what it does. If your customers are buying your new algorithm or integrated processor and sensor system, then licensing an existing ISA may be most attractive. But if the processor itself forms the core of your business or adds some unique value not available in the market, then a custom design may be the right answer. A table comparing the relative risks of different ISA choices is shown in Figure 8-4.

A table has 4 columns of I S A trade-offs. The row is named hardware costs, software engineering costs, existing software ecosystem, time-to-market, manufacturing risk, design flexibility, and royalty fees according to proprietary, licensed, and open source I S A.

Heterogeneous vs. Monolithic Integration – From PCBs to SoCs

Historically, shrinking transistor sizes due to advances in manufacturing and lithographic technologies has enabled semiconductor design companies to continue increasing the performance of their devices without having to pay much attention to device architecture and integration (Gupta & Franzon, 2020). There was enough “wiggle room” so that even if a design included millions of unnecessary transistors, it could still deliver greater performance using less power and space at a lower or equal cost. Even if a company was motivated to make its chip more efficient, the additional design effort required to make fully integrated devices was costly and the rapid pace of Moore’s law made time-to-market a critical constraint. In many cases, by the time a fully integrated device was created, the next generation of manufacturing technology would already have well surpassed whatever the design team could have provided in additional efficiency.

The trend of shrinking transistor sizes for increased performance, often referred to as geometric scaling, continued unabated until the last decade or so, when the industry ran into three main problems (Gupta & Franzon, 2020).

First, as transistors became smaller and logic devices became denser, power management issues became a primary design constraint for advanced systems, displacing frequency as the dominant factor (Gupta & Franzon, 2020). Modern CPUs and other advanced devices often cannot use their full “firepower” since the heat required to run at such speeds would literally fry the circuits in question.

Second, Lithographic Technologies like EUV depend on shrinking light wavelengths that are thin enough to etch increasingly tiny die features (IRDS, 2020). Though these technologies may have sustained geometric scaling to this point, researchers have had trouble finding light wavelengths that can effectively etch smaller features (Brown et al., 2004).

Finally, the thickness of the materials that transistors are made from is now just a few atoms thick with little room for further shrinking (Gallego, 2016). Bluntly put, there won’t be enough atoms to make usable feature patterns below a certain threshold.

In addition to these three physical constraints, each successive technology node requires disproportionally more expensive process technology, which increases the cost to build new fabs and drives up manufacturing unit costs. With some modern fabs costing as much as $20 billion, manufacturing chips using the most advanced process nodes is incredibly expensive, which is pushing designers to squeeze more performance out of older nodes (Lewis, 2019).

In sum, as Moore’s law has slowed down in recent years, the industry has shifted its focus from geometric scaling to functional scaling, boosting performance by optimizing designs for specific applications and by shifting system architectures to include more heterogeneous and monolithic integration (Gupta & Franzon, 2020).

In heterogeneous integration, numerous chips are integrated with one another either on the same board (PCB) or within the same package, called a system-in-package or SiP (Lau, 2017). For board-level heterogeneous integration, different chips and components are soldered to the board and wired to one another. For SiP, different chips and functional modules are enclosed in the same package and connected to one another via interconnects or through silicon vias (TSVs) that connect chips stacked on top of one another using 2.5/3D die stacking technology. By keeping functional components separate, system architects and designers are able to better “plug-and-play,” while still enjoying the performance advantages of a more tightly integrated system.

In monolithic or homogenous integration, numerous functional modules are included on a single integrated circuit, yielding a fully functional system called an SoC (System-on-Chip) (IRDS, 2020). The more integrated a system is, the less distance a signal must travel to reach other parts of the chip. However, fully integrated systems like SoCs are complex to design and have numerous drawbacks that must also be considered.

Across the four PPAC dimensions – performance, power, area, and cost – heterogeneous (SiP) and homogeneous (SoC) integration have significant trade-offs. Smaller form factors (device area) give SoCs significant advantages in area and power efficiency, making them popular choices for smaller, battery powered devices like cell phones. They can suffer on performance, however, depending on the application. Each functional portion of a chip may require different materials and process technologies for peak performance, which is difficult or impossible to do on a single wafer (IRDS, 2020). By integrating all components on a single wafer, like in an SoC, some parts may function very well, while others perform poorly since they were made with materials and processes that are not optimized to their individual requirements (IRDS, 2020).

Cost differences also play a big role in whether to use monolithic or heterogeneous integration. On the one hand, greater levels of integration require more design work and thus have higher design costs and greater manufacturing complexity (Gupta & Franzon, 2020). This may not lead to lower unit costs for heterogeneous devices in all cases, however. Monolithic devices require less area in aggregate, which enables more chips to be printed on a single wafer and can lower net manufacturing costs (Gupta & Franzon, 2020). At the same time, heterogeneous integration enables a sort of “manufacturing arbitrage,” where different process nodes may be used for different parts of the system. In a SiP, for example, advanced modules like memory or core logic could be manufactured using the most advanced 3nm process nodes, while less advanced parts, like analog or RF components, could be manufactured using a 130nm process node (IRDS, 2020). By only manufacturing some components using more advanced process nodes, companies can save considerably on portions of the system that use the older, cheaper technology. They can also build a next-generation system by only changing the memory and logic devices to enable more features, while keeping power management or RF components unchanged. Engineering executives must take great care to balance the additional design costs and manufacturing cost differences when deciding which architecture to employ.

We can see these trade-offs played out in Figure 8-5. Monolithically integrated SoCs consume less power and take up less space than heterogeneously integrated systems integrated at the board level. System-on-Boards (SOBs), however, have greater design flexibility, lower design costs, and can be designed more quickly. System-in-Packages straddle between the two, offering greater design flexibility while reaping higher performance, power, and area advantages due to greater integration.

An illustration of heterogeneous versus monolithic integration. The system-on-chip, system-in-package, and system-on-board in five factors named lower power, smaller area, lower, design cost, greater design flexibility, and quicker time to market.

In addition to the core PPAC factors, system architects and design teams must keep in mind time to market – as a rule of thumb, the more integrated a system is, the more design time is needed. If a competitor is pressuring your team with a new product release, perhaps it’s best to shy away from designing a new SoC from scratch.

Chapter Eight Summary

In this chapter, we first broke down the difference between macroarchitectures and microarchitectures – system-level Instruction Set Architectures (ISAs) describe how data and instructions are delivered to and received from a core processor while microarchitectures describe how an ISA is implemented in a given circuit. We next compared Von Neumann and Harvard macroarchitectures. While in theory, Harvard Architecture can speed up processing by enabling a CPU to retrieve data and instructions from two separate memory banks at the same time, in practice, increased bus complexity and other factors have limited its performance. Von Neumann architecture limits data and instruction exchange between the CPU and one memory bank, reducing design complexity and lowering costs, and has been the most common architecture used in practice since its inception in 1945.

In the second half of the chapter, we teased apart the differences between CISC and RISC. First arising in the mid-1970s, CISC predates the widespread adoption of RISC by almost a decade and a half (RISC was invented in 1980 but wasn’t widely used until the early 1990s). While each has its pros and cons, RISC is widely seen as an improvement due to its pipelining advantages. Moving on from broad macroarchitecture categories, we explored the various ISAs and how to choose between them. Finally, we asked a key consideration facing silicon design teams today – heterogeneous or monolithic integration? Each approach has its advantages and drawbacks. While monolithic ICs like SoCs require less power and take up less area, initial design can take months if not years, which can be prohibitively high. Heterogeneous integration, on the other hand, enables us to capture some PPAC advantages, while retaining design flexibility and quicker time-to-market.

Your Personal SAT (Semiconductor Awareness Test)