In this blog post, we’ll explain three key techniques for boosting CPU performance—pipelining, memory hierarchy, and multithreading—in a simple and accessible way.

Overview

Speed is a key factor when evaluating a computer’s performance, and one of the primary ways to increase that speed is by improving CPU performance. The CPU plays a role similar to the human brain in a computer system and has become increasingly faster over time. In this post, we’ll examine three widely used techniques for improving CPU speed—pipelining, memory hierarchy, and multithreading—in order.

Pipelining

To understand pipelining, imagine the serving process at a restaurant. If a single person is in charge of serving everything, customers behind the first one must wait while that customer receives their rice, soup, and side dishes one after another. On the other hand, if different people are assigned to serve the rice, soup, and side dishes, multiple customers can be served simultaneously, speeding up the overall process. This is the idea behind the CPU’s pipelining technique.
Inside the CPU, the process of executing a single instruction is divided into several stages. Generally, these include the stage of fetching the instruction (IF: Instruction Fetch), the stage of decoding the instruction and retrieving the necessary register values (ID: Instruction Decode), the stage of performing the actual operation (Execution), the stage of accessing memory (Data Access or MEM), and the stage of writing the result of the operation to a register (Write-back). In a pipeline, each stage is overlapped so that they can process different instructions simultaneously, thereby increasing the number of instructions that can be processed per second.
When a pipeline is implemented, the moment the first instruction moves to the next stage, the next instruction fills that slot and undergoes the IF stage, followed by the next instruction undergoing the IF stage, and so on. Similar to the restaurant analogy mentioned earlier, this method increases throughput by having multiple stages operate in parallel; however, situations involving dependencies between instructions or branches require additional processing (such as bubble insertion and branch prediction), which complicates the design.

Memory Hierarchy

The most time-consuming part of a pipeline is the memory access stage. To reduce this, memory is divided into multiple levels—the memory hierarchy—arranged from the fastest but smallest-capacity level to the slowest but largest-capacity level.
This structure can be likened to the placement of a desk and books. Books you read frequently are placed on your desk; those you read occasionally are on the bookshelf in the room; and books you rarely read are stored in the library. It’s the same sequence as when you need a book: you first look on your desk, then on the bookshelf, and finally at the library if it’s not there. Similarly, in memory, the CPU searches sequentially starting from the fastest (smallest) tier, and if the data isn’t found, it moves down to the next tier to search for it.
As you move down to lower levels, memory capacity increases while access time lengthens. For example, cache memory is located between the CPU and main memory (DRAM), significantly reducing access time. Hard drives and SSDs are slower but provide much larger storage capacity. This hierarchical memory structure is a key design element for optimizing the overall system’s execution speed.

Multithreading and Multicore

Multicore and quad-core processors, commonly seen in advertisements, represent another approach to increasing CPU speed. Multicore technology does not increase the CPU’s internal processing speed; rather, it physically incorporates multiple cores to enable the simultaneous processing of multiple processes. This is similar to a group of people working together to handle multiple tasks in parallel.
Multithreading is a concept that is similar yet distinct. A thread is a unit of execution flow within a process; a single program can be divided into multiple threads to perform various tasks simultaneously. Multithreading can be likened to a single person (process) handling multiple tasks concurrently, which allows for more efficient use of CPU resources and increases overall throughput.
Multicore and multithreading are complementary technologies, both of which require complex support from the operating system, hardware design, and parallel programming techniques. Simply increasing the number of cores or creating more threads does not always result in a linear increase in performance; practical constraints such as the limits of parallelization and synchronization overhead exist.

Conclusion

So far, we have examined the fundamental concepts behind CPU speed improvements, focusing on three key techniques: pipelining, the memory hierarchy, and multithreading. While the explanations were simple, implementing each of these techniques in practice requires numerous additional technologies and complex designs. The combination of these technologies has driven the advancement of computer performance today, and progress toward even higher performance will continue in the future.