The fall of Von Neumann architecture
Imagine a chef that is effectively superhuman - they can chop, cook, or mix anything in the matter of one second flat. Unfortunately, this chef has a terrible memory and has to check the recipe book to remember the ingredients infrequently. They have to walk down a long hallway to a room where the recipe book and ingredients are stored. Now the rules of the kitchen also imply that only one person or item can travel down the single lane hallway at a time.
Imagine the chef is making a complex dish where a lengthy-step recipe is being cooked up. Every step is different so the chef has to consequently check the recipe book for every action. An example order of operations would look like the chef walking down the hallway to the room where they read step one of the recipe and grab the ingredients and then walking back to the kitchen to execute that step (no bringing the recipe book to the kitchen lol, explanatory purposes). The chef then executes that step. They then walk back to the recipe room to read step two from the book and consequently walk back to execute that second step. The chef repeats this until all the steps are performed from the recipe book.
This is highly ineffective. To understand and perform EACH and every step, the chef has to walk to and fro the recipe room to the kitchen.
For 80 years, computer engineers have tried to make the chef run faster, tried building a small mini-fridge next to the cutting board (cache), and tried to guess what ingredient the chef might need next.
Now along comes artificial intelligence where a complex recipe looks different → instead of a sequential ordering of tasks, running an AI model looks more like “Dice 10,000 potatoes in a row, each in the same way.”
When we try to use our Von Neumann chef here to dice these 10000 potatoes, the chef has to walk down the hall to read the recipe (“dice a potato”) and grab a potato, walk back and dice it, and then walk back again to put it back in the room. We have to repeat this process 9999 more times.
Two main problems happen here:
- The chef’s hands are doing nothing for 98% of the time because they are preoccupied with simply walking. Giving the chef faster hands (aka a faster CPU clock speed) does nothing because they still have to walk the hallway. This is the speed bottleneck
- Walking the long hallway back and forth that many times burns a massive amount of calories compared to simply chopping a potato. In a real silicon chip, moving a number across the wire from memory to the processor burns over 1000 times more electricity than actually multiplying the number. Your laptop consequently gets blazing hot and your battery dies → not from doing math, but from moving numbers back and forth repetitively. This is the Power / Battery bottleneck
Step in GPUs, TPUs, and NPUs.
Every modern AI chip is essentially a re-design of this kitchen layout because someone needs to read the complex and changing recipes.
The CPU is the head chef. It reads unpredictable, branching recipes on the fly.
The GPU stops reading the recipe one potato at a time. A single expeditor calls out “DICE!” over a loud speaker or intercom and a bunch of cooks execute that exact same dice simultaneously while an elevator dumps crates of potatoes onto their stations. This way, massive throughput is achieved, but serious power is also consumed.
The TPU and NPU are built for pure flow. The chefs stand shoulder-to-shoulder with their tools locked in place. Order of ops here would look like chef 1 makes a cut and passes the ingredient straight into chef 2’s hands. Nobody takes a look at the recipe book and nobody walks down the hallway until the final dish comes off this line.
In more technical terms, the CPU maintains serial control and the classic kitchen layout because branching software requires infrequent trips to Instruction Fetch.
The GPU amortizes the recipe check. One instruction is broadcast across a large amount of workers (CUDA cores / threads) who all perform their task at the same time, fed by high-bandwidth memory.
The TPU and NPU rewire the kitchen into a pipeline where each station holds its weight stationary, and those raw ingredients stream directly from one worker’s hands to the next (Systolic Array). Intermediate results never get to the hallway (DRAM) until the entire layer is done.
Now, if TPUS and NPUS are so much more efficient than GPUS, how is NVIDIA still in business? Why do GPUS still dominate?
Because there is a tradeoff between efficiency and flexibility.
Lets go back to the potato factory line (the TPU / NPU) where the chefs hold their in-place tools next to each other. As long as you are cutting something (doing something that system is built for) that system is unbeatable. It uses less electricity and runs nonstop.
But what happens when instead of cutting, we want to peel each time? Our system is cooked. This is the power of adaptability, and is the core strength of a GPU. The TPU/ NPU workers can only cut, not peel, so now we have to create a whole new architecture for peeling. But for the GPU, because each process is adaptable, we can instruct them to change their task and simply peel.
Technically, if this type of task isn’t supported by the NPU compiler, the task gets sent back to the CPU where it is executed far slower.
Sooo who actually wins the war for kitchen efficiency? Ultimately, the fall of the Von Neumann architecture hasn’t concluded with one single replacement chip - it’s forced a truce across multiple simultaneously.
The entire restaurant works together - you can’t just fire the head chef (the cpu) just because you bought a french fry factory line (the npu), and you don’t get rid of your adaptable prep machine (the gpu) while the menu is changing every week.
We’ve now entered the era of heterogeneous computing, post Von-neumann era - when you train a frontier model in the cloud, GPUs and TPUs do the heavy lifting. And when you deploy that model on the edge - whether on a laptop, a phone, or a board in a robot, you freeze the recipe, quantize the weights, and lock it into an NPU so you can run that algo with constrained power while the CPU executes the rest.
The best engineers of this decade won’t just be people who know how to train a neural network, they will be the ones who understand how to shape their software so it never gets stuck in the hallway.