Quick summary
A processor repeatedly fetches operands from memory, computes and writes results back. Arithmetic can happen in a tiny local circuit, while data may cross caches, on-chip networks, memory controllers and external memory. Charging those wires and activating memory structures consumes energy, so moving a value can cost more than the operation performed on it.
Why a wire costs energy
A digital transition charges or discharges electrical capacitance in transistors and interconnects. Energy grows with capacitance and the square of voltage. Longer wires and larger structures usually have more capacitance, and signals may need repeaters or drivers. Clocking, control and leakage add further cost even when useful data are not changing.
The memory hierarchy
Registers and small caches sit near computing units and are fast but limited. Larger caches are farther away. Dynamic random-access memory (DRAM) provides capacity outside the processor package, and storage lies farther still. Each level trades capacity and cost against latency and energy. Good software reuses data locally so an expensive transfer supports many calculations.
Why AI exposes the bottleneck
AI workloads apply many operations to large arrays of weights and activations. Accelerators can provide enormous arithmetic throughput, but idle if data do not arrive quickly enough. High-bandwidth memory, wider links, compression, sparsity and tiling all try to feed computing units while reducing unnecessary movement.
Compute near or inside memory
Near-memory computing places processors close to memory, shortening links and increasing bandwidth. Processing-in-memory performs selected operations within or beside memory arrays. Analogue approaches may use physical current accumulation for matrix operations, while digital designs place conventional logic near memory banks.
The tradeoffs
Memory technologies are optimized for storage, not every kind of calculation. In-memory systems may have limited precision, device variation, conversion overhead and restricted operations. Moving work into memory complicates programming, scheduling and consistency. A specialized design wins only if reduced traffic outweighs these costs.
Reality check
“Data movement costs more than computing” is workload- and technology-dependent, not a universal constant. Energy varies by chip, voltage, memory level, data type and access pattern. Peak operations per second do not reveal how efficiently a real application moves data or how much time hardware sits idle.
How to evaluate an efficiency claim
Look for energy per completed task, accuracy, memory capacity, utilization and measurements including data conversion and off-chip transfers. Compare against a strong baseline on the same process technology and workload. System-level results matter more than an isolated array demonstrating exceptional efficiency.
