Meta and Panmnesia improve datacenter integration using CXL technology

Panmnesia, a fabless chip design company, has teamed up with Meta to put forward a new vision for AI datacenter design  one where an entire datacenter functions as though it were a single processor. The proposal appears as an invited Review in Nature Reviews Electrical Engineering, part of the Nature Portfolio.

As AI models balloon to trillions of parameters, training runs now depend on hundreds or thousands of accelerators working in concert and moving terabytes of data between them. Because these systems only move as fast as their slowest member, adding more accelerators increases raw compute power but also raises the odds of slowdowns and failures caused by lagging components. A single delayed device can force everything else to idle, and the more inconsistent that delay becomes, the harder it is to predict completion times.

Reducing that inconsistency  and in turn building a bigger, more dependable unit of computation is a challenge the whole industry is wrestling with. Inside a single rack, devices already communicate over dedicated high-bandwidth connections. The harder problem is what happens between racks, a link that today typically depends on general-purpose networking technology like Ethernet or InfiniBand. Each transaction over these networks has to pass through a network interface card and software-managed coordination, and each step adds to the unpredictability. The new proposal targets exactly this inter-rack gap.

The architecture is built on Compute Express Link (CXL), an open standard developed jointly across the semiconductor and infrastructure industries, meaning it isn't tied to any one vendor. Panmnesia and Meta's design applies CXL to cut down on latency inconsistency outside the rack, with the aim of making a whole datacenter behave as predictably as a single chip.

The proposal unifies CPUs, accelerators, and memory into one CXL domain, stretching cache coherence  normally confined to a single rack  across the entire facility. Three pieces of hardware make this possible: a highly-connected, non-blocking switch, a link acceleration unit (LAU), and a fabric controller, all working to keep latency variation in check. Resource placement follows the same design logic normally used inside a chip's own layout. The Review also describes how the CXL fabric's reach can be extended using optical links (CXL-over-optics) to get around the distance limits of electrical signaling, with further detail provided in an appendix.

Using a standard rack setup  one CPU paired with two accelerators as a baseline for comparison, the proposed system lets a single CPU manage eight times as many accelerators, going from two up to sixteen, while the coherence domain that behaves as a unified system can scale up to 960 accelerators, about 13 times larger than the baseline. Data that previously had to leave the rack and cross a network can now travel a fixed path instead, cutting round-trip latency from the microsecond range down to a few hundred nanoseconds  up to a tenfold improvement. It also shrinks the unit that needs replacing after a failure, from an entire server down to just one device. At this scale, tightly linking devices into a single functioning system could enable training much larger AI models without interruptions, or running several workloads at once on shared infrastructure without one affecting the others.

NREE Reviews are invitation-only, with the journal selecting researchers or organizations it considers leaders in a given area. Panmnesia's inclusion as an author reflects Nature's view of the company's technology as representative of the field  making it the first semiconductor startup anywhere to lead an NREE Review, and marking the journal's first Review focused on CXL and AI datacenter architecture.

Panmnesia CEO Myoungsoo Jung said that as AI systems keep growing, being able to link large numbers of accelerators and memory devices quickly and efficiently is becoming as critical as the performance of any single accelerator, and that the research points toward a future for AI infrastructure in which CXL allows a datacenter to function as one unified computing system.