PCIe 4/5: Higher Bandwidth, But at What Cost?

Article By : Rick Merritt

PCIe 4.0 and 5.0 doubles the bandwidth, at the expense of signal reach

SAN JOSE, Calif. — Engineers are coming to grips with how much they are willing to pay for the new speeds that PCI Express 4.0 and 5.0 will deliver — and who will get the money. In the end, its likely to be a mix of board makers for premium materials, chip makers for retimers, connector and cable makers for new kinds of channels, and tooling shops for new mechanical designs.

“Speed costs money, so as we go to higher signaling rates, we will see how much people are willing to pay for it and how,” said Michael Krause, an interconnect expert at Hewlett Packard Enterprise.

The good news is that the PCIe is on track to ratify its 5.0 version by next April, delivering up to 32 giga-transfers/second. That will be less than two years since it delivered version 4.0 at 16 GT/s.

Cloud computing is the main driver accelerating a PCIe roadmap that used to double data rates every three, four, or even seven years. Data center networks need faster speeds for their transition to 400-GbE, and a growing chorus of deep-learning accelerators feels the need for speed, too.

The big tradeoff of the higher speeds is that signals won’t travel as far on existing designs. In the days of PCIe 1.0, the spec sent signals as much as 20 inches over traces in mainstream FR4 boards, even passing through two connectors. The fast 4.0 signals will peter out before they travel a foot without going over any connectors.

So system makers are sharpening their pencils on the costs of upgrading boards and connectors, adding chips to amplify signals, or redesigning their products to be more compact.

Retimer chips for a full 16-lane full PCIe 4.0 could cost $15 to $25 — if you can find them. Upgrading an adapter card from Megtron-2 to Megtron-4 materials might only add a dollar or so. However, the cost of a similar upgrade for a motherboard is about $100, and if the upgrade is to even higher quality Megtron-6 it would cost about $300.

“The data center will go to Megtron-4 for PCIe 4.0 and that will add maybe $10 cost — and you may still need retimers,” said Krause. “For version 5.0, people will weigh even higher-cost PCB materials and retimers or move to cables.”

Arnaud Schleich thinks that the situation is even worse. PCIe 4.0 signals are only travelling three to five inches, said the CEO of PLDA Inc., a designer of PCIe controller cores that came out with 4.0 products three years ago.

“We cannot use FR4. We needed to move to Megtron-6; that’s really clear to me. If you want to get more distance, you need to use retimers, and that’s a bit expensive and tricky.”

“What we have been using for 4.0 and expect to use for 5.0 is twinax cables and firefly connectors,” he added. “The cost is very low compared to retimers, you can get whatever you want in distance, and the latency is really good.”

Indeed, Krause noted that “there’s been a lot of interest in using cables … for every inch on a board, you can go 10 inches on cables for the same power and loss budget, but cables have costs in being routed and connected.”

Considering a shift to new mechanical designs

An even bigger shift that engineers are discussing is to new mechanical designs. The industry has thrived for nearly 18 years on the PCIe mechanicals, but some say that it’s time for a change.

A shift would be slow and painful given the market of hundreds of PCIe cards and millions of sockets. The GenZ interconnect group is already exploring new designs that put a connector 60 mm closer to a processor.

“You can recover 4-dB loss at 16 GT/s and 8 dB at 32 GT/s — that can make the difference,” said Krause.

“We’re just starting education on new form factors and new connectors. I would expect a lot of people to stick with what they know and pay the cost of new PCB materials and retimers. I don’t think anyone will change mechanicals quickly, but PCIe Gen 4 or Gen 5 could be the intercept point for some designs.”

The 700-member-plus PCI Special Interest Group (SIG) that sets the PCIe standards expects to have some ideas for dealing with the reach issues within a couple of months. It’s the main issue that the group is grappling with as it prepares a 0.9 version of the 5.0 standard expected by October.

“We’re in the midst of the analysis and simulations of channels,” said Al Yanes, president of the PCI SIG. “We haven’t gotten down to the final distance numbers yet, but that’s the main focus.”

The 4.0 standard supports FR4 boards for all uses but the longest reaches where retimers are needed, said Yanes. Whether boards will have to move off of FR4 to get the 5.0 speeds “is still part of the discussion … some motherboards may only have a couple of 5.0 slots or attach points.”

Many designers are waiting for a new server reference design from Intel to see how the x86 giant handles its first implementation of 4.0. “Gen 4 is not fully mature,” said Schleich of PLDA. “The whole range of solutions is not fully released yet.”

Meanwhile, he already has three customers hungry for the 32-GT/s 5.0 interconnects. They include a storage and a test OEM and one vendor of a multicore AI accelerator.

The fast designs are putting pressure on signal integrity. “Most of our customers will ask support for that even though we are doing the digital controller,” he said.

— Rick Merritt, Silicon Valley Bureau Chief, EE Times<