On 1/3/20 11:07 AM, Lennart Sorensen wrote:

On Thu, Jan 02, 2020 at 11:36:05PM -0500, Nicholas Krause via talk wrote:

...

Greetings, I mentioned this to some of you at the meeting in December but IBM and others are working on creating a new standard to replace PCI express and some of its limitations called Open CAPI. The current versions is as fast as NVLink at 50GBs in a x16 lane from memory which future versions being able to hit over 100GBs in the same lane speed and 250ms latency guaranteed as compared to the 500ms to 750ms in PCI express.

This is a link:https://opencapi.org/ to anyone who wants to read the spec, through you will have to give them a email and I believe where you work. Currently POWER is the only real architecture supporting it through or NVlink for that matter. In order for that to have any chance of hitting consumer machines it would have to be much better than PCIe due to not being backwards compatible.

AGP solved the serious bandwidth limitations of PCI that video cards needed, but was also a single slot for a single purpose and didn't replace PCI. PCIe replaced PCI since it was much faster, much easier to implement, and had future scalability which PCI couldn't have (a parallel shared bus just isn't going to scale).

Given PCIe is improving over time and stays backwards compatible, being two the speed or half the latency likely isn't enough to make anyone bother. It is but the problem is its hitting the limits currently. PCI Express 5 is near the end of the limits it seems.

So for special purpose like super computers, both nvlink and opencapi have good use cases, but I don't expect to see it in my home PC. It just doesn't add enough versus PCIe to warrent a change. PCIe isn't dead end after all, the way PCI was. That's debatable if you consider the issues with DMA and GPUs. Or PCI Express not being hard aligned for the bytes it requires and having to all of them when accessing it. The problem is similar to SATA for SSDs in that for a lot of cases its fine but actually as a whole in HPC not just supercomputers, SATA is a bad issue outside of cheap storage. Its also not full duplex which is fun in its own right.

Lastly, PCIe has been known to be needed to be replaced at some point being near EOL in terms of scaling. The question is what for both accelerator cards, high end networking and GPUs will be the future. DDR ram has the same issue do to my knowledge.

I do find it interesting that AMD, Google, IBM, Mellanox, nvidia, etc but NOT intel are members.

Of course intel is proposing Compute Express Link (CXL) instead, and has a few of the opencapi members part of their group too (Like Google and HP for example and even AMD). There is also CCIX which is an extension to PCIe to make it batter for super computers and such. Given CXL uses PCIe as its base, I actually expect it to have a much better chance of a future than opencapi.

The problem is that OpenCapi did that and ran into bottlenecks as soon as they went over 50GBs per second or version 3 actually. PCIe has serious limitations and just working around them isn't going to solve it. Just hit the limit of ram in a modern GPU and start going across the PCI bus for each frame and you have issues. What I'm curious about is why try and avoid what seems to be unavoidable in the need of a new standard to replace PCIe as we did not do it with the connectors when creating type C which is going to replace probably all other USB type connectors at some point. Honestly I think it will just tickle down as need be from supercomputers and servers, Nick

On 1/3/20 1:08 PM, Lennart Sorensen wrote:
> On Fri, Jan 03, 2020 at 12:16:53PM -0500, Nicholas Krause wrote:
>> It is but the problem is its hitting the limits currently. PCI Express
>> 5 is near the end of the limits it seems.
> Really?  They have plans for a PCIe v6 next year with double the speed
> again (although it sure sounds like they have to do some complicated
> stuff to get there).
That's the problem is that its hitting end of life. Not sure if it
was PCI v6 or v5 that was the limit or near it. :( It may
scale more but the newer versions run much, much hotter
and use more power. Not sure about the heat issues on
version 6 but its there for SSDs on v4.
>
>> That's debatable if you consider the issues with DMA and GPUs. Or
>> PCI Express not being hard aligned for the bytes it requires and having
>> to all of them when accessing it. The problem is similar to SATA for
>> SSDs in that for a lot of cases its fine but actually as a whole in
>> HPC not just supercomputers, SATA is a bad issue outside of cheap
>> storage. Its also not full duplex which is fun in its own right.
> SATA certainly isn't good for SSDs.  NVMe is much much better.
>
> No idea why SATA was made half duplex while SAS is full duplex.
Its one of the reasons a lot of server people like SAS :).
And yes there is no good reason unless it was a limitation
at the time but that's doubtful.
>
>> Lastly, PCIe has been known to be needed to be replaced at some
>> point being near EOL in terms of scaling. The question is what for
>> both accelerator cards, high end networking and GPUs will be the
>> future. DDR ram has the same issue do to my knowledge.
> Got any links to something showing PCIe has to be replaced?  I haven't
> found any yet.
Its common knowledge for three reasons:
1. The issues with scaling past 100GBs in a lane and PCI express
6 gets there only with extra complexity
2. Latency from and to the bus for DMA or second tier memory
3. Not good common for multi threading environments due to
hardware align not always known when accessing the bus

IBM has decided its the future for POWER and others think
so in HPC. If you want a reason why try encoding and running
out of video ram and having to always go across the PCIe
bus for more memory.  PCI 6 is only 124GB which is only
2 and a half time the speed of version 3 of OpenCAPI.
OpenCAPI if its doubling like PCIe than the next version
will be 100GBs per second at least, maybe faster without
complexity. Its to my knowledge and I could be wrong, able
to hit speeds without complexity of between 200GBs
and 300GBs with better latency.

>> The problem is that OpenCapi did that and ran into bottlenecks
>> as soon as they went over 50GBs per second or version 3 actually.
>> PCIe has serious limitations and just working around them isn't
>> going to solve it. Just hit the limit of ram in a modern GPU and
>> start going across the PCI bus for each frame and you have issues.
> Yes CAPI did run as an extension on PCIe.
>
>> What I'm curious about is why try and avoid what seems to be
>> unavoidable in the need of a new standard to replace PCIe as
>> we did not do it with the connectors when creating type C which
>> is going to replace probably all other USB type connectors at
>> some point.
> Well the USB-C connector may be common but not all ports support every
> protocol.  Some are just USB ports after all.
>
>> Honestly I think it will just tickle down as need be from supercomputers
>> and servers,
> Well we will have to see.  It might happen eventually, if people even
> use desktops at that point.  In a laptop I guess it doesn't matter how
> the GPU is connected since that's only an issue at design time.
> Intel apparently still thinks they can make do with PCIe.  They can be
> pretty stubborn.
PCIe is going to be around I assume until the heat issues are too
much or power issues. Its already there on version 4 but lets
see if they can scale without this becoming more and more of
a issue. Interesting fact is that most PCI 3.0 and 4.0 use the
same wiring actually.

Nick
>