AMD - Zen chitchat

[watercooled]

Again, that's not true. The CCX modules communicate with each other, the IO hub and the memory controller within the die over IF, and hence the Fabric is very much a part of the CCX blocks. The external IF obviously requires its own PHY blocks, but IF isn't exclusively external on the Zeppelin die. I'll link the AMD block diagram later when I'm not typing on my phone.

But like I say, a fabric is not just a block, however the PHYs for external communication are obviously a block, but they are not one and the same as the Fabric. They are just one layer of the protocol.

[Corky34]

No, they really don't. They use the SDF, IF is the nomenclature given to the scalable data fabric, the control fabric, the IFOP and IFIS, the coherent socket extender, the cache-coherent master, and loads of other things. A CCX block does not communicate over the SDF plane, it can't as if it did you'd end up with different data in different parts of the L3, it's the job of the Cache-Coherent Master to do the communication.

The point is there's no such thing as 'A' infinity fabric as it's made up of multiple different things.

[CAT-THE-FIFTH]

Well considering Intel has EMIB,I would expect they might eventually do something similar. 10NM seems to only work so far for small chips,which from what I gather are close to the size of these "chiplets" or are a bit larger.

Intel has a presentation in December - I wonder if they will go the same way??

[watercooled]

No, they really don't. They use the SDF, IF is the nomenclature given to the scalable data fabric, the control fabric, the IFOP and IFIS, the coherent socket extender, the cache-coherent master, and loads of other things. A CCX block does not communicate over the SDF plane, it can't as if it did you'd end up with different data in different parts of the L3, it's the job of the Cache-Coherent Master to do the communication.

You're completely sidestepping the statement I made and spouting a load of random acronyms which are irrelevant here. The CCX blocks communicate over an implementation of IF exactly as I said.



See the path between the CCX, IO and IMC? Yeah, the one labelled 'Infinity Fabric'...

The point is there's no such thing as 'A' infinity fabric as it's made up of multiple different things.

No, there isn't such thing as 'a' fabric i.e. it's not just 'a block' which can be omitted from a theoretical chiplet - the 'a' fabric cannot exist solely on the Northbridge-like chip, that's ridiculous. This Northbridge needs some way to communicate with the chiplets, and guess what they will use for that! That's right, IF, just like they used to communicate between CCX, IMC and IO on Zeppelin. Check AMD's own slides.

[watercooled]

I'm home and a bit less hangry now (working late) and I think I get what you were trying to say but the acronym soup wasn't digesting in that state of mind. I think you were saying the CCX doesn't talk directly to the IF as it goes via the CCM? The CCM is an integral part of the IF, hence my point - and just to reiterate, the fabric isn't just this logic block, it's a sum of its parts, of which the CCM, CAKE, etc are all parts as you can see in that fairly concise diagram above (straight from AMD themselves).

You previously said

Strictly speaking IF is only the pink sections

which is what I was responding to - that part is untrue, the pink parts are essentially the PHYs/SerDes (called IFOP) for the Fabric i.e. parts which let the fabric physically/electrically connect across dies or packages. The PHYs allow other dies/packages to connect to this 'fabric', they are not the fabric itself.

Edit: I've also just noticed my phone keyboard had seemingly auto-capitalised 'fabric' which was meant as a generic term in that context (not as in Infinity Fabric), hence 'a fabric' is grammatically correct. Corrected for clarity.

[watercooled]

And yeah I guess the io chip is sorta an on-package Northbridge - just make a few different versions for each market, as required?

Northbridge gets separated out again

It really does add another dimension to modularity without having to tape out so many 7nm designs - the same chiplets could potentially be used in everything from EPYC to consoles, just Lego up the blocks and bin the chiplets as needed, and maybe build a custom 'Northbridge' for the target market on a cheaper node.

I imagine at would allow the IO chip and the chiplets to run on a separate cadence, too. I'm not sure if there's any real advantage to that but it could allow e.d. core updates without needing IO updates, and vice-versa, allowing e.g. things like PCIe/DDR updates without needing to change the cores.

arggh the forums double posted when i wanted to edit it

Given those "zen cores" are 2x in core size (4 to 8) 16c ryzen is a done thing i believe

Core = 1 physical processing core, appearing as 2 logical cores to the OS due to SMT.
CCX = a group of cores with shared L3 cache e.g. 4 cores in Zeppelin, and likely either 4 or 8 with Zen2.
Chiplet = the physical pieces of silicon containing 1 (if 8 cores/CCX) or 2 (if 4 cores/CCX) CCX blocks and any ancillary stuff necessary to connect it to the IO die i.e. IF PHY.

In answer to Corky's question, I wouldn't expect most of those highlighted blocks (i.e. PCIe, IMC, Southbridge, etc) to appear in a chiplet given that's the reason for actually producing separate chiplets, besides some sort of PHY to connect to the IO chip as I said above (and similar to what's highlighted in pink in that picture), but don't forget there are other parts that aren't highlighted in those simplistic block diagrams.

[Corky34]

I'm home and a bit less hangry now (working late) and I think I get what you were trying to say but the acronym soup wasn't digesting in that state of mind. I think you were saying the CCX doesn't talk directly to the IF as it goes via the CCM? The CCM is an integral part of the IF, hence my point - and just to reiterate, the fabric isn't just this logic block, it's a sum of its parts, of which the CCM, CAKE, etc are all parts as you can see in that fairly concise diagram above (straight from AMD themselves).

That will teach me to read more than a single post, i had a long reply to the post before this one and now that's unnecessary.

Yes the CCX doesn't talk directly to what i guess we could loosely call the IF, i mean it could be argued that it's not really the IF as when it's on a single die it's just a collection of wire traces that connect one thing (block) to another but i guess on the level of a single die IF can be used to describe the SDF, SCF, and everything that connects different parts.

Personally i wouldn't put to much earnest in a logical diagram as they tend to misplace things and move things around.

You previously said which is what I was responding to - that part is untrue, the pink parts are essentially the PHYs/SerDes (called IFOP) for the Fabric i.e. parts which let the fabric physically/electrically connect across dies or packages. The PHYs allow other dies/packages to connect to this 'fabric', they are not the fabric itself.

I think untrue is a bit strong. What i was trying to get at is they were the closest thing to what we assume Infinity Fabric to be, that's not to say Infinity Fabric is not part of a single die as all the bit's an bob's included in a single piece of silicon could well be considered to a part of that, i guess what i was getting at was is the SDF, SCF, CCM and all those other thing really a part of IF when they're contained in a single unit.

Does the notional idea of Infinity Fabric end when looking at a single piece of silicon, does it then become a collection of bit's an bob's connecting to each other via wire traces.

[watercooled]

Typing on phone again so forgive any random autocorrect errors.

Yeah the way I see it, the Fabric is a collective term for the the logical interface, kinda like PCIe I suppose. There are different ways of physically connecting to PCIe e.g. over the typical motherboard interfaces, but also over cables such as DP and IIRC some optical ones too.

Similarly IP can be carried over Ethernet cables, or over VDSL, optics, coax, satellite, etc. The web browser isn't particularly bothered which though. It's an overly simplistic analogy but can't think of a better way to put it. The electrical interfaces don't define the protocols running on them.

Infinity Fabric, like some similar fabric technologies, is the link between logic blocks on the processor, but when moving from on-die to external interfaces, you need some sort of block to add things like error correction/line codes such as 8b/10b, buffers, maybe step up voltages, etc.

[DanceswithUnix]

..., i mean it could be argued that it's not really the IF as when it's on a single die it's just a collection of wire traces that connect one thing (block) to another ...

No, that can't be argued. Two things can be connected with wires but they still need some communication protocol over those wires. More than two things can be connected with a MUX if they are a simple master/multi-slave setup, or you need a connection bus or fabric.

[Corky34]

Sure but AFAIK we don't know what protocol is being used within the silicon itself, we know it uses a SerDes when exiting and leaving a single die but one would assume that's not what's being used within the die itself as if it was there'd be no need for everything coming in and out to pass through a CAKE.

If as you're suggesting it carries on within a single silicon die where do you believe the serialized data is converted to parallel, and if my guess is right and you're suggesting the serialized / parallel nature isn't relevant as it's the actual protocol being used that's important then what's encoding and decoding the data packets as the HyperTransport™ I/O Link Specification document states that...

The HyperTransport™ link is designed to deliver a scalable and high performance interconnect between CPU, memory, and IO devices. The HyperTransport link uses low-swing differential signaling with on-die differential termination to achieve high data rates:

If I'm reading that right HT is an interconnect between devices and requires termination at some point.

EDIT: Just to clarify: IF is not a protocol it's an umbrella term AMD came up with to describe the two main fabrics, what I'm questioning is if that term still applies when talking about only the data or only the control fabric and where, in the engineering sense, is the cut off point? Personally I'd say at the IFOP and ISOP SerDes as that's (afaik) when the signals from the two fabrics come together or are split in two.

[watercooled]

The SerDes blocks are endpoints of an electrical link, not the protocol itself; they are responsible for converting it from one 'format' to another, hence my modem analogy. Similarly, you can send Ethernet frames over cat5, coax, twinax, multimode fibre, single mode fibre, radio links, etc - it's all still Ethernet as far as the protocol is concerned. Just, where twinax might work well for say 100GBe over a few metres between machines in the same room, you need to look towards fibre and maybe coherent optics to send it across the Atlantic.

The electrical interfaces at the edges of the silicon are not the end points of Infinity Fabric, rather they're endpoints of that longer-range 'link', similar to how an optical signal might be changed back to electrical at an SFP module in a switch, and maybe on to another copper link to an adjacent machine. The Ethernet protocol doesn't start and end at the SFP though.

Inside the chip, to my knowledge, the Fabric acts as a crossbar between the nodes on the die. WRT the data and control planes, they're separate even outside of the silicon AFAIK - the control plane interface sits inside the Southbridge block.

[Foxdragon]

Does the Zen behave like a system on a chip or a normal CPU?

[Corky34]

Yea, that part i agree with but we're not (afaik) just talking about a single protocol. Aren't the control and data protocols distinctly different from each other, they're certainly transmitted on two electrically separate planes within a die.

I just spotted this short sentence on wikichips with regards to the control fabric "The SCF has its own dedicated IFIS SerDes that allows the SCF of multiple chips within a system to talk to each other. The SCF also extends to the dies on a second socket in multi-way multiprocessing configurations" So in essence the data and control communications are always separate from each other.

Because the control fabric SerDes was never shown in any (most?) diagrams and images i had assumed it shared the wires with the data link.

That's kind of disappointing as that means there's no single thing we could call Infinity Fabric as it's two entirely separate things.

[watercooled]

That's what I was getting at originally, there is no one thing you can point at and say 'thats the IF'. And back to the original debate, the chiplets will have a SerDes, similar to IFOP blocks, to communicate with the NorthBridge. They need that because they obviously have to extend the fabric off the die to be able to communicate with the NorthBridge, you can't just strip the fabric out because then there would be no way to communicate between CCX blocks on-die (if there are multiple) and between different dies on-package. On-package blocks don't have to be as big despite doing much the same thing, due to smaller distances involved, as you can see in the highlighted die shot you posted.

The off-package blocks will be on the NorthBridge die, and that's the whole reasoning behind doing this - those large off-socket SerDes blocks take up a heck of a lot of die space and don't scale well with smaller nodes so it makes absolute sense to do what AMD are doing. This is precisely the sort of modularity that AMD have been working towards with IF and I very much doubt they'll be the last to do this. Even Intel are going multi-die on Xeon now!

[DanceswithUnix]

Yea, that part i agree with but we're not (afaik) just talking about a single protocol. Aren't the control and data protocols distinctly different from each other, they're certainly transmitted on two electrically separate planes within a die.

I don't believe they are two different competing things, just a representation that a fabric has more than one job to do. We live in a complicated and messy world, and often the best engineering solutions are the ones that reflect that rather than the ones that are nice and simple.

Really I think you are over thinking this. AMD have a fabric that connects blocks together within a die, which they can also drive between the die of an MCM and between sockets in a multi socket system. It goes through conversion blocks along the way. Unless you work in AMD in the right department, you aren't going to pull it apart more than that.

For example:

Because the control fabric SerDes was never shown in any (most?) diagrams and images i had assumed it shared the wires with the data link.

I have yet to see anything meaningful written about the control fabric. It's all hand waving, broken links and irrelevances. However, any connection system requires things like boot configuration and out of band signals like interrupts. Getting that off the critical path of data handling makes a lot of sense, so I presume that is what they are doing. I suspect in this case the control plane just isn't considered interesting enough to document. Much like in a Ferrari people don't write much about the electric motors that wind the windows, they just concentrate on the big one that drives the wheels.

[Foxdragon]

Does ryzen function more of a system on chip or more general cpu?