Processor subsystem interconnect architecture for a large symmetric multiprocessing system

IBM Journal of Research and Development, May-Jul 2004 by Mak, P, Strait, G E, Blake, M A, Kark, K W, Et al

* Fetch Exclusive (CP operand fetch for subsequent store). Fetch which misses local L2.

* Read-Only Invalidate. Spawned from local CP Fetch Exclusive when L2 cache ownership state in requesting book is MC = 1. No data transfer performed, since requesting book already has a copy of the data.

* Least Recently Used (LRU) Castout. Stores aged, changed L2 data to memory on remote book.

With the exception of the Read-Only Invalidate command (which never transfers data), data for remote-fetch-type operations is sourced from a remote L2 cache when IM = 1 on a remote book. If IM = O on all remote books, data is sourced from the target memory book (i.e., the book which contains the memory for the specified address). Note that the target memory book (also known as the memory master, or MM, book) may exist on either the local book or one of the remote books, but there is one and only one target memory book for any given address.

For all remote operations in closed-ring configurations, an incoming ring message is received on each ring, and an outgoing ring message is generated on the same ring. Following receipt of the first incoming ring message, a coherency interrogation of the local L2 directory is performed, yielding a local response. To formulate the outgoing ring response, the local response is merged with the incoming ring response using the response coherency ordering shown in Table 3.

Before giving an example of how this response coherency ordering is used, we provide a brief description of each of the responses listed in Table 3:

* IM Hit - This local response is generated if L2 cache ownership state is IM = 1 and no local IM Reject or MM Reject conditions (described below) are indicated.

* IM Reject - This local response is generated if another remote operation contending for the same address has already indicated an IM hit response.

* MM Reject - This local response is generated if another remote operation contending for the same address could potentially access data at the target memory address.

* Memory Data - This local response is generated as a second response accompanied by data from the target memory book if the local and incoming ring responses at the target memory book are any combination of Read Only Hit, Miss, or No Status (described below).

* Read Only Hit - This local response is generated if the L2 cache ownership state is Read-Only, IM = O, and no local IM Reject or MM Reject conditions are indicated.

* Normal Completion - This local response is generated whenever processing for an LRU Castout operation has been completed at the target memory book.

* Miss - This local response is generated if the L2 cache ownership state is found to be Invalid and no local IM Reject or MM Reject conditions are indicated.

* No Status - This response is launched on both rings of all closed-ring configurations whenever a new remote fetch or remote store command is launched.

As an example of how the coherency-response-merging scheme in Table 3 is applied, if the incoming ring response is No Status and the local response is IM Hit, the outgoing ring response will be IM Hit (i.e., the higherorder response takes precedence). Note that the response forwarded as the first outgoing ring response (i.e., the early or first response) reflects the merged status from coherency interrogations on those books that have received an incoming first ring message on this ring. The response forwarded as the second outgoing ring message (i.e., the final or second response) reflects the merged status from coherency interrogations on all books.