Processor subsystem interconnect architecture for a large symmetric multiprocessing system

IBM Journal of Research and Development, May-Jul 2004 by Mak, P, Strait, G E, Blake, M A, Kark, K W, Et al

Table 3 shows that the IM Hit response is the highest-ordered ring response. This means that for any remote book which observes a local IM Hit response, the final responses returned to the requesting book will be IM Hit responses. For remote fetch operations requiring data transfer (e.g., Fetch Read-Only), a local IM Hit response also ensures the generation of a special intermediate data response on one of the rings. The intermediate data response is always transmitted on the ring which results in the shortest path between the book sourcing the data and the requesting book. Intermediate data responses may precede or follow an IM Hit first response, but always precede an IM Hit second response. The intermediate data response does not factor into the coherencyresponse-merging scheme shown in Table 3.

For diagonally opposite books (which exist only in four-book closed-ring configurations), the incoming first response and the incoming second response are merged with the local response to formulate the outgoing message on both rings. This process is called diagonal book accumulation. Diagonal book accumulation ensures that all incoming second responses reflect the merged status of coherency interrogations on all books in four-book closed-ring configurations. Intermediate data responses sourced by any adjacent book in a four-book closed-ring configuration are returned to the requesting book in the minimal number of hops. (That is, the direction of the intermediate data response at an adjacent book will always be on the ring opposite to the first incoming ring message.) Note that an intermediate data response sourced by a diagonal book can be sent on either ring, since the paths between the data source book and the requesting book on the two rings are equidistant.

A critical component of the IBM eServer z990 ring coherency management system is the establishment of two system coherence points known as the intervention master (IM) and memory master (MM) coherence points.

As described earlier, an IM = l L2 cache ownership state indicates that this book was the most recent to receive cache ownership for that address, and that there can be only one book with IM = 1 for a given address. Remote operations which generate an IM Hit local response simultaneously set a token called IM Pending.

Activation of this cache-based token permits the operation which set the token to proceed while any subsequent contending requests for the same address are rejected via the IM Reject response described earlier. In general, the IM Pending token is set if

* L2 cache or memory data is to be sourced from a book.

* L2 cache or memory data or an IM Hit ring response is in the process of passing through a book (as a result of being sourced from another book).

* The cache ownership states for a Read-Only Invalidate operation are currently in the process of being updated.

* An LRU Castout operation is being processed at the target memory book, or store data for an LRU Castout operation is passing through a book (en route to the target memory book).