Disaster recovery: regulatory issues

Computer Technology Review, April, 2004 by James Dow

In response to the events of September 11th, the Office of the Comptroller of the Currency, Federal Reserve Board, and Securities and Exchange Commission jointly issued a white paper on September 5, 2002 titled Draft Interagency White Paper on Sound Practices to Strengthen the Resilience of the U.S. Banking System. The purpose of this paper was to solicit commentary from the banking industry on proposed regulatory changes designed to increase the security and resiliency of the banking system in a post-September 11 environment.

The issuance of this paper combined the resiliency requirements for multiple regulatory agencies, which had not been common practice before. Issued to retail banks, clearance and settlement firms, investment banks, technology companies, and state and local officials, the paper received significant response from the collective finance industry. The original draft suggested requirements between 200 and 300 miles for separation between primary and secondary processing facilities, with an expectation of close-to-zero data loss and 2-4 hours recovery time.

The final Interagency Paper on Sound Practices to Strengthen the Resilience of the U.S. Financial System, promulgated four key practices:

* Identify clearing and settlement activities in support of critical financial markets

* Determine appropriate recovery and resumption objectives for clearing and settlement activities in support of critical markets

* Maintain sufficient geographically dispersed resources to meet recovery and resumption objectives

* Routinely use or test recovery and resumption arrangements.

While these key practices are accompanied by definitions and commentary on scope, areas remain that are subject to interpretation. The report used the language "significant distance away from primary" in lieu of an explicit requirement to refer to separation between primary and secondary processing facilities. The final paper differentiates between the larger and smaller financial firms by indicating the guidelines would be applied more stringently to "Firms that Play Significant Roles in Critical Financial Markets." The net result has been that feedback from the financial industry incorporated into the final white paper, and that the resiliency and recovery requirements--from a technology perspective--allow broad discretion on the part of the regulators with respect to their interpretation on an institution-by-institution basis.

The broad latitude afforded the regulators has engendered a condition wherein the Chief Examiners assigned to specific institutions may differ markedly from one another with respect to intent or application of the guidelines. Taking the example of a financial institution with the primary trading workforce located on the island of Manhattan with a primary data center tens of miles away in another state, the guidelines may be interpreted such that little marginal resiliency is required to meet the requirements, inasmuch as the primary processing facility is not located within a key target region (i.e., Manhattan). Alternatively, the guidelines may be interpreted such that significant marginal resiliency, viz., a full-scale secondary data center, may be required to protect the firm from the potential loss of the primary facility, regardless of whether it is in a high-risk geography or not. The crux of the issue is whether the guidelines are interpreted to apply to primary facilities in high-risk geographies such as Manhattan, or primary facilities regardless of geography. The difference in this interpretation can lead to $100MM decisions as one considers the full cost of implementing a secondary data center with appropriate system duplication.

The first key to the current regulatory focus is separation: Separation of primary and secondary processing facilities, separation of user communities from their primary processing to segregate workplace recovery from system recovery, and separation of in-region and out-of-region system recovery. The second key to the current regulatory focus is resiliency: Continued resiliency of books and records (essentially unchanged from previous regulatory guidelines (see Securities Exchange Act of 1934 (Amended), Rules 17(a)(3) and 17(a)(4)); resiliency of intraday processing to reduce exposure to lost or unsettled transactions; and overall resiliency of the financial markets as a whole through an industry-wide synchronized but continually shrinking recovery time objective (i.e., 2-4 hours). The shrinking of recovery times from days to hours, and the requirement to minimize data loss, forces firms to move from tape-based restoration procedures to data replication procedures to ensure the timely availability of data.

The paper specifically addresses risks associated with the likelihood of "wide-scale disruption." Prior to September 11th, most institutions had business continuity and disaster recovery plans based primarily on the likelihood of an institution or facility-specific event. The expansion of planning scope to include a wide-scale event has significant implications for the shared-services disaster recovery market. In the shared-services market, vendors develop system and workplace recovery capacities including system hardware and hot-desk arrangements for workers well short of the total subscription base (i.e., for 100 subscribers, there may only be sufficient capacity to recover 15-20% of the systems and workplaces for 15-20% of the subscriber base, a 1:50 - 1:100 ratio). In the event of an institution-specific disaster (such as loss of a single building, data center, or trading floor), this will provide sufficient capacity to recover the institution. In the event of a wide-scale event, however, 80-85% of the institutions, based on the vendor's business model, may be unrecoverable. This shortfall is inducing some large financial firms to move their recovery capacity in-house as well as driving changes in the shared services vendors to lower overlap ratios, viz., 1:10 - 1:25, which greatly increases the cost base.