Evaluating the impact of electronic business systems: lessons learned from three cases at the defense logistics agency

Defense AR Journal, Jan-April, 2004 by Jonathan A. Morell

This article synthesizes our experience evaluating three electronic business (eBusiness) systems in the Defense Logistics Agency. The focus was on actual impact in real life operational settings. We summarize our experience in terms of lessons learned and make a case that our experience can help others do similar evaluation. Lessons learned are grouped into six categories: metrics and data sources, methodology, program logic, adaptive systems, realistic expectations, and dependencies among the previous five.

**********

This article synthesizes our experience evaluating the impact of three electronic business (eBusiness) systems in the Defense Logistics Agency (DLA). Our intention is to show the tactics that emerged when general principles of evaluation were applied for the context-specific purpose of determining whether, and how, an eBusiness system is affecting its environment. The first section outlines our emphasis on impact assessment and makes a case for evaluating eBusiness systems. The second section presents lessons learned that were abstracted from our experiences and that can be applied to other, similar evaluation exercises. Finally, we illustrate how the lessons learned were combined to produce impact assessments of particular eBusiness programs.

IMPACT ASSESSMENT--DIFFICULTIES AND IMPERATIVES

Our evaluation activities assumed that programs that have been deployed should have measurable consequences. In this we are firmly rooted in the tradition of evaluation for impact assessment. This view, as summarized in a classic evaluation textbook, states that:

   The critical issue in impact evaluation,
   therefore, is whether a program
   produces desired effects
   over and above what would have
   occurred either without the intervention,
   or in some cases, with an
   alternate intervention. (Rossi,
   Freeman, & Lipsey, 1999, p. 239)

Our core challenge in making such an assessment was the need for a methodology that could produce causal information within a context that from our evaluator's perspective, was totally uncontrolled. We had to evaluate a natural experiment, i.e., a situation in which "... program variants (or other treatments of interest) are not experimentally controlled but vary in the natural environment and in which causal inference is still desired." (Mark, Henry, & Julnes, 2000, p. 265).

In the present case, not only was the situation uncontrolled, but also entirely post hoc. Evaluation did not begin until after the programs in question were well established. As a result of the timing, it was impossible to influence implementation schedules, to anticipate data needs, or to establish data collection mechanisms. Of necessity, the evaluation design was quasi-experimental, an approach defined by Rossi, Freeman, & Lipsey (1999) "An impact assessment in which 'experimental' and 'control' groups are formed by a procedure other than random assignment" (1999, p. 234). Data limitations, however, made it necessary to formulate tactics that went beyond simple comparisons of nonequivalent control groups. Success required knitting together many disparate data sources and analyses. Much of what will be reported below is the story of the search for those sources and the logic and methodologies used to integrate them.

Because of our emphasis on outcome assessment, we did not dwell on process metrics such as percentage of time a system was running, average time to resolve complaints, or number of users. Rather, we focused on whether, because the system was working, there was measurable impact on dollars, quality, time, or readiness. The objective was to determine whether, for operational eBusiness systems, it would be possible to:

* Obtain relevant data.

* Draw conclusions about what the program accomplished.

* Develop practical recommendations to facilitate further evaluation.

The answer was by no means certain because very few eBusiness programs are implemented in a way that is conducive to impact evaluation. To anticipate the later discussion, limitations on IT systems and inter-organizational agreements conspire to constrain evaluation possibilities. We discovered that despite these problems, it was possible to assess impact for each of these systems. This finding gives us confidence (but no guarantee) that impact evaluation can also be conducted on other operational eBusiness systems. By presenting this information, we hope to convey a sensibility about how this kind of work can be done, and thus, to spur more such activity by a larger number of people. At the DLA's request, three eBusiness systems were studied: Electronic Document Access (EDA), Central Contractor Registration (CCR), and the Department of Defense (DoD) Emall.

EDA (http://eda.ogden.disa.mil/ eda_main.htm): The Electronic Document Access Web (EDA Web) combines Internet and World Wide Web technologies with electronic document management to eliminate paper files and facilitate information sharing among DoD communities to provide access to single-source DoD official documents. The information is maintained and available for access to authorized users in Portable Document Format (PDF). Documents included in EDA include contracts and contract modifications, MAAPR (materiel acceptance and accounts payable report), government bills of lading, and DD1716 forms (Contract Data Package).