Forecast: The Perfect Data Storm - Industry Trend or Event

Computer Technology Review, Nov, 2000 by Fred Moore

Terabytes? Petabytes?? And Then...???

It would be too easy to begin this article by saying "as our industry matures" when a deeper look indicates that we may still be at the real creation of the information technology business. How could we actually be fifty years into IT and just be at the beginning? The automobile industry for example is mature with small incremental changes occurring at best. The IT industry however is building the foundation for dramatic changes; some of which are revolutionary representing discontinuities from the past.

We observe that the price per unit of storage purchased is asymptotically approaching zero as it falls between 35-40% per year (see "The Paradox: As Storage And Bandwith All Drop To Zero Dollars, Who Can Make A Buck?", July 2000 CTR). A little later in time, the price of compute power will approach zero per unit purchased as it is falling at 2515% per year. Still further out in the near future, the price of bandwidth per unit purchased is also approaching zero as its price erosion curve now surpasses 20% annually with some expectations for pricing erosion reaching as high as 50% occurring during the next two years. Bandwidth is now growing faster than storage demand using annualized percentages. These curves are rapidly heading downward as they approach a near-zero state. However, the demand curve for these technologies, at least for the next several years, is headed upwards and at times has been nearly vertical. The supply of storage/networking expertise and personnel required to manage this growth is stead ily falling behind the demand.

Along the way, the value of data is increasing on a near vertical curve while the capability of storage management technologies to successfully manage data doesn't keep up with the demand. In addition, bandwidth outside of the computer (optical transmission) is now getting faster than is bandwidth inside the computer (electronic transmission). Current speed limits on the electronic highway (the Internet) will go away in the not too distant future. Plot these conditions on a x/y graph and take a good look at curves going up, down, converging, diverging and crossing over each other. What is going on here? Is this another technological upheaval? No, these are the ingredients for The Perfect Storm.

Like a forecast for any storm, we want to know what the landscape will look like when it passes so that we can plan our next steps accordingly. As the storage devices, computers, and data transport mechanisms become commodities, the value chain moves from those key and required components of the IT foundation to what we will actually do with these components. As an analogy, when the television set first arrived, the value was thought to be in the TV set itself with the frame, knobs, and screen being the items of perceived as having the most value, or at least the most interest. Once the television became a general commodity, the real value was no longer the television set itself, it became what the television delivered: its contents. Out of the Perfect Storm, the same transformation(s) are shaping up.

Computing is not just about computing anymore; it is really about the way we live. We now live on an evershrinking planet that links every place on Earth, though in reality not every person, through the electronic pathways of cable, telephone, satellite, and terrestrial broadcasting. The primary agent of change is the Internet and all of the above mentioned forces are fundamental to its future. Interestingly enough, the Internet has evolved on a global basis without any single entity in charge. Soon, photons or light waves will become the fifth pathway, and we will see a photon circumnavigate the Earth in nearly a tenth of a second.

Companies are more aggressively attempting to move out of the comfort zones of their past business models and move up the value chain. They have to do so to survive. Moving up the value chain as the traditional hardware, compute, and transmission infrastructure becomes commoditized means reaping more value from the data itself. The IT industry is now poised to use the fundamental technologies to store, retrieve, categorize, and deliver the tools necessary to bring meaning to what was previously simply providing access to unstructured data from numerous sources.

A new wave is emerging based on many forms of content that will become understanding and the wisdom of expert advice in electronic form rather than presenting data without interpretation and meaning. This is often referred to as the path to knowledge. The ones and zeros form the DNA that matures into continually richer content. In turn the value of all of the IT processes in the value chain that lead to the creation, management, and delivery of content also grows. The challenge of using electronic information to solve life's problems seems clear enough, but the constantly improving power of the storage, compute, and bandwidth technologies make the path to truly rich content quite feasible. If content is king, then usability is its kingdom.