Actions speak louder than words - Digital Knowledge - Column

Communication World, June-July, 1999 by Sheri Rosen

So much for asking. "Survey results showed people's opinions on things - but didn't tell us exactly what was the issue, or why it might be so," said Jodi Freedman, communication specialist at Bose Corp., with headquarters in Framingham, Massachusetts.

To find out how people navigated off the intranet home page or reacted to search engine successes or failures to pinpoint useful information, Freedman chose to watch instead of ask.

The Bose intranet was already in existence when she joined the sound reproduction technology and manufacturing company in 1996. But like many early intranets, this one was grass-roots driven with minimal oversight by a team drawn from several disciplines.

"Our home page had grown up haphazardly and was originally organized by each department. As more and more departments became involved, two things happened: it became challenging to find information, and the home page design - term used loosely; we had no designer on our team - was outgrown. We knew we needed to make changes," Freedman said. "We needed a way to find out what our users thought."

The decision to use observers who documented how people actually used the intranet also served another purpose. "Part of our goal in doing the test was to nurture a culture of webmasters who paid attention to their users." So her team sought volunteers among intranet publishers around the company to be the observers. They received training and materials and were linked up with people who volunteered as testers who would be observed. "Observers made the contact with the volunteer tester, set up a time, went to the tester's desk, and observed and took notes."

Act on Information

Testers embarked on a 30-minute scavenger hunt on the Bose intranet, with three minutes to search for answers to eight questions. "We had two versions of the test to ensure we weren't biasing results through the selection of our questions," Freedman said. As the testers hunted, observers took notes. They recorded demographic information, marked the browsing path from the home page to find each answer, and recorded just how long it took.

"Observers noted browsing paths and terms used in the search engine, and also recorded relevant comments made by the testers, as well as noting their own observations of what they saw," Freedman said. In addition, the testers completed a survey about their experience as soon as they finished the test.

The intranet committee organized comments into categories, identified patterns, prioritized issues, and suggested solutions. "We learned quite a bit! Our home page was too crowded and not an effective information gateway. Our search engine was not very effective - and most people didn't even know it was there. Many people were relying on browsing to uncover information."

The result: a revamped home page that is less crowded. A series of index pages link to the home page. Even these modifications were tested and refined. "Then, we published an article in the employee newsletter and put a Q&A on the intranet to explain what was changing, why it was changing, and when it would happen," she said.

"While some people were unhappy at first that their favorite site hadn't made the home page, most people were pleased to see improvements, and finally to physically see more of the resources available to them online - resources that had been previously buried under departments."

Test Again

It helped, and Freedman has the research to back it up. "We ran our second usability test and found significant improvement in users' ability to find information," she said. "Our intranet has grown significantly and we are now getting close to outgrowing our current structure. So challenges remain."

The observation option is one she'll use again. "Through observation, we have been able to discover patterns in expectations and behavior that we can act upon. So observed results are usable in a way that survey results are not."

Though not expensive, it does take effort. But Freedman is convinced it's worth it. "Communicators should care about the navigation and architecture aspects of usability, because they are just as critical as clear writing and focused graphic design in ensuring that users can get your message and have a good experience," she said. "Most intranets are collaborations between technology people, designers, writers and others. All these people have a stake in creating a usable web."

But there's a trap: it's easy to make assumptions about what users want. "Lots of factors influence how users interact with an intranet - and users are the only ones who can explain what works for them, and what doesn't."

Another trap: measuring what's beyond your control. "Observers focused on the home page, top level navigation, indexes, and the search engine - the corporate-administered structure and tools. We made no attempt to measure the value of the content of the whole site," Freedman explained, because the management group isn't responsible for departmental content. "Measure only what you own and can make improvements on; then make improvements, and test again. Make sure that people understand that you are making improvements based on their suggestions; then you will get more suggestions on future tests."