Small world and beyond: A rejoinder

Canadian Journal of Administrative Sciences, Dec 2002 by Erhan Erkut

In David Lodge's (1984) Small World, the Persse character asked, "What do you do if everybody agrees with you?" Arthur Kingfisher's reaction was, "That is a very good question. A very in-ter-est-ing question. I do not remember that question being asked before. You imply, of course, that what matters in the field of critical practice is not truth but difference. If everybody were convinced by your arguments, they would have to do the same as you and then there would be no satisfaction in doing it. To win is to lose the game. Am I right?" "It sounds plausible," said Persse. "I don't have an answer myself, just the question."

I am thankful to the authors of the two commentaries addressing my recent paper (Erkut, 2002), which indicate that, while I may have to ponder a number of different and difficult questions, one question I do not have to worry about is Persse's question above. Both commentaries are thoughtful and contain much fuel for thought. The two commentaries have both similarities and differences. As an example of a similarity, both indicate that the omission of scholarly books from an assessment of research output and impact is a serious one. As an example of a difference between the two commentaries, while Baba (2002) suggests that my study is "an honest attempt to look at scholarship more comprehensively than most studies do," Dery and Toulouse (2002) argue that I provide a "small world depiction" that measures "cognitive output rooted in the classical model of science and a nomothetic epistemology, part of a sociocognitive framework that is subject-based, decontextualized, and published in only one language," which "does not correspond at all to the reality of Canadian research." In the following sections I respond to the comments by the authors, and expand on some of the points raised by them.

Baba (2002)

In his commentary, Baba asks questions about strategic directions for Canadian business schools, and challenges us to "take the next step, to go beyond measurement, and generate knowledge that is empirically sound, theoretically valuable, and practically useful." I agree with many of the points in his commentary, and most of my comments below are not so much a response to Baba as they are an attempt to further engage in this discussion.

Early in his commentary, Baba makes the point that "in matters of scholarship, output is necessary but not sufficient to pronounce judgement as to its quality." This view supports my belief that, when it comes to evaluating research, citation counts are a superior measure to publication counts for several reasons. Studies that use publication counts go to great lengths to differentiate between publications in different journals, either by using a limited set of top-tier journals, or by calculating standardized page counts and then multiplying them by a factor that represents the quality or the prestige of the journal. Not only are such corrections open to debate, but also paper-count studies implicitly assume that journals do not make errors in judging the publishability of papers. Yet anyone who has ever dealt with the refereeing process is aware that it is prone to Type I and Type 2 errors. Despite the efforts of the editors to minimize these errors, it is impossible to completely eliminate them. Citation counts take care of the journal prestige issue, as well as the Type 1 and Type 2 errors. Unfortunately citation counts are not a perfect measure of research impact for the reasons outlined in my study. Nevertheless, I believe they are superior to a paper count metric. Given how much easier it is to use citation counts today than it was in the recent past, one of the goals of my study has been to encourage more extensive use of this metric by both administrators, in assessing research output, and the popular media, in ranking schools. (It is interesting to note that in the Kirkpatrick and Locke (1992) study, they "had to look up all articles and all citation data by hand, which along with list compilation and data analysis took a full-time doctoral research assistant plus two summer clerks more than 13 months to accomplish." While the study reported in Erkut (2002) was very labour-intensive, it did not consume anywhere near that much effort.)

In his commentary, Baba underlines the need for "future research to speculate and debate possible strategic directions at the macro, meso and micro levels." While I hesitate to make any suggestions about what strategies to adopt at the government level, I have some opinions about specialization at the level of institutions and individuals. It seems to me that expecting the same research output from every business school in the country, from every department at a school, and from every individual in a department is an excellent recipe for suboptimal productivity. Research indicators (publication and citation records, as well as research grants) suggest that a small number of Canadian universities produce most of the research. Hence, it seems that there is already some de facto specialization at the university level. It should be noted, however, that Canadian business schools do not have much freedom in this respect since their research agendas cannot stray too far away from the mission of their host universities.