Measuring Canadian business school research output and impact

Canadian Journal of Administrative Sciences, Jun 2002 by Erhan Erkut

The Use of Citation Counts as a Measure

While the measurement of research activity has been relatively popular with business academics, most studies have focused simply on the number of papers published, which is an obvious measure of research output. However, some published papers produce no measurable impact on the discipline, while others have a profound impact on the direction of future research. Focusing on paper counts alone ignores such differences. We suggest that a paper-count analysis ought to he accompanied by a citation-count analysis, which offers an objective way of quantifying a paper's impact on the relevant research area.

There is a recognized field of study, called bibliometrics (also called scientific bibliography, informetrics, or scientometrics), that uses statistical and mathematical methods to analyze the literature of a discipline as it is patterned in its bibliographies. There also exists a journal, Cybermetrics: International Journal of Scientometrics, Informetrics and Bibliometrics, devoted to the study of the quantitative analysis of scholarly and scientific communications. Citation analysis is the main tool used in this field. Citation-based measures have been used to evaluate the impact of journals (Harter, 1996; DuBois & Reeb, 2000; Laband & Piette, 1994; Tahai & Meyer, 1999; Vokurka, 1996) and research institutions, including universities, faculties, and departments (Dusansky & Vernon, 1998). For example, they have been used to rank Canadian universities across a range of 20 disciplines (Science Watch, 1996). They have also played a role in personnel decisions, such as hiring, tenure, and promotion (Cameron, 1997). The National Science Foundation (USA) maintains a bibliometrics database to monitor publication and citation trends at the disciplinary and specialty level, and coauthorship or cooperation between countries or sectors.

The central assumption, in citation-based studies such as ours, is that a reference to a particular journal article reflects the scholarly impact of that article on the author of the citing work. While widely used, citation analysis is by no means a perfect measure and must be used with caution. We provide a (possibly incomplete) list of reasons why citation counts may not be reflective of the intellectual impact of research:

1. Classic works in a field may become common wisdom and authors may stop citing them.

2. A paper may be cited for the wrong reason, for example, a paper that misapplies a method may be used as an example of bad research.

3. An author may be citing his/her own work more diligently than the work of others.

4. Cliques may be citing works of the members of the clique to improve the intellectual status of the entire clique.

5. Authors may be citing marginal papers written by potential referees to appease them.

6. Authors may suppress or neglect works of authors with competing theories.

7. Review articles may receive many citations and authors may stop citing papers that were published prior to the review and simply refer readers to the review.