Towards research performance in the humanities - bibliometrics in qualitative analysis of Flemish law literature - Statistical Data Included

Library Trends, Wntr, 2002 by Henk F. Moed, Marc Luwel, A.J. Nederhof

ABSTRACT

THIS PAPER DESCRIBES A general methodology for developing bibliometric performance indicators. Such a description provides a framework or paradigm for application-oriented research in the field of evaluative quantitative science and technology studies, particularly in the humanities and social sciences. It is based on our study of scholarly output in the field of Law at the four major universities in Flanders, the Dutch speaking part of Belgium. The study illustrates that bibliometrics is much more than conducting citation analyses based on the indexes produced by the Institute for Scientific Information (ISI), since citation data do not play a role in the study. Interaction with scholars in the fields under consideration and openness in the presentation of the quantitative outcomes are the basic features of the methodology. Bibliometrics should be used as an instrument to create a mirror. While not a direct reflection, this study provides a thorough analysis of how scholars in the humanities and social sciences structure their activities and their research output. This structure can be examined empirically from the point of view of its consistency and the degree of consensus among scholars. Relevant issues can be raised that are worth considering in more detail in followup studies, and conclusions from our empirical materials may illuminate such issues. We argue that the principal aim of the development and application of bibliometric indicators is to stimulate a debate among scholars in the field under investigation on the nature of scholarly quality, its principal dimensions, and operationalizations. This aim provides a criterion of "productivity" of the development process. We further contend that librarians are not infrequently requested to provide assistance in collecting data related to research performance assessments, and that the methodology described in the paper aims at offering a general framework for such activities, and can be used by librarians as a line of action whenever they become involved.

1. INTRODUCTION

The study presented in this paper focused on the fundamental questions: How does one recognize a "good" scholar? How does one recognize an "important" scholarly contribution? The approach adopted in this study can be defined as bibliometric. It aims at identifying characteristics of scholarly publications that can validly be assumed to reflect the "quality" or "importance" of a scholar or a scholarly work. Therefore, a first answer to the question "How does one recognize a `good' scholar?" is: One should examine his or her scholarly publications. In other words, in a bibliometric approach, it is assumed that important contributions to scholarly progress are sooner or later communicated in scholarly publications. This is considered to be a universal characteristic of scholarly development in natural sciences, life sciences, social sciences, and humanities.

A bibliometric approach is a quantitative approach. It attempts to calculate statistics of quantitative aspects derived from scholarly publications. Bibliometric indicators result from the statistical analysis of bibliographic information retrieved from the scholarly literature. This determines both their strength and their limitations. The strength of the bibliometric method is that, once established, it can be applied in a uniform or objective manner, eliminating the influence of subjective or personal factors. On the other hand, being a statistical method, it cannot take into account all particularities or special features of the objects to be assessed. As a consequence, bibliometric data should always be applied in combination with qualitative knowledge about the scholars involved and the subdisciplines in which they are active.

Bibliometric indicators have been successfully applied in many subdisciplines in the natural and life sciences. Data from the Science Citation Index (SCI), produced by the Institute for Scientific Information (ISI), play an important role in analyses of research performance in these subdisciplines (e.g., van Raan, 1996; Van Den Berghe et al., 1998). Thus far, the social sciences and humanities have not often been subjected to such analyses. At the same time, the academic authorities of many universities have expressed the need to obtain an insight into the research performance of all faculties and in all fields of scholarship.

Fundamental differences exist between the natural and life sciences, on the one hand, and the humanities and social sciences, on the other hand, with respect to the research object, the methodologies applied, and the structure of scholarly communication. As a result, those who are involved in the development of performance indicators for the humanities and social sciences are confronted with the following situation.

Firstly, they need to develop methodological tools to assist evaluation agencies or policymakers in carrying out their tasks, in the same way that the current SCI-based methodologies provide supplementary research assessment tools in the natural and life sciences. Secondly, this methodology should take into account the characteristics of the field of scholarship, the nature of the scholarly research object, and particularly the communication practices among scholars and the structure of the communication system in their fields. A study dealing with these challenges can indeed be considered an endeavour.