The Impact Factor-Based Quality Assessment of Biomedical Research Institutes in Iran: Effect of Impact Factor Normalization by Subject

 

Nasrollah Rezaei-Ghaleh MD*, Fereidoun Azizi MD^•*

 

 

Authors’ affiliation: *Endocrine Research Center, Shaheed Beheshti University of Medical Sciences, Tehran, Iran.

•Corresponding author and reprints:   Fereidoun Azizi MD, Endocrine Research Center, Shaheed Beheshti University of Medical Sciences, Tehran, Iran. P.O. Box: 19395-4763, Tel:+98-212-240-9309, Fax: +98-212-240-2463, E-mail: azizi@erc.ac.ir.

 

Accepted for publication: 28 February 2007

 

Background: The journal impact factor is increasingly employed to evaluate the quality of scientific research. This is in contrary to its critical limitations, e.g., its marked variation across scientific disciplines. This study was conducted to describe, in both quantitative and qualitative terms, the contribution of Iranian universities and research institutes to biomedical publications in 2004 and to examine the possible effect of impact factor normalization by subject on their rankings.

Methods: The Iranian biomedical articles were recruited from Thomson Science Citation Index Expanded and PubMed databases according to Thomson Scientific subject categorization of journals. Three subject-normalized impact factors were employed: rank-normalized impact factor, journal to field impact score, and standardized impact factor.

Results: Tehran and Shiraz Universities of Medical Sciences and University of Tehran were the top three institutes regarding the sum of impact factor in 2004. On the basis of the mean crude impact factor, Sharif University of Technology, University of Shiraz, and Baqiyatollah University were ranked as the first to third. However, the subject normalization of impact factor made some considerable changes in impact factor-based rankings of research institutes.

Conclusion: The Iranian scientific community and science development policy makers are recommended to employ subject-normalized impact factor, rather than crude impact factor, in quality assessment of biomedical research held in various academic and research institutes.

 

 Archives of Iranian Medicine, Volume 10, Number 2, 2007: 182 – 189.

Keywords:  Biomedical research · Iranian universities · journal impact factor · quality assessment · subject normalization

 

 

Introduction

 

 he quality assessment of the academic and research institutes are increasingly recognized as critical components of development programs.¹ Evaluation of quality in scientific research is a complex problem. Frequently, systematic review of research outputs by experts in the field is considered as an ideal solution of this problem.² The use of more objective scientometric indices in research evaluation emerged in the 1960s and 1970s.^{3 – 5} These  scientometric  indicators,  among  which the most common one is probably the journal impact factor (IF), are increasingly employed to evaluate the quality of scientific research performed by individual scientists, research groups, or institutes.^{6 – 8} The journal IF, first introduced by Garfield in 1960s, is a measure of the frequency with which the average article in a journal has been cited in a particular year.⁴ This indicator, calculated annually by the Thomson Scientific (formerly Thompson ISI), is derived by dividing the number of citations in year 3 to any items published in the journal in years 1 and 2 by the number of substantive articles published in that journal in years 1 and 2.⁴ In Iran, the use of journal IF has recently been advocated to evaluate the quality of research held in universities and research institutes.⁹

In spite of its obvious advantages including conceptual simplicity, and the fact that it provides a convenient method to assess the impact of journals and articles more immediately than citation itself, the employment of journal IF to evaluate the quality of research has been widely criticized.^{2, 10 – 12} Some of these criticisms, however, are more related to the ISI policy in calculation of IF (e.g., its 2-year citation window) than the overall concept of IF. Among the most critical shortcomings of journal IF is that it strongly varies across different scientific disciplines.^{2, 5} Several factors have been found to contribute to such differences among fields of research: variations in the intrinsic citation pattern, the dynamic or stable state of the field, etc.¹³ For example, biochemistry and molecular biology articles are cited about five times as often as pharmacy articles.¹⁴ Several methods have been proposed to normalize journal IFs by their subject,^{15 – 20} among them the rank-normalized journal IFs and “journal to field impact score” (JFIS) are two examples.^{15 – 16}  

In recent years, Iran has substantially increased its presence in world science.^{21, 22} This growing trend is evident in the annual number of scientific articles published by Iranian researchers in international journals.^{21, 23 – 25} It has been reported that Iran experienced the most prominent rise in the number of scientific publications among the Middle East countries.²³ Especially, this upward trend has been observed in biomedical publications.^{24 – 25}

The universities and research institutes of Iran seem to contribute unevenly in science production process. In the present study, the contribution of Iranian universities and research institutes to international biomedical publications in 2004 is described, in both quantitative and qualitative terms. Specifically, the aim of this study is to investigate how the normalization of journal IF by subject may affect the IF-based ranking of Iranian institutes active in biomedical research.

 

Materials and Methods

 

Subject categorization of journals

The subject categories of journals provided by Thomson Scientific were employed. The journals are classified into various subject categories on the basis of an indicator called relatedness index. This index is calculated for each pair of journals on the basis of the amount of citations between them. According to the subject categorization of journals done by Thomson Scientific, some journals may be assigned to more than one subject categories.^{26 – 27}

 

Normalization of journal IFs by subject

To factor out the inherent variation of journal IF among different disciplines,¹³ three normalized indicators were calculated:

1) Rank-normalized IF (rnIF): Each journal was assigned its rnIF after calculation of its fractional rank (in increasing order) within its related subject category. So, rnIF varies between 0 and 1 and journals with the highest IF within their related categories were described by rnIF equal to 1.¹⁵

2) JFIS: The journal IFs were divided by the aggregate IFs calculated for their related subject categories.¹⁶ The aggregate IFs, provided as part of the Journal Citation Report (JCR), reflects how many times the average article of the field is cited.

3) Standardized IF (SIF): A novel method was also employed to normalize journal IFs by their subjects. Since the journal IFs were found to obey, globally and within the subject categories, a log-normal distribution, the following transformations were made to fit the observed IF distribution across subject categories: First, the journal IFs were logarithmically transformed and a new normally-distributed variable was obtained. Second, using the mean and standard deviation of this new variable, the standardization process was made and identical standard normal distributions were reached for SIFs for all subject categories. This normalized indicator could so reflect the relative position of each journal in its own related category, hence, making it possible to compare journals from various categories.

 

Recruitment and classification of Iranian biomedical articles

The articles from Iranian researchers published during 2004 in biomedical research fields were recruited from the Science Citation Index Expanded (SCI-Expanded) database of Thomson ISI Web of Science, through limiting the search to biomedical subject categories. The joint publications with researchers from other countries were also included in this study. To avoid any missing of the Iranian articles due to possible errors in ISI indexing, the PubMed database was also searched for Iranian articles. In PubMed, articles were considered to be from Iran only if “Iran” was mentioned in the affiliation address of the first author. The published Iranian articles were then assigned to diverse scientific subfields, according to the subject category of the publishing journal. If the journal had been assigned to more than one category, the category with most relevance to the specialty of the first author was chosen. In case of any doubt, the category with the highest normalized IF for that journal was selected. The quality of articles published by the country was then evaluated in terms of crude IFs and the three above-mentioned normalized IFs.

 

IF-based ranking of Iranian academic/research institutes and statistical analysis

The Iranian biomedical articles published in 2004, recruited from either ISI SCI-Expanded or PubMed databases and categorized by their subjects were attributed to various universities and research institutes on the basis of the affiliation address of the author. The universities and research institutes were then ranked on the basis of the mean crude or subject-normalized IF of journals publishing their articles. The rankings of Iranian research institutes on the basis of crude and normalized IFs were finally compared through calculating the Pearson’s and Spearman’s correlation coefficients between them.

 

Results

 

The total number of biomedical journals studied was 3,842, attributed to 69 different subject categories. The SIF of these journals, calculated as described earlier, exhibited an excellent correlation with their rnIFs, with Pearson’s and Spearman’s correlation coefficients being 0.946 and 0.992, respectively. The correlation of SIF with JFIS was also strong, although it was relatively weaker than rnIF (Pearson’s and Spearman’s coefficients were 0.745 and 0.955, respectively). The observed correlations were strong enough to justify the employment of SIF in parallel to rnIF and JFIS in the present study.

The total number of Iranian biomedical articles published in 2004 was 1,172 from which 876 articles were published in journals with a calculated IF. Table 1 presents the descriptive statistics of crude and normalized IFs related to journals publishing Iranian biomedical articles in 2004. The maximum crude IF belonged to an article by A. Gharehbaghian of the Iranian Blood Transfusion Center, published in The Lancet with an IF of 21.713. This article was a joint publication with the Transplantation Sciences group of the University of Bristol. The rnIF of this journal was 0.98 and its JFIS and SIF values were 5.078 and 2.599, respectively. The second and third high-impact articles were published in Neuron and EMBO J, with crude IFs of 14.439 and 10.492, respectively. The mean and median crude IF of journals publishing Iranian articles in 2004 were 1.663 and 1.392, respectively. The journal IFs showed a significant positive skewness and kurtosis. The journal IF distribution could be satisfactorily described by a log-normal distri­bu­tion (Figure 1), with scale and shape parameters of 1.322 and 0.701, respectively.

 

 

 

The mean ± SD rnIF for journals publishing Iranian biomedical articles in 2004 was 0.471 ±  0.252 (Table 1). The mean ± SD JFIS was 0.670 ± 0.446 and the corresponding value of SIF was (‑0.108 ± 0.771). In agreement with each other, these three indicators reveal that Iranian articles were published in 2004 in those journals slightly weaker than the average journal of the field.

Table 2 demonstrates the top 20 Iranian universities according to the sum of their IFs in 2004. This criterion represents simultaneously both the quantitative and qualitative aspects of biomedical publications. Obviously, Tehran University of Medical Sciences with the sum of IF of 313.9 is far higher than other Iranian universities. This is almost 22% of the total IF the Iranian biomedical articles achieved in this year. Regarding this criterion, Shiraz University of Medical Sciences and University of Tehran had the second and third ranks.

To evaluate the quality of research, the Iranian universities and research institutes were separately ranked on the basis of their mean crude IF in 2004 (Table 3). Sharif University of Technology, University of Shiraz, and Baqiyatollah University were the top three universities, in respect with the mean crude IF. Among the research institutes, the Institute for Advanced Studies in Basic Sciences (IASBS, Zanjan), Institute for Theoretical Physics and Mathematics (IPM) and National Research Institute for Tuberculosis, affiliated to Shaheed Beheshti University of Medical Sciences, were respectively located in first to third positions.

To take into account the marked variation of journal IFs in different disciplines, the subject-normalized IFs (rnIF, JFIS, and SIF) were calculated as described earlier and the universities and research institutes were again separately ranked according to these normalized variables. Tables 4 to 6 demonstrate the ranking of institutions on the basis of rnIF, JFIS and SIF, respectively.

 

 

 

 

 

 

 

 

The rankings changed considerably when the journal IFs were normalized by their subjects and University of Tabriz, Islamic Azad University, and University of Ferdowsi, Mashhad, were recognized as the top three universities. Among the  research institutes, Digestive Disease Research Center, Tehran University of Medical Sciences (DDRC,  TUMS), National Research Center for Genetic Engineering and Biotechnology (NRCGEB), and Iran Polymer and Petrochemical Institute were moved to top ranks after IF normalization by subject.

To compare the rankings of Iranian universities according to the crude and subject-normalized IFs, the Pearson’s and Spearman’s correlation coefficients between them were calculated (Table 7). The Pearson’s correlation coefficients between the crude IF and rnIF, JFIS, or SIF were 0.577, 0.657, and 0.640, respectively. The corresponding Spearman’s coefficients were 0.694, 0.716, and 0.734, respectively.

 

 

Discussion

The quality of biomedical research held in Iranian academic and research institutes in 2004 was evaluated in terms of the crude and subject-normalized IF of journals publishing their articles. It was shown that the subject normalization of IF influences the IF-based ranking of Iranian research institutes.

Various methods of IF normalization by subject have been proposed in recent years.^{15 – 20} For example, the journal IFs may be normalized through dividing the IF by the maximum IF in the field or the average IF of highest-rank journals, the aggregate IF, the mean IF, the median IF, or any other point indicator of IF distribution in the field.^16-18 Alternatively, the subject normalization of journal IFs may be performed after the IF-based ranking of journals in their related subject categories.¹⁵ Needless to say, all these methods disregard the variation of journal IF  distribution in

different subject categories, hence, they cannot reflect the real position of journals within their categories. On the contrary, the SIF seems to represent well the situation of each journal within their subject categories. Using this subject-normalized IF, the inter-category comparison of journal IFs becomes possible.

After calculating the mean of crude and subject-normalized IFs for each Iranian research institute in 2004, these institutes were ranked according to each of these four indicators. Obviously, the rankings changed considerably when the journal IFs were normalized by their subjects. While Sharif University of Technology, University of Shiraz, and Baqiyatollah University were the top three universities in respect to the mean crude IF, after subject normalization of journal IFs, University of Tabriz, Islamic Azad University, and University of Ferdowsi, Mashhad, were recognized as the top three universities. Similarly among the research institutes, the DDRC, NRCGEB, and Iran Polymer and Petrochemical Institute were moved to top ranks after IF normalization by subject.

To quantify the correlation between various IF-based rankings of Iranian biomedical research institutes, Pearson’s and Spearman’s correlation coefficients between them were calculated. The Pearson’s coefficients between the crude IF-based rankings and three subject-normalized IFs were lower than 0.66 and the Spearman’s coefficients were below 0.74. The presently introduced indicator of SIF exhibited strong correlation with both JFIS and especially rnIF, which is known to behave superior to some of the other known indicators in terms of IF subject normalization¹⁵. The calculated correlation coefficients can be taken to suggest that the subject normalization of IF may considerably influence the ranking of Iranian universities.

In Iran, there is a growing tendency to use the journal IF to evaluate the quality of research performed by individual scientists or research institutes.⁹ The increasing awareness of journal IF and its possible use in evaluation are now reshaping Iranian scientists’ publication behavior towards publishing in journals with higher IFs, often at the expense of apparently low-impact specialist journals. However, the results of this study, along with several other studies,^{2, 10 – 12} indicated that the straightforward use of the crude journal IFs in quality assessment of research may be quite misleading. Based on the obtained results, it is suggested that the Iranian scientific community and science development policy makers should be cautious in using the crude journal IFs in research quality assessment. Instead, employing a subject-normalized IF, which may provide a more realistic picture of research quality, is recommended.

It is noteworthy that only biomedical articles published in journals have been included in this study. So, institutes which often publish their articles in nonbiomedical journals should cautiously be judged according to the results of this study. Such institutes like IASBS (Zanjan), IPM, Sharif, or Amir Kabir University of Technology may have qualitatively dissimilar situations in fields other than biomedical area. Therefore, the results of this study should be regarded to reflect only the situation of biomedical research held in various institutes.

 

Acknowledgment

 

This study was supported by a grant from the Endocrine Research Center, Shaheed Beheshti University of Medical Sciences. The assistance of Dr. A. Etemadi and Mr. M. Khorsand is gratefully acknowledged. 

 

References

 

1         Chen DHC, Dahlman CJ. Knowledge and development: a cross-section approach. In: Chen DHC, Dahlman CJ, eds. World Bank Policy Research Working Paper Series. 2004. Available from: URL: http://info.worldbank.org/etools/docs/library/1173 33/37702_wps3366.pdf.

2         Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ. 1997; 314: 498 – 502.

3         Garfield E. Citation indexing for studying science. Nature. 1970; 227: 669 – 671.

4         Garfield E. Citation analysis as a tool in journal evaluation. Science. 1972; 178: 471 – 479.

5         Garfield E. Journal impact factor: a brief review. Can Med Assoc J. 1999; 161: 979 – 980.

6         Taubes G. Measure for measure in science. Science. 1993; 260: 884 – 886.

7         Calza L, Garbisa S. Italian professorships. Nature. 1995; 374: 492.

8         Vinkler P. Evaluation of some methods for the relative assessment of scientific publications. Scientometrics. 1986; 10: 157 – 177.

9         Development and Coordinator of Research and Technology Center, Iranian Ministry of Health and Medical Education, I.R. Iran. The monitoring forms for evaluating the research performance in Iranian universities of medical sciences. Available from: URL: http://www.hbi.ir/index.htm.

10      Colquhoun D. Challenging the tyranny of impact factors. Nature. 2003; 423: 479.

11      Glanzel W, Moed HF. Journal impact measures in bibliometric research. Scientometrics. 2002; 53: 171 – 193.

12      Bordons M, Fernandez MT, Gomez I. Advantages and limitations in the use of impact factor measures for the assessment of research performance in a peripheral country. Scientometrics. 2002; 53: 195 – 206.

13      Vincler P. Possible causes of differences in information impact of journals from different subfields. Scientometrics. 1991; 20: 145 – 161.

14      Narin F, Hamilton KS. Bibliometric performance measures. Scientometrics. 1996; 36: 293 – 310.

15      Pudovkin AI, Garfield E. Rank-normalized impact factor: a way to compare journal performance across subject categories. In: Pudovkin AI, Garfield E, eds. Proceedings of the 67th Annual Meeting of the American Society for Information Science and Technology.  Rhode Island, USA: Springer; 2004.

16      van Leeuwen THN, Moed HF. Development and application of journal impact factor measures in the Dutch science system.  Scientometrics. 2002; 53: 249 – 266.

17      Sen BK. Normalized impact factor. J Document. 1992; 48: 318 – 329.

18      Marshakova-Shaikevich I. The standard impact factor as an evaluation tool of science and scientific journals. Scientometrics. 1996; 35: 283 – 290.

19      Sombatsompop N, Markpin T, Premkamolnetr N. A modified method for calculating the impact factors of journals in ISI journal citation reports: polymer science category in 1997-2001. Scientometrics. 2004; 60: 217 – 235.

20      Sombatsompop N, Markpin T. Making an equality of ISI impact factors for different subject fields: research article. J Am Soc Inf Sci Technol. 2005; 56: 676 – 683.

21      Moin M, Mahmoudi M, Rezaei N. Scientific output of Iran at the threshold of the 21st century. Scientometrics. 2005; 62: 239 – 248.

22      Stone R. Science in Iran. An Islamic science revolution? Science. 2005; 309: 1802 – 1804.

23    Science Watch. Middle Eastern Nations Making Their Mark. Available from: URL: http://www. sciencewatch. com/nov-dec2003/sw_nov-dec2003_page1.htm.

24      Ghaleh NR, Siadat F, Azizi F. Quantitative and qualitative assessment of biomedical publications from Iran, Pakistan, and Egypt through their impact factor. J Pakistani Med Assoc. 2004; 54: 528 – 529.

25      Azizi F, Etemadi A, Hajipoor R, Mortazavi N. Trends of publication in medical journals in Iran in the 1990s. Saudi Med J. 2004; 25 (suppl): S34.

26      Magri MH, Solari A. The SCI journal citation reports: a potential tool for studying journals? I. Description of the JCR journal population based on the number of citations received, number of source items, impact factor, immediacy index, and cited half-life. Scientometrics. 1996; 35: 93 – 117.

27      Pudovkin AI, Garfield E. Algorithmic procedure for finding semantically related journals, J Am Soc Inf Sci Technol. 2002; 53: 1113 – 1119.

---

 AIM Home | Table of Contents