Citation Analysis of Hepatitis Monthly by Journal Citation Report (ISI), Google Scholar, and Scopus
Background: Citation analysis as one of the most widely used methods of bibliometrics can be used for computing the various impact measures for scholars based on data from citation databases. Journal Citation Reports (JCR) from Thomson Reuters provides annual report in the form of impact factor (IF) for each journal.
Objectives: We aimed to evaluate the citation parameters of Hepatitis Monthly by JCR in 2010 and compare them with GS and Sc.
Materials and Methods: All articles of Hepat Mon published in 2009 and 2008 which had been cited in 2010 in three databases including WoS, Sc and GS gathered in a spreadsheet. The IFs were manually calculated.
Results: Among the 104 total published articles the accuracy rates of GS and Sc in recording the total number of articles was 96% and 87.5%. There was a difference between IFs among the three databases (0.793 in ISI [Institute for Scientific Information], 0.945 in Sc and 0.85 GS). The missing rate of citations in ISI was 4% totally. Original articles were the main cited types, whereas, guidelines and clinical challenges were the least ones.
Conclusions: None of the three databases succeed to record all articles published in the journal. Despite high sensitivity of GS comparing to Sc, it cannot be a reliable source for indexing since GS has lack of screening in the data collection and low specificity. Using an average of three IFs is suggested to find the correct IF. Editors should be more aware on the role of original articles in increasing IF and the potential efficacy of review articles in long term impact factor.
Keywords: Citation Analysis; Impact Factor; Hepatitis Monthly; ISI; Scopus; Google Scholar
The journal of “Hepatitis Monthly” (Hepat Mon) as one of the leading journals in the field of hepatology in the region has been published since 2001. The Journal context is devoted to particular compilation of the latest worldwide and interdisciplinary approach and findings including original manuscripts, meta-analyses, review articles, health economic papers, debates and consensus statements of clinical relevance of hepatological field especially liver diseases. By the end of 2010, more than 220 papers have been published in 28 issues from them 210 articles are indexed in ISI (Institute for Scientific Information) Web of Science since 2007. Full-text of articles can be found in many indexing sources like Scopus and EMBASE which allows the wide exposure and opportunity for sound and insightful content to be cited by other researchers throughout the world. Citation analysis, as the examination of the frequency, patterns, and graphs of citations in articles and books, is one of the most widely used methods of bibliometrics and is suitable application for the identification of expert referees to review papers, to provide transparent data in support of academic merit review, tenure, and promotion decisions. The citation analysis tools can be used for computing the various impact measures for scholars based on data from citation indices.
Web of Science (WoS) is an online academic citation indexing source provided by Thomson Reuters since 1900. Journal Citation Reports (JCR) is a comprehensive resource that allows readers to evaluate and compare journals using citation data drawn from over 11,000 scholarly and technical journals of more than 3,300 publishers in over 80 countries. JCR shows the most frequently cited journals and the highest impact factor (IF) of the journals in a field. Google Scholar (GS) is a freely accessible web search engine that indexes the full text of scholarly literature across an array of scholarly publishing formats either from peer-reviewed or non-peer-reviewed journals since 2004. Scopus (Sc), officially named SciVerse Scopus, is a bibliographic database containing abstracts and citations for academic journal articles which covers nearly 18,000 titles from over 5,000 international publishers, including coverage of 16,500 peer-reviewed journals in the scientific, technical, medical, and social sciences (including arts and humanities). It is owned by Elsevier and is available online by subscription.
We aimed to evaluate the accuracy of Hepatitis Monthly IF presented by JCR in 2010 comparing with GS and Sc and evaluate the advantages and threats of JCR in comparison with GS and Sc.
3. Materials and Methods
Based on JCR 2010, we selected all articles of Hepat Mon published in 2009 and 2008 which had been cited in 2010. Using JCR 2010, Sc and GS report, we gathered all data in a spreadsheet then cleared dataset and prepared them for analysis. Calculation of the IF was done based on the JCR method provided by Thomson Reuters (Table 1). The IF from JCR was assumed as the gold standard when calculating others from Sc and GS.
Method of Calculating Impact Factor of Hepatitis Monthly in 2010 Based on JCR
3.1. Method of data collection and analysis
For the citation indicator of Hepatitis Monthly, the published articles during 2008 and 2009 have been considered for searching in three databases: ISI, Sc, and GS. Each article searched separately in the databases and all citations recorded in a spreadsheet. Afterwards IFs were manually calculated from the spreadsheet which can be found in Table 2. To calculate IF of Hepat Mon in WoS, it was searched through the Cited Reference Search in WoS as well as the final report of JCR, 2010. According to its policy, ISI don’t count the letters to the editors and editorial articles in IF calculation. Notably ISI was considered as the gold standard for calculations.
Total Number of Articles, Citations and Self-Citations of Two Volumes (2008 and 2009) of Hepatitis Monthly in Google Scholar, Scopus and ISI
Among 104 total published articles in Hepatitis Monthly during 2008 and 2009, 87, 100, and 91 articles were recorded in ISI, GS, SC respectively (Table 2). Due to omission of “letters to editor” and “editorials” in calculation of IFs of ISI, the real number of published articles were different from those indexed in ISI. Consequently and thanks to assumption of ISI as the gold standard, the accuracy rates of GS and Sc in recording the total number of articles were 96% and 87.5%, respectively. There was a difference between IF among three databases. As shown in Table 2 the IF of Hepat Mon in 2010 was 0.793 in ISI, while it was 0.945 and 0.85 in SC and GS. There were 72 citations in ISI in 2010 to the published articles in 2008 and 2009, while according to JCR, total number of citations in 2010 to the published articles in the same year (2008 and 2009) were 69. Therefore, the missing rate of citations in ISI was 4% totally (Table 2).
Among all published articles, two systematic reviews from Iran had the most citations which one of them was published in 2008 (n = 9) and another in 2009 (n = 7). Even though, there was a difference between the calculated IF in GS and Sc with ISI, Table 3 demonstrated that the mentioned difference was shown only in a few articles. The main discrepancy was in one original article published in 2009. According to data from GS this article is cited 5 times comparing to ISI and Sc which was two times. Figure 1 shows that among articles published in Hepat Mon during 2008 and 2009, original articles were the main cited types while guidelines and clinical challenges were the least types. Whereas if we consider proportion of citations to the types of articles, review articles have the most citation. Furthermore, case reports were the only type of articles which were the same in citation numbers in all the databases (Figure 1). No significant difference was detected in all other types of articles except original articles which represented a significant difference between ISI and the two other databases.
Comparison of Citations to the Top 20 Articles of Hepatitis Monthly in ISI, GS and Sc During 2008-9
Comparison of Number of Citations in ISI, GS, and Sc Based on Article Types Published in Hepatitis Monthly (2008, 2009)
Based on JCR, out of all citations to Hepatitis Monthly in 2010, “IRAN RED CRESCENT MED” had the most citations (n = 24) and then “ASIAN PAC J CANCER P” (n = 5) and “ARCH MED RES” (n = 4) were in the second and third levels (Table 4). Moreover, the most citations to Hepat Mon were from Iran (15.78%), Japan (3.28%) and USA (2.63%). Analyzing IF of the journals that cited to Hepat Mon doesn’t show any relationship between the journals IF and number of citation to Hepat Mon (the average IF of 1.86 for the first 10 journals compare to 2.139 for the second 10 journals). Among all citations in Hepat Mon, 90 citations belonged to “HEPATOLOGY” (IF: 10.885), 50 to “J HEPATOL” (IF = 9.334), 36 to “GASTROENTEROLOGY” (IF = 12.032), and 36 to “NEW ENGL J MED” (IF = 53.486). Therefore, there was no correlation between IF and number of citations in Hepat Mon (Table 5). Generally, the average IF of the first 10 journals is 10.517 and the average IF for the second 10 journals is 9.05.
Distribution of the Top 20 Journals Which Cited in 2010 to Published Articles of Hepatitis Monthly (Data Gathered From JCR, ISI, 2010)
Distribution of the Top 20 Journals Which Are Cited in Hepatitis Monthly in 2010 (Data Gathered From JCR, ISI, 2010)
The accuracy rates of GS and Sc in recording the total number of articles were 96% and 87.5%. Although GS had indexed more articles than Sc, it is not reasonable to rely on it for counting the total number of articles since fundamentally GS is a robot which collects data via the websites with neither human interaction nor data cleaning while Sc is guided and supervised by operators. by another meaning, GS has high sensitivity and low specificity vs. Sc with high specificity, while GS reports all citations from different types of publications, therefore, it cannot be as specific as the two other databases (1). GS have found to be sometimes inadequate, and less often updated (2). Moreover, Using GS does not guarantee to access the full-text availability for the readers since it is regulated via the journal’s publisher (3). Another important factor is that the relative coverage of GS varies by discipline compared to other general databases (4). It is also indicated that reliability of each database depends on some factors such as year and the subject of publication (5). Despite much more reliability and validity of Sc, there are some doubts about Sc. Since the Elsevier is the owner of Sc, and is also one of the main international publishers of scientific journals, some users have misgivings about a potential conflict of interest in the choice of the periodicals to be included in the database (3).
There was a difference between IF among three databases. (GS = 0.85, Sc = 0.945). Notably Sc covers more international and open access journals than the other databases (5) and this could be the potential reason for higher IF among all. Although we found difference among those three databases, Bauer et al. in 2005 showed no significant difference between results by Sc and WoS (1). Just similar to Bauer (1), Bakkalbasi et al. in 2006 reported differences in citations of all pairs databases except GS and Sc (5). In contrast with previous studies by Bauer and Bakkalbasi and like to ours, another study revealed higher coverage of articles by Sc than WoS (6). “Cited by” feature in GS, found to be inadequate and less often updated in comparison to the other two databases (2). Respect to the nature of GS which is a crawler, a significant problem with GS is the secrecy about its coverage because some publishers do not allow it to crawl their journals. For instance Elsevier journals were not included before mid-2007, when Elsevier began to make most of its ScienceDirect content available to GS and Google's web search (1, 7). List of journals crawled are not known in GS as well as the frequency of its updates (2). Copyright issues in accessing to the contents of the most expensive commercial databases are still discussed between Google and the publishers like Thomson Rheuters. GS crawlers collect blindly citations and manipulated hitting in a webpage raises its ranking consequently brings the page to the tops of the search results. This false effect of GS on citation counts is criticized with many searchers. Therefore citation counts from GS are not a reliable metrics for counting the h-index and impact factor.
Another issue for searchers in GS is searching with Interpunctuation characters in titles which produce wrong search results, and authors are assigned to wrong papers, which leads to erroneous additional search results. Some editors prefer to negotiate with ISI to recategories the published articles to be considered as citable articles (8). Bagatin et al. believe IF cannot completely represent quality of articles, since review articles have a high potential for citations while the publishing case reports usually have a negative effect on citation on account of few citations (9). Based on the types of articles published in Hepat Mon during 2008 and 2009, no significant difference was detected in all other types of articles except original articles which represented a significant difference between ISI and the two other databases. Review articles receive extremely high citations and review journals have some of the highest impact factors (10). There is a skewed distribution of citations in most fields. The so-called 80/20 phenomenon applies, in that 20% of articles may account for 80% of the citations (11). Publishing mediocre review papers will not necessarily boost a journal’s impact (11).
As the free access of PubMed and GS comparing with WoS and Sc absorbs more readers and citations while Sc offers about 20% more coverage than WoS, whereas GS offers results of inconsistent accuracy. Till now PubMed remains an optimal tool in biomedical electronic research. Currently Sc is limited to the recent articles that have been published since 1995 following which it cannot overcome WoS, in spite of wider journal coverage (2). However, some studies emphasize that Sc is more user friendly with shorter time span (12). Thanks to alerting features via Sc which allows authors to track all published articles, it also offers authors a profile which covers affiliations, number of publications and their bibliographic data, references, and details on the number of citations each published document has received. H-index then will be calculated in each profile (12). Garfield believes that 24% of references contain only 25 journals (little more than 1 percent of SCI coverage) (13). This analysis gives good reason for concern about any increase in the number of scientific and technical journals. Increased number of journals makes coverage of the literature more difficult, but it seems that most of the current journals play only a marginal role, if any, in the effective transfer of scientific information (13). Of 2200 journals covered by the SCI in 1969, 500 journals publish about 70% of all published articles (13).
None of the three databases succeed to record all articles published in the journal. Despite high sensitivity of GS comparing to Sc, it cannot be a reliable source for indexing since GS has lack of screening in the data collection and low specificity. According to the highest IF in Sc comparing to GS and ISI it is suggested using an average of three IFs to find the correct one. Editors should be more aware on the role of original articles in increasing IF and the potential efficacy of review articles in long term impact factor.
The authors would like to appreciate Prof. Seyed Moayed Alavian, Mr. Amir Hasani and Dr. Parisa Bahmani for their Valuable comments and their assistance in analyzing data and other technical editor in Kowsar for their guidance during writing the article.
- 1. Bauer K. An examination of citation counts in a new scholarly communication environment. D-Lib magazine. 2005;11(9):1. [DOI]
- 2. Falagas ME, Pitsouni EI, Malietzis GA, Pappas G. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: strengths and weaknesses. FASEB J. 2008;22(2):338-42. [DOI] [PubMed]
- 3. Kousha K, Thelwall M. Sources of Google Scholar citations outside the Science Citation Index: A comparison between four science disciplines. Scientometrics. 2008;74(2):273-94. [DOI]
- 4. Harzing AWK, Van der Wai R. Google Scholar as a new source for citation analysis. Ethics in Science and Environmental Politics(ESEP). 2008;8(1):61-73. [DOI]
- 5. Bakkalbasi N, Bauer K, Glover J, Wang L. Three options for citation tracking: Google Scholar, Scopus and Web of Science. Biomed Digit Libr. 2006;3:7. [DOI] [PubMed]
- 6. Vieira ES, Gomes JANF. A comparison of Scopus and Web of Science for a typical university. Scientometrics. 2009;81(2):587-600. [DOI]
- 7. Pauly D, Stergiou KI. Equivalence of results from two citation analyses: Thomson ISI’s Citation Index and Google’s Scholar service. Ethics Sci Environ Polit. 2005;2005:33-5.
- 8. Chew M, Villanueva EV, Van Der Weyden MB. Life and times of the impact factor: retrospective analysis of trends for seven medical journals (1994-2005) and their Editors' views. J R Soc Med. 2007;100(3):142-50. [DOI] [PubMed]
- 9. Bagatin E, Gontijo B. The expansion of a measure: what is a scientific journal impact factor and how important is it for academic Brazilian dermatologists. Int J Dermatol. 2011;50(11):1432-4. [DOI] [PubMed]
- 10. Coleman R. Impact factors: use and abuse in biomedical research. Anat Rec. 1999;257(2):54-7.
- 11. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295(1):90-3. [DOI] [PubMed]
- 12. Bar-Ilan J. Which h-index?—A comparison of WoS, Scopus and Google Scholar. Scientometrics. 2008;74(2):257-71. [DOI]
- 13. Garfield E. Citation analysis as a tool in journal evaluation. Science. 1972;178(4060):471-9. [DOI] [PubMed]