Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc
1
Department of Applied Economics, and Department of Finance, National Chung Hsing University, Taichung 402, Taiwan
2
Department of Quantitative Finance, National Tsing Hua University, Hsinchu 300, Taiwan
3
Econometric Institute, Erasmus School of Economics, Erasmus University Rotterdam, Rotterdam 3000, The Netherlands
4
Tinbergen Institute, Rotterdam 3000, The Netherlands
5
Department of Quantitative Economics, Complutense University of Madrid, Madrid 28040, Spain
*
Author to whom correspondence should be addressed.
Econometrics 2013, 1(3), 217-235; https://doi.org/10.3390/econometrics1030217
Submission received: 15 October 2013
/
Revised: 6 November 2013
/
Accepted: 7 November 2013
/
Published: 18 November 2013
Abstract
:The paper focuses on the robustness of rankings of academic journal quality and research impact of 10 leading econometrics journals taken from the Thomson Reuters ISI Web of Science (ISI) Category of Economics, using citations data from ISI and the highly accessible Research Papers in Economics (RePEc) database that is widely used in economics, finance and related disciplines. The journals are ranked using quantifiable static and dynamic Research Assessment Measures (RAMs), with 15 RAMs from ISI and five RAMs from RePEc. The similarities and differences in various RAMs, which are based on alternative weighted and unweighted transformations of citations, are highlighted to show which RAMs are able to provide informational value relative to others. The RAMs include the impact factor, mean citations and non-citations, journal policy, number of high quality papers, and journal influence and article influence. The paper highlights robust rankings based on the harmonic mean of the ranks of 20 RAMs, which in some cases are closely related. It is shown that emphasizing the most widely-used RAM, the two-year impact factor of a journal, can lead to a distorted evaluation of journal quality, impact and influence relative to the harmonic mean of the ranks. Some suggestions regarding the use of the most informative RAMs are also given.
Keywords:
research assessment measures; citations; impact; influence; harmonic mean; robust journal rankings; econometricsJEL Classifications:
C18, C81, Y101. Introduction
It is an unavoidable fact of academic life that the actual and/or perceived research performance of scholars is important in hiring, tenure and promotion decisions. Where a paper is published is frequently regarded as being of greater importance than the quality of the paper itself which, among other reasons, leads to rankings of a journal’s perceived quality. Such perceived quality of academic journals is routinely based on both testable and untestable assessments of journal impact and influence, the number of high quality papers, and quantitative or qualitative information about a journal, as well as quantifiable bibliometric Research Assessment Measures (RAMs) that are based on citations.
In this context, the leading database for generating RAMs to evaluate the research performance of individual researchers and the quality of academic journals is the Thomson Reuters ISI Web of Science [1] database (hereafter ISI), where most RAMs are based on alternative weighted and unweighted transformations of citations data. Virtually all existing RAMs are static, with two being dynamic in capturing changes in impact factors over a period of two to five years, as well as escalating journal self-citations.
Seglen [2] and Chang et al. [3,4,5,6,7], among others, have raised important warnings regarding the methodology and data collection methods underlying the ISI database. Such caveats would generally apply to any citation databases. Nevertheless, the ISI citations database is the oldest and most widely-used source of citations-based RAMs, and is undoubtedly the benchmark against which other citations databases, such as SciVerse Scopus, Google Scholar and Microsoft Academic Search, social science open access repositories, such as the Social Science Research Network (SSRN), and discipline-specific databases, such as Research Papers in Economics (RePEc), are compared.
The perceived quality of academic journals has long been used as a (sometimes highly questionable) proxy for the quality of published papers, especially for less established scholars, and especially in the social sciences. In comparison, citations are used far more frequently in the sciences to evaluate the quality of published papers than they are in the social sciences. As stated elsewhere, and as is well known, journal publishers promote the ISI impact factor of their journals and, if their journals do not yet have an impact factor, publicize the fact that their journals have either been selected for coverage in ISI or they have applied for inclusion in ISI.
Various RAMs have been used to compare journals in a wide range of ISI disciplines in terms of citations, quality and impact, such as the 40 leading journals in Economics and the leading 10 journals in each of Management, Finance and Marketing [3], the six leading journals in each of 20 disciplines in the Sciences [4], the 10 leading journals in a sub-discipline of Economics, namely Econometrics, and four leading journals in Statistics [5], the 26 leading journals in Neuroscience [6], the 299 leading journals in Economics [7], the 110 leading journals in Statistics and Probability [8], and the leading 34 journals in Finance [9].
Although [5] evaluated the 10 leading journals in econometrics using 13 RAMs from ISI for seven journals and 10 RAMs from ISI for three journals, the data were downloaded from ISI on April 28, 2010. In this paper, we use 15 RAMs from ISI for all 10 journals using data that were downloaded on September 28, 2013. As ISI data are made available in June of each year, this is four years more current than the previous rankings paper of econometrics journals, which will enable a comparison of whether the previous rankings have changed over time.
This paper also uses five RAMs from the highly accessible RePEc database (see [10]) which, to the best of our knowledge, has not previously been compared with citations RAMs using ISI data. In addition, the five RAMs from RePEc will be compared with each other to determine which RAMs provide distinctive information. Therefore, 20 RAMs will be used to rank the 10 leading journals in econometrics, as well as determine which RAMs are able to provide informational value relative to others from ISI and RePEc.
This paper examines the importance of RAMs as viable rankings criteria in 10 leading econometrics journals from the ISI category of Economics, and suggests a robust rankings method of alternative RAMs using the harmonic mean of the ranks. Together with the arithmetic and geometric means, the harmonic mean is one of the three Pythagorean means, and is defined as the reciprocal of the arithmetic mean of the reciprocals. The rankings based on any single RAM, such as the two-year impact factor, are placed in context, and may be seen as an extreme as it is clearly subsumed by the harmonic mean of the ranks when all other RAMs are given zero weights, except the RAM in question. Moreover, emphasizing the two-year impact factor of a journal to the exclusion of other informative RAMs can lead to a distorted evaluation of journal quality, impact and influence based on citation data.
The use of a mean measure based on the individual RAMs has a firm foundation in statistical theory. In hypothesis testing, the test of a null against a specific alternative hypothesis will have high power if the alternative is true. However, if the alternative is not true, the test will be inconsistent. Such a test is not robust. If the null hypothesis is tested against multiple alternatives, the test will more likely be consistent, and hence will be robust. The same principle applies to rankings of multiple criteria, which is precisely why we aggregate a large number of RAMs to obtain a robust set of rankings.
The plan of the remainder of the paper is as follows. Section 2 presents some key RAMs using ISI data that may be calculated annually or updated daily and key RAMs from RePEc that are updated daily. Section 3 discusses and analyses 20 RAMs for 10 leading journals in econometrics drawn from the ISI category of Economics, and provides a harmonic mean of the ranks as a robust rankings method of alternative RAMs. Section 4 summarizes the ranking outcomes, gives some practical suggestions as to how to rank journal quality and impact using citations data, and emphasizes the inherent usefulness and informational value of some RAMs relative to others.
2. Research Assessment Measures (RAM) for ISI and RePEc
A widely-used RAM database for evaluating journal impact and quality in the sciences and social sciences is the Thomson Reuters ISI Web of Science [1]. An alternative data source that is widely used in economics, finance and related disciplines is the Research Papers in Economics (RePEc) database. As discussed in a number of recent papers (for example, [3,4,5,7]), the RAMs available using data from ISI are intended as descriptive statistics to capture journal impact and performance, and are not based on a mathematical model. Hence, in what follows, no optimization or estimation is required in calculating the alternative RAMs using data from ISI. The data for all journals are given from 1970, and were downloaded from ISI on September 28, 2013.
(i) ISI Data
As the alternative RAMs that are provided in ISI and in several recent publications may not be widely known, this section provides a brief description and definition of 15 RAMs using ISI data that may be calculated annually or updated daily.
2.1. Annual RAM
With three exceptions, namely Eigenfactor, Article Influence and Cited Article Influence, existing RAMs are based on citations data and are reported separately for the sciences and social sciences. RAMs may be computed annually or updated daily. The annual RAMs given below are calculated for a Journal Citations Reports (JCR) calendar year, which is the year before the annual RAMs are released in mid-year.
(1) Two-year impact factor including journal self-citations (2YIF)
The classic two-year impact factor including journal self-citations (2YIF) of a journal is typically referred to as “the impact factor”, is calculated annually, and is defined as “Total citations in a year to papers published in a journal in the previous two years/Total papers published in a journal in the previous two years”. The choice of two years by ISI is arbitrary. It is widely held in the academic community, and certainly by the editors and publishers of journals, that a higher 2YIF is better than a lower one.
(2) Two-year impact factor excluding journal self-citations (2YIF*)
ISI also reports a two-year impact factor without journal self-citations (that is, citations to a journal in which a citing paper is published), which is calculated annually. As this impact factor is not widely known or used, [5] refer to this RAM as 2YIF*. Although 2YIF* is rarely reported (for reasons that are obvious), a higher value would be preferred to lower.
(3) Five-year impact factor including journal self-citations (5YIF)
The five-year impact factor including journal self-citations (5YIF) of a journal is calculated annually, and is defined as “Total citations in a year to papers published in a journal in the previous five years/Total papers published in a journal in the previous five years”. The choice of five years by ISI is arbitrary. Although 5YIF is not widely reported, a higher value would be preferred to a lower one.
(4) Immediacy, or zero-year impact factor including journal self-citations (0YIF)
Immediacy is a zero-year impact factor including journal self-citations (0YIF) of a journal, is calculated annually, and is defined as “Total citations to papers published in a journal in the same year/Total papers published in a journal in the same year”. The choice of the same year by ISI is arbitrary, but the nature of Immediacy makes it clear that a very short run outcome is under consideration. Although Immediacy is rarely reported, a higher value would be preferred to a lower one.
(5) 5YIF Divided by 2YIF (5YD2)
As both 2YIF and 5YIF include journal self-citations, if it is assumed that journal self-citations are uniformly distributed over the five-year period for calculating 5YIF, their ratio will eliminate the effect of journal self-citations and capture the increase in the citation rate over time. In any event, the impact of journal self-citations should be mitigated with the ratio of 5YIF to 2YIF. Chang, McAleer and Maasoumi [7] define a dynamic RAM as 5YD2 as “5YD2 = 5YIF/2YIF”. In the natural, physical and medical sciences, where citations are observed with a frequency of weeks and months rather than years, it is typically the case that 5YIF < 2YIF (see [4,6]), whereas the reverse, 5YIF > 2YIF, seems to hold generally in the social sciences, where citations tend to increase gradually over time (see [3,5]). Although this is essentially an empirical issue, [7] discusses the different speeds at which citations are accrued over time, and suggest that a higher 5YD2 would generally be preferred to a lower one.
(6) Eigenfactor (or Journal Influence)
The Eigenfactor score is calculated annually (see [11,12,13,14]), and is defined as: “The Eigenfactor Score calculation is based on the number of times articles from the journal published in the past five years have been cited in the JCR year, but it also considers which journals have contributed to these citations so that highly cited journals will influence the network more than lesser cited journals. References from one article in a journal to another article from the same journal are removed, so that Eigenfactor Scores are not influenced by journal self-citation.” The value of the threshold that separates ‘highly cited’ from ‘lesser cited’ journals, as well as how the former might ‘influence the network more’ than the latter, are based on the Eigenfactor score of the citing journal. Thus, Eigenfactor might usefully be interpreted as a weighted total citations score, or a “Journal Influence” measure. A higher Eigenfactor score would be preferred to a lower one.
(7) Article Influence (or Journal Influence per Article)
Article Influence (see [12,13,14]) measures the relative importance of a journal’s citation influence on a per-article basis. Despite the misleading suggestion of measuring “Article Influence”, as each journal has only a single “Article Influence” score, this RAM is actually a “Journal Influence per Article” score. Article Influence is a scaled Eigenfactor Score, is calculated annually, is standardized to have a mean of one across all journals in the Thomson Reuters ISI database, and is defined as “Eigenfactor score divided by the fraction of all articles published by a journal”. A higher Article Influence would be preferred to a lower one.
(8) IFI
The ratio of 2YIF to 2YIF* is intended to capture how journal self-citations can inflate the impact factor of a journal, whether this is an unconscious self-promotion decision made independently by publishing authors or as an administrative decision undertaken by a journal’s editors and/or publishers. Chang, McAleer and Oxley [3] define Impact Factor Inflation (IFI) as “IFI = 2YIF/2YIF*” [3]. The minimum value for IFI is 1, with any value above the minimum capturing the effect of journal self-citations on the two-year impact factor. A lower IFI would be preferred to a higher one.
(9) H-STAR
ISI has implicitly recognized the inflation in journal self-citations by calculating an impact factor that excludes self-citations, and provides data on journal self-citations, both historically (for the life of the journal) and for the preceding two years, in calculating 2YIF. Chang, McAleer and Oxley [4] define the Self-citation Threshold Approval Rating (STAR) as the percentage difference between citations in other journals and journal self-citations [4]. If HS = historical journal self-citations, then Historical STAR (H-STAR) is defined as “H-STAR = [(100-HS) − HS] = (100-2HS)”. If HS = 0 (minimum), 50 or 100 (maximum) percent, for example, H-STAR = 100, 0 and −100, respectively. A higher H-STAR would be preferred to a lower one.
(10) 2Y-STAR:
If 2YS = journal self-citations over the preceding two-year period, then the 2-Year STAR is defined by Chang, McAleer and Oxley [4] as “2Y-STAR = [(100 – 2YS) – 2YS] = (100 – 2(2YS))”. If 2YS = 0 (minimum), 50 or 100 (maximum) percent, for example, 2Y-STAR = 100, 0 and −100, respectively. A higher 2Y-STAR would be preferred to a lower one.
(11) Escalating Self-Citations (ESC)
As self-citations for many journals in the sciences and social sciences have been increasing over time, it would seem useful to present a dynamic RAM that captures such an escalation over time. The difference 2YS – HS measures Escalating Self Citations in journals over the most recent two years relative to the historical period for calculating citations, which will differ across journals. Chang, Maasoumi and McAleer [7] define a dynamic RAM as “ESC = 2YS – HS = (H-STAR – 2Y-STAR)/2”. Given the range of each of H-STAR and 2Y-STAR is (−100, 100), the range of ESC is also (−100, 100), with −100 denoting minimum, and 100 denoting maximum, escalation. A lower ESC would be preferred to a higher one.
2.2. Daily Updated RAM
Some RAMs are updated daily and are reported for a given day in a calendar year rather than for a JCR year.
(12) C3PO
ISI reports the mean number of citations for a journal, namely total citations up to a given day divided by the number of papers published in a journal up to the same day, as the “average” number of citations. In order to distinguish the mean from the median and mode, the C3PO of an ISI journal on any given day is defined by Chang, McAleer and Oxley [3] as “C3PO (Citation Performance Per Paper Online) = Total citations to a journal/Total papers published in a journal”. A higher C3PO would be preferred to a lower one. (Note: C3PO should not be confused with C-3PO, the Star Wars android.)
(13) h-index
The h-index [15] was originally proposed to assess the scientific research productivity and citations impact of individual researchers. However, the h-index can also be calculated for journals, and should be interpreted as assessing the impact or influence of highly cited journal publications. The h-index of a journal on any given day is based on historically cited and citing papers, including journal self-citations, and is defined as “h-index = number of published papers, where each has at least h citations”. The h-index differs from an impact factor in that the h-index measures the number of highly cited papers historically. A higher h-index would be preferred to a lower one.
(14) PI-BETA
In comparison with the rejection rate of a journal before publication, there is an equally important implicit rejection rate after publication. This RAM measures the proportion of papers in a journal that has never been cited. As such, PI-BETA is, in effect, a rejection rate of a journal after publication, namely the proportion of published papers that is ignored by the profession, and possibly by the authors themselves. Chang et al. [5] argue that lack of citations of a published paper, especially if it is not a recent publication, reflects on the quality of a journal by exposing: (i) what might be considered as incorrect decisions by the members of the editorial board of a journal; and (ii) the lost opportunities of papers that might have been cited had they not been rejected by the journal [5]. Chang et al. propose that a paper with zero citations in ISI journals can be measured by PI-BETA (=Papers Ignored (PI) – By Even The Authors (BETA)), which is calculated for an ISI journal on any given day as “Number of papers with zero citations in a journal/Total papers published in a journal” [5]. As journals would typically prefer a higher proportion of published papers being cited rather than ignored, a lower PI-BETA would be preferred to a higher one.
(15) CAI
Article Influence is intended to measure the average influence of an article across the sciences and social sciences. As an article with zero citations typically does not have any (academic) influence, a more suitable measure of the influence of cited articles would seem to be Cited Article Influence (CAI). Chang et al. [4] define CAI as “CAI = (1 − PI-BETA)(Article Influence)”. If PI-BETA = 0, then CAI is equivalent to Article Influence; if PI-BETA = 1, then CAI = 0. As Article Influence is calculated annually and PI-BETA is updated daily, CAI may be updated daily. A higher CAI would be preferred to a lower one.
(ii) RePEc Data and Daily Updated RAM
As the alternative RAMs that are provided in RePEc may not be widely known, this section provides a brief description and definition of five RAMs using RePEc data that may be updated daily (see [16,10]). RePEc counts citations from books, chapters and working papers that are listed in its archives, and hence has a broader base compared with ISI. As in the case of RAMs based on ISI data, the RAMs available using data from RePEc are intended as descriptive statistics to capture journal impact and performance, and are not based on a mathematical model, so that no optimization or estimation is required in calculating the alternative RAMs.
Two distinguishing features of the RePEc citations database are as follows:
- (i)
- the impact factors are calculated for each journal from the time of their inclusion in the RePEc database, so there is no fixed duration for calculating the impact factors;
- (ii)
- journal self-citations are excluded in calculating the impact factors.
Although the RePEc impact factors are calculated for the life of each journal, conceptually they are closer to 2YIF* and Article Influence, which exclude journal self-citations, than to Immediacy, 2YIF or 5YIF, which include journal self-citations. The data were downloaded from RePEc on October 4, 2013 for the September 2013 update, at which time there were 1,797 journals and 37,599 authors in the RePEc database.
(16) SIF
The simple impact factor (SIP) is defined as the number of citations divided by the number of published articles. SIP is conceptually similar to Immediacy, 2YIF and 5YIF, though it is calculated over the entirety of the journal’s inclusion in the RePEc database. A higher SIF would be preferred to a lower one.
(17) RIF
The recursive impact factor (RIF) weights each citation by the impact factor of the citing items, which is also computed recursively. The recursive impact factors are normalized so that the average citation has a weight of 1. RIF is conceptually similar to Article Influence, except that it is calculated over the entirety of the journal’s inclusion in the RePEc database. A higher RIF would be preferred to a lower one.
(18) DIF
The discounted impact factor (DIF), wherein each citation is divided by the age in years of the citing article, so that a citation from an article published n years earlier counts for 1/(n + 1), n = 0, 1, 2, … (with n = 0 for the same year). DIF is conceptually different from all three ISI impact factors. A higher DIF would be preferred to a lower one.
(19) RDIF
The recursive discounted impact factor (RDIF) weights each citation by the impact factor of the citing items, which is also computed recursively. Each citation is also divided by the age in years of the citing article, so that a citation from an article published n years earlier counts for 1/(n + 1), n = 0, 1, 2, … (with n = 0 for the same year). RDIF is conceptually similar to Article Influence, except that it is calculated over the entirety of the journal’s inclusion in the RePEc database. A higher RDIF would be preferred to a lower one.
(20) h-RePEc
This RAM has the same definition as the original h-index, which is used for ISI data, except that journal self-citations are excluded in RePEc. A higher h-RePEc would be preferred to a lower one.
3. Analysis of RAM for 10 Leading Journals in Econometrics
The acronyms for the 10 leading econometrics journals are taken from the ISI Economics subject category, and are given (in alphabetical order) as follows:
- ECONOMET J = Econometrics Journal
- ECONOMET REV = Econometric Reviews
- ECONOMET THEOR = Econometric Theory
- ECONOMETRICA = Econometrica
- J APPL ECONOMET = Journal of Applied Econometrics
- J BUS ECON STAT = Journal of Business & Economic Statistics
- J ECONOMETRICS = Journal of Econometrics
- J FINANC ECONOMET = Journal of Financial Econometrics
- OXFORD B ECON STAT = Oxford Bulletin of Economics and Statistics
- REV ECON STAT = Review of Economics and Statistics
No single RAM captures adequately the quality, impact and influence of a journal. Therefore, any general measure of journal quality and impact, such as a harmonic mean of the ranks as a robust rankings method (see, for example, [7]), should depend on all the available RAMs. Of the 20 RAMs, 17 are ranked from high to low. The three RAMs that rank from low to high are PI-BETA, IFI and ESC.
In what follows, we compare the RAMs that are based on ISI citations data (Table 1 , Table 3, Table 4 and Table 5) and RePEc citations data (Table 2, Table 3, Table 4 and Table 5). Only articles from the ISI Web of Science and RePEc are included in the citations data, which were downloaded from ISI on September 28, 2013 and from RePEc on October 4, 2013, for all journals. As will be seen below, all 10 econometrics journals are among the leading journals in both the Economics category of ISI and in RePEc.
In Table 1 we evaluate 15 RAMs for the 10 leading econometrics journals, which are ranked according to 2YIF. The means and ranges for 2YIF, respectively, are 1.665 and 0.707–3.823; for 2YIF*, 1.538 and 0.707–3.425; for 5YIF, 2.440 and 1.252–5.702; and for Immediacy, 0.294 and 0.091–0.740. These impact factors are all considerably higher than their counterparts in the Economics category, which are 1.665, 1.538, 2.440 and 0.294, respectively (see [7]).
The mean and range of 5YD2 in Table 1 are 1.521 and 0.997–2.499, respectively, so that 5YIF is considerably higher than 2YIF, which is to be expected in Econometrics. In Economics, 5YD2 is 1.380 (see [7]), so that citations increase more over time for the leading econometrics journals than for Economics as a whole.
Journal self-citations in the 10 leading econometrics journals are very low, with a mean IFI of 1.086 and a range of 1–1.187. On average, the 299 leading journals in Economics have 2YIF that is inflated by a factor of 1.442 through journal self-citations (see [7]), which is considerably higher.
The h-index has a mean of 63 and a range of 11–181, with the mean being more than double the mean of 27 for the 299 Economics journals in ISI (see [7]). The journals with lower h-indexes tend to have been included in ISI more recently than those journals with higher h-indexes.
In terms of mean citations, C3PO has a mean of 17.63 and a range of 3.46–52.21, as compared with a considerably lower mean of 5.51 for Economics (see [7]). As in the case of the h-index, the journals with lower C3PO values tend to have been included in ISI more recently than those journals with higher C3PO.
Eigenfactor has a mean of 0.01638 and a range of 0.00304–0.04620, which is more than three times the mean of 0.005 for Economics as a whole (see [7]). Article Influence has a mean of 3.181 and a range of 1.533–9.684, which is more than double the mean of 1.334 for the 299 Economics journals in ISI (see [7]). As Article Influence is standardized to have a mean of one across all social science and science journals in the Thomson Reuters ISI database, the mean article influence in econometrics is considerably greater than for all the Economics journals, and even higher still than the full list of journals in the ISI database. Cited Article Influence (CAI) has a mean of 2.432 and a range of (1.035, 6.895), which is much higher than for all Economics journals, with a mean of 0.925.
The h-index has a mean of 63 and a range of 11–181, with the mean being more than double the mean of 27 for the 299 Economics journals in ISI (see [7]). The journals with lower h-indexes tend to have been included in ISI more recently than those journals with higher h-indexes.
In terms of mean citations, C3PO has a mean of 17.63 and a range of 3.46–52.21, as compared with a considerably lower mean of 5.51 for Economics (see [7]). As in the case of the h-index, the journals with lower C3PO values tend to have been included in ISI more recently than those journals with higher C3PO.
Eigenfactor has a mean of 0.01638 and a range of 0.00304–0.04620, which is more than three times the mean of 0.005 for Economics as a whole (see [7]). Article Influence has a mean of 3.181 and a range of 1.533–9.684, which is more than double the mean of 1.334 for the 299 Economics journals in ISI (see [7]). As Article Influence is standardized to have a mean of one across all social science and science journals in the Thomson Reuters ISI database, the mean article influence in econometrics is considerably greater than for all the Economics journals, and even higher still than the full list of journals in the ISI database. Cited Article Influence (CAI) has a mean of 2.432 and a range of (1.035, 6.895), which is much higher than for all Economics journals, with a mean of 0.925.
| Rank | Journal | 2YIF | 2YIF * | IFI | 5YIF | Imm | 5YD2 | h-index | C3PO | PI-BETA | Eigenf | AI | CAI | H-STAR | 2Y-STAR | ESC |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | ECONOMETRICA | 3.823 | 3.425 | 1.116 | 5.702 | 0.740 | 1.491 | 181 | 52.21 | 0.288 | 0.04620 | 9.684 | 6.895 | 96 | 80 | 8 |
| 2 | REV ECON STAT | 2.346 | 2.307 | 1.017 | 3.699 | 0.325 | 1.564 | 95 | 27.03 | 0.100 | 0.02670 | 4.264 | 3.838 | 98 | 100 | −1 |
| 3 | J BUS ECON STAT | 1.932 | 1.852 | 1.043 | 2.369 | 0.217 | 1.226 | 58 | 19.32 | 0.175 | 0.01037 | 2.986 | 2.463 | 96 | 92 | 2 |
| 4 | J APPL ECONOMET | 1.867 | 1.765 | 1.058 | 2.521 | 0.315 | 1.350 | 54 | 16.61 | 0.188 | 0.01005 | 2.368 | 1.923 | 96 | 90 | 3 |
| 5 | J ECONOMETRICS | 1.710 | 1.441 | 1.187 | 2.713 | 0.265 | 1.587 | 105 | 25.84 | 0.121 | 0.04103 | 3.272 | 2.876 | 88 | 70 | 9 |
| 6 | ECONOMET THEOR | 1.477 | 1.321 | 1.180 | 1.473 | 0.188 | 0.997 | 44 | 9.52 | 0.310 | 0.01285 | 2.491 | 1.719 | 84 | 80 | 2 |
| 7 | ECONOMET J | 1.000 | 0.929 | 1.076 | 1.252 | 0.227 | 1.252 | 15 | 4.02 | 0.329 | 0.00420 | 1.724 | 1.157 | 94 | 86 | 4 |
| 8 | J FINANC ECONOMET | 0.976 | 0.881 | 1.108 | 1.580 | 0.091 | 1.619 | 11 | 3.46 | 0.404 | 0.00304 | 1.736 | 1.035 | 82 | 80 | 1 |
| 9 | ECONOMET REV | 0.811 | 0.755 | 1.074 | 1.321 | 0.259 | 1.629 | 17 | 5.17 | 0.347 | 0.00429 | 1.748 | 1.141 | 96 | 88 | 4 |
| 10 | OXFORD B ECON STAT | 0.707 | 0.707 | 1.000 | 1.767 | 0.317 | 2.499 | 46 | 13.15 | 0.167 | 0.00508 | 1.533 | 1.277 | 98 | 100 | −1 |
| Mean | 1.665 | 1.538 | 1.086 | 2.440 | 0.294 | 1.521 | 63 | 17.63 | 0.243 | 0.01638 | 3.181 | 2.432 | 93 | 87 | 3 |
Notes: The journal acronyms are taken from the ISI Economics subject category, and the journals are ranked according to 2YIF. The data for all journals are given from1970, and were downloaded from ISI on 28 September 2013. Imm = Immediacy, Eigenf = Eigenfactor, and AI = Article Influence.
| Rank | Journal | SIF | RIF | DIF | RDIF | h-RePEc |
|---|---|---|---|---|---|---|
| 1 | ECONOMETRICA | 46.688 | 2.839 | 9.622 | 2.746 | 174 |
| 2 | REV ECON STAT | 15.544 | 0.886 | 3.524 | 0.905 | 95 |
| 3 | J BUS ECON STAT | 17.116 | 0.920 | 3.868 | 0.912 | 77 |
| 4 | J APPL ECONOMET | 16.357 | 0.856 | 4.251 | 0.941 | 59 |
| 5 | J ECONOMETRICS | 21.559 | 0.863 | 5.022 | 0.985 | 113 |
| 6 | ECONOMET THEOR | 6.948 | 0.332 | 1.597 | 0.400 | 47 |
| 7 | ECONOMET J | 9.463 | 0.111 | 2.714 | 0.157 | 26 |
| 8 | J FINANC ECONOMET | 7.227 | 0.320 | 2.475 | 0.560 | 20 |
| 9 | ECONOMET REV | 7.561 | 0.295 | 2.201 | 0.461 | 26 |
| 10 | OXFORD B ECON STAT | 9.827 | 0.302 | 2.205 | 0.328 | 46 |
| Mean | 15.829 | 0.772 | 3.748 | 0.840 | 68 |
Notes: The journal acronyms are taken from the ISI Economics subject category, and the journals are ranked according to 2YIF, as in Table 1. The data were downloaded from RePEc on October 4, 2013 for the September 2013 update, at which time there were 1,797 journals in the RePEc data base. SIF = Simple Impact Factor, RIF = Recursive Impact Factor, DIF = Discounted Impact Factor, RDIF = Recursive Discounted Impact Factor, and h-RePEc = h-index for RePEc, which excludes journal self-citations.
| 2YIF | 2YIF * | IFI | 5YIF | Imm | 5YD2 | h-index | C3PO | PI-BETA | Eigenf | AI | CAI | H-STAR | 2Y-STAR | ESC | SIF | RIF | DIF | RDIF | h-RePEc | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2YIF | 1 | |||||||||||||||||||
| 2YIF * | 0.996 | 1 | ||||||||||||||||||
| IFI | 0.136 | 0.054 | 1 | |||||||||||||||||
| 5YIF | 0.956 | 0.954 | 0.016 | 1 | ||||||||||||||||
| Imm | 0.817 | 0.804 | −0.044 | 0.879 | 1 | |||||||||||||||
| 5YD2 | −0.287 | −0.276 | −0.483 | −0.011 | 0.125 | 1 | ||||||||||||||
| h-index | 0.916 | 0.896 | 0.190 | 0.955 | 0.861 | −0.001 | 1 | |||||||||||||
| C3PO | 0.938 | 0.926 | 0.092 | 0.977 | 0.885 | 0.007 | 0.991 | 1 | ||||||||||||
| PI-BETA | −0.263 | −0.297 | 0.297 | −0.335 | −0.170 | −0.242 | −0.421 | −0.413 | 1 | |||||||||||
| Eigenf | 0.811 | 0.775 | 0.413 | 0.846 | 0.701 | −0.078 | 0.941 | 0.897 | −0.412 | 1 | ||||||||||
| AI | 0.949 | 0.931 | 0.172 | 0.954 | 0.902 | −0.125 | 0.929 | 0.943 | −0.105 | 0.825 | 1 | |||||||||
| CAI | 0.966 | 0.955 | 0.122 | 0.982 | 0.884 | −0.100 | 0.965 | 0.977 | −0.267 | 0.874 | 0.985 | 1 | ||||||||
| H-STAR | 0.249 | 0.304 | −0.761 | 0.341 | 0.504 | 0.328 | 0.264 | 0.339 | −0.483 | 0.064 | 0.228 | 0.288 | 1 | |||||||
| 2Y-STAR | −0.146 | −0.058 | −0.944 | −0.061 | −0.004 | 0.401 | −0.202 | −0.123 | −0.354 | −0.420 | −0.206 | −0.144 | 0.713 | 1 | ||||||
| ESC | 0.425 | 0.350 | 0.666 | 0.387 | 0.449 | −0.279 | 0.518 | 0.472 | 0.075 | 0.651 | 0.492 | 0.457 | −0.128 | −0.786 | 1 | |||||
| SIF | 0.915 | 0.888 | 0.164 | 0.935 | 0.907 | −0.053 | 0.937 | 0.955 | −0.205 | 0.838 | 0.952 | 0.945 | 0.286 | −0.262 | 0.622 | 1 | ||||
| RIF | 0.949 | 0.931 | 0.126 | 0.956 | 0.903 | −0.090 | 0.925 | 0.953 | −0.164 | 0.797 | 0.972 | 0.960 | 0.274 | −0.196 | 0.520 | 0.981 | 1 | |||
| DIF | 0.897 | 0.868 | 0.177 | 0.917 | 0.878 | −0.074 | 0.905 | 0.925 | −0.171 | 0.821 | 0.926 | 0.916 | 0.259 | −0.304 | 0.658 | 0.992 | 0.968 | 1 | ||
| RDIF | 0.936 | 0.913 | 0.168 | 0.946 | 0.880 | −0.090 | 0.911 | 0.937 | −0.124 | 0.799 | 0.963 | 0.946 | 0.219 | −0.250 | 0.547 | 0.974 | 0.995 | 0.969 | 1 | |
| h-RePEc | 0.918 | 0.898 | 0.203 | 0.943 | 0.827 | −0.050 | 0.993 | 0.987 | −0.442 | 0.944 | 0.916 | 0.957 | 0.261 | −0.229 | 0.553 | 0.940 | 0.923 | 0.912 | 0.909 | 1 |
Table 4.
Rankings by the harmonic mean and 20 RAM from ISI and RePEc for 10 leading econometrics journals.
| Journal | HM | 2YIF | 2YIF * | IFI | 5YIF | Imm | 5YD2 | h-index | C3PO | PI-BETA | Eigenf | AI | CAI | H-STAR | 2Y-STAR | ESC | SIF | RIF | DIF | RDIF | h-RePEc |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ECONOMETRICA | 1 | 1 | 1 | 8 | 1 | 1 | 6 | 1 | 1 | 6 | 1 | 1 | 1 | 3 | 7 | 9 | 1 | 1 | 1 | 1 | 1 |
| REV ECON STAT | 2 | 2 | 2 | 2 | 2 | 2 | 5 | 3 | 2 | 1 | 3 | 2 | 2 | 1 | 1 | 1 | 5 | 3 | 5 | 5 | 3 |
| OXFORD B ECON STAT | 3 | 10 | 10 | 1 | 6 | 3 | 1 | 6 | 6 | 3 | 7 | 10 | 7 | 1 | 1 | 1 | 6 | 8 | 8 | 9 | 7 |
| J ECONOMETRICS | 4 | 5 | 5 | 10 | 3 | 5 | 4 | 2 | 3 | 2 | 2 | 3 | 3 | 8 | 10 | 10 | 2 | 4 | 2 | 2 | 2 |
| J BUS ECON STAT | 5 | 3 | 3 | 3 | 5 | 8 | 9 | 4 | 4 | 4 | 5 | 4 | 4 | 3 | 3 | 4 | 3 | 2 | 4 | 4 | 4 |
| J APPL ECONOMET | 6 | 4 | 4 | 4 | 4 | 4 | 7 | 5 | 5 | 5 | 6 | 6 | 5 | 3 | 4 | 6 | 4 | 5 | 3 | 3 | 5 |
| ECONOMET REV | 7 | 9 | 9 | 5 | 9 | 6 | 2 | 8 | 8 | 9 | 8 | 7 | 9 | 3 | 5 | 7 | 8 | 9 | 9 | 7 | 8 |
| ECONOMET THEOR | 8 | 6 | 6 | 9 | 8 | 9 | 10 | 7 | 7 | 7 | 4 | 5 | 6 | 9 | 7 | 4 | 10 | 6 | 10 | 8 | 6 |
| J FINANC ECONOMET | 9 | 8 | 8 | 7 | 7 | 10 | 3 | 10 | 10 | 10 | 10 | 8 | 10 | 10 | 7 | 3 | 9 | 7 | 7 | 6 | 10 |
| ECONOMET J | 10 | 7 | 7 | 6 | 10 | 7 | 8 | 9 | 9 | 8 | 9 | 9 | 8 | 7 | 6 | 7 | 7 | 10 | 6 | 10 | 8 |
Notes: The journals are ranked according to the harmonic mean (HM) of the ranks. Imm=Immediacy, Eigenf=Eigenfactor, AI=Article Influence, SIF = Simple Impact Factor, RIF = Recursive Impact Factor, DIF = Discounted Impact Factor, RDIF = Recursive Discounted Impact Factor, and h-RePEc = h-index for RePEc, which excludes journal self-citations.
Table 5.
Correlations of Rankings of the Harmonic Mean (HM) and 20 RAM for 10 Leading Econometrics Journals.
| HM | 2YIF | 2YIF* | IFI | 5YIF | Imm | 5YD2 | h-index | C3PO | PI-BETA | Eigenf | AI | CAI | H-STAR | 2Y-STAR | ESC | SIF | RIF | DIF | RDIF | h-RePEc | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| HM | 1 | ||||||||||||||||||||
| 2YIF | 0.539 | 1 | |||||||||||||||||||
| 2YIF* | 0.539 | 1.000 | 1 | ||||||||||||||||||
| IFI | 0.261 | –0.103 | –0.103 | 1 | |||||||||||||||||
| 5YIF | 0.867 | 0.768 | 0.770 | 0.030 | 1 | ||||||||||||||||
| Imm | 0.842 | 0.418 | 0.418 | 0.309 | 0.697 | 1 | |||||||||||||||
| 5YD2 | 0.249 | –0.527 | –0.527 | 0.249 | 0.042 | 0.297 | 1 | ||||||||||||||
| h-index | 0.879 | 0.770 | 0.770 | –0.042 | 0.903 | 0.709 | –0.103 | 1 | |||||||||||||
| C3PO | 0.903 | 0.806 | 0.806 | 0.055 | 0.915 | 0.746 | –0.115 | 0.988 | 1 | ||||||||||||
| PI-BETA | 0.746 | 0.442 | 0.442 | 0.346 | 0.685 | 0.600 | 0.030 | 0.758 | 0.770 | 1 | |||||||||||
| Eigenf | 0.758 | 0.758 | 0.758 | –0.273 | 0.794 | 0.564 | –0.261 | 0.927 | 0.915 | 0.649 | 1 | ||||||||||
| AI | 0.624 | 0.891 | 0.891 | –0.297 | 0.782 | 0.406 | –0.321 | 0.830 | 0.842 | 0.455 | 0.891 | 1 | |||||||||
| CAI | 0.806 | 0.879 | 0.879 | –0.055 | 0.879 | 0.661 | –0.297 | 0.964 | 0.976 | 0.733 | 0.939 | 0.879 | 1 | ||||||||
| H-STAR | 0.677 | 0.240 | 0.240 | 0.786 | 0.404 | 0.743 | 0.240 | 0.448 | 0.524 | 0.513 | 0.229 | 0.153 | 0.393 | 1 | |||||||
| 2Y-STAR | 0.312 | –0.019 | –0.019 | 0.973 | 0.070 | 0.337 | 0.146 | 0.006 | 0.121 | 0.375 | –0.159 | –0.197 | 0.032 | 0.793 | 1 | ||||||
| ESC | 0.024 | –0.224 | –0.224 | 0.694 | –0.106 | –0.106 | 0.129 | –0.294 | –0.188 | 0.176 | –0.306 | –0.318 | –0.224 | 0.269 | 0.763 | 1 | |||||
| SIF | 0.733 | 0.685 | 0.685 | 0.018 | 0.806 | 0.649 | –0.042 | 0.879 | 0.842 | 0.612 | 0.673 | 0.624 | 0.806 | 0.437 | –0.032 | –0.447 | 1 | ||||
| RIF | 0.709 | 0.891 | 0.891 | –0.079 | 0.879 | 0.394 | –0.309 | 0.855 | 0.867 | 0.539 | 0.818 | 0.891 | 0.879 | 0.262 | –0.006 | –0.118 | 0.733 | 1 | |||
| DIF | 0.539 | 0.746 | 0.746 | –0.152 | 0.782 | 0.515 | –0.139 | 0.757 | 0.721 | 0.442 | 0.576 | 0.624 | 0.733 | 0.197 | –0.210 | –0.518 | 0.927 | 0.709 | 1 | ||
| RDIF | 0.588 | 0.746 | 0.746 | –0.273 | 0.842 | 0.406 | –0.067 | 0.782 | 0.746 | 0.358 | 0.685 | 0.794 | 0.721 | 0.142 | –0.286 | –0.471 | 0.794 | 0.842 | 0.842 | 1 | |
| h-RePEc | 0.794 | 0.831 | 0.831 | –0.151 | 0.857 | 0.642 | –0.252 | 0.983 | 0.970 | 0.705 | 0.957 | 0.882 | 0.983 | 0.350 | –0.085 | –0.364 | 0.844 | 0.857 | 0.756 | 0.768 | 1 |
H-STAR and 2Y-STAR for the 10 econometrics journals are very high, with a mean of 93 and a range of 82–98 for H-STAR, compared with a much lower mean of 73 for all Economics journals in ISI, and a lower mean of 87 and a wider range of 70–100 for 2Y-STAR, compared with a much lower mean of 64 for all economics journals (see [7]). The H-STAR and 2Y-STAR means of 93 and 87 reflect journal self-citations of 3.5% and 6.5%, respectively, historically and for the preceding two years, which are very low compared with all of Economics. On average, journal self-citations have increased over the preceding two years as compared with historical levels. The ESC mean is 3, with a range of −1–9. On average, self-citations are escalating, with two journals decreasing in self-citations in the preceding two years relative to historical levels, and eight journals increasing in self-citations.
The PI-BETA scores are illuminating. The mean is 0.243, with a range of 0.1–0.404 so that, on average, one in every four papers published in the 10 leading econometrics journals is not cited, not even by the authors. In comparison, with a mean PI-BETA of 0.492, one in every two papers that are published in the leading 299 journals in Economics is not cited (see [7]). The PI-BETA values in Table 1 are much lower than for Economics journals listed in ISI, but are very similar to those in many disciplines in the sciences (see [4]).
The RePEc RAMs in Table 2 are illuminating. The simple impact factor (SIF) has a mean of 15.829 and a range of 6.948–46.688. The mean is considerably higher than the means of 2YIF and 5YIF in Table 1, but this can be explained by the fact that the citations base of journals in RePEc is roughly six times as large as in ISI, even though RePEc excludes journal self-citations. The recursive, discounted and recursive discounted impact factors, namely RIF, DIF and RDIF, respectively, have means of 0.772, 3.748 and 0.840, and ranges of 0.111–2.839, 1.597–9.622 and 0.157–2.746, respectively.
The mean h-RePEc is 68, with a range of (20, 174). Despite excluding journal self-citations, the mean h-RePEc of 68 is very similar to the mean h-index of 63 in Table 1 for ISI, which includes journal self-citations. The range of 11–181 for the h-indexes in Table 1 is also very similar to the range of 20–174 for h-RePEc in Table 2.
The pairwise correlations of 20 RAMs for the 10 leading econometrics journals based on the raw RAM scores in Table 1 and Table 2 are given in Table 3. There are 66 pairs of RAMs for which the correlations exceed 0.9 (in absolute value) in Table 3.
The correlations of 0.996 for the pair (2YIF, 2YIF*), 0.995 for (RIF, RDIF), 0.993 for (h-index, h-RePEc), 0.992 for (SIF, DIF), and 0.991 for (h-index, C3PO) are extremely high, which suggest that, among others, the two-year impact factors including and excluding self-citations are very similar for the leading econometrics journals. A similar comment applies to the very high correlations for the other four pairs, including RIF and RDIF, SIF and DIF, and the h-index with each of h-RePEc and C3PO. The 10 pairwise correlations for the five RePEc RAMs are all very high and lie in the range 0.909–0.995, which suggests that they provide similar information to each other, whether simple, recursive, discounted, or recursive discounted impact factors are used. The five RePEc RAMs are also very highly correlated with most of the 15 ISI RAMs. Interestingly, there are numerous pairs for which the pairwise correlations are relatively low, which suggests that they provide useful additional information about journal impact and influence.
One of the primary purposes of the paper is to provide robust rankings and to determine if reliance on the classic 2YIF, to the exclusion of the other RAMs, might lead to a distorted evaluation of journal quality, impact and influence. In order to provide a robust rankings measure based on the 20 RAMs, the rankings of the 10 leading econometrics journals given in Table 4 are based on the harmonic mean.
The journals in Table 4 are ranked according to the harmonic mean of the ranks (given as HM). Bearing in mind that no standard errors are available for these rankings, in comparison with the rankings in Table 1 that are based on 2YIF, only two journals remain unchanged in Table 4, namely Econometrica at number 1 and the Review of Economics and Statistics at number 2. These two journals were ranked identically in [5]. The other eight econometrics journals have changed positions relative to their rankings based on 2YIF in Table 1. The Oxford Bulletin of Economics and Statistics has shifted its ranking by seven places from 10 to 3, the Econometrics Journal has moved from 7 in Table 1–10 in Table 4, and the remaining six journals have shifted by one or two places in either direction.
The rankings based on the h-index and h-RePEc are virtually identical, with seven journals having the same ranking according to either RAM, and the remaining three journals being shifted by only one position. Thus, it would seem that whether journal self-citations are included or excluded does not seem to affect the relative rankings of the 10 leading econometrics journals.
It is widely acknowledged that the use of the harmonic mean of the ranks may be seen as rewarding or penalizing widely-varying rankings across alternative RAMs, with high rewards for particularly high rankings or, equivalently, low rank scores. The harmonic mean of the ranks tends to reward journals with strong individual performances according to one or more RAMs, with one or more strong performances leading to greatly improved rankings. This is most evident for the Oxford Bulletin of Economics and Statistics, which has a wide range of 1–10, with five scores of 1 and three scores of 10. Econometrica also has a wide range of 1–9, with 14 scores of 1 and individual scores of 8 for IFI and 9 for ESC, while the Journal of Econometrics also has a wide range of 2–10, with six scores of 2 and three scores of 10. The Journal of Business & Economic Statistics and Econometric Reviews both have a range of 2–9, while the Journal of Financial Econometrics has a range of 3–10 and Econometric Theory has a range of 4–10. Three journals have relatively narrow ranges, with the Review of Economics and Statistics having a range of 1–5, the Journal of Applied Econometrics having a range of 3–7, and the Econometrics Journal having a range of 6–10.
There may be strong disagreement among the weights to be used, as well as about whether the harmonic, geometric or arithmetic means of the ranks might be an appropriate Pythagorean mean for purposes of obtaining ranks of journals. The RAMs provided in Table 1, Table 2, Table 3, Table 4 allow alternative weights to be used for different journals, but a concentration on 2YIF alone, with corresponding zero weights for all other RAMs, would seem to be excessively restrictive. A similar comment would apply to the use of any single RAM as compared with a broader number of RAMs, especially the harmonic mean. Regardless of whether the harmonic mean should be preferred to its arithmetic or geometric mean counterparts, it is clear that the harmonic mean should be preferred to any single RAM on the basis of its robustness to a broader range of citation criteria.
The simple ranking correlations of the 20 RAMs for the 10 leading econometrics journals, based on the rankings in Table 4, are given in Table 5. The correlations in Table 5 are not very close (in absolute value) to the correlations in Table 3 for the raw RAM scores. There are 16 RAM pairs for which the correlations exceed 0.9, with the two highest correlations being for the pair (2YIF, 2YIF*) at 1.0 and (h-index, C3PO) at 0.988, which show that the rankings according to 2YIF and 2YIF* would be identical, and would be virtually identical according to the h-index and C3PO. For the RePEc rankings, unlike the very high pairwise correlations in Table 3, the highest correlation is for the pair (SIF, DIF) at 0.927.
In Table 5, the five highest correlations with the Harmonic Mean (HM) are for C3PO (at 0.903), h-index (at 0.879), 5YIF (at 0.867), Immediacy (at 0.842), and CAI (at 0.806), which suggests that the classic two-year impact factor including journal self-citations (2YIF) is less correlated (at 0.539) with the Harmonic Mean than are numerous other RAMs. For the RePEc rankings, the highest correlation with the Harmonic Mean is 0.794 for h-RePEc, while the lowest correlation is 0.539 for DIF, which is the same as for 2YIF. Thus, 2YIF would not seem to be a robust individual RAM to use if it were intended to capture the harmonic mean of the ranks. Indeed, using 2YIF as a single RAM to capture the quality of a journal would lead to a distorted evaluation of a journal’s impact and influence.
4. Concluding Remarks
The paper focused on the robustness of rankings of academic journal quality and research impact of 10 leading econometrics journals taken from the Thomson Reuters ISI Web of Science (ISI) Category of Economics, using 15 quantifiable Research Assessment Measures (RAMs).based on weighted and unweighted citations data from ISI and five RAMs from the Research Papers in Economics (RePEc) database, which is widely used in economics, finance and related disciplines. The harmonic mean of the ranks of the 20 RAMs, which in some cases are closely related, were also presented for these 10 leading econometrics journals as a robust rankings method.
The similarities and differences in various RAMs, which are based on alternative weighted and unweighted transformations of citations, were highlighted to show which RAMs are able to provide incremental informational value relative to others in capturing the impact and performance of the leading econometrics journals. Other RAMs were shown not to be highly correlated with each other, in which case they did provide useful additional information.
The increasingly prominent problem of coercive self-citations (see [17]) would seem to be one reason as to why RePEc excludes self-citations, both for journals and for individuals. Nevertheless, journal self-citations were shown not to have a serious effect on the journal rankings as the appropriate RAMs, namely 2YIF*, Article Influence, RIF and RDIF, were highly correlated with each other in terms of their raw scores. Moreover, the h-index and h-RePEc values were highly correlated, both in terms of their raw scores and also in terms of the journal rankings.
The correlation coefficient between the harmonic mean and the most widely-used RAM, 2YIF, was only 0.539, which is the equal 14th highest correlation coefficient. This relatively low value emphasizes the fact that the two-year impact factor of a journal could lead to a distorted evaluation of journal quality, impact and influence relative to the harmonic mean of the ranks of RAMs that included the impact factor, mean citations and non-citations, number of high quality papers, journal influence and article influence. The highest simple correlations with the harmonic mean were for C3PO at 0.903, the h-index at 0.879, 5YIF at 0.867, and Imm at 0.842, all of which were superior to 2YIF if an individual RAM were to be chosen.
Acknowledgments
The authors are most grateful for very informative discussions and correspondence with Esfandiar Maasoumi and Christian Zimmermann, and the helpful comments and suggestions of an external editor and two reviewers. For financial support, the first author wishes to thank the National Science Council, Taiwan, and the second author wishes to acknowledge the Australian Research Council and the National Science Council, Taiwan.
Conflicts of Interest
The authors declare no conflict of interest.
References
- ISI Web of Science. Journal Citation Reports; Essential Science Indicators. New York, NY, USA: Thomson Reuters, 2011. [Google Scholar]
- P.O. Seglen. “Why the impact factor of journals should not be used for evaluating research.” BMJ: Br. Med. J. 314 (1997): 498–502. [Google Scholar] [CrossRef]
- C.-L. Chang, M. McAleer, and L. Oxley. “What makes a great journal great in economics? The singer not the song.” J. Econ. Surv. 25 (2011): 326–361. [Google Scholar] [CrossRef]
- C.-L. Chang, M. McAleer, and L. Oxley. “What makes a great journal great in the sciences? Which came first, the chicken or the egg? ” Scientometrics 87 (2011): 17–40. [Google Scholar] [CrossRef]
- C.-L. Chang, M. McAleer, and L. Oxley. “Great expectatrics: Great papers, great journals, great econometrics.” Econ. Rev. 30 (2011): 583–619. [Google Scholar]
- C.-L. Chang, M. McAleer, and L. Oxley. “How are journal impact, prestige and article influence related? An application to neuroscience.” J. Appl. Stat. 38 (2011): 2563–2573. [Google Scholar] [CrossRef]
- C.-L. Chang, E. Maasoumi, and M. McAleer. “Robust ranking of journal quality: An application to economics.” Econ. Rev. 2013. (submitted). Available online: http://ideas.repec.org/s/emo/wp2003.html. (accessed on 4 October 2013).
- C.-L. Chang, and M. McAleer. “Ranking journal quality by harmonic mean of ranks: An application to ISI Statistics & Probability.” Stat. Neerland. 67 (2013): 27–53. [Google Scholar] [CrossRef]
- C.-L. Chang, and M. McAleer. “What do experts know about forecasting journal quality? A comparison with ISI research impact in finance.” Ann. Financ. Econ. 8 (2013): 1–30. [Google Scholar]
- C. Zimmermann. “Academic rankings with RePEc.” Econometrics. 2013. (submitted) Available online: as Working Paper 2012-023A, Federal Reserve Bank of St. Louis: St. Louis, MO, USA, 2012: http://www.research.stlouisfed.org/wp/2012/2012-023.pdf..
- “Eigenfactor.” Available online: www.eigenfactor.org (accessed on 28 September 2013).
- C. Bergstrom. “Eigenfactor: Measuring the value and prestige of scholarly journals.” C&RL News 68 (2007): 314–316. [Google Scholar]
- C.T. Bergstrom, and J.D. West. “Assessing citations with the Eigenfactor™ metrics.” Neurology 71 (2008): 1850–1851. [Google Scholar] [CrossRef]
- C.T. Bergstrom, J.D. West, and M.A. Wiseman. “The Eigenfactor™ metrics.” J. Neurosci. 28 (2008): 11433–11434. [Google Scholar] [CrossRef]
- J.E. Hirsch. “An index to quantify an individual’s scientific research output.” Proc. Natl. Acad. Sci. USA 102 (2005): 16569–15572. [Google Scholar] [CrossRef] [Green Version]
- “IDEAS:Rankings.” Available online: http://ideas.repec.org/top/ (accessed on 4 October 2013).
- C.-L. Chang, M. McAleer, and L. Oxley. “Coercive journal self-citations, impact factor, journal influence and article influence.” Math. Comput. Simul. 93 (2013): 190–197. [Google Scholar] [CrossRef]
© 2013 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/3.0/).
Share and Cite
MDPI and ACS Style
Chang, C.-L.; McAleer, M. Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc. Econometrics 2013, 1, 217-235. https://doi.org/10.3390/econometrics1030217
AMA Style
Chang C-L, McAleer M. Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc. Econometrics. 2013; 1(3):217-235. https://doi.org/10.3390/econometrics1030217
Chicago/Turabian StyleChang, Chia-Lin, and Michael McAleer. 2013. "Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc" Econometrics 1, no. 3: 217-235. https://doi.org/10.3390/econometrics1030217
APA StyleChang, C. -L., & McAleer, M. (2013). Ranking Leading Econometrics Journals Using Citations Data from ISI and RePEc. Econometrics, 1(3), 217-235. https://doi.org/10.3390/econometrics1030217
