Yuko Nagai
(Secretary-General, Zoological Society of Japan; Director, UniBio Press;
doctoral candidate, Graduate School of Library, Information and Media Studies, University of Tsukuba)

● The Evaluation of Japanese Academic Journals

For a long time now, we have been hearing that Japanese academic journals are no good. Yet, sad to say, no thorough study has ever been done to find out what is wrong with them or why. Nor is there a White Paper that provides an overview.1 On what grounds, then, have Japanese journals been dismissed? First, we are told, “They can’t even attract contributions from researchers in Japan, let alone overseas.” But no one has ever looked into the reasons for this. Until very recently, in fact, nothing had been done to identify causal factors or develop measures to address the problem. Then, “Toward Strengthening the Functions of Academic Associations,”2 an external report issued on June 28, 2007, by the Science Council of Japan, broached the fact that the research evaluation system used in Japan is causing problems for the nation’s journals. Listing the issues that face Japan’s academic associations, the report first cites declining membership, then goes on to say: “Secondly, as journal publishers, Japan’s academic associations are fighting an uphill battle for subscribers, citations, and so on, against intense competition from their European and American counterparts and commercial publishing houses, especially in fields where there is international rivalry on the frontiers of research. The problem is compounded by Japan’s evaluation system, which induces young, ambitious researchers to publish their work in overseas journals, thus sapping the vitality of Japan’s own academic associations and creating a vicious circle.” The first point in the quoted passage is not unique to Japanese journals, since competition is intense throughout the world of scholarly communication in which journals play a central role, and every publication is fighting for survival. But the warning that Japan’s evaluation system makes it difficult for researchers to contribute and leads to researchers either not contributing or not wanting to contribute to their own country’s journals, thereby weakening Japan’s academic associations, was without question long overdue. The problem is largely a historical one, of course; Europe and the U.S. have many excellent academic journals, and it is only natural for any researcher—Japanese or otherwise—to want to have a submission accepted by them. That said, the fact remains that the issue of scientific evaluation in Japan has gone a long way toward defining Japanese journals.

Japan’s evaluation system is founded on the Impact Factor (IF). In a system that favors publication in high-IF journals, as long as research funds are at stake, even a researcher who is well aware of the IF’s limitations will tend to submit papers first to the high-IF journals in his or her field. How high on such a researcher’s wish list are Japanese journals likely to rank?

In view of the excellent quality of the researchers who make up this country’s academic associations, their journals should not be inferior. And yet, in practice, we have created a situation where these researchers cannot contribute to Japanese journals even if they want to. Not recognizing that fact, some have actually insisted all along that Japanese journals are “no good” and “of no interest.” It would be out of touch with reality, as I have shown, to argue that Japanese researchers should only contribute to Japanese journals as a way of stanching the flow of first-class research overseas. Yet, by the same token, how could those whose job it was to defend Japanese scholarship have insisted that the sole proof of a paper’s quality is publication in an overseas journal with a high IF? If researchers themselves decided the evaluation guidelines that affect funding, then I suppose they have made their own bed and now they must lie in it. As one who has long been involved at the day-to-day level with publishing a Japanese journal, I cannot say that this gives me any comfort.

In this article, while giving examples of our IF-dependent research evaluation system, I reexamine the meaning of the IF in light of the fact that being perennially regarded as second-best in research evaluations is sapping the vitality of Japanese journals. I then offer an approach that should allow the associations to do more than gaze wistfully at the movements of their IFs; by utilizing the Journal Citation Report (JCR)3 and other useful databases, they should be able to gain a more composite picture of their own journals and thus propose a course of action to help develop a new evaluation system. With regard to evaluation, it should be noted, however, that as part of a general trend, digitization may well be ushering in an era of evaluation of individual articles rather than of the journals.

● Examples of Research Evaluation Based on the IF

Let us start by looking at what a news item and a call for submissions can tell us about the state of scientific evaluation in Japan. These examples are taken from actual documents posted on websites; I have deleted any identifying information such as names of institutions and departments.

Personally, I do not understand what is meant by “total IF,” but the expression seems to be used in Japan as if its meaning were obvious, and researchers have at times been required to provide this figure.

When the final scores of all papers, obtained in this way, are added together, the total is the research performance evaluation of the project leader who applies for funding.

In this call for submissions, the basic values that push up the score “D” derive from being published in a journal with the highest possible rank (the attached table was not available on the website) and the highest possible IF.

● What is the IF?

On September 8, 2005, the Ministry of Education, Culture, Sports, Science and Technology (MEXT) published “Revision of MEXT Evaluation Guidelines for Research and Development” on its website and also sent it to academic associations and other related institutions. The document contained the following words of caution:

“The Impact Factor is a metric that serves as a guide to the degree of influence a journal exerts in a specific research field. It is essential to be aware that it is not an indicator of the quality of the papers the journal publishes, and to exercise due caution in its use.”4

Yet, for some reason, misconceptions about the IF still abound. Librarians alone seem to be immune, viewing the IF as an index to help “choose important titles from among the ever-growing number of academic journals.”5 Although the situation has become especially complicated in Japan due to the IF’s role in scientific evaluation, in other countries, too, it continues to exert a strong influence on the assessment of research, and many researchers are already aware of its problematic nature.6,7 It is a very simple index, and researchers are surely not incapable of grasping its true significance.

Exactly when the IF began to affect research evaluation in Japan is hard to say. Since 1996, applicants for journal publishing subsidies have been required to enter their IF on the application form for “Scientific Periodicals Subsidized by Kakenhi (Grant-in-Aid for Publication of Scientific Research Results),” which was then under the jurisdiction of the Ministry of Education and is now under that of the Japan Society for the Promotion of Science. As this form is used to decide whether to subsidize publication of a particular journal, it is quite appropriate to employ the IF in this way. The form also includes a space for the number of cites in a specific year, according to JCR data, for all the papers the journal has ever published. In this connection, I would like to see it become common practice, when entering IFs on application forms or writing about them in newspaper articles and the like, to indicate how many papers the journal published in the years in question. Since the IF is derived from papers published in the previous two years, when stating the IF for 2009, for example, one should always note the number of papers published by the journal in 2007 and 2008.

Given the IF formula, as I see it, a journal that takes on a large number of papers runs the risk of publishing some that will not be cited. These days, the workings of the IF are well understood, and in view of the possibilities—choosing papers for their citability, gaming the system by publishing as few papers as possible—manipulation of the IF cannot be entirely ruled out. But academic associations do not publish journals to boost their IF. They accept papers that are right for the journal and valuable to the branch of learning that they represent. They publish in order to advance the state of knowledge in their field.

Let me explain how an IF calculation is affected by the size of the journal’s total output.

For example, in the JCR for 2005, Wildlife Monographs headed Zoology with an IF of 5.286, but only three papers were published under this title in 2004 and four in 2003. Of course, the high IF is due to the fact that those seven papers were cited 37 times. But let us adopt a different perspective: how many of the papers published in its field over two years might one journal be responsible for?

The table at right gives the IF and Total Cites in JCR 2005 for Zoological Science, the journal of the Zoological Society of Japan (ZSJ), together with my calculation of the ratio of the number of papers in Zoological Science to the total number of papers published in JCR-listed zoology journals in 2004.

An academic association is responsible for guaranteeing the quality of the papers it publishes. This involves investing human, physical, and financial resources to support publication, which in turn helps the field advance. I have taken Zoological Science as an example here because, while high-IF journals are undoubtedly good journals, it helps to understand the IF correctly if one realizes that a journal’s quality is actually made up of many aspects. It seems to me that academic associations are unquestioningly accepting the IF, an apparently objective numerical value assigned to them by a third party. Certainly, it has a meaning, and that needs to be understood. But shouldn’t those responsible for publishing a journal be the ones who define it to others? For the association that publishes a journal surely understands it best. Utilizing JCR and several other databases, Japan’s academic associations should draw on the results they see there in a continual endeavor to make their journals better.

Thomson Reuters, which calculates the IF values, has explained on numerous occasions that “the IF is an indicator of a journal’s performance, not of the quality of its papers.” Dr. Eugene Garfield, the IF’s originator, has said the same.8 Please note the following points in regard to the simple calculation formula, which is shown below:

• ISI, now Thomson Reuters, is a commercial enterprise. It is not commissioned by the government to calculate the IF and other data, and it is not itself a public institution.
• The IF does not indicate the quality of the individual papers in a journal.
• Only citations in journals selected by Thomson Reuters are counted. The Web of Science (WoS), the basis of IF calculation, does not include every journal in the world, and naturally a new journal does not automatically receive an IF. The IF is calculated according to the presence or absence of cites among the journals found in WoS.

● Method of Calculation of the IF for 2009
　 (published in June 2010)

Figuer 1: Schematic Diagram of Journals Selected by Thomson Reuters

The IF is always announced for the previous year. The formula remains unchanged since it was devised by Dr. Garfield, and my description of it here contains nothing new. And yet, clearly, many users, with the exception of librarians, do not really understand the IF.

Although the IF does not measure the quality of individual papers, it does provide an overview of a journal’s performance in a specified period. The JCR data can also be used to show a cross-section of trends within a particular journal, for example, as an aid to studying editorial policies. In fact, the IF is just one of a number of indices offered by JCR. The Thomson Reuters website “JCR on the Web” explains its products as follows:

• Impact Factor: a measure of the frequency with which an average article in a specific journal is cited in a particular year. A journal’s IF represents its degree of influence. The number can be used to compare a journal’s relative importance to others in the same field.
• Immediacy Index: an indicator of the number oftimes an article published in a specific year within a specific journal is cited over the course of that same year. This number is useful for comparing journals that publish cutting-edge research.
• Articles Counts: the number of articles (original and review) published in a journal in a specific year.
• Cited Half-life: This benchmarks the age of cited articles by showing the number of years back from the current year that account for 50 percent of the total number of citations to a journal in the current year. For libraries, this number is useful in making collection management and archiving decisions; a publisher may use it to make comparisons with rival journals in different fields and adjust editorial policies.
• Source Data: The Source Data provides information regarding the number of review articles versus original research articles that are published by a particular journal. This number also includes the number of references cited by the articles.

Figuer 2: Proportion of papers cited at least once (Endocrinology, Nature, Zoological Science)
Data from Thomson Scientific Journal Performance Indicators database 1981-2004; prepared by Nobuko Miyairi, Senior Analyst, Thomson Scientific (now a division of Thomson Reuters)

Source: Seibutsu Kagaku Nyūsu (Biological Science News) (ZSJ edition)

The graph in Figure 2 is taken from an article by Noriko Miyairi of Thomson Reuters in Seibutsu Kagaku Nyūsu (Biological Sciences News; co-published by the Botanical Society of Japan and the ZSJ).9

It shows the percentage of papers in Endocrinology, Nature, and Zoological Science that have been cited at least once.

As can be seen, Zoological Science has been making small gains, albeit slowly. At the same time, the graph reveals that even Nature carries some papers that are never cited. To rate work highly on the basis of its being accepted by a journal as prestigious as Nature is a valid means of evaluation, but it would be misleading to depend on it exclusively.

In recent years, there has been a constant search for a new system of scientific evaluation. Japan’s journals are looking to the eventual creation of a more effective system as the key to their survival. The IF will likely continue to be an influential indicator for associations that publish journals, but no longer the only one; indeed, we need to be aware that the era of relying on the IF is coming to an end. Times have changed in favor of electronic journals. Print journals will, of course, survive in some fields and are unlikely to disappear completely. However, e-journals will be the tools that researchers employ in their work; print journals will no longer fill this role in the era and the generation to come. Further, an innovation offered by e-journals, the access log, holds out the possibility of measuring the performance not only of individual journals but of individual papers. Many Japanese academic associations lack the financial resources to respond flexibly to these new developments or to take the initiative. With the outstanding advancement of science and technology, digital data is becoming harder to manage than we had imagined. As journals work with ideas for better ways to deliver content to researchers and attractive ways of publishing papers—ideas that did not exist in the print era—will these new approaches have an effect on the IF?

I would like now to list a number of projects being developed as new evaluation systems, or products recently launched as evaluation and search tools, together with the URLs of the relevant sites. No doubt they will already be familiar to many readers, but please bear with me, as frequent repetition is the best way to share information. Searching the Web I found various other efforts to develop new evaluation methods, but here I will present only those projects where I have actually met and talked with the people concerned and gained a good understanding of the contents. As one involved in scholarly communication, what I look for in evaluations is diversity, the use of several indicators. I would like to see a rethinking of scientific evaluation in Japan, starting from the recognition that it is a difficult and inherently risky task, like quantifying one’s evaluation of a human being. This reexamination should take into account the problem pointed out in the Science Council of Japan’s report quoted at the beginning of this article, namely, that the current research evaluation system is causing problems for Japanese journals.

● The Search for New Evaluation Methods

１. Usage Factor

In the digital era, it has become technologically possible to measure the performance of individual papers by counting accesses. Because the number of accesses and the number of downloads (which are related) are influenced by the scale of the platform and the quality of the system, among other things, their validity as an evaluation system is being examined with great care. There is also the problem of excluding manipulation of download statistics. Further, the UF will not solve the problem of papers in different fields being used differently. It should be noted that papers in more flourishing fields with bigger populations of researchers will obtain higher UFs.10

2. Scopus

This is a commercial database developed by Elsevier. Scopus differs from WoS in that it contains a wider range of journals and it can calculate the H-index, which measures the research activity of individual scientists or scholars.11

3．Eigenfactor

This noncommercial project grew out of an attempt by a team consisting mainly of graduate students at the University of Washington to find a new method of evaluating research taking WoS as their basic data. They were initially sponsored by a university laboratory, but Thomson Reuters introduced the Eigenfactor starting with the 2009 WoS. Metrics such as the Eigenfactor, which measures the influence of a journal, and the Article Influence score, which measures the influence of an individual paper, can be visualized instantly and also viewed as numerical values using free source code.12

4. Impact Deviation Value

Table 1: IDV of Japanese Journals
Source: Negishi, Masamitsu. “IDV or Impact Deviation Value: A Proposal of a Normalized Impact Factor for Performance Evaluations.”

This is a bold attempt that, in an entirely new approach, treats the IF as a useful approximation for assessing papers published in the previous year as part of a performance evaluation, and at the same time uses the deviation values of IFs in each field to derive a normalized index comparable across fields. The IDV has some interesting features, such as being far simpler to calculate than the Eigenfactor, and the fact that it results in shifts in the upper ranks of Japanese journals. The question of how to close the gaps between fields is a major problem with the IF; even within biology, one cannot compare the IFs of journals in developmental biology and taxonomy. As a result, nineteen Japanese journals have a deviation of over 50.0, a situation which presents a severe problem. (See Table 113)

Dr. Garfield, the originator of the Impact Factor, had a truly great idea, simple though the underlying concept may be. Due to its very simplicity, however, coupled with its catchy name, the IF has tended to be misunderstood in Japan and elsewhere. One of the main characteristics of academic journals and scholarly publishing in this country is the fact that they face difficult problems stemming from misconceptions about the IF in the area of research evaluation, together with a disadvantageous history and the need to compete with overseas journals that have long enjoyed the support of researchers.

(Continued in the next issue)

References