● How It All Started

Lecture by Michiel Kolman (Senior Vice President, Elsevier)


Lecture by Yoko Hirose (Solution Management, Scientific & Scholarly Research Division, Thomson Reuters)


Lecture by Sun Yuan (Associate Professor, National Institute of Informatics)

I assumed the current position at the Organization for Research Promotion & Collaboration of Okayama University five years ago and this is my first time to work at a university. Although I had worked with university professors through my previous job, I was completely new to the actual situation and inner reality of universities. In this regard, I might be privileged to see things through fresh eyes.

In getting started in my current position, I was unsure about the status of research activities within the university. Thus, hoping to create something like a research map, I participated in various seminars relating to research evaluation, which led to my encounter with bibliography and citation databases.

Being a complete newcomer, I started with studying how I can utilized such databases myself, participating in seminars organized by various database providers including Elsevier and Thomson Reuters, from which we had lecturers for the 1st SPARC Japan Seminar 2012. Those seminars are organized partly for advertisement purposes and therefore never focus on anything negative about databases provided by sponsoring companies. After attending several seminars of the kind and having learned only good aspects of databases, I felt I would be able to make use of them and thus started off with my work.

In retrospect, it is laughable just how naïve and simple-minded I was back then.

● Limitations of Bibliography and Citation Databases and Understanding Thereof

To begin with, I went over to the library and sought advice from a library staffer presumed to be knowledgeable about bibliography and citation databases. There, I met with a faculty member who was also intending to start work on research evaluation. With the help of the library staffer, I started working on analysis using databases. As the results of my analysis turned out to be quite different from what I had expected, I examined each piece of data closely. Only then did I realize what a bibliography and citation database is supposed to be. That is, I finally realized that rigorous analysis inevitably requires enormous amount of work and that even the results of such rigorous analysis can lead to a wrong conclusion if not applied properly.

Just like scissors and kitchen knives, bibliography and citation databases can be either useful or harmful depending on how they are used. It is necessary to utilize databases based on thorough understanding of their limitations and shortcomings.

Essentially, science and technology can be considered to be useful only when they are proven to be useful for the existence of human beings or when they satisfy the intellectual interest of people. It is obvious to everyone that a research work cannot be considered an excellent one simply because it has been published in a journal with a high impact factor or cited by a number of other papers. That is, a research work is recognized as an excellent achievement only by actually contributing to the well-being of people at large. In this regard, we must always keep in mind that the use of bibliography and citation databases provides just one of many measures to evaluate research.

● Expectations on Bibliography and Citation Databases from the Viewpoint of Evaluators

The currently available bibliography and citation databases are problematic in that they do not include papers written in Japanese and thus put Japanese researchers at disadvantage. However, it is too costly, time-consuming, and thus unrealistic to create a database including all scholarly journals or add detailed bibliographic information that would allow for complete statistical treatment. Nevertheless, I believe that certain problems—e.g., the ambiguity of researcher ID and other problems in identifying researchers, and the handling of researchers moving from one organization to another—need to be addressed. Although these may be problems on the side of researchers, I hope to see something done about them.

● Meaning of My Participation in the SPARC Japan Seminar

For me, the best part of the seminar was listening to the exchange of ideas in the last session. I have encountered various problems myself in the course of analyzing research conducted within the university using bibliography and citation databases. It was indeed worthwhile and helpful just to know that there exist diverse views on databases.

It is true that bibliography and citation databases are playing a significant role today where there exist no absolute criteria for evaluating research. Thus, we need to continue to make good use of them.

With a great deal of importance attached to evaluation, it is expected that the role of bibliography and citation databases will become even more important. Given this prospect, further research and discussions involving third parties should take place to explore ways to use and improve such databases. Toward that end, there should be an open forum for discussion. I hope that organizations such as NII will provide information not only by holding seminars such as the one I attended but also by organizing open forums so as to aggregate a greater amount of opinions and views.

This is just a layman’s view based on what I learned by attending the seminar but I do hope that bibliography and citation databases will someday become more accessible and easier to use in a way to lead to a more accurate conclusion.