How to cite this article: Afshari R, Monzavi SM. Qualitative versus Quantitative Evaluation of Scientists' Impact: A Medical Toxicology Tale. Asia Pac J Med Toxicol 2014;3:134-40.

introduction

Evaluation of scientists working in a specific area of science is necessary, as they may strive for same limited resources, grants and academic promotions. Nevertheless, determining the performance parameters of a scientist has always been under debate (1). One of the most common and accepted methods of assessing the performance and impact of a scientist is calculating the number of citations for their publications (1,2). In that sense, the h-index proposed by Hirsch in 2005 has gained popularity (3). However, such method suffer from certain shortcomings. For example, the value of h-index is time dependent and relies upon the database used for calculating the number of citations (1). In addition, many scientists from the developing countries speak and produce science in non-English languages which are published in journals in their own language. These journals are less likely to be taken into account in major citation systems.
For some scientists, success resides within publishing more articles, gaining more citations and having higher h-index (4). However, it should be kept in mind that the impact a research leading to a successful change in health system policy or a new invention may have on improvement of public health, a highly cited paper may not. Besides, creating a meaningful change in system policies or invention of a new device takes more time and involves more people than studying on a specific subject matter and publishing its results. Therefore, it has become more and more obvious that evaluation of scientists should be qualitative in addition to quantitative (2). Moreover, it should be pragmatic and reflective of the priorities of an institution, a country or an intended population (1,2). In this context, in addition to scientometric analysis, other aspects for evaluation of scientist's impact have been suggested to be considered including research applicability and integrity, research economical rate of return, assistance in peer review of research papers and grant applications, benchmarking and capacity building (1,2,5-7). In the present paper, a scoring scale has been proposed which can be used as a prototype for evaluation of scientist's impact both qualitatively and quantitatively.
Details of the scoring scale for evaluation of scientist's impact
According to the proposed scoring scale, which can be called "360-degree researcher evaluation score", scientists are evaluated in 5 independent domains including (I) science development, (II) economic impact, (III) policy impact, (IV) societal impact and (V) stewardship of research (Table 1). For each domain the maximum score is determined to be 20. The domains contain many items that are categorized under subdomains. The allocated scores for items are researcher-driven, and further investigations are needed to be standardized.

It is noteworthy that the scale is designed for evaluation of impacts resulted from research activities and thus it excludes the educational programs done by a scientist. For some items the score that can be allocated to a given scientist is based on the rank of that scientist among the scientists working in a specific (target) institution (or organization). The important details of the domains and items are explained hereunder:
1. H-index can be calculated from Web of Science, Scopus, Google Scholar or some software designed for this purpose (8). Other indexing databases may also be applicable for calculating the h-index. However, Google scholar seems to be superior for this purpose as it is more inclusive, especially regarding the language diversity of articles included compared to other databases. Nevertheless, Google scholar includes many non-peer reviewed articles. It is therefore upon the policy of the target institution to opt for which database.
2. For calculation of the mean number of citations minus self-citations, the number of citations minus self-citations of all articles published by a scientist should be calculated and then the mean of them should be measured. The standard deviation (SD) of this mean will be used for the item "reciprocal value of SD of number of citations minus self-citation".
3. In order to calculate the score of published articles of a scientist, an additional score from the impact factor of the journal that published the article divided by 30 should be added to maximum score for each article.
4. For a same amount of grant that two scientists receive, the scientific productivity of the research done should be compared by the number of citations minus self-citations of articles produced.
5. Although research, in its nature, is useful to develop science, if it only remains in papers and does not translate into policy or change in society may not be influential. Two domains of policy impact and societal impact are added in this scale to encourage scientists not just focusing on increasing their publications.
Comparison of cost effectiveness and cost benefit analysis Comprehension of these two sets of scales is somehow difficult as they have close meanings. Cost benefit is a measure of total money return (or saving) expected from a new medical method per unit of money spent. On the other hand, in cost effectiveness, the effectiveness and applicability of alternative methods, strategies and interventions to achieve a specific set of results is compared. Cost effectiveness of research is referred to the impacts of research which led to improved quality in healthcare approaches or decrease in morbidity or mortality but currently may not be translated to economical values due to lack of supporting evidence or needing more time for evaluation (9). A cost beneficial method is undoubtedly a cost effective method that is also shown to be economical.
Role of the expert committee policy
As for some items, the level of an individual scientist working in an intended institution should be ranked; an expert committee should be formed. It is better to be comprised of senior policy makers, members of the board of directorate of an institution, a group of experts in that field of science and lay people. This committee will allocate comparative ranks to the scientists.
For some items an exact numerical value for the impact or benefit of a research is not possible to be calculated, which necessitates the ranking of scientists to be subjectively assessed by the expert committee based on priorities, their experience and personal judgment.

Discussion

Importance of evaluation scales in assessment of scientist’s impact
All institutions or organizations that fund for research are inclined to support science and scientists that make a progress. But there is no fixed formula for identifying the exact value of research and researchers (1,2). Some institutions consider the amount of science produced or in a better word, measurement of the quantity of the science production. Recently we have shown that the productivity of the science of toxicology in Asia Pacific region has been expanding faster than Northern America (10). Potential reasons were given in phenomena so called "Ceiling effect of science production" in Northern America and the "the rise of the rest" in Asia Pacific region. As a result, the gap between these regions is narrowing down (10). Nevertheless, measurement of the quantity of the science production might not be insightful, as there may be papers with limited or redundant findings (11). As a reason, and quite rightfully, the number of citations of articles and h-index of individuals or institutions have been suggested to overcome this weakness (3). Hence, the majority of institutions and organizations are now trusting on the h-index; despite having its own limitations and shortcomings. For an organization, defining and prioritizing outcome of research topics requires value judgments and decision-making (1,2). Most directors are seeking for a researcher that creates a real advancement, not only in basic levels but also in improvement of health policies in the administrative levels. The evaluation scale proposed in this article can be helpful in this matter.
Some scientists are only eager to increase their publications and citations as they may be encouraged by their institution (6). This may create a one-dimensional scientist (Figure 1).

Figure 1. Template of a two potential scientists including one whose research impact is mainly coming out from production of science (red line) and the other whose impact is coming out from wide range of research-related activities (blue line). It seems while science production is necessary to expand the edge of knowledge, it is probably more important to encourage scientist to work on other aspects as well.

Notwithstanding, an effective scientist is probably a multi-aspect one who opens new scientific fronts and connections to other disciplines (2), trains junior researchers to come up with him/her during the research, attempts to improve policies in healthcare, tries to build up improved capacities for research, helps other scientists write high quality papers (5), and is also eager to find effective and cost-beneficial approaches that strengthens healthcare and ultimately brings prosperity to the society.
In less affluent countries, there are scientists who dedicated their life to development of science in their society and in improvement of public health. However, they were not able to increase their citations as their knowledge has never been published or has only been disseminated in local journals. Probably a better metric than only h-index and scientometric analysis can provide a clearer image for their scientific impact. It seems that applicability, benchmarking and implementation of research are more important in underprivileged societies. Moreover, as in the developing world, the governmental support and non-governmental organizations (NGOs) are less developed; therefore, the independent impact of an individual scientist is vital. Perhaps these individuals should be defined as non-governmental individuals (NGIs). The value of their impact is heavily underrepresented if h-index alone is taken into account.
There has been an ongoing battle between basic and applied sciences. This article tries to favor applied sciences in scoring the research activities. Furthermore, it is indeed necessary to magnify the value of implementation of research-based principles. We have to set strategies in which scientists are also directly responsible for advocacy, science to policy translation and promoting public health literacy. This is where real effective scientists show their quality. Scientists with highly cited papers who spend most of their time in laboratories and libraries and are not inclined to change methods and implement policies have minor impacts on people’s life.

A medical toxicology tale
It is a privilege to see that in recent decades, the number of scientists working in the field of medical toxicology especially in Asia Pacific region has been increasing (12). As a result, the productivity in this field of science has increased having an upward slope (10). Nevertheless, several questions rise when the amount of the production by a medical toxicologist is taken into account: How does this amount of research contributed to growth of medical toxicology? Has the performed research been effective and cost beneficial in development of newer diagnostic approaches, treatment protocols, improved therapeutic formulations, updated interventions and ultimately upgrading public health? Have these pieces of research resulted in implementation of newer diagnostic approaches, treatment protocols, improved therapeutic formulations and updated interventions? Has medical toxicologist been able to collaborate with other toxicologists or scientists in other disciplines? Has medical toxicologist been capable of making change in policies? Has medical toxicologist been able to prepare fresh young researchers for the future of this discipline? Indeed, an effective and multidimensional medical toxicologist is the one who can receive positive answers for the majority of these questions.

Conclusion

For evaluation of a scientist, in addition to the amount of their scientific production and the citations, the economic impact, policy impact and societal impact of their research as well as their ability in stewardship of research is necessary to be assessed and quantified. This strategy is more important in the developing world where the opportunities and resources are scarce. For this purpose, the evaluation process should be moved from only scintometric indices to a combination of quantitative (including scientometric) and qualitative indices as we called "360-degree researcher evaluation score". This proposed scale seems to be a good start. Its overall coverage, subdomains and proposed weights of scores need to be standardized. Your comments on this scale are highly welcomed to be published in the next issues of the journal.

Conflict of interest: None to be declared.
Funding and support: None.