Article Text
PDF
Abstract
Introduction Academic papers are an essential manner for describing new ideas and consolidating existing concepts in the field of military medicine. The academic impact of military medical publications reflects the extent and depth of recognition, acceptance and utilisation of the concepts transmitted in these publications. The aim of this research was to construct an evaluation index system suitable for evaluating the academic influence of scholars in the field of military medicine.
Methods Using the Delphi consensus methodology, 30 experts from the field of military medicine, military medical information and library and information science were asked during three rounds of questioning to score the feasibility and importance of indicators that could be used to determine academic impact. An analytic hierarchy process method was used to calculate the relative weighting of each indicator in determining the final level of academic impact.
Results Eight evaluation indicators were agreed on to potentially determine academic impact. These comprised: ‘Web of Science documents’, ‘Citation impact’, ‘h-index’, ‘Percentage of international collaborations’, ‘Percentage of the top 10% of the cited frequency’, ‘Category normalised citation impact’, ‘Percentage of documents cited’ and ‘The number of F1000 Recommended papers’.
Conclusions The evaluation index system determined from this study combines the advantages of both qualitative and quantitative recognised evaluation indicators, which are subsequently weighted according to their importance in the field of military medicine. It is hoped that this framework will provide a manner in the future for comparing the potential academic impact of military medical scholars.
- health informatics
- academic evaluation
- evaluation index system
- scholars
- military medicine
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/
Statistics from Altmetric.com
Key messages
In this paper, we used Delphi method and analytic hierarchy process method to construct an evaluation index system.
The constructed comprehensive evaluation index system combines the advantages of qualitative indicators and quantitative indicators.
The evaluation index system will be used to evaluate military medical scholars’ academic impact according to their published articles.
Introduction
Academic papers are an essential manner for describing new ideas and consolidating existing concepts in the field of military medicine. By limiting their size and by conforming to a relatively set structure, information can be transmitted more effectively. Most importantly though the use of academic papers ensures peer review and often enables the general public to understand advances that are happening in the field. The academic impact of military medical publications reflects the extent and depth of recognition, acceptance and utilisation of the concepts transmitted in these publications.1 2 Appraising their academic impact may in some cases enable scholars to adjust their research direction towards that perceived to be more influential. It can also promote more extensive academic communication and cooperation.
Current methods of evaluating scholars’ academic impact include peer review, bibliometric methods,3 social network analysis4 and comprehensive evaluation.5 Peer review is a qualitative method for evaluating scientific research activities and related elements (such as researchers, research projects or achievements) by one or more people in a similar research field with proper competence. The bibliometric method statistically analyses certain numerical values to reveal the characteristics and value of the literature. An example of this is the InCites database, published by the Thomson Reuters Group in 2011. It combines nearly 30 years’ information from Web of Science to deliver reliable metrics and indicators. Due to its inherent wealth of bibliometric indicators, standardised national benchmark data for each subject, and intuitive visualisation of the results, InCites has become a powerful research tool. The social network analysis method evaluates the cooperation of scholars and the effect on a citation network. This includes the node degree, shortest path, weight of nodes and edge data. Finally when faced with complex issues, the comprehensive evaluation method uses multiple indicators to determine the academic impact. The aim of this research was to construct an evaluation index system suitable for evaluating the academic influence of scholars in the field of military medicine.
Method
Evaluation indicators for academic papers
The following 10 evaluation indicators were selected, of which the first nine were determined from the InCites database:
Web of Science documents
The total number of documents published by the scholar in this database within a certain period was used as a measure of a scholar’s academic productivity; the more documents published by an author, the more likely the author is to be a scholar with a high academic impact.
Citation impact
The value of the citation impact is obtained by dividing the total cited frequency of the published documents by the total number of documents of a scholar. Although the author’s total cited frequency also reflects the extent of his or her influence, this study selects citation impact because it shows the average number of citations of a scholar, which can eliminate the influence of the total number of papers on the total cited frequency. Additionally, citation impact has stronger stability.
H-index
The h-index6 was suggested in 2005 by Jorge E Hirsch. A scholar with an index of ‘h’ will have published ‘h’ papers, each of which will have been cited in other papers at least ‘h’ times. Thus, the h-index reflects both the number of publications and the number of citations per publication. The h-index is considered to be a measure of a scholar’s personal achievement in scholarly evaluations.
Percentage of international collaborations
This is the percentage of the number of published papers with international coauthors out of the total number of papers published by a scholar. Some researchers have suggested that international cooperation can help enhance the scientific research impact. This indicator reflects the scholar’s ability to attract international cooperation. The cooperation impact and citation impact complement each other and reflect the scholar’s academic impact together.7
Percentage of domestic collaborations
This is the percentage of the number of published papers with domestic coauthors out of the total number of papers published by a scholar.
Percentage of the top 10% of the cited frequency
This is the ratio of the number of papers in the top 10% of the cited frequency to the total number of papers published by the scholar in a subject area, in a particular year and in a specific document type. The indicator reflects a high level of scientific research.
Category normalised citation impact
Each of the papers belongs to one or several disciplines and the citations may be a discrepancy between different disciplines. This indicator means the actual cited frequency divided by the expected cited frequency—which is in the same subject area—with the same type of document and published in the same year. The scholar’s category normalised citation impact is the average of the category normalised citation impact of all the published papers by the scholar. This indicator can eliminate the influence of the subject area, publication year and type of literature, and so it can be used to evaluate the literature collection of different disciplines.
Category normalised citation impact: country adjusted
The category normalised citation impact is divided by the square root of the category normalised citation impact in its country. This indicator can eliminate the influence of different political backgrounds, different languages and different scales of scientific research networks on the publication and citation of the paper due to different regions.
Percentage of documents cited
The percentage of the number of papers being cited at least once, out of the total number of papers published by the scholar. This indicator reveals the extent of the citation of all the documents published by the scholar.
The number of F1000 Recommended papers
The F1000 database is a secondary literature database for evaluation and recommendation. Top scientists and outstanding researchers in biology and medicine read, evaluate and recommend the published papers. This indicator is based on the expert’s qualitative evaluation. The number of F1000 Recommended papers is converted into quantitative indicators. Although this indicator is not from the InCites database, it can be retrieved through F1000 or PubMed. It combines qualitative and quantitative evaluation so that the construction of the evaluation index system will be more scientific and reliable.5
Constructing the comprehensive evaluation index system using the Delphi method and analytic hierarchy process (AHP)
To evaluate the impact of academics in the field of military medicine, different weights should be given to indicators. In this paper, we chose 30 Chinese experts from the field of military medicine, military medical information and library and information science. To be included in this study, experts had to researchers who are active in the field of military medicine, and they were representative of all subspecialties within the discipline (Table 1). Web page links of the questionnaire were sent to experts through email, WeChat (a popular social platform in China) or SMS so that they could complete the survey online. They were asked to score the feasibility and importance of the indicators, and feedback was provided on the results of each round of the questionnaire. After three rounds of the questionnaire survey, the relative importance of each indicator was calculated. The AHP was then used to construct the judgement matrix to calculate the weight of each indicator.
Basic information on the experts
Questionnaire design
The questionnaire comprised three parts. The first part described the purpose of the study, how to complete it and the criteria for scoring. The second part was the question-and-answer section and listed the preliminary evaluation indicators and the connotations of each indicator. It asked the experts to evaluate and score the indicators according to feasibility and importance while using them to evaluate scholars’ academic impact. Moreover, we set up open-ended questions and asked experts to make suggestions on the evaluation index system. Finally, the third part of the questionnaire asked for experts’ personal information and the basis for estimate.
First round of questioning
The first round of questioning produced a response rate of 100%.8 It asked experts to evaluate the feasibility of the primary evaluation indicators. Several experts proposed deleting the indicator of ‘percentage of domestic collaborations’, because the indicator ‘percentage of international collaboration’ could better reflect the scholar’s academic impact. Additionally, seven experts believed that the feasibility of the ‘category normalized citation impact – country adjusted’ was poor. More than 85% believed that the other eight indicators had satisfactory feasibility, and therefore the other two indicators were removed before the second round.
Second round of questioning
The second questionnaire was sent to 30 people with a response rate of 93%. It asked experts to evaluate and score the indicators according to importance while using them to evaluate the scholars’ academic impact.9 For scoring, the importance of each indicator was divided into five levels: not important: 1 point; generally important: 3 points; comparatively important: 5 points; important: 7 points; and very important: 9 points (Table 2).
The results of the second round of the expert questionnaire
Third round of questioning
Twenty-eight questionnaires were sent out in the third round with a 100% response rate.10 Indicators were sorted by the average score of importance of the two rounds of survey results. The order of the indicators was basically the same in the second and third rounds of the expert questionnaire; almost every indicator’s average score variability was less than 5% (Table 3).
The results of the third round of the expert questionnaire
Calculating the weight of each evaluation indicator through the AHP
The AHP is a commonly used method to calculate the relative importance of each indicator. In this study, the average score of each indicator minus the average score of other indicators was used to construct the judgement matrix.11
(1)
According to formula 1, we got the judgement matrix B (Figure 1).
Then, according to the reciprocity of the comparison matrix, the form of matrix B was converted by the exponential function ex to construct judgement matrix A (Figure 2).
The eigenvector of judgement matrix A (Figure 3) was calculated, and then the weight of each indicator was obtained.
The judgement matrix B.
The judgement matrix A.
Calculating the eigenvector of the judgement matrix and the weight of each indicator by MatLab.
Determining the authority of experts
The experts’ authority has considerably impacted the reliability of questionnaire results. In this study, we considered the factors as ‘the experts’ own academic achievements’, ‘the main basis for the estimate’ and ‘the familiarity of evaluation’,12 to calculate the expert’s authority coefficient (Figure 4). The lowest expert’s authority coefficient was 0.83 in the case of one expert. Hence, the chosen experts had a relatively high degree of authority.
Experts’ authority coefficient.
Determining the reliability of the questionnaire
Reliability is an estimate of the consistency and stability of the results of the questionnaire. In this study, we used the Cronbach’s α method13 to detect the credibility of the questionnaire results. The larger the Cronbach’s α coefficient is, the higher is the reliability of the questionnaire results. Between 0 and 1, if the α coefficient is more than 0.8, this indicates that the reliability is very good; if the α coefficient ranges from 0.7 to 0.8, then it has considerable reliability; if the α coefficient is less than 0.7, there are indicators in the evaluation index system that need to be modified or deleted; and if the α coefficient is less than 0.35, then the credibility is low. In this study, SPSS software was used to test the reliability of the questionnaire results.
Determining the consistency of the judgement matrix
Inconsistencies are inevitable when experts compare different indicators. Therefore, it is necessary to test the consistency of the judgement matrix. Consistency ratio (CR), proposed by Satty, is generally used as the criterion for measuring the consistency of a judgement matrix. When CR is less than 0.1, it can be considered that the judgement matrix has satisfactory consistency.14
(2)
CI represents the consistency index, and RI represents random index.
(3)
λmax means the largest eigenvalue of the judgement matrix, and n means the order of the matrix.
Results
Indicators and weight value of each indicator
Following three rounds of questioning, eight evaluation indicators were agreed on (Table 4).
The average score and weight value of each evaluation indicator
Determining the reliability of the questionnaire
The Cronbach’s α coefficient of the expert questionnaire results was 0.777, indicating that the results of the study had considerable intrinsic reliability.
Determining the consistency of the judgement matrix
The value of the RI of the reciprocal matrix of 1–10 orders is given in Table 5 below.
The value of random index (RI) of the reciprocal matrix of 1–10 orders
Using MatLab to calculate the maximum eigenvalue of the matrix, as shown in (Figure 5), the λmax was found to be 8, and the order of the matrix n was also 8. CI=(8−8)/(8−1)=0, so CR=0, indicating that the study of the structure of the judgement matrix had strong consistency.
Calculating the maximum eigenvalue of a matrix using MatLab.
Discussion
The aim of this research was to use a Delphi methodology and AHP, in conjunction with a qualitative evaluation and quantitative evaluation to construct an evaluation index system for evaluating the academic impact of military medical scholars. Thirty experts from the field of military medicine, military medical information and library and information science were asked three rounds of questioning to score the feasibility and importance of the indicators. This resulted in a list of evaluation indicators comprising: ‘Web of Science documents’, ‘Citation impact’, ‘h-index’, ‘Percentage of international collaborations’, ‘Percentage of the top 10% of the cited frequency’, ‘Category normalised citation impact’, ‘Percentage of documents cited’ and ‘The number of F1000 Recommended papers’. An AHP was subsequently used to construct a judgement matrix to calculate the weight of each indicator. After analysing the authority of the experts, the questionnaire results and consistency of the matrix were found to be reliable.
No evaluation method can ever fully describe the potential academic impact of military medical publications as there will always be a degree of individual opinion and thereby bias when interpreting scientific research. However, the evaluation index system determined from this study combines the advantages of both qualitative and quantitative recognised evaluation indicators, which are subsequently weighted according to their importance in the field of military medicine. It is hoped that this framework will provide a manner in the future for comparing the potential academic impact of military medical scholars.
Footnotes
Contributors JYH provided the overall idea and methodological guidance of the article, besides, she wrote part of the article and completed the verifying and revision of the article. XL acquired and analysed all the data of the article and wrote most of the paper.
Funding This work was supported by General Logistics Department of the Chinese People’s Liberation Army. Grant Number (CWS14R075).
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Data sharing statement All data are included in the article.