Meta-assessment and responsible use of scientometric indicators: Theoretical framework and empowerment path of scientific and technological innovation

Authors

Jintao BAO, Office of Scientific Research, Peking University, Beijing 100871, ChinaFollow
Yi BU, Department of Information Management, Peking University, Beijing 100871, ChinaFollow

Keywords

meta-assessment; scientometric indicator; scientific and technological innovation; research evaluation

Document Type

Policy & Management Research

Abstract

Against the backdrop of China’s effort to deepen the reform of the scientific research system and optimize the ecological environment for scientific and technological innovation, constructing a science and technology assessment system oriented towards innovation value, capability, and contribution has become a critical component. Addressing the core challenge of establishing a new paradigm following the current “Breaking the Five-Onlys” reform, this study introduces meta-assessment as the core theoretical framework, aiming to systematically examine the validity, reliability, and fairness of scientometric indicators, thereby breaking the “indicator-behavior” mutual construction cycle and preventing the risk of new indicator alienation. To this end, the current study systematically analyzes the inherent challenges faced by scientometric indicators in terms of conceptual measurement, result stability, and unbiasedness, and reviews cutting-edge explorations in international principles for responsible assessment. Building on this foundation and rooted in the Chinese context, this study proposes a systematic pathway for constructing a responsible indicator ecosystem: through the localized adaptation of indicator design, the fine-grained deconstruction of assessment scenarios, and the synergistic empowerment of institutional and technological measures, it aims to promote a paradigm shift in science and technology assessment from “metric management” to “value governance” ultimately serving the construction of the national new scientific research system and the systematic enhancement of scientific and technological innovation capabilities.

First page

1813

Last Page

1820

Language

Chinese

Publisher

Bulletin of Chinese Academy of Sciences

References

1. 中国科学技术信息研究所. 2024年中国科技论文统计报告. 北京: 中国科学技术信息研究所, 2024. Institute of Scientific and Technical Information of China, China Scientific and Technological Papers Statistical Report 2024. Beijing: Institute of Scientific and Technical Information of China, 2024. (in Chinese)

2. 徐芳, 李晓轩. 科技评价改革十年评述. 中国科学院院刊, 2022, 37(5): 603-612. XU F, LI X. Review on reform of research evaluation in past decade. Bulletin of Chinese Academy of Sciences, 2022, 37(5): 603-612. (in Chinese)

3. Waltman L, Traag V A. Use of the journal impact factor for assessing individual articles: Statistically flawed or not? F1000 Research, 2021, 9: 366.

4. Waltman L, Van Eck N J. A systematic empirical comparison of different approaches for normalizing citation impact indicators. Journal of Informetrics, 2013, 7(4): 833-849.

5. Moed H F. Citation Analysis in Research Evaluation. Dordrecht: Springer Dordrecht, 2005.

6. Purkayastha A, Palmaro E, Falk-Krzesinski H J, et al. Comparison of two article-level, field-independent citation metrics: Field-Weighted Citation Impact (FWCI) and Relative Citation Ratio (RCR). Journal of Informetrics, 2019, 13(2): 635-642.

7. Bu Y. Understanding the Citation-Based Impact of Scientific Publications Through Ego-Centered Citation Networks. Bloomington: Indiana University, 2020.

8. Bar-Ilan J. Web of Science with the Conference Proceedings Citation Indexes: The case of computer science. Scientometrics, 2010, 83(3): 809-824.

9. Chen H, Bornmann L, Bu Y. Dynamic disruption index across citation and cited references windows: Recommendations for thresholds in research evaluation. arXiv, 2025, doi: 10.48550/arXiv.2504.07828.

10. 赵琪. 非洲学术期刊面临不公平竞争环境. 中国社会科学报, 2022-07-10(A10). Zhao Q. African academic journals face an unfair competitive environment. Chinese Social Sciences Today, 2022-07-10(A10). (in Chinese)

11. Larivière V, Ni C, Gingras Y, et al. Bibliometrics: Global gender disparities in science. Nature, 2013, 504: 211-213.

12. Bladek M. DORA: San Francisco declaration on research assessment (May 2013). College & Research Libraries News, 2014, 75(4): 191-196.

13. Diana H, Paul W, Ludo W, alet, The Leiden Manifesto for research metrics. Nature, 2015, 520(7548): 9-11.

14. 徐芳, 晋新新, 吕雯洁, 等. 国际科研评价改革实践与启示——以DORA为代表的探索. 华南理工大学学报 (社会科学版), 2022, 24(03): 94-100. Xu F, Jin X, Lv W, et al. International science and technology evaluation reform’s practice and enlightenment—A exploration represented by DORA. Journal of South China University of Technology (Social Science Edition), 2022, 24(3): 94-100. (in Chinese)

15. Huang Y, Liu W, Fu H, et al. Toward responsible scientometrics: Normative data practices for research evaluation// 20th International Conference on Scientometrics and Informetrics (ISSI 2025). Yerevan: International Society for Scientometrics and Informetrics (ISSI), 2025: 47-65.

16. Bornmann L. Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics. Journal of Informetrics, 2014, 8(4): 895-903.

Recommended Citation

BAO, Jintao and BU, Yi (2024) "Meta-assessment and responsible use of scientometric indicators: Theoretical framework and empowerment path of scientific and technological innovation," Bulletin of Chinese Academy of Sciences (Chinese Version): Vol. 40 : Iss. 10 , Article 15.
DOI: https://doi.org/10.3724/j.issn.1000-3045.20250829002
Available at: https://bulletinofcas.researchcommons.org/journal/vol40/iss10/15