1 INTRODUCTION

In recent years, publications on medical research by clinical investigators in China have been steadily increasing, amounting to 14.63% of the global total in 2019 [1]. Nonetheless, China's medical research has had a limited impact on clinical decision-making [2, 3], with a notable lack of high-quality clinical studies [4, 5]. A fundamental issue lies in dynamic appraisal of the clinical relevance of medical research that cements the measure of a hospital's research competence. China has developed several evaluative methods concerning hospital research capability, such as Fudan University's China hospital ranking, the Science and Technology Evaluation Metrics (STEM) ranking of China's hospitals released by the Chinese Academy of Medical Sciences, and Innovation and Transformation ranking of China's hospitals, among others [6]. Generally, these measures quantify hospital research capacities based on various factors encompassing academic reputation, quantity of patents or papers, talent pool, academic appointments, and available research resources [7]. Although composite indices are valuable in terms of providing a comprehensive estimation of a hospital's research competence and prospective capabilities, they inadequately tackle the essential question of how much clinical value medical research actually generates.

As an applied science rooted in natural sciences, the aim of medical research should ideally be to discover new patterns of disease and to establish novel diagnostic and therapeutic techniques, all while improving patients' quality of life; these are key indicators of the caliber and usefulness of such research [8]. The inclusion of medical research outcomes as evidence in clinical guidelines released by authoritative associations and reflect its academic and application value because these guidelines summarizing the latest advances in disease diagnosis and therapy are pivotal references for clinicians in daily practice [9, 10]. Hence, we postulated that valuable medical research would be directly or indirectly incorporated into clinical guidelines [11]. We developed an accessible and objective evaluation metric, termed the “Clinical Influence and Timeliness Evaluation” or (CITE), to dynamically appraise the clinical importance of a medical research study and the clinical influence of the hospital from which it originated. Using chronic obstructive pulmonary disease (COPD) as an example, we applied the CITE metric to evaluate the clinical relevance of medical research carried out by hospitals in China and verify the applicability of this metric.

2 DATA COLLECTION AND ANALYTICAL METHODS

2.1 Medical study data set

In the CITE analysis, we investigated medical studies published from 2000 to 2020 that were directly or indirectly referenced in Chinese and international clinical guidelines for COPD. The Chinese guidelines comprised 20 clinical guidelines issued before August 2021 by China's academic organizations such as the Chinese Medical Association and Chinese Physician Association. The international guidelines comprised 59 UpToDate topics updated through August 2021. The 878 articles cited in the Chinese guidelines and 2888 articles cited in the UpToDate topics constituted primary studies, and 74,870 articles cited in primary studies constituted the secondary studies (Figure 1a,b). The medical studies that corresponded to Chinese institutions accounted for 8.8% and 1.9% of Chinese and international guideline sources (Figure 1c). The Chinese medical studies investigated in this research originated from the Chinese mainland and did not include those from Chinese Hong Kong, Macao, or Taiwan.

2.1.1 Calculating progressive clinical importance of medical studies

The CITE metric provides a progressive evaluation of the clinical importance of a medical study (CIMS) using two dimensions: the weight of the research (W) and its timeliness (T). The CIMS for a given year X of assessment, designated the $${\mathrm{CIMS}}_{x}$$, is determined using the following formula:

$$${\mathrm{CIMS}}_{x}={W}_{\mathrm{CS}}\times {T}_{x},$$$()

$${W}_{\mathrm{CS}}$$ denotes the weight metric for a clinical study, which is calculated as

$$${W}_{\mathrm{CS}}={{n}_{1}w}_{1}+{{n}_{2}w}_{2},$$$()

where $${n}_{1}$$ and $${n}_{2}$$ are the number of citations as a primary and secondary clinical study in the corresponding data set, respectively. $${w}_{1}$$ and $${w}_{2}$$ are the assigned weight values for these two situations, $${w}_{1}$$ is assigned a value of 1 whereas $${w}_{2}$$ is determined by the number of references as primary studies; as the number increases, the weight value assigned to each individual secondary study decreases. In this case, $${w}_{2}$$ is given a value of 0.03 owing to an average of 32 secondary studies cited in a primary study in this analysis.

$${T}_{x}$$ denotes the timeliness of a clinical study in year X, which is calculated as

$$${{T}_{x}=y}_{p}-{y}_{x}+{t}_{W},$$$()

where $${y}_{p}$$ and $${y}_{x}\,$$ are the numerical values of the publication year and the evaluation year, respectively. $${t}_{W}$$ signifies the “effective period” set for medical research in the CITE analysis, encapsulating the intuitive concept that the practical implications of a medical study resolve steadily over a specific duration. As the publication year gets closer to the evaluation year, the timeliness $${T}_{x}$$ of the study increases, and vice versa. If $${T}_{x}\,$$ holds has a negative value, this implies a disproportionately extended gap between the years of publication and evaluation, suggesting that the research has outlived its “effective period,” in which case the value of $${T}_{x}$$ is assigned 0. A larger $${t}_{W}$$ value, corresponding to an extended “effective period,” leads to a slower annual degradation of clinical value. In contrast, a smaller $${t}_{W}$$ value accelerates the exhaustion of clinical value and makes the evaluation results more susceptible to annual hospital research volume. From another perspective, this might induce substantial annual oscillations in the clinical influence of hospitals. We evaluated the distribution of hospital research values under varying $${t}_{W}$$ values, and set $${t}_{W}$$ at 10 to address extreme annual oscillations in the evaluation results while preserving the differentiation among hospitals (Table S1).

2.2 Calculating the clinical influence of hospitals

The contribution value of authors' affiliated institution in a certain assessment year X ($${\mathrm{CVAA}}_{x}$$) can be derived from the cumulative clinical importance of medical studies in which the hospital is engaged either as the corresponding author's affiliation or as a coauthoring entity. This is mathematically expressed as

$$${\mathrm{CVAA}}_{x}={w}_{\mathrm{cor}}\sum {\mathrm{CIMS}}_{x}^{\mathrm{cor}}+{w}_{\mathrm{coa}}\sum {\mathrm{CIMS}}_{x}^{\mathrm{coa}},$$$()

where $${w}_{\mathrm{cor}}$$ denotes the assigned weight value for the corresponding author's affiliation, set at 1. $${w}_{\mathrm{coa}}$$ represents the assigned weight for the coauthoring unit. In this study, $${w}_{\mathrm{coa}}\,$$ was determined to be 0.17 because there is a median of six coauthoring entities apart from the corresponding author affiliation in medical studies involving multiple authoring entities. Σ($${\mathrm{CIMS}}_{x}$$)^cor and Σ($${\mathrm{CIMS}}_{x}$$)^coa represent the sum of $${\mathrm{CIMS}}_{x}$$ for hospitals serving as the corresponding author's affiliation and a coauthoring entity, respectively (Figure 2).

The clinical influence of the author's affiliation in a certain assessment year X ($${\mathrm{CIAA}}_{x}$$) refers to the proportion of that hospital's $${\mathrm{CIMS}}_{x}$$ in relation to the cumulative $${\mathrm{CIMS}}_{x}$$ across all hospitals in China. $${\mathrm{CIAA}}_{x}$$ comprises two integral components, namely, $${\mathrm{CIAA}}_{x}^{U}$$ and $${\mathrm{CIAA}}_{x}^{C}$$, each accounting for 50%, as shown in the following formula:

$$$\begin{array}{ccc}{\mathrm{CIAA}}_{x} & = & 50 \% {(\mathrm{CIAA}}_{x}^{U}+{\mathrm{CIAA}}_{x}^{C})\\ & = & 50 \% \left(\frac{{\mathrm{CVAA}}_{x}^{U}}{\sum {\mathrm{CVAA}}_{x}^{U}}\times 100+\frac{{\mathrm{CVAA}}_{x}^{C}}{\sum {\mathrm{CVAA}}_{x}^{C}}\times 100\right)\end{array}$$$()

where $${\mathrm{CVAA}}_{x}^{U}$$ and $${\mathrm{CIAA}}_{x}^{U}$$ denote the values derived from the reference data set of UpToDate topics; $${\mathrm{CVAA}}_{x}^{C}$$ and $${\mathrm{CIAA}}_{x}^{C}$$ are derived from Chinese clinical guidelines.

3 CLINICAL INFLUENCE ASSESSMENT RANKING OF HOSPITALS IN CHINA

A total of 499 hospitals in China were included in the clinical influence assessment ranking of COPD, comprising 350 hospitals in the Chinese guideline data set and 334 hospitals in the UpToDate topics data set. The sum of CIAA scores for the 20 leading hospitals from 2000 to 2020 accounted for 51.0%, 54.5%, and 50.4% of the total CIAA score in the Chinese guideline data set, the UpToDate topics data set, and the merged data set, respectively. The ranking results for the 10 leading hospitals in each data set are shown in Tables 1–3, and that of the 100 leading hospitals are shown in Table S2.

4 DISCUSSION AND FUTURE DIRECTIONS

Clinical and laboratory studies advance medical technology and treatment and eventually improve patient care and outcomes. In recent years, the Chinese government has been actively promoting the construction of research hospitals, hoping to improve population health and the national health system by increasing investment in biomedical research [12]. In line with this, it is crucial to gauge the clinical relevance of medical research.

Clinical guidelines, which assist clinicians in delivering optimal patient care, are an ideal reference for assessing the practical value of medical research [13, 14] and have been adopted by national health commissions as essential tools for improving the quality of medical care [15]. In a prior investigation, we conducted a quantitative examination of the frequency with which Chinese clinical research was referenced in clinical guidelines. Interestingly, we discovered that the citation rate of Chinese research in clinical guidelines lagged significantly behind the burgeoning growth in the number of Chinese medical papers in various international medical articles [16]. This staggering discrepancy underscores the pressing need to clarify the clinical relevance of Chinese medical research. In a bid to systemically and progressively appraise the clinical value of medical research emerging from Chinese hospitals, we developed the CITE evaluation method, which hinges primarily on clinical guidelines as reference.

Compared with existing domestic metrics evaluating the research competence and academic impact of hospitals using composite indicators [7], the CITE metric focuses on evaluating the contribution of studies in improving clinical practice and patient care. The practicality of the CITE metric is underscored from several aspects. First, CITE zeroes in on the clinical relevance of medical research, thereby dramatically streamlining the assessment indicators compared with other methods. This sharply curtails the risk of overwhelming the analysis process with extraneous assessment indicators and compromising its initial purpose [17]. This approach strategically sets the assessment scope for medical studies as a dual-tiered citation centering on clinical guidelines. The exponential surge in medical study assessments ensures the inclusion of numerous hospitals while still giving attention to clinical relevance. Furthermore, the CITE metric is exhaustive in nature. In current medical research, collaboration across various medical centers is increasingly becoming the norm. Regrettably, existing assessment systems offer little recognition to the research input of collaborating healthcare institutions, often reserving this for the principal investigator's institution. Using various evaluation components, our metric captures clinical research outcomes that are directly or indirectly incorporated into guidelines by hospitals that are either leading or participating in the research. Moreover, CITE offers insight into the evolution of clinical value over time. It is well established that medical research, a process that often spans years, generates clinical value that also endures for several years. With this in mind, we instituted the “timeliness” concept, effectively portraying the longevity of clinical value over its “validity period.” This more scientifically reflects the sustained clinical value of medical research and prevents extreme fluctuations in the evaluation results.

This study has several limitations. One prominent issue lies in the lack of reference data and insufficient processing capacity of manual data to determine reasonable values within this field. For example, the factor $${w}_{2}$$, assigned to secondary studies, is calculated as the ratio of the number of primary to secondary studies; and the factor $${w}_{\mathrm{coa}}$$, representing the degree of contribution from participating hospitals, is defined as the reciprocal of the median number of hospitals participating in studies in addition to the corresponding hospital. It is important to concede that the validity of these values may require future exploration to ensure their appropriateness. Similarly, it is more reasonable to assign specific values to parameters such as $${w}_{2}\,$$ and $${w}_{\mathrm{coa}}$$ for each study rather than uniform values, which can be achieved in the future through automated analysis using software algorithms. Furthermore, introduction of the value $${t}_{W}$$ = 10 in this study was a calculated move aimed at creating a balance between sensitivity to the variations in research output and mitigating unreasonable annual volatility. However, we acknowledge that this approach may fall short in terms of accurately illustrating the lifespan of the clinical impact generated by all varieties of medical research. Therefore, this factor too invites further scrutiny and assessment so as to refine the evaluation process. Additionally, in this analysis, we did not examine the publication of studies directly; instead, we examined studies referenced in guidelines. This can lead to delays in the results (as shown in Figure 1b) and could also change past-year evaluation rankings depending on the prevailing citation in the guidelines. Therefore, a single year's hospital rankings may not be unequivocally credible, warranting a review over several years.

In our study, the practical applicability of the CITE metric was successfully established within the domain of COPD. In the future, the extension of this approach to encapsulate a wider range of disease fields holds promise. This could facilitate a thorough and dynamic analysis of the clinical value and transformation of medical investigations conducted across various specialties and hospitals, which could prove invaluable by serving as a robust reference for gauging the input–output relationship within medical research. Ultimately, this could stimulate a more problem-centric approach to medical investigations, thereby encouraging the pursuit of research endeavors that are more aligned with solving real-world clinical issues.

AUTHOR CONTRIBUTIONS

Gao Jianchao: Conceptualization (lead); data curation (lead); formal analysis (lead); funding acquisition (lead); investigation (equal); methodology (equal); project administration (lead); resources (lead); software (lead); supervision (equal); validation (equal); visualization (equal); writing—original draft (equal); writing—review and editing (equal). Chen Xiaoyuan: Conceptualization (equal); data curation (supporting); formal analysis (supporting); investigation (equal); methodology (equal); project administration (supporting); writing—original draft (equal); writing—review and editing (equal). Gao Chenyan: Conceptualization (equal); data curation (supporting); formal analysis (supporting); investigation (equal); methodology (equal); project administration (equal); supervision (equal); validation (equal); visualization (equal); writing—original draft (equal); writing—review and editing (equal).

CONFLICT OF INTEREST STATEMENT

The authors declare no conflict of interest.

ETHICS STATEMENT

Not applicable.

INFORMED CONSENT

Not applicable.