Reflections on defining a standard for computable expression of scientific knowledge: What teach us Yoda can

Science advances at a slow pace. Some factors contributing to the slow pace are unavoidable, such as the need for replication of results to provide sufficient certainty in our knowledge before we build on it for further research advancements or apply it to decision-making. But many factors slowing scientific progress can be overcome.

Scientists often use words and phrases that have more than one meaning or interpretation. Even when they use unambiguous language, different scientists may use different terms for the same concepts. Even if scientists use the same terms, they may combine those terms in diverse ways that lead to different meanings and interpretations, thereby reintroducing ambiguity. Add the power of computing, and reproduction and dissemination of all these variations is multiplied exponentially. It becomes more challenging to find specific scientific knowledge, to know when all the relevant scientific knowledge is detected, and to combine the available information for aggregation, analysis, and synthesis of knowledge.

Universally accepted forms for all aspects of this communication could overcome these problems, and universally agreed-upon forms for how we transfer this communication between computers can accelerate such a solution.

A standard for computable expression of scientific knowledge can be more precisely described as a technical standard for electronic data exchange of machine-interpretable data expressing scientific knowledge.

A standard for computable expression of scientific knowledge will bring a paradigm shift to science—greatly increasing the pace, accuracy, and inclusiveness of scientific communication. Creating such a standard is changing the world. Doing it is, all at once, daunting, humbling, awe-inspiring, fulfilling, motivating, and uplifting.

I have been working to define a standard for computable expression of scientific knowledge for at least five years. In 2016, in a blog post I wrote about “Connecting with systematic review developers (and consortia of such groups) to facilitate shared systems and interoperability to help make the great effort currently expended (with much duplication and thus limits on our ability to cover the scope of medicine) something that can transform our societal ability to know what we know and know how well we know it.”¹

In 2017, in a Guidelines International Network (GIN) technology working group meeting, members representing impressive technical systems for developing and disseminating evidence and guidance for healthcare expressed a desire for a common standard for sharing data between systems. The group tasked me with taking the first step in developing a common interoperable approach. In 2018, I learned how Health Level Seven International (HL7) Fast Healthcare Interoperability Resources (HL7(R) FHIR(R)) is working to solve the once-intractable multifaceted problem of interoperability for health data exchange. To leverage the successful solutions for the health domain to meet similar needs in the science domain, I started an HL7 project to extend FHIR to Evidence-Based Medicine knowledge assets (EBMonFHIR).²

In 2019, we developed a stable model for the expression of evidence with elements for the definition of the variables and their relationship, expression of the statistics, and expression of aspects of certainty regarding the statistics for the combination of variables. This model was a critical first step in defining the standard for computable evidence, and we accomplished the model with open once-weekly meetings. At the time, my attention to this development was about 10% of my professional effort. This model for computable evidence could be used to express findings from interventional trials and observational studies, findings from single studies and syntheses of studies, and findings from original investigators and critically reinterpreted findings from others. This model for computable evidence could provide the precise data sought for various contexts and perspectives such as “evidence-based medicine (EBM),” “real-world evidence (RWE),” and machine-interpretable form to support artificial intelligence (AI) and machine learning (ML).

In 2020, coronavirus disease 2019 (COVID-19) changed everything. The urgency and importance of accelerating scientific discovery and communication compelled me to change my 25-year mission and make solving this problem 100% of my professional effort.³ My new mission is to enable standard-based machine-interpretable expression of knowledge, especially related to healthcare and scientific evidence. I started the COVID-19 Knowledge Accelerator (COKA) as an open virtual group to expand the EBMonFHIR effort⁴ and COKA now has eleven active working groups meeting twelve times weekly.⁵ I also started Computable Publishing LLC as a kind of venture philanthropy to support the software development needed to achieve this mission.

In 2021, with twelve board members from six countries, we started the Scientific Knowledge Accelerator Foundation as a 501(c)(3) nonprofit organization to support virtual scientific knowledge accelerator groups like COKA. For a technological infrastructure, Computable Publishing LLC created the Fast Evidence Interoperability Resources (FEvIR) Platform (https://fevir.net/) to demonstrate and facilitate FHIR-based data exchange for scientific knowledge.⁶

Starting so many groups, projects, and technologies in a brief time is necessary for the foundation of what is yet to come. The end of the year (2021) is a natural time to reflect on what has happened and seek to learn from it to best serve in the next year. But what is the model for reflecting on such things?

HL7 FHIR has shown success as a standard for health data. Learning what makes FHIR successful for a health data exchange standard is a tremendous accelerant for the development of an evidence data exchange standard. Although we learned enough in 2018 to choose FHIR as a basis, we still have a lot to learn given how much is involved, and we still regularly seek guidance from others more expert in the development of FHIR.

A key to the FHIR solution is the “Resource,” which is a small unit of structured data. The term “Resource” is used to both describe the data unit that is shared (an instance of a Resource) and the specification of how that data unit is expressed (a structure definition for the Resource). To achieve interoperability, standards for the small sharable value units are more efficient than standards for large documents containing the core knowledge in unstructured contents. When creating a large document (such as a guideline, a systematic review, a research report, or the article you are currently reading), one focuses on the overall structure and message and can easily miss how often the many bits of knowledge are ambiguous and prone to misinterpretation and re-interpretation.

Creating a standard for computable evidence started with defining FHIR Resources for Evidence and EvidenceVariable data units, and, for the FHIR Evidence Resource, defining structure for the evidence variable, statistic, and certainty components. Focusing on these smaller units leads to a clearer understanding of the core knowledge and minimizes the sensationalism, spin, or undue influence that can occur when conveying the scientific story.

For example, a randomized trial evaluating anticoagulation intensity in patients hospitalized with COVID-19 was reported with a conclusion of “In noncritically ill patients with COVID-19, an initial strategy of therapeutic-dose anticoagulation with heparin increased the probability of survival to hospital discharge with reduced use of cardiovascular or respiratory organ support as compared with usual-care thromboprophylaxis.”⁷ We created an Evidence Resource to express the study findings for the primary outcome.⁸ Explicit attention to statistical analyses and certainty components emphasized multiple concerns including:

(1) unequal distribution of calendar time between the groups being compared;

(2) early trial termination with stopping criteria based on statistical significance and not the magnitude of effect; and

(3) reporting limited to Bayesian analysis with no data to determine if the results are sensitive to the analytic method.

Scholars often report evidence for a treatment with a model describing the Population, Intervention, Comparator, and Outcome (the PICO model). We can consider these variables as a precise way to describe what the evidence is about.

Evidence is also an interpretation of findings from observed variables to inform expected results, but there can be a “change in translation” between the observed variable and the intended variable. In a common approach to rating the quality of evidence, the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group describes indirectness as differences between the observed variable and the variable of interest.⁹

We built into the Evidence Resource a way to express the observed evidence variable, the intended evidence variable, and the degree of matching between them. In 1998, headlines reported a study showing a nasal spray prevents new spinal fractures in women with osteoporosis, including a quote that “The study results should provide encouragement to the millions of women who have suffered from the debilitating effects of a spinal fracture.”¹⁰ If the published study results¹¹ were reported in computable form, the observed variable would be precisely defined as a new radiographic vertebral fracture without information regarding symptoms. Structured reporting of an observed variable of radiographic vertebral fracture, an intended variable of symptomatic vertebral fracture, and low-quality directness match would be less sensational but more representative of what we know.

In the trial of anticoagulation intensity,^{7, 8} we found it challenging to express the observed outcome in precise terms. We created an EvidenceVariable Resource titled “Organ support-free days” and noted both “Organ support was defined as oxygen delivered by high-flow nasal cannula, noninvasive or invasive mechanical ventilation, or the use of vasopressors or inotropes.” and “The methods for one of the individual trials stated ‘Organ Support is defined as receipt of invasive or noninvasive mechanical ventilation, high flow nasal oxygen, vasopressor therapy, or ECMO support’.”¹²

The typical clinician may interpret “organ support” for COVID-19 as mainly intubation and mechanical ventilation, and thus confuse “organ support-free days” with “survival without intubation.” Deeper inspection of the available data finds a 4% absolute difference in the primary outcome and only a 1% absolute difference in survival without intubation. Presumably, most of the apparent effect is from the 3% absolute difference in “organ support without intubation.” This difference in precise specification of the primary outcome changes our interpretation of what effect is expected. In this case, it also further questions the certainty of whether any effect is related to the assigned treatment. In this unblinded trial, the treating physician aware of the use of therapeutic-dose anticoagulation may have been less likely to prescribe “organ support without intubation”—bleeding is the most frequent complication associated with extracorporeal membrane oxygenation (ECMO) support.¹³

A common challenge reported among researchers communicating within their discipline (let alone to the public) is a lack of operational definitions and a casual use of labels.¹⁴ In the era of digital data, a major barrier to repurposing data between patient care and clinical research systems is the information systems in both domains use different data models and terminology systems.¹⁵ A standard for data exchange for science needs to include standard terminologies.

We discovered the need to create standard terminologies for multiple domains used in scientific communication, including study design, statistics, and reporting the risk of bias. To create standard terminologies (specifically code systems as the technical representation of standard terminologies) that would be sustainable and suitable to serve as a foundation for scientific communication, we created a 13-step Code System Development Protocol and an open process for global participation in expert working groups to define the terms.¹⁶

Through many discussions, we found that we all learned and applied these terms in different ways and existing definitions were sometimes incompatible. We often had to “unlearn” and develop consensus to create standard re-usable interoperable definitions for these terms.

After a year, we learned enough about the processes for terminology development that we needed to re-tool for efficiency and sustainability. In November 2021, we consolidated these efforts and revised the protocol to create a single Scientific Evidence Code System (SEVCO) which will have about six hundred terms in the first published-for-use release.¹⁷ As of December 19, 2021, we have reached approval (100% agreement) for 113 terms in the developing SEVCO terminology.¹⁸ By March 19, 2022, we reached approval of 168 out of 536 (31%) drafted terms. Beyond the obvious value of subject matter expertise and vocabulary development experience, we found it especially valuable to maintain attention to the context for application of the vocabulary terms and direct involvement by people who would use the terms but may not have subject matter expertise.

The problems with current processes and systems for communicating science include imprecision, ambiguity, and inefficiency, all of which leads to massive duplication of effort and wasted resources, tremendous delays in realizing the results of scientific advances, and mistakes and misinterpretations that can cause death and suffering on an unimaginable scale. In 1998, the Lancet published a report of twelve cases of pervasive developmental disorder in children associated with the measles, mumps, and rubella (MMR) vaccine.¹⁹ Despite a full retraction many years later,²⁰ this single report stimulated and fueled anti-vaccination movements that to this day interfere with efforts to prevent and eradicate infectious diseases.

A precise, unambiguous specification of the data reported to summarize these twelve cases would be associated with so many reasons for uncertainty in the findings that it would be difficult to provide substantial influence in a system designed to display the available evidence in forms structured to support decision making.

The problem we are overcoming is Vast And Deep and Everywhere-Reaching (VADER). Overcoming this VADER problem will take persistence, collaboration, cooperation, inclusiveness, engagement, careful judgment, and agile responsiveness. We will do it, not just for the expected reasons of improving lives, but also because once we start communicating science in a precise, unambiguous, and efficient form, we cannot return to the problems with our previous methods.

With a Foundation Of Re-usable Consistent Expressions (FORCE), we will succeed in providing a standard for computable evidence. May the FORCE be with us.