Applying evidence in the real world: a case study in library and information practice

Introduction

Previously, we reported the development and validation of a search filter to find information relevant to palliative care in the general medical literature using the OVID medline database. Using a method where performance of a search strategy is compared with a gold standard reference set of hand-searched journal articles, we validated the performance of a particular strategy, called the Master Search. This search consisted of 12 search terms [9 Medical Subject Headings (MeSH) terms and three textwords] combined with the Boolean OR operator (column 1, Table 1).¹ In our judgement, this search represented the best compromise between sensitivity and precision. Given that implementation of evidence is as integral to evidence-based practice as research evidence is,² we now consider issues related to making this evidence available for use ‘in the real world’. In particular, we identify three issues relating to the generalizability of our findings and consequent implications for the utility of the search, report the results of some bibliometric analyses and detail progress in translating this evidence into practice.

Efficacy versus effectiveness

Generalizability of findings is an important issue for all health research. One criticism of randomized controlled trials investigating the efficacy of interventions, for example, is that the initial selection of study participants is usually such that certain people are excluded.³ While this may be necessary to ensure the internal validity of a trial, it also limits external validity; that is, the generalizability of results. This means that a trial's results may not be immediately generalizable to many of the patients actually seen in clinical practice.^4,5 Conversely, a criticism of randomized controlled trials investigating the effectiveness of interventions, is that some rigour may be sacrificed in order to achieve improved generalizability.

Similarly, methodological choices underpinned the validation of the Master Search palliative care filter. Ultimately, these decisions support the internal validity of the findings, but they also have implications for generalizability.

Objectives

The objective of the current study was to identify and explore the possible implications of several key decisions with a view to improving the relevance and generalizability of the search filter. Three such issues were identified and investigated. These are summarized in Table 2.

First, the three textwords in the Master Search were originally restricted to the search field of .tw. (textword), rather than using the .mp. (title, original title, abstract, name of substance word, subject heading) default in OVID for non-MeSH term searching. The basis for this decision was that .mp. would search for textwords in several fields, including MeSH terms, but this would obscure some of the very detail we sought, to assess for example, the performance of individual MeSH terms in retrieving relevant citations. In the real world, however, if searchers do utilize the .tw. (textword) restriction or, as is more likely, the default .mp. delimiter, they may not be exploiting OVID's software functionality to maximize citation retrieval. In particular, using .af. (all fields) would enable a searcher to find textwords occurring in any field, possibly maximizing sensitivity further. Given the textwords used in the filter (palliat$, hospice$ or terminal care) are almost certainly uniquely palliative (compared with, for example, the textword ‘hand’ which can have multiple uses and meanings), the .af. option may be particularly yielding.

Second, the Master Search was originally developed to find relevant palliative care content in the general medical literature (Annals of Internal Medicine, British Medical Journal, Journal of the American Medical Association, The Lancet). Reasonably, searchers will also want citations indexed in the specialized palliative care journals, such as the Journal of Palliative Care, Journal of Palliative Medicine and BMC Palliative Care. While we might expect the Master Search would be most effective for these citations, it is in fact unknown.

Third, while OVID medline was chosen as the medline interface, many searchers use PubMed. Not only is there a difference in journal coverage (for example, BMC Palliative Care is indexed on PubMed and not OVID medline), but they both use different software. To maximize use of a validated search, it should be available for use in both interfaces. However, there needs to be some validation process to ensure the equivalency of the Master Search.

We sought to explore these three issues further using bibliometric analyses to:

Methods

Results

Discussion

This case study illustrates the need for caution in relation to implementing research evidence. Our analysis of three key methodological decisions, made in the creation of a specialist search filter, confirmed the proposition underlying this study. Namely, in the same way that evidence needs to be appraised for its generalizability before applying it in clinical practice, so must it be in library and information practice. The original filter was, we believe, an example of good-quality evidence. However, the parameters used in its generation had implications for its generalizability. In this case, we identified three issues with a potential to influence implementing this evidence in the real world practice of searching for palliative care literature. The original choices of search delimiters, journal titles and database interfaces all impacted on the generalizability of the evidence.

It has been demonstrated that sensitivity could be improved marginally by using the delimiter of .af., without compromising precision. Indeed, a slight improvement in precision was achieved. However, the overall sensitivity was still unacceptably low, highlighting our original finding that further effort is needed to identify MeSH terms, textwords and phrases that can contribute to improving the search strategy. Importantly, the investigation of the filter's performance in the specialist palliative journals (where all content might be seen as being relevant), revealed that MeSH indexing terms do not necessarily capture the full content. While the use of .af. goes some way toward overcoming the deficit, it relies on one of the core textwords appearing in the journal title or institutional field. However, this is still suboptimal because some journals, such as the Journal of Pain and Symptom Management, publish a large number of highly relevant palliative care papers, yet none of the three textwords is contained in the journal titles. Understanding why some content is not indexed as palliative and what combination of terms is implicitly palliative still remains fundamental to improving the efficacy of the filter. In the interim, specifying search terms for a palliative-specific list of journals in addition to the Master Search using .af. may ensure the most comprehensive retrieval of palliative care literature.

Therefore, on the basis of research reported so far, and in consideration of generalizability, we offer two key suggestions for those requiring exhaustive searches for palliative care literature in medline, such as those undertaking systematic reviews. First, the delimiter ‘all fields’ (.af. for OVID medline and [all fields] for PubMed) should be used for uniquely palliative textwords to maximize sensitivity from both the general medical and specialist palliative journals. This strategy is probably relevant for other health-related fields which use specialized terminology with restricted meanings. The word ‘sleep’ would be an example of a likely highly sensitive term relevant for the sleep medicine field. We note that, while the development and use of specialized subject search filters is widespread, such as those published as part of the Cochrane Collaborative Review Groups, the published formal evaluation (i.e. objective validation) of these subject search filters’ performances in medline is somewhat of a scarcity. Indeed, the majority of published validated filters focus on the identification of studies that use particular methods, such as those filters referenced in an impressive website dedicated to the identification of methodological search filter resources.⁷ While validated subject filters are reported in the formal literature, such as for sleep,⁸ paediatrics⁹ and the current palliative care example,¹ these are far less frequent.

Second, one way of enhancing the sensitivity of a search for all palliative literature, would be to combine the Master Search (i.e. the most sensitive strategy for relevant literature in the general medical journal literature) with a strategy designed to find all the content published in a core group of specialist palliative journals. The syntax would vary depending on whether PubMed or OVID medline was used, but conceptually, such a strategy would be represented by the search: Master Search (using .af.) OR specialist palliative journals search. Such a combination, using different journal titles may be necessary because simply including .af. as a delimiter will not retrieve citations that do not use the given textwords; yet we speculate that most of what is published in the specialist journal literature is likely to be relevant.

Implementation

We have applied the findings of this research in creating resources to support palliative care clinicians and educators in accessing literature and evidence. Thereby, we have not only translated this research into practice, but are also supporting the process of translating other palliative research into practice. Subject-based search strategies using PubMed had earlier been developed and hosted on the CareSearch website (http://www.caresearch.com.au).⁶ These are presented as hyperlinks, so users can select a topic, which then runs a real-time search of PubMed. Topics include, for example, dyspnoea, vomiting and anorexia. In order to address the likely differing needs of searchers, these searches embed different search limits; for example, restricting results to randomized controlled trials or to free full-text articles. These searches are simple and not validated, however, offer an immediate entry point into PubMed in relation to topics commonly of interest in palliative care.

Subsequent to our research, these subject-based searches were modified to include the palliative care filter. In this way, we have implemented the best evidence that was currently available in relation to searching for these topics. We also provided the PubMed version of the filter online. In light of the results of bibliometric analyses presented here demonstrating that use of the .af. delimiter outperforms .tw, all online searches using PubMed have now been modified to use the equivalent PubMed [all fields] delimiter for the selected textwords. It is noted that, despite these efforts, the identification of further relevant MeSH terms and textwords to improve the underlying Master Search remains an outstanding research direction.

In conclusion, we argue that it is as important in this field as it is in clinical research, that evidence does not simply accumulate in journal articles. Neither should it be randomly or inappropriately applied. Issues of generalizability are therefore critical to evidence-based library and information practice. Evidence must be made accessible, which we have facilitated through development of a range of different searches for users. However, the process for doing this should be evidence based in its own right, which is reflected in our incorporation of the validated palliative care filter into those searches and subsequent modifications in the light of the current paper.

Key Messages

Implications for Policy

Implications for Practice