Flawed citation practices facilitate the unsubstantiated perception of a global trend toward increased jellyfish blooms

Introduction

The identification of patterns in nature usually occurs through the accumulation of frequent consistent observations, typically involving a synthesis of a large number of scattered scientific reports addressing specific questions – a ‘development-by-accumulation’ of accepted facts and theories (Kuhn, 1962). Generally, the consistency among primary observations will be captured in narrative reviews and eventually in formal meta-analyses. This process of accumulation leads to the establishment of patterns and, once formally tested by the scientific method, theory. The path from primary observations to theory is provided by a network of citations that guide the reader from the final synthesis to the source of individual observations, thereby ensuring that the process is reproducible. This entire process, therefore, critically depends on the quality of the primary observations and on the accuracy of citation practices carrying knowledge across networks of papers (Todd et al., 2010). As Dupps (2008) notes citations are ‘a form of academic succession, a lineage of ideas and proofs, into which we place our own work’.

Citations, however, are subject to errors of bias and inaccuracy (Dupps, 2008). Mis-citation and publication bias are artefacts of paradigm development across fields. The steep growth of the scientific literature, at 7.7% per year for ecological sciences (Andersen et al., 2008) with over 2.2 million papers added to the Web of Knowledge in 2012 alone, is now producing a Babelian tower of information challenging the capacity of individual scientists to maintain a solid grasp on the literature that underpinned rigorous citation practices in the past. As a result, there is growing concern and evidence that citation practices may be prone to considerable errors of accuracy and bias (Harzing, 2002; Todd et al., 2007, 2010).

Following a qualitative study pointing at gross misquotations (Harzing, 1995), Harzing (2002) conducted a pioneering quantitative analysis of citation networks to show that poor citation practices had created the false perception that repatriation rates of foreign workers were high. More recently, Todd et al. (2007, 2010) observed high and consistent rates of mis-citation in the general ecology and marine biology literature with only 76.1% and 75.8% of citations, respectively, clearly supporting the assertions made in each discipline. Errors of accuracy in citation are particularly concerning because they may have considerable influence on the development of perceptions within a discipline if they are persistent and biased in a particular direction (Harzing, 2002).

A particular case of a perception lacking robust scientific evidence is that of the perceived global increase in jellyfish blooms (Condon et al., 2012). This perception percolated from scientific literature into media reports (Condon et al., 2012) and policy statements (e.g. Turley & Boot, 2010) prior to the two meta-analyses that finally tested the hypotheses (Brotz et al., 2012; Condon et al., 2013). These meta-analyses provided evidence of increasing jellyfish blooms in 28 of 45 (i.e. 62%) large marine ecosystems investigated (Brotz et al., 2012) and 30% of long-term records of jellyfish abundance (Condon et al., 2013), with the latter also indicating that global jellyfish populations undergo multi-decadal cycles. Hence, whereas the hypothesis that jellyfish may be rising globally cannot be rejected, it cannot be fully supported with the data available and the global trend of jellyfish blooms remains inconclusive. Here we examine how the perception that jellyfish blooms are rising developed in absence of quantitative meta-analysis and solid evidence.

Given that mis-citation is widespread in ecology (Todd et al., 2007) we ask here whether chains of citations may have contributed to transforming suggestions that jellyfish blooms ‘appear to be rising in some areas’ into assertive statements on a ‘global rise in jellyfish’. We analysed a network of citations on trends in jellyfish populations to assess the hypothesis that mis-citation practices contributed to the perception that jellyfish blooms were increasing globally in the absence of rigorous analysis supporting the assertion.

Methods

Citation network and its topological analysis

We used the classification of trend statements, citation threads and citation assessment to build a citation network (Fig. 1). Each node represents a paper with a trend statement about jellyfish populations classified according to spatial category and degree of affirmation. Links from node i to node j indicates that paper i was cited by paper j, thereby representing the information flow in the network, which is directional as, of course, citing papers are published later than those they cite. The colour of nodes indicates the affirmation degree and its shape represents the spatial category of the statement (Fig. 1; Appendix S1 in the Supporting Information contains a colour-blind friendly network).

The citation network (Fig. 1) allows the evolution of the perception of rising jellyfish to be traced, identifying key papers involved in establishing this notion. Specific properties of the network were analysed to quantify the relevance of each paper (or node) for the optimized flow of citations through the network (Freeman, 1977; Albert & Barabási, 2002): (1) in-degree and out-degree of nodes, which are the number of incoming arrows and outgoing arrows, respectively, and represent how often paper i cites other papers (citing frequency) and how often it is cited by other papers (cited frequency); and (2) betweenness centrality, b_i, that estimates the fraction of all shortest paths connecting any pair of nodes that pass through node i (Freeman, 1977). Specific properties of the nodes (or papers) of the network are available in Sanz-Martín et al., 2016. The library Igraph, version 0.7.1, within the statistical software R version 2.13 was used to build and analyse the citation network (Csardi & Nepusz, 2006).

Results

Our search identified 225 papers on jellyfish ecology published between 1987 and April 2012, of which 159 (70.7%) were involved, by citing or being cited, in contributing to the perception of a global rise in jellyfish. A total of 51.6% of papers in the network were ‘citing only’ papers, 27% were ‘cited only’ papers, and 15.7% of papers were both ‘citing & cited’ papers. Three point eight per cent of papers claimed jellyfish trend statements, but they were classified as ‘neither citing nor cited’ papers because they did not support their claims with citations and no other papers cite them and 1.9% of papers were cited papers that could not be verified. The ‘citing only’ papers and the ‘citing & cited’ papers contained at least one statement that cited between one and eight papers in support of a statement on jellyfish trends. Seventy-seven per cent of the ‘cited only’ papers provided statements on jellyfish trends, whereas 23% did not contain any statement on jellyfish trends despite being cited to this end.

From the 115 ‘citing only’ and ‘citing & cited’ (hereafter citing papers) that have contributed to the perception of increasing jellyfish population, 70 papers (60.9%) stated that jellyfish populations are or have been increasing, of which 27.2% referred to increases at the global or generic scale and 34.3% at the scale of multiple regions (Table 1). One in every four citing papers (26.1%) provided a statement indicating that populations may be increasing, of which 11.4% referred to global or generic increases, 14% to increases in multiple regions and 0.9% to possible regional increases (the regional study was included because it was a ‘citing & cited’ paper) (Table 1). A total of 10.4% of the citing papers stated that trends were equivocal and 2.6% could not be verified (Table 1). Whereas no papers would have had a basis to state that jellyfish are either increasing or decreasing globally, since the first global analyses were not available until 2012 (i.e. Brotz et al., 2012), 27% of the papers contained such a statement and all argued for a global rise in jellyfish (Table 1). Similar inferences, however, would have been legitimate for the multi-regional trend statements (34%; Table 1), which often appropriately cited several region-specific papers to support their statement. In our network, 16% of the total evaluated papers (25 out of 159) were ‘cited only’ studies. As has been shown, most statements focused on increases and neglected the evidence for equivocal and variable trends (12 papers cite 29 other studies in reference to equivocal trends and only 1.3% were inappropriate; Tables 1 & 3), despite monotonous jellyfish declines being almost as represented as reports of increases (cf. Condon et al., 2013).

The cumulative number of citing papers classified as ‘are increasing’ and ‘may be increasing’ displayed a rapid increase, at a rate of 64.1% and 23.8% per year, and papers classified as ‘equivocal’ at a slower rate of 9.7%, from the first papers that discussed possible trends and cited any source (Fig. 2). The first cited paper we found in support of statements on trends in jellyfish populations was Legovic (1987), but it was not until the late 1990s that cited papers started to increase in the literature. The first highly cited paper within the network (which received 18 citations in support of jellyfish population trends) was Brodeur et al. (1999), which concluded that in the Bering Sea, ‘the biomass of large jellyfish has increased dramatically in the 1990s compared with the previous decade’. This paper attracted considerable attention (Fig. 3, Table 2) and could be considered to have seeded the question of whether jellyfish may be rising in other areas. This question was subsequently addressed by Mills (2001), in what was by far the most influential paper in this citation network (Figs 1 & 3, Table 2). Mills (2001) triggered a large flux of research on trends in jellyfish populations with other prominent papers in the network of citations including reviews by Brodeur et al. (2002), Purcell (2005), Purcell et al. (2007) and Richardson et al. (2009) (Figs 1 & 3, Table 2). Despite Mills (2001) being a balanced paper, including statements on jellyfish increases (e.g. ‘Some blooms appear to be long-term increases in native jellyfish populations’, Mills, 2001) and decreases (e.g. ‘Lest one conclude that the next millennium will feature only increases in jellyfish numbers worldwide, examples are also given in which populations are decreasing in heavily impacted coastal areas’, Mills, 2001), she may have been inadvertently responsible for seeding a chain of inappropriate citations since the title posed the question ‘Jellyfish blooms: are populations increasing globally in response to changing ocean conditions?’, which left it up to the readers to draw their own conclusions and provided grounds for selective and inappropriate citations.

The dominance of Mills (2001) in the network was evident in that it was the most cited paper (54 citations) but also had the highest mis-cited score (= 66), as 85% of the statements supported by reference to Mills (2001) were inappropriate (Fig. 3, Table 2). Specifically, Mills was selectively cited 50% of the time, ambiguously cited 9% of the time, unsupported 10% of the time and supported only 15% of the time. Hence, Mills (2001) was particularly influential in conforming views about jellyfish trends before rigorous, quantitative analyses of the evidence were attempted. The case of Mills (2001) also reflects the difficulties authors had in correctly assigning conclusions to papers that provide ambiguous conclusions.

Five of the eleven most-cited papers in our network were reviews. Of the remaining six papers, which reported primary observations, two papers reported definite regional and multi-regional increases (Brodeur et al., 1999; Lynam et al., 2006), two papers reported possible increases (Brodeur et al., 2002; Link & Ford, 2006) and one paper reported equivocal trends (Graham, 2001) (Fig. 3, Table 2). The top three and the top 11 most-cited papers accumulated a cited frequency of 37% and 59% of all the citations, respectively (Table 2).

Three of the 11 most cited papers also ranked within the 10 papers with the highest betweenness centrality, showing their importance in connecting groups of papers with divergent perceptions on the issue and being inappropriately cited on 14–85% of occasions (Fig. 3, Table 2). Five of the 11 most cited papers do not cite previous studies regarding global or multi-regional trends in jellyfish populations (i.e. they are ‘cited only’ papers). Thus, these papers had zero betweenness centrality and they contributed the initiate the perception. Richardson et al. (2009) was one of the 11 most cited papers and also had the highest betweenness centrality (b_i = 199.5) because it connected early views regarding possible trends with more recent research focused on the drivers of those putative trends. Of the citations made by Richardson et al. (2009) in support of jellyfish population trends (eight citations) 62.5% were ‘unsupported’ and 37.5% were ‘supported’, having a mis-citing score of 15, the third highest mis-citing score of the network (Sanz-Martín et al., 2016). In turn, Richardson et al. (2009) was always appropriately cited (100% ‘supported’), because this paper could be cited as evidence for concern about increases in jellyfish numbers (Fig. 3, Table 2).

Mis-citation was evident in the network (Fig. 1) with 48.9% of the citations being considered inappropriate for the statements they purport to support (Fig. 4, Table 3). The citing papers that had the most inappropriate citations were those that asserted that jellyfish are increasing at both global/generic and multi-regional scales (34.6%; Fig. 4, Table 3) whereas those papers that stated that jellyfish may be increasing at global, generic or multi-regional scales were less prone to poor citation practices (13.1%; Fig. 4, Table 3).

Discussion

This network of citations shows that jellyfish demography is a vibrant research field and the rate of papers referring to jellyfish population trends has increased by 87.1% per year since 1998, and far exceeds the 7.7% annual increase in the growth rate of scientific literature in ecology (Andersen et al., 2008). Examination of the network allows the identification of four stages involved in the development of this perception. First, the report of Brodeur et al. (1999) of an increase in jellyfish populations for the Bering Sea, followed by the review of Mills (2001), which is by far the most influential paper in this research topic. Despite the author, Claudia Mills, presenting balanced views in her conference talk at the first Jellyfish Blooms Symposium in 1999 and in the associated paper (Mills 2001), the possibility that jellyfish populations may be increasing was raised. The second stage involved papers that selectively cited the statements in Mills (2001) to assume that jellyfish populations were or may be increasing globally or across multiple regions and moved on to develop the narrative through reviews aimed at identifying the environmental drivers of purported increases (Purcell et al., 2007; Richardson et al., 2009). Following a decade of speculation, a third stage developed two quantitative meta-analyses to establish whether increases in jellyfish populations are a global phenomenon (Brotz et al., 2012; Condon et al., 2013). Interestingly, Brotz et al. (2012) and Condon et al. (2013), differ in their conclusions, despite reporting comparable results. Brotz et al. (2012) concluded that evidence for increasing jellyfish blooms was available for 28 of 45 (i.e. 62%) of the large marine ecosystems since 1950, of which 21% showed increases with high certainty. Condon et al. (2013) report that 30% (11 of 31) of the long-term records of jellyfish abundance included in their analysis showed a significant increase in jellyfish abundance since 1970. Brotz et al. (2012) conclude that ‘Jellyfish populations appear to be increasing in the majority of the world's coastal ecosystems and seas’ whereas Condon et al. (2013) conclude that ‘the perception of a global rise in jellyfish, possibly prompted by more jellyfish blooms in the 1990s, may therefore be best interpreted as part of an oscillation’. The fourth phase involves the present study, which follows the thread of citations in support of jellyfish trend statements flowing from the quantitative analyses of Brotz et al. (2012) and Condon et al. (2013) to previous speculative studies because available evidence is still subject to different interpretations. Poor citation practices lend credence to the perception of a global trend toward increased jellyfish blooms which may be less likely once more quantitative data are available and increasing data availability limits the margin for misinterpreting results.

A bias towards the possibility of increasing jellyfish blooms is further illustrated by three cases. First, Jackson et al. (2001), a highly cited paper in the context of global impacts in the ocean ecosystem (4523 cites according to Google Scholar, January 2016), proposed that oceanic degradation resulted, among other consequences, in outbreaks of jellyfish, providing only anecdotal evidence from a single location and study (Newell, 1988) to support this claim. Jackson (2008) further concluded in the abstract that ‘synergistic effects … are transforming complex food webs into … simplified … ecosystems with boom and bust cycles of … jellyfish …’ without offering any evidence in the paper to support this statement. Second, Brodeur et al. (2002) reported a major increase in jellyfish in the Bering Sea, but subsequently reported declines in populations (Brodeur et al., 2008). Numerous papers published after Brodeur et al. (2008), however, continue to cite Brodeur et al. (2002) as an example of a sustained increase in jellyfish and ignore the later paper showing that jellyfish populations in the Bering Sea are variable. Third, a United Nations Environment Report (Turley & Boot, 2010) and one of the most widely cited reviews on impacts of ocean acidification (Doney et al., 2009; cited 1602 times in GS, accessed January 2016), cited Attrill et al. (2007) to claim a link between ocean acidification and increased jellyfish numbers, but neither papers acknowledged the rebuttal of Attrill et al. (2007) by Haddock (2008), the erratum published by Attrill & Edwards (2008) or the expanded analysis of Attrill et al. (2007) by Richardson & Gibbons (2008) that showed no link between acidification and jellyfish populations. Indeed, the misconceptions resulting from mis-citation are even more dangerous when they are contained in papers published in highly influential journals, which provide a platform for those papers to be highly cited. Duarte et al. (2015) have argued that poor citation practices are one of the elements that have perpetuated perceptions on ocean calamities (including rising jellyfish populations) that are contributing to an overly negative perception on the state of the ocean. Our study confirms that that mis-citation facilitated the perception of rising jellyfish populations.

Our analysis suggests that inappropriate citations may have been facilitated by the presence of imprecise or ambiguous language and contradictory statements in papers with variable conclusions. For instance, we determined that 85% of papers referencing Mills (2001) were inappropriate, including 19% that were unsupported. In that paper Mills poses a question in the title that is not clearly answered in the text and, in searching for answers, the reader might focus on selected sentences thereby choosing to ignore the balanced account of evidence for increases and decreases that Mills (2001) presented throughout her paper (Fig. 3, Table 2). Imprecise and ambiguous language may also introduce uncertainty into our analysis as we assigned statements to one of several categories, implying that authors citing such papers will also find difficulties in constraining the domain of the statement. For instance, Lynam et al. (2006) reports ‘Jellyfish biomass has increased substantially in several locations worldwide, perhaps as a consequence of fishing (Mills, 2001)’ showing that vague statements have facilitated mis-interpretation, potentially through greater emphasis on the use of examples showing population increases than those showing declines. Mills (2001) was also cited as evidence of both increasing and decreasing populations, but in most cases increasing examples are discussed first and decreasing examples used as caveats. For instance, Barz & Hirche (2007) stated ‘in recent years the abundance of scyphomedusae is increasing in many ecosystems, … but decreases have also been reported (Dawson et al., 2001; Mills, 2001)’, where Dawson et al. (2001) is the only paper whose maximum degree of affirmation statement was classified as ‘decreasing’ jellyfish trends.

The accuracy of citations was not aided by the fact that some influential review papers were often cited indirectly, as shown by the betweenness centrality pattern of the network (i.e. Purcell et al., 2007; Richardson et al., 2009; Table 2). Mills (2001) was clearly the most influential paper in the development of the perception of rising jellyfish populations (54 citations, 17.6% of the 306 citations used in support of trend statements). However, the development of a perception based on a concept introduced in an influential seminal paper, which then is followed by other researchers, is not unusual. In fact, it has been proposed to be the process through which paradigms and even disciplines develop (Krishnan, 2009). Paradigms are not static, but tend to change as science advances through what Thomas Kuhn (1962) called scientific revolutions. A decade after Mills (2001) was published we may be at a point where the perception of rising jellyfish populations (Condon et al., 2013) may be shifting. Indeed, Richardson et al. (2012) proposed that researchers should move beyond the question of global trends to focus on factors controlling those populations that are certainly increasing by managing the purported anthropogenic drivers of blooms, including eutrophication, overfishing and species translocations.

Todd et al. (2010) concluded that ‘one in four citations in marine biology papers is inappropriate’. Our analyses, however, indicate a rate of inappropriate citations of 49% in the literature addressing jellyfish population trends. We also provide evidence of inappropriate citation of papers with variable or ambiguous conclusions, propelling the rapid growth of statements that jellyfish numbers are, or may be, rising globally. We submit that the reason for differential citation bias for papers asserting or questioning that jellyfish blooms are increasing may be derived from the aversion of the ecological community towards Type II errors or false negatives, as the implications of erroneously concluding that jellyfish blooms are not rising when in fact they might be, are higher than the reverse, a concern that has been recently highlighted by Richardson et al. (2012). Our results suggest the accumulation of multiple lines of evidence occasionally forms an imprecise framework statement (e.g. assertions of jellyfish increasing globally or in generic populations based on ‘equivocal’ and ‘regional’ conclusions) that can be easily propagated. Guidelines to robust citation practices highlight the importance of including the citation immediately after the phrase that calls on it and avoiding the clustering of references at the end of a sentence (Dupps, 2008). This helps clarify which statement each reference supports. It is also important to provide a balanced account of the research question discussed, acknowledge the caveats the authors introduced, and actively search for counter-evidence (Harzing, 2002). However, the onus of avoiding mis-citations also rests on the authors, who should not only ensure they adhere to robust citation practices but also that they too avoid ambiguous statements that may lead to misinterpretations. Catchy, declaratory titles are conducive to misinterpretations as they typically capture only some of the findings reported, which might direct citing readers to focus on the conclusion highlighted in the title. Furthermore, we highlight the importance of adhering to best practices in hypothesis testing, enabling strong inferences in science (Platt, 1964). As a community, we must ensure our statements about scientific findings in general are accurately substantiated and carefully communicated, thereby minimizing the possibility of being mis-cited, such that incorrect perceptions, as in the case of jellyfish blooms, do not develop in the absence of rigorous testing and robust evidence.

We are making the database available in Sanz-Martín et al. (2016) to allow readers to challenge the robustness of our analysis by reading the sources cited and decide the extent to which the statements are or not supported by references used, reclassify the statements according to its affirmation degree and spatial scale, and rerun our analysis.