Evidence Maps: Communicating Risk Assessments in Societal Controversies: The Case of Engineered Nanoparticles

1. INTRODUCTION

Debates inside scientific communities—especially those regarding risks—can reach the political and public arena and raise societal concerns and fears and stoke political controversies. Examples come to mind easily: the ongoing nuclear debate, the GMO issue, the concerns about the impact of cell phones on health, and the debate on nanotechnology-related risk potentials. It seems that “science” no longer belongs only to scientists.⁽¹⁾

Therefore, science communication, especially risk communication, has become a fast growing topic in the academic world.⁽²⁾ No doubt, science communication is not easy. It faces many hurdles. The public and science speak different languages and refer to the same issues with different perspectives and knowledge backgrounds. One striking example is the difference between risk assessment and risk perception. Many studies have revealed that lay people use their own criteria and heuristics in order to evaluate risks, which differ from the principles used by professional risk assessors.^(3,4)

Nevertheless, there is a link between both worlds. In case of new and emerging technologies the base of intuitive risk perception is not experience, but information,⁽⁵⁾ which has its initial origin in science. However, the scientific evidence about adverse effects of new technologies to human health is often quite limited and inconclusive. This makes a transparent and fair characterization of scientific evidence, which is one of the crucial issues in hazard and risk assessment, a demanding task. Especially in those cases where the evidence is incoherent and open to different interpretations, risk assessors have to cope with a difficult communication task: How should they describe their scientific data, connect them together into an overall picture, and justify their conclusions so that decisionmakers, the public as well as various stakeholders, can understand their assessment? Furthermore, how can risk assessors themselves avoid traps such as confirmation biases that may distort their judgments about the weight of the various parts of evidence?^(6,7)

In the following, we explore how these challenges can be met in order to improve the reporting of hazard assessments. Hereby, we will focus on a special example, the hazard assessment of nanoparticles. This topic has gained public attention in many countries because of public concerns that nanoparticles might have adverse health effects.⁽⁸⁾ The question is how the reporting of hazard assessments can be improved so that interested nonexperts can obtain a reasonable and balanced picture of the scientific evidence?

The remainder of the article is structured in five parts. First, a short overview is provided on issues of evidence reporting and the concept of evidence maps is introduced. Second, we discuss the hazard assessment of nanoparticles as an illustration of the challenges of reporting such assessments. Third, the example case is applied to show how evidence maps can be used to improve the quality of reporting. Fourth, we provide empirical evidence that supports the usefulness of evidence maps for communicating hazard assessments. Finally, we discuss the advantages as well as the limitations of evidence maps as a tool for risk communication.

2. REPORTING THE QUALITY OF EVIDENCE

A vast amount of literature has explored the hurdles and failure sources in summarizing and evaluating scientific evidence from various angles.^(9-11) The Equator Network,⁴ which promotes the development and dissemination of guidelines for transparent reporting of health research, has published many recommendations for summarizing evidence accurately and reliably. One of them is the PRISMA statement, which lists 27 recommendations for communicating systematic reviews.⁽¹²⁾ A special problem emerges when a quantitative integration of the data is not possible and the evidence characterization must rely on a qualitative approach. Usually, this is the case in risk assessments where many different research types—cell studies, animal research, and epidemiology—have to be synthesized.

Several institutions have stressed the importance of improving the presentation of hazard and risk assessment findings and related scientific opinions. For instance, the EU⁽¹³⁾ underlines: “More consistency in the format of the opinions would facilitate the use of opinions, at least for non-professional readers who wish to quickly understand the answers” (p. 173). The U.S. EPA⁽¹⁴⁾ has pointed out four criteria for evaluating the quality of risk characterization. They suggest that “risk characterization is … judged by the extent to which it achieves the principles of Transparency, Clarity, Consistency, and Reasonableness” (p. 14). This raises the question which formats of communicating are appropriate in order to reach these aims.

At least three main options are available: the level of evidence can be described qualitatively by verbal phrases integrated in a free-form narrative, by a standardized set of verbal descriptors, or quantitatively, that is, by using confidence intervals.

In its cancer risk assessment guideline, EPA advises to use “evidence narratives,” that is, “the narrative explains the kinds of evidence available and how they fit together in drawing conclusions, and it points out significant issues/strengths/limitations of the data and conclusions” (pp. 1–12).⁽¹⁵⁾ The evidence narrative should include among others: (1) the conclusions of the assessment, (2) the key evidence on which the conclusion are based, (3) the conditions that describe the conditions of expression of the adverse health effect, and (4) the data that describe the mode of action if available. EPA⁽¹⁵⁾ uses five standardized descriptors (e.g., “Likely to Be Carcinogenic to Humans,”“Suggestive Evidence of Carcinogenic Potential,” etc.) as part of the narrative to express the conclusion regarding the weight of evidence for carcinogenic hazard potential. Similar descriptors are used by IARC for characterization of carcinogenic hazard potential.⁽¹⁶⁾

A different approach has been pursued in the California EMF project, where a qualitative Bayesian approach was used for evidence characterization, combining qualitative descriptors with quantitative terms.⁽¹⁷⁾ But there, too, the need for structured argumentation in order to increase reliability and transparency of the reporting is emphasized.⁽¹⁸⁾

2.1. Evidence Maps

The concept of evidence maps was introduced by Schütz et al.⁽¹⁹⁾ These maps are designed to depict the reasons that lead experts to their conclusions when summarizing and evaluating the scientific evidence about a (potential) hazard. The theoretical starting point of the evidence map approach is to view evidence characterization as a form of argumentation. This approach is motivated by the idea that transparent evidence characterization should make clear how a conclusion regarding the causal relationship between the exposure to an agent and its effect on human health (i.e., the hazardousness of an agent) is reached.

Toulmin has provided a detailed analysis of how a conclusion or claim is substantiated in an argumentation.⁽²⁰⁾ According to his view, an argument typically includes six main components: claim, data, warrant, backing, rebuttal, and a qualifier. In substantiating a claim, one usually refers to some data. The justification for using the data as grounds for the claim is expounded in the warrant. Often, a backing might be used to further strengthen the warrant. While in a line of argumentation the data are explicitly appealed to, the warrant and the backing are often not explicitly indicated. However, it is often the warrant and the backing rather than the data that provide reasons for attacking a claim. In the extreme, a rebuttal might completely reject the claim, or at least formulate conditions of exception. In many cases, this will result in a qualifying statement. Thus, one may use qualifiers by saying that a claim is, for instance, “probably” true, or holds “unless” certain conditions apply.

Often, evidence characterization regarding a hazard or a risk is confronted with inconsistent or conflicting evidence. How can inconsistent or conflicting evidence be represented in the argumentation scheme? One way is to take those pieces of evidence as data that, say, support the existence of a hazard, and then use the other pieces of evidence, which do not speak for the existence of a hazard, as (part of) the rebuttal. Although it is surely possible to rebut a claim that is built on certain data by referring to other contradicting data, this has the disadvantage that the logical status of data and rebuttal are confounded.

A more appropriate conceptualization would be to put forward two independent lines of argumentation instead of including all data into one single line of argumentation: a pro argument that is built on the data that support the claim that a hazard exists, and a con argument that rests upon evidence that does not support the existence of a hazard. This would allow for each argument to distinguish clearly the data, the warrant, and backing as well as the rebuttal. The combined conclusion will then rest on both lines of argumentation. Depending on which argumentation is more convincing, it will either claim the existence or absence of a hazard. However, since the state of scientific evidence in hazard or risk assessment is rarely unambiguous, the claim will often include qualifying statements.

Conceptualizing evidence characterization as two lines of argument—a pro and a contra argument—introduces another theoretical link: the weight-of-evidence (WOE) approach. In a WOE evaluation, positive and negative evidence should be included and weighed against each other according to their scientific quality. Note, however, that apart from this basic idea, there is no standard WEO approach, but a diversity of qualitative and quantitative procedures.^(21-24)

Building on these ideas, evidence maps provide a graphical representation of the arguments that speak for and against the existence of a causal relationship between exposure to a (potentially) hazardous agent and the conclusions that are drawn. Fig. 1 shows a template of an evidence map. On the left in the evidence map, boxes with the pro and con arguments are drawn. These basically represent the data. Both the pro and con arguments are extended by supporting or attenuating arguments (depicted as ellipses) that typically refer to theoretical or methodological issues, which either strengthen or weaken the argumentative power of the pro or con arguments.

The conclusion, which represents the claim in the Toulmin scheme, is drawn on the right-hand side of the evidence map and is linked with arrows to the pro and con arguments. Typically, an evidence map will address only one endpoint, for example, cancer.

Two elements are included in an evidence map that are not part of the Toulmin scheme. The first is the remaining uncertainties that are located as a box underneath the conclusion and are linked to it with an arrow. Explicitly stating the uncertainties that remain with regard to a hazard or a risk is an important element of characterization⁽¹⁴⁾ and thus also for evidence characterization. The remaining uncertainties can be considered to correspond to the qualifier in the Toulmin scheme, although qualifying statements will often already be found in the conclusion itself. The second additional element is a box on the upper right side of an evidence map that informs about the “evidence base.” In this box, the number of studies that are considered in the evidence map is listed. This information is indispensable for evaluating the state of knowledge that exists with regard to a particular hazard. Note that this argumentation-based evidence map is different from another evidence map approach that presents medical research findings in a table format without addressing pros, cons, and uncertainties (see http://evidencemap.org).

3. THE CONSTRUCTION OF EVIDENCE MAPS ON ENGINEERED NANOPARTICLES

The increasing use of engineered nanoparticles in industrial production calls for a sound assessment of their potential hazards to human health.^(25,26) Recently, the U.S. EPA has conducted a preliminary assessment of the toxicity of nanoscale titanium dioxide (TiO₂) that provides a good example for evidence characterization in a scientific report.⁽²⁷⁾ Of special interest in the current context is the final chapter that summarizes the results of nano-TiO₂ carcinogenicity studies in animals, as this illustrates the difficulties in summarizing conflicting evidence. On the one hand, the report informs the reader that TiO₂ (including nano-TiO₂) has been classified as “possibly carcinogenic to humans” by IARC⁽¹⁶⁾ and as “carcinogenic” by the Canadian Centre for Occupational Health and Safety.⁽²⁸⁾ On the other hand, the report advises that the U.S. National Institute for Occupational Safety and Health is not considering TiO₂ as a “potential occupational carcinogen” because of insufficient evidence.⁽²⁹⁾ Finally, the reviewers present their own summary of the available evidence—five studies—in a table that depicts a controversial picture. The reviewers do not attempt to give their own conclusions, nor do they explain the heterogeneity of the findings. However, in order to remain fair, it should be acknowledged that the authors of the U.S. EPA report⁽²⁷⁾ claim that their report does not represent a completed or even preliminary hazard assessment. In our perspective, however, this EPA document can be used to demonstrate how better reporting of conflicting evidence via evidence maps can be obtained.

As an example of how evidence maps can be used to describe conflicting evidence, we refer to the project NanoHealth. This project, conducted by a consortium of German and Swiss research institutes⁵ in the years 2006–2009, aimed at the hazard assessment of selected nanoparticles, among them nano-TiO₂. In a first step, the current state of knowledge of adverse effects of various nanoparticles with respect to selected endpoints was summarized in an expert opinion report. The main results were presented in form of evidence maps. Fifteen experts⁶ from Germany and Switzerland then discussed, at a two-day workshop, these preliminary evidence maps. Based on the discussion results, the evidence maps were revised.⁽³⁰⁾

3.1. Application of Evidence Maps in Assessing Health-Relevant Effects of Engineered Nanoparticles

The following example of an evidence map serves as an illustration for the evidence map concept in the domain of health-related effects of engineered nanoparticles. For space reasons, we present here only one of the 22 end points evaluated in the project NanoHealth. Our choice was generation of reactive oxygen species (ROS) or oxidative stress by TiO₂ nanoparticles, as all sections of the evidence map drafted for the purpose are occupied (i.e., pro, contra, supporting, attenuating, remaining uncertainties), so that the schema is well suited to illustrate a case. Note that this section is meant to provide an example for the application of an evidence map that reports a hazard assessment of TiO₂ nanoparticles.

ROS are noxious forms of oxygen that play a significant pathophysiological role in oxidative stress and thus in various diseases as well as in the aging process. Within the organism, ROS develop in the mitochondria as a by-product of cell respiration or through inflammatory cells to fight viruses and bacteria. The ROS generated during the normal cellular metabolism are faced with numerous antioxidative mechanisms in the cell. A change in this steady state in favor of the oxidants, which is accompanied by damage to the cellular components, is called oxidative stress. ROS generation is thus the preliminary stage of oxidative stress. If ROS are quickly degraded, and are thus “deactivated,” oxidative stress does not occur despite ROS generation. If, however, ROS are generated constantly and antioxidative processes are exhausted then oxidative stress possibly may induce DNA damage and hence tumor formation can result.⁽³¹⁾ Oxidative stress triggered in tissues may lead to a chronic inflammatory reaction. Under these circumstances the immune system is weakened, aging accelerated, and allergies, cardiovascular ailments, and cancer may occur.

For hazard evaluation purposes, we consider both studies proving ROS generation and those that detect oxidative stress as the two are inseparable in cell cultures since it is only a matter of time before induced ROS generation in a monoculture leads to oxidative stress, which would never be the case in the body.

3.1.1. Basis of Evidence

Methodologically acceptable studies published before October 15, 2007, that examine the connection between engineered TiO₂ nanoparticles and ROS generation or oxidative stress were selected by the 15 nanotoxicological experts participating in the NanoHealth project in a consensus process. The primary criterion in choosing relevant studies was involvement of the inhalation exposure pathway for TiO₂ in the lung. In determining methodologically acceptable nanotoxicological studies, quality criteria listed in the relevant literature were decisive.⁷ Good physical and chemical characterization of nanoparticles (particle size, surface characteristics, etc.) and correct performance of biological methods (including use of positive and negative controls, concentrations in an acceptable range, and standard operation procedures, for example, using reference materials, etc.) were of paramount importance. The more completely a publication met these criteria the more reliable are the results concerning the health-related effect. Studies sometimes include methodological sources of errors, making the results unsuitable for hazard characterization. For instance, test procedures involving interaction between nanomaterials and analytes or contaminants and solvents used for suspensions could be much more toxic than the nanomaterial itself and, thus, falsify the results.^(32-34) Studies suffering from such methodological shortcomings are excluded from hazard assessment.

After screening, only seven methodologically acceptable studies were left. These studies provide the basis of evidence in this example. The evidence base consists of six cell studies and only one study in living animals (see Table I).

Table I. Studies on ROS (Reactive Oxygen Species) and Oxidative Stress by Titanium Dioxide Nanoparticles

Material	Species	Result	Reference
TiO2 P25 (80:20 anatase/rutile), size 20–30 nm, BET surface area: 50 m2/g, negative surface charge, rel. low hydrophobicity index, photocatalytic	Murine phagocytic cell line (RAW 264.7). Cells treated with 10 μg/mL of each individual NP for 16 hours.	ROS generation detected by using the fluorescent dyes, dichlorofluorescin diacetate (DCFH-DA), and MitoSOX Red. Measurement of TNF-α production. Result: no formation of ROS despite overload and uptake.	Xia et al.(40)
Ultrafine TiO2 (size <30 nm)	Rat alveolar macrophages, single intratracheal instillation of 2 mg nanoparticles per rat (female albino rats, 175–200 g). Animals sacrificed 1, 4, 8, and 16 days after treatment.	Glutathione peroxidase (GPx) and glutathione reductase (GR) were assayed. Result: Weak oxidative stress and antioxidative response (enhancement of antioxidant enzymes), which could not diminish the enhanced lipid peroxidation and increased rate of hydrogen peroxide generation; level of glutathione remained decreased.	Afaq et al.(35)
Pure anatase TiO2 NP, size 20–75 nm, conc. 30 ppm (30 μg/mL)	Human lung epithelial cells A549.	The relative amount of expressed ROS calculated by [(Ftest)/(Fref) × 100]; use of dichlorofluorescein as a sensitive probe for ROS inside a cell. Exposure resulted in a promoted increase in reactive oxidizing species after 4 hours—oxidative stress.	Limbach et al.(39)
TiO2 P25 (70:30 anatase/rutile), photocatalytic, negative surface charge, particle size 30 nm, 5–120 ppm slurry conc. Aggregates in both HBBS and DMEM	Brain microglia BV2 (mice); 1, 6, and 18 hours exposure to P25 using the CellTiter-Glo assay, release of H2O2 measured with Image-iT and OxyBURST.	Release of ROS after 1 hour.	Long et al.(38)
TiO2 (Degussa—80% anatase, 20% rutile), 20–80 nm, high cationic purity, both native and with surface methylation, BET specific area 50 m2/g−1	Human lung epithelial cells A549. The TiO2 samples suspended in HBSS, sonicated and added to the cells for a total of 2 or 4 hours in HBSS at 400 μg/cm2. One hour after start of treatment, DMPO or TEMPOL added.	Oxidative stress, but unrealistic high concentration for an effect.	Singh et al.(36)
TiO2 (Degussa), specific surface area of the particles 6.64 m2/g; concentrations of 15, 31, 62, 125 and 250 μg/mL	Human alveolar epithelial type II-like cell line A549 treated for 4, 6, 12, or 24 hours with the particles. Prior to treatment cells were suspended in DMEM without FCS for 24 hours.	Amount of total glutathione (both (GSSG) and reduced GSH, with the GSSG converted to GSH) after 4 hours treatment measured. A significant depletion of GSH with all concentrations for TiO2 occurred: oxidative stress.	Monteiller et al.(37)
TiO2 pure anatase NP, size 5 nm, surface 242 m2/g, surface mean diameter 6 nm, mass conc. 0.53, 5.3, 53 μg/cm2 (resp. 1, 10, and 100 μg/mL)	Human bronchial epithelial BEAS-2B cells, exposure for 24 hours.	Measurement of IL-6 by ELISA showed small and not biologically important secretion of IL-6; no indication for ROS formation.	Veranth et al.(41)

3.1.1.1. Pro and con arguments for generation of ROS or oxidative stress by TiO₂.  The evaluated studies provide both pro and con arguments (see Fig. 2). Some include aspects that influence proof of effect or absence thereof in their result. These aspects are mainly methodological strengths or weaknesses affecting results concerning effect strengths, health relevance versus effect weakness or lack of any effect on human health, and side effects that support or contradict the main biological effect. These are shown in the evidence map as either supporting or attenuating the main arguments pro or contra. By illustrating this, the evidence map helps one to see at a glance the strengthening or weakening of the main arguments for and against.

3.1.1.2. Pro argument.  The evidence map shows five studies supporting the pro argument, which means that these studies showed generation of ROS or oxidative stress (see Fig. 2). The only in vivo study in this area is that of Afaq et al.⁽³⁵⁾ It tested various biochemical and chemical parameters in alveolar macrophages of rats in order to examine the lung toxicity of ultrafine TiO₂. A single intratracheal⁸ exposure with 2 mg of this nanomaterial per rat caused weak oxidative stress, to which the alveolar macrophages showed an adaptive reaction. The antioxidative response, however, was low, resulting in a permanent displacement of the oxidative steady state. The data indicate that the induction of antioxidant enzymes does not suffice for the cells to protect themselves and prevent a toxic effect of ultrafine TiO₂. This result strengthens the meaningfulness of the pro argument. The positive findings of the study are, however, of limited validity due to the method of administering the particles, which does not correspond to any physiological process or expected exposure scenario. For this reason, the method of administration is considered as attenuating factor in the evidence map.

The second study supporting the pro argument was carried out by Singh et al.⁽³⁶⁾ Subject of examination was the role of the particle size and the surface chemistry in triggering proinflammatory effects in vitro in human pulmonary epithelial cells A549 exposed to ultrafine TiO₂ particles. The exposure caused oxidative stress, although the effect manifested at an unrealistically high concentration of particles. This methodological aspect makes the biological response shown less convincing where the hazard potential of TiO₂ nanoparticles in the inhalation pathway is concerned and is therefore shown in the evidence map as attenuating where the arguments in favor are concerned. Because of this serious shortcoming, the study of Singh et al.⁽³⁶⁾ is included in the assessment with reservations.

The study of Monteiller et al.⁽³⁷⁾ tested the proinflammatory reactions of a human alveolar type II epithelial-type cell line to an exposure to, among others, TiO₂ nanoparticles at different concentrations and with varying exposure duration. “Proinflammatory reactions” in a cell culture mean formation of proinflammatory mediators like interleukins. Oxidative stress was observed. There was also a much stronger proinflammatory reaction to (smaller) nanoparticles than to (larger) fine particles with the same mass dose. Based on the entirety of the results, the authors concluded that the large particle surface is the key factor for the inflammogenicity of the particles rather than their quantity.

Long et al.⁽³⁸⁾ tested in vitro the effect of the nanomaterial Degussa P25⁹ on brain microglia (BV2) of mice. These cells are comparatively sensitive to oxidative damage. The biological reaction of the BV2 brain microglia to concentrations of P25 is not considered cytotoxic (10–20 μg/mL) but rather as oxidative stress. However, the relevance of this result is restricted by the fact that the microglia remained stable and resistant in spite of the generation of ROS (see on the evidence map the section “attenuating” with regard to the pro argument). This is obviously due to the fact that microglia cells are macrophage-like cells that phagocytize particles and respond to this with the production of oxygen radicals. This is a particle-induced but a normal reaction and no severe effect will occur.

The study of Limbach et al.⁽³⁹⁾ tested the development of ROS in human epithelial cells through nanoparticles of pure TiO₂ (particle size 20–75 nm, concentration 30 μg/mL). The exposure resulted¹⁰ in oxidative stress within the cells. The pro argument is supported by the found catalytic activity of the nanomaterial. The positive findings of the study are, however, of limited validity due to the fact that the biological system of the A549 pulmonary cells used in this study is insufficient to prove the tested biological effect as these cells produce per se only small amounts of ROS, even under severe circumstances. It should nevertheless be considered that in this study TiO₂ caused the weakest reaction compared to other tested pure metal oxides.

To summarize: Attenuating arguments appear to dominate overall on the “pro” side of the evidence map. Therefore, no definite conclusion can be drawn that TiO₂ nanoparticles can cause ROS or oxidative stress.

3.1.1.3. Con argument.  As shown in the evidence map, two studies support the con argument, which means that two of the evaluated studies showed generation of ROS or oxidative stress (see Fig. 2). The study of Xia et al.⁽⁴⁰⁾ tested the cellular effects of Degussa P25 among others. For the test, macrophage-type cells were used because macrophages in the lung represent the first line of defense against particles. Subsequently, the overproduction of ROS would be a possible defense mechanism against the particles’ toxicity. This hypothesis was not confirmed by the study's findings: in spite of overload conditions and particle uptake, the cells showed no development of ROS. This strengthens the meaningfulness of the con argument. The sound methodological quality of this study, which included a comparative analysis of multiple substances under investigation with positive controls and multiple measurement indicators, increases its weight in the assessment of possible health-related effects.

The second study supporting the contra argument was carried out by Veranth et al.⁽⁴¹⁾ It examined the effect of anatase TiO₂ nanoparticles (size 5 nm) on BEAS-2B cells, among others. No generation of ROS by amorphous TiO₂ despite high concentrations was shown. In addition, it was found that (smaller) nanostructured particles are not consistently more effective than the same number of (larger) micrometer-sized particles of the same nominal composition, as far as the induction of proinflammatory cytokines into the tested in vitro models is concerned. The ability of some nanoparticle variants to induce the production of IL-6 through BEAS-2B cells did not correlate with the generation of ROS in the cell-free media. Based on these results, the authors concluded that the metal oxide particles are to a small extent capable of producing IL-6 secretion in the BEAS-2B cells. Experiments of this kind must consider particle artifacts of nonbiological effects, as well as inherent restrictions of cell culture studies, when in vitro results are interpreted. The study's authors underlined that the technologically produced metal oxide nanoparticles are not highly toxic to pulmonary cells compared to environmental particles also tested in this study.

Three supportive and one attenuating arguments are found on the con side overall. The con side of arguments is hence weaker than the pro as it is presented by only two studies. However, it provides stronger evidence that TiO₂ nanoparticles do not cause ROS generation or oxidative stress. Nanotoxicological judgment is that the methodological excellence of the study of Xia et al.⁽⁴⁰⁾ supplies comparatively the strongest con arguments in the presented hazard assessment.

3.1.2. Remaining Uncertainties

The studies’ results do not allow a judgment of the difference in effects caused by the anatase versus the rutile particle form (see footnote 9). Furthermore, the question remains unanswered whether there is a threshold level for a noxious ROS generation. A general problem with the issue of “oxidative stress” is that there are hardly any in vivo studies compared to in vitro studies, since oxidative stress in vivo can hardly be reproduced or directly measured. However, in vitro tests in general are less conclusive for hazard assessments. Although it may seem striking that many cell studies find a ROS release, this is actually not astounding because mechanistic studies are often carried out with very high particle concentrations in order to trigger the effect reliably. Findings of experiments with unrealistically high particle concentrations are, however, only to a limited extent conclusive for hazard assessments and irrelevant for risk evaluations.⁽³⁶⁾ An additional problem arises if the methodology used in studies is not convincingly suitable to prove that the found ROS actually caused oxidative stress.⁽³⁹⁾ Finally, the only in vivo study evaluated by us for the field of ROS or oxidative stress by TiO₂ nanoparticles (Afaq et al.⁽³⁵⁾) has methodological shortcomings, as it uses the method of administering particles that does not correspond to any physiological process (see footnote 8).

3.1.3. Conclusion

No consistent trend toward increased toxicity can be observed with five positive opposed to two negative findings because of the many limitations of the positive studies and the high quality of one negative study. Thus, the appraisal of evidence leads to the conclusion that because of the contradictory test results no conclusive assessment of the causality between TiO₂ nanoparticles and oxidative stress is possible.

As aforementioned, 22 evidence maps were established as part of the project NanoHealth. What conclusions concerning health-relevant effects of engineered nanoparticles can we draw from this cornucopia of information? Some experience has already been gained in the fields of medicine and biology as nanoparticles are not really new, but since they started making them specifically for particular uses there has been such a lot of particles of varying sizes, chemical composition, morphology, etc., as to be confusing. It is hence obvious they have to be evaluated on a case-by-case basis to determine what differences or duplications exist where biological effects are concerned. Differentiated risk evaluation using evidence maps makes it possible to distinguish between nanomaterials where their biological activities are concerned. Evaluated specialized literature, for example, yielded clues to weaker acute toxicity of amorphous TiO₂ compared to the crystalline variety. Testing these forms of material for health relevance individually seems reasonable.

The evidence map presented here indicates that there are major knowledge gaps. There is no adequate database for a conclusive hazard assessment. For the whole project, our evaluation only showed consistent clues to the ability of nanoparticles to cross a variety of tissue and cell barriers except for the intact epidermal barrier. No further evaluation of any other endpoints was feasible due to the lack of data.

4. EMPIRICAL EVIDENCE IN SUPPORT OF EVIDENCE MAPS

An important question that needs to be addressed when evaluating the usefulness of evidence maps as a tool supporting nonexperts in reaching an informed judgment about a hazard claim is whether nonexperts actually value the aspects that are the essence of expert hazard assessments and that constitute the components of an evidence map: the arguments that speak for and against the existence of a causal relationship between exposure to a (potentially) hazardous substance or condition and the endpoints that are considered, as well as the conclusions that are drawn and the remaining uncertainties. Do nonexperts consider these aspects as relevant for their own judgment and how important are these aspects relative to each other?

This question was addressed in a self-recruited online survey among staff and students of the University of Innsbruck (Austria). The survey was made known to the prospective participants via announcements at the university. From 193 volunteers, 171 completed the survey, which was conducted January to March 2011. Due to the self-recruitment, no further information is available about response rates and about differences between responders and nonresponders.

Participants were aged between 19 and 64 years, median age was 24; 55% of the participants were women, and 90% of the participants were university students or had an academic background. Their professional background was as follows: 35% psychology, 11% economy, 7% educational science, 5% law, 5% biology, and 4% medicine. The remaining belong to various other disciplines. This sample pool is particularly suitable for this kind of complex issue because it represents the target group—people with basic knowledge in scientific argumentation—for which evidence maps are constructed.

Participants were presented a list of items, which included all combinations of the four evidence map components (information about the evaluation base, pro risk information, con risk information, and information about the remaining uncertainty). Each component could be either present or absent, resulting in a list of 16 items ranging from no information at all to information about all four components.

We decided to use this generic approach to the evaluation of the usefulness of evidence maps in order to eliminate a possible confounder effect, that is, the impact of various levels of knowledge in toxicology. In addition, the generic version allows a simple test strategy. For economic reasons, a within-subject design was chosen, that is, each study participant had to evaluate all 16 items. To avoid order effects, the 16 items were presented in a random order. Study participants evaluated each item on a five-point Likert scale with regard to its usefulness for reaching an informed judgment (1 = not useful at all, 5 = very useful).

As shown by Fig. 3, the nonoverlapping of the 95% confidence intervals indicates statistically significant differences among the means. The respondents evaluate the comprehensive evidence map option (information about evidence base, pro risk, con risk, and uncertainty) as most useful. Clearly, the “no information” item is considered as least useful. The usefulness of the other combinations of the evidence map components receive judgments in between these two extremes, with ratings of usefulness roughly increasing with the number of components. Thus, the data suggest that the respondents prefer to get all information included in an evidence map in order to consider it as “very useful for evaluation.”

A question in the online survey addressed the participants’ attitude toward technological progress (How do you evaluate the technical progress?). In order to test whether this attitude toward technological progress influences the evaluation of information components, ANOVAs were conducted for each of the 16 tested combinations of information with technology skeptics and technology supporters (i.e., high versus low scores on the technology progress attitude scale) as the independent variable and information evaluation as the dependent variable. No statistically significant differences between technology skeptics and technology supporters were found (all p-values are between 0.07 and 0.897, after Bonferroni correction).

Fig. 4 presents the mean values of the four evidence map components on a Likert scale in terms of importance for risk evaluation (with the endpoint 1 = least important, and 4 = most important). Here, pro risk information is at the top. As in Fig. 3, the nonoverlapping of the 95% confidence intervals indicates a significant difference. Again, we did not find significant differences between technology skeptics and technology supporters (all p-values in the four ANOVAs >0.197) with respect to the perceived importance of the evidence map components. These results suggest a “pro risk” bias, similar to the negativity bias,⁽⁴²⁾ and reminds one to describe explicit, transparent, and concise pro hazard argumentations in an evidence map, especially with respect to the attenuating factors.

These findings support the usefulness of the “evidence map” format for communicating evidence about hazard assessments. Further results from recent experimental but not yet published studies are promising. For instance, they suggest that nonexperts evaluate evidence maps as better formats for communicating complex and contradictory risk information compared with evidence narratives.⁽⁴³⁾ Another experiment revealed that evidence maps support faster information processing.⁽⁴⁴⁾ However, more research is needed to show how nonexperts integrate the information components presented in an evidence map into a judgment about the strength of evidence.

5. THE BENEFITS AND LIMITATIONS OF EVIDENCE MAPS

Evidence maps are designed to depict the reasons that lead experts to their conclusions when evaluating and summarizing the scientific evidence about a potential hazard. That is, they build upon systematic reviews and do not substitute them. Their specific value lies in the organization and easy-to-grasp graphical representation of the expert reasoning. They provide the arguments that speak for or against the existence of a causal relationship between exposure to a (potentially) hazardous substance or condition and the endpoints that are considered, as well as the conclusions that are drawn and the remaining uncertainties. In doing so, they might provide several advantages compared to the conventional narrative summary of hazard or risk characterizations.

Above all, evidence maps promote the openness and transparency of an assessment. Thus, evidence maps help to meet the criteria of good risk characterization of EPA.⁽¹⁴⁾ By making explicit the overall structure of the argumentation with respect to a cause-effect relation, evidence maps help scientists give their assessment a clear argumentative structure.⁽¹⁸⁾ Furthermore, evidence maps can facilitate a constructive dialogue in a debate about potential hazards between the experts with conflicting opinions. They may help to describe them in a more transparent way and thus to agree about the disagreements in assessment.

As a communication tool, evidence maps seem to satisfy the information preferences of nonexperts, as suggested by the empirical data presented earlier. In addition, evidence maps may assist nonexperts to pay attention to the entire scientific picture. The graphic format of the evidence maps is easy to grasp and draws attention to both the pro and con arguments. In so doing, evidence maps work against the conformation bias; they motivate people to take into account both sides of argumentation in a risk controversy.

Theories on cognition and text comprehension such as mental model theory⁽⁴⁵⁾ and assimilation theory⁽⁴⁶⁾ support this view. Additionally, empirical research shows that graphical overviews and hypertext structures might enhance text comprehension,⁽⁴⁷⁾ especially when the given information is complex.⁽⁴⁸⁾ These effects are significant for various measures of text comprehension.⁽⁴⁹⁾

Three issues with respect to the construction of evidence maps need to be researched. First, evidence maps could be improved by using a quantitative approach to the weighting of evidence. For instance, an interesting approach developed by Swaen and van Amelsvoort⁽⁵⁰⁾ is using the Bradford Hill criteria in a quantitative manner, thus providing an estimate of the probability of the causality of an association. However, those advanced procedures have still to be checked against their possible side effects on communicating with nonexperts. Second, evidence maps summarizing the evidence across various endpoints—that is, giving an overall picture of the hazardousness of a substance—would be a useful extension of the current evidence map format. Third, even when evidence maps are a step in the right direction with regard to evidence reporting, it remains open whether nonexperts are able to use the full range of information provided in the maps. A recent qualitative study,⁽⁵¹⁾ testing various plain language formats of Cochrane reviews, documented that lay people have difficulties to grasp some basic information, such as the distinction between review and a single study, the concept of quality of evidence and the concept of uncertainty as well as its expression in study results. Consequently, more has to be done to improve lay people's skills for the critical appraisal of risk information. Moreover, it has to be taken into account that evidence maps do not provide a “plain language summary” addressed to people with only limited risk literacy. Evidence maps provide complex information in a more structured way, but still require some elaborated knowledge of risk assessment methodology. Therefore, the intended consumers of evidence maps are not primarily lay people or the general public but stakeholders, public health decisionmakers, and the educated public.