CRAG rocks

In this month's issue, Ortiz et al. report the results of a clinical trial to compare outcomes of dogs presenting to a primary care practice with haemorrhagic diarrhoea and treated with either co-amoxiclav or co-amoxiclav plus metronidazole. The study was a prospective randomised blinded trial for which the study design is the first to have been examined and agreed with the CRAG (Clinical Research Assessment and Guidance) panel (see Jeffery 2015). The authors’ conclusion is that it is not necessary to supplement co-amoxiclav with metronidazole when treating dogs presenting with haemorrhagic diarrhoea.

This study has raised a number of important issues, which have relevance for use of antibiotics and the practicalities of carrying out clinical trials in small animal primary care practice. In the discussion that follows it is important to bear in mind that data published in almost every published article require a reader's careful interpretation; it is always necessary to weigh up the strength of evidence they provide in answering a research question.

When interpreting the results of a clinical trial the most important aspect is whether the results can reliably guide clinical practice. This can be analysed by examining both “internal” and “external” validity. Internal validity refers to whether the trial was conducted in a logical, unbiased, internally consistent manner and which fulfilled its aims. On the whole, the Ortiz et al. study was conducted according to the pre-study protocol, although slightly fewer animals were recruited than was deemed appropriate by the study design. The power calculation indicated the need for 20 dogs in each group, whereas only 14 were assigned to the co-amoxiclav group. This means that the power of the study to detect the difference between the groups was reduced. The original study power of 80% means that a difference of at least 14 hours hospitalisation time between groups would be detected in 80% of studies carried out in this way, if such a difference were to truly exist. This means that there was already a 20% chance of not detecting the difference anyway and it can be calculated that the absence of six dogs from one arm of the trial would only have changed this to approximately 26% (http://clincalc.com/stats/Power.aspx). Therefore, it might be concluded that the study should still be reasonably reliable in detecting a difference of the stated magnitude if it truly existed. Examination of the data mean and spread (Ortiz et al. 2018, Fig. 2) suggests that it is improbable that there would be a useful treatment effect (defined as 14 hours difference in hospitalisation time) even if there had been full recruitment. So – in summary, although not perfect, there is reasonable internal validity.

As an aside about internal validity, the authors recalculated their sample size based on an interim analysis, which had not been discussed with the CRAG panel. Interim analysis is a complicated matter in clinical trials because it can represent an opportunity to detect a statistically important outcome at an early stage of the study that then might either suggest a need to terminate the study early or, alternatively, it might bias the assessors for the remainder of the study. Such interim analysis can therefore call into question the meaning of the final P value (because of repeated testing). Although it was not ideal for it to have been done, it would seem improbable that the interim analysis in this trial would have been able to bias the assessors, because they were kept blinded to treatment allocation of each individual throughout the trial.

How about external validity? For a study to be externally valid it means that if someone else – especially journal readers themselves – were to apply these treatment protocols to their own cases they would be likely to find the same results. This largely relies on comparability between the cases that were studied in this trial and those that are treated by others. The aim of achieving external validity is a key reason for carrying out this trial in a primary care practice – it is much more likely that the results will transfer to other similar practices than if the study had been carried out in a veterinary school or referral practice. In this trial the recruited cases were subject to little selection: dogs had to have diarrhoea of less than 3 days and for it to be bloody. This means that the authors have recruited a broad sample of these cases, which should imply good (general) external validity. However, because the sample is quite small it is possible that they could have inadvertently recruited a more selected cohort, although there is little evidence in support of this. Their listed exclusions are also those that clinicians working elsewhere might have knowledge of before treatment and so similar cases can be excluded from those that readers may wish to treat according to these trial results. However, as the authors quite rightly point out, some clinicians at the practice were sufficiently concerned for the safety of the dogs under their care that they did not wish to enter some severely affected cases into this study in which they might have only received one antibiotic. This may make the dogs that were included less severely affected than the average, meaning that one might need to be a little cautious in extrapolating the results of this study to include severely affected individuals. On the other hand, there is also other evidence suggesting the lack of need for antibiotics in similarly affected dogs (Unterer et al. 2011), although this previous study specifically excluded dogs with signs of sepsis.

The related question that might arise here is whether a dog presenting to a reader's clinic is in the same clinical “group.” The case definition is of “profuse bloody diarrhoea” as reported by the owners. As clinicians we all know that one owner's “profuse bleeding” can be another's “minor cut” and so there may be a problem in deciding whether a dog presented elsewhere would be similar enough to those included in this study to fairly extrapolate the findings. This aspect can be considered both a strength and a weakness of the study design. It is a strength because it means that a dog presented to a primary care practice with a reported clinical sign of “profuse bloody diarrhoea” should likely be similar to cases included in this study. On the other hand, the weakness is that we cannot be sure of that, and the small sample size implies that including even only a handful of less severe cases would represent a large proportion of the total study population, and so skew the results.

In terms of study design it is difficult to know how the case definition could be more tightly controlled. How could it be quantified? Perhaps by measuring the volume of diarrhoea and having a colour grading sheet etc? The problem with such (over-)refinement of a study is that it will inevitably reduce the number of dogs included (fewer owners will wish/be able to take part) and so then the cases become a sample from within a larger population and thus risk being non-representative of the “real” primary care practice population as a whole.

In all, the wide recruitment eligibility means that this trial represents an example of a “pragmatic trial,” in which a wide range of cases is potentially included. On the whole, pragmatic trials are less often carried out in veterinary medicine compared to human medicine but have a much higher likelihood of changing practice, because they are easier to generalise and therefore more widely relevant in treatment of cases treated elsewhere. In fact, many trials that could be carried out in primary care veterinary practice would likely need to be pragmatic in design because it is necessary to reduce costs and make follow-up straightforward and not dependent upon equipment that is expensive or time-consuming to use.

A major concern voiced by the reviewers about this report is that they considered that use of any antibiotic in such cases to be inappropriate and that publication of this report might be seen as evidence that antibiotics should be used for haemorrhagic gastroenteritis cases. This is an important point of view, with evidence in its support (Unterer et al. 2011), but the authors of this current study were concerned with addressing what actually happens currently in practice. Far from not using antibiotics at all for such cases, it appears that many primary care veterinarians would prefer to use two antibiotics – as evidenced by the reluctance of some clinicians in the practice to enter some dogs into the trial. Therefore, this study with its strong suggestion that use of metronidazole and co-amoxiclav together is inappropriate over-treatment supports the reviewers’ point and should provide reassurance about reduction of antibiotic use in practice.

So – what is the summary of the message that this study is sending? Really this is for the readers to decide for themselves, weighing up the questions outlined above. However, it would seem reasonable to suggest that this study overall (despite the expressed reservations) strongly suggests that it is unnecessary to give metronidazole in addition to co-amoxiclav to dogs with haemorrhagic gastroenteritis.