Evaluating firearm examiner conclusion variability using cartridge case reproductions

The forensic science pattern comparison areas, including fingerprints, footwear, and firearms, have been criticized for their subjective nature. While much research has attempted to move these disciplines to more objective methods, examiners are still coming to conclusions based on their own training and experience. To complement this subjectivity, black box studies are necessary to establish the accuracy of these feature-comparison methods. However, when cartridges are fired by a firearm to create cartridge case test sets there may be significant variability within the resulting impressions. This can result in different participants receiving test sets with varying levels of difficulty based on differences in impression quality. Therefore, comparison of accuracy between examiners is not straightforward. To compare accuracy between examiners, a method called double-casting was used to create plastic cartridge case reproductions. Double-casts of twenty-one test sets of master cartridge cases were created and mailed to firearm examiners. The double-casts ensured that all participants were comparing exhibits with the same level of detail. The examiners were tasked with determining if the unknown cartridge case in each set was fired by the same firearm as the three knowns. Automated comparisons were also used to compare the cartridge cases within each set. The results from this study showed that there are differences in examiner conclusions when examining the same evidence. Furthermore, it was shown that automated comparison metrics would benefit examiners as a quality control measure to correct any potential errors and strengthen conclusions.