Addressing the National Academy of Sciences’ Challenge: A Method for Statistical Pattern Comparison of Striated Tool Marks
Nicholas D. K. Petraco Ph.D.
Department of Sciences, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
Faculty of Chemistry, Graduate Center, City University of New York, 365 5th Avenue, New York, NY 10016.
Search for more papers by this authorPeter Shenkin Ph.D.
Department of Mathematics and Computer Science, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
Search for more papers by this authorJacqueline Speir M.S.
Department of Chemical and Physical Sciences, Forensic Science Program, Cedar Crest College, 100 College Drive, Allentown, PA 18104.
Search for more papers by this authorPeter Diaczuk B.S.
Department of Sciences, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
PEDICO Research, RR2 Box 62, Waymart, PA 18472.
Search for more papers by this authorPeter A. Pizzola Ph.D.
Special Investigations Unit, 421 East 26th Street, New York, NY 10016.
Search for more papers by this authorCarol Gambino M.S.
Department of Sciences, Borough of Manhattan Community College, City University of New York, 199 Chambers Street, New York, NY 10007.
Search for more papers by this authorNicholas Petraco M.S.
Department of Sciences, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
Petraco Forensic Consulting, 73 Ireland Place, Amityville, NY 11701.
New York City Police Department Crime Laboratory, 150-14 Jamaica Avenue, Jamaica, NY 11432.
Search for more papers by this authorNicholas D. K. Petraco Ph.D.
Department of Sciences, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
Faculty of Chemistry, Graduate Center, City University of New York, 365 5th Avenue, New York, NY 10016.
Search for more papers by this authorPeter Shenkin Ph.D.
Department of Mathematics and Computer Science, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
Search for more papers by this authorJacqueline Speir M.S.
Department of Chemical and Physical Sciences, Forensic Science Program, Cedar Crest College, 100 College Drive, Allentown, PA 18104.
Search for more papers by this authorPeter Diaczuk B.S.
Department of Sciences, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
PEDICO Research, RR2 Box 62, Waymart, PA 18472.
Search for more papers by this authorPeter A. Pizzola Ph.D.
Special Investigations Unit, 421 East 26th Street, New York, NY 10016.
Search for more papers by this authorCarol Gambino M.S.
Department of Sciences, Borough of Manhattan Community College, City University of New York, 199 Chambers Street, New York, NY 10007.
Search for more papers by this authorNicholas Petraco M.S.
Department of Sciences, John Jay College of Criminal Justice, City University of New York, 899 10th Avenue, New York, NY 10019.
Petraco Forensic Consulting, 73 Ireland Place, Amityville, NY 11701.
New York City Police Department Crime Laboratory, 150-14 Jamaica Avenue, Jamaica, NY 11432.
Search for more papers by this authorAbstract
Abstract: In February 2009, the National Academy of Sciences published a report entitled “Strengthening Forensic Science in the United States: A Path Forward.” The report notes research studies must be performed to “…understand the reliability and repeatability…” of comparison methods commonly used in forensic science. Numerical classification methods have the ability to assign objective quantitative measures to these words. In this study, reproducible sets of ideal striation patterns were made with nine slotted screwdrivers, encoded into high-dimensional feature vectors, and subjected to multiple statistical pattern recognition methods. The specific methods employed were chosen because of their long peer-reviewed track records, widespread successful use for both industry and academic applications, rely on few assumptions on the data’s underlying distribution, can be accompanied by standard confidence levels, and are falsifiable. For PLS-DA, correct classification rates of 97% or higher were achieved by retaining only eight dimensions (8D) of data. PCA-SVM required even fewer dimensions, 4D, for the same level of performance. Finally, for the first time in forensic science, it is shown how to use conformal prediction theory to compute identifications of striation patterns at a given level of confidence.
References
- 1 Edwards HT, Gatsonis C, Berger MA, Cecil JS, Bonner-Denton M, Fierro M, et al. Strengthening forensic science in the United States: a path forward, 1st edn. Washington, DC: National Academies Press, 2009.
- 2 Moran B. A report on the AFTE theory of identification and range of conclusions for tool mark identification and resulting approaches to casework. AFTE J 2002; 34(2): 227–35.
- 3 Grzybowski RA, Murdock JE. Firearm and toolmark identification meeting the Daubert challenge. AFTE J 1998; 30(1): 3–14.
- 4 Bohan TL, Heels EJ. The new scientific evidence “Standard” and the standards of several states. J Forensic Sci 1995; 40(6): 1030–44.
- 5 Thompson E, Wyant R. Knife identification project (kip). Ann Firearm Toolmark Exam J (AFTE) 2003; 35(4): 366–70.
- 6 Tomasetti KA. Analysis of the essential aspects of striated tool mark examinations and the methods of examination. Ann Firearm Toolmark Exam J (AFTE) 2002; 34(3): 289–301.
- 7 Collins JM. The language of toolmarks. Ann Firearm Toolmark Exam J (AFTE) 1998; 30(1): 366–70.
- 8 AFTE. Theory of identification as it relates to toolmarks. AFTE J 1998; 30(1): 86–8.
- 9 Duda RO, Hart PE, Stork DG. Pattern classification, 2nd edn. New York, NY: Wiley, 2001.
- 10 Theodoridis S, Koutroumbas K. Pattern recognition, 3rd edn. San Diego, CA: Academic Press, 2006.
- 11
Shawe-Taylor J,
Cristianini N.
Kernel methods for pattern analysis. London, UK: Cambridge University Press, 2004.
10.1017/CBO9780511809682 Google Scholar
- 12 Leon FP. Automated comparison of firearm bullets. Forensic Sci Int 2006; 156: 40–50.
- 13 Gardner GY. Computer identification of bullets. AFTE J 1979; 11(2): 26–33.
- 14 Taroni F, Champod C, Margot P. Statistics: a future in tool marks comparison? AFTE J 1996; 28(4): 222–32.
- 15 Champod C, Baldwin D, Taroni F, Buckleton JS. Firearm and tool marks identification: the Bayesian approach. AFTE J 2003; 35(3): 307–16.
- 16 Buckleton J, Nichols R, Triggs C, Wevers G. An exploratory Bayesian model for firearm and toolmark interpretation. AFTE J 2005; 37(4): 352–61.
- 17 Biedermann A, Taroni F. A probabilistic approach to the joint evaluation of firearm evidence and gunshot residues. Forensic Sci Int 2006; 163: 18–33.
- 18 Katterwe H, Baldwin D, vanBeest M, Besler C, Birkett J, Girod A, et al. Conclusion scale for shoeprint and toolmark examinations. J Forensic Identification 2006; 56(2): 255–81.
- 19 Neel M, Wells M. A comprehensive statistical analysis of striated tool mark examinations. Part 1: comparing known matches to known non-matches. AFTE J 2007; 39(3): 176–98.
- 20 Biasotti AA. A statistical study of the individual characteristics of fired bullets. J Forensic Sci 1959; 4(1): 34–50.
- 21 Miller J, Neel M. Criteria for identification of toolmarks part III: supporting the conclusion. AFTE J 2004; 37(1): 7–38.
- 22 Brackett J. A study of idealized striated marks and their comparison using models. J Forensic Sci Soc 1970; 10(1): 27–56.
- 23 Blackwell R, Framan E. Automated firearms identification systems AFIDS: Phase I. AFTE J 1980; 12(4): 11–37.
- 24 Deinet W. Studies of models of striated marks generated by random processes. J Forensic Sci 1981; 26(1): 35–50.
- 25 Stone R. How unique are impressed toolmarks? AFTE J 2003; 35(4): 376–83.
- 26 Collins E. How unique are impressed toolmarks? An empirical study of twenty worn hammer faces. AFTE J 2005; 37(4): 252–95.
- 27 Bunch S. Consecutive matching striation criteria: a general critique. J Forensic Sci 2000; 45(5): 955–62.
- 28
Banno A.
Estimation of bullet striation similarity using neural networks.
J Forensic Sci
2004; 49(3): 1–5.
10.1520/JFS2002361 Google Scholar
- 29
Bachrach B.
Development of a 3D-based automated firearms evidence comparison system.
J Forensic Sci
2002; 47(6): 1–12.
10.1520/JFS15557J Google Scholar
- 30 Bachrach B. A statistical validation of the individuality of guns using 3D images of bullets. National Institute of Justice, Grant Report: 97-LB-VX-0008. Washington, DC: National Institute of Justice, 2006.
- 31 Geradts Z, Keijzer J. TRAX for toolmarks. AFTE J 1996; 28(3): 183–90.
- 32 Geradts Z, Keijzer J, Keereweer I. A new approach to automatic comparison of striation marks. J Forensic Sci 1994; 39(4): 974–80.
- 33 Geradts Z, Bijhold J, Hermsen R, Murtaugh F. Image matching algorithms for breech face marks and firing pins in a database of spent cartridges of firearms. Forensic Sci Int 2001; 119: 97–106.
- 34 Senin N, Groppetti R, Garofano L, Fratini P, Pierni M. Three-dimensional surface topography acquisition and analysis for firearm identification. J Forensic Sci 2006; 51(2): 282–95.
- 35 Faden D, Kidd J, Craft J, Chumbley LS, Morris M, Genalo L, et al. Statistical confirmation of empirical observations concerning tool mark striae. AFTE J 2007; 39(3): 205–14.
- 36 DeKinder J, Tulleners F, Thiebaut H. Reference ballistic imaging database performance. Forensic Sci Int 2004; 140: 207–15.
- 37 DeKinder J, Bonfanti M. Automated comparisons of bullet striations based on 3D topography. Forensic Sci Int 1999; 101: 85–93.
- 38 Song J, Vorburger T, Renegar T, Rhee H, Zheng A, Ma L, et al. Correlation of topography measurements of NIST SRM 2460 standard bullets by four techniques. Measurement Sci and Tech 2006; 17(3): 500–3.
- 39 Booker JL. Examination of the badly damaged bullet. J Forensic Sci Soc 1980; 20: 153–62.
- 40 Uchiyama T. A criterion for land mark identification. AFTE J 1988; 20(3): 236–51.
- 41 Uchiyama T, Igarashi N, Nagai M. The frequency of occurrence of individual characteristics of firearms on fired bullets and cartridge cases. AFTE J 1988; 20(4): 376–90.
- 42 Nichols RG. Firearm and toolmark identification criteria: a review of the literature. J Forensic Sci 1997; 42(3): 466–74.
- 43 Nichols RG. Firearm and toolmark identification criteria: a review of the literature-part 2. J Forensic Sci 2003; 48(2): 318–27.
- 44 Nichols RG. Defending the scientific foundations of the firearms and tool mark discipline: responding to recent challenges. J Forensic Sci 2007; 53(3): 586–94.
- 45 Silbert RW. Drugfire revolutionizing forensic firearms identification and providing a foundation for a national firearms identification network. USA Crime Lab Digest 1994; 21: 63–8.
- 46 Baldur R, inventor. Forensic Technology Wai Inc., assignee. Fired cartridge examination method and imaging apparatus. US patent 5654801 (file date: 1/3/1995; issue date: 8/5/97).
- 47 Baldur R, Barrett MR, inventors. Forensic Technology Wai Inc., assignee. Computer automated bullet analysis apparatus. US patent 5390108 (file date: 5/24/1991; issue date: 2/14/95).
- 48 DL Cork, JE Rolph, ES Meieran, CV Petrie, eds. Ballistic imaging. Washington, DC: National Academies Press, 2008.
- 49 Geradts Z. Content-based information retrieval from forensic image databases. Utrecht, The Netherlands: University of Utrecht, 2002.
- 50 Geradts Z, Keijzer J, Keereweer I. Automatic comparison of striation marks and automatic classification of shoe marks. Proceedings SPIE 1995, http://spie.org/x648.html?product_id=218471 (accessed March 8, 2012).
- 51 DeKinder J, Prevot P, Pirlot M, Nys B. Surface topology of bullet striations: an innovating technique. AFTE J 1998; 30: 294–9.
- 52 Bachrach B, Jain A, Jung S, Koons RD. A statistical validation of the individuality and repeatability of striated tool marks: screwdrivers and tongue and groove pliers. J Forensic Sci 2010; 55(1): 348–57.
- 53 Chu W, Song J, Vorburger T, Yen J, Ballou S, Bachrach B. Pilot study of automated bullet signature identification based on topography measurements and correlations. J Forensic Sci 2010; 55(2): 341–7.
- 54 Banno A, Masuda T, Ikeuchi K. Three dimensional visualization and comparison of impressions on fired bullets. Forensic Sci Int 2004; 140: 233–40.
- 55 Sakarya U, Leloglu UM, Tunali E. Three-dimensional surface reconstruction for cartridge cases using photometric stereo. Forensic Sci Int 2008; 175: 209–17.
- 56 Evans JPO, Smith CL, Robinson M. Validation of line scan imaging technique for imaging cylindrical forensic ballistics specimens. AFTE J 2004; 36(4): 275–80.
- 57 Zographos A, Robinson M, Evans J, Smith CL. Ballistics identification using line-scan imaging techniques. Proceedings of the Institute of Electrical and Electronics Engineers 31st Annual International Carnahan Conference on Security Technology; 1997 Oct 15–17; Canberra, Australia. IEEE 1997; 82–7, doi 10.1109/CCST.1997.626243.
- 58 Smith C. Fireball: a forensic ballistic imaging system. Proceedings of the Institute of Electrical and Electronics Engineers 31st Annual International Carnahan Conference on Security Technology; 1997 Oct 15–17; Canberra, Australia. IEEE 1997; 64–70 doi 10.1109/CCST.1997.626240.
- 59 vanBeest M, Zaal D, Hardy H. The forensic application of the Mikrocad 3D imaging system. Inform Bull Shoeprint/Toolmark Examiners 2000; 6: 65–72.
- 60 Howitt D, Tulleners F, Cebra K, Chen S. A calculation of the theoretical significance of matched bullets. J Forensic Sci 2008; 53(4): 868–75.
- 61 Doble P, Sandercock M, DuPasquier E, Petocz P, Rouxa C, Dawson M. Classification of premium and regular gasoline by gas chromatography/mass spectrometry, principal component analysis and artificial neural networks. Forensic Sci Int 2003; 132(1): 26–39.
- 62 Sandercock PML, Pasquier ED. Chemical fingerprinting of unevaporated automotive gasoline samples. Forensic Sci Int 2003; 134(1): 1–10.
- 63 Sandercock PML, Pasquier ED. Chemical fingerprinting of gasoline 2. Comparison of unevaporated and evaporated automotive gasoline samples. Forensic Sci Int 2004; 140(1): 43–59.
- 64 Sandercock PML, Pasquier ED. Chemical fingerprinting of gasoline 3. Comparison of unevaporated automotive gasoline samples from Australia and New Zealand. Forensic Sci Int 2004; 140(1): 71–7.
- 65 Egan W, Galipo RC, Kochanowski BK, Morgan SL, Bartick EG, Miller ML, et al. Forensic discrimination of photocopy and printer toners. III. Multivariate statistics applied to scanning electron microscopy and pyrolysis gas chromatography/mass spectrometry. Anal Bioanal Chem 2003; 376: 1286–97.
- 66 Egan W, Morgan SL, Bartick EG, Merrill RA, Taylor HJ. Forensic discrimination of photocopy and printer toners. II. Discriminant analysis applied to infrared reflection absorption spectroscopy. Anal Bioanal Chem 2003; 376: 1279–85.
- 67 Morgan SL, Bartick EG. Discrimination of forensic analytical chemical data using multivariate statistics. In: RD Blackledge, editor. Forensic analysis on the cutting edge: new methods for trace evidence analysis. New York, NY: Wiley, 2007; 331–72.
- 68 Ratle F, Gagne C, Terrettaz-Zufferey AL, Kanevski M, Esseiva P, Ribaux O. Advanced clustering methods for mining chemical databases in forensic science. Chemometrics Intelli Lab Sys 2008; 90(2): 123–31.
- 69 Vapnik VN. Statistical learning theory. New York, NY: Wiley, 1998.
- 70 Wold S, Eriksson L, Trygg J, Kettaneh N. The pls method-partial least squares projections to latent structures-and its application in industrial rdp (research, development and production). Proceedings of COMPSTAT 2004, 16th Symposium of IASC; 2004 Aug 23–27; Prague, Czech Republic. The Hague, The Netherlands: International Association for Statistical Computing, 2004; 1–44.
- 71 Petraco N, Petraco NDK, Faber L, Pizzola PA. Preparation of tool mark standards with jewelry modeling waxes. J Forensic Sci 2009; 54(2): 353–8.
- 72 Rasband W. ImageJ [computer program]. National Institute of Health, http://rsb.info.nih.gov/ij/ (accessed April, 2010).
- 73 Wolfram Development. Mathematica [computer program], 7th edn. Champaign, IL: Wolfram Research, Inc., 2009.
- 74 Jolliffe IT. Principal component analysis, 2nd edn. New York, NY: Springer, 2004.
- 75
Vapnik VN.
The nature of statistical learning theory. New York, NY: Springer, 2000.
10.1007/978-1-4757-3264-1 Google Scholar
- 76 R Core Development Team. R: a language and environment for statistical computing [computer program], version 2.9.1. Vienna, Austria: R Foundation for Statistical Computing, 2009.
- 77 Bertsekas DP, Tsitsiklis JN. Introduction to probability, 1st edn. Belmont, MA: Athena Scientific, 2002.
- 78
Rencher AC.
Methods of multivariate analysis, 2nd edn. Hoboken, NJ: Wiley, 2002.
10.1002/0471271357 Google Scholar
- 79 Petraco NDK, Gil M, Pizzola PA, Kubic TA. Statistical discrimination of liquid gasoline samples from casework. J Forensic Sci 2008; 53(5): 1092–101.
- 80 Kuhn M. Building predictive models in R using the caret package. J Stat Softw 2008; 8(5): 1–26.
- 81
Cristianini N,
Shawe-Taylor J.
An introduction to support vector machines and other kernel-based learning methods. London, UK: Cambridge University Press, 2000.
10.1017/CBO9780511801389 Google Scholar
- 82
Karatzoglou A,
Smola A,
Hornik K,
Zeileis A.
Kernlab—an s4 package for kernel methods in R.
J Stat Softw
2004; 11(9): 1–20.
10.18637/jss.v011.i09 Google Scholar
- 83 Barker M, Rayenes W. Partial least squares for discrimination. J Chemometrics 2003; 17(6): 166–73.
- 84 Mevik BH, Wehrens R. The pls package: principal component and partial least squares regression in R. J Stat Softw 2007; 18(2): 1–24.
- 85 Smith CAB. Some examples of discrimination. Ann Eugenics 1947; 18: 272–83.
- 86 Efron B. Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 1983; 78(382): 316–31.
- 87
Efron B,
Tibshirani RJ.
An introduction to the bootstrap, 1st edn. Boca Raton, FL: Chapman & Hall/CRC, 1993.
10.1007/978-1-4899-4541-9 Google Scholar
- 88
Li M,
Vitanyi P.
An introduction to Kolmogorov complexity and its applications, 3rd edn. New York, NY: Springer, 2008.
10.1007/978-0-387-49820-1 Google Scholar
- 89 Vovk V, Gammerman A, Shafer G. Algorithmic learning in a random world, 1st edn. New York, NY: Springer, 2005.
- 90 Shafer G, Vovk V. A tutorial on conformal prediction. J Mach Learn Res 2008; 9: 371–421.
- 91 Dekking FM, Kraaikamp C, Lopuhaa HP, Meester LE. A modern introduction to probability and statistics: understanding how and why, 1st edn. New York, NY: Springer, 2005.
- 92 Lehmann EL, Romano JP. Testing statistical hypotheses, 3rd edn. New York, NY: Springer, 2005.
- 93 Rosner B. Fundamentals of biostatistics. Belmont, CA: Thomson, 2006.
- 94
Bishop CM.
Pattern recognition and machine learning, 1st edn. New York, NY: Springer, 2006.
10.1007/978-0-387-45528-0 Google Scholar
- 95 Gammerman A, Vovk V. Hedging predictions in machine learning. Comp J 2007; 50(7): 151–77.
- 96 Vanderlooy S, Maaten L, Sprinkhuizen-Kuyper I. Off-line learning with transductive confidence machines: an empirical evaluation. In: P Perner, editor. Proceedings of the 5th International Conference on Machine Learning and Data Mining in Pattern Recognition; 2007 July 18–20; Leipzig, Germany. Berlin, Germany: Springer-Verlag, 2007; 310–23.
