Spatial determinants of the mandibular curve of Spee in modern and archaic Homo

1 Introduction

The curve of Spee (COS) is a bilateral alignment of the canine through distal molar cusps in the mesiodistal plane (Figure 1) (Spee, 1890) and is found in some mammals including modern humans and fossil hominins (Braun & Schmidt, 1956; Hitchcock, 1983; Mohl, 1988; Ferrario, Sforza, Miani, Colombo, & Tartaglia, 1992; Farella, Michelotti, Van Eijden, & Martina, 2002). Among modern humans, the COS is not present in the deciduous dentition, but develops with the eruption of the mandibular permanent incisors and first molar (Marshall et al., 2008). Following the eruption of the permanent dentition, the degree of concavity is relatively invariant throughout adulthood (Osborn, 1987).

Despite the presence of a COS across several mammalian taxa, it is unclear whether the presence/absence of this feature is tied to functional variation in the masticatory apparatus or whether the COS is linked to larger dentognathic spatial dynamics. Two primary hypotheses have been proposed that underscore the potential functional dynamics influencing COS development. The first hypothesis suggests that the COS is part of a larger complex of features including incisor overbite, slope of the articular eminence, and height of the molar cusps necessary for maintaining balanced occlusal function1 in modern humans (Ash & Wheeler, 1984). However, presence of a COS in non-human mammals without incisor overbite and a steep articular eminence indicates that the curve's primary function is likely unrelated to balancing occlusion (Osborn, 1993). An alternative model proposes that increased concavity of the COS maximizes work potential during the breakdown of a food bolus by directing the bite force down the long axis of the curve (Baragar & Osborn, 1987). Osborn (1993) found that the tilt of the superficial masseter muscle and the orientation of the posterior molar teeth were highly correlated among primates and mammals, supporting the hypothesis that a COS maximizes superficial masseter work potential during biting.

In contrast to the potential functional influences, morphological variation in the degree of curvature of the COS may be a secondary consequence of larger patterns of craniodental morphology. Indeed, across fossil and extant non-human primates there is a correlation between the curvature of the COS and alveolar prognathism (i.e., lower facial prognathism restricted to the maxillary and mandibular alveolus) such that relatively orthognathic taxa (e.g., Hadropithecus and Alouatta) have a more pronounced COS when compared with prognathic taxa (Tattersall, 1973; Osborn, 1993). Highly prognathic papionins commonly exhibit a convex or “reversed” COS (Jablonski, 1993). As such, the presence of a COS may be tied to the spatial dynamics of the dentognathic region such that reduced prognathism results in less space for the erupting dentition and as a result, the COS is a spatial compromise between the length of the dental arch and the length of the mandibular corpus.

In a similar manner, differences in COS morphology across genus Homo may be tied to larger patterns of facial prognathism. In modern humans, for example, Begg and Kesling (1977) noted that more prognathic Australian Aborigines had markedly reduced concavity of the COS when compared with relatively orthognathic European populations. Nevertheless, to date there have only been a limited number of studies that have directly assessed the relationship between the degree of COS curvature and alveolar prognathism in geographically variant modern human populations. Moreover, there is a general lack of consensus regarding the significance of the relationship between lower facial prognathism and the COS. Merz, Isaacson, Germane, and Rubenstein (1991) documented that African-Americans, who are characterized by greater alveolar prognathism, also had a shallower, albeit non-significantly different, COS when compared with European-Americans. Regarding the relationship between the COS and more general aspects of lower facial morphology, Farella et al. (2002) found that the COS was significantly related to the distal positioning of the dental arch with respect to the mandibular condyle, location of the mandible relative to the anterior cranial base, and facial height. However, Cheon et al. (2008), examining lateral cephalograms from Korean orthodontics patients, found that craniofacial morphology had only a minor influence on the concavity of the COS, although the curve was relatively deeper in individuals with posteriorly positioned mandibles.

The degree to which COS variation in fossil hominins can be explained as a function of variation in facial prognathism is also unclear. Indeed, some fossil hominins have been described as having a COS, including specimens attributed to Paranthropus boisei (KNM-ER 729) and early Homo (Stw 53, SK 45, and SK 847) (Leakey, Mungai, & Walker, 1972; Leakey & Wood, 1974; White & Johanson, 1982). A particularly deep curve has been noted in the Homo floresiensis LB1 specimen (Brown et al., 2004). In addition to the presence of a COS in some fossil hominins, these taxa have a greater range of alveolar prognathism than modern populations (Spoor, Leakey, & Leakey, 2005a). Among the features distinguishing anatomically modern Homo sapiens (AMHS) is a significant decrease in prognathism (i.e., orthognathism) compared with other late Pleistocene fossil hominin taxa such as Homo neanderthalensis (Neandertals) (Lieberman, McBratney, & Krovitz, 2002; Trinkaus, 2003; Lieberman, Krovitz, & McBratney-Owen, 2004). If the degree of curvature in the COS is influenced by patterns of facial prognathism, then the concavity of the COS is expected to be greater in AMHS compared with Archaic Homo. To date there have been no studies examining the potential interaction between prognathism and the COS across archaic and modern genus Homo. As such, our goal is not to assess competing functional and non-functional hypotheses, but rather to establish larger patterns of morphological covariation. These patterns can be used to subsequently test causal mechanisms that influence COS evolution and development.

The goal of the present study is to examine patterns of covariation between alveolar prognathism and the COS. Using a large sample of European- and African-derived subjects, we first test the hypothesis that maxillary and mandibular alveolar prognathism is significantly negatively correlated with concavity of the COS in extant modern humans. Individuals with greater alveolar prognathism are predicted to have a flatter COS, whereas orthognathic individuals are predicted to exhibit greater curvature. To determine whether the patterns established in the extant sample can be extended to fossil Homo, we also examine differences in alveolar prognathism and concavity of the COS in Neandertals and fossil AMHS. Due to greater levels of absolute alveolar prognathism, we predict that Neandertals will exhibit a less concave COS relative to both fossil and recent AMHS mandibles.

2 Materials and methods

To examine the relationship between alveolar prognathism and the COS, we utilized a sample of recent and fossil Homo specimens that exhibit considerable variation in alveolar prognathism. Our modern human sample consisted of n = 123 European-American (116 male and 7 female) and n = 166 African-American (130 male and 36 female) individuals with associated crania and mandibles (for a total of n = 289) from the Hamann–Todd Collection at the Cleveland Museum of Natural History. Previous studies have found no differences in curvature between males and females (Ferrario et al., 1992; Farella et al., 2002; Cheon et al., 2008; Marshall et al., 2008); so despite their small sample, females were retained in the sample to extend the range of variation across possible sexual dimorphism. The Hamann–Todd Collection was selected for its large sample sizes and documented ancestry and sex.

Genetic admixture and common environmental factors potentially reduce the morphological differences in African-American and European-American samples when compared with African and European founding populations. Nevertheless, previous researchers have documented that the African- and European-American samples from the Hamann-Todd Collection and other widely used collections (e.g., the Terry Collection at the Smithsonian Institution) consistently show significant differences in alveolar projection comparable to their source populations (Jantz & Ousley, 1993; Gill, 1998; Weinberg, Putz, Mooney, & Siegel, 2005; Wescott & Jantz, 2005).

We selected adult individuals with no obvious craniofacial pathology or damage to regions of interest. Further, given that the COS does not vary between the left and right sides of the mandible (Ferrario et al., 1992; Ferrario, Sforza, Poggio, Serrao, & Colombo, 1999; Xu, Suzuki, Muronoi, & Ooya, 2004; Marshall et al., 2008), we were able to include subjects with canine to M₃ dentition present on only one side. Only individuals with Class I occlusion, where the mesiobuccal cusp of the M¹ aligned with the buccal groove of the M₁, were included in order to focus on variation associated with alveolar prognathism and normal molar relationships. A previous study noted that the COS is more pronounced in individuals with Class II occlusion, where the buccal groove of M₁ is distal to the M¹ mesiobuccal cusp (Andrews, 1972).

Our fossil hominin sample of Neandertals and AMHS was composed of n = 10 research quality casts housed in the Paleoanthropology Laboratories at Washington University in St. Louis and The University of Iowa (Table 1). In order to maximize sample sizes for both taxa, only mandibular specimens were included in the analysis. Fossil hominin mandibles from these taxa were included in the analysis if the dentition, symphyseal morphology, and landmark areas of the ascending ramus were sufficiently preserved and unaltered by actual or suspected pathology.2

We collected a series of unilateral skeletal and dental coordinate landmarks on the cranium and associated mandible (Table 2, Figure 2) using a MicroScribe 3DX digitizer (Immersion Corp., San Jose, CA). To assess shape variation in the COS, we digitized a series of occlusal landmarks from the highest canine to M₃ cusps. This approach differs from previous assessments of COS morphology that often quantify the concavity of the COS simply as the perpendicular distance between the deepest cusp tip and a chord across the occlusal row (Merz et al., 1991; Farella et al., 2002). In contrast, the coordinate landmark-based approach used here allows us to capture, with greater detail, the contours of the entire post-canine occlusal surface.

Since we collected landmarks on the occlusal surface of the dentition, it is possible that dental wear could affect patterns of variation in the COS. While there is evidence from archaeological samples to suggest that the degree of COS concavity is unrelated to dental wear (Sengupta, Whittaker, Barber, Rogers, & Musgrave, 1999), we nevertheless examined the relationship between dental wear and the COS in our sample to ensure that our results were not affected by attrition. Dental wear in the recent human sample was scored using the Scott (1979) system that rates the degree of wear on individual cusps by the amount of dentin exposure. Wear scores for all of the cusps included in the COS measurement were summed to calculate a total wear score for each individual and standardized by age at death. Wear scores were standardized because the European-American sample had significantly higher ages at death and wear scores than the African-American sample (p < .001). Dental wear was positively correlated with age at death (r² = .245, p < .001). We used Procrustes ANOVA on occlusal landmarks to test for a significant interaction between shape variation and age-standardized wear scores. The influence of dental wear on the COS was not tested in the fossil hominin sample because of unknown ages at death of these specimens, the potential for different wear patterns resulting from thinner enamel in Neanderthals (Olejniczak et al., 2008), and the complicating factor of unknown dietary differences.

To determine whether the curvature of the COS is associated with facial prognathism we first assessed variation across our recent human sample. Coordinate landmarks were first superimposed using generalized Procrustes analysis (GPA) and then subjected to a series of multivariate analyses. First, using principal components analysis (PCA), we examined the axes of morphological variation across the combined European-American and African-American sample. In particular, we examined whether greater subnasal alveolar prognathism was associated with a flatter COS, while a deeper curve was associated with subnasal orthognathism. Additionally, we examined morphological differences between the European- and African-American samples using discriminant function analysis (DFA) with leave-one-out cross validation (Bookstein, 1997).

In order to statistically examine covariation between the COS and alveolar prognathism, we employed a test of morphological integration using a two-block partial least squares analysis (PLS). Within the single landmark configuration, we defined two blocks of landmarks. The first block was comprised of occlusal landmarks used to assess variation in the COS, and the second block consisted of all non-occlusal landmarks used to quantify variation in craniomandibular skeletal shape (Table 2). A GPA was used to scale landmarks in each dataset, after which the scaled landmarks were divided into blocks. Within the PLS analysis, the RV coefficient measured the strength of association between the two landmark blocks (Klingenberg, 2009). Statistical significance of the correlations between blocks was tested using permutation tests (n = 999 random landmark permutations).

All specimens were digitized and scored for dental wear by MFL, and intraobserver errors for landmarks and dental wear were calculated using three skulls measured on three separate occasions. Euclidean distances were calculated between landmarks for each skull and scoring occasion. Intraobserver errors for dental wear scoring and landmark distances were not significant (p < .05). All statistical analyses were run in R (R Core Team, 2014).

3 Results

We tested the hypothesis that maxillary and mandibular alveolar prognathism was negatively correlated with concavity of the COS using craniomandibular/occlusal and mandibular/occlusal landmarks from the modern sample. Variation in the COS and alveolar prognathism was captured on PC1 in both the craniomandibular/occlusal and mandibular/occlusal landmark sets (Table 3). Along with variation in the COS, PC1 also captured variation in the mediolateral placement of the molars relative to the more mesial dentition. The low percentage of total variation captured on PC1 likely reflects the high number of occlusal landmarks. Along PC1 in both landmark sets, there are significant differences between the African-American and European-American groups (p < .001) (Figure 3). On average, African-Americans had greater alveolar prognathism, and a more anteriorly positioned dental arch relative to the subnasal maxillary and symphyseal regions. This was further associated with a flatter COS, captured on lower PC1 scores in the modern craniomandibular/occlusal landmarks, and higher PC1 scores in the mandibular/occlusal landmarks. In contrast, European-Americans had greater orthognathism and a more concave COS corresponding with higher PC1 scores of the modern craniomandibular/occlusal landmarks and lower PC1 scores in the mandibular/occlusal landmarks. The association between dental arcade length and PC1 was tested using the Euclidean distance between infradentale and a transverse posterior to the third molars. In both sets of landmarks, longer dental arcades were correlated with lower scores on PC1, largely composed of the African-American sample (craniomandibular/occlusal: p < .001, r² = .12; mandibular/occlusal: p < .001, r² = .10). Ln centroid size was positively correlated with PC1 in both datasets such that individuals with larger mandibles had higher PC1 scores (greater alveolar prognathism and a flatter COS) (craniomandibular/occlusal: p < .001, r² = .07; mandibular/occlusal: p < .001, r² = .08). In these landmark sets, the percentage of individuals correctly classified by ancestry based on variation in COS and alveolar prognathism morphology was tested using DFAs for each dataset's procrustes-aligned shape variables. In the modern craniomandibular/occlusal landmark set, the DFA correctly classified 80.28% of African-Americans and European-Americans. A DFA for the modern mandibular/occlusal landmark set also correctly classified 80.28% of individuals.

The statistical covariation between the COS and alveolar prognathism was evaluated using a PLS analysis. For the modern craniomandibular/occlusal landmark set, the occlusal surface comprised one block, and the other craniomandibular Procrustes-aligned landmarks composed the respective second block. Blocks one and two were positively correlated with an RV coefficient of 0.91 (p = .001) (Figure 4). Low scores along blocks one and two capture a face with greater alveolar prognathism with a flat to concave COS. High scores on both blocks describe an orthognathic face with a concave COS. This result indicates that variation in the COS and alveolar prognathism blocks are highly significantly covariant. For the mandibular/occlusal landmarks, COS landmarks and Procrustes aligned mandibular landmarks formed the two respective blocks. Again, blocks one and two were positively correlated, with a correlation coefficient of 0.96 (p = .001) (Figure 5). Low scores on blocks one and two describe a mandible with alveolar prognathism and a flat COS; high scores reflect orthognathic mandibles and a concave COS. Similar to the analyses that also included cranial points, variation in concavity of the COS and of mandibular prognathism was highly correlated.

We tested whether the patterns of covariation between alveolar prognathism and concavity of the COS can be extended to fossil Homo using a fossil and modern mandibular/occlusal landmark set. In this landmark set, individuals with higher PC1 scores had a flatter COS, greater alveolar prognathism, an anteriorly positioned dental arch, and a less projecting mentum osseum (Figure 6). Neandertals have the highest PC1 values in this dataset (average = 0.07), followed by the African-American sample (average = 0.015), fossil AMHS (average = 0.014), and the European-American sample (average = −0.03) (Figure 6). Ln centroid size was positively correlated with PC1 such that individuals with larger mandibles had higher PC1 scores (greater alveolar prognathism and a flatter COS) (p < .001, r² = .14). African-Americans, European-Americans, and Neanderthals each were significantly different in their Ln centroid sizes (p < .001); however, fossil AMHS were not significantly different from the other groups (p < .31).

4 Discussion and conclusions

Across anthropoids and within Homo sapiens, variation in prognathism has been correlated with a suite of craniofacial features including supraorbital projection, presence/absence of a canine fossa, and projection of the chin (e.g., Howells, 1976; Smith & Ranyard, 1980; Vinyard & Smith, 2001; Fiscella & Smith, 2006; Maddux & Franciscus, 2009; Scott et al., 2010; Marshall et al., 2011; Holton et al., 2015). Our results indicate that dentognathic variation, specifically in the magnitude of COS curvature, may be added to the suite of morphologies influenced by facial prognathism.

The results of our analyses indicate that the COS covaries with maxillary and mandibular alveolar prognathism across African-Americans and European-Americans. Although only explaining a small portion of the total craniodental variation captured in our landmarks, this covariation is evident from the results of our PCA and morphological patterns from the PLS analysis. The PLS analysis indicated a high degree of covariation between the COS and other craniofacial regions. On average, the African-American sample, characterized by greater alveolar prognathism, exhibited a flatter COS, whereas the more orthognathic European-American sample exhibited a greater degree of COS curvature. Morphological differences between these groups are also related to allometry in that PC1 was correlated with the Ln centroid size. This suggests that differences in shape are related to differences in size. African-American populations generally have larger skulls, with greater alveolar prognathism, and longer dental arcades. Collectively, these results suggest that the COS is part of a larger suite of population-specific features (Howells, 1989; Hennessy & Stringer, 2002; Viðarsdóttir, O'Higgins, & Stringer, 2002; Weinberg et al., 2005; Rosas, Bastir, Alarcón, & Kuroe, 2008) that distinguish between European- and African-derived groups.

While our results suggest a significant relationship between craniomandibular form and the magnitude of COS curvature, there is little consensus on this relationship in previous studies. Farella et al. (2002) suggest that the COS exhibited a greater degree of curvature in subjects with a more anteriorly positioned lower jaw relative to the mandibular condyle. However, positioning of the condyle varies allometrically and is more posterior in smaller individuals (Nicholson & Harvati, 2006). In our analyses, PC1 in both the craniomandibular/occlusal and mandibular/occlusal landmarks sets did not capture variance in the horizontal position of the mandibular condyle relative to COS curvature. This difference may relate to the removal of size in our analyses. Cheon et al. (2008) documented a significant relationship between a greater COS and posterior mandibular position in Korean subjects; however, they found that craniomandibular form was poorly correlated with the curvature of the occlusal surface.

In the only other study to examine the relationship between the COS and craniomandibular form in European- and African-American subjects, Merz et al. (1991) found that African-Americans, who are characterized by greater alveolar prognathism, also had a shallower COS, albeit not significantly different when compared with European-Americans. The discrepancy in COS concavity pattern between Merz et al. (1991) and our study is likely due, at least in part, to differences in the quantification of the COS. The former study measured the COS as the perpendicular distance between the P₄ cusp tip and a chord across the occlusal row. In contrast to their univariate measure, we were able to take into account the entire contour of the molar and premolar occlusal surfaces using a multivariate coordinate landmark approach. This likely gave us better resolution to identify meaningful differences between our samples. Indeed, previous studies have found that the deepest point along the COS can vary from M₁ to P₄ (Koyama, 1979; Garcia, 1985; Shannon & Nanda, 2004). As such, a univariate measure like that used by Merz et al. (1991) may be too overly simplistic to account for the full range of variation along the occlusal surface.

While our study tested patterns of covariation between the COS and facial prognathism, it is likely that this morphological relationship relates to developmental and functional integration between the maxillary and mandibular dental arches. During development, variation in anterior projection of the maxillary dental arch affects the position of the lower dental arch via occlusal interlocking (You, Fishman, Rosenblum, & Subtelny, 2001; Marshall et al., 2011). That is, due to intercuspation of the maxillary and mandibular molars, the dental arches form an integrated dentoalveolar complex that varies independently from the basal and ramal components of the mandible. As such, individuals with reduced facial prognathism are likely to have a smaller skull, and a shorter and more posteriorly positioned mandibular dental arch relative to the mandibular condyle. This in turn positions the third molar more distally relative to the lingual tuberosity and mandibular ramus, and during eruption this would produce greater curvature from the distal molars. However, this relationship requires additional testing using ontogenetic series.

We note that this same integrative occlusal locking dynamic has been used to explain the relationship between chin projection and facial prognathism both within and between modern human populations (Scott et al., 2010; Marshall et al., 2011; Holton et al., 2015). As is evident from the patterns of occlusal and mandibular covariation in the present study, decreased dentoalveolar prognathism (and associated increase in COS concavity) is further associated with greater prominence of the bony chin. Thus, the integration between the upper and lower alveolar processes, coupled with the relative modularity between the lower alveolar process and other mandibular components, has a significant influence on population-specific patterns of skeletal and occlusal morphological variation in the mandible.

The relationship between facial prognathism and the COS was further evident when examining mandibular variation across the recent and fossil human samples. The results of this analysis were consistent with the pattern documented in the analysis of modern human craniomandibular shape, with the fossils extending the range of morphological variation. As with the analysis of craniomandibular shape in the recent human sample, there was a morphological distinction between the European- and African-American samples, with the African-Americans exhibiting increased mandibular alveolar prognathism and a reduced COS when compared with the European-Americans. The fossil AMHS sample spans much of the range of variation in recent humans, especially when compared with the distribution of PC1 scores for the African-American sample. Shape variation in both the extant and fossil samples was related to allometry such that larger individuals had greater prognathism and a flatter COS.

In spite of the relatively small AMHS sample, these fossils are arrayed temporally along PC1. The Tabun 23 mandible, which was found at the extreme end of the range of variation for alveolar prognathism and COS concavity in all fossils, including Neandertals, is derived from level C where dates range from 100–120 ka (Grün & Stringer, 1991, 2000) to 170–180 ka (Mercier et al., 1995). Skhul 5 and Qafzeh 9, both of which had positive scores along the PC1 axis, date to approximately 100 ka (McDermott, Grün, Stringer, & Hawkesworth, 1993). In contrast, the Zhoukoudian 3 and Ohalo 2 specimens, which are found on the other end of the fossil range of morphological variation, date much later, to around 34–10 ka and 22.5-23.5 ka, respectively, and have greater orthognathism and a more concave COS (Hedges et al., 1992; Nadel, Carmi, & Segal, 1995; Harvati, 2009; Snir et al., 2015). This suggests that the degree of curvature of the COS is associated with a well-established temporally arrayed decrease in facial prognathism from archaic to modern humans (Howells, 1989; Hennessy & Stringer, 2002; Lieberman et al., 2002, 2004; Stringer, 2002; Viðarsdóttir et al., 2002; Trinkaus, 2003; Spoor, Leakey, & Leakey, 2005b; Weinberg et al., 2005; Rosas et al., 2008).

In contrast to the fossil AMHS sample, the Neandertals were clustered toward the extreme positive end of PC1. As such, Neandertals and some of the fossil AMHS sample (Tabun 2, Qafzeh 9, and Skhul 5) were characterized by greater alveolar prognathism and a flatter COS. A flat occlusal surface is perhaps not surprising given that Neandertals (as well as Skuhl 5 and Tabun 2) exhibit a characteristic retromolar space. As with the COS, the presence/absence of a retromolar space is a function of the spatial interaction between the length of the dental arch and mandibular ramus relative to mandibular length (Franciscus & Trinkaus, 1995). Thus, if the presence of a COS is due to spatial constraints between the position and overall size of the lower dental arch and surrounding elements of the mandible (as our recent human results suggest) then the presence of a retromolar space alleviates these constraints. This spatial dynamic may further explain variation in the COS curvature across a wider range of hominin variation in mandibular morphologies (Leakey et al., 1972; Leakey & Wood, 1974; White & Johanson, 1982; Brown et al., 2004), although this remains to be tested.

While we did not test Baragar and Osborn's (1987) hypothesis that the COS functions to orient the masseter muscle over the mesial molars in order to maximize the muscle's work potential, our results indicate that variation in COS concavity may be more parsimoniously explained as a consequence of the spatial interactions between the dental arches and surrounding mandibular structures. Moreover, in vivo analysis of masticatory function in living humans suggests that muscle orientation may be less important in bite force production than other aspects of craniofacial morphology (e.g., Van Spronsen, Weijs, van Ginkel, & Prahl-Andersen, 1996). As such, it is possible that there is a close correspondence between the COS and muscle orientation across a wide range of morphological variation.

There is a wide range of variation in the COS (i.e., flat to highly curved) both across recent human populations and archaic human comparisons. While we have only examined a narrow range of population and taxonomic variation in this study, given that the relationship between the magnitude of the COS and alveolar prognathism is manifest across our recent African- and European-American, fossil AMHS, and Neandertal samples, it is likely that this pattern would extend to other recent human populations and other archaic Homo taxa. It is likely that additional variables not examined in the present study contribute to the magnitude of COS curvature such as tooth size, variation in cusp dimensions, variation in eruption timing, or soft tissue dimensions. However, our results highlight the influence of craniodental spatial dimensions, particularly alveolar prognathism, on variation in the COS.