1 Introduction

A classical view of facial displays holds that they express the emotional states of the sender, with the communicative aspect arising as a by-product deriving from the receiver experience (Darwin 1872; Buck 1994). Although the matching between facial expressions and emotions has been experimentally highlighted by several studies (Tomkins 1962; 1963; Levenson, Ekman, and Friesen 1990; Liu et al. 2017), focussing attention exclusively on the emotional aspect of facial displays introduces some methodological issues that can reduce the replicability of results (Waller, Whitehouse, and Micheletta 2017; Waller, Julle-Daniere, and Micheletta 2020). One main issue is that facial displays are often defined on the basis of the supposed underlying emotion (Waller, Whitehouse, and Micheletta 2017). An elucidating example is the Silent Bared-Teeth display (SBT), which has a negative emotional valence (e.g., submission, subordination) in most anthropoid species, while in some species like Sulawesi macaques (including Tonkean macaques, Macaca tonkeana), bonobos (Pan paniscus) and humans (Homo sapiens), it is mainly used to communicate affiliative intentions with a positive valence (van Hoof 1967; Thierry et al. 1989; Preuschoft 1992; Maestripieri 1997; Beisner and McCowan 2014; Vlaeyen et al. 2022). Because of such functional bivalence, SBT is often referenced with terms that indicate opposite affective states like fear (fear grimace) and pleasure (smile) depending on the species (Preuschoft and van Hooff 1995). Therefore, the use of emotions to reference facial expressions may affect the way researchers perceive them, potentially leading to confusing descriptions and ambiguity in the study of facial communication (Waller et al. 2008).

This issue can be solved by putting the spotlight on the morphology of facial expressions before exploring their functional nature (Waller, Julle-Daniere, and Micheletta 2020). The morphological approach to the study of facial expressions allows detection of possible homologies because it focuses on the physical substrate producing the display (i.e., facial musculature), which is continuous across species and indicative of a common origin (van Hooff 1967; Preuschoft and van Hooff 1995; Waller, Whitehouse, and Micheletta 2008, 2017, 2020).

The Facial Action Coding System (FACS) is a valuable quantitative tool to describe facial expressions in humans (Human FACS, Ekman & Friesen 1978) and other species (e.g., MaqFACS, Parr et al. 2010). Via FACS it is possible to associate an unequivocal code to a specific morphological change in the face (AU). Therefore, FACS helps not only to quantify the complexity of a single facial expression objectively (e.g., number of AUs present in a given display, Waller, Caeiro, and Davila-Ross 2015), but also the diversity and numerosity of the facial displays of certain species thus favouring objective comparisons (Parr et al. 2007; Rincon et al. 2023). Yet, determining the exact number of facial displays is not as easy as it sounds, because of the graded nature of facial expressions (Thierry et al. 1989; Clark et al. 2020; Clark et al. 2022).

In this study, we investigated the facial displays of Tonkean macaques (Macaca tonkeana), a species showing a high communicative complexity related to its highly tolerant social style (Thierry 2004, 2007; Freeberg, Dunbar, and Ord 2012; Rincon et al. 2023). We focused on “bared-teeth faces” (BTFs) as a model category of facial expressions. According to the literature, BTFs include silent bared-teeth, half-open mouth, open-mouth bared-teeth, and open-mouth bared-teeth scream (van Hooff 1967; Thierry, Iwaniuk, and Pellis 1989, 2000; Pellis et al. 2011; Scopa and Palagi 2016; Table S1). However, previous categorization of BTFs was based mainly on their context or their combination with a vocalization and/or postures, and not on morphological facial output. In fact, these facial expressions resemble each other, especially regarding mouth and lip configuration thus introducing a certain level of ambiguity (Thierry et al. 1989; Clark et al. 2020).

Following the literature (van Hooff 1967; Thierry, Iwaniuk, and Pellis 1989, 2000; Pellis et al. 2011; Scopa and Palagi 2016), we hypothesized that there are two structurally different displays of BTF in Tonkean macaques: the bared-teeth (BT), where the lips and mouth corners are retracted exposing teeth, and sometimes the gums, and the open-mouth bared-teeth (OMBT), where a wide opening of the jaw is added to the bared-teeth configuration. In this case, we expected that BT and OMBT would differ in terms of which AUs they include (Prediction 1).

If the two facial configurations convey different meanings, we predicted that they would be expressed under different contexts, with BT more frequent during peaceful interactions than in other contexts (Prediction 2a) and OMBT relatively more frequent during interactions with competitive elements such as playful and agonistic contacts, as the literature suggests (Thierry, Iwaniuk, and Pellis 1989, 2000, Pellis et al. 2011; Scopa and Palagi 2016; Clark et al. 2020) (Prediction 2b).

FACS also facilitates the study of flexibility of a facial display, including how external factors can affect display complexity such as the stereotypy and specificity of the recruited AUs. For example, in crested macaques (M. nigra), a species closely related to Tonkean macaques, the application of MaqFACS (Parr et al. 2010) revealed that the SBT comprises four morphological variants related to the social context. Interestingly, SBTs displayed during submission, playful or sexual interactions were characterized by the presence of a specific AU in each context (Clark et al. 2020). SBTs during affiliative contacts were characterized by higher overall variability (Clark et al. 2020). Under particularly unpredictable situations such as during play fighting, the risk of misunderstanding can be high, and clearer signals should have been positively selected to reduce such risk (risk reduction hypothesis; Clark et al. 2022). By contrast, peaceful contexts may allow a higher degree of freedom in the performance of facial expressions. Therefore, we hypothesized an effect of context on the level of stereotypy (Margoliash, Staicer, and Inoue 1991; Maglieri et al. 2022, 2024) and specificity (Mielke et al. 2022) of the facial expressions in Tonkean macaques. Since we predicted that OMBTs would be used mainly in managing competitive interactions, we expected OMBTs to show a higher degree of stereotypy (Prediction 3a) and specificity (Prediction 3b) than BTs.

2 Methods

3 Results

3.1 Two Different Bared-Teeth Displays (Prediction 1 Supported)

The a priori classification of BT and OMBT was confirmed by the analyses. The first cluster identified by the k-means included 88.5% of OMBTs, while the second cluster included all BTs and 11.5% of OMBTs (Figure 1). The results of the exact Fisher test and the power test are reported in Table 2. All subsequent analyses of BT and OMBT are based on the outcome of this k-means unsupervised cluster analysis.

3.2 The Emission of Different BTFs is Affected by the Context (Prediction 2a Supported and 2b Not Supported)

The full model differed from the control model (χ² = 1670.94, df = 4, p < 0.001, N_cases = 1216). Context had a significant effect on the type of BTF (Table 1A). The probability of BT emission was higher during peaceful than playful (post-hoc Tukey pairwise t-ratio_{peaceful versus playful} = −11.269, df = inf, p < 0.001) and agonistic contexts (t-ratio_{peaceful versus agonistic} = −7.294, df = inf, p < 0.001). The probability of BT and OMBT did not differ between playful and agonistic contexts (t-ratio_{playful versus agonistic} = 0.397, df = inf, p = 0.917) (Table 1; Figure 2).

Table 1. Estimated parameters (Coeff), Standard Error (SE), and results of the Likelihood Ratio Tests (χ²) of the models. Significant p values are shown in bold.

A - Prediction 2 (GLMM) – The emission of different bared-teeth displays is affected by the immediate social context (partially supported) – Response Variable: BT/OMBT
Fixed effects	Coeff	SE	χ2	df	p
Intercept	−1.738	0.204	n/a	n/a	n/a
Tested variables
Sex	−0.327	0.249	1.723	1	0.189
Age	0.535	0.946	0.320	1	0.572
Context			141.711	2	< 0.001
Context[PLAY]	1.970	0.174
Context[AGONISTIC]	1.866	0.255
Control variable
Screaming	1.326	0.305	18.933	1	< 0.001
Ncases = 1216, NID = 63, Variance for the random factor: ID = 0.440 ± 0.663 SD

B - Prediction 3a (LMM) – OMBTs show a higher degree of stereotypy than BTs, especially in agonistic and playful context (supported) - Response Variable: LD values
Fixed effects	Coeff	SE	χ2	df	p
Intercept	−1.606	0.499	n/a	n/a	n/a
Context			675.32	2	< 0.001
Context[PLAY]	−0.015	0.015
Context[AGONISTIC]	−0.468	0.030
Facial expression	−0.626	0.024	4093.31	1	< 0.001
Context*Facial expression			151.19	2	< 0.001
Context[PLAY]*Facial expression	−0.606	0.027
Context[AGONISTIC]*Facial expression	0.355	0.040
Ncases = 118500, NID1 = 58, NID2 = 58, Variance for the random factors: ID1 = 0.051 ± 0.226 SD; ID2 = 0.091 ± 0.301 SD

C - Prediction 3b (LMM) – OMBTs show a higher degree of specificity than BTs, especially in agonistic and playful context (not supported) - Response Variable: Specificity values
Fixed Effects	Coeff	SE	χ2	df	p
Intercept	0.445	0.031	n/a	n/a	n/a
Context			31.782	2	< 0.001
Context[PLAY]	−0.065	0.032
Context[AGONISTIC]	−0.041	0.032
Facial expression	0.0125	0.029	60.042	1	< 0.001
Context*Facial expression			16.592	2	< 0.001
Context[PLAY]*Facial expression	−0.104	0.045
Context[AGONISTIC]*Facial expression	0.080	0.041
Ncases = 858, NCOMBINATION SIZE = 7, Variance for the random factor: COMBINATION SIZE = 0.003 ± 0.050 SD; ID2 = 0.091 ± 0.301 SD

• Abbreviations: BT, bared teeth; df, degree of freedom; LD, Levenshtein distance; n/a, not applicable; OMBT, open mouth bared teeth.

3.3 OMBTs are More Stereotyped Than BTs Especially Under Potentially Risky Situations (Prediction 3a Supported)

The full model of LD differed significantly from the null model (χ² = 8473.9, df = 5, p < 0.001, N_cases = 118500). Facial expression, context and their interaction were significant predictors of the facial movement (Table 1B). Overall, BTs were less stereotyped than OMBTs in each context (peaceful, t-ratio = 25.647, df = 118491, p < 0.001; play, t-ratio = 59.720, df = 118,491, p < 0.001; agonistic, t-ratio = 8.459, df = 118491, p < 0.001; Figure 3).

BTs in peaceful contexts were less stereotyped than in playful and agonistic contexts (t-ratio_{peaceful versus playful} = 22.013, df = 118491, p < 0.0001; t-ratio_{peaceful versus agonistic} = 15.697, df = 118491, p < 0.0001). BTs in a playful context were less stereotyped than those in an agonistic context (t-ratio_{playful versus agonistic} = 8.134, df = 118491, p < 0.0001; Figure 3).

OMBTs in peaceful context were also less stereotyped than those in playful and agonistic contexts (t-ratio_{peaceful versus playful} = 10.679, df = 118491, p < 0.0001; t-ratio_{peaceful versus agonistic} = 3.724, df = 118491, p = 0.0027). OMBTs in an agonistic context were less stereotyped than in a playful context (t-ratio_{playful versus agonistic} = −8.613, df = 118491, p < 0.0001; Figure 3).

3.4 OMBT Specificity is Higher Than BT Specificity Especially Under Risky Situations (Prediction 3b Not Supported)

The networks representing the probability of each context to be present when an AU was active (specificity) for BT and OMBT are visualized in Figure 4a,b respectively. Among BTs, the AU combinations with high specificity (> 0.8) were found only in the agonistic context (25%; Figure 5). In playful and peaceful context, no combinations had high specificity values (Figure 5). For OMBT, the AU combinations with high specificity (> 0.8) were found in the agonistic and peaceful context (34% for both). In the playful context, no combinations scored high specificity values (Figure 5).

The full model differed significantly from the null model (χ² = 105.14; df = 5, p < 0.001, N_cases = 858). The variables context, facial expression and their interaction affected the AU combination specificity (Table 1, Figure 6). Within BTs, specificity did not differ across contexts (t-ratio_{peaceful versus playful} = 2.306, df = 850, p = 0.193; t-ratio_{peaceful versus agonistic} = 1.708, df = 850, p = 0.527; t-ratio_{agonistic versus playful} = 0.592; df = 850; p = 0.992; Table 1). Within OMBTs, specificity of the playful context was significantly lower than peaceful (t-ratio_{peaceful versus playful} = 5.823, df = 850, p < 0.001) and agonistic context (t-ratio_{agonistic versus playful} = 5.947, df = 850, p < 0.001). OMBT specificity did not differ between peaceful and agonistic contexts (t-ratio_{peaceful versus agonistic} = −0.135; df = 850; p = 1.000).

BT scored a lower specificity than OMBT under peaceful (t-ratio_{peacefulBT versus peacefulOMBT} = −5.216, df = 850, p < 0.001) and agonistic context (t-ratio_{agonisticBT versus agonisticOMBT} = −7.576, df = 850, p < 0.001). The specificity of BT and OMBT did not differ in the playful context (t-ratio_{playfulBT versus playfulOMBT} = −0.478, df = 850, p = 0.997) (Table 1, Figure 6).

4 Discussion

Using Tonkean macaque facial expressions as a study system, we revealed a standardized distinction between two similar expressions (bared-teeth and open-mouth bared-teeth) illustrating an iterative approach that can be applied to other facial expressions. Initially, BT and OMBT were distinguished a priori based on existing literature (Table S1). Subsequently, using MaqFACS (Parr et al. 2010) we coded specific AUs for each facial expression and subjected the AUs to a rigorous clustering procedure using an unsupervised k-means algorithm. The resulting clusters aligned with the a priori classification of the two bared-teeth facial expressions in Tonkean macaques (Prediction 1 supported, Figure 1), which differed mainly in the degree of mouth-opening (BT, AU25 or AU25 + 26; OMBT, AU25 + 27; Table 2).

The morphological separation of BT and OMBT did not completely reflect their context specificity. Neither expression was exclusively performed during a specific context, although BT was mostly displayed in peaceful contexts, and was probably associated with a relaxed affective state (Prediction 2a supported). We noted that BT and OMBT are occasionally used as a submissive signal by one partner, but are more often used for reconciliation (Thierry et al. 1989). In Tonkean macaques, affiliative facial displays (e.g., lipsmacking, BT, OMBT) that end a conflict are often accompanied by conciliatory contacts. Contrary to expectations, BT and OMBT performance did not differ between playful and agonistic contexts, indicating some flexibility in the use of these facial expressions during risky situations (Prediction 2b not supported). The absence of a preferential emission of one of the two facial expressions during an agonistic event could relate to the sender's role as aggressor or victim. Because we did not consider whether the facial expression was emitted by the attacker or the victim, it is possible that the coarse grouping under the “agonistic” context may have concealed differences in signaling. We therefore recommend that future studies focus on the direction of the agonistic behavior to clarify the use of these signals in agonistic contexts.

When Tonkean macaques are playing, OMBT substitutes for the relaxed open mouth (ROM) display, a metacommunicative playful signal that is widespread among primates (van Hoof 1967; Thierry et al. 1989; Preuschoft and van Hooff 1995; Pellis et al. 2011; Palagi, Norscia, and Spada 2014). In several primates, ROM involves wide opening of the jaw, but without retraction of the mouth corners (Preuschoft 2000). The substitution of ROM with OMBT in Sulawesi macaques may be linked to the motivational shift of the BT, from a submissive towards a peaceful meaning. In fact, evolutionarily speaking, the OMBT may reflect a merger of the BT and the play face (Preuschoft 2000), resulting in an expression that resembles the full play face well described in hominoids (Palagi 2008; Davila-Ross, Owren, and Zimmermann 2009; Waller and Cherry 2012; Waller, Caeiro, and Davila-Ross 2015; Bresciani, Cordoni, and Palagi 2022) and geladas (Theropithecus gelada; Palagi and Mancini 2011). The presence of BT and OMBT during play reflects the dual nature of this behavior, which can be both cooperative and competitive at the same time (Pellis and Pellis 1998; Reinhart et al. 2010; Palagi et al. 2016; Burghardt et al. 2024). Accordingly, we could hypothesize that as the degree of competition during play increases (i.e., more asymmetrical play session), the proportion of OMBTs compared to BTs should increase as well. However, another explanation might involve the arousal state of the interacting animals. For example, we could hypothesize that a higher arousal state in the sender (i.e., more intense play session) would lead to a higher proportion of OMBTs (Waller, Caeiro, and Davila-Ross 2015). This idea is worthy of further investigation.

Our results indicate that BT and OMBT differ not only in the AUs they contain but also in their degree of stereotypy and the specificity of AU combinations. In particular, BT showed lower stereotypy and specificity than OMBT (Prediction 3a supported, Figure 3 and 5). These results support the risk reduction hypothesis as BTs were displayed mostly during peaceful contexts (Figure 2). In terms of BT stereotypy in each context, we found that the peaceful context predicted more variability than the agonistic and playful contexts (Figure 3). During affiliative interactions there is a weaker selective pressure to develop highly stereotyped facial expression that unequivocally manifest senders’ intentions, because misunderstandings are likely to have negligible consequences. This same pattern seems to hold for OMBT, which showed the highest Levenshtein Distance (LD) values during peaceful interactions (Figure 3). Surprisingly, OMBT scored the lowest LD values during play (Figure 3), a context requiring clear signals of motivation by both players to prolong the session (Mancini, Ferrari, and Palagi 2013; Scopa and Palagi 2016). Although comparisons with other primates are not possible at present, some comparisons can be made with social carnivores. A similar result was found in dogs whose playful facial expression (Maglieri et al. 2022), the ROM, is emitted in a highly stereotyped way showing LD values overlapping with those recorded for OMBT in Tonkean macaques. In dogs, the stereotyped ROM serves a metacommunicative function in maintaining the play session balanced between the players (Maglieri et al. 2022). A similar function may apply to Tonkean macaques, a species that engages in play at every age (Ciani et al. 2012).

Given the weak intensity and the high bidirectionality characterizing the aggressive interactions of Tonkean macaques (Ciani et al. 2012; Thierry et al. 2004), it is not surprising that OMBTs emitted during agonistic encounters can be more plastic and less stereotyped, probably because of the necessity of social modulation of the signal in tolerant species (Rincon et al. 2023). This result goes in tandem with the high level of AU specificity recorded for OMBT in both peaceful and agonistic contexts compared to the play context (Prediction 3b not supported, Figure 6). This finding could relate to the presence of specific AUs (not present during play) that can be recruited for unambiguous signaling for peaceful or agonistic purposes (Figure 5). Specifically, the network analysis (NetFACS, Mielke et al. 2022) revealed the presence of some context specific AUs (Figure 4). The AD19 (tongue showing) is present in the 100% of the OMBTs during peaceful contacts. The AU41 (glabella lowerer) occurs in the 100% of the OMBTs (Figure 4b) and in the 95% of BTs during agonistic encounters (Figure 4a). Accordingly, AU41 predicts the agonistic context in Tonkean macaques. A similar result has been found by Clark et al. (2022) in Macaca nigra, another tolerant macaque. The authors found that some AUs and Action Descriptors (ADs) distinguish social contexts such as play (jaw stretching, AU27), sex (jaw wobble, AD184), and submission (teeth chattering, AD182). In wolves, a socially complex carnivore, Maglieri et al. (2024) demonstrated that, despite sharing several AUs, the full-ROM (analogous to the full play face in great apes) and the high Threatening-Face differed in two highly context specific AUs. The brow lowerer (AU104) and gum exposure (GUM) were never present during play fighting and recruited only during real fighting. This also aligns with findings from studies of both humans (Waller et al. 2008; Shariff and Tracy 2011) and mice (Defensor et al. 2012), indicating that the narrowing of the eyelid opening is associated with increased aggression and may function as an indicator of dominance or an imminent threat.

Overall, our results show the multifaceted facial expression system of Tonkean macaques, with a single facial expression that can acquire multiple meanings depending on the social context. The use of BTs and OMBTs seems to be extremely flexible, reflecting the social complexity of the species (Rebout et al. 2020, 2021; Waller et al. 2022). Concurrent analysis of stereotypy and specificity would be valuable, as they would allow direct comparisons and could reveal the role of variability in signal emission as related to different degrees of social complexity and tolerance.

Our study has some limitations that need to be addressed in future work. First, vocalizations should be included in the analysis, because unimodal communication is generally rare in animals, whose signals often comprise elements from different sensory channels (Partan and Marler 1999; Micheletta et al. 2013). Other communicative elements can be associated with facial expressions (e.g., postures, gestures), determining different meanings. Finally, the identity of the receiver and its attentional state are crucial factors to be considered (message-meaning analysis), since they can make the difference in the signal production, in shaping its designed features and clarifying precise functions.

Author Contributions

Conceptualization: Elisabetta Palagi. Data curation: Federico Fantoni, Veronica Maglieri, Chiara Scopa, Virginia Pallante. Investigation: Federico Fantoni, Veronica Maglieri, Nicolò Giusti, Chiara Scopa, Virginia Pallante. Formal analysis: Federico Fantoni, Veronica Maglieri, Antonio Lorenzo Loprete, Elisabetta Palagi. Methodology: Federico Fantoni, Veronica Maglieri, Antonio Lorenzo Loprete. Supervision: Elisabetta Palagi. Validation: Elisabetta Palagi: Visualization: Federico Fantoni, Veronica Maglieri, Antonio Lorenzo Loprete. Writing–original draft: Federico Fantoni, Veronica Maglieri, Elisabetta Palagi. Writing–review and editing of the first version of the manuscript: Federico Fantoni, Veronica Maglieri, Nicolò Giusti, Chiara Scopa, Virginia Pallante, Antonio Lorenzo Loprete, Elisabetta Palagi.

Conflicts of Interest

No competing interest to declare.