Do emojis influence social interactions? Neural and behavioral responses to affective emojis in bargaining situations
Corresponding Author
Martin Weiß
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Correspondence
Martin Weiß, Julius-Maximilians-Universität Würzburg, Marcusstraße 9–11, 97070 Würzburg, Germany.
Email: [email protected]
Search for more papers by this authorJulian Gutzeit
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Search for more papers by this authorJohannes Rodrigues
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Search for more papers by this authorPatrick Mussel
Division Personality Psychology and Psychological Assessment, Freie Universität Berlin, Berlin, Germany
Search for more papers by this authorJohannes Hewig
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Search for more papers by this authorCorresponding Author
Martin Weiß
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Correspondence
Martin Weiß, Julius-Maximilians-Universität Würzburg, Marcusstraße 9–11, 97070 Würzburg, Germany.
Email: [email protected]
Search for more papers by this authorJulian Gutzeit
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Search for more papers by this authorJohannes Rodrigues
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Search for more papers by this authorPatrick Mussel
Division Personality Psychology and Psychological Assessment, Freie Universität Berlin, Berlin, Germany
Search for more papers by this authorJohannes Hewig
Differential Psychology, Personality Psychology, and Psychological Diagnostics, Julius-Maximilians-Universität Würzburg, Würzburg, Germany
Search for more papers by this authorFunding information
European Union “Individualisierung Digital” project (Fonds 823881) in the Europäischer Fonds für regionale Entwicklung (EFRE).
Abstract
Emojis are nowadays a common substitute for real facial expressions to integrate emotions in social interaction. In certain contexts, emojis possibly could also transport information beyond emotions, reflecting interindividual differences or social aspects. In this study, we investigated the influence of emojis as socioemotional feedback stimuli on behavior and neural responses in a social decision game. We modified the Ultimatum Game by including emotional feedback provided by the proposer as response to the decision of the participant as receiver. Therefore, we generated identities that differed in their feedback behavior to identify differences in the processing of emotional feedback in a positive (acceptance) versus negative (rejection) frame. Regarding offer sizes, we replicated the valence effect of feedback-related negativity for small offer sizes evoking more negative brain potentials compared to larger ones. Further, we found an effect of affective emojis on distinct ERPs: A face-detecting neural component (N170) was examined to be a part of the processing of emojis, which resulted in significantly more negative amplitudes in response to a sad-looking emoji compared to smiling and neutral ones. Furthermore, P3 amplitudes indicate transmission effects from the feedback emoticons to the neural processing of different offer sizes. In contrast to previous findings, P3 responses of our subjects did not depend on the offer size, but rather by which kind of partner they were made. Since some evaluative processes did not reveal any effects, emojis seem to be less effective than real facial expressions, which convey more information that is socially meaningful.
Supporting Information
Filename | Description |
---|---|
psyp13321-sup-0001-Supinfo.pdfPDF document, 185.9 KB |
Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.
REFERENCES
- Amodio, D. M. (2010). Coordinated roles of motivation and perception in the regulation of intergroup responses: Frontal cortical asymmetry effects on the P2 event-related potential and behavior. Journal of Cognitive Neuroscience, 22(11), 2609–2617. https://doi.org/10.1162/jocn.2009.21395
- Baker, T. E., Stockwell, T., Barnes, G., & Holroyd, C. B. (2011). Individual differences in substance dependence: At the intersection of brain, behaviour and cognition. Addiction Biology, 16(3), 458–466. https://doi.org/10.1111/j.1369-1600.2010.00243.x
- Batty, M., & Taylor, M. J. (2003). Early processing of the six basic facial emotional expressions. Brain Research Cognitive Brain Research, 17(3), 613–620. https://doi.org/10.1016/S0926-6410(03)00174-5
- Becker, M. P., Nitsch, A. M., Miltner, W. H., & Straube, T. (2014). A single-trial estimation of the feedback-related negativity and its relation to BOLD responses in a time-estimation task. Journal of Neuroscience, 34(8), 3005–3012. https://doi.org/10.1523/JNEUROSCI.3684-13.2014
- Blau, V. C., Maurer, U., Tottenham, N., & McCandliss, B. D. (2007). The face-specific N170 component is modulated by emotional facial expression. Behavioral and Brain Functions, 3(1), 7. https://doi.org/10.1186/1744-9081-3-7
- Boksem, M. A., & De Cremer, D. (2010). Fairness concerns predict medial frontal negativity amplitude in ultimatum bargaining. Society for Neuroscience, 5(1), 118–128. https://doi.org/10.1080/17470910903202666
10.1080/17470910903202666 Google Scholar
- Bublatzky, F., Gerdes, A. B., White, A. J., Riemer, M., & Alpers, G. W. (2014). Social and emotional relevance in face processing: Happy faces of future interaction partners enhance the late positive potential. Frontiers in Human Neuroscience, 8, 493. https://doi.org/10.3389/fnhum.2014.00493
- Calvo, M. G., & Beltran, D. (2014). Brain lateralization of holistic versus analytic processing of emotional facial expressions. NeuroImage, 92, 237–247. https://doi.org/10.1016/j.neuroimage.2014.01.048
- Camerer, C. (2003). Behavioral game theory: Experiments in strategic interaction. New York, NY: Russell Sage Foundation.
- Chaumon, M., Bishop, D. V., & Busch, N. A. (2015). A practical guide to the selection of independent components of the electroencephalogram for artifact correction. Journal of Neuroscience Methods, 250, 47–63. https://doi.org/10.1016/j.jneumeth.2015.02.025
- Churches, O., Nicholls, M., Thiessen, M., Kohler, M., & Keage, H. (2014). Emoticons in mind: An event-related potential study. Society for Neuroscience, 9(2), 196–202. https://doi.org/10.1080/17470919.2013.873737
10.1080/17470919.2013.873737 Google Scholar
- Cialdini, R. B., & Goldstein, N. J. (2004). Social influence: Compliance and conformity. Annual Review of Psychology, 55, 591–621. https://doi.org/10.1146/annurev.psych.55.090902.142015
- Comesana, M., Soares, A. P., Perea, M., Pineiro, A. P., Fraga, I., & Pinheiro, A. (2013). ERP correlates of masked affective priming with emoticons. Computers in Human Behavior, 29(3), 588–595. https://doi.org/10.1016/j.chb.2012.10.020
- Delorme, A., & Makeig, S. (2004). EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. Journal of Neuroscience Methods, 134(1), 9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
- Delorme, A., Sejnowski, T., & Makeig, S. (2007). Enhanced detection of artifacts in EEG data using higher-order statistics and independent component analysis. NeuroImage, 34(4), 1443–1449. https://doi.org/10.1016/j.neuroimage.2006.11.004
- Derks, D., Bos, A. E., & von Grumbkow, J. (2008). Emoticons in computer-mediated communication: Social motives and social context. Cyberpsychological Behavior, 11(1), 99–101. https://doi.org/10.1089/cpb.2007.9926
- Donchin, E., & Cohen, L. (1967). Averaged evoked potentials and intramodality selective attention. Electroencephalography & Clinical Neurophysiology, 22(6), 537–546. https://doi.org/10.1016/0013-4694(67)90061-2
- Eckel, C. C., & Wilson, R. K. (1998). Reciprocal fairness and social signalling: Experiments with limited reputations. Paper presented at the American Economic Association Meeting, New York, NY. Retrieved from https://pdfs.semanticscholar.org/5459/a24d5ce3cddca52807249ed77b1545558755.pdf
- Eimer, M. (2000). The face-specific N170 component reflects late stages in the structural encoding of faces. NeuroReport, 11(10), 2319–2324. https://doi.org/10.1097/00001756-200007140-00050
- Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392. https://doi.org/10.1037/0003-066X.48.4.384
- Ekman, P., Friesen, W. V., & Ellsworth, P. (2013). Emotion in the human face: Guidelines for research and an integration of findings. Amsterdam, The Netherlands: Elsevier.
- Folstein, J. R., & Van Petten, C. (2008). Influence of cognitive control and mismatch on the N2 component of the ERP: A review. Psychophysiology, 45(1), 152–170. https://doi.org/10.1111/j.1469-8986.2007.00602.x
- Foti, D., Weinberg, A., Dien, J., & Hajcak, G. (2011). Event-related potential activity in the basal ganglia differentiates rewards from nonrewards: Temporospatial principal components analysis and source localization of the feedback negativity. Human Brain Mapping, 32(12), 2207–2216. https://doi.org/10.1002/hbm.21182
- Grill-Spector, K., Henson, R., & Martin, A. (2006). Repetition and the brain: Neural models of stimulus-specific effects. Trends in Cognitive Science, 10(1), 14–23. https://doi.org/10.1016/j.tics.2005.11.006
- Grill-Spector, K., & Malach, R. (2001). fMR-adaptation: A tool for studying the functional properties of human cortical neurons. Acta Psychologica, 107(1–3), 293–321. https://doi.org/10.1016/S0001-6918(01)00019-1
- Güth, W., Schmittberger, R., & Schwarze, B. (1982). An experimental analysis of ultimatum bargaining. Journal of Economic Behavior & Organization, 3(4), 367–388. https://doi.org/10.1016/0167-2681(82)90011-7
- Hammerschmidt, W., Kulke, L., Broering, C., & Schacht, A. (2018). Money or smiles: Independent ERP effects of associated monetary reward and happy faces. PLoS ONE, 13(10), e0206142. https://doi.org/10.1371/journal.pone.0206142
- Hareli, S., & Rafaeli, A. (2008). Emotion cycles: On the social influence of emotion in organizations. Research in Organizational Behavior, 28, 35–59. https://doi.org/10.1016/j.riob.2008.04.007
- Hewig, J., Kretschmer, N., Trippe, R. H., Hecht, H., Coles, M. G., Holroyd, C. B., & Miltner, W. H. (2010). Hypersensitivity to reward in problem gamblers. Biological Psychiatry, 67(8), 781–783. https://doi.org/10.1016/j.biopsych.2009.11.009
- Hewig, J., Kretschmer, N., Trippe, R. H., Hecht, H., Coles, M. G., Holroyd, C. B., & Miltner, W. H. (2011). Why humans deviate from rational choice. Psychophysiology, 48(4), 507–514. https://doi.org/10.1111/j.1469-8986.2010.01081.x
- Hinojosa, J. A., Mercado, F., & Carretie, L. (2015). N170 sensitivity to facial expression: A meta-analysis. Neuroscience & Biobehavioral Reviews, 55, 498–509. https://doi.org/10.1016/j.neubiorev.2015.06.002
- Holroyd, C. B., & Coles, M. G. H. (2002). The neural basis of human error processing: Reinforcement learning, dopamine, and the error-related negativity. Psychological Review, 109(4), 679–709. https://doi.org/10.1037/0033-295X.109.4.679
- Holroyd, C. B., Nieuwenhuis, S., Yeung, N., & Cohen, J. D. (2003). Errors in reward prediction are reflected in the event-related brain potential. NeuroReport, 14(18), 2481–2484. https://doi.org/10.1097/01.wnr.0000099601.41403.a5
- Holroyd, C. B., Pakzad-Vaezi, K. L., & Krigolson, O. E. (2008). The feedback correct-related positivity: Sensitivity of the event-related brain potential to unexpected positive feedback. Psychophysiology, 45(5), 688–697. https://doi.org/10.1111/j.1469-8986.2008.00668.x
- Ito, T. A., Larsen, J. T., Smith, N. K., & Cacioppo, J. T. (1998). Negative information weighs more heavily on the brain: The negativity bias in evaluative categorizations. Journal of Personality & Social Psychology, 75(4), 887–900. https://doi.org/10.1037/0022-3514.75.4.887
- Itti, L., & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2(3), 194–203. https://doi.org/10.1038/35058500
- Johnson, R. (1988). The amplitude of the P300 component of the event-related potential: Review and synthesis. Advances in Psychophysiology, 3, 69–137.
- Johnson, R. Jr, & Donchin, E. (1985). Second thoughts: Multiple P300s elicited by a single stimulus. Psychophysiology, 22(2), 182–194. https://doi.org/10.1111/j.1469-8986.1985.tb01584.x
- Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(11), 4302–4311. https://doi.org/10.1523/JNEUROSCI.17-11-04302.1997
- Kreussel, L., Hewig, J., Kretschmer, N., Hecht, H., Coles, M. G., & Miltner, W. H. (2012). The influence of the magnitude, probability, and valence of potential wins and losses on the amplitude of the feedback negativity. Psychophysiology, 49(2), 207–219. https://doi.org/10.1111/j.1469-8986.2011.01291.x
- Levine, D. K. (1998). Modeling altruism and spitefulness in experiments. Review of Economic Dynamics, 1(3), 593–622. https://doi.org/10.1006/redy.1998.0023
- Luck, S. J., & Hillyard, S. A. (1994). Electrophysiological correlates of feature analysis during visual search. Psychophysiology, 31(3), 291–308. https://doi.org/10.1111/j.1469-8986.1994.tb02218.x
- Luo, Y., Wu, T., Broster, L. S., Feng, C., Zhang, D., Gu, R., & Luo, Y. J. (2014). The temporal course of the influence of anxiety on fairness considerations. Psychophysiology, 51(9), 834–842. https://doi.org/10.1111/psyp.12235
- Ma, Q., Hu, Y., Jiang, S., & Meng, L. (2015). The undermining effect of facial attractiveness on brain responses to fairness in the Ultimatum Game: An ERP study. Frontiers in Neuroscience, 9, 77. https://doi.org/10.3389/fnins.2015.00077
- Melo, C. D., Marsella, S., & Gratch, J. (2016). People do not feel guilty about exploiting machines. ACM Transactions on Computer-Human Interaction (TOCHI), 23(2), 8. https://doi.org/10.1145/2890495
- Miltner, W. H., Braun, C. H., & Coles, M. G. (1997). Event-related brain potentials following incorrect feedback in a time-estimation task: Evidence for a “generic” neural system for error detection. Journal of Cognitive Neuroscience, 9(6), 788–798. https://doi.org/10.1162/jocn.1997.9.6.788
- Mognon, A., Jovicich, J., Bruzzone, L., & Buiatti, M. (2011). ADJUST: An automatic EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology, 48(2), 229–240. https://doi.org/10.1111/j.1469-8986.2010.01061.x
- Muchnik, L., Aral, S., & Taylor, S. J. (2013). Social influence bias: A randomized experiment. Science, 341(6146), 647–651. https://doi.org/10.1126/science.1240466
- Mussel, P., Hewig, J., & Weiß, M. (2018). The reward-like nature of social cues that indicate successful altruistic punishment. Psychophysiology, 55(9), e13093. Advance online publication. https://doi.org/10.1111/psyp.13093
- Mussel, P., Hewig, J., Allen, J. J., Coles, M. G., & Miltner, W. (2014). Smiling faces, sometimes they don't tell the truth: Facial expression in the ultimatum game impacts decision making and event-related potentials. Psychophysiology, 51(4), 358–363. https://doi.org/10.1111/psyp.12184
- Nachson, I. (1995). On the modularity of face recognition: The riddle of domain specificity. Journal of Clinical and Experimental Neuropsychology, 17(2), 256–275. https://doi.org/10.1080/01688639508405122
- Nakamura, K., Kawashima, R., Ito, K., Sugiura, M., Kato, T., Nakamura, A., … Kojima, S. (1999). Activation of the right inferior frontal cortex during assessment of facial emotion. Journal of Neurophysiology, 82(3), 1610–1614. https://doi.org/10.1152/jn.1999.82.3.1610
- Navalpakkam, V., & Itti, L. (2005). Modeling the influence of task on attention. Vision Research, 45(2), 205–231. https://doi.org/10.1016/j.visres.2004.07.042
- Nieuwenhuis, S., Aston-Jones, G., & Cohen, J. D. (2005). Decision making, the P3, and the locus coeruleus-norepinephrine system. Psychological Bulletin, 131(4), 510–532. https://doi.org/10.1037/0033-2909.131.4.510
- Nolan, J. M., Schultz, P. W., Cialdini, R. B., Goldstein, N. J., & Griskevicius, V. (2008). Normative social influence is underdetected. Personality and Social Psychology Bulletin, 34(7), 913–923. https://doi.org/10.1177/0146167208316691
- Osinsky, R., Mussel, P., & Hewig, J. (2012). Feedback-related potentials are sensitive to sequential order of decision outcomes in a gambling task. Psychophysiology, 49(12), 1579–1589. https://doi.org/10.1111/j.1469-8986.2012.01473.x
- Osinsky, R., Mussel, P., Ohrlein, L., & Hewig, J. (2014). A neural signature of the creation of social evaluation. Social and Cognitive Affective Neuroscience, 9(6), 731–736. https://doi.org/10.1093/scan/nst051
- Peirce, J. W. (2007). PsychoPy—Psychophysics software in Python. Journal of Neuroscience Methods, 162(1–2), 8–13. https://doi.org/10.1016/j.jneumeth.2006.11.017
- Peterburs, J., Voegler, R., Liepelt, R., Schulze, A., Wilhelm, S., Ocklenburg, S., & Straube, T. (2017). Processing of fair and unfair offers in the ultimatum game under social observation. Scientific Reports, 7, 44062. https://doi.org/10.1038/srep44062
- Polezzi, D., Daum, I., Rubaltelli, E., Lotto, L., Civai, C., Sartori, G., & Rumiati, R. (2008). Mentalizing in economic decision-making. Behavioral Brain Research, 190(2), 218–223. https://doi.org/10.1016/j.bbr.2008.03.003
- Polich, J. (2007). Updating P300: An integrative theory of P3a and P3b. Clinical Neurophysiology, 118(10), 2128–2148. https://doi.org/10.1016/j.clinph.2007.04.019
- Potts, G. F., Martin, L. E., Burton, P., & Montague, P. R. (2006). When things are better or worse than expected: The medial frontal cortex and the allocation of processing resources. Journal of Cognitive Neuroscience, 18(7), 1112–1119. https://doi.org/10.1162/jocn.2006.18.7.1112
- Proudfit, G. H. (2015). The reward positivity: From basic research on reward to a biomarker for depression. Psychophysiology, 52(4), 449–459. https://doi.org/10.1111/psyp.12370
- Qu, C., Wang, Y., & Huang, Y. (2013). Social exclusion modulates fairness consideration in the ultimatum game: An ERP study. Frontiers in Human Neuroscience, 7, 505. https://doi.org/10.3389/fnhum.2013.00505
- Rhodes, G., Brake, S., & Atkinson, A. P. (1993). What's lost in inverted faces? Cognition, 47(1), 25–57. https://doi.org/10.1016/0010-0277(93)90061-Y
- Riepl, K., Mussel, P., Osinsky, R., & Hewig, J. (2016). Influences of state and trait affect on behavior, feedback-related negativity, and P3b in the Ultimatum Game. PLoS ONE, 11(1), e0146358. https://doi.org/10.1371/journal.pone.0146358
- Robertson, L. C., & Delis, D. C. (1986). ‘Part-whole’processing in unilateral brain-damaged patients: Dysfunction of hierarchical organization. Neuropsychologia, 24(3), 363–370. https://doi.org/10.1016/0028-3932(86)90021-7
- Rossion, B., Gauthier, I., Tarr, M. J., Despland, P., Bruyer, R., Linotte, S., & Crommelinck, M. (2000). The N170 occipito-temporal component is delayed and enhanced to inverted faces but not to inverted objects: An electrophysiological account of face-specific processes in the human brain. NeuroReport, 11(1), 69–72. https://doi.org/10.1097/00001756-200001170-00014
- San Martín, R., Manes, F., Hurtado, E., Isla, P., & Ibañez, A. (2010). Size and probability of rewards modulate the feedback error-related negativity associated with wins but not losses in a monetarily rewarded gambling task. NeuroImage, 51(3), 1194–1204. https://doi.org/10.1016/j.neuroimage.2010.03.031
- Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
- Von Neumann, J., & Morgenstern, O. (1944). Game theory and economic behavior. New York, NY: John Wiley and Sons.
- Winkler, I., Haufe, S., & Tangermann, M. (2011). Automatic classification of artifactual ICA-components for artifact removal in EEG signals. Behavioral and Brain Functions, 7(1), 30. https://doi.org/10.1186/1744-9081-7-30
- Wu, Y., Hu, J., van Dijk, E., Leliveld, M. C., & Zhou, X. (2012). Brain activity in fairness consideration during asset distribution: Does the initial ownership play a role? PLoS ONE, 7(6), e39627. https://doi.org/10.1371/journal.pone.0039627
- Yeung, N., Holroyd, C. B., & Cohen, J. D. (2005). ERP correlates of feedback and reward processing in the presence and absence of response choice. Cerebral Cortex, 15(5), 535–544. https://doi.org/10.1093/cercor/bhh153
- Yeung, N., & Sanfey, A. G. (2004). Independent coding of reward magnitude and valence in the human brain. Journal of Neuroscience, 24(28), 6258–6264. https://doi.org/10.1523/JNEUROSCI.4537-03.2004
- Yuasa, M., Saito, K., & Mukawa, N. (2011). Brain activity associated with graphic emoticons. The effect of abstract faces in communication over a computer network. Electrical Engineering in Japan, 177(3), 36–45. https://doi.org/10.1002/eej.21162