Deep Bidirectional LSTM for Emotion Detection through Mobile Sensor Analysis
D. Roja Ramani
Computer Science and Engineering, New Horizon College of Engineering, Bangalore, India
Search for more papers by this authorNaveen Chandra Gowda
School of Computer Science and Engineering, REVA University, Bengaluru, India
Search for more papers by this authorS. Sreejith
Department of Artificial Intelligence and Data Science, New Horizon College of Engineering, Bangalore, India
Search for more papers by this authorShrikant Tangade
Department of Mathematics, University of Padova, Padova, Italy
Search for more papers by this authorD. Roja Ramani
Computer Science and Engineering, New Horizon College of Engineering, Bangalore, India
Search for more papers by this authorNaveen Chandra Gowda
School of Computer Science and Engineering, REVA University, Bengaluru, India
Search for more papers by this authorS. Sreejith
Department of Artificial Intelligence and Data Science, New Horizon College of Engineering, Bangalore, India
Search for more papers by this authorShrikant Tangade
Department of Mathematics, University of Padova, Padova, Italy
Search for more papers by this authorAli Kashif Bashir
Search for more papers by this authorSummary
In the era of rapid sensor and information technology advancements, machines now possess the remarkable capability to not only recognize but also analyze the intricate tapestry of human emotions. The pursuit of emotion recognition has become a focal point across various domains, unveiling a captivating landscape where facial expressions, speech patterns, behavior, and physiological signals serve as the rich palette for understanding human emotions. These diverse signals, artfully captured by different sensors, pave the way for accurate identification, propelling the field of affective computing into an exciting frontier. This journey underscores the profound importance of decoding the multifaceted layers of human experience, marking a transformative stride towards a more emotionally intelligent technological landscape. This research looks into the use of Deep Bidirectional Long Short-Term Memory (LSTM) for the nuanced task of emotion detection using mobile sensor analysis. Emotion detection is at the heart of several applications, most notably human-robot interaction. Traditionally, this work includes fusing data from several sources, such as physiological signals, environmental inputs, video feeds, and so on. These methodologies frequently wrestle with the complexities of feature engineering, a process fraught with inherent limits. Our novel technique harnesses the power of deep learning, aided by an iterative process that weaves a tapestry of insights from a wide range of sensor data kinds and modalities. What distinguishes us is our ability to avoid the time-consuming task of manual feature engineering. Our dataset is a synthesis of information from on-body, ambient, and geographical sensors, providing a comprehensive view of the ever-changing landscape of human emotions. Our foray into uncharted territory has yielded truly remarkable results. Our approach surpasses traditional deep neural networks and ensemble methods that depend on feature engineering, achieving an exceptional average accuracy of 97%. This study underscores the groundbreaking potential of Deep Bidirectional LSTM for emotion detection through mobile sensor analysis, emphasizing the paramount importance of accuracy and effectiveness.
References
-
Picard , R.W.
,
Affective Computing
, pp.
4
–
5
,
MIT Press
,
Cambridge, MA, USA
,
2000
, [Google Scholar].
10.7551/mitpress/1140.003.0008 Google Scholar
-
Nayak , S.
,
Nagesh , B.
,
Routray , A.
,
Sarma , M.
,
A Human-Computer Interaction Framework for Emotion Recognition through Time-Series Thermal Video Sequences
.
Comput. Electr. Eng.
,
93
,
107280
,
2021
, [Google Scholar] [CrossRef].
10.1016/j.compeleceng.2021.107280 Google Scholar
- Colonnello , V. , Mattarozzi , K. , Russo , P.M. , Emotion Recognition in Medical Students: Effects of Facial Appearance and Care Schema Activation . Med. Educ. , 53 , 195 – 205 , 2019 , [Google Scholar] [CrossRef] [PubMed].
- Feng , X. , Wei , Y.J. , Pan , X.L. , Qiu , L.H. , Ma , Y.M. , Academic Emotion Classification and Recognition Method for Large-Scale Online Learning Environment-Based on a-CNN and LSTM-Att Deep Learning Pipeline Method . Int. J. Environ. Res. Public Health , 17 , 1941 , 2020 , [Google Scholar] [CrossRef] [PubMed][Green Version].
-
Fu , E.
,
Li , X.
,
Yao , Z.
,
Ren , Y.X.
,
Wu , Y.H.
,
Fan , Q.Q.
,
Personnel Emotion Recognition Model for Internet of Vehicles Security Monitoring in Community Public Space
.
Eurasip J. Adv. Signal Process.
,
2021
,
81
,
2021
, [Google Scholar] [CrossRef].
10.1186/s13634-021-00789-5 Google Scholar
-
Oh , G.
,
Ryu , J.
,
Jeong , E.
,
Yang , J.H.
,
Hwang , S.
,
Lee , S.
,
Lim , S.
,
DRER: Deep Learning-Based Driver's Real Emotion Recognizer
.
Sensors
,
21
,
2166
,
2021
, [Google Scholar] [CrossRef] [PubMed].
10.3390/s21062166 Google Scholar
-
Sun , X.
,
Song , Y.Z.
,
Wang , M.
,
Toward Sensing Emotions with Deep Visual Analysis: A Long-Term Psychological Modeling Approach
.
IEEE Multimed.
,
27
,
18
–
27
,
2020
, [Google Scholar] [CrossRef].
10.1109/MMUL.2020.3025161 Google Scholar
-
Mandryk , R.L.
,
Atkins , M.S.
,
Inkpen , K.M.
,
A continuous and objective evaluation of emotional experience with interactive play environments
, in:
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
,
Montréal, QC, Canada
, 22–27 April
2006
, pp.
1027
–
1036
, [Google Scholar].
10.1145/1124772.1124926 Google Scholar
-
Ogata , T.
and
Sugano , S.
,
Emotional communication between humans and the autonomous robot which has the emotion model
, in:
Proceedings of the 1999 IEEE International Conference on Robotics and Automation (Cat. No. 99CH36288C)
,
Detroit, MI, USA
, 10–15 May
1999
, pp.
3177
–
3182
, [Google Scholar].
10.1109/ROBOT.1999.774082 Google Scholar
- Malfaz , M. and Salichs , M.A. , A new architecture for autonomous robots based on emotions . IFAC Proc. , 37 , 805 – 809 , 2004 , [Google Scholar] [CrossRef].
-
Rattanyu , K.
,
Ohkura , M.
,
Mizukawa , M.
,
Emotion monitoring from physiological signals for service robots in the living space
, in:
Proceedings of the ICCAS 2010
,
Gyeonggi-do, Republic of Korea
, 27–30 October
2010
, pp.
580
–
583
, [Google Scholar].
10.1109/ICCAS.2010.5669914 Google Scholar
-
Hasnul , M.A.
,
Aziz , N.A.A.
,
Alelyani , S.
,
Mohana , M.
,
Aziz , A.A.J.S.
,
Electrocardiogram-based emotion recognition systems and their applications in healthcare—A review
.
Sensors
,
21
,
5015
,
2021
, [Google Scholar] [CrossRef].
10.3390/s21155015 Google Scholar
-
Feidakis , M.
,
Daradoumis , T.
,
Caballé , S.
,
Emotion measurement in intelligent tutoring systems: What, when and how to measure
, in:
Proceedings of the 2011 Third International Conference on Intelligent Networking and Collaborative Systems
,
Fukuoka, Japan
, 30 November–2 December
2011
, pp.
807
–
812
, [Google Scholar].
10.1109/INCoS.2011.82 Google Scholar
-
Saste , S.T.
and
Jagdale , S.
,
Emotion recognition from speech using MFCC and DWT for security system
, in:
Proceedings of the 2017 international Conference of Electronics, Communication and Aerospace Technology (ICECA)
,
Coimbatore, India
, 20–22 April
2017
, pp.
701
–
704
, [Google Scholar].
10.1109/ICECA.2017.8203631 Google Scholar
- Zepf , S. , Hernandez , J. , Schmitt , A. , Minker , W. , Picard , R. , Driver emotion recognition for intelligent vehicles: A survey . ACM Comput. Surv. (CSUR) , 53 , 1 – 30 , 2020 , [Google Scholar] [CrossRef].
- Houben , M. , Van Den Noortgate , W. , Kuppens , P. , The relation between short-term emotion dynamics and psychological well-being: A meta-analysis . Psychol. Bull. , 141 , 901 , 2015 , [Google Scholar] [CrossRef][Green Version].
- Bal , E. , Harden , E. , Lamb , D. , Van Hecke , A.V. , Denver , J.W. , Porges , S.W. , Emotion recognition in children with autism spectrum disorders: Relations to eye gaze and autonomic state . J. Autism Dev. Disord. , 40 , 358 – 370 , 2010 , [Google Scholar] [CrossRef].
-
Martínez , R.
,
Ipiña , K.
,
Irigoyen , E.
,
Asla , N.
,
Garay , N.
,
Ezeiza , A.
,
Fajardo , I.
,
Emotion elicitation oriented to the development of a human emotion management system for people with intellectual disabilities
, in:
Trends in Practical Applications of Agents and Multiagent Systems: 8th International Conference on Practical Applications of Agents and Multiagent Systems
, pp.
689
–
696
,
Springer
,
Berlin/Heidelberg, Germany
,
2010
, [Google Scholar].
10.1007/978-3-642-12433-4_81 Google Scholar
- Plasqui , G. and Westerterp , K.R. , Physical activity assessment with accelerometers: an evaluation against doubly labeled water . Obesity , 15 , 10 , 2371 – 2379 , 2007 .
-
Supratak , A.
,
Wu , C.
,
Dong , H.
,
Sun , K.
,
Guo , Y.
,
Survey on feature extraction and applications of biosignals
, in:
Machine Learning for Health Informatics
, pp.
161
–
182
,
Springer
,
2016
.
10.1007/978-3-319-50478-0_8 Google Scholar
- Ronao , C.A. and Cho , S.-B. , Human activity recognition with smartphone sensors using deep learning neural networks . Expert Syst. Appl. , 59 , 235 – 244 , 2016 .
- Längkvist , M. , Karlsson , L. , Loutfi , A. , A review of unsupervised feature learning and deep learning for time-series modeling . Pattern Recognit. Lett. , 42 , 11 – 24 , 2014 .
- Ordóñez , F.J. and Roggen , D. , Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition . Sensors , 16 , 1 , 115 , 2016 .
-
Yan , F.
and
Mikolajczyk , K.
,
Deep correlation for matching images and text
, in:
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, pp.
3441
–
3450
,
2015
.
10.1109/CVPR.2015.7298966 Google Scholar
- Jozefowicz , R. , Zaremba , W. , Sutskever , I. , An empirical exploration of recurrent network architectures , in: Proceedings of the 32nd International Conference on Machine Learning (ICML-15) , pp. 2342 – 2350 , 2015 .
- Hinton , G.E. , Osindero , S. , Teh , Y.-W. , A fast learning algorithm for deep belief nets . Neural Comput. , 18 , 7 , 1527 – 1554 , 2006 .
-
Haag , A.
,
Goronzy , S.
,
Schaich , P.
,
Williams , J.
,
Emotion recognition using bio-sensors: First steps towards an automatic system
, in:
Tutorial and Research Workshop on Affective Dialogue Systems
,
Springer
,
Berlin/Heidelberg, Germany
, pp.
36
–
48
,
2004
.
10.1007/978-3-540-24842-2_4 Google Scholar
- Mehta , D. , Siddiqui , M.F.H. , Javaid , A.Y. , Facial emotion recognition: A survey and real-world user experiences in mixed reality . Sensors , 18 , 416 , 2018 , [CrossRef] [PubMed].
-
Marechal , C.
,
Mikolajewski , D.
,
Tyburek , K.
,
Prokopowicz , P.
,
Bougueroua , L.
,
Ancourt , C.
,
Wegrzyn-Wolska , K.
,
Survey on AI-Based Multimodal Methods for Emotion Detection
, in:
High-Performance Modelling and Simulation for Big Data Applications
, vol.
11400
,
J. Kołodziej
and
H. González-Vélez
(Eds.), pp.
307
–
324
,
Springer
,
Berlin/Heidelberg, Germany
,
2019
.
10.1007/978-3-030-16272-6_11 Google Scholar
- Dzedzickis , A. , Kaklauskas , A. , Bucinskas , V. , Human Emotion Recognition: Review of Sensors and Methods . Sensors , 20 , 592 , 2020 , [CrossRef].
- Landowska , A. , Brodny , G. , Wrobel , M.R. , Limitations of Emotion Recognition from Facial Expressions in e-Learning Context . CSEDU , Gdansk, Poland , pp. 383 – 389 , 2017 .
-
Coito , T.
,
Firme , B.
,
Martins , M.S.
,
Vieira , S.M.
,
Figueiredo , J.
,
Sousa , J.
,
Intelligent Sensors for Real-Time Decision-Making
.
Automation
,
2
,
62
–
82
,
2021
, [CrossRef].
10.3390/automation2020004 Google Scholar