The week's pick
Random Articles
Reseach Article
MotionScript: Sign Language to Voice Converter
| International Journal of Computer Applications |
| Foundation of Computer Science (FCS), NY, USA |
| Volume 186 - Number 6 |
| Year of Publication: 2024 |
| Authors: Nidhi Kadam, Chaitanya Kakade, Vishal Kaira |
10.5120/ijca2024923399
|
Nidhi Kadam, Chaitanya Kakade, Vishal Kaira . MotionScript: Sign Language to Voice Converter. International Journal of Computer Applications. 186, 6 ( Jan 2024), 20-26. DOI=10.5120/ijca2024923399
Abstract
Sign language serves as a vital mode of communication for the deaf and mute community, yet it presents a significant barrier in their interaction with the larger society, which often lacks proficiency in sign language. This paper presents MotionScript, an innovative sign language to voice conversion system that leverages computer vision, deep learning, Convolutional Neural Networks (CNN), Natural Language Processing (NLP) and Large Language Model (LLM) to facilitate interaction between individuals from the deaf and mute community and the rest of the world. This paper outlines a thorough comparison of four distinct neural network models, utilizing metrics to identify the most accurate model for transforming American Sign Language (ASL) into coherent and meaningful sentences voiced in natural language. This conversion process incorporates essential components such as autocorrection and the integration of a large language model.
References
- Hsien-I Lin, Ming-Hsiang Hsu, Wei-Kai Chen, “Human Hand gesture recognition using a convolution neural network”, 10.1109/CoASE.2014.6899454, August 2014
- Bikash K. Yadav, Dheeraj Jadhav, Hasan Bohra, Rahul Jain, “Sign Language to Text and Speech Conversion”, International Journal of Advance Research, Ideas, and Innovations in Technology, www.IJARIIT.com.
- Garcia, B., & Viesca, S. A. (2016), “Real-time American sign language recognition with convolutional neural networks, Convolutional Neural Networks for Visual Recognition, 2, 225-232
- S. S Kumar, T. Wangyal, V. Saboo and R. Srinath, “Time Series Neural Networks for Real Time Sign Language Translation,” 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, FL, 2018, pp. 243-248, doi: 10.1109/ICMLA.2018.00043.
- AlKhuraym, Batool Yahya et al. “Arabic Sign Language Recognition using Lightweight CNN-based Architecture.” International Journal of Advanced Computer Science and Applications (2022)
- X. Han, F. Lu and G. Tian, “Sign Language Recognition Based on Lightweight 3D MobileNet-v2 and Knowledge Distillation,” ICETIS 2022; 7th International Conference on Electronic Technology and Information Science, 2022, pp. 1-6.
- M. Mahesh, A. Jayaprakash and M. Geetha, “Sign language translator for mobile platforms,” 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI), Udupi, 2017, pp. 1176-1181, doi: 10.1109/ICACCI.2017.8126001.
- V. N. T. Truong, C. Yang and Q. Tran, “A translator for American sign language to text and speech,” 2016 IEEE 5th Global Conference on Consumer Electronics, 2016, pp. 1-2, doi: 10.1109/GCCE.2016.7800427.
- Yang, Jie and Y. Xu. “Hidden Markov Model for Gesture Recognition.” (1994), doi: 10.21236/ada282845.
- Lee, C. K. M. et al. “American sign language recognition and training method with recurrent neural network.” Expert Syst. Appl. 167 (2021).
- Sari, Winda & Rini, dian Palupi & Malik, Reza. (2020), “Text Classification Using Long Short-Term Memory With GloVe Features”, Jurnal Ilmiah Teknik Elektro Komputer dan Informatika. 5. 85. 10.26555/jiteki.v5i2.15021
- Saleh, Yaser and Ghassan F. Issa. “Arabic Sign Language Recognition through Deep Neural Networks Fine-Tuning.” Int. J. Online Biomed. Eng. 16 (2020): 71-83.
- M. S. Nair, A. P. Nimitha and S. M. Idicula, “Conversion of Malayalam text to Indian sign language using synthetic animation,” 2016 International Conference on Next Generation Intelligent Systems (ICNGIS), Kottayam, 2016, pp. 1-4, doi: 10.1109/ICNGIS.2016.7854002.
- V. Lopez-Ludena, R. San-Segundo, R. Martin, D. Sanchez and A. Garcia, “Evaluating a Speech Communication System for Deaf People,” in IEEE Latin America Transactions, vol. 9, no. 4, pp. 565-570, July 2011, doi: 10.1109/TLA.2011.5993744.
- D. Kelly, J. Mc Donald and C. Markham, “Weakly Supervised Training of a Sign Language Recognition System Using Multiple Instance Learning Density Matrices,” in IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), vol. 41, no. 2, pp. 526- 541, April 2011, doi: 10.1109/TSMCB.2010.2065802.
- R. San Segundo, B. Gallo, J. M. Lucas, R. Barra-Chicote, L. D’Haro and F. Fernandez, “Speech into Sign Language Statistical Translation System for Deaf People,” in IEEE Latin America Transactions, vol. 7, no. 3, pp. 400- 404, July 2009, doi: 10.1109/TLA.2009.5336641.
- Ss, Shivashankara & S, Dr.Srinath. (2018). American Sign Language Recognition System: An Optimal Approach. International Journal of Image, Graphics and Signal Processing. 10. 10.5815/ijigsp.2018.08.03.
- S. Liu and W. Deng, “Very deep convolutional neural network based image classification using small training sample size,” 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015, pp. 730-734, doi: 10.1109/ACPR.2015.7486599.
- Akash Nagaraj. (2018).ASL Alphabet [Data set]. Kaggle. https://doi.org/10.34740/KAGGLE/DSV/29550
Index Terms
Keywords