Real-Time Sign Language Recognition and Translation: A Mobile Solution Using Convolutional Neural Network

Abstract

This study presents a mobile application for sign language recognition and translation using a convolutional neural network (CNN) to overcome communication barriers for the deaf community. Unlike existing solutions, the app uses a CNN trained on a dataset of 200–450 images per sign to process hand images via preprocessing, feature extraction, and hand landmark detection, accurately recognizing sign language gestures. The application underscores a user-friendly interface and is designed for real-time mobile use. Employing CNN-based image processing, it translates hand movements into gestures with high precision, achieving 96% accuracy and a loss of 0.069 after 100 training epochs with a batch size of 40. Usability testing, conducted using the System Usability Scale (SUS) questionnaire, revealed high user satisfaction, with positive feedback on usability, functionality, maintainability, and efficiency. The average SUS score indicates an excellent usability. Further evaluation criteria included precision, recall, and F1-score, all of which demonstrated strong performance. The system effectively bridges the communication gap between the deaf and hearing communities, fostering more accessible and meaningful interactions.