Automated 3D sign language caption generation for video

Mehta, Nayan and Pai, Suraj and Singh, Sanjay Automated 3D sign language caption generation for video. Universal Access in the Information Society. ISSN 1615-5289

[img] PDF
9787.pdf - Published Version
Restricted to Registered users only

Download (1MB) | Request a copy


Eforts to make online media accessible to a regional audience have picked up pace in recent years with multilingual cap�tioning and keyboards. However, techniques to extend this access to people with hearing loss are limited. Further, owing to a lack of structure in the education of hearing impaired and regional diferences, the issue of standardization of Indian Sign Language (ISL) has been left unaddressed, forcing educators to rely on the local language to support the ISL structure, thereby creating an array of correlations for each object, hindering the language building skills of a student. This paper aims to present a useful technology that can be used to leverage online resources and make them accessible to the hearing�impaired community in their primary mode of communication. Our tool presents an avenue for the early development of language learning and communication skills essential for the education of children with a profound hearing loss. With the proposed technology, we aim to provide a standardized teaching and learning medium to a classroom setting that can utilize and promote ISL. The goals of our proposed system involve reducing the burden of teachers to act as a valuable teaching aid. The system allows for easy translation of any online video and correlation with ISL captioning using a 3D cartoonish avatar aimed to reinforce classroom concepts during the critical period. First, the video gets converted to text via subtitles and speech processing methods. The generated text is understood through NLP algorithms and then mapped to avatar cap�tions which are then rendered to form a cohesive video alongside the original content. We validated our results through a 6-month period and a consequent 2-month study, where we recorded a 37% and 70% increase in performance of students taught using Sign captioned videos against student taught with English captioned videos. We also recorded a 73.08% increase in vocabulary acquisition through signed aided videos

Item Type: Article
Uncontrolled Keywords: Indian Sign Language (ISL) · Profound and severe hearing loss · Bilingual education · Oralism · Total communication (TC)
Subjects: Engineering > MIT Manipal > Electronics and Communication
Engineering > MIT Manipal > Information and Communication Technology
Depositing User: MIT Library
Date Deposited: 21 Apr 2022 04:46
Last Modified: 21 Apr 2022 04:46

Actions (login required)

View Item View Item