Um sistema de reconhecimento de sinais em Libras usando CNN e LSTM

Gabriel Stefano, Wesley Lobato Passos, Jonathan Gois, Gabriel Araujo, Amaro de Lima

DOI: 10.14209/sbrt.2021.1570727292

Evento: XXXIX Simpósio Brasileiro de Telecomunicações e Processamento de Sinais (SBrT2021)

Keywords: Libras Computer Vision Convolutional Neural Network Recurrent Neural Network

Abstract

In Brazil, there are around 10 million hard of hearing and deaf people. However, the majority of Brazilians are not fluent in the Brazilian Sign Language (Libras). Many members of the hearing impaired and deaf community have communication issues in everyday life situations. Technological solutions can aid in mitigating this problem. This work proposes a semi-supervised method to identify and classify signals in Libras from Youtube videos. The routine starts by segmenting the videos through a measure of movement intensity. We catalog the video segments according to the corresponding subtitles, which we extract by employing Optical Character Recognition (OCR) if embedded. It composes an ad-hoc dataset that we use to train a Libras recognition system. A Convolutional Neural Network (CNN) performs feature extraction frame-by-frame, and a Recurrent Neural Network (RNN) models the time correlation between the features, thus classifying the signal. The proposed method can achieve accuracy up to 61.6% in the ad-hoc dataset used in this work.

Download