Augmented Reality Based Spelling Assistance to Dysgraphia Students

Dysgraphia, a learning disability associated with writing skills, hinders students to put their thought on paper and write correctly. Writing problems hit students most frequently that one third students become failed to acquire writing skill. Different IT based assistance solutions available for dysgraphia students but most of them are accommodations based or provides writing alternatives rather than developing writing skills of a dysgraphia student. Handwriting is an essential skill for academic life and developed handwriting skill helps student to protect their selfesteem and build student’s confidence to participate in other activities during class. Most of available writing assistance solutions do not provide interesting ways to acquire writing skills. To handle this problem, augmented reality (AR) based dysgraphia assistance solution has presented in this work. This study utilized AR to develop dysgraphia student’s interest in writing and used it to assist in writing activity by providing help in spellings. AR based dysgraphia assistance writing environment (AR-DAWE) modal use Google cloud API of speech-to-text and addressed one of the important issues of dysgraphia student that is associated with spelling mistakes.


INTRODUCTION
Dysgraphia is a learning disability which affects writing skills, that requires complex set of motor and information processing skills [1]. It is a component within the wide-ranging definition of written expression disorders; refers to illegible handwriting and spelling errors [2]. According to a report, about 13% of students (age [3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21] in all public schools receiving special education. Among 35% of these have learning disability issues [3]. Handwriting is the basic academic but complex skill that acquired over extended period of time and it combines motor and linguistic components [2]. It usually produces problem in spellings and poor handwriting [4]. Student face immense difficulty to put their thoughts on paper hence, produces low quality of script [5]. They often make letter formation inconsistencies; make capital/small letter mistake and letter space inconsistency. Associated problems of the disability are sign orientations through symptoms (addition, deletion, omission, transposition of punctuation sign and symbol, transliteration/ transcription of one alphabet to another) [1]. Due to these reasons, children affected with dysgraphia avoid *Address correspondence to this author at the Computer Science Department, Federal Urdu University of Arts, Science & Technology MSc. Block, Gulshan-e-Iqbal, Karachi, Pakistan; Tel: +92-308-2735377; E-mail: mfkj@live.com writing task, being considering it a cumbersome activity and become frustrated early. Handwriting problems are in general due to motor nature, they are not related to poor spellings or any kind of psycholinguistic issues [5]. Another worth noting fact is that one third students fail to get efficient writing skills. Handwriting is the basic academic but complex skill that acquired over extended period and it combines motor and linguistic components [2]. For learning disable students, it is necessary to discover innovative solution that not only help them but also motivate them in learning process [6].
Augmented reality integrates virtual object or virtual view in real object or real environment. It is similar to virtual reality (VR) concept but it did not immerse user completely instead user can view virtual object in augmentation with real object. AR adds a virtual layer that can include 3D-objects, animation, texture/images, sound into real world view [7]. In simple manner, it is a technique that augments virtual view into the real world and doesn't break the user connection/interaction with real world unlike VR. Augmented reality (AR) constitutes with three main characteristics which include (i) combination of real and virtual world, (ii) real time user interaction ability and (iii) it being happen/register in 3d space [8]. Augmented reality is one of the widely research area and its implementation in educational fields has enormous potential [9] to solve different issues. Augmented reality implementation in formal educational programs is a leading trend in upcoming learning environments and it has positive impacts in learning and educational fields [7]. Augmented reality provides interesting ways of learning [9] therefore, it can be an ideal learning means for Dysgraphia students who usually get bore early and consider writing a cumbersome task. Currently, AR implementation in learning and education sector is a leading trend and it has positive impact on learning outcomes. Teachers become able to show virtual objects to their student instead of defining and explaining things in plain words or asking student for imaginations. AR has also enabled students to directly view lesson on the objects and there are many apps available that help in learning [7]. Cloud computing consider as an idea that emphasize on central storage of program and data which can be access from everywhere and anytime, which has many advantages including access flexibility, ubiquity of data and resilience [10]. Cloud computing provide great scalability, flexibility and it helps to reduce cost [11] Cloud computing make distance learning easier through providing access to information and data sharing [12]. Massive open online courses (MOOCs) like Udacity [13], Coursera [14], or EdX [15] are some the example of distance learning through cloud computing.
Most of the dysgraphia assistance solutions are based on accommodations that are unable to enhance student's writing skill and do not motivate student to write which is essential to acquire handwriting skills. Under consideration of these facts, this study aims to present an assistive environment for dysgraphia students using augmented reality and cloud computing. The proposed systematic modal has referred to AR-DAWE (augmented reality -dysgraphia assistance writing environment). In this work, a technical model (AR-DAWE) has discussed in detail which states about the mechanism of spelling assistance to dysgraphia students with the help of augmented reality and cloud based speech to text service. There are different writing problems that constitute overall writing difficulty for children with Dysgraphia, including spelling mistakes which has addressed in this work with augmented reality based solution.

DYSGRAPHIA ASSISTANCE AND THEIR LIMITATIONS
There are three ways or strategies to assist dysgraphia including prevention strategy, remediation and accommodation. In prevention, early training provides to grade one or kindergarten students about letter formation and writing which prevents many problems but it is difficult to eradicate incorrect habits at this stage. Remediation strategy brings exercise and writing drill for dysgraphia students. Specialist suggested muscle trainings and recommended exercise plan help students to increase dexterity and strength. Remediation requires daily practice of alphabets and exercises period may prolong to several months. Lastly, accommodation refers to assistance through providing alternatives method like word processors and oral answer facility to students. Accommodation by pass complex motor training and provides alternatives to handwriting [16] but not primarily help to enhance handwriting skill. It is necessary to evaluate these solution, based on three characteristics, which includes writing skill development, interesting or attentive solution and provides motivations.
Different solutions available for Dysgraphia affected children, including voice-recorded notes, speech-totext, simultaneous recording & typing or handwritten notes and therapeutic hand exercises to improve children motor skills [1]. Therapeutic hand exercises are good to acquire the writing skills as it does improve motor skills and dexterity. Some accommodations including word processing and different IT solution help to minimize the problem and provide alternatives but it didn't improve or help to acquire writing skills.
Detail tools have outlined in [17] for struggling writers that includes word processing, word predictions, spell check and speech recognition (SR). These IT based solutions provide alternatives to handwriting which although provide some motivations but does not contribute in writing skill development. Best way to help dysgraphia students is to refine motor skills and ability using exercises/activities aimed at developing their visual spatial co-ordination and consequently their writing skill and ability [4]. There are also different widgets accommodation available like Pencil grip, Slant board, Raised Paper, highlighted paper [18] to assist dysgraphia student during writing. These accommodations can bring good result in combination with some other IT based writing solutions.
Different apps, aimed at assisting dysgraphia students, that detect user's handwriting on touch surface using finger or stylus and provide interesting way to write and enhance writing skills. Handwriting without Tears called Wet-Dry-Try app [19] provides writing facility on touchscreen and enable a student to learn letter writing in interesting way and motivate students awarding stars or points. It can be classified as a preventive solution that assists early grade one / kindergarten students. It can help to develop student's writing skill at early stage of education but can't assist in common writing task like essay or paragraph writing. Writing Pad [20] recognizes writing on touchscreen and provides spell checking facility. Although this app is not mainly developed to assist dysgraphia students but it can help dysgraphia students. Typ-O HD -Writing is for everybody! [21] is iOS based app that specially designed for dyslexia (reading disorder) users. It helps to write without spelling mistakes and support voice input. Accommodation based solutions for dysgraphia students are also available, like Dragon Dictation [22], Dragon-GO which is an instant voice to text conversion application. SnapType [23,24] is another specially designed app for dysgraphia that provide ability to students to snap the worksheet and type answer on it directly. These apps just provide alternatives to dysgraphia students. They are not developed to enhance the writing skills of dysgraphia students. These solutions are somehow lack of motivations or not directly impact students' handwriting. Therefore, a solution that not only motivate dysgraphia students to write but also contributes in handwriting skill development, is necessary. Augmented reality is the most exciting way to deal with this problem being interesting in nature [9]. It allows superimpose virtual object in real environment which can be see through camera.

DYSGRAPHIA
Dysgraphia affected children avoid writing task being considering it a cumbersome activity and become frustrated early. Therefore, it requires such a solution that not only creates interest of dysgraphia students in writing activity but also motivate them to write and participate in this activity profoundly.

A. Common Characteristics of Dysgraphia
There are common characteristics of Dysgraphia define in different literatures [16,[25][26][27][28][29], which have classified in Table 1 according to the type of the problem and presented its taxonomy. Classification helps in addressing specific issue and present solution accordingly. A Dysgraphia student can exhibit one or more characteristics [25].
Dysgraphia affected children often make spelling mistakes which contributes in erroneous and poor quality of writing. This study also provides assistance in spelling to avoid the core writing problem, therefore, augmented reality based Dysgraphia Assistive Writing Environment (AR-DAWE) model has presented with the scope to address the spelling mistake issue. AR-DAWE model has primarily tried to address the spelling mistake issue of surface or mixed dysgraphia student who are able to pronounce the sound of the word but unable to write it correctly [30]. Phonological dysgraphia creates problem in only phonetically irregular words, non-words or unfamiliar words spelling [25] so AR-DAWE model can help phonological dysgraphia students if they are able to speak the regular words correctly.

AUGMENTED REALITY BASED DYSGRAPHIA ASSISTIVE WRITING ENVIRONMENT (AR-DAWE)
To handle the problem during writing, augmented reality based assistive environment AR-DAWE (Augmented Reality -Dysgraphia Assistive Writing Environment) model has presented in Figure 1. Currently the model addressed the core writing problems of dysgraphia students. AR-DAWE is an augmented reality based assistive environment that help dysgraphia students by providing real-time spelling assistance based on user voice input. The real-time spelling feedback will be provided to dysgraphia students in augmented reality fashion with backend support by speech-to-text cloud service. Using AR-DAWE system, dysgraphia student speaks the desired word, difficult for him/her to write, during writing. Then, system will recognize the word and will provide exact spelling to the student on notebook using 3D Text in the real physical 3D space. The displayed 3D text output's coordinates will be adjusted near to the student's notebook so that it can conveniently, attentively and interestingly enable student to understand spelling of the desired word and learn spellings profoundly. By minimizing the spelling mistakes, AR-DAWE model will enable dysgraphia students to write correctly, make fewer mistakes and help to build their confidence to participate in their academic activities. The system has enough potential to keep students' interest in writing and assist them during copying the text or spontaneously writing. Further the characteristics of AR based solution will also allow to solve the contextual and sensual problems of dysgraphia students.

A. AR-DAWE Components and Layers
Technically, AR-DAWE model consists of two components which are client and cloud component, it can be classified into client and server model. The client component consist of two layers i.e. hardware and application layer while cloud components only consist of single speech to text conversion component. Cloud component's Google speech to text API [31] has used to convert student speech input into text format.
In client component, hardware layer can be Augmented Reality Smart Glasses [32] or any Smartphone that can get voice input and must have a camera to detect object and render AR contents. Application layer consist of the application that can be developed on using any game engine like unity3d [33] or unreal [34].
To enable AR content and object tracking in unity3d, it requires third party libraries like Vuforia [35] or ARtoolkit [36] that helps to generate AR contents and detect objects in real word. The application layer will manage to take voice input and send it to the cloud service which return converted text and present it to the user through AR libraries (e.g., ARtoolkit or Vuforia).

Client Components
The application layer consists of the mobile app which is the actual implementation of the proposed concept. The mobile app will manage to take user voice input (e.g., word input) and send it to cloud component and get the cloud service response to show the user spelling output in augmented reality fashion.
The mobile application can be developed through any game engine like Unity or Unreal. For generating AR contents within Unity, it requires third party libraries like Vuforia [35] or ARtoolkit [36] that work with Unity to develop AR application. For Unreal game engine unreal4AR plug-in [37] can be used that build on ARtoolkit [36] to generate or track AR contents.
The functionality of the app will be: i.
Get input in user's voice ii. Send speech to cloud service iii.
Received text output from cloud iv.
Transform text into 3d text v.
Apply user selected color scheme in 3D text.
vi. Detect the user notebook object through mobile camera with the help AR SDKs.
vii. Augment 3d Text (spelling) in physical space, on top of the notebook using AR SDKs/Algorithm.

Cloud Component
Cloud components, classified as Sever component, consist of Google Speech API that provides real time speech to text conversion based on neural networks model. The API recognized over 80 languages with different variants [31]. This is an important component of the AR-DAWE model as it will convert user audio speech input into text format. The speech API will be used by the app that will be run on client device to transform the audio into text. Then, the output text will be show to the user within 3D format with user specified color materials. Dysgraphia student can speak word correctly but failed to write it with correct spellings. Speech-to-text cloud service will help dysgraphia student to understand the spelling of the word.

B. AR-DAWE Flow
Dysgraphia student required an internet enabled Smartphone or Augmented Reality Glasses with application. Whenever Dysgraphia student found difficulty in writing any word, he/she has to provide voice input of that particular word:

User Voice Input
The student speaks the word which will be recognize by the application and application send it to Google speech engine through its API.

Voice Input to Text
Google speech API at cloud side (server side) will convert speech input into text and send the text back to client application.

Text to 3D/2D Text Conversion
Application will convert the text into 3D text. 3D text representation will allow the disabled student to keep their attention and understand the spelling without any third person intervention.

3D Text Output Display
3D text output will be displayed to user on top of the notebook. Notebook will be recognized by image processing algorithms of ARToolkit or Vuforia SDK and text augmentation with real word objects (e.g., student notebook etc.) will also be enabled by these libraries.
3D text augmentation with real object (like student's notebook) can become more interesting by replacing 3D text into animation. An animation based text display can impact more positively and it can also help students to learn letter formation technique. Animated words/letters text augmentation with real world objects can also help students to learn and memorize the formation of a letter or word by visualization each letter formation technique. The propose modal developed in Unity and Unreal game engine that support AR content generation and tracking.

C. Proof-of-Concept
For the proof of the presented concept, a prototype application has developed in unity3d using the above defined model. Figure 2a, b & c depicted the typical view of this application where two buttons including start speech input and spelling output available along with the color picker panel for the color of spelling text. Whenever user wants to view the spelling of a desired word, he/she press the start speech input button to record the desired word. After recording the speech input, user press speech button to stop speech input. For watching the spelling result, user focus on his/her notebook object and need to touch the show output button. It will show the augmented text on user notebook object as shown in Figure 2d. The augmented spelling text is also further customizable so that end user can adjust its color according to his/her own choice.

DISCUSSION
Dysgraphia is a writing disability and different solutions are available that address various aspects of writing and related issues but most of them are not primarily developed to enhance students writing skills. These solutions also lack of providing motivations to students to participate in writing activity. The AR-DAWE system will help to solve these serious issues of dysgraphia students.
In current proposed solution, augmented reality based system (AR-DAWE) has presented that primarily help dysgraphia students in spellings. It converts user speech into text and show the correct spellings through AR generated contents which is an interesting and exciting way to deal with this problem. AR will help to motivate learning disable students who are usually avoid writing activity and it will also assist them in writing activity real-time. Proposed AR-DAWE model addressed limited number of Dysgraphia students that can speak words correctly but unable to write it correctly. The model is expendable and reusable to address wide-range of dysgraphia student along with their different problems. Beside significant advantages of the proposed system, AR-DAWE system posed some of the weaknesses especially in the cloud component context. The system requires internet connection convert speech into text. Further, it can delay the result of speech-to-text conversion due to increase network traffic or slow internet connection. Moreover, speech to text conversion still is not 100% mature [38] yet, so it can produce erroneous result.
The maturity of the AR third party libraries like ARtoolkit or Vuforia can also be a concern in proposed model. In AR applications tracking accuracy has an important role [39]. As AR-DAWE system will be used within the range of 1 meter (distance between notebooks and students' will be less than 1 meter) it will not make any problem with AR toolkits to track the notebook object in real space in order augment text on it. But tracking problems can be occurred if the distance exceeds with 1 meter. Another issue students can face, during writing, is the position of augmented 3d text which will be superimposed on students writing notebook. It can create problems for student to learn the spelling if notebook page is already filled of text. This problem is controllable through providing the color option of the 3d text which a user can adjust according to his/her own preferences and visibility.
To get the better results and ease in assistance through AR-DAWE model it is preferred to use AR smart glasses over Smartphone. These glasses merge virtual information with physical information and show to their user field of view [32]. The reasons to prefer smart glasses over Smartphone is that smart glasses are more easy and convenient to use. While in Smartphone case, user has hold the device and need to focus his/her notebook to view augmented 3d spelling text which can be a cumbersome task.

CONCLUSION
Technological solution provides great assistance to learning disable students through accommodation or through providing alternatives, but they also assist to upgrade learning disable students' skill. AR-DAWE system has adopted this trend to explore dysgraphia and augmented reality along with the power of cloud computing. It is better to provide such solutions to learning disabled students that help to enhance their skills rather than provide them alternatives.
Currently AR-DAWE system addressed only spelling issue of the dysgraphia students. The work can also expand through adding more facilities in to AR-DAWE systems. Like 3D graphical guide line for students about the text placement position on paper. The feature of AR-DAWE system can also be upgraded through replacing spelling feedback text with animation. Animation can visualize the letter formation that can also help students to learn letter formation.