LaDiff ULMFiT: A Layer Differentiated Training Approach for ULMFiT

Abstract

In our paper we present Deep Learning models with a layer differentiated training method which were used for the SHARED TASK @ CONSTRAINT 2021 sub-tasks COVID19 Fake News Detection in English and Hostile Post Detection in Hindi. We propose a Layer Differentiated training procedure for training a pre-trained ULMFiT [8] model. We used special tokens to annotate specific parts of the tweets to improve language understanding and gain insights on the model making the tweets more interpretable. The other two submissions included a modified RoBERTa model and a simple Random Forest Classifier. The proposed approach scored a precision and f1-score of 0.96728972 and 0.967324832 respectively for sub-task COVID19 Fake News Detection in English. Also, Coarse Grained Hostility f1 Score and Weighted Fine Grained f1 score of 0.908648 and 0.533907 respectively for sub-task Hostile Post Detection in Hindi. The proposed approach ranked 61st out of 164 in the sub-task “COVID19 Fake News Detection in English” and 18th out of 45 in the sub-task “Hostile Post Detection in Hindi”. The complete code implementation can be found at: GitHub Repository (https://github.com/sheikhazhanmohammed/AAAI-Constraint-Shared-Tasks-2021).