Tweet normalization using conditional random fields and distributed representations of words - Flores, Elen Rose S. (2016)
Abstract: Social interaction has been in its all-time high in the onset of the 21st century, wherein various social media platforms have emerged to cater the need of social interactions in the global scene. Twitter is one of the social media platforms which came forth for this purpose - information sharing in a massive scale. Given this, the diversity of twitter data shows noisy texts, which can lower the quality of information. Tweets that contain very valuable information were however impeded by noisy texts or ill-formed words which blur the meaning of rich data relayed by tweets. Tweet normalization is applied to formalize tweets, to make them understandable and to retain the value of the information. This paper describes three techniques of normalizing noisy texts in Twitter data. The first approach is a supervised rule-based method that performs corrections to transform each out-of-vocabulary (OOV) word to its canonical form. The second approach is an unsupervised model using distributed representations of words. Using word vectors, the latter approach finds the most similar word of each OOV token. The last approach is the combination of the rule-based and word vector methods. An evaluation was performed to the test data provided by the ACL 2015 W-NUT: Workshop on Noisy User-generated Text shared task to compare the difference between the two approaches, also including a hybrid of both. The supervised method achieved a higher F-score of 0.5114 compared to the unsupervised method which achieved an F-score of 0.1676. The hybrid approach yielded an improved F-score of 0.6086 which is higher than the rule-based approach and word vector approach. On the experiment of normalizing Philippine tweets, the rule-based model yielded an F-score of 0.2575, the word vector approach had an F-score of 0.0963 and the hybrid model ranked highest with an F-score of 0.2741. This study showed that the hybrid normalization model was able to capture the lexical and contextual standard equivalents of the out-of-vocabulary words.
Thesis (Bachelor of Science in Computer Science). University of the Philippines Cebu