Títol:
|
Automatic normalization of short texts by combining statistical and rule-based techniques
|
Autor/a:
|
Ruiz Costa-Jussà, Marta; Banchs, Rafael E.
|
Altres autors:
|
Universitat Politècnica de Catalunya. Departament de Teoria del Senyal i Comunicacions; Universitat Politècnica de Catalunya. VEU - Grup de Tractament de la Parla |
Abstract:
|
The final publication is available at link.springer.com http://link.springer.com/article/10.1007%2Fs10579-012-9187-y#page-1 |
Abstract:
|
Short texts are typically composed of small number of words, most of which are abbreviations, typos and other kinds of noise. This makes the noise to signal ratio relatively high for this specific category of text. A high proportion of noise in the data is undesirable for analysis procedures as well as machine learning applications. Text normalization techniques are used to reduce the noise and improve the quality of text for processing and analysis purposes. In this work, we propose a combination of statistical and rule-based techniques to normalize short texts. More specifically, we focus our attention on SMS messages. We base our normalization approach on a statistical machine translation system which translates from noisy data to clean data. This system is trained on a small manually annotated set. Then, we study several automatic methods to extract more general rules from the normalizations generated with the statistical machine translation system. We illustrate the proposed methodology by conducting some experiments with a SMS Haitian-Créole data collection. In order to evaluate the performance of our methodology we use several Haitian-Créole dictionaries, the well-known perplexity criteria and the achieved reduction of vocabulary. |
Matèries:
|
-Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural -Machine translating -Normalization chats -Normalization of short texts -Statistical machine translation -Automatic extraction of rules -Perplexity -Traducció automàtica |
Drets:
|
http://creativecommons.org/licenses/by-nc-nd/3.0/es/ |
Tipus de document:
|
Article - Versió presentada Article |
Compartir:
|
|