Para acceder a los documentos con el texto completo, por favor, siga el siguiente enlace:

Automatic normalization of short texts by combining statistical and rule-based techniques
Ruiz Costa-Jussà, Marta; Banchs, Rafael E.
Universitat Politècnica de Catalunya. Departament de Teoria del Senyal i Comunicacions; Universitat Politècnica de Catalunya. VEU - Grup de Tractament de la Parla
The final publication is available at
Short texts are typically composed of small number of words, most of which are abbreviations, typos and other kinds of noise. This makes the noise to signal ratio relatively high for this specific category of text. A high proportion of noise in the data is undesirable for analysis procedures as well as machine learning applications. Text normalization techniques are used to reduce the noise and improve the quality of text for processing and analysis purposes. In this work, we propose a combination of statistical and rule-based techniques to normalize short texts. More specifically, we focus our attention on SMS messages. We base our normalization approach on a statistical machine translation system which translates from noisy data to clean data. This system is trained on a small manually annotated set. Then, we study several automatic methods to extract more general rules from the normalizations generated with the statistical machine translation system. We illustrate the proposed methodology by conducting some experiments with a SMS Haitian-Créole data collection. In order to evaluate the performance of our methodology we use several Haitian-Créole dictionaries, the well-known perplexity criteria and the achieved reduction of vocabulary.
Àrees temàtiques de la UPC::Informàtica::Intel·ligència artificial::Llenguatge natural
Machine translating
Normalization chats
Normalization of short texts
Statistical machine translation
Automatic extraction of rules
Traducció automàtica

Mostrar el registro completo del ítem

Documentos relacionados

Otros documentos del mismo autor/a

Farrus, Mireia; Ruiz Costa-Jussà, Marta; Cobo, German; García Solórzano, David; Villarejo Muñoz, Luis; Banchs, Rafael E.
Ruiz Costa-Jussà, Marta; Gupta, Parth; Banchs, Rafael E.; Rosso, P.
Ruiz Costa-Jussà, Marta; Henríquez, Carlos; Banchs, Rafael E.
Centelles, Jordi; Ruiz Costa-Jussà, Marta; Banchs, Rafael E.