Browsing by study line "Språkteknologi"

Social media provides huge amounts of potential data for natural language processing but using this data may be challenging. Finnish social media text differs greatly from standard Finnish and models trained on standard data may not be able to adequately handle the differences. Text normalization is the process of processing non-standard language into its standardized form. It provides a way to both process non-standard data with standard natural language processing tools and to get more data for training new tools for different tasks. In this thesis I experiment with bidirectional recurrent neural network models and models based on the ByT5 foundation model, as well as the Murre normalizer to see if existing tools are suitable for normalizing Finnish social media text. I manually normalize a small set of data from the Ylilauta and Suomi24 corpora to use as a test set. For training the models I use the Samples of Spoken Finnish corpus and Wikipedia data with added synthetic noise. The results of this thesis show that there are no existing tools suitable for normalizing Finnish written on social media. There is a lack of suitable data for training models for this task. The ByT5-based models perform better than the BRNN models.

There are comprehensive requirements in Finland for procurement by any government organization to go through a tendering process where information about each tender is made available not only to vendors and service providers, but to everyone else in Finland as well. This is accomplished through the website Hilma and should make tenders easy to find. Moreover, in Finnish, variance in domain terminology is not thought to be the problem that it is in English. For instance, the last four years of tenders on Hilma never refer to jatkuva parantaminen as toiminnallinen erinomaisuus whereas “continuous improvement” and “operational excellence” could be used interchangeably in English. And yet, it is considered very difficult for a vendor or service provider to find applicable tenders on Hilma. Rather than lexical variability being the cause as it might be in English, the differences in concept paradigms between the private and public sectors in Finland pose the challenge. Whereas a taxi company representative would be looking for tenders about transportation services, a public officer could be posting a tender about social equity for the disabled. The second difficulty is that the Hilma search engine is purely Boolean with restrictive string match criteria rather than inviting natural language questions. Finally, the Hilma search engine does not account for Finnish being a highly inflecting and compounding language where single words usually morph instead of taking on adpositions, and where compound words are affixed together without hyphenation. Many information retrieval approaches would look outside the corpus for query expansion terms. Natural language processing might also offer the potential to look for paraphrases in existing parallel corpora on tenders throughout the European Union rather than in Hilma. However, this thesis focuses on clustering the tenders posted in Finnish on Hilma, applying the comprehensive workflow of the very recent BERTopic package for Python. All documents in each cluster are concatenated and the highest TFIDF-scoring words in the concatenated document are slated to be “search extension terms.” If one of the terms were to be entered by a Hilma user, the user would be invited to perform parallel searches with the remaining terms as well. The first main contribution of this thesis is to use state of the art models and algorithms to represent the corpus, reduce dimensionality of the representations and hierarchically cluster the representations. Second, this thesis develops analytical metrics to be used in automatic evaluation of the efficacy of the clusterings and in comparisons among model iterations that programmatically remove more and more distractions to the clustering that are discovered in the corpus. Finally, this thesis performs case studies on Hilma to demonstrate the remarkable efficacy of the search extension terms in generating tremendous numbers of additional useful matches, addressing paradigm-based differences in terminology, morphovariance and affixation.

Computer-Assisted Language Learning (CALL) is one of the sub-disciplines within the area of Second Language Acquisition. Clozes, also called fill-in-the-blank, are largely used exercises in language learning applications. A cloze is an exercise where the learner is asked to provide a fragment that has been removed from the text. For language learning purposes, in addition to open-end clozes where one or more words are removed and the student must fill the gap, another type of cloze is commonly used, namely multiple-choice cloze. In a multiple-choice cloze, a fragment is removed from the text and the student must choose the correct answer from multiple options. Multiple-choice exercises are a common way of practicing and testing grammatical knowledge. The aim of this work is to identify relevant learning constructs for Italian to be applied to automatic exercises creation based on authentic texts in the Revita Framework. Learning constructs are units that represent language knowledge. Revita is a free to use online platform that was designed to provide language learning tools with the aim of revitalizing endangered languages including several Finno-Ugric languages such as North Saami. Later non-endangered languages were added. Italian is the first majority language to be added in a principled way. This work paves the way towards adding new languages in the future. Its purpose is threefold: it contributes to the raising of Italian from its beta status towards a full development stage; it formulates best practices for defining support for a new language and it serves as a documentation of what has been done, how and what remains to be done. Grammars and linguistic resources were consulted to compile an inventory of learning constructs for Italian. Analytic and pronominal verbs, verb government with prepositions, and noun phrase agreement were implemented by designing pattern rules that match sequences of tokens with specific parts-of-speech, surfaces and morphological tags. The rules were tested with test sentences that allowed further refining and correction of the rules. Current precision of the 47 rules for analytic and pronominal verbs on 177 test sentences results in 100%. Recall is 96.4%. Both precision and recall for the 5 noun phrase agreement rules result in 96.0% in respect to the 34 test sentences. Analytic and pronominal verb, as well as noun phrase agreement patterns, were used to generate open-end clozes. Verb government pattern rules were implemented into multiple-choice exercises where one of the four presented options is the correct preposition and the other three are prepositions that do not fit in context. The patterns were designed based on colligations, combinations of tokens (collocations) that are also explained by grammatical constraints. Verb government exercises were generated on a specifically collected corpus of 29074 words. The corpus included three types of text: biography sections from Wikipedia, Italian news articles and Italian language matriculation exams. The last text type generated the most exercises with a rate of 19 exercises every 10000 words, suggesting that the semi-authentic text met best the level of verb government exercises because of appropriate vocabulary frequency and sentence structure complexity. Four native language experts, either teachers of Italian as L2 or linguists, evaluated usability of the generated multiple-choice clozes, which resulted in 93.55%. This result suggests that minor adjustments i.e., the exclusion of target verbs that cause multiple-admissibility, are sufficient to consider verb government patterns usable until the possibility of dealing with multiple-admissible answers is addressed. The implementation of some of the most important learning constructs for Italian resulted feasible with current NLP tools, although quantitative evaluation of precision and recall of the designed rules is needed to evaluate the generation of exercises on authentic text. This work paves the way towards a full development stage of Italian in Revita and enables further pilot studies with actual learners, which will allow to measure learning outcomes in quantitative terms

Hostility in the player communication of video games (and by extension, mobile games) is a well-documented phenomenon that can have negative repercussions for the well-being of the individual being subjected to it, and the society in general. Existing research on detecting hostility in games through machine learning methods is scarce due to the unavailability of data, imbalanced existing data (few positive samples in a large data set), and the challenges involved in defining and identifying hostile communication. This thesis utilizes communication data from the Supercell game Brawl Stars to produce two distinct machine learning models: a support vector classifier and a multi-layer perceptron. Their performance is compared to each other as well as to that of an existing sentiment analysis classifier, VADER. Techniques such as oversampling and using additional data are also used in an attempt to reach better results by improving the balance of the data set. The support vector classifier model was found to have the best performance overall, with an F1 score of 64.15% when used on the pure data set and 65.74% when combined with the SMOTE oversampling algorithm. The thesis includes an appendix with a list of the words that were found to have the strongest influence on the hostile/non-hostile classification.

Monolingual paraphrases are semantically equivalent sentences in one single language transmitting the same meaning but not necessarily using the same words. Also, the same word can have different meanings in different contexts. Understanding the meaning of a text behind its words is essential for many natural language processing and deep-learning tasks such as machine translation, plagiarism detection, question-answering, and information extraction. Paraphrases have been studied extensively, mainly from an English-only or sometimes multilingual perspective. There are not many studies about paraphrase detection in Finnish and even fewer about detecting paraphrases between different registers of the Finnish language, such as Standard Finnish and Easy Finnish. In this thesis, three different pre-trained sentence-BERT models are tested in a paraphrase detection task. The aim of the task is to find paraphrase pairs and triples between three distinct registers of the Finnish language; Standard Finnish, Easy Finnish, and Colloquial Finnish. As the data Yle News articles in Standard and Easy Finnish mostly from the year 2014 are used, as well as Ylilauta online discussions. The applied BERT models are paraphrase-multilingual-MiniLM-L12-v2 sentence-transformers model and FinBERT model. The first mentioned is also fine-tuned with Finnish paraphrase corpus' data. According to the manual evaluation based on the models' precisions, the fine-tuned model outperforms the other two. The same three models are tested on two different balanced test sets of 50 paraphrase sentence pairs and 50 non-paraphrase sentence pairs. The FinBERT model reaches the best F1 score in this research setting. Among the precision and the F1 score, the average sentence lengths and the repetitiveness of the paraphrase sentence pairs and triples are compared and discussed. The FinBERT model detected the shortest sentences and the most repetition, but its total number of detected sentence pairs was also the highest. As a result of this study, a new Easy Finnish - Standard Finnish paraphrase corpus is collected to facilitate further studies in paraphrase detection or simplification in Finnish. The corpus is presented in this thesis. It contains 5881 sentence pairs of which approximately 98 % can be assumed to be true paraphrases according to the manual evaluation of randomly selected sentence pairs. The corpus is created by using the fine-tuned paraphrase-multilingual-MiniLM -L12-v2 sentence-transformers model and it includes paraphrase sentence pairs from Yle News articles in Easy Finnish and in Standard Finnish from the years 2014-2018.

Neural machine translation (NMT) has been a mainstream method for the machine translation (MT) task. Despite its remarkable progress, NMT systems still face many challenges when dealing with low-resource scenarios. Common approaches to address the data scarcity problem include exploiting monolingual data or parallel data in other languages. In this thesis, transformer-based NMT models are trained on Finnish-Simplified Chinese, a language pair with limited parallel data and the models are improved using various techniques such as hyperparameter tuning, transfer learning and back-translation. Finally, the best NMT system is an ensemble model that combines different single models. The results of our experiments also show that different hyperparameter settings can cause a performance gap of up to 4 BLEU scores. The ensemble model shows a 35% improvement over the baseline model. Overall, the experiments suggest that hyperparameter tuning is crucial for training vanilla NMT models. Back-translation offers more benefits for model improvement than the transfer learning method. The results also show that adding sampling in back-translation does not improve NMT model performance in this low-data setting. The findings may be useful for future research on low-resource NMT, especially the Finnish-Simplified Chinese MT task.

Automatic question answering and question generation are two closely related natural language processing tasks. They both have been studied for decades, and both have a wide range of uses. While systems that can answer questions formed in natural language can help with all kinds of information needs, automatic question generation can be used, for example, to automatically create reading comprehension tasks and improve the interactivity of virtual assistants. These days, the best results in both question answering and question generation are obtained by utilizing pre-trained neural language models based on the transformer architecture. Such models are typically first pre-trained with raw language data and then fine-tuned for various tasks using task-specific annotated datasets. So far, no models that can answer or generate questions purely in Finnish have been reported. In order to create them using modern transformer-based methods, both a pre-trained language model and a sufficiently big dataset suitable for question answering or question generation fine-tuning are required. Although some suitable models that have been pre-trained with Finnish or multilingual data are already available, a big bottleneck is the lack of annotated data needed for fine-tuning the models. In this thesis, I create the first transformer-based neural network models for Finnish question answering and question generation. I present a method for creating a dataset for fine-tuning pre-trained models for the two tasks. The dataset creation is based on automatic translation of an existing dataset (SQuAD) and automatic normalization of the translated data. Using the created dataset, I fine-tune several pre-trained models to answer and generate questions in Finnish and evaluate their performance. I use monolingual BERT and GPT-2 models as well as a multilingual BERT model. The results show that the transformer architecture is well suited also for Finnish question answering and question generation. They also indicate that the synthetically generated dataset can be a useful fine-tuning resource for these tasks. The best results in both tasks are obtained by fine-tuned BERT models which have been pre-trained with only Finnish data. The fine-tuned multilingual BERT models come in close, whereas fine-tuned GPT-2 models are generally found to underperform. The data developed for this thesis will be released to the research community to support future research on question answering and generation, and the models will be released as benchmarks.

Using language that is easy to understand when presenting information in a written form is critical for ensuring effective communication. Yet, using language that is too complex or technical for its intended audience is a common pitfall in many domains, such as legal and medical text. Automatic text simplification (ATS) aims to automatize the conversion of complex text into a simpler, more easily comprehensible form. This study explores ATS models for English that can be controlled in terms of the readability of the output text. Readability is measured with an automatically calculated readability level that corresponds to a school grade level. The readability- controlled models take a readability level as a parameter and simplify input text to match the reading level of the intended audience corresponding to the parameter value. In total, six readability-controlled sentence simplification models with different control attribute configurations are trained in this study. The models use a pretrained sequence-to-sequence model architecture that is finetuned on a dataset of sentence pairs in regular and simple English. The trained models are evaluated using automatic evaluation metrics and compared to each other and ATS systems from previous research. Additionally, the simplified sentences produced by the best performing model are evaluated manually to identify errors and the types of text transformations that the model employs to simplify sentences. When the readability level input value is optimized to maximise model performance on validation data, the readability-controlled models surpass systems from previous works in terms of automatic evaluation metrics, suggesting that the addition of readability level as a control attribute results in improved simplification quality. Manual evaluation shows that readability-controlled models are capable of splitting long sentences to multiple shorter sentences to reduce syntactic complexity of text. This finding suggests that readability level metrics can be used to effectively control syntactic complexity in ATS models as a lightweight alternative to previously applied, more computationally demanding methods that rely on dependency parsing. Finally, this study discusses the different types errors produced by the models, their potential causes and ways to reduce errors in future ATS systems.

Modernit sanaupotusmenetelmät, esimerkiksi Word2vec, eivät mallinna leksikaalista moniselitteisyyttä luottaessaan kunkin sanan mallinnuksen yhden vektorirepresentaation varaan. Näin ollen leksikaalinen moniselitteisyys aiheuttaa ongelmia konekääntimille ja voi johtaa moniselitteisten sanojen käännökset usein harhaan. Työssä tarkastellaan mahdollisuutta mallintaa moniselitteisiä sanoja merkitysupotusmenetelmän (sense embeddings) avulla ja hyödynnetään merkitysupotuksia valvomattoman konekäännösohjelman (unsupervised machine translation) opetuksessa kieliparilla Englanti-Saksa. Siinä missä sanaupotusmenetelmät oppivat yhden vektorirepresentaation kullekin sanalle, merkitysupotusmenetelmän avulla voidaan oppia useita representaatioita riippuen aineistosta tunnistettujen merkitysten määrästä. Näin ollen yksi valvomattoman konekääntämisen perusmenetelmistä, sanaupotusten kuvaus joukosta lähde- ja kohdekielten yksikielisiä vektorirepresentaatioita jaettuun kaksikieliseen vektoriavaruuteen, voi tuottaa paremman kuvauksen, jossa moniselitteiset sanat mallintuvat paremmin jaetussa vektoriavaruudessa. Tämä mallinnustapa voi vaikuttaa positiivisesti konekäännösohjelman kykyyn kääntää moniselitteisiä sanoja. Työssä merkitysupotusmalleja käytetään saneiden alamerkitysten yksiselitteistämiseen, ja tämän myötä jokainen konekäännösmallin opetusaineistossa esiintyvä sane annotoidaan merkitystunnisteella. Näin ollen konekäännösmalli hyödyntää sanaupotusten sijaan merkitysupotuksia oppiessaan kääntämään lähde- ja kohdekielten välillä. Työssä opetetaan tilastollinen konekäännösmalli käyttäen tavanomaista sanaupotusmenetelmää. Tämän lisäksi opetetaan sekä tilastollinen että neuroverkkokonekäännösmalli käyttäen merkitysupotusmenetelmää. Aineistona työssä käytetään WMT-14 News Crawl -aineistoa. Opetettujen mallien tuloksia verrataan aiempaan konekäännöstutkimuksen automaattisessa arvioinnissa hyvin menestyneeseen tilastolliseen konekäännösmalliin. Lisäksi työssä suoritetaan tulosten laadullinen arviointi, jossa keskitytään yksittäisten moniselitteisten sanojen kääntämiseen. Tulokset osoittavat, että käännösmallit voivat hyötyä merkitysupotusmenetelmästä. Tarkasteltujen esimerkkien perusteella merkitysupotusmenetelmää hyödyntävät konekäännösmallit onnistuvat kääntämään moniselitteisiä sanoja sanaupotusmenetelmää hyödyntävää mallia tarkemmin vastaamaan referenssikäännöksissä valittuja käännöksiä. Näin ollen laadullisen arvioinnin kohdistuessa yksittäisten moniselitteisten sanojen kääntämiseen, merkitysupotusmenetelmästä näyttää olevan hyötyä konekäännösmallien opetuksessa.

Tekstin normalisointi on prosessi, jossa epästandardia kirjoitettua kieltä muutetaan standardisoituun muotoon. Murteet ovat yksi esimerkki epästandardista kielestä, joka voi poiketa huomattavastikin standardisoidusta yleiskielestä. Lisäksi suomen kieli on ortografialtaan varsin pitkälti foneemista, minkä ansiosta myös puhutun kielen ominaispiirteet on mahdollista tuoda esille kirjoitetussa muodossa. Etenkin epävirallisilla alustoilla ja arkikielisessä kontekstissa, kuten sosiaalisessa mediassa, suomen kielen puhujat saattavat kirjoittaa sanat kuten ääntäisivät ne normaalisti puhuessaan. Tällaista epästandardista kielestä koostuvaa aineistoa voi löytää myös luonnollisen kielen käsittelyn tarpeisiin esimerkiksi Twitteristä. Perinteiselle yleiskieliselle tekstiaineistolle suunnatut luonnollisen kielen käsittelyn työkalut eivät kuitenkaan välttämättä saavuta toivottavia tuloksia puhekieliselle aineistolle sovellettuna, jolloin ratkaisuna voidaan käyttää välivaiheena tekstin normalisointia. Normalisointiprosessissa syötteenä käytettävä puhekielinen tai muutoin epästandardia kieltä sisältävä teksti muutetaan standardisoituun kirjoitusasuun, jota luonnollisen kielen käsittelyn työkalut paremmin ymmärtävät. Tämä työ pohjaa aiempaan tutkimukseen, jota on tehty suomen murteiden normalisoinnin parissa. Aiemmissa tutkimuksissa on todettu, että merkkipohjaiset BRNN-neuroverkkomallit (Bidirectional Recurrent Neural Nerwork) saavuttavat hyviä tuloksia suomen kielen murteiden normalisoinnissa, kun syötteenä käytetään sanoja kolmen kappaleen lohkoissa. Tämä tarkoittaa, että järjestelmä saa syötteenä kerrallaan kolmen sanan joukon, ja jokainen sana on edelleen pilkottu välilyönnein eroteltuihin kirjoitusmerkkeihin. Tässä työssä pyrittiin käyttämään samoja metodeja ja aineistoa kuin aiemmassa tutkimuksessa, jotta tulokset olisivat vertailukelpoisia. Aineistona on käytetty Kotimaisten kielten keskuksen ylläpitämää Suomen kielen näytteitä -korpusta, ja normalisointiin on käytetty OpenNMT-nimistä avoimen lähdekoodin kirjastoa. Työssä toteutetuista kokeiluista saadut tulokset näyttävät vahvistavan aiempien tutkimustulosten pohjalta tehdyt löydökset, mutta lisäksi on viitteitä siitä, että neuroverkkomallit saattaisivat pidemmistä lohkoista koostuvista syötteistä. BRNN-mallin lisäksi työssä kokeillaan myös muita neuroverkkoarkkitehtuureja, mutta vertailtaessa sanavirheiden suhdelukua mittaavaa WER-arvoa (Word Error Rate) voidaan todeta, että BRNN-malli suoriutuu normalisointitehtävästä muita neuroverkkoarkkitehtuureja paremmin.