green logo
green logo

Welsh National Language Technologies Portal

Data

Welsh place-names

Dyma rhestr o enwau yng Nghymru a ddefnyddir yn y Gymraeg, a rhestr sydd hefyd yn cynnwys gwybodaeth hydred a lledred yn ogystal ag enwau Saesneg sy’n cyfateb.

Corpora

Many Welsh language (and bilingual) corpora, such as the CEG corpus and the Proceedings for the National Assembly of Wales, are available through the Welsh National Corpus Portal. The Welsh National Corpus Portal allows the user to easily search and find examples of the use of words and terms in many contexts.
Below, we have provided links for downloading corpus data from some of the corpora from the Welsh National Corpus Portal as well as links to data that we would not be able to provide otherwise.

Here are corpora in the form of audio files used to aid development of various speech technologies:

A corpus of Welsh language sentences tagged with part-of-speech tags and released under CC0 to enable the training of statistical part-of-speech taggers for Welsh.

Corpus of CC0 Sentences

This is a corpus of Welsh language sentences released under a CC0 licence collected by members of the Language Technologies Unit, Bangor University, expressly to serve as prompts for developing Welsh speech recognition. The sentences come from various CC0 sources.

This is a collection of 14,857 sentences released under a CC0 licence. They were collected by members of the Language Technologies Unit, Bangor University, expressly to serve as prompts for Welsh Speech Recognition. The sentences come from various CC0 sources and include:

  • Original sentences
  • Sentences from novels, essays and other out of copyright material
  • Sentences from the Welsh Wicipedia where authors gave us permission to release them under a CC0 licence
  • Tweets, emails, and other electronic material gifted to the project to be used as prompts

In a number of cases, the language was adapted and the sentences heavily edited to make them suitable for reading aloud by volunteers.

The corpus was also given to the Mozilla Common Voice project, and these sentences were therefore used to record volunteers.

We wish to thank everyone who helped us collect these sentences, including those who gave us their materials under a CC0 licence, and to Mozilla for their help and leadership with the Common Voice project.

Download ‘Brawddegau Cymraeg’ resource from GitHub.

Lexicons

Welsh Pronunciation Dictionary

The Welsh Pronunciation Dictionary is a dictionary suitable for use with speech technology. It is hosted by Bangor University’s School of Linguistics and Bangor University’s Language Technologies Unit.

Hunspell

Hunspell is an open source spell checker used in a number of software applications.

In October 2020, we revised and significantly updated Hunspell, and we continue to update it regularly. The latest update includes new prime forms (‘actiwari’, ‘biodreulio’ a ‘seiberfwlio) and 98 additional international place names (including ‘Irac’).

The Hunspell files can be found on our GitHub page click on the green ‘code’ button then select ‘Download Zip’ to download.

We offer a separate version of our Hunspell data, specifically designed for Y Trawsgrifiwr, contributing to Bangor University Language Technologies Unit’s transcription bank. This version includes spoken forms that align with the project’s verbatim transcription conventions (see Bangor Transcription Conventions for details). Examples of included spoken forms are ch’mod, rwbath, sicir, and gweud. These Hunspell files are available on our GitHub page.

To install Hunspell within LibreOffice, follow the instructions in the README.

Wordlists of the most common wordforms in Welsh, and the most common English words used in Welsh

These wordlists are intended to help improve Welsh speech technology by identifying the most common words likely to be uttered for processing in a Welsh language transcription system.

Our most frequently written Welsh and English wordlists are available on our GitHub site.

A requirement of the Welsh Government funded “Macsen” Speech Recognition project was that we publish wordlists of the 2,500 most common words written in Welsh, and the 500 most common English words used in Welsh. These wordlists are intended to help improve Welsh speech technology by identifying the most common words likely to be uttered for processing in a Welsh language transcription system. These wordlists will be used to test the ability of our prototype transcription once it is ready, and measure its ability to recognise these common words.

Other projects, such as that to develop the Bangor Siarad Corpus (http://bangortalk.org.uk/speakers.php?c=siarad), have followed lexical principles in assigning words to either Welsh or English categories, i.e. their evaluation of what is a ‘Welsh’ or an ‘English’ word has been based on their attestation or otherwise in Welsh or English dictionaries. We have added other principles for our wordlists, based on whether their pronunciation follows Welsh or English letter-to-sound rules, as we are specifically interested in improving the recognition and transcription of spoken Welsh.

Wordlists are often thought of as lists of canonical forms, or the headwords found in dictionary entries. However, in a language such as Welsh, which has many inflected forms, and in contrast to English which has far fewer inflections, we decided to publish lists of the wordforms and their frequency counts, rather than the canonical forms or lemmas. This will help identify the most common forms, as some wordforms are much more common than their lemma, e.g. ‘mae’ (3rd person singular of the verb ‘to be’), is much more common that its lemma ‘bod’. Sometimes more than one wordform deriving from the same lemma comes into the 2,500 top most commonly used words, e.g. the lemma ‘Mehefin’ (‘June’) and the wordform ‘Fehefin’ with its soft mutation, both come into the list of the top 2,500 wordforms in Welsh.

Welsh makes much use of apostrophes to merge two words and indicate missing vowels. Wordforms that include apostrophes are regarded as legitimate wordforms in the Welsh wordlist, since doing so will help Welsh speech recognition and the creation of a Welsh transcription system. As illustrations, the wordform ‘hi’n’ occurs 1,117719 times in the corpus, and the wordform ‘a’i’ occurs 1,05456 times. For the avoidance of ambiguity, apostrophes are shown as underscores in the wordlist, therefore ‘hi_n’ and ‘a͏_i’ are the forms shown in the Welsh wordlist.

In order to gather objective evidence for the word frequency counts, we needed a Welsh corpus or corpora of sufficient size, if possible in the form of a balanced corpus. The largest Welsh corpus available for us was the Cysill Ar-lein Corpus (Prys, Prys a Jones 2016), which has by now reached over 200 million words and which continues to grow. It was created from texts input by users in order to check their spelling and grammar on-line, between 2009 and the present. It was not therefore intended as a balanced corpus, but it contains a great variety of text types, as Wooldridge’s (2011) analysis of it demonstrates:

categories of texts in the Cysill ar-lein corpus ” class=”wp-image-826 alignnone size-full”></p> <p>The CEG Corpus (Ellis et al, 2001), was also examined, but although designed from the outset as a balanced corpus, it only contains one million words. Also, the original texts were collated during the early 1990s, and the vocabulary of Welsh has changed a great deal since then, with new words and concepts such as ‘rhyngrwyd’ (‘internet’) coming into the language. The most common words in Welsh according to the CorcenCC project were also examined. This corpus will eventually contain 10 million words in a balanced corpus across different modalities, both written and spoken (http://www.corcencc.cymru/). This project is still ongoing, and at present there are only 3,597 unique wordforms and 14,876 in its demo list of wordforms, making for a comparatively small sample. </p> <p>The most common wordforms at the top of the lists were compared across the three corpora, and were found to be markedly similar, with more divergence appearing further down. This helped confirm that the methodology used was basically sound. The Cysill Ar-lein Corpus was therefore used as the source of the Welsh wordforms, with the data being taken from the January 2019 version of the corpus. </p> <p>Finding a source for the most common English words used in Welsh was more of a challenge. We examined the Siarad Corpus (Deucher et al, 2009), which is a speech corpus of spoken dialogue created explicitly to study code switching between Welsh and English. It contains 447,507 words, and an estimated 84% of the corpus is in Welsh, with 4% in English, with 13% undetermined. It was transcribed by hand, and the English words in it are tagged as English. We conducted a few experiments with this corpus, but the sample proved to be too small for a meaningful list of the 500 most common words in Welsh. Also, since it was in essence a speech corpus, the register used was very informal, in contrast to what is used in written texts, even informal written texts such as Twitter messages. </p> <p>On the other hand, we observed that there were numerous English words in the Cysill Ar-lein Corpus. Many of them come from English sentences appearing in the middle of Welsh text, but after having excluded those from the list, a substantial number of English words remained, although they only formed a small part of the corpus. Compare, for example, the frequency count of 10,952,674 of the most common word in Welsh, namely ‘yn’, with 7,361 occurrences for the most common English words in the corpus, namely ‘the’ (note however that it is possible that some examples of the Welsh word ‘te’ (‘tea’) with the aspirate mutation are mixed in here). There were 11 English words each occurring 107 times in the Cysill Ar-lein Corpus, this being the lowest frequency wordcount to come within the 500 English words limit. </p> <p>The Cysill Ar-lein Corpus, January 2019 version, was also used to collate the wordlist of the most common English words in Welsh.</p> <p>Where there are orthographic variations in Welsh, with more than one form accepted as legitimate, all legitimate forms coming within the 2,500 word limited were included in the list. For example, ‘siŵr’ (with the circumflex accent) occurs 54,228 times in the corpus, and ‘siwr’ (without the circumflex accent) occurs 24,009 times. Both forms are correct according to the University of Wales Welsh Dictionary. Therefore both forms were included since both occur with the 2,500 most frequently used Welsh words. </p> <p>In the wordlists, each wordform has the number of times it occurs in the Cysill Ar-lein Corpus listed alongside.</p> <p>In order to identify the Welsh words, and any words that should be classified as English words in the Cysill Ar-lein Corpus, the Hunspell Welsh and English lexicons were used for comparison. It was noted that some words could be both Welsh and English (e.g. ‘plant’), and if so it was presumed that they were more likely to be Welsh words in a corpus derived from Welsh language materials and tagged as such. </p> <p>We used Langdetect CLD2 Google components to identify English phrases and sentences in the Cysill Ar-lein Corpus and to exclude them so as not to contaminate the data on individual words used in code switching. SRILM language modelling software was also used to count ngrams so if there were ambiguous words which could be either Welsh or English it was possible to see what was the language of the word in front of it. If that language was English, then it was more likely that the ambiguous word was also English, and we counted it as such, even though ambiguous words were otherwise counted as Welsh ones. </p> <p>The words that made it to the top of the English list were also the most frequent English words in English in general, e.g. ‘the’, ‘of’, ‘and’. Many of these may come from named entities in the corpus, such as names of films and organisations, e.g. <i>The Lion King, The Royal Society</i>, and these were retained. </p> <p>We do not claim that these methods have succeeded in every instance to identify Welsh and English words correctly. For example, as noted above, ‘the’ can be the mutated form of the Welsh word ‘te’, or ‘of’ above can be the mutated form of the Welsh word ‘gof’ (‘blacksmith’). However in the above contexts the English ‘the’ and ‘of’ were judged to be more common since they correlated with the cluster of most common English words rather than mutated form of ‘gof’ a ‘te’. On the other hand, the ambiguous ‘to’ was taken out of the English list and placed in the Welsh list (there are 7,990 examples of ‘to’ in the Cysill Ar-lein Corpus) because, on examining the collocations, the Welsh word ‘to’ (corresponding to English ‘roof’), the Welsh word was by far the most common occurrence. </p> <p>Editorial judgement was used to move some named entities that had been tagged as Welsh by Hunspell, especially proper nouns (e.g. Jones, Williams) to the English list as they are pronounced according to English phonetic conventions. The basic principle used was that the Welsh list had to conform to how Welsh is pronounced according to Welsh letter-to-sound rules, and the English list had to conform to the letter-to-sound rules for English. </p> <p>This also meant that some loanwords from English, perceived, according to some interpretations, to be part of code-switching between Welsh and English, are included as Welsh words. In the Cysill Ar-lein Corpus, the spelling of such words had been Welshified, e.g. ‘plîs’ (English: ‘please’), and ‘jyst’ (English: ‘just’). If this was the case, they were accepted as Welsh words for the purpose of these wordlists. </p> <p>Some ambiguous words were not recognised as either Welsh or English because they did not occur in either of the two Hunspell lexicons. These were edited by hand and included in the correct list if they had the requisite word frequency count for inclusion. </p> <p>Numbers (e.g. ‘2’) were excluded from the wordlists, unless they were written out fully as words, e.g. ‘dau’ or ‘dwy’ (masculine and feminine forms). All acronyms and abbreviations were also excluded, as any transcribing system would not be able to deal with these without further tokenization work. </p> <p>There are many named entities in both the Welsh and the English wordlists. These are usually written with an initial capital letter in both languages. These were converted to lower case letters as the data was processed. An attempt was made to restore capitalization in named entities by comparing with Hunspell, which is case-sensitive. However, many words can be both common nouns and named entities, e.g. ‘gwyn’ a ‘Gwyn’ and ‘heather’ a ‘Heather’. In such cases, the common noun was prioritized over the named entity. There may well be instances where the named entity is the most common form, e.g. with ‘may’ in English, it is possible that the name of the former prime minister, Mrs May, is what is meant in many of the examples, or the month of May, as names of months also appear in the corpus. </p> <p>Lastly, we decided to check the wordlists for any words with derogatory connotations, which should be removed from a publically available resource. We are happy to report that only one word needed to be excluded for this reason. </p> <h4>References</h4> <p>Prys, D., Prys G., and Jones, D.B. (2016) <em>Cysill Ar-lein: A Corpus of Written Contemporary Welsh Compiled from an On-line Spelling and Grammar Checker.</em> Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016) Portoroz, Slovenia. <a href=Paper

Wooldridge, D. (2011) Gwella Cysill at Ddefnydd Cyfieithwyr: adnabod ymyrraeth gan yr iaith Saesneg mewn testunau Cymraeg. Traethawd MRes, Prifysgol Bangor. Dissertation

Ellis, N. C., O’Dochartaigh, C., Hicks, W., Morgan, M., & Laporte, N. (2001) Cronfa Electroneg o Gymraeg (CEG) Website

Knight, D. et al (2019) Corpws Cenedlaethol Cymraeg Cyfoes (CorCenCC) Website

Deuchar, M. et al (2009) Corpws Siarad Bangor Website

Chan, D., and Jones, D.B. Hunspell Cymraeg (2013) Website

Delyth Prys
Dewi Bryn Jones
September 2019

This site is registered on wpml.org as a development site. Switch to a production site key to remove this banner.