Creating a Multilingual Corpus from Wikipedia Files

Luke Lindemann

February 19, 2021

Wikipedia currently has versions in over 300 different languages from about forty different language families and more than thirty different scripts. This makes it ideal for the collection of sample texts for multilingual corpora. I have collected and processed Wikipedia entries for every language version that contains 100+ articles. I limited each sample to 200,000 words. The fully cleaned text corpus (110 MB) can be downloaded here, and the raw extracted files (130 MB) can be downloaded here. Basic information about each language is in a chart at the end of this document. Below, I detail the process for creating a text corpus from Wikipedia.

Processing Wikipedia Files

The most recent text of any Wikipedia version can be downloaded as a Wikimedia dump file here. The complete list of languages and the Wikicodes for each language are here. The dump files extract to xml with Wikipedia mark-up code. In the past, I have used the WikiCorpus function from the Python Gensim module to process the texts. I believe this method works well for building a single massive English corpus, but I have been unable to use it adequately for my purpose - building a corpus of smaller sample texts from hundreds of dump files in different languages.

I typically only want to extract the first 200 or so articles from a file rather than process the entire file, which can run multiple hours for languages with millions of articles like Spanish and English. Processing all 300+ wikipedia versions has taken me multiple days in the past. Furthermore, for many languages the processing fails to remove much of the markup code (especially text formatting), tables, and long irrelevant lists of topics or languages at the bottom of an article. As far as I am aware, the cleaning is done entirely through regular expressions, which is inadequate for handling recursively-nested templates. Additionally, it deletes characters in some scripts, particularly Brahmi-derived Indic scripts like Devanagari.

I wrote a Python program, WikiTextExtract, which is available here, to extract text from a folder containing Wikipedia dump files. My priority is to capture long stretches of running text, so it is aggressive in filtering out markup code, lists, tables, and templates. You can set the maximum number of lines to read, which allows a portion of the file to be processed quickly, as well as a maximum number of either full articles or words to extract, and it can filter out short articles that pass a minimum word threshold. Each article is listed separately on a metadata line along with its title and the number of words in the article.

For this corpus, I set a limit of 200,000 words for each language (or, for languages with scripts that do not use spaces for words - Burmese, Chinese, Classical Chinese, Dzongkha, Gan, Japanese, Khmer, Lao, Mon, Shan, Thai, Tibetan, Yue - the first 100 articles). About half of the versions contain fewer than 200,000 words total. Each accepted article must be 100 words or longer, to avoid articles which contain formulaic entries and lists (for a small number of languages which only contain short entries - Cheyenne, Cree, Inuktitut, Inupiak, Kongo, Greenlandic, Kashmiri, Lak, Madurese, Nauruan, Norfolk, Kirundi - I lowered the threshold to ten words and deleted repetitive entries by hand). I then did a quick eye scan through the resulting files and deleted articles or portions of text in irrelevant languages, as well as remaining lists and headings (often I deleted the head article because it does not contain encyclopedia-style running text).

Cleaning the Texts

WikiTextExtract does not clean the text for processing. In other words, the resulting text leaves intact the uppercase letters, punctuation, indentation, etc. This is because I wanted it to be easy to read the extracted articles. For the cleaned texts, I used R to scan the text (ignoring the titles and word counts), making everything lowercase, and (for the Latin, Greek, Cyrillic, Arabic, and Hebrew scripts) deleting rare characters which occur with a less than .01% frequency.

I then restricted the text to characters that occur within the Unicode block of that language's script so that, for example, a Cyrillic text is cleaned of Latin and Arabic characters used in loanwords or quoted text. Six Wikipedia versions use multiple scripts (Cree, Tatar, Buginese, Pali, Inupiak, and Kashmir). For these languages, about half the articles are written in one script and half are written in the other. For these I created two separate texts, restricting by one script and then the other.

Deleting punctuation is trickier than using a [[:punct:]] regular expression, because many languages use punctuation symbols in their main orthographies. A noteworthy example is the apostrophe, and variant symbols, which are used phonemically in the Latin-based scripts of Acehnese, Aymara, Buginese, Central Bicolano, Chamorro, Cheyenn, Fula, Gorontalo, Guarani, Hawaiian, Igbo, Lojban, Madurese, Navajo, Malagasy, Maori, Nias, Quechua, Samoan, Tatar, Tetum, Uzbek, and Xhosa. There are also languages which use punctuation symbols in digraphs, including Breton and Catalan.

Update from January 24, 2024: The author of this blog post helpfully pointed out that the [[:punct:]] regular expression deletes characters (\u104c-\u104f) in the Burmese text. I have updated that text to fix the issue.

For the other scripts, I did my best to research the punctuation and numeral characters for each script so that I could exclude them. I hope to have created a fairly simple and effective method for extracting text from Wikipedia pages.

The chart below gives basic information about each language.