The dataset used to create the self-supervised problems is a collection of Tweets collected from the open stream for several years, i.e., the Spanish collection started on December 11, 2015; English on July 1, 2016; Arabic on January 25, 2017; Russian on October 16, 2018; and the rest of the languages on June 1, 2021. In all the cases, the last day collected was June 9, 2023. The collected Tweets were filtered with the following restrictions: retweets were removed; URLs and usernames were replaced by the tokens _url and _usr, respectively; and only tweets with at least 50 characters were included in the final collection.

The corpora are divided into two sets: the first set is used as a training set, i.e., to estimate the parameters, while the second set corresponds to the test set, which could be used to measure the model’s performance. The basis for this division is a specific date, with tweets published before October 1, 2022, forming the first set. Those published on or after October 3, 2022, are being used to create the test set.

The procedure has two stages. Two datasets were created for each country and language in the first stage. The first one contains \(2^{23}\) (8 million) tweets, and the second has \(2^{12}\) (4,096) tweets; the former will be used to create the training set, and the latter corresponds to the test set. These two sets were constructed using tweets with geographic information and filtered according to the language information provided by Twitter. Each set was meticulously crafted to follow, as closely as possible, a uniform distribution of the days. Within each day, near duplicates were removed. Then, a three-day sliding window was used to remove near duplicates within the window. The final step was to shuffle the data to remove the ordering by date.

In the second stage, training sets are created for each language. Each training set contains \(2^{21}\) (2 million) tweets. The procedure used to develop the training consists of drawing tweets from the sets created in the first stage, which have a size of \(2^{12}\). The sampling procedure aims to develop training sets that follow a uniform distribution by country. We also produce a smaller training set containing \(2^{18}\) (262 thousand) tweets. The procedure is equivalent to the previous one; the aim is to have a uniform distribution of the countries.

It is worth mentioning that we did not have enough information for all the countries and languages to follow an exactly uniform distribution. For example, it can be observed in Table 4 (Spanish) that for Puerto Rico (pr), there are only 12,407 tweets in the training set and 1,487 tweets in the test set, which correspond to the total number of available tweets that met the imposed restrictions.

The performance of different algorithms is presented in Figure 1 using macro-recall. The best-performing system in almost all cases is DialectId, which is trained on 2 million tweets and has a vocabulary of 500,000 tokens. The exception are Turkish and Dutch, where the best systems is StackBoW trained with only 262k tweets.

The remaining figures provide details on macro-recall by presenting the system’s recall in each country.