The dataset used to create the self-supervised problems is a collection of Tweets collected from the open stream for several years, i.e., the Spanish collection started on December 11, 2015; English on July 1, 2016; Arabic on January 25, 2017; Russian on October 16, 2018; and the rest of the languages on June 1, 2021. In all the cases, the last day collected was June 9, 2023. The collected Tweets were filtered with the following restrictions: retweets were removed; URLs and usernames were replaced by the tokens _url and _usr, respectively; and only tweets with at least 50 characters were included in the final collection. The column Corpus in Table 1 and Figure 1 show the number of tweets collected for the Arabic-speaking countries. The figure shows that there are days when more tweets are collected, and there is a tendency to collect fewer tweets in 2023 due to changes in the Twitter API. The data corresponding to German, English, Spanish, French, Dutch, Portuguese, Russian, Turkish, and Chinese are shown in Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, and Table 10; and Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, and Figure 10.

The corpora are used to create two pairs of training and test sets. The training sets are drawn from tweets published before October 1, 2022, and the test sets are taken from tweets published on or after October 3, 2022. The procedure for creating the set pairs consists of two stages. In the first stage, the tweets were organized by country and then selected to form a uniform distribution by day. Within each day, near duplicates were removed. Then, a three-day sliding window was used to remove near duplicates within the window. The final step was to shuffle the data to remove the ordering by date, respecting the limit between the training and test sets.

The tweets of the first pair were selected to follow a uniform distribution by country as closely as possible. In this pair, the size of the training set is roughly 2 million tweets, whereas the test set size is \(2^{12}\) (4,096) tweets per country. We also produce a smaller training set containing 262 thousand tweets. The procedure is equivalent to the previous one, aiming to have a uniform distribution of the countries. The column identified with the legend train in Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, and Table 10 shows the size of the training set, and the column test indicates the size of the test set in the first pair of training and test sets. It is worth mentioning that we did not have enough information for all the countries and languages to follow an exactly uniform distribution. For example, Table 4 (Spanish) notes that for Puerto Rico (pr), 1,487 tweets in the test set correspond to the total number of available tweets that meet the imposed restrictions.

The second pair of tweets was selected to follow the original distribution of the corpus; in this case, the training and test set has a maximum size of 2 million tweets. The process of selecting the tweets was set as a convex optimization problem where the objective is to maximize the number of tweets subject to a maximum of 2 million (\(2^{21}\)), and the availability of tweets for each country, and the distribution is given by all the tweets available. The column identified with the legend train (orig. dist.) in Table 1, Table 2, Table 3, Table 4, Table 5, Table 6, Table 7, Table 8, Table 9, and Table 10 shows the size of the training set, and the column test (orig. dist.) indicates the size of the test set in the second pair of training and test sets.

DialectId is a text classifier based on a Bag of Words (BoW) representation with a linear Support Vector Machine (SVM) as the classifier.

The normalization procedure used in the BoW corresponds to setting all characters to lowercase, removing diacritics, and replacing usernames and URLs with the tags “_usr” and “_url”, respectively.

The BoW representation weights the tokens with the term frequency and inverse document frequency (TF-IDF). The tokens correspond to words, bi-grams of words, and q-grams of characters (with q=4, 3, 2). The tokens and weights were estimated using each language’s training dataset (2 million tweets). The tokens (vocabulary) with higher frequency in the training set were kept. We developed systems for different vocabulary sizes, i.e., \(2^{17}\), \(2^{18}\), and \(2^{19}.\)

The BoW can be used by importing the BoW class, as seen in the following example, where the good morning text is transformed into a vector space. The first line imports the class, the second line instantiates the class, where the parameter token_max_filter indicates the vocabulary size, and the third line converts the text into a vector space.

Each text in the training set is represented in the vector space, and the associated country is retained for use in a linear SVM using the one-vs-all strategy. The approach creates as many binary classification problems as there are different classes. In the binary problems, each class corresponds to the positive class exactly once, and it is the negative class in the remaining cases. Traditionally, one uses all the information in the approach, which is the case for the reduced training set (262 thousand tweets). However, the negative examples were limited to the maximum number of positive classes or \(2^{14}\) tweets in the whole training set. In both cases, the examples are weighted inversely proportional to class frequencies to treat an imbalanced dataset.

Complementing the previous example, the following code instantiates the DialectId in Spanish using a vocabulary size of \(2^{18}\) indicated by the parameters lang and token_max_filter, respectively.

from dialectid import DialectId
detect = DialectId(lang='es', token_max_filter=2**18)
detect.predict(['comiendo unos tacos'])

array(['mx'], dtype='<U2')

A drawback of using SVM is that it does not estimate the classification probability. For some applications, it is more amenable to calculate the probability instead of the decision-function value. Thus, the developed systems are calibrated to estimate the probability by training a logistic regression using the SVM’s decision function as inputs. The calibration procedure involves predicting the SVM’s decision function on the reduced training set using stratified k-fold cross-validation (k = 3). The decision functions predicted are the inputs of the logistic regression, and the classes are the ones in the reduced training set; the parameters that weight each example inversely proportional to class frequencies are used in this case. To invoke the model using probability, the parameter probability must be set to true, as shown in the following example.

from dialectid import DialectId
detect = DialectId(lang='es', probability=True)
detect.predict_proba(['comiendo unos tacos'])

array([[1.9695617e-06, 4.8897579e-07, 1.8095196e-05, 3.6204598e-05,
        1.5155481e-03, 1.1557303e-05, 5.4983921e-06, 2.8960928e-06,
        2.2128135e-05, 5.0534654e-05, 1.7656431e-01, 2.0665383e-02,
        6.3909459e-01, 1.3323617e-01, 1.0678391e-04, 7.2897665e-06,
        2.6716828e-02, 6.4306505e-08, 1.9328916e-03, 8.0165564e-06,
        2.7745039e-06]], dtype=float32)

As described previously, there are two training sets: one that follows a uniform distribution in the countries as closely as possible, and the second one that follows the distribution seen in the corpus, namely the original distribution (identified as orig. dist.). The parameter uniform_distribution indicates which training set is used to estimate the parameters. By default, the parameter is set to true to use the training sets with uniform distribution in the countries.

from dialectid import DialectId
detect = DialectId(lang='es',
                   uniform_distribution=False,
                   probability=True)
detect.predict_proba(['comiendo unos tacos'])

array([[5.8839246e-06, 4.0277046e-06, 1.5494716e-05, 3.9904633e-05,
        1.5577762e-02, 6.6911220e-05, 9.4599171e-05, 2.6315629e-05,
        1.4272113e-05, 4.7489698e-06, 5.5686768e-02, 2.2258271e-02,
        8.8939697e-01, 1.3783798e-02, 3.9202135e-04, 8.4752110e-06,
        4.6369270e-05, 9.1043790e-07, 2.5635022e-03, 5.8783221e-06,
        7.1929417e-06]], dtype=float32)

The performance of different algorithms is presented in Figure 11 using macro-recall. The best-performing system in almost all cases is DialectId, which is trained on 2 million tweets and has a vocabulary of 500,000 tokens. The exception are Turkish and Dutch, where the best systems is StackBoW trained with only 262k tweets.

The remaining figures provide details on macro-recall by presenting the system’s recall in each country.