DialectId is a text classifier based on a Bag of Words (BoW) representation with a linear Support Vector Machine (SVM) as the classifier.

The normalization procedure used in the BoW corresponds to setting all characters to lowercase, removing diacritics, and replacing usernames and URLs with the tags “_usr” and “_url”, respectively.

The BoW representation weights the tokens with the term frequency and inverse document frequency (TF-IDF). The tokens correspond to words, bi-grams of words, and q-grams of characters (with q=4, 3, 2). The tokens and weights were estimated using each language’s training dataset (2 million tweets). The tokens (vocabulary) with higher frequency in the training set were kept. We developed systems for different vocabulary sizes, i.e., \(2^{17}\), \(2^{18}\), and \(2^{19}.\)

The BoW can be used by importing the BoW class, as seen in the following example, where the good morning text is transformed into a vector space. The first line imports the class, the second line instantiates the class, where the parameter token_max_filter indicates the vocabulary size, and the third line converts the text into a vector space.

Each text in the training set is represented in the vector space, and the associated country is retained for use in a linear SVM using the one-vs-all strategy. The approach creates as many binary classification problems as there are different classes. In the binary problems, each class corresponds to the positive class exactly once, and it is the negative class in the remaining cases. Traditionally, one uses all the information in the approach, which is the case for the reduced training set (262 thousand tweets). Nonetheless, in the full training set, the negative examples were limited to the maximum number of positive elements or \(2^{14}\) tweets. In both cases, the examples are weighted inversely proportional to class frequencies to treat an imbalanced dataset.

Complementing the previous example, the following code instantiates the DialectId in Spanish using a vocabulary size of \(2^{18}\) indicated by the parameters lang and token_max_filter, respectively.

from dialectid import DialectId
detect = DialectId(lang='es', token_max_filter=2**18)
detect.predict(['comiendo unos tacos'])

array(['mx'], dtype='<U2')

A drawback of using SVM is that it does not estimate the classification probability. For some applications, it is more amenable to calculate the probability instead of the decision-function value. Thus, the developed systems are calibrated to estimate the probability by training a logistic regression using the SVM’s decision function as inputs. The calibration procedure involves predicting the SVM’s decision function on the reduced training set using stratified k-fold cross-validation (k = 3). The decision functions predicted are the inputs of the logistic regression, and the classes are the ones in the reduced training set; the parameters that weight each example inversely proportional to class frequencies are used in this case. To invoke the model using probability, the parameter probability must be set to true, as shown in the following example.

from dialectid import DialectId
detect = DialectId(lang='es', probability=True)
detect.predict_proba(['comiendo unos tacos'])

array([[1.9695617e-06, 4.8897579e-07, 1.8095196e-05, 3.6204598e-05,
        1.5155481e-03, 1.1557303e-05, 5.4983921e-06, 2.8960928e-06,
        2.2128135e-05, 5.0534654e-05, 1.7656431e-01, 2.0665383e-02,
        6.3909459e-01, 1.3323617e-01, 1.0678391e-04, 7.2897665e-06,
        2.6716828e-02, 6.4306505e-08, 1.9328916e-03, 8.0165564e-06,
        2.7745039e-06]], dtype=float32)

As described previously, there are two training sets: one that follows a uniform distribution in the countries as closely as possible, and the second one that follows the distribution seen in the corpus, namely the original distribution (identified as orig. dist.). The parameter uniform_distribution indicates which training set is used to estimate the parameters. By default, the parameter is set to true to use the training sets with uniform distribution in the countries.

from dialectid import DialectId
detect = DialectId(lang='es',
                   uniform_distribution=False,
                   probability=True)
detect.predict_proba(['comiendo unos tacos'])

array([[5.8839246e-06, 4.0277046e-06, 1.5494716e-05, 3.9904633e-05,
        1.5577762e-02, 6.6911220e-05, 9.4599171e-05, 2.6315629e-05,
        1.4272113e-05, 4.7489698e-06, 5.5686768e-02, 2.2258271e-02,
        8.8939697e-01, 1.3783798e-02, 3.9202135e-04, 8.4752110e-06,
        4.6369270e-05, 9.1043790e-07, 2.5635022e-03, 5.8783221e-06,
        7.1929417e-06]], dtype=float32)

The performance, using macro-recall, of DialectId with different parameters and StackBoW (Graff, Moctezuma, and Téllez (2025), Graff et al. (2020)), used as baseline, is presented in Table 11 and Table 12. Table 11 shows the performance on the test set that has at most 2048 in all countries, and Table 12 presents the performance on the test set that follows the original distribution across countries.

The notation used is as follows: the number in brackets indicates the vocabulary size, the systems with the number 626k in parentheses show the performance of the systems trained with the small training set. The system identified with the label Orig. Dist. indicates that it is trained on the training set that follows the original distribution; the rest of the systems are trained with the training set that has a uniform distribution across countries.

It can be observed in Table 11 that DialectId outperforms the baseline (StackBoW) in almost all languages except in Dutch and Turkish. It is essential to note that the training set size of these languages is less than 600k tweets. DialectId trained with the uniform distribution training set outperformed the system trained with the original distribution training set; this behaviour is expected because the original distribution training set provides fewer examples in the minority classes; however, macro-recall gives the same weight to all classes. DialectId with a vocabulary size of \(2^{19}\) obtained the best performance in Spanish, English, Arabic, French, Portuguese, Russian, and Chinese. DialectId using a vocabulary size of \(2^{18}\) obtained the best performance in German; this could be the result that the training set size is 94k tweets, which might not be enough to train a greater vocabulary size. The other language with fewer examples is Chinese; in this case, the difference in performance of DialectId with \(2^{18}\) and \(2^{19}\) is not statistically significant, as can be seen in Figure 11.

DialectId aims to estimate the likelihood of origin of a text; one of its applications could be to calculate the distribution of dialects of a set of texts. The performance has been presented using macro-recall; however, this measure does not provide information about the closeness of the distribution computed with DialectId. To provide information, Table 13 presents the Pearson correlation coefficient in the test set, which follows the original distribution of the DialectId with a vocabulary size of \(2^{19}\) trained with the two training sets. It can be observed that in all the countries, the correlation is above 0.9. The lowest value is in Spanish, where the system trained with the uniform distribution achieved 0.9063, while the other system achieved 0.9824. DialectId trained with the original distribution has correlation coefficients above 0.98 in all the cases; however, it is the system with the lowest macro-recall in all the cases. The table includes the system DialectId(Bias), which is equivalent to DialectId trained on the uniform distribution, with the difference that the probabilities are weighted by the proportion of each country measured in the training set of the original distribution. It can be observed that in all the cases, DialectId(Bias) has a correlation greater than 0.97.

To complement the information presented in Table 13, Figure 12, Figure 13, Figure 14, Figure 15, Figure 16, Figure 17, Figure 18, Figure 19, Figure 20, and Figure 21 present these distributions for Arabic, German, English, Spanish, French, Dutch, Portuguese, Russian, Turkish, and Chinese; all the figures follow an equivalent notation. For example, Figure 15 shows in the blue line the distribution measured in the test set, the broad orange line presents the distribution obtained with the prediction made by DialectId trained in the uniform distribution, and the wide green line presents the distribution obtained with the DialectId (trained with the original distribution) predictions.

The figure also includes, in thin lines, estimated distributions, with the two versions of DialectId, from a dataset where there is no geographic information, so it is impossible to measure the actual distribution. The dataset comes from the same period as the test set, and it follows a treatment equivalent to the test set, such as the near duplicates are removed among other constraints. It can be observed that the thin lines follow the wide lines in almost all countries, except in the Dominican Republic. These later distributions (i.e., thin lines) show one of the applications of DialectId, which is to estimate the dialect of texts from a collection where the information is not available.