CompStats

Bootstrap-based statistical comparison of supervised-learning systems evaluated on a shared test set

CompStats is an open-source Python package for the statistical analysis of the results of supervised-learning systems evaluated on a shared held-out test set, the setting of collaborative competitions (also called challenges). From the gold standard and the predictions of each system, it estimates by bootstrap the standard error and the confidence interval of every performance measure, tests whether the difference between the best system and each of the others is statistically significant, and corrects those tests for multiple comparisons.

Competitions rank systems by a point estimate computed on a single test set. The differences between consecutive systems are often in the hundredths or thousandths. Consequently, a higher score does not imply that a system would remain better on the population from which the test set was sampled. Moreover, the statistical tests for comparing learning algorithms require access to the underlying algorithm (Dietterich, 1998), whereas a competition gives access only to the predictions, and the procedures for comparing classifiers over multiple datasets (Demšar, 2006) do not apply because a competition involves a single dataset.

CompStats implements the methodology of Nava-Muñoz et al. (2024), Analysis of systems’ performance in natural language processing competitions (Pattern Recognition Letters). The methodology is designed to complement the standard winner selection process, which in most cases relies solely on the ranking of the scores, with well-established statistical tools and graphics. It also provides measures that allow organizers to assess the difficulty of a competition. It is designed to be universally applicable. The article illustrates it with eight natural language processing competitions as case studies, involving classification and regression problems.

Follows the scikit-learn metrics API

A function of CompStats.metrics takes (y_true, y_pred) like its sklearn.metrics namesake and returns a Perf object instead of a float.

Uncertainty for every measure

The standard error and the percentile confidence interval are computed from the bootstrap distribution of the measure.

Paired significance testing

Every system is evaluated on the same bootstrap samples, so the differences are paired. The p-values come from the bootstrap distribution of the difference, with optional Bonferroni, Holm or Benjamini-Hochberg correction.

Quickstart

The package is installed with pip install CompStats or with conda install --channel conda-forge CompStats. See Install for details. The following example compares three classifiers on the digits problem.

from sklearn.datasets import load_digits
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import GaussianNB
from sklearn.svm import LinearSVC
from CompStats.metrics import f1_score

X, y = load_digits(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.3, random_state=0)

svm = LinearSVC(random_state=0).fit(X_train, y_train)
score = f1_score(y_val, svm.predict(X_val), average='macro', name='Linear SVM')
score(RandomForestClassifier(random_state=0).fit(X_train, y_train).predict(X_val),
      name='Random Forest')
score(GaussianNB().fit(X_train, y_train).predict(X_val), name='Naive Bayes')

score                 # point estimate and standard error of every system
score.ci              # 95% percentile bootstrap confidence intervals
diff = score.difference()            # paired differences against the best system
diff.p_value(correction='holm')      # p-values adjusted for multiple comparisons
score.plot()                         # performance with confidence intervals

Where to go next

Install describes the requirements and the installation methods, and Methodology states the statistical basis of the package. The tutorials are scikit-learn Users, Competition Organizers and Multiple Testing. These three pages are Jupyter notebooks that can be opened in Colab from the badge at their top. Citing gives the references to cite, and the architecture and the complete API are in the developer docs.

Colophon

As with the rest of the project, this site was built by the CompStats team together with Claude, Anthropic’s coding assistant, through Claude Code. The assistant helped set up the Quarto site and draft this page, with the authors reviewing and validating every change before it was committed.

References

Demšar, Janez. 2006. “Statistical Comparisons of Classifiers over Multiple Data Sets.” Journal of Machine Learning Research 7 (1): 1–30. http://jmlr.org/papers/v7/demsar06a.html.
Dietterich, Thomas G. 1998. “Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms.” Neural Computation 10 (7): 1895–1923. https://doi.org/10.1162/089976698300017197.
Nava-Muñoz, Sergio, Mario Graff, and Hugo Jair Escalante. 2024. “Analysis of Systems’ Performance in Natural Language Processing Competitions.” Pattern Recognition Letters 186: 346–353. https://doi.org/10.1016/j.patrec.2024.03.010. Preprint: https://arxiv.org/abs/2403.04693.