CompStats

https://img.shields.io/badge/GitHub-CompStats-black?logo=github https://github.com/INGEOTEC/CompStats/actions/workflows/test.yml/badge.svg https://coveralls.io/repos/github/INGEOTEC/CompStats/badge.svg?branch=develop https://badge.fury.io/py/CompStats.svg https://dev.azure.com/conda-forge/feedstock-builds/_apis/build/status/compstats-feedstock?branchName=main https://img.shields.io/conda/vn/conda-forge/compstats.svg https://img.shields.io/conda/pn/conda-forge/compstats.svg

Version 0.2.0.

Note

This is the developer reference of CompStats: the architecture of the package and the complete API. Installation, tutorials and the description of the methodology for users are published on the project site, https://ingeotec.github.io/CompStats/.

Introduction

Collaborative competitions (challenges) evaluate systems on a single held-out test set whose gold standard is kept by the organizers. The systems are ranked by a performance measure computed on that set. The organizers have access only to the predictions, to a fixed dataset size and to a small number of submissions; consequently, the classical statistical tests that require access to the algorithm, or that require performance estimates on several datasets, do not apply.

CompStats implements a bootstrap-based methodology that complements the ranking with standard errors, confidence intervals, significance tests of paired differences and corrections for multiple comparisons. The same setting covers any comparison of supervised-learning systems on a shared test set, whether the task is classification or regression. The methodology is described in Analysis of systems’ performance in natural language processing competitions (doi:10.1016/j.patrec.2024.03.010), which extends Comparison of Classifiers in Challenge Scheme (doi:10.1007/978-3-031-33783-3_9).

Statistical background

Each concept below is stated with its formula and the symbol of the package that computes it.

Setting. Let \(y\) be a gold standard of size \(n\), let \(\hat{y}_1, \ldots, \hat{y}_m\) be the predictions of \(m\) systems, and let \(s(y, \hat{y})\) be a performance measure. A measure is either bigger-is-better (a score) or smaller-is-better (an error); the flag BiB carries that direction.

Bootstrap. \(B\) samples (num_samples, default 500) of \(n\) indices are drawn with replacement from \(\{1, \ldots, n\}\). The measure is evaluated on every sample for every system, which gives the bootstrap distribution \(s_j^{*1}, \ldots, s_j^{*B}\) of system \(j\) (StatisticSamples). The same index samples are used for all the systems, which is what makes the comparisons paired.

Standard error. The standard deviation of the bootstrap distribution,

\[\widehat{\mathrm{SE}}_j = \sqrt{\frac{1}{B} \sum_{b=1}^{B} \left(s_j^{*b} - \bar{s}_j^{*}\right)^2}, \qquad \bar{s}_j^{*} = \frac{1}{B} \sum_{b=1}^{B} s_j^{*b},\]

computed by SE and exposed as the se property of Perf.

Confidence interval. The percentile interval

\[\left[ q_{\alpha/2}(s_j^{*}),\; q_{1-\alpha/2}(s_j^{*}) \right],\]

where \(q_p\) is the \(p\)-th quantile of the bootstrap distribution and \(\alpha = 0.05\) by default. It is computed by CI and exposed as the ci property of Perf.

Paired difference. For a reference system \(A\) (by default the best one) and another system \(B\), the bootstrap distribution of

\[\delta^{*} = s^{*}(y, \hat{y}_A) - s^{*}(y, \hat{y}_B)\]

is computed on the same samples. The sign is flipped for smaller-is-better measures, so that a positive difference always means that the reference is better. It is built by difference, which returns a Difference.

Significance. Let \(\delta(x)\) be the difference observed on the test set. The p-value of the null hypothesis \(H_0\): “the reference is not better” is estimated as the fraction of bootstrap differences that are at least \(2\delta(x)\),

\[\hat{p} = \frac{1}{B} \sum_{b=1}^{B} \mathbb{1}\left[\delta^{*b} \geq 2\delta(x)\right],\]

(p_value), following Berg-Kirkpatrick, Burkett and Klein (2012). The bootstrap distribution is centred on \(\delta(x)\) rather than on zero, hence the factor 2.

Multiple comparisons. Comparing the best system against the other \(m - 1\) systems tests several hypotheses at once. The p-values can be adjusted with any method of statsmodels.stats.multitest.multipletests through the correction argument (for example 'bonferroni', 'holm' or 'fdr_bh'). When several measures are evaluated, each measure is corrected as its own family.

Competitiveness of a competition. The legacy functional API (performance_multiple_metrics) summarizes the competitiveness of a competition with the coefficient of variation of the systems’ performance,

\[CV = 100 \, \frac{s_x}{\bar{x}},\]

where \(s_x\) is the sample standard deviation and \(\bar{x}\) the mean of the systems’ performance, and with the distance between the winner and the system in the middle of the table. The potential for improvement of a task is assessed separately with the possible percentage improvement, \(PPI = 100\,(1 - x_{best})\), for bigger-is-better scores bounded by 1.

Architecture

CompStats is organized around one core data flow: raw predictions from one or more systems are turned into bootstrap-resampled statistics, from which derived comparisons and plots are computed. This section gives a map of how the modules and classes relate to each other before diving into the per-function API references that follow.

Data flow

Given a set of systems’ predictions and a shared ground truth (\(y\)), the same bootstrap resample indices are drawn once per population size and reused across every system being compared – this pairing is what makes the pairwise comparisons statistically valid. A performance statistic (e.g. accuracy, F1) is evaluated on each resample for each system, producing an empirical distribution per system. Everything else – standard errors, confidence intervals, and pairwise significance testing between systems – is derived from these bootstrap distributions rather than from parametric assumptions.

Module and class relationships

        classDiagram
    class StatisticSamples {
        statistic: Callable
        num_samples: int
        n_jobs: int
        BiB: bool or ndarray[bool]
        calls: dict~str, ndarray~
        __call__(*args, name)
        __sklearn_clone__()
    }
    class Perf {
        y_true: ndarray
        predictions: dict~str, ndarray~
        func: Callable or list~Callable~
        BiB: bool or list~bool~
        statistic_samples: StatisticSamples
        best
        difference(wrt)
        dataframe()
        plot()
        __call__(y_pred, name)
    }
    class Difference {
        statistic_samples: StatisticSamples
        statistic: dict
        best: str or ndarray
        p_value(right, correction)
        dataframe()
        plot()
    }
    class MetricsWrappers {
        <<CompStats.metrics>>
        accuracy_score(y_true, *y_pred)
        balanced_accuracy_score(y_true, *y_pred)
        f1_score(y_true, *y_pred)
        ...
        .measure(**kwargs) Callable
    }
    class PerformanceModule {
        <<CompStats.performance>>
        performance(dataframe, score)
        difference(statistic_samples)
        all_differences(statistic_samples)
        plot_performance(statistic_samples)
        plot_difference(statistic_samples)
    }
    class Measurements {
        <<CompStats.measurements>>
        CI(samples, alpha)
        SE(samples)
        difference_p_value(samples, BiB)
    }

    Perf "1" *-- "1" StatisticSamples : owns
    Difference "1" *-- "1" StatisticSamples : owns (cloned)
    Perf ..> Difference : difference() clones\nvia sklearn.base.clone
    MetricsWrappers ..> Perf : constructs, tagging\nfunc.BiB
    PerformanceModule ..> StatisticSamples : operates on directly
    Measurements ..> StatisticSamples : reads .calls
    
  • StatisticSamples (CompStats.bootstrap) is the foundational primitive: given a statistic callable, it draws num_samples bootstrap resamples of the population and evaluates the statistic on each one, optionally in parallel via n_jobs. Results for a named system are cached in calls (name -> ndarray of bootstrap samples); resample indices are cached per population size so every system sharing a population reuses the same resampling. __sklearn_clone__ lets sklearn.base.clone produce a fresh instance that carries over parameters – used to build a Difference from a Perf without recomputing bootstrap samples.

  • Perf (CompStats.interface) is the main user-facing entry point. It wraps one or more systems’ predictions against shared ground truth, holding one StatisticSamples instance keyed by system name. func is a single callable or a list of callables (a multi-measure Perf); direction (bigger-is-better vs. smaller-is-better) comes from each callable’s own BiB attribute when present (tagged by a CompStats.metrics wrapper’s .measure factory), falling back to the constructor’s BiB default otherwise – BiB is the single source of truth for direction; there is no separate score/error split. Perf.difference(wrt=...) produces a Difference comparing every system against the best (or an explicit reference).

  • Difference (CompStats.interface) is created from Perf.difference() via sklearn.base.clone, reusing the already-computed bootstrap samples rather than recomputing them. Its p_value() is computed directly from the bootstrap distribution of paired differences, with no parametric test assumptions.

  • CompStats.metrics wrappers (accuracy_score, balanced_accuracy_score, top_k_accuracy_score, f1_score, …) are factories that close over an sklearn.metrics function (plus its metric-specific keyword arguments) and construct a Perf with that closure as func, tagging it with .BiB so direction travels with the callable. The shared @metrics_docs decorator injects the common Perf-style docstring into each wrapper.

  • CompStats.performance is an alternative, more functional API operating directly on a pandas.DataFrame (one gold column plus one column per system): performance(), difference()/all_differences(), and the plot_performance*/plot_difference* family, plus *_multiple variants for comparing several metrics at once. It predates Perf and is intentionally not composed with Perf/Difference.

  • CompStats.measurements provides stateless helpers – CI (percentile bootstrap confidence interval), SE (bootstrap standard error), and difference_p_value – each accepting either a raw ndarray of bootstrap samples or a StatisticSamples instance, in which case it maps itself over .calls.

Invariants to preserve

Note

Bootstrap resampling must stay paired across systems. StatisticSamples caches resample indices by population size precisely so every system’s bootstrap replicate i uses the same resampled indices. Do not introduce per-system independent resampling – it would invalidate every pairwise comparison.

Note

BiB (Bigger is Better) must be threaded consistently, never re-derived. A callable’s own BiB attribute wins when present; otherwise Perf’s BiB constructor argument is the default. Sorting, best, and p-value sign logic throughout CompStats.interface/CompStats.performance read this flag rather than re-deriving direction from which argument a function was passed as.

Note

Bootstrap samples must be reused via __sklearn_clone__, not re-instantiation. sklearn.base.clone duplicates Perf/StatisticSamples instances while carrying over already-computed bootstrap samples (e.g. Perf.difference(), CompStats.performance.difference). Plain re-instantiation silently redraws new bootstrap samples and breaks paired comparisons.

Development

Install the package with its test dependencies and the documentation dependencies:

pip install -e ".[test]"
pip install -r docs/requirements.txt

Run the test suite:

pytest CompStats

Build the developer reference; the HTML is written to docs/build/html/index.html:

python -m sphinx -n -W --keep-going -b html docs/source docs/build/html

The same strict build runs in continuous integration, and any warning (for example, a cross-reference that does not resolve) makes it fail.

Releases are published by running the workflow .github/workflows/publish-pypi.yml manually on main. Creating a tag or a GitHub release does not publish the package. The version gate scripts/check_package_versions.py verifies that the version is a single step ahead of the one published on PyPI.

User site

The user site is written with Quarto in quarto/ and is previewed with QUARTO_PYTHON="$PWD/.venv/bin/python" quarto preview quarto (live reload) or rendered statically into quarto/_site/ with QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto. Continuous integration never executes the notebooks of the site: it renders them from the cache committed in quarto/_freeze/. After changing a notebook, the site must therefore be rendered locally and quarto/_freeze/ committed together with the notebook; a test fails when they disagree.

Citing

The first reference below is the one to cite for the methodology; the second is the work it extends.

@article{Nava-Munoz2024,
  author  = {Nava-Mu{\~n}oz, Sergio and Graff, Mario and Escalante, Hugo Jair},
  title   = {Analysis of systems' performance in natural language processing competitions},
  journal = {Pattern Recognition Letters},
  year    = {2024},
  doi     = {10.1016/j.patrec.2024.03.010},
  url     = {https://doi.org/10.1016/j.patrec.2024.03.010}
}

@inproceedings{Nava-Munoz2023,
  author    = {Nava-Mu{\~n}oz, Sergio and Graff Guerrero, Mario and Escalante, Hugo Jair},
  title     = {Comparison of Classifiers in Challenge Scheme},
  booktitle = {Pattern Recognition},
  series    = {Lecture Notes in Computer Science},
  volume    = {13902},
  pages     = {89--98},
  year      = {2023},
  publisher = {Springer Nature Switzerland},
  doi       = {10.1007/978-3-031-33783-3_9},
  url       = {https://doi.org/10.1007/978-3-031-33783-3_9}
}

References

  • Berg-Kirkpatrick, T., Burkett, D. and Klein, D. (2012). An empirical investigation of statistical significance in NLP. EMNLP-CoNLL 2012.

  • Chernick, M. R. and LaBudde, R. A. (2011). An Introduction to Bootstrap Methods with Applications to R. Wiley.

  • Efron, B. and Tibshirani, R. J. (1994). An Introduction to the Bootstrap. Chapman and Hall/CRC.

  • Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65-70.

  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 289-300.

API