CompStats¶
Version 0.2.0.
Note
This is the developer reference of CompStats: the architecture of the package and the complete API. Installation, tutorials and the description of the methodology for users are published on the project site, https://ingeotec.github.io/CompStats/.
Introduction¶
Collaborative competitions (challenges) evaluate systems on a single held-out test set whose gold standard is kept by the organizers. The systems are ranked by a performance measure computed on that set. The organizers have access only to the predictions, to a fixed dataset size and to a small number of submissions; consequently, the classical statistical tests that require access to the algorithm, or that require performance estimates on several datasets, do not apply.
CompStats implements a bootstrap-based methodology that complements the ranking with standard errors, confidence intervals, significance tests of paired differences and corrections for multiple comparisons. The same setting covers any comparison of supervised-learning systems on a shared test set, whether the task is classification or regression. The methodology is described in Analysis of systems’ performance in natural language processing competitions (doi:10.1016/j.patrec.2024.03.010), which extends Comparison of Classifiers in Challenge Scheme (doi:10.1007/978-3-031-33783-3_9).
Statistical background¶
Each concept below is stated with its formula and the symbol of the package that computes it.
Setting. Let \(y\) be a gold standard of size \(n\), let
\(\hat{y}_1, \ldots, \hat{y}_m\) be the predictions of \(m\) systems, and let
\(s(y, \hat{y})\) be a performance measure. A measure is either bigger-is-better (a score) or
smaller-is-better (an error); the flag BiB carries that direction.
Bootstrap. \(B\) samples (num_samples, default 500) of \(n\) indices are drawn with
replacement from \(\{1, \ldots, n\}\). The measure is evaluated on every sample for every
system, which gives the bootstrap distribution \(s_j^{*1}, \ldots, s_j^{*B}\) of system
\(j\) (StatisticSamples). The same index samples are used
for all the systems, which is what makes the comparisons paired.
Standard error. The standard deviation of the bootstrap distribution,
computed by SE and exposed as the se property of
Perf.
Confidence interval. The percentile interval
where \(q_p\) is the \(p\)-th quantile of the bootstrap distribution and
\(\alpha = 0.05\) by default. It is computed by CI and
exposed as the ci property of Perf.
Paired difference. For a reference system \(A\) (by default the best one) and another system \(B\), the bootstrap distribution of
is computed on the same samples. The sign is flipped for smaller-is-better measures, so that a
positive difference always means that the reference is better. It is built by
difference, which returns a
Difference.
Significance. Let \(\delta(x)\) be the difference observed on the test set. The p-value of the null hypothesis \(H_0\): “the reference is not better” is estimated as the fraction of bootstrap differences that are at least \(2\delta(x)\),
(p_value), following Berg-Kirkpatrick, Burkett and
Klein (2012). The bootstrap distribution is centred on \(\delta(x)\) rather than on zero,
hence the factor 2.
Multiple comparisons. Comparing the best system against the other \(m - 1\) systems tests
several hypotheses at once. The p-values can be adjusted with any method of
statsmodels.stats.multitest.multipletests through the correction argument (for
example 'bonferroni', 'holm' or 'fdr_bh'). When several measures are evaluated, each
measure is corrected as its own family.
Competitiveness of a competition. The legacy functional API
(performance_multiple_metrics) summarizes the competitiveness of
a competition with the coefficient of variation of the systems’ performance,
where \(s_x\) is the sample standard deviation and \(\bar{x}\) the mean of the systems’ performance, and with the distance between the winner and the system in the middle of the table. The potential for improvement of a task is assessed separately with the possible percentage improvement, \(PPI = 100\,(1 - x_{best})\), for bigger-is-better scores bounded by 1.
Architecture¶
CompStats is organized around one core data flow: raw predictions from one or more systems are turned into bootstrap-resampled statistics, from which derived comparisons and plots are computed. This section gives a map of how the modules and classes relate to each other before diving into the per-function API references that follow.
Data flow¶
Given a set of systems’ predictions and a shared ground truth (\(y\)), the same bootstrap resample indices are drawn once per population size and reused across every system being compared – this pairing is what makes the pairwise comparisons statistically valid. A performance statistic (e.g. accuracy, F1) is evaluated on each resample for each system, producing an empirical distribution per system. Everything else – standard errors, confidence intervals, and pairwise significance testing between systems – is derived from these bootstrap distributions rather than from parametric assumptions.
Module and class relationships¶
classDiagram
class StatisticSamples {
statistic: Callable
num_samples: int
n_jobs: int
BiB: bool or ndarray[bool]
calls: dict~str, ndarray~
__call__(*args, name)
__sklearn_clone__()
}
class Perf {
y_true: ndarray
predictions: dict~str, ndarray~
func: Callable or list~Callable~
BiB: bool or list~bool~
statistic_samples: StatisticSamples
best
difference(wrt)
dataframe()
plot()
__call__(y_pred, name)
}
class Difference {
statistic_samples: StatisticSamples
statistic: dict
best: str or ndarray
p_value(right, correction)
dataframe()
plot()
}
class MetricsWrappers {
<<CompStats.metrics>>
accuracy_score(y_true, *y_pred)
balanced_accuracy_score(y_true, *y_pred)
f1_score(y_true, *y_pred)
...
.measure(**kwargs) Callable
}
class PerformanceModule {
<<CompStats.performance>>
performance(dataframe, score)
difference(statistic_samples)
all_differences(statistic_samples)
plot_performance(statistic_samples)
plot_difference(statistic_samples)
}
class Measurements {
<<CompStats.measurements>>
CI(samples, alpha)
SE(samples)
difference_p_value(samples, BiB)
}
Perf "1" *-- "1" StatisticSamples : owns
Difference "1" *-- "1" StatisticSamples : owns (cloned)
Perf ..> Difference : difference() clones\nvia sklearn.base.clone
MetricsWrappers ..> Perf : constructs, tagging\nfunc.BiB
PerformanceModule ..> StatisticSamples : operates on directly
Measurements ..> StatisticSamples : reads .calls
StatisticSamples (
CompStats.bootstrap) is the foundational primitive: given astatisticcallable, it drawsnum_samplesbootstrap resamples of the population and evaluates the statistic on each one, optionally in parallel vian_jobs. Results for a named system are cached incalls(name -> ndarray of bootstrap samples); resample indices are cached per population size so every system sharing a population reuses the same resampling.__sklearn_clone__letssklearn.base.cloneproduce a fresh instance that carries over parameters – used to build aDifferencefrom aPerfwithout recomputing bootstrap samples.Perf (
CompStats.interface) is the main user-facing entry point. It wraps one or more systems’ predictions against shared ground truth, holding oneStatisticSamplesinstance keyed by system name.funcis a single callable or a list of callables (a multi-measurePerf); direction (bigger-is-better vs. smaller-is-better) comes from each callable’s ownBiBattribute when present (tagged by aCompStats.metricswrapper’s.measurefactory), falling back to the constructor’sBiBdefault otherwise –BiBis the single source of truth for direction; there is no separate score/error split.Perf.difference(wrt=...)produces aDifferencecomparing every system against the best (or an explicit reference).Difference (
CompStats.interface) is created fromPerf.difference()viasklearn.base.clone, reusing the already-computed bootstrap samples rather than recomputing them. Itsp_value()is computed directly from the bootstrap distribution of paired differences, with no parametric test assumptions.CompStats.metrics wrappers (
accuracy_score,balanced_accuracy_score,top_k_accuracy_score,f1_score, …) are factories that close over ansklearn.metricsfunction (plus its metric-specific keyword arguments) and construct aPerfwith that closure asfunc, tagging it with.BiBso direction travels with the callable. The shared@metrics_docsdecorator injects the commonPerf-style docstring into each wrapper.CompStats.performance is an alternative, more functional API operating directly on a
pandas.DataFrame(one gold column plus one column per system):performance(),difference()/all_differences(), and theplot_performance*/plot_difference*family, plus*_multiplevariants for comparing several metrics at once. It predatesPerfand is intentionally not composed withPerf/Difference.CompStats.measurements provides stateless helpers –
CI(percentile bootstrap confidence interval),SE(bootstrap standard error), anddifference_p_value– each accepting either a raw ndarray of bootstrap samples or aStatisticSamplesinstance, in which case it maps itself over.calls.
Invariants to preserve¶
Note
Bootstrap resampling must stay paired across systems.
StatisticSamples caches resample indices by population size precisely so every system’s
bootstrap replicate i uses the same resampled indices. Do not introduce per-system independent
resampling – it would invalidate every pairwise comparison.
Note
BiB (Bigger is Better) must be threaded consistently, never re-derived.
A callable’s own BiB attribute wins when present; otherwise Perf’s BiB constructor
argument is the default. Sorting, best, and p-value sign logic throughout
CompStats.interface/CompStats.performance read this flag rather than
re-deriving direction from which argument a function was passed as.
Note
Bootstrap samples must be reused via __sklearn_clone__, not re-instantiation.
sklearn.base.clone duplicates Perf/StatisticSamples instances while carrying
over already-computed bootstrap samples (e.g. Perf.difference(),
CompStats.performance.difference). Plain re-instantiation silently redraws new bootstrap
samples and breaks paired comparisons.
Development¶
Install the package with its test dependencies and the documentation dependencies:
pip install -e ".[test]"
pip install -r docs/requirements.txt
Run the test suite:
pytest CompStats
Build the developer reference; the HTML is written to docs/build/html/index.html:
python -m sphinx -n -W --keep-going -b html docs/source docs/build/html
The same strict build runs in continuous integration, and any warning (for example, a cross-reference that does not resolve) makes it fail.
Releases are published by running the workflow .github/workflows/publish-pypi.yml manually
on main. Creating a tag or a GitHub release does not publish the package. The version gate
scripts/check_package_versions.py verifies that the version is a single step ahead of the one
published on PyPI.
User site¶
The user site is written with Quarto in quarto/ and is previewed with
QUARTO_PYTHON="$PWD/.venv/bin/python" quarto preview quarto (live reload) or rendered
statically into quarto/_site/ with QUARTO_PYTHON="$PWD/.venv/bin/python" quarto render quarto.
Continuous integration never executes the notebooks of the site: it renders them from the cache
committed in quarto/_freeze/. After changing a notebook, the site must therefore be rendered
locally and quarto/_freeze/ committed together with the notebook; a test fails when they
disagree.
Citing¶
The first reference below is the one to cite for the methodology; the second is the work it extends.
@article{Nava-Munoz2024,
author = {Nava-Mu{\~n}oz, Sergio and Graff, Mario and Escalante, Hugo Jair},
title = {Analysis of systems' performance in natural language processing competitions},
journal = {Pattern Recognition Letters},
year = {2024},
doi = {10.1016/j.patrec.2024.03.010},
url = {https://doi.org/10.1016/j.patrec.2024.03.010}
}
@inproceedings{Nava-Munoz2023,
author = {Nava-Mu{\~n}oz, Sergio and Graff Guerrero, Mario and Escalante, Hugo Jair},
title = {Comparison of Classifiers in Challenge Scheme},
booktitle = {Pattern Recognition},
series = {Lecture Notes in Computer Science},
volume = {13902},
pages = {89--98},
year = {2023},
publisher = {Springer Nature Switzerland},
doi = {10.1007/978-3-031-33783-3_9},
url = {https://doi.org/10.1007/978-3-031-33783-3_9}
}
References¶
Berg-Kirkpatrick, T., Burkett, D. and Klein, D. (2012). An empirical investigation of statistical significance in NLP. EMNLP-CoNLL 2012.
Chernick, M. R. and LaBudde, R. A. (2011). An Introduction to Bootstrap Methods with Applications to R. Wiley.
Efron, B. and Tibshirani, R. J. (1994). An Introduction to the Bootstrap. Chapman and Hall/CRC.
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65-70.
Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 289-300.