CompStats.interface

class Perf[source]

Perf is an entry point to CompStats

Parameters:
  • y_true (numpy.ndarray or pandas.DataFrame) – True measurement or could be a pandas.DataFrame where column label ‘y’ corresponds to the true measurement.

  • func (Function, or list of functions, where the first argument is \(y\) and the second is \(\hat{y}.\)) – Function (or list of functions) to measure the performance. Whether the best algorithm has the highest or the lowest value is given by BiB – either the constructor’s default, or, when a callable already carries its own BiB attribute (e.g. built by a CompStats.metrics wrapper’s .measure factory), that tag takes precedence. A list of functions combines them into a single, multi-measure Perf.

  • BiB (bool or list of bool) – Bigger is Better; the default direction used for any measure in func that doesn’t already carry its own BiB attribute. A single bool applies to every measure; a list applies element-wise, one entry per measure in func.

  • measure_names (list) – Display name for each measure, only relevant when more than one measure is given; defaults to each function’s __name__.

  • y_pred (numpy.ndarray) – Predictions, the algorithms will be identified with alg-k where k=1 is the first argument included in args.

  • kwargs (numpy.ndarray) – Predictions, the algorithms will be identified using the keyword

  • n_jobs (int) – Number of jobs to compute the statistic, default=-1 corresponding to use all threads.

  • num_samples (int) – Number of bootstrap samples, default=500.

  • use_tqdm (bool) – Whether to use tqdm.tqdm to visualize the progress, default=True.

>>> from sklearn.svm import LinearSVC
>>> from sklearn.linear_model import LogisticRegression
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.base import clone
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> perf = Perf(y_val, hy, name='LinearSVC')
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf(ens.predict(X_val), name='forest')
>>> perf
<Perf>
Statistic with its standard error (se)
statistic (se)
0.9792 (0.0221) <= LinearSVC
0.9744 (0.0246) <= forest

If an algorithm’s prediction is missing, this can be included by calling the instance, as can be seen in the following instruction. Note that the algorithm’s name can also be given with the keyword name.

>>> lr = LogisticRegression().fit(X_train, y_train)
>>> perf(lr.predict(X_val), name='Log. Reg.')
<Perf>
Statistic with its standard error (se)
statistic (se)
1.0000 (0.0000) <= Log. Reg.
0.9792 (0.0221) <= alg-1
0.9744 (0.0246) <= forest

The performance function used to compare the algorithms can be changed, and the same bootstrap samples would be used if the instance were cloned. Consequently, the values are computed using the same samples, as can be seen in the following example.

>>> perf_error = clone(perf)
>>> perf_error.func = lambda y, hy: (y != hy).mean()
>>> perf_error.BiB = False
>>> perf_error
<Perf>
Statistic with its standard error (se)
statistic (se)
0.0000 (0.0000) <= Log. Reg.
0.0222 (0.0237) <= alg-1
0.0222 (0.0215) <= forest

When several algorithms are compared, difference’s p-values can be adjusted for multiple comparisons by passing a statsmodels.stats.multitest.multipletests method name (e.g. 'bonferroni', 'holm', 'fdr_bh') as correction – both to Difference.p_value directly and, so that plots reflect the same adjusted significance, to plot/dataframe.

>>> diff = perf.difference()
>>> diff.p_value()
{'alg-1': np.float64(0.3), 'forest': np.float64(0.2)}
>>> diff.p_value(correction='bonferroni')
{'alg-1': np.float64(0.6), 'forest': np.float64(0.4)}
>>> perf.plot(correction='bonferroni')

Two or more measures can be combined into a single Perf instance (e.g. macro-F1 together with macro-recall) by passing a list of functions to func – see CompStats.metrics’s .measure factories (e.g. f1_score.measure). Every measure is evaluated on the same bootstrap resamples, so comparisons across algorithms remain paired for each measure.

>>> from CompStats.metrics import f1_score, recall_score
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val),
...             func=[f1_score.measure(average='macro'),
...                   recall_score.measure(average='macro')])

With multiple measures, correction is applied independently per measure (i.e. per column), so one metric’s correction never mixes with another’s.

>>> diff = perf.difference()
>>> diff.p_value()
{'alg-1': array([1. , 0.3]), 'forest': array([0.2, 1. ])}
>>> diff.p_value(correction='bonferroni')
{'alg-1': array([1. , 0.6]), 'forest': array([0.4, 1. ])}
__init__(y_true, *y_pred, name: str = None, func=<function balanced_accuracy_score>, BiB: bool = True, measure_names: list = None, num_samples: int = 500, n_jobs: int = -1, use_tqdm=True, **kwargs)[source]
get_params()[source]

Parameters

difference(wrt: str = None)[source]

Create the Difference between a reference system and every other system, computed on the bootstrap samples already stored in this instance (so the comparison is paired). By default the reference is the best system.

Parameters:

wrt (str) – Name of the reference system; None (default) uses best.

Returns:

Instance holding the bootstrap distribution of the paired differences, signed so that a positive value means the reference is better.

Return type:

CompStats.interface.Difference

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.base import clone
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val))
>>> perf.difference()
<Difference>
difference p-values w.r.t alg-1
forest 0.06
property best

System with best performance

property sorting_func

Rank systems when multiple performances are used

property statistic

Statistic

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val))
>>> perf.statistic
{'alg-1': 1.0, 'forest': 0.9500891265597148}
property se

Standard Error

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val))
>>> perf.se
{'alg-1': 0.0, 'forest': 0.026945730782184187}
property ci

Confidence interval

>>> from sklearn.svm import LinearSVC
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, name='LinearSVC')
>>> perf.ci
(np.float64(0.9333333333333332), np.float64(1.0))
plot(value_name: str = None, var_name: str = 'Performance', alg_legend: str = 'Algorithm', perf_names: list = None, CI: float = 0.05, kind: str = 'point', linestyle: str = 'none', col_wrap: int = 3, capsize: float = 0.2, comparison: bool = True, right: bool = True, correction: str = None, comp_legend: str = 'Comparison', winner_legend: str = 'Best', tie_legend: str = 'Equivalent', loser_legend: str = 'Different', palette: object = None, **kwargs)[source]

plot with seaborn

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy,
                func=lambda y, hy: (y != hy).mean(), BiB=False,
                forest=ens.predict(X_val))
>>> perf.plot()
dataframe(comparison: bool = False, right: bool = True, alpha: float = 0.05, correction: str = None, value_name: str = 'Score', var_name: str = 'Performance', alg_legend: str = 'Algorithm', comp_legend: str = 'Comparison', winner_legend: str = 'Best', tie_legend: str = 'Equivalent', loser_legend: str = 'Different', perf_names: str = None)[source]

Dataframe

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val))
>>> df = perf.dataframe()
property n_jobs

Number of jobs to compute the statistics

property statistic_func

Statistic function

A single func callable is returned as-is; when more than one measure is given (a list passed to func), a composite callable is returned that concatenates every measure’s output into a single vector, evaluated on the same bootstrap samples.

property measure_names

Display name for each measure, used when combining more than one

Defaults to each measure’s function __name__ (available because every CompStats.metrics wrapper’s inner function is functools.wraps-decorated with the corresponding sklearn metric).

property statistic_samples

Statistic Samples

property num_samples

Number of bootstrap samples

property predictions

Predictions

property y_true

True output, gold standard o \(y\)

property func

Function (or list of functions) used to measure the performance

property BiB

Bigger is Better; default direction for measures without their own BiB tag

class Difference[source]

Bootstrap distribution of the paired differences between a reference system (by default the best one) and every other system. It is built by difference and provides the p-values (p_value) and the plot of the differences.

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.base import clone
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val))
>>> diff = perf.difference()
>>> diff
<Difference>
difference p-values w.r.t alg-1
0.0780 <= forest
__init__(statistic_samples: StatisticSamples = None, statistic: dict = None, best: str = None) → None
property sorting_func

Rank systems when multiple performances are used

p_value(right: bool = True, correction: str = None)[source]

Compute p_value of the differences

Parameters:
  • right (bool) – Estimate the p-value using \(\text{sample} \geq 2\delta\)

  • correction (str) – Method to adjust for multiple comparisons, passed to statsmodels.stats.multitest.multipletests (e.g. 'bonferroni', 'holm', 'fdr_bh'); None (default) leaves the p-values uncorrected. With a single measure, the family of comparisons is every other system against best; with multiple measures, each measure is corrected as its own family (i.e. per column) so metrics do not contaminate each other’s correction.

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.naive_bayes import GaussianNB
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.base import clone
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> perf = Perf(y_val, hy, name='LinearSVC')
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf(ens.predict(X_val), name='forest')
>>> nb = GaussianNB().fit(X_train, y_train)
>>> perf(nb.predict(X_val), name='bayes')
>>> diff = perf.difference()
>>> diff.p_value()
{'forest': np.float64(0.3), 'bayes': np.float64(0.2)}
>>> diff.p_value(correction='bonferroni')
{'forest': np.float64(0.6), 'bayes': np.float64(0.4)}
dataframe(value_name: str = 'Score', var_name: str = 'Best', alg_legend: str = 'Algorithm', sig_legend: str = 'Significant', perf_names: str = None, right: bool = True, alpha: float = 0.05, correction: str = None)[source]

Dataframe

plot(value_name: str = 'Difference', var_name: str = 'Best', alg_legend: str = 'Algorithm', sig_legend: str = 'Significant', perf_names: list = None, alpha: float = 0.05, right: bool = True, correction: str = None, kind: str = 'point', linestyle: str = 'none', col_wrap: int = 3, capsize: float = 0.2, set_refline: bool = True, **kwargs)[source]

Plot

>>> from sklearn.svm import LinearSVC
>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import load_iris
>>> from sklearn.model_selection import train_test_split
>>> from sklearn.base import clone
>>> from CompStats.interface import Perf
>>> X, y = load_iris(return_X_y=True)
>>> _ = train_test_split(X, y, test_size=0.3)
>>> X_train, X_val, y_train, y_val = _
>>> m = LinearSVC().fit(X_train, y_train)
>>> hy = m.predict(X_val)
>>> ens = RandomForestClassifier().fit(X_train, y_train)
>>> perf = Perf(y_val, hy, forest=ens.predict(X_val))
>>> diff = perf.difference()
>>> diff.plot()