CompStats.interface¶
- class Perf[source]¶
Perf is an entry point to CompStats
- Parameters:
y_true (numpy.ndarray or pandas.DataFrame) – True measurement or could be a pandas.DataFrame where column label ‘y’ corresponds to the true measurement.
func (Function, or list of functions, where the first argument is \(y\) and the second is \(\hat{y}.\)) – Function (or list of functions) to measure the performance. Whether the best algorithm has the highest or the lowest value is given by
BiB– either the constructor’s default, or, when a callable already carries its ownBiBattribute (e.g. built by aCompStats.metricswrapper’s.measurefactory), that tag takes precedence. A list of functions combines them into a single, multi-measurePerf.BiB (bool or list of bool) – Bigger is Better; the default direction used for any measure in
functhat doesn’t already carry its ownBiBattribute. A single bool applies to every measure; a list applies element-wise, one entry per measure infunc.measure_names (list) – Display name for each measure, only relevant when more than one measure is given; defaults to each function’s
__name__.y_pred (numpy.ndarray) – Predictions, the algorithms will be identified with alg-k where k=1 is the first argument included in
args.kwargs (numpy.ndarray) – Predictions, the algorithms will be identified using the keyword
n_jobs (int) – Number of jobs to compute the statistic, default=-1 corresponding to use all threads.
num_samples (int) – Number of bootstrap samples, default=500.
use_tqdm (bool) – Whether to use tqdm.tqdm to visualize the progress, default=True.
>>> from sklearn.svm import LinearSVC >>> from sklearn.linear_model import LogisticRegression >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from sklearn.base import clone >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> perf = Perf(y_val, hy, name='LinearSVC') >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf(ens.predict(X_val), name='forest') >>> perf <Perf> Statistic with its standard error (se) statistic (se) 0.9792 (0.0221) <= LinearSVC 0.9744 (0.0246) <= forest
If an algorithm’s prediction is missing, this can be included by calling the instance, as can be seen in the following instruction. Note that the algorithm’s name can also be given with the keyword
name.>>> lr = LogisticRegression().fit(X_train, y_train) >>> perf(lr.predict(X_val), name='Log. Reg.') <Perf> Statistic with its standard error (se) statistic (se) 1.0000 (0.0000) <= Log. Reg. 0.9792 (0.0221) <= alg-1 0.9744 (0.0246) <= forest
The performance function used to compare the algorithms can be changed, and the same bootstrap samples would be used if the instance were cloned. Consequently, the values are computed using the same samples, as can be seen in the following example.
>>> perf_error = clone(perf) >>> perf_error.func = lambda y, hy: (y != hy).mean() >>> perf_error.BiB = False >>> perf_error <Perf> Statistic with its standard error (se) statistic (se) 0.0000 (0.0000) <= Log. Reg. 0.0222 (0.0237) <= alg-1 0.0222 (0.0215) <= forest
When several algorithms are compared,
difference’s p-values can be adjusted for multiple comparisons by passing astatsmodels.stats.multitest.multipletestsmethod name (e.g.'bonferroni','holm','fdr_bh') ascorrection– both toDifference.p_valuedirectly and, so that plots reflect the same adjusted significance, toplot/dataframe.>>> diff = perf.difference() >>> diff.p_value() {'alg-1': np.float64(0.3), 'forest': np.float64(0.2)} >>> diff.p_value(correction='bonferroni') {'alg-1': np.float64(0.6), 'forest': np.float64(0.4)} >>> perf.plot(correction='bonferroni')
Two or more measures can be combined into a single
Perfinstance (e.g. macro-F1 together with macro-recall) by passing a list of functions tofunc– seeCompStats.metrics’s.measurefactories (e.g.f1_score.measure). Every measure is evaluated on the same bootstrap resamples, so comparisons across algorithms remain paired for each measure.>>> from CompStats.metrics import f1_score, recall_score >>> perf = Perf(y_val, hy, forest=ens.predict(X_val), ... func=[f1_score.measure(average='macro'), ... recall_score.measure(average='macro')])
With multiple measures,
correctionis applied independently per measure (i.e. per column), so one metric’s correction never mixes with another’s.>>> diff = perf.difference() >>> diff.p_value() {'alg-1': array([1. , 0.3]), 'forest': array([0.2, 1. ])} >>> diff.p_value(correction='bonferroni') {'alg-1': array([1. , 0.6]), 'forest': array([0.4, 1. ])}
- __init__(y_true, *y_pred, name: str = None, func=<function balanced_accuracy_score>, BiB: bool = True, measure_names: list = None, num_samples: int = 500, n_jobs: int = -1, use_tqdm=True, **kwargs)[source]¶
- difference(wrt: str = None)[source]¶
Create the
Differencebetween a reference system and every other system, computed on the bootstrap samples already stored in this instance (so the comparison is paired). By default the reference is the best system.- Parameters:
wrt (str) – Name of the reference system;
None(default) usesbest.- Returns:
Instance holding the bootstrap distribution of the paired differences, signed so that a positive value means the reference is better.
- Return type:
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from sklearn.base import clone >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> perf.difference() <Difference> difference p-values w.r.t alg-1 forest 0.06
- property best¶
System with best performance
- property sorting_func¶
Rank systems when multiple performances are used
- property statistic¶
Statistic
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> perf.statistic {'alg-1': 1.0, 'forest': 0.9500891265597148}
- property se¶
Standard Error
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> perf.se {'alg-1': 0.0, 'forest': 0.026945730782184187}
- property ci¶
Confidence interval
>>> from sklearn.svm import LinearSVC >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, name='LinearSVC') >>> perf.ci (np.float64(0.9333333333333332), np.float64(1.0))
- plot(value_name: str = None, var_name: str = 'Performance', alg_legend: str = 'Algorithm', perf_names: list = None, CI: float = 0.05, kind: str = 'point', linestyle: str = 'none', col_wrap: int = 3, capsize: float = 0.2, comparison: bool = True, right: bool = True, correction: str = None, comp_legend: str = 'Comparison', winner_legend: str = 'Best', tie_legend: str = 'Equivalent', loser_legend: str = 'Different', palette: object = None, **kwargs)[source]¶
plot with seaborn
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, func=lambda y, hy: (y != hy).mean(), BiB=False, forest=ens.predict(X_val)) >>> perf.plot()
- dataframe(comparison: bool = False, right: bool = True, alpha: float = 0.05, correction: str = None, value_name: str = 'Score', var_name: str = 'Performance', alg_legend: str = 'Algorithm', comp_legend: str = 'Comparison', winner_legend: str = 'Best', tie_legend: str = 'Equivalent', loser_legend: str = 'Different', perf_names: str = None)[source]¶
Dataframe
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> df = perf.dataframe()
- property n_jobs¶
Number of jobs to compute the statistics
- property statistic_func¶
Statistic function
A single
funccallable is returned as-is; when more than one measure is given (a list passed tofunc), a composite callable is returned that concatenates every measure’s output into a single vector, evaluated on the same bootstrap samples.
- property measure_names¶
Display name for each measure, used when combining more than one
Defaults to each measure’s function
__name__(available because everyCompStats.metricswrapper’s inner function isfunctools.wraps-decorated with the corresponding sklearn metric).
- property statistic_samples¶
Statistic Samples
- property num_samples¶
Number of bootstrap samples
- property predictions¶
Predictions
- property y_true¶
True output, gold standard o \(y\)
- property func¶
Function (or list of functions) used to measure the performance
- property BiB¶
Bigger is Better; default direction for measures without their own
BiBtag
- class Difference[source]¶
Bootstrap distribution of the paired differences between a reference system (by default the best one) and every other system. It is built by
differenceand provides the p-values (p_value) and the plot of the differences.>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from sklearn.base import clone >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> diff = perf.difference() >>> diff <Difference> difference p-values w.r.t alg-1 0.0780 <= forest
- __init__(statistic_samples: StatisticSamples = None, statistic: dict = None, best: str = None) None¶
- property sorting_func¶
Rank systems when multiple performances are used
- p_value(right: bool = True, correction: str = None)[source]¶
Compute p_value of the differences
- Parameters:
right (bool) – Estimate the p-value using \(\text{sample} \geq 2\delta\)
correction (str) – Method to adjust for multiple comparisons, passed to
statsmodels.stats.multitest.multipletests(e.g.'bonferroni','holm','fdr_bh');None(default) leaves the p-values uncorrected. With a single measure, the family of comparisons is every other system againstbest; with multiple measures, each measure is corrected as its own family (i.e. per column) so metrics do not contaminate each other’s correction.
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.naive_bayes import GaussianNB >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from sklearn.base import clone >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> perf = Perf(y_val, hy, name='LinearSVC') >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf(ens.predict(X_val), name='forest') >>> nb = GaussianNB().fit(X_train, y_train) >>> perf(nb.predict(X_val), name='bayes') >>> diff = perf.difference() >>> diff.p_value() {'forest': np.float64(0.3), 'bayes': np.float64(0.2)} >>> diff.p_value(correction='bonferroni') {'forest': np.float64(0.6), 'bayes': np.float64(0.4)}
- dataframe(value_name: str = 'Score', var_name: str = 'Best', alg_legend: str = 'Algorithm', sig_legend: str = 'Significant', perf_names: str = None, right: bool = True, alpha: float = 0.05, correction: str = None)[source]¶
Dataframe
- plot(value_name: str = 'Difference', var_name: str = 'Best', alg_legend: str = 'Algorithm', sig_legend: str = 'Significant', perf_names: list = None, alpha: float = 0.05, right: bool = True, correction: str = None, kind: str = 'point', linestyle: str = 'none', col_wrap: int = 3, capsize: float = 0.2, set_refline: bool = True, **kwargs)[source]¶
Plot
>>> from sklearn.svm import LinearSVC >>> from sklearn.ensemble import RandomForestClassifier >>> from sklearn.datasets import load_iris >>> from sklearn.model_selection import train_test_split >>> from sklearn.base import clone >>> from CompStats.interface import Perf >>> X, y = load_iris(return_X_y=True) >>> _ = train_test_split(X, y, test_size=0.3) >>> X_train, X_val, y_train, y_val = _ >>> m = LinearSVC().fit(X_train, y_train) >>> hy = m.predict(X_val) >>> ens = RandomForestClassifier().fit(X_train, y_train) >>> perf = Perf(y_val, hy, forest=ens.predict(X_val)) >>> diff = perf.difference() >>> diff.plot()