crossval

crossval#

Source code: sensai/evaluation/crossval.py

class VectorModelCrossValidationData(trained_models: Optional[List[TModel]], eval_data_list: List[TEvalData], predicted_var_names: List[str], test_indices_list=None)[source]#

Bases: ABC, Generic[TModel, TEvalData, TEvalStats, TEvalStatsCollection]

property model_name#
get_eval_stats_collection(predicted_var_name=None) → TEvalStatsCollection[source]#
iter_input_output_ground_truth_tuples(predicted_var_name=None) → Generator[Tuple[PandasNamedTuple, Any, Any], None, None][source]#
track_metrics(tracking_context: TrackingContext)[source]#
class CrossValidationSplitter[source]#

Bases: ABC

Defines a mechanism with which to generate data splits for cross-validation

abstract create_folds(data: InputOutputData, num_folds: int) → List[Tuple[Sequence[int], Sequence[int]]][source]#
Parameters:

  • data – the data from which to obtain the folds

  • num_folds – the number of splits/folds

Returns:

a list containing numFolds tuples (t, e) where t and e are sequences of data point indices to use for training and evaluation respectively

class CrossValidationSplitterDefault(shuffle=True, random_seed=42)[source]#

Bases: CrossValidationSplitter

create_folds(data: InputOutputData, num_splits: int) → List[Tuple[Sequence[int], Sequence[int]]][source]#
Parameters:

  • data – the data from which to obtain the folds

  • num_folds – the number of splits/folds

Returns:

a list containing numFolds tuples (t, e) where t and e are sequences of data point indices to use for training and evaluation respectively

class CrossValidationSplitterNested(test_fraction: float)[source]#

Bases: CrossValidationSplitter

A data splitter for nested cross-validation (which is useful, in particular, for time series prediction problems)

create_folds(data: InputOutputData, num_folds: int) → List[Tuple[Sequence[int], Sequence[int]]][source]#
Parameters:

  • data – the data from which to obtain the folds

  • num_folds – the number of splits/folds

Returns:

a list containing numFolds tuples (t, e) where t and e are sequences of data point indices to use for training and evaluation respectively

class VectorModelCrossValidatorParams(folds: int = 5, splitter: Optional[CrossValidationSplitter] = None, return_trained_models=False, evaluator_params: Optional[Union[RegressionEvaluatorParams, ClassificationEvaluatorParams]] = None, default_splitter_random_seed=42, default_splitter_shuffle=True)[source]#

Bases: object

Parameters:

  • folds – the number of folds

  • splitter – the splitter to use in order to generate the folds; if None, use default split (using parameters for random seed and shuffling below)

  • return_trained_models – whether to create a copy of the model for each fold and return each of the models (requires that models can be deep-copied); if False, the model that is passed to evalModel is fitted several times

  • evaluator_params – the model evaluator parameters

  • default_splitter_random_seed – [if splitter is None] the random seed to use for splits

  • default_splitter_shuffle – [if splitter is None] whether to shuffle the data (using randomSeed) before creating the folds

class VectorModelCrossValidator(data: InputOutputData, params: VectorModelCrossValidatorParams)[source]#

Bases: MetricsDictProvider, Generic[TCrossValData], ABC

Parameters:

  • data – the data set

  • params – parameters

static for_model(model: VectorModel, data: InputOutputData, params: VectorModelCrossValidatorParams) → Union[VectorClassificationModelCrossValidator, VectorRegressionModelCrossValidator][source]#
eval_model(model: VectorModel, track: bool = True)[source]#
Parameters:

  • model – the model to evaluate

  • track – whether tracking shall be enabled for the case where a tracked experiment is set on this object

Returns:

cross-validation results

create_metrics_dict_provider(predicted_var_name: Optional[str]) → MetricsDictProvider[source]#

Creates a metrics dictionary provider, e.g. for use in hyperparameter optimisation

Parameters:

predicted_var_name – the name of the predicted variable for which to obtain evaluation metrics; may be None only if the model outputs but a single predicted variable

Returns:

a metrics dictionary provider instance for the given variable

class VectorRegressionModelCrossValidationData(trained_models: Optional[List[TModel]], eval_data_list: List[TEvalData], predicted_var_names: List[str], test_indices_list=None)[source]#

Bases: VectorModelCrossValidationData[VectorRegressionModel, VectorRegressionModelEvaluationData, RegressionEvalStats, RegressionEvalStatsCollection]

class VectorRegressionModelCrossValidator(data: InputOutputData, params: VectorModelCrossValidatorParams)[source]#

Bases: VectorModelCrossValidator[VectorRegressionModelCrossValidationData]

Parameters:

  • data – the data set

  • params – parameters

class VectorClassificationModelCrossValidationData(trained_models: Optional[List[TModel]], eval_data_list: List[TEvalData], predicted_var_names: List[str], test_indices_list=None)[source]#

Bases: VectorModelCrossValidationData[VectorClassificationModel, VectorClassificationModelEvaluationData, ClassificationEvalStats, ClassificationEvalStatsCollection]

class VectorClassificationModelCrossValidator(data: InputOutputData, params: VectorModelCrossValidatorParams)[source]#

Bases: VectorModelCrossValidator[VectorClassificationModelCrossValidationData]

Parameters:

  • data – the data set

  • params – parameters