imitation.scripts.tuning#
Tunes the hyperparameters of the algorithms.
Functions
|
Evaluate a given trial of a parallel run on a separate set of seeds. |
|
Find the trial with the best mean return across all seeds. |
|
Tune hyperparameters of imitation algorithms using the parallel script. |
- imitation.scripts.tuning.evaluate_trial(trial, num_eval_seeds, run_name, parallel_run_config, resources_per_trial, return_key, print_return=False)[source]#
Evaluate a given trial of a parallel run on a separate set of seeds.
- Parameters
trial (
Trial) – The trial to evaluate.num_eval_seeds (
int) – Number of distinct seeds to evaluate the best trial on.run_name (
str) – The name of the evaluation run.parallel_run_config – Dictionary of arguments passed to the parallel script to get best_trial.
resources_per_trial (
Dict[str,int]) – Resources to be used for each evaluation trial.return_key (
str) – The key of the return metric in the results dataframe.print_return (
bool) – Whether to print the mean and std of the evaluation returns.
- Returns
The result of the evaluation run.
- Return type
eval_run
- imitation.scripts.tuning.find_best_trial(experiment_analysis, return_key, print_return=False)[source]#
Find the trial with the best mean return across all seeds.
- Parameters
experiment_analysis (
ExperimentAnalysis) – The result of a parallel/tuning experiment.return_key (
str) – The key of the return metric in the results dataframe.print_return (
bool) – Whether to print the mean and std of the returns of the best trial.
- Returns
The trial with the best mean return across all seeds.
- Return type
best_trial
- imitation.scripts.tuning.tune(parallel_run_config, eval_best_trial_resource_multiplier=1, num_eval_seeds=5)[source]#
Tune hyperparameters of imitation algorithms using the parallel script.
The parallel script is called twice in this function. The first call is to tune the hyperparameters. The second call is to evaluate the best trial on a separate set of seeds.
- Parameters
parallel_run_config – Dictionary of arguments to pass to the parallel script. This is used to define the search space for tuning the hyperparameters.
eval_best_trial_resource_multiplier (
int) – Factor by which to multiply the number of cpus per trial in resources_per_trial. This is useful for allocating more resources per trial to the evaluation trials than the resources for hyperparameter tuning since number of evaluation trials is usually much smaller than the number of tuning trials.num_eval_seeds (
int) – Number of distinct seeds to evaluate the best trial on. Set to 0 to disable evaluation.
- Raises
ValueError – If no trials are returned by the parallel run of tuning.
- Return type
None