Hyperparameter Optimization

A typed SearchSpace/Configuration/Trial core plus grid/random search, Bayesian optimization (Gaussian-process and TPE surrogates), and bandit-based early stopping (Successive Halving, Hyperband, ASHA) – also consumed directly by Evolutionary Computation CMA-ES and Population Based Training. More...

Files
file	acquisition_functions.hpp
	Bayesian-Optimization acquisition functions over a GP posterior: Expected Improvement, Probability of Improvement, GP-Upper-Confidence-Bound (Snoek, Larochelle, Adams, "Practical Bayesian Optimization of Machine Learning Algorithms," NeurIPS 2012).
file	asha.hpp
	ASHA (Asynchronous Successive Halving Algorithm; Li, Jamieson, Rostamizadeh, Gonina, Hardt, Recht, Talwalkar, "A System for Massively Parallel Hyperparameter Tuning," MLSys 2020): unlike synchronous Successive Halving/Hyperband (which process one full rung across every candidate before any candidate advances to the next), ASHA promotes a candidate to the next rung as soon as it qualifies (its metric is in the top 1/eta fraction of every candidate that has ever completed that rung so far), never waiting on the rest of the current rung's population.
file	gaussian_process.hpp
	Gaussian-Process regression surrogate for Bayesian Optimization – the "gaussian bands" technique named at this campaign's own drafting (Snoek, Larochelle, Adams, "Practical Bayesian Optimization of Machine Learning Algorithms," NeurIPS 2012).
file	gp_bo.hpp
	The GP-BO trial loop: random-initialize, then repeatedly fit a GP to every trial observed so far and propose the next point by maximizing an acquisition function over random candidates in the unit hypercube.
file	hpo_sampling.hpp
	Grid search and random search over a SearchSpace: the two simplest, baseline hyperparameter-optimization algorithms, proving the SearchSpace/Trial abstraction's instantiate/train/read-back-metric loop end-to-end before Phase 2's GP-BO/TPE.
file	hyperband.hpp
	Hyperband (Li, Jamieson, DeSalvo, Rostamizadeh, Talwalkar, JMLR 2018): runs several Successive Halving "brackets" with different (num_configs, initial_budget) trade-offs – covering the tension between "few configs trained long" and "many configs trained short, halved down" that a single Successive Halving run commits to in advance – and returns the best result across every bracket.
file	search_space.hpp
	Typed hyperparameter search-space description: named parameters, each continuous, log-uniform, integer, or categorical.
file	successive_halving.hpp
	Successive Halving (the rung-based promotion mechanism underlying Hyperband and ASHA, Li, Jamieson, DeSalvo, Rostamizadeh, Talwalkar, "Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization," JMLR 2018): train a population of configurations for a small budget, keep the best fraction, repeat with a larger budget, until one survives.
file	tpe.hpp
	Tree-structured Parzen Estimator (Bergstra, Bardenet, Bengio, Kegl, "Algorithms for Hyper-Parameter Optimization," NeurIPS 2011) – a structurally distinct second surrogate family from GP-BO (gp_bo.hpp), handling mixed/categorical search spaces GP-BO's own vanilla kernel cannot (gp_bo.hpp restricts to Continuous/LogUniform; this file supports every ParameterKind).
file	trial.hpp
	A single hyperparameter-optimization trial: the configuration tried, plus every metric value recorded against it.

Detailed Description

A typed SearchSpace/Configuration/Trial core plus grid/random search, Bayesian optimization (Gaussian-process and TPE surrogates), and bandit-based early stopping (Successive Halving, Hyperband, ASHA) – also consumed directly by Evolutionary Computation CMA-ES and Population Based Training.