tpe.hpp File Reference

Tree-structured Parzen Estimator (Bergstra, Bardenet, Bengio, Kegl, "Algorithms for Hyper-Parameter Optimization," NeurIPS 2011) – a structurally distinct second surrogate family from GP-BO (gp_bo.hpp), handling mixed/categorical search spaces GP-BO's own vanilla kernel cannot (gp_bo.hpp restricts to Continuous/LogUniform; this file supports every ParameterKind). More...

#include <algorithm>
#include <cmath>
#include <limits>
#include <stdexcept>
#include <string>
#include <vector>
#include "pulsatrix/hpo_sampling.hpp"
#include "pulsatrix/search_space.hpp"
#include "pulsatrix/trial.hpp"

Include dependency graph for tpe.hpp:

Go to the source code of this file.

Namespaces
namespace	pulsatrix

Functions
double	pulsatrix::GaussianKdeDensity (const std::vector< double > &observations, double bandwidth, double x)
	Fixed-bandwidth Gaussian KDE: density(x) = mean over every observation o of N(x; o, bandwidth^2).
double	pulsatrix::CategoricalDensity (const std::vector< std::string > &observations, size_t num_categories, const std::string &category)
	Laplace(add-one)-smoothed empirical probability of category among observations.
double	pulsatrix::LogDensityRatio (const SearchSpace &space, const Configuration &candidate, const std::vector< Configuration > &good_configs, const std::vector< Configuration > &bad_configs)
	log(l(candidate)) - log(g(candidate)): the TPE scoring function, summed independently over every parameter in space (so a candidate's mixed continuous/categorical parameters each contribute their own term, composing naturally rather than needing a joint density over the whole space).
template<typename ObjectiveFn , typename RNG >
std::vector< Trial >	pulsatrix::RunTPELoop (const SearchSpace &space, ObjectiveFn objective_fn, size_t num_initial_random, size_t num_iterations, double gamma, size_t num_candidates, RNG &rng)
	Runs TPE: num_initial_random uniformly-random trials (RandomSample), then num_iterations trials each chosen by splitting all trials so far into good/bad by the gamma quantile (maximization convention: good = highest objective values) and picking, among num_candidates uniformly-random candidates, the one maximizing LogDensityRatio.

Detailed Description

Tree-structured Parzen Estimator (Bergstra, Bardenet, Bengio, Kegl, "Algorithms for Hyper-Parameter Optimization," NeurIPS 2011) – a structurally distinct second surrogate family from GP-BO (gp_bo.hpp), handling mixed/categorical search spaces GP-BO's own vanilla kernel cannot (gp_bo.hpp restricts to Continuous/LogUniform; this file supports every ParameterKind).

Note

Scope simplifications from the original paper, both logged here rather than left implicit: (1) per-parameter densities use a fixed-bandwidth Gaussian mixture (bandwidth derived once from each parameter's own declared range), not the original paper's adaptive per-observation bandwidth (spacing to each point's nearest neighbors); (2) candidates are proposed by drawing uniformly from the SearchSpace's own prior (RandomSample, hpo_sampling.hpp) and scoring each by l(x)/g(x), not by drawing directly from the fitted l(x) density itself (the original paper's actual strategy, which requires being able to sample from an arbitrary fitted mixture, not just evaluate its density). Both are well-precedented, simpler variants that preserve TPE's defining idea (score candidates by a good/bad density ratio, built independently per parameter so mixed/categorical spaces compose naturally) without the original paper's full generative-sampling machinery.