pulsatrix 1.1.0
Loading...
Searching...
No Matches
uncertainty.hpp File Reference

How sure a model is of each prediction (KS-3): Monte Carlo dropout (Gal and Ghahramani, ICML 2016), deep ensembles (Lakshminarayanan, Pritzel and Blundell, NeurIPS 2017), and the split of a prediction's uncertainty into the part more data would remove and the part it wouldn't. More...

#include <cstdint>
#include <functional>
#include <vector>
#include "pulsatrix/device_backend.hpp"
#include "pulsatrix/json.hpp"
#include "pulsatrix/module.hpp"
#include "pulsatrix/tensor.hpp"
Include dependency graph for uncertainty.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::UncertaintyDecomposition
 DecomposeUncertainty()'s result: one value per input, entropies in nats. More...
 
struct  pulsatrix::GaussianMixturePrediction
 CombineGaussians()' result: one value per input. More...
 
class  pulsatrix::GaussianNLLLoss
 The Gaussian negative log likelihood, for regressors that predict their own noise (Nix and Weigend, 1994; Lakshminarayanan et al., 2017): the network outputs two values per input, a mean mu and s, and its variance is softplus(s) + min_variance, which keeps it positive. More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< Tensor > pulsatrix::McDropoutSamples (Module &model, const std::function< Tensor(const Tensor &)> &predict, const Tensor &input, int64_t samples)
 Monte Carlo dropout: runs predict on input samples times with every dropout layer under model in training mode and everything else in eval mode, so BatchNorm uses its running statistics and only the dropout masks change between passes.
 
std::vector< Tensor > pulsatrix::EnsembleSamples (const std::vector< std::function< Tensor(const Tensor &)> > &members, const Tensor &input)
 A deep ensemble's samples: each member's output for input.
 
UncertaintyDecomposition pulsatrix::DecomposeUncertainty (const std::vector< Tensor > &samples)
 Splits the uncertainty of a set of probability samples into its aleatoric and epistemic parts, input by input.
 
GaussianMixturePrediction pulsatrix::CombineGaussians (const std::vector< std::vector< double > > &means, const std::vector< std::vector< double > > &variances={})
 Mixes M Gaussian predictions per input into one, as Lakshminarayanan et al. do: the mixture's mean is the mean of the means, and its variance the mean of (variance + mean^2) minus the mixture mean squared.
 
JsonValue pulsatrix::ToJson (const UncertaintyDecomposition &decomposition)
 The decomposition as a pulsatrix.uncertainty.v1 document, without the mean probabilities.
 

Detailed Description

How sure a model is of each prediction (KS-3): Monte Carlo dropout (Gal and Ghahramani, ICML 2016), deep ensembles (Lakshminarayanan, Pritzel and Blundell, NeurIPS 2017), and the split of a prediction's uncertainty into the part more data would remove and the part it wouldn't.

One deterministic network gives one probability per class, and nothing says how much to trust it. Several predictions of the same input that disagree do. Monte Carlo dropout gets them from one network by keeping dropout on at inference; a deep ensemble from several networks trained from different seeds. Either way the result is a set of samples of the predictive distribution, and DecomposeUncertainty() splits its entropy:

  • total, the entropy of the mean prediction, H[E p];
  • aleatoric, the mean entropy of the samples, E H[p]: what each one is unsure of, noise in the data that more data wouldn't remove;
  • epistemic, the difference, the mutual information between the prediction and the model (BALD; Houlsby et al., 2011): how much the samples disagree, which more data would reduce.

For regression, each member predicts a mean and a variance, trained with GaussianNLLLoss, and CombineGaussians() mixes them the same way.

The decomposition is only as good as the samples. Dropout samples come from one network and tend to agree more than an ensemble's; neither is the true posterior.