53 const std::vector<bool>& train,
const std::vector<bool>& test,
DeviceBackend* backend,
67[[nodiscard]] std::vector<int>
ControlTaskLabels(
const std::vector<int64_t>& token_types,
const std::vector<int>& labels,
int classes,
89 const std::vector<int64_t>& token_types,
const std::vector<bool>& train,
113 int64_t baseline_dim,
const std::vector<int>& labels,
int classes,
const std::vector<bool>& train,
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Abstract interface isolating vendor-specific memory/compute operations from Tensor/ComputationGraph.
Definition acquisition_functions.hpp:16
ProbeGain CompareToBaseline(const std::vector< float > &features, int64_t dim, const std::vector< float > &baseline, int64_t baseline_dim, const std::vector< int > &labels, int classes, const std::vector< bool > &train, const std::vector< bool > &test, DeviceBackend *backend, const ProbeOptions &options={})
Trains the same probe on features and on baseline, rows matched, and reports how much better the repr...
std::vector< int > ControlTaskLabels(const std::vector< int64_t > &token_types, const std::vector< int > &labels, int classes, uint64_t seed)
Hewitt and Liang's control labels: each distinct value in token_types gets one label,...
ProbeResult TrainLinearProbe(const std::vector< float > &features, int64_t dim, const std::vector< int > &labels, int classes, const std::vector< bool > &train, const std::vector< bool > &test, DeviceBackend *backend, const ProbeOptions &options={})
Trains a linear softmax probe (standardized features, AdamW, cross-entropy) on the rows marked in tra...
ProbeSelectivity MeasureSelectivity(const std::vector< float > &features, int64_t dim, const std::vector< int > &labels, int classes, const std::vector< int64_t > &token_types, const std::vector< bool > &train, const std::vector< bool > &test, DeviceBackend *backend, const ProbeOptions &options={})
Trains one probe on labels and one on ControlTaskLabels(token_types, labels, classes,...
A probe on a representation beside the same probe on a baseline representation.
Definition probe_controls.hpp:93
ProbeResult probe
Definition probe_controls.hpp:94
double balanced_gain
The same with balanced accuracy.
Definition probe_controls.hpp:99
double gain
probe.accuracy - baseline.accuracy.
Definition probe_controls.hpp:97
ProbeResult baseline
Definition probe_controls.hpp:95
double auc_gain
The same with ROC AUC, for two classes; NaN otherwise.
Definition probe_controls.hpp:101
Definition probe_controls.hpp:27
int64_t epochs
Definition probe_controls.hpp:28
uint64_t seed
Definition probe_controls.hpp:32
float learning_rate
Definition probe_controls.hpp:30
int64_t batch
Definition probe_controls.hpp:29
float weight_decay
Definition probe_controls.hpp:31
A probe's quality on held-out rows.
Definition probe_controls.hpp:36
double auc
For two classes, the ROC AUC of class 1's probability; NaN otherwise.
Definition probe_controls.hpp:41
double balanced_accuracy
The mean recall over classes, so a rare class counts as much as a common one.
Definition probe_controls.hpp:39
int64_t train
Definition probe_controls.hpp:42
double accuracy
Definition probe_controls.hpp:37
int64_t test
Definition probe_controls.hpp:43
A probe on the real task beside the same probe on the control task.
Definition probe_controls.hpp:71
double selectivity
task.accuracy - control.accuracy, Hewitt and Liang's selectivity.
Definition probe_controls.hpp:75
ProbeResult task
Definition probe_controls.hpp:72
double balanced_selectivity
The same with balanced accuracy, for imbalanced labels.
Definition probe_controls.hpp:77
ProbeResult control
Definition probe_controls.hpp:73
int64_t train_types
How many token types the training rows hold. Few types make the control easy to learn; with no more t...
Definition probe_controls.hpp:80