pulsatrix 1.1.0
Loading...
Searching...
No Matches
probe_controls.hpp
Go to the documentation of this file.
1
18#pragma once
19
20#include <cstdint>
21#include <vector>
22
24
25namespace pulsatrix {
26
28 int64_t epochs = 20;
29 int64_t batch = 1024;
30 float learning_rate = 1e-2f;
31 float weight_decay = 1e-4f;
32 uint64_t seed = 0;
33};
34
37 double accuracy = 0;
41 double auc = 0;
42 int64_t train = 0;
43 int64_t test = 0;
44};
45
52[[nodiscard]] ProbeResult TrainLinearProbe(const std::vector<float>& features, int64_t dim, const std::vector<int>& labels, int classes,
53 const std::vector<bool>& train, const std::vector<bool>& test, DeviceBackend* backend,
54 const ProbeOptions& options = {});
55
67[[nodiscard]] std::vector<int> ControlTaskLabels(const std::vector<int64_t>& token_types, const std::vector<int>& labels, int classes,
68 uint64_t seed);
69
82
88[[nodiscard]] ProbeSelectivity MeasureSelectivity(const std::vector<float>& features, int64_t dim, const std::vector<int>& labels, int classes,
89 const std::vector<int64_t>& token_types, const std::vector<bool>& train,
90 const std::vector<bool>& test, DeviceBackend* backend, const ProbeOptions& options = {});
91
93struct ProbeGain {
97 double gain = 0;
99 double balanced_gain = 0;
101 double auc_gain = 0;
102};
103
112[[nodiscard]] ProbeGain CompareToBaseline(const std::vector<float>& features, int64_t dim, const std::vector<float>& baseline,
113 int64_t baseline_dim, const std::vector<int>& labels, int classes, const std::vector<bool>& train,
114 const std::vector<bool>& test, DeviceBackend* backend, const ProbeOptions& options = {});
115
116} // namespace pulsatrix
Vendor-agnostic compute/memory backend. CPUBackend, CUDABackend (Phase 1.5), and HIPBackend (Phase 1....
Definition device_backend.hpp:248
Abstract interface isolating vendor-specific memory/compute operations from Tensor/ComputationGraph.
Definition acquisition_functions.hpp:16
ProbeGain CompareToBaseline(const std::vector< float > &features, int64_t dim, const std::vector< float > &baseline, int64_t baseline_dim, const std::vector< int > &labels, int classes, const std::vector< bool > &train, const std::vector< bool > &test, DeviceBackend *backend, const ProbeOptions &options={})
Trains the same probe on features and on baseline, rows matched, and reports how much better the repr...
std::vector< int > ControlTaskLabels(const std::vector< int64_t > &token_types, const std::vector< int > &labels, int classes, uint64_t seed)
Hewitt and Liang's control labels: each distinct value in token_types gets one label,...
ProbeResult TrainLinearProbe(const std::vector< float > &features, int64_t dim, const std::vector< int > &labels, int classes, const std::vector< bool > &train, const std::vector< bool > &test, DeviceBackend *backend, const ProbeOptions &options={})
Trains a linear softmax probe (standardized features, AdamW, cross-entropy) on the rows marked in tra...
ProbeSelectivity MeasureSelectivity(const std::vector< float > &features, int64_t dim, const std::vector< int > &labels, int classes, const std::vector< int64_t > &token_types, const std::vector< bool > &train, const std::vector< bool > &test, DeviceBackend *backend, const ProbeOptions &options={})
Trains one probe on labels and one on ControlTaskLabels(token_types, labels, classes,...
A probe on a representation beside the same probe on a baseline representation.
Definition probe_controls.hpp:93
ProbeResult probe
Definition probe_controls.hpp:94
double balanced_gain
The same with balanced accuracy.
Definition probe_controls.hpp:99
double gain
probe.accuracy - baseline.accuracy.
Definition probe_controls.hpp:97
ProbeResult baseline
Definition probe_controls.hpp:95
double auc_gain
The same with ROC AUC, for two classes; NaN otherwise.
Definition probe_controls.hpp:101
Definition probe_controls.hpp:27
int64_t epochs
Definition probe_controls.hpp:28
uint64_t seed
Definition probe_controls.hpp:32
float learning_rate
Definition probe_controls.hpp:30
int64_t batch
Definition probe_controls.hpp:29
float weight_decay
Definition probe_controls.hpp:31
A probe's quality on held-out rows.
Definition probe_controls.hpp:36
double auc
For two classes, the ROC AUC of class 1's probability; NaN otherwise.
Definition probe_controls.hpp:41
double balanced_accuracy
The mean recall over classes, so a rare class counts as much as a common one.
Definition probe_controls.hpp:39
int64_t train
Definition probe_controls.hpp:42
double accuracy
Definition probe_controls.hpp:37
int64_t test
Definition probe_controls.hpp:43
A probe on the real task beside the same probe on the control task.
Definition probe_controls.hpp:71
double selectivity
task.accuracy - control.accuracy, Hewitt and Liang's selectivity.
Definition probe_controls.hpp:75
ProbeResult task
Definition probe_controls.hpp:72
double balanced_selectivity
The same with balanced accuracy, for imbalanced labels.
Definition probe_controls.hpp:77
ProbeResult control
Definition probe_controls.hpp:73
int64_t train_types
How many token types the training rows hold. Few types make the control easy to learn; with no more t...
Definition probe_controls.hpp:80