pulsatrix 1.1.0
Loading...
Searching...
No Matches
embedding_export.hpp File Reference

Hand embeddings or activations to UMAP and t-SNE tools, and bring their coordinates back as a projection document. More...

#include <cstdint>
#include <string>
#include <vector>
#include "pulsatrix/tensor.hpp"
Include dependency graph for embedding_export.hpp:

Go to the source code of this file.

Classes

struct  pulsatrix::MetadataColumn
 One column of per-point metadata: a label, a class, a value as text. More...
 

Namespaces

namespace  pulsatrix
 

Functions

std::vector< uint8_t > pulsatrix::SerializeNpy (const Tensor &x)
 A tensor as a .npy file: format version 1.0, little-endian float32 (<f4), C order, with the tensor's shape.
 
void pulsatrix::WriteNpy (const std::string &path, const Tensor &x)
 SerializeNpy() written to path.
 
Tensor pulsatrix::ParseNpy (const std::vector< uint8_t > &bytes, DeviceBackend *backend)
 A .npy file as a float32 tensor on backend, with the file's shape.
 
Tensor pulsatrix::ReadNpy (const std::string &path, DeviceBackend *backend)
 ParseNpy() of the file at path.
 
std::string pulsatrix::ProjectorVectorsTsv (const Tensor &x)
 The Embedding Projector's tensor file: a line per row of x, its values separated by tabs. Dimensions after the first are flattened.
 
std::string pulsatrix::ProjectorMetadataTsv (const std::vector< MetadataColumn > &columns)
 The Embedding Projector's metadata file: a line per point, a column per entry of columns, separated by tabs.
 
void pulsatrix::ExportEmbeddings (const std::string &directory, const Tensor &x, const std::vector< MetadataColumn > &columns={})
 Writes x for UMAP and t-SNE tools into directory, which must exist: vectors.npy, vectors.tsv, and metadata.tsv when there are columns.
 

Detailed Description

Hand embeddings or activations to UMAP and t-SNE tools, and bring their coordinates back as a projection document.

pulsatrix doesn't implement UMAP or t-SNE. Both are expensive, have many settings that change the picture, and are well served elsewhere. Both also distort the data: they keep each point's nearest neighbors, but distances between clusters, cluster sizes and densities in their plots carry little meaning, and an elongated or empty region can come from the settings alone. Use PCA (pca.hpp) for a view whose distances mean something, and these for a view of the neighborhoods.

This header writes the data in the formats those tools read:

  • **.npy** (SerializeNpy(), WriteNpy()): numpy's own format, which umap-learn, openTSNE and scikit-learn's TSNE take after np.load.
  • TSV for the Embedding Projector (ProjectorVectorsTsv(), ProjectorMetadataTsv()): projector.tensorflow.org runs UMAP, t-SNE and PCA in the browser on uploaded files, with nothing to install.

ExportEmbeddings() writes all of them to a directory. ParseNpy() and ReadNpy() read the coordinates the tools return, and MakeProjectionDocument() in viz/document.hpp turns them into a pulsatrix.projection.v1 document for the SVG, HTML and notebook views, which mark UMAP and t-SNE plots as distorting distances.