A single recipe — pretext task on unlabeled data, then transfer — now powers almost every modality:

The empirical payoff is huge — but the theory is scattered across paradigms.

1. Sufficiency — spectral factorization of the PMI \(\frac{\mathbb{P}(x',x)}{\mathbb{P}(x')\mathbb{P}(x)}\) gives a universal sufficient representation.
2. Unified framework — every SSL method learns it; they differ only in parametrization × optimization.
3. New algorithms — better stochastic estimators, by design.
4. Applications — regression, causal inference, RL.

Let \(\psi(x)\) span the union of all SVD subspaces of \(\mathbb{P}(y\mid x)\) over every task (the core set, with \(\mathbb{P}(z\mid x) = \langle \nu(z),\, \psi(x)\rangle\)). Then for any neighbor \(x'\):

Since \(A = B^\top B\) is full-rank, define \(\varphi(x) := B\,\psi(x)\). Then

\(\varphi\) and \(\psi\) span the same space ⇒ \(\mathbb{E}[y|x] = \langle \varphi(x), w\rangle\) for every task.

This recipe will now walk through the four parametrizations in turn.

Recall the spectral parametrization of the PMI as an inner product of spectral factors:

Rescale by the marginal densities to get a tractable target:

Spectral learning = find \(\varphi\) such that \(\langle\varphi(x'),\varphi(x)\rangle\) matches \(T\) on observed pairs.

Mean-square match of \(T\) against the rank-\(d\) reconstruction \(\sqrt{\mathbb{P}(x)\mathbb{P}(x')}\langle\varphi(x'),\varphi(x)\rangle\):

• First term: alignment on positive pairs (from \(\mathbb{P}(x,x')\)).
• Second term: uniformity on negative pairs (independent \(\mathbb{P}(x)\mathbb{P}(x')\)).
• Both expectations admit unbiased minibatch estimators ⇒ SGD-friendly.

Introduce a full-rank linear projection \(\xi(x) = Z^\top\varphi(x)\) with \(Z = U\Lambda^{-1}\) and \(A = \mathbb{E}[\varphi(x)\varphi(x)^\top] = U\Lambda U^\top\). The SVD condition becomes

Minimizing squared error from \(\mathbf{I}\) with a regularization coefficient \(\lambda\) recovers the exact Barlow-Twins objective:

Linear projection \(Z=U\Lambda^{-1/2}\) — orthonormality (Eq. 15) & variational SVD characteristic (Eq. 16):

Combine via the penalty method:

Fixed-point condition of the spectral rep \(\xi = Z^\top\varphi\):

Iterate three conceptual steps:

1. \(\Lambda_{t+1} \leftarrow \mathbb{E}[\xi_t(x)\xi_t(x)^\top]\)  — EMA of covariance.
2. match \(\xi_{t+1}(x) \approx \Lambda^{-1}\!\int T\,\xi_t(x')\,dx'\)  — predictor head on the online branch.
3. orthogonalize  — self-normalization or Gram-Schmidt.

Folklore: contrastive needs large batches because of negatives. Actual culprit: bias in the stochastic gradient.

Implication: the contrastive-vs-non-contrastive debate is miscast — the real axis is bias vs unbiased gradient.

Replace the linear factorization with an energy-based one:

\(v(x)\in\mathbb{R}^d\) is low-dimensional, yet the spectral factor \(\varphi\) implied by this parametrization will be infinite-dimensional.

Expand the quadratic exponent \(\exp(v(x')^\top v(x)) = e^{\|v(x')\|^2/2}\,e^{-\|v(x')-v(x)\|^2/2}\,e^{\|v(x)\|^2/2}\). The middle factor is a Gaussian kernel; Bochner gives

Generalizes to any positive-definite kernel on \(v(x),v(x')\) — used later for nonlinear projector view.

Plug EBM into the \(k\)-way posterior \(p_\theta(k|\{x'_i\},x) = \dfrac{e^{v(x'_k)^\top v(x)}}{\sum_j e^{v(x'_j)^\top v(x)}}\). Minimizing the multi-class log-loss:

Restricting \(\|v(x)\|_2=1\) (L2-normalize) gives the exact SimCLR loss.

Gaussian-corrupt \(\tilde x'\sim\mathcal{N}(\sqrt{1-\beta}x', \beta\sigma^2 I)\) — still EBM spectral. Tweedie's identity:

For the Gaussian case, with \(\zeta(\tilde x',\beta):=\nabla_{\tilde x'}\mu(\tilde x',\beta)\):

\(\Rightarrow\) unbiased denoising loss; one network = generator + spectral rep \(v(x)\):

The latent marginal \(\mathbb{P}(z|x)\) gives a spectral decomposition of the transition operator:

where \(\Lambda(z)=\tfrac{1}{\mathbb{P}(z)}\) is a diagonal operator, so \(\sqrt{\Lambda(z)}\,\mathbb{P}(z|x)\) spans the same function space as \(\mathbb{P}(z|x)\).

Trade-off: simplex constraint can cost dimensions (Prop 2 of Agarwal et al. 2020) but enables probabilistic modeling.

Softmax \(\mathbb{P}(z|x)\) as an EBM (Eq. 78), equal to the Gaussian form (Eq. 79):

connection: \(W_z = \tfrac{Cz}{\sigma^2}\),   \(b_z = \tfrac{1}{2\sigma^2}\,z^\top C^\top C z\),   \(z\in\{0,1\}^k,\ z^\top\mathbf{1}=1\) — DeepCluster treats \(C,W\) as independent (suboptimal).

Maximum likelihood = variational ELBO, optimized by EM:

Stationary property \(\int \mathbb{P}(z|x')\,\mathbb{P}(x'|x)\,dx' = \mathbb{P}(z|x)\); match by KL (power iteration):

Softmax \(\mathbb{P}(z|x)\propto\exp(v(x)^\top W_z + b_z)\) gives:

Differ only in \(\mathbb{P}_t(z|x)\): DINO = previous-iteration teacher (EMA, centering/sharpening); SwAV = optimal-transport (Sinkhorn) assignment — the same fixed-point rule.

Given paired \((x,y)\in\) (image, text), (audio, video), …

energy-based conditionals (Eq. 120):

• CLIP: rank-NCE on the multi-modal EBM — log-sum-exp over a batch.
• SigLIP: binary NCE — unbiased gradient, but adds a temperature to compensate for the unnormalized implicit assumption.

Downstream classification (linear) & retrieval (inner product) follow the same sufficient-rep argument.

MLE of the multi-modal EBM requires computing \(Z(x), Z(y)\). Three ways to sidestep:

With sufficient spectral rep \(\varphi(x)\), so that \(\mathbb{P}(y\mid x) = \langle \varphi(x), \mu(y) \rangle\), the optimal regressor and Bayes classifier both reduce to linear maps:

This justifies why linear probes are the accepted SSL evaluation protocol — they are the optimal downstream.

Spectral factorization of the transition \(\mathbb{P}(s'|s,a) = \langle\varphi(s,a),\mu(s')\rangle\)  +  \(r(s,a)=\langle\varphi(s,a),\theta_r\rangle\) implies, for any policy \(\pi\):

• Linear LSTD ⇒ efficient TD learning.
• Posterior over \(\eta^\pi\) ⇒ tractable exploration bonuses.
• Works for low-rank, partial-observable, and continuous-control MDPs (Ren · Zhang · Shribak · Gao · Nabati et al.).

Same spectral rep that underlies SSL vision models can be learned for dynamics ⇒ one encoder for perception, control, and planning.

Controllable synthesis in generative models, animal-behavior discovery, and off-policy evaluation also fit — see the paper for details.

They differ only in how they parametrize the rep and how they optimize the underlying decomposition.

Acknowledgments. NSF AI Institute (2112085), NSF ECCS-2401390, ECCS-2401391, ONR N000142512173, NSF IIS-2403240.

Collaborators across the line of work: Na Li, Dale Shuurmans, Tongzheng Ren, Haotian Sun, Chenxiao Gao, Tianjun Zhang, Hongming Zhang, Zhaolin Ren, Haitong Ma,Dmitry Shribak, Yang Hu, Zhaohan Daniel Guo, Jiyi Wang, Ofir Nabati, Niao He, Hanjun Dai, Ofir Nachum, and many more.

Denote \(A := \mathbb{E}[\varphi(x)\varphi(x)^\top] = U\Lambda U^\top\). Choose \(Z=U\Lambda^{-1}\), \(\xi(x)=Z^\top\varphi(x)\). SVD variational form:

Therefore \(\min_\xi \|\mathbf{I} - \mathbb{E}_{\mathbb{P}(x,x')}[\xi(x)\xi(x')^\top]\|_F^2\). Let

Bias. \((1-C_{ii})^2\) is a square-of-expectation ⇒ stochastic gradient is biased; requires large batches.

Set \(Z=U\); fixed-point eigenvalue condition:

Iterate:  learn \(\Lambda\) with \(\xi_t\) fixed:

then learn \(\xi_{t+1}\) with \(\Lambda\) fixed:

Together = exact BYOL (Grill et al. 2020). \(\Lambda\) = predictor head; \(\xi_t\) = target network.

MINC variant uses a generalized Hebbian rule with lower-triangular matrix operation for orthogonalization (Guo et al. 2025).

EBM \(\mathbb{P}(x'|x) \propto \mathbb{P}(x')\exp(v(x')^\top v(x))\). Plug into rank posterior:

Multi-class logistic regression:

With L2-normalization \(v \to v/\|v\|_2\): exact SimCLR.

Partition-function handling. EBM with normalizer \(Z(x) = \int\mathbb{P}(x')e^{v(x')^\top v(x)}dx'\) can be absorbed into \(\phi_\omega(x)\) via Fourier decomposition (Eq. 53 of the paper).

Latent spectral property: \(\int\mathbb{P}(z|x')\mathbb{P}(x'|x)dx' = \mathbb{P}(z|x)\). KL-match at iteration \(t\):

Softmax \(\mathbb{P}(z|x) \propto \exp(v(x)^\top W_z + b_z)\) gives

• DINO: \(\mathbb{P}_t(z|x)\) = previous iteration teacher ⇒ EMA + centering + sharpening.
• SwAV: \(\mathbb{P}_t(z|x)\) via optimal-transport Sinkhorn (same as SeLa E-step).

Gaussian-noised sample \(\tilde x'\sim\mathcal{N}(\sqrt{1-\beta}x',\beta\sigma^2 I)\):

Tweedie's identity: \(\nabla_{\tilde x'}\log\mathbb{P}(\tilde x'|x;\beta) = \mathbb{E}_{\mathbb{P}(x'|\tilde x',x;\beta)}[\nabla_{\tilde x'}\log\mathbb{P}(\tilde x'|x';\beta)]\), Gaussian case:

Learn \(v, \zeta\) by the denoising objective (above). Same network provides the generative model and the spectral representation.

Softmax \(\mathbb{P}(z|x)\) with Gaussian emission \(\mathbb{P}(v(x)|z) = \mathcal{N}(Cz,\sigma^2 I)\):

Blockwise-Gibbs sampling of \((x,z)\) via Langevin on \(x\)-side, softmax on \(z\)-side ⇒ Hinton's contrastive divergence on the latent spectral model.

Fenchel dual of the intractable f-divergence:

Continuous \(z\): reparameterization.  Discrete \(z\): enumerate.  Min-max saddle — needs stabilization in practice.