Spectral Representation-based Reinforcement Learning

Balancing modeling, exploration and planning via spectral representations

Accepted by JMLR

Bo Dai

Google DeepMind & Georgia Tech

Team

Collaborators

Tongzheng Ren

UT Austin Alumni

Chenjun Xiao

UAlberta

Tianjun Zhang

UC Berkeley

Haitong Ma

Harvard

Zhaolin Ren

Harvard

Chenxiao Gao

Georgia Tech

Lisa Lee

Google DeepMind

Sherry Yang

Google DeepMind & NYU

Sujay Sanghavi

UT Austin

Na Li

Harvard

Csaba Szepesvári

Google DeepMind & UAlberta

Dale Schuurmans

Google DeepMind & UAlberta

Outline

Roadmap

1 · Dilemma in RL

Difficulties in model-free and model-based RL (world models).

2 · An inspiration from representation for control

Provable and practical stochastic nonlinear control.

3 · Coherent solution: RL with linear representation

Linear representation for MDPs.

4 · Beyond states: POMDPs, videos and the real world

Linear representation for POMDPs, foundation-model encoders, video-based RL.

Dilemma in Reinforcement Learning

Model-free and model-based RL each win on some axes and lose on others — modeling, exploration, planning.

Motivation

The power of reinforcement learning

Games — AlphaGo vs. Lee Sedol

Control — autonomous driving, robotics

Background

Markov Decision Processes (MDPs)

\(\mathcal{M} = \langle \mathcal{S}, \mathcal{A}, r, T, \mu, H/\gamma \rangle\)

State space: \(\mathcal{S}\)
Action space: \(\mathcal{A}\)
Reward: \(r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\)
Transition: \(T:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})\)
Initial distribution: \(\mu\)
Policy: \(\pi(\cdot|s):\mathcal{S}\to\Delta(\mathcal{A})\)

\[ V^\pi(s) := \mathbb{E}_{T,\pi}\Bigl[\textstyle\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)\,\Big|\,s_0=s\Bigr] \]

\[ Q^\pi(s,a) := \mathbb{E}_{T,\pi}\Bigl[\textstyle\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)\,\Big|\,s_0=s,a_0=a\Bigr] \]

Finite-horizon: \(V_h^\pi, Q_h^\pi\) with \(\sum_{t=h}^{H-1}\) instead.

\[ \max_\pi\; J(\pi) = \mathbb{E}_{\mu(s)}\bigl[V^\pi(s)\bigr] \]

the RL objective

Model-free RL

(Deep) Q-learning

Dynamic programming via the Bellman recursion:

\[ Q(s,a) = R(s,a) + \gamma \sum_{s'} T(s'|s,a)\,\max_{a'} Q(s',a') \]

TD update

\[ Q_t(s,a) = Q_{t-1}(s,a) + \alpha\Bigl(R(s,a) + \gamma\max_{a'}Q(s',a') - Q_{t-1}(s,a)\Bigr) \]

Deep version

\[ \theta_t = \theta_{t-1} + \alpha\Bigl(R(s,a) + \gamma\max_{a'}Q_{t-1}(s',a') - Q_{t-1}(s,a)\Bigr)\nabla_\theta Q(s,a) \]

Model-free RL

Policy gradient

Direct policy optimization:

\[ J(\pi_\theta) = \sum_{s} d^{\pi_\theta}(s)V^{\pi_\theta}(s) = \sum_s d^{\pi_\theta}(s)\sum_a \pi_\theta(a|s)\,Q^{\pi_\theta}(s,a) \]

Policy gradient

\[ \nabla_\theta J(\theta) = \sum_s d^\pi(s)\sum_a Q^\pi(s,a)\,\pi_\theta(a|s)\nabla_\theta\log\pi_\theta(a|s) = \mathbb{E}_{T,\pi}\bigl[Q^\pi(s,a)\nabla_\theta\log\pi_\theta(a|s)\bigr] \]

PG update (→ natural PG, soft actor-critic, …)

\[ \theta_t = \theta_{t-1} + \alpha\,\mathbb{E}_{T,\pi}\bigl[Q^\pi(s,a)\nabla_\theta\log\pi_\theta(a|s)\bigr] \]

Model-free RL

Model-free RL: pros & cons

Pros

Modeling: easy to combine with function approximators (deep nets) and gradient-based learning.

Cons

Exploration: hard to capture uncertainty with arbitrary nonlinear functions.
Planning: no global-convergence guarantee for the optimal policy with general nonlinear functions.

Model-based RL

The model-based RL loop

Three instantiations of the same loop — differing only in the model class:

Generic MBRL

Estimate the dynamics model and reward; model predictive control on the learned model.

LQR

Estimate a linear dynamics model and quadratic reward; optimize the estimated LQR.

Deep MBRL / world model

Estimate a deep world model and reward; MPC on the learned model.

Model-based RL

Linear Quadratic Regulator

\[ \begin{aligned} \min_{u_t,x_t}\;&\mathbb{E}\Bigl[\tfrac12\textstyle\sum_{t=0}^{N}\bigl\{x_t^\top Q x_t + u_t^\top R u_t\bigr\} + \tfrac12 x_{N+1}^\top S x_{N+1}\Bigr] \\ \text{s.t.}\;& x_{t+1} = A x_t + B u_t + e_t,\quad t=0,1,\dots,N \end{aligned} \]

With a given model, we have efficient solutions & elegant analysis.

Pros

Exploration: theoretically rigorous and computationally efficient uncertainty estimation.
Planning: elegant planner with global-convergence guarantee.

Cons

Modeling: linear dynamics is too restrictive.

Model-based RL

Deep model-based RL / world models

ME-TRPO MB-MF PETS-RS / PETS-CEM DINO-WM: DINO as a foundation for video world models

Pros

Modeling: exploits deep models for better approximation.

Cons

Exploration: hard to capture uncertainty with arbitrary nonlinear functions.
Planning: difficult to control with a nonlinear dynamics model.

Dilemma

The impossible triangle: modeling, exploration, planning

	Modeling	Exploration	Planning
Model-free (deep)	✓	✗	✗
LQR	✗	✓	✓
Deep MBRL / WM	✓	✗	✗
Spectral Rep. RL	✓	✓	✓

A practical algorithm with rigorous theoretical guarantee that balances all three? → Spectral Representation-based Reinforcement Learning

Representation View for Provable Control

Stochastic nonlinear control becomes linear in the right feature space.

Control

Stochastic nonlinear control as an MDP

Stochastic nonlinear control

\[ \min_\pi\;\mathbb{E}_\pi\Bigl[\textstyle\sum_{h=1}^{H} r(s_h,a_h)\Bigr]\quad \text{s.t.}\;\; s_{h+1} = f(s_h,a_h) + \epsilon_h,\;\; \epsilon_h\sim\mathcal{N}(0,\sigma^2 I) \]

MDP reformulation \(\mathcal{M} = \langle\mathcal{S},\mathcal{A},r,T,\mu,H\rangle\)

\[ T(s'|s,a) \;\propto\; \exp\Bigl(-\frac{\|s'-f(s,a)\|_2^2}{2\sigma^2}\Bigr) \]

\[ T(s'|s,a) \;\propto\; \bigl\langle \phi_\omega(f(s,a)),\, \phi_\omega(s') \bigr\rangle_{\mathcal{N}(0,\sigma^{-2}I)},\qquad \phi_\omega(x) = [\cos(\omega^\top x),\,\sin(\omega^\top x)] \]

Gaussian kernel = inner product of random Fourier features

Control

Factorized dynamics ⇒ linear Q-functions

Transition and reward factorize with \(\phi(s,a)\):

\[ T(s'|s,a) = \langle \phi(s,a),\, \mu(s') \rangle,\qquad r(s,a) = \langle \phi(s,a),\, \theta_r\rangle \]

Integration representable:

\[ \int V^\pi_{h+1}(s_{h+1})\,T(s_{h+1}|s_h,a_h)\,ds_{h+1} = \Bigl\langle \phi(s_h,a_h),\, \int V^\pi_{h+1}(s_{h+1})\,\mu(s_{h+1})\,ds_{h+1}\Bigr\rangle_{\mathcal{H}} \]

Q-function is linearly representable:

\[ Q^\pi_h(s_h,a_h) = r(s,a) + \int T(s_{h+1}|s_h,a_h)V^\pi_{h+1}(s_{h+1})\,ds_{h+1} = \Bigl\langle \phi(s,a),\, \underbrace{\theta_r + \textstyle\int V^\pi_{h+1}\,\mu(s_{h+1})\,ds_{h+1}}_{w}\Bigr\rangle_{\mathcal{H}} \]

KeyNot a special model, but a generic structure.

Control

Planning for stochastic nonlinear control

Given an arbitrary bounded nonlinear \(f\), construct \(\phi(s,a)\) for value functions; then optimize by linear dynamic programming in the obtained space.

Least-squares policy evaluation

for \(t=0,\dots,T-1\):

\[ \hat w_{k,t+1} = \arg\min_w \sum_{i}\Bigl(\phi(s_i,a_i)^\top w - r(s_i,a_i) - \gamma\,\phi(s_i',a_i')^\top \hat w_{k,t}\Bigr)^2 \]

Natural policy gradient for control

\[ \theta_{k+1} = \theta_k + \eta\,\hat w_{k,T} \]

\[ \pi_{k+1}(a|s) \propto \exp\bigl(\phi(s,a)^\top\theta_{k+1}\bigr) \]

Ren et al., CDC 2023 / IEEE TAC 2025

Control · Experiments

Empirical performance

SDEC (random-feature / Nyström spectral dynamics embedding) vs. iLQR, DKUC, energy-based, SAC.

Control · Summary

Summary and gaps

Take-home

Linearization makes nonlinear control potentially solvable.
Linearization bridges model-free and model-based RL.

Gaps: theory vs. practice

Infinite-dim linearization approximation.
Gaussian noise assumption.

Could we do better and avoid these limitations?

RL with Linear Representation

Learn \(\phi(s,a)\) from data — tractably — then explore and plan linearly.

Linear MDP

Spectral view of MDPs

\[ T(s'|s,a) \propto \exp\Bigl(-\frac{\|s'-f(s,a)\|_2^2}{2\sigma^2}\Bigr) = \bigl\langle \phi_\omega(f(s,a)),\,\phi_\omega(s')\bigr\rangle_{\mathcal{N}(0,\sigma^{-2}I)} \]

Generalize: learn the factorization directly instead of fixing the kernel.

\[ \boxed{\;T(s'|s,a) = \langle\phi(s,a),\,\mu(s')\rangle,\qquad r(s,a) = \langle\phi(s,a),\,\theta_r\rangle\;} \]

The MLE of \(T\) under this constraint is intractable — four tractable surrogates follow.

Contrastive (NCE) SVD Latent variable (ELBO) Diffusion (score)

Linear MDP · Contrastive

Contrastive representation (NCE / CPC)

A contrastive loss as a tractable alternative to the MLE:

\[ (s,a,s')\sim\mathcal{D},\qquad s_l\sim p(s') \]

\[ \max_{\phi,\mu}\;\hat{\mathbb{E}}\Bigl[\langle\phi(s,a),\mu(s')\rangle - \log\sum_l \langle\phi(s,a),\mu(s_l')\rangle\Bigr] \]

The objective leads to the solution

\[ T(s'|s,a) = \bigl\langle \phi(s,a),\; {\color{#3b82f6}p(s')}\,\mu(s')\bigr\rangle \]

Zhang, Ren et al., ICML 2022 (CTRL)

Linear MDP · SVD

Spectral decomposition representation (SVD)

SVD as a tractable alternative to the MLE:

\[ T(s'|s,a) = \langle\phi(s,a),\,{\color{#3b82f6}p(s')}\mu(s')\rangle \;\;\Rightarrow\;\; \frac{T(s',s,a)}{\sqrt{p(s,a)}\sqrt{p(s')}} = \sqrt{p(s,a)}\sqrt{p(s')}\,\phi(s,a)^\top\mu(s') \]

SVD decomposition

\[ \int\Bigl\|\frac{T(s',s,a)}{\sqrt{p(s,a)}\sqrt{p(s')}} - \sqrt{p(s,a)}\sqrt{p(s')}\,\phi(s,a)^\top\mu(s')\Bigr\|^2 d(s,a)\,ds' \]

\[ \propto\; -2\,\mathbb{E}_{T(s',s,a)}\bigl[\phi(s,a)^\top\mu(s')\bigr] + \mathbb{E}_{p(s,a)p(s')}\bigl[(\phi(s,a)^\top\mu(s'))^2\bigr] \]

a simple, unbiased SGD objective

Ren, Zhang et al., ICLR 2023 (SPEDE)

Linear MDP · Latent Variable

Latent variable models are linear MDPs

\[ T(s'|s,a) = \int p(s'|z)\,p(z|s,a)\,dz = \bigl\langle p(z|s,a),\, p(s'|z)\bigr\rangle_{L_2} \]

\[ \phi(s,a) = p(z|s,a),\qquad \mu(s') = p(s'|z) \]

Evidence lower bound (ELBO)

\[ \begin{aligned} \log T(s'|s,a) &= \log\int p(s'|z)\,p(z|s,a)\,dz \\ &= \max_{q\in\Delta(\mathcal{Z})}\;\mathbb{E}_q\bigl[\log p(s'|z)\bigr] - KL\bigl(q(z|s,a,s')\,\|\,p(z|s,a)\bigr) \end{aligned} \]

Ren, Xiao et al., ICLR 2023 (LV-Rep)

Linear MDP · Diffusion

Diffusion spectral representation

Connect the diffusion model with a spectral decomposition of the MDP:

\[ T(s'|s,a) \propto \exp\bigl(\psi(s,a)^\top\upsilon(s')\bigr) = \bigl\langle \phi_\omega(\psi(s,a)),\, \nu_\omega(\upsilon(s'))\bigr\rangle \]

Score-based representation learning

\[ \min_{\psi,\upsilon}\;\mathbb{E}_\beta\,\mathbb{E}_{s,a,s',\tilde s'}\Bigl[\bigl\|\tilde s' + \beta\,\psi(s,a)^\top\nabla_{\tilde s'}\upsilon(\tilde s',\beta) - \sqrt{1-\beta}\,s'\bigr\|^2\Bigr] \]

Shribak et al., NeurIPS 2024 (Diff-SR)

Linear MDP · Algorithm

The algorithm

Collect data \(s\sim d^{\pi_n},\; a\sim U(\mathcal{A}),\; s'\sim T(\cdot|s,a)\); \(\mathcal{D}_n = \mathcal{D}_{n-1}\cup\{s,a,s'\}\)
Learn representation via NCESVDELBODiffusion
(Optional) UCB bonus
\[ \hat b_n(s,a) = \alpha_n\sqrt{\hat\phi_n(s,a)^\top\hat\Sigma_n^{-1}\hat\phi_n(s,a)},\qquad \hat\Sigma_n = \textstyle\sum_{(s,a)\in\mathcal{D}_n}\hat\phi_n(s,a)\hat\phi_n(s,a)^\top + \lambda_n I \]
Policy evaluation with Bellman recursion
\[ Q^\pi(s,a) = r(s,a) + \hat b_n(s,a) + \gamma\,\mathbb{E}_P\bigl[V^\pi(s')\bigr] \]
Policy optimization with the learned \(Q\); repeat.

Linear MDP · Experiments

Empirical performance — MuJoCo / MBBL

Linear MDP · Experiments

Empirical performance — 27 DeepMind Control tasks

Same tuned TD3 backbone, fixed hyperparameters: Speder · linear / spectral contrastiveLV-Rep · latent variable / ELBODiff-SR · EBM / score matchingCTRL-SR · EBM / ranking perturbed NCE vs. TD3, SAC, TD7. Energy-based reps lead, especially on dog-* and humanoid-*.

Gao, Sun, Li, Schuurmans, Dai, 2025 (arXiv:2512.15036), Fig. 4 · first panel = average over 27 tasks

Linear MDP · Ablation

Ablation — representation dimension

Speder vs CTRL-SR across representation dimensions

FindingBoth improve with larger \(d\), but the energy-based CTRL-SR is better at equal capacity and keeps scaling — it implicitly learns an infinite-dim feature map, truncated by random Fourier features chosen through the critic.

Linear MDP · Ablation

Ablation — noise perturbation & coupling with the critic

Number of noise levels \(M\)

Diff-SR and CTRL-SR with different numbers of noises

Robust to the noise magnitude — but CTRL-SR collapses without noise: perturbation stabilizes contrastive learning.

With / without critic loss on \(\phi\)

Speder, Diff-SR, CTRL-SR with or without critic loss

Finite-dim truncation can drop Q-relevant directions; using the rep objective as a regularizer on top of the critic loss helps uniformly.

Linear MDP · Summary

Summary and gaps

Linearization enables RL with nonlinear models

Efficient exploration.
Efficient planning.

Still not applicable to practical settings

RL from observations, e.g., images / videos / texts.

Linear Representation for POMDPs

Rich observations, no complete state — and how foundation-model encoders fit the picture.

POMDP

Rich observations in the real world

…but no complete state information.

POMDP

Partially observable MDPs

\(\mathcal{P} = \langle\mathcal{S},\mathcal{A},\mathcal{O},r,H,\rho_0,T,O\rangle\)

HardComputation: PSPACE-complete (Papadimitriou & Tsitsiklis, 1987). Statistics: exponential in the horizon (Jin et al., 2020a).

But not allStructured POMDPs admit efficient sample complexity (Jin et al., 2020a; Golowich et al., 2022; Liu et al., 2022, 2023; Efroni et al., 2022; Guo et al., 2023).

POMDP

\(L\)-decodable POMDPs

Definition (\(L\)-decodability, Efroni et al., 2022). Define \[ x_t \in \mathcal{X} := (\mathcal{O}\times\mathcal{A})^{L-1}\times\mathcal{O},\qquad x_t = (o_{t-L+1}, a_{t-L+1},\dots,o_t). \] A POMDP is \(L\)-decodable if there exists a decoder \(p:\mathcal{X}\to\Delta(\mathcal{S})\) such that \(p(x_t) = s_t\).

The \(L\)-step window is a sufficient statistic for the state — e.g. frame stacking in visual RL.

POMDP

Spectral representation of the \(L\)-step dynamics

\(L\)-decodability lets the value live on the window \(x_t\) — an \(L\)-step Bellman equation:

\[ Q^\pi(x_t,a_t) = \mathbb{E}_{x_{t+1:t+L}\sim\mathbb{P}^\pi(\cdot|x_t,a_t)}\Bigl[\textstyle\sum_{i=0}^{L-1}\gamma^i r_{t+i} + \gamma^L V^\pi(x_{t+L})\Bigr] \]

A moment-matching policy \(\tilde\pi\), conditioned only on \((x_t,a_t)\), removes the intermediate steps.

Spectrally decompose the \(L\)-step transition and reward:

\[ \mathbb{P}^{\tilde\pi}_L(x_{t+L}|x_t,a_t) = \bigl\langle \phi(x_t,a_t),\, \mu^{\tilde\pi}(x_{t+L})\bigr\rangle_{\mathcal{H}},\qquad r^{\tilde\pi}_L(x_t,a_t) = \textstyle\sum_{i=0}^{L-1}\gamma^i r_{t+i} = \bigl\langle \phi(x_t,a_t),\, \theta_r^{\tilde\pi}\bigr\rangle_{\mathcal{H}} \]

\[ \boxed{\;Q^\pi(x_t,a_t) = \bigl\langle \phi(x_t,a_t),\; \theta_r^{\tilde\pi} + \gamma^L\textstyle\int \mu^{\tilde\pi}(x_{t+L})\,V^\pi(x_{t+L})\,dx_{t+L}\bigr\rangle_{\mathcal{H}}\;} \]

Beyond LV\(\phi(x_t,a_t)=p(z|x_t,a_t)\) (μLV-Rep) is just one instance.

POMDP

Learning \(\phi(x_t,a_t)\): every formulation carries over

Formulation	Representation	Objective on windows \((x_t,a_t)\to o_{t+1:t+L}\)
Linear	\(\phi_{\text{Linear}}(x_t,a_t)\)	Spectral contrastive: \(-2\,\mathbb{E}\bigl[\phi(x_t,a_t)^\top\nu(o_{t+1:t+L})\bigr] + \mathbb{E}_{p\otimes p}\bigl[(\phi^\top\nu)^2\bigr]\)
Latent variable	\(\phi_{\text{LV}} = p(z\|x_t,a_t)\)	Multi-step ELBO; predicting single frames as a cheaper approximation
Energy-based	\(\phi_{\text{EBM}}\) via random features of \(\exp(\psi^\top\upsilon)\)	Score matching, or NCE with \(\phi(o_{t-L+1:t},a_t)\) vs. \(\nu(o_{t+1:t+L})\) on \(L\) consecutive frames

e.g. the latent-variable instance — multi-step ELBO:

\[ \log p(o_{t+1:t+L}|x_t,a_t) = \max_{q\in\Delta(\mathcal{Z})}\;\mathbb{E}_{q}\bigl[\log\mathbb{P}^\pi(o_{t+1:t+L}|z)\bigr] - D_{KL}\bigl(q(z|x_t,a_t,o_{t+1:t+L})\,\|\,p(z|x_t,a_t)\bigr) \]

The RL algorithm (LSPI / UCB bonus) and its guarantees carry over ⇒ a family of practical and provable algorithms for a subclass of POMDPs.

Gao, Sun, Li, Schuurmans, Dai, 2025 (arXiv:2512.15036), Sec. 6 · Zhang et al., ICML 2024

POMDP · Foundation Models

Foundation models for representation learning

Latent-variable instance: \(p(z_{h+1}|x_h,a_h)\) = Transformer over the encoded history · \(p(o_{h+1:h+3}|z_{h+1})\): MLP over the encoded future.

POMDP · Foundation Models

Connection to DINOv2

Frozen DINO encoder for past & future features; action-conditioned predictor trained on future-feature loss.

POMDP · Experiments

Empirical performance — partially observable control

Video-based RL

From pixels to manipulation.

Video-based RL

Video-based reinforcement learning

Video-based RL · Experiments

Empirical performance — Meta-World success rates

Video-based RL · Experiments

Empirical performance — learning curves

μLV-Rep vs. MWM and DreamerV2 on Meta-World.

Video-based RL · Experiments

Empirical performance — DMC from pixels

normalized score vs. training time for 1M frames

84×84 frames, 3-frame stack; conv encoder → spectral rep.
Model-free Diff-SR / CTRL-SR match or beat model-based TDMPC2 & Dreamer-V3 (best on dog-stand).
Far cheaper than model-based: no planning at decision time.

Gao et al., 2025 (arXiv:2512.15036), Figs. 5–6

Closing

One structure, three axes balanced. References and code.

Take-away

If you remember one thing

A spectral (linear) representation of the dynamics makes Q-functions linear — giving deep-net modeling, provable exploration, and efficient planning at once.

MDP

NCE / SVD / ELBO / diffusion to learn \(\phi(s,a)\).

POMDP

\(L\)-decodable windows; \(Q = \langle \phi(x_t,a_t), w\rangle\) for any formulation.

Real world

Foundation-model encoders, video-based RL.

References

Tongzheng Ren*, Tianjun Zhang*, Csaba Szepesvári, Bo Dai. A Free Lunch from the Noise: Provable and Practical Exploration for Representation Learning. UAI 2022.
Tianjun Zhang*, Tongzheng Ren*, Mengjiao Yang, Joseph Gonzalez, Dale Schuurmans, Bo Dai. Making Linear MDPs Practical via Contrastive Representation Learning. ICML 2022.
Tongzheng Ren*, Tianjun Zhang*, Lisa Lee, Joseph Gonzalez, Dale Schuurmans, Bo Dai. Spectral Decomposition Representation for Reinforcement Learning. ICLR 2023.
Tongzheng Ren, Chenjun Xiao, Tianjun Zhang, Na Li, Zhaoran Wang, Sujay Sanghavi, Dale Schuurmans, Bo Dai. Latent Variable Representation for Reinforcement Learning. ICLR 2023.
Tongzheng Ren, Zhaolin Ren, Na Li, Bo Dai. Stochastic Nonlinear Control via Finite-dimensional Spectral Dynamic Embedding. CDC 2023.
Hongming Zhang, Tongzheng Ren, Chenjun Xiao, Dale Schuurmans, Bo Dai. Provable Representation with Efficient Planning for Partially Observable Reinforcement Learning. ICML 2024.

References

More recent progress

Dmitry Shribak, Chen-Xiao Gao, Yitong Li, Chenjun Xiao, Bo Dai. Diffusion Spectral Representation for Reinforcement Learning. NeurIPS 2024.
Haitong Ma, Zhaolin Ren, Bo Dai, Na Li. Skill Transfer and Discovery for Sim-to-Real Learning: A Representation-Based Viewpoint. IROS 2024.
Zhaolin Ren, Runyu Zhang, Bo Dai, Na Li. Scalable Spectral Representations for Network Multi-Agent Control. AISTATS 2025.
Yang Hu, Tianyi Chen, Na Li, Kai Wang, Bo Dai. Primal-Dual Spectral Representation for Off-policy Evaluation. AISTATS 2025.
Haitong Ma, Bo Dai, Zhaolin Ren, Yebin Wang, Na Li. Offline Imitation Learning upon Sub-optimal Demonstrations by Primal-Dual Representation. IROS 2025.
Chenxiao Gao, Haotian Sun, Na Li, Dale Schuurmans, Bo Dai. Spectral Representation-based Reinforcement Learning. arXiv:2512.15036, 2025.

Thank you. Questions?

Spectral representation for RL spectral-rl.github.io

Spectral representation for control repr-control-orgnaization.readthedocs.io

Sim-to-real congharvard.github.io/steady-sim-to-real

Collaborators: Chenxiao Gao, Tongzheng Ren, Chenjun Xiao, Tianjun Zhang, Haitong Ma, Zhaolin Ren, Lisa Lee, Sherry Yang, Sujay Sanghavi, Na Li, Csaba Szepesvári, Dale Schuurmans, and many more.

Appendix

Appendix A1

Spectral view of classical representations

Representation	Decomposed dynamics
Successor feature	\(\texttt{svd}\bigl((I-\gamma P^\pi)^{-1}\bigr)\)
Proto-value function	\(\texttt{eig}\bigl(\Lambda P^\pi + (P^\pi)^\top\Lambda\bigr)\)
Krylov basis	\(\bigl\{(P^\pi)^i r\bigr\}_{i=1}^{k}\)
Spectral state-aggregation	\(\texttt{svd}(P^\pi)\)