DSAD: Dynamic Soft Anisotropic Diagrams for Reduced-Order Video Representation

Dou, Zhiyang; Iinbor, Laki; Matusik, Wojciech

DSAD: Dynamic Soft Anisotropic Diagrams for Reduced-Order Video Representation

One soft anisotropic diagram assigns both appearance and motion: dense sites own the colors, and a coarser set of anchors on the same geometry spans a reduced-order motion basis for each group of pictures.

NeurIPS 2026

Zhiyang Dou¹Laki Iinbor²Wojciech Matusik¹

¹ MIT² Independent Researcher

Paper Code (coming soon) Models & records (coming soon)

Abstract Overview Method Results Speed Analysis BibTeX

Abstract

Explicit video primitives decompose a group of pictures (GoP) into local units that can be rendered, inspected, and edited. This is challenging for implicit neural representations, which typically encode a clip holistically in network weights. Existing GoP-based methods model appearance with explicit primitives such as 2D Gaussians, but their motion network consumes appearance budget and obscures which parameters move which region. Our insight is that in a short GoP, nearby primitives often move together, so the rule that decides which primitive owns a pixel can also decide which primitives share a motion, and a small set of shared motions can efficiently encode a GoP’s deformation. We introduce Dynamic Soft Anisotropic Diagrams (DSAD), which apply one soft-distance geometry at two scales: dense sites of a Soft Anisotropic Diagram (SAD), a top-K-masked softmax over anisotropic Apollonius scores, own local colors, and coarser anchors reuse the same geometry to form a partition-of-unity motion basis, which a reduced-order model (ROM) pairs with knot-based temporal coefficients. The ROM uses only 0.24–0.46 of the parameters read by D2GV’s MLP, which frees budget for appearance. On Bunny, UVG, and DAVIS, DSAD has the highest PSNR among compared GoP primitives, though D2GV-AR has higher MS-SSIM on UVG. Against D2GV with its original MLP at 1280×720, at matched training length and decoder-read parameters, DSAD gains a mean +1.92 dB and wins 223 of 302 GoP–budget pairs, leading on every DAVIS scene while D2GV leads on 3 of 7 UVG scenes and 3 of 32 in a wider DAVIS pool; the margin holds at 1080p. DSAD decodes at 706–1,095 FPS from a cold start on one RTX 2080 Ti.

Overview

One soft-distance geometry at two scales: for appearance ownership and for motion coordinates.

DSAD pipeline: input GoP, SAD appearance with dense sites, anchor ROM with temporal coefficients, deformation field, rendered frame — **DSAD representation.** Given observed GoP frames, dense SAD cells govern local RGB ownership while coarser ROM anchors with knot-based temporal coefficients induce the low-rank deformation field that warps canonical sites to render each frame. Appearance and motion share the same soft-distance geometry. UVG frames © Digiturk, CC BY-NC 3.0.

Method

Three layers over a shared soft-distance geometry, fitted to each group of pictures (GoP).

1. Soft Anisotropic Diagram

A softmax restricted to the K highest-scoring anisotropic-Apollonius sites at each pixel. Dense sites in a canonical coordinate frame encode local RGB ownership.

2. Anchors as a motion basis

A coarser set of anchors reuses the SAD geometry, so that the anchor weights φ_j(x) form a partition-of-unity motion basis, without an additional neural module.

3. Reduced-order model

Deformation factorizes into this spatial basis and knot-based temporal coefficients a_j(t), one curve per anchor: many sites, few independent motion patterns.

Δ(x, t) = ∑_j φ_j(x) a_j(t)applied to each site's canonical position before the tanh that maps it to the image

Soft-distance diagram family: Voronoi, Apollonius, power and anisotropic Apollonius cells of the same anchors, computed from their scores — **Soft-distance diagram family.** Illustrative panels share anchors (dots), with cells minimizing each diagram’s score. Circles and equal-area ellipses show each anchor’s scale and metric. DSAD softens the SAD score with a top-K-masked softmax.

Learned deformation on three UVG GoPs — **Learned deformation on UVG.** Rows show Jockey, ReadySetGo and YachtRide GoP₁ at the UVG-Small budget under the 60k protocol (60k fine iterations). Panels show the first frame (a), dominant ownership with white anchors (b), and first/last-frame displacement (c, d). Magnitudes are normalized per panel. UVG frames © Digiturk, CC BY-NC 3.0.

Results

UVG and DAVIS at 1280×720 under D2GV-AR's budgets, against GoP primitive representations.

Main comparison under the D2GV-AR budgets. Per-video PSNR (dB) and MS-SSIM (higher is better). Italics: transcribed from D2GV-AR; upright: our runs (DSAD decoded from its stored parameters). DSAD and the paired D2GV (original MLP) fit identical GoPs with an L2 loss and 120k fine iterations, at budgets matched on the parameters each decoder reads; D2GV-AR’s training loss includes an SSIM term. The unranked second DSAD row uses the ten-scalar-per-site storage count (10% fewer sites). The last row is the per-GoP paired margin of DSAD over the paired D2GV in dB (GoP–budget pairs won). Ranks are shaded among the GoP primitives, ties sharing a rank: 1st, 2nd, 3rd. The paper’s table also lists full-clip implicit representations (INRs), ranked separately.

Method	Source	UVG-Small		UVG-Large		DAVIS-Small		DAVIS-Large
Method	Source	PSNR ↑	MS-SSIM ↑	PSNR ↑	MS-SSIM ↑	PSNR ↑	MS-SSIM ↑	PSNR ↑	MS-SSIM ↑
GoP primitives: baselines
D-3DGS	from D2GV-AR	26.27	0.8868	28.54	0.9124	24.54	0.8385	26.48	0.8808
D2GV-AR	from D2GV-AR	35.03	0.9663	35.83	0.9728	29.82	0.9457	31.87	0.9668
D2GV (original MLP, paired)	reproduced	34.80	0.9592	35.61	0.9636	29.82	0.9361	32.48	0.9673
GoP primitives: ours
DSAD	ours	35.50	0.9635	36.43	0.9704	33.97	0.9745	35.93	0.9826
DSAD (10 numbers per site)	ours	35.36	0.9624	36.33	0.9698	33.63	0.9726	35.69	0.9821
DSAD − D2GV (paired), ΔPSNR	ours	+0.81 (65/98)		+0.97 (55/98)		+4.18 (53/53)		+3.47 (50/53)

Reconstruction of UVG ReadySetGo GoP 1: ground truth, D2GV, DSAD, HiNeRV — **UVG *ReadySetGo* GoP₁ reconstruction.** Fits from the table above at 1280×720 and the UVG-Small budget (HiNeRV fits the full clip). Top: full frame; bottom: zoomed crop with per-frame PSNR (dB). The GoP was chosen before the comparison, and the crop is the region of largest ground-truth motion. DSAD preserves the gate’s thin structures more sharply than D2GV, gaining +2.42 dB on this GoP against a UVG-Small median margin of +0.58 dB, while full-clip HiNeRV remains sharpest overall. UVG frames © Digiturk, CC BY-NC 3.0.

Ground truth and DSAD reconstructions on five UVG and DAVIS scenes — **UVG and DAVIS reconstructions.** Five scenes (Jockey, YachtRide, Bosphorus, bmx-trees, camel) use GoP₁ at 1280×720 and Small budgets. Ground truth is above DSAD decoded from stored parameters under the 60k protocol at t = 0.2, 0.4, 0.8 and 1. Each reconstruction shows PSNR (dB). UVG frames © Digiturk, CC BY-NC 3.0; DAVIS 2017 frames, CC BY-NC 4.0.

Speed

Decode and encode speed at 1280×720 on one idle RTX 2080 Ti.

Decode and encode speed at 1280×720, measured on GoP₁ of every scene, as the mean over the scenes of each split. Decoding starts from the stored parameters, one frame at a time, and is timed end to end: the DSAD cold start rebuilds site axes and exact top-8 candidates; the steady state reuses cached candidates. The paired D2GV’s first row omits the opacity branch its renderer never reads and builds its frame times on the GPU once per GoP (the frames bit-identical), the row below is its unmodified original implementation; the lightweight D2GV is timed the same way (unmodified: 365–466 FPS). D2GV’s encode times are estimated from timed windows of its fit (the paper’s Appendix J). Decode FPS: higher is better; encode seconds per frame: lower is better.

Method	UVG-Small	UVG-Large	DAVIS-Small	DAVIS-Large
GoP primitives
D2GV, paired control, decode FPS	444	279	501	294
original implementation, decode FPS	336	224	376	238
D2GV, lightweight MLP (60k), decode FPS	587	428	597	452
DSAD, decode FPS, cold start	909	769	1,095	706
DSAD, decode FPS, steady state	4,946	4,196	5,187	4,361
D2GV, paired control, encode s/frame (120k)	161.2	177.0	187.5	161.0
D2GV, lightweight MLP, encode s/frame (60k)	78.5	66.5	89.6	66.4
DSAD, encode s/frame (120k)	16.1	19.3	16.7	20.9

PSNR against decode and encode speed on UVG and DAVIS for DSAD, the paired D2GV and HiNeRV — **Quality and speed at 1280×720.** Small/Large budgets of the main comparison (up and right is better). Lines join cold-start decoding (rebuilding axes and exact candidates) to steady-state decoding (reusing candidates). Lightweight D2GV (428–597 FPS, 66.4–89.6 s/frame at 60k) uses a different schedule and budget and is reported in the table above.

Analysis

A short GoP holds many sites but only a few independent motion patterns.

Cumulative singular-value energy of the motion field and top-1 energy per scene — **UVG-Small motion spectra.** Left shows cumulative singular energy for DSAD (solid) and RAFT (dotted). Right shows top-1 energy per scene, ordered by decreasing paired margin. The dashed line marks one half.

BibTeX

To be updated with the proceedings entry.

@inproceedings{dou2026dsad,
  title     = {{DSAD}: Dynamic Soft Anisotropic Diagrams for Reduced-Order Video Representation},
  author    = {Dou, Zhiyang and Iinbor, Laki and Matusik, Wojciech},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year      = {2026}
}