TrackEverything: Long Horizon Dense Tracking via De-Duplicating 3D Scene Representations

The authors introduce TrackEverything, a 3D point tracker representing videos as persistent 3D scene tracks in world coordinates. By employing a voxelization-based de-duplication mechanism, separating endpoint refinement from a lightweight trajectory refiner, and replacing memory-heavy 4D correlation volumes with efficient scene cloud feature sampling, the model breaks the trade-off between sparse long-horizon and dense short-clip tracking. It tracks all visible points across videos exceeding 1000 frames within 40 GB of GPU memory and outperforms open-source all-frame dense 3D trackers by more than 20% APD on short clips.

TrackEverything is the first 3D tracker capable of tracking all visible points across videos exceeding 1000 frames within 40 GB of GPU memory.

On TAPVid-3D, TrackEverything outperforms all open-source all-frame dense 3D trackers by more than 20% APD on short clips.