Track4World

Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels

Jiahao Lu¹, Jiayi Xu¹, Wenbo Hu^2†, Ruijie Zhu², Chengfeng Zhao¹,
Sai-Kit Yeung¹, Ying Shan², Yuan Liu^1†
1 HKUST ² ARC Lab, Tencent
ECCV 2026

---

🖼️ Framework

Track4World estimates dense 3D scene flow of every pixel between arbitrary frame pairs from a monocular video in a global feedforward manner, enabling efficient and dense 3D tracking of every pixel in the world-centric coordinate system.

---

⚙️ Setup and Installation

1. Clone the Repository

Clone the repository with submodules to ensure all dependencies are included:

git clone --recursive https://github.com/TencentARC/Track4World.git
cd Track4World

2. Environment Setup

We provide an installation script tested with CUDA 12.1 and Python 3.11.

# Create and activate environment
conda create -n track4world python=3.11
conda activate track4world

# Install PyTorch
pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121

# Install dependencies
pip install -r requirements.txt

3. Install Third-Party Modules

We utilize several external repositories. Please run the following commands to set them up correctly:

# Install utils3d
git clone https://github.com/jiah-cloud/utils3d.git 

# Setup Pi3 (Sparse checkout)
git clone --no-checkout https://github.com/yyfz/Pi3.git track4world/nets/external/pi3_repo
cd track4world/nets/external/pi3_repo
git sparse-checkout init
git sparse-checkout set pi3
git checkout main
find . -maxdepth 1 -type f -exec rm -f {} \;
mv pi3 ../pi3
cd ../../../..

# Setup Grounded-SAM-2
git clone https://github.com/IDEA-Research/Grounded-SAM-2.git submodules
cd submodules
pip install -e .
pip install --no-build-isolation -e grounding_dino
cd ..

4. Download Weights

Download the pre-trained model weights and place them in the checkpoints/ directory.

mkdir -p checkpoints

# Download SAM2 weights
wget https://dl.fbaipublicfiles.com/segment_anything_2/092824/sam2.1_hiera_large.pt -O ./checkpoints/sam2.1_hiera_large.pt

# Download Track4World weights
wget https://huggingface.co/TencentARC/Track4World/resolve/main/track4world_da3.pth -O ./checkpoints/track4world_da3.pth
wget https://huggingface.co/TencentARC/Track4World/resolve/main/track4world_pi3.pth -O ./checkpoints/track4world_pi3.pth
wget https://huggingface.co/TencentARC/Track4World/resolve/main/track4world_moge.pth -O ./checkpoints/track4world_moge.pth

• Manual Download: HuggingFace Link

---

🚀 Demo

Run the following commands to perform tracking and reconstruction on the provided demo video (demo_data/cat.mp4).

1. First Frame 3D Tracking (3d_ff)

Reconstructs 3D motion based on the geometry of the first frame.

python demo.py \
    --mp4_path demo_data/cat.mp4 \
    --mode 3d_ff \
    --Ts -1 \
    --save_base_dir results/cat

2. Dense Tracking: Every Pixel, Every Frame (3d_efep)

Performs dense 3D tracking for every pixel across all frames.

Option A: Camera-Centric Coordinate System

python demo.py \
    --mp4_path demo_data/cat.mp4 \
    --coordinate world_depthanythingv3 \
    --mode 3d_efep \
    --Ts -1 \
    --ckpt_init checkpoints/track4world_da3.pth \
    --save_base_dir results/cat

Option B: World-Centric Coordinate System

For world-centric reconstruction, you can also directly run Step 2 to obtain world-centric 3D tracking results. However, for better visualization, especially to clearly separate foreground and background objects,it is recommended to first segment dynamic objects using DINO and SAM2 in Step 1. You can use either world_depthanythingv3 or world_pi3 for world coordinate system.

# 1. DINO + SAM2 Segmentation
# Use --text-prompt to specify the dynamic objects in your video (e.g., "cat.", "person.", "car.").
python scripts/run_dino_sam2.py \
    --video-path demo_data/cat.mp4 \
    --sam2-checkpoint checkpoints/sam2.1_hiera_large.pt \
    --output-dir results/cat \
    --text-prompt "cat."
    
# 2. Run Track4World 3D EFEP
python demo.py \
    --mp4_path demo_data/cat.mp4 \
    --coordinate world_depthanythingv3 \
    --mode 3d_efep \
    --Ts -1 \
    --ckpt_init checkpoints/track4world_da3.pth \
    --save_base_dir results/cat

3. 2D Tracking (2d)

Performs standard 2D tracking in image space.

python demo.py \
    --mp4_path demo_data/cat.mp4 \
    --mode 2d \
    --Ts -1 \
    --save_base_dir results/cat

4. Metric Scale Output

When using the DA3 backbone, you can enable --metric_scale to output all geometric results (points, flow3d, world_points, camera_poses) in metric (meter-level) coordinates:

python demo.py \
    --mp4_path demo_data/cat.mp4 \
    --coordinate world_depthanythingv3 \
    --mode 3d_efep \
    --Ts -1 \
    --ckpt_init checkpoints/track4world_da3.pth \
    --metric_scale \
    --save_base_dir results/cat

Note: Metric scale recovery is currently only supported with the DA3 backbone (--use_model depthanythingv3). MoGe and Pi3 backbones output in relative scale.

---

✨ Visualization

Visualize the dense 4D trajectories and reconstructed scenes using the generated output files.

Visualize First Frame 3D Tracking:

python visualization/vis_3d_ff.py --ply_dir results/cat/3d_ff_output

Visualize Dense Tracking (Every Pixel):

# Camera Centric Visualization
python visualization/vis_3d_efep.py --ply_dir results/cat/3d_efep_output

# World Centric Visualization (Foreground-Background Separation, Static Background)
python visualization/vis_3d_efep_world.py --ply_dir results/cat/3d_efep_output

---

📊 Evaluation

For detailed instructions on how to evaluate the model on standard benchmarks (Sintel, KITTI, Kubric, etc.), please refer to the evaluation guide:

👉 Evaluation Guide (evaluation/eval.md)

📈 WorldTrack: comparison with OpenD4RT

Track4World and OpenD4RT are scored on the four WorldTrack subsets (50 clips each) under both projects’ evaluation protocols. In each protocol only the predictor is swapped — the data loader, the alignment and the metric code are the host protocol’s own. Track4World is track4world_da3.pth with --coordinate world_depthanythingv3; OpenD4RT is its released OpenD4RT_32CLIP_9Dataset_NoAUG.

Both models are fed the same image information: every clip is downsampled to the 256x256 OpenD4RT consumes and then upsampled back to Track4World’s own canvas. The bottleneck is bit-identical to OpenD4RT’s input (both go through Open-d4rt’s _resize_video, i.e. cv2.INTER_AREA), so neither model sees more of the image than the other.

How to reproduce

OpenD4RT lives beside this repo as a git submodule, so both directions of the swap run against its unmodified code:

git submodule update --init third_party/Open-d4rt
# then follow third_party/Open-d4rt/README.md to fetch OpenD4RT_32CLIP_9Dataset_NoAUG
# into third_party/Open-d4rt/checkpoints/

Two adapters under evaluation/opend4rt_comparison/ each keep the host protocol’s data loader, alignment and metrics and only replace the predictor. --bottleneck-hw / --bottleneck_hw route each clip through 256x256 and back up, matching the image information available to OpenD4RT:

# Track4World under Open-d4rt's protocol
python evaluation/opend4rt_comparison/eval_track4world_in_worldtrack.py \
    --data-root evaluation/track --num-frames 64 --bottleneck-hw 256,256

# OpenD4RT under Track4World's protocol
python evaluation/opend4rt_comparison/eval_opend4rt_in_t4w.py \
    --dataset adt --num_frames 16

Results (image information matched)

Track4World’s numbers are its info-matched score; OpenD4RT’s are unchanged, since it already runs at 256x256. Each protocol reports its own metrics; best per cell in bold.

Under Open-d4rt’s protocol (frame-0 visible queries, global median-scale alignment). APD shown as a percentage, higher APD is better, lower EPE is better:

Model	PStudio		PO		DR		ADT		Mean
	APD ↑	EPE ↓	APD ↑	EPE ↓	APD ↑	EPE ↓	APD ↑	EPE ↓	APD ↑	EPE ↓
Track4World · 16f	78.51	0.1968	73.32	0.2589	69.96	0.3091	86.76	0.1478	77.14	0.2282
OpenD4RT · 16f	81.42	0.1663	68.80	0.3330	72.11	0.3023	75.34	0.2421	74.42	0.2609
Track4World · 50f	73.95	0.2200	69.19	0.3041	70.32	0.3062	82.14	0.1862	73.90	0.2541
OpenD4RT · 50f	80.53	0.1693	67.93	0.3216	72.84	0.2941	71.97	0.2712	73.32	0.2640
Track4World · 64f	72.86	0.2274	66.67	0.3323	69.91	0.3088	79.71	0.2212	72.29	0.2724
OpenD4RT · 64f	78.63	0.1812	66.04	0.3397	72.65	0.2944	69.91	0.2966	71.81	0.2780

Under Track4World’s protocol (isotropic scale + 3D shift alignment, TAPVid-3D thresholds). APD, AJ and OA shown as percentages, higher is better:

Model	PStudio			PO			DR			ADT			Mean
	APD ↑	AJ ↑	OA ↑	APD ↑	AJ ↑	OA ↑	APD ↑	AJ ↑	OA ↑	APD ↑	AJ ↑	OA ↑	APD ↑	AJ ↑	OA ↑
Track4World · 16f	59.14	51.86	93.67	53.86	39.65	80.04	49.75	43.89	97.21	62.83	58.17	98.78	56.39	48.39	92.42
OpenD4RT · 16f	59.93	52.33	92.91	51.43	37.32	81.64	50.73	44.47	96.20	50.51	45.33	97.24	53.15	44.86	92.00
Track4World · 50f	54.56	44.42	87.91	51.31	35.55	76.81	50.95	44.07	94.64	56.69	51.94	98.12	53.38	44.00	89.37
OpenD4RT · 50f	58.53	49.98	91.02	50.26	34.46	79.69	52.34	44.53	93.32	50.89	43.90	93.50	53.00	43.22	89.38
Track4World · 64f	53.65	42.44	86.05	50.45	34.66	76.49	51.01	43.91	94.03	54.40	49.65	97.92	52.38	42.66	88.62
OpenD4RT · 64f	56.66	47.39	89.61	48.87	33.00	79.33	52.34	44.18	92.60	48.64	41.46	91.85	51.63	41.51	88.35

---

📝 Citation

If you find Track4World useful for your research or applications, please consider citing our paper:

@article{lu2026track4world,
  title={Track4World: Feedforward World-centric Dense 3D Tracking of All Pixels},
  author={Lu, Jiahao and Xu, Jiayi and Hu, Wenbo and Zhu, Ruijie and Zhao, Chengfeng and Yeung, Sai-Kit and Shan, Ying and Liu, Yuan},
  journal={arXiv preprint arXiv:2603.02573},
  year={2026}
}

---

🤝 Acknowledgements

Our codebase is built upon MoGe, Alltracker, Pi3, and Depth Anything 3. We also gratefully acknowledge TrackingWorld and VGGT for their excellent work!