Awesome-Physics-Cognition-based-Video-Generation

Awesome-Physics-Cognition-based-Video-Generation

🚀🚀🚀 This is a repository for organizing papers, codes, and other resources related to physics cognition-based video generation. This repo is being actively updated, please stay tuned! For more detailed information, please refer to our survey paper: Exploring the Evolution of Physics Cognition in Video Generation: A Survey.

This paper list covers T2V, V2V, and dynamic 3D and 4D generation based on physics, and includes some world models and world simulators.

If you find this repository useful, please consider giving us a star 🌟 and a cite.

⚡ Contributing

We welcome feedback, suggestions, and contributions that can help improve this survey and repository and make them valuable resources for the entire community. We will actively maintain this repository by incorporating new research as it emerges. If you have any suggestions about our taxonomy, please take a look at any missed papers, or update any preprint arXiv paper that has been accepted to some venue.

If you want to add your work or model to this list, please do not hesitate to pull requests.

Markdown format:

 * Paper Name. [[Paper]](link) [[Code]](link) [[Website]](link) [**Name of Conference or Journal + Year**]

📖 Table of Contents

• ⚡ Contributing
• 📖 Table of Contents
  • Workshops
  • Surveys
  • Basic Schematic Perception for Generation
  • Passive Cognition of Physical Knowledge for Generation
  • Active Cognition for World Simulation
  • Datasets, Benchmarks and Metrics
  • Physical Understanding
    • VLMs, LLMs, MLLMs on Physical Understanding
    • Benchmarks
• ♥️ Star History
• 📑 Citation

Workshops

Title	Website	Date
Building Physically Plausible World Models		ICML 2025 Workshop, Vancouver Saturday, July 19 (Whole-Day Workshop)
1st Workshop on Vision Meets Physics: Synergizing Physical Simulation and Computer Vision		CVPR 2025 Workshop, June 12 (full day), 2025

Surveys

Title	arXiv	Github	WebSite	Pub. & Date
From Specialist to Generalist: A Comprehensive Survey on World Models	-	-	-	Jan., 2026
Video Generation Models in Robotics - Applications, Research Challenges, Future Directions		-	-	Jan,. 2026
Simulating the Visual World with Artificial Intelligence: A Roadmap				Nov., 2025
Aligning Perception, Reasoning, Modeling and Interaction: A Survey on Physical AI			-	Oct., 2025
A Survey of Interactive Generative Video		-	-	Apr., 2025
Digital Gene: Learning about the Physical World through Analytic Concepts		-	-	Apr., 2025
Simulating the Real World: A Unified Survey of Multimodal Generative Models			-	Mar., 2025
Grounding Creativity in Physics: A Brief Survey of Physical Priors in AIGC		-	-	Feb., 2025
Generative Physical AI in Vision: A Survey			-	Jan., 2025
Physics-Informed Computer Vision: A Review and Perspectives		-	-	ACM Computing Surveys, 2024

Basic Schematic Perception for Generation

Title	arXiv	Github	WebSite	Pub. & Date
Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models		-		Nov., 2025
Geometry-aware 4D Video Generation for Robot Manipulation				Jul., 2025
CoCo4D: Comprehensive and Complex 4D Scene Generation				Jun., 2025
Generative Blocks World: Moving Things Around in Pictures		-	-	Jun., 2025
UniRelight: Learning Joint Decomposition and Synthesis for Video Relighting		-		Jun., 2025
IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation				Jun., 2025
MotionPro: A Precise Motion Controller for Image-to-Video Generation				May, 2025, CVPR
ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D Physics Modeling for Complex Motion and Interaction		-		Apr., 2025
Any2Caption:Interpreting Any Condition to Caption for Controllable Video Generation				Mar., 2025
Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach		-		Feb, 2025
SG-I2V: Self-Guided Trajectory Control in Image-to-Video Generation				ICLR, 2025
TrackGo: A Flexible and Efficient Method for Controllable Video Generation		-	-	AAAI, 2025
3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation				ICLR, 2025
Generative Photography: Scene-Consistent Camera Control for Realistic Text-to-Image Synthesis				CVPR, 2025
Lux Post Facto: Learning Portrait Performance Relighting with Conditional Video Diffusion and a Hybrid Dataset				CVPR,2025
Identity-Preserving Text-to-Video Generation by Frequency Decomposition				CVPR, 2025
Motion Modes: What Could Happen Next?		-		CVPR, 2025
Motion Prompting: Controlling Video Generation with Motion Trajectories		-		CVPR, 2025 (Oral)
Spectral Motion Alignment for Video Motion Transfer using Diffusion Models				AAAI, 2025
Video Creation by Demonstration		-		Dec., 2024
InterDyn: Controllable Interactive Dynamics with Video Diffusion Models		-		Dec., 2024,CVPR 25
LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis				Dec., 2024
GenLit: Reformulating Single-Image Relighting as Video Generation		-		Dec., 2024
Motion Dreamer: Realizing Physically Coherent Video Generation through Scene-Aware Motion Reasoning				Nov., 2024.
AnimateAnything: Consistent and Controllable Animation for Video Generation				Nov., 2024
InTraGen: Trajectory-controlled Video Generation for Object Interactions			-	Nov., 2024
DreamVideo-2: Zero-Shot Subject-Driven Video Customization with Precise Motion Control		-		Oct., 2024
LumiSculpt: A Consistency Lighting Control Network for Video Generation		-	-	Oct., 2024
Tora: Trajectory-oriented Diffusion Transformer for Video Generation				Jul., 2024
UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation				Jun., 2024
Image Conductor: Precision Control for Interactive Video Synthesis				Jun., 2024
Motion Inversion for Video Customization				Mar., 2024
VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models				CVPR, 2024
MotionDirector: Motion Customization of Text-to-Video Diffusion Models				ECCV, 2024, Oral
FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video Synthesis		-		CVPR, 2024, Highlight
Generative Image Dynamics				CVPR, 2024, Best Paper Award
MotionCtrl: A Unified and Flexible Motion Controller for Video Generation				SIGGRAPH, 2024
MOFA-Video: Controllable Image Animation via Generative Motion Field Adaptions in Frozen Image-to-Video Diffusion Model				ECCV, 2024
Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling				SIGGRAPH, 2024
DragAnything: Motion Control for Anything using Entity Representation				ECCV, 2024
FLATTEN: optical FLow-guided ATTENtion for consistent text-to-video editing				ICLR, 2024
Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation				CVPR, 2024
Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion				SIGGRAPH, 2024
Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling		-		SIGGRAPH, 2024
Compositional 3D-aware Video Generation with LLM Director		-		NIPS, 2024
TC4D: Trajectory-Conditioned Text-to-4D Generation				ECCV, 2024
DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory		-		Aug., 2023
Control-A-Video: Controllable Text-to-Video Diffusion Models with Motion Prior and Reward Feedback Learnin				May., 2023
VideoComposer: Compositional Video Synthesis with Motion Controllability				NeurIPS, 2023
Adding Conditional Control to Text-to-Image Diffusion Models			-	ICCV, 2023, Best Paper Award

Passive Cognition of Physical Knowledge for Generation

Title	arXiv	Github	WebSite	Pub. & Date
OptiWorld: Optimal Control for Video World Generation under Physical Constraints		-		Jun., 2026
NEWTON: Agentic Planning for Physically Grounded Video Generation		-		May, 2026
TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction			May, 2026
PhysVid: Physics Aware Local Conditioning for Generative Video Models		-	-	CVPR 2026
MotionPhysics: Learnable Motion Distillation for Text-Guided Simulation				AAAI, 2026
Phys4D: Fine-Grained Physics-Consistent 4D Modeling from Video Diffusion		-		Mar., 2026
PhysAlign: Physics-Coherent Image-to-Video Generation through Feature and 3D Representation Alignment		-		Mar., 2026
Physical Simulator In-the-Loop Video Generation				CVPR, 2026
PhysVideo: Physically Plausible Video Generation with Cross-View Geometry Guidance		-		Mar., 2026
Chain of Event-Centric Causal Thought for Physically Plausible Video Generation		-	-	CVPR, 2026
RealWonder: Real-Time Physical Action-Conditioned Video Generation				Mar., 2026
Goal Force: Teaching Video Models To Accomplish Physics-Conditioned Goals				CVPR, 2026
PI-Light: Physics-Inspired Diffusion for Full-Image Relighting			-	ICLR 2026
Self-Refining Video Sampling				Jan., 2026
FoundationMotion: Auto-Labeling and Reasoning about Spatial Movement in Videos				Dec., 2025
VDAWorld: World Modelling via VLM-Directed Abstraction and Simulation		-		Dec., 2025
Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement		-	-	ICCV 2025 Physics-IQ Challenge Third Place
ProPhy: Progressive Physical Alignment for Dynamic World Simulation		-		Dec., 2025
Planning with Sketch-Guided Verification for Physics-Aware Video Generation				Nov., 2025
Bootstrapping Physics-Grounded Video Generation through VLM-Guided Iterative Self-Refinement		-	-	Nov., ICCV 2025 Physics-IQ Challenge Third Place Solution
CP4D: Compositional physics-aware 4D scene generation	-	-	-	Oct., 2025
3DPhysVideo: 3D Scene Reconstruction and Physical Animation Leveraging a Video Generation Model via Consistency-Guided Flow SDE	-	-	-	Oct., 2025
PhyMAGIC: Physical Motion-Aware Generative Inference with Confidence-guided LLM	-	-	-	Oct., 2025
DiffuPhyGS: Text-to-Video Generation with 3D Gaussians and Learnable Physical Properties via Diffusion Priors	-	-	-	Oct., 2025
Fracture-GS: Dynamic Fracture Simulation with Physics-Integrated Gaussian Splatting	-	-	-	Oct., 2025
PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Generation	-	-	-	Oct., 2025
Learning to Generate Object Interactions with Physics-Guided Video Diffusion		-		Oct., 2025
CONTROLHAIR: Physically-based Video Diffusion for Controllable Dynamic Hair Rendering		-		Sep., 2025
NewtonGen: Physics-Consistent and Controllable Text-to-Video Generation via Neural Newtonian Dynamics			-	Sep., 2025
PIRF: Physics-Informed Reward Fine-Tuning for Diffusion Models			-	Sep., 2025
PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation	-	-		NeurIPS, 2025
PhysGM: Large Physical Gaussian Model for Feed-Forward 4D Synthesis		-		Aug., 2025
Physics-Grounded Motion Forecasting via Equation Discovery for Trajectory-Guided Image-to-Video Generation		-	-	Jul., 2025
Dreamland: Controllable World Creation with Simulator and Generative Models		-		Jun., 2025
Vid2Sim: Generalizable, Video-based Reconstruction of Appearance, Geometry and Physics for Mesh-free Simulation		-		Jun., 2025, CVPR
Physics-Guided Motion Loss for Video Generation Model		-	-	Jun., 2025
Think Before You Diffuse: LLMs-Guided Physics-Aware Video Generation		-		May, 2025
Force Prompting: Video Generation Models Can Learn and Generalize Physics-based Control Signals				May, 2025
WonderPlay: Dynamic 3D Scene Generation from a Single Image and Actions		-		May, 2025
MAGIC: Motion-Aware Generative Inference via Confidence-Guided LLM		-	-	May, 2025
UniPhy: Learning a Unified Constitutive Model for Inverse Physics Simulation				May, 2025, CVPR
FinePhys: Fine-grained Human Action Generation by Explicitly Incorporating Physical Laws for Effective Skeletal Guidance				May, 2025, CVPR
Generating Physically Stable and Buildable LEGO Designs from Text				May, 2025
ReVision: High-Quality, Low-Cost Video Generation with Explicit 3D Physics Modeling for Complex Motion and Interaction		-		Apr., 2025
Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learning		-	-	Apr., 2025
VLIPP: Towards Physically Plausible Video Generation with Vision and Language Informed Physical Prior				Mar., 2025
Articulated Kinematics Distillation from Video Diffusion Models		-		Apr., 2025; CVPR, 2025
RainyGS: Efficient Rain Synthesis with Physically-Based Gaussian Splatting		-		Mar., 2025; CVPR, 2025
PhysGen3D: Crafting a Miniature Interactive World from a Single Image				Mar., 2025; CVPR, 2025
AccidentSim: Generating Physically Realistic Vehicle Collision Videos from Real-World Accident Reports		-		Mar., 2025
Synthetic Video Enhances Physical Fidelity in Video Synthesis		-		Mar., 2025
PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos				Mar., 2025
C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation			-	Feb., 2025
PhysAnimator: Physics-Guided Generative Cartoon Animation		-	-	Jan., 2025
OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation		-	-	ICLR, 2025
AutoVFX: Physically Realistic Video Editing from Natural Language Instructions				3DV, 2025
Gaussian Splashing: Unified Particles for Versatile Motion Synthesis and Rendering		-		CVPR, 2025
Unleashing the potential of multi-modal foundation models and video diffusion for 4d dynamic physical scene simulation		-		CVPR, 2025
FluidNexus: 3D Fluid Reconstruction and Prediction from a Single Video		-		CVPR, 2025
OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation				ICLR,2025
DreamPhysics: Learning Physical Properties of Dynamic 3D Gaussians with Video Diffusion Priors			-	AAAI, 2025
GauSim: Registering Elastic Objects into Digital World by Gaussian Simulator		-		Dec., 2024
GaussianProperty: Integrating Physical Properties to 3D Gaussians with LMMs				Dec., 2024
Phys4DGen: A Physics-Driven Framework for Controllable and Efficient 4D Content Generation from a Single Image		-		Nov., 2024
Teaching Video Diffusion Model with Latent Physical Phenomenon Knowledge				Nov., 2024
Llmphy: Complex physical reasoning using large language models and world models		-	-	Nov., 2024
Automated 3D Physical Simulation of Open-world Scene with Gaussian Splatting		-		Nov., 2024
Enhancing Sketch Animation: Text-to-Video Diffusion Models with Temporal Consistency and Rigidity Constraints		-	-	Nov., 2024
PhysMotion: Physics-Grounded Dynamics From a Single Image		-		Nov., 2024
Trans4D: Realistic Geometry-Aware Transition for Compositional Text-to-4D Synthesis			-	Oct., 2024
Phy124: Fast Physics-Driven 4D Content Generation from a Single Image		-	-	Sep., 2024
Kubrick: Multimodal Agent Collaborations for Synthetic Video Generation		-	-	Aug., 2024
Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion				Jun., 2024
Sync4D: Video Guided Controllable Dynamics for Physics-Based 4D Generation		-		May., 2024
ElastoGen: 4D Generative Elastodynamics		-		May, 2024
MotionCraft: Physics-based Zero-Shot Video Generation				Nips, 2024
PhysGen: Rigid-Body Physics-Grounded Image-to-Video Generation				ECCV, 2024
PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation				ECCV, 2024 Oral
Video2Game: Real-time, Interactive, Realistic and Browser-Compatible Environment from a Single Video				CVPR, 2024
PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF				CVPR, 2024
VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Reality		-		SIGGRAPH, 2024
PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics				CVPR, 2024
Feature Splatting: Language-Driven Physics-Based Scene Synthesis and Editing				ECCV, 2024
Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics				NIPS, 2024
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis				3DV, 2024
LLM-grounded Video Diffusion Models				ICLR, 2024
Compositional 3D-aware Video Generation with LLM Director		-		NIPS, 2024
GPT4Motion: Scripting Physical Motions in Text-to-Video Generation via Blender-Oriented GPT Planning				CVPR, 2024, workshop
DeformGS: Scene Flow in Highly Deformable Scenes for Deformable Object Manipulation				WAFR 2024
Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics				ICML, 2023
Pac-nerf: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification				ICLR, 2023, Spotlight

Active

README 内容过长，已截取前 64 KB。完整内容请前往 GitHub 仓库查看。