PIE-NeRF🍕: Physics-based Interactive Elastodynamics with NeRF

CVPR 2024

Yutao Feng1,2, Yintong Shang1, Xuan Li3, Tianjia Shao2, Chenfanfu Jiang3, Yin Yang1
University of Utah1, Zhejiang University2, University of California, Los Angeles3
Paper arXiv Video Code Supp
Utah ZJU UCLA

Pie-Nerf🍕 is an efficient and versatile pipeline that synthesizes physics-based novel motions of complex NeRF models interactively.

Abstract

We show that physics-based simulations can be seamlessly integrated with NeRF to generate high-quality elastodynamics of real-world objects. Unlike existing methods, we discretize nonlinear hyperelasticity in a meshless way, obviating the necessity for intermediate auxiliary shape proxies like a tetrahedral mesh or voxel grid. A quadratic generalized moving least square (Q-GMLS) is employed to capture nonlinear dynamics and large deformation on the implicit model. Such meshless integration enables versatile simulations of complex and codimensional shapes. We adaptively place the least-square kernels according to the NeRF density field to significantly reduce the complexity of the nonlinear simulation. As a result, physically realistic animations can be conveniently synthesized using our method for a wide range of hyperelastic materials at an interactive rate.

Pipeline



The input of PIE-NeRF is the same as other NeRF-based frameworks, which consists of a collection of images of a static scene. An adaptive Poisson disk sampling is followed to query the 3D geometry of the model, which are sparsified into n Q-GMLS kernels. Integrator points are placed over the model, including centers of Q-GMLS kernels (i.e., kernel IPs). Discretization at kernels and numerical integration at IPs enable efficient synthesis of novel and physics-based elastodynamic motions. The quadratic warping scheme helps to better retrieve the color/texture of a deformed spatial position to render the final result.

Interactive showcase


We interact with all types of objects and generat physically realistic dynamic motions.

Swaying excavator

Dragging ficus tree

Chopper riding a truck

Tyrannosaurus dance

Dragging chair

BibTeX

@misc{feng2023pienerf,
      title={PIE-NeRF: Physics-based Interactive Elastodynamics with NeRF}, 
      author={Yutao Feng and Yintong Shang and Xuan Li and Tianjia Shao and Chenfanfu Jiang and Yin Yang},
      year={2023},
      eprint={2311.13099},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}