ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis

Wangbo Yu1*, Jinbo Xing2*, Li Yuan1*, Wenbo Hu3†, Xiaoyu Li3, Zhipeng Huang1, Xiangjun Gao5, Tien-Tsin Wong6, Ying Shan4,3, Yonghong Tian1†,
1 Peking University   2 The Chinese University of Hong Kong   3 Tencent AI Lab
4 ARC Lab, Tencent PCG  5 Hong Kong University of Science and Technology   6 Monash University  
* Equal Contribution † Corresponding Authors
arXiv Video Code 🤗 Demo

Abstract

Despite recent advancements in neural 3D reconstruction, the dependence on dense multi-view captures restricts their broader applicability. In this work, we propose ViewCrafter, a novel method for synthesizing high-fidelity novel views of generic scenes from single or sparse images with the prior of video diffusion model. Our method takes advantage of the powerful generation capabilities of video diffusion model and the coarse 3D clues offered by point-based representation to generate high-quality video frames with precise camera pose control. To further enlarge the generation range of novel views, we tailored an iterative view synthesis strategy together with a camera trajectory planning algorithm to progressively extend the 3D clues and the areas covered by the novel views. With ViewCrafter, we can facilitate various applications, such as immersive experiences with real-time rendering by efficiently optimizing a 3D-GS representation using the reconstructed 3D points and the generated novel views, and scene-level text-to-3D generation for more imaginative content creation. Extensive experiments on diverse datasets demonstrate the strong generalization capability and superior performance of our method in synthesizing high-fidelity and consistent novel views.



Zero-shot Novel View Synthesis Results (Single View)

Left: Camera trajectory; Right: Generated novel view video along the camera trajectory.


Zero-shot Novel View Synthesis Results (2 Views)




3D Reconstruction Results (Single View)





Text-to-3D Generation Results


Visualization of Point Cloud Render Results

The first row displays the point cloud render results, while the second row shows the corresponding novel views generated by ViewCrafter. ViewCrafter can not only fill in occlusions in the point cloud but also handle incorrect geometry.



Method Overview

Given a single reference image or sparse image sets, we first build its point cloud representation using a dense stereo model, which enables accurately moving cameras for free-view rendering. Subsequently, to address the large missing regions, geometric distortions, and point cloud artifacts exhibited in the point cloud render results, we train a point-conditioned video diffusion model to serve as an enhanced renderer, facilitating the generation of high-fidelity and consistent novel views based on the coarse point cloud renders. To achieve long-range novel view synthesis, we adopt an iterative view synthesis strategy that involves iteratively moving cameras, generating novel views, and updating the point cloud, which enables a more complete point cloud reconstruction and benefits downstream tasks such as 3D-GS optimization.


To facilitate more consistent 3D-GS optimization, we leverage the iterative view synthesis strategy to progressively complete the initial point cloud and synthesize novel views using ViewCrafter. We then use the completed dense point cloud to initialize 3D-GS and employ the synthesized novel views to supervise 3D-GS training.

 
   
     

BibTeX

      

  @article{yu2024viewcrafter,
    title={ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis},
    author={Yu, Wangbo and Xing, Jinbo and Yuan, Li and Hu, Wenbo and Li, Xiaoyu and Huang, Zhipeng and Gao, Xiangjun and Wong, Tien-Tsin and Shan, Ying and Tian, Yonghong},
    journal={arXiv preprint arXiv:2409.02048},
    year={2024}
  }
   
Â