LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation

Shuai Yang1,4*, Jing Tan2,4*, Mengchen Zhang3,4, Tong Wu2,4✉, Yixuan Li2,4, Gordon Wetzstein5, Ziwei Liu6, Dahua Lin2,4✉

1 Shanghai Jiao Tong University, 2 The Chinese University of Hong Kong, 3 Zhejiang University,
4 Shanghai AI Laboratory, 5 Stanford University, 6 S-Lab, Nanyang Technological University

Paper Youtube Code

LayerPano3D: Layered 3D Panorama
for Hyper-Immersive Scene Generation

Shuai Yang1,4*, Jing Tan2,4*, Mengchen Zhang3,4, Tong Wu2,4✉, Yixuan Li2,4, Gordon Wetzstein5, Ziwei Liu6, Dahua Lin2,4✉

1 Shanghai Jiao Tong University, 2 The Chinese University of Hong Kong, 3 Zhejiang University,
4 Shanghai AI Laboratory, 5 Stanford University, 6 S-Lab, Nanyang Technological University

Paper Youtube Code

Abstract

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D , a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.


Abstract

3D immersive scene generation is a challenging yet critical task in computer vision and graphics. A desired virtual 3D scene should 1) exhibit omnidirectional view consistency, and 2) allow for free exploration in complex scene hierarchies. Existing methods either rely on successive scene expansion via inpainting or employ panorama representation to represent large FOV scene environments. However, the generated scene suffers from semantic drift during expansion and is unable to handle occlusion among scene hierarchies. To tackle these challenges, we introduce LayerPano3D , a novel framework for full-view, explorable panoramic 3D scene generation from a single text prompt. Our key insight is to decompose a reference 2D panorama into multiple layers at different depth levels, where each layer reveals the unseen space from the reference views via diffusion prior. LayerPano3D comprises multiple dedicated designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for high-quality, consistent panorama generation. 2) We pioneer the Layered 3D Panorama as underlying representation to manage complex scene hierarchies and lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes with unconstrained viewing paths. Extensive experiments demonstrate that our framework achieves state-of-the-art performance in both full view consistency and immersive exploratory experience. We believe that LayerPano3D holds promise for advancing 3D panoramic scene creation with numerous applications.


Gallery


Free Rendering 0° -45° 45°

A vibrant city avenue, bustling traffic, towering skyscrapers, cartoon style.

teaser v1_1 v3_11 v1_0 v2_6 v2_8 v2_19 v2_23 v3_1 v3_2 v3_8 v3_9 v3_10 v2_13 v3_12 v4_7

Method Overview

Explanation Overview

Overview of pipeline. Our framework consists of three stages, namely reference panorama generation, multi-layer panorama construction and panoramic 3D scene optimization. LayerPano3D streamlines an automatic generation pipeline without any manual efforts to design scene-specific navigation paths for expansion or completion.

LayerPano3D is robust to render consistent new panorama images at various locations other than the original camera location in the center.

Additional Results in 2D Panorama

❮ ❯

Additional Results in Multi-Layer 3D Representation (Image Guided)

❮ ❯

Citation


		@article{yang2024layerpano3d,
		  title={LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation},
		  author={Yang, Shuai and Tan, Jing and Zhang, Mengchen and Wu, Tong and Li, Yixuan and Wetzstein, Gordon and Liu, Ziwei and Lin, Dahua},
		  journal={arXiv preprint arXiv:2408.13252},
		  year={2024}
		}