MirrorVerse: Advancing Diffusion Models to Master Realistic Reflections

Challenges in Modeling Physical Phenomena in AI

Generative AI has sparked a surge in research focused on teaching models to understand and replicate physical laws, such as gravity and liquid dynamics. Latent diffusion models (LDMs), which have been dominant since 2022, struggle with accurately representing physical phenomena, especially reflections.

The Reflection Problem in Diffusion Models

Reflections pose a unique challenge for LDMs compared to other physics-based tasks like gait simulation or particle physics. While ray-tracing in CGI attempts to simulate light interactions for realistic reflections, it is computationally expensive and limited by the number of light bounces. Diffusion models lack explicit structural rules, relying heavily on training data quality and diversity to learn reflection behavior.

Previous Approaches to Reflection Modeling

Methods like Neural Radiance Fields (NeRF) and Gaussian Splatting have tried to model reflections with varying success, using complex scene modeling and ray calculations. However, diffusion models, due to their semantic nature, find it harder to embed reflection logic reliably.

Introducing MirrorVerse and MirrorFusion 2.0

The MirrorVerse project addresses reflection limitations by providing an improved dataset and training method. MirrorFusion 2.0, the diffusion-based generative model developed by researchers from IISc Bangalore and Samsung R&D Institute, utilizes the newly curated MirrorGen2 dataset. This dataset introduces randomized object positioning, rotations, grounding, and semantically consistent object pairings to simulate realistic reflective interactions.

Dataset Generation and Training

MirrorGen2 dataset construction involves placing 3D objects from Objaverse and Amazon Berkeley Objects onto textured floors with HDRI backgrounds and mirrors, ensuring visibility and realistic lighting. Multiple objects are paired semantically to simulate occlusions and spatial complexity.

Training MirrorFusion 2.0 follows a three-stage curriculum: starting with single-object synthetic scenes, progressing to multi-object scenes to handle occlusions, and finally fine-tuning on real-world data (MSD dataset) with depth maps. This staged approach improves model generalization and reflection accuracy.

Evaluation and Results

MirrorFusion 2.0 outperforms previous models in quantitative metrics (PSNR, SSIM, LPIPS) and qualitative tests on datasets like MirrorBenchV2, GSO, and MSD. It preserves spatial integrity, accurate geometry, and realistic reflections even on out-of-distribution and complex real-world scenes. A user study reported 84% preference for MirrorFusion 2.0 outputs over baselines.

Limitations and Future Directions

Despite improvements, the reflection problem remains challenging due to the diffusion architecture's limitations and dependency on training data. Enhancing dataset diversity and annotation related to reflections could improve future models. However, balancing efforts among various physical simulation challenges in LDMs remains a complex decision.

The MirrorVerse project and MirrorFusion 2.0 represent significant steps forward in enabling diffusion models to realistically reflect the world, though further work is needed to approach the fidelity of more structured methods like NeRF and Gaussian Splatting.