GECONERF: FEW-SHOT NEURAL RADIANCE FIELDS VIA GEOMETRIC CONSISTENCY

Abstract

We present a novel framework to regularize Neural Radiance Field (NeRF) in a few-shot setting with a geometry-aware consistency regularization. The proposed approach leverages a rendered depth map at unobserved viewpoint to warp sparse input images to the unobserved viewpoint and impose them as pseudo ground truths to facilitate learning of NeRF. By encouraging such geometry-aware consistency at a feature-level instead of using pixel-level reconstruction loss, we regularize the NeRF at semantic and structural levels while allowing for modeling viewdependent radiance to account for color variations across viewpoints. We also propose an effective method to filter out erroneous warped solutions, along with training strategies to stabilize training during optimization. We show that our model achieves competitive results compared to state-of-the-art few-shot NeRF models.

1. INTRODUCTION

Recently, representing a 3D scene as a Neural Radiance Field (NeRF) Mildenhall et al. (2020) has proven to be a powerful approach for novel view synthesis and 3D reconstruction Barron et al. (2021) ; Jain et al. (2021); Chen et al. (2021) . However, despite its impressive performance, NeRF requires a large number of densely, well distributed calibrated images for optimization, which limits its applicability. When limited to sparse observations, NeRF easily overfits to the input view images and is unable to reconstruct correct geometry Zhang et al. (2020) . The task that directly addresses this problem, also called a few-shot NeRF, aims to optimize highfidelity neural radiance field in such sparse scenarios Jain et al. ( 2021 2022), but their necessity for handcrafted methods or inability to extract local and fine structures limited their performance. To alleviate these issues, we propose a novel regularization technique that enforces a geometric consistency across different views with a depth-guided warping and a geometry-aware consistency modeling. Based on these, we propose a novel framework, called Neural Radiance Fields with Geometric Consistency (GeCoNeRF), for training neural radiance fields in a few-shot setting. Our key insight is that we can leverage a depth rendered by NeRF to warp sparse input images to novel viewpoints, and use them as pseudo ground truths to facilitate learning of fine details and highfrequency features by NeRF. By encouraging images rendered at novel views to model warped images with a consistency loss, we can successfully constrain both geometry and appearance to boost fidelity of neural radiance fields even in highly under-constrained few-shot setting. Taking into consideration non-Lambertian nature of given datasets, we propose a feature-level regularization loss that captures contextual and structural information while allowing for modeling view-dependent color differences. We also present a method to generate a consistency mask to prevent inconsistently warped information from harming the network. Finally, we provide coarse-to-fine training strategies for sampling and pose generation to stabilize optimization of the model. We demonstrate the effectiveness of our method on synthetic and real datasets Mildenhall et al. (2020); Jensen et al. (2014) . Experimental results prove the effectiveness of the proposed model over the latest methods for few-shot novel view synthesis.



); Kim et al. (2022); Niemeyer et al. (2022), countering the underconstrained nature of said problem by introducing additional priors. Specifically, previous works attempted to solve this by utilizing a semantic feature Jain et al. (2021), entropy minimization Kim et al. (2022), SfM depth priors Deng et al. (2022) or normalizing flow Niemeyer et al. (

