NeRF - Neural Radiance Fields

Implicit surfaces rendering in PyTorch3D

3D Graphics Systems - IMPA 2021 - Lab Class #10

Goal

  • Given a set of sparse views of a scene, we would like to learn how to render a new viewpoint of this scene.

Strategy

  • Scene is encoded in a neural network
    • Input: 3D spatial coordinates + 2D viewing direction coordinates
    • Output: 3D color + 1D density
  • "Volumetric version of light field"

View Dependent Effects

Strategy

Two Pass Rendering: Coarse

h:450

Two Pass Rendering: Fine

h:450

Volume Rendering is Differentiable

Positional Encoding

Positional Encoding

  • Learn on a higher dimensional space (from RR to R2LR^{2L})

γ(p)=(sin(20πp),cos(20πp),...,sin(2L−1πp),cos(2L−1πp))\gamma(p) = (sin(2^0\pi p), cos(2^0\pi p), ..., sin(2^{L-1}\pi p), cos(2^{L-1}\pi p))

  • γ(.)\gamma(.) is applied separately to each coordinate

Results

Results

Limitations

  • Speed - hundreds of thousands of iterations for high-res
  • Sampling resolution along rays
  • Generalization - specific for one scene
    • No more? PixelNeRF (CVPR, 2021 - happening NOW!)

Implicit Surfaces in PyTorch3D

Let's check the code on Colab!