Antialiasing Complex Global Illumination Effects in Path-space

Antialiasing Complex Global Illumination Effects
in Path-space

Laurent Belcour

Montréal University

Ling-Qi Yan

UC Berkeley

Ravi Ramamoorthi

UC San Diego

Derek Nowrouzezahrai

Montréal University

Antialiasing Surface Appearance

Image from Mitsuba [Wenzel Jakob]

Antialiasing Surface Appearance

Idea: limit materials frequency to reduce noise
- Adapt texture resolution to screen resolution
- Better cache handling for textures

Antialiasing Surface Appearance

Idea: limit materials frequency to reduce noise
- Adapt texture resolution to screen resolution
- Better cache handling for textures

Method: use the pixel's footprint
- Geometrical method: what is the pixel's projection?
- Use the pixel/ray derivatives [Igehy 1999]

Antialiasing Surface Appearance

Lets assume that we are in this simple configuration where a pinehole camera sees an object lit by a distance point light. *click*

And that the surface's appearance has a very high frequency content. In this case, the texture has very small dots of various color. *click*

To know what area of the object a pixel sees, we need to trace the pixels geometry throught the camera hole onto the geometry. For that we use the derivative of the camera ray with respect to the pixel position (here marked with little arrows) and we project the quadrilateral it defines on the object (here in red). *click*

This quadrilateral defines a surface area P that we can use to integrate the object appearance to obtain its average color in the pixel footprint. Then we can use a single sample to integrate the light contribution and perform shading.

Problem Statement

Differentials work fine for specular interaction
- Extended to rough interaction by [Suykens and Willems 2001]
- But limited ...

Fail short to model subtle effects
- Binary footprints: no pixel filter

Decorrelated light in antialiasing
- Can build differentials from light [Schjøth et al. 2007]

image from [Suykens and Willems 2001]

Now while this sounds like a working model, many flaws appear when we try to use ray differentials into a modern path tracing engine. *click*

The first limitation we usually encounter is that ray differentials are defined for specular transport only. Once a rough material is hit, it requires a different formulation to propagate the geometrical footprint (namely the method of the path differentials). The issue of this later method is that its build a geometry that is no longer a quadrilateral and that the number of facets increases as we perform more bounces.*click*

Also, since the differentials create a geometrical domain, it can only represent the square geometry of the pixel and it will not account for any pixel filter. As you can see on this image, a ray differential footprint will have this quadrilateral shape while the pixel footprint projection will correspond more to the red gaussian falloff.*click*

Another limitation comes from the use of eye path ray differentials. Since only the information from the eye is accounted to separate the integration of spatially varying materials and light, if the lighting contains higher frequency than the material, we will be introducing a non-negligeable bias in rendering. *click*

It is however possible to build ray differentials from the light source and this was used to improve photon mapping. However those differentials do not account for the eye frequency limit.

Problem Statement

Bidirectional path tracing requires symmetric light transport

That leads us to the most important limitation I would like to raise in this talk is the issue of kernel symmetry. Bidirectional path tracing methods require BSDF symmetry to ensure correct rendering. *click*

Lets look at the following example: a spot light illuminating a diffuse textured wall but the camera sees only the rough reflection of this surface. With BDPT we create two paths, one from the camera and one from the eye and average the contribution of all connecting combinaison with multiple importance sampling. Lets assume for a moment that we can build antialiasing footprints for both paths and that the light path has much wider footprint than the eye path at the same position. *click*

On one hand, we can connect the eye path vertex on the wall directly to the spot light. Since we use the BSDF evaluated during the construction of the eye path and that it is the smallest footprint between the two methods it wont introduce too much bias. *click*

On the other hand, we can also connect the eye path vertex on floor to the light path vertex on the wall. This will result in an incorrect filtering of the texture since we will use the footprint of the spotlight to filter the texture. *click*

Here is the resulting image. Clearly overblurred. *click*

In this talk, I will present a connection and antialiasing technique that avoid this issue and symmetrize the antialiasing of surfaces.

Our Solution

Stop thinking in terms of geometry!
- Use frequency analysis to define antialiasing kernels
- Adapt textures frequency to incoming light-field frequency

Make modern path tracing (e.g., BDPT) a first class citizen
- Support multiple non-specular bounce
- Antialiasing kernels using both eye and light paths

Simple implementation
- Extension of ray class, similar to ray differentials

Specifically, I will present the following elements of our method. *click*

First I will present how we change the formalism of antialiasing. With our solution, we got rid of the geometrical formulation of ray and path differentials. Instead, we used frequency analysis to defined antialiasing kernels to adapt the texture frequency limit. *click*

I will also show how to support modern path tracing engines. We enable to specify the antialiasing kernel after multiple boucnes using results from frequency analysis and formulate the combine kernel of eye and light path connection. *click*

Finally, I will give a little details on our implementation and give some results.

Previous Models for Antialiasing

Filtering is defined by surface area: $\color{red}{\mathcal{P}}$

Our Model for Antialiasing

Pixel filter is a kernel applied on the surface!

Frequency Analysis Perspective

This kernel is a low-pass filter on the SV-BRDF:

Fourier domain

What is the Kernel?

$$\Sigma = \begin{pmatrix} \sigma_{xx} & \sigma_{xy} & \sigma_{xu} & \sigma_{xv} \\ \sigma_{yx} & \sigma_{yy} & \sigma_{yu} & \sigma_{yv} \\ \sigma_{ux} & \sigma_{uy} & \sigma_{uu} & \sigma_{uv} \\ \sigma_{vx} & \sigma_{vy} & \sigma_{vu} & \sigma_{vv} \\ \end{pmatrix} $$

$$\Sigma = \begin{pmatrix} \color{red}{\sigma_{xx}} & \sigma_{xy} & \sigma_{xu} & \sigma_{xv} \\ \sigma_{yx} & \color{black}{\sigma_{yy}} & \sigma_{yu} & \sigma_{yv} \\ \sigma_{ux} & \sigma_{uy} & \color{blue}{\sigma_{uu}} & \sigma_{uv} \\ \sigma_{vx} & \sigma_{vy} & \sigma_{vu} & \color{black}{\sigma_{vv}} \\ \end{pmatrix} $$

Use statistical analysis in Fourier space [Durand 2005]
- Second order information is relevant [Belcour 2012]
- Similar to a Gaussian approximation of the kernel
Construction similar to Ray Differentials
- Initialize at light/eye vertex and propagate
- Account for rough interactions and volumes
How to specify it?
- Using the pixel filter function
- Similar to Pre-Filtered IS [Colbert and Krivanek 2009]

Now the question we need to solve is the following: how can we compute this kernel? *click*

Our solution for was rely on Frequency Analysis of light transport and to use covariance tracing. Covariance tracing works directly in Fourier space and store the second order statistics matrix of the light-field Fourier transform. *click*

This matrix contains information of the amount of variation in the spatial domain *click* and in the angular domain. *click*

During my presentation, I will illustrate this matrix with a Gaussian representing a 2D slice in the first spatial and angular dimension. *click* Here the spatial dimension 'x' is displayed horizontally and the angular dimension 'u' is displayed vertically. *click*

This matrix is initiated at the sensor or light vertex and later updated with respect to the path interactions. This very much similar to Ray Differentials but it accounts for rough interactions and participating media out of the box. *click*

To input covariance tracing, we either use the Fourier transform of the pixel filter function or we can use a method similar to pre-filtered importance sampling. The number of sample per pixels will define the input covariance matrix. See our paper for details.

Covariance Antialiasing

This is how it works in the unidrectional case: we first start from the sensor where we can ... *click*

... express the pixel filter as a light field function. That is the kernel is a function of both space and direction. In the case of a pinehole camera this function is a dirac in space at the aperture and has the angular footprint of the pixel in directional. *click*

We can express this function in Fourier space and define its moments. In the case of the pinehole camera the fourier transform is infinite along the spatial axis and its angular bandlimit correspond to the solid angle of the pixel. *click*

Then, we perform covariance tracing. As we build the path, we update the covariance information to account for travel, curvature, BRDF, and so on and we store this covariance at vertices. This is similar to what is needed to propagate a ray differential except that it can account for rough materials and participating media. *click*

At the surface location, we can evaluate the antialiasing kernel on the surface using the inverse Fourier transform of the spatial slice of the Fourier transform. There is a closed form in the case of Gaussian kernels.

We generate antialiasing kernel from the light the same way.

Bidirectional Antialiasing

For that, let's go back to the example of the textured wall and the rough ground plane. Using covariance tracing from the eye, we can build a path and ... *click*

... evaluate its antialiasing kernel at each intersections. Here for the ground plane. *click*

And here for the wall. *click*

Now the issue we are facing is when we want to connect this path to a light path (say here the light vertex). *click*

The light path kernel might not match the eye path kernel. If the light path kernel is spatially much larger in spread than the eye path kernel, everything would be fine and we would not introduce a big bias. *click*

However, as in this case, if the spatial spread of the light kernel is narrow, then we will be overblurring and introducing bias.

Bidirectional Antialiasing

eye kernel

material

light

light kernel

material

light

Bidirectional Antialiasing

$$ \color{blue}{\Sigma_e} \; + \; \color{green}{\Sigma_l} \; = \; \Sigma $$

Bidirectional Antialiasing

Now there is a subtlety in this evaluation because we usually don't know the covariance for both eye and light direction for all vertices. A vertex is depend on its sampling strategy. *click*

Let say for a moment that we want to evaluate the summed covariance for the red vertex here. Since it was generated by light path sampling, it already stores the covariance for the light path. However, the covariance for the eye path is unkown. *click*

What we need to do is to take the eye path vertex at which we are doing the connection and ... *click*

... propagate this covariance matrix to the select vertex. *click*

Then we can sum them once they are in the same coordinate frame.

Implementation details

We need to re-evaluate BSDFs at each connection
- To account for the mean covariance
- In practice we can re-evaluate at connection vertices only
Mitsuba Implementation
- Added a RayCovariance class
- Convertion to RayDifferential before BSDF evaluation
Providing a Small PT [Beason 2010] example
- Available on GitHub

Theoretically, to compute the contribution of a connection in BDPT, we should re-evaluate the BSDF of every vertex to account for the summed covariance of the resulting path. However, this might be too costly. In practice, we only re-evaluate the BSDF using the summed covariance at a predefined vertex of the connection. This still produces a symmetric evaluation of the path and we did not experienced overblurring with it. *click*

To quickly test covariance antialiasing in a renderer already supporting RayDifferentials, it is possible to convert on the fly a covariance matrix to equivalent differentials. We show in our paper how to do so and we used this technique for our Mitsuba implementation. *click*

In our supplemental code, that you can find on github, we provide an implementation of our covariance kernel prediction insde Small Path Tracer from Kevin Beason.

Kernel Validation

Kernel after one bounce

Results

Results: Snails

Unidirectional antialiasing kernels
- Using [Yan 2016] normal map model
- Focusing on indirect footprints with glossy materials

Results: Spoon

Bidirectional antialiasing kernels
- Using [Yan 2016] normal map model
- Focusing on long light paths (caustics)

Ours (~50min)

Reference (24 days)

BDPT (equal-time)

Results: Christmas

Bidirectional antialiasing kernels
- Highly indirect illumination
- Complex lighting (light bulbs)

Ours (~4h)

Reference (141 days)

BDPT (equal-time)

Limitations and Future Work

Limitations
- Only treated spatial antialiasing
- Stationary footprint
Future directions of work
- Incorporate geometry in antialiasing (curvature, ...)
- Antialiasing kernels for participating media
- Other uses of covariance tracing

Finally, I would like to talk a little bit of the limitations and potentialities of our work. *click*

In this work, we only tackled the problem of surface antialiasing without accounting for the angular componnent of the BSDF signal. For example a glint model such as the one of Yan et al. or Jakob et al. still contain high frequency content in the angular domain that needs to be resolved by dense integration. Note that this is not limited by our framework, but more by the model we had at hand.

In this work, we assumed that the footprint was completely covered by the surface to antialias, and the surface planar. This is definitively not the case when kernel get wide. But then a surface modelisation might not be the best solution. *click*

Regarding future works, it would be nice to have antialiasing models that incorporate elements of geometry such as curvature.

We are also interested to look at the extension of our work to antialias volumetric media such as the SGGX model.

We would like to look at other use of covariance tracing as well as it is quite a general framework to solve issues in rendering.

Thank you for your attention


paper	code

available at belcour.github.io/blog