Real-time Rendering of High-quality Effects using Multi-frame Sampling Daniel Limberger

Jürgen Döllner

Hasso Plattner Institute, University of Potsdam, Germany

1

Abstract

3

4

Excerpt of Multi-frame Approaches

In a rendering environment of comparatively sparse interaction, e.g., digital production tools, image synthesis and its quality do not have to be constrained to single frames. This work analyzes strategies for highly economically rendering of state-of-the-art rendering effects using progressive multi-frame sampling in real-time. By distributing and accumulating samples of sampling-based rendering techniques (e.g., anti-aliasing, orderindependent transparency, depth-of-field and shadowing, ambient occlusion, screen-space reflections) over multiple frames, images of very high quality can be synthesized with unequaled resource-efficiency.

Our multi-frame sampling approach distributes samples over a well-defined number nMF of consecutive frames. With each frame we progressively increase image quality using a unique sample or set of samples specified by a precomputed sampling kernel. Consecutive frames are accumulated until nMF frames are computed and the rendering eventually pauses. On any update request, the accumulation process is restarted. Since all samples are precomputed and designed for a specific multi-frame number nMF, accumulating additional frames on top of that is futile. Especially when passing low multi-frame numbers, this may lead to temporal clustering.

2

Multi-frame Anti-Aliasing

Introduction

Real-time rendering usually provides a continuous stream of individual high-quality images. However, this strict continuity is only required for frequent input changes (e.g., virtual camera movement, dynamic objects). Even though most non-gaming applications are faced by less frequent data changes (e.g., previews in CAD, DCC, and data visualization tools), they prefer to apply rendering techniques designed for single-frame execution. These techniques usually increase the computational complexity and resources requirements in order to produce high-quality frames. With multi-frame sampling, instead of rendering a single frame in response to changed inputs, multiple frames are rendered and accumulated in order to reduce the computational complexity of a single frame. The accumulation result is immediately displayed while its quality progressively increases: Shadow Mapping, 4 lights

Shading

DoF separated

SF SSAO, Composition 24 samples, separated blur

Geometry Rendering, 2 draw calls, 4x SSAA t

time

1st Accumulation

...

By favoring interactivity in terms of low response times over per-frame quality, multi-frame sampling facilitates scalable resource demands and an increases overall image quality. In our implementations we achieve massive resource reduction and frame rate increases of within an order of magnitude. Furthermore, the presented techniques are well suited for implementation in web and mobile clients.

Multi-frame Soft Shadows

32 frames for 10242px shadow map sampling an area light

1st

4th

For soft shadows the lights’ surfaces are randomly sampled: each frame, a random position on each of the selected lights is used for basic shadow mapping, scene lighting, and shading. Accumulating multiple frames results in realistic and aliasing-free soft shadows with large penumbras. Multi-frame Transparency

5 16th

For anti-aliasing a subpixel offset is generated using shuffled, poisson-disk sampling and added to the vertices’ xy-coordinates in normalized device coordinates (NDC), effectively shifting the complete NDC space per frame. Noticeable shifts for first frames are eliminated by constraining the first sample to the center of the pixel. Furthermore, to avoid clustering at pixel edges and corners of adjacent pixels, we use a tile-based poisson-disk sampling.

1024th frame using per-fragment OIT in conjunction with DoF, AA, and Soft Shadows

For order independent transparency (OIT) transparent fragments are discarded depending on a random opacity threshold either per fragment or object. For it, a mapping of n distinct opacity values in the range [0, 1] to associated bitmasks is precomputed [Enderton et al. 2010] and randomly shuffled for adjacent fragments. By using per-fragment thresholding with back-face culling disabled, high-quality OIT can be achieved.

Multi-frame Depth of Field

MF

The presented techniques have been implemented using the open-source, header-only libraries glm [Riccio 2015] and glkernel [Limberger 2015] used for dynamic computation of kernels of required characteristics at runtime. Examples for industry applications taking advantage of multi-frame sampling comprise MFAA by Nvidia [Nvidia 2015], the Sketchfab viewer [Sketchfab 2012], and Nitrous viewports of 3ds Max [Autodesk 2012]. Acknowledgment This work was funded by the Federal Ministry of Education and Research (BMBF), Germany, within the InnoProfile Transfer research group “4DnD-Vis” (www.4dndvis.de). Related Work

SSAO, 8 samples, no blur

Autodesk, I., 2012. 3ds Max. http://www.autodesk.com/products/3ds-max.

Illustration of a single-frame composition (SF) transformed into multi-frame (MF): One shadow pass instead of 4 and 8 SSAO samples instead of 24 are used. DoF and AA are inherently available due to camera and NDC shifting. Even though this approach is well-known [Fuchs et al. 1985; Haeberli and Akeley 1990], it is neglected or at least could be better exploited in most of todays rendering systems. In this work, optimized sampling strategies for common rendering effects are discussed in detail to ease the adoption of efficient and responsive high-quality rendering [Limberger et al. 2016]. Therefore, underlying sampling characteristics such as (1) number of samples, (2) spatial or value-based distribution, (3) sample regularity and completeness, and (4) temporal convergence constraints, w.r.t. finite quality convergence and resource demands are discussed.

Computer Graphics Systems Group Hasso Plattner Institute

Implementation and References

64th

nth Accumulation

Daniel Limberger, M.Sc. [email protected] +49(0)331 5509 3907 daniellimberger.de

Results

Enderton, E., Sintorn, E., Shirley, P., and Luebke, D. 2010. Stochastic transparency. In Proc. of the 2010 ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, ACM, I3D ’10, 157–164.

64th frame

128th frame using star shape 1st frag. samples

For depth-of-field (DoF) a two-dimensional vector is used to indicate for each point in a scene, where on its circle of confusion (CoC) it should be rendered on the image plane. The sample vector is added to the vertices’ xy-coordinates and scaled by their z-distance to the chosen focal plane or point in view space. Fragments of points distant to the focal plane are spread widely, causing a substantially unsteady convergence. In order to gradually increase the DoF effect, the sequence of samples is sorted by their distance to the center. Furthermore, arbitrary Bokeh shapes can be easily accounted for by masking the samples with the desired shape.

Prof. Dr. Jürgen Döllner [email protected]

Hasso Plattner Institute Prof.-Dr.-Helmert-Str. 2–3 D-14482 Potsdam, Germany

www.hpi3d.de

4 frag. samples

16 frag. samples

1024 frag. samples

Fuchs, H., Goldfeather, J., Hultquist, J. P., Spach, S., Austin, J. D., Brooks, Jr., F. P., Eyles, J. G., and Poulton, J. 1985. Fast spheres, shadows, textures, transparencies, and imgage enhancements in pixel-planes. SIGGRAPH Comput. Graph. 19, 3 (July), 111–120. Haeberli, P., and Akeley, K. 1990. The Accumulation Buffer: Hardware Support for High-quality Rendering. In Proc. ACM SIGGRAPH, 309–318. Limberger, D., Tausche, K., Linke, J., and Döllner, J. 2016. Progressive rendering using multi-frame sampling. In GPU Pro 7. Limberger, D., 2015. Opengl kernel utils (glkernel). https://github.com/cginternals/glkernel.

1st obj. samples

4 obj. samples

16 obj. samples

1024 obj. samples

Convergence for per-fragment (top row) and per-object (bottom row) transparency thresholding. For per-fragment thresholding, back-face culling is off.

Nvidia, 2015. Multi-frame sampled anti-aliasing (mfaa). technology/mfaa/technology.

http://www.geforce.com/hardware/

Riccio, C., 2015. Opengl mathematics (glm). http://glm.g-truc.net/. Sketchfab, I., 2012. 3ds Max. https://sketchfab.com/developers/viewer.