THREE.js-PathTracing-Renderer

Real-time PathTracing with global illumination and progressive rendering, all on top of the Three.js WebGL framework.

Quick Controls Notes

LIVE DEMOS

For comparison, here is a real photograph of the original Cornell Box vs. a rendering with the three.js PathTracer:


Natural Environment Ray Marching Demos

BVH Acceleration Structure Demos

Raytraced Shapes Demos

The above image was my inspiration to embark on the years-long (and still ongoing!) journey to implement a complete library of analytically ray-traced mathematical shapes that can be rendered in realtime inside a browser. The image is a computer screen grab from an old cinema magazine article showing how the vintage CG company MAGI made their iconic imagery for the 1982 movie, TRON. I saw that movie in theaters when it came out (I was 9 years old, ha) and at first I thought, since it was a Disney movie, that their artists had hand-drawn all the crazy scenes and sci-fi vehicles. As the end credits rolled though, it said ‘computer imagery and animation by MAGI’. Mind blown! At 9 years old in the early 1980’s, I hadn’t seen anything like that in a movie - I couldn’t even comprehend how they made all those cool scenes/vehicles inside of a computer! The film really peaked my interest in computer graphics and nearly 40 years later, I am happy to report that my quest to be able to render all the shapes that MAGI could has been largely successful! For those that are interested in the math, these are all quadric shapes - shapes that can be defined implicitly (i.e., a unit sphere: x2 + y2 + z2 - 1 = 0) and reduced to a quadratic equation in the ray’s ‘t’ value, which can be easily solved by a computer to quickly find the roots (t0, t1). Using these mathematical primitives, MAGI was able to construct all the cool vehicles featured in the movie. An interesting side note: they did not use triangles/polygon modeling like the CG industry does today - it was mainly these math shapes with pixel-perfect continuous-looking curves. Also noteworthy is that they used ray tracing to render the final animations. Each frame took 30 minutes to multiple hours. Well I’m happy to say that you won’t have to wait that long to see an image now - my shapes render at 30-60 FPS inside your browser, even on mobile! ;-)

CheapTorus2

Constructive Solid Geometry(CSG) Demos

The following section deals with different techniques in Constructive Solid Geometry(CSG) - taking one 3D mathematical shape and either adding, removing, or intersecting a second shape.

All of the following 4 demos feature a large dark glass sculpture in the center of the room, which shows Ellipsoid vs. Sphere CSG.

Important note! - There is a hidden Easter Egg in one of these 4 Museum demo rooms. Happy hunting!

Sometimes we don’t want any curves at all (ha) - rather, we’d like to be able to render simple, straight-edged, faceted shapes such as pyramids and prisms. Using a technique developed by Eric Haines for his article in Graphics Gems II (published 1991), I created several routines to efficiently raycast these special shapes in the browser. The algorithm takes in a set of user-defined cutting planes. These planes are of type ‘vec4’ and consist of a 3D normal vector to the plane, as well as a float distance to the plane from the shape’s origin (its center of gravity). For instance, to define a simple unit cube, you would provide 6 planes, all with normals pointing in different cardinal axis directions, and all with a distance of 1.0 from the cube’s origin. All pyramids, prisms, and Platonic solids can be built up in a similar manner. The ray intersection algorithm then takes these small lists of cutting planes and uses them as ‘half-spaces’, resulting in perfectly sharp, straight-edged, faceted shapes.
Check out the demo below, which renders the most common convex polyhedra (faceted shapes) that we might want to have in a scene. The demo includes a couple of material presets such as diffuse, metal, glass, and clearCoat diffuse, as well as a roughness slider. Finally, the shapes’ transforms are exposed in the GUI menu, so you can easily adjust the position, scale, rotation, and skew of all the shapes in real time!

Materials Demos

Quadric Shapes BVH

The above image of the classic scene entitled “Invisible Date” (by Christa Marx, 2000) inspired me to try and render it in real time with the browser. I knew the amount of geometry was going to be a challenge, simply because the original scene was composed of several thousands of triangles. Now my standard triangle BVH system could have handled it, and it would have run smoothly on a desktop with a nice GPU - but not so much on mobile devices. Devices like cell phones have a harder time traversing the deep BVH trees that these kind of triangle-heavy scenes produce. Then, I noticed that most of ‘Invisible Date’s geometry (except for the Utah Teapot) was built up from simple, quadric shapes like spheres, cylinders, cones, and rectangular boxes of various dimensions. So I decided to put together a custom ‘Shapes BVH’ system - instead of handling thousands of small triangles as all other ray tracing BVH builders do, it instead works with larger, simpler shapes like spheres and boxes that are ray tracing-friendly. Since a larger, quadric shape can cover an area that would have required hundreds of triangles, now the geometry count goes down exponentially! Instead of several thousands of triangles and all of their bounding boxes required for the original scene, our new ‘Shapes BVH’ boils everything down to… 54 shapes! And a BVH tree with only 54 leaves (the shapes themselves) will basically run on a toaster, lol! When I first tried out this classic scene on my phone, I was delightfully surprised that it ran at a smooth 60 FPS! Check it out for yourself:

I think that this system might help us get closer to the dream of having most graphics ray traced in real-time with the browser, especially when it comes to under-powered devices like cell phones, tablets, and mobile VR headsets. I don’t believe that we can just throw the standard ‘triangle BVH’ system at phones and tablets and expect them to perform like a desktop with an NVIDIA RTX ray tracing card. I think that for the time being, until small mobile devices get way faster and more energy efficient, that we will have to rely on more ‘outside-the-box’ thinking and techniques, if we are to have any hope of having high quality ray-traced graphics available everywhere around us, on any device. I’m really excited to see what the future of graphics on the web has in store for us. Imagine visiting a website or playing a game in a photorealistic real-time path traced environment…in VR…on the go!

Path Tracing 3D Fractals

CubeFractal

Another good use case for my Shapes BVH technique is for path tracing 3D fractals. By using these raytracing-friendly shapes, we can have over 200,000 shapes, each with their own unique transform and material, all running at 30-60 fps…even on your cell phone! Here is a demo that generates many iterations of the ‘cube-frame’ 3D fractal. This scene also features a physical sky model that renders a realistic, atmospheric sky as seen from Earth. In the GUI menu, you can rotate the Sun to get different times of day as well as play with the lighting and soft shadows in real time. You can also select from 10 interesting camera viewpoint presets that show off the epic fractal nature of the building as well as its dramatic lighting effects.

Classic Scenes / Ray Tracing History

<img src=”https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/readme-Images/4-Figure7-1.png” width=30% height=30%>

Arthur Appel is credited with the first formal mention of Ray Tracing (raycasting and shadow rays, shown above) in his 1968 paper Some Techniques for Shading Machine Renderings of Solids while working at IBM Research (TJW Center). Mr. Appel used this new technique to help visualize machine parts and architectural concepts on printed paper in black and white. The scene data was sent to an IBM 1627 (Calcomp) digital plotter that cleverly used text characters (like ‘+’) with different spacing and brightness to differentiate the various shading of sides of a 3D model under a virtual light source. Here are a few examples of Mr. Appel’s digital plot renderings from his 1968 paper:

<img src=”https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/readme-Images/2-Figure2-1.png” width=20% height=20%> <img src=”https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/readme-Images/4-Figure3-1.png” width=70% height=70%>

For reference, here is a link to all the images featured in the research paper: Original Appel Renderings (click on the ‘View All 14 Figures and Tables’ button below the first images).

And here is a demo that lets you literally ‘jump into’ Appel’s 1968 research paper and experience his groundbreaking techniques of per-pixel raycasting and shadow rays:

Scenes that used to take several minutes on Appel’s digital plotting device now run at 60 fps in your browser! I think Arthur would get a kick out of dragging the sunlight around in real time on his classic scenes!

Until now (2021), actual photos of Arthur Appel were not publicly available (none can be found with a thorough internet search). All that was known was that he was working at IBM Research (TJW Center) at the time he wrote this seminal 1968 paper. I really wanted to see what Mr. Appel looked like, and to share and celebrate his image and contributions to the field of Ray Tracing and Rendering. With a little hesitation at first, I reached out to the IBM Corporate Archives in New York to see if they might have any remaining employee portraits of Arthur Appel. I’m so glad I did, because I met (via email) a wonderful IBM Archive employee, Max Campbell, who kindly searched the entire archives and found 2 rarely-seen photos of Mr. Appel. Since these images are copyrighted by IBM (and NOT a part of my repo’s CC License), Max also kindly and graciously helped me to obtain permission from IBM to share these historic photos of the man who started it all! Click on the images to see the full resolution photos:

<img src=”https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/readme-Images/1982_December_Arthur%20Appel_IBM%20Research%20Magazine.png” width=20% height=20%>
Arthur Appel, from the IBM Research Employee Gallery, ca. 1982 Reprint Courtesy of IBM Corporation ©

<img src=”https://github.com/erichlof/THREE.js-PathTracing-Renderer/blob/gh-pages/readme-Images/1983_December_Arthur%20Appel_IBM%20Research%20Magazine.png” width=20% height=20%>
Arthur Appel demonstrating display architecture, from IBM Research Magazine ca. 1983 Reprint Courtesy of IBM Corporation ©

Many thanks to Max Campbell at IBM Research Archives for locating these rare photos and helping me to obtain permission to share them with everyone who is interested in ray tracing! It is so nice to be able to finally put a face with the name of one of my ray tracing heroes. Thank you Arthur Appel for your historic contributions to the field of Computer Graphics!

While working at Bell Labs and writing his now-famous paper An Improved Illumination Model for Shaded Display, J. Turner Whitted created an iconic ray traced scene which showcased his novel methods for producing more realistic images with a computer. Beginning work in 1978, he rendered a handful of scenes featuring spheres and planes with various materials and reflectivity, so that these images would be included in his paper (which would be published in June 1980). Then for an upcoming SIGGRAPH conference submission, Whitted decided to create an animated sequence of individual rendered images. Thus the first ever ray traced animation was born! This style of putting together single frames of pre-rendered images would continue through a great lineage of movies such as Tron, Toy Story, Cars, all the way to current animated feature films.

Vintage 1979 Video: ‘The Compleat Angler’ by J. Turner Whitted

Although this movie appears as a smooth animation, it took around 45 minutes to render each individual frame back in 1979! Fast forward to today and using WebGL 2.0 and the parallel processing power of GPUs, here is the same iconic scene rendered at 60 times a second in your browser! :

Thank you Dr. Whitted for your pioneering computer graphics work and for helping to start the rendered animation industry!

In 1986 James T. Kajiya published his famous paper The Rendering Equation, in which he presented an elegant and profound unifying integral equation for rendering. Since the equation is infinitely recursive and hopelessly multidimensional, he suggests using Monte Carlo integration (sampling and averaging) in order to converge on a solution. Thus Monte Carlo path tracing was born, which this repo follows very closely. At the end of his paper he included a sample rendered image that demonstrates global illumination through Monte Carlo path tracing:

And here is the same scene from 1986, rendered in real-time:

The next classic ray traced scene comes from Eric Haines. In 1987 for the SIGGRAPH Art Show, Haines presented an image of several thousand spheres arranged in his custom 3D fractal pattern, which he called ‘Sphereflake’. The fractal is generated by first placing the large root parent sphere in the center. Then 9 smaller child spheres are placed at equidistant angles on the parent sphere’s surface. On the next iteration, those 9 child spheres become parents themselves, spawning 9 even smaller child spheres on their surfaces. The process continues in fractal fashion, leading to an exponential increase in the amount of spheres on each iteration.

For this demo, I chose 5 iterations of Haines’ fractal pattern, which means that in the end we have: 1 root parent sphere + (9) + (9x9) + (9x9x9) + (9x9x9x9) + (9x9x9x9x9) = 66,430 spheres total! This dense fractal structure relies on my new custom Shape_BVH builder, which instead of handling typical triangles of a triangular model, handles quadric shapes (spheres, boxes, cylinders, cones, paraboloids, etc.) for fractal and CSG models. These simple math shape primitives are ray-tracing friendly and with the help of my BVH tree builder, it accelerates the rendering to real-time, even on your cell phone! Also, this demo allows you to change the entire Sphereflake material to common material presets, like metal, clearCoat, glass, etc. Just for fun, I included a ‘Random’ material option which assigns a randomized unique material type to each of the 66,430 spheres!

Here is Haines’ Sphereflake fractal, path traced in real-time:

Bi-Directional Path Tracing

In December of 1997, Eric Veach wrote a seminal PhD thesis paper on methods for light transport http://graphics.stanford.edu/papers/veach_thesis/ In Chapter 10, entitled Bi-Directional Path Tracing, Veach outlines a novel way to deal with difficult path tracing scenarios with hidden light sources (i.e. cove lighting, recessed lighting, spotlights, etc.). Instead of just shooting rays from the camera like we normally do, we also shoot rays from the light sources, and then later join the camera paths to the light paths. Although his full method is difficult to implement on GPUs because of memory storage requirements, I took the basic idea and applied it to real-time path tracing of his classic test scene with hidden light sources. For reference, here is a rendering made by Veach for his 1997 paper:

And here is the same room rendered in real-time by the three.js path tracer:

The following classic scene rendering comes from later in the same paper by Veach. This scene is intentionally difficult to converge because there is no direct light, only indirect light hitting the walls and ceiling from a crack in the doorway. Further complicating things is the fact that caustics must be captured by the glass teapot on the coffee table, without being able to directly connect with the light source.

And here is that scene rendered in real-time by the three.js path tracer: Try moving the GUI slider to open and close the door!

I only had the above images to go on - there are no scene dimensions specifications that I am aware of. However, I feel that I have captured the essence and purpose of his test scene rooms. I think Veach would be interested to know that his scenes, which probably took several minutes if not hours to render back in the 1990’s, are now rendering real-time in a web browser! :-D

For more intuition and a direct comparison between regular path tracing and bi-directional path tracing, here is the old Cornell Box scene again but this time there is a blocker panel that blocks almost all of the light source in the ceiling. The naive approach is just to path trace normally and hope that the camera rays will be lucky enough to find the light source:

Game Engine path tracer for Desktop and Mobile

Before I got into this world of path tracing, I was a 3D game programmer (and still am, although path tracing is consuming most of my coding time!). My first game was way back in 1998, using OpenGL 1 and the C language, back when these new things called graphics cards were all the rage! my old Binary Brotherz page Although using OpenGL back then and WebGL today was/is cool, I always wanted more in terms of lighting, shadows, reflections, diffuse color sharing, etc., in my game engines that I just couldn’t get from rasterizing graphics APIs. Well, fast forward to 2019 and NVidia is releasing graphics cards dedicated to real-time ray tracing! I couldn’t have imagined this back in the 90’s! However, at the time I’m writing this, NVidia is only doing specular ray tracing as a bonus feature on top of the old rasterization technique. I wanted to see if I could ‘overclock’ my full path tracer’s convergence so that you could see the beautiful light effects in real time, being able to possibly move a game character or 1st-person camera through a path-traced dynamic game environment at 30-60 fps, even on mobile. If you’re willing to sacrifice some ultimate physical reality (like perfect converged reflected/refracted caustics), then you can have this!:

PATH TRACED GAMES

I am pleased to announce the first ever path traced game for desktop and mobile: AntiGravity Pool! If you’ve ever played American 8-ball before, then you already know how to play - except that gravity has been shut off! LOL. I tried to imagaine how our distant future descendants would enjoy the game of billiards while in the HoloDeck. Warping the 2D classic pool table into a 3D cube presents some unique and interesting challenges for the player. AntiGravity Pool features real-time raytraced reflections, soft shadows, and path traced global illumination from 8 light sources (which is challenging for path tracers). Since it uses a physics engine and various custom components, I decided to create a dedicated repository for just this new game. Be sure to check it out!

Continuing my series of path traced games for desktop and mobile, I happily present: Path Traced Pong! The iconic game of Pong holds a special place in my heart as it was my first computer game experience as a 6 year old in 1979, played on my brand new Atari 2600! My version of Pong brings the classic into 3D, and is played inside the CG-famous ‘Cornell Box’. Path Traced Pong features real time raytraced reflections, soft shadows, transparency, dynamic light sources, and path traced global illumination. As with AntiGravity Pool, I made a dedicated repository for just this new game. I must say, once you start playing, it’s hard to stop! I didn’t realize how addictive it would become!

In 1986 when I was 13 years old and on my Commodore 64 (I know, I’m old), Geoff Crammond released his masterpiece, The Sentinel. This iconic game featured true 3D filled polygons (an amazing feat running on underpowered 80’s hardware!) and had a haunting look and atmosphere like no other before it (or after). This was the first game that I played that truly immersed me, surrounding the player from all angles with its sterile, other-worldly environment. I’ve always wanted to pay homage to my favorite game of all time, while maybe adding some of my personal path tracing touch to it. So it is with much joy that I present, The Sentinel: 2nd Look. This fully path traced remake contains a random landscape generator (which I had to figure out from looking at the classic over several months), an added day cycle, pixel-perfect raytraced shadows on the terrain and game objects, object self-shadowing, and true raytraced reflections on the white/black connector panels of the landscape.

Creating this remake forced me to figure out how to make a dynamic top-level BVH over many moving, rotating game objects/models, each with their own unique BVHs for their own triangle geometry. I’m happy to report that not only does my new system work, it can completely rebuild and update the whole top-level BVH in a split second, allowing for more complex, path traced dynamic game environments! As of now, this project is a W.I.P. (gameplay and game logic to be added soon), but I just wanted to share this passion project of mine, as well as the technical major step forward (in BVH technology) that will allow a wider range of real time games and applications to be path traced right inside your browser!



My New YouTube series! The Joy of Ray Tracing

You may be interested to know that I have started my own video tutorial series on YouTube all about ray tracing! It’s called The Joy of Ray Tracing and together we will make several different kinds of ray tracers - from scratch !



JoyOfRayTracing_thumbnail7 JoyOfRayTracing_thumbnail JoyOfRayTracing_thumbnail4


My YouTube Channel intro


The Joy of Ray Tracing Video Series


If you want to know how all of my ray tracing and path tracing demos work under the hood, then look no further. And when I say ‘from scratch’, I really mean it! Chapter 0 of the video series is all about installing and setting up a productive coding environment for the web (mainly VS Code and some helpful plugins), the basics of HTML/CSS/JavaScript, GitHub basics plus how to use GitHub’s integration with VS Code to easily create your 1st GitHub repo, how to use VS Code’s awesome Live Server plugin to automatically run and test your code on any device on your WiFi network and to instantly see the visual results of your code changes, and finally an introduction to the web’s Canvas element that allows us to draw our 1st pixels to the screen!

Therefore, if you’re an experienced web developer who already has a working coding environment and who is comfortable with HTML/CSS/JavaScript basics, then you can safely skip Chapter 0 and meet us in Chapter 1. In this chapter we start talking about concepts and techniques that are specific to Ray Tracing. As of this Readme update today (2/7/2023), I am still in the process of filming the rest of the videos for Chapter 1. But by the end of this chapter, you will have created your first basic ray tracer that runs on any device! The following chapters will explore more intermediate techniques and our ray tracers will get more sophisticated (and awesome!). Along the way we’ll also learn about the fascinating history of Ray Tracing, which is over 55 years old by now! (wow, something that’s actually older than I am, lol)

As we go through the series making our ray tracers, I code every single line on-screen, and then all of the code (and live demos!) will be placed in a dedicated GitHub repository with the same name (Joy of Ray Tracing):

The Joy of Ray Tracing companion code repository

I hope to see you over on YouTube! I’m really excited to have you along for the ride!
So…

Let's get started on our journey to discover... The Joy of Ray Tracing!




A random sample rendering from the three.js pathtracing renderer as it was back in 2015!

FEATURES

Experimental Works in Progress (W.I.P.)

The following demos show what I have been experimenting with most recently. They might not work 100% and might have small visual artifacts that I am trying to fix. I just wanted to share some more possible areas in the world of path tracing! :-)

Some pretty interesting shapes can be obtained by deforming objects and/or warping the ray space (position and direction) around these objects. This demo applies a twist warp to the spheres and mirror box and randomizes the positional space of the top purple sphere, creating an acceptable representation of a little cloud.

Normally, all of my demos on this repo use a single pixel sample per frame, which keeps the framerate high (30-60 fps) for all devices, even cell phones. But a naive implementation of just 1 pixel sample per frame results in large amounts of distracting noise. I use my custom-built denoiser to smooth out and quiet down the noise, giving much better picture quality for all devices at interactive framerates. However, several users have requested demos of more than 1 sample per animation frame, because they have more powerful systems with the latest dedicated GPUs, and they want to utilize these resources to the max. So here are a couple of demo examples that allow multiple pixel samples per frame. The first demo is the Geometry Showcase Demo, but with a slider that lets you crank up the pixel samples anywhere from 1 to 100. The second demo is a similar scene, but with dynamic, moving objects like you might have in a game, and also lets you adjust the number of samples per frame. The reason why these multiSPF demos are here in the experimental section is because they do not have the denoiser in place yet. My denoiser relies on the number of samples (which has previously been 1) in its calculations over time to smooth out and converge the image. I will have to convert the denoiser so that it will work properly with multi-sample scenes like these:

When rendering/raytracing Terrain, you can either raymarch a Perlin noise texture (as I have demonstrated in the above Terrain_Rendering and Planet_Rendering demos), or you can just load in a large pre-existing triangle terrain mesh and raytrace it in the traditional way. Both have their advantages and disadvantages. However, if you want to go the classical raytracing route, to make the land contours a little more convincing, there needs to be a lot of triangles! The following WIP preview demo uses the BVH acceleration structure to load in and quickly render a huge terrain mesh consisting of no less than 734,464 triangles! It really pushes my BVH code to the max - we’re pretty near a million triangles here, pathtracing in WebGL! For now I just stuck a checker texture across the terrain and the environment is simply a large skylight dome. But the good news is that it doesn’t crash the browser, and it runs slightly above 20 fps even on my humble laptop - it’s amazing that all of this is happening inside a browser webpage! Note: because of the large BVH data set that needs to be built at startup, this demo might take a few seconds to compile - please be patient, it’s worth the wait! ;-)

Inspired by an older Shadertoy demo by user koiava that I came across - https://www.shadertoy.com/view/MtBSzd - I noticed that my mobile device didn’t have any problems when trying that particular demo with 1000 triangles. I copied / edited / optimized the traversal code and then, I did the unthinkable (for me anyway) - I sent down over 2 million triangles to the engine to be raytraced, then raytraced yet again for the reflection/shadow ray pass (so effectively 4,200,000 triangles in a single frame, and …. my Samsung 9 still runs at nearly 60 fps! It didn’t even blink an eye. Compilation takes maybe 1 second. I couldn’t believe what I was seeing at first.

A technical note about what you are seeing: The data arrives to the fragment shader through a 1024x1024 heightmap texture (I randomly chose a DinoIsland.png heightmap, but it can be anything, even a realtime video texture). The acceleration structure handles sampling the texture and stepping the ray through each sample cell. The number of cells is up to you. At first I tried 32x32 cells, and each cell is a square, and each of the 4 corners of that square is a vertex that is used by 2 triangles sandwiched together back-to-back. So to get the number of triangles that you must raytrace, you take 32 cells width times 32 cells height and remember that each square cell contains 2 triangles, so multiply all that times 2, so 32Wx32Hx2t which is 2048 triangles representing the polygon heightmap. Now 2048 triangles sounds like a lot, and it is for raytracing, but the result mesh looks like an old-school low-poly terrain - it is not detailed enough. On a whim, I tried a resolution of 1024, so each little texel of the 1024x1024 source texture image has its own quad cell, and 2 triangles for every one of those quad cells. So now we have 1024x1024x2, or 2,097,152 triangles every frame! And since the grid looks up the texture to get the triangle vertices every frame, you can animate the height/depth of the displacement as well as even play an HD video (saved as textures) with an embossed-effect on the terrain in real time!
Oddly, it turns out that my Mobile device (a Samsung S9) trounces my laptop at this demo. The humble old laptop achieves maybe 20fps, whereas the newer smartphone rocks at 60fps. It may have to do with the cheap integrated graphics of my laptop, but in any case, this is a true testament to the power of modern smartphone GPUs!

Updates

TODO

ABOUT

More examples, features, and content to come!