Rendering Using Sky Images as Light Sources

I rendered the following bunny images in Photorealizer using equirectangular HDR sky images as light sources.

Bunny at sunrise / sunset.

Bunny during twilight.

Below are LDR versions of the sky images that I used for the bunny renders above. These are the same images that I posted and described in the previous post.

Sunrise / sunset.

Twilight.

Note the realistic deep blue tone of the twilight images. This blue is a result of ozone absorption. Without ozone, these images would be a dull gray.

Converting Between Fisheye and Equirectangular

I used Photorealizer to convert some of my recent fisheye sky renders to equirectangular (latitude–longitude) format. To give Photorealizer the capability to do this, I copied the panoramic cameras from the sky renderer to Photorealizer, and gave Photorealizer the ability to use a high dynamic range fisheye image as an environment map. Then, to do the actual conversion, I simply set up an environment map using a sky renderer render, and rendered a picture of it using the equirectangular camera (an analogous process could be used to convert from equirectangular to fisheye).

I wanted to convert the fisheye images to equirectangular format because Photorealizer already supports importance sampling for equirectangular environment maps. I plan to use these images as light sources for Photorealizer renders, so being able to sample them efficiently—even when they contain the sun—is very important.

Alternatively, I could have used existing software to do the conversion, but I prefer to do things myself when possible. Or I could have re-rendered the sky images in equirectangular format, but that would have taken a long time. The way I did the conversion worked very well, and it let me add features to and make improvements to Photorealizer in the process.

Here are two sets of before and after images:

Before: sunrise / sunset fisheye render.

After: converted to equirectangular format using Photorealizer.

Before: twilight fisheye render.

After: converted to equirectangular format using Photorealizer.

I recently made some other relevant improvements to Photorealizer as well, which you can read about on my Photorealizer blog:
http://photorealizer.blogspot.com/2012/12/environment-map-improvements.html
http://photorealizer.blogspot.com/2012/12/photon-mapping-with-hdr-environment-maps.html

Twilight with Aerosols

Here are two new twilight images that I rendered with aerosols. I haven't tone-mapped either of these images—they are display-linear. In the second image, the sun is lower, there is a lower concentration of aerosols in the atmosphere, and the ground has a desert-like reflectivity. Because the aerosol concentrations and ground reflectivities differ, the images are not directly comparable, but I believe that a few of the differences are due to factors besides the aerosols. Based on my research (including Color and Light in Nature and The Influence of Ozone and Aerosols on the Brightness and Color of the Twilight Sky) and observations, at −4.8° I would expect a more defined twilight arch, an area of low saturation purple above the twilight arch, and more saturated blue in the zenith direction. All of these phenomena seem to be present in the −4.8° render.

−3° solar elevation angle.

−4.8° solar elevation angle. Half the amount of aerosols as in the
image above.

Comparison

Here's a current render from my sky simulator compared with one from Hosek and Wilkie's (one of the path tracer renders from their paper, scaled up to the same size as my render). I purposely matched the lens, the solar elevation angle, and the exposure, however I did not try to match the atmospheric properties, ground albedo, or tone-mapping. There are a few images in the Hosek–Wilkie paper with a solar elevation angle of 4°, and I simply chose the one that matched my render from the previous post most closely (which was the one with a turbidity of 6), then I re-exposed my (HDR) render to match the Hosek–Wilkie one as well as possible. Besides the numerous differences in the simulations themselves, there a few notable differences between these two images: Hosek and Wilkie's is tone-mapped while mine is not, mine uses a fisheye lens that extends beyond 180° to show the ground, and mine includes the solar disc. In spite of the differences, the images look remarkably similar to me. Using the lightbox, you can toggle between the two images.

My sky simulator.

Hosek and Wilkie's sky simulator.

Aerosols

With aerosols. Solar elevation angle 4°.

No aerosols. Solar elevation angle 4°. Same exposure as the image above.

I made a first pass at adding aerosols to the atmosphere. Aerosols have a significant impact on the appearance of the sky, but simulating them isn't particularly straightforward: the types and levels of aerosols vary widely by region (e.g., city vs. forest vs. ocean) and other factors, and computing their scattering properties is quite complicated.

I used the representative profile of aerosols in Elterman's paper UV, Visible, and IR Attenuation for Altitudes to 50 km, 1968. The paper provides aerosol extinction coefficients that vary based on altitude and wavelength. It's the same data used in The Influence of Ozone and Aerosols on the Brightness and Color of the Twilight Sky, which is what led me to the paper in the first place. I used the provided scale height relationship (scale height of 1.2 km) to extrapolate the data beyond 50 km.

For the scattering phase function, for now I just used the Henyey-Greenstein phase function, which I had already implemented for Photorealizer, including analytic scattering direction sampling. For now I used a constant asymmetric parameter of 0.7 (which means that the mean cosine of the scattering angle is 0.7, which implies strong forward scattering), which seems to be a pretty good average value based on my research. I plan to implement Cornette and Shanks's modified version of the Henyey-Greenstein phase function, which better approximates actual Mie scattering phase functions and converges to the Rayleigh phase function as the asymmetry parameter approaches zero.

I used a constant single scattering albedo of 0.9, which, like my asymmetry parameter, seems to be a pretty good average value based on my research. At the sampled extinction distance, I scatter the photon/ray with 90% probability, and absorb it with 10% probability.

I changed the Earth's surface to have an albedo of 31%, which seems to be the accepted average albedo of Earth's surface. At some point, I would like to make a procedural ground, or use a map of the actual Earth. Procedural mountains would be particularly nice for showing off atmospheric effects.

My aerosols system could be improved in many ways. In particular, I could use a realistic particle size distribution, use realistic particle type proportions (each type having a certain index of refraction, with real and imaginary parts), and then compute scattering and absorption properties using the Mie solution to Maxwell's equations. That sounds like it might be overkill for now, although I'm sure I would learn a lot in the process. 

Unbiased Distance Sampling

I implemented an algorithm for unbiased distance sampling in heterogeneous media, which replaces the biased ray marching in my sky renderer. It's a Monte Carlo algorithm, "Algorithm 1" in the paper Unbiased Global Illumination with Participating Media. I learned about that paper from Hosek and Wilkie's sky model paper. The algorithm originates from a 1968 paper by Coleman, which I'm interested in looking at, but which I haven't been able to locate.

The algorithm samples a random distance based on the highest extinction (or scattering or absorption) coefficient along the ray (or line segment in my case), or any coefficient greater than or equal to that (such as the highest in the entire medium, which is often easier to find). Then it probabilistically takes another leap based on the actual coefficient at the sampled location.

Algorithm from Unbiased Global Illumination with Participating Media.

Using the lowest max coefficient allows larger steps to be taken, which decreases the number of steps needed and increases performance. Since Rayleigh and ozone scattering and absorption coefficients vary predictably based on wavelength, the lowest max coefficient can be found for given wavelength, and with Rayleigh scattering, the lowest coefficient along the line segment can be found by using the coefficient at the point on the line segment that is closest to the center of the Earth (assuming density is monotonically decreasing with altitude).

After implementing this algorithm and the related optimizations, not only are my renders unbiased, but they are also 5 to 10 times faster—a pretty huge speed increase.

Earth from Space

Because I'm performing a brute force atmospheric simulation, I'm able to render the atmosphere from any point of view. After a few tweaks, I'm now able to render images from outside the atmosphere. Here are a few images that show what the Earth looks like from space. The planet's surface is currently an 18% gray Lambertian diffuse reflector.

The planet as seen from 200 km up.

Same view as above, but with no atmosphere.

A different view. Also 200 km up. Sun right below the horizon.

Adaptive Step Size

To increase accuracy and significantly decrease simulation times, I made the ray marching step size vary based on a few heuristics: the rate of change of the altitude at the current location, the altitude of the current location, and an element of randomness.

The first heuristic, the rate of change of the altitude, has the biggest impact on the step size. When a short line segment is nearly parallel to the ground, the altitude doesn't varying much along the segment, which means that the density and other properties of the air also don't vary much along the segment. So if we're travelling parallel to the ground we can take much larger step sizes than if we're travelling perpendicular to the ground. When we're travelling up and down, the atmospheric properties change much more quickly and we need to smaller step sizes to accurately capture the changes and details of atmospheric properties.

These are all just a temporary improvements because I just realized that I could replace ray marching with an unbiased technique, which should also be cleaner and faster. I'll write more about this later, after I implement it.

If I were to stick with ray marching, I would want to additionally vary the step sizes based on wavelength and other factors, and I would want to take different step sizes when computing different types of scattering and absorption.

Twilight Fisheye

Here's a new angular fisheye render taken at twilight (solar elevation angle −3°) in which you can clearly see the twilight arch, the anti-twilight arch, and the Earth shadow.

To produce the image below, I first took advantage of the symmetry of the render to effectively double the number of samples, using Photoshop to duplicate the high-dynamic range EXR image, flip it horizontally, and composite it onto the original. Then I used Photomatix to tone-map the processed image. The solar (top) side is much brighter than the anti-solar (bottom) side, so without tone-mapping it wouldn't be possible to expose the entire sky nicely at one time—part of the image would be overexposed or part would be underexposed. The tone-mapped image is over-saturated and the relative intensities of different parts of the sky are not accurate, but you can see the entire sky and you can see the colors clearly.

Twilight, tone-mapped.

And below is the PNG that came directly out of my renderer. I applied a mild S-shaped transfer curve before writing to PNG.

Twilight, directly from my renderer.

The sky was clear this evening in Philadelphia and I was able to see the Earth shadow. The colors in the real sky closely resembled those in this render.

New Panorama Cameras

I made two cool new cameras that allow the entire sky to be viewed in a single image.

The first uses latitude-longitude panoramic format, also known as an equirectangular mapping. This is the same format that I use for HDR environment maps in Photorealizer, and my sky renderer already saves images to EXR (in addition to PNG), so I will be able to use my sky renderer renders as HDR environment maps for my Photorealizer renders. Plus, I have HDR environment map importance sampling in Photorealizer, so I can leave the tiny, bright, influential sun in the sky images and Photorealizer will automatically know to heavily sample the sun.

Equirectangular projection.

The second new camera uses an angular fisheye projection. This is a pretty intuitive way to view the entire sky (and part of the ground in my implementation). This kind of picture can be captured in real life in a single shot with a fisheye lens. Real fisheye lenses are pretty advanced. Here is a site that shows how a real fisheye lens works, with nice diagrams of the fancy optics.

Angular fisheye projection.

The environment is identical in both images above. The solar elevation angle is 10°. I think that the two bright peaks above the horizon in the solar and anti-solar directions are due to the forward and backward peaks of the Rayleigh scattering phase function.

Vertical Distribution of Ozone

I noticed that the sky right above the horizon seemed too blue after I implemented ozone absorption. I realized that since the most Rayleigh scattering happens in the low, dense part of the atmosphere near the ground, scattered light would be passing through too much ozone if the concentration were too high at these altitudes. I also thought more generally about the geometry of the atmosphere and the differing amounts of ozone that light would pass through based on the distribution of the ozone. I had made the ozone concentration a constant value of 0.6 parts per million, which had made the total ozone amount correct, but it had also made the amount near the ground far too high, and the amount higher in the atmosphere far too low. That amount of ozone at ground level would be dangerous to your health. In reality, most of the ozone in the atmosphere is found in the ozone layer in the stratosphere. The ozone layer is responsible for absorbing most of the sun's harmful UV rays.

To improve my results, I implemented a realistic distribution of ozone. Average data for the Earth's atmosphere was difficult to find, and the exact distribution of ozone is pretty variable anyway, so I read data from the NASA graph below, and made a few tweaks based on other information I had found (namely, lowering the concentration in the troposphere, and weighting the data up to make the total concentration closer to 0.6 ppm). I might want to tweak or replace the data in the future, but it's already much more accurate than the constant value I was using before.

Ozone concentration in the atmosphere. Image from NASA.

Below is a new render, and some old ones for comparison. In the new render, compared to the old ozone render, the sky around the zenith is bluer, and the the color and brightness near the horizon is more accurate.

Ozone with realistic vertical concentration profile.

Ozone with uniform concentration throughout the atmosphere.

No ozone.

Here's another new render, this one taken during twilight in the direction of the sun, with the sun −3° below the horizon:

Twilight + ozone.

By the way, ozone absorption should make the color of the Earth shadow at twilight more accurate (darker and bluer), so I'm looking forward to rendering some shots that illustrate that.

Ozone Absorption

A couple weeks ago, I noticed that, in my sunset and twilight renders, towards the zenith, the sky looked fairly gray, while in real life it looks fairly blue. I realized that implementing the actual solar spectral irradiance would make the sky appear slightly bluer, but I didn't expect the effect to be strong enough to explain the discrepancy between my images and real life. Then I stumbled across something very interesting on Wikipedia:

According to Craig Bohren, "preferential absorption of sunlight by ozone over long horizon paths gives the zenith sky its blueness when the sun is near the horizon".

I proceeded to check the referenced Craig Bohren paper, where the statement was explained in more detail. In particular, the Chappuis absorption band of ozone, "which extends from about 450 to 750 nm and peaks at around 600 nm", is responsible for absorbing light of longer wavelengths in the visible spectrum, thus making the sky appear more blue. He also said that

In the absence of molecular absorption, the spectrum of the zenith sky would be essentially that of the zenith sun (although greatly reduced in radiance)

Bohren referenced a 1953 paper by Hulbert, who originally pointed out this explanation of this blueness: Explanation of the Brightness and Color of the Sky, Particularly the Twilight Sky. Here's an excerpt from the abstract of that paper:

Calculation showed that during the day the clear sky is blue according to Rayleigh, and that ozone has little effect on the color of the daylight sky. But near sunset and throughout twilight ozone affects the sky color profoundly. For example, in the absence of ozone the zenith sky would be a grayish green-blue at sunset becoming yellowish in twilight, but with ozone the zenith sky is blue at sunset and throughout twilight (as is observed), the blue at sunset being due about 1/3 to Rayleigh and 2/3 to ozone, and during twilight wholly to ozone.

I also found a paper from 1973, The Influence of Ozone and Aerosols on the Brightness and Color of the Twilight Sky, that confirms the effect of ozone on the color of the twilight sky using computer simulations.

I decided that I needed to include ozone absorption in my sky renderer, so I searched around for spectral ozone absorption data, and eventually came across recent (2011), high-precision, absorption cross-section measurements here. The data is provided in 0.01 nanometer increments, which is way too precise for my purposes, so I wrote some code to convert the data to 5 nanometer increments, averaging 500 data points to create each new data point. The data is provided for 11 different temperatures. They're similar enough that I probably could have just used one, or averaged them all together, but I decided to use all of them anyway. When a query is made, I just look up the data at the closest temperature and wavelength—I decided that interpolation was probably overkill in this case.

Ozone absorption cross-sections. Image from IUP.

To convert the absorption cross-sections (units of cm^2/molecule) to absorption coefficients (units of cm^-1), I multiply by the molecular number density of standard air (units of molecules/cm^3), multiply by the concentration of ozone in the atmosphere (unitless), and then correct for altitude by multiplying by the relative density (unitless) at the altitude in question. The concentration of ozone varies a lot with altitude, but I just used the overall concentration, which is around 0.00006% (I wonder to what extent a more realistic distribution would affect my results). I packaged all of this up in a new Ozone class.

Below are some new renders with and without ozone absorption. These images are taken during twilight, looking straight up along the zenith.

Twilight zenith sky with ozone.
Solar elevation angle: −5°.

Twilight zenith sky without ozone.
Solar elevation angle: −5°.

Below are some more new renders (all with the same exposure) that show the effect of ozone absorption on the color of the sky. The previously intense orange at sunset is now much more subdued. The orange should become somewhat more saturated when I add aerosols to the atmosphere—an effect described in The Influence of Ozone and Aerosols on the Brightness and Color of the Twilight Sky and in Color and Light in Nature.

With ozone.
Solar elevation angle: 1°.

Without ozone.
Solar elevation angle: 1°.

With ozone.
Solar elevation angle: 30°.

Without ozone.
Solar elevation angle: 30°.

I'm pretty excited about these results. I think it's very interesting that Rayleigh scattering alone does not explain the blueness of the sky. According to The Effect of Stratospheric Dust on the Color of the Twilight Sky, adding dust will increase the blueness of the twilight sky even more. I wonder whether implementing absorption by other gases would have any other noticeable effects on my results.

By the way, as far as I know, the existing sky simulations and analytic sky models in computer graphics do not consider the effects of ozone absorption.

Spectral Solar Irradiance

Image created by Robert A. Rohde / Global Warming Art

I implemented the spectral irradiance of the sun based on modern AM0 data. AM0 (Air Mass Zero) means that the data is for sunlight that hasn't passed through any atmosphere, which is what I needed because I am simulating the atmosphere myself. This extraterrestrial spectral solar irradiance data is often used for space applications. AM1.5 data is also available, for ground level applications such as solar energy. I found the data on NREL's website, and it originally came from a 2003 paper by Guemard. In my renderer I convert irradiance to radiance instead of using the data directly. Previously, I was using a constant value for the sun's energy, uniform across the spectrum. I could have alternatively used a black body radiation curve to approximate the spectral irradiance, however spectral solar irradiance does not exactly follow a black body radiation curve.

Here are some renders comparing the new and old solar spectra. To clearly see the differences, you can click an image to bring it up in the lightbox, then you can then switch between images. One of the most noticeable differences is that the new images are bluer.

New.

Old.

New.

Old.

To use the spectral solar irradiance data, I wrote a LookupTable class with a custom binary search and linear interpolation, and then made a SpectralSolarIrradiance subclass with an array of wavelength keys and spectral irradiance values. I made another subclass of LookupTable called RayleighScatteringDepolarizationFactor to store the depolarization factor data for Rayleigh scattering. I could also make my CIEXYZColorMatchingFunctions class extend the LookupTable superclass, however it currently uses optimized code that I wrote specifically for data given in uniform increments. I will probably also use the LookupTable class for ozone absorption data. Ozone absorption is an important factor that I will write about in a future post. I also now have new classes for USStandardAtmosphere1976, RayleighScattering, which I didn't write right away just because the code was in a lot of flux. I'll post more information about the structure of my code another time.

Shadow of the Earth

When flying from San Francisco to Los Angeles this summer, we took off right before sunset, and I was lucky enough to have a window seat. As we emerged from the clouds into the clear sky above, I had a brief but amazing view of the bright orange sun shining and scattering through the clouds. I was on the left side of the plane, so as we turned to the south, the view of the sun went away, but the sun set soon anyway.

What I saw after the sun had set was really remarkable. The sky was very clear, and above the horizon I started noticing an expansive dark area with a pretty sharp edge, gradually consuming the sky. I soon realized that what I was seeing was the shadow of the Earth in the atmosphere. Inside the shadow, the sun had already set, but in the brighter area above the shadow, the sun would have still been directly visible.

I am able to replicate this effect in my sky renderer.

Here's a render at twilight, from the ground, facing away from the sun. You can clearly see the shadow of the Earth in the lower part of the image above the horizon (the ground is the sharp, rectangular, solid-colored area at the bottom). You can even see the anti-twilight arch!

Twilight, facing
away from sun.

And below is my earlier twilight render, looking in the direction of the sun, for comparison. The exposure on this image is three stops lower than the shadow image above. I raised the exposure on the image above because the sky was much darker in the direction opposite the sun.

Twilight, facing
towards sun.

Visible Spectrum

Here's what the visible spectrum looks like at a few different exposures when rendered in my sky renderer. I used my new spectral rendering and display system (spectrum → CIE XYZ → linear RGB → sRGB) to create these images.

CIE XYZ

The sRGB gamut in the CIE xy chromaticity diagram.
Image found on Dietrich Zawischa's colorimetry page.

One of the first things I did for this project is to write a spectral rendering system that associates each ray with a wavelength instead of an RGB color. Initially, in order to convert the spectral data to RGB primaries, I made up response curves for the camera's RGB sensors, but this was just a rough placeholder system. I considered implementing the response curves of a real-life camera, however, I decided on a more scientific approach. I decided to convert my spectral data to CIE XYZ by integrating it with the CIE XYZ color matching functions (specifically, the CIE 1931 2° XYZ CMFs modified by Judd and Vos), then transforming XYZ to linear RGB (Rec. 709 primaries), then finally transforming that to sRGB. This approach yields colorimetrically accurate results and is very flexible. I'm very interested in color, so doing this was also fun and good practice.

Here are after and before shots of twilight. Notice how the hues shift:

Updated twilight
image.

Old twilight image,
for comparison.

Here are a few more new renders:

Updated day
image.

Notice the multicolored noise in this quick render, resulting from spectral rendering coupled with Monte Carlo scattering:

Updated sunrise /
sunset image.