---

Jon wanted to convert his demos into proper language tests, and add in new ones. Testing becomes increasingly important as we approach a closed-group release. As it turns out, the literature on compiler testing is scarce, and getting access to quality test suites is pricey (see C or FORTRAN). I've also disliked test frameworks in general, which added to the problem.

In the interest of feeling joyful and productive, I decided to sit and think about the specific reasons why I've been unhappy with frameworks, and to use the language to write one I'd enjoy using.

How I Like to Test

I like to ask personally interesting questions which the tests ought to help me answer. Often, they relate to the behavior of software at the threshold of cross-cutting concerns. For the jai compiler, they could be questions such as:

• Can we make a procedure's return type randomly change at compile-time?
• What if a polymorphic procedure is used as a constructor?
• How do we prevent certain structs from using a procedure as a constructor?
• Can we generate bytecode for a slightly different procedure because of the struct that is using it as a constructor?

Note: We tend to use the term procedure instead of function.

Don't worry if the questions seem bizarre (the language is new and hasn't been released yet). The common theme is they all require very different language features to interact with each other (i.e. cross-cutting). For example, the first item could be handled polymorphically with the modify directive, enums, a switch statement, and some random bit generator. The answer to the second item is that nothing bad happens; in fact, this opens the door for multiple structs to use the same polymorphic procedure as their constructor. This observation naturally leads to questions three and four. When I wrote the tests for some of these, compiler bugs were uncovered naturally and without much effort.

Note how these questions are answered by just writing programs. They are the language tests. Anything not helping me towards that feels like noise, and that explains why I haven't been happy with many testing systems. Here's the criteria I want a framework to meet to feel good about writing tests:

1. Ability to write tests as if they were real, independent programs.
2. Simple asserts the tests can use.
3. By default, virtually zero interaction with the test framework.
4. Minimal test suite reports that are nice to look at.

By a minimal report I mean showing me which programs ran and when, how many test asserts were called and where to find them if they failed. The report should look good so I can help my brain enjoy the overall experience.

Our Framework

To showcase the framework we ended up creating, let's go through an example, transforming a program into a language test.

Here's the program that tackles one of the simpler inquiries from earlier: What if a polymorphic procedure is used as a constructor? While we're at it we'll ask the same of destructors.

#import "Basic";

Thing :: struct {
    mem : *u8;
    value := 42;
    #constructor init_thing;
    #destructor free_thing;
}

init_thing :: (using thing : *$T) {
    mem = alloc(1000);
}

free_thing :: (using thing: *$T) {
    free(mem);
    mem = null;
    print("Thing memory freed.\n");
}

main :: () {
    {
        our_thing : Thing; // Constructor fires off.
        print("%\n", our_thing);
    } // Destructor fires off.
}

If we compile it, we see that it's valid:

 >>> jai poly_constructor.jai
 >>> poly_constructor.exe
 Thing { value = 42; mem = 1e4e6b4f790; }
 Thing memory freed.
 >>>

As I had mentioned, this means multiple structs can use the procedure as their respective constructors.

 #import "Basic";

Thing :: struct {
    mem : *u8;
    value := 42;
    #constructor init_thing;
    #destructor free_thing;
}

AnotherThing :: struct {
    mem : *u8;
    message := "I don't hold the meaning of life.";
    #constructor init_thing;
    #destructor free_thing;
}

init_thing :: (using thing : *$T) {
    mem = alloc(1000);
}

free_thing :: (using thing: *$T) {
    free(mem);
    mem = null;
    print("Thing memory freed.\n");
}

main :: () {
    //
    // Test with multiple structs.
    //
    {
        our_thing : Thing;
        print("%\n", our_thing);

        different_thing : AnotherThing;
        print("%\n", different_thing);
    }
}

Output:

 >>> jai poly_constructor.jai
 >>> poly_constructor.exe
 Thing { mem = 1d7df5bf790; value = 42; }
 AnotherThing { mem = 1d7df5c0190; message = "I don't hold the meaning of life."; }
 Thing memory freed.
 Thing memory freed.
 >>>

Great! And remember: this is just a program. We can explore the possibility space even more, define any number of procedures, declare hundreds of complex structs, import 3rd-party modules—you name it. However, let's pretend we're done answering our polymorphic constructor questions. To turn this into a language test, let's make two straightforward modifications (we'll add a couple of asserts while we're at it):

polymorphic_constructor :: () {
    //
    // Test multiple things.
    //
    {
        our_thing : Thing; // Constructor fires off.
        assert(our_thing.mem, "Thing didn't get an memory address.");

        different_thing : AnotherThing;
        assert(different_thing.mem, "AnotherThing didn't get an memory address.");
     }
 } @TestProcedure

Notice I changed the name from main to something else and added the simple note TestProcedure.
Notes are essentially comments recognized by the compiler that do nothing by default.

Let's put that file in a tests/ folder.

 >>> mv poly_constructor.jai tests/
 >>> ls
 tests/
 tests.jai
 >>>

Ah, the reader will notice I have a file conveniently called tests.jai. Let's compile it.

 >>> jai tests.jai
 >>> ls
 tests/
 tests.exe
 tests.jai
 >>>

That produced tests.exe; let's run it! We'll use some real screenshots this time.

And we just made a full test suite run.

Note: The reader might ask why we have 82 more asserts than what we wrote, plus another test procedure
from a String file. More on that soon.

Notice a log file was generated. Here's the summary portion of that report:

Report format is based off Aras Pranckevicius' API doc

That's a lot of useful output just by compiling a source file! In addition, we can have multiple test procedures on the same test file:

    polymorphic_constructor :: () {
        //
        // Test multiple things.
        //
        {
            our_thing : Thing; // Constructor fires off.
            assert(our_thing.mem != null, "Thing points to null.");

            different_thing : AnotherThing;
            assert(different_thing.mem != null, "AnotherThing points to null.");
        }
    } @TestProcedure

    polymorphic_constructor_uninit :: () {
        //
        // Control when the initializer and constructors
        // actually get called.
        //
        {
            our_thing : Thing = ---; // Uninitialized var.
            memset(*our_thing, 0, size_of(Thing));

            T :: type_of(our_thing);
            tis := cast(*Type_Info_Struct) type_info(Thing);
            initializer := cast((*T) -> void) tis.initializer;
            constructor := cast((*T) -> void) tis.constructor;

            initializer(*our_thing);

            assert(our_thing.value == 42, "Expected % but got %\n", 42, our_thing.value);
            assert(our_thing.mem == null, "Expected % but got %\n", null, our_thing.mem);

            constructor(*our_thing);

            assert(our_thing.value == 42, "Expected % but got %\n", 42, our_thing.value);
            assert(our_thing.mem != null, "Didn't want % but got % anyway", null, our_thing.mem);

            //
            // We can do the same for AnotherThing.
            //

            // ...
        }
} @TestProcedure

Opening the report we get:

If we wanted to, we could drop in more test files in the tests/ folder, each containing their own sub-programs. Finally, if invariances in our tests don't hold (i.e. an assert fails), that will also be reported. Let's add a deliberate failure on the first test procedure:

    polymorphic_constructor :: () {
        //
        // Test multiple things.
        //
        {
            our_thing : Thing; // Constructor fires off.
            assert(our_thing.mem != null, "Thing points to null.");

            //
            // Deliberate failure.
            //
            assert(our_thing.value == 999, "Meaning of life isn't 999? It's %", our_thing.value);

             different_thing : AnotherThing;
             assert(different_thing.mem != null, "AnotherThing points to null.");
         }
} @TestProcedure

And we get that on the log file too:

If the test writer heads to the test files section, the procedure that failed would be made apparent.

Library Tests

So how do we explain that weird String file from the reports earlier? We didn't write that! Well, Ignacio found it useful to use this framework for the language's system modules. System modules are special .jai files we provide programmers. Users can import them to leverage OpenGL / D3D, font rendering, audio playback, and more.

The test file from earlier imported Basic (a catch-all module we use to experiment new features). Basic
happens to indirectly import the String module. When I opened up String, I noticed Ignacio had written this somewhere on the file:

string_tests :: () {

    assert(join(..{:string: "foo", "bar", "puf"}, ", ") == "foo, bar, puf");
    assert(join("foo", "bar", "puf", separator="/") == "foo/bar/puf");

    assert("foo/bar/puf" == join(..split("foo/bar/puf", "/"), "/"));
    assert("foo\\bar\\puf" == join(..split("foo/bar/puf", "/"), "\\"));
    assert("foo/bar/puf/grr" == join(..split("foo, bar, puf, grr", ", "), "/"));

    // More asserts
    // ...
} @TestProcedure

Which accounted for the 82 additional asserts. That's because the framework detects tests written in external modules, outside of our own test files. Now anyone on the team writing modules can write test procedures inside their module, tag them, and they're done. In fact, I have a file under tests/ that just imports all the modules. The actual tests can live right next to the module code, outside of our folder, and we register them for free just as we automatically register the test files inside the folder for free. And fret not, a module being imported multiple times will still have its test recognized only once.

Future Updates

I'd like to note there are special settings and configurations the user doesn't know about (good! that's the point!), and those who care can open up the build file, which is just the tests.jai source file. Documentation amounts to a few well-placed comments.

Upon release (if we have time to make it happen), I'd like people to write their numerous test procedures, compile tests.jai, and have them witness Jon's bananas.jai but in the context of tests: A visualization of a growing tree of test files as branches, where each test procedure is a leaf from the branch.

How It All Works

A common theme back at Kennedy Space Center was that we wanted to automate most of the "bookkeeping" so the engineer's time and effort went into thinking about the real problem. Interacting with build systems was not in our best interest! Even so, sometimes we were stuck with what I think were high-friction tools. I can't properly express to the reader how great it's been to write a tool that reduces friction nontrivially.

Let's take a quick look at how the language played a part in letting me create this framework, as I think it has the right kind of features all statically-typed languages should have going forward.

Minimizing Framework Interaction

Initially, the main struggles in making the testing system work were psychological, working to unlearn years of how I was told automated test frameworks should be structured. My first attempt was writing a hybrid C++/Jai program with wacky command-line parameters. I think Jon nearly had a heart attack. It was difficult to implement something that met the criteria listed at the beginning of this post, because my model was tied to old-school software testing models.

The closest implementation I have ever found is CxxTest. What caught my interest was the second sentence on their homepage:

CxxTest is easy to use because it does not require precompiling a CxxTest testing library, it employs no advanced features of C++ (e.g. RTTI) and it supports a very flexible form of test discovery.

Aha! It's trying to make itself invisible, and the term test discovery sounds like it's doing work on behalf of the user. Indeed, later the website goes on to say:

Additionally, CxxTest supports test discovery. Tests are defined in C++ header files, which are parsed by CxxTest to automatically generate a test runner. Thus, CxxTest is somewhat easier to use than alternative C++ testing frameworks, since you do not need to register tests.

Aha por dos! Test discovery is the property I want frameworks to exhibit. With CxxTest, though, we still need to understand its build system steps to reap the benefits, but that's a difficult limitation to overcome with C++. In jai, however, we can specify our build system in the language.

So I made a source file, tests.jai. In said file, we can ask the compiler to compile any number of different files, and request it to open up its compilation stages to us as things get lexically scanned, parsed, and code generated. That's more than we care to ask for, so our entire build system can amount to compiling a single file, in the same way we compile any normal code.

Tagging Procedures

As I just said, in tests.jai we can ask the compiler to pause at any compilation stage for us to intervene in the process. So I used that for the test discovery. The current mechanism, as we have seen, is to tag procedures that want to behave like an independent test program with the note @TestProcedure. Then, once the compiler has finished type-checking all source files, I ask it to give me the declarations tagged with the note. Specifically:

for decl.notes {
    if it.text == {
        case "TestProcedure";
        proc_info := cast(*Type_Info_Procedure) type;
        if proc_info.argument_types.count > 0 {
            print("[Test Suite] WARNING: '%' tagged with @TestProcedure should take no arguments.\n", proc.name);
            continue;
        }
        user_test : TestProcedure;
        user_test.filename = decl.filename;
        user_test.procname = proc.name;
        array_add(*tests_to_register, user_test);

        //...
        case;
        // Do nothing.
    }
}

Where decl is the current code declaration we're inspecting from the AST.

In the past, I used to check for procedure names starting with test, but Allen and Nico made me question whether that was too constraining. Turns out it was. The downside with the notes here, though, is there might be tests importing external modules that tagged some of their code with @TestProcedure for unrelated reasons. I mitigate that by ignoring structs with that tag, and by verifying whether the procedure acts as an entry point (i.e. main with no parameters). There are other options available to us, but for now this is a good-enough solution.

Hidden Test Suite

I register the discovered tests by creating register_test procedure calls on-the-fly--meaning I update the program's AST before things enter the bytecode pipeline. Where is the register_test procedure defined? Well, the dirty secret is I still wrote an actual test suite, contained in a single file living inside the tests/ folder. The build file has access to that, as do people. If anyone wishes to avoid all the automatic magic and also believe editing the build file is not enough, they are free to ignore it altogether and figure out how the suite works. They can register tests manually, generate reports however they see fit, and perform the entire job of a framework themselves.

Test Asserts

As we can see from the CxxTest homepage, it has different types of asserts for us to use. I had something similar going, but Ignacio reminded me that Jon added an assertion handler as part of the implicit context. That means we can override the behavior of regular asserts! This is what allows external modules to write officially-recognized tests inside their file and still compile anywhere--we're not requiring them to do anything special.

What's Next

I now enjoy writing tests whenever I need to, and I hope this straightforward style sounds appealing to some people.

In other news, for the purpose of testing a compiler, will an automated test suite suffice? Not really, though it's a necessary first step. Although we've uncovered dozens upon dozens of important compiler bugs, we're starting to hit the ceiling of what human-made tests can offer. It's no surprise we've started to seriously look into a different beast entirely—a compiler fuzzer.

But that's a story for another blog post!

P.S. Shoutout to Josh, who among other things, helped write a bunch of the language tests and makes sure everything keeps running smoothly on Mac and Linux.

We've been revising the modeling and texturing for areas near the end of the game. There are some nice treats in store!

We've been working on some of the "special effects" in the game, most notably particle effects. Our particle system was old and slow, so we are speeding this up, and then we will make some modifications to help make particle authoring an easier process.

We've been making some baseline functionality smoother. For example, so that you don't need to think about saving your game, The Witness autosaves your progress every minute or so. On a high-end PC, you don't notice the autosave, but on a low-end PC or on a console (because consoles tend to have different memory access properties than PCs), scanning the world for stuff that needs to be saved can be pretty slow, like, it could take 1/10 of a second, which would result in a hitch in the frame rate happening every minute. So we are hammering on this to make it faster, which should also result in saved game files being smaller, since we'll be excluding a lot of things that don't need to be saved.

The D3D shader compiler provides a backward compatibility flag (D3DCOMPILE_ENABLE_BACKWARDS_COMPATIBILITY) that I was hoping would allow us to use our shaders 'as is'. Unfortunately, it seems the backward compatibility mode has not received much love and things were not that simple. I run into several compiler internal errors, and even after finding workarounds most of our shaders did not actually compile, but would have required significant modifications, the resulting vertex and pixel shaders would often not link at runtime, and the use of register annotations on samplers only allowed us to control the layout of the samplers, but not of the associated textures.

For other targets (Cg & PSSL) we had been able to get away with some preprocessor macros and simple text transformations, but it seemed that was not going to be an option in this case. Not only that, having to write and adapt our shader code for the idiosyncrasies of each compiler was starting to become a challenge.

Instead, I thought it would be much cleaner and much more powerful to standardize on a fixed subset of HLSL and transform it to the shading language of the target by building a syntax tree, applying syntactic transformations, and generating shader code by traversing the resulting syntax tree.

This is essentially what we already do in the OpenGL backend to transform GLSL into HLSL. For that we use HLSL2GLSL and GLSLOptimizer. I considered using these same projects, either modifying HLSL2GLSL to output HLSL1x code instead of GLSL, or modifying GLSLOptimizer to accept and produce HLSL code. I avoided the former, because HLSL2GLSL's code was a big mess and it was not very attractive to work with it. The latter was more interesting, and I believe it's the approach taken by the UE4 engine. However, while much cleaner, GLSLOptimizer is still a fairly large project and I did not care too much about the optimization aspects that it offered.

Another option that I had just come across is Max McGuire's HLSLParser. HLSLParser was a lot smaller and cleaner than either of those code bases. It was pretty much the code I would have written if I were to do this from scratch, so it seemed like the best possible starting point.

One of the main annoyances when working in the OpenGL backend is that GLSL errors are extremely hard to track down. They can be reported by either HLSL2GLSL, GLSLOptimizer, or the driver's GLSL compiler. Each of the inputs to these compilers has different file and line information and none of them matches the original source code. Whenever an error occurs we have to dump the intermediate code and figure out from there what's going on. It's usually a waste of time and if D3D11 is to become our primary backend I did not want to have to deal with this issue.

I was happy to see that HLSLParser handled this correctly associating file and line info to the tree nodes and producing correct #file preprocessor statements in the generated output.

In order to parse our shaders with HLSLParser we had to extend it in several ways, here's a partial list of the features I added:

• Support for 3D, 2DMS and shadow samplers, and most of the associated intrinsics.
• Added support for many missing intrinsics.
• out argument modifier.
• Default argument values.
• Block statements.
• Attribute annotations.
• Static, inline, uniform and other type modifiers.
• Bitwise operators.
• Hex literals.
• Full declaration syntax, including multiple declarations in the statement.
• Add support for full declarations in buffer fields.
• Effect syntax (techniques, passes, sampler states, state assignments).

I'm very thankful for the work Max McGuire did on HLSLParser and the least I can do is to release these our improvements to the public as well. You can find our fork of the project in our github repository:

https://github.com/Thekla/hlslparser

Once we have the syntax tree in memory we can do many cool code transformations. In particular, we do:

• Dead code elimination.
• Determine used parameters and resources.
• Assign registers/offsets explicitly to control layout of parameters and resources.
• Sort resource registers and group parameters in buffers based on frequency of change.
• Rename semantics.
• Reorder pixel shader arguments so that vertex shader outputs match pixel shader inputs and avoid linkage errors.

The shaders that result from these transformations are minimal, that is, they only contain the definitions and declarations that are necessary for a specific entry point. We have complete control of the layout of resources and parameters and extract all the necessary information about the shader from the tree instead of relying on the platform-specific reflection APIs. The shader transformation is very fast and the resulting shaders compile faster than the original ones. Our new D3D11 shader processor and compiler is about 4 times faster than its D3D9 counterpart.

While it currently accepts all of our shaders, HLSLParser still has some limitations. Here are a few that come to mind:

• No type inference for sampler types. It assumes sampler == sampler2D.
• No support for {} initializers. Instead of float2 p = {0, 0} you have to write float2 p = float2(0, 0). This particular case is not complicated, but it does get more hairy when initializing structures. For now, I've simply modified our shaders to avoid this.
• No support for rectangular matrices. We did not support them in the GLSL translators either, so we avoided their use in our shaders. It would be nice to add this eventually.
• No support for do, while or switch statements. These should be easy to add, but we just didn't need them!
• No proper error recovery. We stop reporting errors after first one. This would be nice, but is not critical.
• Semantic analysis is incomplete. The parser does some, but not a complete semantic check. The generated shader is not guaranteed to be correct in the absence of errors, but since we have proper file/line annotations, this usually doesn't really matter.

Looking forward, I expect we will add code generators to target other platforms. It should be really easy to bring the existing GLSLGenerator up to date, or add new generators for PSSL and Metal.

I'm not particularly fond of our shader code. On one side, we abuse the preprocessor to generate shader variations and the result is often a mess that is hard to read and modify. On the other, our code is often highly repetitive, but there doesn't seem to be a clean way to share it across different materials or structure it in a way that's more compact. I'm particularly interested in experimenting with higher level code transformations, or coming up with new constructs to facilitate writing and sharing shader code.

Ignacio has posted the conclusion of his series on irradiance caching -- at long last! -- and released the source code.

This code is a significant chunk of what makes lighting in The Witness as nice as it is.

Here's the blog post.

Here's the video:

Update: And here's another post, complete with a video showing in detail how the grass planting works in the actual game.

Working on The Witness, Part 5

Working on The Witness, Part 6

In this one he talks about how to distribute grass. Legally. As well as the different colors of noise one might want to be familiar with in order to do this well.

I'm not going to try to address that here, but just as I was writting this Matt Pettineo released a good overview of some of the different techniques available that is a good starting point. Instead, I'll describe some of the problems that I faced when implementing and optimizing our shadow mapping implementation, some of the techniques that I tried and the solutions that worked best for us in the end.

While there are many techniques that I haven't tried and I'm sure our solution could be improved further, for this game I'm pretty comfortable with our implementation in terms of quality and performance.

Jon wrote our initial shadow mapping system a few years ago and with the exception of some small improvements I made early on, it had remained pretty much unchanged until very recently, when it became obvious that shadow map rendering and sampling was too slow and needed to be improved drastically.

The posts where Jon describes our old shadow mapping system are here and here, but to summarize, we were using stable cascade shadow maps, with 4 partitions, distance based selection, and some innovative tricks to reduce unused shadow map space. For filtering we used randomized PCF sampling and a combination of gradient, slope and constant biases to prevent acne artifacts.

Filtering

One of the first thing I changed in the original implementation was the filtering method. Initially we used a Poisson distribution of PCF samples that was rotated randomly, similar to what is used in the Crysis engines. This approach has many advantages: it scales well reducing the number of samples, the kernel radius can be adjusted easily, but it often resulted in noise artifacts.

In many games, these artifacts would not be very noticeable in the presence of detailed textures and normal maps, but in The Witness the artifacts really stood off in contrast with our stylized geometry and smooth precomputed lighting.

We experimented with higher number of samples, but in our case the number of samples required to obtain smooth results defeated the purpose of using a sparse distribution. We were able to obtain much higher quality results with a compact box filter such as the one described by Bunnell and Pellacini in Shadow Map Antialiasing. The box filter produced smooth results, but didn't do a good job smoothing out the stair stepping artifacts resulting from the limited resolution of the shadow maps. A straightforward extension of their method that looked considerably better is to weight each of the samples using a Gaussian curve.

The naive implementation of this filter uses (N-1)x(N-1) PCF texture samples. In "Fast Conventional Shadow Filtering", Holger Grün shows how to reduce the box filter to (N/2)x(N/2) samples, but argues that separable filters require (N-1)x(N/2), and proposes to use Gather4 in order to reduce the number of samples down to (N/2)x(N/2). I think Grün was trying too hard to sell the Gather4 feature and didn't fully explore the bilinear PCF sampling approach. His approach to bilinear PCF sampling is too complicated and does not result in the lowest possible number of samples. Here I'll show how to compute the filter weights in a more efficient and much more simple way.

Let's first imagine a symmetric horizontal kernel q with 5 taps:

  q = {0, a, b, c, b, a, 0}

Each of these kernel weights corresponds to a bilinear PCF sample, that is, the actual footprint of the kernel is 6 shadow map texels and each of the kernel weights is a linear combination of our original kernel:

  k[x_] := s*q[[x]] + (1 - s)*q[[x + 1]]

where x is the integer texel coordinate and s is the fractional part.

The traditional way to apply this kernel is by simply taking 5 bilinear samples and weighting each one by the weights in q. However, you could achieve the same result taking 6 point samples and weighting each of them by the factors in k.

The important observation is that we can reduce the number of samples to a half (just 3 in this case) by spacing out the PCF samples so that their footprints do not overlap and adjusting the subtexel location of the samples and the corresponding weighting factors. That is, for every sample we want to find a new subpixel coordinate 'u' and a new weight 'w' that meets these constrains:

  eq[1, x_] := (k[x] == (1 - u) w)
  eq[2, x_] := (k[x + 1] == u*w)

Solving this equation system in Mathematica gives us the following result for each of the 3 samples:

  Simplify[Solve[{eq[1, 1], eq[2, 1]}, {u, w}]]

    u -> (b + a s - b s)/(a + b - b s)
    w -> a + b - b s

  Simplify[Solve[{eq[1, 3], eq[2, 3]}, {u, w}]]

    u -> (b - b s + c s)/(b + c)
    w -> b + c

  Simplify[Solve[{eq[1, 5], eq[2, 5]}, {u, w}]]

    u -> (a s)/(a + b s)
    w -> a + b s

This may be equivalent to what Grün does to half the number of samples in each of the rows in the kernel, but contrary to his method, this can be extended to separable 2D kernels.

Let's say we have a separable 5x5 filter like this:

  q = {{0, 0, 0, 0, 0, 0, 0},
    {0, a*a, a*b, a*c, a*b, a*a, 0},
    {0, b*a, b*b, b*c, b*b, b*a, 0},
    {0, c*a, c*b, c*c, c*b, c*a, 0},
    {0, b*a, b*b, b*c, b*b, b*a, 0},
    {0, a*a, a*b, a*c, a*b, a*a, 0},
    {0, 0, 0, 0, 0, 0, 0}}

In this case, the filter touches a 6x6 range of texels and each of the weights is a bilinear combination of the 4 nearest factors:

  k[x_, y_] := s*t*q[[x, y]] +
    (1 - s)*t*q[[x + 1, y]] +
    s*(1 - t)*q[[x, y + 1]] +
    (1 - s)*(1 - t)*q[[x + 1, y + 1]]

To find the new subpixel coordinates 'u' and 'v' and the corresponding weight 'w' we have the following equations:

  eq[1, x_, y_] := (k[x, y] == (1 - u) (1 - v) w)
  eq[2, x_, y_] := (k[x + 1, y] == u (1 - v)*w)
  eq[3, x_, y_] := (k[x, y + 1] == (1 - u) v*w)
  eq[4, x_, y_] := (k[x + 1, y + 1] == u*v*w)

Solving this equation system for the left-top sample gives us the following result:

Solve[{eq[1, 1, 1], eq[2, 1, 1], eq[3, 1, 1], eq[4, 1, 1]}, {u, v, w}]

    u -> (b + a s - b s)/(a + b - b s) 
    v -> (b + a t - b t)/(a + b - b t) 
    w -> (a + b - b s) (a + b - b t)

We can repeat this for the remaining 8 PCF samples to compute the corresponding uv coordinates and weights. Looking at the results we can notice some common terms that appear due to the symmetries in the kernel. Once we factor the common sub-expressions and replace the a,b,c constants with scalars we can simplify the formulas and end up with fairly simple expressions. For example, in our case, in the 5x5 kernel we set the constants to 1, 3 and 4 to achieve a crude approximation to a Gaussian filter and this is the resulting code:

    float uw0 = (4 - 3 * s);
    float uw1 = 7;
    float uw2 = (1 + 3 * s);

    float u0 = (3 - 2 * s) / uw0 - 2;
    float u1 = (3 + s) / uw1;
    float u2 = s / uw2 + 2;

    float vw0 = (4 - 3 * t);
    float vw1 = 7;
    float vw2 = (1 + 3 * t);

    float v0 = (3 - 2 * t) / vw0 - 2;
    float v1 = (3 + t) / vw1;
    float v2 = t / vw2 + 2;

    float sum = 0;

    sum += uw0 * vw0 * do_pcf_sample(base_uv, u0, v0, z, dz_duv);
    sum += uw1 * vw0 * do_pcf_sample(base_uv, u1, v0, z, dz_duv);
    sum += uw2 * vw0 * do_pcf_sample(base_uv, u2, v0, z, dz_duv);

    sum += uw0 * vw1 * do_pcf_sample(base_uv, u0, v1, z, dz_duv);
    sum += uw1 * vw1 * do_pcf_sample(base_uv, u1, v1, z, dz_duv);
    sum += uw2 * vw1 * do_pcf_sample(base_uv, u2, v1, z, dz_duv);

    sum += uw0 * vw2 * do_pcf_sample(base_uv, u0, v2, z, dz_duv);
    sum += uw1 * vw2 * do_pcf_sample(base_uv, u1, v2, z, dz_duv);
    sum += uw2 * vw2 * do_pcf_sample(base_uv, u2, v2, z, dz_duv);

    sum *= 1.0f / 144;

The main advantage of the PCF separable sampling approach over Gather4 is not just that it works in a wider range of hardware, but that is also more efficient. With PCF sampling we need to compute the new sampling coordinates and weights, while with Gather4 we have to perform depth comparisons and filtering in the shader. The latter uses a lower number of instructions, but consumes more bandwidth between the ALUs and texture units and more importantly, it requires more registers to hold the results of the texture samples. That said, despite the lower performance, the use of Gather4 has some advantages that we will discuss in the next section.

For those that simply want to use our shadow sampling code without deriving formulas for their own weights, you can find our implementation integrated in Matt's shadow mapping example under the "OptimizedPCF" option. Running his demo in the two GPUs that I have available, a NVIDIA GTX 480 and an Intel HD 4000, I've measured that our technique is between 10 and 20% faster than the Gather4 method.

Acne

Shadow acne is one of the major issues of most shadow mapping implementations. Eliminating shadow acne is such a pain, that I wonder why it's not received much more attention.

In our case I ended up employing what I believe is the most common solution: A combination of constant, slope-scale, normal and receiver plane depth biases. Finding the right magnitudes for each of these offsets is complicated. If the offsets are too big, that results in peter panning or light leaking artifacts, if the offsets are too small, they don't eliminate acne entirely.

In Eliminating Surface Acne with Gradient Shadow Mapping Christian Schuler proposes the use of linearly filtered depth values and fuzzy depth comparisons to mitigate the acne artifacts. However, these techniques are not possible when using PCF sampling.

Another alternative is to render backfaces to the shadow maps, but this approach is more prone to light leaking and does not really solve the acne problems on thin walls. Dual or midpoint shadow maps are interesting, but it seems like they would be more expensive, do not entirely solve the problem near silhouettes or with very thin objects, and impose additional modeling constrains.

The magnitude of the offsets is proportional to the size of the texels and the filtering radius, so a simple way of reducing them is using higher resolution shadows, but larger shadow maps also reduce performance. In the end we just try to find the best compromise between the two forms of artifacts and as a last resort constrain the modelling to prevent the most egregious artifacts.

One of the unfortunate disadvantages of our shadow filtering method is that we rely on the hardware texture units to sample 4 shadow texels simultaneously and perform the depth comparison. When using receiver plane depth bias we would like to use a different bias for each of the texels, but there's no way we can provide a depth gradient to the texture unit. As a result, the receiver plane depth bias is not very effective with our method. In fact, it's so ineffective, that we chose to disable it entirely and compensate with a larger normal and slope-scale bias.

For these biases to work well it's important to scale their magnitude correctly based on the surface normal and light direction. In particular, the slope-scale bias should be proportional to the tangent of the angle between the normal (N) and the light direction (L), and the normal bias should be proportional to the sine:

float2 get_shadow_offsets(float3 N, float3 L) {
    float cos_alpha = saturate(dot(N, L));
    float offset_scale_N = sqrt(1 - cos_alpha*cos_alpha); // sin(acos(L·N))
    float offset_scale_L = offset_scale_N / cos_alpha;    // tan(acos(L·N))
    return float2(offset_scale_N, min(2, offset_scale_L));
}

Finally, one of the disadvantages of the normal bias is that it does not only bias the depth, it also offsets the shadow. On its own this offset is not very noticeable, but the offset amount is different for each of the cascades and that results in an obvious shift between cascades. Blending the cascades helps, but to reduce the problem further we also interpolate the magnitude of the bias from one cascade to the next, so that the shift is barely noticeable.

Coming Next

In part 2, I'll describe our cascade placement algorithm and finish with the optimizations that we use in order to speed up shadow map rendering.

Despite not even being finished, The Witness is already one of my all-time favorite games. I’ve been playing it since Jon started developing it several years ago, and I could not be more excited about its upcoming release.

Unlike Jon’s previous game Braid, the asset and programming scale for The Witness is much further toward the “triple-A” side of the scale than the “indie game” side. As anyone who works on these types of projects knows, the work required to successfully pull off a game goes up dramatically when you go that direction. The Witness already has several more people working on it than Braid did at its peak, but as with any project of this size, there are still many parts of the project that need more attention than the principals can afford to give them.

For this reason, it was always my intention to try to find some time to help out on The Witness when it came time to ship it. So this past Thanksgiving, Jon and I sat down and he went through a list of things in the codebase that he thought would benefit from the attention of an extra programmer. After considering the relative importance of the items, we decided that the player movement code was the place where improvements could have the biggest impact on the player’s experience.

Walkmonster in Wall

In the context of The Witness, the goal of player movement code is to be as unobtrusive as possible. The player is supposed to fully immerse themselves in an alternate reality, and every last detail is important to this experience. The last thing you ever want to happen is to have the player notice the fact that they’re sitting at a computer asking it to move a virtual viewpoint around.

For that reason, the player movement code has to be rock solid. If you can get stuck on edges, trapped in walls, fall through the floor, walk through a cliff, walk down a hill but not back up it, etc., it immediately breaks the immersion and reminds you that you are playing an artificial experience mitigated by an unreliable system. It can also lead to disastrous consequences for the player in some circumstances if there’s no way for them to recover from the problem with restarting or reloading a (possibly very old) save game. If you play games often, you’ve probably encountered one or more of these problems in shipping titles, and you know what I’m talking about.

Last week I began to work on this problem. I decided the first thing to do was to write a few integrated tools for working with the player movement code so we could analyze it and see how it was currently behaving. Immediately, when I opened up the project, I found myself with a familiar conundrum: what to name my first source file. This is always the most important part of any project (as Bob Pollard once said of band names and album names). If you have the right name for your source file, it’s smooth sailing from there. Pick the wrong name, and you might as well call the project off.

But what to call a system for bulletproofing player movement code? I’d never been asked to write code for that before. When I thought about it, I realized that I’d only personally seen evidence of similar code once before: when playing an early beta of Quake. There were some bugs with the monster placement, and in the console window, you could see error messages announcing the fact that the monsters, instead of spawning on the ground, had spawned partially intersecting with the level geometry. Each debug message began with the phrase “walkmonster in wall at. . .”

Bingo. It’s hard to ask for a better source file name than “walk_monster.cpp”. I was pretty sure, from that point forward, the code was going to come along nicely.

Drive Toward Point

When you’re trying to test a system, the most important thing to do is actually test the system. Even though that sounds like a straightforward rule, people writing tests often fail to observe it.

In this specific case, there are very easy ways to think that you were testing the player movement code without actually testing it. One example would be to do some kind of analysis on the collision volumes and walkable surfaces in the game, looking for small areas, gaps, etc. Once you’d eliminated all of these, you’d then proclaim the world safe to traverse and move on.

But this is testing the data, not the actual code. It’s still easily possible to have bugs in the movement code that result in bad behavior even with sanitized data.

To avoid this kind of trap, I wanted the testing system to remain as close as possible to what a human does to actually control player movement in the game. I started out by writing two routines to serve as the building blocks of this kind of testing.

The first is the closest to real human usage. It’s an update call that hooks into The Witness’s input processing and feeds it synthetic mouse and keyboard events. It can do some basic things that humans might do, such as looking around, walking toward a point, looking toward a point, and so on. It does these through nothing but emulation of human interaction with the mouse and keyboard, so I know that everything in The Witness’s input path will be running as it actually would during testing. I’ll talk more about this system and how it’s used in later installments.

The second piece of code is one step removed from that level. It’s a function called DriveTowardPoint that takes two points in the world and, calling the existing player collision system, attempts to move cleanly from one to the other. When it returns, it provides information about the attempt such as obstacles encountered and whether or not the destination was successfully reached.

This function is not quite as reliable as the synthetic input method of testing, because it eliminates a portion of the actual player movement system from testing. For example, any erroneous state associated with where the player is that the collision system might have built up won’t affect tests using this function. Nevertheless, I felt it valuable to have this level of testing available because it can much more rapidly test large areas, since it does not require the entire game loop to run, and thus can be employed much more frequently and over the entire world instead of just isolated test runs.

It’s also worth noting that there are no physics inputs to this function; there are no velocities specified for the starting point, for example. This is because The Witness is not an action game, so there are few stateful physical properties for the player. Players can’t jump, they can’t wall-run, they can’t enter bullet-time. Supporting these types of behaviors is possible with systems of the kind I’ll be discussing, but they add a layer of complexity that we don’t have to worry about on this project.

Anyhow, with DriveTowardPoint in place, it was possible for me to start on my first goal for the system: determining everywhere the player can go on the island of The Witness.

Rapidly Exploring Random Trees

Where can the player go? It seems like a simple question, but you’d be surprised how many games ship without the development team knowing the real answer. If possible, I want The Witness to be one of the few games where the team knows prior to shipping exactly where the player can and can’t go, no surprises.

This makes the problem statement, but perhaps not the problem, very simple: given a DriveTowardPoint function that faithfully determines whether a player could move in a straight line between two points, produce a coverage map showing everywhere the player could end up.

For reasons that I can’t necessarily explain, I had in my head, even before writing a line of code, that the best way to do this would be to use a Rapidly Exploring Random Tree. For those of you unfamiliar with this algorithm, it’s a very simple process whereby you record all the points you’ve visited along with a reference to the point from which you walked there. To add points to the tree, you pick a random target point anywhere in the world, select the closest point in your tree thus far, and try to get from that tree point to the target point. Wherever you end up, that’s a new sample point.

Normally, this is more of a pathfinding algorithm, so interleaved with random points, you repeatedly select some goal point as a target. This biases the exploration of the space toward the target, which is what you want when your only aim is to reach the goal. But in this case, I wanted to produce a complete map of everywhere the player could go, so I was exclusively using random samples.

After implementing this (it is thankfully a very simple algorithm to write, so it doesn’t take long), I could see that it did do a reasonable job of exploring the space (the white etchings show paths that have been explored, and the red vertical lines show places where an obstacle was hit):

However, once I actually looked at how it behaved, it was clear to me that it wasn’t really the algorithm I wanted. For example, even after many iterations, it often barely explores rooms like this one, despite densely covering the area just outside, simply because it doesn’t pick enough random targets inside it:

If I’d thought about it before I started, I would have realized that the benefit of something like a Rapidly Exploring Random Tree is that it explores high-dimensional spaces efficiently. In fact that’s the primary reason you normally use it. But in The Witness, we don’t have a high-dimensional space. We have a two-dimensional space (distributed on an intricate manifold, yes, but a two-dimensional space nonetheless).

In this low-dimensional space, the benefits of a Rapidly Exploring Random Tree are muted, and the drawback is critically bad for my stated purpose: it’s designed to most efficiently find ways to connect pairs of points in a space, not to efficiently find all reachable points in that space. If you care about the latter, then Rapidly Exploring Random Trees will actually take an excruciatingly long time to do so.

So, I quickly realized that what I should be looking for is an algorithm that efficiently covers low-dimensional spaces in their entirety.

3D Flood Filling

Once I actually thought about the choice of algorithm, it seemed obvious that what I actually wanted was something like the old 2D flood fills we used to use to fill regions of a bitmap. Given a starting point, I just wanted to fill up the entire space, exhaustively probing every possible way you could go. Unfortunately, this is a lot less straightforward in the world of The Witness than it is in a 2D bitmap, for a number of reasons.

First, there’s no clear concept of finite connectivity around a point. Everything is continuous. It’s not like pixels where you can easily enumerate 4 possible places to go from every given location, and check each one in turn.

Second, there’s no fixed size for a locus in space like there is for a pixel in a bitmap. Walkable surfaces and obstacles can be anywhere, they have no minimum or maximum feature size, and no alignment to any external grid.

Third, although walking around in The Witness locally acts as if it were on a plane, the space itself actually is a deeply interconnected and varying manifold, with walkable regions directly above other regions (stacked perhaps several times over), and special connectivity that changes based on world states (such as doors that open, elevators that go up and down, etc.)

Given these complexities, it is very easy to think you have devised a way to do a flood fill, only to find after you’ve implemented it that it gets bogged down oversampling areas, misses important paths, produces erroneous connectivity information at places where the manifold is complex, or is simply too cumbersome to use because it must be restarted to cope with changes in the world state.

No good solution to all of this immediately came to mind, so I tried some simple experimentation. Using the Rapidly Exploring Random Tree code I had written, I changed the target point selection from random selection to a very controlled selection. Each time a new point was added to the tree, I would say that points a unit step along the cardinal directions from that point where to be considered future targets, just like a basic 2D flood fill.

But of course, this would produce a useless sampling loop if I wasn’t careful. A point would branch out to a neighborhood of 8 points around it, but those 8 points would all then want to try the original point again, and this would go back and forth forever. So in addition to the controlled target selection, I also needed the simple constraint that any target which wasn’t some minimum useful distance away from an existing target point would not be considered. To my surprise, these two rather simple rules did produce a somewhat usable flood fill:

Not bad for just trying the obvious thing. But it suffers from what I’ve taken to calling “boundary echo”. You can see this effect in the following screenshot, taken while the map exploration was still in progress:

In regions with no obstacles, this algorithm proceeds nicely, sampling at a relatively even distance. But once it hits a boundary, the intersections there produce points that are “off grid”, in that they are not aligned along the sampling pattern with which the algorithm filled the adjacent open region. The reason “on grid” points don’t produce overly dense tessellation is because any new point that tries to step back onto one of the previous points finds the previous point there and declines to reconsider it. But when new points are produced on the boundary, they are completely unaligned, so there is nothing to stop them from stepping back into the already explored space. This creates a wave of offset sampling which will continue until it runs into a fortuitous line of points somewhere else that happen to be close enough to its advancing front for it to consider them coincident.

Although this may not seem like a huge problem, it is actually critical. The entire goal of an algorithm like this is to concentrate the samples in the areas where they are most likely to yield productive results. The more time you spend sampling and resampling wide open regions, the less time you spend mapping the actual edges of that region, which is the information you actually wanted. Because this is a continuous space and only an infinite number of samples can truly reveal its exact shape, your ratio of meaningful to meaningless samples is literally the measure of how effective your algorithm is at producing the walkable area.

Now, there is an easy fix for this specific problem: expand the distance at which two points are considered “close enough”. But by doing this to reduce sampling density in places you don’t care about, you lose sampling density in places you do care about, like around boundaries where you are trying to carefully probe for holes.

Localized Directional Sampling

Perhaps because I had started with a Rapidly Exploring Random Tree, my brain had been pigeonholed into thinking about proximity to the exclusion of everything else. The previous algorithms had all been about using proximity to do things, like figure out whether a new point should be considered, or which point to start from when trying to get to a new target point.

But after thinking about the problem for a while, I came to realize is that everything falls more neatly into place if you think in terms of directionality as well as proximity. While obvious in hindsight, if you’ve worked on these sorts of problems before, you know it’s easy to get trapped in one way of thinking for a while and not see the bigger picture, even if it turns out to be a simple one. That is precisely what had happened to me.

Once I shifted perspective, the correct sampling behavior was obvious. Each time I wanted to expand the exploration from a point, I would do a proximity query for a local neighborhood of existing nearby points. Instead of using the distance to those points to guide the exploration, however, I would categorize them by their direction (currently, I just use the eight cardinal directions, but I’d like to experiment with different kernels here).

In any direction where I don’t “see” a point, I walk a predetermined distance and add a point wherever I end up (whether I hit something or not). In any direction where I do see a point, I walk there and make sure I can get to it. If I can, I just add a visible edge, so it’s easy for the user to see that we’re connected. If I can’t, then I do add a new point at the collision, revealing the obstacle boundary.

This sampling method works beautifully. It allows you to very accurately control the sampling with nicely tunable parameters, keeping all the points you want while preventing unnecessary tesselation, which leads to a very fast filling of the space:

Because it searches along directions rather than just using proximity, it is immune to boundary echo, and restricts oversampling to the boundaries where you actually want it:

Furthermore, it’s completely impervious to state changes or complex manifold problems. It deals strictly with points and those points can be anywhere, and you can add new ones at any time. If you’ve already made a map of an area with a door closed, and you open the door, all you have to do is plant a single new exploration point on the other side of the door and say “continue expanding this map”, and it’ll connect itself back up and explore the entire area beyond the door properly.

You can also change any of the core parameters at any time, and everything still works. Want to sample an area more densely? Just set the default spacing lower. You can do this in the middle of a map build, and it will start sampling more densely, with no need to jettison the existing results (which may have taken some time to produce).

Rudimentary Edge Probing

Although the algorithm by default will already sample boundaries more thoroughly, because intersections produce additional points beyond the sampling pattern, it didn’t necessarily probe them as thoroughly as I wanted, because it doesn’t do anything special when it encounters obstacles. It occurred to me that since I know which points were produced from collisions and which were not, whenever two collision points found themselves connected by an edge, I could choose to invoke extra sampling to try to find more boundary points nearby.

I haven’t yet worked on this extensively, but I put in a rudimentary method to test the theory, and it shows promise. For any two collision points connected by an edge, I step to the midpoint of the edge and probe outwards perpendicular to the edge. If I don’t collide with the boundary within a very short distance, I assume that the boundary is more complex, and add a new target there so the search will continue in that area.

Even this simple scheme produces nicely dense sampling along the boundary, without any oversampling of open regions nearby. Here’s a region with several boundaries, but without edge probing:

Now here’s the same region with edge probing:

As pleased as I am with this, I’d be surprised if there weren’t significantly better algorithms for sampling the boundary, and I look forward to trying a few more methods in the future.

Early Victories

Even with little development time, and the resulting rather simplistic code, Walk Monster already produces very usable output that can find real problems in the game. Here are just a few examples of problems I’ve noticed in passing while developing the algorithm:

The slopes on the side of this extended platform should not have been walkable, but they were. This was because the player movement code has a pathologically bad way in which it deals with sloping geometry. I now know that’s in there, and I’ll be sure to fix it when the time comes to bulletproof it.

The Witness is meant to be a contemplative game, but wondering why there seemed to be a rock where there was no rock was not meant to be one of its koans. As you might guess, this was because someone left a collision volume in the game when the geometry it represented had already been removed. This is an easy thing to have happen, and it’s nice to have a tool that trivially reveals these mistakes quickly so people don’t have to worry about it.

These were supposed to be impenetrable rocks, but Walk Monster revealed them to be anything but. What’s worse, Walk Monster revealed that the path was somehow only traversible in one direction (from left to right in this top-down screenshot), which is never supposed to happen. I haven’t actually looked at this problem yet, besides verifying that I could actually do it via player motion (I could). It’ll be interesting to see what’s going on!

Open Questions

It’s encouraging to already have good results upon which we can build. Like I said, if you pick the right name for your source files, it all flows smoothly from there! But all of this work was done in the span of a few days, so it is obviously far from comprehensive and at lot of things are still completely ad hoc. If I have time to develop these systems further, there are a number of questions worth answering.

First, what kinds of post-processing on the data should I do to make it easiest to visualize? Drawing the raw network of points and edges can be very difficult for people to visually untangle, and some sort of better representation might make it possible for people to quickly understand more complex walkable areas at a glance.

Second, how can I improve the sampling patterns around boundaries to ensure that I find the most number of holes? Are there good methods for characterizing the ways in which shapes fall into a lattice, and are there good tessellation schemes that maximize the chances of intersecting and passing through these shapes?

Third, are regular sampling patterns or randomized sampling patterns better for filling spaces? I could easily modify the target point selection criteria to produce randomized patterns, but it’s not clear if this is worthwhile or, if it is, what kind of randomized patterns would be best.

Fourth, what other information would we like to get from our walkable area maps, now that we can build them? For example, it would be a straightforward extension of the existing system to do things like pathfinding or distance mapping, where a user could pick a point and ask for the shortest path between it and some other point, or to see a heat map of the distance between that point and all other points in the map. Are queries like this useful? What other queries might be?

For now, the walkable area visualizations we’re already getting from Walk Monster are more than enough to show us that the player movement code is pretty broken. I had planned to move on to creating a system for overnight testing via the user input method, but it seems clear that we already have enough problems to fix that there’s no need for that yet. So my next order of business, before improving these tools any further, will be to see what I can do to make the player movement code more robust. While I’m at it, I’d also like to see if I can get it an order of magnitude faster or two, because currently, the thing that slows down Walk Monster rather severely is the slow speed of the collision system.

With luck, I’ll report back soon in another installment on how that work is going.

Right now, we save the thumbnail and the game data as separate files, but the names are mostly the same; so if you go into your game data folder in Windows you can see them living there very plainly:

(It might be interesting to do it like Spore and embed the save data in the image file, so that there's only one file per save. Maybe we'll do this, though it's not a no-brainer; right now the image file is usually bigger than the save data, so this would involve tying a relatively small amount of crucial data to a relatively large amount of cosmetic data. Maybe that doesn't matter, though.)

As I mentioned in the previous post, we are starting to get the game out to a limited number of players. It's been a long time since I've had anyone new play through the game as a whole, so I really have no idea how it plays now, and it's important to find out. However, it's not so easy, because if you sit behind someone and watch them play a game, they are going to play it differently than if you weren't there. You invalidate your own playtest data.

So, I wanted to set up the game to record some minimal playtest data, that would be easy for me to browse through later on, and that would get recorded without me being there to directly watch the play session.

One of the first approaches that programmers often consider is an input recording method. You just journal a stream of all the player's inputs, and you can play that back deterministically to reproduce the entire game session. I have never liked this method. Firstly, you need to play back using exactly the same version of the game executable that the player used, which is a pain; secondly, this kind of playback code is brittle and breaks often. An uninitialized memory read bug can render your playback data useless! (Of course, one reason you might want playback data is to track down an uninitialized memory read bug... so... there's a problem here.)

For these reasons I didn't want to use input journaling. It seemed to me that our saved games are small enough, and the thumbnails are nice enough, that together they almost solve the problem. All I need to do is save a series of snapshots / game states over time, without overwriting previous saves:

Whereas the earlier folder contained a collection of different play sessions, this one contains one play session at different moments in time. Every time the game made an autosave for this play session (overwriting the old one in the normal way), it also also copied the autosave here. The date and time that are in each filename just indicate when that game was started; the number at the end tells you which snapshot this is.

There's one immediate, very nice property of this: I don't even need to run the game in order to get a general idea of how someone's playtest went. Once they send me the files, I can just click on the first one in Windows to invoke the image viewer, and then click the forward/backward buttons in the image viewer to animate my way through the game.

But, if I have questions about what was going on, I can also dig into the save file that corresponds to any given image. I made a mode that lets me browse these saves in-game, load up any particular one into the current world state, and play from there. I invoke it by running a console command:

This loads all the saved games and thumbnails and pops them into a scrolling list at the top of the screen, in chronological order:

I just scroll to the one I want:

... then press Enter to load that game state:

It's very fast to use and it only took a few hours to program.

This whole system is also pretty cool because it's much more robust than the input-recording method. If I want to look into some detail that is super-nitpicky, I may need to use the same build of the game executable that the player used, but this will almost never be necessary. Usually I won't even need to use the in-game browser -- as I mentioned, I can just click through the thumbnails in Windows to get a basic sense of what's going on.

The fact that we only have discrete snapshots is sort of a side-bonus: because we only snapshot every minute or so, the data is not so creepy and spy-like; players can relax while they are playtesting and know that we won't later be obsessing over their every move.

Turns out this can be implemented easily using some of the new C++ features. I'm not a big fan of C++11; for the most part I think it adds even more complexity to an already complex language. However, I have to admit that some of the features can be useful, when used with care and moderation.

I think this is one of such cases. If you can get past the weird syntax, lambdas can be very powerful. What I find most useful is to be able to define local functions (closures) that can access the local variables of the enclosing scope.

In our case, what we want to do is to define some code that is executed at the end of the scope to cleanup objects allocated in the current scope. This is easily achieved allocating an object on the stack that invokes a lambda function that wraps our custom cleanup code:

#include <functional>

struct ScopeExit {
    ScopeExit(std::function f) : f(f) {}
    ~ScopeExit() { f(); }
    std::function f;
};

#define STRING_JOIN2(arg1, arg2) DO_STRING_JOIN2(arg1, arg2)
#define DO_STRING_JOIN2(arg1, arg2) arg1 ## arg2
#define SCOPE_EXIT(code) \
    ScopeExit STRING_JOIN2(scope_exit_, __LINE__)([=](){code;})

Then you can write code like:

int * ip = new int[16];
SCOPE_EXIT(delete [] ip);

FILE * fp = fopen("test.out", "wb");
SCOPE_EXIT(fclose(fp));

I know you can go to greater lengths to mimic the D syntax, and that Boost also provides similar functionality, but this does all we need in less than 10 lines of code.

Sadly we cannot really use this on The Witness, because I'm not sure we are yet ready to drop support for platforms and compilers that do not implement the required C++11 functionality.

Update:

As mentioned by Daniel in the comments, you don't really want to use std::function in this case. A quick look at the assembly output confirms that the generated code is horrible. It makes a lot more sense to wrap the closure explicitly and rely on type inference to allocate the wrapper object on the stack:

template <typename F>
struct ScopeExit {
    ScopeExit(F f) : f(f) {}
    ~ScopeExit() { f(); }
    F f;
};

template <typename F>
ScopeExit<F> MakeScopeExit(F f) {
    return ScopeExit<F>(f);
};

#define SCOPE_EXIT(code) \
    auto STRING_JOIN2(scope_exit_, __LINE__) = MakeScopeExit([=](){code;})

I ended up running a straightforward test where we would change the game minimally and build consecutive patches. The patches turned out to be quite large, but upon investigation, it was clear there usually wasn't a good reason for this. Our compiled shaders would contain small differences, as would many meshes in the game.

These looked like uninitialized memory errors and, upon further investigation, that's indeed what they were. They are fixed now, so patches are a lot smaller.

However, patches may never be as small as we would like, and I find there is a pretty deep philosophical issue where.

Right now we have 17MB of files that change in response to minimal changes in the game (say, moving a few entities in the world by small distances). These are mostly cube maps for Light Probe entities, which are used to do lighting on nearby objects. Here's what a Light Probe looks like in the game:

(Well, you don't see that sphere the game, only in the editor!) To generate one of these cube maps, we render the scene in all directions, from the point at the center of that sphere, then apply a blur to the result.

Imagine that we move a couple of entities in the world, even slightly. All Light Probes that can see those entities are going to change, by at least a pixel or two. Because we are applying a big fat blur kernel, a 1-2 pixel change becomes small changes spread across the entire image.

We have an automated script that preps the game for release, clearing out data that we only use for development, and ensuring that automatically-computed things get recomputed (like LODs for faraway objects). Recently we added a step to this process that re-generates all the Light Probes, because it just makes sense to update those automatically. If we have to keep track of them manually somehow we are just going to make mistakes.

What this means is that you can pretty much guarantee that a lot of light probes are going to change if you move some entities and patch the game. If we change anything basic about any of the shaders, which seems very likely, then all of the light probes are going to change (because the output of the shaders feeds back into the light probes).

That said, maybe it's the case now that more light probes are changing than I would expect. I am wondering if other things in the game are causing them to change. For example, when you are playing the game, foliage waves in the wind. I wonder if the animation timer for plants is ticking during the process when we generate light probes, causing them to move over time? This would cause most light probes to be different every time we bake them, regardless of other factors.

Now, there's a bigger issue. In addition to light probes, we have pre-baked lightmaps all over all geometry in the game:

In a way that is reminiscent of the light probes (but a bit different), we generate these by placing a camera at many points along each surface, rendering the scene from each camera, and interpolating those into a final result.

So, just like light probes change when entities move slightly, lightmaps are going to change also. Lightmaps will also change if shaders change, but furthermore, lightmaps will change if light probes change (because light probes determine the look of entities that shine on the surfaces we are lightmapping), and light probes will change if lightmaps change (because the lightmaps influence the objects rendered in the light probes). So we have a circular dependency, which is not so bad in practice, except it means that if you change one entity in the game, then repeat this process iteratively, you may well end up changing most lightmaps and light probes in the game.

The light probes are not so big, but last I checked there are between 500MB and 700MB of lightmaps. However, re-baking the lightmaps is not part of the automated build process; we have to trigger a bake manually. But it seems like good policy to do a full rebake any time we change the game significantly. So, we are probably looking at a minimum patch size in the 500MB range.

I guess that is not such a big deal, since we are not an online game that would be patching all the time. Hopefully once we release the initial version of the game, the "content" won't change very much (sorry, but the word "content" as used in video games is kind of absurd, so it's hard for me to use it without sarcasm quotes; it's almost as bad as "title", which is a word I refuse to use, except to make fun of its use)... the "content" won't change very much, so patches would be rare.

We'll see. We have a while to figure these things out.

When textures are static a common solution is to pre-process them in an attempt to eliminate the edge seams. In a short siggraph sketch, John Isidoro proposed averaging cube map edge texels across edges and obscuring the effect of the averaging by adjusting the intensity of the nearby texels using various methods. These methods are implemented in AMD's CubeMapGen, whose source code is now publicly available online. While this seems like a good idea, a few minutes experimenting with CubeMapGen make it obvious that it does not always work very well!

Embedded Texture Borders

A very simple solution that even works for dynamic cube maps is to slightly increase the FOV of the perspective projection so that the edges of adjacent faces match up exactly. Ysaneya shows that in order to achieve that, the FOV needs to be tweaked as follows:

fov = 2.0 * atan(n / (n - 0.5))

where n is the resolution of the cube map.

What this is essentially doing is to scale down the face images by one texel and padding them with a border of texels that is shared between adjacent faces. Since the texels at the face edges are now identical the seams are gone.

In practice this is much trickier than it sounds. While the fragments at the adjacent face borders should sample the scene in the same direction, rasterization rules do not guarantee that in both cases the rasterized fragments will match.

However, if we take this idea to the realm of offline cube map generation, we can easily guarantee exact results. Cube maps are often used to store directional functions. Each texel has an associated uv coordinate within the cube map face, from which we derive a direction vector that is then used to sample our directional function. Examples of such functions include expensive BRDFs that we would like to precompute, or an environment map sampled using angular extent filtering.

Usually these uv coordinates are computed so that the resulting direction vectors point to the texel centers. For an integer texel coordinate x in the [0,n-1] range we map it to a floating point coordinate u in the [-1, 1] range as follows:

map_1(x) = (x + 0.5) * 2 / n - 1

We then obtain the corresponding direction vector as follows:

dir = normalize(faceVector + faceU * map_1(x) + faceV * map_1(y)

When doing that, the texels at the borders do not map to -1 and 1 exactly, but to:

map(0) = -1 + 1/n
map(n-1) = 1 - 1/n

In our case we want the edges of each face to match up exactly to they result in the same direction vectors. That can be achieved with a function like this:

map_2(x) = 2 * x / (n - 1) - 1

If we use this map to sample our directional function, the resulting cube map is seamless, but the face images are scaled down uniformly. In the first case the slope of the map is:

map_1'(x) = 2 / n

but in the second case it is slightly different:

map_2'(x) = 2 / (n - 1)

This technique works very well at high resolutions. When n is sufficiently high, the change in slope between map_1 and map_2 becomes minimal. However, at low resolutions the stretching on the interior of the face can become noticeable.

A better solution is to stretch the image only in the proximity of the edges. That can be achieved warping the uv face coordinates with a cubic polynomial of this form:

warp3(x) = ax^3 + x

We can compose this function with our original mapping. The result around the origin is close to a linear identity, but we can adjust a to stretch the function closer to the face edges. In our case we want the values at 1-1/n to produce 1 instead, so we can easily determine the value of a by solving:

warp3(1-1/n) = ax^3 + x = 1

which gives us:

a = n^2 / (n-1)^3

I implemented the linear stretch and cubic warping methods in NVTT and they often produce better results than the methods available in AMD's CubeMapGen. However, I was not entirely satisfied. While this removed the zero-order discontinuity, it introduced a first-order discontinuity that in some cases was even more noticeable than the artifacts it was intended to remove.

You can hover the cursor over the following image to show how the warp edge fixup method eliminates the discontinuities, but sometimes still results in visible artifacts:

Any edge fixup method is going to force the slope of the color gradient across the edge to be zero, because it needs to duplicate the border texels. The eye seems to be very sensible to this form of discontinuity and it's questionable whether this is better than the original artifact. Maybe other warp functions would make the discontinuity less obvious, or maybe it could be smoothed like Isidoro's method do. At the time I implemented this I thought the remaining artifacts did not deserve more attention and moved on to other tasks.

Modifed Texture Lookup

However, a few days ago Sebastien Lagarde integrated these methods in AMD's CubeMapGen. See this post for more results and comparisons against other methods. That got me thinking again about this and then I realized that the only thing that needs to be done to avoid the seams is to modify the texture coordinates at runtime the same way we modify them during the offline cube map evaluation. At first I thought that would be impractical, because it would require projecting the texture coordinates onto the cube map faces, but turns out that the resulting math is very simple. In the case of the uniform stretch that I first suggested, the transform required at runtime is just a conditional per-component multiplication:

float3 fix_cube_lookup(float3 v) {
   float M = max(max(abs(v.x), abs(v.y)), abs(v.z));
   float scale = (cube_size - 1) / cube_size;
   if (abs(v.x) != M) v.x *= scale;
   if (abs(v.y) != M) v.y *= scale;
   if (abs(v.z) != M) v.z *= scale;
   return v;
}

One problem is that we need to know the size of the cube map face in advance, but every mipmap has a different size and we may not know what mipmap is going to be sampled in advance. So, this method only works when explicit LOD is used.

Another issue is that with trilinear filtering enabled, the hardware samples from two contiguous mipmap levels. Ideally we would have to use a different scale factor for each mipmap level. That could be achieved sampling them separately and combining the result manually, but in practice, using the same scale for both levels seems to produce fairly good results. We can easily find a scale factor that works well for fractional LODs as a function of the LOD value and the size of the top level mipmap:

float scale = 1 - exp2(lod) / cube_size;
if (abs(v.x) != M) v.x *= scale;
if (abs(v.y) != M) v.y *= scale;
if (abs(v.z) != M) v.z *= scale;

If you are using cube maps to store prefiltered environment maps, chances are you are computing the cube map LOD from the specular power using log2(specular_power). If that's the case, the two transcendental instructions cancel out and the scale becomes a linear function of the specular power.

The images below show the results using the warp filtering method (these were chosen to highlight the artifacts of the warp method). Hover the cursor over the images to visualize the results of the new approach:

I'd like to thank Sebastien Lagarde for his valuable feedback while testing these ideas and for providing the nice images accompanying this article.

Note: This article is also published at AltDevBlogADay.

A small team with limited manpower, we wanted a low-effort way to use our existing version control system (svn) to facilitate concurrent world editing for our open-world 3D game (i.e. we desire that multiple users can edit the world at the same time and merge their changes).  We developed a system wherein the data for each entity is stored as an individual text file.  This sounds crazy, but it works well: the use of individual files gives us robustness in the face of merge conflicts, and despite the large number of files, the game starts up quickly.  Entities are identified using integer serial numbers.  We have a scheme for allocating ranges of these numbers to individual users so there is no overlap (though this scheme could certainly be better).  The system seems now to be robust enough to last us through shipping the game, with seven people and one automated build process concurrently editing the world across 18 different computers.

We don't know whether this method would work for AAA-sized teams, though it seems that it might with some straightforward extensions.  But for us the method works surprisingly well, given how low-tech it is.

About the Game

We have been developing a 3D open-world game called The Witness since late 2008.  For much of that time, development happened in a preproduction kind of way, where not many people were working on the game.  Recently, though, we went into more of a shipping mode, so the number of developers has grown: we're up to about 10 or 11 people now (pretty big for a non-high-budget, independent game).  Not all of these people directly touch the game data, but about 7 do now, and this number will rise to 9 in the near future.

The design of the game requires the world to be spatially compact, relative to other open-world games, so it is far from Skyrim-esque in terms of area or number of entities.  At the time of this writing, there are 10,422 entities in the game world.  I expect this number to grow before the game is done, but probably not by more than 2x or 3x the current amount.

Because the world is not too big, and some game-level events require entities to coordinate across the length of the world, it makes sense to keep all the entities in one space, that is to say, there is no zoning of the world or partitioning of it into separate sets of data.  This makes the engine simpler, and it reduces the number of implementation details that we need to think about when editing the world.  It also introduces a bit of a challenge, because it requires us to support concurrent editing of the world without an explicit control structure governing who can edit what.  One such control structure might be: if you imagine the world were divided into a 10x10 grid; so there are 100 different zones; and where anyone editing the world must lock whatever zones they want to edit, make their changes, check them in, then unlock those zones; this would provide a very clear method of ordering world edits.  (Because of the locking mechanism, two people would not be able to change the same world entity at the same time, so there doesn't have to be any resolution process to decide whose changes are authoritative). However, the straightforward implementation of such a control scheme would make for inconvenient workflow, as it is tedious to lock and unlock things.  Even with further implementation work and a well-streamlined UI, there would still be big inconveniences (I really want to finish building a particular puzzle right now, but a corner of that puzzle just borders on some region that a different user has locked, and he is sick this week so he probably won't finish his work and unlock it until next week.  Or, if my network connection is flaky then I may not be able to edit; if I am off the net then I definitely could not edit.)

Thus, rather than implementing some kind of manual locking, I found it preferable to envision a system that Just Magically Works.  Since we already used the Subversion source-control system to manage source code and binary source assets such as texture maps and meshes, it didn't seem like too big of a stretch for Subversion to handle entity data as well.

The World Editor and Entity Data

Our world editor is built into the game; you press a key to open it at any time.  Here's what it looks like:

The editor view is rendered in exactly the same way as the gameplay view, but you fly the camera around and select entities (I have selected the red metal door at the center of the scene, so it is surrounded by a violet wireframe).  In the upper right, I have a panel where I can edit all the relevant properties of this Door; the properties are organized into four tabs and here you can see only one, the "Common" properties.

These properties, which you can edit by typing into the fields of this Entity panel, are all that the game needs to instantiate and render this particular entity (apart from heavier-weight shared assets like meshes and texture maps).  If you serialize all these fields, then you can unserialize them and generate a duplicate of this Door.  That's how the editor saves the world state: serialize every entity, and for each entity, serialize all fields.

Originally, the game saved all entities into one file in a binary format.  This is most efficient, CPU-performance-wise, and the natural thing that most performance-minded programmers would do first.  Unfortunately, though, svn can't make sense of such a file, so if two people make edits, even to disjoint parts of the world, svn is likely to produce a conflict; once that happens, it's basically impossible to resolve the situation without throwing away someone's changes or taking drastic data recovery steps.

Our natural first step was to change this file from binary into a textual format.  Originally I had worries about the load-time performance of parsing a textual file, but this turned out not to be an issue.  I decided to use a custom file format, because the demands in this case are very simple, and we have maximum control over the format and can modify it to suit our tastes.

Many modern programmers seem to have some kind of knee-jerk inclination to use XML whenever a textual format is desired, but I think XML is one of the worst file formats ever created, and I have no idea why anyone uses it for anything at all, except that perhaps they drank the XML Kool-Aid or have XML Stockholm Syndrome.  Yes, there are XML-specific tools out there in the world, and whatever, but I didn't see how any such tool would be of practical use to us.  One of our primary concerns is human-readability, so that people can make intelligent decisions when attempting to resolve revision control conflicts.  XML is not particularly readable, but we can modify our own format to be as readable as it needs to be in response to whatever real-world problems arise.  You'll see a snippet below of the format we use.

Evolution of the Textual Format

The first version of our textual format was a straightforward adaptation of our binary format.  The gameplay-level code that saves and loads entities is the same for either format; the only difference is just a boolean parameter, choosing textual or binary, which is passed to the lower-level buffering code.  After the buffer is done being packed, it's written to a file.

In our engine, entities are generally referenced using integer serial numbers (that act as handles).  These numbers are the same when serialized as they are at runtime.  Originally, in the binary file, the order in which the entities were written didn't matter.  But if we want to handle concurrent edits, we want to make merging as easy as possible for the source control system, so we sort entities by serial number, writing them into the file in increasing order.  Thus we didn't generate any spurious file changes due to haphazardly-changing serialization order.

In theory, Subversion could now handle simultaneous world edits and just merge them, the same way it does with our source code.  Unsurprisingly, though, this initial attempt was not sufficient.  Though the file was now textual, it contained a bare minimum of annotations and concessions to readability.  Though I knew I would eventually want to add more annotations, I didn't want to go overboard if it wasn't necessary, because these inflate the file size and the time required to parse the files (though I never measured by how much, so it's likely the increases are negligible performance-wise and I was engaging in classic premature optimization).  As it happens, the kind of redundant information that increases human-readability also helps the source control system understand file changes unambiguously, so as we made the format more-readable, the number of conflicts fell.

Here's what our file format currently looks like, for entity #6555 (the Door we had selected in the editor screenshot).

Door
72
6555
; position
    46.103531 : 42386a04
    108.079681 : 42d828cc
    8.694907 : 410b1e57
; orientation
    0.000000 : 0
    0.000000 : 0
    -0.069125 : bd8d9184
    0.997608 : 3f7f633d
; entity_flags
    160
; entity_name
    !
; group_id
    4059
; mount_parent_id
    6554
; mount_position
    0.035274 : 3d107ba5
    0.006069 : 3bc6e19c
    0.000038 : 381ffffc
; mount_orientation
    0.000000 : 0
    0.000000 : 0
    0.000027 : 37e5f004
    1.000000 : 3f800000

(The first line, "Door", is the type name; 72 is the entity data version, which is used to load older versions of entities as their properties change over time; 6555 is the entity's ID.  root_z and cluster_id (seen in the screenshot) are not saved into the file, because these properties have metadata that say they are only saved or loaded under certain conditions which are not true here.  For the sake of making the listing short, I've shown only the properties listed on that first tab in the screenshot.)
You'll note that the floating-point numbers are written out a bit oddly, as in this first line under 'position':

    46.103531 : 42386a04

This is how we preserve the exact values of floating-point numbers despite saving and loading them in a text file.  If you do not take some kind of precaution, and use naive string formatting and parsing, your floating-point values are likely to change by an LSB or two, which can be troublesome.

The number on the left is a straightforward decimal representation of the value, written with a reasonable number of digits.  The : tells us that we have another representation to look at; the number on the right, which the : introduces, is the hexadecimal representation of the same number.  The IEEE 754 floating-point standard defines this representation very specifically, so we can use this hexadecimal representation to reproduce the number exactly, even across different hardware platforms.  The decimal representation is mostly just here to aid readability.  However, if we decide we want to hand-edit the file, we can just delete the : and everything after it, changing that first number to a 47.5 or something; seeing no :, the game will parse the decimal representation and use that instead.

There are other ways to solve this floating-point exactness problem; for example, you can implement an algorithm that reads and writes the decimal representation in a manner that carefully preserves the values.  This is complicated, but if you're interested in that kind of thing, here's what Chris Hecker says:

Everybody just uses David Gay's code:
http://www.netlib.org/fp/dtoa.c
http://www.netlib.org/fp/g_fmt.c

You need to get the flags set right on the compiler for it to work. /fp:precise at least, I also do #pragma optimize ("p",on), and make sure you write a test program for it with your compiler settings.

(Twitchy, and look at how much code that is!  What we did is a couple of lines.) In a document on his web site, Charles Bloom discusses a scheme that was in use at Oddworld Inhabitants, similar to what we're doing in The Witness but involving a comparison between the two representations:

... Plain text extraction of floats drifts because of the ASCII conversions; to prevent this, we simply store the binary version in the text file in addition to the human readable version; if they are reasonably close to each other, we use the binary version instead so that floats can be persisted exactly.

Subversion Confusion

After switching to this format, conflicts still happened.  I categorize conflicts into two types: "legitimate", where two people edit the same entity in contradictory ways and the source control system doesn't know who to believe; and "illegitimate", where Subversion was just getting confused.  We did have a number of legitimate conflicts for a while, yet which were not the users' fault, because our world editor tended to do things that are bad in a concurrent-editing environment.  (See the Logistics section at the end of this article for some examples).  But even after we ironed these out, we still found that Subversion was still getting confused.

Here's an example of a common problem: suppose we start with a world containing data on entities A, B, and D, so our file contains: ABD.  You edit the file and delete entity B.  I edit the file and also delete entity B, but add another entity C.  (That we both deleted B may be a very common use-case, especially if this is due to some automatic editor function).  So, you want to check in a file that just says: ABD - B = AD, so the new result is AD.  I want to check in a file that says: ABD - B + C = ACD (the C goes in the middle, because remember, the entities are sorted!)  In theory this is an easy merge: the correct result is just ACD.  But Subversion would get confused in cases like this, requiring the user to hand-merge the file.

This is a disaster, because even with a visual merge tool such as the one that comes with TortoiseSVN, you don't want non-engine-programmers trying to merge your entity data file.  Even if that file is extremely readable, at some point someone will make a mistake; the file will fail to parse, and then you have a data recovery problem that requires the attention of an engine programmer.  When this happens it kills the productivity of two people (the engine guy, because he has to merge the data, and also the guy who generated the conflict, because until it gets resolved his game is in an un-runnable state, and anyway, the engine guy has probably commandeered the other guy's computer to fix this).

After we had this problem a few times too many, Ignacio suggested moving to a more-structured file format like JSON, which, in theory, source-control systems would handle more solidly.  But I thought that this would only reduce the frequency of the problem, but not solve anything fundamental; even if we achieved a situation of only generating legitimate conflicts, each conflict would still be a potential game-crippling disaster.

Splitting the File

It occurred to me, finally, that it could hardly get cleaner than to store each entity in its own file.  This initial thought might cause a performance-oriented programmer to recoil in horror; we have 10,422 entities right now, and it must be very slow to parse and load 10,000 text files every time the game starts up!

But I knew that we already had over 4,000 individual files laying around for assets like meshes and texture maps, currently loaded individually, the idea being that at ship time we would combine these into one package so as to avoid extra disk seeks and the like.  But on our development machines, these files still loaded quickly.  I figured that if we can already load 4,000 big files, why not try loading an extra 10,000 small files?  If loading all these files turned out to be slow, we could resort to this only when entity data has changed via a source control update; after loading all the text files, we can save out a single binary file containing all the entity data, which would be much faster to load, just like we used to do before we supported concurrent editing.

I tried this and it worked great.  So, now we have a data folder called 'entities', which holds one file per serialized entity.  The Door selected in the editor screenshot is stored in a file named after its entity ID, "6555.entity_text", which contains the text you see in the listing above.

As mentioned, there are currently 10422 entity files; the text files take up about 8MB of space, whereas the binary file is 1.4MB.  We're not making any effort to compress either of these formats; they are very small compared to the rest of our data.

Benefits of Using Individual Files

Storing each entity in a separate file solved our problems with merging: both the illegitimate source control confusions as well as the problems due to hand-guided file merging.

The illegitimate conflicts go away because transactions become very clear.  If I delete an entity, and you delete an entity, then we both just issue a file delete command, and it's very easy for the source control system to see that we agree on the action.  Furthermore, if we both delete B but I add C, because the change to C is happening in a completely separate file, there is no way for the system to get confused about what happened.

Legitimate conflicts do still happen, as multiple people edit the same entities, but these cases are much easier to handle than they were before.  The usual response is to either revert one whole entity or stomp it with your own.  Because we usually want to treat individual entities atomically -- using either one person's version of that entity or the other's -- individual files are a great match, because now the usual actions we want to perform are top-level source control commands (usually from the TortoiseSVN pulldown menu); before, this wasn't true, and merging a single entity in an atomic way required careful use of the merge tool.  If an entity ever gets corrupted somehow, for whatever reason, we can just revert that one file, knowing the damage is very limited in scope; before we've reverted it, the worst that can happen is that that single entity will not load; it won't prevent the rest of the entities in the world from loading.

It would be difficult to overemphasize the robustness gained.  I feel that these sentences do not quite convey the subtle magic; it feels a little like the state change when a material transitions from a liquid to a solid.

Sometimes, someone will update their game data in a non-vigilant way; not noticing that a conflict happened, so they won't resolve it; and then they will run the game.  Before, this would have been a disaster; Subversion inserts annotations into text files to try to help humans manually resolve conflicts (this is an unfortunate throwback to programs like SCCS), which would have caused much of the world to fail to parse.  Now, though, the damage from these annotations is constrained to individual entities. It's also easier to detect: if there is a conflict in the file for our Door #6555, then in addition to writing these annotations into 6555.entity_text (making it unparseable by the game), Subversion will also write out files 6555.entity_text.mine and 6555.entity_text.theirs, which contain the two intact but conflicting versions of the file.  The game can detect these very straightforwardly, put an error message on the HUD, and even choose one to load by default.  It is all very clean.

This is one of those interesting cases where reality-in-practice is starkly different from computer-science-in-theory.  In theory, a directory is just a data structure organizing a bunch of black-box data, and we happen to view this organization in a way that looks like a list of files; we could just do the same thing interior to our own file format, and it would be no big deal.  In practice, due to the way our tools work, these two arrangements are tremendously different.  The individual files just feel solid and reliable to work with, whereas one big structured file is always going to be precarious.

Startup Procedure

When the game starts up, we need to load all the entities from disk.  In the simplest case, this is straightforward: we just iterate through every file in the entities directory and load it.  However, as I mentioned, we wanted to keep a binary version of the entities that is preferred by the game if it exists and is up to date; partly, this might help startup speed a little (we don't have to parse all these strings), but mainly, that is the file format and code path we are going to use when the game ships, and we don't want that code to rot in the meantime.  Using it every day is the best way to prevent it from rotting.

So: when the game first starts up, we check for the binary entity package.  If it's not there, we load all the individual entities files, then immediately save out the binary package.  (This binary package is a private, local file; if we were to add it to source control, everyone would always conflict with everyone else.)  But suppose we start up and the binary package is already there (which it usually will be).  We need then to determine if any of the textual entities have changed since the binary package was built.  The most-general way to do this is just by looking at the file timestamps; if we were to rely on some other signal being written to the disk that says "hey, entity data has changed," then we would only be allowed to modify the entity files using sanctioned and customized tools; that would be annoying and would cause all kinds of robustness issues.

In concrete terms: we want to know the timestamp of the newest .entity_text file, inclusive of the directory containing these files (if we just did a Subversion update that only deleted entities, then all the .entity_text timestamps will be old, but the directory timestamp will be new; we want to ensure we properly rebuild the binary file in this case!)  To find the newest timestamp, we just iterate through all the timestamps in a brute-force way:

bool os_get_newest_file_modification_time(char *input_dir_name, Gamelib_Int64 *file_time_return) {
    //
    // We include the modification time of the directory as well.
    //

    char *dir_name = mprintf("%s/*", input_dir_name);
    Free_On_Return(dir_name);
    if (strlen(dir_name) >= MY_MAX_PATH - 5) return false;

    //
    // best_time starts as the time for the directory (if it exists!)
    //
    Gamelib_Int64 best_time = 0;
    bool success = get_file_modification_time(input_dir_name, &best_time);
    if (!success) {
        Log::print("Unable to get modification time for directory '%s'!\n", input_dir_name);
    }

    //
    // Now consider the file timestamps.
    //
    WIN32_FIND_DATA find_data;
    HANDLE handle = FindFirstFile(dir_name, &find_data);
    if (handle == INVALID_HANDLE_VALUE) return false;

    while (1) {
        if (find_data.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY) {
        } else {
            Gamelib_Int64 file_time = to64(find_data.ftLastWriteTime);
            if (file_time > best_time) best_time = file_time;
        }

        BOOL success = FindNextFile(handle, &find_data);
        if (!success) break;
    }

    FindClose(handle);

    if (file_time_return) *file_time_return = best_time;

    return true;
}

bool get_file_modification_time(const char *name, Gamelib_Int64 *result_return) {
    HANDLE h = CreateFile(name, 0, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE,
                          NULL, OPEN_EXISTING, FILE_ATTRIBUTE_READONLY | FILE_FLAG_BACKUP_SEMANTICS,
                          NULL);
    if (h == INVALID_HANDLE_VALUE) return false;

    FILETIME write_time;
    GetFileTime(h, NULL, NULL, &write_time);
    CloseHandle(h);

    if (result_return) *result_return = to64(write_time);
    return true;
}

You'll notice that this is Windows-specific code, but we don't have to worry about porting it, because we only run the world editor on Windows.  Fortunately, in Windows, the file timestamp is right there in the directory data, so we don't have to open each file.  (At least in NTFS this is true, rather than being an API illusion; the timestamps are stored in the Master File Table [MFT].)

What's the performance like?  I typically run the game on two different machines: a high-performance Tower PC (CPU: Intel Core i7-970, hard drive: OCZ Vertex 3 SSD, RAM: 12GB) and a gaming Laptop (CPU: Intel Core i7-2820QM, hard drive: Intel SSD 500 [SSDSC2MH250A2], RAM: 12GB).

The time spent in os_get_newest_file_modification_time is:  Tower: .0095 sec, +/- .0015 sec; Laptop: .039 sec, +/- .005 sec.  (Timings were computed by averaging six trials, after a discarded warm-up run.)  This is definitely one of those cases where I'm glad I didn't prematurely optimize, as I would have expected this to be much slower!

We typically install SSDs in all our work machines, but even when people have home machines that use spinning-platter hard drives, startup times are not much slower (likely due to the MFT data staying in cache).

If for a large enough number of entities this timestamp-gathering process becomes too slow, we can always write some code to read the timestamps out of the MFT directly, avoiding all the API call overhead, as in this forum posting.  To date we've had no need to do this.

Some people have reported performance problems when Windows has more than 100,000 files in one directory, but this can be worked around: just make 10 directories, with IDs ending with each digit from 0-9 grouped in common directories.  (Why use the least-significant "ending" digit, rather than the most-significant "starting" digit, which might be the impulsive choice?  Because if you do it that way, Benford's Law says your entities are likely to be unevenly distributed across the directories.  Why not just hash them into a folder?  Because it's useful to be able to manually navigate to the entity you want, without searching.  (However, partitioning by least-significant digit may be inconvenient if you want to manually edit a sequential range of entities for some reason, because that range will span many directories.  If this is an important use case, perhaps there's a good compromise like using the tens digit or hundreds digit -- though I have the itching feeling that a clearly superior solution would exist.))

As things stand in terms of editing the game world, we are probably not yet at the peak entity count, but there's unlikely to be more than 3x the number of entities we have now.  Given how low the timing measurements are now, it's clear that performance will not be a factor in the ability to use this system up until shipping the game.

Partitioning the ID Space

As I mentioned, in our engine, entity IDs are integer serial numbers.  We like this because they are easy to type (we have to type them in the editor sometimes) and at runtime, looking up an entity compiles to a fast indexing of a flat array, with a bounds check.

This introduces another problem for us to deal with in world editing, though: we need to ensure that two users, who are both creating new entities, do not both try to use the same number.  This would result in both users attempting to add completely incompatible .entity_text files with the same name.

One obvious and bulletproof solution is to have the editor connect to the network and atomically grab an ID from a central server each time one is needed.  But again, we don't want to require users to be on the network in order to edit.

My solution was a somewhat lazy one: we allocate a range of IDs to each computer that edits the world.  These are kept in a file called "id_ranges"; at startup time, the game queries its machine's MAC address, then looks in the file for the corresponding ID range; if not found, it claims a new range and adds a new line to the file:

 6000 10000 BC-AE-C5-66-28-E9 #Jonathan_office
10000 13000 BC-AE-C5-66-28-48 #Ignacio_office
13000 16000 00-1C-7B-75-00-D8 #Jonathan_laptop
16000 19000 BC-AE-C5-66-28-93 #Shannon_office
19000 22000 BC-AE-C5-56-42-51 #Jonathan_home
22000 25000 C0-3F-0E-AA-F3-F2 #icastano_home
25000 28000 F4-6D-04-1F-D4-36 #Shannon_home
28000 31000 Baker #Baker
31000 34000 BC-AE-C5-56-43-FE #Misc_PC_office
34000 37000 00-E0-81-B0-7A-A9 #eaa
37000 40000 00-21-00-A0-8F-E3 #Orsi
40000 43000 64-99-5D-FD-97-64 #Jonathan
43000 46000 08-86-3B-42-0B-8D #Misc
46000 49000 90-E6-BA-88-9B-B7 #andrewlackey
49000 52000 48-5B-39-8C-80-5E #eaa
52000 55000 50-E5-49-55-64-BF #facaelectrica
55000 58000 00-FF-D4-C7-15-A3 #facaelectrica
58000 70000 Clusters #Clusters
71000 75000 Overflow #Overflow
75000 78000 F4-6D-04-1F-CC-77 #Nacho
78000 81000 BC-AE-C5-74-7B-A3 #Thekla

On each line, everything after the # is just a comment telling us in a human-readable way whose machine is responsible for that range (it defaults to the Windows username, but we can hand-edit that if we want).

("Baker" is for an automated process, so I made a special case that uses the string "Baker" instead of the MAC address, just to minimize the potential for the build machine to claim a new range without us noticing and failing to check that in; see below.  "Clusters" is for an editor operation that is also run in an automated process, but which can also be done manually by anyone; cluster generation removes all previous clusters from the world, so if clusters didn't have their own space, you would end up frequently deleting entities in other peoples' ranges; this seems like it might be a bad idea, though maybe, in reality, it's fine and there would be no problem.  But as it stands we know that anything in the Clusters range is completely disposable, so if a conflict ever arises there we can just revert or delete those entities without a second thought.)

This ID space is naturally going to have a lot of big holes in it, but before we ship we will renumber the entities so that we can pack them tightly into an array.  In the meantime, it is sparse but not pathologically so; we can still get away with using a linear array to index the entities, though it will use more memory now than at ship.

This id_ranges file is a bit of a lazy solution, and it causes a few annoyances.  The most annoying thing is that, any time someone runs the editor for the first time on a new machine, they need to immediately check in the id_ranges file to publish their allocation to the rest of the team; and they need to coordinate so that nobody else is trying to do the same thing.  Because the editor just allocates the next 3000 IDs starting with the highest ID in the file, if two people try to allocate IDs simultaneously, they will both claim the same range.  We could solve this by requiring the editor to hold a source control lock to modify the id_ranges file, but I just haven't gotten around to it, because this hasn't been a big-enough problem.  Furthermore, we could have the editor automatically commit the changes to the file; but currently, nothing is ever published to the rest of the team automatically -- publishing always requires manual action, and this seems like it may be a good thing to preserve.

But, not having implemented these solutions, we have been taken by surprise a couple of times by interactions with a second lazy choice of this system: the fact that we file only records one MAC address per computer.

Most modern computers have multiple MAC addresses (for example: if your computer has a Wifi adapter and two Ethernet ports, it's probably got at least 3 MAC addresses).  My lazy code just asks Windows to iterate the MAC addresses and reports the first one for use in the file.  Sometimes, though, the reported address will change, which causes someone's machine to modify the id_ranges file when they didn't expect it; we had this problem once when an audio contractor, who works offsite, installed some VPN software.  If that user is not vigilant about source control (which non-technical people often aren't), it can be a while before this problem is discovered; meanwhile, that user has been creating entities that may conflict with someone else's.

Most or all occurrences of this problem could be solved by recording all of the user's MAC addresses in the file, instead of one; I just haven't done so.

If we weren't using integer serial numbers, we wouldn't really need id_ranges at all.  For example, if we looked up entities at runtime by hashing unique string identifiers, then we would only need to ensure that each user has a unique string prefix.  We take a slight overkill approach, where each entity ID is a somewhat long string starting with the user's MAC address; or we could even generate a Windows GUID for each entity.  MAC addresses are 48 bits wide.  If you concatenate the MAC address with a 4-byte integer, that's just 10 bytes per entity ID, which isn't much (we currently use 4, already!).  Of course encoding these as strings and storing them in a hash table is going to require more space, but it isn't ridiculous.  Looking things up in a hash table all the time is slower than indexing a flat array, so we are sticking with the way we do things for now; but this would solve some problems and simplify the overall system, so it sounds like the kind of decision I would change my mind on later, as computers get even faster than they currently are.  So if you implement a scheme like this, you can ignore everything I said about id_ranges.  The biggest problem becomes making it easy to type longer entity IDs into the in-game editor.

Logistical Issues in Editor and File Format Design

In closing, I want to discuss changes we made to the editor and file format, in order to make them compatible with the paradigm of concurrent editing.

Arrays

Just about every game wants to serialize arrays into files.  When transitioning from a binary format, the most natural thing for a performance-minded programmer is to write the array length out first, then write each member of the array.  That way, when reading the file back, we know how much memory to allocate, and we avoid all the slow things one has to do when reading an array of unknown length.  However, encoding arrays in this way results in a textual format that always conflicts, because even when people are making compatible edits to an array, nobody is going to agree on what that array count should be; also, it becomes impossible to manually edit conflicts without typing the count in: if two people add elements to an array, the merged array will contain all the items, but neither version of the file will contain the correct count!

To solve this, of course, we go to the common text-file convention of using a syntactic designation that an array is now over, rather than pre-declaring how many elements there are.  This is obvious, but I mention it here because it is a relatively universal concern.

Automatic Changes

You will also want to minimize or work around cases where the editor automatically changes entity properties without user intervention, because often these will lead to conflicts.  A simple example from our game world: the sun is represented by a few entities that are instantiated in the world, a glowing disc and a couple of particle systems.  Every frame, the gameplay code positions these entities with respect to the current camera transform so that, when the player moves around the world, the sun will seem to be infinitely far away.  This happens in the editor too, because when editing we want to see the world exactly the way it will look in-game.  So, as soon as we enabled concurrent editing, everyone was checking in different coordinates for the sun (depending on where the camera happened to be when they saved!) and this caused a lot of conflicts.  My first solution was to add a hack that zeroes the position of sun entities when saving, but upon loading, this interacted badly with the way particle systems interpolate their positions.  Instead, I added an entity flag that signifies that an entity's pose doesn't matter, with respect to thinking about whether it has changed or not, so we will only modify the sun's .entity_text files if some more-important properties are modified.  (This article is already long, but I'll say that in general, we detect differences between an entity's current state and its state when it was loaded, and we only write out .entity_text files when properties have changed.)

Relative Transforms

In our engine, all entities have 'position' and 'orientation' properties, which always represent the entity's frame in worldspace, flat and non-hierarchically.  If we want an entity to be attached to another entity's transform, we set the properties 'mount_parent_id', 'mount_position', and 'mount_orientation' (you can see all these properties in the listing for Door #6555 above); once per frame, the game engine computes the entity's new worldspace transform based on its mount information, properly handling long dependency chains.  Once we started using textual entity files, we found that entities' worldspace coordinates would often differ by an LSB or two, even when their parents had not moved (differences especially happening, for example, between Debug and Release builds of the game).  Even without concurrent editing, it's good to clean this up as a matter of basic hygiene, because otherwise you're checking in spurious changes all the time, increasing load on the source control system and making it harder for a human to see useful information in the change list.  I chose to solve this problem by making the mount-update code more complex: it detects whether an entity has moved of its own accord, and only updates child entities' worldspace positions if the parent entity really moved (or if the child entity really moved!).

Terrain Editing

There is one piece of editing workflow that used to make sense before concurrent editing, but which now does not.  We have a terrain system wherein we can move control points around in worldspace, then select a terrain entity, which has a height-field mesh, and press a key to recompute that height field with respect to the new control points.  I originally made this recomputation happen only on manually-selected entities, rather than automatically on the entire world, for UI smoothness purposes: the recomputation takes some time.  But when you only recompute one block of terrain and not its neighbors, you get discontinuities in the terrain; the longer things go on, the more discontinuities you get.  Once in a while I would fix this by selecting all terrain entities in the world, recomputing them simultaneously.

A terrain discontinuity

The catch is that when terrain changes, it affects other entities.  We re-plant grass, so that the grass matches the new elevations; we change the positions of trees and rocks and other objects that are rooted to the terrain at a predetermined height.  Whenever you recompute some terrain, we collect all grass/trees/rocks within a conservative bounding radius, as these are all the things that might possibly change.  When dealing with small groups of terrain blocks, this already is a bit of a recipe for trouble (it is not too hard to overlap someone else's edits this way); but it also means that if I recompute the entire terrain to eliminate those discontinuities, I am in turn editing all grass, trees, and rocks in the world, and will inevitably conflict with anyone else who has moved any of them. For now, we simply shy away from this kind of global operation; when you get a conflict in some entity that you probably didn't care about, like grass, it is easy just to accept the remote version of the file.  Long-term, I expect some kind of heavier-weight solution would have to happen here, though I hope it does not involve much locking.

Issuing Source Control Commands

Our editor automatically issues source control commands when entities are created or destroyed, because it would be extremely tedious and error-prone to expect users to do this.  Initially we had some problems with source control commands unexpectedly failing, which quickly leads to chaos.  After some work, issuing source control commands seems to work with full consistency, but we are still a little wary of it; so we don't, for example, want to issue the commands via a background process, adding more complexity to the system.  The problem with this is that svn commands are a bit slow.  Sometimes when I have been editing the world for a while, it can take 15 to 30 seconds to save my changes, which would have happened instantaneously without the svn commands.

Conclusion

We've put together a system that lets us concurrently edit an open world, using source control to manage our world edits.  It works well enough for us to ship the game, so I am happy with it.  If the scheme ever fails in some way, or requires heavy modification, I will post an update!

• Lightmap Parameterization
• Hemicube Rendering and Integration

Unfortunately, by the time I was done with these, the work I had done on the gradients was not so fresh anymore, which is why it has taken me so long to complete this article.

In my previous post I mentioned the need to find an algorithm that would allow us to extrapolate the irradiance to fill lightmap texels with invalid samples, to smooth the resulting lightmaps to reduce noise, and to sample the irradiance at a lower frequency to improve performance. As some avid readers suggested Irradiance Caching solves all these problems. However, at the time I didn't know that, and my initial attempts at a solution were misguided.

Texture Space Approaches

I had some experience writing baker tools to capture normal and displacement maps from high resolution meshes. In that setting it is common to use various image filters to fill holes caused by sampling artifacts and extend sampled attributes beyond the boundaries of the mesh parameterization. Hugo Elias' popular article also proposes a texture space hierarchical sampling method, and that persuaded me that working in texture space would be a good idea.

However, in practice texture space methods had many problems. Extrapolation filters only provide accurate estimates close to chart boundaries, and even then, they only use information from one side of the boundary, which usually results in seams. Also, the effectiveness of irregular sampling and interpolation approaches in texture space is greatly reduced by texture discontinuities and did not reduce the number of samples as much as desired.

After trying various approaches, it became clear that the solution was to work in object space instead. I was about to implement my own solution, when I learned that this is what irradiance caching was about.

Irradiance Caching

This technique has such a terrible name that I would have never learned about it if it wasn't due to a friend's suggestion. I think that a much more appropriate name would be adaptive irradiance sampling or irradiance interpolation. In fact, in the paper that introduced the technique for the first time Greg Ward referred to it as lazy irradiance evaluation, which seems to me a much more appropriate name.

There's plenty of literature on the subject and I don't want to repeat what you can learn elsewhere. So, in this article I'm only going to provide a brief overview, focus on the unique aspects of our implementation, and point you to other sources for further reference.

The basic idea behind irradiance caching is to sample the irradiance adaptively based on a metric that predicts changes in the irradiance due to proximity to occluders and reflectors. This metric is based on the split sphere model, which basically models the worst possible illumination conditions in order to predict the necessary distance between irradiance samples based on their distance to the surrounding geometry.

Interpolation between these samples is then performed using radial basis functions, the radius associated to each sample is also based on this distance. Whenever the contribution of the nearby samples falls below a certain threshold a new irradiance record needs to be generated, in our case this is done by rendering an hemicube at that location. In order to perform interpolation efficiently the irradiance records are inserted in an octree, which allows us to efficiently query the nearby records at any given point.

For more details, the Siggraph '08 course on the topic is probably the most comprehensive reference. There's also a book based on the materials of the course; it's slightly augmented, but in my opinion it does not add much value to the freely available material.

The Physically Based Rendering book by Matt Pharr & Greg Humphreys is an excellent resource and the accompanying source code contains a good, albeit basic implementation.

Finally, Rory Driscoll provides a good introduction from a programmer's point of view: part 1, part 2.

Our implementation is fairly standard, there are however a two main differences: First, we are sampling the irradiance using hemicubes instead of a stratified Montecarlo distribution. As a result, the way the we compute the irradiance gradients for interpolation is somewhat different. Second, we use a record placement strategy more suited to our setting. In part 1 I'll focus on the estimation of the irradiance gradients, while in part 2 I'll write about our record placement strategies.

Irradiance Gradients

When interpolating our discrete irradiance samples using radial basis functions the resulting images usually have spotty or smudgy appearance. This can be mitigated by increasing the number of samples and increasing the interpolation threshold, which effectively makes them overlap and smoothes out the result. However, this also increases the number of samples significantly, which defeats the purpose of using irradiance caching.

One of the most effective approaches to improve the interpolation is estimating the irradiance gradients at the sampling points, which basically tell how the irradiance changes when the position and orientation of the surface changes. We cannot evaluate the irradiance gradients exactly, but we can find reasonable approximations with the information that we already have from rendering an hemicube.

In my previous article I explained that the irradiance integral was just a weighted sum over the radiance samples, where the contribution of each sample is the product of the radiance through each of the hemicube texels , the solid angle of the texel , and the cosine term . That is, the irradiance at a point is approximated as:

The irradiance gradients consider how each these terms change under infinitesimal rotations and translations and can be obtained by differentiating .

The radiance term is considered to be constant, so the equation reduces to:

In reality, the radiance is not really constant, but the goal is to estimate the gradients without using any information in addition to what we have already obtained by rendering the scene on a hemicube. So that we can improve the quality of the interpolation by only doing some additional computations while integrating the hemicube.

Rotation Gradient

The rotation gradient expresses how the irradiance changes as the hemicube is rotated in any direction:

The most important observation is that as the hemicube rotates, the solid angle of the texels with respect to the origin remains constant. So, the only factor that influences the gradient is the cosine term:

With this in mind, the rotation gradient is simply the rotation axis scaled by the rate of change, that is, the angle gradient:

And the rotation axis is given by the cross product of the z axis and the texel direction :

Therefore:

This can be simplified further by noting that is the length of before normalization and that results in a very simple expression:

Finally, the rotation gradient for the hemisphere is the sum of the gradients corresponding to each texel.

The resulting code is trivial:

foreach texel, color in hemicube {
    Vector2 v = texel.solid_angle * Vector2(texel.dir.y, -texel.dir.x);
    rotation_gradient[0] += v * color.x;
    rotation_gradient[1] += v * color.y;
    rotation_gradient[2] += v * color.z;
}

Translation Gradients

The derivation of the translation gradients is a bit more complicated than the rotation gradients because the solid angle term is not invariant under translations anymore.

My first approach was to simplify the expression of the solid angle term by using an approximation that is easier to differentiate. Instead of using the exact texel solid angle like I had been doing so far, I approximated it by the differential solid angle scaled by the constant area of the texel, which is a reasonable approximation if the texels are sufficiently small.

In general, the differential solid angle in a given direction is given by:

For the particular case of the texels of the top face of the hemicube, n is equal to the z axis. So, the differential solid angle is:

We also know that:

which simplifies the expression further:

Using this result and factoring the area of the texels out of the sum we obtain the following expression for the gradient:

The remaining faces have similar expressions, slightly more complex, but still easy to differentiate along the x and y directions. We could continue along this path and we would obtain closed formulas that can be used to estimate the translation gradients. However, this simple approach did not produce satisfactory results. The problem is that these gradients assume that the only thing that changes under translation is the projected solid angle of the texels, but that ignores the parallax effect and the occlusion between neighboring texels. That is, samples that correspond to objects that are close to the origin of the hemicube should move faster than those that that are farther, and as they move they may occlude nearby samples.

As you can see in the following lightmap, the use of these gradients resulted in smoother results in areas that are away from occluders, but produced incorrect results whenever the occluders are nearby.

Left: Reference. Middle: Irradiance caching without gradients. Right: Irradiance caching with flawed gradients

The hexagonal shape corresponds to the ceiling of a room that has walls along its sides and a window through which the light enters. The image on the left is a reference lightmap with one hemicube sample per pixel. The image on the middle is the same lightmap computed using irradiance caching taking only 2% of the samples. Note that the number of samples is artificially low to emphasize the artifacts. On the right side the same lightmap is computed using the proposed gradients. If you look closely you can notice that on the interior the lightmap looks smoother, but close to the walls the artifacts remain.

This is exactly the same flaw of the gradients that Krivanek et al proposed in Radiance Caching for Efficient Global Illumination Computation, but was later corrected in the followup paper Improved Radiance Gradient Computation by approaching the problem the same way Greg Ward did in the original original Irradiance Gradients paper.

I then tried follow the same approach. The basic idea is to consider the cosine term invariant under translation so that only the area term needs to be differentiated, and to express the gradient of the cell area in terms of the marginal area changes between adjacent cells. The most important observation is that the motion of the boundary between cells is always determined by the closest sample, so this approach takes occlusion into account.

Unfortunately we cannot directly use Greg Ward's gradients, because they are closely tied to the stratified Montecarlo distribution and we are using a hemicube sampling distribution. However, we can still apply the same methodology to arrive to a formulation of the gradients that we can use in our setting.

In our case, the cells are the hemicube texels projected onto the hemisphere and the cell area is the texel solid angle. To determine the translation gradients of the texel's solid angle is equivalent to computing the sum of the marginal gradients corresponding to each of the texel walls.

These marginal gradients are the wall normals projected onto the translation plane, scaled by the length of the wall, and multiplied by the rate of motion of the wall in that direction.

Since the hemicube uses a gnomonic projection, where lines in the plane map to great circles in the sphere, the texel boundaries of the hemicubes map to great arcs in the hemisphere. So, the length of these arcs is very easy to compute. Given the direction of the two texel corners , the length of their hemispherical projection is:

The remaining problem is to compute the rate of motion of the wall, which as Greg Ward noted has to be proportional to the minimum distance of the two samples.

The key observation is that we can classify all the hemicube edges in wo categories: Edges with constant longitude ( is invariant), and edges with constant latitude ( is invariant).

In each of these cases we can estimate the rate of change of edges with respect to the normal direction by considering the analogous problem in the canonical hemispherical parametrization:

For longitudinal edges we can consider how changes with respect to motion along the x axis:

and for latitudinal edges we can consider how changes with respect to motion along the y axis:

As we noted before, the rate of the motion is dominated by the nearest edge, so takes the value of the minimum depth of the two samples adjacent to the edge.

A detailed derivation for these formulas can be found in Wojciech Jarosz's dissertation, which is now publicly available online. His detailed explanation is very didactic and I highly recommend reading it to get a better understanding of how these formulas are obtained.

Note that these are just the same formulas used by Greg Ward and others. In practice, the only thing that changes is the layout of the cells and the length of the walls between them.

As can be seen in the following picture, the new gradients produce much better results. Note that the appearance on the interior of the lightmap is practically the same, but closer to the walls the results are now much smoother. Keep in mind that artifacts are still present, because the number of samples is artificially low.

Left: No gradients. Middle: Flawed translation gradients. Right: Improved translation gradients.

In order to estimate the translation gradients efficiently we precompute the direction and magnitude of the marginal gradients corresponding to each texel edge without taking the distance to the edge into account:

// Compute the length of the edge walls multiplied by the rate of motion
// with respect to motion in the normal direction
foreach edge in hemicube {
    float wall_length = acos(dot(d0, d1));
    Vector2 projected_wall_normal = normalize(cross(d0, d1).xy);
    float motion_rate = (constant_altitude(d0,d1) ?
        -cos_theta(d0,d1) :      // d(theta)/dx = -cos(theta) / depth
        -1 / sin_theta(d0,d1);   // d(phi)/dy = -1 / (sin(theta) * depth)
 
    edge.translation_gradient = projected_wall_normal * wall_length * motion_rate;
}

Then, during integration we split the computations in two steps. First, we accumulate the marginal gradients of each of the edges scaled by the minimum edge distance:

foreach edge in hemicube {
    float depth0 = depth(edge.x0, edge.y0);
    float depth1 = depth(edge.x1, edge.y1);
    float min_depth = min(depth0, depth1);
 
    texel[edge.x0, edge.y0].translation_gradient -= edge.gradient / min_depth;
    texel[edge.x1, edge.y1].translation_gradient += edge.gradient / min_depth;
}

Once we have the per texel gradient, we compute the final irradiance gradient as the sum of the texel gradients weighted by the texel colors:

foreach texel, color in hemicube {
    Vector2 translation_gradient = texel.translation_gradient * texel.clamped_cosine;
    translation_gradient_sum[0] += translation_gradient * color.x;
    translation_gradient_sum[1] += translation_gradient * color.y;
    translation_gradient_sum[2] += translation_gradient * color.z;
}

Conclusions

Irradiance Caching provides a very significant speedup to our lightmap baker. On typical meshes we only need to render about 10 to 20% of the samples to approximate the irradiance at very high quality, and for lower quality results or fast previews we can get away with as few as 3% of the samples.

The use of irradiance gradients for interpolation allows reducing the number of samples significantly, or for the same number of samples produces much higher quality results. The following picture compares the results without irradiance gradients and with them, in both cases the number of samples and their location is the same, note how the picture at the bottom has much smoother lightmaps.

I think that without irradiance caching the hemicube rendering approach to global illumination would have not been practical. That said, implementing irradiance caching robustly is tricky and a lot of tuning and tweaking is required in order to obtain good results.

Note: This article is also published on #AltDevBlogADay.

Sun beams. These are accomplished with a radial blur centered on the sun, during postprocessing. The nice thing about this effect is how dynamic it is -- as you walk beneath the trees, the shape and intensity of the beams are changing all the time.

Due to the way the effect works, sometimes beams are drawn that are a little bit unphysical. You probably have to be looking closely to see that, but I am a little bit uncomfortable with it given that the game is all about the opposite of that. We'll see how this evolves. We've been talking about augmenting this with other effects.

A revised water reflection effect.

Softening of the shoreline, so that it doesn't look like one big alpha-blended water polygon intersecting the ground.

Here's what the island looks like now:

It's fun to look at this compared to another shot of the island, say, from about a year ago:

The current puzzle count is 328.

Here's what it looks like now:

It does not look shipping-quality, but then, almost nothing in the game does! As the first results obtained by just slapping some pre-existing grass meshes down on the terrain without trying too hard, I find it highly encouraging.

Note that grass doesn't cast shadows right now. There are two reasons for that: rendering speed and aliasing of the shadow map. I suspect we will try to make grass cast shadows before we ship, though, and if we make something good happen there, I think it will really boost the visual quality.

The one piece of code that pushed the grass over the threshold into general usability is that it now has the ability to sample terrain blend maps. (The terrain uses a blend shader, where a non-visible blend texture map controls the degree of mixing of different terrain materials. This is becoming a common thing in 3D games; Uncharted 2 and Crysis 2 do it, among others.) The grass can now peek at the blend map and reduce its density in areas of particular blend values, which is why in this screenshot that path going up the hill is free of grass. This tremendously reduces the amount of manual tweaking required to make the grass not do obviously-bad things.

However, the grass doesn't check for buildings yet, which means if you place a field of grass across the area of a building, the grass will probably stick up through the floor inside the building! Fixing that is next on my to-do list, but it involves a lot of raycasting, so hopefully it doesn't make grass placement too slow.

In the future I plan to try using this same "grass" system for other kinds of scattered ground objects (fallen leaves, twigs, small rocks, etc).

I was talking to David Hellman about the game the other day, and he was asking when the island was going to become less like a flat plane where you can just walk to any destination from anywhere else.

It's been on my to-do list for a while to add more topology to the island. That week I had been working on an area full of puzzles, and I wanted the buildings that define this area to have more of a purpose than they currently do; a topological purpose seemed like the right thing.

So I figured I'd add a canyon, and the building containing these puzzles would also serve as a bridge across the canyon.

The natural way to build the canyon in this engine is to shape it out generally with terrain control points, and then make the walls steep by putting mesh objects in there (so that you can't walk up them, and so that the walls of the canyon won't look super-distorted the way heightfield terrains usually do when there are steep slopes.)

Here was the first cut at it:

As you can see, there are a lot of weird ripples and curves happening there. This is what signal processing people might call "ringing". What's happening is that the Kriging system that we are using for terrain modeling is always trying to reconstruct a smooth function, but what we are trying to build here is a discontinuous function (or else a function so steep that it has a lot of high frequencies in it).

There are a number of ways that one might fix this, but the cheap and hacky one, which would work not only for Kriging but for any other terrain modeling stuff we might stick in in the future, is to add the notion of separating planes that represent discontinuities in the terrain. When evaluating the height of the terrain at any point, we just ignore any control points that are on the other side of the separating plane.

Adding discontinuities has been on my to-do list since last year when Kriging was added to the engine. It was pretty obvious that it would be needed eventually, but in the spirit of getting things done and not rat-holing on technology, I chose to put it off for as long as possible. But it makes sense that if we are starting to add more topology to the island, we are going to want to have more steep places, and so it's time to add this.

(We've already had this steepeness problem before, though: if you look at this plateau that is near the camera in our regular island snapshots, you'll see that the whole thing is made out of mesh objects, rather than terrain; we did this in order to avoid this very same ringing problem:)

It didn't take long to add discontinuity entities into the editor; here's how they are represented:

To start with, I just used a really dumb heuristic to decide when control points should be excluded. When we use this, here's the result:

There is still a bunch of heightfield terrain sticking out. What's going on? If we look back at our red discontinuity planes, it is clear that there are some terrain sample points inside the canyon that are getting treated as if they were outside:

What's going on is that my simple criterion said "if you are on the opposite side of a discontinuity plane from a control point, ignore that control point". The problem is that I am treating them as infinite planes, so those points that you see, which are in front of one plane, are actually outside a neighboring plane, if you were to extend that plane infinitely. This is no surprise; I knew it would be a problem, but I am always about implementing the simple thing first and seeing what happens. But now I need a heuristic that will fix this problem.

One solution would be to combine the discontinuity objects into sequences of points, and then do some more-heuristicky things that may be easier because we have an ordering between the points.  But I don't want to resort to that if I don't have to, because that may make the system less general, and may make it harder to edit.  (Then again it may make it easier to edit, because we have editor support for paths!  But if we want to, we can easily go the other way, from a path down to individual slabs, so it's better to take the most-primitive thing as input if we are able to.)

I added a check that would make discontinuity planes irrelevant if the sample points are closer to the side planes of these very thin slabs than they are to the face planes.  In theory that would at least help, but upon running this, the scene looked almost exactly the same.

Then upon further investigation I found that I was computing totally wrong planes in the Kriging code; I was transforming the coordinates incorrectly, and so the planes were not in fact where the red slabs in the picture would show them to be.  I fixed this:

Ahhh, much better.  This seems like it will be good enough for now.  There's still the question of how to seamlessly blend the heightfield terrain into the mesh objects, but that can wait until closer to ship.  For now, it doesn't need to be pretty; it just needs to let us build things of roughly the right shape.

The heuristic I use for discarding points is still pretty simplistic and may cause problems.  My hope is that most of those problems can be worked around just by changing the planes a bit.  If that turns out not to be true, a little more work will need to be done here!

The general requirement was that we need to be able to capture a pre-rendered scene with a really wide viewing angle. At first I was confused about the technical aspects of the problem (hopefully forgivable, since I spend most of my time thinking about gameplay these days, so my tech is a little rusty), so I thought that we might need to have a linear projection in order to solve the specific problem under consideration. So Shannon made a mock-up scene in 3D Studio MAX and we started making prerenders with increasingly wide camera angles, in order to test our special environment mapping. Here are three shots of the same scene with fields-of-view of 90 degrees, 120 degrees, and 150 degrees:

(Click on the images to see actual sizes.)

It's clear that as the field of view becomes wider, we can see more of the scene at once, laterally. This is good for our application, because we want to see as much as possible! But at the same time, when the angle is wide, objects at the center of the scene appear further away, occupying fewer pixels in the rendered image. This is bad because it means we only have low-resolution imagery for the most important things in the scene! (These screenshots are also badly artifacted because for some reason the avi encoder we were using interlaces the output by default, and we couldn't find a way to not interlace, without using a different codec! Hint to anyone making a video codec, anywhere in the past or future of this universe or any other: NOBODY LIKES INTERLACING. IT IS UGLY AND CAUSES A LOT OF PROBLEMS. DON'T DO IT. Please do your part to make the world a better place.)

It's around this time that I realized we didn't have to use a linear projection, which was good -- if we can warp the image however we need to, then we can have the important parts of the scene landing really big in the middle of the texture, using lots of pixels, and squeeze the rest of the scene nonlinearly around the edges of the bitmap, using fewer pixels.

A 180-degree fisheye lens seemed like the right tool to do this. For other reasons that I won't go into, another nice property of the fisheye lens (as opposed to some arbitrary distortion) is that it is physically plausible -- you can mount a fisheye lens onto a physical camera and generate a similar image.

We found some plugins for Mental Ray (a renderer that you can use from Max or Maya) that seemed to do the trick. Here's the output of one of the plugins we found (this is from Jan Sandström's JS_fisheye.c, which appears to be a modification of a simple fisheye lens shader in the Mental Ray reference manual):

Unfortunately the field of view in this image we saved is less than 180 degrees, which makes it hard to directly compare with the later images, but just look at the basic character of it for now.

At first I assumed that this shader was implementing actual fisheye lens math, though there were no real comments to go on. Here's the math used in the stripped-down Mental Ray Reference Manual:

    mi_vector_to_camera(state, &camdir, &state->dir);
    t = state->camera->focal / -camdir.z /
           (state->camera->aperture/2);
    x = t * camdir.x;
    y = t * camdir.y * state->camera->aspect;
    r = x * x + y * y;
    if (r < 1) {
        dir.x = camdir.x * r;
        dir.y = camdir.y * r;
        dir.z = -sqrt(1 - dir.x*dir.x - dir.y*dir.y);
        mi_vector_from_camera(state, &dir, &dir);
        return(mi_trace_eye(result, state, &state->org, &dir));
    }

(JS_fisheye.c has more parameters but is basically the same thing.)

Essentially, this code takes an input texture coordinate and converts it into a view vector in 3D space. In order to use the rendered image from within a shader in the game, I need to be able to invert this function: turn it from a view vector into a 2D texture coordinate. But when I tried to do this, I had all kinds of problems: the math was messy and ugly. I got the inkling that this shader may not be acting as a physical lens really would, but rather, was a 2D image-warping effect that gives the same general impression as a fisheye lens.

(The reason for thinking this is that light behaves bidirectionally; there's no difference between light moving forward and light moving backward. So if something is simple when light is going in one direction, it ought to be simple in the other direction too. If the equation starts looking a lot more complicated, that is sort-of a violation of the way that the physical universe works, mathematically, and so one starts thinking that something is wrong. That was the intuition I had, anyway.)

Over on this page I found a nice reference for the way that a real fisheye lens bends light. Indeed the equation is very simple (though as it is written on that page, it's not quite in the proper form for us to use). Based on that, I wrote a new Mental Ray shader that looks like this:

    mi_vector_to_camera(state, &camdir, &state->dir);
    miScalar x = state->raster_x / state->camera->x_resolution * 2 - 1;
    miScalar y = state->raster_y / state->camera->y_resolution * 2 - 1;
    
    miScalar r2 = x * x + y * y;

    if (r2 < 1) {
        miScalar c = 1 - r2;
        miScalar s = sqrtf(2 - r2);

        dir.x = x * s;
        dir.y = y * s;
        dir.z = -c;  

        mi_vector_from_camera(state, &dir, &dir);
        return mi_trace_eye(result, state, &state->org, &dir);
    }

Superficially it doesn't look too different from the previous example, but in fact this version refracts light in a physical way and is invertible. Here's what a render looks like using this lens:

As I mentioned, it's hard to compare with the earlier shot because the field-of-view is different (sorry about that), but I think it's evident that the nature of the distortion is fairly different between the two shots. In our version, it feels milder.

Here's the runtime pixel shader code that inverts it:

    // (xprime, yprime, zprime) is the view vector in the same space where the environment map was rendered.

    float c = abs(zprime);

    video_uv.xy = float2(xprime, yprime);
    float scale = 1 / sqrt(1 + c);
    video_uv.xy *= scale;

    // Now we have video_uv in [-1, 1]; for [0, 1], scale appropriately.

It's very simple, and the only real math required is a multiplication by a reciprocal square root (very fast for shaders!) So that was pleasant.

Once we had these both hooked up and working, the scene rendered perfectly, and we knew that the effect we wanted to create was achievable. By way of improving it, Ignacio suggested using a cylinder map instead of a fisheye lens, because that is better for this particular shot: we need to render a wide room, and see a lot laterally, but the vertical span is roughly constant and much shorter than the horizontal span. (The fisheye lens is more general, and we can use it for any scene, but to optimize texture resolution for specific cases, we might go to other things like the cylinder map. You can think of the cylinder map as being fisheyed along only one axis.)

Here's the cylinder map version of the scene:

This is what we are going with for now. Problem solved, job seemingly well-done.

Ignacio put the cylinder mapping code into the same shader as the fisheye lens, and also added a latitude-longitude distortion.

Our work was sped up drastically by the fact that we were able to find JS_fisheye.c free on the internet, as well as information for the way a fisheye lens works. So here's our attempt to give back a little: our final Mental Ray shader and the .mi file that defines the interface for it:

witness_fisheye.c
witness_fisheye.mi

If you are an experienced graphics programmer asking yourself why the hell didn't they just use a cube map for this, well, there is a very good answer to that, but it involves the spoiler. You'll see when the game is released!

I had been viewing a bunch of online portfolios of 3D modelers (and lamenting the quality of these portfolios) with an eye to hiring someone. Sometimes these portfolios are shown in steps: here's the wireframe, here's the filled-in geometry, here it is with lighting, and here is the final thing after it's been textured. Casey pointed out something interesting about these that I knew was true as soon as he said it: most of these portfolios look worse than they need to because the texturing ruins them. When you run a radiosity algorithm on a scene full of unpainted geometry, it has an elegant, sophisticated, scent-of-the-real kind of thing going on; but it is very easy to spoil this sophistication through texturing. And, it seems, in video games we have developed a tradition of "bad" texturing, whatever that means: unsubtle, unrefined, or just plain ugly.

We're on a low budget here, being an indie game; so not only was it an open question how to solve the Ugly Texturing problem, but we also needed to solve it with relatively little manpower.

I knew that, ultimately, we could just fall back on untextured geometry with lighting, if we had good precomputed lighting. That's why we spent so much time and effort putting together our lighting system (as has been written about many times in this blog). The question was, would we stick to vertex colors on most objects, or would we use some kind of texturing?

Early on, we identified a texturing style that would work for human-sized objects and would gel with the gameplay of The Witness. The style prioritizes color, and tries not to break up the geometry of the object by imposing conflicting details through the texture. We did some objects like a 1-person boat in this style in order to refine it. [I'd like to include a screenshot of the boat here to illustrate it, but the boat is messed up right now, so I'll just have to be vague.]

Because that style tries to use the texture map to accentuate the object, the contents of the texture must change with the local neighborhood of the object, which means you can't use tiling textures very much. This means it would be very expensive to texture something like a big building, since you'd need unique texels everywhere.

Thankfully, the Western game development world has a tradition of developers being very open with their techniques and processes, and communicating with other developers very honestly about what is effective. (This is not something that one finds elsewhere in the world, and we should work hard to keep that spirit alive because it benefits everyone; but that's a totally separate rant.) Here's one good example of this kind of information exchange: at the GDC this year, some guys from Naughty Dog gave a talk on the art direction for Uncharted 2 (pdf link to their slides here.) Somewhere around page 220 of this very informative slideshow, they start talking about the blend shaders that they use in order to give more character to buildings and other geometry while still using tiling textures. The idea is that you interpolate between two (or more) textures with a blend function that fits the shape of your building nicely. You put some smarts into the interpolation so that it produces something that is visually interesting, rather than the blurry mess you would get via naive interpolation.

For The Witness it seemed like a good idea to use this kind of technique, but in a minimalist way, to create patterns that accentuate the shapes of buildings, and help tie them together visually, without stumbling into the realm of Ugly Texturing.

(There are a lot of reasons for the minimalism; one reason, though, is to provide clarity about what you can interact with, versus what is inert. Interactive things are colored in a way that makes them stand out in an obvious way. Think of it as something like what Mirror's Edge did, but in a different way.)

We've started experimenting with these blend shaders; here are some images:

The upper-left shows a scene before we implemented the blend shader for buildings. In the upper-right, we are using a blend shader that blends the original texture with an alternate one that is a bit darker and rougher (but the difference is a bit subtle, by game standards at least). Below these images are the same scenes with precomputed lighting mixed in.

It is pretty clear to me that the lighting is actually the biggest ingredient in tying the scene together, but the blend shader is, I think, a good extra kick. It helps the area feel like there is a little bit of life and history to it (for example, the lighter-colored paths near the puzzle panels in the foreground, which imply that people have been walking that way). What you see here is not of shipping quality, but I see how we can reach shipping quality by iterating on this style and putting production-level effort into the assets.

Here's a close-up of what the stairway looks like:

As you can see, the texture maps have some grain to them; from far away, the grain melts away and the structure is mostly a solid color, accented by lighting.

These experiments are still early yet but I find them very promising. The plan is that we will continue to hone the style and end up with something very special.

Using the same engine makes it very easy to keep the global illumination in sync with the rest of the game. If we add new shaders or materials, change the representation of the geometry or alter the world, keeping the global illumination up to date requires little o no effort. Another interesting benefit is that optimizations to the rendering engine also benefit the global illumination pre-computation. However, as we will see later, in practice each application stresses the system in different ways and end up limited by different bottlenecks, what improves the rendering time does not necessary improve the pre-computation and vice-versa.

Projection Methods

I tried several different approaches to render the scene and estimate the incoming radiance. The most common approach is to render hemicubes; that requires rendering the scene to 5 different viewports. Several sources suggested the use of spherical or parabolic projections, but in my experience these approaches produced terrible results due to the non linear projection. Without very high tessellation rates the scene would become unacceptably distorted, creating gaps between objects or making parallel surfaces self-intersect.

In GPU Gems 2, Coombe et al. suggested to project the z value independently of (x,y) coordinates using an orthographic projection in order to maintain a correct depth ordering. As seen in the next picture, that removed the most obvious artifacts; note how the staircase inside the tower is not visible anymore:

However, while this works OK from some points of view, the distortion becomes unacceptable when the camera is close to a surface, which is always our case. So, in practice these approaches are not very useful. Note how the outdoors are visible when sampling the lighting from the interior of the tower:

Other implementations use a single plane projection. This kind of projection does not capture the whole hemisphere, but by using a wide field of view it's possible to capture about 80% of it. I haven't actually tried this approach, but to get an idea of what the results would look like, I modified the hemicube integrator to ignore 20% of the samples near the horizon and found out that this introduced artifacts that were unacceptable for high-quality results.

Pascal Gautron estimates that on typical scenes this approach introduced a 18% RMS error, but suggests various tricks to reduce it by re-balancing the solid angle weights, or extrapolating the border samples and filling the holes. The latter method reduces the error down to a mere 6%.

However, these improvements do not eliminate banding artifacts caused when an intense light source visible from one location is suddenly not visible anymore in the next when it falls out of the projection frustum. To make things worse, the projection lacks rotational invariance, which enhances the problem when hemicubes are rendered with random rotations. So, while in many cases the results look alright, the few cases that do not render the method impractical.

Another problem is that the wide frustum is 4 times larger than the equivalent hemicube and easily intersects with the surrounding geometry; as I'll discuss next, this is one of the main problems that I had to deal with.

In the end I simply stayed with the traditional hemicubes for simplicity. I think it would have been possible to do somewhat better using a fancy tetrahedral or pyramidal projection with 3 or 4 planes, but that didn't seem worth the trouble.

Hemicube Overlaps

As I just mentioned, one of the main problems that I had to solve was to handle overlaps between the hemicubes and the geometry. These overlaps are unavoidable unless you impose some severe modeling restrictions that are not desirable for production.

When the geometry is well formed and does not have self-intersections, these overlaps can sometimes be avoided with centroid sampling. Without centroid sampling the location where the hemicube is rendered from can easily end up being outside of the polygon that contains the sample and thus inside another volume.

With centroid sampling the origin of the hemicube is now always inside the surface. However, it can still be very close to the surface boundary and have overlaps with nearby surfaces. A simple solution is to move the near plane closer to the camera until the overlap dissapears, but how close? The distance to the nearest boundary can be determined analytically, but there's a limit in how much the near plane distance can be reduced without introducing depth sampling artifacts. An interesting solution for this problem is to render multiple hemicubes at the same location, each hemicube enclosing the previous one with its far plane at the near plane of the next.

That would solve most the problems when the geometry is watertight and centroid sampling is used, but in many cases our assets are not always well conditioned. So, I had to find a more general solution.

I tried all sort of heuristics, to render backfaces in black, to fill them with the average color, to extrapolate the nearest colors, etc. None of these solutions worked well in all cases. In the end, something that produced satisfactory results was to render the backfaces with a special tag to distinguish them from front facing faces when integrating the hemicubes. Whenever a hemicube had too many tagged texels it would be discarded entirely, and instead its output radiance would be estimated at that location interpolating or extrapolating it from the nearest valid samples.

The extrapolation is not always correct, and the detection of invalid hemicubes is not 100% accurate, but so far this is the method that I'm happiest with. In the next article I'll explain how this extrapolation is done in more detail.

Shadow Rendering

While the goal was to use the same rendering engine, using its shadow system 'as is' would have been overkill. We are using cascade shadow maps, which are updated on every frame, and to maximize the shadow map resolution the shadow maps are tightly aligned to the view frustum slices. When rendering the hemicubes, shadow map resolution is not that important, and re-rendering the shadows for each hemicube face is unnecessarily expensive. To keep things simple I still used cascade shadow maps, but instead of rendering them for each view, I only rendered them once for each object positioning the cascades around the center of the object.

Rasterization

Now that I've explained how to capture the irradiance of the scene at any point, it's necessary to sample it for every lightmap texel. To do that I simply rasterize the geometry in the lightmap UV space using a conservative rasterizer and render one hemicube at every fragment whose coverage is above certain threshold.

One nice thing of using our own rasterizer is that I can evaluate the exact coverage analytically. Since the parameterization does not have overlaps, the coverage is just the area of the intersection of the texel square with the polygon being rasterized. Similarly, the centroid is the center of mass of the clipped polygon.

This approach gives us a position and a direction to render the hemicube from, but I still need to determine the orientation of the hemicube. Initially, I simply chose an arbitrary orientation using the traditional/standard method, but that resulted in banding artifacts:

The source of the problem is just the limited hemicube resolution, but it's surprisingly hard to get rid of it entirely by only increasing the hemicube size. A better solution is to simply use random orientations:

Note that I am trading banding artifacts by noise, but the latter is much easier to get rid of by using some smoothing.

Integration

Once the hemicubes have been rendered it's necessary to integrate the sampled irradiance to compute the output radiance. The integral of the irradiance hemicube is simply a weighted sum of the samples, where the weights are the cosine-weighted areas of the hemicube texels projected on the hemisphere, that is, the cosine-weighted solid angles of the texels. These weights are constant, so they can be precomputed in advance.

In the article by Hugo Helias that I referenced in the introduction the same approach is used and this weight table is called the "The Multiplier Map". Beware of that part of the article, though, because there's a bug in the formulas. The subtended solid angle of the texel is not "the cosine of the angle between the direction the camera is facing in, and the line from the camera to the pixel":

That term needs to be divided by the squared distance between the location of the camera x and the center of the texel y:

See formula 25 in the Global Illumination Compendium for more details.

Also, since this is a precomputation it made sense to evaluate the solid angle exactly. That can be derived using Girard's formula or directly using the handy formula in Manne Ohrstrom's thesis. Doing so improved the invariance of the integral with respect to rotations of the hemicube and that reduced noise significantly.

In the end, the integration is a fairly simple operation. It could be done very efficiently using compute shaders and I might still do that in the future, but for prototyping purposes it was much easier to perform the integration in the CPU. Unfortunately, that required transferring the data across the bus and, if not done properly, it severely reduced performance.

Asynchronous Memory Transfers

Contrary to popular belief, Direct3D 9 provides a mechanism to perform memory copies asynchronously. In Direct3D 10 and OpenGL this is done through staging resources or pixel buffer objects; in Direct3D 9 something similar can be achieved with offscreen plain surfaces. The problem is that this is not very well documented, it's very easy to fall out of the fast path, and the conditions in which that happens are slightly different for each IHV.

In the first place, you have to create two render targets and offscreen plain surfaces:

dev->CreateRenderTarget(..., &render_target_0);
dev->CreateRenderTarget(..., &render_target_1);
 
dev->CreateOffscreenPlainSurface(..., &offscreen_surface_0);
dev->CreateOffscreenPlainSurface(..., &offscreen_surface_1);

If you have more than two targets the copies become synchronous in some systems, but two is sufficient to generate enough work to do in parallel with the memory transfers.

The most important observation is that the only device method that appears to be asynchronous is GetRenderTargetData. After calling this method, any other device method blocks until the copy is complete. So, instead of overlapping the memory transfer with the rendering code, you have to find some CPU work to do simultaneously. In my case, I just do the integration of one of the render targets while copying the other:

while hemicubes left {
 
    [...] // Render hemicube to render_target_0
 
    // Initiate asynchronous copy to system memory.
    dev->GetRenderTargetData(render_target_0, offscreen_surface_0);
 
    // Lock offscreen_surface_1
    // at this point the corresponding async copy has finished already.
    offscreen_surface_1->LockRect(..., D3DLOCK_READONLY);
 
    [...] // Integrate hemicube.
 
    offscreen_surface_1->UnlockRect();
 
    swap(render_target_0, render_target_1);
    swap(offscreen_surface_0, offscreen_surface_1);
}

Another problem is that the buffer needs to be sufficiently big to reach transfer rates that saturate the available bandwidth. In order to achieve that I batch multiple hemicubes in a single texture atlas as seen in the following picture:

In practice I do not render the hemicubes using the depicted cross layout, but instead arrange the faces tightly in a 3x1 rectangle that does not waste any texture space and maximizes bandwidth. However, the cross layout was useful to visualize the output for debugging purposes.

Performance and Conclusions

The performance of our global illumination solution is somewhat disappointing. While we do most of the work on the GPU we are not really making a very effective use of it. The main GPU bottleneck is in the fixed function geometry pipeline; we are basically rendering a large number of tiny triangles on a small render target.

But that's not the main problem. Traversing the scene in the CPU and submitting commands to render it actually takes more time than it takes the GPU to process the commands and render the scene, so the GPU is actually underutilized.

Our engine was designed for quick prototyping, to make it very easy to draw stuff with different materials, but not to do so in the most efficient way possible. These inefficiencies became a lot more pronounced in the lightmap baker.

On the bright side, as we optimize the CPU side of the engine we expect performance to improve. Modern GPUs also have functionality that make it possible to render to multiple hemicube faces at once. We expect that will also reduce the CPU work significantly.

While it was easy to get our global illumination solution up and running by reusing our rendering engine, bringing it to a level of sufficient quality and performance took a considerable amount of time. In the next installment I'll describe some of the techniques and optimizations that I implemented to fix some of the remaining artifacts and improve performance.

Here's an example. The following tree looked OK close to the camera:

but as we moved away, the leafs started to fade and thin out:

until they almost disappeared:

I had never encountered this problem before, neither had I heard much about it, but after a few google searches I found out that artists are often frustrated by it and usually work around the issue using various hacks. These are some of the proposed solutions that are usually suggested:

• Manually adjusting contrast and sharpening the alpha channel of each mipmap in Photoshop.
• Scaling the alpha in the shader based on distance or on an lod factor estimated using texture gradients.
• Limiting the number of mipmaps, so that the lowest ones aren't used by the game.

These solutions may work around the problem in one way or another, but none of them is entirely correct and in some cases add a significant overhead.

In order to address the problem it's important to understand why the geometry fades out in the distance. That is because when computing alpha mipmaps using the standard algorithms, each mipmap has a different alpha test coverage. That is, the proportion of pixels that pass the alpha test changes, in most cases going down and causing the texture to become more transparent.

A simple solution to the problem is to find a scale factor that preserves the original alpha test coverage as best as possible. We define the coverage of the first mipmap as follows:

coverage = Sum(a_i > A_r) / N

where A_r is the alpha test value used in your application, a_i are the alpha values of the mipmap, and N is the number of texels in the mipmap. Then, for the following mipmaps you want to find a scale factor that causes the resulting coverage to stay the same:

Sum(scale * a_i > A_r) / N == coverage

However, finding this scale directly is tricky because it's a discrete problem, in general, there's no exact solution, and the range of scale is unbounded. Instead, what you can do is to find a new alpha reference value a_r that produces the desired coverage:

Sum(a_i > a_r) / N = coverage

This is much easier to solve, because a_r is bounded between 0 and 1. So, it's possible to use a simple bisection search to find the best solution. Once you know a_r the scale is simply:

scale = A_r / a_r

An implementation of this algorithm is publicly available in NVTT. The relevant code can be found in the following methods of the FloatImage class:

float FloatImage::alphaTestCoverage(float alphaRef, int alphaChannel) const;
void FloatImage::scaleAlphaToCoverage(float desiredCoverage, float alphaRef, int alphaChannel);


And here's a simple example of how this feature can be used through NVTT's public API:

// Output first mipmap.
context.compress(image, compressionOptions, outputOptions);

// Estimate original coverage.
const float coverage = image.alphaTestCoverage(A_r);

// Build mipmaps and scale alpha to preserve original coverage.
while (image.buildNextMipmap(nvtt::MipmapFilter_Kaiser))
{
    image.scaleAlphaToCoverage(coverage, A_r);
    context.compress(tmpImage, compressionOptions, outputOptions);
}

As seen in the following screenshot, this solves the problem entirely:

even when the trees are far away from the camera:

Note that this problem does not only show up when using alpha testing, but also when using alpha to coverage (as in these screenshots) or alpha blending in general. In those cases you don't have a specific alpha reference value, but this algorithm still works fine if you choose a value that is close to 1, since essentially what you want is to preserve the percentage of texels that are nearly opaque.

1. Use an external terrain editing tool and then write an importer for its file format
2. Build a terrain editing tool into the game's editor (this will require a bunch of work to do a good job, and will have fewer features than an external tool but they may be more appropriate for your game; also, you can make reasonable things happen to entities placed in the world when you interactively move the terrain around)
3. Paint heightmaps in an image editor like Photoshop and hotload those from the game engine (requires much less programming; doesn't provide many features, but you have very fine control over the results; unfortunately, the editing process becomes pretty cumbersome)

These seemed to be the main approaches on offer.  However, I wasn't enthusiastic about doing any of the traditional things; I have written a number of terrain systems for games, and what bugged me, with respect to this game, was the way in which terrain handling tends to be very different from laying out objects in the world, and how this creates an extremely disjoint work process.

In concrete terms, this means that terrain was not going to be as malleable as the other objects in the world.  I am taking a very rough-drafty approach to the game; we are building areas, then moving them across the world if we feel they will work better somewhere else, rotating them as a group, etc.  The goal is to build a very-cohesive game world in which carefully-tuned gameplay is the most important thing -- where we more-or-less have a rough draft of the whole world set up before we go in and try to get serious about how everything looks.

This kind of approach just dies when you have elements that behave very differently, and are difficult to manipulate together, such as terrain and objects.  If I have two hills with a path winding between them, a bench by the path with a book upon the bench, and a signpost nearby, what happens when I want to move that scene across the map and then rotate it by 30 degrees?  With the entities, it is no problem, we just do that in the editor.  But with the terrain it just becomes a mess to move the hills at all, regardless of which of the above approaches are used.  And if you want to move and rotate the hills in sync with the entities so that the whole scene is intact... good luck!

To solve this problem, terrain needed to be built in the in-game editor, in a way that would be maximally malleable -- so that I can pick up a hill and rotate and drag it.  Because of the requirement for rotation, you can't really be manipulating samples directly; it has to be control surfaces of some kind (because if you are just blitting samples around, a few rotation operations are going to mangle whatever original shapes you made).

As for control surfaces, the first ideas that come to mind are things like spline patches.  But the problem with spline patches is that you have to be very careful managing the control points in a patch, and they ultimately aren't that malleable -- what if I want to split a patch in half without changing the shapes of areas near the edges?  What if I want to move one small, high-detail terrain area so that it overlaps somewhat with a big, low-detail area?  How is that going to work? And if I have spline patches, then any time I instantiate a new grid of patches, I have to manage a bunch of individual control points, which is labor-intensive.  If I am rough-drafting, I want to do the minimum possible work to lay out a given area.  If I just generally want to build a hill but don't care much about its shape, I ought to be able to do that with just 2 or 3 control points, arbitrarily-placed.

Clearly what I wanted was a system that would just look at a set of unstructured control points -- no topology or anything -- and generate a terrain from those points in the local area.  When there is no topology or parameterization, you can merge and split groups of control points at will; you can have them overlap; you can do whatever you want.

I started putting together an ad hoc scheme for doing this, but did not take it very far before discovering that the mathematics for doing a good job at exactly this problem had already been invented decades ago.  The method is called Kriging, and it is well known in geostatistics.  If you've got a survey of height field data (or any other field data -- the density of a mineral at certain points, for instance), and that data is missing values for some locations, Kriging will give you a reasonable guess at the values of the field at the unknown points.

For rough-draft terrain modeling, we can turn the idea around; if we give a Kriging function only a small set of known data points, it will do a quite fantastic job of filling in the values at all the other points.  The basic math involves the inverse of an n by n matrix, where n is the number of known data points. In theory that could become a huge computation, but for the number of points we may typically have in one local area of the map, the computation is easy for modern CPUs. Kriging is not hard to implement; I started with some public domain code written at a university and modified it for our needs (though this basically involved rewriting it, as it was messy and had some serious bugs!  If anyone wants our version of the Kriging code, drop me a line).

Here's how we have it integrated into our game engine.  There's a type of object that exists as a set of terrain control points, which can be moved around the world arbitrarily, just like anything else.  It looks like this:

[See image gallery at the-witness.net]

The only meaningful data in this object, with respect to terrain shaping, are the positions of the control points (purple boxes).  The surface is just a visualization; it tells you what surface these points would define if they were the only control points in the universe.  (Though actually, the surface defined by those points would extend infinitely in all directions; this visualization stays within the convex hull of the control points, which helps you see what you are doing when editing).

Right now these control points haven't actually been applied to the game's terrain (green surface below); they are just hovering above it.  Here's a shot from another angle:

[See image gallery at the-witness.net]

I can press a key to add a new control point to this object:

[See image gallery at the-witness.net]

The new control point doesn't affect the shape of the old surface in any way, which is very convenient. I can drag the new control point at will:

[See image gallery at the-witness.net]

Or add many more control points:

[See image gallery at the-witness.net]

Here's that same visualization surface, with the control points and gridding turned off, so you can see it better:

[See image gallery at the-witness.net]

Then, when I decide I like the shape, I can press another key to apply these control points to the game's terrain:

[See image gallery at the-witness.net]

(Now there's a lot of z-fighting between the control surface and the terrain because they are in pretty much the same place.)

Here is the terrain with the control surface hidden.  Note that the terrain values have been smoothly interpolated, even in areas outside the control object:

[See image gallery at the-witness.net]

(By the way, those seams you see in the previous two pictures are lighting seams.  These don't have anything to do with Kriging, it's just that the terrain blocks don't know how to look up their neighbors' height values when generating their surface normals.  Any block-based terrain system has to solve a similar problem.  It's easy to solve, but at a low priority on the to-do list, hence it hasn't been done yet.  The distance between seams tells you how big the terrain blocks are, so the seams are sort of useful in this image!)

Suppose this control surface is part of a scene that I want to reorient.  I'll rotate it about 70 degrees clockwise (now you'll see the surface intercutting more-heavily with the existing terrain values):

[See image gallery at the-witness.net]

And then once again, I apply Kriging to the terrain, generating the same hills in a rotated orientation:

[See image gallery at the-witness.net]

To illustrate the way that interpolation is not confined to a single control object, and that in fact this is a control point soup, here I've created three control objects:

[See image gallery at the-witness.net]

And here's what the terrain looks like with them applied:

[See image gallery at the-witness.net]

And here's the final terrain along with the control objects (lots of z-fighting again):

[See image gallery at the-witness.net]

Hopefully that's enough screenshots to illustrate the point.  Compared to splines, Kriging has the following highly beneficial properties (aside from the huge benefits of not requiring a topology, discussed above):

• Always interpolates every control point exactly
• Adding a new control point does not change the existing surface at all.
• No funny spline-like behavior (curves appearing in strange places, huge overshoots, etc)

Hacking In Compact Support

When doing Kriging in a video game, you probably don't want to compute the terrain for the entire world at once for every change, because that would be very expensive, and it also is a little bit unintuitive (since a change to a control point very far away could change some interpolated terrain nearby, even if only slightly).  However, the vanilla Kriging algorithm would require global computation and thus behave nonlocally.  For The Witness we clearly wanted our control points to affect only nearby areas, i.e. have compact support. My first attempt at this was:

• Pick a radius r0 at which any control point has maximum effect, and a larger radius r1 at which the point has no effect, with smooth interpolation in-between.
• Terrain consists of contiguous blocks 33x33 samples in size.
• For each block of terrain you want to Krige, find all control points whose distance in the XY plane from any sample in the block is less than r1 (these are all the control points that could possibly affect any sample in the block).
• Pass all these points to the Kriging routine, which computes every sample in the block.  It does this by building an inverse variogram matrix -- square with dimension (n+1) -- and performing one matrix-vector multiply for each terrain sample, which is pretty fast.
• For each sample, the Kriging routine gives us the set of weights we would use to blend each control point if the control points did not have compact support.  To provide compact support, just compute an additional factor that is (distance - r1) / (r0 - r1) clamped to [0, 1], multiply all the weights by these factors, and then re-normalize the weights (ensure they all add to 1).

This almost works, but not quite; we get discontinuities between terrain blocks (which should not happen if everything is correct, since two vertices at the same XY position should always get the same Z value). At first I thought this was a programming error, but after looking more closely at the situation I convinced myself that the reason is mathematical.  My assumption had been that a control point with a weight of 0 will have the same effect as a control point that never enters into the calculation, but this seems not to be true.

The step that I added, multiplying the compact-support factors against the weights provided by the matrix-vector multiply, is -- by the distributive property -- the same as multiplying each row of the matrix by the support factor before doing the matrix-vector multiply.  If a support factor is 0, then this makes the matrix rank-deficient, i.e. it could not have been the result of an inversion, which means that this step is not compatible with the mathematical technique being applied.  The idea of a 0 blend factor just isn't valid.  (This also brings into question whether the idea of multiplying by the support factors is valid at all, but for now I will just observe that it produces very useful and convenient behavior.  However, there are some small stepping artifacts that I see in some locations.  I think these may be due to having too-low a density of control points in the world (some terrain samples having 0 control points inside r1) but there's a possibility that they aren't, and have something to do with the weighting procedure itself.  I haven't investigated yet.)

Now, as I type this, I am not 100% convinced that it is the right reasoning, and that something more like a typical programming error might be the problem.  My math is pretty rusty these days.  However, my updated version of the routine does not have discontinuities.  

Unfortunately, the fix for discontinuities has made the Kriging a bit slower.  What I do is, instead of building just one inverse variogram for the entire terrain block, I build a new variogram for every vertex, and include in that variogram only the control points that have nonzero support factors.  Thus the dimensions of the matrix will change from sample to sample, and I am inverting a new matrix for each output sample.

Unoptimized, this version of the algorithm takes between 5 and 25 seconds to recompute the terrain for the entire world, depending on how large r1 is.  This is fine for now, especially since editing operations only hit about 1/50 of the world or less at any given time.  However, it'll get slower as more control points are added to the world.

Fortunately, this first version of the discontinuity-fixing was intended to be slow -- I programmed it in the most straightforward way possible, to maximize the likelihood that it would work.  It is easy to think of ways to make it faster by caching the matrices, but it's unclear which of these would be fastest.  For example, if you just cache one copy of the matrix and invalidate it any time a new control point comes into or out of range, then you might easily find yourself in a situation where you oscillate between recomputing two different matrices as you work your way along the sample grid.  This is a little reminiscent of the kinds of problems that 3D hardware has, which store images in a non-linear way so that they can be cache-efficient when sampled along lines in arbitrary directions... but it's not clear to me how far to take that analogy or whether any applicable wisdom can be derived from it.

So I intend to optimize it, or come up with a faster solution to the discontinuity problem, but it's not clear yet what form that will take.

This time I'll show how to reduce the memory usage by 25% on machines that support non-power-of-two textures; or, on machines that don't, how to fit 5 shadow maps in the space previously used for 4.

Valient's shadow system generates very stable shadow maps, but this costs a lot of texture memory.  Recall that so much memory is needed because you are inscribing your frustum slice inside a circle, and then inscribing that circle in the texture map:

When implementing cascading shadow maps, you will be using more than one frustum slice: say, four or five.  Valient's ShaderX 6 paper suggests a way to sample from four different shadow slices quickly in the shader by packing them into an atlas, perhaps like this:

This was the way we implemented it at first, but very soon after that I was desiring a little more versatility.  If your graphics hardware will do non-power-of-two textures, then you may wish to have the number of shadow slices be an adjustable property; but if the texture maps are packed into a square like that, it's hard to know where to put a fifth one.  So we changed the atlas layout to this:

This still works on power-of-two hardware (power-of-two textures don't have to be square!), and on more-versatile hardware, we can now easily add new textures onto the end.

When your atlas is laid out in a line like this, and you add some runtime visualization to your game, it becomes very clear how much texture space is going unused.  Here's a shot from The Witness's shadow maps:

The frustum slices are shown in green.  Obviously there is a lot of empty space between them.  (For the purposes of visualization, we have left the pixels in the "empty space" filled with shadow map data, but anything that is not covered by green will never be visible in-game, so the game will run faster if we cull all that stuff!)

Because there's so much empty space, we could move our shadow maps closer together, and use less memory.  But we need to be careful about it, because the whole point of the algorithm is that we need enough shadow map space to handle the worst case.  Indeed, if you turn the camera a bit, you get this:

Now there is very little space between the maps, so you can't squeeze them together.  Instead, all the space is toward the bottom of the atlas, and it is unclear how to use it effectively.

But there's a trick we can apply.  Even though these shadow maps are all packed into one atlas, they are still indexed as separate units in the shader, which means we can transform them as separate units.  So why don't we just rotate each individual shadow map by 90 degrees?  Here's what happens to the image above when you do that:

Now there is plenty of horizontal space and we can pack the maps all together!  (Of course we don't need to render our shadow maps at the default orientation and then copy-and-rotate them in a post-pass; we can just rotate the up-vector of the camera used to render the shadow maps, so that we get the rotation for free.)

When we rotate the shadow maps like this, in order to render properly, our shader needs to realize that they are rotated; but Valient's system already provided a way to transform the shadow map coordinates based on map index, so rotating the maps fits directly into that scheme.  Furthermore, if we are using any filter offsets to index into our shadow map, we need to ensure that those offsets get rotated (by either rotating the constants before uploading them to the shader, or by adding them before doing the map-to-atlas transform, depending on how your system is implemented).

So how do we figure out how to pack the frustum slices together?  Our implementation slides them to the left as far as we can make them go.  First, we compute bounding rectangles for the frustum slices (bounding rectangles shown in yellow):

These rectangles are all oriented in the same direction, so we can just slide them to the left until they nearly touch:

We figure out the rectangle orientation by performing a brute-force uv-bounding computation on one of the frustum slices, then using that same orientation for all slices.  By "brute-force uv-bounding computation" I mean that we just try a whole bunch of different angles and see which one fits the tightest.  In the current implementation we try 100 different orientations spread over 90 degrees, so we find the best angle to within less than 1 degree of accuracy.  (You only need to try 90 degrees, because at that point every angle has been covered by at least one of the sides of your rectangle).

The initial instinct was to use Shadow Map 0 to compute the rectangle orientation, but this didn't quite work out; because of the extreme pointiness of the first frustum slice, we sometimes end up with bounding rectangles that do not suit the other maps.  Using any of the other shadow maps solves this problem (because their near plane tends to create a coherent bounding edge).

We're not done moving the shadow maps around.  Besides just shifting the later shadow maps leftward, as above, we save some more space by shifting the first shadow map leftward based on the leftmost vertex in its frustum slice.  (We don't do it based on the yellow rectangle, because that rectangle will almost always be poking outside of the atlas already.  We don't need to make sure the whole yellow rectangle is inside the atlas; just the green frustum slice!  Remember that the rectangles are just a tool to help us know how to pack.)  So then we get this:

And we can shave off even more space by shifting the rightmost map upward or downward, whichever direction will allow it to slide further to the left (in this case, it's downward:)

Clearly we've saved a lot of space -- about half for this viewing angle. But we have to allocate enough texture space for the worst case, and the question is, what is that worst case? Empirically, for our game's FOV, we find that in this worst-case viewing angle we can chop off 25% of the texture map.

If you're on non-power-of-two texture hardware, you can just cut your map down and you are happy.  If you are power-of-two-only, you can't do that, but you could use five frustum slices in the space that previously held only four.

Dealing with Overlapping

You can pack these rectangular areas together however you want, as long as they don't overlap -- at scene rendering time, your shader doesn't know or care how close these regions are to each other.  Actually building the shadow maps in the first place has become a bit trickier, though.  When each shadow map was in its own self-contained square region, we would just set up a viewport for that square, erase the contents, and render into it.  But now, suppose we are rendering the shadow maps from left to right: while we are rendering the second shadow map, its viewport is going to inevitably overlap the first shadow map.  We have to be careful not to write into the valid region of that first shadow map, or else we'll mess up the scene.

In our current implementation, we use DirectX9's API-provided clipping planes for this purpose.  (The long sides of the yellow rectangles tell us exactly where to put our clipping planes. )  Clipping planes are deprecated in future versions of the API, so we would either switch to stencil testing or else do the clipping ourselves in the shaders.

Here's what the neighborhood of the worst case looks like:

It happens when you are looking sort-of in the direction of the light vector, but you are turned enough to the side that the view frustum has rotated to be diagonal relative to the axes of the texture map space.  (If you are pointing directly along the light vector, your view frustum extrema are likely to be parallel to the axes of the texture map space.)

Deeper Thinking We Have Not Yet Done

We've saved this texture memory by doing what are essentially image-space operations.  But it seems to me that one could somehow project these operations back to the 3D space that we use to compute the bounding spheres in the first place, and generate smaller spheres.  The operation of rotating our texture maps means that we are folding our output space over on itself, using only half the possible orientations.  The whole point of the bounding sphere is to permit the use of any orientation at all -- which we clearly, in retrospect, do not need.  So it's probably possible to reformulate this algorithm in terms of a bounding hemisphere of some kind (or some shape that is somewhat near a hemisphere but allows frustum corner protrusions to poke out of it).

Saving Tremendously More Memory, if you are willing to copy your maps or else render them in multiple pieces...

The reason our system works is that once the shadow maps are packed into an atlas, the shader doesn't know or care where the boundaries of the shadow maps are.  We can address the atlas at will, because in reality it is just one continuous texture map.  This means that we can go even further, and place some of our shadow maps in the atlas in such a way that they cross the boundaries of the atlas and wrap to the other side.  It looks discontinuous, but the shader being run at scene-render time does not care, if the texture is addressed in a wrapping mode.  Here is a messy mock-up (not the output from an actual running version, because we haven't implemented this):

Unwrapped, easy-to-see version of texture space:

Wrapped vertically:

Wrapped horizontally:

In my done-by-hand mockup, the result is 46% the size required by the default packing in Valient's original algorithm.  That's a lot of memory savings.

The catch is that, while it's easy to employ this mapping without additional expense, it's more expensive to generate.  You have to either render a non-wrapped version of each shadow map and then copy it into its multiple destination pieces, or else you have to render each map in multiple passes.  (It's always possible to arrange your maps so that one of them is unsplit, so for 4 frustum slices, you probably need 3 additional passes.  Each pass has fewer objects, though, so maybe it's not too bad.)

If you are using pre-filtered shadow maps, then the filtering step is just a kind of copy anyway, so you would be able to do this packing essentially for free.  We aren't doing any pre-filtering on The Witness, but if we do, it is likely we will switch over to this packing scheme.  The memory savings are so huge, how could we not?

But I'd like to reiterate for clarity that at this time we have not implemented this wrapping-shadow-map scheme, so if there are any hidden gotchas, I don't know.  We have only implemented the above 25% savings scheme (sliding the maps around, no wrapping).

I have developed some interesting technology to compute automatic paramaterizations, to formulate the lighting computations in a way that can be easily mapped to the GPU, and to speed them up performing them at a lower frequency by using irradiance caching. None of these things are very cutting edge, but I run into some interesting problems that are usually overlooked in the literature and came up with some creative solutions that I think would be interesting to share. However, in retrospect, I think I also made some mistakes, but I hope that being honest about them will help me avoid them in the future.

In this first post I will describe the lightmap parameterization algorithm that I have implemented and in the next few posts I'll cover other details of our lighting system such as our approach to perform final gathering using hemicubes and our implementation of irradiance caching.

In order to use precomputed lighting we need a way of mapping these lighting computations to the surfaces. One possibility is to store them per vertex, but in many cases that does not provide enough resolution and the interpolation of the lighting makes the triangulation of the mesh fairly obvious. Many games get away with that by compensating the lack of detail in the lighting with high texture and normal map detail, but that does not go along well with the graphical style that Jon wants to achieve. So, instead of that we decided to create full parameterizations of our objects in order to store the precomputed lighting into lightmap textures.

Creating these parameterizations by hand is possible, but it is time consuming and I'd rather have our artists work on something more enjoyable. So, we decided to create the parameterizations automatically. However, the greatest mistake I've made so far is to not look more carefully at the available solutions to accomplish that. I had some pieces of the implementation already available, and I overestimated how much effort it would take to put them together and produce robust results. While talking about this to some friends at GDC one of them asked why I wasn't using Microsoft's UVAtlas tool, and to my dismay and embarrassment my only answer was that it had not occurred to me. I think that goes to prove how valuable it is to be open about our development in a small team like ours.

Now that I got that out of the way, let's talk about the tech. We only use lightmaps to represent indirect illumination and the direct illumination of static area lights. So, our expectation was that lightmaps would be fairly smooth and that seams would not be a huge problem. For this reason we decided not to invest in a solution that guaranteed seamless parameterizations and instead used a fairly traditional approach: We first divide the input geometry into charts with disk topology, parameterize them individually, and finally pack them into a rectangular texture.

Segmentation

Dividing the mesh into charts is by far the most complex step of the process. In the past I had tried using hierarchical clustering without much success, so this time around I decided to use an iterative clustering algorithm similar to Lloyd's algorithm, in the same vein as the algorithm used in the Multi-chart geometry images (MCGI) paper.

Like Lloyd's clustering algorithm, the MCGI algorithm iterates between chart growing and reseeding stages. The growth of the charts is guided by a metric based on the normal deviation and the distance between the face centroids. After each chart growing iteration a new seed is selected taking the most central face of the chart, and the process is repeated until the seed location does not change anymore.

My implementation of the MCGI algorithm worked moderately well, but it didn't always produce good results. It was possible to tweak the number and location of the seeds to obtain the desired output, but that required a certain amount of user intervention that was not desired.

Part of the problem is that the meaning of "good" in this setting is very subjective. It usually means nicely shaped charts, that fit well to the surface, produce parameterizations with little distortion, have fairly straight and convex boundaries and are more or less equaly sized.

In order to adapt the algorithm to our purposes I did two things, modified the bootstraping method so that the initial number of seeds is chosen automatically and used a combination of metrics to obtain better control over the results.

In order to select the initial number of seeds, I simply insert the seeds one by one, and grow the respective charts until a threshold is met. At that point I pick an arbitrary face that is not assigned to any chart yet and repeat the process. When no faces are left, the growing process is restarted from all seeds in parallel.

A common strategy to remove the sensitivity to the initial conditions is to use the farthest point heuristic. However, instead of doing that I found that an approach that works well is to pick the worst candidate face of the already chosen charts as the next seed.

The second important change to the algorithm was to use a different set of metrics. The MCGI paper proposes a metric that combines normal deviation with distance between face centroids. I found that using absolute distances made the algorithm very dependent on the tessellation of the mesh and that measuring the distances from the centroids favored triangles with low aspect ratios.

Instead of using distance between faces I considered only the normal deviation and combined it with other metrics similar to the ones proposed in D-Charts: Quasi-Developable Mesh Segmentation and that take roundness and straightness into account.

These metrics also had their own problems and I had to tweak them in various ways. The most significant issue is that they constrained the chart growth in undesired ways. For example, to encourage chart convexity you would associate a high factor to the corresponding metric, but that would result in a large number of almost convex charts, as soon as a convex configuration was found no additional faces could be added to the chart. So, instead of using these metrics to constrain the growth I only use them to favor convex charts or straight boundaries; that allowed the charts to grow freely, but still retain the most convex and most straight shape possible.

A trick that helped in many cases was to incorporate some input from the artist, but without imposing any additional burden to them. A simple way of doing that is taking into account the existing parameterization of the mesh. Our objects already have a parameterization that is used for texturing and the artists tend to place texture seams at locations that are not easily visible. Snapping the boundaries of our charts to the existing seams did not only reduce the overall number of seams, but also caused the charts to evolve in a way that matches better what the artist would have done manually.

I considered texture and normal seams independently. Since our parameterization is used for lighting I follow normal seams more strictly and proportionally to their angle, not doing so would cause the corners to appear less defined or more blurry. That also helps the process to converge much more quickly in architectural meshes that have many hard edges.

Finally, I don't impose any chart topology constrains. I specially wanted charts go grow around holes in the mesh as long as the metric was below the desired threshold. I was tempted to use a better proxy fitting metric instead of the normal deviation metric, but that would have generated cylindrical or conical charts that would have to be cut, and I did not want to deal with that. Closing holes on the other side is fairly easy, it's just matter of detecting the boundaries and closing all, but the longest one.

Parameterization

For the chart parameterization algorithm we simply use Least Squares Conformal Maps since I had an implementation already available. My implementation is based on the description of the algorithm in the notes of the SIGGRAPH 2007 Mesh Parameterization course, which I found much more accessible than the one in the original paper. Particularly useful is the chapter about numerical optimization, although if I had to write a sparse matrix solver again I would probably use OpenNL, since it has now been released under a liberal BSD license.

LSCM is not the best parameterization algorithm out there. It worked pretty well in most cases, although in some rare situations it resulted in small self-overlaps. That happened in two cases:

• Zero area triangles in the input sometimes flipped in the parameterization.
• Charts with complex boundaries that self-intersect when parameterized.

The first case turned out that was caused by T-junctions in the input mesh that were detected as holes and filled with a degenerate triangle. In practice the small overlaps in the parameterization were not really an issue so instead of fixing the input we simply plan to report these problems to the artist in the exporter and have him fix the them if it turns out it's causing artifacts.

In the second case, increasing the weight of the convexity metric used during the clustering process allowed us to prevent these errors. So, I didn't really do anything to handle this problem, but so far it has not been an issue.

Even after that we had some problematic cases remaining. I noticed that the implementation in Blender was not having this same issues and was handling these cases more gracefully. So, I asked Brecht Van Lommel, who suggested the use of alternate matrix weights based on the ABF formulation of LSCM, and that seemed to eliminate the remaining issues.

While LSCM is very easy to implement, it's definitely not the best parameterization method out there. If I were to implement something better I would probably look at LinABF or Fitting to Guidance Gradient Fields. However, since parameterization of the individual charts is currently the fastest step it may make sense to also look at more expensive deformation analysis methods.

Packing

Our packing algorithm is fairly simple. Like many other implementations we first sort the charts by parametric area, rotate them to align the charts with their best fit rectangle and greedily introduce them in the atlas one by one. Ideally I should test all 4 possible orientations, but currently I simply pick one of the orientations randomly.

Instead of using a tetris-like scheme to find the best location for each chart, I use a more brute force approach in which all the possible locations are considered. I achieve that using a conservative rasterizer that tags all the texels touched by a chart, plus an extra texel for padding. I progressively increase the texture extents as new charts are inserted. If the a location within the current extents is found, the chart is automatically accepted, otherwise a metric is evaluated to determine the location that least increases the extents of the atlas. Since I use a greedy algorithm, minimizing area resulted in atlases with very high aspect ratios. So, instead I minimize a metric that combines extent and perimeter and results in mostly square atlases.

Here are some examples of the results:

Once we use lightmap compression it will probably make sense to align the charts to DXT block boundaries in order to prevent bleeding artifacts.

Future Improvements

This is one of the areas where I think there's a lot of room for improvement, but at the same time, there's not much !/$ in spending more time improving the results. Even though it has not been tested in production, I think that our current solution works fairly well. It would be nice to have larger charts with less distortion or a lower number of seams, and I would certainly have fun working on that. However, that's not going to make the game run significantly faster or look dramatically better. We plan to release the source code of these algorithms at some point so my hope is that the implementation will progressively get better as other people adopt it and improve it to fit their needs.

That said, if there's one area where I think it's worth investing, that is in the use of signal specialized parameterizations. The research papers on the subject (Signal-specialized parametrization and Signal-specialized parameterization for piecewise linear reconstruction) are probably overkill in our setting, and I'm not sure they would work very well with our low resolution meshes.

Half-Life 2 achieved that that by using artist specified sampling rates for each surface, but I think it's worth doing that automatically using an approach like the one proposed by Bungie, which seems fairly simple and provides most of the gains the more complex approaches would give us.

My expectation is that the current implementation won't suffer significant changes, other than small tweaks and slight adjustments to deal with problematic cases that may show up in the future. If someone has any experience with these kind of algorithms, I would love to hear about it. In particular I'd like to know if there's anything you would have done differently, or if there's any optimization or improvement that I may have overlooked and you think is worth considering.

First, I've taken shots from 3 positions inside a house, with four different lighting settings.  For all four settings, we are computing only the first lighting bounce.  The parameters are:

1. 32x32 cube maps, adaptive sampling, quality = 0.5.  Time to compute: 197 seconds.
2. 32x32 cube maps, non-adaptive sampling.  Time to compute: 648 seconds.
3. 64x64 cube maps, non-adaptive sampling.  Time to compute: 738 seconds.
4. 128x128 cube maps, non-adaptive sampling.  Time to compute: 1570 seconds.

(Click for full size).

Front Door:

32x32 adaptive

32x32

64x64

128x128

Corner:

32x32 adaptive

32x32

64x64

128x128

Side Wall:

32x32 adaptive

32x32

64x64

128x128

The hardware used to compute these timings is an Intel Core i7-860 with an ATI Radeon 5870 (most of the process is GPU-bound).

The non-adaptive sampling renders one cube map for each shadow map texel, whereas the adaptive sampling uses heuristics to place samples and then smoothes the map between them.  At higher quality settings, adaptive sampling provides more-accurate results, but there always seem to be artifacts, so for the time being we are using it as a fast preview mode.  If at some point the quality becomes shippable, then we would switch to adaptive sampling for everything.

We also have a sub-texel sampling mode that improves quality, but I didn't play with that during this run.

This second set of images shows progressive bounces of the lighting with 64x64 cube maps, non-adaptive sampling.  (The first one is the same 64x64 image as above).  The times are: 1 bounce, 738 seconds; 2 bounces, 1438 seconds; 3 bounces, 2481 seconds.  The precompute time is basically linear in the number of bounces, as one might expect (the algorithm is basically just doing the same thing n times.)

1 bounce

2 bounces

3 bounces

]]> https://the-witness.net/news/2010/03/graphics-tech-lighting-comparison/feed/ 10 https://the-witness.net/news/2010/03/graphics-tech-shadow-maps-part-1/ https://the-witness.net/news/2010/03/graphics-tech-shadow-maps-part-1/#comments Wed, 03 Mar 2010 18:22:04 +0000 https://the-witness.net/news/?p=77 Continue reading

For The Witness, I wanted to develop a graphical style that values simplicity.   It would be suited to mellower environments, with both indoor and outdoor settings.  Some kind of global illumination solution seemed like the right idea here.  (Global illumination is when you simulate light bouncing around in a scene; the resulting look is usually much richer and subtler than the direct lighting that games usually do.)

Clearly, full real-time global illumination would be the most versatile solution.  I investigated some licensable packages that provide this, such as Geomerics' SDK.  However, these solutions will invariably use a lot of texture space, and consume a bunch of processing time, and they seem like overkill for this game (The Witness does not have many moving light sources, relative to other types of games).

So, some form of precomputed global illumination seemed like the right thing.  3D modeling packages have plugins to compute global illumination, but they are very difficult to interface with, and they could not even come close to handling a full game scene.   The only thing that knows where all the objects in the world are at once is the game itself (and the in-game editor), so it seemed appropriate to develop our own in-game system for global illumination.  It could have been radiosity, could have been something like Monte Carlo Path Tracing, but Charles Bloom suggested the very simple solution of just rendering the scene for every pixel in a lightmap, and that seemed like a good idea.  With an approach like that, you don't have to do a bunch of monkeying around to make radiosity or ray tracing match the completely different algorithm that is used to render your realtime game scene; light transport is computed by that same algorithm, so it will automatically match unless you mess things up.

For several months, Ignacio Castaño has been working on this system.  He is writing up a highly technical explanation of how the system works, which I think he'll be done with in a few days, but in the meantime, here is a lighter-weight overview.

At preprocess time, we just walk a camera over every surface, pointing the camera away from the surface and rendering the game scene into a hemicube in video memory.  The render target textures of a hemicube are packed together into an atlas, and when the atlas fills up, we download it into system memory, integrate the values in each hemicube to find an average light value, and then store that value in the lightmap.  Most of the shots below were generated using 128x128 cube maps and multiple samples per texel.

Because we are just using our regular rendering code, this precomputation process is hardware-accelerated by default.

Note: In the images below, all geometry and textures are placeholders. As with Braid, we are using rough-draft versions of all these things while the core gameplay is built.  Bloggers, these are not to be considered screenshots of the game.  They aren't supposed to look good yet -- they are just supposed to show the precomputed lighting.

First, some shots of the house.  The precomputed lighting is responsible for most of the light in the room.  The places where you see sunlight hitting surfaces directly are dynamic; everything else is a static precompute.  (Without the precomputed lighting, the entire room would be black except for the patches of sunlight).

Note the ambient occlusion, soft shadows, and other nice effects that all come naturally from this system. (In the third image, the picture on the wall hasn't been lightmapped yet, which is why it sticks out so brightly.)

Here's a simple bunker interior consisting mostly of a single material and color.

Lastly, the tower:

There are lots of caveats on these images.  The game renders in HDR, but we haven't balanced our lighting constants or done tone mapping yet, so I have gamma-corrected the output images by various amounts.  For now, we are computing only the first lighting bounce.  We'll probably do at least 2 or 3 bounces for the final game, and when we have that running I'll post some comparison images.  I believe our model of sky illumination is still overly simple and wrong (though I'd need to check with Ignacio on that, maybe he has fixed it up).  I haven't played around with the results of the current system in outdoor areas (e.g. lots of small branches and leaves), though if we find that we have too many aliasing problems, we can always make the lightmaps higher-resolution in those areas.

In areas that have doors or windows that open and close, we are planning on layering a supplemental lightmap that gets scaled by the openness of that light aperture, then added to the base lightmap.  This will provide a pretty good approximation for the way the indirect light in the room changes as the aperture opens and closes.  Direct light will be completely correct since it uses the fully dynamic shadow system.  I'll do a post about that shadow system sometime in the future.