Here’s hoping you’re using a 1.6/1.7 client, and enjoy the merge conflict reduction I’ve seen.

Cheers.

Very cool, I really like the deprecation system. Thanks for the info! I might tromp around your blog again if I have more questions, or do you respond to email?

We define a couple of macros to make it simple, so it looks like…

void Entity_Type_Terrain::register_metadata() {
NULL_PROXY(e, Entity_Terrain);

metadata = make_entity_metadata();

Metadata_Item *item;
item = Item(e, texture_name);

item = Item(e, normal_map_name);
item->minimum_revision_number = 25;
item->flags |= Metadata_Item::ADJUSTABLE_WITHOUT_RECREATE;

item = Item(e, map_size);
item->minimum_revision_number = 63;
item->maximum_revision_number = 85;
item->flags |= Metadata_Item::DO_NOT_DISPLAY_IN_UI;

…

The ‘NULL_PROXY’ macro just declares ‘e’ as an Entity_Terrain and sets it up in the right way. It’s an idiom that only exists in that file that you don’t really have to think about, apart from ensuring it’s at the top of the function.

Ah, interesting. I hadn’t thought about the compatibility issues. So essentially you have to manually update a map of strings to variables (or something to that effect)? That seems manageable. I can look up more info on member fields and the like.

Also when I said “entities in code” I did mean in memory, heh. Whoops. But cool, thanks for the info.

The editor is editing the entities that are loaded into memory. Then when you press the Save button, the entities are serialized into both the text and binary files.

Serialization is pretty straightforward, it is reflection that is involved. Since C++ does not provide a good way to do this, we create a function for each entity type that registers its members. But this is actually good, because this is also how we handle compatibility as entities change. When you register a member field of an entity, you can specify the per-entity-type revision number at which the field started being valid, or at which it stopped being valid. If you try to use automated reflection provided by the programming language, there isn’t an obvious way to do this, since if you delete a field it is just gone. (You then probably tend toward a file format that is a little more self-descriptive, though that will be somewhat less space-efficient).

I’m relatively new to building game engines (so forgive my ignorance) and I have some questions about editing game data. I see from this post that you have the ability to save and load from both binary and text files. My main question: what, specifically, is the editor actually editing? Is it editing the binary package, the text files, or is it directly editing the entities themselves in code? If it’s the last one (I suspect that it is), at what point are you saving out to files? Is there a specific save function or is it automatic?

Finally, in any case, are there any resources you could point me to on implementing robust serialization and/or reflection in c++? I come from a C# background (don’t judge me! :) ) where reflection is trivial, so I’m less familiar with pulling together an entity’s data in a form useful for serialization.

Thanks!

I would suggest Git, but looks like I would be wrong judging by the commits!

I think so :D!

xx

Dude… We have to make custom T-shirts. :)

It’s a shame I learned so much from reading your article. That makes your post more valuable, but my formal education less.

I recently landed my first job as a Software Engineer. I’m finding that working full-time for other people is less fulfilling than I’d hoped. How will I ever find the time to create awesome things now that I have this commitment? :(

Using file per entity was the first step. I chose binary files, because I didn’t want to bother with serialization. Merging individual files was not important in this system.

For handling multiple users we used a simple identifier in editor settings (username). That mapped to a folder where entity files user created were saved. Every new entity file had a number for a name, which increased for each entity.

Simple rule was applied: “Only the user can create files in his folder, he can change files in any folder”.

I think the last time I was checking the entity count was 300k. We’ve obviously had problems with this, so a simple hack was used, when user folder had a lot of files in it the username was appended with an number username_1.

Now for the main difference, we used Dropbox instead of svn. That saved me from doing any work related to syncing/committing files since it was automatic. I don’t think there is a real need to have a full version history of map, but do make backups.

Solving conflicts – we didn’t! Go work on you part of the map.

Terrain was a heightfield, chopped up into grid and each grid was a file. Same rules as above, with terrain having been created at start.

Textures, models, shaders and all other resources were also managed with Dropbox.

The grand plan was to have every resource automatically synced on every computer and have the tools detect changes and reload them as they happen. You could have an editor open and see the changes other people make. It worked for a while.

But as with all things, it didn’t last. With more files in there is started to get slower. I had to work on other parts of the game, and the slowness was compensated with removing features – automatic resource reloading was gone.

Hahaha no, that is not what that means. That’s our main engine programmer!

Has this been previously announced, or did I just scoop everyone? ;)

The only thing the database would provide is atomic access, which as I said we can already do just by making that file lockable; we just didn’t care enough to do it yet.

Apart from that, keep in mind we already have a database: it’s called the filesystem.

But since you are saying “but you already need to be connected to use commits… So why not use a database” it’s clear that you actually don’t understand the concern. That’s fine, I have no problem with that; just don’t claim that you do!

The point is we can edit the world for weeks if we want to, in isolation, then come back on the net and do a commit… It’s kind of different!

I think having a database will solve the unique ID issues, format representation issues and most of the conflicts.

This is likely true! It definitely helps to use a better merge program, but part of the point is that whatever merge program you use, at some point someone is going to fail a merge in a way that causes the file to stop parsing.

Right, but %g seems to be mainly a zero-trimming thing, not an exact-transcription thing…

I found WinMerge a much better visual merge tool than the one which comes with Tortoise, it handles conflicts better and seems “smarter” in interpreting text changes.

It can set it as default conflicts handler in TortoiseSVN.

Because git cannot handle game data. See comments above.

Could well be, they certainly didn’t used to, but I don’t know when that changed. I used to install the command line binaries from somewhere else too, but noticed they were included in Tortoise recently when building a new dev machine.

BTW, probably not directly useful, but I recently needed to enumerate the MAC addresses of “physical” network devices only, ignoring the virtual ones, etc. As far as I could tell, the information returned by GetAdaptersAddresses is insufficient to make this distinction, so I had to combine it with some information from the registry.

Yeah, someone notified me of this and said it is new as of svn 1.7?

The thing is, we want the majority of objects to be independent of terrain. So we would have to have some kind of hybrid system about how things are anchored or not, and I just haven’t put in the time to think about it.

It needed to be per-machine, because something that happens often is, say, I am editing on my computer at home, and I forget to check it in when I go to work; throw a laptop into the mix, and there becomes this whole check-in-management discipline that I would need to maintain. With per-machine ranges that basically goes away.

The entity data version is for adding fields to entities. The game can load an older entity that wasn’t saved with those fields. It’s a completely separate issue from source control, really.

A small correction: TortoiseSVN does include the command line binaries, but they’re not installed by default, just select them in the installer.

Did the ID ranges need to be per-machine rather than per-user? I’d have thought going off SVN username would be easier than MAC address if not.

And can you elaborate on what you used the entity data version for? What does that give you on top of what is already held in SVN history?

Hi John,

you are right about problems with “big files” with git and mercurial. However, last version of mercurial tries to fix exactly this problem with providing a default-implementation plug-in that is (if I’m correct) the same as Kiln’s plugin. It’s called largefiles (see http://mercurial.selenic.com/wiki/LargefilesExtension ).
As said in the page, it’s too soon to use it in production, it needs some works, but it’s a good thing they are working on this problem. I guess some years from now the problem will be solved for us game devs using decentralized tools.

Whoops – my bad, looks like comments covering my question got posted while I was reading. :)

Have you thought about how git’s change-centric model might affect the problems you’re solving? It seems like some of your pain comes because SVN is state-centric.

I agree with you. I have tried quite a few source control systems, each progressively better than the previous (SourceSafe, SVN, hg, git). Now we use git in all team projects and so far all merge conflicts we’ve encountered are ‘legitimate’. But then again, I haven’t been doing many projects the same scale as The Witness and might be wrong.

Also, I agree with the XML bit, though the way you said it was quite funny :).

But I can tell you try to say things as simple as possible in layman terms, but is still ahrd to follow and get everything.

By the way, did you know that Python’s repr() function returns your float as a string in exactly as many decimals you need to convert it back to its original float representation? I.e. f == float(repr(f)) is guaranteed for all floats f.