Games from Within

The Game Asset Pipeline
by Ben Carter
Charles River Media, 2004

Did you watch Star Wars Episode I: The Phantom Menace? Do you remember how eagerly everybody was anticipating a new Star Wars movie after 16 years since The Return of the Jedi? If you asked people what they thought of the movie right as they were walking out of the theater, you would have heard all sort of mixed responses. It was a Star Wars movie after all. It had cool computer graphics, the mystical force, and, above all, dark jedis! On the other hand, it was also the movie that brought us Jar-Jar Bink, and some very forgettable scenes. It was really hard to combine all those facts and come up with a summary or rating that would do the movie justice. You hated some parts, and loved some others.

That's exactly how I feel about the book The Game Asset Pipeline.

I've been awaiting a book covering the asset pipeline for a long time, but there really was nothing out there covering that topic, which is ironic considering the importance of the asset pipeline in modern game development. My article “Optimizing the Content Pipeline” in Game Developer Magazine earlier this year was an attempt to start bringing the topic to people's minds and stimulate more discussion on the topic. When I saw the announcement for Ben Carter's The Game Asset Pipeline book a few months ago I couldn't help but salivate in anticipation.

Would I recommend the book to other game developers? Absolutely! Go and buy it today. If the words “asset pipeline” are not brought up several times a day in your company, you definitely need to read this book and pass it around. You have a lot of work ahead of you. Even if you've carefully thought about your pipeline and are working towards making it more efficient, you will get a few gold nuggets out of this book.

Unfortunately the book is not without flaws. And some are so big that they might overshadow the rest of the book (think Jar-Jar). Hopefully this review will get you ready for it, so keep an open mind, go for the interesting parts of the book, and ignore the rest.

The book does a very good job at giving an overview of the game asset pipeline. What it is, why you should care about it, how it can be implemented, etc. Don't expect any magic formulas or fixed approaches. It's more of a discussion of the topic, weighing the different approaches when there are alternatives, mixed with a bit of personal experience and anecdotes. That's really the best way to approach a discussion of the asset pipeline because there isn't a single solution that will be best for all the projects. Games and companies are different enough from each other that they'll benefit from having a pipeline closely tailored to their specific situation. This book makes you think about the different aspects and proposes different options.

So, what's not to like then?

Let's start with the major flaws. The structure of the book leaves much to be desired. The table of contents suggests that the book is reasonably well structured, but frankly, when I was reading it, I always had to keep looking back at the table of contents to remember what chapter I was reading and where I was headed. The first few chapters are titled Introduction, Where Do Assets Come From?, A High-Level View of Asset Management, and Building an Asset Management System. To me, they all blurred together into one big amorphous chapter. The book would have greatly benefited from a quick overview and motivation chapter, and then a detailed explanation of each step of the pipeline.

Following those chapters we have four chapters (which represent over a third of the whole book--a whopping 130 pages) that are completely, utterly useless and out of place. They attempt to explain how you can process textures, geometry, sound, and level data into something more efficient that can be used in the game. Of course, you need to do that, but this is not the place or the time to read about it. I don't want to read a discussion on efficient triangle stripping or lossy texture compression here! There are much better books and resources out there for that. Give us a link and move on. There is plenty to be said about the asset pipeline, so why waste precious pages with generic material that can be best found elsewhere?

The rest of my complaints are more about topics that were not mentioned or covered in enough depth than actual problems with the book. But given that a full third of the book was totally wasted, it's frustrating to be left wanting more details in other topics. For example, there's very little discussion on the integration between the game engine and the tools (loose coupling with external data-definition languages, or tight-coupling reusing some of the same libraries and code for example). Multiplatform development is completely glossed over even though it can completely change the way the asset pipeline needs to be structured. Similarly, a bunch of other topics didn't even get a mention: fast data paths, resource hotloading, concurrent editing of resources, etc.

Another area where the book falls short is the lack of diagrams. For a book that is solely about the asset pipeline, I would have liked to see extensive diagrams showing me how the data flows from one stage to the next. Where it gets created, how it gets processed, how it makes its way into the game, etc.

If you're willing to overlook all those shortcomings, you will find some interesting material inside the book. Apart from a good (although disorganized) overview of the pipeline, there are some very interesting discussions on advanced topics. One of my favorite ones is the idea of using “views” into the resources instead of using the files directly. This is very much the same way that Perforce maps files from the depot into your local file system. Different people could have different views of the resources depending on what they're working on, and even the game itself could use a different view to create the final master.

Another very interesting topic discussed is that of fetching assets on demand from the network and caching them at the local PC. It's a very intriguing idea and I'll definitely want to explore it further in the near future. In the past, I've always worked with assets bundled into large “pack” files that need to be copied and updated and passed around during development. The idea of just having the game request whatever resources it needs over the network is extremely intriguing and, if integrated well into the pipeline, could really improve iteration times.

Finally, one really cute trick that I hadn't encountered before: Ben recommends using odd-numbered UIDs for resources (or anything else for that matter). The idea is that pointers, in most platforms, will always be even-numbered, so it'll be very easy to distinguish between them at a glance in the debugger window.

All in all, this is a frustrating book because it could have been so much more. Right now it has no competition, so I heartily recommend it to anybody. However, I don't think it's going to fare very well with the passage of time, and as soon as other books in the topic come out there will be little reason to revisit it. Until then, may the force guide your work on the asset pipeline.