Understanding the Content Pipeline, Part 1

Of all of the great things that XNA Game Studio and Creator’s Club provides, none of them in my opinion is as interesting and as underrated as the Content Pipeline build process.  This is a very cool feature that builds your raw game assets into compact and formatted binary representations that are quicker to load at runtime.

Not only is this my favorite feature, it is perhaps something I struggled with for quite a while.  I know people that hate the content pipeline and use it only for basic texture and model loading.  At first glance it is very complicated and hard to understand.  I didn’t go much beyond the basics at first, but now I tend to think about how I can use the content build system to tackle a problem before anything else.

I think learning by example is always a good idea so I will give an example of how I use the content pipeline.  Hopefully if you are struggling with it as I did, this example will help.

My example is focused around a big problem my game has to solve.  Since it is a word-based game, it needs a list of words that define its dictionary and this list needs to be fast to load, fast to verify a word as correct, and fast to search for the AI to make use of.  My problem statement is this:  I want to take an arbitrary list of N words and build a data structure that will make it possible to fulfill the requirements as stated above.

After some searching around, it was decided that I should build a directed acyclic word graph for word lookup and searching.  This data structure is the runtime piece that we use in the AI and during the game to validate the correctness of words.

Since the word list can be quite long, the word graph can take some time to load.  A question arises: do we really want to load this word list into the fast word graph at runtime when the game or level loads?  Well, we can try it out.  After getting a hold of my 170,000 or so word list, I tried to load it in my game.  Result: not so good.  Over 1 minute was required to load and transform these words.  Now my algorithm is probably less then optimal, but another question arises from that perspective:  do I want to spend the time to try to make this faster or just pre-compute the word graph during build?  I thought about it for a while and concluded that attempting to optimize the word graph build algorithm was pointless.  Pre-computation won out.

Since I’m going to be pre-computing, the content pipeline build process will be how I will do it.  My goal will be to drop in a text file containing all my words and transform this file into a binary representation of the word graph.  This binary file will then be loaded directly from the ContentManager in-game and should already be transformed and should have a small footprint on disk.  It shouldn’t take any longer to load the word graph than a large texture or sprite font.

So let’s begin with a simple review of how to build a custom processor that will build my word graph via the XNA Game Studio Content Pipeline…

There are 3 parts to the build process of a game asset:

  1. Content Importing – this is the process of opening a game asset file (in my case, a text file) and transforming it into a managed object.  Importers are all about understanding file formats and getting file contents into managed objects so that the data may be used farther down the pipe.
  2. Content Processing –  this is the process of massaging and transforming the managed object obtained from importing.  Processing is a very wide-open aspect of the pipeline.  Almost anything could be done here.  For my example, I will be running my word list through my own custom WordGraphBuilder which is an object that does the word graph build algorithm.
  3. Content Writing – this is the process of taking the final processed object from step #2 and serializing it to binary.  In XNA Game Studio 3.1, this process is automatic and the system can generate this step for you.  However, in my code I wrote it before 3.1 was released and I’m doing some fancier logic to conditionally write certain aspects of the object to the binary stream.  I do think it could be rewritten to take advantage of the automatic serialization capabilities in 3.1, but so far I have not explored that avenue out of time constraints (in other words, if ain’t broke, don’t fix it!).

When you place a content file in your Content project in XNA Game Studio, you have the option to specify the Importer, and Processor that will be used to build your asset file.  Many importers and processors ship with the XNA framework, but you can write your own.  Obviously for example, I needed to write my own because there isn’t a text file importer suitable for my needs and there certainly isn’t a processor that builds a word graph that ships with the XNA Framework!

In the next part of the series, I will start going over the individual three parts in detail.  Stay tuned…


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: