Resource subsystem - part one

Finally. I have been very busy researching and coding since my last post, but at last it seems to be paying off.

The subsystems were the first group of classes I developed for Derange (the skeleton, remember?), and as I tried different design approaches, they had to adapt to lots of structural changes, which ultimately led them in quite an amount of disarray (i.e. they were a total fucking mess). This made them perfect candidates for a complete makeover, so I took some time to redesign them to make them more flexible and hopefully, more elegant as well.

The one requiring the outmost care was clearly the ResourceSubsystem, because most other classes and even subsystems work with resources, and thus interact with it closely. So I decided to start there, and since right now I think it is much more robust and flexible, I will try to explain how it was designed and how it works.

The ResourceSubsystem is responsible for operations involving reading, modifying or writing files in disk, and performing them in an efficient way, reusing resources as needed or unloading them if they are no longer in use or if too much memory is in use.

Fortunately, Ogre already offers that functionality (and much more) and can be easily extended by implementing custom resource types. However, explaining in detail how it works is quite a big subject, so I'll try to stick with what is relevant to Derange only.

Ogre works with named groups of resources, using the interface provided by the ResourceGroupManager class. These groups may contain any type of resource that has been previously registered, and they can be stored in a file, compressed archive or even a remote URL. This is very useful as I can set up a default or "shared" resource group which contains all global resources, and then separate groups for each Location in the game, which can be loaded and unloaded as the player travels from one location to another.

Each resource type has its corresponding resource manager (which is derived from the ResourceManager class), and a resource serializer (derived from the Serializer class), which is simply the class that knows how to load and save a file of a specific type. If multiple resources come from say, a text file, you might be able to reuse the same serializer in all of them.

Regarding resources and resource managers, here's what the Ogre API says:

Resources are data objects that must be loaded and managed throughout an application. A resource might be a Mesh, a Texture, or any other piece of data - the key thing is that they must be identified by a name which is unique, must be loaded only once, must be managed efficiently in terms of retrieval, and they may also be unloadable to free memory up when they have not been used for a while and the memory budget is under stress.

A resource manager is responsible for managing a pool of resources of a particular type. It must index them, look them up, load and destroy them. It may also need to stay within a defined memory budget, and temporarily unload some resources if it needs to stay within this budget.

So, taking all this into account and after searching a long time for documentation on some very specific tasks (implementing resources with creation parameters seems to be a topic no one usually discusses -or I couldn't find anything on it-, so I had to take a look under the hood and figure it out myself), I came up with a new design. Here's a diagram that illustrates a part of it:

ResourceSubsystem class diagram. (click to enlarge)

As this class diagram shows, the ResourceSubsystem class keeps an instance of each resource manager class and two Log class instances: one for the system, and the other specifically for scripts. This helps debugging the engine and scripts separately. Additionally, it uses the CoreConfiguration class to access the engine's configuration, which is stored in an XML file and parsed as soon as the engine starts. The ResourceFileManager template class derives from Ogre's ResourceManager class and performs common operations like telling resources to load themselves and registering/unregistering the derived resource manager with the ResourceGroupManager class. All derived classes are supposed to implement only resource-specific operations. Additionally, all the classes in the diagram are Singletons, since there's no point in having multiple instances of them.

In practice, a simple usage example would be having a Lua script request that a specific GameObject is created and initialized to a certain state. These would be the steps involved in performing that:

1. The script asks the SceneSubsystem class to load the GameObject, and after this, the GameObjectManager will require a GameObjectState to be retrieved from disk.
   
2. The ResourceSubsystem::loadGameObjectState() method is called with the name of the GameObject wanted and the name of the state as parameters.
3. The load() method of the GameObjectState resource manager is called, passing the object name and a list of creation parameters. The state name is in this list.
4. The resource manager calls its base class ResourceManager::getByName() method passing the name as a parameter.
5. The resource is searched in all the registered resource locations (i.e. directories). Let's assume it was found and the resource is valid for this example's sake.
   
6. If the resource instance was not created yet (i.e. it was not used before), it is now, passing the creation parameters from step 3 along.
7. The GameObjectState::load() method is called.
8. The ResourceGroupManager::openResource() method is called, and a data stream is returned from the file.
9. A GameObjectSerializer class instance is created, and it is instructed to load() the data stream into the GameObjectState by parsing it and then using the setter methods on the former class.
10. That's it. The GameObjectState should now be loaded and ready to be passed back to the GameObjectManager that requested it in the first place. The next time someone requires that object with that state, it won't need to be loaded again. Unless it was unloaded to preserve memory, of course.

Right now there are only four resource types in the engine, but adding a new one is really easy. Just subclass the ResourceFileManager, ResourceFileSerializer and ResourceFile classes, add some method to handle those on the ResourceSubsystem class and voilá. There you go.

And this is how the resource subsystem interacts with resource managers. But there's still more, I haven't yet shown in detail how the manager-serializer-resource interactions work.

However, I'll be dealing with that on the next post. Hopefully sooner than what it took to write this one.

Intermission

I've learned a lot since I first designed the subsystems, and thus things have changed. I'll be taking some time to redesign them all now, before they grow too large and it gets much more difficult.

As I previously said, I want to disclose details on how some subsystems work, but there's no point in showing obsolete blueprints. So right now I'm working on the ResourceSubsystem.

The good news is that when I'm done, all resources will be handled in an efficient an consistent way, thanks to Ogre's own resource management. This means, for example, that if I want to create 3 instances of the same Actor it won't get loaded 3 times. It also means there will be a common interface to all resource types, which right now is not the case.

The bad news is, it's taking a little longer than I'd like to, and some of you guys might get impatient. Or maybe not care at all, but still.

I've added a Twitter widget to show what I'm currently working on. I hope this helps filling the gaps between posts and make the blog look more alive during development.

In the meantime, if you've got any other suggestions, I'd love to read them.

Engine name changed to "Derange"

I'm still wondering how on earth I missed this, but it seems that there are at least two game engines already using the "SAGE" acronym. The letters don't mean the same, but they're still too close. And even closer if you take into account that one of them is co-developed by Electronic Arts.

So I had to bite the dust and choose a new name, and I came up with "Derange".

What? I like it!

Anyway, this time I chose not to pick a name that relates to the kind of engine I'm making, because for example, I might implement adventure game specific functionality as a plugin, to be supported by the core classes. Or maybe not. But the bottom line is that I think this approach gives more freedom. That, and I'm sick of making up weird acronym based names.

So that's it. Bricks have officially been shat. In a painful way.

Meet Derange™.