Current Features (March 2009)

Derange has been in development for quite some time now, so I figured it might be a good idea to review which features have been implemented so far to see where I’m currently standing. Many of these features have been rewritten from scratch at least a couple of times as I learned better ways to implement them.

Implemented features

• 3D graphics rendering using Ogre3D v1.6 (Shoggoth).
• Scriptable API using Lua v5.1 and LuaBind v0.7.
• Input device management using OIS v1.2.
• Support for multiple Locations with separate resource groups that can be initialized using DotScene files.
• Importing of DotScene files using OgreMax v1.6.23, creating necessary game objects as needed.
• Configuration through XML files using Xerces-C v3.0 for parsing and validating against XML schemas.
• Efficient resource management using Ogre3D’s capabilities.
• Flexible game object design, tightly bound to Ogre3D’s scene graph implementation.
• Multiple Controllers can be combined and attached to game objects to separate representation from the actual game logic applied.
• Easy to create game objects using a factory class which only requires the names of the object and the state to initialize it to.
• Skeletal animation support with smooth blending between different animations.
• Actors rotate smoothly in the direction they’re moving to using Slerp interpolation.
• Actors move using configurable speed and inertia parameters.
• Third person player control, with movement relative to the current camera’s position using the keyboard.

Near-middle future plans

• Collision detection, possibly physics simulation too. Need to figure out the pros and cons of the different third party physics engines available to decide.
• Automatic camera selection based on target proximity.
• Walking through predefined path nodes.
• Path finding.
• Mouse picking.
• Overlay manager for HUD display, menus, and on-screen debugging.

Resource subsystem – part two

In my previous post, I explained how the Resource subsystem interacted with resource managers. This time, I’ll show you in more detail how each Derange resource manager deals with their specific resources using Ogre’s resource management scheme. Again, I’ll try to stick to what’s relevant to the engine, because that is quite a lengthy topic.

Resource life cycle

A Resource goes through several state transitions until it is actually loaded. This is taken care of by its corresponding Resource manager silently. Here’s a chart with the different states and what they mean:

State	Declaration?	Instance?	Available?
Undefined	No	No	No
Declared	Yes	No	No
Unloaded	Yes	Yes	No
Loaded	Yes	Yes	Yes

1. Undefined: This is the initial state for all Resources. Basically means that the engine knows nothing about them yet.
2. Declared: The engine knows about the Resource, but no steps have been taken towards loading it, and there is no valid instance yet. Also, no memory is used at this stage.
3. Unloaded: Resources enter this state when the ResourceGroup they belong to is initialized. By this time, they are only using a small amount of memory: enough to hold an instance, but with no available content yet. They can also be forced back into this state by manually unloading them or when their ResourceManager exceeds its memory quota, which means the instance is kept, but not the actual content.
4. Loaded: This is the “fully active” state of a Resource. It means there’s a valid instance available and the data it contains has been loaded into memory, so it is available for the game to use. Resources are loaded automatically when they are needed, unless their ResourceManager exceeded its memory quota.

Class diagram showing Resource and Serializer hierarchies. (click to enlarge)

Serializers

Whenever a ResourceManager needs to load a Resource from disk, a Serializer is used. They are classes that implement the logic needed to read or write files of a certain type.

Typically, a ResourceManager is asked to load a resource  of a given name. Then it creates an instance of the appropriate Serializer and calls its load() method passing a DataStream with the file contents and a Resource instance as parameters. The file is then read and parsed in order to initialize the Resource with the content. And that’s pretty much all they do.

Parsing scripts

Some Resource types might depend on other Resources being loaded. Take Material scripts, for example. You might be loading many of them that use the same Texture, and it would be really sloppy to have it reloaded every time a Material requests it.

Instead of that, it is declared, which basically means that it will be loaded only once when the ResourceGroup containing the declaration is bulk loaded. This saves a considerable amount of loading time, depending on the times it is referenced.

Loading Scenes

Scenes are stored in DotScene format, as exported from 3DStudio Max by OgreMax. This is basically an XML file defining each object in the scene, their properties, and SceneNode relationships.

OgreMax also provides code to create the necessary Ogre objects when importing it into the engine, and a callback class that can be used to trap specific events generated by the process.

Derange traps some of these events. For example, if an Ogre Camera is imported, a suitable GameObject of type “Camera” will be created and attached to it as an UserDefinedObject. This saves the need to have two different scene graphs, since each GameObject can be reached from its Ogre counterpart.

As far as loading goes, the DotSceneSerializer only takes care of loading the .scene file and store the XML text contained into a DotScene resource. All importing is triggered from the resource itself.

Loading Game Objects

GameObject definitions are also XML files, but they contain several GameObjectStates that can be applied to an object, and are referenced by an unique name.

There’s only one GameObject class. It’s the GameObjectState that defines the type of GameObject: cameras, lights, actors, path nodes, etc. I will explain the details on these in another post.

This Resource, however, is a little different from the others: In order to load a GameObject, not only its name is required. we also need the name of the GameObjectState to apply. To achieve this, Ogre provides a way to define creation parameters per Resource through the StringInterface class. This is essentially a list of properties that can either be used by a Serializer as loading parameters (In this case, the name of the GameObjectState to load) or by an editor to display onscreen.

Resource subsystem - part one

Finally. I have been very busy researching and coding since my last post, but at last it seems to be paying off.

The subsystems were the first group of classes I developed for Derange (the skeleton, remember?), and as I tried different design approaches, they had to adapt to lots of structural changes, which ultimately led them in quite an amount of disarray (i.e. they were a total fucking mess). This made them perfect candidates for a complete makeover, so I took some time to redesign them to make them more flexible and hopefully, more elegant as well.

The one requiring the outmost care was clearly the ResourceSubsystem, because most other classes and even subsystems work with resources, and thus interact with it closely. So I decided to start there, and since right now I think it is much more robust and flexible, I will try to explain how it was designed and how it works.

The ResourceSubsystem is responsible for operations involving reading, modifying or writing files in disk, and performing them in an efficient way, reusing resources as needed or unloading them if they are no longer in use or if too much memory is in use.

Fortunately, Ogre already offers that functionality (and much more) and can be easily extended by implementing custom resource types. However, explaining in detail how it works is quite a big subject, so I'll try to stick with what is relevant to Derange only.

Ogre works with named groups of resources, using the interface provided by the ResourceGroupManager class. These groups may contain any type of resource that has been previously registered, and they can be stored in a file, compressed archive or even a remote URL. This is very useful as I can set up a default or "shared" resource group which contains all global resources, and then separate groups for each Location in the game, which can be loaded and unloaded as the player travels from one location to another.

Each resource type has its corresponding resource manager (which is derived from the ResourceManager class), and a resource serializer (derived from the Serializer class), which is simply the class that knows how to load and save a file of a specific type. If multiple resources come from say, a text file, you might be able to reuse the same serializer in all of them.

Regarding resources and resource managers, here's what the Ogre API says:

Resources are data objects that must be loaded and managed throughout an application. A resource might be a Mesh, a Texture, or any other piece of data - the key thing is that they must be identified by a name which is unique, must be loaded only once, must be managed efficiently in terms of retrieval, and they may also be unloadable to free memory up when they have not been used for a while and the memory budget is under stress.

A resource manager is responsible for managing a pool of resources of a particular type. It must index them, look them up, load and destroy them. It may also need to stay within a defined memory budget, and temporarily unload some resources if it needs to stay within this budget.

So, taking all this into account and after searching a long time for documentation on some very specific tasks (implementing resources with creation parameters seems to be a topic no one usually discusses -or I couldn't find anything on it-, so I had to take a look under the hood and figure it out myself), I came up with a new design. Here's a diagram that illustrates a part of it:

ResourceSubsystem class diagram. (click to enlarge)

As this class diagram shows, the ResourceSubsystem class keeps an instance of each resource manager class and two Log class instances: one for the system, and the other specifically for scripts. This helps debugging the engine and scripts separately. Additionally, it uses the CoreConfiguration class to access the engine's configuration, which is stored in an XML file and parsed as soon as the engine starts. The ResourceFileManager template class derives from Ogre's ResourceManager class and performs common operations like telling resources to load themselves and registering/unregistering the derived resource manager with the ResourceGroupManager class. All derived classes are supposed to implement only resource-specific operations. Additionally, all the classes in the diagram are Singletons, since there's no point in having multiple instances of them.

In practice, a simple usage example would be having a Lua script request that a specific GameObject is created and initialized to a certain state. These would be the steps involved in performing that:

1. The script asks the SceneSubsystem class to load the GameObject, and after this, the GameObjectManager will require a GameObjectState to be retrieved from disk.
   
2. The ResourceSubsystem::loadGameObjectState() method is called with the name of the GameObject wanted and the name of the state as parameters.
3. The load() method of the GameObjectState resource manager is called, passing the object name and a list of creation parameters. The state name is in this list.
4. The resource manager calls its base class ResourceManager::getByName() method passing the name as a parameter.
5. The resource is searched in all the registered resource locations (i.e. directories). Let's assume it was found and the resource is valid for this example's sake.
   
6. If the resource instance was not created yet (i.e. it was not used before), it is now, passing the creation parameters from step 3 along.
7. The GameObjectState::load() method is called.
8. The ResourceGroupManager::openResource() method is called, and a data stream is returned from the file.
9. A GameObjectSerializer class instance is created, and it is instructed to load() the data stream into the GameObjectState by parsing it and then using the setter methods on the former class.
10. That's it. The GameObjectState should now be loaded and ready to be passed back to the GameObjectManager that requested it in the first place. The next time someone requires that object with that state, it won't need to be loaded again. Unless it was unloaded to preserve memory, of course.

Right now there are only four resource types in the engine, but adding a new one is really easy. Just subclass the ResourceFileManager, ResourceFileSerializer and ResourceFile classes, add some method to handle those on the ResourceSubsystem class and voilá. There you go.

And this is how the resource subsystem interacts with resource managers. But there's still more, I haven't yet shown in detail how the manager-serializer-resource interactions work.

However, I'll be dealing with that on the next post. Hopefully sooner than what it took to write this one.

Intermission

I've learned a lot since I first designed the subsystems, and thus things have changed. I'll be taking some time to redesign them all now, before they grow too large and it gets much more difficult.

As I previously said, I want to disclose details on how some subsystems work, but there's no point in showing obsolete blueprints. So right now I'm working on the ResourceSubsystem.

The good news is that when I'm done, all resources will be handled in an efficient an consistent way, thanks to Ogre's own resource management. This means, for example, that if I want to create 3 instances of the same Actor it won't get loaded 3 times. It also means there will be a common interface to all resource types, which right now is not the case.

The bad news is, it's taking a little longer than I'd like to, and some of you guys might get impatient. Or maybe not care at all, but still.

I've added a Twitter widget to show what I'm currently working on. I hope this helps filling the gaps between posts and make the blog look more alive during development.

In the meantime, if you've got any other suggestions, I'd love to read them.

Engine name changed to "Derange"

I'm still wondering how on earth I missed this, but it seems that there are at least two game engines already using the "SAGE" acronym. The letters don't mean the same, but they're still too close. And even closer if you take into account that one of them is co-developed by Electronic Arts.

So I had to bite the dust and choose a new name, and I came up with "Derange".

What? I like it!

Anyway, this time I chose not to pick a name that relates to the kind of engine I'm making, because for example, I might implement adventure game specific functionality as a plugin, to be supported by the core classes. Or maybe not. But the bottom line is that I think this approach gives more freedom. That, and I'm sick of making up weird acronym based names.

So that's it. Bricks have officially been shat. In a painful way.

Meet Derange™.

Core and subsystems overview

I think it's time I started talking about the specifics of the current Derange design, so I think I'll start from the core.

The Derange core -or kernel if you prefer-, is divided into the Core class and a few derivations of the SubSystem class, each dealing with specific areas of functionality. They need to be accessed by the rest of the engine all the time, as they do the "lower level" stuff, making things shorter to develop and easier to maintain afterward.

Some of them also encapsulate the functionality of some of the third-party components I use. That way, if I ever decide to switch to a different component to do the same task, I keep the original subsystem interface, and just adapt the inner workings.

The core can be configured by modifying the configuration file, which is an XML file with things like desired screen resolution, rendering options, resource locations, etc. When instanced, it instructs each subsystem to initialize itself and in turn, shutdown. More importantly, it starts the game loop.

Here are some subsystems and a brief description of what they're supposed to do:

• Input: Provides access to any available input devices. Right now it's basically an OIS wrapper.
• Resources: Allows to access and keep track of game resources like meshes, material scripts, textures, shaders, XML files, locations, actors, etc. It should do so in an efficient way -for example caching or sharing resources across requests-, so that if it's asked for a specific resource multiple times, it only gets loaded once.
  
• Video: Provides means to change how things are rendered, and it also creates the Ogre3D render window. Keeps the game loop running until exit is requested.
  
• Scripting: Allows to execute Lua scripts, which tell the engine what to do.
  
• Scene: Allows to create, access, destroy and keep track of all the game objects that interact in a single scene. It also notifies them when they should be updated.

click to enlarge

As you can see in this class diagram, all subsystem classes inherit from the Subsystem class, which implements the Singleton design pattern. This is because it only makes sense to have one instance of each subsystem. Additionally, the Subsystem class has access to the engine configuration, as specified in the XML file.

All classes in Derange derive either directly or indirectly from the CoreObject class. This way if I need to introduce any changes that should be applied to every single class, like for example keeping a count of instances and amount of memory used, I can do it from a single place.

So, when an instance of the Core class is created, it creates an instance of each subsystem class. Then it starts the game loop. And when it's done, it destroys the instances it created. Each subsystem is responsible for both initializing itself and cleaning after its own mess upon shutdown.

As Derange keeps growing, I'll most definitely add at least a couple more subsystems (for example one to handle all audio), but at this stage those should be enough.

I might disclose details on how each subsystem works soon, but I think I'll take a few days to tidy them up a little. As I was writing this post, I noticed some really dumb design mistakes that I previously overlooked.

So how am I going to do this?

Ok, so you now know why I'm making this. It would only be fair to tell you how I plan to do it, or at least how I think I'm going to.

Derange is being fully written in Visual C++ 2008. This was not a very easy decision to make, since I had never written anything in that language before. I'm very comfortable developing in C# and .NET, since I've been using those at work for a long time now. I love how tidy, flexible and powerful they are. However, there are some important drawbacks to using them for game development, as I found out:

• Ogre3D is written using VC++. Even though there are ports to use it with other languages, they will always be behind schedule when releasing updates, and there might be bugs in the porting process.
• .NET code is executed in a virtual machine, which is of course much more resource intensive and slower than plain old native code.
• I think most of the game development resources out there are targeted to the C++ developer, and being one in a learning process, this was quite important.
• I understand VC++ is the industry standard when it comes to game development.

Usually I like to plan development prior to get my hands on it, and have the big picture in mind to identify any possible design flaws before they become a problem that's hard to fix. Given the size of this project (Don't forget I'm the only one working on it) and that I'm still learning, that didn't sound realistic at all, as I can only anticipate what's coming up to a certain point. So I decided I would design and redesign as needed, on the fly, as I see fit.

This means that I might end up completely rewriting entire sections of code. Many times. And actually, I already have by now. I believe each code component should be as modular, reusable, and decoupled from other components as I can make it to be, so that when I need to take a different approach with something, there's not so much I need to change.

Before getting into game specific tasks like handling entities, animation, conversations, etc., I decided to focus on the "skeleton" of the engine. Things like basic engine initialization, handling input, resources and events in general, running Lua scripts and the like. I think this really needs to be dealt with first, because everything else I code will depend on how the skeleton is laid out. If it's not flexible enough, there will come a time when I have to redesign it, and probably rewrite most of the code that depends on it. Which would of course make me shit bricks and is not acceptable in any way.

I redesigned this skeleton many many times back when I started, and it still needs a little tidying, but by now I'm fairly comfortable with its general design. I'll get into the specifics and some diagrams in the following posts.

I don't plan on reinventing any wheels, so I also chose some free, open source, third party tools for some specific tasks. (I might add more or switch to different ones in the way)

• Lua: A scripting language widely used for game development, it's what makes a specific game "drive" the engine.
  
• LuaBind: Binds VC++ classes to Lua classes, making them available from scripts.
  
• Xerces C: Allows to read, write, and validate XML data.
  
• OgreMax: Imports scenes created in 3DStudio Max into Ogre3D.
  
• OIS: Handles input devices like keyboards, mice, gamepads, etc.

After the skeleton was mature enough, I started focusing on the actual game objects and their functionality. Things that can move, animate, walk a given route, talk, interact, and much more. Some examples of game objects would be cameras, lights, actors, particle systems, and the player himself.

So the plan up to this point is to keep polishing this skeleton as it becomes necessary, developing more game objects and testing them in a sandbox scene where I can see all their functions in action. I'll be deciding the order in which to implement this functionality on the fly, most probably based on whatever I feel like doing next at each point. :)

So that's it for now. I'll give more details on how the current design looks like in the following posts.

Introduction: Why I made this blog

I have wanted to develop games since I was a child.

I actually made a couple of horrible games back when I had a Commodore 64 using a Basic compiler and some Assembly (and I wasn't even 12 yet!), but never had any experience developing games for PC before, let alone working in a 3D environment. On the other hand, I have always been a quick learner, specially when it came to software development, so I figured I might as well give it a try.

In most tutorials I've read for newbie game developers you're advised to start very small, maybe just a Pacman or Tetris clone until you get a hold of the basics. However, I've always been very ambitious (and stubborn) when starting projects, so I figured a good way to learn the works was to make a reusable engine and deal with all topics at once instead of separately. Time will tell if this was a wise decision. :)

The first thing I noticed is that learning how to render 3D graphics seemed like way too much to study by myself considering I have a day job, a daughter, and -hopefully- a life, which doesn't leave me with THAT much spare time. And even if I did, I would have to constantly update the render code so that it stays fresh, which would be too much work for me alone.

So I decided to focus only on the game logic, and leave the rendering to a third-party engine. This is when Ogre3D comes in. Ogre3D is a free open source 3D rendering engine with a very big community using it, and I've already seen it in action in "Ankh", by Deck13 Interactive, so I knew it was mature enough to handle the kind of job I needed.

I play lots of games, and have particularly enjoyed point-and-click adventure games ever since Maniac Mansion came out on the C-64. So, considering they're not so resource intensive and dependent on cutting edge graphics, I decided that my engine would be tailored to make adventure games.

This engine is called Derange, and has already been in development for several months, making quite a slow progress as my spare time allowed. This will be its development diary.

This blog does NOT intend to teach ANYONE how to develop a game engine. Actually, I've never developed one. I'm just learning how to do it by practice.

So don't forget, I just made it because I wanted to:

• Keep track of the development -and learning- process somewhere, for my personal archiving purposes. Might be nice to have some day if I finish it.
  
• Allow anyone that might be interested in it to follow the progress.
• Get feedback. Remember, I'm learning, so it's very possible that I pick a completely fucked up way to do things, and feedback helps me get back in track before it's too late.

And who knows, it might even help a fellow newbie in a similar quest.

Hope you enjoy it!