Levellass' Keen Patching Tutorial
This is a basic guide to understanding and creating Patches for the Commander Keen games. It is intended to be a step-by-step guide from simple to complex patch techniques for the novice patcher (the current status of this Tutorial is yet incomplete).
It is highly recommended to first read Ceilick's or Levellass' modding tutorial. This will inform about basic mod setup in general as well as getting some rudimentary patches and an idea of how things work, which is required to follow this tutorial.
- 1 Setting up
- 2 Organisation of Patches
- 3 Hexadecimal code
- 4 Edit and Mimic existing Patches
- 5 Text Patches
- 6 Arrangement of Code Segments
- 7 Prospects
- 8 Credits For This Tutorial
- 9 See also
Besides a clean copy of a Keen Galaxy game you need to use Keengraph for graphics editing, this will make things easier later on. Also you need a patch file (you can simply create one by using the windows editor called notepad; open and save as patch.pat. Or just take a look at the earlier mentioned tutorials patch section. You could even use a tool called The Neural Stunner). Furthermore you need a hex editor with which you can view raw Keen code. For preference use XVI32.
Organisation of Patches
This section is dealing with the structure and organization of patches. Therefore already existing patches getting probed and investigated.
Copy-and-past existing patches from the Keen Wiki straight into your patch file is the simplest way to get patches. The best place to start is the main Patch Index, which lists all the pages with patches on them. Every entry is annotated with detailed and comprehensible explanations.
As an example, say we wish to increase the number of lives Keen has in Keen 5 when he starts a new game. We will thus be searching the Wiki for
new game As it so happens the Patch:Lives has a section with the patch we want:
|Patch: Lives at game start|
This can be copied straight from the Wiki as-is. It's quite obvious here that the $0003W somehow stands for 3 lives. We don't need to know how just yet, we can change this number to change the number of lives Keen has. Helpfully the Wiki has most of such changeable numbers color-coded in blue for quick identification.
Using this approach one can build an entire mod based on the work of what others have done before. But there are issues; for example, if we give Keen $0010W lives he starts the game not with 10, but with 16 lives, what's going on? Maybe if we look deeper at what we're doing...
Tracking Your Changes
By now you should have a patch file full of things you want in your mod or just want to test. But is there any way to track just what's going on? Yes, and it involves the hex editor we downloaded earlier.
First we need to create a dump file for our mod. This is a complete copy of the patched executable as it exists after
CKPatch has finished doing everything. We should also have a clean dump, one that nothing has been changed in. To do this we will put two dump commands into our patch file, one at the start, one at the end. Set up your patch file to look something like this:
%ext ck5 #Clean dump %dump DCLEAN.DMP #Patches, #More patches #Modified dump %dump DMOD.DMP %endWhen you run your mod two files will be created in your mod folder. These will be overwritten each time you run your mod, updating any changes. (As these are for testing be sure and remove the %dump commands when you release your mod.)
Now attempt to open the files. Hopefully you can't, the .DMP file type shouldn't be used by anything. You should get a suggested list of programs you can use that contains XVI32, if not you can browse your own computer to find XVI32 and select that. Make sure to tick
always use this program!
XVI32 will open with the files in two windows. It looks like a whole bunch of numbers on the left side and weird ASCII symbols on the right. This is the raw code of the Keen games. To look at our patch press
G to open the
goto menu. There you can type in the very first number of your patch, in the case of our example this is $5C9B. You don't need the
$ but you can type it in if you want. Make sure you to check
You should now see a different set of numbers in the window, this is the code at the location of the patch. Do the same for the other window\file. Now you can see a before and after view of the game's code side-by-side. If you did everything right you should see a string of numbers like
6A $6F $10 $00 This seems to be a slightly garbled version of our patch's numbers,
$6F6AW $0010W Ok, but what does that mean?
This section is dealing with hexadecimal and how patches are working with this numeral system. Reading this short introduction may help, but isn't necessary: Wikipedia:Hexadecimal
DEC, HEX, BYTES, WORDS and LONGS
Now we move on to the structure of patches themselves and how they may be more precisely edited. Patches are nearly universally based on hexadecimal where numbers are based on 16 instead of 10.
Basically patch numbers are a bit odd. Fortunately anyone with windows doesn't actually need to know much about this. Simply fire up windows inbuilt calculator and press
3 to covert it to programming mode. At the top left of the window there are four possible options:
Hex. Switching between these will automatically convert a number you have typed in.
Normal numbers, and the mode you will start in, are
Dec for decimal. If we type in
10 here we can see the proper patch number by switching to
Hex mode. We see
A meaning for 10 lives we should use this patch:
|Patch: Lives at game start|
This is very useful; a lot of patches can be hard to figure out if the numbers are large and in hex. This simple method will often shed light on things.
But why is the code arranged differently? Why was it scrambled when we looked at it? This is part of how
CKPatch deals with information. As an example the following three patches do exactly the same thing:
|Patch: Lives at game start|
The three patches are written using bytes, words and longs (double words). These are different ways of arranging the same data. Keen's code is arranged in bytes, in patches single two-digit numbers. Two of these make a word and four a long. But due to the interesting way (some) computers work to get these you need to rearrange the numbers slightly. $0003W makes sense, but as bytes the second part ($03) comes before the first ($00) When you build up a word or long from bytes you need to reverse the byte order!
This explains the odd way the code appears, but why use words and longs at all? For one thing, clarity; $0003W is simpler to deal with. 258 is $0102W in word form which is how it appears in windows calculator. Split up into bytes it's $02 $01 which is a little confusing. Many numbers in Keen use the word format, requiring two bytes (and some require four).
A note of cation, while you can rearrange patches like this, do NOT attempt to change a single byte into a word or long or vice-versa. The result will not 'fit' into Keen's code. The equation must balance, two bytes to a word, two words to a long.
Editing Existing Patches
But something else is odd here, what does $6F6AW mean? It's coded in brown and a very big number, why is it there? The patch description mentions two hotlinks, one to the new game patch for more patches and one to game stats. The Game Stats page has a number of interesting sections dealing in just how such numbers work. While not necessary reading for us yet, investigation will explain a lot about how patches work.
The vital takeaway is that by exploring the Wiki instead of just straight copying a patcher can get a better idea of what their patches actually do. If in doubt looking at similar patches can be informative. More complex explanations including a full one about the brown numbers will be later. For now we focus on one question: What if the Wiki doesn't have what you want?
Edit and Mimic existing Patches
This section is dealing with already exiting patches which need to be converted to ones special needs, say an ascertained Keen 4 patch needs to be converted to Keen 5. Here is how to handle something like that.
Find and Replace
Say we want to change Keen's standing animation so it's the same as his looking up one, but for some reason the Wiki doesn't have a patch for Keen's standing animation. In this case we must look for possible patches in the game's code. Then we can write our own patch at that location! As such we open up the clean dump file in XVI32.First we need to know what we're looking for. Obviously somewhere is a number for Keen's standing image. To find out what that is we open the mod's graphics folder and find Keen's standing sprite image. As it turns out there are two, left and right. In Keengraph the two images are
5SPR0116.BMP(if using Modkeen things get tricky). These two numbers, 108 and 116 are what are used by the game (in Modkeen they're not).
Likewise Keen's looking up image (there's only one!) is
5SPR0124.BMP. Converting these in windows calculator gives us
$74 as the originals and
$7C as our desired change. In XVI32 we goto the search menu, select
count and type in
74 in the 'hex string' box.
1.946 matches. Oh.
But wait! Most numbers in Keen are words! That's
$74 $00! We search for that.
31 matches. Still a lot. Maybe try them later. What about
$6C $00? 18 matches. Using
F we can find the first one, then use the
F3 key to find the next, and next, and next...
The first match is at
$2AE4 (look at the
Adr. Hex counter at the bottom left of the window). The second at
$3041E and so on. We thus start building a possible patch list that looks like this:
|Patch: ''Possible'' Standing Animation|
But do we really want to test each of these out one at a time? That sounds like hard work. So let's patch smarter. We can eliminate a few right off the bat. The second patch for example. In XVI32 we can see on the right of the window a bunch of text, in fact the patch is part of the
normal, one of the difficulty options! That's probably not what we're looking for. The third patch is the same. We can toss out quite a few patches this way.
What's left we can test half at a time; if we have ten patches test 5 at once, if nothing happens, test the other five, if something happens discard the second set and split the 5 possible patches into 3 and 2. In this way a large number of patches can be tested in a short time (though some might crash the game, which makes things harder).
But as it so happens a smart patcher won't even need to do this. The fourth possible match is at
$30BCA, in fact the hex code is buried in
$74 $00 $6C $00 $03 $00. Isn't
$74 the other thing we were searching for? Looks suspicious! And won't you know it? These patches change Keen's standing animation:
|Patch: Standing Animation|
For the true novice this is probably the only way of finding new patches. It's slow and inefficient but it works surprisingly often. But with all the patches we have, is there a way of being faster about things?
One method of being faster is getting an idea of where you should be looking. Quite often a patch exists for one game or situation but not the exact one you want. The Keen Vorticons and Keen Galaxy games form two series based on the same basic code, most patches work the same way between games. If you know a patch for one game you can often use it to make a patch for another.
We should expect that the game's code is organized in a similar manner in each game in a series, so stuff in one game will be in sort of the same place as it is in another. As an example consider the lives at start patch from above. In Keen 4, 5 and 6 it is as follows:
|Patch: Lives at game start|
Notice the locations are quite similar, when looking for matching patches we should test those closest to an existing patch first. This works within and across games.
We can also use an existing patch to get an idea where a related patch may be. For example the following patches list stuff Keen has when starting a new game in Keen 5:
|Patch: Stuff Keen has at game start (Defaults)|
It's simple common sense that the patch for Keen's starting ammo will be near his starting lives and a simple look at the code is enough to confirm this and write the patch. Now let's look in more detail how we can use the structure of existing patches to make new ones.
Let's suppose you want to change the default difficulty of demo mode games. There is a patch for Keen 4 on the Wiki, but not one for Keen 5, which you want:
|Patch: Difficulty of demo mode -Keen 4|
$0002Wgets you hundreds of possible matches, that could take days to track down. but it's likely that the code around the patch is similar in the three Galaxy games. Opening a Keen 4 dump and going to the first patch we see the following code at
$C7 $06 $70 $7A $05 $00 $C7 $06 $4A $47 $02 $00 $9A $A8 $08 $35 $1B
$C7 $06 $4A $47 $02 $00 gives us 4 results in the entire game; one of these matches the second patch at
$5209, these must set the game difficulty in various spots! We can test this and create new patches for Keen 4, but that's not what we want. So we then pen up Keen 5's dump and look for that same data.
No matches. Rats! In hindsight this makes sense, we can't expect everything to be identical. So let's look for some other stuff nearby and see what we get. For various reasons stuff following
$C7 $06s tends to change between games and even dump files. getting a feel for what does and doesn't change takes time and practice, but less then you'd think. From experience I expect
$05 $00 $C7 $06 and
$02 $00 $9A not to change much between games, both involve numbers affecting the same thing across games for example.
Looking for the first one in Keen 4 gives us only TWO matches, both of which match the demo difficulty patches we have. And for Keen 5? Two matches! Right next to each match is a
$02 $00 that we need to change in our new patches. We thus make this patch and test it, and it works!
|Patch: Difficulty of demo mode -Keen 5|
This section is dealing with one of the simpler type of patches, the text patch, that alters what text the game displays at certain points. These patches consist of two parts, the text itself and the text 'pointer'.
Locate a Text Passage
For an example, one could alter the text which appears when Keen tries to enter water in Keen 4 without the Wetsuit. By default this is "I can't swim!" which can be easily seen in-game.
Finding the text, simply open the Keen 4 game dump in XVI32 and open the search menu and select
Find.... There select the
Text string option and look for the text. (Be sure and use the right case and punctuation. To be more sure of success you might want to search for only a short part of the text, such as "I can't".) This gives exactly one result at
XVI32's window is divided into two parts, a hex side on the left and a text side on the right. Looking on the result one can see the
can't swim text in the executable is shown plain on the right side of the window, as well as all the other texts that surround it. The following patch can be made:
|Patch: Text message -Keen 4|
The text in this patch is commonly referred to as a text 'string'. (Because it's one long line.) Note something important here: the text string ends with
$00. This tells the game to stop reading text there, which is important, if changed into
$20 (A space) for example the message becomes "I can't swim! Cool! I can breathe under water now!" This happens because the game keeps reading the text and accidentally reads TWO texts. In fact doing this will crash the game with a
string exceeds width error.
Another thing to note comes to light when looking at the next text string, about breathing under water. Properly patched it looks like this:
|Patch: Text message -Keen 4|
The $0A means 'Go to the next line down' and it stops all the text being on one long line. This is VERY important, if you have a line of text that is too long the game will crash (with that
string exceeds width error mentioned above). One of the hardest and most annoying things about text patching is knowing where to move down to the next line.
With a little thinking this reveals another limitation of text patches; it is easy to have a patch that is shorter than the old text; nothing needs to really be changed. But one that is longer will probably overwrite something important. For example, if we change the swim message to:
|Patch: Text message -Keen 4|
The game crashes when Keen tries to enter the water. When he instead gets the wetsuit the following message displays: "ithout a wetsuit!". This is because that text string has been overwritten by your new message. What we need then is a way to move about all the text strings. This can be done, it involves something called pointers.
A pointer 'points' to a location in the game executable. There are two types, words and longs, but for now only words will be considered. A word pointer is two bytes, like a normal word, except that they refer to a location instead of a value.
This location will need to have another value added to it to give the true location. Sometimes, such as in the Keen Vorticons games, this value is zero, but often it isn't. The location which is needed to be added for text patches is called the 'data segment location'. It is different for each game. The following is a list for the 'default' versions of Keen games:
|Patch: Default data segment location|
Keen 1: $13050 Keen 2: $17780 Keen 3: $19820 Keen 4: $2EE70 Keen 5: $30340 Keen 6: $30D30 Keen D: $23A70
The location can easily be found. It is 4 bytes less than where the text string
Borland C++ is in the executable (This text was added in automatically when the game was created to mark the data segment's start).
As the location of the swimming text is known and the value added, one can work out what the game's pointer will look like. Therefore the Windows Calculator needs to be opened and the following sum needs to be entered:
30A3D - 2EE70. This gives the value
1BCD. Looking in game dump for this value (type
1B into the search field!) the result will be one match at
$F3A5, from that one can create the following patch:
By changing this value, one can influence were the game looks for the can't swim message, adding
1 will move the text's location 1 and so on. This means that if there isn't enough space the text can be moved somewhere more roomy. Alternatively one small text string can made longer and moved the start of the next string so it doesn't get overwritten (This will shrink the next text string of course).
Assumed this large I can't swim without a wetsuit! message is favoured, it could jiggled around with the wetsuit message. Otherwise something unimportant can be spotted in order to overwrite it. Fortunately Keen, especially Keen Galaxy, has a large number of checks and error messages that shouldn't come up in regular gameplay. These can be changed or overwritten with little concern. The keen wiki page Patch:Game errors contains an exhaustive list of game error messages complete with their text pointers and text locations.
For demonstration purpose we choose to override the MarkTileGraphics errors. These appear when the tileset properties are messed up somehow, no released game should ever trigger them if properly tested. The first patch looks like this:
%patch $168F9 $4099W
There are six patches in total which allows quite a bit of text to be stuffed in. To move the can't swim patch just take the MarkTile pointer,
$4099W and replace
$1BCDW with it. Launching the game Keen gives the error message when he tries to enter the water. Now this error message can be edited and changed into whatever, for e.g.:
But what about the error message? It might be needed if something got messed up while building the game. Or even if it ever did appear it should make sense. Well, it can just swapped around with the swim message, and be made a smaller error message. It could be as short as E1 (Error 1) or Back anim 0, as long as it makes sense to you and\or any possible players:
It is easily possible to compress the game's dozens of errors into just a few lines, freeing up massive amounts of 'text space'. It should also be noted that looking at what have been done in the game dump is a useful way of keeping track of how much space is left and whether anything has been overwritten.
Pointer Tool LText
Finally if this seems a little tricky there's a small tool that will scan the Keen games for any desired text and try to automatically generate a text patch plus text pointer for that very text. This is LText and it can be found in the TutoriTools folder. Simply type some text into LText.TXT and run the executable. It will create an output folder giving you all possible patches for all possible Keen games that contain that text. It's not perfect, it will often miss text pointers that are hidden in tricky spots, but it can be used to drastically cut all the manual work out of finding new text patches.
Arrangement of Code Segments
This section is dealing with how code pointers and locations are arranged in segments.
All that effort in the last section was taken to make the games internal search most efficiently (with prospects to extremely limited processing power). Because the executable is larger than 64 KB, addresses would have to be in DWORDs what the game finds difficult to deal with. Also every time an address is given the game must 'look up' the location, which is difficult plus time consuming to look up to go to a 'faraway' address. Therefore it is very advantageous for the game to be organized in a way to minimize the 'lookup times'.
The game executable is organized into a number of 'segments', each specializing in a certain type of code. Due to the somewhat automated creation of these segments they can vary wildly in size and specialty (for e.g. each level name is its own segment). Some segments contain sprite code, others drawing routines or the main game loop.
Each segment attempts to be organized such that code in a segment most commonly calls code in the same segment. This is why, for example, sprite behaviors are all clustered in one or two segments.
Whenever the game wishes to call a piece of code that is not in the same segment it uses a segment call, also known as a far call (As it is capable of 'calling' the farthermost bits of code.) (Same segment calls involve something else, a near call.) These calls have several interesting features. The one that will be examining one tells the game how Keen should act while standing still:
(The specifics of how this patch was obtained will be covered later. For now its format shall be focused.) The call is a DWORD, composed of two parts. The first WORD is the segment address, the second WORD is the address within that segment. This raises a problem, with only a WORD for the segment address how can the game call segments further than $FFFF? It does so by dividing the address by 16 ($10). In hex this has the effect of lopping off the last digit of the address. Our example here calls a segment located at $B800. As a consequence this has the side effect that segments can only start at multiples of $10.
This example is thus calling code located at $B800 + $395 or $BB95. Opening the Keen 4 dump in XVI32 and going to the address, shows the first byte is $55, which is nearly always the start of a piece of game code. At this point this copy of the dump, preferably produced some time ago, should be kept open for alter comparison.
This is in fact Keen's standing behavior. It can be tested by replacing the specific call with a blank call that calls nothing:
In this case, running the game causes Keen to do nothing when standing still, he doesn't get bored or respond to arrow presses.
Why does this new patch use L while the original used RL? Attempting the above patch with RL causes the game to crash messily when Keen appears. This is because segment calls except blank calls require a relative address.
(Delete the patch and produce a new dump of the Keen 4 executable. Open this in XVI32 while also keeping the older dump from earlier open.) Go to the patch address and compare the four bytes there in both versions. What do you notice? If all went well and the dumps were from different days then while the first two bytes should be the same, the next two should differ. This doesn't seem to make sense, the same game run the same way produces two (slightly) different dumps.)
What is happening is that these collections of bytes are giving the proper address in computer memory for the code. But the game itself is being loaded into a different location in memory each time. This means that everything in the game has an 'extra' value added to it. This could be added to the segment or inter-segment value (or both.) but is by convention added to the segment value. The value is largely random and can change each time the game is started. It is impossible to write an ordinary patch that can guess the game's location and work, even if it were possible the patch would only work once!
Fortunately CKPatch has the RL functionality to automatically detect and add the game's memory offset to any existing patch. It is applied only to far calls.
Now we have seen a bare far call, we can move on to see a 'complete' one.
Full Far Calls
The patch above was an action patch, which will be discussed much later. It is just an address the game loads then runs. Most far calls in code are several bytes longer, such as follows:
In this example a lot is happening (a lot of this will be covered later). It plays sound 4 when called. The first four bytes take the two-byte value 0004 and save it for the called code to use. $9A tells the game the next four bytes are a far call Finally $83 $C4 $02 tells the game to unload two bytes now the game is done with them.
The most basic calls will not load anything before being run, they are just five bytes, $9A and the address. Other calls will load (and dismiss) several values in several ways. But there are several important things to note here.
First of all simply scanning for $9A will identify where a far call is in most code. Secondly replacing this and the next four bytes with $90 will cause the call to be deleted and the code not to be run. As an example this patch takes the above code and stops the playsound code being called, muting the sound in that situation:
Finally, the far call will have nearly an identical setup everywhere in the game if it calls the same code. Even though the call is different in each dump, all identical calls will be altered the same way by the same amount. If knowing where one is one can look at that location in the dump and then look for other versions of that call, tracking down each and every time that code is called in the game (which is an immensely powerful thing to do).
Far Call Searches
Starting from scratch we'll take a random sound patch from the Keen Wiki, we could also use a patch from a mod or one we discovered ourselves:
The first thing to do is to open up a dump of Keen 4 and go to the patch's location. There the same setup as the last example can be found.; value 2 is loaded, the code called then unloaded. Now pay especial attention to the far call. Though yours will differ and example might be:
$9A $F1 $09 $49 $1F
This is the version of the far call for sounds in this dump. Searching for this sequence of bytes gives back no less than 78 matches, all of them different calls to play a sound in the game.
Many things can be done with this, if the sound should get altered is known one can search the matches until it is found. If a special code for tweaking is known (such as the location of Keen's walking behaviour) one can look for matches at or near that location for the sounds that code uses. Or, if nothing is unknown, one can simply try all of the 78 patches, changing the sounds they play and testing the game, allowing to potentially label every single sound and situation in the game, providing endless clues as to the locations of other bits of code.
There are a few other things to note when dealing with segments. The first is that one can make up segment addresses when patching and the game will work fine, so long as the address points to the right location. However doing so can cause some minor problems since in effect the game is getting fragmented and is probably slowing its performance. If at all possible default segment values should be used. These can be guessed from other patches or the Keen Wiki has an incomplete list.
Another thing is that only action patch calls can be blanked with the $00000000L value.
The data segment (as a final wrap-up) is special since it doesn't usually have to be called; instead usually an address within the segment is given, and usually not for code, but for data like text. This topic will in fact make up several sections on its own, after we have learned about jumps and variables. Suffice to say it is an incredibly powerful resource to master.
Finally, converting between a location and call can be tricky. To help the TutoriTools package contains a utility called RL that will take a far call patch and convert it into a code location. Simply paste the patch in INPUT.TXT and run RL, the address will be printed in OUTPUT.TXT.
This is the end of the tutorial at present. There's still a lot to cover, mostly involving understanding the way the game does things and using this to better build and modify patches. Next up we'll look at near calls and jumps. But that's for another day.
Credits For This Tutorial
- Ceilick's Keen Galaxy Modding Tutorial
- Levellass' Keen Galaxy Modding Tutorial
- TheMagician's Keen Galaxy Modding Tutorial