Westwood SHP Format (Dune II)
The sprite format used in Dune II is a collection of compressed 8-bit frames, where each frame can have its own set of dimensions.
The format comes in two versions; one used in v1.00 of the game, the other in the patched v1.07. The difference between the two types is that the newer type uses UINT32LE addressing, whereas the older version uses UINT16LE. Sadly, there is no clear version indicator in the header, meaning the type has to be derived from the data itself.
|0x00||UINT16LE||NrOfFrames||The number of frames in the file.|
|0x02||UINTXXLE[NrOFFrames+1]||FrameOffsets||List of offsets to the headers of each frame. As mentioned, these are either UINT16LE or UINT32LE, depending on the version. In the v1.00 format, these offsets are relative to the beginning of the file. In v1.07, however, they are relative to the beginning of the FrameOffsets array.|
As you see, the array has one more entry than the amount of frames. This last offset points to the end of the file. As noted, in v1.07 this is relative to the start of the FrameOffsets array, meaning the value in there will be two bytes less than the actual file size.
A classic check to distinguish the two types is to see if the 5th and 6th bytes in the file are zero; in a 16-bit array this would be the address of the second frame, while in 32-bit mode this would be the higher-than-0xFFFF part of the very first frame, which would require a ridiculously high frame count in the file to be anything else than zero.
Since the final entry in the array always equals the file size, this can also be used as version test. This check should test v1.00 first; even though chances are really small, it is technically possible that data NumImages*2 bytes into the file data behind the FrameOffsets array happens to contain data exactly matching the file end address. However, in 32-bit addressing, it is impossible for either the first or second half of a valid offset halfway down the array to ever match the file end value, since that should only occur once, at the end of the array.
Each of the addresses in the header (except for the last one) points to a frame, which is comprised of a header followed by compressed image data.
|0x00||BYTE||Flags||A series of bit flags that give extra options on how to handle the data. The three flags are HasRemapTable (bit 1), NoLCW (bit 2) and CustomSizeRemap (bit 3). Bit 3 should never be enabled if bit 1 isn't.|
|0x02||UINT8||Slices||The format's transparency-collapsing compression works per row. This indicates the number of rows. This value should always match the frame height.|
|0x03||UINT16LE||Width||The frame width.|
|0x05||UINT8||Height||The frame height.|
|0x06||UINT16LE||Filesize||The full size of the frame's data in the file, including this header.|
|0x08||UINT16LE||ZeroCompressedSize||Size of the data before RLE transparency decompression. If the NoLCW flag is not enabled, this gives the amount of space that needs to be reserved for the LCW decompression process. Otherwise it just equals the image data size behind the header.|
|0x0A||UINT8||RemapSize||This byte is only added if the CustomSizeRemap flag is enabled. If not, RemapSize defaults to 16.|
|0x0A or 0x0B||UINT8[RemapSize]||RemapTable||This table is only added if the HasRemapTable flag is enabled.|
Image data decompression
This header is followed by the actual image data, which should first be decompressed using LCW (unless NoLCW is enabled), and then expanded using the transparency-expanding RLE algorithm.
The RLE compression is a classic flag-based RLE triggered by the value 00. Whenever a 00 byte is encountered, the value behind it will indicate how many times this 00 needs to be repeated. This RLE compression never crosses over to a next line, and should be treated line per line. If overflows occur they should be ignored, not wrapped around. This detail is important when writing a compression algorithm.
If a remap table is present, the image data has to be transformed with a simple remapping operation, in which every byte value
p in the final uncompressed image data is replaced by
RemapTable[p]. The remap is required for the game's house colour remapping algorithm, so unit graphics that have red remap colours on them need to contain such a remap table.
The final image data is classic compact 8-bit data with a stride equalling the frame width.
The following tools are able to work with files in this format.
|Name||Platform||View images in this format?||Convert/export to another file/format?||Import from another file/format?||Access hidden data?||Edit metadata?||Notes|
|Engie File Converter||Windows||Yes||Yes||Yes||N/A||N/A||Supports viewing, reading, and writing, including the remapping tables.|
|d2shpset||Windows (command line)||No||No||Yes||N/A||N/A||A small Dune II SHP writing utility written by OmniBlade of the RedAlert++ team to help Dune II mod makers. It supports generating the remap tables the game needs for applying house colours.|
|Red Horizon Utilities||Java (command line)||No||Yes||Yes||No||N/A||Support for Dune II SHP in it is experimental, and apparently it works better in certain older versions than in the newest ones.
Original site is defunct. Backups of the tools can be found here.
|XCC Mixer||Windows||Yes||Yes||No||N/A||N/A||Only supports viewing the format and converting it to frames. Has no support for creating files of this type.|