Unpacking LZRW-compressed game assets from resource forks

Learning about resource and data forks

After cleaning up the source code of Reckless Drivin’, my first goal was to read the contents of the file called Data. This file includes the images, sprites, textures, level data, sounds, and other game assets. The commit that introduced the file is titled “resource fork”. The project’s readme says:

To be able to upload this to git… the resource forks of the rsrc files have been moved to the data fork.

Until this point I hadn’t ever heard of resource or data forks, so I turned to Wikipedia and archived Macintosh articles to learn more.

On some file systems, like Apple’s HFS+ that Reckless Drivin’ was developed on, files can contain forks). Forks are sets of data associated with the file in the file system. Although there was support for an arbitrary number of forks, most Macintosh software only used the data and resource forks. On PowerPC machines the executable was stored in the data fork, and the resource fork stored structured data associated with the file like icons, images, and sounds. These resource forks are essentially databases stored alongside an executable.

Each resource stored in the resource fork has a four-byte type identifier, usually a four-character label, and a two-byte ID number. An example resource type would be PICT for picture. This table lists some of the most common types of data stored in resource forks.

From C source code, a resource can be accessed with the following call, passing the type¹ and the id:

Handle picture = GetResource('PICT', 1000);

This returns a handle pointing to memory where the resource was stored. Many tools like git only read information from the data fork, which is why Jonas moved all the data from the resource fork to the data fork before committing in git.

$ derez -useDF Data > unpacked_data
$ head unpacked_data
data 'Pack' (128, "Object Definitions") {
	$"0000 2C78 0000 0000 A010 009E 0000 0001"            /* ..,x.......... */
	$"0280 0102 04F8 0081 0102 0538 0022 2282"            /* ........8."" */
	$"0204 7800 8302 01B8 0084 0201 F800 8501"            /* ..x........ */
	$"0106 3844 4400 8602 0578 0087 0202 B800"            /* ..8DD...x.... */
	$"8802 02F8 0089 0101 0788 8838 008A 0206"            /* ......8... */
	$"7800 8B02 03B8 008C 0203 F800 8D01 0110"            /* x.......... */
	$"1108 3800 8E02 0778 008F 0204 B800 9002"            /* ..8...x..... */
	$"00F8 0091 2122 0100 0938 0092 0200 7800"            /* ..!"...8...x. */
	$"9302 00B8 0094 0200 F800 4244 9501 000A"            /* ......BD... */

struct Resource {
  char type[8];
  uint32_t id;
  uint32_t length;
  char *data;
};

Reading the packs

Most of the game data is stored as Pack resources. These packs are read into memory by the uint32_t LoadPack(int num) function.

uint32_t LoadPack(int num) {
  uint32_t check = 0;
  if (!packs[num]) {
    packs[num] = GetResource("Pack", num + 128);
    if (packs[num]) {
      if (num >= PACK_ENCRYPTED) {
        check = CryptData((uint32_t *)*packs[num], GetHandleSize(packs[num]));
      }
      LZRWDecodeHandle(&packs[num]);
    }
  }
  return check;
}

This function loads a Pack resource given an ID number and stores a handle in a global array. This function also contains two roadblocks to interpreting the resources:

1. Some packs are encrypted, as indicated by the call to CryptData() when the pack number is above a certain value. This is because levels 4 through 10 were restricted to registered players.
2. Each pack is compressed by the LZRW algorithm which is decompressed by a call to LZRWDecodeHandle().

Because decompression is needed for all Pack and PPic resources, that is what I will focus on for the remainder of this post. I will describe the process of decrypting the level data in a later post.

LZRW decompression

LZRW (Lempel-Ziv Ross Williams) is a family of compression algorithms created by Ross Williams that are focused on speed rather than high compression. Ross created seven variants of the algorithm before fears related to software patents caused him to leave the world of data compression behind. In 1997 he released all of his algorithms on his “compression crypt” website. Ross’ website is not always available online, so here’s an archived link to his homepage where the seven compression algorithms are located. A compression benchmarking repo on GitHub also contains the source code for most of the variants, with an additional header file to create a common interface for the different source files.

Unfortunately the Reckless Drivin’ source code shared by Jonas is missing the file for LZRW decompression. Ross Williams’ website includes all the source code for the algorithms, but there are seven variations, and Reckless Drivin’s code doesn’t mention which of the seven was used.

The variants all follow a similar interface, so I created a test program to load a pack and decompress it with the chosen algorithm. When compiling this program I enabled the address sanitizer in gcc with -fsanitize=address. With no other indication of success, I decided that If the pack would decompress without memory issues, then the LZRW variant worked.

At this point I thought this task would be simple: Decompress all the packs with each variant until one variant decompressed without memory issues. I soon found a problem because more than one of the variants succeeded without issue. Both lzrw3 and lzrw3-a would successfully decompress all of the files with no memory issues. Some of the other algorithms like lzrw1 would decompress some of the resource packs, but would fail with out-of-bounds memory accesses with other packs.

Because multiple variants decompressed successfully, I would now need to look at the bytes in the decompressed resource packs to determine if the data was decoded properly. Because the Pack resources are cast directly to various structs in Reckless Drivin’, it would be very difficult to find any obvious patterns in the data. So I turned to the PPic resources instead.

data 'PICT' (128) {
  $"0078" /* picture size; don't use this value for picture size */
  $"0000 0000 006C 00A8" /* bounding rectangle of picture at 72 dpi */
  $"0011" /* VersionOp opcode; always $0011 for extended version 2 */
  $"02FF" /* Version opcode; always $02FF for extended version 2 */
  $"0C00" /* HeaderOp opcode; always $0C00 for extended version 2 */
          /* next 24 bytes contain header information */
    $"FFFE" /* version; always -2 for extended version 2 */
    $"0000" /* reserved */
    $"0048 0000" /* best horizontal resolution: 72 dpi */
    $"0048 0000" /* best vertical resolution: 72 dpi */
    $"0002 0002 006E 00AA" /* optimal source rectangle for 72 dpi horizontal and 72 dpi vertical resolutions */
  $"0000" /* reserved */

$ xxd ppic-1000 | head -n 2
00000000: 0000 0000 7f10 0000 0000 01e0 0280 0011  ................
00000010: 0c31 3233 00ff fe00 0248 3132 3334 3536  .123.....H123456

data 'PPic' (1000) {
        $"0005 7F10 0000 0000 0000 7F10 0000 0000"
        $"01E0 0280 0011 02FF 0C00 7400 FFFE 0002"

Ghidra saves the day

Ghidra, an open source reverse engineering tool developed by the NSA is a wonderful resource for decompiling software. This isn’t the place to explain how to use Ghidra, but if you haven’t tried it, it is incredible. Within minutes I had the original LZRWDecodeHandle() function decompiled in Ghidra.

The highlighted line of decompiled code is:

.compress(2,uVar3,**param_1 + 4,iVar1 + -4,*piVar2,auStack24);

This function call has the same signature as the one found in Ross Williams’ code that I used when blindly reimplementing LZRWDecodeHandle(), but with one interesting difference that stands out. The first four bytes of the packed data are ignored, as indicated by the +4 offset to the array, and the -4 offset to the length of the array. I added the same offsets to my implementation and recompiled.

lzrw3a_compress(COMPRESS_ACTION_DECOMPRESS, working_mem, **handle + 4,
                handle_len - 4, dst_mem, &dst_len);

The following is PPic 1000 after decompression with the memory offset.

$ xxd ppic-1000 | head -n 2
00000000: 7f10 0000 0000 01e0 0280 0011 02ff 0c00  ................
00000010: fffe 0000 0048 0000 0048 0000 0000 0000  .....H...H......

This exactly lined up with the format described in the QuickDraw specifications, with 0011 02ff 0c00 FFFE identifying this as an extended version 2 image, and the remainder of the header specifying the dimensions (640x480) and other metadata. This confirmed that the lzrw3-a algorithm was the correct variant! I was super excited, and this breakthrough motivated some very fast paced development on Open Reckless Drivin’ over the next few weeks.

Although at this point there still wasn’t anything besides bytes to show for all my work, it was both rewarding and motivating to have access to Reckless Drivin’s game assets in my reimplementation. This post has grown long enough, so I won’t describe the process of writing a QuickDraw interpreter until a later post. This process of unpacking the resource forks taught me a lot of interesting things about Macintosh computers, and I hope you found this interesting too!

1. The four characters of the type identifier were written inside a character literal which would be interpreted as a single 32-bit integer. This code emits a -Wmultichar warning in gcc, but it does work. In my reimplementation I chose to use four-character string literals instead of relying on the implementation-defined behavior of character literals.↩︎︎

2. The full output was 362,666 lines of text (27MB of data).↩︎︎

3. The conventional resource type for a QuickDraw image is PICT. I assume that PPic means Packed PICT, but unless Jonas reveals what he meant by PPic we will never know.↩︎︎