DOS Games Discussion

Hi all,

Let me write about a collection of repositories, found here: https://bitbucket.org/gamesrc-ver-recreation/

What you can find in these repositories for most is reverse-engineering of code related to DOS games. However, rather than reversing a whole DOS game, what I usually do here is reverse different versions of games which were already open-sourced.

This idea came to me after the open-source release of not just one, but multiple DOS versions of Softdisk's Keen Dreams title. To be more specific, I downloaded the git repository and then had a look at the revision matching shareware v1.13. Using what I assumed to be the exact process for making the EXE, I got precisely the original EXE from the 90s, byte-by-byte. This process included usage of the right compiler version, as well as packing the EXE with LZEXE 0.91.

As for the above-mentioned repositories, the success rates greatly change, depending on the game in question, and especially the compiler in use (including the compiler version).

A more successful example is Wolfenstein 3D. Here, the executables from multiple versions of the game, including differing Apogee versions, can be fully recreated byte-by-byte, if done with the right tools and the right steps. It's possible that some luck might also assist, due to unintentional dependencies of the compiler and/or any other tool on the environment.

In other cases, there may still be differences in the output files. For instance, due to the way certain versions of Watcom C32 (v10.0b and older from what I know) behave, it may fill gaps between C string literals with data which depends on the environment, and/or on the textual contents of the input source code. This applies to the game code from Duke Nukem 3D: Atomic Edition v1.5, just for the example.

My most recent work is on the Heretic and Hexen sources; This only covers id Tech 1 code, not DMX. I did technically use DMX files in order to build the exes, but that's it; The changed code was from id or Raven.

For Heretic, this covers shareware and registered v1.0, which are quite similar. Another covered version is 1.2, which isn't very different from 1.3. The open source release itself turned out to match 1.3 in behaviors.

For Hexen, the open-source release basically matches one of two versions identified as "1.1". It's the latter out of the two, and the only practical difference between them is the addition of two checks to A_SoAExplode before spawning a monster, with one of them being a check of the boolean variable "nomonsters". The VERSION_ID string in the MAKEFILE was also changed from CBI to BCP.
Most of the work was actually in reversing version 1.0, or as it was later found out, two very similar variations of it again. The only actual difference between them is that the latter build had the addition of a check of the boolean variable "i_CDMusic" to P_SetupLevel, before calling S_StartSongName.

Regarding sound effects and music, back when Nuke.YKT was working on his PCDoom port, he made a wrapper over DMX, which is actually using the Apogee Sound System. While it obviously sounds different from DMX, it's a GPL-compatible alternative.
Therefore, I decided to add this wrapper under its own git submodule named "apodmx", with a few modifications. It can be built as a .LIB file which can be used with the Heretic or Hexen sources. You also need a compatible AUDIO_WF.LIB file from the Apogee Sound System.

Hey there,

I've gotten another addition for the Hexen repository. To make this short, it should now cover code which is more-or-less fully equivalent in behaviors to the 4-level beta (Oct 2 1995).

I also forgot to mention here earlier, that before working on Heretic 1.0-1.2, I also covered a later 4-level demo of Hexen (Oct 18 1995).

Regarding the 4-level beta, there's probably too much to write about the code itself, so I'll simply mention the following examples of information:
- This revision has code for the removed fly creature (https://doomwiki.org/wiki/Fly).
- The 4-level demo from Oct 18 1995, previously named HEXDMO10 in my repository, was renamed HEXDM10B, while the earlier demo was named HEXDM10A.
- I considered renaming HEXDM10B back to HEXDMO10, while HEXDM10A would be renamed using (a subset of) the EXE's original modification date; Reason being, the latter is identified as a beta in-game, and I already did a similar thing with a Wolf3D proto. beforehand.
- However, both versions are referred to as demos in the README.TXT files from 1995. I also don't currently recall any mention of a version number, like 1.0. For now, I just keep using the names of HEXDM10A and HEXDM10B.
- I originally started to inspect the 4-level beta as a possibility after finishing with Heretic 1.0-1.2. I eventually returned to the beta more recently. What's clear is that it required more work to recreate the code than the previously covered versions of Hexen; Maybe even more than all previously covered versions of Heretic and Hexen, combined.
- In addition to the previously known issue of global variables not being fully ordered as in the original EXE, there are also a few functions that I couldn't get their compiler-generated layouts to fully match. Unless I missed anything, they should still match in behaviors. The functions in question are A_Quake, P_XYMovement and P_ZMovement. The latter's C code was actually not changed by me at all.

So, it's possible that anybody who tries to cover all versions from a single EXE might find it difficult to do so at this point.
That is, unless you go for what I did in Reflection Wolfenstein 3D, as it more-or-less covers separate builds of different game versions in a single exe.

As of this post, the following additions are now in:
- 3-level shareware beta of Heretic (Dec 20 1994).
- Retail store beta of Hexen (Sep 26 1995).

These will probably be the last additions related to Heretic and Hexen. As usual, there's no promise regarding any future endeavor.
To summarize, we ended with 5 distinct DOS builds of Heretic, and 7 of Hexen.

As I suspected, it still took significantly more time to work on the beta release of the 4-level Hexen demo (given preceding work on v1.0 and the demo re-release), compared to the work on the aforementioned two betas of Heretic and Hexen.

In certain technical manners, the two Hexen betas might be close, even if obviously not identical. The aforementioned removed fly code is the same, just for one example.

I did have issues with the generated layouts of the following functions in the Hexen betas again: A_Quake, P_XYMovement and P_ZMovement. As mentioned earlier, the latter's C code was actually not changed by me at all.

In the case of Heretic, I suddenly spotted a different layout for M_FindResponseFile in a recent build of 1.2. I did it see beforehand, also in Hexen, but I thought that it was not reproduced after I finished working on each of the differing Heretic versions, excluding the beta. At the least, the code size, including padding for alignment, remained the same, so the other functions did match in layouts and locations.

Having some updates, both major and minor.

- Thanks to Nuke.YKT's efforts, Doom and Strife are now both covered. He used the Heretic sources and other sources in order to transform Linux Doom into faithful Final Doom DOS sources. I added older Doom versions to the tree, and occasionally also assisted with Strife a bit, here and there. He originally made a post in a separate thread within other forums, which I'll quote here.
- Nuke.YKT further updated the apodmx repository, i.e., his DMX wrapper that uses the Apogee Sound System. This technically impacts not just Doom or Strife, but also Heretic and Hexen.
- Ken Silverman figured out how was Ken-Build's GAME.EXE (probably) built from the sources as uploaded in November 2002. So, while the code was there beforehand, I added it as an option to the BAT files and make file.
- For a few exes that can be built from the duke3d tree, 3 bytes would differ before adding these linker directives: "segment type code lo", "segment type data lo". As it recently turned out, an alternative fix was the simple removal of the directive "system dos4g".

Additionally, Nuke.YKT is now a part of gamesrc-ver-recreation.

There might still be restrictions on what's uploaded to gamesrc-ver-recreation. For instance, a reversed engineered game is generally not covered. Exceptions are still a possibility. For instance, after Blake Stone: Planet Strike was open-sourced by Apogee, it was stated that the Aliens of Gold sources were assumed to be lost, thus explaining their lack. Therefore, I didn't mind building upon Blzut3's earlier reverse-engineering efforts, and later uploading reconstructed sources for the game.

Nuke.YKT wrote: Hi,

As you know Doom source code as released in 1997 was not for original DOS version, but rather was Linux version cleaned up by Bernd Kreimeier. During clean up, code specific for DOS version was stripped out. Another big change was reorganizing header files, specifically doomdef.h, p_local.h and r_local.h were split to smaller header files. This is where Heretic sources comes in handy. Raven released their code as is, without any sort of clean up. Specifically it has majority of Doom's DOS specific code and header files organization is very close to Doom's original form.

The goal of this project is to combine both these sources and get codebase that is close to the Doom's original codebase as possible. Actually I had this idea for a pretty long time and had previous attempts in the form of PCDoom and PCDoom-v2. This time I decided to start mostly from Heretic sources, gradually reverting Raven's code changes by comparing it to Linux Doom sources. I targeted id Anthology Final Doom EXE as it was closest to linuxdoom sources. Eventually I got compiling EXE, but as expected it had lots of differences to original EXE files. Then I gradually fixed all the differences and got EXE that identical to original EXE by behaviours. Once I covered both revisions of Final Doom, NY00123 volunteered and also covered a lot more Doom revisions down to prototype v1.666. This also includes Chex Quest.

Meanwhile I decided to try to do the same for Strife executable using restored Doom code. As Strife source code was never released, I had to rely on reverse engineering of original EXE file. After finishing initial reversing resulted EXE had some differences to original EXE. Eventually after lots of trial and error and not without some help from NY00123 we eventually got EXE that identical to original EXE (up to usual garbage data between string literals and differences due to __LINE__ macro).

As original release of Doom this does not include proprietary DMX code. You can use APODMX replacement library instead to get complete GPLv2 compatible codebase.

TL;DR These repositories recreate different revisions of Doom and Strife codebase. Compiled EXE files are identical to original EXE in term of behaviours. In total both repositories cover 15 different revisions of Doom, Strife and Chex Quest.

List of covered Doom revisions:

Code: Select all

Doom II prototype v1.666
Doom Shareware / Doom II early v1.666
Doom v1.666
Doom II v1.7
Doom II v1.7a
Doom II French v1.8
Doom / Doom II v1.8
Doom / Doom II v1.9
Doom v1.9 Special Edition prototype exe
The Ultimate Doom
Final Doom
Final Doom, later id Anthology revision
Chex Quest

List of covered Strife revisions:

Code: Select all

Strife v1.3
Strife v1.31

Doom repository
Strife repository

Going to quote Nuke.YKT again:

Nuke.YKT wrote:Update to the Strife restoration

A couple more revisions of the Strife executable are covered now: registered v1.1(aka v1.0) and registered v1.2. Both reconstructed EXEs are identical to the original EXE files (up to garbage data between string literals and differences due to the __LINE__ macro). Thus gamesrc-ver-recreation now covers all known registered versions of Strife.

The next obvious step is to try to cover the demo versions of the Strife, but I expect much more differences because both demo versions use much earlier revisions of the executable, so I guess I'll leave this for later.

As always, great work!

Here's another quote of text from Nuke.YKT:

Nuke.YKT wrote:To celebrate upcoming doom's 30th anniversary gamesrc-ver-recreation now covers original shareware v0.99/v1.0 release

https://bitbucket.org/gamesrc-ver-recreation/doom

And so, Nuke.YKT decided to go even further than version 1.2, all the way back to the very first proper shareware release! On a side-note, so far, outside of Chex Quest, all covered versions included clear support for Doom II, to different extents. The sprite and mobj definitions were also the same. With shareware v0.99/1.0, the situation is obviously different now.

By the way, I've had really minor updates to batch scripts in the Doom, Strife, Build and Duke3D sub-modules. Earlier in this November, it was discovered that the lack of use of the command-line switch /S for the program CHOICE was apparently a mistake. It might not be observed as a problem in DOSBox v0.74-3, but it otherwise was. Let's hope this is fixed now.

To begin with, that was supposed to go earlier:

MrFlibble wrote: ↑ As always, great work!

Thanks for showing interest again!

Now, for a small update, after bringing up the topic of running a reconstructed Doom v0.99/1.0 binary that uses APODMX, Nuke.YKT realized that APODMX was incompatible with v0.99/1.0. That was correct, due to API changes in the functions SFX_PlayPatch and SFX_SetOrigin: https://doomwiki.org/wiki/Random_sound_pitch_removed

Thus, I decided to update the APODMX repository, so it should let you build two versions using the different APIs in separate sub-folders now. I also updated all of the Doom, Heretic, Hexen and Strife repositories accordingly. I further applied technical modifications to the Heretic and Hexen repositories related to the DOBUILD.BAT and DOCLEAN.BAT scripts, making them and related files closer to the ones for Doom and Strife (and partially also Build and Duke3D).

Hi,

I'm having additions to the wolf3d repository after more than 6.5 years, covering two added revisions of code. But there are also a few more general points to add:
- The various repositories aren't submodules of the "gamesrc-ver-recreation" repository itself anymore. Nuke.YKT and I haven't really been using this feature, so I've decided to do this change. The repositories still exist, I just don't use the submodule feature.
- While currently done in the wolf3d tree only, doing so in other repositories isn't necessarily out of the question. So, a significant subset of the file notes-restoration.md was replaced with a new README.md file, aiming to replace it. It should be much smaller, albeit it's still not necessarily a small file. I left various notes under notes-restoration.md but I currently have no guarantee of them being up-to-date. Interestingly, the older incarnation titled "game-srccode-ver-recreation" had a quite small README.md file added to it, so this can be seen in part as going back to the roots.

As for wolf3d, before getting to the aforementioned two code revisions, I also made these changes:
- Output build directories were collapsed. That should be more consistent with other repositories, the Blake Stone repository being just one example. For instance, WL1AP10.PRJ's output dir was changed from "OBJ\WL1AP10" to "WL1AP10".
- One directory was mistakenly named "STATIC\WOLF3D\WL920512". Instead of "WL920512", "WL920312" is used now.

Let's get to the added revisions themselves.
- The first one is what I suspect to be a quite unknown April 16 1992 prototype executable, somewhere in-between WL920312 and shareware v1.0. There wasn't a lot of new code added but I did have to go through existing macros, so it took some time. There were a few exceptions here and there, like placeholder code under the function "Victory", but what was there beforehand could be reused otherwise.
- Secondly, under a new directory, I added a separate modification of the open-source release of Wolf3D into code more-or-less matching a very early ROTT prototype. Known as ROTT0993, it's still essentially a modified Wolf3D. There wasn't a lot of code added and I didn't introduce new game data to the repository. This revision draws textured floors and ceilings and also has additional hotkey checks for debugging. As for the output EXE, expect differences in debugging symbols. I was reducing them, but even after making timestamps match, I still had 29 differing bytes.

To finish, here's one more thing. Getting back to the wolf3d repository after about 6.5 years, short of the few occasional edits done in-between, is not something I recall doing beforehand under gamesrc-ver-recreation right now. One point was known to me earlier, but having another look after a while further clarified it. Basically, the use of pre-processor macros did make it possible to cover many versions under a single codebase - currently 19 in total. On the other hand, over time, it may become difficult to maintain this code and track it with the added pre-processor macro checks. Then again, they did make it quite convenient to see differences between versions under a single window, without using a diff program. But that can be a challenge when it comes to supporting multiple versions in source ports. In ReflectionHLE's case, I mostly bypassed it by repeatedly rebuilding ported Wolf3D sources with a bit different configurations and then linking the builds into a single binary. Definitions of pre-processor macros inherited from gamesrc-ver-recreation were changed across these sets of objects. I still made adjustments, if due to the replacement of the macro UPLOAD with a variable of the same name, or any other technical reason.

I'll ask a somewhat unrelated question: consider someone wanted to build a limit-removing Heretic binary from the source, what code changes are required to accomplish that?

MrFlibble wrote: ↑ I'll ask a somewhat unrelated question: consider someone wanted to build a limit-removing Heretic binary from the source, what code changes are required to accomplish that?

I haven't done the right work on the sources to know the answer for sure. You might get a possible answer if you check the changes listed, e.g., for Doom2-plus. A good example also coming to my mind earlier (and having its uses in Doom2-plus) is changing the values of macros representing array / data sizes. I still don't have enough technical familiarity to be sure that is sufficient, though.

Thanks!

I'll start by quoting Nuke.YKT first:

Nuke.YKT wrote:Hi,
To celebrate 30th anniversary of Raptor, gamesrc-ver-recreation project now covers all known EXE revisions of Raptor. As a reminder, gamesrc-ver-recreation is a project which aims to reconstruct EXEs of various games accurately as possible using their released source codes as a base. Like with Doom engine games EXE restorations, this release does not include proprietary DMX code.
This project started in November 2021, when copy of source code of Raptor and parts of surrounding code (GFX library) were received from original developer, Scott Host. Using code we got and copy of DMX code we already had, me and NY00123 starting piecing them together to get functioning DOS exe. Eventually we managed to compile it, but as expected, there were few differences compared to original v1.2 EXE. With some more effort we found and fixed all the differences, and got EXE that matched original EXE byte-by-byte. Using this, I immediately started reconstructing sources for earlier releases. v1.1 release was reconstructed fairly easily within a day or so. Reconstruction of v1.0 though, proved to be quite difficult, as the DMX version used in this version was much earlier than any version we had (for example, Gravis ultrasound code was completely different, and still was using hardware mixing like Doom 1.3 and earlier did). We weren't able to finish this at the time. Shortly after this we moved to the Doom and Strife source code restoration, and DMX restoration we did for Raptor v1.2, turned out to be useful there. Specifically, we found out that Doom 1.7/1.7a uses exact same DMX revision.
Much later though, in September 2023, I was trying to reconstruct Doom 1.2 executable. As it uses DMX version quite close to Raptor 1.0, I decided to look at this again. This time I had much more luck, and thus Raptor v1.0 finally was reconstructed.
Shortly after this, branch of v1.2 reconstruction was published on github by skynettx as official GPL release of Raptor source code. Based on this, NY00123 put together this repository for public release.

https://bitbucket.org/gamesrc-ver-recreation/raptor

Now, adding my words as well.

1. Work on Raptor v1.2 was started by Nuke.YKT after receiving original files from Scott Host of Mountain King Studios. I joined later with the efforts, albeit I consider most work to still be done by Nuke.YKT. v1.1 was indeed covered by Nuke.YKT during the span of a single day, more-or-less. For v1.0, I helped a bit, mostly with Sound Blaster related code in DMX, but it was otherwise Nuke.YKT's work again. What we ended with were 3 separate private git branches representing versions 1.0, 1.1 and 1.2. They weren't ready for becoming public, due to the presence of proprietary DMX code and other possible aspects of code review. I eventually took the open-source release of v1.2 from skynettx' repository and added versions 1.0 and 1.1 to the same codebase, while preparing additional files (like Raptor's makefile and helper scripts) for support of all covered versions. I also added README.md, partially based in form on notes previously written for other games.

2. One more thing is the time span, given that it all started back in November 2021. While originally preparing the open-source release of Raptor v1.2, we were waiting to see if we'd maybe be able to use DMX. There were also internal pending code review notes. At some point, Scott Host seemed to want something a bit different; While he'd be working on a remaster of Raptor without a legacy mode, Nuke.YKT would work on a more direct port of the original game, with the idea that - possibly as an alternative to the DOS sources - the port itself would be the one to go open-source. Eventually, as written by Nuke.YKT, he had finally gotten back to Raptor v1.0 source reconstruction and finished it in September 2023. Host showed support for the original plan again, and not much later, skynettx published the sources.

3. I'll finish with the topic of forum threads. People have maybe noticed that the Duke4.net and/or Doomworld threads on the topic were new ones. In fact, Nuke.YKT already started a separate related Doomworld thread earlier, covering reconstructed Doom and Strife sources. Beforehand, there was generally just a single thread of mine about gamesrc-ver-recreation in each forum which had any; For a portion of these threads, each of them would be mostly specific to one game or engine, but that wasn't the case for all of them. I had actually considered deprecating my Doomworld thread covering Heretic+Hexen and migrating to Nuke.YKT's thread about Doom+Strife as a new general one. However, it was eventually decided with Nuke.YKT to start a new one for Raptor (at least on Duke4.net and Doomworld for now). A few reasons to bring up:
- If a post about a newly covered game (e.g., Raptor) were to be made in a more general existing thread, then people who see the title wouldn't immediately be aware of Raptor being the topic; They wouldn't know without inspecting the thread.
- As an added post to a previously created thread, chances are it would look like a small update, rather than something entirely new.
- Finally, I think that usually, one person's reverse-engineering projects covering different code-bases (e.g., games) are considered to be separate endeavors. Sure, we've had varying repositories under gamesrc-ver-recreation, and there's more than enough to share (like the Apogee Sound System). But I still think the notion of such projects being separate makes sense.

NY00123 wrote: ↑

MrFlibble wrote: ↑ I'll ask a somewhat unrelated question: consider someone wanted to build a limit-removing Heretic binary from the source, what code changes are required to accomplish that?

I haven't done the right work on the sources to know the answer for sure. You might get a possible answer if you check the changes listed, e.g., for Doom2-plus. A good example also coming to my mind earlier (and having its uses in Doom2-plus) is changing the values of macros representing array / data sizes. I still don't have enough technical familiarity to be sure that is sufficient, though.

Nevermind, recently I found out that someone has already created limit-removing versions of all Doom engine games based on your work:
https://sourceforge.net/projects/doom-dos-ports/files/