Kotor 1 (GoG) Reverse Engineering

[Lane 12]

Hey everyone!

My Name is Lane. Some of you know me from Kotor Speedrunning, and others from my various Youtube exploits.

I don't ever really post on Deadly Stream, but I've been lurking in and around the KotOR modding community for about a decade now.
I have a degree in computer Science and decided to put it to good use, and reverse engineer KotOR 1 (the GoG version). This has been an on-going project for about 2-3 years now, and I've been sharing my progress with friends, and in some Youtube Videos. 

However, I've pretty much hit a wall with what I can do with this effort now. So I wanted to release my progress publicly, so other smart and clever people can start doing fun stuff with this. Linked below is a Google Drive with several useful things:

• A Ghidra SARIF export that contains all data types, function labels, parameters, Classes, and other additions I've made
• A Ghidra Format XML that contains the labels and function adjustments I've worked on. This is lighter weight than the SARIF file, but has more limitations when it comes to import fidelity.
• A generated `.h` file, that contains the Header information I have pieced together over time. Even lighter-weight, and more limited than the above

What this is not:

•     True Debug Symbols for kotor
•     BioWare Intellectual Property
•     A runnable or compilable program
•     Kotor's Source Code
•     A reverse engineering of Kotor 2
•     A reverse engineering of the Steam version *

* A note about Kotor 1 Steam: While this reverse engineering effort targeted the GoG release of KotOR 1, the Steam version has MANY similarities; often times having identical memory addresses for most functions. Any Patch made for the GoG version can be pretty reasonably ported to Steam with a little bit of effort.

What this is:

•     A decently representative result of what Kotor's debug symbols might look like (format and terminology pulled from the MacOS symbols, and existing NWN docs)
•     A research-based labeling and reverse engineering of the GoG version of Kotor 1
•     A labor of love for the past several years that I'm happy to share

Why this is cool/important:

•     This provides a stepping off point for creating proper patches for KotOR 1
•     This also provides a means for researching underlying issues with things such as memory management, graphical limitations, and compatibility
•     This also provides a researching angle for coming to understand some of the more mysterious file formats, and how they interact with the game itself
•     There are also a variety of fundamental similarities between this and KotOR 2. Which may unlock some insights for that game
•     This is also the first step towards a proper re-compilation (though that is a long-ways off)

How do I use this?

•     You need Ghidra installed, with a modern Java Runtime
•     Create a new project, and import swkotor.exe (as purchased from GoG)
•     Open the EXE in Ghidra's code browser
•     When it asks if you want it analyzed, select 'Yes"
•     The default analyzers are fine, technically you could speed this up by stripping out a few unneeded analyzers
•     The analyzers will take several minutes to complete (progress can be tracked in the bottom right)
•     Once the analyzers have run, we can proceed
•     Select "File > Add To Program..." and select the SARIF (or XML) file (download below)
•     The importer will analyze the symbols and apply them to the project
•     You now have a decently labeled/decompiled instance of KotOR 1

Limitations:

• 99.9% of the functions have been labeled, however there were a few stragglers that I was never able to work out. These will appear as `FUN_<address>`
• 92.3% of the Data is labeled, with stragglers being named `DAT_address`
• Data Types are VERY incomplete. The labeled ones consist mostly of frequently used types, and known fields. Unknown fields are marked `field<index>_<offset>`
• Virtual Function calls are very under labeled (largely due to the difficultly of labeling vtables in Ghidra). Though you can determine the underlying function by applying the offset to the associated Class vtable.
• Most functions have only automatic variables defined within their decomp. Typing and purpose of underlying variables beyond function names, and parameter types, are left up to inference.
• Overlapping functions. Certain functions overlap in this compilation, due optimizations within the Visual C++ runtime. As a result some functions such as `GetProperty0x30` are shared by multiple classes, and thus lack a name-space. You can usually work out their purpose by checking the associated data type at that offset.
• If you used the XML import, you will be missing a lot of typing and Function Class/Namespace info

Final Notes:

• Please feel free to ask me any questions about this effort, or any thing strange you might find within the decomp. I've grown to be quite the kotor expert over the years, and I'd be happy to share any insights.
• You can reach out to me on Discord @lane_d, I'm in the Kotor Reddit server, as well as the kotor speedrunning server.
• I will be periodically posting updates to this drive, whenever I get the chance to work on this more. If anyone has any major contributions they'd like to see added, please reach out! I'd be happy to chat.
• Both the XML and SARIF formats have some limitations, unfortunately they are the best I can do without publicly sharing Bioware intellectual property. If there is something missing from these that you are in need of, please reach out and I can see if I can help you. (Note: I will not be abetting piracy though. Buy the damn game, it goes on sale all the time)

Google Drive Link Here

 

 

[Lane 12]

(Cross posted from Discord)

More progress has been made on doing some patching in KotOR 1!
Here I demonstrate adding in a simple test function mto the NWScript VM, and running it in-game.
The summarize what all is happening here:

• We patch the space allocated and existing bounds checks for script IDs (I extend the space such that we could have up to 1000 total functions, but I can always extend it more in the future)
• I patch the existing initialize commands function in the game to continue onto my own block of code where I initialize this new `TestScriptExtension` command to call a simple execution script (using an identical pattern to the existing scripts in this game)
• I added this command to nwscript.nss so `nwnnsscomp.exe` would recognize and compile the script with this new function (special thanks to Edward T. Smith for making a tool that is so well designed that it works out of the box for our very custom situation here)
• Set it to run when I talked to carth
• The actually function just prints some debug text to Windows Debug, however, we can also have them call other game functions, implement custom stack logic, among other things. 

There's still A LOT of work to be done:

• I want to clean up the patcher/launcher so that using it is actually intuitive
• I want to add a bunch more commands to the script extender
• I want to tutorialize the creation of both new commands and patches, so that the lovely and creative people of kotor modding world can contribute in the future
• Setting up Kotor 2 compatibility
• much much more

Let me know if ya got any questions!

Edited Saturday at 07:01 PM by Lane

[Lane 12]

(Cross posted from Discord)
https://www.youtube.com/watch?v=dAHxy0tlDms
A new day, a new Demo. Here I show File I/O working in NWScript.

• These just pipe into the existing c-standard `fopen`, `fclose`, `fread_s`, and `fwrite_s` functions.
• This allows for a rudimentary way for scripts to save data to be recalled later.

[Lane 12]

The early progress on the patch manager I've been using as well as releases can now be found on GitHub

[Lane 12]

Another demo, this time showing off some of the Kotor 2 creature stats functions in Kotor 1.
A few discoveries during this process:

• The various "BonusForcePoints" functions kotor 2 implements are going to be non-trivial to implement unfortunately, as Force point calculations are very hard-coded, and I'm not aware of any convenient ways to tip the scales there. `SetBonusForcePoints`, `AddBonusForcePoints`, and `GetBonusForcePoints` are still on the wishlist, they'll just take a bit longer as I patch the FP logic.
• `Modify<Reflex/Fort/Will>SavingThrowBase` functions are in a similar camp. Anytime the game calls `GetBaseWillSavingThrow` (or related), it live calculates the value based on class level and specific feats. Kotor 2 gets around this by adding an additional field to the creature stats structure that stores a modifier they use in this calculation. So again, these are still on the list, but will require a bit more love to get working than the commands I show off in the demo video

All-in-all progress is going well. I'll be cutting another alpha release later, for anyone interested in experimenting.

As always please share questions and comments on this effort!

[Lane 12]

Two demos in one day! In this one I show, kotor 1 working with more than 256 rows in placeables 2DA (something that previously wasn't possible).

 

[DarthParametric 3,898]

Something extremely useful for mod testing/debugging would be to make the ShipBuild() function functional. That would allow external script logging as well as on-screen script notifications using the vanilla include debug functions like Db_PostString. Although I'm not sure how much of the related logging code was just excised altogether.