DrMcCoy

Huh, interesting. That might be a bug in xoreos-tools' erf packer? I might be able to look a bit at that during the holidays. Unless you want to tackle debugging this for me?

I didn't say that. No one is currently actively working on an Android port. But I have that on my radar and I do have ideas and plans. So I'm afraid I can't give you an estimate. Heck, I can't even give you an estimate on when KotOR is playable in xoreos, or when xoreos is "user-ready", or anything, really. I can only say that more contributors = faster work, and we really need more contributors. I know, I probably sound like a broken records, but it is what it is.

That...depends. In C, you can't. In C++, you can overload a function, as long as the signatures, i.e. the number of parameters and their types, are different (internally, C++ "mangles" the identifiers, using a scheme that encodes the parameter types into the name). In NWScript, you can't either. The compiler wouldn't know what to do. When you write a script in NWScript, you use the compiler (for example, a modifed nwnnsscomp for Kotor, based on a open source reimplementation by the "OpenKnights Consortium" of the original NWN compiler, based on the work of Torlack who RE'd the stuff back in the day) to turn the C-like NWScript source (nss) into NWScript bytecode (ncs). Internally, the bytecode doesn't know much about the engine functions anymore. The compiler looks into the nwscript.nss file to look for the name of the function, and then counts at the index of the function within the file. I.e. if it's the first function in nwscript.nss, the second, the third, etc. That number it puts into the bytecode file ncs. So in the bytecode, you only have pushes onto the stack, for the parameters, that the compiler arranged there (and space for the return value), and then "call engine function 523". The game, when running the script bytecode, takes that number, then knows what to do, and which parameters it takes. I.e. with NWScript, you can't change the number of parameters, and the types, because that knowledge is hardcoded into the game. (Well, it's not impossible, just quite hard.) What you could, theoretically, do is add a new function to the bottom, then manipulate the look-up function in the game to recognize it. You'd still have to add the functionality itself, though. It's still hard, just slightly less so. (This could also be used to emulate overloading, by extending the compiler to map the overloaded functions onto later indices. Might not be worth the hassle, though.) We, err, don't have that implemented yet. It's not really that it's forbidden. It's just that you're setting yourself up for a lot of hard work and pain. And potentially failure, of course. I know, coming from me, who's RE'ing games for fun, and who has too many projects and stuff to do for 10 lifetimes, that's a bit ironic. Still, I still think the "best" way to resolve hardcoded things in KotOR is going over xoreos: getting xoreos in a playable and "done" state, so that improvements can be build upon xoreos. Not that this isn't a massive amount of work either, far from it, just that this opens up so many more possibilities. If you're adding a single thing to KotOR using the manual binary-hackery method, it's way harder to be sure you didn't miss anything, you can't verify easily when you continue to work on it that you're not breaking it, it's harder to integrate with other people's work, etc. You could instead do something similar, figure out how things for in the game and reimplement that in xoreos, bringing xoreos further along. It will not give you the gratification of having changed something in the original game as quickly, and still need to wait for other people to do similar things for other areas, but once it's all done, that's a far more stable base. At least that's how I see it. YMMV and all that jazz. And to be fair, I started xoreos low-key in 2010, and while there has been progress, it's still far from done, nearly a decade later. It is what it is.

In C/C++-parlance, these are called "function declarations". They declare the name and parameters of a function, so that the function can be called. (Additionally, you have a comment explaining the function, which is not strictly necessary to call the function but helps the programmer.) Yes. The whole functionality for these functions is inside the engine, inside the EXE. It can't be changed readily at all. Wrong. The code and documentation of xoreos show explictly that the functionality is within the engine. That's pretty much expected, yeah. These engine functions usually check the parameters to be within their valid ranges. The rest of the code expects them to be still within the valid ranges, too. If you suddenly violate this expectancy, a crash is the most likely thing to happen. What you are doing is most probably invoking what's called "Undefined Behaviour", and the program might actually do everything it wants to in such a case. Common joking examples include eating your grandmother or letting demons fly out of your nose. On a more serious note, the game might also not crash immediately, and just leave your game in a broken state (even carrying on in future save states), so that you might, for example, always get a crash in a later section of the game, or just not progress through the game at all. Because you're changing the game's memory to values the original programmers never thought about. Essentially, what you'd need to do is to check every single location in the game that uses the value you changed, and verify that it still does the originally intended actions. And if you want it to do something additional or different, you need to make sure that it still makes logical sense in combination with the rest of the code. In addition to adding the new functionality, that is, which in itself is a tall order. Frankly, I for one would disadvise you from going down that road, but it's your life.

Just to re-iterate: from what I've seen while taking the games apart, the menus are pretty hard-coded. The games in general know what widgets are there for each menu, how they hang together, how they behave and what they do, and they only use the GUI files to read the look. I.e. there is no way to get fully working menus from one game into the other, just by exchanging data files. You will need to hook into the binary itself

IANAL, but from a legal standpoint, you won't be able to distribute these converted data files. What you can distribute (probably, I guess, but again, IANAL) is a converter that everyone will have to run for themselves, and everybody who does that will need to have legal, bought copies of both KotOR Android/iOS and KotOR2 PC. This is actually not *that* difficult, at least the Android part. Primarily, what we need is an OpenGL ES renderer. Currently, the renderer subsystem is being rewritten, so adding an OpenGL ES render path would probably clash, but after the rewrite is finished, that would be a very valid thing to do (in fact, that is already on my TODO list). Apart from that, all our libraries, including SDL2, would already work on Android. You can look at ScummVM for a proof of concept of sorts. There's certain things we'll have to keep in mind, like input, but it's not magic. iOS, on the other hand... the problem there is licensing, and the walled garden that is iOS. There's no chance xoreos will be allowed on the official Apple App Store. Not with Apple being Apple, and also because xoreos is GPLv3, a license that's fundamentally at odds with the closed-off nature of an App Store. The GPLv3 grants its users to modify the program, and run the modified program, and you can't do that with App Store things. I don't think there's really any sideloading on iOS, right?

And just as a note, the task to extend the xoreos NWScript disassembler into a decompiler is still open :P. More information on that can be found here: https://github.com/xoreos/xoreos-tools/issues/16 Likewise, the task for an assembler, and then compiler, is also still open, probably. The person who started working on it has been silent for a few months now: https://github.com/xoreos/xoreos-tools/issues/17

Frankly, I'd advise against using assets from the games as logo or promotional material. Likewise, I'd advise against having game assets in the project repository. That's just a legal minefield.

Huh, it didn't come across as harsh, though? It's all good, I'm not slighted in the slightest (), as far as I'm concerned we're good. We're all good . Heck, I don't want any beef or rivalries with anybody, I want us all to help each other along. That's one of the reasons I'm in FLOSS to begin with. And yes, this KotOR JS project is obviously further along, and that's absolutely fine, and great work. I mean, this (and this, a bit later) is the current state of KotOR in xoreos, more or less, still stuck in the first room. There's only a handful of script functions implemented. The intro cinematic doesn't work play, and you can't get Trask to open the door. Similarily, there's a person implementing their own NWN2 engine over on the Neverwinter Vault. The renderer is way further along than xoreos, with proper shaders and everything. I'd of course be happy to have more contributors to xoreos, the xoreos-tools and Phaethon, but I can also understand that people want to do their own thing. Sorry for taking up this much space here, I just wanted to be absolutely clear on that.

Damn, this is impressive. Good work! 😀 And yes, it is further along than xoreos, from a raw playability standpoint. By what you're skipping, It's mostly "just" (without trying to downplay your work) script functions, though, yes? From the underlying code framework, systems and concepts that are missing in the code, I assume it's probably quite similar to xoreos, correct? How does the start of KotOR look for you? That's where we're "stuck" at the moment, in the first room with Trask, because there is a lot of party management stuff necessary there. I'm certainly curious and interested in your code, how you solved certain things, and what I could pilfer for xoreos ;). So please, do get the code ready. The main issue for xoreos, though, is that we need people to work on it, if you allow me my recruitment spiel here. You can reach me here, by mail, on Freenode IRC, wherever :P (I also think that you're selling yourself way short if you think you couldn't meaningfully contribute to xoreos. For the most part, C++ isn't some wildly different animal from JS. It's not, say, Haskell.) xoreos, too, is of course also just standing on the shoulders of giants, building on work by others. I too owe a lot to those, and also to the people here. <3 In either case, again nice work, and good luck :)! Feel free to contact me if you have any questions about xoreos' code, or would like to talk shop or something :P

Yeah, ncsdis in xoreos-tools is a disassembler, not a decompiler. You won't get anything like the original script source out of it, but a more direct representation of the underlying bytecode. With some prior knowledge of assembly and the internals of NWScript, it is readable and understandable. But it's hardly comfortable, yes. There is an open task, up for grabs, to extend the disassembler into a full-fledged decompiler: https://github.com/xoreos/xoreos-tools/issues/16. If anybody reading here feels up for it, please work on this. I'd just request that you comment on that GitHub tracker issue, so that this can be coordinated properly. Likewise, there's an open task to create an NWScript compiler (which will then be able to handle all the games under one single "roof", so to speak): https://github.com/xoreos/xoreos-tools/issues/17. I made some hubbub to advertise these issues yesterday, and there's already two people who expressed interest in working on the compiler. If we're lucky, this might be there in the not so distant future. I know, I know, there do exists versions of the OpenKnights nwnnsscomp that work with KotOR/KotOR2, but I still think this might be of interest to some people here. (Also, this might be, AFAIK, the first NWScript compiler that can handle Dragon Age: Origins and Dragon Age II scripts, what with the 4 new opcodes (two for arrays and two for references). Me, I'm pretty excited about that. YMMV, though. :P) In either case, anybody wanting to help there, please chime in on these GitHub issues.

Yes and no. The naming is a bit inconsistent, unfortunately. Both KotOR Xbox .txb files and KotOR Windows PC .tpc files are fundamentally the same format, only that the Xbox one supports different encodings (and also swizzles the textures [1]). xoreos-tools can read both of them (both via the --tpc switch of xoreostex2tga, because for xoreos, both of them are "TPC"), converting them into TGA (and unswizzles where necessary). xoreos-tools can't write TPC at the moment, unfortunately. There's also another TXB format, used by Jade Empire (both the Xbox and the Windows PC version). That is a different format from KotOR Xbox TXB. To have somewhat consistent naming, xoreos always calls the KotOR format, whether Xbox or Windows PC, "TPC", and the Jade Empire format, whether Xbox or Windows PC, "TXB". (And if you're being pedantic, all of these formats, and also the non-standard .dds files in Neverwinter Nights, are still similar in concept. They're clearly different formats, but also very clearly variations on the same theme. Why they felt they had to reinvent the wheel three times, I don't know.) [1] In a swizzled texture, the pixels aren't laid out row-by-row, line-by-line, but in a zig-zag pattern. This is so that pixels that are near each other, i.e. also pixels that are below and above each other, are near each other in memory as well. This can help rendering performance. A more in-depth explanation is here: https://fgiesen.wordpress.com/2011/01/17/texture-tiling-and-swizzling/

Frankly, I for one would rather have the information free for anybody to access and use.

No. This is, unfortunately, very hard to trace and verify just by reading the disassembly. You basically need to find every place the structure is accessed and check if it reads from the relevant offset. If they stuff the pointer into other places too, it's even harder. If you can run the game in a debugger, either in IDA's, winedbg, gdb or what-have-you, you might be break on model load and watch memory access, I guess. But that's rather fiddly in general. You also can't say 100% for sure that the data is never accessed, only that it's not accessed in the render paths on your system, for the model types you looked at. They might be used on some special models or on different hardware (though the render paths for old, older hardware are probably not relevant anymore anyway). Ah, so no one has actually figured out how those work and you're just faking them? Interesting to know. IIRC, Farmboy0 looked at those some time back, tried to figure that out just by staring at a few files in 010 Editor. I don't think he managed to crack them, though, unfortunately. The only thing I could contribute was "I have no damn clue". Since xoreos doesn't yet support anything more complex than the simple whole-mesh animations NWN does for the most part, I didn't really need those data points yet. But this is something that'll be vital in the future.

I wouldn't really say anything is planned in that direction. The main goal of xoreos is, of course, making the games themselves playable. There is, however, Phaethon, a graphical resource explorer I started and worked on briefly as a side-thing while I was thinking through how to implement something else. Right now, it can look into the game archives, play sounds and display images/textures. It could, conceivably, in the future, also house model viewers and format editors. Maybe not editing images or even models, but editing GFFs and 2DAs would probably work quite well. That's kinda how I envision a tool like this to work. However, I have currently stopped working on it. If anybody is interested in taking up that task, please contact me. Also, for GUI stuff, it uses wxWidgets, which...well, is a bit annoying, IMHO. It would probably be best if it could be rewritten using, I guess, QT. Or if anyone has another idea, one that still works in a portable manner (i.e. not something like MFC), talk to me. And yes, I'm aware that I'd be replicating a lot of things already out there modding-wise. But this would be portable and FLOSS'd. I see worth there, but, of course, YMMV. Also, yes, there is a bit of a Not-Invented-Here-feeling involved there, I admit that. Thirdly, yes, this is all just talking the talk without walking the walk for now, "vapourware" if you want to be blunt; I'm also very aware of that as well.

Just FYI, xoreos main dev here. Hej hej. Yes, I do keep an eye on the forums. Especially when "xoreos" is mentioned, I'll probably see it eventually. I'd appreciate an explict poking when you're talking about xoreos, though, so that I can jump in and answer questions, clear up misunderstandings, etc. There is. I've looked at the disassembled original EXEs of both KotOR and KotOR2 using IDA, and they both have a parser for ASCII models, filling out structures resembling the binary model format, just like NWN does. The meshes themselves are more complicated (because that gets parsed into IBO/VBO structures), but the top-most model header and the node headers get parsed into a struct that's 1:1 what's in the binary model. That's, for example, how I found the transparencyhint field. Which, unfortunately, like in NWN, isn't always set when it should be, but that's BioWare for you. I haven't really mapped out all of it yet, though. And yes, the controllers stuff is also more difficult to follow in the disassembly, unfortunately. Of course, since I'm going to need it in the future, I'd be very interested in more details and informations about the format. So far, I've only done the bare minimum of what's necessary to get xoreos to display the basics. Also, IIRC, Farmboy0 has started crawling through the binary format using 010 Editor (I don't know how far, though). That sounds like a quite useful tool in general. In combination with following the ASCII model loader in the disassembly, that's probably the best way to map out the format completely, maybe.

Fair enough. No, it's separate from the kernel. It's a normal userspace library, just like GTK. It doesn't use GTK, though, it's an alternative to GTK (or maybe the other way round: Tk predates GTK). But both were meant, in a way, as a free and easier-to-use alternative to Motif (with different other agendas). For a bit of a look into history, Tk has been working on Linux since 1992, before it even worked on Windows (which came after the Windows 95 release). And yes, it does still work.

...That's kinda rich, if you consider Tk's history.

Hey there, xoreos main dev/lead chiming in. The name is not exactly Greek, no. There's an entry in our FAQ that answers the backstory for the name. This is exactly what it's supposed to be, yes. And yes, the xoreos-tools are partially to help the development of xoreos and partially to provide something potentially useful for modders. However, I'm not necessarily striving to replace KotORTool (or any other tool [1]), but add to it and increase the options people have. I'm not going for the hostility/rivalry route here. I'm also approaching things from a more stricter copyleft (GPLv3) and free software perspective, because that's important to me personally. Other modding tools developers might not care as much, or prefer to go either closed or more permissive. I'm also a proponent of staying as wildly portable as possible. As for the model formats, there's currently no xoreos tool to do anything with them (except extracting the binary MDL files from the archives). If you can read C++, the model loaders for xoreos are in the model_* files in src/graphics/aurora/. Not everything is supported yet, though. And lastly, my standard recruiting call: I'm still looking for more contributors to xoreos (and xoreos-tools). There's a long TODO list on our wiki, but it is, of course, non-exhaustive. If you feel up to tackling any of these or have your own suggestions, please contact me and let me know.

There's nothing in the bytecode that says "this here is an included script". The include basically just means that the compiler copies the contents of the include into your script. The effect would be the same if you manually copy-pasted the contents of the include yourself. You couldn't tell one method apart from the other in the bytecode. So if it fails for certain includes, that would just mean that these includes contain recursive functions. And yes, this is what I have found: some generic utility includes contain functions that use recursion to solve their task. This means that the bytecode of every script that uses this utility include contains recursive functions. Because the #include just means "copy what's in this file here wholesale". That's still entirely consistent with my guess that this is the same issue between DeNCS and my ncsdis. EDIT: See also this Wikipedia article on the include directive: https://en.wikipedia.org/wiki/Include_directive, especially the part about C/C++ (because NWScript takes a lot of cues from C).

Okay, experimenting with your tools, Fair Strides, I fail to get them to break the file. You basically read and write the strings verbatim, without converting. So you write CP-1252 to the txt file, and read it back as CP-1252, all the while strictly assuming that one character is one byte (as would be correct for CP-1252). So if the text is CP-1252, everything is okay. If someone modifies it and writes, say, UTF-8 into it, the text for that entry is wrong in the game, but further entries are not broken. While that's not the way I chose to go about it [*], I see nothing wrong. I can't see a way how that would break the file in a way cornel experienced, so I guess I am barking up the wrong tree with the encoding thing. [*] I am explictly converting read TLK string to UTF-8, and always write UTF-8 to the XML file. I always read UTF-8 from the XML file and explictly write in an encoding the game wants. This is way more work and headaches, but something I needed anyway for xoreos. I can also break in other interesting ways.

Can the compare plugin spit out a file of only the differences? Can you post that (if it's not too long, I guess)? Best not even copy-pasted, but straight-up what that spits out, so we know it's not getting mangled. Do you see any "weird" character there?

The place in the TLK file where the game looks for the text. No. But the tool that writes the new TLK needs to count how many characters there are. And it's possible that Fair Strides' tool might count wrong. That would be wrong, then. The length field in the file needs to be how many *bytes* the string is long. And the offset is, of course, also byte-based. So it's vital to count how many actual bytes you write to the file, and calculate the running offsets with that. Still, the strings should be CP-1252 encoded for US and western European versions of the game, no? So it should map 1:1 bytes to characters. How are you reading the string, i.e. in which encoding and how it is stored in the variables, and how do you write them? I'm afraid I have no clue at all how Perl (that is Perl, right?) handles strings, and different encodings. Of course, I might be barking up the completey wrong tree here as well.

Just throwing my two cents in the ring, while knowing absolutely nothing about how Fair Strides' tool works: That looks like the offset to the strings is off by one starting with an entry somewhere in the middle of the file? So I guess the string length counting messed up somewhere? Did you maybe modify a string to contain a non-ASCII character, like an umlaut or a smart-quote, and it chockes there? At least that's what I'd guess, that it counts something by characters instead of by bytes, leading to it writing two bytes but only counting one character.