• Content Count

  • Joined

  • Last visited

  • Days Won


Everything posted by DrMcCoy

  1. I don't remember seeing this exacty case. Then again, it's been a while since I touched that code and I'm not known to have a good memory. :P Also, my disassembler just follows the starting segment and IIRC I don't even check for unreachable blocks without an edge leading into them. That's maybe something that could use improvement. I take GitHub pull requests! ;) However, I can say that I have found a few bugs in BioWare's script compiler (I explained two in my replies above, I think), so I wouldn't particularily rule that out. And while I can't name any specifics right now, I also never got the feeling that their compiler was all that great in optimizing, fusing instructions, or removing outright unnecessary instructions. I don't think they focussed on that at all, rather keeping it simple and working without any surprises (bugs nonwithstanding).
  2. As for operators, you should probably also make sure you know the precedence rules. I'm not sure NWScript follows the C rules there exactly. I haven't yet looked at that, since it's unambiguous in the bytecode. Oh, and also, there's another bug I've found, but I'm not sure in which games it was present. It had to do with parameter shadowing. I.e. void foobar(int blah) { int blah = barfoo(blah); } (excuse the quirky formatting, this editor is weirder than I remember) I've seen the produced bytecode use the uninitialized value of the just created local variable blah as in input for barfoo(), instead of the parameter of foobar(). Maybe something to have an eye on. Hmm, depends on how DeNCS operates. Does it also try to decompile functions that are not called? I.e. if you have an include with foobar1(), foobar2() and foobar3(), and the script itself just calls foobar1() (and that doesn't call the other functions), does DeNCS also decompile foobar2() and foobar3()? Because IIRC ncsdis completely ignores foobar2() and foobar3() in that case and wouldn't see any recursion there.
  3. Yeah, do have a look at the nwscript directory: https://github.com/xoreos/xoreos-tools/tree/master/src/nwscript Specifically, the instruction.h: https://github.com/xoreos/xoreos-tools/blob/master/src/nwscript/instruction.h , which contains a list of opcodes (including 2 each that were introduced by Dragon Age: Origins and Dragon Age II). You might also want to look at https://github.com/xoreos/xoreos-tools/blob/master/src/nwscript/game_kotor.h and https://github.com/xoreos/xoreos-tools/blob/master/src/nwscript/game_kotor2.h for a list of engine functions and their signatures (return type + types of parameters). Alternatively, you can parse the nwscript.nss from the games yourself. If you haven't already, have a read over my blog post about disassembling the bytecode: https://xoreos.org/blog/2016/01/12/disassembling-nwscript-bytecode/ . It also shows a bug in BioWare's compiler (though that particular bug is not present in the KotOR scripts, it was fixed before KotOR development started it seems). If you want to decompile all scripts, you'll find that you're going to have problems with recursion, i.e. functions that call themselves. You need to analyze how a function leaves the stack to be able to continue with the code after the call, so you need to branch into a callee first before continuing with the caller. If the callee is the caller itself (or A calls B calls C calls A again)... that's a problem. This is essentially the halting problem, and there's no general solution, unfortunately.
  4. (Btw, please feel free to tag me in when talking about xoreos or something related. Google takes a while until it picks up new posts :P) In either case, I'd be happy to have any help willing to work on xoreos and its sub-projects (xoreos-tools and Phaethon). It'd very much like xoreos-tools and Phaethon to be a collection of tools helpful for modders as well. My general idea was that xoreos-tools was to be a CLI collection and Phaethon a tool incorporating complimentary GUI functionality. Both, I feel, are a good base to add more things that can edit and create, as opposed to just read, the files for the targetted BioWare games. I'd also be willing to have similar sub-projects under the same xoreos banner as well [1][2][3]. Basically, what I'm saying: I for one think xoreos would be the ideal grounds for modernized KotOR modding tools [4]. We're not quite dependency-free, because there are libraries we depend on. Especially Boost is somewhat of a beast, I'm aware. In the future, I do want to phase it out as stuff gets approved into new the C++ standards and those become mature to be used in userland code. [1] Though it might maybe be a good idea to migrate some of the code into a library first. Right now, xoreos-tools and Phaethon contains (mostly) unmodified copies of the xoreos files and that's a bit eeeeeeh [2] I do want them to be portable, though and of course license-compatible with the GPLv3'd xoreos code. I can understand if people disagree with me on these points :P [3] And yes, I am sometimes a stickler for form and following a consistent coding style. Again, I can understand if people feel I'm too strict there and wouldn't want to me near any of their code [4] And also the other targetted BioWare games, to be honest
  5. Huh, interesting. That might be a bug in xoreos-tools' erf packer? I might be able to look a bit at that during the holidays. Unless you want to tackle debugging this for me?
  6. Wait, why did I have to (re?)upload my profile photo? Or did I just completely forget to do that when I joined ages ago? Am I getting old? O_o

    1. DarthParametric


      I think some older images got nuked during the site update a little while back. A lot of attachments got the Thanos treatment.

    2. DrMcCoy


      Ah, okay, thanks

  7. I didn't say that. No one is currently actively working on an Android port. But I have that on my radar and I do have ideas and plans. So I'm afraid I can't give you an estimate. Heck, I can't even give you an estimate on when KotOR is playable in xoreos, or when xoreos is "user-ready", or anything, really. I can only say that more contributors = faster work, and we really need more contributors. I know, I probably sound like a broken records, but it is what it is.
  8. That...depends. In C, you can't. In C++, you can overload a function, as long as the signatures, i.e. the number of parameters and their types, are different (internally, C++ "mangles" the identifiers, using a scheme that encodes the parameter types into the name). In NWScript, you can't either. The compiler wouldn't know what to do. When you write a script in NWScript, you use the compiler (for example, a modifed nwnnsscomp for Kotor, based on a open source reimplementation by the "OpenKnights Consortium" of the original NWN compiler, based on the work of Torlack who RE'd the stuff back in the day) to turn the C-like NWScript source (nss) into NWScript bytecode (ncs). Internally, the bytecode doesn't know much about the engine functions anymore. The compiler looks into the nwscript.nss file to look for the name of the function, and then counts at the index of the function within the file. I.e. if it's the first function in nwscript.nss, the second, the third, etc. That number it puts into the bytecode file ncs. So in the bytecode, you only have pushes onto the stack, for the parameters, that the compiler arranged there (and space for the return value), and then "call engine function 523". The game, when running the script bytecode, takes that number, then knows what to do, and which parameters it takes. I.e. with NWScript, you can't change the number of parameters, and the types, because that knowledge is hardcoded into the game. (Well, it's not impossible, just quite hard.) What you could, theoretically, do is add a new function to the bottom, then manipulate the look-up function in the game to recognize it. You'd still have to add the functionality itself, though. It's still hard, just slightly less so. (This could also be used to emulate overloading, by extending the compiler to map the overloaded functions onto later indices. Might not be worth the hassle, though.) We, err, don't have that implemented yet. It's not really that it's forbidden. It's just that you're setting yourself up for a lot of hard work and pain. And potentially failure, of course. I know, coming from me, who's RE'ing games for fun, and who has too many projects and stuff to do for 10 lifetimes, that's a bit ironic. Still, I still think the "best" way to resolve hardcoded things in KotOR is going over xoreos: getting xoreos in a playable and "done" state, so that improvements can be build upon xoreos. Not that this isn't a massive amount of work either, far from it, just that this opens up so many more possibilities. If you're adding a single thing to KotOR using the manual binary-hackery method, it's way harder to be sure you didn't miss anything, you can't verify easily when you continue to work on it that you're not breaking it, it's harder to integrate with other people's work, etc. You could instead do something similar, figure out how things for in the game and reimplement that in xoreos, bringing xoreos further along. It will not give you the gratification of having changed something in the original game as quickly, and still need to wait for other people to do similar things for other areas, but once it's all done, that's a far more stable base. At least that's how I see it. YMMV and all that jazz. And to be fair, I started xoreos low-key in 2010, and while there has been progress, it's still far from done, nearly a decade later. It is what it is.
  9. In C/C++-parlance, these are called "function declarations". They declare the name and parameters of a function, so that the function can be called. (Additionally, you have a comment explaining the function, which is not strictly necessary to call the function but helps the programmer.) Yes. The whole functionality for these functions is inside the engine, inside the EXE. It can't be changed readily at all. Wrong. The code and documentation of xoreos show explictly that the functionality is within the engine. That's pretty much expected, yeah. These engine functions usually check the parameters to be within their valid ranges. The rest of the code expects them to be still within the valid ranges, too. If you suddenly violate this expectancy, a crash is the most likely thing to happen. What you are doing is most probably invoking what's called "Undefined Behaviour", and the program might actually do everything it wants to in such a case. Common joking examples include eating your grandmother or letting demons fly out of your nose. On a more serious note, the game might also not crash immediately, and just leave your game in a broken state (even carrying on in future save states), so that you might, for example, always get a crash in a later section of the game, or just not progress through the game at all. Because you're changing the game's memory to values the original programmers never thought about. Essentially, what you'd need to do is to check every single location in the game that uses the value you changed, and verify that it still does the originally intended actions. And if you want it to do something additional or different, you need to make sure that it still makes logical sense in combination with the rest of the code. In addition to adding the new functionality, that is, which in itself is a tall order. Frankly, I for one would disadvise you from going down that road, but it's your life.
  10. Just to re-iterate: from what I've seen while taking the games apart, the menus are pretty hard-coded. The games in general know what widgets are there for each menu, how they hang together, how they behave and what they do, and they only use the GUI files to read the look. I.e. there is no way to get fully working menus from one game into the other, just by exchanging data files. You will need to hook into the binary itself
  11. IANAL, but from a legal standpoint, you won't be able to distribute these converted data files. What you can distribute (probably, I guess, but again, IANAL) is a converter that everyone will have to run for themselves, and everybody who does that will need to have legal, bought copies of both KotOR Android/iOS and KotOR2 PC. This is actually not *that* difficult, at least the Android part. Primarily, what we need is an OpenGL ES renderer. Currently, the renderer subsystem is being rewritten, so adding an OpenGL ES render path would probably clash, but after the rewrite is finished, that would be a very valid thing to do (in fact, that is already on my TODO list). Apart from that, all our libraries, including SDL2, would already work on Android. You can look at ScummVM for a proof of concept of sorts. There's certain things we'll have to keep in mind, like input, but it's not magic. iOS, on the other hand... the problem there is licensing, and the walled garden that is iOS. There's no chance xoreos will be allowed on the official Apple App Store. Not with Apple being Apple, and also because xoreos is GPLv3, a license that's fundamentally at odds with the closed-off nature of an App Store. The GPLv3 grants its users to modify the program, and run the modified program, and you can't do that with App Store things. I don't think there's really any sideloading on iOS, right?
  12. And just as a note, the task to extend the xoreos NWScript disassembler into a decompiler is still open :P. More information on that can be found here: https://github.com/xoreos/xoreos-tools/issues/16 Likewise, the task for an assembler, and then compiler, is also still open, probably. The person who started working on it has been silent for a few months now: https://github.com/xoreos/xoreos-tools/issues/17
  13. Frankly, I'd advise against using assets from the games as logo or promotional material. Likewise, I'd advise against having game assets in the project repository. That's just a legal minefield.
  14. Huh, it didn't come across as harsh, though? It's all good, I'm not slighted in the slightest (), as far as I'm concerned we're good. We're all good . Heck, I don't want any beef or rivalries with anybody, I want us all to help each other along. That's one of the reasons I'm in FLOSS to begin with. And yes, this KotOR JS project is obviously further along, and that's absolutely fine, and great work. I mean, this (and this, a bit later) is the current state of KotOR in xoreos, more or less, still stuck in the first room. There's only a handful of script functions implemented. The intro cinematic doesn't work play, and you can't get Trask to open the door. Similarily, there's a person implementing their own NWN2 engine over on the Neverwinter Vault. The renderer is way further along than xoreos, with proper shaders and everything. I'd of course be happy to have more contributors to xoreos, the xoreos-tools and Phaethon, but I can also understand that people want to do their own thing. Sorry for taking up this much space here, I just wanted to be absolutely clear on that.
  15. Damn, this is impressive. Good work! 😀 And yes, it is further along than xoreos, from a raw playability standpoint. By what you're skipping, It's mostly "just" (without trying to downplay your work) script functions, though, yes? From the underlying code framework, systems and concepts that are missing in the code, I assume it's probably quite similar to xoreos, correct? How does the start of KotOR look for you? That's where we're "stuck" at the moment, in the first room with Trask, because there is a lot of party management stuff necessary there. I'm certainly curious and interested in your code, how you solved certain things, and what I could pilfer for xoreos ;). So please, do get the code ready. The main issue for xoreos, though, is that we need people to work on it, if you allow me my recruitment spiel here. You can reach me here, by mail, on Freenode IRC, wherever :P (I also think that you're selling yourself way short if you think you couldn't meaningfully contribute to xoreos. For the most part, C++ isn't some wildly different animal from JS. It's not, say, Haskell.) xoreos, too, is of course also just standing on the shoulders of giants, building on work by others. I too owe a lot to those, and also to the people here. <3 In either case, again nice work, and good luck :)! Feel free to contact me if you have any questions about xoreos' code, or would like to talk shop or something :P
  16. Yeah, ncsdis in xoreos-tools is a disassembler, not a decompiler. You won't get anything like the original script source out of it, but a more direct representation of the underlying bytecode. With some prior knowledge of assembly and the internals of NWScript, it is readable and understandable. But it's hardly comfortable, yes. There is an open task, up for grabs, to extend the disassembler into a full-fledged decompiler: https://github.com/xoreos/xoreos-tools/issues/16. If anybody reading here feels up for it, please work on this. I'd just request that you comment on that GitHub tracker issue, so that this can be coordinated properly. Likewise, there's an open task to create an NWScript compiler (which will then be able to handle all the games under one single "roof", so to speak): https://github.com/xoreos/xoreos-tools/issues/17. I made some hubbub to advertise these issues yesterday, and there's already two people who expressed interest in working on the compiler. If we're lucky, this might be there in the not so distant future. I know, I know, there do exists versions of the OpenKnights nwnnsscomp that work with KotOR/KotOR2, but I still think this might be of interest to some people here. (Also, this might be, AFAIK, the first NWScript compiler that can handle Dragon Age: Origins and Dragon Age II scripts, what with the 4 new opcodes (two for arrays and two for references). Me, I'm pretty excited about that. YMMV, though. :P) In either case, anybody wanting to help there, please chime in on these GitHub issues.
  17. Yes and no. The naming is a bit inconsistent, unfortunately. Both KotOR Xbox .txb files and KotOR Windows PC .tpc files are fundamentally the same format, only that the Xbox one supports different encodings (and also swizzles the textures [1]). xoreos-tools can read both of them (both via the --tpc switch of xoreostex2tga, because for xoreos, both of them are "TPC"), converting them into TGA (and unswizzles where necessary). xoreos-tools can't write TPC at the moment, unfortunately. There's also another TXB format, used by Jade Empire (both the Xbox and the Windows PC version). That is a different format from KotOR Xbox TXB. To have somewhat consistent naming, xoreos always calls the KotOR format, whether Xbox or Windows PC, "TPC", and the Jade Empire format, whether Xbox or Windows PC, "TXB". (And if you're being pedantic, all of these formats, and also the non-standard .dds files in Neverwinter Nights, are still similar in concept. They're clearly different formats, but also very clearly variations on the same theme. Why they felt they had to reinvent the wheel three times, I don't know.) [1] In a swizzled texture, the pixels aren't laid out row-by-row, line-by-line, but in a zig-zag pattern. This is so that pixels that are near each other, i.e. also pixels that are below and above each other, are near each other in memory as well. This can help rendering performance. A more in-depth explanation is here: https://fgiesen.wordpress.com/2011/01/17/texture-tiling-and-swizzling/
  18. Frankly, I for one would rather have the information free for anybody to access and use.
  19. No. This is, unfortunately, very hard to trace and verify just by reading the disassembly. You basically need to find every place the structure is accessed and check if it reads from the relevant offset. If they stuff the pointer into other places too, it's even harder. If you can run the game in a debugger, either in IDA's, winedbg, gdb or what-have-you, you might be break on model load and watch memory access, I guess. But that's rather fiddly in general. You also can't say 100% for sure that the data is never accessed, only that it's not accessed in the render paths on your system, for the model types you looked at. They might be used on some special models or on different hardware (though the render paths for old, older hardware are probably not relevant anymore anyway). Ah, so no one has actually figured out how those work and you're just faking them? Interesting to know. IIRC, Farmboy0 looked at those some time back, tried to figure that out just by staring at a few files in 010 Editor. I don't think he managed to crack them, though, unfortunately. The only thing I could contribute was "I have no damn clue". Since xoreos doesn't yet support anything more complex than the simple whole-mesh animations NWN does for the most part, I didn't really need those data points yet. But this is something that'll be vital in the future.
  20. I wouldn't really say anything is planned in that direction. The main goal of xoreos is, of course, making the games themselves playable. There is, however, Phaethon, a graphical resource explorer I started and worked on briefly as a side-thing while I was thinking through how to implement something else. Right now, it can look into the game archives, play sounds and display images/textures. It could, conceivably, in the future, also house model viewers and format editors. Maybe not editing images or even models, but editing GFFs and 2DAs would probably work quite well. That's kinda how I envision a tool like this to work. However, I have currently stopped working on it. If anybody is interested in taking up that task, please contact me. Also, for GUI stuff, it uses wxWidgets, which...well, is a bit annoying, IMHO. It would probably be best if it could be rewritten using, I guess, QT. Or if anyone has another idea, one that still works in a portable manner (i.e. not something like MFC), talk to me. And yes, I'm aware that I'd be replicating a lot of things already out there modding-wise. But this would be portable and FLOSS'd. I see worth there, but, of course, YMMV. Also, yes, there is a bit of a Not-Invented-Here-feeling involved there, I admit that. Thirdly, yes, this is all just talking the talk without walking the walk for now, "vapourware" if you want to be blunt; I'm also very aware of that as well.
  21. Just FYI, xoreos main dev here. Hej hej. Yes, I do keep an eye on the forums. Especially when "xoreos" is mentioned, I'll probably see it eventually. I'd appreciate an explict poking when you're talking about xoreos, though, so that I can jump in and answer questions, clear up misunderstandings, etc. There is. I've looked at the disassembled original EXEs of both KotOR and KotOR2 using IDA, and they both have a parser for ASCII models, filling out structures resembling the binary model format, just like NWN does. The meshes themselves are more complicated (because that gets parsed into IBO/VBO structures), but the top-most model header and the node headers get parsed into a struct that's 1:1 what's in the binary model. That's, for example, how I found the transparencyhint field. Which, unfortunately, like in NWN, isn't always set when it should be, but that's BioWare for you. I haven't really mapped out all of it yet, though. And yes, the controllers stuff is also more difficult to follow in the disassembly, unfortunately. Of course, since I'm going to need it in the future, I'd be very interested in more details and informations about the format. So far, I've only done the bare minimum of what's necessary to get xoreos to display the basics. Also, IIRC, Farmboy0 has started crawling through the binary format using 010 Editor (I don't know how far, though). That sounds like a quite useful tool in general. In combination with following the ASCII model loader in the disassembly, that's probably the best way to map out the format completely, maybe.
  22. Fair enough. No, it's separate from the kernel. It's a normal userspace library, just like GTK. It doesn't use GTK, though, it's an alternative to GTK (or maybe the other way round: Tk predates GTK). But both were meant, in a way, as a free and easier-to-use alternative to Motif (with different other agendas). For a bit of a look into history, Tk has been working on Linux since 1992, before it even worked on Windows (which came after the Windows 95 release). And yes, it does still work.
  23. ...That's kinda rich, if you consider Tk's history.
  24. Hey there, xoreos main dev/lead chiming in. The name is not exactly Greek, no. There's an entry in our FAQ that answers the backstory for the name. This is exactly what it's supposed to be, yes. And yes, the xoreos-tools are partially to help the development of xoreos and partially to provide something potentially useful for modders. However, I'm not necessarily striving to replace KotORTool (or any other tool [1]), but add to it and increase the options people have. I'm not going for the hostility/rivalry route here. I'm also approaching things from a more stricter copyleft (GPLv3) and free software perspective, because that's important to me personally. Other modding tools developers might not care as much, or prefer to go either closed or more permissive. I'm also a proponent of staying as wildly portable as possible. As for the model formats, there's currently no xoreos tool to do anything with them (except extracting the binary MDL files from the archives). If you can read C++, the model loaders for xoreos are in the model_* files in src/graphics/aurora/. Not everything is supported yet, though. And lastly, my standard recruiting call: I'm still looking for more contributors to xoreos (and xoreos-tools). There's a long TODO list on our wiki, but it is, of course, non-exhaustive. If you feel up to tackling any of these or have your own suggestions, please contact me and let me know.
  25. There's nothing in the bytecode that says "this here is an included script". The include basically just means that the compiler copies the contents of the include into your script. The effect would be the same if you manually copy-pasted the contents of the include yourself. You couldn't tell one method apart from the other in the bytecode. So if it fails for certain includes, that would just mean that these includes contain recursive functions. And yes, this is what I have found: some generic utility includes contain functions that use recursion to solve their task. This means that the bytecode of every script that uses this utility include contains recursive functions. Because the #include just means "copy what's in this file here wholesale". That's still entirely consistent with my guess that this is the same issue between DeNCS and my ncsdis. EDIT: See also this Wikipedia article on the include directive: https://en.wikipedia.org/wiki/Include_directive, especially the part about C/C++ (because NWScript takes a lot of cues from C).