DrMcCoy

Members
  • Content Count

    71
  • Joined

  • Days Won

    3

DrMcCoy last won the day on April 8 2018

DrMcCoy had the most liked content!

Community Reputation

29 Jedi Padawan

1 Follower

About DrMcCoy

  • Rank
    Jedi Padawan
  • Birthday 08/05/1985

Contact Methods

  • Website URL
    https://drmccoy.de/

Profile Information

  • Gender
    Male
  • Location
    Braunschweig, Germany
  • Interests
    Geek. Atheist+. Leftist. Metal-Head. Discordian. Lefty.
    ScummVM dev, xoreos lead.
    Software freedom zealot.

Recent Profile Visitors

3,138 profile views
  1. Yeah, xoreos-tools lacks support for animated textures, for example. It also doesn't export the TXI information. And it can't convert textures back to TPC. That's all in the TODO still. We do take patches and pull requests, though
  2. I'm not sure about the race condition theory, considering that the game isn't even, AFAIK, multi-threaded. I don't see how the checks could happen out of order. And how that would then persist for the next checks. Likewise, a memory leak also sounds implausible. Not that I have any better idea, mind. I'm pretty sure there's even a lot of legitimate entries that have delay 0 that should be skipped: ones with only developer commentary. TSL has commentary in basically all of its dialogue files, mostly talking about how the camera is moved, the mental state of a character, etc. I think that's meant for the animators and voice actors? The commentary is enclosed in curly brackets and the game filters those out. Likewise, there's entries that exist solely for camera movement. Can't say anything about your proposed changes, though. However, 0xFFFFFFFF is often used as a sentinel value of sorts, so it might be there's some other functional difference between delay 0 and delay 0xFFFFFFFF in the game. In either case, the solution should of course be properly tested. Maybe even cross-checked with a disassembly of the game, or even attaching a debugger like OllyDbg while the game is running (and/or even already in the buggy state). That is, however, time-consuming and complex.
  3. You might want to see if the round-trip holds, at least using nwnnsscomp and existing source files. Like, source file -> nwnnsscomp (ncs #1) -> decompiler -> nwnnsscomp (ncs #2), then compare whether ncs #1 and ncs #2 identical. There shouldn't be any differences in the bytecode, probably. Possibly also run it through the decompiler again to check that the output is stable. You can automate that steps and just focus on the examples where it fails. That's possible also good as a sort of test suite between big changes or releases. There never was a public release of a KotOR-era BioWare compiler, though, right? Has anybody ever tried rigging the compiler in the Neverwinter Nights toolset (does it read the nwscript.nss or is it more hardcoded)? If it can be rigged, that would give you another angle to test on, though it's not necessarily guaranteed any of the NWN releases matches the version (versions?) that was used during KotOR development.
  4. For correctness' sake, the OBB file, at least in the Android version (I haven't looked at the iOS version at all so far), isn't exactly encrypted. In fact, the files themselves aren't even compressed, but the resource list is zlib-compressed. I've looked at the files a few days ago, planning to extent the unobb tool in xoreos-tools to make it read them as well: https://github.com/xoreos/xoreos-tools/issues/69 TL;DR: The KotOR2 Android OBBs, just like the Jade Empire Android OBBs, are a "virtual filesystem" (their terminology, they called their obb reading library libobbvfs.so). Essentially, the Jade Empire files were stored in 4096 byte blocks, zlib-compressed, with a zlib-compressed resource list at the end of the file. The KotOR2 ones are similar in principle, just that the files themselves are uncompressed for some reason. The resource list is still zlib-compressed though and has the exact same layout. There are some differences in how I have to go about locating the resource list, though, and I'm currently a bit unsure how to go about it. Potentially, the solution is in finding out how one thing I so far ignored in the file works, a list of blocks. If that exists in the KotOR2 files as well, that is, I haven't checked. If everything fails, I'll just add a simple auto-detection heuristics with a command line flag to override that, but that feels a bit dirty. I'll play around it a bit more over the coming days. If anybody else has already looked at the OBB files, feels free to ping me, maybe we can collaborate a bit there. If nobody even cares about reading those, feel free to ignore me EDIT: See https://github.com/xoreos/xoreos-tools/blob/master/src/aurora/obbfile.h and https://github.com/xoreos/xoreos-tools/blob/master/src/aurora/obbfile.cpp for more details on the current code that can read the Jade Empire files. Still unextended with my KotOR2 stuff, because that's still WIP and hacky
  5. Yeah, nwnnsscomp is not optimizing. But my point is that lots of roads lead to Rome, and denoting one as more correct than others is the wrong way to think about this. That different compilers written by different people lead to different bytecode that still accurately perform the same visible actions is normal, that should be dealt with. Similar for dead code. Dead code is entirely fine and should be handled properly without throwing errors there. And I feel that saying "nwnnsscomp is wrong" does it a huge disservice, because it's technically not wrong at all. So I'm more hung up about your phrasing here. As long as the decompiler still works as such (and doesn't just outright refuse to work) for cases were different idiosyncrasies slip through, be it nwnnsscomp or maybe the BioWare compiler used for Dragon Age, everything is entirely fine with me (for what that's worth, which is maybe not a lot). Guaranteeing a 100% match with the original source in all cases is pretty much impossible anyway, and not something I expected in the first place.
  6. Frankly, saying this is "incorrect" compilation is a misunderstanding of the compilation process. If you look at modern C compilers, they're doing far more transformations. That's all legal and often even wanted (to keep pipelines from stalling, etc). Don't stoop down to clickbait. And in general, for many targets, a nested if like that is indistinguishable from an unnested if, else if, else in the final machine code anyway. This one reason compilation-decompilation is never a lossless process, different things in the source can compile down to the same machine code. What you have found is a "tell" in the original BioWare compiler, that lets you distinguish certain things. That nwnnsscomp doesn't have the same tell is not a fault of nwnnsscomp, it's the nature of the thing. Ask me about gcc and clang differences some day, or different styles of mangling C++ symbols in gcc and msvc.
  7. It's probably easier and safer just query the user on what game this script is for (via a command line option, for example), than to dependant on some brittle heuristic to detect that. If you lack files to test this on, you could get Neverwinter Nights and install its toolset, because that came with an official BioWare NWScript compiler. Yes, it's for NWN, but the control structure stuff should be identical. Dunno if I already wrote that here or just in IRC, on a GitHub issue or wherever, but: the interesting thing about the DESTRUCT opcode is that it's used to single out individual struct members. I.e. whenever the nss used a struct member, the ncs copies the whole struct to the top of the stack and used DESTRUCT on this new block in the stack to get rid of everything but the single struct member it's interested in. Which means you can use the existance of DESTRUCT to identify structs.
  8. Yeah, the way the original game create the UI is a bit...meh. It's mostly hardcoded and the GUI files are only read to apply some styling afterwards. Though I can't really tell you outright which things are changeable and which aren't.
  9. Interesting that you're encountering the same short-circuiting bug I've seen in NWN (I wrote about it here, though everyone reading is probably aware: https://xoreos.org/blog/2016/01/12/disassembling-nwscript-bytecode/). Up until now, I had assumed they fixed that bug by the time they moved to other games. I know that the bug is in some scripts in NWN, but not in all. And I know that the compiler in the last released toolset does not have that bug anymore. I'd be interested to know when exactly it was fixed...
  10. Technically, this is not quite correct. xoreos-tools is GPLv3+, with the additional "or later" term. I.e. "licensed under the terms of the GNU General Public License version 3 or (at your option) any later version", as it's stated in the README.md and the comment-header of all source files. The reason this exists is so that code can be upgraded from one GPL to the next version without having to ask every contributor for approval. You don't specify that in ncs2nss, so with just the GPLv3 license text, this code is by default GPLv3 exclusive. There's no separate full license text for the variant, it's part of §14, you only have to state this intend (preferably with that exact phrasing) where you say that the code is GPLv3. (Best practice is also that you add a small blurb to the top of all source files stating the license, just to make sure that it's absolutely clear. You can see an approach in all xoreos project source files, for example. Then again, it's also best practice to explictly state the copyright holder in all files, but I've outsourced this into the AUTHORS file and only reference that file in the comment header.)
  11. xoreos and xoreos-tools are C++11 now, Phaethon is C++14. Though there are still vestiges of old stuff in there, like our own ScopedPtr which isn't necessary anymore now with std::unique_ptr. Gradually replacing that is on the TODO. As for newer standards, these are also gradually on my purview. Moving over xoreos and xoreos-tools to C++14 shouldn't be that much trouble. C++17 needs some more evaluations. Probably still a bit off, just to make sure. And I need to check if our CI systems (Travis CI and AppVeyor) and my personal cross-compilers can be upgraded. And probably also need to wait on Coverity Scan, that has a few issues with some C++14 stuff in Phaethon already, from verdigris we pull in (which lets you write Qt code without moc). Oh, and about the library request, I've written down my thoughts about it here: https://github.com/xoreos/xoreos-tools/issues/58#issuecomment-662419079 TL;DR: That's planned, but I don't think the time for that is now.
  12. There's also these discussions on the xoreos-tools issue tracker: https://github.com/xoreos/xoreos-tools/issues/56 , https://github.com/xoreos/xoreos-tools/issues/57 , that might give you some pointers and ideas. Also shows a case where the current control flow analysis in xoreos-tools' ncsdis fails lachjames also wrote me a few thoughts and ideas on Discord, but since this is non-public 1:1 talk, I can't really paste this without permission. (Also a reason I generally prefer public conversations, to be frank. Information wants to be free and all that jazz.)
  13. See also https://github.com/xoreos/xoreos-tools/blob/master/src/nwscript/game_kotor.h vs https://github.com/xoreos/xoreos-tools/blob/master/src/nwscript/game_kotor2.h . Btw, @lachjamesis also currently working on a dencs. You might want to talk to each other to see if you can maybe combine your efforts. As for me, just to lay my cards on the table, I'm also quite interested in this. With some caveats: I'd want to integrate it into the xoreos-tools package, written in C++ and targetting not just KotOR/KotOR2, but all the Aurora-derivate games that xoreos supports and of course based on the NCS disassembly code that already exists in xoreos-tools (which includes a few more opcodes for the two Dragon Age games, since those can do references and arrays). So what I'd like to do, once there is a working dencs that supports KotOR/KotOR2, is take that and port it to C++ and add it to the xoreos-tools. Provided there's sources, licenced compatibly with the GPLv3, and of course giving proper credits in the source file and AUTHORS text file. Probably not directly 1:1 either, but more seeing how you do things and redo them within the current framework in xoreos-tools.
  14. To explain what this warning means: ncsdis tries to run through the whole disassembly to analyze the stack. Essentially, how the stack looks after each operation: how many stack slots are allocated and what types (int, string, float, ...) are in there. When there is a fork, for example due to branching in an if-clause, ncsdis follows both paths. And when the paths merge again, for example after the if when the excution continues on linearly, the stack has to look the same for both paths. Otherwise, I wouldn't know what to make out of this situation. (ncsdis only checks that the size of the stack matches, and ignores the types at the moments.) In this case, for some reason ncsdis found that the stack sizes of two paths don't match. It's 2 slots for one path and 5 slots for another path. This might be a bug in ncsdis. This could also be a bug in the compiler that produced the NCS in question. For example, IIRC I found the same message for one of the bugs I wrote about in my blog post, the one where the short-circuiting boolean OR. In that case, it didn't really matter that the stack is broken for one path, because that path was logically dead anyway, it would never have been called. So to work around this bug in the original compiler, I added a dead branch check, and the stack analysis won't follow dead branches. It might be that this here is a similar problem and my dead branch check is not catches it and needs to be expanded. Or this script is broken after all. (Or it could still be a bug in ncsdis.) I'd need to check that, I'll add it to my TODO list. Or if anybody else would like to tackle this, feel free
  15. I don't remember seeing this exacty case. Then again, it's been a while since I touched that code and I'm not known to have a good memory. :P Also, my disassembler just follows the starting segment and IIRC I don't even check for unreachable blocks without an edge leading into them. That's maybe something that could use improvement. I take GitHub pull requests! ;) However, I can say that I have found a few bugs in BioWare's script compiler (I explained two in my replies above, I think), so I wouldn't particularily rule that out. And while I can't name any specifics right now, I also never got the feeling that their compiler was all that great in optimizing, fusing instructions, or removing outright unnecessary instructions. I don't think they focussed on that at all, rather keeping it simple and working without any surprises (bugs nonwithstanding).