A New DeNCS

[AmanoJyaku 184]

Life's been hectic, it's been a while since I've worked on this. This is on hold indefinitely unless I get some free time.

Also, I'm waiting for the KoTOR remake. It would be silly to find out the remake either includes a reverse compiler or replaces NCS scripts with something else.

[DrMcCoy 40]

6 minutes ago, AmanoJyaku said:

Also, I'm waiting for the KoTOR remake. It would be silly to find out the remake [...] replaces NCS scripts with something else.

Why? Why should that have any say on whether an NCS decompiler is useful?

[DarthParametric 3,795]

The remake is in UE4. There won't be any NWScript in it. It doesn't even use UnrealScript any more, just Blueprints.

If you are giving up, then please at least make the current source code available. DS now has its own Github repo for various tool projects. If you'd like to create a repo for it there, jump on the DS Discord - https://discord.gg/h68Z9Ds2 - or send me a PM.

[AmanoJyaku 184]

15 hours ago, DrMcCoy said:

Why? Why should that have any say on whether an NCS decompiler is useful?

I'm not saying a reverse compiler isn't useful. But I'd like to know how the remake will change things. If the mod community moves from NWScript/NCS to something new in the remake, a reverse compiler for NCS won't be useful for much beyond historical reasons.

And we have no idea if TSL will get a remake, so the reverse compiler may be necessary regardless of the KoTOR remake.

So for the moment assume this is on hold rather than canceled.

 

12 hours ago, DarthParametric said:

The remake is in UE4. There won't be any NWScript in it. It doesn't even use UnrealScript any more, just Blueprints.

So far I've only seen rumors, no definitive proof beyond switching the game engine to UE. And I don't see why NWScript/NCS can't coexist with UE?

 

12 hours ago, DarthParametric said:

If you are giving up, then please at least make the current source code available. DS now has its own Github repo for various tool projects. If you'd like to create a repo for it there, jump on the DS Discord - https://discord.gg/h68Z9Ds2 - or send me a PM.

I'm not giving up, just busy. And the code isn't important. What's important is the documentation I've compiled explaining how, to my knowledge, NCS works.

What I've been documenting is a step-by-step model that takes individual NCS opcodes and recreates higher-level operations, which often lead to higher-level operations of their own. It's rarely a 1-to-1 reversal like "CONSTI 1, CONSTI 1, ADDII" (which is "1 + 1" in NWScipt), and often requires examining operations in various contexts.

For example, the JMP opcode shows up in four types of statements:

1. Expression statements
2. Jump statements
3. Iteration statements
4. Selection statements

And there are three jump statements and two selection statements. That's a total of seven contexts to examine (1 + 3 + 1 + 2).

And I've identified an actual error in at least one NCS compiler. I need to figure out if the reverse compiler should try to reverse those broken files, or just quit and complain.

[DrMcCoy 40]

1 hour ago, AmanoJyaku said:

And the code isn't important.

I'd disagree. The proof is in the pudddingcode.

But sure, do also release proper, structured documentation, please.

[Blue 68]

My 2 cents: The remake will never be KotOR. It has the potential to recapture the magic, but there will always be a desire to preserve the original experience. If you truly do end up walking away from this project I hope you do decide to release the source. One of the saddest parts of the Lucasforums Era was the lack of source code sharing with the community and as tool makers left the scene so did any hopes of getting a proper source code release from the author. 

 

Anyways thanks for what you have done so far to share the things you have found!

[DrMcCoy 40]

1 hour ago, Blue said:

One of the saddest parts of the Lucasforums Era was the lack of source code sharing with the community

Agreed. I come from the FLOSS side of things, for me it's natural to release sources. And while in the FLOSS space, we disagree on different licenses, we agree that source releases, with a proper license attached to allow for re-use, is important.

And "release early, release often" is also a common mantra: things might happen, so while we wait for a drop of perfectly finished and cleaned up sources, maybe you get hit by a bus. Or, less crass, your HDD dies and you don't have any backups. Or something else catches your interest or your RL gets bogged down or a global panini hits and your headspace is just weird as all f. It happens, that's okay.

Don't get me wrong, good documentation and detailed documented specs of file format is also important. Ideally, there's both docs and code. With code, though, if you can get it to run, you can immediately see if it's correct.

This isn't meant as an attack or rant or anything, just for an explanation of where I'm coming from here.

[AmanoJyaku 184]

On 5/13/2022 at 6:00 PM, DrMcCoy said:

With code, though, if you can get it to run, you can immediately see if it's correct.

This is why I'm not releasing the current source, because it doesn't run in the manner you're expecting.

A reverse compiler is not like your typical program. Let's say we're collaborating to write a web browser, I'm an expert in image decoding and you're an expert in network transmission. The two pieces are independent of each other, if you inform me you can no longer work on the networking component that does not impact my ability to complete the image decoder. Anyone can pick up where you left off, or scrap your work and start from the beginning.

That's not how a reverse compiler works. A reverse compiler works in stages, and each stage requires a formatted input. Without a properly formatted input subsequent stages will fail, which is what's happening now.

The documentation I'm writing details what I understand of that format, and that understanding has changed as recently as December. Consequently, the source code has changed. In the last year alone I've rewritten the program from the ground up four times, along with dozens of minor modifications. If I released sources you'd have to toss them in the trash after you realized they don't work.

This is also true for your source code. After reviewing your GIT I see you made the same mistakes I did, and that's why you've been struggling to progress.

tl;dr

This is not about ideology, it's about pragmatism. There's nothing special in the 1,500 lines of code I've written I feel I need to hide. I just know much of that is going to get replaced again. You're better off with a document detailing the format so if I can't figure out the remaining pieces someone else can.

And the source will practically write itself.

[DrMcCoy 40]

5 minutes ago, AmanoJyaku said:

After reviewing your GIT

Yes, you can actually inspect my git tree, see what I did, retrace the changes. That's exactly why it's public, so that other people can benefit from it. This is how this works.

10 minutes ago, AmanoJyaku said:

why you've been struggling to progress

I mean, no, not really? I haven't exactly tackled the decompiler stage yet, I've so far concentrated on the disassembler. The problem with xoreos is that there are so many different subsystems to have a got at, and I can only go at one thing at a time. And if you look at the git of all the xoreos repos, I haven't done much work on any of them. My struggle is rather, quite frankly, with trying to balance the requirements in our current nightmare capitalistic society and my mental health in the pandemic that's still ongoing despite everyone behaving like it's over, i.e. RL issues outside of xoreos. If you look at what FLOSS work I've done at all in the past 2 years, it's virtually all related to DM'ing tabletop RPG games via Foundry VTT, because I'm DM'ing 4 groups, I've crammed all my free time full with doing that, because that keeps me stable at the moment.

I also reject your claims that a decompiler is such a unique untertaking. Especially since I already have written one in the past, for the script language used in Coktel Vision's line of adventure games (Gobliiins, Woodruff and the Schnibble, but also the Adi/Adibou/Addy edutainment games) for my work with the ScummVM project. You'll find the same dependency on previous output when, for example, REing codecs.

And heck, if you look at what a complex beasts modern browsers are, those command the utmost respect from me. All the things you and I brought up here as an example can be done by one person alone, but a modern browser? Even compiling Firefox takes like multiple hours (and I do that regularily, since I run Gentoo Linux). I've filed multiple bug reports with Firefox, some I managed to trace, one I even managed to fix and send a patch upstream (which then wasn't used because someone else beat me to the punch), but most of them are too complex for me to understand. For example this here: https://bugs.gentoo.org/821898#c4 , where compiling Firefox with a specific version of Rust breaks it in a completely weird way.

[AmanoJyaku 184]

1 hour ago, DrMcCoy said:

Yes, you can actually inspect my git tree, see what I did, retrace the changes. That's exactly why it's public, so that other people can benefit from it. This is how this works.

That doesn't work, and there are whole books explaining the importance of documentation. The "why" is far more important that the "how", because why we do something tends to be fixed. How we achieve that thing can take many forms.

The only reason I understood your GIT tree is because it resembled the thought processes and code I had written independently in the first two years. However, perusing the comments in your GIT made it clear there's a lot of trial-and-error that's lacking a methodology. I'm documenting the methodology, and in the process discovering things none of us anticipated that has hindered our progress.

1 hour ago, DrMcCoy said:

I mean, no, not really? I haven't exactly tackled the decompiler stage yet, I've so far concentrated on the disassembler. The problem with xoreos is that there are so many different subsystems to have a got at, and I can only go at one thing at a time. And if you look at the git of all the xoreos repos, I haven't done much work on any of them.

Fair enough, but disassembly of NCS is trivial because it's a 1-to-1 conversion and can't be compared to reverse compiling back to NWScript.

1 hour ago, DrMcCoy said:

I also reject your claims that a decompiler is such a unique untertaking. Especially since I already have written one in the past, for the script language used in Coktel Vision's line of adventure games (Gobliiins, Woodruff and the Schnibble, but also the Adi/Adibou/Addy edutainment games) for my work with the ScummVM project. You'll find the same dependency on previous output when, for example, REing codecs.

I can't speak for that project, but I can speak for NCS. You've said things that make it clear you don't understand NCS. Structs don't disappear, recursive functions are easy to handle, and you're wasting your time recreating basic blocks and control-flow graphs.

But to understand why you need documentation, and I'm not alone in this. Read any specification and you rarely see example code, and certainly not code you can use in production.

Regardless, my position on this will not change. I'll release code when I have code to release, which will take longer than assembling the documentation I have.

I think it's better if I give an example in a language most people have encountered: HTML.

Let's say you're new to HTML and I gave you the following:

<!DOCTYPE html>
<html lang="en">
	<head>
		<title>Sample page</title>
	</head>
	<body>
		<h1>Sample page</h1>
		<p>This is a <a href="demo.html">simple</a> sample.</p>
		<!-- this is a comment -->
	</body>
</html>

You might look at every tag and think I made a mistake with the <!DOCTYPE> because it doesn't have an end tag like the rest.

But, how would you know?

And what if I modified the example to include a break?

<!DOCTYPE html>
<html lang="en">
	<head>
		<title>Sample page</title>
	</head>
	<body>
		<h1>Sample page</h1>
		<p>This is a <a href="demo.html">simple</a> sample.</p>
		<!-- this is a comment -->
		<br>
		<p>This is another paragraph.</p>
	</body>
</html>
</!DOCTYPE html>

Should the <BR> tag be terminated with an end tag?

[DrMcCoy 40]

1 hour ago, AmanoJyaku said:

You've said things that make it clear you don't understand NCS. Structs don't disappear, recursive functions are easy to handle, and you're wasting your time recreating basic blocks and control-flow graphs

I mean, you talk all you want about it, but it's all hot air, because you're not showing your work. Anyone can make grandiose claims like that.

I never said that struct "disappear", I have no idea where you get that from. Struct don't technically exist in the bytecode, but you can detect them by seeing stack elements that are copied around together, and when the DESTRUCT opcode is used to select the element that's to be operated on (which is essentially the bytecode view of saying "foo.x"). Tracing that is a bit of a hassle. And technically, a smarter compiler could implemented structs differently: instead of copying the whole struct and then calling DELETE, it could just copy the one single element. And the DELETE opcode could be used in other places.

That's the thing, I'm not only concerned with the OpenKnights compiler and the BioWare compiler from the KotOR era, I'm also concerned about later revision of the latter, used by Dragon Age II. I have already added 4 new opcodes to the xoreos disassembler (and the xoreos interpreter) used by Dragon Age: Origins and Dragon Age II, that, as far as I am aware, haven't been documented by anybody else prior to my work. I called them WRITEARRAY, READARRAY, GETREF and GETREFARRAY. If you think that shows I "don't understand NCS", sure. In either case, my work is out here in the open, for anybody to judge.

The general case of stack-analysing a recursive functions is impossible to handle, that's just a fact of the halting problem. If you can dispute that, you shouldn't waste your time on this, you should write a computer science or mathematics paper, whatever makes you more likely to be eligble for a Fields Medal or Abel Prize. If you've found a pattern that holds for both the BioWare compiler and the OpenKnights compiler, that is distinctive enough to not be triggered for other constructs, that's great too, but, like I said, anybody can make claims like that.

Let me be blunt here: I don't believe most of your claims. There, I said it. I don't believe your swagger that this is just all easy-peasy and you have it all working and figured out. You might well have found a pattern for some things, they might even hold up for the majority of cases, but that's the rub: there's always weird corner cases where it doesn't work, where there's ambiguity between two different control structures. This is why this is hard. For x86, there's only a few tools out there of varying quality and prices (HexRays is the old top dog in the infosec space, Ghidra is the newcomer, for example). NCS is no x86 (it has more in common with the JavaVM or AVM structurally), but a lot of the theoretical underpinnings are still relevant. I'm assuming you haven't look at what BioWare, if anything, changed up to Dragon Age II.

2 hours ago, AmanoJyaku said:

Should the <BR> tag be terminated with an end tag?

In XHTML, it should. But all that is irrelvant, because HTML (nowadays) is a set of standards, regulated by a standards body, reasoned about people, interoperability focused. We have specs written by that standards body.

For NWScript we don't have that, nevermind for its bytecode. In NWScript, everything is legal that the original compiler eats. And in NCS, everything is legal that the original game interprets as intended. That's it, that's all the securities we have. NWScript is literally "the proof is in the code".

2 hours ago, AmanoJyaku said:

Read any specification and you rarely see example code, and certainly not code you can use in production

Except in, for example, the codecs space. They release something called a "reference implementation". In short, what that implementation eats and produces, that's the file format.

Some might also release written specs. But have a look at what funny things the MPEG4 specs proclaim, including 3D rendering of arbitrary polygon data, which no one implements, not even they themself, and tell me that's the format. And then have a look at the code in ffmpeg, where you have decades of person-hours poured into getting all the files that actually exist out there in the world read.

Even your HTML example violates it, because look at the difference between what the specs say is legal and what browsers accept.

[AmanoJyaku 184]

4 hours ago, DrMcCoy said:

I mean, you talk all you want about it, but it's all hot air, because you're not showing your work. Anyone can make grandiose claims like that.

I suggest you take a breath and actually read what I wrote:

Quote

The documentation I'm writing details what I understand of that format, and that understanding has changed as recently as December. Consequently, the source code has changed. In the last year alone I've rewritten the program from the ground up four times, along with dozens of minor modifications. If I released sources you'd have to toss them in the trash after you realized they don't work.

I made it very clear I have nothing to give. Complain all you want, that isn't going to make code that doesn't exist magically appear.

 

4 hours ago, DrMcCoy said:

The general case of stack-analysing a recursive functions is impossible to handle, that's just a fact of the halting problem.

This is what I mean when I say you understand NCS. The halting probem has nothing to do with handling recursion in NCS. I've given hints in various posts as to why this is the case, but a formal document would better explain this. You're making this more difficult than it needs to be, which is exactly what I was doing for two years. And my document will explain why.

 

4 hours ago, DrMcCoy said:

Even your HTML example violates it, because look at the difference between what the specs say is legal and what browsers accept.

This line of reasoning is why I wouldn't release source code without documentation. The reason browsers accept invalid HTML is because they're designed to. They have a defined specification they know how to work around. That code snippet comes directly from the HTML specification document, and because I know the HTML spec I can write invalid HTML that parses just fine and produces identical output as valid HTML.

No such specification exists for NCS, at least nothing we've been able to recover.

And you already have an example of NCS documentation without source code: Skywing's NCS bytecode reference. There isn't a single line of code there.

 

This is not up for discussion. In order to write the source code you want I have to finalize the documentation. I've tried to avoid documentation, only to be forced to write it. End of story.

[DrMcCoy 40]

1 hour ago, AmanoJyaku said:

And you already have an example of NCS documentation without source code: Skywing's NCS bytecode reference. There isn't a single line of code there.

Except that this document is older than Skywing. It was created by Torlack, together with code, his compiler, which was released under the OpenKnights banner.

Skywing build upon the documentation and code, creating, among other things, an NWScript JIT environment.

The docs are also not without errors. SAVEBP, for example, does modify the SP, because that's where the BP is push onto. In German, we have a saying: "Papier ist geduldig", meaning you can write down a lot of things, none of which have to be true. Cicero knew it as "epistula non erubescit", a letter doesn't blush.

 

The rest of your post is more hot air, because you have nothing to show for yourself except unsupported claims. Even if your documentation materilized some day, which I find doubtful, I don't expect it to contain more than those sorts of half-wrong claims with no proof.

We're done here. Good day, Sir.

[Salk 376]

Just chiming in to say I'm sorry to see two valuable contributors arguing over something that should be a common interest and passion.

I barely understand a fraction of the whole problem but l know both of you put lots of effort and time into your work and it seems to me that the great divide is mostly a formality but again... I might have misunderstood it all.

[AmanoJyaku 184]

6 hours ago, Salk said:

Just chiming in to say I'm sorry to see two valuable contributors arguing over something that should be a common interest and passion.

I barely understand a fraction of the whole problem but l know both of you put lots of effort and time into your work and it seems to me that the great divide is mostly a formality but again... I might have misunderstood it all.

I appreciate the kind words, this is a silly argument that never should have gone beyond my first explanation.

Like DrMcCoy, I have to feed the capitalist machine. And like many, I have real life concerns. I'm havinng to dealing with the aftermath of two deaths and two attempted suicides, so I'm not amused when someone has a meltdown over a video game.

[Salk 376]

I take your work very seriously, Amano Jyaku.

And since I'm old enough l can tell you for personal experience that many a argument on a Forum or a chat would hardly ever take place between smart people talking face to face.

I've not the slightest doubt both you and DrMcCoy are both smart and competent.

It's been an awful period for the world (still is) and you're dealing with your own real life issues.

Hard sometimes to keep tempers and words in check but I appeal to your intelligence, hoping we can all dismiss the latest exchange as unworthy of you both.

Keep up the excellent job done on either side when you can.

We'll be here waiting and thankful for future and past contributions!

Cheers! 

[InSidious 237]

And on that note, I would like to issue a friendly reminder to please keep the discussion civil.

[JoeClaudus 0]

So I suppose the project was canned without anything being shared, either code or documentation?

[Guest Qui-Gon Glenn]

On 5/16/2022 at 9:30 AM, AmanoJyaku said:

I appreciate the kind words, this is a silly argument that never should have gone beyond my first explanation.

Like DrMcCoy, I have to feed the capitalist machine. And like many, I have real life concerns. I'm havinng to dealing with the aftermath of two deaths and two attempted suicides, so I'm not amused when someone has a meltdown over a video game.

Months later.... What a thread, and what a disappointing ending to it.

 

None of us knew what you were going through, and couldn't because you.... hadn't told us. We are all very sorry for your loss.

Nobody had a meltdown, but there was one person that acted like a tool in this thread, and I quoted that person.  Between passive aggression, pedantry (which, trust me, I usually appreciate) and then you take your toys and go home?

All too typical, to be honest. I hope you can recover from what you've gone through, and come back and apologize. We will gladly accept you back into the community, even if you never release a damned thing.

 

I don't really care much  about these games anymore, but then an interesting thread like this comes along, gets me interested enough to read and think about all of it, and finally the OP implodes. Fucking classic.

[AmanoJyaku 184]

9 hours ago, Qui-Gon Glenn said:

None of us knew what you were going through, and couldn't because you.... hadn't told us. We are all very sorry for your loss.

I told you as soon as I realized I likely wouldn't be working on the project for a while. The last update was January, my hiatus was May. I had no reason to explain my personal life, I wasn't looking for sympathy. I explained why because I was forced to remind someone this is a hobby, which takes a backseat to real life.

 

9 hours ago, Qui-Gon Glenn said:

I hope you can recover from what you've gone through, and come back and apologize.

I don't owe you, or anyone, an apology. Like other modders, I work on this because I want to, when I am able. I am currently unable, and anyone who cannot accept that is being unreasonable.

 

9 hours ago, Qui-Gon Glenn said:

We will gladly accept you back into the community, even if you never release a damned thing.

If earning your acceptance means begging for your forgiveness and subjecting myself to your insults, then I don't want it.

You also forget your place: it is you who rely upon me to provide this work. I don't have to do anything, and I honestly wish someone else would continue the work. That's why I've documented my progress in this thread. But no one has, so it continues to fall to me to do what I can. Not for you specifically, because quite frankly you don't deserve it. It's for everyone else who, despite their disappointment, anger, etc... refrained from attacking me even though they didn't get what they want when they wanted it.

If you're so set on getting this done, feel free to start a GoFundMe or Patreon. In the meantime, life beckons.

[VarsityPuppet 1,085]

Locking