Recommended Posts

Shouldn't matter. The game doesn't have a problem with this, it's the decompilers that do. Guess it's a good idea to create a new one, after all.

Share this post


Link to post
Share on other sites

I went through some of K1CP's own scripts and found a few that won't decompile. Running them through ncsdis, cp_end_trasksp_d produces a message I haven't seen before - "WARNING: Script analysis failed Because: Unbalanced stack in block fork merge @00000464: 2 != 5". It looks like cp_tar_brejikatk is recursion, but that one is actually just a renamed vanilla file.

Non-decompile_additional.7z

Share this post


Link to post
Share on other sites

To explain what this warning means:

ncsdis tries to run through the whole disassembly to analyze the stack. Essentially, how the stack looks after each operation: how many stack slots are allocated and what types (int, string, float, ...) are in there. When there is a fork, for example due to branching in an if-clause, ncsdis follows both paths. And when the paths merge again, for example after the if when the excution continues on linearly, the stack has to look the same for both paths. Otherwise, I wouldn't know what to make out of this situation. (ncsdis only checks that the size of the stack matches, and ignores the types at the moments.)

In this case, for some reason ncsdis found that the stack sizes of two paths don't match. It's 2 slots for one path and 5 slots for another path. This might be a bug in ncsdis.

This could also be a bug in the compiler that produced the NCS in question. For example, IIRC I found the same message for one of the bugs I wrote about in my blog post, the one where the short-circuiting boolean OR. In that case, it didn't really matter that the stack is broken for one path, because that path was logically dead anyway, it would never have been called. So to work around this bug in the original compiler, I added a dead branch check, and the stack analysis won't follow dead branches.

It might be that this here is a similar problem and my dead branch check is not catches it and needs to be expanded. Or this script is broken after all. (Or it could still be a bug in ncsdis.) I'd need to check that, I'll add it to my TODO list. Or if anybody else would like to tackle this, feel free :)

Share this post


Link to post
Share on other sites

Here's the source if that helps. I removed an include function to see if that was it and it still fails to decompile. Also tried not declaring GetUserDefinedEventNumber() as an int and just popping it directly in the switch, but again still fails. Although the ncsdis error has now changed to Unbalanced stack in block fork merge @000001AB: 1 != 4

void main() {
	
	switch (GetUserDefinedEventNumber())
	{
		case 50:
			object oTrask = GetPartyMemberByIndex(1);
			object oPC = GetFirstPC();
			location lTraskSp = Location(Vector(16.34,20.5,-1.27), 180.0);
			
			DelayCommand(0.25, AssignCommand(oTrask, ActionDoCommand(SetFacingPoint(GetPosition(oPC)))));
			DelayCommand(0.6, AssignCommand(oTrask, ActionStartConversation(oPC, "", FALSE, CONVERSATION_TYPE_CINEMATIC, TRUE)));
			SetGlobalFadeIn(0.9, 0.5);
			break;
	}
}

Apologies AmanoJyaku for derailing your thread. I appreciate you confirming my suspicions regarding the problem being something to do with the includes by the way. It's nice to have that mystery finally resolved.

cp_end_trasksp_d.7z

  • Like 1

Share this post


Link to post
Share on other sites

Not derailed at all. This is exactly what I will encounter when I get far enough. May actually be easier to troubleshoot than the parser and lexer I'm writing. 🤣

  • Like 1
  • Thanks 1

Share this post


Link to post
Share on other sites

I can confirm that your appraisal was correct. At least with the two files I have tried so far. If I hex edit them and replace 03 00 00 7E 21 05 00 00 EF 01 with 05 00 0E 44 45 43 4F 4D 50 5F 52 45 50 4C 41 43 45 (effectively replacing string RACE_DEFAULT = GetStringByStrRef(32289); with string TEST_REPLACE = "DECOMP_REPLACE";) then DeNCS decompiles them. Many thanks, this will open up a lot of previously closed doors.

  • Like 2
  • Thanks 2

Share this post


Link to post
Share on other sites

"Closed doors"

*AmanoJyaKreia, Dark Lord of the Script, mentors DarthParametric and boosts their security skill*

  • Like 1
  • Haha 2

Share this post


Link to post
Share on other sites

Greetings, thread master - and fellow Jedi as well!

Salute for the progress made regarding the development of the tool, and recent discovery concerning script's decompiling -- a promising future in script-modding indeed. :cheers: Not to being off-thread and hoped what I've got here can provide an insights more so to the tool's development [and/or discussion].

So I found another script that failed to be decompiled by DeNCS, it's k_creditsplay; a script that fires the lightside credit sequence fired in STUNT_57 RIM/module. It had these with the bytecodes --

Spoiler

// k_creditsplay
// STUNT_57/Lightside ending ceremony

00000008 42 00000062              T 00000062
0000000D 1E 00 00000008           JSR fn_00000015
00000013 20 00                    RETN
00000015 04 05 0000 str           CONSTS ""
00000019 04 03 00000000           CONSTI 00000000
0000001F 05 00 0206 02            ACTION StartCreditSequence(0206), 02
00000024 04 03 00000001           CONSTI 00000001
0000002A 04 05 000A str           CONSTS "CREDITPLAY"
00000038 05 00 0243 02            ACTION SetGlobalBoolean(0243), 02
0000003D 04 04 00000000           CONSTF 0.000000
00000043 04 04 00000000           CONSTF 0.000000
00000049 04 04 00000000           CONSTF 0.000000
0000004F 04 04 00000000           CONSTF 0.000000
00000055 04 04 00000000           CONSTF 0.000000
0000005B 05 00 02D0 05            ACTION SetGlobalFadeOut(02D0), 05
00000060 20 00                    RETN	

which in NSS could be --

Spoiler

// k_creditsplay
// STUNT_57/Lightside ending ceremony

void main()
{
	SetGlobalFadeOut(0.0f, 0.0f, 0.0f, 0.0f, 0.0f);
	StartCreditSequence(FALSE);
	SetGlobalBoolean("CREDITPLAY", TRUE);
}

Meanwhile, the other script which is slightly longer; k_stunt_end -- fires the darkside credit sequence fired in STUNT_55a RIM/module, can be decompiled and had these with the bytecode --

Spoiler

// k_stunt_end
// STUNT_55a/Darkside ending ceremony

00000008 42 000000BB              T 000000BB
0000000D 1E 00 00000008           JSR fn_00000015
00000013 20 00                    RETN
00000015 04 04 00000000           CONSTF 0.000000
0000001B 04 04 00000000           CONSTF 0.000000
00000021 04 04 00000000           CONSTF 0.000000
00000027 04 04 00000000           CONSTF 0.000000
0000002D 04 04 00000000           CONSTF 0.000000
00000033 05 00 02D0 05            ACTION SetGlobalFadeOut(02D0), 05
00000038 04 03 00000001           CONSTI 00000001
0000003E 04 05 0002 str           CONSTS "55"
00000044 05 00 0301 02            ACTION QueueMovie(0301), 02
00000049 04 03 00000001           CONSTI 00000001
0000004F 05 00 0302 01            ACTION PlayMovieQueue(0302), 01
00000054 2C 10 00000000 00000000  STORE_STATE 10, 00000000, 00000000
0000005E 1D 00 0000002B           JMP off_00000089
00000064 2C 10 00000000 00000000  STORE_STATE 10, 00000000, 00000000
0000006E 1D 00 0000000E           JMP off_0000007C
00000074 1E 00 00000021           JSR fn_00000095
0000007A 20 00                    RETN
0000007C 04 04 3D4CCCCC           CONSTF 0.050000
00000082 05 00 0007 02            ACTION DelayCommand(0007), 02
00000087 20 00                    RETN
00000089 05 00 0224 00            ACTION GetFirstPC(0224), 00
0000008E 05 00 0006 02            ACTION AssignCommand(0006), 02
00000093 20 00                    RETN
00000095 04 03 00000000           CONSTI 00000000
0000009B 05 00 0206 01            ACTION StartCreditSequence(0206), 01
000000A0 04 03 00000001           CONSTI 00000001
000000A6 04 05 000A str           CONSTS "CREDITPLAY"
000000B4 05 00 0243 02            ACTION SetGlobalBoolean(0243), 02
000000B9 20 00                    RETN

which in NSS will be --

Spoiler

// k_stunt_end
// STUNT_55a/Darkside ending ceremony

void ST_StartCreditSequence()
{
	StartCreditSequence(FALSE);
	SetGlobalBoolean("CREDITPLAY", TRUE);
}

void main()
{
	SetGlobalFadeOut(0.0f, 0.0f, 0.0f, 0.0f, 0.0f);
	QueueMovie("55", TRUE);
	PlayMovieQueue(TRUE);
	AssignCommand(GetFirstPC(), DelayCommand(0.05f, ST_StartCreditSequence()));
}

Therefore the question is --

  • Why two similar scripts had different result, as in one can be decompiled while other isn't?

From my limited view and understanding, there's a difference in how the StartCreditSequence were called - observed by the bytecodes. k_creditsplay has --

00000013 20 00                    RETN
00000015 04 05 0000 str           CONSTS ""
00000019 04 03 00000000           CONSTI 00000000
0000001F 05 00 0206 02            ACTION StartCreditSequence(0206), 02

while k_stunt_end --

00000093 20 00                    RETN
00000095 04 03 00000000           CONSTI 00000000
0000009B 05 00 0206 01            ACTION StartCreditSequence(0206), 01

That's only far as I can go with what I've got - and much thanks for considering this.

  • Like 2

Share this post


Link to post
Share on other sites

Not off-topic at all!!!

My guess is because the compiled script is wrong. Look at what it does:

//Simplified code
void main()
{
	StartCreditSequence();
	SetGlobalBoolean();
	SetGlobalFadeOut();
}

So, what's wrong? The very first function:

// 518: StartCreditSequence
void StartCreditSequence( int bTransparentBackground );

It takes one parameter, an integer. However, look at the byte code:

04 05 0000	//CONSTS - Place Constant String Onto the Stack, 0 characters
04 03 00000000	//CONSTI - Place Constant Integer Onto the Stack, value of 0
05 00 0206 02	//ACTION - Call an Engine Routine, 0x0206 [518, or StartCreditSequence(int)], with two arguments

Can you see what's wrong here?

 

Off-Topic: Sometimes the code blocks have colored code. Then, it goes white. Why???

  • Like 1

Share this post


Link to post
Share on other sites

It's trying to call it like:

StartCreditSequence(0, "");

AmanoJyaku can correct me if I'm interpreting the bytecode incorrectly, being further along the get a degree in computer science in order to make better modding tools for a 15-year-old game career track, but I believe the number after the function is the number of arguments to call the function with, and then those arguments are taken off the stack. Things are taken off a stack last in, first out, so it takes the integer 0 and then the empty string.

My guess is that at some point in development, the function took two arguments - maybe the string was for the name of a movie file - but they changed how the function behaves sometime after this script was compiled, and never recompiled it.

17 hours ago, AmanoJyaku said:

Veeeery interesting. I'm seeing the same thing in all files, so I hope it's the culprit.

Basically, every NCS file I've worked on uses global variables initialized from constants. However, the files you've given me all include a global that's initialized from an engine routine! I'm wondering if DeNCS is choking on that.

That seems to be the case. I tested with the following:

int testInt() {
	return 1;
}

int TEST_INT = testInt();

And that will cause DeNCS to fail to decompile any script that includes it.

For completion's sake, I also tried it this way:

int testInt() {
	return 1;
}

int x;
x = testInt();

Interestingly, this would not even compile.

It seems the tools aren't very fond of variables with global scope, but I'm not sure why it would care whether one was initialized or not.

  • Like 1

Share this post


Link to post
Share on other sites
1 hour ago, DarthParametric said:

Or a string and an int.

 

21 minutes ago, JCarter426 said:

It's trying to call it like:


StartCreditSequence(0, "");

 

Yep, the problem is the script is compiled so that StartCreditSequence(int) is told to accept two arguments, despite being defined as accepting one. DeNCS is failing to decompile, because the code doesn't match the function definition. It doesn't know what to do.

31 minutes ago, JCarter426 said:

It seems the tools aren't very fond of variables with global scope, but I'm not sure why it would care whether one was initialized or not.

Crappy coding. Given the fact that I've only just now seen a function call used to initialize a global, my guess is DeNCS was never written with that in mind. It's taking me forever to write my compiler/decompiler because I'm accounting for this and more. And, I still expect the finished product to have issues. 🤣

Share this post


Link to post
Share on other sites
4 hours ago, AmanoJyaku said:

DeNCS is failing to decompile

Per some discussion we were having about this in the Mod Development channel of the /r/kotor Discord, is it actually nwnnsscomp that is choking? I'm kind of hazy on how exactly it works, but my assumption is that DeNCS calls nwnnsscomp to decompile the NCS into bytecode, it then converts that into a script, then it calls nwnnsscomp to compile and decompile that and compares that bytecode back to the original's. I would be interested to know what DeNCS would do if you could feed it the bytecode from ncsdis, since that apparently is able to generate it from that script. But the recompiling step would be a roadblock.

Edit: Hrm, trying nwnnsscomp directly it seems not to be the case. It can apparently produce bytecode for those scripts just fine:

00000C67 02 05                    RSADDS
00000C69 04 03 00007E21           CONSTI 00007E21
00000C6F 05 00 00EF 01            ACTION GetStringByStrRef(00EF), 01
00000C74 01 01 FFFFFFF8 0004      CPDOWNSP FFFFFFF8, 0004
00000C7C 1B 00 FFFFFFFC           MOVSP FFFFFFFC

So I guess it really is all down to DeNCS.

Did you get a chance to have a look at that third set of non-decompiling scripts btw? Going through some of the Tatooine modules I also found some new ones that still fail after hex editing them to remove the GetStringByStrRef from the include (although I gather a couple of them could just be recursion):

Additional_04.7z

Share this post


Link to post
Share on other sites

One file has RSADDI in the _start() function, which I don't think is legal.

But, most files seem OK at first glance. Will have to investigate some more as this is beyond the capabilities of the current analyzer.

Share this post


Link to post
Share on other sites

Time for a monthly update. And what a month it's been...

 

The list of to-do's hasn't seen much progress, unfortunately:

  •     Identifying iteration, selection and jump statements
  •     Operator associativity and precedence
  •     Type conversions
  •     Byte code conversion to source code
  •     Source code conversion to byte code
  •     GUI
  •     Setup new dev laptop (dropped current laptop last night, awaiting delivery of new one) 😢
  •     Probably more stuff, but I don't know what I don't know, you know?

 

I was cursing my luck at having dropped my old laptop, but the timing couldn't have been any better. I got the new one right before deliveries started to be impacted due to COVID-19. Additionally, the laptop DOUBLED in price two weeks after I purchased mine. I won't be buying from that vendor again...

 

As for NCS byte code, the following progress has been made:

  • Identifying iteration statements (do-while, while, and for)
  • Identifying selection statements (if and switch)

Jump statements break and continue are still being worked on. Return is the only jump statement that is easily identified, because NWScript compiles each subroutine with only one RETN instruction. An NWScript function that has multiple returns:

Spoiler

//:: k_sup_galaxymap

void main()
{
    int nSelected = GetSelectedPlanet();
    int nPrevPlanet = GetCurrentPlanet();

    if(nSelected == -1) return;	//Explicit return;
	//More code

	//Implicit return;
}

 

Simply jumps from the basic block with the selection statement to the basic block that has the return.

 

In the category of "Probably more stuff", DarthParametric and I (and probably other people, fuzzy old man brain!) discovered another reason to replace DeNCS: it cannot decompile scripts that have global variables initialized by functions:

Spoiler

//Sample file

//Global variables
string RACE_DEFAULT = GetStringByStrRef(32289);
//End Global variables
void main()
{
	//Do something with RACE_DEFAULT
}

 

The example is perfectly valid code, but DeNCS doesn't like it. Replace GetStringByStrRef(32289) with a fixed string, e.g. "Amano's Test String", and DeNCS happily goes to work. The problem is not GetStringByStrRef() as DeNCS has no problem decompiling it inside functions. My decompiler isn't affected by this, but it's definitely something to be aware of.

A problem my current NCS analyzer does have is that I wasn't handling the STORE_STATE instruction correctly. My analyzer saw blocks of code as dead code, when in fact they were part of the STORE_STATE flow. I'll have to fix this.

 

I should be further along than this, but I've been sidetracked by:

  • A demanding client, one of the few still in business due to COVID-19 (I need the money)
  • My concern over weak healthcare and labor laws, which my country doesn't give a damn about (COVID-19 has made things really bad)
  • My family and friends, who are all older and dealing with health issues and in many cases still working face-to-face (COVID-19!!!)
  • My new obsession with assembly language, which I thought would help me with decompiling from NCS (it hasn't, and can't)
  • Learning parsing and lexing, which will allow this work in progress to compile to NCS
  • My new laptop, which allows me to play games that I haven't touched in months (I'm kind of stressed)
  • This girl I've been seeing 😘

The work continues...

  • Like 1

Share this post


Link to post
Share on other sites
54 minutes ago, AmanoJyaku said:

//Do something with RACE_DEFAULT

Note that since Bioware's compiler (and nwnnsscomp too) pull in all globals listed in listed includes, the script doesn't actually need to make any use of RACE_DEFAULT. Simply being in the global list is enough to bork it. Which is why half the Tatooine and Manaan scripts don't decompile, despite none of them appearing to actually make use of it (not that I have seen anyway). Doing this:

#include "k_inc_tat"

void main() {

}

is enough to choke DeNCS.

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.