GFF Compiler/Decompiler

[bead-v 251]

@JCarter426 and me were talking on Discord and we came up with this idea. We were talking about editing .pth files directly in kotormax, but since kotormax can't export binary files, we'd need a compiler to produce the actual .pth file.

So the idea is that the compiler could turn any gff file (dlg, pth, git, utc, ifo, ...) into a .gff.ascii file. The ascii file would contain the information about the file type, but otherwise the files wouldn't be distinguished.

The ascii file might look something like this:

# GFF ASCII
# Created ....

type dlg
version 3.2

newstruct
    dword DelayEntry 0
    dword DelayReply 0
    dword NumWords 0
    resref EndConverAbort script1
    resref EndConversation script2
    byte Skippable 0
    resref AmbientTrack mus_amb_1
    byte AnimatedCut 0
    resref CameraModel 003ebocam
    byte ComputerType 0
    int ConversationType 0
    list EntryList
    newstruct 0
    	cexostring Speaker "Atris"
        cexostring Listener "Kreia"
        list AnimList
        endlist
        cexolocstring Text -1
        	0 "This is a cexolocstring English male variant."
        	1 "This is a cexolocstring English female variant."
        	2 "This is a cexolocstring French male variant."
        endcexolocstring
        resred VO_ResRef someresref
        resref Script script3
        dword Delay -1
        ...
        int RecordNoVOOverri 0
    endstruct
    newstruct 1
    	...
    endstruct
    newstruct 2
    	...
    endstruct
    endlist
    byte OldHitCheck 0
    list ReplyList
    endlist
    ...
    cexostring EditorInfo "v2.3.2 Apr 30, 2008 LastEdit: 13-Dec-16 16:31:52"
endstruct

Does anyone have any objections, ideas, concerns, opinions etc... about this?

[DarthParametric 3,785]

Could be useful if it can generate triggers and so forth. You could essentially import the entire level into Max, rather than just the room geometry. Perhaps you could have the importer read the GIT as well as the LYT? You could place dummies for things like placeables and creatures.

[bead-v 251]

Yep. If it comes to that, I'd also make the compiler part of mdledit as well, so it would simply compile everything.

Although getting things ready for this on the kotormax side will take its time... for now, all that I'm really considering is the compiler.

[VarsityPuppet 1,085]

Good thinking @bead-v!  Been thinking about an idea like this for awhile as well. It will be a lot easier to merge gff files in any *ahem* future installers that any of us may want to try to make.

I have been thinking about using the JSON file format, but it has its own issues, particularly with type-casting and that will simply not work with any GFF type stuff.

 

[bead-v 251]

Yeah, there's also a problem with the ascii format for the VOID data type, because that one stores binary code. I was thinking we could convert to Base64 or the like. Though I don't know if it's even used in any gff file in Odyssey. But for the other data types, I think an ascii format would work fine.

[JCarter426 1,215]

13 hours ago, DarthParametric said:

You could essentially import the entire level into Max, rather than just the room geometry. Perhaps you could have the importer read the GIT as well as the LYT? You could place dummies for things like placeables and creatures.

Yeah, that's on my wish list for this. It's something VP and I talked about before, though I figured it would just be a simple export of coordinates and then maybe we could import that into an existing GIT file somehow. bead-v is the one who thought of going to the trouble of inventing an entire new format for GFFs instead. Though I'd be happy to see either.

15 minutes ago, VarsityPuppet said:

It will be a lot easier to merge gff files in any *ahem* future installers that any of us may want to try to make.

I was thinking that, too. I don't know the specifics of TSLPatcher's current issues, beyond what it can and can't do with sort of an inking of why because I bugged Fair Strides about it for an explanation, but if we have a new, modder-designed format to work with that can actually go back and forth between GFF and text, then some other things on our modder wish lists might be doable.

For example, perhaps the new compiler (whether it's included in MDLEdit or whatever) could detect what different GFF trees actually are, so it would know not to add tons and tons of duplicate items to a creature's inventory if run multiple times, as TSLPatcher does, or perhaps it could allow for deleting trees.

[bead-v 251]

2 minutes ago, JCarter426 said:

though I figured it would just be a simple export of coordinates and then maybe we could import that into an existing GIT file somehow. bead-v is the one who thought of going to the trouble of inventing an entire new format for GFFs instead.

Well, it's either
a) configure kotormax (or other relevant tool) to export coordinates AND create a new tool to convert specifically that type of coordinate list file to a specific gff format using some logic, or
b) configure kotormax (or other relevant tool) and force it to export in .gff.ascii AND create a general compiler from .gff.ascii to the GFF format.
In b), the compiler can handle any type of gff, so unlike in a), you don't need a new conversion tool for every type of gff file and also, a compiler takes no effort to make because many of us have a GFF library by now.
 

8 minutes ago, JCarter426 said:

perhaps the new compiler (whether it's included in MDLEdit or whatever) could detect what different GFF trees actually are, so it would know not to add tons and tons of duplicate items to a creature's inventory if run multiple times, as TSLPatcher does, or perhaps it could allow for deleting trees.

Since it would be merely a compiler, all that would be done in the ascii file. The compiler's only job would be to convert the ascii into a binary gff. If the structure in the ascii is valid/correct/as intended, then the one in the compiled gff will also be valid/correct/as intended.

[JCarter426 1,215]

Yeah, I guess I should've said compiler + merger that I guess you'll be forcing @VarsityPuppet to make.

What I mean though is if you toolmasters are designing the new GFF text format and tools to read it, it may enable more read/write functionality in the future. So we could say "only add this item if this exact GFF tree doesn't already exist" or "delete this exact GFF tree" where TSLPatcher is currently limited to "add this GFF tree" or "delete GFF tree # whatever without knowing what that actually is".

[bead-v 251]

6 minutes ago, JCarter426 said:

Yeah, I guess I should've said compiler + merger that I guess you'll be forcing @VarsityPuppet to make.

Well, it's his dream

8 minutes ago, JCarter426 said:

So we could say "only add this item if this exact GFF tree doesn't already exist" or "delete this exact GFF tree" where TSLPatcher is currently limited to "add this GFF tree" or "delete GFF tree # whatever without knowing what that actually is".

I take it that by "trees" you mean "structs". If so, then you'd basically need to specify for every struct operation what fields in the struct have to match for it to be considered identical. There are struct IDs in the GFF, but they aren't even used a lot of the time and even if they were, there's a big chance that still wouldn't help you find the right struct to run the operation on.

[JCarter426 1,215]

I believe the struct IDs are by type rather than... by ID. Like every inventory item will have the same ID. So that's a problem, yeah. TSL Patcher can only count structs - so you can change the field # in struct # to whatever or delete a specific struct # or of course add a struct. I believe it can delete, anyway, but it doesn't support it because of the problems that would cause - if something were deleted, all the numbers would be changed, so your mod wouldn't work with any other mod that alters the same part of that GFF, thus defeating the point of the Patcher. And when it's told to add, it just keeps making more structs. If you're VP is making a new format, though, I would imagine that it would allow for more functionality than that. Theoretically, in the future.

As for what would be considered identical, hmm... I guess that's situational, and it depends on the specifics of the mod being installed, but it could be as simple as a resref. "Add g_w_lghtsbr02 if there isn't a struct with one already", that sort of thing.

[bead-v 251]

From what you've said, I gather that the patcher is missing the functionality to find a struct not only by number but also by the value of one or several of the fields that it contains. The new format itself can't solve that. It's a TSLPatcher problem not a format problem.

The benefits of the format are:
1. It's easier to write a program that handles a text file than a binary file (at least in the specific case of GFF files). Any new tool could benefit from it.
2. No need to use the GFF editor, you can edit the file in a text editor, which can speed some operations up a little bit, but will at the same time make errors more likely.
3. Kotormax could read and write GFF files with the help of the compiler, since it can only deal with text files (except maybe when used with 3ds, but even that wouldn't be optimal).

 

20 minutes ago, JCarter426 said:

I guess that's situational

Exactly! You need a way to specify for every case specifically how to compare the structs. And since you can't generalize this, there's no way you could put this into a format, it needs to be part of the program that handles the GFF. That's what's missing from the Patcher.

[JCarter426 1,215]

11 minutes ago, bead-v said:

It's a TSLPatcher problem not a format problem.

Yeah, I get that, but as you said - it's easier to write a program that handles a text file. I guess this is a thing for VP to figure out though. And even if that doesn't pan out, the features of just the text format alone sound good.

11 minutes ago, bead-v said:

No need to use the GFF editor, you can edit the file in a text editor, which can speed some operations up a little bit, but will at the same time make errors more likely.

Like editing models in a text editor? I'm used to that.

11 minutes ago, bead-v said:

Exactly! You need a way to specify for every case specifically how to compare the structs. And since you can't generalize this, there's no way you could put this into a format, it needs to be part of the program that handles the GFF. That's what's missing from the Patcher.

Mm, I see... I was thinking that with your compiler, or this might be another matter for the merger/installer/whatever new tool, it would be easier to see that structs were identical. As in, not similar, but 100% the same data, because it was added multiple times, or they came from the same source in such a case like merging different copies of the same file. But maybe checking the entire struct would take a lot of processing time, and then any shortcuts for that would be situational....

[VarsityPuppet 1,085]

Ah I do remember my goals behind using JSON a little more clearly now. 

 

The idea was to represent the data in an agnostic format that could be used to load up in some hypothetical project management software, compiling to a GFF format or some other third thing. 

 

It would be slightly different than your proposed ascii format Bead in that the compiler would have to know about how to turn the JSON representation to the proper GFF file.

 

Put simply, the JSON representation of your example Bead, might look like this:

 

{

    “DelayEntry”: 0,

    “DelayReply”: 0,

    “NumWords “: 0

    “EndConverAbort”: “script1”,

    “EndConversation”: “script2”,

    “Skippable”: 0,

    “AmbientTrack”: “mus_amb_1”

    “AnimatedCut”: 0,

    “CameraModel”: “003ebocam”,

    ”ComputerType”: 0,

    “ConversationType”: 0,

    “EntryList”: [{

        “Speaker”: "Atris",

        “Listener”: “Kreia",

        “AnimList”: [],

        “Text”: {

            “0”: “This is a cexolocstring English male variant.",

            “1”: “This is a cexolocstring English female variant."

            “2”: “This is a cexolocstring French male variant."

        },

        “VO_ResRef”: “someresref”,

        “Script” “script3”,

        “Delay”:  -1

    }, {

        //new node

    }]

    //etc...

}

 

Though this has it’s downfalls. Since you lose any concept of type, it will make input validation client-side more troublesome (like a KOTORTool-esque GFF editor).

 

I am just a fan of simplifying file representations, because let’s be honest, GFF files are a bit complex sometimes.

[VarsityPuppet 1,085]

For purposes of merging, I think it might be best to try to figure out what our common GFF merging situations are. I mean, is it mostly adding structs? Adding/changing fields?

[JCarter426 1,215]

I'm not the best source of intelligence on this because I've usually avoided dabbling with GFF files in TSLPatcher precisely because of its problems, but a few common changes I have made are:

• Changing a character's appearance. (Changing fields.)
• Adding items to a character or placeable's inventory. (Adding structs - and the source off one of my major gripes with TSLPatcher, because if you keep running it, you get more and more items.)
• Making an item droppable or not droppable. (Changing fields. It's somewhat of a minor miracle that this works both as adding/removing a struct or just changing the field.)
• Changing what items a character has equipped. (Changing fields.)
• Changing an object's scripts. (Changing fields.)
• Making an item upgradeable. (Changing fields.)
• Changing an item's model and/or texture variation. (Changing fields.)

Some less common ones:

• Giving a character the necessary feats to equip an item. (Adding structs.)
• Changing a character's soundset. (Changing fields.)
• Changing a character's conversation. (Changing fields.)

Some things TSLPatcher cannot do, but that have come up before:

• Removing an item from a character or placeable's inventory, or one that's equipped.
• Changing an item name or description.
• A whole mess of things with dialogue files that I'd be surprised if they were ever resolved.

[ndix UR 218]

I would definitely recommend using a standard object serialization format for your textual representation. Either JSON or YAML. YAML is messier as a standard (far more complex parser and encoder), but more human-readable and can have comments. JSON is simple, but hand editing in a text editor is a little trickier because its parser is more rigid (also making it much easier to implement).

With either format, you wind up using a more complicated structure if you're modeling the actual GFF data. The format VP showed is nice for export/viewers/other contexts where reconstruction isn't necessary. Here's one way to do the typed version:

[ {
  type: "struct",
  value: {
    "DelayEntry": { value:0, type:"dword" },
    "DelayReply": { value:0, type:"dword" },
    "NumWords": { value:0, type:"dword" }
  }
} ]

An approach for retaining type while using the simplified/export version of the JSON might be to have a tool that reads a full GFF file and exports another json object with pathed type info. Like /DelayEntry: "dword", /EntryList/Text: "cexolocstring", etc. Then have the 'compiler' combine that info. Could even just run it on sample files for all types and put it into a structured type dictionary with path lists per type.

Personally I've been using GFF->JSON for awhile and it would certainly be my preference, but once you're creating the structured objects, yaml or json is a one line code change. Avoiding writing your own parser is a good thing. I'll just leave this here:

        cexolocstring Text -1
        	0 "This is a cexolocstring English male variant.
 I "hate" my life now."
        	1 "This is a cexolocstring English female variant.

Why did someone put a descriptive paragraph 1 \"in here\". "
        	2 "This is a cexolocstring French male variant.

\tIt is also a long string that contains an internal multiline string

\"Write your own parser, they said.

It'll be fun, they said.\""

Parsing that isn't my idea of fun. I guess the question and limiting factor here is whether 3DS gives you a native JSON/YAML(or any standard format) parser? Also whether it gives you base64 encoding/decoding (there's a "void" type in GFF which is just raw binary data)...being a proprietary language, your ability to extend and rely on existing libraries and language features is probably not so good?

[bead-v 251]

8 hours ago, ndix UR said:

Avoiding writing your own parser is a good thing. I'll just leave this here:

Parsing that isn't my idea of fun.

The reason I created this thread was precisely because I wanted everybody who wanted to to contribute their ideas and find the best solution together. I wrote that whole format in a minute right before posting, just to show approximately what the format could look like, to get the discussion started. No need to keep picking holes in it like that ☹️

I've discussed this some more with VP on discord and it does seem like going with JSON is the better way. A typed version is fine by me, maybe something like ndix UR proposed. But I can also see VP's point – we could potentially be forcing the user to remember types, which could easily go wrong, especially if that causes problems with the game once it's compiled.
 

8 hours ago, ndix UR said:

An approach for retaining type while using the simplified/export version of the JSON might be to have a tool that reads a full GFF file and exports another json object with pathed type info.

Also an option, but since the correspondence between the type and the label (+path) doesn't ever change, it makes little sense to put it in a separate file where the user could potentially change it and in so doing ruin it. That information may be best kept inside the program. And there's only so many types of GFF files.
 

8 hours ago, ndix UR said:

Personally I've been using GFF->JSON for awhile

Does that mean a (de)compiler already exists? It's been a running joke, but I don't think we actually do need another mdledit/mdlops situation.
 

9 hours ago, ndix UR said:

 I guess the question and limiting factor here is whether 3DS gives you a native JSON/YAML(or any standard format) parser?

The answer to this is no. There may be a script somewhere on the Internet that does it though. A quick search did reveal some attempts, but not the full thing. The problem is also that later versions implemented python support, which is what some used to solve the problem. Kotormax should remain compatible with gmax however, so that doesn't help.
 

9 hours ago, ndix UR said:

Also whether it gives you base64 encoding/decoding (there's a "void" type in GFF which is just raw binary data)...

Two points about that. FIrst, do we even have an occurrence of the void type in the Kotors' GFF files? Second, if we do, do we really need to be able to import that anywhere in this form? Another way to export the void format on decompilation would be to put the binary code into a separate file and specify the name of the file for the value in the JSON export file. That filename could then also get imported into max (and other editors).