Kotor/TSL Model Format (MDL/MDX) Technical Details

[ndix UR 218]

This thread is for discussing and retaining technical details of the Kotor MDL/MDX binary formats, with the goal of cracking the meaning of some of the remaining unknowns.  In implementing a substantially improved MDLOps, I have been extensively leaning on the technical work of cchargin (original author of MDLOps) and MagnusII (original author of KAurora?), who are serious heroes.

 

State of Knowledge and Tools:

Many people point to the kotor/mdl_info document of cchargin as a source of reference. We need to produce something better than this. It has certain information which is completely vital that I have found nowhere else, but we really need to produce a redone version, maybe distributed as a PDF modding resource or something, that incorporates MagnusII's info from the KAurora source.  All I was able to find (thanks DarthSapiens for code post in a forum article on bump mapping) of the KAurora source so far is the trimesh header structure from the bump mapping thread, and it is much more nuanced and correct than the original cchargin info. Differentiation between signed and unsigned can actually matter, especially in the byte & short types. My MDLOps is using the improved data layout (for the mesh header) and accuracy has gone up.

 

Background, relation to NWN:

Currently, the way I understand modeling to work is like this: Kotor modelers basically rely on tools developed for working with NWN. NWN modelers have life fairly easy, because they can actually work with text/ascii models and the game will compile them on the fly to its own binary format. So, in essence, Kotor models get decompiled/recompiled in order to be compatible with NWN text-based modeling tools. There are a couple important/interesting points to note here. 1) There is no 'official' text format for Kotor models. Such a thing never existed, and has only been agreed upon by the modding community. 2) The binary formats and capabilities of NWN and Kotor are substantially different. They are far from directly compatible. What this also means is that a lot of things the NWN tools put into ascii models just don't exist in Kotor, and a lot of things that Kotor can do are totally unknown to NWN toolsets.

 

Kotor Model Unknowns:

    Big Caveat: I won't know the full extent of what is still unknown until I can access KAurora's source, if someone can get me that, this list will improve a lot. I have some suspicions though, based on extensive reading online.  I am starting the list with the question marks that I can tell remain in MagnusII's work.  I want to expand it to an exhaustive list once I actually get a handle on the full scope from KAurora...

 

  Small Caveat: It is important to realize that some of these 'unknown's may turn out to be just useless padding. Padding is regularly used in binary formats for various reasons.

• (Animation) Controller 240
• Common Mesh header unknowns: an array of 3 signed integers [ -1, -1, 0 ], an unknown 'Rotation' (4 floats?), an unknown uint16 (probably a flag of some kind), an unknown UInt32 = 0, and 2 unknown UInt32's specific to Kotor2:TSL.
• Unknown Struct in MDX: there is an undeciphered structure in the MDX files pointed to in the common mesh header.
• TO BE CONTINUED...

Methodology:

I have been developing a number of personal use dev tools for following in the footsteps of cchargin and MagnusII. One of the main necessities for reverse-engineering this stuff is being able to easily compare and analyze across the entire corpus of models present in both games. To this end, I have made a couple interesting things that I am happy to use to generate reports for people in the process of tracking down the meaning of some of these things... My core goal is to make it so that decompiling with MDLOps and recompiling will produce byte-for-byte identical results, or as close as reasonably possible. To this end, I have a test script that operates on all models, decompiling them, recompiling them, and then doing a byte-wise comparison of the resulting MDL and MDX files, measuring the error/difference (in actual bytes as well as as a ratio against the total bytes). So, when I casually said earlier that 'accuracy is up' in my MDLOps, it's actually verifiably true.

 

Of course, that being said ... I have only *just* fixed MDLOps enough to be able to decompile/recompile every model without going into infinite recursion or loops on a few problem children, so I don't have a *lot* of accuracy data yet. In case you're wondering, the numbers are approximate and the verification takes hours.  The best I could come up with for purely accurate verification would take weeks to run, the approximation is fine though because it is mainly being compared with itself as a means of knowing whether I'm moving in the right direction or not. I can use these tools on small numbers of models basically instantaneously, which I also do.

 

I have another tool that generates some statistics and compares (currently only) 2 model features. This is in my effort to understand Animation Controller 240. For instance, I can tell you that whatever 240 does, everywhere it appears as an animation controller, birthrate is also being animated. There is a comment in MDLOps.pm mentioning that the author thought it might be life expectancy, but now thinks lifeExp is 120. Whatever it is, it is related. I can also produce a text file with the controller 240 values (and their key animation values) for every model in the game, with comparison to the birthrate numbers. (Produce as in, I have this file already).

 

So, my medium term agenda here is to produce a 'debug mode' for MDLOps. What it is going to do is put *all* of the non-derived values into your ascii decompiled files, and then make sure that everything it puts into ascii files gets back into (debug mode) compiled files. This will make it a lot easier I think for people to certain modifications on large numbers of models, making it more likely that they can maybe figure out what these things are doing. Up to now I've just been making MDLOps able to handle all the things that are actually known (haslightmap aka lightmapping, shininess, sparkle). A slight time-sink issue is that I am also modifying NWN toolset components to not mess up all the things I am adding, and that is more of a challenge.

 

So yeah, I hope that we can engage in a productive technically focused discussion here and maybe eventually shed some new light on aspects of the model format that have gone dark in recent years. I know some of you guys know a *lot* on this topic with your hex editing mad(awesome)ness, and also I know that some of you have KAurora source access and I hope that you'll pass that along via PM so I can flesh this out (I'm not about posting it ... or even working on it (probably), but I would love to save the knowledge represented therein).

[xander2077 80]

Awesomeness. Purifier and i are also coming at this from other angles as well. Im currently slogging throung comparing samples of NWN1 models for each category to their equivalents in Kotor. Once i produce a list then i will peobably have to pass that on and rely on someone else for comparison of the TSL files since there is a slight difference. I dont have any extracted from my copy of TSL.

[DarthParametric 3,790]

until I can access KAurora's source, if someone can get me that, this list will improve a lot.

Fair Strides is the KAurora source keymaster. I'm not sure whether you'll be able to get access to it though. Presumably Magnus would have open sourced it from the get-go (or at least after dropping it) if he intended to grant access to all and sundry.

[VarsityPuppet 1,085]

Good to hear this is being looked at. Not sure I would be much help, but Fair Strides and I recently did things with vertex normals, so we might be able to contribute.

[Fair Strides 509]

*sigh* I'm not trying to sound cruel or ungrateful or anything like that, but I've kind of been holding off on talking about any of this stuff or dealing with it because in all of the two pages of discussion in the MDLOps thread and all the weeks since that started, no one thought to contact me and ask for any input or help. My name's at the top of the thread and the first post, and several times in the discussion you guys mentioned getting help and contacting people, but I'm not sure if that ever happened.

 

@ndix UR: I am very grateful for your most recent post in the MDLOps thread and that does help a lot. I'd be interested in further contact and I can work up some tables for the formats at a later date.

 

Most of the time, I juggle between projects, so I can't guarantee on when.

[xander2077 80]

@ fair strides and @ DP: we all kind of figured FS was the keymaster for the code, but with him being busy lately we didnt want to burden him with a lot of things. So each of us has been studying dutifully and hopeing that once FS has some time we can go over all of it and compare notes to improve things. being that a lot of us are Blender users, and Blender handles it differently, as well as neverblender, this is a passionate subject for all of us. Not to bypass anyone, just that things are a bit different from this side than the Max side of things. So while we may be going over things already known or learned, it is still best to learn in spite of that to bring a better understanding to the table so FS wont feel like he is repeating things over and over. Or less so because we all have a better understanding before going onto the discussion.

 

I think this is going to help expand Kotor modding to a new pinnacle, and even help us figure out some things that have been brick walls for a while. Who knows, we may even be able to figure out how to add the correct lines to an mdl so it can do things it could not before with textures. One can dream.

[ndix UR 218]

@DP Thanks for clarifying the chain of custody on KAurora for me, it is actually a little difficult to piece together. I found redrob41 taking over stewardship in 2008 (it looked like), the current upload is by Sithspecter, and the description page mentions "The source code will be released with a later version along with the necessary license." which raises questions about whether the closed source status is because of a ) authorial intent or b ) not wanting to figure out licensing restrictions. I personally have not open sourced things just to not have to deal with it, so ... yeah. Thanks again. If it were Magnus' intent to have someone else do the format research in the open, I absolutely would/will respect that and do it myself. I did not mean to make it seem assumed that I would be granted such access, which, admittedly in the quote you picked out, it kind of did  .

 

My basic position is just that the original MDL/MDX format tables from 2003?4? are what some people are still working from, they have issues (that some people have solved), and I would like a better source of information to be put into public view if possible. If it were to require me to do clean-roomed research, I would go about it and contribute what I could. It would just be less, and slower.

 

@Fair Strides Thanks for jumping in, I will PM you.

 

Edit: Eugh ... Just found and reread the change of stewardship of KAurora post that I had found earlier. My memory of it was poor, sorry. It was 2014 and much clearer than I remembered. Never try and type something w/o looking directly at what you're talking about...

[xander2077 80]

I have been reading up a lot on this and even delving into the format a bit, trying to comprehend it. And I think I am getting closer to understanding some of the issues that still plague us from the modeling side.

 

An idea that popped into my head today was a way to handle the emitter problem and some of the other issues that just can’t reliably be fixed by using a text editor on the ASCII or a hex editor on the binary.

 

One idea I had was an additional tool separate from MDLOPS that can steal the emitter data from a donor model, and inject the proper lines into the model by selecting a drop down, and this could be effective for a lot of the properties that we know about, but cant reliably edit into the mesh by text or hex editing methods. Why dropdowns? Well it eliminates errors in typing and limits us to the correct variables for emitters, so no matter what emitter type it is, it will only show the options available to us based on what the game already has available. There are other variables this could be expanded on as well, such as animations, supermodel and other things like particle texture. It would be kind of cool to see the ability to do this. Especially putting Bastila's animations on HK 47 or something like that on the fly.

 

I’m not sure if that is even an option really, but it seems like an interactive tool that checks the model with these template variables, that is more in depth, similar to the tools we use to edit the 2da or other files that have fields we can input data into or use a dropdown to select the proper variable, might be something worth looking into. the reason I thought it might be possible is because there are certain things you can and can’t do with regards to running programs, or editing INI files or whatever, I’m sure people have experienced it, where an exe or a batch has to do it for you, or the edit fails. try as you might you follow the instructions edit that registry entry or whatever and it fails and then you find a batch file or a REG file and it works. now of course I’m speaking from the Linux perspective, where this is an issue repeatedly, where I have attempted to edit files that I can’t edit, and I have to compile things by using "make", that simply copy pasting won’t work with, or opening it in text editor will not work, or it screws the format up somehow no matter how correct the syntax is... and the only way for things to work is by using a method that operates outside of text editing... and this might have to be one of those instances where typing it in has to be eliminated. Not due to syntax, but due to it having to be injected into the model by computer rather than by human. I may not be explaining it right but maybe you guys can understand what I’m getting at.

 

now im no tool writer, but as a joke i suggest we call it xanders mdl spell checker to poke fun at my lack of typing skills. or not.

[Blue 68]

It is possible to decompile the Kaurora.exe with dotPeek from JetBrains though I'm not sure that the Kaurora in the download section of DeadlyStream is built with the latest version of the source code. I've been working with the MDL format myself over the past couple of months and will be sure to share anything I discover as well.

 

Don't forget to talk to the guys who are working on xoreos if you haven't already. They may have some insight and might benefit from anything we discover.

[ndix UR 218]

Thanks for the advice Blue, but line 3 of the download page does mention "Please do not disassemble, reverse engineer, copy, distribute etc. the program." So I won't really be going that route. It would also be a bit difficult for me to understand since I can't even run KAurora in the first place   Also it would be reverse engineering a reverse engineer job ... kinda like cloning a clone.  I have looked at xoreos (the code mostly, but also some helpful stuff in their dev blog, and of course the docs folder in their github repo) but haven't actually talked to anyone working on it at this point in time.

 

In general I'm making pretty good progress in learning and discerning things. (Not even really intentionally, just ... finding things while looking for and attempting other things).  I am in the midst of writing up something more complete, but here is a first update related to some of the unknowns I spelled out in my first post.

 

Animation Controller 240: this is a RandomBirthRate, a float value I suspect. I don't know exactly how it functions, but it has similar values to the birthRate (which is particles emitted per unknown time unit). I suppose the two options that occur to me for how it might work are:

1. when it is set, it defines a range between the normal birthRate and the randomBirthRate. there are a number of places where it is 2 - 5 apart (above AND below) from birthrate, which in this scenario would mean that the effective birthrate was somewhere between the two at any given time. the problem with this scenario is that randomBirthRate would be meaningless w/o birthRate, and the few places where they are set to the same value would be not doing anything.
2. consider birthing new particles at the given frequency. in this scenario, randomBirthRate is basically additive over the basic birthRate. So, if birthRate == randomBirthRate, the effective birthrate would be somewhere between birthRate and 2*birthRate, depending on the random effect.

Unknowns in Common Trimesh Header: I've actually learned a lot of things here, but for now I'm going to keep it simple and just talk about the 'short' (aka uint16) values starting at offset 308 and going to offset 316. Rather than 4 shorts, this is actually 8 (unsigned) bytes. The reason the original interpretation was 'good enough' has to do w/ the little endian byte order. My first clue was the 'value of 256 = cast shadow' ... in a big endian short, in binary, 256 looks like 0000 0001 0000 0000. Translate that to little endian (which is how everything in kotor binary files is), and you get 0000 0000 0000 0001. This is the same as a 0 byte at 310 and a 1 byte at 311. Which, incidentally, I believe to be the correct interpretation. Here's how it breaks down AFAICT: 308 - lightmapped 0|1, 309 - rotatetexture 0|1 (i assume this enables the "Rotation" numbers after offset 316 but haven't tried it), 310 - needs study, 311 - shadow 0|1, 312 - beaming 0|1 (made tree light beams in nwn i think? not sure if it makes an appearance in k1/2), 313 - render 0|1, 314-316 is still unknown maybe padding or something.

 

I haven't yet had time to figure out how many models each byte is set in (and whether any are more than just 0|1 values).

 

Anyway, just a couple notes from the front lines. I am kind of hoping that some intrepid modder will take some of these bits of knowledge into their hex editor and maybe help confirm or deny some of my claims, if anyone tries it, good hunting (and thank you)!

 

Also, just, a little methodology note. The converse of what I mentioned above about the '256 value in a short that is actually bytes' thing is maybe worth spelling out because it is kind of weird. Suppose you are putting a value of 1 as a short (or uint16 ... 2 bytes) at offset 308. In terms of individual bytes, that will put a 1 at 308 and a 0 at 309. In fact, even if you were putting a value of 1 as a long (or uint32) at offset 308, it would still be a 1 in byte 308, with 0's at 309, 310, and 311.  This is not news to people who've been hexediting on this stuff for a long time ... buuuut, on a side note, I now hate little endian byte order more than anything in this world 

[Blue 68]

Thanks for the advice Blue, but line 3 of the download page does mention "Please do not disassemble, reverse engineer, copy, distribute etc. the program." So I won't really be going that route. 

 

Good to know. I have to admit I didn't read the fine print. 

 

 

Anyway, just a couple notes from the front lines. I am kind of hoping that some intrepid modder will take some of these bits of knowledge into their hex editor and maybe help confirm or deny some of my claims, if anyone tries it, good hunting (and thank you)!

 

 

I'll try to give it a go with my model viewer i'm working on. Not sure when it will come up but i'll be sure to report back if i find anything useful.

 

Edit*

 

 

 

I have looked at xoreos (the code mostly, but also some helpful stuff in their dev blog, and of course the docs folder in their github repo) but haven't actually talked to anyone working on it at this point in time.

 

I've talked to DrMcCoy a little. He keeps an eye on the forums over here so he's a good point of contact if you need one.

[DarthParametric 3,790]

I tried playing around with some trimesh options and comparing the compiled results. Test model was a simple cube trimesh (8 verts, 12 faces, 24 tverts) and an AuroraBase. Here's what I found.

 

"scale" - Float. Default is 1.0. Changing it yielded no result. Compiled models were identical regardless.

"transparencyhint" - Int. Default is 0. Changing it yielded no result. Compiled models were identical regardless.

"alpha" - Float. Default is 1.0. 2 Bytes. http://dpimages.wheb.org/KOTOR_MDL_Test_01_Alpha.jpg

"render" - Int., 0 or 1. Default is 1. 1 Byte (?). 1 = 256. ID'd in CChargin's data table (common mesh header, although # is different) http://dpimages.wheb.org/KOTOR_MDL_Test_02_Render.jpg

"shadow" - Int., 0 or 1. Default is 0. 1 Byte (?).. 1 = 256. ID'd in CChargin's data table (common mesh header, although # is different) http://dpimages.wheb.org/KOTOR_MDL_Test_03_Shadow.jpg

"rotatetexture" - Int., 0 or 1. Default is 0. Changing it yielded no result. Compiled models were identical regardless.

"beaming" - Int., 0 or 1. Default is 0. Changing it yielded no result. Compiled models were identical regardless.

"inheritcolor" - Int., 0 or 1. Default is 0. Changing it yielded no result. Compiled models were identical regardless.

"tilefade" - Int. Appears it has 4 set values. 0 for "not a cap", 1 for "fadable, 2 for "base", 4 for "neighbour". Default is 0. Changing it yielded no result. Compiled models were identical regardless.

 

I assume all those values that did nothing are redundant Aurora options that NWMax spits out. Or at the very least MDLOps just ignores them.

[DrMcCoy 40]

Just FYI, xoreos main dev here. Hej hej.

 

Yes, I do keep an eye on the forums. Especially when "xoreos" is mentioned, I'll probably see it eventually. I'd appreciate an explict poking when you're talking about xoreos, though, so that I can jump in and answer questions, clear up misunderstandings, etc.

 

1) There is no 'official' text format for Kotor models

 

There is. I've looked at the disassembled original EXEs of both KotOR and KotOR2 using IDA, and they both have a parser for ASCII models, filling out structures resembling the binary model format, just like NWN does.

 

The meshes themselves are more complicated (because that gets parsed into IBO/VBO structures), but the top-most model header and the node headers get parsed into a struct that's 1:1 what's in the binary model.

 

That's, for example, how I found the transparencyhint field. Which, unfortunately, like in NWN, isn't always set when it should be, but that's BioWare for you.

 

I haven't really mapped out all of it yet, though. And yes, the controllers stuff is also more difficult to follow in the disassembly, unfortunately.

 

Of course, since I'm going to need it in the future, I'd be very interested in more details and informations about the format.

 

So far, I've only done the bare minimum of what's necessary to get xoreos to display the basics.

 

Also, IIRC, Farmboy0 has started crawling through the binary format using 010 Editor (I don't know how far, though). That sounds like a quite useful tool in general. In combination with following the ASCII model loader in the disassembly, that's probably the best way to map out the format completely, maybe.

[xander2077 80]

That is awesome Dr McCoy! Keeping eyes peeled! What exactly is planned on xoreos end with regards to model viewers, format specific editors or anything else.

[DrMcCoy 40]

I wouldn't really say anything is planned in that direction. The main goal of xoreos is, of course, making the games themselves playable.
 
There is, however, Phaethon, a graphical resource explorer I started and worked on briefly as a side-thing while I was thinking through how to implement something else. Right now, it can look into the game archives, play sounds and display images/textures. It could, conceivably, in the future, also house model viewers and format editors. Maybe not editing images or even models, but editing GFFs and 2DAs would probably work quite well. That's kinda how I envision a tool like this to work.

 

However, I have currently stopped working on it. If anybody is interested in taking up that task, please contact me.

 

Also, for GUI stuff, it uses wxWidgets, which...well, is a bit annoying, IMHO. It would probably be best if it could be rewritten using, I guess, QT. Or if anyone has another idea, one that still works in a portable manner (i.e. not something like MFC), talk to me.

 

And yes, I'm aware that I'd be replicating a lot of things already out there modding-wise. But this would be portable and FLOSS'd. I see worth there, but, of course, YMMV. Also, yes, there is a bit of a Not-Invented-Here-feeling involved there, I admit that. Thirdly, yes, this is all just talking the talk without walking the walk for now, "vapourware" if you want to be blunt; I'm also very aware of that as well.

[ndix UR 218]

@DrMcCoy thanks for your comments and your work on xoreos-tools. I would be lost without unkeybif! Also your xoreos dev blog and wiki are great resources that I would recommend to people. I arrived at IDA through the xoreos wiki linking to ScummVM's reverse engineering howto (prior to that I had used borg and/or straight vim/xxd. IDA is better.).
 

There is. I've looked at the disassembled original EXEs of both KotOR and KotOR2 using IDA, and they both have a parser for ASCII models, filling out structures resembling the binary model format, just like NWN does.

Thank you for this clarification. 

 

Yes, I too have noticed the same thing regarding the presence of an ASCII model parsing routine. There definitely is an 'official' Kotor ascii model dialect of which I was wholly unaware.  A problem now is how much of the 'official' kotor ASCII naming can be adopted while maintaining compatibility with the NWN-centric tools like NWMax etc. that people rely on. A good example is the 'lightsaber' mesh type ... among many others. I have been using the the code of NWMax, Neverblender, NWN-Tools etc to do string and formatting checks as well as fill in the picture of what things actually do in-game.

In terms of progress, I've got a lot of bits mapped out, especially things completely missing from the cchargin mdl info table. Lights, emitters, and mesh (w/ a large number (probably all?) of the controllers). Enough that I'm starting to get to the hard parts. Like the random pointed-to data segments ... and how bezierkeys are differentiated from linear in the binary (even ascii, i know that they are 'controllernamebezierkey' instead of 'controllername key', but do they have any extra data? knowing about bezier easing and whatnot I would expect so ... just not sure) model ... among other things. Basically I am double checking things, writing things up, and trying to verify as much as I can ... when I'm not working on other projects.

And, a major question I have is this: from that parsing routine, it is easy to tell that the game engine is stashing the data into memory structures, but, do you have a sense of whether it actually uses everything that it stashes? For a random example, it knows about the 'beaming' setting, but does the engine actually use that information at some point? Especially because a few of the bits (not many) don't seem to be used in any models in either game. If you have an impression on that, I would love to hear, because I don't exactly want to attempt to verify a bunch of stuff that the engine is likely to not even support anyway

Incidentally, figuring out the lightsaber mesh implementation is a pretty substantial challenge it seems like ... and the final 3 chars of controller headers are driving me a little mad.

@DarthParametric, thanks for your research and screenshots. The only one of those bits/properties that I don't think has any kotor equivalent is tilefade. I can tell you with certainty that the currently released mdlops does not use transparencyhint or beaming in compiled models (it seems that the game can know about them on some level though... even when I figured out where beaming should be and hexedited it into models I couldn't get a test where any change was apparent. unfortunately my in-game testing skills are nowhere near good enough yet to be able to make conclusions based on their results). Also scale is more than 99% implemented in the current mdlops compiler but not really 'enabled' sort of. It is easy to do silly things w/ scale ... quite a bit harder to do anything that looks really good and works 100%.

[DarthParametric 3,790]

Ah, so it is more of a MDLOps issue then, unsurprisingly.

 

Scale could be useful for placeables, given they are static. I'd suspect it wouldn't be so great for animated meshes like characters (I'd have though Bioware would have just used that instead of having small/medium/large bodies if it worked).

[Fair Strides 509]

Just want to clarify something for those that aren't aware of it: when ndix UR is referring to scale in the models, there are two different scales.

 

The first is simply "scale" and is in the mesh headers. Honestly not sure if the game cares about it's value in the mesh header when not animating.

 

The second is an animation controller and is called "scalekeys" in the ascii. This allows you to animate the "scale" entry of the mesh header to change over time. This only place this is used as far as I know is in lightsabers.

 

The lightsaber blades themselves are made up of four segments in a single-blade lightsaber. When the lightsaber powers on or off, the scale is animated on the lightsaber blade pieces to change from 0 to 1 and vice-versa.

 

The only reason MDLOps can't natively seem to do lightsabers is that each animation controller has three pieces of data at the end that seem to be calculated somehow. For Orientation and Position, MDLOps just shoves three values in there with no calculations or anything and those work. But Scale is different. Reading in a single and a double lightsaber and printing out the last three numbers for each of the scale keys of each animation and the numbers are all different every time.

 

I could simply write those numbers down and "hardcode" those in by checking the name of the current animtation, but then you have to consider whether that's the right solution or just the hacky solution (even if "hacky" is all we need right now).

[xander2077 80]

Hacky might be the way to go, that's one of those values drop downs would be great for, but starting off with the right structures might be something for neverblender, and keeping it that way would be the job of mdlops.

[ndix UR 218]

Just want to clarify something for those that aren't aware of it: when ndix UR is referring to scale in the models, there are two different scales.

 

The first is simply "scale" and is in the mesh headers. Honestly not sure if the game cares about it's value in the mesh header when not animating.

 

@FairStrides, When I sat down to work on "scale" my understanding was exactly what you wrote above.  Which is part of why I thought it was going to be so easy.  However, the more I looked for where to put the scale value in the headers ... the more I started to realize it's not there.  The only "scale" value I was able to find in headers is the "animationscale" field in the Model header.

 

 

The second is an animation controller and is called "scalekeys" in the ascii. This allows you to animate the "scale" entry of the mesh header to change over time. This only place this is used as far as I know is in lightsabers.

 

So yeah, as far as I know, this controller is the only implementation of "scale" or "scaling" of any kind. It can, however, be used just like a lot of other controllers in any of 3 different ways: 'single value', 'keyed', or 'bezier keyed'. The single value usage is the same as how alpha and selfillumcolor work. It's sort of the same as how position and orientation work, but those actually are copied to the mesh header for whatever reason (maybe because they are required, whereas scale 1 can be default). Basically when you connect "scale" in the ascii model to a single-value controller entry, you get working per-mesh scale (per-node actually, because it is available on all node types).

 

Incidentally, fx_part_04 seems to be the only non-lightsaber TSL model that actually *uses* this feature (single-value scale controller for 'initial' scale). It has scale 2.5.  The sabers also use it to start their scale at 0.0 (and yeah, then they have animations using the scale controller in keyed mode to scale up and down like you describe). fx_part_04 also animates the scale. The following other TSL models animate scale but start at scale 1.0: fx_koltobub, plc_starmap, v_grnadhs_imp.

 

 

The lightsaber blades themselves are made up of four segments in a single-blade lightsaber. When the lightsaber powers on or off, the scale is animated on the lightsaber blade pieces to change from 0 to 1 and vice-versa.

 

The only reason MDLOps can't natively seem to do lightsabers is that each animation controller has three pieces of data at the end that seem to be calculated somehow. For Orientation and Position, MDLOps just shoves three values in there with no calculations or anything and those work. But Scale is different. Reading in a single and a double lightsaber and printing out the last three numbers for each of the scale keys of each animation and the numbers are all different every time.

 

 

Yeah ... sigh ... the numbers almost-but-not-really-at-all make sense I am going through the same thing with those numbers for emitters right now...

 

 

@DarthParametric ... that actually does sound like a really good use for it. The characters just get weird ... even though I have had fun laughing at a tiny-hands atton.

 

 

Also, speaking of controllers, I "figured out" the thing with how bezier keys are represented in binary, and there are actually some uses of them in K1 (haven't checked TSL yet). I say "figured out" because what I actually figured out was to RTFTorlack, and it is in there, and it is actually the same in kotor (it seems so far).

[DrMcCoy 40]

do you have a sense of whether it actually uses everything that it stashes?

No. This is, unfortunately, very hard to trace and verify just by reading the disassembly. You basically need to find every place the structure is accessed and check if it reads from the relevant offset. If they stuff the pointer into other places too, it's even harder.

 

If you can run the game in a debugger, either in IDA's, winedbg, gdb or what-have-you, you might be break on model load and watch memory access, I guess. But that's rather fiddly in general. You also can't say 100% for sure that the data is never accessed, only that it's not accessed in the render paths on your system, for the model types you looked at. They might be used on some special models or on different hardware (though the render paths for old, older hardware are probably not relevant anymore anyway).

 

The only reason MDLOps can't natively seem to do lightsabers is that each animation controller has three pieces of data at the end that seem to be calculated somehow.

Ah, so no one has actually figured out how those work and you're just faking them? Interesting to know.

 

IIRC, Farmboy0 looked at those some time back, tried to figure that out just by staring at a few files in 010 Editor. I don't think he managed to crack them, though, unfortunately. The only thing I could contribute was "I have no damn clue".

 

Since xoreos doesn't yet support anything more complex than the simple whole-mesh animations NWN does for the most part, I didn't really need those data points yet. But this is something that'll be vital in the future.

[redrob41 86]

@DP Thanks for clarifying the chain of custody on KAurora for me, it is actually a little difficult to piece together. I found redrob41 taking over stewardship in 2008 (it looked like)

I don't know how you figured that, because I'm just a user of the tools, I don't know how to make them . I might try a little guerrilla style hex editing now and then, but it's all hit or miss, paired with blind luck. When I test things, it's always on the user side, and I hope that my experiments can help someone else (more qualified than me) on the programming side.

 

I wish you guys the best of luck , and I'll keep hoping that someday there will be a set of tools that allow me to make significant changes to models, while retaining smooth model seams, accurate shadows, and to correctly use bump maps (Curse you male Twi'lek head  )

[VarsityPuppet 1,085]

I don't know how you figured that, because I'm just a user of the tools, I don't know how to make them . I might try a little guerrilla style hex editing now and then, but it's all hit or miss, paired with blind luck. When I test things, it's always on the user side, and I hope that my experiments can help someone else (more qualified than me) on the programming side.

 

I wish you guys the best of luck , and I'll keep hoping that someday there will be a set of tools that allow me to make significant changes to models, while retaining smooth model seams, accurate shadows, and to correctly use bump maps (Curse you male Twi'lek head  )

 

We're pretty close. I know we have those problems solved individually. Not sure what's up with the shadows not showing though.. is that a side-effect of our mdlops vertex weight fix?

[redrob41 86]

We're pretty close. I know we have those problems solved individually. Not sure what's up with the shadows not showing though.. is that a side-effect of our mdlops vertex weight fix?

I know that with 0.7alpha2 you fixed the shadows that are cast on the ground, but I'm talking about how it treats some mesh edge seams as separate, but every uvw seam on the same mesh gets treated as smooth (even when you would want a hard edge). I'll refer back to my earlier posted images in post #26 http://deadlystream.com/forum/topic/4036-mdlops/?p=42484

1) If you look at the image of the male shirtless underwear, using Taina's replacer:

    - There is a ring around the top of the chest, this is due to the original mesh not being welded, so it is treated like a hard cut

    - Where the neck of the body meets the neck of the head, the shadow is distinctly showing as light falling on his right side, and shadow on his left side of his neck

    - Where the arm meets the shoulder, there is only one hard edge

2) using MDLOps 0.7 in that same image:

    - The ring is gone because I welded the verts

    - the neck of the head is shaded properly, but the body has a highlight on both sides of the neck. I think that this is because the body's "neck hole" has faces there, and they are being treated like a smooth edge instead of a hard one.

    - there is an extra hard edge below the shoulder (above the bicep) because this is where the arm mesh meets the torso mesh.

 

I know that there are several ways to work around this (like attaching all meshes into one, which requires re-weighting a ton of verts) or deleting the "neck hole" faces, but they are a lot of labor for just one model. Multiply that for EVERY model that I want to fix, and it would take ages. If the MDLOps shading issues could be ironed out, then I could fix the dancer model completely (add mesh faces for the inside of the skirt thing, move all the hooks to the proper locations, so that force powers come out of her hand rather than the air).

 

By the way, for the Mira underwear model, you can's see the neck highlight, because her black choker hides it. That's also how I tackled the bicep seams; I just put short sleeves on her shoulders to hide the hard edge.

[VarsityPuppet 1,085]

I know that with 0.7alpha2 you fixed the shadows that are cast on the ground, but I'm talking about how it treats some mesh edge seams as separate, but every uvw seam on the same mesh gets treated as smooth (even when you would want a hard edge). I'll refer back to my earlier posted images in post #26 http://deadlystream.com/forum/topic/4036-mdlops/?p=42484

1) If you look at the image of the male shirtless underwear, using Taina's replacer:

    - There is a ring around the top of the chest, this is due to the original mesh not being welded, so it is treated like a hard cut

    - Where the neck of the body meets the neck of the head, the shadow is distinctly showing as light falling on his right side, and shadow on his left side of his neck

    - Where the arm meets the shoulder, there is only one hard edge

2) using MDLOps 0.7 in that same image:

    - The ring is gone because I welded the verts

    - the neck of the head is shaded properly, but the body has a highlight on both sides of the neck. I think that this is because the body's "neck hole" has faces there, and they are being treated like a smooth edge instead of a hard one.

    - there is an extra hard edge below the shoulder (above the bicep) because this is where the arm mesh meets the torso mesh.

 

I know that there are several ways to work around this (like attaching all meshes into one, which requires re-weighting a ton of verts) or deleting the "neck hole" faces, but they are a lot of labor for just one model. Multiply that for EVERY model that I want to fix, and it would take ages. If the MDLOps shading issues could be ironed out, then I could fix the dancer model completely (add mesh faces for the inside of the skirt thing, move all the hooks to the proper locations, so that force powers come out of her hand rather than the air).

 

By the way, for the Mira underwear model, you can's see the neck highlight, because her black choker hides it. That's also how I tackled the bicep seams; I just put short sleeves on her shoulders to hide the hard edge.

 

Ah, yes that's right. Sorry, I forgot where we had left off with mdlops work. There are a couple of different algorithms for determining vertex normals, and I think we used the most basic one. There is a more complex one that will use hard edges if the angle between the planes is acute enough. Additionally, we could factor in smoothing group information when we determine the vertex normals. That should provide better and more configurable results