Area Research

[JCarter426 1,214]

I've started experimenting with editing and creating new areas, so I thought I'd take advantage of our new forum here to document my tinkerings. Areas have been the most problematic matters for KOTOR modding since the early days, mainly because the modeling tools couldn't handle vital things like lights and animations. Also, different tools were necessary for different procedures, and they each had things they couldn't do, and would break different things. Fortunately, recent developments courtesy of @bead-v have sorted that; KOTORMax and MDLEdit can handle any type of model and can successfully import and export everything necessary for area models. But that was only recently, and as such I haven't dabbled much in areas because I thought it was too much of a hassle before. I've always been interested in it, though - I first got into modding making maps for Jedi Academy, and I was thoroughly disappointed when I learned things weren't so simple with KOTOR. I wasn't the only one who felt that way either. Even after making new areas was technically possible, few did. And with the new tools, the limits have yet to be tested. I've decided to do some of that testing.

I'm not going into this completely blind - so far I've been able to half-remember several discoveries made by my predecessors, and I've made a few of my own already. The idea of this thread is to document things a little better as I go through them so nobody in the future will have to half-remember anything. And anyone else experimenting with areas is welcome to join in, of course.

I think we'll get started with a little show and tell:

Spoiler

I've set up a new area as a proof of concept, and I've attached the files below. There's not much of a point to it now, beyond letting you take a look at some empty areas, though I expect I might end up breaking things and require troubleshooting in the future, so here's a working version of it as a baseline I guess. For now you can look at it, and I've included everything so you should be able to fiddle with it for personal experimentation if you want, though obviously don't put it in a mod because go get your own.

Anyway, I've created a new area for Citadel Station by rearranging various rooms from three different areas and making some other adjustments. I took the hallway from 204TEL, doubled the length, added doorways at either end, and changed what each door leads to. Most of the side rooms are copies of the apartment complexes from 203TEL, with a few rooms from the Ithorian compound at the end of the hall. I also changed the exit, as I decided to use the existing one as a sort of lobby area, and instead added a doorway to the wall. I pulled the new exit hallway from 208TEL. I made some other edits like putting in new signs and made some changes to the lightmaps so everything blended together better. But for the most part I just pieced together existing models in a new layout. I'd done this before, mostly by editing layout files in Notepad, but I figured more extensive changes would be possible with the new tools, so I wanted to try it out.

I was impressed by how easy it was. I did the basic layout with all the rearranging and had a working area in just one evening. It took a bit longer to make the other edits, especially dealing with the lightmaps, and to debug a few things. Though, even a lot of what I didn't know was intuitive enough, like making the new walkmesh. This isn't a brand new area from scratch, but I think it is distinct enough compared to the other Citadel areas - all the residential ones at least have the same architecture, with copies of the same basic rooms but different variations on them. Similar, but not the same. And it was the work of a few days by someone who didn't know what they were doing yet. Pretty easy.

Now for some technical details. There were a few things that gave me trouble and between half-remembering things and experimenting, I've come up with solutions for them and perhaps more importantly a guess for why the game is the way it is.

Animate UV

This is one of the properties on the OdysseyTrimesh modifier in KOTORMax. It's a form of animated texture, but rather than have an animation on the texture file, the game animates the texture mapping. 204TEL has two instances of this: the water fountain and the sign to the Ithorian compound. During my early edits, they wouldn't animate. I was able to conjure up a memory of reading something elsewhere on Deadly Stream and realize the reason for this is that objects with animated UVs cannot be linked directly to the model base. They have to be linked to a dummy that's linked to the base. The dummies in the game seem to be named with the room name + "a" (e.g. trimesh linked to dummy 212tel03a linked to room 212tel03) though I don't know if the name matters.

Alpha Shenanigans

Spoiler

The parts I've circled are rendering on top of the window rather than behind it. It stands out more if you see it in 3D space, but in the picture you can tell by how they're dark brown when they should be tinted green due to the glass of the window. It's happening because both the window and the offending objects - the skyline, the antennae - make use of alpha channels to have part of the texture transparent. The game doesn't render overlapping alpha channels correctly. The game will always pick one object to render on top of the other object(s) regardless of how it's actually supposed to be. This is actually a known bug in OpenGL, which makes it like the hard-codedest of hard-coded problems.

Let's say you have a room with two windows, Window A on the west side of the room and Window B on the east. If you're outside the room on the west side, Window A should appear in front of Window B; if you're outside of the room on the east side, Window B should appear in front of Window A. But the game will always render A on top of B regardless of where you are. You can trick it to render B on top of A if you prefer, because the game determines the rendering order in a very specific way. The object that is rendered most recently is always rendered on top. Normally, the way to do this is to edit the hierarchy of a model's objects - put the one you want rendered on top at the bottom, so it's most recent. The easiest way to do this is to unlink all the transparent objects, then link them in the desired order. But that wasn't applicable here because the rooms are on different models. Yet it wasn't a problem on the original game models, so it had to be something I was doing differently.

I tried fiddling with the layout file, the visibility file, even the module's .are, and none worked. It took me a while to figure it out, but the answer again was that transparent objects should not be linked directly to the model base. Like the animated UV issue, they should be linked to a dummy that's linked to the base. I'm not exactly sure how this fixes it, but I'm guessing the game renders all objects linked to all the room bases first before rendering things linked to things rendered to the base. There is some sort of delay, in any case, that makes the window rendered after the skybox room, which is what we want.

Lightmaps and Lack Thereof

I had another problems with windows when I first started working on my area - they were totally invisible. It didn't take me long to guess this was because they weren't lightmapped; any part of an area model that lacks a lightmap or self illumination is rendered totally black. Since windows use an additive blending mode, and black adds a value of 0, my windows were appearing completely transparent rather than the intended partially transparent. It didn't take me long to guess that, though I had to deal with the other two problems before I figured out the real reason this was happening. Objects that don't have a lightmap or self illumination should not be linked directly to the model base. It was the same damn thing a third time. Once again, they should be linked to a dummy that's linked to the base.

To be fair, all the areas I was editing were set up that way. However, I didn't notice because I had to create new rooms to get my new walkmesh to behave, so I had unlinked everything from the start and then linked everything to the new bases. I'm actually glad I overlooked this, though, as I don't think I would've figured out as much as I did if I hadn't broken three things. My guess is that objects linked directly to a room base behave in a certain way, and can only do certain things. Linking them to another dummy is a way of telling the game "don't treat these objects like area model objects" and you if you do that, you can do different things with them. It's already affected three things, so this might have applications in other areas.

Smoothing

This is kind of a no-brainer - I'm used to having to do my own smoothing for character models, but after all the other troubles I had, when I ran into what turned out to be a smoothing issue, I expected it to be a bigger problem. Generally, area models aren't as complex as character models, and MDLEdit is pretty good figuring them out, but it did mess up one part of my area. So you should double-check your smoothing groups because the tools don't always calculate them correctly.

That's all for now. I'll be back as I poke at more things.

212TEL v1.zip

[DarthParametric 3,782]

2 hours ago, JCarter426 said:

Normally, the way to do this is to edit the hierarchy of a model's objects - put the one you want rendered on top at the bottom, so it's most recent.

It actually works in the opposite manner for mesh alpha. Anything at the top of the hierarchy will occlude everything below it. And per @ndix UR's thread here, alpha blending ignores hierarchy.

[JCarter426 1,214]

Hmm, that's just for mesh alpha though, right? I'm pretty sure for textures with alpha channels, it's the lower one that's rendered on top. That's how my Darksaber works, for example. Although, now that you mention it, I do remember it being backwards when I was working with alpha on the trimesh modifier instead.

I didn't know about the alpha blending, though. That's cool.

So, it seems like there may be more options for the window/skybox issue, though the weird linking to a dummy thing does work. I'm still curious which factor exactly is in play there, but perhaps we'll never know.

[DarthParametric 3,782]

I managed to find this in a PM, which I am sure @ndix UR won't mind me sharing publicly:

On 6/14/2018 at 11:09 AM, ndix UR said:

I've recently done a bit of new learning on the subject of 'placing things under the a-suffixed dummy in area models', so I thought I'd pass it along since I recall you were keenly interested in it

So, the crux is that nodes placed under the 'a' node are considered 'Dynamic', and everything else in a level is 'Static'. From this point forward, I will use Dynamic & Static to mean 'under a' or 'not under a' respectively and specifically. So, the biggest reason things MUST be Dynamic is if they have animated controllers. So, for example, if you have animloop1 on your area model and no 'a' node... I can only imagine the game gets upset. I haven't proven to myself that the Dynamic requirement holds for animated lights yet, but I'm pretty convinced it does for emitters and trimeshes. It is interesting to note that you can have emitters that are Static, and they still visibly change, it's just that how they change doesn't change. The second big important thing you use Dynamic & Static for is to arrange your area lighting. I don't know what KMax calls the nDynamicType/isDynamic value, or how it denotes the 0/1/2 values. I believe it just gives you the raw integer access. I've recently figured out how to correctly describe these values. 0 is 'Static Only', 1 is 'Static and Dynamic', and 2 is 'Dynamic Only'. So a lot of vanilla area models use fully lightmapped areas in conjunction with exclusively type 2 'Dynamic Only' lighting. When you add Static, non-lightmapped meshes to area models like those, they will be fully black.

I've also seen semi-translucent meshes made Dynamic, but I'm not sure if this is just to take them out of potentially being lit up (an effect that would be like reflective glare).

On 6/14/2018 at 5:28 PM, ndix UR said:

It seems like the node *must* be named ROOTNAMEa ... no more, no less. I tried other things ending in 'a', and adding stuff after. I also looked through every model. There is only one model that contains an animation without 'a'. In 152har33, the room w/ sabotaged protocol droid, there is supposed to be a selfillumcolor animation on the 4 wall sconce light things, but it does not function because their 'a' node got sloppily renamed 152har33a02. Another one of those minor things you can fix ... package it up w/ the escape pod room fix? Anyway, as far as the vanilla models go, I only checked animations. For Static/Dynamic pertaining to lights, I tested using an area I'm working on. Another thing I saw was that you can definitely nest under the 'a' node, things don't have to be direct children of it to be Dynamic/animatable.

Interestingly though, the nDynamicType thing did not work as I expected. Nor did the Dynamic/Static exactly. So, changing my test mesh from Static to Dynamic (putting it under the 'a' mesh), made the 'ambient' light level apply to the object, but it did *not* make my nDynamicType 2 lights affect it. So apparently nDynamicType 2 is more like "Do not affect level geometry". Changing 'Affect Dynamic' on the light had no effect (under any case, nor did adding 'background geometry' to the mesh I was testing with). When I change nDynamicType to 1, things get interesting. So, changing that to 1, the lamp lights the Static & Dynamic geometry of the area. However, for the Dynamic mesh, when I run away (making the priority 3 light source 'turn off'), the Dynamic mesh becomes unlit and returns to ambient light level, while all the Static meshes around it stay lit. So, Static meshes use an initial, unchanging light calculation (from area lighting), while Dynamic meshes recalculate. I said "(from area lighting)" there, because transient light sources (the flash from the clashing of swords) still affects the Static mesh.

Interestingly, if you animate an nDynamicType 1 light, it *does* change the lighting on Static meshes. And animating a light only works if it is under the 'a' node.

[JCarter426 1,214]

Ah, that's interesting. I figured dummies might be necessary for animations, as I noticed the shuttles are linked to them. Also figured it would want the -a specifically because if linking to any random dummy would change how the objects behave, then you'd have an issue if you wanted to link use dummies just for organizing and placing objects. So by the sound of it, the -a dummy is a sort of room within the room that behaves more like a placeable than a room. Though they still have some area characteristics, as objects linked to them can be lightmapped. And if it's used for dynamic objects, it would make sense that it's rendered at a different point than the static rooms, which would explain why it fixes the window issue.

[bead-v 251]

24 minutes ago, JCarter426 said:

Though they still have some area characteristics, as objects linked to them can be lightmapped.

Any mesh may use bitmap2 as a lightmap, but I don't know what happens if you try to do it on a placeable or door or the like... But anyway, from the point of view of the format, being an area model is not a prerequisite for being lightmappable. Hell, there's isn't even a proper Classification number for area models.

[JCarter426 1,214]

I thought it worked too, though @DarthParametric once said he couldn't get it to.

[DarthParametric 3,782]

Not in K1 at any rate. But I abandoned further efforts after the placeable count limit came to light.

[LiliArch 115]

Why would you lightmap a placeable anyway? That would make it un-reusable, since lighting would be wrong in other locations.

[DarthParametric 3,782]

Some placeables are only used in one place, and that's especially likely if they are custom ones added for a mod. If would be a lot easier to add certain things via placeables rather than screwing with room models. But it's not particularly practical in K1 anyway given the placeable limits. In TSL you can apply tints on a per-instance level to sort of fake lightmapping.

[LiliArch 115]

What makes a single-use placeable easier than making a room model?

[JCarter426 1,214]

In the past, if you wanted to add something to an area model, you would have to add it as a new room to the layout and visibility files because directly editing area models wasn't a thing. So a lot of older mods used placeables for that reason.

Currently, placeables still have some advantages - like they can be scripted in, which is better for compatibility because there are many ways to script something in but usually only one model in a room to edit. Plus, if you want to give it an inventory or make it at all interactive, you're going to need to put in some sort of placeable, so it's easier to make a new one rather than add an invisible one on top of editing the area geometry. If you want your placeable to be animated, you may also run into some issues depending on which room you're in, because rooms can only play one animation at a time.

Also, the lighting isn't that big a deal for reuse. The game's lighting is not super complicated and realistic - how else would they be able to get away with tons and tons of placeables with no lighting at all? It's all situational, of course, but there are advantages to lightmapping that still apply. You can achieve the same result by baking the lighting onto your diffuse textures, but that depends on the texture mapping.

[JCarter426 1,214]

It's been an eventful week. I have more discoveries to report, on another case of me breaking something and then everybody putting it back together again. Only this time, it broke because I was doing things the right way! It was only when I screwed up that I got the results I wanted....

So, I've been porting areas for a while. During the MDLEdit beta testing, once editing areas became a thing, I got some of Dantooine ported to K2 quite nicely:

Spoiler

But for every attempt after that, something went wrong, like the latest one on Taris here:

Spoiler

Some of the rooms don't render. When I got to Taris, this was after a handful of failed modules like this, and I finally realized the rooms that aren't rendering are only those without any real walkmesh data (no aabb node on the model and an empty WOK file). Even though the layout and visibility files were set up correctly (they're the original ones from K1) and all the textures were there, and so on, these rooms would not render. On Taris, all the walkable parts of the area would show, but the skybox and background buildings would not.

I'd noticed before that K2 rooms have a tendency to include walkmeshes even when they shouldn't need them. Skyboxes were immediately obvious and suspect. You're not supposed to walk in the sky. Yet they were there, and though I didn't understand why, I tried adding a walkmesh to one of my problematic rooms on Taris and got it to render:

Spoiler

After deleting the unneeded WOK file but leaving the MDL & MDX intact, I was able to confirm that K2 does not actually require a WOK file. Also, copying the original (empty) WOK files from K1 did not help with the rendering. It seemed only the presence of the aabb node was needed to get it to show.

But the Dantooine skybox has no such node. The game was rendering it even without one. It couldn't be that K2 requires an aabb node for rooms. I had a working Dantooine to point to. I kept saying "This works! Why does this work?!"

Well, lots of thanks to @bead-v for troubleshooting, and also @ndix UR for providing some suggestions. It took days of thinking and testing to figure out, but now the answer is clear: room models without an aabb node might not be visible in K2. Because obviously it was that. Of course K2 has different requirements from K1 that nobody predicted.

The reason Dantooine was working for me was because when I set up the module, a year ago, I forgot to include the VIS file. Apparently, the lack of any visibility list forces the game to render every room all the time... even when it would have trouble rendering them otherwise. The counter-intuitive consequence of this was that when my module was set up incorrectly, everything that was supposed to be there would be there; when the module was set up correctly, stuff would fail to render. Naturally, I only forgot the VIS file the one time, and included it in every subsequent porting attempt, and therefore got unsatisfactory results. Every time I set things up correctly, the game would not behave, because apparently K2 has a higher standard for what qualifies as an area model than K1 does.

So the upshot is that areas ported from K1 to K2 will have problems because of rooms lacking aabb nodes (probably every skybox, and more). Also, all rooms created for K2 must meet the same requirements, or else they won't render; everything needs an aabb node even if it doesn't require a walkmesh. bead is working on an update to MDLEdit to address this issue - an option to add a dummy aabb node when needed. The specifics are still being debated because of a complication in detecting whether a model is an area model. Apparently, the only way to do that is to check for an aabb node or the presence of a WOK file, since those are things that only areas have. I suspect the complication in solving this problem may in fact be the cause of it - that at some point in development for whatever reason, Obsidian introduced an aabb node requirement because it was the only viable way to check if the model was an area model and should be rendered... and then that meant every model required them. Given that K1 rooms can lack aabb nodes (that's the whole problem) the only other option is to check for a WOK file. Therefore, areas should be compiled with their original WOK files present, when applicable - in the same way that character models require supermodels.

The part of this that I can't get over is I happened to misplace one file one time, and that mistake is what made KOTOR behave. If I hadn't screwed up, there wouldn't have been a working example for us to look at. We wouldn't have identified the problem without that, or at least we would've had a harder time of it. So the moral of the story is you can't look at how things work and use logic and reason to plan your modding course of action, because the game won't accept that. You have to approach KOTOR like a blunt instrument. Make mistakes that work.

[Thor110 494]

I have converted and ported 79 modules from K1 into K2 ( Take a look at the link in my signature )
I did not experience all the same graphical issues with certain maps like you did, Taris all seems to load just fine, however I was missing portions of Dantooine and Korriban does not convert properly through MDLEdit from ASCII to Binary it halts / gets stuck on kor_m38aa_03 roughly.

Tatooine all loads fine but I am missing most of Anchorhead, I have come to the conclusion for the time being that some assets, textures, skyboxes and other such items are either hidden in the swoop tracks / extra modules / cut-scenes or some information fails to convert properly, but I think the former is more likely because 95% of the modules I have converted load and play for the moment what seems to be me perfectly with no crashes.

Hopefully some of the information / research / work I put in can go towards helping you and others. ( have been awake for three days porting all those modules, only just slept last night ) don't know why it just kept happening and have some other troubles / issues in my personal life keeping me up at the moment.

With a bit more work I am hoping to completely load every single module in the game, cut-scenes, swoop tracks, cut modules and anything else included, also to be used as a modding resource / asset same as you intended to, though I started with a different goal I have decided to work on a clean port version for general / open use as well.

The issues I have left over are very minor and some are not even noticeable, there are only a few major issues, which seem to be due to a lack of textures I believe because I am missing the Jedi Academy on Dantooine which you are not ( and I had the buildings on Taris, which you did not ) which points me to me porting a file for Taris you did not and you porting a file for Dantooine I missed.

And I believe the same issue on Anchorhead is caused by similar missing files, so at the least I can compare my Taris modules to my Dantooine and Tatooine/Anchorhead module and figure out which file I am missing / vice-versa.

Due to it being such a lengthy process it is incredibly easy to miss files along the way as you will know.

Big thanks for the tips on using MDLEdit, it has helped greatly.

Thor110

I look forward to seeing if there will be an update to MDLEdit for the AABB node by bead and will be seeing how every single level reacts without the .vis file for the moment, I have every single module ported including wok's,vis's,lyt's,dwk's,pwk's,tpc's and lightmaps, pth and utd files built into the module and all entries for camera's, creature's, encounter's, list's, storelist's and triggerlist's have been removed for the moment as well as any entries pertaining to scripts in the are, git or ifo files. Such as onentry, onload, onaqitem and so on.

Did you have any luck with Korriban? for me it fails to batch convert on the last two modules and subsequently it seems every prior module fails to convert correctly, either that or I missed something, I will check if perhaps it is because of the conflict between K2 Korriban and K1 Korriban, though that could give similar results for Dantooine it doesn't appear to do so.

Edited December 31, 2018 by Thor110
Korriban

[Kexikus 995]

I've seen your research on texture alpha / semitransparent textures and mesh hierarchy but I was wondering if you or anyone else knows how semitransparent textures work within a single mesh.

Like in your example if window A and B would belong to the same mesh. In that case it's obviously not the way they're linked to the odysseybase but something else that determines the order of their rendering. I have yet to experiment with this myself but since I don't know when I'll have the chance to do so, I figured I'd take my chance and see if anyone else has come across this issue before.

I've noticed the issue in K1s Dune Sea area where there are backdrop planes for Anchorhead. Those all belong to the same mesh but the ones in front create holes in the ones in the back where they are semitransparent. Not sure if the vanilla textures have semitransparent pixels, but it's visible with custom textures.

I could of course split them to different meshes but that doesn't seem like an elegant solution.

[DarthParametric 3,782]

You might want to look at this if you haven't already - https://deadlystream.com/topic/5992-tpc-compressed-texture-transparency-alpha-blending/

[ebmar 893]

On 8/18/2018 at 7:32 AM, JCarter426 said:

Objects that don't have a lightmap or self illumination should not be linked directly to the model base.

Thank you for the information. I'm so lucky to stumbled upon this thread prior to my attempt that somehow related with. You just saved one's soul, hahah.

While here I also want to attach a visualization of this idea, for one to easily understand what actually happened --

On 8/18/2018 at 7:32 AM, JCarter426 said:

...any part of an area model that lacks a lightmap or self illumination is rendered totally black.

Spoiler