ndix UR

On 12/3/2020 at 9:30 AM, Jenko said:

There still seems to be an issue with batch processing many textures. I can get about 30 textures done before the red "compression failure" happens on almost every texture. No issue if I do them alone.

Version 4.0.0 has just been released. It uses an entirely new compression engine (as long as you select a compressor setting better than 'Super Fast') that definitely resolves this particular issue (and also produces higher visual quality outputs).

Previous attempts to fix this issue helped but were ultimately unsuccessful.

I would definitely recommend using a standard object serialization format for your textual representation. Either JSON or YAML. YAML is messier as a standard (far more complex parser and encoder), but more human-readable and can have comments. JSON is simple, but hand editing in a text editor is a little trickier because its parser is more rigid (also making it much easier to implement).

With either format, you wind up using a more complicated structure if you're modeling the actual GFF data. The format VP showed is nice for export/viewers/other contexts where reconstruction isn't necessary. Here's one way to do the typed version:

[ {
  type: "struct",
  value: {
    "DelayEntry": { value:0, type:"dword" },
    "DelayReply": { value:0, type:"dword" },
    "NumWords": { value:0, type:"dword" }
  }
} ]

An approach for retaining type while using the simplified/export version of the JSON might be to have a tool that reads a full GFF file and exports another json object with pathed type info. Like /DelayEntry: "dword", /EntryList/Text: "cexolocstring", etc. Then have the 'compiler' combine that info. Could even just run it on sample files for all types and put it into a structured type dictionary with path lists per type.

Personally I've been using GFF->JSON for awhile and it would certainly be my preference, but once you're creating the structured objects, yaml or json is a one line code change. Avoiding writing your own parser is a good thing. I'll just leave this here:

        cexolocstring Text -1
        	0 "This is a cexolocstring English male variant.
 I "hate" my life now."
        	1 "This is a cexolocstring English female variant.

Why did someone put a descriptive paragraph 1 \"in here\". "
        	2 "This is a cexolocstring French male variant.

\tIt is also a long string that contains an internal multiline string

\"Write your own parser, they said.

It'll be fun, they said.\""

Parsing that isn't my idea of fun. I guess the question and limiting factor here is whether 3DS gives you a native JSON/YAML(or any standard format) parser? Also whether it gives you base64 encoding/decoding (there's a "void" type in GFF which is just raw binary data)...being a proprietary language, your ability to extend and rely on existing libraries and language features is probably not so good?

IIRC, the WOK files have a specific byte or uint32 set (or unset, or something like that) when their mesh data is empty. Maybe K1 used that info to ignore whether a model has AABB, while K2 just ignores it and requires the AABB... I guess it's unlikely because the whole division between WOK & AABB node is a remnant of the 'client-server' architecture, with the WOK being used (largely) to establish & enforce positioning on the 'server' side and the AABB node used for the 'client' side.

9 minutes ago, JCarter426 said:

Mm, I've checked for that. I zoomed out with GLIntercept when the rooms should have been visible, and they're definitely not rendering at all. Even if the skybox had been offset by that much, I would think at least some of it would be visible given its size and the coordinates.

Cool, sounds like you're definitely aware of this. I wasn't really expecting it to be a factor here, mostly putting it out there as a general debugging tip. Using GLIntercept is another nice thing to add to that apparently.

18 hours ago, JCarter426 said:

I noticed that it seemed to be limited to rooms that don't have a walkmesh, like the skybox and some of the background buildings.

It seems like there's some Kotormax-relative terminology here. I use the older terminology where 'walkmesh'es are WOK files, and the walkmesh in-model are AABB nodes; bead-v folded them into one thing in the interface so that people couldn't get them out of sync w/ each other. In these terms, m14aa's skybox does have a walkmesh, it does not have an AABB node. When you're discussing issues around this stuff, it is very helpful to differentiate.

bead-v chose to collapse the walkmesh functionality around the AABB nodes, ignoring the WOK files as much as possible (except for the information that uniquely resides within them), so, for these models where the area doesn't have an AABB node, it seems highly likely that there would be some kind of breakage, and I would expect MDLEdit not to be able to produce a WOK file (during ASCII=>Binary compilation phase). Probably the fix for bead-v will be to auto-add an AABB node from the WOK file (during Binary=>ASCII decompile) if the model is lacking an AABB but a WOK is present (assuming he hasn't already had to add such a feature).

Unfortunately, in several places within the tools (particularly the editor components), the presence of the AABB node is used to classify area models. Meaning, within the model-processing toolchain, there's a decent chance that the WOK file wouldn't even be loaded in cases where there's no AABB node, because we don't even know that the model in question is an area. It's pretty annoying that there's no classification for area models... I'm sure kotorblender would also experience issues with area models lacking AABB nodes, based on the shared use of that AABB-node == area model classification method. There's a chance that mdlops would work where MDLEdit doesn't though because it uses an ASCII WOK file during compilation, while MDLEdit never looks at such a file.

18 hours ago, JCarter426 said:

I've noticed before that many rooms have walkmeshes when they shouldn't need to, like the skyboxes on Citadel Station. Yet I went back and checked m14aa and its skybox does not have a walkmesh, and my ported version of it did render. So I wanted to confirm whether the issue was K2 requiring walkmeshes when K1 didn't, or something MDLEdit was doing. I kept the compiled WOK file and reverted the MDL & MDX files for that room, and that turned it invisible again. I also tried removing the WOK while retaining the newer MDL & MDX files and the room was visible with that. So the fault seems to lie in MDLEdit. When it compiles a room that doesn't have a walkmesh, there's a chance that it won't render in the game.

In these cases, you have to (often with difficulty) determine whether something "didn't render" or whether it's location was just messed up. They often seem similar. The most common thing that gets messed up is that instead of using the LYT/WOK position, an area model gets placed at 0,0,0 (the second most common is that its location gets 'double' applied, so instead of being at 30,30,45, the model's origin will be at 60,60,90). So sometimes I will load up the LYT in Blender, and move the misbehaving model to 0,0,0 and figure out in there where I might be able to see the model in-game if that were the situation. Anyway, that's just a debugging tip.

I've heard before something related to area models without WOK files or AABB nodes (I can't remember which) not rendering in-game, which I assume is what you're experiencing. bead-v was involved in debugging that and only relayed it to me secondhand, so he'll probably have something to say about it when his time permits. I think the idea that K2 requires an AABB node is probably right (or currently prevailing understanding), and that K1 not requiring an AABB node might be something we were unaware of. There might be more to it though, which would be nice to know. Seems like an easy enough thing to test w/ a hexedit... I'll give it a go when I get a bit of time.

50 minutes ago, bead-v said:

https://github.com/DrMcCoy/NWNTools/blob/master/_NmcLib/NmcMesh.cpp

NmcGenerateBumpmapData()

Took me a second to find it

It seems to be tracking some Mirror boolean, could be what we need? I'll have to make the real dive into it another time though, way too tired right now..

Spent a fair bit of time looking at that today too I think the mirror boolean is just how it tracks the UV faces whose normal vector Z components go "down" or "into" the texture image. It definitely seems to use a simpler algorithm in general, which may or may not be accurate for our needs. I'm currently trying to make sure that the algorithm we use actually chooses a consistent tangent vector in the first place. Since there are numerous vectors that are tangent to the surface, and it's just a convention to choose one along an axis defined by the UV's.

5 hours ago, JCarter426 said:

Ah, ok. I did some testing and it didn't seem to make a difference whether the height map was there or whether it was all white. It really didn't like having an inverted height map on the alpha channel, however. Did some weird things to it.

Interesting... well... maybe I'm wrong. That's always an option. Often a good one. It's possible that the tests I ran were the wrong ones.

42 minutes ago, DarthParametric said:

So you're saying it doesn't ever make use of an alpha mask for transparency, even though there's at least one vanilla normal map with an alpha mask? Although the map in question appears to be unused in the actual game, at least judging by the TXI semantics of its parent diffuse.

There's at least one vanilla normal map w/ a non-white alpha channel? This is the first I am hearing of such a phenomenon. If the game actually uses the alpha channel on a normal map for ... reasons, that would definitely be interesting news.

1 hour ago, JCarter426 said:

Ah, so it actually doesn't make a difference if I save the height map on the alpha channel. Or are you saying it generates a solid white channel only if there isn't an alpha channel already?

It generates a solid white channel only if there isn't an alpha channel given in the input image. On the plus side, I've tested stuffing unrelated things into the alpha channel of normal maps and the game doesn't seem to care. I put ambient occlusion in there sometimes... so you should be fine storing your height maps in there.

1 hour ago, JCarter426 said:

Yeah, 1.0.0. That's the most recent version on the site?

So it is...

If I am reading the issue correctly, the wrist issue should be present on the default p_handmaidenba.mdl file when adding tangent space & normal map? Just in terms of being able to easily replicate the issue for debugging.

@bead-v interesting. I haven't done any world translation for the tangent space basis yet (IIRC). I don't actually expect this issue to be quite so directly similar. I'm thinking you need to do something more like rotating the UV map coordinates (per-island?) to match the coordinate system of the world in order for all tangent space bases to be compatible (whereas, right now, they are all mesh/UV island relative, so don't smooth together properly). How that works mathematically isn't something I know off the top of my head, that's just an initial idea.

On 8/3/2018 at 3:38 PM, eNoodles said:

Thanks a lot for the info! Could you link that documentation? 

Can be found several places, here's one:

https://github.com/kucik/nwn-docs

Just have to remember that they are for Aurora (NWN), and Odyssey (KotOR) is slightly different. I am unaware of a source for updated documentation.

On 8/5/2018 at 11:34 PM, DarthParametric said:

As I said above, ndix UR seems to have implemented an undocumented feature into TGA2TPC (or at least one I was unaware of) that converts an indexed colour image to RGB and normalizes it (i.e. generates Z data in the B channel). So that was why you got a functional map out of it. But it doesn't know what handedness the source image is.

This was essentially the first thing implemented, because when we first started, I had to rule out all the various indexed color modes as necessary or helpful in order to show that 32-bit RGBA is the one true way I don't think it does normalization though, possibly just R/B inversion. Some old implementation detail memories there...

On 8/5/2018 at 11:53 PM, DarthParametric said:

Apologies. Yes, you'll want to set the DXT5 flag in TGA2TPC. The DTX1 implementation is RGB only, hence why it crashes I would guess. I'm not sure why uncompressed would cause a crash. That would be a question for @ndix UR. Edit: Ah, you said RGB. That would likely be the problem. It needs to be RGBA. The DXT5 conversion is presumably adding an alpha if the source image lacks one.

Why the game would require a useless alpha channel on normal maps is a question for BW actually. I would love to know the answer though. As for the RGB/RGBA DXT1/5 thing ... I think I went the other direction. If you request DXT5 on an image with an all-white alpha channel, or missing alpha channel, you get DXT1 instead. I believe this was the choice based on people not really understanding the differences and probably erring towards "oh, DXT5, higher number, better". Actually, unless I'm missing something in the code, it looks like I talked myself down from helping people in that way. So yeah, if you DXT5 an RGB image, you just get a flat white alpha channel.

11 hours ago, bead-v said:

Have you tried compiling with MDLOps? Unless ndix UR has done some changes to the algorithm, it should produce the same result, because the tangent space calculation was a copy paste of ndix UR's algorithm after he figured it out in the bumpmapping research thread (we should still check though). Because it was just a copy paste, I never went very deep into why it is the way it is. I can try to get into it when I find the time, and maybe I can find some way to account for this situation. I can't know for sure, but it feels like this issue should be solvable with an updated algorithm. @ndix UR, what do you think?

I'd assume that frankencompile was mdlops 1.0.0? That one had some serious issues w/ orientations. Usual dot-zero release stuff...

Yeah, the tangent space calculations are due for a closer look, certainly. We can finally do the comparison with the nwntools compiler's code (which I found after painstakingly figuring out the algorithm), to see if it had anything else figured out...

I'd be interested in seeing whether the wrist highlight problem is affected by changing the hands and fingers to 'down' in the parlance of the infographic. I'm wondering if the issue has to do with something absolute or relative ... is it actual 'upness' that matters, or is it just that issues arise at the places where 'up' meet 'down' (this might indicate it's essentially the same problem we had w/ normal vectors, and we just need to reorient the tangent space basis according to 'world coordinates' before accumulating them).

The documentation for Aurora says that LinkedToFlags controls area transition, so is what you need possibly. 0 = links to nothing, 1 = links to another door, 2 = links to a Waypoint. LinkedTo is the Door or Waypoint in question (probably).

For triggers, I think the area transition behavior is controlled by the 'Type' in the UTT file, and should be set to 1 for area transitions.

Yeah, I mean, I do think it's the same poly, having now looked at the model in game for a bit. Changing the weight on vert 307 from 0.6 0.4 to 0.0 1.0 was successful in fixing the issue... I'm surprised that it's not easier to find that triangle, visually, distorted somewhere, as it should be connected at 308 & 309 still.

@DarthParametric Looks like the usual weight mismatch to me. Torso vert lcollar_g 1.0 meeting larm vert lcollar_g 0.4 lbicep_g 0.6. Bigger mismatch than "usual".

Definitely a good thing to fix. I recently fixed that in kotorblender for the same reason. Although in Blender, it's easier to just import model 1, save it as a blend file, start new document, import model 2, and use the 'append' function to pull in specific nodes directly from the other blend file. That bypasses the text editor part. Before that I was using a text-based approach, similar to you, then a script that did it (but that was for merging specific nodes into a vanilla ascii model file before the compilers had skills).

NPCs won't have a PWK file. The PWK file lets you specifically locate where the 'use' points are. Like, if you have a chest that opens from the 'front', the two 'use' points will be in the front. If you have something without a direction, like a barrel, the use points are usually in 'front' and 'back'.

There might be a way to combine an NPC 'placeable' with triggers to get a behavior that tracks closer to a normal placeable. I'm interested in this topic, but I haven't done any experiments yet.

fx_ritual01.tpc is my enemy now. here you go:

fx_ritual01.tga

@bead-v I believe your problem with the vanilla model is that you're not applying an orientation transform to the normals. The torso skin mesh has a super cool 180 degree Z orientation (quat(0, 0, 1, 1.27e6)), so, actually all those torso vertex normals you drew pointing away from the cape vertex normals, all "3" of those normals go in the same outward direction (and two of them match). If not complete slop/oversight it was probably done to workaround a bug in the vanilla toolset's smoothing. I suspect when you fix that (apply orientation translation to normal vectors), the patch-based normal construction is going to work out better. I don't fully comprehend why this would be necessary, other than, interestingly, apparently we need to be doing world coordinate vector comparisons in our smoothgroup computations where we've been doing something more like object space comparisons. When I opened the binary model straight into blender, it looked ... well ... like it does in-game. For the record, I wouldn't consider the smoothing on the vanilla pfbim model something worth emulating/holding up as an example. It's reaaaaal sloppy and horrible. The weights along the torso-robeback interface don't even match, such that you can see gapping in-game when you look from the right angle. Here's a screenshot of the vanilla model hilighting the garbage smoothing between torso and robeback (basically it's OK on the 5 rear seams, but kind of sucks on the side seams)

@JCarter426 if you're tweaking this model, I'd definitely recommend applying the rotation on the torso skin mesh if that's not one of the for-some-reason-sensitive-for-skin-weights-in-3DS operations. If the model you posted in thread already has this done, sorry and nevermind! (I only looked at vanilla while debugging)

OK, I've got these issues sorted. All kotorblender issues. Will be fixed in 1.0.2.

One complete typo was making the particle effect on the sphere wrong (Motion instead of Motion_Blur in two places).

The issue w/ the lines & symbols being visible came down to my setting of minimum values for alphaMid and lifeExp that didn't include the full range of possible values. I had them set to 0.0, but apparently -1.0 is the actual minimum value. Because there are 50+ emitter controllers, I think I probably relied on logic, rather than exhaustive cataloging, to establish some of the min/max values. And anyone who tries to apply logic to the functioning of the game engine gets bitten eventually...

Here's the code patch:

--- a/nvb/nvb_node.py
+++ b/nvb/nvb_node.py
@@ -1625,7 +1625,7 @@ class Emitter(GeometryNode):
             elif attrname == 'render':
                 attrname = 'render_emitter'
                 if value not in ['Normal', 'Billboard_to_Local_Z', 'Billboard_to_World_Z',
-                                 'Aligned_to_World_Z', 'Aligned_to_Particle_Dir', 'Motion']:
+                                 'Aligned_to_World_Z', 'Aligned_to_Particle_Dir', 'Motion_Blur']:
                     value = 'NONE'
             elif attrname == 'blend':
                 if value.lower() == 'punchthrough':
@@ -1639,6 +1639,8 @@ class Emitter(GeometryNode):
                 else:
                     obj.nvb.p2p_type = 'Gravity'
                 # p2p_type has update method, sets p2p_sel
+                # except it doesn't seem to initially
+                obj.nvb.p2p_sel = self.p2p_sel
                 continue
             #print(attrname)
             setattr(obj.nvb, attrname, value)
--- a/nvb/nvb_props.py
+++ b/nvb/nvb_props.py
@@ -389,7 +389,7 @@ class KB_PG_OBJECT(bpy.types.PropertyGroup):
     frameend = bpy.props.IntProperty(name="End Frame", description="Frame End", default=0)
     framestart = bpy.props.IntProperty(name="Start Frame", description="Frame Start", default=0)
     grav = bpy.props.FloatProperty(name="Gravity", description="Gravity (m/s²)", default=0.0, min=0.0, unit='ACCELERATION')
-    lifeexp = bpy.props.FloatProperty(name="Lifetime", description="Life Expectancy", default=1.0, min=0.0)#, update=kb_update_lifeexp_prop)
+    lifeexp = bpy.props.FloatProperty(name="Lifetime", description="Life Expectancy", default=1.0, min=-1.0)#, update=kb_update_lifeexp_prop)
     mass = bpy.props.FloatProperty(name="Mass", description="Mass", default=1.0, min=0.0)
     p2p_bezier2 = bpy.props.FloatProperty(name="Bezier 2", description="???", default=0.0)
     p2p_bezier3 = bpy.props.FloatProperty(name="Bezier 3", description="???", default=0.0)
@@ -411,7 +411,7 @@ class KB_PG_OBJECT(bpy.types.PropertyGroup):
     lightningsubdiv = bpy.props.IntProperty(name="Subdivisions", description="Lightning Subdivisions", default=0, min=0, max=12, update=nvb_update_emitter_prop)
     lightningscale  = bpy.props.FloatProperty(name="Scale", description="Lightning Scale", default=1.0, min=0.0, max=1.0)
     lightningzigzag = bpy.props.IntProperty(name="ZigZag", description="Lightning Zig-Zag", default=0, min=0, max=30)
-    alphamid = bpy.props.FloatProperty(name="Alpha mid", description="Alpha mid", default=1.0, min=0.0, max=1.0)
+    alphamid = bpy.props.FloatProperty(name="Alpha mid", description="Alpha mid", default=1.0, min=-1.0, max=1.0)
     percentstart = bpy.props.FloatProperty(name="Percent start", description="Percent start", default = 1.0, min=0.0, max=1.0)
     percentmid = bpy.props.FloatProperty(name="Percent mid", description="Percent mid", default=1.0, min=0.0, max=1.0)
     percentend = bpy.props.FloatProperty(name="Percent end", description="Percent end", default=1.0, min=0.0, max=1.0)
@@ -474,7 +474,7 @@ class KB_PG_OBJECT(bpy.types.PropertyGroup):
                  ('Billboard_to_World_Z', "Billboard to world Z", "Billboard to world Z", 4),
                  ('Aligned_to_World_Z', "Aligned to world Z", "Aligned  to world Z", 5),
                  ('Aligned_to_Particle_Dir', "Aligned to particle dir.", "Aligned to particle direction", 6),
-                 ('Motion', "Motion Blur", "Motion Blur", 7)],
+                 ('Motion_Blur', "Motion Blur", "Motion Blur", 7)],
         default='NONE', options=set())
     blend = bpy.props.EnumProperty(
         name ="Blend", description="Blending Mode",

21 minutes ago, CarthOnasty said:

Just a heads up, I repeated the process using MDLedit 1.0.3. No issues with modeling or maps or anything like that. Only issue I'm seeing is same as above, particle effects have changed and animations are off or not loading.

Cool, if you're interested in getting support, you need to be way, way more specific about what animations are changed and/or what particle effects on what meshes.

Well, that would be after export.

Also, in Blender, if you just parent plc_starmap_wg under the plc_starmap_pwk node, you should get the right result without having to make text edits after exporting from Blender. I think. Maybe.

Yeah, that's more of a MDLOps issue than KotorBlender actually. It happens because kotorblender isn't putting the PWK nodes after all the geometry nodes. There's no particular reason that should be required other than the naive way I implemented the code to remove those nodes in mdlops. Technically, in your model, the PWK nodes are after all the geometry nodes, it's just that your plc_starmap_wg node has the wrong parent (which makes it seem like there's one non-PWK node in the middle of the PWK nodes, which triggers the same bug). If you change its 'parent plc_starmap' line to 'parent plc_starmap_pwk' the model works fine. I had to make some fixes to how that stuff was exported to make it more like mdledit after the 1.0.0 release of mdlops, which, I think is why you are having that issue...