MDLedit bug reporting thread

[Red Hessian 9]

Not sure if this qualifies as a bug, but every time I try converting heads from TSL to Kotor and vice versa, the head has no animations in game.

Can someone maybe tell me if I'm doing something wrong? I'm not very skilled in these things 😁

[DarthParametric 3,795]

Sounds like user error. Since the removal of the porting ban, I have moved models between both games and had no issues. You need to describe your workflow in detail.

[Red Hessian 9]

Well, I extracted the .mdl and .mdx files using Kotortool, load the .mdl in MDLedit and just change it by clicking the KOTOR1/KOTOR2 button, then save it. Like I said, I'm not very good at these things, and I couldn't find a step by step tutorial anywhere.

[JCarter426 1,220]

You need to have the supermodel extracted in the same folder as the model when you compile for animations to work, and specifically the supermodel for the game you are porting to. For most female heads, this is S_Female03; for most male heads, this is S_Female02 (don't ask).

However, you should know that you can't port a head by compiling it willy-nilly and expecting it to work perfectly. The lips will be messed up if you don't adjust them in a modeling program. I'm working on way to automate this process for heads ported from K2 to K1, but I expect heads ported from K1 to K2 will always need these adjustments, due to the numerous new animations in K2.

[bead-v 251]

1 hour ago, JCarter426 said:

You need to have the supermodel extracted in the same folder as the model when you compile for animations to work, and specifically the supermodel for the game you are porting to. For most female heads, this is S_Female03; for most male heads, this is S_Female02 (don't ask).

Aaaaaah, mdledit needs a warning in this case. It simply takes the model as it is with its supernode numbers, compiles it for the other game instead and completely disregards that the supernode numbers might be different. Thanks for reporting this, @Red Hessian, I'll add a warning to the next version of mdledit.

[JCarter426 1,220]

Well, I'm back.

The Handmaiden has some weird issue on her shoulder here, and similar problems on her wrists. It looks like a smoothing issue, but... well, the problem goes away if I don't use a bump map. Just removing it from the TXI data is enough - leaving the model as bumpmappable is fine. The weird highlights go away whenever the bump map goes away. But I don't understand that because that part of the bump map is solid blue. And if I do get rid of the bump map, different weird highlights pop up on her back. And obviously I'd want to not get rid of the bump map in any case.

Files attached. I've included the binary model and all textures for once, in case it's something there I did wrong.

Handmaiden Shoulder Issue.zip

[DarthParametric 3,795]

Your source normal map is incorrectly formatted. Try this and see what happens:

Edit: Whoops, made a meal of the first one. 2nd attempt -  Alt_JC_HandBA_B_v2.7z

Also, for future reference you can dispense with the

isdiffusebumpmap 1

isspecularbumpmap 1

TXI semantics. Per discussions with @ndix UR, they are leftover remnants from NWN that aren't actually used by Odyssey.

 

[JCarter426 1,220]

18 minutes ago, DarthParametric said:

Your source normal map is incorrectly formatted. Try this and see what happens

Hmm, no change. What did I do wrong? I saved as a power of 2 in indexed color mode and had Tupac compile it as usual, or at least I thought I did. Plus it seemed to be working - the highlight issue aside, it was at least bumping the parts that were meant to be bumped.

19 minutes ago, DarthParametric said:

Per discussions with @ndix UR, they are leftover remnants from NWN that aren't actually used by Odyssey.

You know, I'd been wondering if they were doing anything for a while. But I've been copying and pasting from the first bump map I did, so I always left them in. Thanks for clarifying.

[DarthParametric 3,795]

10 minutes ago, JCarter426 said:

indexed color

That's not how you make a normal map. A tangent space normal map is RGB (plus an alpha mask for Odyssey). If you want an actual bump map, that is greyscale. Additionally, your Y direction was wrong. See this post. It seems like ndix UR put some provision in TGA2TPC for dealing with indexed colour maps, judging by converting your original TPC back to TGA, but the Y was still inverted.

[JCarter426 1,220]

Huh, I was under the impression KOTOR maps had to be saved that way.

To clarify, I first make a height map (attached) and use Nvidia's Normal Map Filter for Photoshop to convert it to an RGB normal map. I then save that in indexed color mode as I thought that was the requirement, though this is based on my vague recollections from the initial testing back in the LucasForums days... I think it was Darth_Sapiens' thread. Then I run the TGA and TXI with obsolete data through TGA2TPC and it makes the TPC and I thought it was working. It's strange that the Y axis would be inverted. There is an option for that in the Nvidia filter, but I hadn't selected it. I suppose I could invert the inversion, though I don't see what was causing it to be inverted in the first place.

Edit: I ran TGA2TPC without converting the TGA to indexed color mode first and the game did in fact crash.

 

JC_HandBA_H.tga

[DarthParametric 3,795]

Yes that's outdated, erroneous information. That thread was continued here. For clarity, Odyssey requires OpenGL (i.e. +Y) tangent space normal maps with an alpha mask, formatted as a TPC. The reason the whole indexed colour thing came up in the first place was that the engine would not accept TGA (or DDS) RGBA bumpmaps, but it would accept greyscale TGA bumpmaps. People thought they could "cheat" in a normal map when it was discovered that it also accepted indexed colour TGAs, but it was proven that the game simply read it as a greyscale bumpmap, not a normal map.

As I said above, ndix UR seems to have implemented an undocumented feature into TGA2TPC (or at least one I was unaware of) that converts an indexed colour image to RGB and normalizes it (i.e. generates Z data in the B channel). So that was why you got a functional map out of it. But it doesn't know what handedness the source image is.

The nVidia filter is DirectX based, so it produces -Y normal maps. To switch these to OpenGL/+Y, all you need to do is invert the G channel (CTRL I in PS).

[JCarter426 1,220]

OK, I had another go at it. It seems RGB > DXT5 works, while Indexed Color > anything is also fine, but RGB > Uncompressed and RGB > DXT1 crash the game.

The Nvidia thing makes sense, though it would've been nice for them to say that somewhere. There's no need to muck about with the channels, by the way - it's a simple flag in the plug-in. It's hard to tell with KOTOR quality models which way is correct, but I'll trust your judgement.

Sadly, the weird highlights are still there, though.

[DarthParametric 3,795]

Apologies. Yes, you'll want to set the DXT5 flag in TGA2TPC. The DTX1 implementation is RGB only, hence why it crashes I would guess. I'm not sure why uncompressed would cause a crash. That would be a question for @ndix UR. Edit: Ah, you said RGB. That would likely be the problem. It needs to be RGBA. The DXT5 conversion is presumably adding an alpha if the source image lacks one.

The simple trick to remember for direction is that if the detail is meant to be raised, that detail on the normal map should appear to be coming out of the screen (versus going into the screen in a DirectX/-Y map).

It's often problematic when you are generating normal data from images, especially colour images. It's not uncommon to have some details being flipped in direction vs others in the same image, even when they should be pointing in the same direction, requiring manual flipping of those areas rather than the whole channel.

[JCarter426 1,220]

Yeah, I did notice the colors were inverted right away after you first mentioned, I just wasn't paying attention to the map before because when I'm not doing it wrong it's usually an automated process. Greyscale height > Nvidia filter > Tupac.

I've got an adjusted workflow now... and actually converting to indexed color was a pain anyway, so I prefer this. I've also configured the Nvidia filter to actually save the height on the alpha channel now that it's an option, so that's also good. It's still difficult to see whether it makes any difference in the game, though.

[DarthParametric 3,795]

The alpha should be solid white in your case. As far as I am aware, it is only a transparency mask (although I haven't actually gotten around to testing it with transparency).

Your highlight issue would appear to be along the UV seam on the shoulder?

And I gather it's the same with the wrists.

Edit: How about remapping the arms something like this?

Obviously ignore the scale/placement in the layout, I'm talking more about hiding the seams on the underside/back of the arm and eliminating the seam on the top of the arm altogether.

[JCarter426 1,220]

1 hour ago, DarthParametric said:

The alpha mask should be solid white in your case. As far as I am aware, it is only a transparency mask (although I haven't actually gotten around to testing it with transparency). 

I don't expect it to anything for KOTOR, no. However, with the original height map saved on the alpha channel, it should be possible to retrieve it from the normal map, which as far as I understand is not usually possible. Converting an RGB bump map to greyscale height map even with Nvidia's tool will result in some quality loss because the normal map doesn't save all the information. Of course, it shouldn't be an issue for me because I have my original height map saved, but I think it's nice to have the option.

1 hour ago, DarthParametric said:

Your highlight issue would appear to be along the UV seam on the shoulder?

It's in that area, yeah, but I don't see anything wrong with it. At least, nothing fundamentally different from the original model, or different even from the left shoulder which lacks the problem. I did some cleaning up with the UV seams (and attached new files) but I still see no difference. I also noticed some skin weight issues that I suspect were KOTORMax's doing (it had to remove excess data due to more than 4 bones per vertex) but I can fix those later.

That said, I'd be very surprised if this wasn't my fault. I did edit the shoulders and hands specifically, so it must be. I had to edit them to get around the 16 bone limit and now I'm wondering if I should've chopped them up differently. Still, it is strange they go away when the bump map goes away....

Handmaiden Shoulder Issue_v13.zip

[DarthParametric 3,795]

All your right arm UV faces are inverted, but the left arm shoulder faces are also inverted. This could be related to that, although I would have been more likely to suspect that in the case of overlapping UVs, which these aren't. As I said in the edit above, I wonder if a remap of the arms isn't worth trying.

[JCarter426 1,220]

Mm, ok, I'll give it a try later (probably after some sleep). The thing is that on the original model, the shoulders weren't on the same mesh as the arms. And the underwear was painted on rather than being actual geometry. I split the shoulders from the underwear, then decided to attach them to the arms thinking that would improve the seams. Then as a consequence, I had to remove the hands to get around the 16 bone limit. I didn't remap them because I didn't think I had to. But maybe I have to.

Edit: After successfully transferring the problem from the right shoulder to the left shoulder, and then getting rid of that part, I think I can confirm that mirrored UV islands were indeed the cause there. I just never expected that to be a problem.

So, to summarize for @bead-v:

• I un-mirrored part of a model (the torso) but left other parts (arms, hands) alone because it wasn't necessary for my purposes.
• However, this caused issues along the UV seams. Where the different UV islands met, there would be weird highlights from what I assume was KOTORMax/MDLEdit having trouble applying smoothing across those seams.
• The cause of the problem seemed to be the handedness of the UV islands. Since I mirrored one part of the map but not the other, they were facing different directions. Flipping the others so they're all flipped the same way seemed to solve the problem. No fancy UVW mapping was necessary - simply mirroring the problematic islands and adjusting the texture was enough.
• For some reason, most of the smoothing oddities seemed to go away when I removed my bump map.
• This might be another entry for the "don't do that" category, but I'm hoping any new information will help you somehow.

I'm still figuring out the hands, but I expect I just broke something and there's no helping that. I meant to detach them along the UV seams and I noticed that I had not, in fact. i don't know what I did to cause the initial problem, but it was probably at that point, and any attempts to fix it have only made it worse. I'm now debating whether to start over with new arm meshes and a new skin wrap, or chop the model up differently, or try a new UVW map, or... what, I don't know. Not a lot of appealing options.

[bead-v 251]

22 hours ago, JCarter426 said:

So, to summarize for @bead-v:

Thanks

So, it's likely that this is connected to the tangent space calculation. That calculation also uses smoothing groups to determine which faces to include in the calculation of a particular tangent space. Also, regular smoothing (normal vectors) doesn't care about UVs. They are however used in the tangent space calculation. So again, very likely that that is the cause. It also explains why the issue went away when you removed the bump map.

Have you tried compiling with MDLOps? Unless ndix UR has done some changes to the algorithm, it should produce the same result, because the tangent space calculation was a copy paste of ndix UR's algorithm after he figured it out in the bumpmapping research thread (we should still check though). Because it was just a copy paste, I never went very deep into why it is the way it is. I can try to get into it when I find the time, and maybe I can find some way to account for this situation. I can't know for sure, but it feels like this issue should be solvable with an updated algorithm. @ndix UR, what do you think?

[DarthParametric 3,795]

23 hours ago, JCarter426 said:

I'm still figuring out the hands

Looking at the version I have here (v13), the left hand and thumb and the right fingers have inverted UV faces. As do the left arm, chest, right leg, left boot upper, right boot lower. Although the left arm is intentional presumably, as it shares UVs with the right arm.

Edit: The boots are also overlapping, I'm guessing probably the original UVs? That sort of weird half and half inversion is typical of vanilla UVs.

[JCarter426 1,220]

Dumb problem is dumb. So, so dumb.

After "fixing" my model - I decided to start over with the arms, imported the original meshes, did a new skin wrap without chopping the arms up this time - I still encountered the highlight issues on the wrists. I barely touched the wrists that time, so it turns out it wasn't anything I broke. A surprise, to be sure, but a welcome one. It took me a while to connect the dots, and I couldn't have done it without you guys, but it finally makes sense. It seems that for the tangent space calculations, direction matters. Up always has to be up. And there's probably a very good mathematical reason for that, but without knowing that, and without having reason to check for that, it took me days to see the problem on the original game UVW map. The arm UV islands point down, with the wrists at the bottom of the map. The hand UV islands, however, point up, with the wrists and the base of each finger at the bottom. Just as the polarity of the UV islands was causing a problem on the shoulder seams, the differing orientation was causing the problem on the wrist seams.

10 hours ago, DarthParametric said:

I'm guessing probably the original UVs?

Probably most of what you listed were, yeah. The only part I did an extensive mapping on was the torso. I un-mirrored and realigned the skin part and completely remapped the underwear part. For most everything else, I only moved the UV islands around to make room on the texture. The boots and necklace were from her clothing model.

11 hours ago, bead-v said:

Have you tried compiling with MDLOps?

MDLOps had other problems with it:

[ndix UR 218]

On 8/5/2018 at 11:34 PM, DarthParametric said:

As I said above, ndix UR seems to have implemented an undocumented feature into TGA2TPC (or at least one I was unaware of) that converts an indexed colour image to RGB and normalizes it (i.e. generates Z data in the B channel). So that was why you got a functional map out of it. But it doesn't know what handedness the source image is.

This was essentially the first thing implemented, because when we first started, I had to rule out all the various indexed color modes as necessary or helpful in order to show that 32-bit RGBA is the one true way I don't think it does normalization though, possibly just R/B inversion. Some old implementation detail memories there...

On 8/5/2018 at 11:53 PM, DarthParametric said:

Apologies. Yes, you'll want to set the DXT5 flag in TGA2TPC. The DTX1 implementation is RGB only, hence why it crashes I would guess. I'm not sure why uncompressed would cause a crash. That would be a question for @ndix UR. Edit: Ah, you said RGB. That would likely be the problem. It needs to be RGBA. The DXT5 conversion is presumably adding an alpha if the source image lacks one.

Why the game would require a useless alpha channel on normal maps is a question for BW actually. I would love to know the answer though. As for the RGB/RGBA DXT1/5 thing ... I think I went the other direction. If you request DXT5 on an image with an all-white alpha channel, or missing alpha channel, you get DXT1 instead. I believe this was the choice based on people not really understanding the differences and probably erring towards "oh, DXT5, higher number, better". Actually, unless I'm missing something in the code, it looks like I talked myself down from helping people in that way. So yeah, if you DXT5 an RGB image, you just get a flat white alpha channel.

11 hours ago, bead-v said:

Have you tried compiling with MDLOps? Unless ndix UR has done some changes to the algorithm, it should produce the same result, because the tangent space calculation was a copy paste of ndix UR's algorithm after he figured it out in the bumpmapping research thread (we should still check though). Because it was just a copy paste, I never went very deep into why it is the way it is. I can try to get into it when I find the time, and maybe I can find some way to account for this situation. I can't know for sure, but it feels like this issue should be solvable with an updated algorithm. @ndix UR, what do you think?

I'd assume that frankencompile was mdlops 1.0.0? That one had some serious issues w/ orientations. Usual dot-zero release stuff...

Yeah, the tangent space calculations are due for a closer look, certainly. We can finally do the comparison with the nwntools compiler's code (which I found after painstakingly figuring out the algorithm), to see if it had anything else figured out...

I'd be interested in seeing whether the wrist highlight problem is affected by changing the hands and fingers to 'down' in the parlance of the infographic. I'm wondering if the issue has to do with something absolute or relative ... is it actual 'upness' that matters, or is it just that issues arise at the places where 'up' meet 'down' (this might indicate it's essentially the same problem we had w/ normal vectors, and we just need to reorient the tangent space basis according to 'world coordinates' before accumulating them).

[JCarter426 1,220]

34 minutes ago, ndix UR said:

So yeah, if you DXT5 an RGB image, you just get a flat white alpha channel.

Ah, so it actually doesn't make a difference if I save the height map on the alpha channel. Or are you saying it generates a solid white channel only if there isn't an alpha channel already?

34 minutes ago, ndix UR said:

I'd assume that frankencompile was mdlops 1.0.0? That one had some serious issues w/ orientations. Usual dot-zero release stuff...

Yeah, 1.0.0. That's the most recent version on the site?

34 minutes ago, ndix UR said:

I'd be interested in seeing whether the wrist highlight problem is affected by changing the hands and fingers to 'down' in the parlance of the infographic. I'm wondering if the issue has to do with something absolute or relative ... is it actual 'upness' that matters, or is it just that issues arise at the places where 'up' meet 'down' (this might indicate it's essentially the same problem we had w/ normal vectors, and we just need to reorient the tangent space basis according to 'world coordinates' before accumulating them).

I flipped them all down and now it's fine:

Spoiler

So I assume it's a relative issue. Although, it's possible that up and down matter for other things in relation to tangent space. I haven't noticed any other issues yet, though. For example, the legs are rotate 90 degrees, but I assume there's no issue because they don't smooth with any other UV islands on my model.

[bead-v 251]

32 minutes ago, ndix UR said:

(this might indicate it's essentially the same problem we had w/ normal vectors, and we just need to reorient the tangent space basis according to 'world coordinates' before accumulating them).

I'm already doing that in the latest beta version of mdledit. I do it in paralel with the vertex normals, so I basically take the world coordinates for the verts instead of the object coordinates and do the whole calculation and then at the end rotate the vector to where it needs to be in object space.  I checked a few models and it seemed to get them right, but it could be that I only checked models that don't have any weird stuff going on... so maybe that's not the right way to do it. So yeah, need to find the time to look at the algorithm and compare with the one in NWNTools...

 

11 minutes ago, JCarter426 said:

I flipped them all down and now it's fine:

You never wait for us to fix the tools

[JCarter426 1,220]

Well, like I said, I file the problem under "don't do that". It would be nice if you could have the tools fix it, but if it the problem can be avoided on my end, I don't mind sorting it. Maybe I'm still scarred by the dark times of NWMax & old MDLOps, but I'm used to adjusting my methods to make the game happy. It is good to understand why these things break, I think, even if the problems can be solved.