[TOOL] KotOR Stuff

[ApanLoon 5]

I started to work on a set of small tools for extracting assets from KotOR. These tools are not in any way competing with more complex tools such as "KotOR Tool" or "KotOR Toolset". Those focus on giving a complete experience - a complete solution for all your modding needs. KotOR Stuff does the opposite.

 

KotOR Stuff provides several tiny command-line tools that each perform a small task. With scripting, these tools could be used to simplify batch processing or support other tools that perform more complex tasks.

 

Currently three tools are included:

• bif - Extract assets from KeyTable/BIF repositories
• erf - Extract assets from ERF/MOD/SAV and HAK repositories
• tpc - Convert TPC files to PNG. Use command line options to select which side of a cubemap, what frame of a cycle animation or which mip map to convert. Can also list TXI attributes if any are present.

Future releases of this project may include:

• Creating new BIF/ERF/MOD/SAV and HAK repositories
• Adding resources to BIF/ERF/MOD/SAV and HAK repositories
• Convert PNG to TPC given additional information on the command line
• Improved TPC support: Even though the tpc tool can correctly convert some TPC types that "KotOR Tool" can't, some image types are still not handled correctly. For example, cube maps with only four sides exist but can't be converted yet.

Future releases of this project will probably not include:

• Pretty GUIs that makes modding a breeze for anyone (That is what "KotOR Tool" and "KotOR Toolset" are for.)

C# source code can be found here:

 

https://github.com/ApanLoon/KotORStuff

 

Binary version for Windows (might also work with mono on other platforms) can be found here:

 

http://deadlystream.com/forum/files/file/1128-kotor-stuff/

[VarsityPuppet 1,085]

Ahhh a man after my own heart, with all of that C# code. I'll take a closer look later, but I'm happy just to see this

[ApanLoon 5]

UPDATE: v0.3

 

The tpc tool can now convert PNG files to uncompressed TPC files.

 

The following example creates an uncompressed RGB file with two sub images (One full scale and one half-scale mip map) and a couple of TXI attributes:

tpc -o tpc -s2 --outpath PO_PFHA3d.tpc -txi="proceduretype cycle" -txi="channelscale 4 0.1 0.2 0.3 0.4" PO_PFHA3d.png

Note that it might still not be possible to create cyclic procedurals with this tool, I just added those TXI attributes to illustrate how it should be done.

 

To get some more info about what TXI attributes are used in game and how they need to be formatted for this tool, use the following command to examine an existing TPC file:

tpc -o none -t PO_PFHA3d.tpc

Run this on one, some or all the files in the texture pack to see what's in there.

[VarsityPuppet 1,085]

UPDATE: v0.3

 

The tpc tool can now convert PNG files to uncompressed TPC files.

 

The following example creates an uncompressed RGB file with two sub images (One full scale and one half-scale mip map) and a couple of TXI attributes:

tpc -o tpc -s2 --outpath PO_PFHA3d.tpc -txi="proceduretype cycle" -txi="channelscale 4 0.1 0.2 0.3 0.4" PO_PFHA3d.png

Note that it might still not be possible to create cyclic procedurals with this tool, I just added those TXI attributes to illustrate how it should be done.

 

To get some more info about what TXI attributes are used in game and how they need to be formatted for this tool, use the following command to examine an existing TPC file:

tpc -o none -t PO_PFHA3d.tpc

Run this on one, some or all the files in the texture pack to see what's in there.

Okay, now this stuff is legit. I will be looking very closely at this code. I've been wondering about this for years, but never took the initiative to look into it. Thank you for your hard work, and I'd be honored if you continued to share more of it in the future. If I can learn anything from it I'd love to contribute as well

 

I hope God blesses every single one of you toolmakers. ( You know who you are )

[ApanLoon 5]

Cool. I am happy that you like it.

 

There are still some types of TPC files that this tool can't read and there are some questionable assumptions in the current code. I would appreciate any and all assistance making the code better and more complete.

 

I was told that TGA support is kind of important, so I might look into ways to add that. The library I use can't do TGA - only BMP, EMF, EXIF, GIF, ICO, JPEG, PNG, TIF and WMF. Adding more of those would be trivial, adding TGA is a bit of work but not too complicated, I think.

[ApanLoon 5]

Assistance requested:

 

This will be a long post. It contains some tips about how to use the tpc tool but also questions about further development of the tool.

 

I just added the possibility to extract the TXI to a separate file when reading a TPC and conversely, to read the TXI from a separate file when creating a TPC. However, the command line options are a bit weird at the moment. I would like to get feedback on how these should be changed to make them more intuitive.

 

For the rest of this post, I will be talking about the current development branch on GitHub and NOT the currently released version. It behaves differently in regards to TXI.

 

The current help info looks like this:

$ ./tpc -h
Usage: tpc.exe [OPTIONS]+ [FILE]+

Operate on files in the Bioware TPC format.

Options:
  -n, --nofile               Don't output to a file, instead use stdout.
  -o, --output=VALUE         The format of the output file(s). Possible
                               options are: png, tif, raw, hex, tpc, none
  -i, --image=VALUE          The index of the image to extract.
  -s, --subimage=VALUE       The index of the sub image to extract or the
                               number of sub images to generate in new TPC.
  -t, --txi[=VALUE]          Dump the texture info or add TXI attribute to
                               new TPC. When adding an attribute, use the form -
                               -txi="key value..."
      --txipath=VALUE        Create this separate TXI file when reading a TPC
                               or read this separate TXI file when creating a
                               TPC.
  -c, --compress             Use DXT compression when creating new TPC.
  -f, --format=VALUE         Pixel format for new TPC. Possible options are:
                               Gray, RGB, RGBA, BGRA
  -p, --outpath=VALUE        Path and file name of output. Default is the
                               same as the input with a new extension.
  -h, --help                 Show this message and exit

Unless the nofile option is given, tpc.exe prints information about the file
on stdout. This example explains what is shown:

zc   64   64  6  7 42 RGBA         1 CM_rakata.tpc

  Flags:                      z=DXT compression, c=Cube map

  Width Height:               Of the full-size version of the image(s).
                              Each additional sub image, if any is reduced
                              in size by half width and half height.

  Number of images:           The number of full-size images. Cube maps have
                              six images.

  Sub images per image:       The number of sub images for each image.
                              The first is the full-size image, subsequent
                              sub images are mip maps.

  Total number of sub images: The header of some files indicate more sub
                              images than there is data for in the file.
                              This shows how many sub images there actually
                              are.

  Pixel format:               Gray, RGB, RGBA, SwizzledBGRA

  Unknown1:                   Float value of unknown purpose.

  Full path to the file

I realise that not many of you have even tested this yet, but I would like to have a cleaner way to tell the tool what type of output to generate and where to put that output.

 

The tool basically outputs three things: Info, Image and TXI

 

I think that he most common usage would be to output the image and TXI to files and print the info on stdout. Currently, this is what happens if you, for example, run this:

tpc -o png -t PO_PFHA3d.tpc C_HoloDodonna.tpc

The new files will be named PO_PFHA3d.png, PO_PFHA3d.txi, C_HoloDodonna.png and C_HoloDodonna.txi respectively and the info for the two TPCs are printed on stdout.

 

It is possible that, in some situations, you would want the image data to be sent to stdout instead of a file. An example of this would be if you want to rotate an image with something like ImageMagick before storing the file:

tpc -n -o png CM_HawkBay.tpc | convert - -rotate 90 CM_HawkBay.png

I don't know if this works in Windows "cmd" application, but I do know that it works in "bash" on Linux, OSX and Cygwin under Windows.

 

You also might want to filter stuff out from the info:

tpc -o none *.tpc | grep "RGB "

only lists the info for RGB files. Note the extra space at the end of the grep to make sure that it doesn't also match RGBA.

 

We can also get the TXI on stdout:

tpc -n -o none -t C_HoloDodonna.tpc | grep proceduretype

When you specify that the tool should not send output to a file with -n, the tool won't output the info as that would interfere with the image or TXI output. "convert", for example, would fail to read the png if the data also contained the text of the info.

 

Now we arrive at the more complex situations. What if we want both the info and the TXI on stdout? Like the currently released version does with -t?

It would be nice to see what TXI belongs to what TPC file when listing more than one. We might try:

tpc -n -o none -t *.tpc | grep proceduretype

But this doesn't do exactly what we want. From the result of this we will get a list of all proceduretype attributes, and that could be useful too, especially if you add " | sort | uniq" to the command line. However, we won't see which file has which attribute and that would probably be useful. There is no way to do this yet.

 

Question 1 to you is: What options should we have to make this happen?

 

Note that, when working on a lot of files, you might get an error such as this:

-bash: ./tpc: Argument list too long

If that happens, you should embed your command in a "find":

find AllTPCs/ -exec tpc -n -o none -t "{}" "+" | grep proceduretype | sort | uniq

The "+" will make find call tpc multiple times with an optimal(?) number of files each time.

 

Question 2 to you is: Are there other combinations of outputs that would be useful? Sending TXI to a file while sending the image to stdout or sending the info to a file, or...

 

Question 3 to you is: What additional options do we need? One I know already is that we should have a way to specify the unknown float that is put into new TPCs. Currently, this simply defaults to 1.0.

 

Question 4 to you is: What features/bugs need to be addressed and in what order? I.e. What features are most important to you?

[JCarter426 1,216]

Interesting. However, when I tried to use the TPC converter, it seemed to discard the alpha channel and saved it as transparency data instead. Is there any way to avoid that? Unfortunately, the way it is now makes editing textures with alpha channels impossible.

[ApanLoon 5]

Interesting. However, when I tried to use the TPC converter, it seemed to discard the alpha channel and saved it as transparency data instead. Is there any way to avoid that? Unfortunately, the way it is now makes editing textures with alpha channels impossible.

 

Is the alpha channel used for anything other than transparency in game? If so, what?

[Malkior 476]

Is the alpha channel used for anything other than transparency in game? If so, what?

There are several uses for it, including Reflective or "Cubemaps" and an oft unused glow effect. If you're curious, the [texturename].txi will list its possible applications for the Alpha information.

[sELFiNDUCEDcOMA 98]

Cool. New tools. Definitely looking forward to playing around with the PNG to TPC converter at some point. Would be better if it had a GUI front-end, but, as the code is available and in C#... I guess someone can make one for it at some point.

 

There's likely a few solutions to add TGA support to your tool.

 

The alpha channel in texturing can be used for a number of things -- anything that is a black and white map really. So transparency, specularity, etc. Newer engines use it to store other things like ambient occlusion. Usually it is used to store everything but transparency it seems .

[JCarter426 1,216]

Yeah, the problem is that for those textures that do not use it for transparency - and they are many - when they are rendered transparent by the tool, all the non-alpha data, the visible part, is lost. The Sith armor, for example, becomes almost entirely invisible because of all the shiny parts that are rendered transparent.

[ApanLoon 5]

Interesting. I thought that since normal maps are stored in separate files, reflection maps and such would also be stored in separate files. I will think about how to handle this. Most likely, I will make an option to extract the alpha channel as a separate gray scale image and then add the possibility to provide two PNGs to create a new TPC file.

 

For now, this work-around in Photoshop might work:

1. Open the converted PNG
2. Menu: Layer->Layer Mask->From Transparency
3. Alt-LeftClick the mask
4. Ctrl-a Ctrl-x to cut the mask out. This will make the mask completely white, meaning that you can see all that info that the alpha channel was hiding
5. Menu: Layer->New->Layer
6. Ctrl-v to paste the mask into the new layer
7. Edit the image and alpha layers to your hearts content
8. With the alpha layer selected, ctrl-a ctrl-c
9. Alt-LeftClick the mask of the image layer
10. Ctrl-v to paste in the edited alpha channel
11. Hide the "alpha" layer
12. Save as PNG
13. Use the tpc tool to create a new TPC file based on the new PNG file

I am not 100% certain that this will be perfectly accurate, but I think that it should work even if it is a bit clunky. Also, the tpc tool still can't create DXT-compressed images.

 

@sELFiNDUCEDcOMA about the GUI

Yes, that is one of the main points of this exercise: The code is structured so that there are libraries that handle the file formats and separate console applications for the tools I provide. Anyone could use the same libraries in their own GUI application directly, or call the console applications when needed. Many options are available.

 

@Malkior

The TXI can say many things. Most, as I understand it, has nothing to do with the alpha channel. Is there a list somewhere that explains what the various TXI attributes do? I have guessed many, but it would be nice to build a comprehensive list in a public space.

[ApanLoon 5]

There's likely a few solutions to add TGA support to your tool.

That looks really interesting, perhaps I should make a fork of CSharpImageLibrary, adding TPC support, and then use that in the tpc tool. It even has DXT compression done.

[Malkior 476]

@Malkior

The TXI can say many things. Most, as I understand it, has nothing to do with the alpha channel. Is there a list somewhere that explains what the various TXI attributes do? I have guessed many, but it would be nice to build a comprehensive list in a public space.

As far as cubemaps are concerned, the TXI uses the Alpha layer of the texture to determine what is shiny and what is not. Without an Alpha layer, the texture with a cubemap applied to it either by the TXI or Appearance.2da will apply it to the entire object.

 

As far as what the TXI functions are, I only know of very few. There is, however, a topic on this forum devoted to listing them all that may be of some assistance.

[ApanLoon 5]

As far as cubemaps are concerned, the TXI uses the Alpha layer of the texture to determine what is shiny and what is not. Without an Alpha layer, the texture with a cubemap applied to it either by the TXI or Appearance.2da will apply it to the entire object.

As far as what the TXI functions are, I only know of very few. There is, however, a topic on this forum devoted to listing them all that may be of some assistance.

It would surprise me if the presence of the "cube map" TXI-attribute implies that the alpha is a "shininess" map. I am pretty sure that I have seen specific attributes for that and, until now, I didn't know what they did. I will examine this when I get back to my computer.

 

I would appreciate a link to that forum thread. I tried to search for it but it is difficult on the phone.

[DarthParametric 3,790]

The alpha should always be a transparency mask. The TXI semantics just tell the engine whether or not to blend a second texture, like an environment/cube map.

[ApanLoon 5]

That is what I thought too.

[JCarter426 1,216]

Yes, the actual shininess or whatever is a separate texture (cm_baremetal, for example) but essentially the game can either have a shader or transparency, but not both. The alpha channel is used for either, depending on the situation. And actually you can't just use the alpha channel for transparency, because then it won't render any objects behind the texture for some reason, so you still need a shader for that.

 

For now, this work-around in Photoshop might work:

1. Open the converted PNG
2. Menu: Layer->Layer Mask->From Transparency
3. Alt-LeftClick the mask
4. Ctrl-a Ctrl-x to cut the mask out. This will make the mask completely white, meaning that you can see all that info that the alpha channel was hiding
5. Menu: Layer->New->Layer
6. Ctrl-v to paste the mask into the new layer
7. Edit the image and alpha layers to your hearts content
8. With the alpha layer selected, ctrl-a ctrl-c
9. Alt-LeftClick the mask of the image layer
10. Ctrl-v to paste in the edited alpha channel
11. Hide the "alpha" layer
12. Save as PNG
13. Use the tpc tool to create a new TPC file based on the new PNG file

I am not 100% certain that this will be perfectly accurate, but I think that it should work even if it is a bit clunky. Also, the tpc tool still can't create DXT-compressed images.4

 

Ah, that does look like it works. I should have figured since it was still saved as a separate channel, there would be a way to unbake the cake here. However, I'd still be interested in a more automated process, for two reasons. First, I wrote a batch script to mass convert every TPC file in the game... but having to go fix the alpha channel after that doesn't really save time in the end. So that would certainly help. The other thing is that 3ds Max reads the games' alpha channels as white, which is... not what it should be doing. I'm not sure if it can't figure out how to use the alpha channel properly, or if it just prefers having separate textures, but I've never cared to find out and I end up separating it into two textures anyway when I'm rendering in it. So TPC to 2 PNGs would definitely speed things along there.

 

I look forward to any updates. Thanks for continuing to make tools for a game that's now older than this generation of players.

[ApanLoon 5]

I am pretty sure that you could use imagemagick for making a script that does the same thing as that workaround. Inagemagick works well in Cygwin under Windows or in Linux/OS X.

[JCarter426 1,216]

Ah, I've never used that. I'll take a look.

[ApanLoon 5]

It is Christmas time again and my holiday has started.

 

I decided to look into some of the TPC variations that my little tpc tool still can't read. Lightmaps are weird. Has anyone examined/used those?

 

These images appear to break the rules. The number of pixels in the data matches the width and height specified in the header, but the actual image content does not. Instead it appears to be split up in 10 rows and 10 columns of square blocks padded by black pixels on the right and bottom.

 

Converting 3dgui_a00001.tpc with my tool (as well as any other tool I am aware of) shows a number of 5x5 squares in the top left corner of the converted image. (See attached 3dgui_a00001.png) I can manually assemble these squares into something that looks more like the desired end-result. (See 3dgui_a00001-assembled.png)

 

Doing something similar with PLCaa_a00005.tpc again yields a much more sensible light map. Here the squares are 17x17 pixels and the order of the assembly is reversed - columns become rows.

 

I can see nothing in the header or the TXI that would help the tool to assemble these images correctly.

 

Can anyone shed some light on this? (Pun intended)

 

P.S. The scale of the thumbnails are misleading, the size of the assembled images are 50x50 and 170x170 respectively.

[DarthParametric 3,790]

Your tool is working correctly. That's just how the lightmap UVs are arrayed. For example, Plane02 in PLCaa that uses that lightmap:

 

[ApanLoon 5]

Wow, that is weird - what a terrible waste of texture memory. I assume that these textures are generated by some form of ray tracing. Makes it really tricky to make light map mods though. :-(

 

Thanks for checking it out. Perhaps this is the cause of some other textures looking weird too - they might also be generated by some tool that doesn't care about memory efficiency.

[DarthParametric 3,790]

I would assume their lightmap UVs were probably auto-generated, yeah. Pretty much all of them only use a fraction of the available UV space. Maybe it was intended to keep to some uniform size/pixel density to keep shadows consistent.

 

As to how they were generated, it would have been baked in 3DS Max, probably in a not too dissimilar manner to how modders do it now (albeit most likely with some scripts to automate things).