Jump to content


Photo

[TOOL] KotOR Stuff


  • Please log in to reply
19 replies to this topic

#1 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 17 July 2017 - 07:28 PM

I started to work on a set of small tools for extracting assets from KotOR. These tools are not in any way competing with more complex tools such as "KotOR Tool" or "KotOR Toolset". Those focus on giving a complete experience - a complete solution for all your modding needs. KotOR Stuff does the opposite.

 

KotOR Stuff provides several tiny command-line tools that each perform a small task. With scripting, these tools could be used to simplify batch processing or support other tools that perform more complex tasks.

 

Currently three tools are included:

  • bif - Extract assets from KeyTable/BIF repositories
  • erf - Extract assets from ERF/MOD/SAV and HAK repositories
  • tpc - Convert TPC files to PNG. Use command line options to select which side of a cubemap, what frame of a cycle animation or which mip map to convert. Can also list TXI attributes if any are present.

Future releases of this project may include:

  • Creating new BIF/ERF/MOD/SAV and HAK repositories
  • Adding resources to BIF/ERF/MOD/SAV and HAK repositories
  • Convert PNG to TPC given additional information on the command line
  • Improved TPC support: Even though the tpc tool can correctly convert some TPC types that "KotOR Tool" can't, some image types are still not handled correctly. For example, cube maps with only four sides exist but can't be converted yet.

Future releases of this project will probably not include:

  • Pretty GUIs that makes modding a breeze for anyone (That is what "KotOR Tool" and "KotOR Toolset" are for.)

C# source code can be found here:

 

https://github.com/ApanLoon/KotORStuff

 

Binary version for Windows (might also work with mono on other platforms) can be found here:

 

http://deadlystream....28-kotor-stuff/



#2 VarsityPuppet

VarsityPuppet

    Jedi Master

  • Premium Member
  • 2,014 posts

Posted 18 July 2017 - 12:51 AM

Ahhh a man after my own heart, with all of that C# code. I'll take a closer look later, but I'm happy just to see this :D
  • LDR likes this

Be careful how you think; your life is shaped by your thoughts. - Proverbs 4:23


#3 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 18 July 2017 - 03:39 PM

UPDATE: v0.3

 

The tpc tool can now convert PNG files to uncompressed TPC files.

 

The following example creates an uncompressed RGB file with two sub images (One full scale and one half-scale mip map) and a couple of TXI attributes:

tpc -o tpc -s2 --outpath PO_PFHA3d.tpc -txi="proceduretype cycle" -txi="channelscale 4 0.1 0.2 0.3 0.4" PO_PFHA3d.png

Note that it might still not be possible to create cyclic procedurals with this tool, I just added those TXI attributes to illustrate how it should be done.

 

To get some more info about what TXI attributes are used in game and how they need to be formatted for this tool, use the following command to examine an existing TPC file:

tpc -o none -t PO_PFHA3d.tpc

Run this on one, some or all the files in the texture pack to see what's in there.



#4 VarsityPuppet

VarsityPuppet

    Jedi Master

  • Premium Member
  • 2,014 posts

Posted 18 July 2017 - 04:25 PM

UPDATE: v0.3

The tpc tool can now convert PNG files to uncompressed TPC files.

The following example creates an uncompressed RGB file with two sub images (One full scale and one half-scale mip map) and a couple of TXI attributes:

tpc -o tpc -s2 --outpath PO_PFHA3d.tpc -txi="proceduretype cycle" -txi="channelscale 4 0.1 0.2 0.3 0.4" PO_PFHA3d.png
Note that it might still not be possible to create cyclic procedurals with this tool, I just added those TXI attributes to illustrate how it should be done.

To get some more info about what TXI attributes are used in game and how they need to be formatted for this tool, use the following command to examine an existing TPC file:
tpc -o none -t PO_PFHA3d.tpc
Run this on one, some or all the files in the texture pack to see what's in there.

Okay, now this stuff is legit. I will be looking very closely at this code. I've been wondering about this for years, but never took the initiative to look into it. Thank you for your hard work, and I'd be honored if you continued to share more of it in the future. If I can learn anything from it I'd love to contribute as well

I hope God blesses every single one of you toolmakers. ( You know who you are ;) )

Be careful how you think; your life is shaped by your thoughts. - Proverbs 4:23


#5 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 18 July 2017 - 09:37 PM

Cool. I am happy that you like it. :D

 

There are still some types of TPC files that this tool can't read and there are some questionable assumptions in the current code. I would appreciate any and all assistance making the code better and more complete.

 

I was told that TGA support is kind of important, so I might look into ways to add that. The library I use can't do TGA - only BMP, EMF, EXIF, GIF, ICO, JPEG, PNG, TIF and WMF. Adding more of those would be trivial, adding TGA is a bit of work but not too complicated, I think.



#6 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 19 July 2017 - 01:25 PM

Assistance requested:

 

This will be a long post. It contains some tips about how to use the tpc tool but also questions about further development of the tool.

 

I just added the possibility to extract the TXI to a separate file when reading a TPC and conversely, to read the TXI from a separate file when creating a TPC. However, the command line options are a bit weird at the moment. I would like to get feedback on how these should be changed to make them more intuitive.

 

For the rest of this post, I will be talking about the current development branch on GitHub and NOT the currently released version. It behaves differently in regards to TXI.

 

The current help info looks like this:

$ ./tpc -h
Usage: tpc.exe [OPTIONS]+ [FILE]+

Operate on files in the Bioware TPC format.

Options:
  -n, --nofile               Don't output to a file, instead use stdout.
  -o, --output=VALUE         The format of the output file(s). Possible
                               options are: png, tif, raw, hex, tpc, none
  -i, --image=VALUE          The index of the image to extract.
  -s, --subimage=VALUE       The index of the sub image to extract or the
                               number of sub images to generate in new TPC.
  -t, --txi[=VALUE]          Dump the texture info or add TXI attribute to
                               new TPC. When adding an attribute, use the form -
                               -txi="key value..."
      --txipath=VALUE        Create this separate TXI file when reading a TPC
                               or read this separate TXI file when creating a
                               TPC.
  -c, --compress             Use DXT compression when creating new TPC.
  -f, --format=VALUE         Pixel format for new TPC. Possible options are:
                               Gray, RGB, RGBA, BGRA
  -p, --outpath=VALUE        Path and file name of output. Default is the
                               same as the input with a new extension.
  -h, --help                 Show this message and exit

Unless the nofile option is given, tpc.exe prints information about the file
on stdout. This example explains what is shown:

zc   64   64  6  7 42 RGBA         1 CM_rakata.tpc

  Flags:                      z=DXT compression, c=Cube map

  Width Height:               Of the full-size version of the image(s).
                              Each additional sub image, if any is reduced
                              in size by half width and half height.

  Number of images:           The number of full-size images. Cube maps have
                              six images.

  Sub images per image:       The number of sub images for each image.
                              The first is the full-size image, subsequent
                              sub images are mip maps.

  Total number of sub images: The header of some files indicate more sub
                              images than there is data for in the file.
                              This shows how many sub images there actually
                              are.

  Pixel format:               Gray, RGB, RGBA, SwizzledBGRA

  Unknown1:                   Float value of unknown purpose.

  Full path to the file

I realise that not many of you have even tested this yet, but I would like to have a cleaner way to tell the tool what type of output to generate and where to put that output.

 

The tool basically outputs three things: Info, Image and TXI

 

I think that he most common usage would be to output the image and TXI to files and print the info on stdout. Currently, this is what happens if you, for example, run this:

tpc -o png -t PO_PFHA3d.tpc C_HoloDodonna.tpc

The new files will be named PO_PFHA3d.png, PO_PFHA3d.txi, C_HoloDodonna.png and C_HoloDodonna.txi respectively and the info for the two TPCs are printed on stdout.

 

It is possible that, in some situations, you would want the image data to be sent to stdout instead of a file. An example of this would be if you want to rotate an image with something like ImageMagick before storing the file:

tpc -n -o png CM_HawkBay.tpc | convert - -rotate 90 CM_HawkBay.png

I don't know if this works in Windows "cmd" application, but I do know that it works in "bash" on Linux, OSX and Cygwin under Windows.

 

You also might want to filter stuff out from the info:

tpc -o none *.tpc | grep "RGB "

only lists the info for RGB files. Note the extra space at the end of the grep to make sure that it doesn't also match RGBA.

 

We can also get the TXI on stdout:

tpc -n -o none -t C_HoloDodonna.tpc | grep proceduretype

When you specify that the tool should not send output to a file with -n, the tool won't output the info as that would interfere with the image or TXI output. "convert", for example, would fail to read the png if the data also contained the text of the info.

 

Now we arrive at the more complex situations. What if we want both the info and the TXI on stdout? Like the currently released version does with -t?

It would be nice to see what TXI belongs to what TPC file when listing more than one. We might try:

tpc -n -o none -t *.tpc | grep proceduretype

But this doesn't do exactly what we want. From the result of this we will get a list of all proceduretype attributes, and that could be useful too, especially if you add " | sort | uniq" to the command line. However, we won't see which file has which attribute and that would probably be useful. There is no way to do this yet.

 

Question 1 to you is: What options should we have to make this happen?

 

Note that, when working on a lot of files, you might get an error such as this:

-bash: ./tpc: Argument list too long

If that happens, you should embed your command in a "find":

find AllTPCs/ -exec tpc -n -o none -t "{}" "+" | grep proceduretype | sort | uniq

The "+" will make find call tpc multiple times with an optimal(?) number of files each time.

 

Question 2 to you is: Are there other combinations of outputs that would be useful? Sending TXI to a file while sending the image to stdout or sending the info to a file, or...

 

Question 3 to you is: What additional options do we need? One I know already is that we should have a way to specify the unknown float that is put into new TPCs. Currently, this simply defaults to 1.0.

 

Question 4 to you is: What features/bugs need to be addressed and in what order? I.e. What features are most important to you?



#7 JCarter426

JCarter426

    Jedi Knight

  • M478_Staff
  • 289 posts

Posted 04 August 2017 - 12:30 AM

Interesting. However, when I tried to use the TPC converter, it seemed to discard the alpha channel and saved it as transparency data instead. Is there any way to avoid that? Unfortunately, the way it is now makes editing textures with alpha channels impossible.



#8 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 09 August 2017 - 10:09 PM

Interesting. However, when I tried to use the TPC converter, it seemed to discard the alpha channel and saved it as transparency data instead. Is there any way to avoid that? Unfortunately, the way it is now makes editing textures with alpha channels impossible.

 

Is the alpha channel used for anything other than transparency in game? If so, what?



#9 Malkior

Malkior

    Jedi Master

  • Members
  • PipPipPipPipPip
  • 1,099 posts

Posted 09 August 2017 - 10:43 PM

Is the alpha channel used for anything other than transparency in game? If so, what?

There are several uses for it, including Reflective or "Cubemaps" and an oft unused glow effect. If you're curious, the [texturename].txi will list its possible applications for the Alpha information.


____________________________
|_______________________>>___\
....But I digress....
__________________________
|____________>>_______>>__/

#10 sELFiNDUCEDcOMA

sELFiNDUCEDcOMA

    Sith'ari

  • Members
  • PipPipPipPip
  • 305 posts
  • LocationLand of Oz

Posted 10 August 2017 - 02:24 AM

Cool. New tools. Definitely looking forward to playing around with the PNG to TPC converter at some point. Would be better if it had a GUI front-end, but, as the code is available and in C#... I guess someone can make one for it at some point.

 

There's likely a few solutions to add TGA support to your tool.

 

The alpha channel in texturing can be used for a number of things -- anything that is a black and white map really. So transparency, specularity, etc. Newer engines use it to store other things like ambient occlusion. Usually it is used to store everything but transparency it seems ;).



#11 JCarter426

JCarter426

    Jedi Knight

  • M478_Staff
  • 289 posts

Posted 10 August 2017 - 05:11 AM

Yeah, the problem is that for those textures that do not use it for transparency - and they are many - when they are rendered transparent by the tool, all the non-alpha data, the visible part, is lost. The Sith armor, for example, becomes almost entirely invisible because of all the shiny parts that are rendered transparent.



#12 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 10 August 2017 - 08:46 AM

Interesting. I thought that since normal maps are stored in separate files, reflection maps and such would also be stored in separate files. I will think about how to handle this. Most likely, I will make an option to extract the alpha channel as a separate gray scale image and then add the possibility to provide two PNGs to create a new TPC file.

 

For now, this work-around in Photoshop might work:

  1. Open the converted PNG
  2. Menu: Layer->Layer Mask->From Transparency
  3. Alt-LeftClick the mask
  4. Ctrl-a Ctrl-x to cut the mask out. This will make the mask completely white, meaning that you can see all that info that the alpha channel was hiding
  5. Menu: Layer->New->Layer
  6. Ctrl-v to paste the mask into the new layer
  7. Edit the image and alpha layers to your hearts content
  8. With the alpha layer selected, ctrl-a ctrl-c
  9. Alt-LeftClick the mask of the image layer
  10. Ctrl-v to paste in the edited alpha channel
  11. Hide the "alpha" layer
  12. Save as PNG
  13. Use the tpc tool to create a new TPC file based on the new PNG file

I am not 100% certain that this will be perfectly accurate, but I think that it should work even if it is a bit clunky. Also, the tpc tool still can't create DXT-compressed images.

 

@sELFiNDUCEDcOMA about the GUI

Yes, that is one of the main points of this exercise: The code is structured so that there are libraries that handle the file formats and separate console applications for the tools I provide. Anyone could use the same libraries in their own GUI application directly, or call the console applications when needed. Many options are available. :D

 

@Malkior

The TXI can say many things. Most, as I understand it, has nothing to do with the alpha channel. Is there a list somewhere that explains what the various TXI attributes do? I have guessed many, but it would be nice to build a comprehensive list in a public space.



#13 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 10 August 2017 - 10:16 AM

There's likely a few solutions to add TGA support to your tool.

That looks really interesting, perhaps I should make a fork of CSharpImageLibrary, adding TPC support, and then use that in the tpc tool. It even has DXT compression done. :)



#14 Malkior

Malkior

    Jedi Master

  • Members
  • PipPipPipPipPip
  • 1,099 posts

Posted 10 August 2017 - 10:48 AM

@Malkior
The TXI can say many things. Most, as I understand it, has nothing to do with the alpha channel. Is there a list somewhere that explains what the various TXI attributes do? I have guessed many, but it would be nice to build a comprehensive list in a public space.


As far as cubemaps are concerned, the TXI uses the Alpha layer of the texture to determine what is shiny and what is not. Without an Alpha layer, the texture with a cubemap applied to it either by the TXI or Appearance.2da will apply it to the entire object.

As far as what the TXI functions are, I only know of very few. There is, however, a topic on this forum devoted to listing them all that may be of some assistance.
____________________________
|_______________________>>___\
....But I digress....
__________________________
|____________>>_______>>__/

#15 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 10 August 2017 - 11:07 AM

As far as cubemaps are concerned, the TXI uses the Alpha layer of the texture to determine what is shiny and what is not. Without an Alpha layer, the texture with a cubemap applied to it either by the TXI or Appearance.2da will apply it to the entire object.
As far as what the TXI functions are, I only know of very few. There is, however, a topic on this forum devoted to listing them all that may be of some assistance.


It would surprise me if the presence of the "cube map" TXI-attribute implies that the alpha is a "shininess" map. I am pretty sure that I have seen specific attributes for that and, until now, I didn't know what they did. I will examine this when I get back to my computer.

I would appreciate a link to that forum thread. I tried to search for it but it is difficult on the phone.

#16 DarthParametric

DarthParametric

    Dark Lord of the Sith

  • Members
  • PipPipPipPipPip
  • 1,179 posts
  • LocationOz

Posted 10 August 2017 - 11:13 AM

The alpha should always be a transparency mask. The TXI semantics just tell the engine whether or not to blend a second texture, like an environment/cube map.

#17 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 10 August 2017 - 11:21 AM

That is what I thought too.

#18 JCarter426

JCarter426

    Jedi Knight

  • M478_Staff
  • 289 posts

Posted 10 August 2017 - 01:11 PM

Yes, the actual shininess or whatever is a separate texture (cm_baremetal, for example) but essentially the game can either have a shader or transparency, but not both. The alpha channel is used for either, depending on the situation. And actually you can't just use the alpha channel for transparency, because then it won't render any objects behind the texture for some reason, so you still need a shader for that.

 

For now, this work-around in Photoshop might work:

  1. Open the converted PNG
  2. Menu: Layer->Layer Mask->From Transparency
  3. Alt-LeftClick the mask
  4. Ctrl-a Ctrl-x to cut the mask out. This will make the mask completely white, meaning that you can see all that info that the alpha channel was hiding
  5. Menu: Layer->New->Layer
  6. Ctrl-v to paste the mask into the new layer
  7. Edit the image and alpha layers to your hearts content
  8. With the alpha layer selected, ctrl-a ctrl-c
  9. Alt-LeftClick the mask of the image layer
  10. Ctrl-v to paste in the edited alpha channel
  11. Hide the "alpha" layer
  12. Save as PNG
  13. Use the tpc tool to create a new TPC file based on the new PNG file

I am not 100% certain that this will be perfectly accurate, but I think that it should work even if it is a bit clunky. Also, the tpc tool still can't create DXT-compressed images.4

 

Ah, that does look like it works. I should have figured since it was still saved as a separate channel, there would be a way to unbake the cake here. However, I'd still be interested in a more automated process, for two reasons. First, I wrote a batch script to mass convert every TPC file in the game... but having to go fix the alpha channel after that doesn't really save time in the end. So that would certainly help. The other thing is that 3ds Max reads the games' alpha channels as white, which is... not what it should be doing. I'm not sure if it can't figure out how to use the alpha channel properly, or if it just prefers having separate textures, but I've never cared to find out and I end up separating it into two textures anyway when I'm rendering in it. So TPC to 2 PNGs would definitely speed things along there.

 

I look forward to any updates. Thanks for continuing to make tools for a game that's now older than this generation of players. :)



#19 ApanLoon

ApanLoon

    Jedi Apprentice

  • Members
  • PipPip
  • 14 posts
  • LocationSweden

Posted 10 August 2017 - 01:17 PM

I am pretty sure that you could use imagemagick for making a script that does the same thing as that workaround. Inagemagick works well in Cygwin under Windows or in Linux/OS X.

#20 JCarter426

JCarter426

    Jedi Knight

  • M478_Staff
  • 289 posts

Posted 10 August 2017 - 01:33 PM

Ah, I've never used that. I'll take a look.






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users