Random CSCI Adventures with VP

[Fair Strides 509]

Update: Well, after implementing what I thought would be a faster alternative to Lists, I found out that the main cause of the terrible performance was with one particular chunk of terrible code. After fixing that, performance was pretty damn fast.

 

From my console output:

 

651 ms to read GFF

288 ms to write XML

190 ms to write GFF

 

I'll have to try a bigger file later.

 

...Maybe it's just the file size itself, but this could be a classic example of speed inequality between languages. It seem it'd be longer then a second to do it in Perl, though that could just be because I'm not too aware of the format and there's a big difference between the GFF files, say the .are and the .fac; the size and complexity are very different.

[VarsityPuppet 1,085]

...Maybe it's just the file size itself, but this could be a classic example of speed inequality between languages. It seem it'd be longer then a second to do it in Perl, though that could just be because I'm not too aware of the format and there's a big difference between the GFF files, say the .are and the .fac; the size and complexity are very different.

 

Could be. I mean, I don't think managed to do anything particularly fast, clever or efficient with my code. In my research, I read that Hashes aren't necessarily the fastest in terms of writing things with indexes (indices?) involved, which may or may not be the case with .tlk files. Turned out it was useful for GFFs.

 

I haven't had the chance to test any module files, but I've been testing the "basic" gff templates along the way to ensure I was reading/building correctly.

 

My tests:

 

Saber UTI file (5 kb)

 
131 ms to read GFF
36 ms to write XML
32 ms to write GFF 
2 ms to format XML

Area GIT file (14 kb)

 

146 ms to read GFF
37 ms to write XML
44 ms to write GFF 
3 ms to format

Kreia DLG file (1.4 MB) (Biggest DLG file I could find in the override)

 

4362 ms to read GFF
1257 ms to write XML
2052 ms to write GFF 
499 ms to format

[LiliArch 115]

Random irritated complaining: Why on Earth I have recalculated Y-values of tverts? I didn't change the _height_ of the texture image... no wonder there's white stripes in wrong places.

 

 

Well, the most current info, while I buy myself time to dig into the guts of MDLOps' source code(it is Perl, so I'm on my own here),

 

The two sources of info on it I've found are by the maker of MDLOps himself:

http://cchargin.home.comcast.net/~cchargin/kotor/mdl_info.html

 

and some guy I don't know named Torlack(from 2002, so it's for NWN. You'll need to copy it and make an html document on your computer and then paste the info in there...):

https://github.com/xoreos/xoreos-tools/blob/master/docs/specs/torlack/binmdl.html

Thanks! I'm not perfectly sure if you are actually planning to dig into the MDLOps or am I just irritating you, but those links are really useful. Maybe I'll be able to recalculate those odd values with them... though if I have to manually go through 5 000 entries in mdx file, I think I won't be doing it anytime soon. If it just would be possible to copy text from the MDLOps' tree view, I could use kdiff3 to check what's different, but no. Of course I could kdiff the hex values, hmm...

 

In case someone's interested:

I compared in the said MDLOps' tree view a compiled unmodified model, a compiled modified model, and the original model, and tried to change only the values that were different in the first of two, using the third as a reference. But MDLOps seems to change a lot of values from their original ones, replace a lot of another values with zero... and remove some altogether. Maybe I should try KAurora?

[VarsityPuppet 1,085]

Random irritated complaining: Why on Earth I have recalculated Y-values of tverts? I didn't change the _height_ of the texture image... no wonder there's white stripes in wrong places.

 

 

Thanks! I'm not perfectly sure if you are actually planning to dig into the MDLOps or am I just irritating you, but those links are really useful. Maybe I'll be able to recalculate those odd values with them... though if I have to manually go through 5 000 entries in mdx file, I think I won't be doing it anytime soon. If it just would be possible to copy text from the MDLOps' tree view, I could use kdiff3 to check what's different, but no. Of course I could kdiff the hex values, hmm...

 

In case someone's interested:

I compared in the said MDLOps' tree view a compiled unmodified model, a compiled modified model, and the original model, and tried to change only the values that were different in the first of two, using the third as a reference. But MDLOps seems to change a lot of values from their original ones, replace a lot of another values with zero... and remove some altogether. Maybe I should try KAurora?

I don't know if KAurora will work, so you can certainly give it a try.

 

My guess as to why your method may not be working may be because of the way the model is being recompiled. It's possible (and all too likely) that some offsets are being rearranged upon mdlops compiling (at least in comparison to the original mdl). Best bet might be to just play with mdl_info (Fair Strides first provided link) and just start trying to puzzle together the file format.

[Fair Strides 509]

Random irritated complaining: Why on Earth I have recalculated Y-values of tverts? I didn't change the _height_ of the texture image... no wonder there's white stripes in wrong places.

 

If it just would be possible to copy text from the MDLOps' tree view, I could use kdiff3 to check what's different, but no. Of course I could kdiff the hex values, hmm...

 

I could throw a hack into MDLOps for you that would output the hex data to a file, as you click on the node. The only downside is that you will have to click on each node and there's no safety against clicking the same node twice, so the output could get very screwy very fast...

[LiliArch 115]

My guess as to why your method may not be working may be because of the way the model is being recompiled. It's possible (and all too likely) that some offsets are being rearranged upon mdlops compiling (at least in comparison to the original mdl). Best bet might be to just play with mdl_info (Fair Strides first provided link) and just start trying to puzzle together the file format.

Well, I was mostly using the original mdl to check if a value was same or different than the one in compiled, unmodified model (if it was changed in the modified model, that is, no need to check original values if they are not changed, as there is no need to do nothing with them) and if it was not, trying to figure out what to do with it. So it may be I messed something there, or maybe I just missed some value, which might be even more likely...

 

 

I could throw a hack into MDLOps for you that would output the hex data to a file, as you click on the node. The only downside is that you will have to click on each node and there's no safety against clicking the same node twice, so the output could get very screwy very fast...

I can get the hex data, that's no problem. But if you could make it output the dedimal values in a file, that would be handy. You know, the stuff that is in the rightmost... window? (well, it's not really a window, as the whole thing is in one window, but I can't recall the correct word for it, it's been a while since I've been working with GUIs that are not web pages) of the tree view.

[VarsityPuppet 1,085]

Well, I was mostly using the original mdl to check if a value was same or different than the one in compiled, unmodified model (if it was changed in the modified model, that is, no need to check original values if they are not changed, as there is no need to do nothing with them) and if it was not, trying to figure out what to do with it. So it may be I messed something there, or maybe I just missed some value, which might be even more likely...

My bad.

 

Only issue I could see with changing the hex to all decimals is that you'll probably get some strange numbers every once in a while since some of them, for instance, could be float/double representations. Heck, even some of them are boolean flags.

[VarsityPuppet 1,085]

Into Form Application phase! woop woop!

 

[peedeeboy 23]

I thought having a crack at parsing 2da v2.b files would be good Java revision.:

 

Results:

 

Ubuntu

 

Windows

 

And I thought you might find the following useful in your future endeavours, incase you want to build tools for 2da files:

 

Most of the necessary info is in tk102's post here: http://www.lucasforums.com/archive/index.php/t-178681.html

 

But there a couple of bits slip up on:

 

• The integers in the file (rowcount, data offsets) are in little endian byte order
• The native 2da v2.b files appear to be formatted for memory optimisation.  There is not one one string in the data area for each piece of data - rather several offsets might point to the same piece of data.
• E.g. In the modulesave.2da, several rows might have 6 in the SaveGroup column.  Rather than 6 being stored in the file several times, it will be stored once, and all the offsets for those cells will point to the same instance of 6.

 

Hope that helps!

[VarsityPuppet 1,085]

But there a couple of bits slip up on:

• The integers in the file (rowcount, data offsets) are in little endian byte order

•  

Yep. GFF files were all little endian too.

 

• The native 2da v2.b files appear to be formatted for memory optimisation.  There is not one one string in the data area for each piece of data - rather several offsets might point to the same piece of data.

•  

 

Good to know. GFFs did this with labels, but not with any field values. I'm sure they were just too disparate of values to really even bother. Most of the values that would have mattered anyways fit into 4 bytes anyways

 

Hope that helps!

 

I will most certainly take a look at that! Thank you!

 

Any chance you've written any code to add rows and then write the new file? I'd be real grateful if you could mock up something like that. Looking for just a really "ragtag" way to finish up my Saber mod without any more tedious work - well, tedious file editing, anyhow.

[peedeeboy 23]

 

Any chance you've written any code to add rows and then write the new file? I'd be real grateful if you could mock up something like that. Looking for just a really "ragtag" way to finish up my Saber mod without any more tedious work - well, tedious file editing, anyhow.

 

I'll have a crack at it!    I'm still very much learning as I go too!  When you say you want to add a row, do you mean to the bottom of the table, or in the middle of other rows?  Updating row headers if inserting into the middle of the table will be tricky as there is no guarantee that they are numbers .

 

There's a couple of ways to write the file :

 

1. The slightly less migrane inducing way: write an offset and piece of data for everything in the 2da (tk102's Excel Add-in does this)
2. Attempt to replicate the compression / optimisation in the native 2da files (not sure if KotoR tool does this - need to check)

 

Trying to insert new data into an existing 2da without completely rewriting the offsets, may cause nervous breakdown!

 

I'll probably try 1 first, then have a crack at 2.  No point in re-invented the wheel when tk102's Excel addin already does 1 perfectly!

 

I also want to add in:

 

• the ability to edit row / column headers
• create a brand new 2da from scratch
• open regular (text version) 2da files

 

Should keep me busy for a couple of weeks

[VarsityPuppet 1,085]

Go crazy with it! For my purposes, just to add a row to the bottom. Preferably, something that can read a text file and generate a row (or several rows) based on that input.

[Sithspecter 909]

I don't quite understand all the technical aspects of this as I'm a slightly more artsy type, but good work here! I'm liking where this is going.

[VarsityPuppet 1,085]

I don't quite understand all the technical aspects of this as I'm a slightly more artsy type, but good work here! I'm liking where this is going.

Well, if we have any tools you might find useful (or if you have any suggestions for tools) feel free to post here.

 

Keep in mind though it might be awhile until we get to things like... models

[VarsityPuppet 1,085]

Alright! I'm going to be Marathoning today! I want to release a solidly working version of my new tool BigGFFer by 4 o'Clock Central time. That's a little more than 8 hours from now.

 

I'm trying to kill time waiting for my new iOS tools to download. Upgraded my Mac to 10.9 Mavericks, upgraded my XCode to version 5. I'm probably going to need to upgrade iOS on my iPhone. All so I can get my Sensus devkit in some working format for GDC.

 

Alrighty! We'll see how this goes.

 

By the way - Malachor VI source Code is live!

 

http://deadlystream.com/forum/topic/32-malachor-vi-an-ending-mod/?p=25675

[VarsityPuppet 1,085]

Update on that new tool:

 

Eh, I'm close. I was doing alright, but then there was a power surge and all the power went out. I didn't lose any progress, but now I have to catch up on homework because my procrastinating ass wanted to write code all spring break.

 

But, I ended up with this!

 

Yeah, that's me. Don't make fun of my accent.

 

https://www.youtube.com/watch?v=eP724DnP9nE

[VarsityPuppet 1,085]

Reassignment!

 

 

I'm refocusing my efforts into Duplisaber so I can get that mod out. I have pretty much everything I need, I just need to monkey around with the data now.

[VarsityPuppet 1,085]

Took quite a bit longer than I thought it would, but I have a working 2da reader/writer.

 

I'm that much closer to the desired result.

 

I can read and write to 2DA files

I can read and write to GFF files (It's going to be helpful for DLG files)

 

It's only a matter of time.

 

Time for bed.

[Darth Hayze 19]

What is the difference between your reader/writer and the one that comes with KT?

[VarsityPuppet 1,085]

What is the difference between your reader/writer and the one that comes with KT?

 

Kotor Tools actually has a visual interface lol.

 

What I mean when I say "Read and Write" is that it can read the file's byte code and turn it into recognizable data and write itself back into a working file.

[Darth Hayze 19]

Ok

[LiliArch 115]

I think there is a tool to be made, but I need a bit help with it. Anyone with 3DS Max, who would be willing to give me a hand?

[Fair Strides 509]

I think there is a tool to be made, but I need a bit help with it. Anyone with 3DS Max, who would be willing to give me a hand?

I would need more details, but I'd be willing to at least advise you on it. I have 3DSMax 2011, and I can make models work in-game, most of the time.

 

What do you need? If necessary, you can PM me...

[LiliArch 115]

Of course you need. I'll PM you the details. I'm not sure if it is absolutely necessary to use PMs, but I like to make surprises to people, so I don't want to give it all away before it is done.

[VarsityPuppet 1,085]

It won't be long now! I've done most of the heavy lifting for Duplisaber.