A new converter for Shenmue geometry including textures

[south carmain]

hey reece which files did you extract to get dobuita?

[ReeceKun]

hey reece which files did you extract to get dobuita?

This is a brief list of all the scene files from all 4 discs combined (may not be 100% accurate)


You want folder D000 (there are other Dobuita folders but they are cutscenes or beta areas)
MPK00.PKF/PKS is the Main Doubita Map (there are loads of other PKS/PKF in the folder too = vehicles, bus, outer map areas, coke machines etc)

It has multiple layers for each area , and there is a lights layer, Xmas decorations layer, just tons o stuff, that may or may not appear depending on the time of day/year.

[south carmain]

thanks for some reason only the smaller layers were working when I converted them, turns out it was only a problem with C4D though.

[ReeceKun]

This is just a note for hellwig74 (if he ever comes back) and anyone else who rips models from MT5s

Before the unpacker existed I created a 'Quick BMS' script to extract MT5s from the PKF/PKS archives

Code: Select all

findloc OFFSET string "HRCM"
math i = 0
do
    goto OFFSET
    get DUMMY long
    findloc NEXT_OFFSET string "HRCM" 0 ""
    if NEXT_OFFSET == ""
        get SIZE asize
    else
        math SIZE = NEXT_OFFSET
    endif
    math SIZE -= OFFSET
    string NAME p= "%08x.MT5" i
    log NAME OFFSET SIZE
    math i += 1
    math OFFSET = NEXT_OFFSET
while NEXT_OFFSET != ""


This basically extracts the MultiMt5s into seperate ones by finding the start of the HRCM header to the start of the next one. ( Im sure you know this I just like to explain all details)

MT5Converter is great for converting the base MAPM files, But it wont convert the single extracted MT5 files (that you have output as CHRM) It stops at the MTL phase

Long story short : My unpacking method is better for converting the other 'CHRM' files
Why?...

I tried converting one of these CHRM with YMT5 ripper and it didnt work, but I knew I'd already converted the file in the past using my 'unpacking method'.

I dont know where an MT5 file is supposed to end but when converting with YMT5 adding or deleting a few '0000 0000 0000 0000' in hex makes the difference between the file being converted or not. (which just boggles my mind)

So by adding the 0000s to your CHRM your file converted with YMT5. but the output was identical to the file i'd unpacked.

YMT5 also rips the MAPM types too but MapConverter is far more accurate and gets 'all geometry'' I would certainly like to see a new MT5Converter, if not I am still very grateful for your contribution to the cause

[dotsona]

Wow this is amazing. I haven't posted on here in a long time but its amazing to see we finally have a tool for ripping models. In the past others have tried to make a tool for ripping mt5 (which kinda worked but had major bugs) but nothing has been as complete as this. Back in the earlier days 3d ripper was the best thing we had. Great work!

Would be nice if someone could organize the models by folder and release them publicly so no one has to rip the models manually. Also is there any plans on exporting the character rigs too?

[ReeceKun]

Would be nice if someone could organize the models by folder and release them publicly so no one has to rip the models manually.

This is exactly what im doing, I'm not sure if hellwig74 is ever going to return, BUT other than unpacking 'extracted' PKS/PKF files, its not much different to YMT5 ripper which still remains an important Tool.

My biggest frustration with MT5Converter is if it detects a single thing wrong it will crash , and it usually crashes at the texture uv mapping stage. I know MT5 Converter rips more model data than YMT5 ripper (and YMT5 sucks at getting all the textures too) I would be happy with just the base model data instead of the program crashing and outputting nothing.

However between MT5Converter , YMT5 ripper & 3d Ripper DX I have all the tools to rip everything, It will just take a bit longer than just using 1 program to magically rip everything perfectly placed and perfectly textured.
(Note:MT5Converter Scale is exactly 3937% bigger than YMT5 models, this helps to combine meshes)


I also have a good idea to rip the NPC characters (however they all all need combining into a single mesh and not overlapping bodyparts)

[-ThreeBlades-]

Bit useless, but I obviously found some use in it.. It's a simple batch script to automate the decompressing, unpacking and converting of various file types. I found it helpful when I wanted to extract multiple models using this method but not have to process each file manually.

1. Extract the archive somewhere.
2. Copy all of your files you wish to decompress, unpack & convert to the root folder of the newly-created "Shenmue_Model_Extractor" folder.
3. Run the 'RUNME.bat' batch script and follow the prompt(s) as necessary.

Download link

NOTE: If any updates for the above tools are published then you can simply replace the executables and the script should still work, as long as the parameters do not change.

EDIT: Here's a MAXScript script that will loop through all of the objects you currently have selected and if they have a material with a diffuse map assigned it will copy that diffuse map over to the opacity map and set the Mono Channel Output to Alpha and the Alpha Source to the Image Alpha accordingly. This was useful when previewing the extracted foliage meshes along with their extracted materials, now simply selecting them directly after extraction and importing into 3ds Max, you can get a relatively good representation in a couple of seconds. The script also supports Multi Materials, just in case the unpacker/extractor end up utilising them. Otherwise, I'm sure the functionality will be useful to someone, somewhere!

Code: Select all

MaterialArray = #()
MaterialCount = 0

for obj in selection where (mtl = obj.material) != undefined do (
   if (classOf mtl == multiMaterial) then (
      for m in mtl.materialList do (
         append MaterialArray m
      )
   ) else (
      append MaterialArray mtl
   )
)

for mtl in MaterialArray do (
   if(hasProperty mtl "diffuse") then (
      print(mtl.name)
      mtl.opacityMap = COPY mtl.diffuseMap
      mtl.opacityMap.monoOutput = 1
      mtl.opacityMap.alphaSource = 0
      MaterialCount = MaterialCount + 1
   )
)

print("Processed " + MaterialArray.count As String + " materials.")

[BluePink]

Thank you so much for this little gem, It has helped me no-end since i got it. and its given me more reason to finish what i started got knows how long ago.

Chris.

[Yokosuka]

I found this on a tumblr


It would be fantastic if we could have the same scale models for Shenmue II locations.

[Esppiral]

You found it on tumblr? This image came from the dojo all we love

viewtopic.php?f=37&t=47268&start=15

[ReeceKun]

I contacted Heliwig yesterday and he is still around but it sounds like he just hit a brick wall , I asked for the sources so here is the link, Ive also added the Yazgoos sources too since it works with different files.

SHENMUE 1 MT5 CONVERSION SOUCRES (For anyone with Model Extraction Knowlege)
Yazgoo's (only works with individual MT5s)
https://github.com/yazgoo/mt5_extraction_tools
Hellwig's (only works with MAP files found in compressed PKF/PKS)
https://github.com/hellwig/shencon

Its possible with the increased Shenmue attention someone new will have a look at the model files.
Between both converters all of the Building & Object Model data can be ripped perfectly.
Texture UV placement arnt perfect but since everything needs retexturing anyway it doesnt bother me.
The most deformed objects are Characters,NPCs and Kitty Kat Meshes.

The Model data in general is a massive cluster fuck, the game loads AFS containing PKS containing MT5 to render models. MANY Models are duplicated with different File scructures (which is fortunate because one converter may work where the other doesnt)