2015 Thoughts On MT5/MT7 Model Structure (Unkown Factors)

[ReeceKun]

I came across a perplexing conundrum today and it might explain something to do with why MT5 model extraction has been so difficult. And why textures are never 100% placed (Even when using 3D Ripper DX)

Today I was wondering if I should try to preserve the original texture placement while im re-modeling.

Ingame the Road Texture looks like this, NOTE: The Road Texture is Mirrored


But In 3ds max although the correct texture is applied it looks like this


This is a Model from Mafia NOTE: The Polygons are Mirrored


This is How UV Maps are apllied to Models


UNLESS I am a complete moron (which is possible) I can not Tell 3DS Max to Mirror the texture at an abitary midway point. WHICH CAN ONLY MEAN That the people who designed the MT5 structure were utterly brilliant. They found a way to MINIMISE Polygon count by including information 'somewhere' (In the MT5?) That tells the Shenmue Game Engine To create this Mirrored effect , which could explain why the MT5 has been so difficult to analise. IT ALSO MEANS that even IF a 100% Model Extractor was created The Textures would NEVER render as seen ingame in any 3D Modeling Program

Personally since I will be applying new textures all I need is the Model Data and with a few minor exceptions I already have. I feel its worth noting these kind of thoughts anyway if it helps future model extractors (or just gets people interested in having a go)

Also when you compare the MT5 Pheonix Mirror In Shenmue 1 with the MT7 Pheonix Mirror in Shenmue 2 in Hex Workshop, Although they are slightly different it looks like there is alot of similar data, so I would request that even if non textured Models could be extracted it would be a HUGE help for us all

IF I am just talking crap I will happily re-edit and or delete this thread

[TapamN]

The Dreamcast's 3D processor has a mode that can mirror textures. It's not on an arbitrary location, but whenever the texture repeats. This can be done on one or both axis of the texture. It is exactly equivalent to the following in OpenGL:
glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_MIRRORED_REPEAT);

So if this is your entire texture:
/

Instead of repeating it and getting
////////
////////

It will mirror it each time it repeats and you can get
/\/\/\/\
/\/\/\/\
or
/\/\/\/\
\/\/\/\/

The ripper tools either aren't recording the flip flags, or the model format the ripper generates doesn't support texture mirroring. The original MT5/MT7 will have the flags stored somewhere in them. The "Ninja" model format, used in the DC Bios, Sonic Adventure, Skies of Arcadia, Rez, and more, has what is almost a raw hardware command stored in it to setup how polygons are drawn; Shenmue probably does something similar.

I think Sega used some version of Lightwave for modeling, so Lightwave at least probably supports texture mirroring. Even if a modeling program or format doesn't support mirroring, it would still be possible, with a little extra work, for a convertor to work around it by subdividing polygons with mirrored textures and baking in the UV mirroring manually.

[ReeceKun]

Im just gonna paste this here:

SHENMUE 1 MT5 CONVERSION SOUCRES (For anyone with Model Extraction Knowlege)
Yazgoo's (Mainly works with individual MT5s)
https://github.com/yazgoo/mt5_extraction_tools
Hellwig's (Mainly works with MAP files found in compressed PKF/PKS)
https://github.com/hellwig/shencon

I understand the basics of what your saying and this kind of mirroring is certainly outdated (and i think the classic tomb raiders used the same form of texturing because they didnt use UV maps at all) I think expecting someone to go even further to support this mirroring is a bit too much to ask.

What would be great (especially with MT7/Shenmue Extraction) Is for a ripper to simply identify each polygon and coordinate location for each model and convert them into OBJ (since I dont need UV maps) even if models outputted hundereds of polygon objects they could still be merged in 3ds Max afterwards.
I have already started the basics of looking at model formats in Hex to identify 3D structures but It will be some time before I learn to make a program to extract them

I'm convinced MT5Converter can get all the model structure data from all models now, but it basically refuses to convert if it gets confused at the texture stage, IF someone understands the sources and can bypass the UV texture stage alltogether It would save me a lot of time

[ReeceKun]

I've been getting some amazing rips today

It might not seem amazing but each mesh was output in the right position

When I see stuff like this It really depresses me how close MT5 conversion is to getting everything 100%, it just needs some tweeking , I cant really speculate on MT5 structure but you'd think that the files that convert better (or more complete) would reveal 'what works' when you compare the structure with files that dont.


This is a comparison of The same 2 models (Character Above) but from different game folders, The structure is identical up untill the Texture Stage , One works in MT5Converter but the other Crashes when it gets to the texture stage. YMT5 will convert both but the end result is all fudged up. I have sooo much stuff to do , but hopefully I can use this comparison info to get all the characters to convert , I only really need their faces (since their body structure is ancient and obsolete) and im sure plenty of characters use the same base model

EDIT: Having said that all of the 'Character' faces do convert properly , Its just the 'Street' NPCs that have a different Facial Animation (Mesh Swap) to talk, so Their Mouth Flap is usually Fudged because their are muliple mouth meshes (they swap to talk)

[Fenix]

You mean the extractor can actually export models without destroying them? That's a huge improvement.

[ReeceKun]

You mean the extractor can actually export models without destroying them? That's a huge improvement.

MT5 Convert has been available since Fri Nov 14, 2014, there havent been any recent updates.

YMT5 attempts to convert all models but has frequent errors
MT5Converter is 98% perfect but refuses to complete the export if there is a texture error/confusion.
Combine them both and you get most model structure intact (with the exception of character/NPC files)
3DRipperDx can fill any gaps.
Once I've created a complete archive of every model these converters will be obsolete.

[Fenix]

You mean the extractor can actually export models without destroying them? That's a huge improvement.

MT5 Convert has been available since Fri Nov 14, 2014, there havent been any recent updates.

YMT5 attempts to convert all models but has frequent errors
MT5Converter is 98% perfect but refuses to complete the export if there is a texture error/confusion.
Combine them both and you get most model structure intact (with the exception of character/NPC files)
3DRipperDx can fill any gaps.
Once I've created a complete archive of every model these converters will be obsolete.

I was refearing to this. Actually, I know maps have pretty good results.

[ReeceKun]

Apparently Yazgoo returned to object extraction and it looks like he cracked MT7 extraction , I dont really understand due to the language barrier but this happened LAST YEAR and I only found out today .. I really really wish we could get someone 100% committed to model extraction

A google translated forum where Yazgoo revives the thread to announce his discovery
https://translate.google.co.uk/translat ... rev=search

A strange online MT7 model viewer video
https://translate.googleusercontent.com ... h0KEOm4aOQ

Github sources
https://translate.googleusercontent.com ... Ahs7HhQ8Sw

A strange forum post with a working MT7 extractor that only works on 1 MT7 file?
http://forum.xentax.com/viewtopic.php?f=16&t=11482

[IlDucci]

A strange forum post with a working MT7 extractor that only works on 1 MT7 file?
http://forum.xentax.com/viewtopic.php?f=16&t=11482

SkyBladeCloud's script only managed to extract one set of detailed guide maps (Not actual stages), but in the end Xentax gave no results and I ended up editing the vertex by hand, with an hex editor, for the Spanish translation.

[ReeceKun]

A strange forum post with a working MT7 extractor that only works on 1 MT7 file?
http://forum.xentax.com/viewtopic.php?f=16&t=11482

SkyBladeCloud's script only managed to extract one set of detailed guide maps (Not actual stages), but in the end Xentax gave no results and I ended up editing the vertex by hand, with an hex editor, for the Spanish translation.

How difficult was it to locate the right vertex?
Im wondering how easy it would be to take a MAP.MT5/7 or a charcter.MT5/7 and delete elements OR take objects from a map or character and resave it as smaller MT5/7 file

[IlDucci]

As difficult as having to edit one vertex, then pack things up, then emulate the game to see where the hell I was touching. Pure trial an error. Thankfully there isn't more than a 100 vertex per model. In hexa I did not knew jack. Also, the floats for each vertex seemed to be a multiple of something I didn't know. Most of the times, adding a 0,00001 would make the game crash.

XBOX, on the other hand, has a totally different vertex structure than Dreamcast. DC seems to have an unique format (Based on pieces, per example, a square would have in its own area the four vertex positions, UVs and other normal data), while XBOX seemed to be more standard (Sections segregated by use, UV area, vertex area, normal area). Again, fortunately, the coordinates were exactly the same between formats, it just has them in different structures.

I don't know much more about 3D hacking though.

[ReeceKun]

This is the BEST character rip I have ever found


What's cool is it shows the entire Bone Structure rotation points.
(if you load the DAE with 'bone & animation) it roatates the pivot points on import)
I personally never intended to use the original bone structure but If someone could understand the .MOT file this would be the Perfect basis to understand the relation between character models & their animations. The Sequence ATH files relate only to the camera controls (im 90%)
The missing mesh parts are the meshes that deform which Model rippers have a big problem with

When it comes to MT5s it seems that their isnt a 100% standard form, which also seems to be the case for Shenmue 2's MT7 if you look at the headers some are MDP7 MDC7 etc , which is why (Heliwigs) MT5Converter doesnt work for all files, my guess is there is a lot of experimental stuff that was left in this game, for example if they decided to have meshes overlap in animations or morph/deform. (They didnt go back and standardise all models animations etc)

I've have converted some MT5s by altering their footers and Im trying to develop a way of altering MT5s in HEX inorder to get them to rip as best they can YMT5 ripper seems to rip as much as it can but MT5Converter just crashes if it encounters something it doesnt like , I just wish there was more information on understanding exactly what it doesnt like before it crashes (I sometimes wonder if its a memory issue for the exe itself)

Ultimately I have all the tools to 100% rip all the models, I feel its worth mentioning my musings however innacuurate for future generations to ponder, because once I have everything i wont look back to Shenmue Modding at all.

Secondly as I work though the original files I am creating a file to document all files and what they contain, I wanted to be able to create a graphical text representation of the file structure and the best I have found is making a .H file eventually I might convert it to a html file.

Code: Select all

[HIHY                     // YOUNG RYO BEATEN IN DOJO FLASHBACK
HIHY.PKF                  // 19 PVR TEXTURES + CHRT
   CHARA.CHRT               // SCENE OBJ CHART
   tex_54415749_52.pvr         // IOWA HANDS
   tex_5f414b97_5f464f4b.pvr   // 
   tex_5f414b97_5f465254.pvr   // 
   tex_5f414b97_5f46554d.pvr   // 
   tex_5f414b97_5f495341.pvr   // 
   tex_5f414b97_5f4d414b.pvr   // 
   tex_5f414b97_5f4f414b.pvr   // 
   tex_5f415749_4d5f4954.pvr   // 
   tex_5f415749_4d5f4f54.pvr   // 
   tex_5f415796_5f445241.pvr   // 
   tex_5f415796_5f46554d.pvr   // 
   tex_5f415796_5f49424f.pvr   // 
   tex_5f415796_5f495341.pvr   // 
   tex_5f415796_5f4a414b.pvr   // 
   tex_5f415796_5f4d414b.pvr   // 
   tex_5f415796_5f4d4d55.pvr   // 
   tex_5f418996_5f4a414b.pvr   // 
   tex_92caabd7_5f423261.pvr   //
   tex_c4c7a97e_5f6f79da.pvr   //

HIHY.PKS                  // 7 MT5 3 BIN 1 MOT 1 ATH
   IWA_F.HCM                   // IOWA FACE
   IWA_M.HCM                   // IOWA BODY MT5 MAIN MODEL (INC BONE PIVOT POINTS)
   IWA_TL.HCM                  // IOWA LEFT HAND
   IWA_TR.HCM                  // IOWA RIGHT HAND
   JKA_F.HCM                   // YNG RYO FACE
   JKA_M.HCM                   // YNG RYO MT5 MAIN MODEL (NO MT5RIP)
   MAP12.MAP                   // MT5 DOJO INTERIOR MODEL
   IWA_FTBL.BIN                // IOWA FACE MORPH COMMANDS
   IWA_HM.BIN                  // IOWA (HAND MORPHS?)
   JKA_FTBL.BIN                // YNG RYO FACE COMMANDS
   M_0125.MOT                  // ANIMATION COMMANDS
   SEQDATA5.ATH                // CAMERA COMMANDS
   ]