More info about Scenes/MT5- greetings from AM2

[Master Kyodai]

OK some news about MT5 format.

I have been quiet in the past few weeks, i know, but i have not been completely asleep.

I have come to a point where i can't understand some stuff i found so i think Yazgoo needs to take over, or someone else who is more familiar with these topics.


At first - i got some files with original code from AM2 from the scene compiling process. I am sure i can get more of them by investing more time, so here is what you get, i made a zip archive containing 2 examples


--> Download

http://www.tankraider.com/userup/1264542643.zip


And here is the whole stash including mt7 examples and stuff from shenmue 2:

http://www.tankraider.com/userup/1264579444.zip


OK first lets discuss about what's inside the archive and why i think it's valuable. At first it's a gift from AM2. I am not sure i understood everything as it's all very japanese, but also some delightfull surprises.

Lets first take a look at the .txt file. if i got it right this is the original log file from creating a "scene package". Why is this interesting? As it shows they simply used gzip to zip together some MT5 files. And as we know mt5 is good as we can extract it with yazgoo's method.

So whats the conclusion from the .txt file? If we look closely we notice that AM2 did more. In a first run they seem to gzip the mt5 files together, then they run a special script to add Motion files. As i understand it a scene does not necessarily need motion files, but here we got an example where motion files were added.

The next interesting file is the .chr file. This is the definition file they used to compile the file. at first this seems quite random, they just assign 4 characters to each mt5 model, but they also assign an offset to the file.

So this is where these mt5 files are placed in space of a scene.


Yazgoo, i hope you read this. It might not be much, but i believe this brings us much closer to understanding MAPINFO, PKS and PKF files.

If you look at the 2 examples i added (These are from scenes DCBN and D000), you can see the 4-letter name assigned to each mt5 files. You can also see the position in 3D space there. I think if you look closely into the files you can find the 4 letter name - and i bet - also the coordinates.

The MAPINFO.BIn files are particulary interesting. Here you also find the motion files. As we know from the compile log these were squised in along with the gzipped mt5 files. And what does that mean? The mt5 are gzipped, the motions NOT. So you can actually read the motion descriptions in clear text. As you can imagine japanese people "secretly writing in english", in mind that noone will every read it again we come across some excellent funny "engrish".


But there is more to them.

There are sounds defined like background musics, some funny comments from the programmers and even small "procedures" or "timers" for actions.

For example one timer reads "yamagisi ga iru!" which obviously means "There's Yamagashi-san" and comments like that.





I also discovered a completely new file type, "MOBJ" files. From what i see i believe it is a motion capture format or a motion definition file. But i lack the abilities and experience to properly interpret the meaning.

CYCLEMAN.BIN in D000 is a good example. I believe it is the motions used for riding a bicycle. It would be fun if you could swap the satandard motion for walking Ryo with the bicycle one. Didn't someone wanna ride the bike? ;)


That's it for today, more as soon as i think it's noteworthy.

[yazgoo]

Master Kyodai, this is really neat.
I had seen the TXT files and the gzip reference,
but I had not had a look into the CHR files.
I'll look into it and give you feedback.

I've also started to write my MT5 guesses on the project github wiki.

[Master Kyodai]

The second file, it contains mt7 examples. While we are not talking about mt7 yet (lets first get mt5 working perfect?), if you take a peek at the MT7 compiling process it becomes clear this is very similar.

The Shenmue 2 related files even hold evidence they used python scripts to write their "compiler".


The chr files do not seem to be valuable at first glance - just some temp file that was obviously deleted, but the ones we have give reference about the 4 letter combination making it possible to find the mt5 files referenced and (hopefully) the coordinates. These are all i found, but they seem to be valuable as we have some coordinates, we know the correct values, we just need to find them in the file. But its much easier when you know what you are looking for and how its formatted.

There are also c files from whats shenmue. I can see stuff here like Camera, Move, Motion - so i think this is the source of what they used to compile the motion files. I am aware this is not your focus and maybe never will be - but it is an interesting little view about how AM2 programs. I am not that familiar with ELF binaries, but with Sequenceconv.o we seem to have the matching binary.

F1AZIICH.H also seems to be far off, i think from what i understand it is just a bunch of definitions for sounds, but it might lead to further clues about AM2s programming style.


I really hope these are of some use for you Yazgoo.

[dotsona]

Very very nice work Kyodai, great discovery! I hope this will aid Yazgoo in the development of the extractor

[SiZiOUS]

I just registered for this thread because I want to ask, Master Kyodai: How did you retrieved the source files from AM2 ??! I'm really impressed!!!!

By the way, I made a tool to edit PVR/DDS textures embedded in MT5/MT6/MT7, called Models Textures Editor. Maybe you can check it out if it may help you (the source code is also available).

I've also started to write my MT5 guesses on the project github wiki.

Wow!!! Great job!!

[Master Kyodai]

The source files are a "gift" from AM2, while the Japanese Versions of the game are very "clean", the EURO (PAL) version of the game seems to have been made in great hurry - here is where i got most "forgotten" files from. As you can see they sometimes forgot to delete log files, parts of sources and so on. I also found a few files in "What's Shenmue", but the PAL version of Shenmue 1 has left us by far with the largest number of "leftover" files.


Big thanks here go to David from the french Shenmue scene for supplying me with a large number of files (I was way too dumb to correctly transfer the data with my Serial coders cable).

[SiZiOUS]

The source files are a "gift" from AM2, while the Japanese Versions of the game are very "clean", the EURO (PAL) version of the game seems to have been made in great hurry - here is where i got most "forgotten" files from. As you can see they sometimes forgot to delete log files, parts of sources and so on. I also found a few files in "What's Shenmue", but the PAL version of Shenmue 1 has left us by far with the largest number of "leftover" files.

I already saw log files indeed. But parts of sources, never! So I'm really interested where did you find these In the PAL release of Shenmue 1 then ? And in What's Shenmue too ?

Edit: OK I found these files, THANK YOU for finding these gems!

Big thanks here go to David from the french Shenmue scene for supplying me with a large number of files (I was way too dumb to correctly transfer the data with my Serial coders cable).

I know him, we are working together. It's a really cool guy.

[Master Kyodai]

Yes it is indeed sometimes surprising what treasures you can find in Shenmue after over 10 years after its release. =)

All props go to David - french Shenmue scene is definitely the strongest, so many people have helped me, also big props to Family Guy who often helped me to survive through all the odd file formats and saved me ages of work on conversion. Also of course props to Kingshriek for his excellent knowledge of the sound formats, especially DSF and DTPK.


BTW: DTPK ain't dead. Saw a newer DTPK version in a wii game recently. AM2 ain't dead yet.

[SiZiOUS]

Yes it is indeed sometimes surprising what treasures you can find in Shenmue after over 10 years after its release. =)

Do you have "scanned" every Shenmue release? I just scanned Shenmue 2 Demo for Xbox and I retrieved some log files very interesting... So great!

All props go to David - french Shenmue scene is definitely the strongest, so many people have helped me, also big props to Family Guy who often helped me to survive through all the odd file formats and saved me ages of work on conversion.

I'm working with them both for my translation project and their are really great guys Oh and by the way I'm french people too (that's explaining my bad english!).

[Master Kyodai]

Well i did not check the north american releases as i don't have them, only japanese and european editions so far.

Very nice folks, I am sure with the help of them your project will be successfull. I think your english is quite well. I am german so my english has flaws as well. And i bet my french is much worse than your english, just ask David, we had a conversation in french and i think i sometimes said quite weird stuff.

The support for Shenmue on jeuxvideo.com was actually remarkable. Maybe all the support will get us to S3 in sum. It's good to know so many people are interested in Shenmue.

http://www.youtube.com/watch?v=yxdvHIQMWrw

[Supa]

This is a stupid question, but why did you call this a gift from Am2? Did you actually receive this from them or is that something Am2 does in all of their games to make things easier and you're just thanking them for it?

[Master Kyodai]

A gift as you normally will never see any sources for any game. Obviously rather an accident.

[Supa]

Ah, ok. Thanks for clearing it up.