Shenmue Files, emulation, converting & Ripping Compendiu

(Modding and other series related projects)

Shenmue Files, emulation, converting & Ripping Compendiu

Postby Master Kyodai » Mon Nov 02, 2009 2:54 pm

Shenmue Files, Emulation & Ripping Compendium

Preface

The following article is intended to be a compendium trying to explain the characteristics of the files encountered in Shenmue and their handling. It is based on facts and assumptions gathered by fans of the series. The article may not be complete or perfect. If you find any error or have an idea how to rewrite/correct parts of it then just let us know.


Chapter 1: File types encountered

The following is a listing of the file types encountered in Shenmue and what these files contain. Also file types not directly found on the disc, but related to Shenmue are mentioned.

DSF (PSF)

Dreamcast Sound Format is the proposed title of a digital music format for encoding Dreamcast music, similar to PlayStation Sound Format (PSF, also "Portable Sound Format). So far, such a format has not been publicly released.

The Header starts with "PSF". Dreamcast games don't contain DSP/PSF files, if such files are found they usually originate from someone who ripped the sounds already.

A DSF file contains a raw ARM7 sound program image which is loaded and copied to a specified address inside the program. The ARM7 then resets and executes the image.

DSF files can be loaded into a Winamp plugin called Highly Theoretical.

There is also the SSF, the "Saturn Sound Format".

DSF Files are usually manually (or in batch processes) created from MLT files.

A good way to convert DSF into MP3 or wav is using either vgmstream or a plugin for Foobar.

Good source of tools:

http://snesmusic.org/hoot/kingshriek/ssf/



DTPK

DTPK is an AM2 sound file. Usually uses the AICA sound driver.
There are some python scripts to extract DTPK sound files and the aica/am2 driver. Files start with "DTPK" if viewed in hex-Editor


SND

Just the common extension for DTPK sound files. The files are called like sound.snd. A Hex-Editor reveals the true nature of the file. Check hex header for "DTPK".


SPSD

Dreamcast digital sound file. Files start with "SPSD" in hex Editor. When extracted from afs their file ending in Shenmue is usually "*.STR". can be extracted with VGMstream, a nice utility to extract loads of different game music formats. For mono files it is required to add a Generic header (GENH) in order to rip them with VGMStream. VGMStream requires files to be renamed to "filename.SPSD" in order to extract these.

Newest built usually available at HCS64 site:

http://hcs64.com/vgmstream.html

A tool surfaced that does all of this automatically. It also eliminated all the problems that occured with VGMStream.


AFS

Sega container file. Starts with "AFS" if viewed in a Hex-Editor. This is comparable to archives like "zip" and usually just contain several other files in different formats.
Extract with AFSUtils.


IDX

Index files. These are related to an ASF file of the same name and give information about the contents (filenames) of the AFS archive. Hex-header starts with "IDX"


ADX

CRI Middleware compressed Audio format. Used in many SEGA games. Usually collected in AFS archives. There are plenty tools to extract or play these.


AHX

Basically the same as ADX, but it seems to be optimized for speech.


PVR

Power VR Texture/Image file. The only format Shenmue 1 and 2 use for Textures.


SFD

Movie file. Although an official format by CRI Middleware this format does not seem to be used in Shenmue. It seems all cut-scenes, the intro and other cinematics are done in real-time with the engine.


GENH

File with generic header. Shenmue does not have such files by default, it is a header someone added to make the file usable with VGMStream and other DSF players.

MT5/MT7

AM2's generic 3D model format. Partly directly on the disc, partly grouped in AFS files. Although some french guy found a way to interpret the model format there is no known tool released to convert them. So the only way to get the 3D models is ripping them out of the emulator (More about this in the next chapter)


DDS

DDS files don't directly appear in Shenmue. But if you rip models with 3D Ripper DX the texture format will be DDS. If you have a 3D program that can not handle DDS files as textures you will need to convert the to other formats like jpg or png.



Chapter 2: Extracting, Ripping, Converting and Emulating

There are several approaches to get content from Shenmue. To make things easier i try to group them by the corresponding media types.

Music and sound

Most music and sound files are stored as DTPK files (typically as filename.snd), AM2's proprietary file format that is probably best described as a kind of Sequenced tracks including the samples.
A good tool to convert the proprietary DTPK files to the easier to handle DSF format is the python script by kingshriek:

http://snesmusic.org/hoot/kingshriek/ssf/

Other files (especially speech and sound effects) come in SPSD format. SPSD files are not normally encountered alone on the Shenmue discs, they usually come in large bundles, stored in a single monolithic file with the AFS extension (accompanied by the IDX index files). most AFS Extractors will name the files with a .str extension (file.str). There is a nice batch related tool that can mass convert str files to wave from audio which can be played with all media players.


Textures

All textures come as PVR format files. Most are embedded in AFS bundles. There is a tool called PVR-dissector which can be used to mass-extract PVR files directly into PNG images. Note that they usually come mirrored and upside-down.
When you rip textures from an emulator the textures come in DDS format which you may need to convert to jpg or png in order to use.

3D Models

The 3D Models come in MT5 and MT7 formats, which are proprietary formats by AM2. A french guy has found out how these files are built and successfully managed to completely read the files.

The current method of ripping the 3D models is to rip the whole scene from the current view with 3D Ripper DX in NullDC Emulator. To get the models in 3D you need to use a special stereoscopic plugin for NullDC.
More about Emulation in Chapter 4 - Emulation. Video tutorials are available that demonstrate the whole process of ripping.




Chapter 3: usefull Links and tools

Info about ripping Sound and Music from Dreamcast games:

http://hcs64.com/vgm_ripping.html

DSF collection (spares the pain to create DSF ourselves)

http://dsf.hcs64.com/s/

DDS Converter 2.1 (Mass Converts DDS to jpg or png)

http://www.tankraider.com/userup/1264411639.zip

MT5 Extractor, GUI (requires mono) and sources

http://abdessel.iiens.net/dreamcast/mt5/

MT5 Extractor Mass extraction GUI (Basically same as above but doesnt need mono)

http://www.tankraider.com/userup/1261175783.zip

Flashless GUI & most recent mt5 extractor:

http://www.tankraider.com/userup/1265893559.zip

VB6 mass extractor Source (VB6):

http://www.tankraider.com/userup/1265915914.zip

Null DC 1.00B16 with Stereoscopic Plugin (a must for ripping from the game)

http://www.tankraider.com/userup/1265047692.zip

3D Ripper DX 1.7 (Rip 3D Models while playing the game)

http://www.tankraider.com/userup/1265076820.zip

DSF Plugins for Foobar2000 and Winamp

http://www.tankraider.com/userup/1266756862.zip

Big Package of extraction tools

http://www.tankraider.com/userup/1266735863.zip

AFS Utility V2 - extract/view/manipulate AFS archives

http://www.tankraider.com/userup/1266741951.rar

Shenmue CInematics director - lets you change subtitles

http://www.tankraider.com/userup/1266677884.rar

PVR Dissector - Good tool to extract PVR files from various other resources

http://www.tankraider.com/userup/1266748225.zip

AM2 AHX Extractor Tool

http://www.tankraider.com/userup/1266690927.rar

STR/DTPK mass extraction tool

http://www.tankraider.com/userup/1266713241.rar

SND2WAV - as it says it extracts SND files

http://www.tankraider.com/userup/1266770332.rar

MAPINFO Patcher - you need this if you wanna swap maps in the game

http://www.tankraider.com/userup/1266721825.zip

Shenmue Font hacking pack - a tutorial and tools to change the fonts of the subtitles

http://www.tankraider.com/userup/1266727432.rar

Selfboot Data pack. If you need to make a new selboot image from your edited files this will help.

http://www.tankraider.com/userup/1266772850.rar

Humans dissector - extracts Humans.afs

http://www.tankraider.com/userup/1266757966.rar

Photoshop DDS Plugins - allows editing of DDS files in PS

http://www.tankraider.com/userup/1266733003.zip

AiO Cinematics Translator - good cinematics subtitle editor for shenmue

http://www.tankraider.com/userup/1266725831.zip

Shenmue Free Quest subtitle editor

http://www.tankraider.com/userup/1266737565.zip

Shenmue test environment - To easily test out how your changes will look

http://www.tankraider.com/userup/1266707434.zip

IDX Creator - recreates IDX index for your modified afs

http://www.tankraider.com/userup/1266733513.zip

MT5/MT7 Texture editor - edits the texture of the model

http://www.tankraider.com/userup/1266773066.zip

SPR Editor - Edit textures/graphics in SPR files

http://www.tankraider.com/userup/1266700546.zip

PVR2PNG - Extracts PVR to PNG while keeping transparency information

http://www.tankraider.com/userup/1266679211.zip

SEGA PVR Tools package - all tools SEGA used themselves

http://www.tankraider.com/userup/1266747838.zip

CRI Middleware 2008 - AFS, AHX, ADX tools

http://www.tankraider.com/userup/1271176534.zip

AFS Explorer 3.2

http://www.tankraider.com/userup/1271607190.zip

GDI2Data - Extract GDI images

http://www.tankraider.com/userup/1271623975.zip

KyoDTPK - mass converter i made to convert dtpk files

http://www.tankraider.com/userup/1365733086.zip




Chapter 4: Emulation

There are 2 major emulators available that are able to run commercial games like Shenmue: NullDC and Chankast. The more compatible and generally prefered one is NullDC. Games usually come as .cdi files which represents a Disc Juggler image file. As the original GD-Rom Game discs can not be used for reading in a normal PC CD-Rom drive the only easy solution is to download the CDI files from the internet. A small search in google for "Shenmue" and "torrent" or "Shenmue" and "Rapidshare" should reveal some sources. Transforming the GD-Rom discs into an image is unfortunately quite complicated and requires at least a Dreamcast coder's cable (Dreamcast to serial port PC) or the Broadband adaptor (BBA) while the later is ridiculous expensive and the first one is hard to get.
Last edited by Master Kyodai on Tue Apr 20, 2010 12:20 pm, edited 17 times in total.

Master Kyodai has received a thanks from: nagates
User avatar
Master Kyodai
"After Burner...Great!"
"After Burner...Great!"
 
Joined: May 2003

Postby Master Kyodai » Mon Nov 02, 2009 2:55 pm

- NOTE -


before anyone rants - this is NOT finished. I had to stop in the middle of writing it because i have to care about some other stuff. I will complete it asap, so i beg your pardon this is incomplete yet.
User avatar
Master Kyodai
"After Burner...Great!"
"After Burner...Great!"
 
Joined: May 2003

Postby Bluecast » Mon Nov 02, 2009 3:47 pm

Glad HR made this a new forum.
Congrats you guys!
:D
User avatar
Bluecast
Jean Valjean
Banned
 
Joined: August 2003
PSN: Ryudoadam
XBL: Dogi99
Nintendo FC: Segata
Steam: Ryudo2k9
Favorite title: Shenmue
Currently playing: Some weeb game as always.

Postby Crimson Ryan » Mon Nov 02, 2009 3:52 pm

Ryudo™ wrote:Glad HR made this a new forum.
Congrats you guys!
:D

And pipped you as admin in turn for Master Kyodai. You must be pissed!

Congrats MK and dotsona.
User avatar
Crimson Ryan
Mac Tonight
Shenmue III
 
Joined: February 2004
Location: UK

Postby Bluecast » Mon Nov 02, 2009 3:57 pm

Crimson Ryan wrote:
Ryudo™ wrote:Glad HR made this a new forum.
Congrats you guys!
:D

And pipped you as admin in turn for Master Kyodai. You must be pissed!

Congrats MK and dubkor.

:P Never entered my mind to be honest so I never asked,then again never asked to be mod either.

Anyway back on topic.
I think it's awesome we have dedicated fans again making mods for the game and from people who are respectable forum members.
User avatar
Bluecast
Jean Valjean
Banned
 
Joined: August 2003
PSN: Ryudoadam
XBL: Dogi99
Nintendo FC: Segata
Steam: Ryudo2k9
Favorite title: Shenmue
Currently playing: Some weeb game as always.

Postby Crimson Ryan » Mon Nov 02, 2009 4:28 pm

May I ask what the purpose of it all is? Are you guys working on a new game or just for laughs like the modded cutscenes?
User avatar
Crimson Ryan
Mac Tonight
Shenmue III
 
Joined: February 2004
Location: UK

Postby dotsona » Mon Nov 02, 2009 4:38 pm

Crimson Ryan wrote:May I ask what the purpose of it all is? Are you guys working on a new game or just for laughs like the modded cutscenes?


Well that is what we want the community to decide, personally I want to see a library of shenmue models (Characters, Environments) to allow fans to make comics, game mods (put characters in other games) possibly create our own games (which would be difficult) but alot of it is just for fun. I personally want to make some comics :D, but who knows, we will see where it takes us. Anything to help keep shenmue alive.

dotsona has received a thanks from: nagates
User avatar
dotsona
Man Mo Acolyte
Man Mo Acolyte
 
Joined: September 2009
Location: Winter Springs, USA
Favorite title: What's Shenmue
Currently playing: Shenmue Mods

Postby Crimson Ryan » Mon Nov 02, 2009 4:44 pm

dotsona wrote:
Crimson Ryan wrote:May I ask what the purpose of it all is? Are you guys working on a new game or just for laughs like the modded cutscenes?


Well that is what we want the community to decide, personally I want to see a library of shenmue models (Characters, Environments) to allow fans to make comics, game mods (put characters in other games) possibly create our own games (which would be difficult) but alot of it is just for fun. I personally want to make some comics :D, but who knows, we will see where it takes us. Anything to help keep shenmue alive.

Sounds wicked. It's amazing what people can do when you give them something to play with..

On the comics side, it may even be possible to recreate the Shenmue spinoff comics with the actual 3D models..minus Ziming ofcourse.
User avatar
Crimson Ryan
Mac Tonight
Shenmue III
 
Joined: February 2004
Location: UK

Postby Master Kyodai » Mon Nov 02, 2009 4:48 pm

Crimson Ryan wrote:
Ryudo™ wrote:Glad HR made this a new forum.
Congrats you guys!
:D

And pipped you as admin in turn for Master Kyodai. You must be pissed!

Congrats MK and dotsona.



Not sure if the "Admin Title" is permanent. I don't know if i would have the time for a full time admin anyways. it was actually only granted as i needed admin access to check out the currently installed forum features (Mods, addons, etc) and their position in a scenario with a possible upgrade. It was in discussion whether the installation of this board can still be regarded "recent" so i took a look at it all to give a statement and a short report about the possibilities of potential upgrades, the caveats and requirements. The possibilities are still being evaluated, but if Ziming decides for an upgrade we may get the one or other new feature. Some features that did not "survive" the moval to the new host might also be fixed, so cross your fingers and with a bit luck you get an even better Dojo soon. I'm not going into detail as most we discussed yet was pure speculation and evaluation of ideas, but Ziming might get into detail soon and inform the community whether the updates take place. I think i may already say that we were able to fix the Gallery, so stay tuned for thousands of Shenmue related pictures.


@Crimson ryan: The purpose is hard to describe. For one it might simply be the creation of nice high-res desktop backgrounds, the other wants to make a comic with Shenmue characters and the third want to make a small animated movie. Maybe someone even inserts Lan Di's Black car as a car for GTA 3. I think there is much you can do with the resources of Shenmue, whether it is the Music, the sounds, the textures or the 3D models. We think it is the key to more fan-created content and keeping the game, the scene and the legend alive.


my goal is to make every single aspect of the game and its resources available to the fans, so all kinds of fan-related artworks are available without the hazzle it was for the pioneers like Lan DC and many others, the many hours and days of trying to figure out how to do this or that.
This forum should help people to save the time and pain and directly going over to create content. We want to spread knowledge, content and resources.

I wish we were working on a new game, but I'm not a good game programmer i think. Maybe if we can agree to an engine, but not right now i fear. However - i also enjoyed lan Di's Stereo (and how ryo nearly broke it), so if it leads to more mods like this coming out it won't be my loss...


=)
Last edited by Master Kyodai on Mon Nov 02, 2009 4:53 pm, edited 1 time in total.
User avatar
Master Kyodai
"After Burner...Great!"
"After Burner...Great!"
 
Joined: May 2003

Postby dotsona » Mon Nov 02, 2009 4:50 pm

Edit: ^^ I agree with everything he said above :)

Yeah I plan on using the models for the comics, it would be a 3d comic that looks just like the game (Only slightly better since we can smooth models) I think it would be pretty neat :) Plus once we start rigging the characters positioning there body and putting them into poses will be easy. Making comic creation a much faster process than traditionally drawing out characters.
Last edited by dotsona on Mon Nov 02, 2009 4:53 pm, edited 1 time in total.
User avatar
dotsona
Man Mo Acolyte
Man Mo Acolyte
 
Joined: September 2009
Location: Winter Springs, USA
Favorite title: What's Shenmue
Currently playing: Shenmue Mods

Postby Crimson Ryan » Mon Nov 02, 2009 4:53 pm

Either way it sounds amazing what you plan to do and I wish the best of luck. Evidently, fan support for Shenmue has already brought Ryo back into Sega's games, and when you put your resources into more peoples hands what will come out of it is something I'm very much looking forward to seeing.
User avatar
Crimson Ryan
Mac Tonight
Shenmue III
 
Joined: February 2004
Location: UK

Postby Riku Rose » Mon Nov 02, 2009 4:53 pm

Master Kyodai wrote:Not sure if the "Admin Title" is permanent. I don't know if i would have the time for a full time admin anyways. it was actually only granted as i needed admin access to check out the currently forum features. It was in discussion whether the installation of this board can still be regarded "recent" so i took a look at it all to give a statement and a short report about the possibilities of potential upgrades, the caveats and requirements. The possibilities are still being evaluated, but if Ziming decides for an upgrade we may get the one or other new feature. Some features that did not "survive" the moval to the new host might also be fixed, so cross your fingers and with a bit luck you get an even better Dojo soon. I'm not going into detail as most we discussed yet was pure speculation and evaluation of ideas, but Ziming might get into detail soon and inform the community whether the updates take place. I think i may already say that we were able to fix the Gallery, so stay tuned for thousands of Shenmue related pictures.


Hope so. The forum is looking a bit dated.
User avatar
Riku Rose
Shenmue III
Shenmue III
 
Joined: February 2006
Location: Kent, England
Currently playing: Yakuza 6

Postby dotsona » Tue Nov 10, 2009 11:24 pm

This should be a sticky, would be great to add to it, perhaps we can combine this with the video tutorials as well.
User avatar
dotsona
Man Mo Acolyte
Man Mo Acolyte
 
Joined: September 2009
Location: Winter Springs, USA
Favorite title: What's Shenmue
Currently playing: Shenmue Mods

Re: Shenmue Files, emulation, converting & Ripping Compe

Postby R.Y.O » Fri Jan 08, 2010 1:48 am

Master Kyodai wrote:There are 2 major emulators available that are able to run commercial games like Shenmue: NullDC and Chankast.


Don`t forget Makaron and Demul. The Shenmue emulation in Demul is far better than in NullDC. Unfortunately Demul requires a fast CPU.

Makaron is real great emulator too, but Shenmue has a few graphic errors there.


:) :) :)
R.Y.O
Asia Travel Representative
Asia Travel Representative
 
Joined: August 2006

Postby wude » Mon Jan 11, 2010 4:14 pm

demul doesn't support 3d ripper dx, and makaron needs joystick :? .
User avatar
wude
٩(⁎❛ᴗ❛⁎)۶
Machine Gun Fist
 
Joined: February 2008
Location: Ryuuseigai
Favorite title: Shenmue II
Currently playing: m.a.m.e

Next

Return to Community Projects

Who is online

Users browsing this forum: No registered users and 1 guest

Powered by phpBB © 2000-
ShenmueDojo.net