Difference between revisions of "DatabasePackedFile"

From SimsWiki
Jump to: navigation, search
(Auto-inserted from WakkaWikki)
 
 
(32 intermediate revisions by 8 users not shown)
Line 1: Line 1:
−
{{OldWikiEntry}} Format of Simcity4 Archive data files
+
Archive format used many Maxis developed games, including [[The Sims 2]], [[SimCity 4]] and Spore.
−
--------------------------
+
  
−
Most simcity Data is as you probably know stored in various types of archive files that all have the same internal format such as DAT, SC4, SC4lot, etc. This format is proprietary to maxis and has been used in the Sims 2, and Sims online as well as this one.  
+
=Format=
 +
Most Sims 2 Data is as you probably know stored in various types of archive files that all have the same internal format such as package, cache, etc. This format is proprietary to Maxis and has been used in the [[SimCity 4]], and [[The Sims Online]] as well as this one.
  
−
Entries may be compressed with QFS compression depending on their entry data.
+
Entries may be compressed with [[RefPack]] compression depending on their entry data.
  
−
Header (96 bytes)
+
For an examination of the Spore DBPF format, see [[Spore:DBPF]]
−
File 1
+
−
File 2
+
−
File n
+
−
Index Entry 1 (20 bytes)
+
−
Index Entry 2
+
−
Index Entry n
+
  
−
Header first, individual files following with no filenames and a small file header area at the beginning of each followed by an index of all the files in the archive at 20 bytes per entry.
+
==Overview==
 +
This is the general structure of DatabasePackedFile packages:
  
−
Header:
+
Header (96 bytes)
 +
File 1
 +
File 2
 +
...
 +
File n
 +
Index Entry 1 (20 bytes)
 +
Index Entry 2
 +
...
 +
Index Entry n
  
−
Offset 00 - Identifier (DBPF - the type of dat)
+
Header first, individual files following with no filenames and a small file header area at the beginning of each followed by an index of all the files in the archive. Note that files may appear after the index table in files that are updated.
−
Offset 04 - Version Major (1 in SC4/Ts2 Dats)
+
−
Offset 08 - Version Minor (0 in sc4 dats, 1 in most TS2 packages)
+
−
Offset 12 - DWORD[3] Reserved (Users can store DBPF info data here if they want)
+
−
Offset 24 - Date Created in Hex (Unused in Version 1.1)
+
−
Offset 28 - Date Modified in Hex (Unused in Version 1.1)
+
−
Offset 32 - Index Type/Version (Always 7 in SC4/TS2 dats)
+
−
Offset 36 - Number of entries in the index
+
−
Offset 40 - Location of first index entry
+
−
Offset 44 - Size of index
+
−
Offset 48 - Number of Hole entries in the Hole Record
+
−
Offset 52 - Location of the hole Record
+
−
Offset 56 - Size of the hole Record
+
−
Offset 60 - Index Version 01 = 7.0, 02 = 7.1 (Version 1.1+ in TS2 only)
+
−
Offset 64 - 32 Bytes reserved for Future Use in other versions.
+
  
−
File: (If files are uncompressed, they start at offset 0, if they are compressed, there are 9 byte headers of this format)
+
==Header==
 +
{| width="100%" border="1"
 +
! width="20%" | Name
 +
! width="5%" | Version
 +
! width="10%" | Type/Size
 +
! width="65%" | Info
  
−
Offset 00 - Compressed Size of file
+
|-
−
Offset 04 - Compression ID (0x10FB) QFS Compression. (Compression spec is seperate covered on the [[packages]] page)
+
| Magic
−
Offset 06 - Uncompressed Size of file
+
|
−
Offset 09 - Start of compressed/uncompressed file data
+
| char[4]
 +
| DBPF
  
 +
|-
 +
| Major Version
 +
|
 +
| u32
 +
| rowspan="2" |
 +
* 1.0 seen in Sim City 4, The Sims 2
 +
* 1.1 seen in The Sims 2
 +
* 2.0 seen in SPORE
  
−
There are now 2 known formats of indexes in the game. To determine your version, read offset 60 in version 1.1 or use 7.0 for version 1.0 DBPFs.
+
|-
 +
| Minor Version
 +
|
 +
| u32
  
−
The size of an index entry in version 1.0 is 20 bytes, like this:
+
|-
−
(Version 1.0 and some 1.1's) (7.0)
+
| unknown
 +
| 2.0
 +
| u32
 +
|
  
−
Offset 00 - Type ID (main type of file. picture, texture, model etc)
+
|-
−
Offset 04 - Group ID (group of the file by purpose or type)
+
| unknown
−
Offset 08 - Instance ID (marker used by the format for finding a specific file)
+
| 2.0
−
Offset 12 - Location of the file in the archive
+
| u32
−
Offset 16 - Size of the file
+
|
  
−
And 24 bytes in many 1.1 DBPFs (7.1)
+
|-
 +
| unknown
 +
| 2.0
 +
| u32
 +
| Should always be zero in DBPF 2.0.
  
−
Offset 00 - Type ID (main type of file. picture, texture, model etc)
+
|-
−
Offset 04 - Group ID (group of the file by purpose or type)
+
| Date Created
−
Offset 08 - Instance ID (marker used by the format for finding a specific file)
+
| 1.0
−
Offset 12 - Instance ID 2 (Second Instance ID for GUID Based systems in TS2)
+
| u32
−
Offset 16 - Location of the file in the archive
+
|
−
Offset 20 - Size of the file
+
  
−
HOLE RECORD
+
|-
−
-----------
+
| Date Modified
−
A Hole record contains the location and size of all holes in a DBPF file. Its format is as follows
+
| 1.0
 +
| u32
 +
|
  
−
(repeating)
+
|-
−
DWORD - Hole Location
+
| Index major version
−
DWORD - Hole size
+
| < 2.0
 +
| u32
 +
| Always 7 in The Sims 2, Sim City 4. If this is used in 2.0, then it is 0 for SPORE.
 +
 
 +
|-
 +
| Index entry count
 +
|
 +
| u32
 +
| Number of entries in the Index table.
 +
 
 +
|-
 +
| Offset of first Index entry
 +
| < 2.0
 +
| u32
 +
|
 +
 
 +
|-
 +
| Index size
 +
|
 +
| u32
 +
| Size of the Index table in bytes.
 +
 
 +
|-
 +
| Hole entry count
 +
| < 2.0
 +
| u32
 +
|
 +
 
 +
|-
 +
| Hole offset
 +
| < 2.0
 +
| u32
 +
|
 +
 
 +
|-
 +
| Hole size
 +
| < 2.0
 +
| u32
 +
|
 +
 
 +
|-
 +
| Index minor version
 +
| < 2.0?
 +
| u32
 +
| Version 1.1+ in The Sims 2 only. In DBPF 2.0, it has a value of 3, unknown if used.
 +
 
 +
|-
 +
| Index offset
 +
| 2.0
 +
| u32
 +
| Offset to Index table in DBPF 2.0.
 +
 
 +
|-
 +
| unknown
 +
| 2.0
 +
| u32
 +
|
 +
 
 +
|-
 +
| reserved
 +
|
 +
| char[24]
 +
|
 +
 
 +
|}
 +
 
 +
==Index Table==
 +
There are 2 known formats of indexes in the game. The format version is found in the file header.
 +
 
 +
A third format has been discovered, structure currently unknown. The Index Minor Version is an actual zero, so this would be called "7.-1" by the logic used here. This format has been seen in files that ship with the game, and is believed to have been seen in files generated by the game, but has not yet been seen in user-created files [for obvious reasons ;) ]. These files were discovered by the simple expedient of having a file-scanning tool choke on them during a scan of the entire game.
 +
 
 +
In order that others may investigate, a list of the files with this previously-unknown index format:
 +
 
 +
Program directory:
 +
*Apartment Life\TSData\Res\GlobalLots\CAS!.package
 +
*Bon Voyage\TSData\Res\GlobalLots\CAS!.package
 +
*Double Deluxe\Base\TSData\Res\Catalog\Skins\FaceInfo.package
 +
*Double Deluxe\Base\TSData\Res\Effects\effects.package
 +
*Double Deluxe\Base\TSData\Res\GlobalLots\CAS!.package
 +
*Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\NeighborhoodManager.package
 +
*Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\Tutorial\Lots\Tutorial_Lot1.package
 +
*Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\Tutorial\Lots\Tutorial_Lot3.package
 +
*Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\Tutorial\Lots\Tutorial_Lot4.package
 +
*Double Deluxe\EP2\TSData\Res\GlobalLots\CAS!.package
 +
*Double Deluxe\SP4\TSData\Res\GlobalLots\CAS!.package
 +
*Family Fun Stuff\TSData\Res\GlobalLots\CAS!.package
 +
*FreeTime\TSData\Res\GlobalLots\CAS!.package
 +
*Glamour Life Stuff\TSData\Res\GlobalLots\CAS!.package
 +
*H&M Fashion Stuff\TSData\Res\GlobalLots\CAS!.package
 +
*IKEA Home Stuff\TSData\Res\GlobalLots\CAS!.package
 +
*Kitchen & Bath Interior Design Stuff\TSData\Res\GlobalLots\CAS!.package
 +
*Open For Business\TSData\Res\GlobalLots\CAS!.package
 +
*Pets\TSData\Res\GlobalLots\CAS!.package
 +
*Seasons\TSData\Res\GlobalLots\CAS!.package
 +
*Teen Style Stuff\TSData\Res\GlobalLots\CAS!.package
 +
*University\TSData\Res\Effects\effects.package
 +
*University\TSData\Res\GlobalLots\CAS!.package
 +
 
 +
Documents directory:
 +
*EA Games\The Sims 2\LotCatalog\cx_00000036.package -- This MAY be a copy of the CAS!.package file, and probably is.
 +
*EA Games\The Sims 2\Neighborhoods\Tutorial\Lots\Tutorial_Lot1.package
 +
*EA Games\The Sims 2\Neighborhoods\Tutorial\Lots\Tutorial_Lot3.package
 +
*EA Games\The Sims 2\Neighborhoods\Tutorial\Lots\Tutorial_Lot4.package
 +
 
 +
===DBPF 1.x, Index table 7.0===
 +
(20 bytes, some DBPF 1.1 entries use this format)
 +
 
 +
;DWORD
 +
:Type ID (see [[InternalFormats]])
 +
;DWORD
 +
:Group ID
 +
;DWORD
 +
:Instance ID
 +
;DWORD
 +
:Location of the file in the archive
 +
;DOWRD
 +
:Size of the file
 +
 
 +
===DBPF 1.x, Index table 7.1===
 +
(24 bytes)
 +
 
 +
;DWORD
 +
:Type ID (see [[InternalFormats]])
 +
;DWORD
 +
:Group ID
 +
;DWORD
 +
:Instance ID
 +
;DWORD
 +
:Second Instance ID
 +
;DWORD
 +
:Location of the file in the archive
 +
;DOWRD
 +
:Size of the file
 +
 
 +
==Hole Table==
 +
 
 +
===DBPF 1.x===
 +
A Hole record contains the location and size of all holes in a DBPF file.
 +
 
 +
;repeating
 +
;DWORD
 +
:Hole Location
 +
;DWORD
 +
:Hole size
  
 
Holes are created when the game deletes something from a DBPF. The holes themselves are simply junk data of the appropriate length to fill the hole.
 
Holes are created when the game deletes something from a DBPF. The holes themselves are simply junk data of the appropriate length to fill the hole.
  
 +
==File==
 +
Bulk of DBPF packages.  Each file is either uncompressed or compressed.  To check if a file is compressed you first need to read the [[DIR]] resource, if it exists.  (See the corresponding entry for a description).  If no DIR entry exists, then no files within the package are compressed.
 +
 +
===Compression===
 +
 +
When reading a DBPF file, the first thing you should do is check to see if any chunks inside are compressed.  This is done via the [[E86B1EEF|DIR]] resource.
 +
 +
For example code and a description of the mechanics, see [[DBPF Compression]].
 +
 +
===Directory Files===
 +
 +
These are used for compression lists. See [[E86B1EEF]]
 +
 +
===Header===
 +
Only if the file is compressed
 +
 +
;DWORD
 +
:Compressed Size of file
 +
;WORD
 +
:Compression ID (0x10FB) QFS Compression. (See [[packages]] for compression information)
 +
;3 bytes
 +
:Uncompressed Size of file
 +
 +
===Body===
 +
Raw data of the files.  See [[InternalFormats]] for a list.
 +
 +
=Pseudocode=
 +
 +
To read files inside a .package file you need to perform the following steps, in order
 +
 +
* Read the DBPF header
 +
* Read the Index tables
 +
* Check for a DIR record
 +
* Check for the file you want to extract
 +
* Is this file compressed? If so, uncompress it
 +
* Read the file data and process accordingly
 +
 +
For an example PHP class for reading DBPF files, see [[DBPF/Source Code]]
 +
 +
=Visual Format Guide=
 +
 +
Having recently started poking around with the DBPF format again for the SimCity 4 community on SimTropolis, I created some visual guides to the DBPF archive format. Hopefully they will be found useful here.
 +
{|
 +
| [[Image:DBPF_File_Format_v1.1.png|thumb|left|200x152px|alt=DBPF v1.x Visual Guide|DBPF v1.x]]
 +
| [[Image:DBPF_File_Format_v2.0.png|thumb|left|200x152px|alt=DBPF v2.0 Visual Guide|DBPF v2.0]]
 +
|}
 +
 +
Some information was taken from old DatGen source code, such as the Created/Modified timestamps and possible Flags DWORD. That information may not be accurate, as it was some time ago that I last chatted with Maxoids about the DBPF format.
 +
 +
- DarkMatter
 +
 +
== Examples of packages ==
 +
* [[Body Shop Clothing package]]
 +
* [[Mesh package]]
  
−
[[Category:Modding]]
+
[[Category:Sims 2 Modding]]
 +
[[Category:InternalFormats]]

Latest revision as of 21:39, 17 September 2012

Archive format used many Maxis developed games, including The Sims 2, SimCity 4 and Spore.

Contents

[edit] Format

Most Sims 2 Data is as you probably know stored in various types of archive files that all have the same internal format such as package, cache, etc. This format is proprietary to Maxis and has been used in the SimCity 4, and The Sims Online as well as this one.

Entries may be compressed with RefPack compression depending on their entry data.

For an examination of the Spore DBPF format, see Spore:DBPF

[edit] Overview

This is the general structure of DatabasePackedFile packages:

Header (96 bytes) 

File 1
File 2
...
File n
Index Entry 1 (20 bytes)
Index Entry 2
...
Index Entry n

Header first, individual files following with no filenames and a small file header area at the beginning of each followed by an index of all the files in the archive. Note that files may appear after the index table in files that are updated.

[edit] Header

Name Version Type/Size Info
Magic char[4] DBPF
Major Version u32
  • 1.0 seen in Sim City 4, The Sims 2
  • 1.1 seen in The Sims 2
  • 2.0 seen in SPORE
Minor Version u32
unknown 2.0 u32
unknown 2.0 u32
unknown 2.0 u32 Should always be zero in DBPF 2.0.
Date Created 1.0 u32
Date Modified 1.0 u32
Index major version < 2.0 u32 Always 7 in The Sims 2, Sim City 4. If this is used in 2.0, then it is 0 for SPORE.
Index entry count u32 Number of entries in the Index table.
Offset of first Index entry < 2.0 u32
Index size u32 Size of the Index table in bytes.
Hole entry count < 2.0 u32
Hole offset < 2.0 u32
Hole size < 2.0 u32
Index minor version < 2.0? u32 Version 1.1+ in The Sims 2 only. In DBPF 2.0, it has a value of 3, unknown if used.
Index offset 2.0 u32 Offset to Index table in DBPF 2.0.
unknown 2.0 u32
reserved char[24]

[edit] Index Table

There are 2 known formats of indexes in the game. The format version is found in the file header.

A third format has been discovered, structure currently unknown. The Index Minor Version is an actual zero, so this would be called "7.-1" by the logic used here. This format has been seen in files that ship with the game, and is believed to have been seen in files generated by the game, but has not yet been seen in user-created files [for obvious reasons ;) ]. These files were discovered by the simple expedient of having a file-scanning tool choke on them during a scan of the entire game.

In order that others may investigate, a list of the files with this previously-unknown index format:

Program directory:

  • Apartment Life\TSData\Res\GlobalLots\CAS!.package
  • Bon Voyage\TSData\Res\GlobalLots\CAS!.package
  • Double Deluxe\Base\TSData\Res\Catalog\Skins\FaceInfo.package
  • Double Deluxe\Base\TSData\Res\Effects\effects.package
  • Double Deluxe\Base\TSData\Res\GlobalLots\CAS!.package
  • Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\NeighborhoodManager.package
  • Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\Tutorial\Lots\Tutorial_Lot1.package
  • Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\Tutorial\Lots\Tutorial_Lot3.package
  • Double Deluxe\Base\TSData\Res\UserData\Neighborhoods\Tutorial\Lots\Tutorial_Lot4.package
  • Double Deluxe\EP2\TSData\Res\GlobalLots\CAS!.package
  • Double Deluxe\SP4\TSData\Res\GlobalLots\CAS!.package
  • Family Fun Stuff\TSData\Res\GlobalLots\CAS!.package
  • FreeTime\TSData\Res\GlobalLots\CAS!.package
  • Glamour Life Stuff\TSData\Res\GlobalLots\CAS!.package
  • H&M Fashion Stuff\TSData\Res\GlobalLots\CAS!.package
  • IKEA Home Stuff\TSData\Res\GlobalLots\CAS!.package
  • Kitchen & Bath Interior Design Stuff\TSData\Res\GlobalLots\CAS!.package
  • Open For Business\TSData\Res\GlobalLots\CAS!.package
  • Pets\TSData\Res\GlobalLots\CAS!.package
  • Seasons\TSData\Res\GlobalLots\CAS!.package
  • Teen Style Stuff\TSData\Res\GlobalLots\CAS!.package
  • University\TSData\Res\Effects\effects.package
  • University\TSData\Res\GlobalLots\CAS!.package

Documents directory:

  • EA Games\The Sims 2\LotCatalog\cx_00000036.package -- This MAY be a copy of the CAS!.package file, and probably is.
  • EA Games\The Sims 2\Neighborhoods\Tutorial\Lots\Tutorial_Lot1.package
  • EA Games\The Sims 2\Neighborhoods\Tutorial\Lots\Tutorial_Lot3.package
  • EA Games\The Sims 2\Neighborhoods\Tutorial\Lots\Tutorial_Lot4.package

[edit] DBPF 1.x, Index table 7.0

(20 bytes, some DBPF 1.1 entries use this format)

DWORD
Type ID (see InternalFormats)
DWORD
Group ID
DWORD
Instance ID
DWORD
Location of the file in the archive
DOWRD
Size of the file

[edit] DBPF 1.x, Index table 7.1

(24 bytes)

DWORD
Type ID (see InternalFormats)
DWORD
Group ID
DWORD
Instance ID
DWORD
Second Instance ID
DWORD
Location of the file in the archive
DOWRD
Size of the file

[edit] Hole Table

[edit] DBPF 1.x

A Hole record contains the location and size of all holes in a DBPF file.

repeating
DWORD
Hole Location
DWORD
Hole size

Holes are created when the game deletes something from a DBPF. The holes themselves are simply junk data of the appropriate length to fill the hole.

[edit] File

Bulk of DBPF packages. Each file is either uncompressed or compressed. To check if a file is compressed you first need to read the DIR resource, if it exists. (See the corresponding entry for a description). If no DIR entry exists, then no files within the package are compressed.

[edit] Compression

When reading a DBPF file, the first thing you should do is check to see if any chunks inside are compressed. This is done via the DIR resource.

For example code and a description of the mechanics, see DBPF Compression.

[edit] Directory Files

These are used for compression lists. See E86B1EEF

[edit] Header

Only if the file is compressed

DWORD
Compressed Size of file
WORD
Compression ID (0x10FB) QFS Compression. (See packages for compression information)
3 bytes
Uncompressed Size of file

[edit] Body

Raw data of the files. See InternalFormats for a list.

[edit] Pseudocode

To read files inside a .package file you need to perform the following steps, in order

  • Read the DBPF header
  • Read the Index tables
  • Check for a DIR record
  • Check for the file you want to extract
  • Is this file compressed? If so, uncompress it
  • Read the file data and process accordingly

For an example PHP class for reading DBPF files, see DBPF/Source Code

[edit] Visual Format Guide

Having recently started poking around with the DBPF format again for the SimCity 4 community on SimTropolis, I created some visual guides to the DBPF archive format. Hopefully they will be found useful here.

DBPF v1.x Visual Guide
DBPF v1.x
DBPF v2.0 Visual Guide
DBPF v2.0

Some information was taken from old DatGen source code, such as the Created/Modified timestamps and possible Flags DWORD. That information may not be accurate, as it was some time ago that I last chatted with Maxoids about the DBPF format. 

- DarkMatter

[edit] Examples of packages

Retrieved from "http://direct.simswiki.info/index.php?title=DatabasePackedFile&oldid=59959"
Personal tools
Namespaces

Variants
Actions
Navigation
game select
Toolbox