# pyjmap
**pyjmap** is a high-level implementation of Nintendo's homemade BCSV/JMap data format. This includes methods to construct, analyze, manipulate, deserialize and serialize proper JMap data. Conversion between CSV and BCSV files is also supported. The reverse-engineered specifications of the file format can be accessed on the [Luma's Workshop wiki](https://luma.aurumsmods.com/wiki/BCSV_(File_format)). This flatbuffer-like data type was used in first-party GameCube and Wii games. As the field/column names are hashed, a lookup table needs to be used to retrieve proper field names. For this, the library provides hashtable implementations for *Super Mario Galaxy*, *Super Mario Galaxy 2*, *Luigi's Mansion* and *Donkey Kong Jungle Beat*.
## Setup
This library requires **Python 3.6 or newer**. You can use pip to install *pyjmap*:
```sh
pip install pyjmap
```
## Command usage
Command line operations to convert between JMap and CSV files are supported. The CSV files are required to be in a special format that has been found in some leftover source files from *Super Mario Galaxy 2*. That format is described down below.
You can dump the contents of a BCSV/JMap file to a CSV file using:
```sh
pyjmap tocsv [-le] [-jmapenc JMAP_ENCODING] [-csvenc CSV_ENCODING] {smg,dkjb,lm} JMAP_FILE_PATH CSV_FILE_PATH
```
Proper CSV files can be converted back to BCSV/JMap files using:
```sh
pyjmap tojmap [-le] [-jmapenc JMAP_ENCODING] [-csvenc CSV_ENCODING] {smg,dkjb,lm} CSV_FILE_PATH JMAP_FILE_PATH
```
If ``le`` is set, the data is expected to be stored using little-endian byte order. ``jmapenc`` specifies the encoding of strings in the JMap data and it defaults to ``shift_jisx0213``. ``csvenc`` is the encoding of the CSV file and it uses ``utf-8`` by default. The hash lookup table is specified by ``HASHTABLE``. Supported values are ``smg`` for *Super Mario Galaxy*, ``lm`` for *Luigi's Mansion*, ``sms`` for *Super Mario Sunshine* and ``dkjb`` for *Donkey Kong Jungle Beat*.
## Library usage
The library provides various high-level operations to deal with JMap data. Below is some example code showing the fundamentals of *pyjmap*. Look at [jmap.py](pyjmap/jmap.py) for more information about the different methods.
```python
import pyjmap
# A hash lookup table is required to retrieve the proper names for hashed fields:
hashtbl_smg = pyjmap.SuperMarioGalaxyHashTable() # Lookup table for Super Mario Galaxy 1/2
hashtbl_sms = pyjmap.SuperMarioSunshineHashTable() # Lookup table for Super Mario Sunshine
hashtbl_lm = pyjmap.LuigisMansionHashTable() # Lookup table for Luigi's Mansion
hashtbl_dkjb = pyjmap.JungleBeatHashTable() # Lookup table for Donkey Kong Jungle Beat
# Create JMapInfo data from files and print number of entries
info = pyjmap.from_file(hashtbl_smg, "GalaxySortIndexTable.bcsv", big_endian=True) # Big-endian is True by default
info_from_csv = pyjmap.from_csv(hashtbl_smg, "GalaxySortIndexTable.csv") # Load data from CSV file
print("Number of entries: %d" % len(info)) # >> Number of entries: 55
# Print fields
for field in info.fields:
print(field) # >> name
# >> MapPaneName
# >> OpenCondition0
# >> OpenCondition1
# >> OpenCondition2
# >> PowerStarNum
# >> GrandGalaxyNo
# Checking if a field exists
print("MapPaneName" in info) # >> True
print("StageName" in info) # >> False
# Getting information about a field
field = info.get_field("MapPaneName") # Get field by name
field = info.get_field(0x7991F36F) # Get field by hash
print("[%08X]" % field.hash) # >> [7991F36F]
print(field.name) # >> MapPaneName
print(field.type) # >> JMapFieldType.STRING_OFFSET
print("0x%08X" % field.mask) # >> 0xFFFFFFFF
print(field.shift) # >> 0
# Manually-specified offsets and bit-packed data
collision_pa = pyjmap.JMapInfo(hashtbl_smg)
collision_pa.manual_offsets = True
collision_pa.create_field("camera_id", pyjmap.JMapFieldType.LONG, 0, mask=0x000000FF, shift_amount=0, offset=0)
collision_pa.create_field("Sound_code", pyjmap.JMapFieldType.LONG, 0, mask=0x00007F00, shift_amount=8, offset=0)
collision_pa.create_field("Floor_code", pyjmap.JMapFieldType.LONG, 0, mask=0x001F8000, shift_amount=15, offset=0)
collision_pa.create_field("Wall_code", pyjmap.JMapFieldType.LONG, 0, mask=0x01E00000, shift_amount=21, offset=0)
collision_pa.create_field("Camera_through", pyjmap.JMapFieldType.LONG, 0, mask=0x02000000, shift_amount=25, offset=0)
# Creating an exact copy of the data
copied = info.copy()
# The following creates a new field called CometMedalNum which uses the LONG data type. The field's default value
# that is applied to all fields is -1. The optional bitmask and shift amount are 0xFFFFFFFF and 0, respectively.
copied.create_field("CometMedalNum", pyjmap.JMapFieldType.LONG, -1, mask=0xFFFFFFFF, shift_amount=0)
# This removes the field OpenCondition2 and its data in all entries.
copied.drop_field("OpenCondition2")
# Accessing entries directly
first = copied[0] # Get first entry from copied data
last = copied[-1] # Get last entry from copied data
# Adding and deleting entries
new_entry = copied.create_entry() # Creates a new entry with default data for all fields
del copied[-3:] # Delete the last three entries from the copied data
# Iterate over all entries and set GrandGalaxyNo to 0
for entry in copied:
entry["GrandGalaxyNo"] = 0
# Sort entries by name in lexicographic descending order
info.sort_entries(lambda e: e["name"].lower(), reverse=True)
# Get all entries whose name start with "Koopa"
for entry in filter(lambda e: e["name"].startswith("Koopa"), info):
print(entry) # >> {'name': 'KoopaJrShipLv1Galaxy', ... }
# >> {'name': 'KoopaBattleVs3Galaxy', ... }
# >> {'name': 'KoopaBattleVs2Galaxy', ... }
# >> {'name': 'KoopaBattleVs1Galaxy', ... }
# Write data to files
pyjmap.write_file(info, "GalaxySortIndexTable_edited.bcsv", big_endian=True) # Pack and write binary
pyjmap.dump_csv(copied, "GalaxySortIndexTable_copied.csv", encoding="utf-8") # Dump CSV content
# Pack as little-endian buffer
packed_copied = pyjmap.pack_buffer(copied, big_endian=False)
```
# Data types
The following field data types are supported:
| Identifier | CSV type | Description |
| - | - | - |
| ``JMapFieldType.LONG`` | ``Int`` | 32-bit integer |
| ``JMapFieldType.UNSIGNED_LONG`` | ``UnsignedInt`` | 32-bit unsigned integer |
| ``JMapFieldType.SHORT`` | ``Short`` | 16-bit integer |
| ``JMapFieldType.CHAR`` | ``Char`` | 8-bit integer |
| ``JMapFieldType.FLOAT`` | ``Float`` | single-precission float |
| ``JMapFieldType.STRING`` | ``EmbeddedString`` | embedded SJIS string (occupies 31 bytes at max) |
| ``JMapFieldType.STRING_OFFSET`` | ``String`` | SJIS string (**not supported in Luigi's Mansion**) |
# CSV format
The CSV format is based on the format of known source files that were left in the files of *Super Mario Galaxy 2*:
* All CSV files are comma-delimited and use quote-marks for quoted cell strings. Quoting is only used when necessary.
* The first CSV-row contains the field descriptors. A field descriptor always consists of three components that are separated by double-colons: the field's name, the data type and the default value. All existing CSV types are described in the previous section and are case sensitive!
* The default value for strings is 0 and is always ignored. It is only kept for syntax.
* If an entry's field data is empty, the default value will be used.
* The field name may be a hash if it's a hex-string encapsulated between two square brackets (for example ``[DEADBEEF]``)
Here is an example of a properly-formated CSV file:
```csv
name:String:0,MapPaneName:String:0,OpenCondition0:String:0,OpenCondition1:String:0,OpenCondition2:String:0,PowerStarNum:Char:0,GrandGalaxyNo:Char:0
AstroGalaxy,dummy,,,,0,0
AstroDome,dummy,,,,0,0
LibraryRoom,dummy,,,,0,0
PeachCastleGardenGalaxy,dummy,,,,0,0
EpilogueDemoStage,dummy,,,,0,0
```
Raw data
{
"_id": null,
"home_page": "https://github.com/SunakazeKun/pyjmap",
"name": "pyjmap",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.6",
"maintainer_email": "",
"keywords": "nintendo,jsystem,jmap,bcsv,modding",
"author": "Aurum",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/4b/19/80ce6ccd47347eed8973128fb3802a14ee8f14407900825aef9fbe5b779c/pyjmap-1.0.9.tar.gz",
"platform": null,
"description": "\r\n# pyjmap\r\n**pyjmap** is a high-level implementation of Nintendo's homemade BCSV/JMap data format. This includes methods to construct, analyze, manipulate, deserialize and serialize proper JMap data. Conversion between CSV and BCSV files is also supported. The reverse-engineered specifications of the file format can be accessed on the [Luma's Workshop wiki](https://luma.aurumsmods.com/wiki/BCSV_(File_format)). This flatbuffer-like data type was used in first-party GameCube and Wii games. As the field/column names are hashed, a lookup table needs to be used to retrieve proper field names. For this, the library provides hashtable implementations for *Super Mario Galaxy*, *Super Mario Galaxy 2*, *Luigi's Mansion* and *Donkey Kong Jungle Beat*.\r\n\r\n## Setup\r\nThis library requires **Python 3.6 or newer**. You can use pip to install *pyjmap*:\r\n```sh\r\npip install pyjmap\r\n```\r\n\r\n## Command usage\r\nCommand line operations to convert between JMap and CSV files are supported. The CSV files are required to be in a special format that has been found in some leftover source files from *Super Mario Galaxy 2*. That format is described down below.\r\n\r\nYou can dump the contents of a BCSV/JMap file to a CSV file using:\r\n```sh\r\npyjmap tocsv [-le] [-jmapenc JMAP_ENCODING] [-csvenc CSV_ENCODING] {smg,dkjb,lm} JMAP_FILE_PATH CSV_FILE_PATH\r\n```\r\n\r\nProper CSV files can be converted back to BCSV/JMap files using:\r\n```sh\r\npyjmap tojmap [-le] [-jmapenc JMAP_ENCODING] [-csvenc CSV_ENCODING] {smg,dkjb,lm} CSV_FILE_PATH JMAP_FILE_PATH\r\n```\r\n\r\nIf ``le`` is set, the data is expected to be stored using little-endian byte order. ``jmapenc`` specifies the encoding of strings in the JMap data and it defaults to ``shift_jisx0213``. ``csvenc`` is the encoding of the CSV file and it uses ``utf-8`` by default. The hash lookup table is specified by ``HASHTABLE``. Supported values are ``smg`` for *Super Mario Galaxy*, ``lm`` for *Luigi's Mansion*, ``sms`` for *Super Mario Sunshine* and ``dkjb`` for *Donkey Kong Jungle Beat*.\r\n\r\n## Library usage\r\nThe library provides various high-level operations to deal with JMap data. Below is some example code showing the fundamentals of *pyjmap*. Look at [jmap.py](pyjmap/jmap.py) for more information about the different methods.\r\n\r\n```python\r\nimport pyjmap\r\n\r\n# A hash lookup table is required to retrieve the proper names for hashed fields:\r\nhashtbl_smg = pyjmap.SuperMarioGalaxyHashTable() # Lookup table for Super Mario Galaxy 1/2\r\nhashtbl_sms = pyjmap.SuperMarioSunshineHashTable() # Lookup table for Super Mario Sunshine\r\nhashtbl_lm = pyjmap.LuigisMansionHashTable() # Lookup table for Luigi's Mansion\r\nhashtbl_dkjb = pyjmap.JungleBeatHashTable() # Lookup table for Donkey Kong Jungle Beat\r\n\r\n# Create JMapInfo data from files and print number of entries\r\ninfo = pyjmap.from_file(hashtbl_smg, \"GalaxySortIndexTable.bcsv\", big_endian=True) # Big-endian is True by default\r\ninfo_from_csv = pyjmap.from_csv(hashtbl_smg, \"GalaxySortIndexTable.csv\") # Load data from CSV file\r\nprint(\"Number of entries: %d\" % len(info)) # >> Number of entries: 55\r\n\r\n# Print fields\r\nfor field in info.fields:\r\n print(field) # >> name\r\n # >> MapPaneName\r\n # >> OpenCondition0\r\n # >> OpenCondition1\r\n # >> OpenCondition2\r\n # >> PowerStarNum\r\n # >> GrandGalaxyNo\r\n\r\n# Checking if a field exists\r\nprint(\"MapPaneName\" in info) # >> True\r\nprint(\"StageName\" in info) # >> False\r\n\r\n# Getting information about a field\r\nfield = info.get_field(\"MapPaneName\") # Get field by name\r\nfield = info.get_field(0x7991F36F) # Get field by hash\r\n\r\nprint(\"[%08X]\" % field.hash) # >> [7991F36F]\r\nprint(field.name) # >> MapPaneName\r\nprint(field.type) # >> JMapFieldType.STRING_OFFSET\r\nprint(\"0x%08X\" % field.mask) # >> 0xFFFFFFFF\r\nprint(field.shift) # >> 0\r\n\r\n# Manually-specified offsets and bit-packed data\r\ncollision_pa = pyjmap.JMapInfo(hashtbl_smg)\r\ncollision_pa.manual_offsets = True\r\ncollision_pa.create_field(\"camera_id\", pyjmap.JMapFieldType.LONG, 0, mask=0x000000FF, shift_amount=0, offset=0)\r\ncollision_pa.create_field(\"Sound_code\", pyjmap.JMapFieldType.LONG, 0, mask=0x00007F00, shift_amount=8, offset=0)\r\ncollision_pa.create_field(\"Floor_code\", pyjmap.JMapFieldType.LONG, 0, mask=0x001F8000, shift_amount=15, offset=0)\r\ncollision_pa.create_field(\"Wall_code\", pyjmap.JMapFieldType.LONG, 0, mask=0x01E00000, shift_amount=21, offset=0)\r\ncollision_pa.create_field(\"Camera_through\", pyjmap.JMapFieldType.LONG, 0, mask=0x02000000, shift_amount=25, offset=0)\r\n\r\n# Creating an exact copy of the data\r\ncopied = info.copy()\r\n\r\n# The following creates a new field called CometMedalNum which uses the LONG data type. The field's default value\r\n# that is applied to all fields is -1. The optional bitmask and shift amount are 0xFFFFFFFF and 0, respectively.\r\ncopied.create_field(\"CometMedalNum\", pyjmap.JMapFieldType.LONG, -1, mask=0xFFFFFFFF, shift_amount=0)\r\n\r\n# This removes the field OpenCondition2 and its data in all entries.\r\ncopied.drop_field(\"OpenCondition2\")\r\n\r\n# Accessing entries directly\r\nfirst = copied[0] # Get first entry from copied data\r\nlast = copied[-1] # Get last entry from copied data\r\n\r\n# Adding and deleting entries\r\nnew_entry = copied.create_entry() # Creates a new entry with default data for all fields\r\ndel copied[-3:] # Delete the last three entries from the copied data\r\n\r\n# Iterate over all entries and set GrandGalaxyNo to 0\r\nfor entry in copied:\r\n entry[\"GrandGalaxyNo\"] = 0\r\n\r\n# Sort entries by name in lexicographic descending order\r\ninfo.sort_entries(lambda e: e[\"name\"].lower(), reverse=True)\r\n\r\n# Get all entries whose name start with \"Koopa\"\r\nfor entry in filter(lambda e: e[\"name\"].startswith(\"Koopa\"), info):\r\n print(entry) # >> {'name': 'KoopaJrShipLv1Galaxy', ... }\r\n # >> {'name': 'KoopaBattleVs3Galaxy', ... }\r\n # >> {'name': 'KoopaBattleVs2Galaxy', ... }\r\n # >> {'name': 'KoopaBattleVs1Galaxy', ... }\r\n\r\n# Write data to files\r\npyjmap.write_file(info, \"GalaxySortIndexTable_edited.bcsv\", big_endian=True) # Pack and write binary\r\npyjmap.dump_csv(copied, \"GalaxySortIndexTable_copied.csv\", encoding=\"utf-8\") # Dump CSV content\r\n\r\n# Pack as little-endian buffer\r\npacked_copied = pyjmap.pack_buffer(copied, big_endian=False)\r\n```\r\n\r\n# Data types\r\nThe following field data types are supported:\r\n\r\n| Identifier | CSV type | Description |\r\n| - | - | - |\r\n| ``JMapFieldType.LONG`` | ``Int`` | 32-bit integer |\r\n| ``JMapFieldType.UNSIGNED_LONG`` | ``UnsignedInt`` | 32-bit unsigned integer |\r\n| ``JMapFieldType.SHORT`` | ``Short`` | 16-bit integer |\r\n| ``JMapFieldType.CHAR`` | ``Char`` | 8-bit integer |\r\n| ``JMapFieldType.FLOAT`` | ``Float`` | single-precission float |\r\n| ``JMapFieldType.STRING`` | ``EmbeddedString`` | embedded SJIS string (occupies 31 bytes at max) |\r\n| ``JMapFieldType.STRING_OFFSET`` | ``String`` | SJIS string (**not supported in Luigi's Mansion**) |\r\n\r\n# CSV format\r\nThe CSV format is based on the format of known source files that were left in the files of *Super Mario Galaxy 2*:\r\n* All CSV files are comma-delimited and use quote-marks for quoted cell strings. Quoting is only used when necessary.\r\n* The first CSV-row contains the field descriptors. A field descriptor always consists of three components that are separated by double-colons: the field's name, the data type and the default value. All existing CSV types are described in the previous section and are case sensitive!\r\n* The default value for strings is 0 and is always ignored. It is only kept for syntax.\r\n* If an entry's field data is empty, the default value will be used.\r\n* The field name may be a hash if it's a hex-string encapsulated between two square brackets (for example ``[DEADBEEF]``)\r\n\r\nHere is an example of a properly-formated CSV file:\r\n```csv\r\nname:String:0,MapPaneName:String:0,OpenCondition0:String:0,OpenCondition1:String:0,OpenCondition2:String:0,PowerStarNum:Char:0,GrandGalaxyNo:Char:0\r\nAstroGalaxy,dummy,,,,0,0\r\nAstroDome,dummy,,,,0,0\r\nLibraryRoom,dummy,,,,0,0\r\nPeachCastleGardenGalaxy,dummy,,,,0,0\r\nEpilogueDemoStage,dummy,,,,0,0\r\n```\r\n",
"bugtrack_url": null,
"license": "gpl-3.0",
"summary": "Python library for Nintendo's BCSV/JMap format",
"version": "1.0.9",
"split_keywords": [
"nintendo",
"jsystem",
"jmap",
"bcsv",
"modding"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "4b1980ce6ccd47347eed8973128fb3802a14ee8f14407900825aef9fbe5b779c",
"md5": "b7bc9d26c66be35bc56523d1d996c2fb",
"sha256": "a24ead8a993ac3156ace3dd8fb3e5a797605c375abd346924df387e981365af1"
},
"downloads": -1,
"filename": "pyjmap-1.0.9.tar.gz",
"has_sig": false,
"md5_digest": "b7bc9d26c66be35bc56523d1d996c2fb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.6",
"size": 37628,
"upload_time": "2023-02-11T12:44:11",
"upload_time_iso_8601": "2023-02-11T12:44:11.284177Z",
"url": "https://files.pythonhosted.org/packages/4b/19/80ce6ccd47347eed8973128fb3802a14ee8f14407900825aef9fbe5b779c/pyjmap-1.0.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-02-11 12:44:11",
"github": true,
"gitlab": false,
"bitbucket": false,
"github_user": "SunakazeKun",
"github_project": "pyjmap",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "pyjmap"
}
JSON
