# BONSAI classifications
The BONSAI classifications Python package is a part of the [Getting The Data Right](https://bonsamurais.gitlab.io/bonsai/documentation) project.
Here, all the classifications, which are used in the Bonsai database, are created and stored as csv files.
The csv files can be found under [/src/classifications/data](https://gitlab.com/bonsamurais/bonsai/clean/classifications/-/tree/main/src/classifications?ref_type=heads). The structure of organising these files follows the [Bonsai ontology](https://github.com/BONSAMURAIS/ontology) and thus has the following folders:
- activitytype (includes: `industry_activity`, `government_activity`, `treatment_activity`, `non_profit_institution_serving_household`, `household_production`, `household_consumption`, `market_activity`, `natural_activity`, `auxiliary_production_activity`, `change_in_stock_activity`, `other_activity`)
- flowobject (includes `industry_product`, `material_for_treatment`, `market_product`, `government_product`, `household_product`, `needs_satisfaction`, `emission`, `direct_physical_change`, `natural_resource`, `economic_flow`, `social_flow`)
- location
- unit
- time
Since the Bonsai ontology does not cover all required topics, additional folders are added:
- dataquality
- uncertainty
A comprehensive documentation of the classification package is availbale [here](https://bonsamurais.gitlab.io/bonsai/util/classifications/index.html)
## Format
The csv files (tables) of each folder (datapackage) are organised in tabular format. Each of the mentioned folders represents a valid `dataio.datapackage` created with the Python package [dataio](https://bonsamurais.gitlab.io/bonsai/util/dataio/index.html). The following types of tables with its prefixes are used:
- tree table `tree_`
- concordance table `conc_`
- dimension table `dim_`
- pairwise cocncordance table `concpair_`
### tree table
Tree tables are used for classifications which have a tree structure, meaning that the classification is structured hierarchically with multiple levels. The classification starts with broad categories at the top level and then branches out into more specific subcategories as you move down the hierarchy.
The following column names are used:
- `code`: code of the item
- `parent_code`: code of the items parent
- `name`: name of the item
- `level`: the items level in the tree structure (from 0 to n)
- `prefixed_id`: unique id (uuid4)
### concordance table
A concordance table is used to establish equivalences or relationships between different classification systems. It provides mappings between codes of a classification system and codes from another classification system. A relationship between codes can have four different types:
- **one-to-one (1:1) correspondence**: In a one-to-one correspondence, each category or code in one classification system is mapped to exactly one category or code in another classification system, and vice versa. This type of mapping implies a direct and unambiguous correspondence between the two systems. The skos uri is http://www.w3.org/2004/02/skos/core#exactMatch
- **one-to-many (1:M) correspondence**: In a one-to-many correspondence, each category or code in one classification system is mapped to multiple categories or codes in another classification system. However, each category or code in the second system is only mapped to one category or code in the first system. This type of mapping implies that one category or code in the first system may encompass multiple categories or codes in the second system. The skos uri is http://www.w3.org/2004/02/skos/core#narrowMatch . Indicating `<A> skos:narrowMatch <B>` means "B is narrower than A"
- **many-to-one (M:1) correspondence**: In a many-to-one correspondence, multiple categories or codes in one classification system are mapped to a single category or code in another classification system. However, each category or code in the second system is only mapped to one category or code in the first system. This type of mapping implies that multiple categories or codes in the first system are aggregated or collapsed into a single category or code in the second system. The skos uri is http://www.w3.org/2004/02/skos/core#broadMatch . Indicating `<A> skos:broadwMatch <B>` means "B is broader than A"
- **many-to-many (M:M) correspondence**: In a many-to-many correspondence, multiple categories or codes in one classification system are mapped to multiple categories or codes in another classification system. This type of mapping indicates complex relationships where neither a straightforward one-to-one correspondence exists, nor a parent-child relationship. The skos uri is http://www.w3.org/2004/02/skos/core#relatedMatch
The following column names are used:
- `<tree_classification_A>`: code of classification A
- `<tree_classification_A>`: code of classification B which is mapped to the code of classification A
- `comment`: comment on the type of concordance
- `prefixed_id`: unique id (uuid4)
- `skos_uri`: skos uri
The requirements for these table types are specified [here](https://dataio-bonsamurais-bonsai-util-a55d63cbbbcf635b952059f8b8a12a71.gitlab.io/syntax.html#field).
### dimension table
A dimension table is used for classifications which do not have a tree structure.
The following column names are used:
- `code`: code of the item
- `name`: name of the item
- `description`: description of the item
- `prefixed_id`: unique id (uuid4)
### pairwise concordance table (for Bonsai)
This type of concordance table is used to map pairwise codes. For instance, some data providers such as `UNdata` and `IEA` are using combined codes for an activity (e.g. for "production of", "electricity production by") and `flowobject` (e.g. "coal") to express a `bonsai_activitytype` ("A_COAL", "A_PowC"). In some cases, when the `conc_` tables for `activitytype` and `flowobject`, which map single relations, are not sufficient to create these pairwise concordances, it is reasonable to make it explicit. The mapping relationships between the pairwise codes can be the same as in the `conc_` tables.
The following column names are used:
activitytype_from,flowobject_from,activitytype_to,flowobject_to,classification_from,classification_to
- `activitytype_from`: code for activitytype of `<from>` classification
- `flowobject_from`: code for flowobject of `<from>` classification
- `activitytype_to`: code for the activitytype of `<other>` classification
- `flowobject_to`: code for the flowobject of `<other>` classification
- `classification_from`: name of the `<from>` classification schema
- `classification_to`: name of the `<other>` classification schema
- `skos_uri`: skos uri
- `comment`: comment on the type of concordance
- `prefixed_id`: unique id (created by `uuid4`)
## Usage
To use the classification, you can install the package via pip. Replace `<version>` by a specific tag or branch name.
```
pip install git+ssh://git@gitlab.com/bonsamurais/bonsai/util/classifications@<version>
```
From pypi, do:
```
pip install bonsai_classifications
```
All classifications are provided as `dataio.datapackage` which include the tables as `pandas.DataFrame`. E.g., you can do the following get the classification `tree` for industry activities of Bonsai:
```python
import classifications
bosai_tree = classifications.activitytype.datapackage.tree_bonsai
```
:::{note} The datapackage object includes also the tables of other classifications.
:::
You can also get the concordance tables and external classifications in the similar way, using the `datapackage` object.
The activities and flowobjects of Bonsai can be also used directly as objects. By doing the following, you would get the `name` of the `A_Chick` activity.
```python
classifications.activitytype.bonsai.A_Chick.name
```
### Special methods
- `lookup()` for searching strings in code names
- `get_children()` to get all codes that have the same parent code
- `create_conc()` to create a concordance table
- `disaggregate_bonsai()` for adding new codes, which disaggregate an existing code
- `get_bonsai_schemas_mapping()` returns a dict that maps Bonsai schemas to Bonsai codes
- `print_tree()` prints the tree structure of a given code
To search for certain key words in a table, you can use the line of code below. This returns a pandas DataFrame with rows that have "coal" in the `name` column. Note that this lookup is case sensitive.
```python
bonsai_tree.lookup("coal")
```
To get all children of a certain code (here for treatment activities in Bonsai), you can do use the following method. By setting the option `deep=True`, you get all descandents. With `deep=False` you get only the direct children. The option `return_parent=True` will include the selected parent code. The option `exclude_sut_children=True` will return only the children that are included by another code in the SUT.
```python
classifications.activitytype.datapackage.tree_bonsai.get_children(parent_code="at", deep=True, return_parent=False, exclude_sut_children=False)
```
The package also helps to create new concordance tables. When having two concordance tables, one for mapping codes of classification `a` to `b`, and the other for mapping `b` to `c`, you can use the following:
df_1:
| a | b |
| - | - |
| 01.01 | x |
| ... | ... |
df_2:
| b | c |
| - | - |
| x | YXDA |
| ... | ... |
```python
df_3 = classifications.create_conc(df_1, df_2, source="a", target="c", intermediate="c")
```
df_3:
| a | c |
| - | - |
| 01.01 | YXDA |
| ... | ... |
To disaggregate existing codes of the Bonsai classification, you can use the `disaggregate_bonsai()` method. Depending on the category, e.g. `activitytype` or `flowobject`, you can call that method.
To indicate the which code you want to disaggregate, you need to provide a dictionary, with the old code of Bonsai as keys. The value corresponding to that key is a list of tuples. Each tuple represents a new code. The first entry of that tuple is the code, the second entry is the name, and the third is a mapping dictionary. This mapping dictionary includes the name of another classification scheme (other than Bonsai) as key, and a list of strings, which are the codes of the other classification now represented by the new code.
```python
codes = {"disaggregations":
[
{"old_code" : "A_Paper",
"new_codes":
[
{"code": "New_Paper1",
"description": "new paper production 1",
"mappings": {"nace_rev2": ["10.02","01.13"]}
},
{"code": "New_Paper2",
"description": "new paper production 2",
"mappings": {}
}
]
}
]
}
d = classifications.activitytype.datapackage.disaggregate_bonsai(codes)
```
Get the pandas DataFrames that are modified.
```python
d["tree_bonsai"]
d["conc_bonsai_nace_rev2"]
```
To use that function via terminal, execute `python disaggregate_bonsai.py <bonsai_categorty> <path/to/disaggregaion.yaml> <directory/for/updated/files>`. `<bonsai_category>` can be for instance `activitytype` or `flowobject`.
:::{note} To disaggregate an existing code, you need to provide at least 2 new codes. It is assumed that all entities covered by the new codes are equal to the entities of the existing code.
:::
The `print_tree(toplevelcode)` method helps to inspect the tree structure for a given code.
```python
classifications.flowobject.datapackage.tree_bonsai.print_tree("C_Wine")
```
The differentiation between bold and italic text is only relevant for the Bonsai-SUT. Italic written codes are "not part" of the inspected `toplevelcode`, since these are explicitly in the SUTs. Since these codes are seperatly in the SUT, the definition of the `toplevelcode` is thus "code, excluding the the italic children".
```
𝐂_𝐖𝐢𝐧𝐞
├── 𝐟𝐢_𝟐𝟒𝟐𝟏𝟏
├── 𝘊_𝘎𝘳𝘢𝘱𝘵
```
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/bonsamurais/bonsai/util/classifications",
"name": "bonsai-classifications",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": null,
"author": "admin_bonsai",
"author_email": "admin.gettingthedataright@aau.dk",
"download_url": "https://files.pythonhosted.org/packages/c6/12/654c19cb3300d7ebcea4e2b25e46b15cde0327243d85434abb4110ab1626/bonsai_classifications-0.3.10.tar.gz",
"platform": "any",
"description": "# BONSAI classifications\n\nThe BONSAI classifications Python package is a part of the [Getting The Data Right](https://bonsamurais.gitlab.io/bonsai/documentation) project.\n\nHere, all the classifications, which are used in the Bonsai database, are created and stored as csv files.\nThe csv files can be found under [/src/classifications/data](https://gitlab.com/bonsamurais/bonsai/clean/classifications/-/tree/main/src/classifications?ref_type=heads). The structure of organising these files follows the [Bonsai ontology](https://github.com/BONSAMURAIS/ontology) and thus has the following folders:\n- activitytype (includes: `industry_activity`, `government_activity`, `treatment_activity`, `non_profit_institution_serving_household`, `household_production`, `household_consumption`, `market_activity`, `natural_activity`, `auxiliary_production_activity`, `change_in_stock_activity`, `other_activity`)\n- flowobject (includes `industry_product`, `material_for_treatment`, `market_product`, `government_product`, `household_product`, `needs_satisfaction`, `emission`, `direct_physical_change`, `natural_resource`, `economic_flow`, `social_flow`)\n- location\n- unit\n- time\n\nSince the Bonsai ontology does not cover all required topics, additional folders are added:\n- dataquality\n- uncertainty\n\nA comprehensive documentation of the classification package is availbale [here](https://bonsamurais.gitlab.io/bonsai/util/classifications/index.html)\n\n## Format\nThe csv files (tables) of each folder (datapackage) are organised in tabular format. Each of the mentioned folders represents a valid `dataio.datapackage` created with the Python package [dataio](https://bonsamurais.gitlab.io/bonsai/util/dataio/index.html). The following types of tables with its prefixes are used:\n- tree table `tree_`\n- concordance table `conc_`\n- dimension table `dim_`\n- pairwise cocncordance table `concpair_`\n\n### tree table\nTree tables are used for classifications which have a tree structure, meaning that the classification is structured hierarchically with multiple levels. The classification starts with broad categories at the top level and then branches out into more specific subcategories as you move down the hierarchy.\n\nThe following column names are used:\n- `code`: code of the item\n- `parent_code`: code of the items parent\n- `name`: name of the item\n- `level`: the items level in the tree structure (from 0 to n)\n- `prefixed_id`: unique id (uuid4)\n\n### concordance table\nA concordance table is used to establish equivalences or relationships between different classification systems. It provides mappings between codes of a classification system and codes from another classification system. A relationship between codes can have four different types:\n\n- **one-to-one (1:1) correspondence**: In a one-to-one correspondence, each category or code in one classification system is mapped to exactly one category or code in another classification system, and vice versa. This type of mapping implies a direct and unambiguous correspondence between the two systems. The skos uri is http://www.w3.org/2004/02/skos/core#exactMatch\n\n- **one-to-many (1:M) correspondence**: In a one-to-many correspondence, each category or code in one classification system is mapped to multiple categories or codes in another classification system. However, each category or code in the second system is only mapped to one category or code in the first system. This type of mapping implies that one category or code in the first system may encompass multiple categories or codes in the second system. The skos uri is http://www.w3.org/2004/02/skos/core#narrowMatch . Indicating `<A> skos:narrowMatch <B>` means \"B is narrower than A\"\n\n- **many-to-one (M:1) correspondence**: In a many-to-one correspondence, multiple categories or codes in one classification system are mapped to a single category or code in another classification system. However, each category or code in the second system is only mapped to one category or code in the first system. This type of mapping implies that multiple categories or codes in the first system are aggregated or collapsed into a single category or code in the second system. The skos uri is http://www.w3.org/2004/02/skos/core#broadMatch . Indicating `<A> skos:broadwMatch <B>` means \"B is broader than A\"\n\n- **many-to-many (M:M) correspondence**: In a many-to-many correspondence, multiple categories or codes in one classification system are mapped to multiple categories or codes in another classification system. This type of mapping indicates complex relationships where neither a straightforward one-to-one correspondence exists, nor a parent-child relationship. The skos uri is http://www.w3.org/2004/02/skos/core#relatedMatch\n\n\nThe following column names are used:\n- `<tree_classification_A>`: code of classification A\n- `<tree_classification_A>`: code of classification B which is mapped to the code of classification A\n- `comment`: comment on the type of concordance\n- `prefixed_id`: unique id (uuid4)\n- `skos_uri`: skos uri\n\nThe requirements for these table types are specified [here](https://dataio-bonsamurais-bonsai-util-a55d63cbbbcf635b952059f8b8a12a71.gitlab.io/syntax.html#field).\n\n### dimension table\nA dimension table is used for classifications which do not have a tree structure.\n\nThe following column names are used:\n- `code`: code of the item\n- `name`: name of the item\n- `description`: description of the item\n- `prefixed_id`: unique id (uuid4)\n\n\n### pairwise concordance table (for Bonsai)\nThis type of concordance table is used to map pairwise codes. For instance, some data providers such as `UNdata` and `IEA` are using combined codes for an activity (e.g. for \"production of\", \"electricity production by\") and `flowobject` (e.g. \"coal\") to express a `bonsai_activitytype` (\"A_COAL\", \"A_PowC\"). In some cases, when the `conc_` tables for `activitytype` and `flowobject`, which map single relations, are not sufficient to create these pairwise concordances, it is reasonable to make it explicit. The mapping relationships between the pairwise codes can be the same as in the `conc_` tables.\n\n\nThe following column names are used:\nactivitytype_from,flowobject_from,activitytype_to,flowobject_to,classification_from,classification_to\n\n- `activitytype_from`: code for activitytype of `<from>` classification\n- `flowobject_from`: code for flowobject of `<from>` classification\n- `activitytype_to`: code for the activitytype of `<other>` classification\n- `flowobject_to`: code for the flowobject of `<other>` classification\n- `classification_from`: name of the `<from>` classification schema\n- `classification_to`: name of the `<other>` classification schema\n- `skos_uri`: skos uri\n- `comment`: comment on the type of concordance\n- `prefixed_id`: unique id (created by `uuid4`)\n\n## Usage\nTo use the classification, you can install the package via pip. Replace `<version>` by a specific tag or branch name.\n\n```\npip install git+ssh://git@gitlab.com/bonsamurais/bonsai/util/classifications@<version>\n```\n\nFrom pypi, do:\n```\npip install bonsai_classifications\n```\n\n\nAll classifications are provided as `dataio.datapackage` which include the tables as `pandas.DataFrame`. E.g., you can do the following get the classification `tree` for industry activities of Bonsai:\n```python\nimport classifications\n\nbosai_tree = classifications.activitytype.datapackage.tree_bonsai\n```\n:::{note} The datapackage object includes also the tables of other classifications.\n:::\n\nYou can also get the concordance tables and external classifications in the similar way, using the `datapackage` object.\n\nThe activities and flowobjects of Bonsai can be also used directly as objects. By doing the following, you would get the `name` of the `A_Chick` activity.\n```python\nclassifications.activitytype.bonsai.A_Chick.name\n```\n\n### Special methods\n- `lookup()` for searching strings in code names\n- `get_children()` to get all codes that have the same parent code\n- `create_conc()` to create a concordance table\n- `disaggregate_bonsai()` for adding new codes, which disaggregate an existing code\n- `get_bonsai_schemas_mapping()` returns a dict that maps Bonsai schemas to Bonsai codes\n- `print_tree()` prints the tree structure of a given code\n\nTo search for certain key words in a table, you can use the line of code below. This returns a pandas DataFrame with rows that have \"coal\" in the `name` column. Note that this lookup is case sensitive.\n```python\nbonsai_tree.lookup(\"coal\")\n```\n\nTo get all children of a certain code (here for treatment activities in Bonsai), you can do use the following method. By setting the option `deep=True`, you get all descandents. With `deep=False` you get only the direct children. The option `return_parent=True` will include the selected parent code. The option `exclude_sut_children=True` will return only the children that are included by another code in the SUT.\n```python\nclassifications.activitytype.datapackage.tree_bonsai.get_children(parent_code=\"at\", deep=True, return_parent=False, exclude_sut_children=False)\n```\n\nThe package also helps to create new concordance tables. When having two concordance tables, one for mapping codes of classification `a` to `b`, and the other for mapping `b` to `c`, you can use the following:\n\ndf_1:\n| a | b |\n| - | - |\n| 01.01 | x |\n| ... | ... |\n\ndf_2:\n| b | c |\n| - | - |\n| x | YXDA |\n| ... | ... |\n\n```python\ndf_3 = classifications.create_conc(df_1, df_2, source=\"a\", target=\"c\", intermediate=\"c\")\n\n```\ndf_3:\n| a | c |\n| - | - |\n| 01.01 | YXDA |\n| ... | ... |\n\nTo disaggregate existing codes of the Bonsai classification, you can use the `disaggregate_bonsai()` method. Depending on the category, e.g. `activitytype` or `flowobject`, you can call that method.\nTo indicate the which code you want to disaggregate, you need to provide a dictionary, with the old code of Bonsai as keys. The value corresponding to that key is a list of tuples. Each tuple represents a new code. The first entry of that tuple is the code, the second entry is the name, and the third is a mapping dictionary. This mapping dictionary includes the name of another classification scheme (other than Bonsai) as key, and a list of strings, which are the codes of the other classification now represented by the new code.\n```python\ncodes = {\"disaggregations\":\n [\n {\"old_code\" : \"A_Paper\",\n \"new_codes\":\n [\n {\"code\": \"New_Paper1\",\n \"description\": \"new paper production 1\",\n \"mappings\": {\"nace_rev2\": [\"10.02\",\"01.13\"]}\n },\n {\"code\": \"New_Paper2\",\n \"description\": \"new paper production 2\",\n \"mappings\": {}\n }\n ]\n }\n ]\n}\n\nd = classifications.activitytype.datapackage.disaggregate_bonsai(codes)\n```\n\nGet the pandas DataFrames that are modified.\n```python\nd[\"tree_bonsai\"]\nd[\"conc_bonsai_nace_rev2\"]\n```\n\nTo use that function via terminal, execute `python disaggregate_bonsai.py <bonsai_categorty> <path/to/disaggregaion.yaml> <directory/for/updated/files>`. `<bonsai_category>` can be for instance `activitytype` or `flowobject`.\n\n:::{note} To disaggregate an existing code, you need to provide at least 2 new codes. It is assumed that all entities covered by the new codes are equal to the entities of the existing code.\n:::\n\n\nThe `print_tree(toplevelcode)` method helps to inspect the tree structure for a given code.\n\n```python\nclassifications.flowobject.datapackage.tree_bonsai.print_tree(\"C_Wine\")\n```\n\nThe differentiation between bold and italic text is only relevant for the Bonsai-SUT. Italic written codes are \"not part\" of the inspected `toplevelcode`, since these are explicitly in the SUTs. Since these codes are seperatly in the SUT, the definition of the `toplevelcode` is thus \"code, excluding the the italic children\".\n\n```\n\ud835\udc02_\ud835\udc16\ud835\udc22\ud835\udc27\ud835\udc1e\n\u251c\u2500\u2500 \ud835\udc1f\ud835\udc22_\ud835\udfd0\ud835\udfd2\ud835\udfd0\ud835\udfcf\ud835\udfcf\n\u251c\u2500\u2500 \ud835\ude0a_\ud835\ude0e\ud835\ude33\ud835\ude22\ud835\ude31\ud835\ude35\n\n```\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "\"Python package classifications of the util workflow stage of the BONSAI database\"",
"version": "0.3.10",
"project_urls": {
"Documentation": "http://bonsamurais.gitlab.io/bonsai/util/classifications",
"Homepage": "https://gitlab.com/bonsamurais/bonsai/util/classifications"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "2f7c13d106ebe07e2ba5c0f106465ee64d4a3389443ee27d071a80328a58e72e",
"md5": "346736ade8e4a78ab33ca8180edebd28",
"sha256": "d6f51ecf498463f130771eaa23ab54871764aa43d824347676ee79737a3712e0"
},
"downloads": -1,
"filename": "bonsai_classifications-0.3.10-py3-none-any.whl",
"has_sig": false,
"md5_digest": "346736ade8e4a78ab33ca8180edebd28",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 2924487,
"upload_time": "2025-01-16T12:26:07",
"upload_time_iso_8601": "2025-01-16T12:26:07.881349Z",
"url": "https://files.pythonhosted.org/packages/2f/7c/13d106ebe07e2ba5c0f106465ee64d4a3389443ee27d071a80328a58e72e/bonsai_classifications-0.3.10-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "c612654c19cb3300d7ebcea4e2b25e46b15cde0327243d85434abb4110ab1626",
"md5": "13d179aa104077d72644b3bf063da641",
"sha256": "59c03f792e8e70ee6e2428bdb8cdfefbdf196c0ae346f5421099d1e9f6024f88"
},
"downloads": -1,
"filename": "bonsai_classifications-0.3.10.tar.gz",
"has_sig": false,
"md5_digest": "13d179aa104077d72644b3bf063da641",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 3798485,
"upload_time": "2025-01-16T12:26:09",
"upload_time_iso_8601": "2025-01-16T12:26:09.994159Z",
"url": "https://files.pythonhosted.org/packages/c6/12/654c19cb3300d7ebcea4e2b25e46b15cde0327243d85434abb4110ab1626/bonsai_classifications-0.3.10.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-16 12:26:09",
"github": false,
"gitlab": true,
"bitbucket": false,
"codeberg": false,
"gitlab_user": "bonsamurais",
"gitlab_project": "bonsai",
"lcname": "bonsai-classifications"
}
JSON