# ✂️ CSV Trimming
[](https://badge.fury.io/py/csv-trimming)
[](https://pypi.org/project/csv-trimming/)
[](https://pypi.org/project/csv-trimming/)
[](https://pepy.tech/projects/csv-trimming)
[](https://github.com/LucaCappelletti94/csv_trimming/actions/)
[](https://app.codacy.com/gh/LucaCappelletti94/csv_trimming/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)
[CSV Trimming](https://github.com/LucaCappelletti94/csv_trimming) is a Python package designed to take messy CSVs — the kind you get from scraping websites, legacy systems, or poorly managed data — and transform them into clean, well-formatted CSVs with just one line of code. No need for complex setups or large language models. It’s simple, straightforward, and generally gets the job done.
## How do I install this package?
As usual, just download it using pip:
```shell
pip install csv_trimming
```
## How do I use this package?
The package is very simple to use, just load your CSV and pass it to the trimmer.
```python
import pandas as pd
from csv_trimming import CSVTrimmer
# Load your csv
csv = pd.read_csv("tests/documents/noisy/sicilia.csv")
# Instantiate the trimmer
trimmer = CSVTrimmer()
# And trim it
trimmed_csv = trimmer.trim(csv)
# That's it!
```
For instance, your input CSV to clean up may look like this at the beginning:
| | 0 | 1 | 2 | 3 | 4 |
|---|-----|-------------------------|---------|--------------------------------------------------|-----------|
| 0 | #RIF! | #RIF! | ......... | /// | ----- |
| 1 | ('surname',)('-',)(0,) | region | (""('surname',)('-',)(0,"),)(' ',)(1,) | province | surname |
| 2 | ------ | #RIF! | #RIF! | | |
| 3 | #RIF! | Calabria | ------- | Catanzaro | Rossi |
| 4 | 0 | Sicilia | _____ | Ragusa | Pinna |
| 5 | "" | Lombardia | ------ | Varese | Sbrana |
| 6 | 0 | Lazio | __ | Roma | Mair |
| 7 | _ | Sicilia | #RIF! | Messina | Ferrari |
| 8 | ----- | .. | "" | 0 | --------- |
And after the trimming, it will look like this:
| | region | province | surname |
|---|-----------|-----------|---------|
| 0 | Calabria | Catanzaro | Rossi |
| 1 | Sicilia | Ragusa | Pinna |
| 2 | Lombardia | Varese | Sbrana |
| 3 | Lazio | Roma | Mair |
| 4 | Sicilia | Messina | Ferrari |
Magic!
## Advanced trimming with row correlation
Sometimes, the CSVs you are working with may have a row correlation, meaning part of a given row is inserted in the next row. Such cases are common when the data-entry clerk wants to make the whole table fit in their screen, and in order to do so, they split the row in two. While this is clearly an extremely bad practice, it happens in the real world and the CSV Trimmer can handle it with a little help.
You just need to provide a function that defines which rows are correlated, and the CSV Trimmer will take care of the rest. While in this example we are using a rather simple function and a relatively clean CSV, the package can handle more complex cases.
```python
from typing import Tuple
import pandas as pd
from csv_trimming import CSVTrimmer
def simple_correlation_callback(
current_row: pd.Series,
next_row: pd.Series
) -> Tuple[bool, pd.Series]:
"""Return the correlation between two rows.
Parameters
----------
current_row : pd.Series
The current row being analyzed in the DataFrame.
next_row : pd.Series
The next row in the DataFrame.
Returns
-------
Tuple[bool, pd.Series]
A tuple with a boolean indicating if the rows are correlated
and a Series with the merged row.
"""
# All of the rows that have a subsequent correlated row are
# non-empty, and the subsequent correlated rows are always
# with the first cell empty.
if pd.isna(next_row.iloc[0]) and all(pd.notna(current_row)):
return True, pd.concat(
[
current_row,
pd.Series({"surname": next_row.iloc[-1]}),
]
)
return False, current_row
csv = pd.read_csv("tests/test.csv")
trimmer = CSVTrimmer(simple_correlation_callback)
result = trimmer.trim(csv)
```
In this case, our CSV looked like this at the beginning:
| | region | province |
|----|----------|-----------------|
| 0 | Campania | Caserta |
| 1 | | Ferrero |
| 2 | Liguria | Imperia |
| 3 | | Conti |
| 4 | Puglia | Bari |
| 5 | | Fabris |
| 6 | Sardegna | Medio Campidano |
| 7 | | Conti |
| 8 | Lazio | Roma |
| 9 | | Fabbri |
And after the trimming, it will look like this:
| | region | province | surname |
|----|----------|-----------------|---------|
| 0 | Campania | Caserta | Ferrero |
| 1 | Liguria | Imperia | Conti |
| 2 | Puglia | Bari | Fabris |
| 3 | Sardegna | Medio Campidano | Conti |
| 4 | Lazio | Roma | Fabbri |
## More examples
Here follow some examples of the package in action.
### Case with duplicated schemas
Sometimes, when chaining multiple CSVs in a poor manner, you may end up with duplicated schemas.
The CSV Trimmer detects rows that match the detected header, and it can (optionally) remove them.
```python
import pandas as pd
from csv_trimming import CSVTrimmer
# Load your csv
csv = pd.read_csv("tests/documents/noisy/duplicated_schema.csv")
# Instantiate the trimmer
trimmer = CSVTrimmer()
# And trim it
trimmed_csv = trimmer.trim(csv, drop_duplicated_schema=True)
# That's it!
```
For instance, your input CSV to clean up may look like this at the beginning:
| | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
|----|------------|------------------------------|--------|-------------------------------------------|------------------------------|------------------------------|------------|----------|
| 0 | #RIF! | //// | #RIF! | #RIF! | 0 | .... | 0 | 0 |
| 1 | | ('surname',)('.',)(0,) | region | province | surname | ('province',)('_',)(1,) | | 0 |
| 2 | 0 | //////// | region | province | surname | 0 | 0 | |
| 3 | _____ | /////// | region | province | surname | #RIF! | #RIF! | |
| 4 | | | Puglia | Bari | Zanetti | 0 | -------- |
| 5 | 0 | | Piemonte| Alessandria | Fabbri | | | |
| 6 | 0 | ------- | | #RIF! | #RIF! | 0 | | ---- |
| 7 | ///////// | ///////// | Sicilia| Agrigento | Ferretti | ////////// | | ----------|
| 8 | __ | -------- | Campania| Napoli | Belotti | | /// | |
| 9 | | -------- | 0 | ///// | --- | 0 | ///// | ----------|
| 10 | ----- | #RIF! | Liguria| Savona | Casini | 0 | | #RIF! |
| 11 | ... | 0 | | ----- | | -------- | 0 | 0 |
And after the trimming, it will look like this:
| | region | province | surname |
|---|----------|-------------|---------|
| 0 | Puglia | Bari | Zanetti |
| 1 | Piemonte | Alessandria | Fabbri |
| 2 | Sicilia | Agrigento | Ferretti|
| 3 | Campania | Napoli | Belotti |
| 4 | Liguria | Savona | Casini |
### Case with only padding
Sometimes, the data entry clerk may start filling a table offsetted from the top-left corner, and export it with also
empty cells all around. We call such cells "padding". The CSV Trimmer can detect and remove them.
```python
import pandas as pd
from csv_trimming import CSVTrimmer
# Load your csv
csv = pd.read_csv("tests/documents/noisy/padding.csv")
# Instantiate the trimmer
trimmer = CSVTrimmer()
# And trim it
trimmed_csv = trimmer.trim(csv, drop_padding=True)
```
For instance, your input CSV to clean up may look like this at the beginning:
| | | region | province | surname |
|---|---|----------|----------------|---------|
| 0 | | | | |
| 1 | | | | |
| 2 | | region | province | surname |
| 3 | | Campania | Caserta | Ferrero |
| 4 | | Liguria | Imperia | Conti |
| 5 | | Puglia | Bari | Fabris |
| 6 | | Sardegna | Medio Campidano| Conti |
| 7 | | Lazio | Roma | Fabbri |
| 8 | | | | |
| 9 | | | | |
| 10| | | | |
| 11| | | | |
And after the trimming, it will look like this:
| | region | province | surname |
|---|----------|----------------|---------|
| 0 | Campania | Caserta | Ferrero |
| 1 | Liguria | Imperia | Conti |
| 2 | Puglia | Bari | Fabris |
| 3 | Sardegna | Medio Campidano| Conti |
| 4 | Lazio | Roma | Fabbri |
## Command Line Interface
The package also provides a command line interface to trim CSVs. It comes installed with the `setup.py` of the package, therefore after having pip installed the package, you can immediately use it from the command line.
You can use it by running the following command:
```shell
csv-trim tests/documents/noisy/sicilia.csv tests/documents/noisy/sicilia_trimmed.csv
```
It supports the following options to keep it from attempting some trimmings:
- `--keep-padding`: Do not attempt to remove padding.
- `--keep-duplicated-schema`: Do not attempt to remove duplicated schemas.
- `--no-restore-header`: Do not attempt to restore the header.
For instance:
```shell
csv-trim tests/documents/noisy/sicilia.csv tests/documents/noisy/sicilia_trimmed.csv --keep-padding
```
## How do I contribute to this package?
If you have identified some new corner case that the package does not handle, or you have a suggestion for a new feature, feel free to open an issue. If you want to contribute with code, open an issue describing the feature you intend to add and submit a pull request.
## License
This package is released under MIT license.
Raw data
{
"_id": null,
"home_page": "https://github.com/LucaCappelletti94/csv_trimming",
"name": "csv-trimming",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": null,
"author": "LucaCappelletti94",
"author_email": "cappelletti.luca94@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/6a/3b/6473544d571b9d405f7eec84c1729d6e774b02fe56a482f4e5b8409a26b5/csv_trimming-1.1.1.tar.gz",
"platform": null,
"description": "# \u2702\ufe0f CSV Trimming\n\n[](https://badge.fury.io/py/csv-trimming)\n[](https://pypi.org/project/csv-trimming/)\n[](https://pypi.org/project/csv-trimming/)\n[](https://pepy.tech/projects/csv-trimming)\n[](https://github.com/LucaCappelletti94/csv_trimming/actions/)\n[](https://app.codacy.com/gh/LucaCappelletti94/csv_trimming/dashboard?utm_source=gh&utm_medium=referral&utm_content=&utm_campaign=Badge_grade)\n\n[CSV Trimming](https://github.com/LucaCappelletti94/csv_trimming) is a Python package designed to take messy CSVs \u2014 the kind you get from scraping websites, legacy systems, or poorly managed data \u2014 and transform them into clean, well-formatted CSVs with just one line of code. No need for complex setups or large language models. It\u2019s simple, straightforward, and generally gets the job done.\n\n## How do I install this package?\n\nAs usual, just download it using pip:\n\n```shell\npip install csv_trimming\n```\n\n## How do I use this package?\nThe package is very simple to use, just load your CSV and pass it to the trimmer.\n\n```python\nimport pandas as pd\nfrom csv_trimming import CSVTrimmer\n\n# Load your csv\ncsv = pd.read_csv(\"tests/documents/noisy/sicilia.csv\")\n# Instantiate the trimmer\ntrimmer = CSVTrimmer()\n# And trim it\ntrimmed_csv = trimmer.trim(csv)\n# That's it!\n```\n\nFor instance, your input CSV to clean up may look like this at the beginning:\n\n| | 0 | 1 | 2 | 3 | 4 |\n|---|-----|-------------------------|---------|--------------------------------------------------|-----------|\n| 0 | #RIF! | #RIF! | ......... | /// | ----- |\n| 1 | ('surname',)('-',)(0,) | region | (\"\"('surname',)('-',)(0,\"),)(' ',)(1,) | province | surname |\n| 2 | ------ | #RIF! | #RIF! | | |\n| 3 | #RIF! | Calabria | ------- | Catanzaro | Rossi |\n| 4 | 0 | Sicilia | _____ | Ragusa | Pinna |\n| 5 | \"\" | Lombardia | ------ | Varese | Sbrana |\n| 6 | 0 | Lazio | __ | Roma | Mair |\n| 7 | _ | Sicilia | #RIF! | Messina | Ferrari |\n| 8 | ----- | .. | \"\" | 0 | --------- |\n\nAnd after the trimming, it will look like this:\n\n| | region | province | surname |\n|---|-----------|-----------|---------|\n| 0 | Calabria | Catanzaro | Rossi |\n| 1 | Sicilia | Ragusa | Pinna |\n| 2 | Lombardia | Varese | Sbrana |\n| 3 | Lazio | Roma | Mair |\n| 4 | Sicilia | Messina | Ferrari |\n\nMagic!\n\n## Advanced trimming with row correlation\nSometimes, the CSVs you are working with may have a row correlation, meaning part of a given row is inserted in the next row. Such cases are common when the data-entry clerk wants to make the whole table fit in their screen, and in order to do so, they split the row in two. While this is clearly an extremely bad practice, it happens in the real world and the CSV Trimmer can handle it with a little help.\n\nYou just need to provide a function that defines which rows are correlated, and the CSV Trimmer will take care of the rest. While in this example we are using a rather simple function and a relatively clean CSV, the package can handle more complex cases.\n\n```python\nfrom typing import Tuple\nimport pandas as pd\nfrom csv_trimming import CSVTrimmer\n\ndef simple_correlation_callback(\n current_row: pd.Series,\n next_row: pd.Series\n) -> Tuple[bool, pd.Series]:\n \"\"\"Return the correlation between two rows.\n \n Parameters\n ----------\n current_row : pd.Series\n The current row being analyzed in the DataFrame.\n next_row : pd.Series\n The next row in the DataFrame.\n\n Returns\n -------\n Tuple[bool, pd.Series]\n A tuple with a boolean indicating if the rows are correlated\n and a Series with the merged row.\n \"\"\"\n\n # All of the rows that have a subsequent correlated row are\n # non-empty, and the subsequent correlated rows are always\n # with the first cell empty.\n if pd.isna(next_row.iloc[0]) and all(pd.notna(current_row)):\n return True, pd.concat(\n [\n current_row,\n pd.Series({\"surname\": next_row.iloc[-1]}),\n ]\n )\n\n return False, current_row\n\ncsv = pd.read_csv(\"tests/test.csv\")\ntrimmer = CSVTrimmer(simple_correlation_callback)\nresult = trimmer.trim(csv)\n```\n\nIn this case, our CSV looked like this at the beginning:\n\n| | region | province |\n|----|----------|-----------------|\n| 0 | Campania | Caserta |\n| 1 | | Ferrero |\n| 2 | Liguria | Imperia |\n| 3 | | Conti |\n| 4 | Puglia | Bari |\n| 5 | | Fabris |\n| 6 | Sardegna | Medio Campidano |\n| 7 | | Conti |\n| 8 | Lazio | Roma |\n| 9 | | Fabbri |\n\n\nAnd after the trimming, it will look like this:\n\n| | region | province | surname |\n|----|----------|-----------------|---------|\n| 0 | Campania | Caserta | Ferrero |\n| 1 | Liguria | Imperia | Conti |\n| 2 | Puglia | Bari | Fabris |\n| 3 | Sardegna | Medio Campidano | Conti |\n| 4 | Lazio | Roma | Fabbri |\n\n## More examples\nHere follow some examples of the package in action.\n\n### Case with duplicated schemas\nSometimes, when chaining multiple CSVs in a poor manner, you may end up with duplicated schemas.\nThe CSV Trimmer detects rows that match the detected header, and it can (optionally) remove them.\n\n```python\nimport pandas as pd\nfrom csv_trimming import CSVTrimmer\n\n# Load your csv\ncsv = pd.read_csv(\"tests/documents/noisy/duplicated_schema.csv\")\n# Instantiate the trimmer\ntrimmer = CSVTrimmer()\n# And trim it\ntrimmed_csv = trimmer.trim(csv, drop_duplicated_schema=True)\n# That's it!\n```\n\nFor instance, your input CSV to clean up may look like this at the beginning:\n\n| | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 |\n|----|------------|------------------------------|--------|-------------------------------------------|------------------------------|------------------------------|------------|----------|\n| 0 | #RIF! | //// | #RIF! | #RIF! | 0 | .... | 0 | 0 |\n| 1 | | ('surname',)('.',)(0,) | region | province | surname | ('province',)('_',)(1,) | | 0 |\n| 2 | 0 | //////// | region | province | surname | 0 | 0 | |\n| 3 | _____ | /////// | region | province | surname | #RIF! | #RIF! | |\n| 4 | | | Puglia | Bari | Zanetti | 0 | -------- |\n| 5 | 0 | | Piemonte| Alessandria | Fabbri | | | |\n| 6 | 0 | ------- | | #RIF! | #RIF! | 0 | | ---- |\n| 7 | ///////// | ///////// | Sicilia| Agrigento | Ferretti | ////////// | | ----------|\n| 8 | __ | -------- | Campania| Napoli | Belotti | | /// | |\n| 9 | | -------- | 0 | ///// | --- | 0 | ///// | ----------|\n| 10 | ----- | #RIF! | Liguria| Savona | Casini | 0 | | #RIF! |\n| 11 | ... | 0 | | ----- | | -------- | 0 | 0 |\n\nAnd after the trimming, it will look like this:\n\n| | region | province | surname |\n|---|----------|-------------|---------|\n| 0 | Puglia | Bari | Zanetti |\n| 1 | Piemonte | Alessandria | Fabbri |\n| 2 | Sicilia | Agrigento | Ferretti|\n| 3 | Campania | Napoli | Belotti |\n| 4 | Liguria | Savona | Casini |\n\n### Case with only padding\nSometimes, the data entry clerk may start filling a table offsetted from the top-left corner, and export it with also\nempty cells all around. We call such cells \"padding\". The CSV Trimmer can detect and remove them.\n\n```python\nimport pandas as pd\nfrom csv_trimming import CSVTrimmer\n\n# Load your csv\ncsv = pd.read_csv(\"tests/documents/noisy/padding.csv\")\n\n# Instantiate the trimmer\ntrimmer = CSVTrimmer()\n\n# And trim it\ntrimmed_csv = trimmer.trim(csv, drop_padding=True)\n```\n\nFor instance, your input CSV to clean up may look like this at the beginning:\n\n| | | region | province | surname |\n|---|---|----------|----------------|---------|\n| 0 | | | | |\n| 1 | | | | |\n| 2 | | region | province | surname |\n| 3 | | Campania | Caserta | Ferrero |\n| 4 | | Liguria | Imperia | Conti |\n| 5 | | Puglia | Bari | Fabris |\n| 6 | | Sardegna | Medio Campidano| Conti |\n| 7 | | Lazio | Roma | Fabbri |\n| 8 | | | | |\n| 9 | | | | |\n| 10| | | | |\n| 11| | | | |\n\nAnd after the trimming, it will look like this:\n\n| | region | province | surname |\n|---|----------|----------------|---------|\n| 0 | Campania | Caserta | Ferrero |\n| 1 | Liguria | Imperia | Conti |\n| 2 | Puglia | Bari | Fabris |\n| 3 | Sardegna | Medio Campidano| Conti |\n| 4 | Lazio | Roma | Fabbri |\n\n\n## Command Line Interface\nThe package also provides a command line interface to trim CSVs. It comes installed with the `setup.py` of the package, therefore after having pip installed the package, you can immediately use it from the command line.\n\nYou can use it by running the following command:\n\n```shell\ncsv-trim tests/documents/noisy/sicilia.csv tests/documents/noisy/sicilia_trimmed.csv\n```\n\nIt supports the following options to keep it from attempting some trimmings:\n\n- `--keep-padding`: Do not attempt to remove padding.\n- `--keep-duplicated-schema`: Do not attempt to remove duplicated schemas.\n- `--no-restore-header`: Do not attempt to restore the header.\n\nFor instance:\n \n```shell\ncsv-trim tests/documents/noisy/sicilia.csv tests/documents/noisy/sicilia_trimmed.csv --keep-padding\n```\n\n## How do I contribute to this package?\nIf you have identified some new corner case that the package does not handle, or you have a suggestion for a new feature, feel free to open an issue. If you want to contribute with code, open an issue describing the feature you intend to add and submit a pull request.\n\n## License\nThis package is released under MIT license.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Package python to remove common ugliness from a csv-like file",
"version": "1.1.1",
"project_urls": {
"Homepage": "https://github.com/LucaCappelletti94/csv_trimming"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "6a3b6473544d571b9d405f7eec84c1729d6e774b02fe56a482f4e5b8409a26b5",
"md5": "088457ce0258be47ff5b7bb13f91e06e",
"sha256": "17fa6a276a6a0e9c0eadd328ecf1947e638f580515870f351d32a225026b7ab3"
},
"downloads": -1,
"filename": "csv_trimming-1.1.1.tar.gz",
"has_sig": false,
"md5_digest": "088457ce0258be47ff5b7bb13f91e06e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 17445,
"upload_time": "2024-09-02T13:51:15",
"upload_time_iso_8601": "2024-09-02T13:51:15.286617Z",
"url": "https://files.pythonhosted.org/packages/6a/3b/6473544d571b9d405f7eec84c1729d6e774b02fe56a482f4e5b8409a26b5/csv_trimming-1.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-02 13:51:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "LucaCappelletti94",
"github_project": "csv_trimming",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "csv-trimming"
}
JSON
