==========
csvinsight
==========
.. image:: https://img.shields.io/pypi/v/csvinsight.svg
:target: https://pypi.python.org/pypi/csvinsight
.. image:: https://circleci.com/gh/ProfoundNetworks/csvinsight.svg?style=shield&circle-token=:circle-token
:target: https://circleci.com/gh/ProfoundNetworks/csvinsight
:alt: Build Status
.. image:: https://readthedocs.org/projects/csvinsight/badge/?version=latest
:target: https://csvinsight.readthedocs.io/en/latest/?badge=latest
:alt: Documentation Status
.. image:: https://pyup.io/repos/github/ProfoundNetworks/csvinsight/shield.svg
:target: https://pyup.io/repos/github/ProfoundNetworks/csvinsight/
:alt: Updates
Fast & simple summary for large CSV files
* Free software: MIT license
* Documentation: https://csvinsight.readthedocs.io.
Features
--------
* Calculates basic stats for each column: max, min, mean length; number of non-empty values
* Calculates exact number of unique values and the top 20 most frequent values
* Supports non-orthogonal data (list fields)
* Works with very large files: does not load the entire CSV into memory
* Fast splitting of CSVs into columns, one file per column
* Multiprocessing-enabled
Example Usage
-------------
Given a CSV file::
bash-3.2$ cat tests/sampledata.csv
name|age|fave_color
Alexey|33|red;yellow
Boris|31|blue
Valentina|0|
you can obtain a CsvInsight report with::
bash-3.2$ csvi tests/sampledata.csv --list-fields fave_color
CSV Insight Report
Total # Rows: 3
Column counts:
3 columns -> 3 rows
Report Format:
Column Number. Column Header -> Uniques: # ; Fills: # ; Fill Rate:
Field Length: min #, max #, average:
Top n field values -> Dupe Counts
1. name -> Uniques: 3 ; Fills: 3 ; Fill Rate: 100.0%
Field Length: min 5, max 9, avg 6.67
Counts Percent Field Value
1 33.33 % Valentina
1 33.33 % Boris
1 33.33 % Alexey
2. age -> Uniques: 3 ; Fills: 3 ; Fill Rate: 100.0%
Field Length: min 1, max 2, avg 1.67
Counts Percent Field Value
1 33.33 % 33
1 33.33 % 31
1 33.33 % 0
3. fave_color -> Uniques: 4 ; Fills: 3 ; Fill Rate: 75.0%
Field Length: min 0, max 6, avg 3.25
Counts Percent Field Value
1 25.00 % yellow
1 25.00 % red
1 25.00 % blue
1 25.00 % NULL
Since CSV comes in different flavors, you may need to tweak the underlying CSV parser's parameters to read your file successfully.
CSVInsight handles this via CSV dialects.
For example, to read a comma-separated file, you would use the following command::
bash-3.2$ csvi your/file.csv --dialect delimiter=,
You may combine as many dialect parameters as needed::
bash-3.2$ csvi your/file.csv --dialect delimiter=, quoting=QUOTE_NONE
For a full list of dialect parameters, see the documentation for Python's `csv module <https://docs.python.org/3.6/library/csv.html#dialects-and-formatting-parameters>`_.
Constant values like QUOTE_NONE are resolved automagically.
Once you've discovered the winning parameter combination for your file, save it to a YAML file::
list_fields:
- fave_color
- another_field_name
list_separator: ;
dialect:
- "delimiter=|"
- "quoting=QUOTE_NONE"
You can then invoke CSVI as follows::
bash-3.2$ csvi your/file.csv --config your/config.yaml
Credits
---------
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
=======
History
=======
Unreleased
----------
0.4.0 (2024-04-09)
------------------
* Drop support for Py2.7, get rid of dependency on six, futures imports, etc.
* Drop support for Py3.7 and below
* Get rid of ipynb feature
* Correctly handle empty quotechar and escapechar dialect parameters
* Correctly handle doublequote dialect parameter
0.3.3 (2020-12-02)
------------------
* Handle numeric quoting parameter, e.g. "--dialect quoting=3"
0.3.2 (2019-07-01)
------------------
* Set the field size limit to sys.maxsize
0.3.1 (2019-06-26)
------------------
* Make Jupyter notebook an optional dependency
0.3.0 (2018-07-11)
------------------
* Added --most-common parameter (resolved Issue #14)
* Added --no-tiny parameter
* Refactored temporary file naming
* Improve error message when handling empty CSV files
* Fixed "Argument list too long" error (Issue #15)
* Added --json parameter
* Added --ipynb parameter to generate IPython notebook
0.2.3 (2017-12-09)
------------------
* Fix bug: Unicode column names now work under Py2
0.2.2 (2017-12-04)
------------------
* Fix bug: Unicode characters no longer break CsvInsight on Py2
0.2.1 (2017-11-27)
------------------
* Fix bug: opening gzipped files with Py3 now works
0.2.0 (2017-11-25)
------------------
* Split files using gsplit and process them in parallel for faster processing
* No longer work with streams; works exclusively with files
* Get rid of csvi_summarize and csvi_split entry points
* Integrated plumbum for cleaner pipelines
* Fixed issue #11: added support for more CSV parameters via the --dialect option
* Fixed issue #10: reading from empty files no longer raises StopIteration
* Fixed issue #8: use the correct link to the GitHub project in the documentation
* Fixed issue #2: implemented in-memory mode for smaller files
0.1.0 (2017-10-29)
------------------
* First release on PyPI.
Raw data
{
"_id": null,
"home_page": "https://github.com/ProfoundNetworks/csvinsight",
"name": "csvinsight",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "csvinsight",
"author": "Michael Penkov",
"author_email": "m@penkov.dev",
"download_url": null,
"platform": null,
"description": "==========\ncsvinsight\n==========\n\n\n.. image:: https://img.shields.io/pypi/v/csvinsight.svg\n :target: https://pypi.python.org/pypi/csvinsight\n\n.. image:: https://circleci.com/gh/ProfoundNetworks/csvinsight.svg?style=shield&circle-token=:circle-token\n :target: https://circleci.com/gh/ProfoundNetworks/csvinsight\n :alt: Build Status\n\n.. image:: https://readthedocs.org/projects/csvinsight/badge/?version=latest\n :target: https://csvinsight.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://pyup.io/repos/github/ProfoundNetworks/csvinsight/shield.svg\n :target: https://pyup.io/repos/github/ProfoundNetworks/csvinsight/\n :alt: Updates\n\n\nFast & simple summary for large CSV files\n\n\n* Free software: MIT license\n* Documentation: https://csvinsight.readthedocs.io.\n\n\nFeatures\n--------\n\n* Calculates basic stats for each column: max, min, mean length; number of non-empty values\n* Calculates exact number of unique values and the top 20 most frequent values\n* Supports non-orthogonal data (list fields)\n* Works with very large files: does not load the entire CSV into memory\n* Fast splitting of CSVs into columns, one file per column\n* Multiprocessing-enabled\n\nExample Usage\n-------------\n\nGiven a CSV file::\n\n bash-3.2$ cat tests/sampledata.csv\n name|age|fave_color\n Alexey|33|red;yellow\n Boris|31|blue\n Valentina|0|\n\nyou can obtain a CsvInsight report with::\n\n bash-3.2$ csvi tests/sampledata.csv --list-fields fave_color\n CSV Insight Report\n Total # Rows: 3\n Column counts:\n 3 columns -> 3 rows\n\n Report Format:\n Column Number. Column Header -> Uniques: # ; Fills: # ; Fill Rate:\n Field Length: min #, max #, average:\n Top n field values -> Dupe Counts\n\n\n 1. name -> Uniques: 3 ; Fills: 3 ; Fill Rate: 100.0%\n Field Length: min 5, max 9, avg 6.67\n Counts Percent Field Value\n 1 33.33 % Valentina\n 1 33.33 % Boris\n 1 33.33 % Alexey\n\n 2. age -> Uniques: 3 ; Fills: 3 ; Fill Rate: 100.0%\n Field Length: min 1, max 2, avg 1.67\n Counts Percent Field Value\n 1 33.33 % 33\n 1 33.33 % 31\n 1 33.33 % 0\n\n 3. fave_color -> Uniques: 4 ; Fills: 3 ; Fill Rate: 75.0%\n Field Length: min 0, max 6, avg 3.25\n Counts Percent Field Value\n 1 25.00 % yellow\n 1 25.00 % red\n 1 25.00 % blue\n 1 25.00 % NULL\n\nSince CSV comes in different flavors, you may need to tweak the underlying CSV parser's parameters to read your file successfully.\nCSVInsight handles this via CSV dialects.\nFor example, to read a comma-separated file, you would use the following command::\n\n bash-3.2$ csvi your/file.csv --dialect delimiter=,\n\nYou may combine as many dialect parameters as needed::\n\n bash-3.2$ csvi your/file.csv --dialect delimiter=, quoting=QUOTE_NONE\n\nFor a full list of dialect parameters, see the documentation for Python's `csv module <https://docs.python.org/3.6/library/csv.html#dialects-and-formatting-parameters>`_.\nConstant values like QUOTE_NONE are resolved automagically.\n\nOnce you've discovered the winning parameter combination for your file, save it to a YAML file::\n\n list_fields:\n - fave_color\n - another_field_name\n list_separator: ;\n dialect:\n - \"delimiter=|\"\n - \"quoting=QUOTE_NONE\"\n\nYou can then invoke CSVI as follows::\n\n bash-3.2$ csvi your/file.csv --config your/config.yaml\n\nCredits\n---------\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n\n\n=======\nHistory\n=======\n\nUnreleased\n----------\n\n0.4.0 (2024-04-09)\n------------------\n\n* Drop support for Py2.7, get rid of dependency on six, futures imports, etc.\n* Drop support for Py3.7 and below\n* Get rid of ipynb feature\n* Correctly handle empty quotechar and escapechar dialect parameters\n* Correctly handle doublequote dialect parameter\n\n0.3.3 (2020-12-02)\n------------------\n\n* Handle numeric quoting parameter, e.g. \"--dialect quoting=3\"\n\n0.3.2 (2019-07-01)\n------------------\n\n* Set the field size limit to sys.maxsize\n\n0.3.1 (2019-06-26)\n------------------\n\n* Make Jupyter notebook an optional dependency\n\n0.3.0 (2018-07-11)\n------------------\n\n* Added --most-common parameter (resolved Issue #14)\n* Added --no-tiny parameter\n* Refactored temporary file naming\n* Improve error message when handling empty CSV files\n* Fixed \"Argument list too long\" error (Issue #15)\n* Added --json parameter\n* Added --ipynb parameter to generate IPython notebook\n\n0.2.3 (2017-12-09)\n------------------\n\n* Fix bug: Unicode column names now work under Py2\n\n0.2.2 (2017-12-04)\n------------------\n\n* Fix bug: Unicode characters no longer break CsvInsight on Py2\n\n0.2.1 (2017-11-27)\n------------------\n\n* Fix bug: opening gzipped files with Py3 now works\n\n0.2.0 (2017-11-25)\n------------------\n\n* Split files using gsplit and process them in parallel for faster processing\n* No longer work with streams; works exclusively with files\n* Get rid of csvi_summarize and csvi_split entry points\n* Integrated plumbum for cleaner pipelines\n* Fixed issue #11: added support for more CSV parameters via the --dialect option\n* Fixed issue #10: reading from empty files no longer raises StopIteration\n* Fixed issue #8: use the correct link to the GitHub project in the documentation\n* Fixed issue #2: implemented in-memory mode for smaller files\n\n0.1.0 (2017-10-29)\n------------------\n\n* First release on PyPI.\n",
"bugtrack_url": null,
"license": "MIT license",
"summary": "Fast & simple summary for large CSV files",
"version": "0.4.0",
"project_urls": {
"Homepage": "https://github.com/ProfoundNetworks/csvinsight"
},
"split_keywords": [
"csvinsight"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c3f4e7c8ea14384b7ab535217a432f5cf10fbdc460f464902bcda405cd38fe5e",
"md5": "bbcb58dca7aac6d0bb1d9b750aa0afc8",
"sha256": "202dfa02f87552f66606a304ffe74be8d8f44d3ef3dd966741a3530c1fe9d58c"
},
"downloads": -1,
"filename": "csvinsight-0.4.0-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "bbcb58dca7aac6d0bb1d9b750aa0afc8",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 15251,
"upload_time": "2024-04-09T08:42:28",
"upload_time_iso_8601": "2024-04-09T08:42:28.157985Z",
"url": "https://files.pythonhosted.org/packages/c3/f4/e7c8ea14384b7ab535217a432f5cf10fbdc460f464902bcda405cd38fe5e/csvinsight-0.4.0-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-09 08:42:28",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ProfoundNetworks",
"github_project": "csvinsight",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "csvinsight"
}
JSON
