Name | catalystcoop.ferc-xbrl-extractor JSON |
Version |
1.4.0
JSON |
| download |
home_page | None |
Summary | A tool for extracting data from FERC XBRL Filings. |
upload_time | 2024-04-15 01:15:08 |
maintainer | None |
docs_url | None |
author | None |
requires_python | <3.13,>=3.10 |
license | MIT License Copyright (c) 2022 Catalyst Cooperative Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
keywords |
accounting
data
electricity
energy
federal energy regulatory commission
ferc
finance
gas
natural gas
oil
regulation
utility
xbrl
|
VCS |
|
bugtrack_url |
|
requirements |
No requirements were recorded.
|
Travis-CI |
No Travis.
|
coveralls test coverage |
No coveralls.
|
===============================================================================
FERC XBRL Extractor
===============================================================================
.. readme-intro
.. image:: https://www.repostatus.org/badges/latest/active.svg
:target: https://www.repostatus.org/#active
:alt: Project Status: Active
.. image:: https://github.com/catalyst-cooperative/ferc-xbrl-extractor/workflows/tox-pytest/badge.svg
:target: https://github.com/catalyst-cooperative/ferc-xbrl-extractor/actions/workflows/tox-pytest.yml
:alt: Tox-PyTest Status
.. image:: https://img.shields.io/codecov/c/github/catalyst-cooperative/ferc-xbrl-extractor?style=flat&logo=codecov
:target: https://codecov.io/gh/catalyst-cooperative/ferc-xbrl-extractor
:alt: Codecov Test Coverage
.. image:: https://img.shields.io/readthedocs/catalystcoop-ferc-xbrl-extractor?style=flat&logo=readthedocs
:target: https://catalystcoop-ferc-xbrl-extractor.readthedocs.io/en/latest/
:alt: Read the Docs Build Status
.. image:: https://img.shields.io/pypi/v/catalystcoop.ferc-xbrl-extractor
:target: https://pypi.org/project/catalystcoop.ferc-xbrl-extractor/
:alt: PyPI Latest Version
.. image:: https://img.shields.io/conda/vn/conda-forge/catalystcoop.ferc_xbrl_extractor
:target: https://anaconda.org/conda-forge/catalystcoop.ferc_xbrl_extractor
:alt: conda-forge Version
.. image:: https://img.shields.io/pypi/pyversions/catalystcoop.ferc-xbrl-extractor
:target: https://pypi.org/project/catalystcoop.ferc-xbrl-extractor/
:alt: Supported Python Versions
.. image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black
:alt: Any color you want, so long as it's black.
.. image:: https://results.pre-commit.ci/badge/github/catalyst-cooperative/ferc-xbrl-extractor/main.svg
:target: https://results.pre-commit.ci/latest/github/catalyst-cooperative/ferc-xbrl-extractor/main
:alt: pre-commit CI
.. image:: https://zenodo.org/badge/471019769.svg
:target: https://zenodo.org/doi/10.5281/zenodo.10020145
:alt: Zenodo DOI
The Federal Energy Regulatory Commission (FERC) has moved to collecting and distributing
data using `XBRL <https://en.wikipedia.org/wiki/XBRL>`__. XBRL is primarily designed for
financial reporting, and has been adopted by regulators in the US and other countries.
Much of the tooling in the XBRL ecosystem is targeted towards filers, and rendering
individual filings in a human readable way, but there is very little targeted towards
accessing and analyzing large collections of filings. This tool is designed to provide
that functionality for FERC XBRL data. Specifically, it can extract data from a set of
XBRL filings, and write that data to a SQLite database whose structure is generated from
an XBRL Taxonomy. While each XBRL instance contains a reference to a taxonomy,
this tool requires a path to a single taxonomy that will be used to interpret all
instances being processed. This means even if instances were created from different
versions of a Taxonomy, the provided taxonomy will be used when processing all of these
instances, so the output database will have a consistent structure. For more information
on the technical details of the XBRL extraction, see the docs.
We are currently using this tool to extract and publish the following FERC data:
.. list-table::
:header-rows: 1
* - FERC Form
- Archived XBRL
- SQLite DB
* - `Form 1 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annual>`__
- https://doi.org/10.5281/zenodo.4127043
- https://data.catalyst.coop/ferc1_xbrl
* - `Form 2 <https://www.ferc.gov/industries-data/natural-gas/industry-forms/form-2-2a-3-q-gas-historical-vfp-data>`__
- https://doi.org/10.5281/zenodo.5879542
- https://data.catalyst.coop/ferc2_xbrl
* - `Form 6 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-66-q-overview-orders>`__
- https://doi.org/10.5281/zenodo.7126395
- https://data.catalyst.coop/ferc6_xbrl
* - `Form 60 <https://www.ferc.gov/ferc-online/ferc-online/filing-forms/service-companies-filing-forms/form-60-annual-report>`_
- https://doi.org/10.5281/zenodo.7126434
- https://data.catalyst.coop/ferc60_xbrl
* - `Form 714 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric>`__
- https://doi.org/10.5281/zenodo.4127100
- https://data.catalyst.coop/ferc714_xbrl
Usage
-----
Installation
^^^^^^^^^^^^
To install using conda, run the following command, and activate the environment.
.. code-block:: console
$ conda env create -f environment.yml
Activate:
.. code-block:: console
$ conda activate ferc-xbrl-extractor
CLI
^^^
This tool can be used as a library, as it is in `PUDL <https://github.com/catalyst-cooperative/pudl>`__,
or there is a CLI provided for interacting with XBRL data. The only required options
for the CLI are a path to the filings to be extracted, and a path to the output
SQLite database. The path to the filings can point to a directory full of XBRL
Filings, a single XBRL filing, or a zipfile with XBRL filings. If
the path to the database points to an existing database, the ``--clobber`` option
can be used to drop all existing data before performing the extraction.
.. code-block:: console
$ xbrl_extract {path_to_filings} {path_to_database}
This repo contains a small selection of FERC Form 1 filings from 2021 in the
``examples`` directory. To test the tool on these filings, use the command
.. code-block:: console
$ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite
By default, the CLI will use the 2022 version of the FERC Form 1 Taxonomy to create
the structure of the output database, and it will obtain the taxonomy directly from
the FERC web server. To specify a different taxonomy use the ``--taxonomy`` option. This
can be either a URL pointing to a taxonomy entry point, or a path to a local zipfile.
The following command demonstrates this option using the same URL that would be used by
default.
.. code-block:: console
$ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \
--taxonomy https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd
When using the ``--taxonomy`` option to read the taxonomy from a zipfile, the
``--archive-path`` option is also required. This option is used to indicate the path to
the taxonomy entry point within the zipfile. NOTE: This will likely change in the future
to automatically detect the entry point as this interface is clunky. To demonstrate, the
FERC Form 1 2021 taxonomy is included in the examples directory. This can be used with
the following command.
.. code-block:: console
$ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \
--taxonomy examples/taxonomy.zip \
--archive-path taxonomy/form1/2021-01-01/form/form1/form-1_2021-01-01.xsd
Parsing XBRL filings can be a time consuming and CPU heavy task, so this tool
implements some basic multiprocessing to speed this up. It uses a
`process pool <https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor>`__
to do this. There are two options for configuring the process pool, ``--batch-size``
and ``--workers``. The batch size configures how many filings will be processed by
each child process at a time, and workers specifies how many child processes to
create in the pool. It may take some experimentation to get these options
optimally configured. The following command will use 5 worker processes to process
batches of 50 filings at a time.
.. code-block:: console
$ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \
--workers 5 \
--batch-size 50
There are also several options included for extracting metadata from the taxonomy.
First is the ``--datapackage-path`` command to save a
`frictionless datapackage <https://specs.frictionlessdata.io/data-package/>`__
descriptor as JSON, which annotates the generated SQLite database. There is also the
``--metadata-path`` option, which writes more extensive taxonomy metadata to a json
file, grouped by table name. See the ``ferc_xbrl_extractor.arelle_interface`` module
for more info on the extracted metadata. To create both of these files using the example
filings and taxonomy, run the following command.
.. code-block:: console
$ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \
--taxonomy examples/taxonomy.zip \
--archive-path taxonomy/form1/2021-01-01/form/form1/form-1_2021-01-01.xsd \
--metadata-path metadata.json \
--datapackage-path datapackage.json
Raw data
{
"_id": null,
"home_page": null,
"name": "catalystcoop.ferc-xbrl-extractor",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": "accounting, data, electricity, energy, federal energy regulatory commission, ferc, finance, gas, natural gas, oil, regulation, utility, xbrl",
"author": null,
"author_email": "Catalyst Cooperative <pudl@catalyst.coop>, Zach Schira <zach.schira@catalyst.coop>",
"download_url": "https://files.pythonhosted.org/packages/89/e4/45e89bc69bf1b0288cd80461f2068eb182d8b7601de9fedfacb204e724dd/catalystcoop_ferc_xbrl_extractor-1.4.0.tar.gz",
"platform": null,
"description": "===============================================================================\nFERC XBRL Extractor\n===============================================================================\n\n\n.. readme-intro\n\n.. image:: https://www.repostatus.org/badges/latest/active.svg\n :target: https://www.repostatus.org/#active\n :alt: Project Status: Active\n\n.. image:: https://github.com/catalyst-cooperative/ferc-xbrl-extractor/workflows/tox-pytest/badge.svg\n :target: https://github.com/catalyst-cooperative/ferc-xbrl-extractor/actions/workflows/tox-pytest.yml\n :alt: Tox-PyTest Status\n\n.. image:: https://img.shields.io/codecov/c/github/catalyst-cooperative/ferc-xbrl-extractor?style=flat&logo=codecov\n :target: https://codecov.io/gh/catalyst-cooperative/ferc-xbrl-extractor\n :alt: Codecov Test Coverage\n\n.. image:: https://img.shields.io/readthedocs/catalystcoop-ferc-xbrl-extractor?style=flat&logo=readthedocs\n :target: https://catalystcoop-ferc-xbrl-extractor.readthedocs.io/en/latest/\n :alt: Read the Docs Build Status\n\n.. image:: https://img.shields.io/pypi/v/catalystcoop.ferc-xbrl-extractor\n :target: https://pypi.org/project/catalystcoop.ferc-xbrl-extractor/\n :alt: PyPI Latest Version\n\n.. image:: https://img.shields.io/conda/vn/conda-forge/catalystcoop.ferc_xbrl_extractor\n :target: https://anaconda.org/conda-forge/catalystcoop.ferc_xbrl_extractor\n :alt: conda-forge Version\n\n.. image:: https://img.shields.io/pypi/pyversions/catalystcoop.ferc-xbrl-extractor\n :target: https://pypi.org/project/catalystcoop.ferc-xbrl-extractor/\n :alt: Supported Python Versions\n\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/psf/black\n :alt: Any color you want, so long as it's black.\n\n.. image:: https://results.pre-commit.ci/badge/github/catalyst-cooperative/ferc-xbrl-extractor/main.svg\n :target: https://results.pre-commit.ci/latest/github/catalyst-cooperative/ferc-xbrl-extractor/main\n :alt: pre-commit CI\n\n.. image:: https://zenodo.org/badge/471019769.svg\n :target: https://zenodo.org/doi/10.5281/zenodo.10020145\n :alt: Zenodo DOI\n\nThe Federal Energy Regulatory Commission (FERC) has moved to collecting and distributing\ndata using `XBRL <https://en.wikipedia.org/wiki/XBRL>`__. XBRL is primarily designed for\nfinancial reporting, and has been adopted by regulators in the US and other countries.\nMuch of the tooling in the XBRL ecosystem is targeted towards filers, and rendering\nindividual filings in a human readable way, but there is very little targeted towards\naccessing and analyzing large collections of filings. This tool is designed to provide\nthat functionality for FERC XBRL data. Specifically, it can extract data from a set of\nXBRL filings, and write that data to a SQLite database whose structure is generated from\nan XBRL Taxonomy. While each XBRL instance contains a reference to a taxonomy,\nthis tool requires a path to a single taxonomy that will be used to interpret all\ninstances being processed. This means even if instances were created from different\nversions of a Taxonomy, the provided taxonomy will be used when processing all of these\ninstances, so the output database will have a consistent structure. For more information\non the technical details of the XBRL extraction, see the docs.\n\nWe are currently using this tool to extract and publish the following FERC data:\n\n.. list-table::\n :header-rows: 1\n\n * - FERC Form\n - Archived XBRL\n - SQLite DB\n * - `Form 1 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-1-electric-utility-annual>`__\n - https://doi.org/10.5281/zenodo.4127043\n - https://data.catalyst.coop/ferc1_xbrl\n * - `Form 2 <https://www.ferc.gov/industries-data/natural-gas/industry-forms/form-2-2a-3-q-gas-historical-vfp-data>`__\n - https://doi.org/10.5281/zenodo.5879542\n - https://data.catalyst.coop/ferc2_xbrl\n * - `Form 6 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-66-q-overview-orders>`__\n - https://doi.org/10.5281/zenodo.7126395\n - https://data.catalyst.coop/ferc6_xbrl\n * - `Form 60 <https://www.ferc.gov/ferc-online/ferc-online/filing-forms/service-companies-filing-forms/form-60-annual-report>`_\n - https://doi.org/10.5281/zenodo.7126434\n - https://data.catalyst.coop/ferc60_xbrl\n * - `Form 714 <https://www.ferc.gov/industries-data/electric/general-information/electric-industry-forms/form-no-714-annual-electric>`__\n - https://doi.org/10.5281/zenodo.4127100\n - https://data.catalyst.coop/ferc714_xbrl\n\nUsage\n-----\n\nInstallation\n^^^^^^^^^^^^\n\nTo install using conda, run the following command, and activate the environment.\n\n.. code-block:: console\n\n $ conda env create -f environment.yml\n\nActivate:\n\n.. code-block:: console\n\n $ conda activate ferc-xbrl-extractor\n\n\nCLI\n^^^\n\nThis tool can be used as a library, as it is in `PUDL <https://github.com/catalyst-cooperative/pudl>`__,\nor there is a CLI provided for interacting with XBRL data. The only required options\nfor the CLI are a path to the filings to be extracted, and a path to the output\nSQLite database. The path to the filings can point to a directory full of XBRL\nFilings, a single XBRL filing, or a zipfile with XBRL filings. If\nthe path to the database points to an existing database, the ``--clobber`` option\ncan be used to drop all existing data before performing the extraction.\n\n.. code-block:: console\n\n $ xbrl_extract {path_to_filings} {path_to_database}\n\nThis repo contains a small selection of FERC Form 1 filings from 2021 in the\n``examples`` directory. To test the tool on these filings, use the command\n\n.. code-block:: console\n\n $ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite\n\nBy default, the CLI will use the 2022 version of the FERC Form 1 Taxonomy to create\nthe structure of the output database, and it will obtain the taxonomy directly from\nthe FERC web server. To specify a different taxonomy use the ``--taxonomy`` option. This\ncan be either a URL pointing to a taxonomy entry point, or a path to a local zipfile.\nThe following command demonstrates this option using the same URL that would be used by\ndefault.\n\n.. code-block:: console\n\n $ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \\\n --taxonomy https://eCollection.ferc.gov/taxonomy/form1/2022-01-01/form/form1/form-1_2022-01-01.xsd\n\nWhen using the ``--taxonomy`` option to read the taxonomy from a zipfile, the\n``--archive-path`` option is also required. This option is used to indicate the path to\nthe taxonomy entry point within the zipfile. NOTE: This will likely change in the future\nto automatically detect the entry point as this interface is clunky. To demonstrate, the\nFERC Form 1 2021 taxonomy is included in the examples directory. This can be used with\nthe following command.\n\n.. code-block:: console\n\n $ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \\\n --taxonomy examples/taxonomy.zip \\\n --archive-path taxonomy/form1/2021-01-01/form/form1/form-1_2021-01-01.xsd\n\nParsing XBRL filings can be a time consuming and CPU heavy task, so this tool\nimplements some basic multiprocessing to speed this up. It uses a\n`process pool <https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor>`__\nto do this. There are two options for configuring the process pool, ``--batch-size``\nand ``--workers``. The batch size configures how many filings will be processed by\neach child process at a time, and workers specifies how many child processes to\ncreate in the pool. It may take some experimentation to get these options\noptimally configured. The following command will use 5 worker processes to process\nbatches of 50 filings at a time.\n\n.. code-block:: console\n\n $ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \\\n --workers 5 \\\n --batch-size 50\n\nThere are also several options included for extracting metadata from the taxonomy.\nFirst is the ``--datapackage-path`` command to save a\n`frictionless datapackage <https://specs.frictionlessdata.io/data-package/>`__\ndescriptor as JSON, which annotates the generated SQLite database. There is also the\n``--metadata-path`` option, which writes more extensive taxonomy metadata to a json\nfile, grouped by table name. See the ``ferc_xbrl_extractor.arelle_interface`` module\nfor more info on the extracted metadata. To create both of these files using the example\nfilings and taxonomy, run the following command.\n\n.. code-block:: console\n\n $ xbrl_extract examples/ferc1-2021-sample.zip ./ferc1-2021-sample.sqlite \\\n --taxonomy examples/taxonomy.zip \\\n --archive-path taxonomy/form1/2021-01-01/form/form1/form-1_2021-01-01.xsd \\\n --metadata-path metadata.json \\\n --datapackage-path datapackage.json\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2022 Catalyst Cooperative Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A tool for extracting data from FERC XBRL Filings.",
"version": "1.4.0",
"project_urls": {
"Documentation": "https://catalystcoop-ferc-xbrl-extractor.readthedocs.io",
"Homepage": "https://github.com/catalyst-cooperative/ferc-xbrl-extractor",
"Issue Tracker": "https://github.com/catalyst-cooperative/ferc-xbrl-extractor/issues",
"Source": "https://github.com/catalyst-cooperative/ferc-xbrl-extractor"
},
"split_keywords": [
"accounting",
" data",
" electricity",
" energy",
" federal energy regulatory commission",
" ferc",
" finance",
" gas",
" natural gas",
" oil",
" regulation",
" utility",
" xbrl"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "c144775d70b83c9306891f46c1f9fd5fe76460b20931ce0cc4ed35c808fbce01",
"md5": "b15477a9c7391a25495d53b300c2c6d3",
"sha256": "867e75809bda9632700d0c2afd68263146174f3c7f4d26dcac641b509b290c5f"
},
"downloads": -1,
"filename": "catalystcoop.ferc_xbrl_extractor-1.4.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b15477a9c7391a25495d53b300c2c6d3",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 29215,
"upload_time": "2024-04-15T01:15:05",
"upload_time_iso_8601": "2024-04-15T01:15:05.323558Z",
"url": "https://files.pythonhosted.org/packages/c1/44/775d70b83c9306891f46c1f9fd5fe76460b20931ce0cc4ed35c808fbce01/catalystcoop.ferc_xbrl_extractor-1.4.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "89e445e89bc69bf1b0288cd80461f2068eb182d8b7601de9fedfacb204e724dd",
"md5": "8b1d4841f8232c5511488e57e13b6671",
"sha256": "3ab8e265e0285993ca691077c090006146f65d6baabd067def7800250d1d3d4b"
},
"downloads": -1,
"filename": "catalystcoop_ferc_xbrl_extractor-1.4.0.tar.gz",
"has_sig": false,
"md5_digest": "8b1d4841f8232c5511488e57e13b6671",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<3.13,>=3.10",
"size": 19974548,
"upload_time": "2024-04-15T01:15:08",
"upload_time_iso_8601": "2024-04-15T01:15:08.134538Z",
"url": "https://files.pythonhosted.org/packages/89/e4/45e89bc69bf1b0288cd80461f2068eb182d8b7601de9fedfacb204e724dd/catalystcoop_ferc_xbrl_extractor-1.4.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-15 01:15:08",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "catalyst-cooperative",
"github_project": "ferc-xbrl-extractor",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "catalystcoop.ferc-xbrl-extractor"
}