| Name | fpds JSON |
| Version |
1.3.0
JSON |
| download |
| home_page | |
| Summary | A parser for the Federal Procurement Data System (FPDS) Atom feed |
| upload_time | 2024-01-18 06:12:03 |
| maintainer | |
| docs_url | None |
| author | |
| requires_python | >=3.8 |
| license | The MIT License Copyright (c) 2022 Derek Herincx Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
fpds
python
atom feed
cli
xml
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
aiohttp
click
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# fpds
A no-frills parser for the Federal Procurement Data System (FPDS)
at https://www.fpds.gov/fpdsng_cms/index.php/en/.
## Motivation
The only programmatic access to this data via an ATOM feed limits each
request to 10 records, which forces users to deal with pagination.
Additonally, data is exported as XML, which proves annoying for most
developers. `fpds` will handle all pagination and data
transformation to provide users with a nice JSON representation of the
equivalent XML data.
## Setup
To install this package for development, create a virtual environment
and install dependencies.
```
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install -e .
```
## Usage
For a list of valid search criteria parameters, consult FPDS documentation
found at: https://www.fpds.gov/wiki/index.php/Atom_Feed_Usage. Parameters
will follow the `URL String` format shown in the link above, with the
following exceptions:
+ Colons (:) will be replaced by equal signs (=)
+ Certain parameters enclose their value in quotations. `fpds` will
automatically determine if quotes are needed, so simply enclose your
entire criteria string in quotes.
For example, `AGENCY_CODE:”3600”` should be used as `"AGENCY_CODE=3600"`.
Via CLI:
```
$ fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/05/01]" "AGENCY_CODE=7504"
```
By default, data will be dumped into an `.fpds` folder at the user's
`$HOME` directory. If you wish to override this behavior, provide the `-o`
option. The directory will be created if it doesn't exist:
```
$ fpds parse "LAST_MOD_DATE=[2022/01/01, 2022/05/01]" "AGENCY_CODE=7504" -o {some-directory}
```
Same request via python interpreter:
```
from fpds import fpdsRequest
request = fpdsRequest(
target_database_url_env_key="SOME_ENVIRONMENT_VAR",
LAST_MOD_DATE="[2022/01/01, 2022/05/01]",
AGENCY_CODE="7504"
)
# handles automatic conversion of XML --> JSON
data = request.process_records()
# URL magic method for assitance / debugging
url = request.__url__()
```
For linting and formatting, we use `flake8` and `black`.
```
$ make lint
$ make formatters
```
Lastly, you can clean the clutter and unwanted noise.
```
$ make clean
```
### Testing
```
$ make test
```
## What's New
`fpds` now supports asynchronous requests! As of `v1.3.0`, users can instantiate
the class as usual, but will now need to call the `process_records` method
to get records as JSON. Note: due to some recursive function calls in the XML
parsing, users might experience some high completion times for this function
call. Recommendation is to limit the number of results.
#### Timing Benchmarks (in seconds):
| v1.2.1 | v.1.3.0 |
-------- | --------
188.46 | 29.40
190.38 | 28.14
187.20 | 27.66
Using `v.1.2.1`, the average completion time is 188.68 seconds (~3min).
Using `v.1.3.0`, the average completion time is 28.40 seconds.
This equates to a <u>**84.89%**</u> decrease in completion time!
As of `v1.3.0`, `fpds` now supports the use of over 100 keyword tags when searching
for contracts using the `v1.5.3` ATOM feed.
Raw data
{
"_id": null,
"home_page": "",
"name": "fpds",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "fpds,python,atom feed,cli,xml",
"author": "",
"author_email": "Derek Herincx <derek663@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/aa/75/3cbc5dea6a5cb0f8141c619c27441bca327d2ae04f37f5085b0f7c54c729/fpds-1.3.0.tar.gz",
"platform": null,
"description": "# fpds\nA no-frills parser for the Federal Procurement Data System (FPDS)\nat https://www.fpds.gov/fpdsng_cms/index.php/en/.\n\n\n## Motivation\nThe only programmatic access to this data via an ATOM feed limits each\nrequest to 10 records, which forces users to deal with pagination.\nAdditonally, data is exported as XML, which proves annoying for most\ndevelopers. `fpds` will handle all pagination and data\ntransformation to provide users with a nice JSON representation of the\nequivalent XML data.\n\n\n## Setup\nTo install this package for development, create a virtual environment\nand install dependencies.\n\n```\n$ python3 -m venv venv\n$ source venv/bin/activate\n$ pip install -e .\n```\n\n\n## Usage\nFor a list of valid search criteria parameters, consult FPDS documentation\nfound at: https://www.fpds.gov/wiki/index.php/Atom_Feed_Usage. Parameters\nwill follow the `URL String` format shown in the link above, with the\nfollowing exceptions:\n\n + Colons (:) will be replaced by equal signs (=)\n + Certain parameters enclose their value in quotations. `fpds` will\nautomatically determine if quotes are needed, so simply enclose your\nentire criteria string in quotes.\n\n For example, `AGENCY_CODE:\u201d3600\u201d` should be used as `\"AGENCY_CODE=3600\"`.\n\n\nVia CLI:\n```\n$ fpds parse \"LAST_MOD_DATE=[2022/01/01, 2022/05/01]\" \"AGENCY_CODE=7504\"\n```\n\n\nBy default, data will be dumped into an `.fpds` folder at the user's\n`$HOME` directory. If you wish to override this behavior, provide the `-o`\noption. The directory will be created if it doesn't exist:\n\n```\n$ fpds parse \"LAST_MOD_DATE=[2022/01/01, 2022/05/01]\" \"AGENCY_CODE=7504\" -o {some-directory}\n```\n\nSame request via python interpreter:\n```\nfrom fpds import fpdsRequest\n\nrequest = fpdsRequest(\n target_database_url_env_key=\"SOME_ENVIRONMENT_VAR\",\n LAST_MOD_DATE=\"[2022/01/01, 2022/05/01]\",\n AGENCY_CODE=\"7504\"\n)\n\n# handles automatic conversion of XML --> JSON\ndata = request.process_records()\n\n# URL magic method for assitance / debugging\nurl = request.__url__()\n```\n\nFor linting and formatting, we use `flake8` and `black`.\n\n```\n$ make lint\n$ make formatters\n```\n\nLastly, you can clean the clutter and unwanted noise.\n\n```\n$ make clean\n```\n\n### Testing\n```\n$ make test\n```\n\n## What's New\n`fpds` now supports asynchronous requests! As of `v1.3.0`, users can instantiate\nthe class as usual, but will now need to call the `process_records` method\nto get records as JSON. Note: due to some recursive function calls in the XML\nparsing, users might experience some high completion times for this function\ncall. Recommendation is to limit the number of results.\n\n#### Timing Benchmarks (in seconds):\n\n| v1.2.1 | v.1.3.0 |\n-------- | --------\n188.46 | 29.40\n190.38 | 28.14\n187.20 | 27.66\n\nUsing `v.1.2.1`, the average completion time is 188.68 seconds (~3min).\nUsing `v.1.3.0`, the average completion time is 28.40 seconds.\n\nThis equates to a <u>**84.89%**</u> decrease in completion time!\n\n\nAs of `v1.3.0`, `fpds` now supports the use of over 100 keyword tags when searching\nfor contracts using the `v1.5.3` ATOM feed.\n",
"bugtrack_url": null,
"license": "The MIT License Copyright (c) 2022 Derek Herincx Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ",
"summary": "A parser for the Federal Procurement Data System (FPDS) Atom feed",
"version": "1.3.0",
"project_urls": {
"Issues": "https://github.com/dherincx92/fpds/issues",
"Repository": "https://github.com/dherincx92/fpds"
},
"split_keywords": [
"fpds",
"python",
"atom feed",
"cli",
"xml"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "33502dc15731a822f1a4c33f293980d206a8614ba2bf2f397d00014439d23fb4",
"md5": "254869e5a6e2471d4c611a15cc0fff0b",
"sha256": "e6a4760d4594fafa5717ba4e34872faa0a874c59e0671a6dc5b8207ff91d098b"
},
"downloads": -1,
"filename": "fpds-1.3.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "254869e5a6e2471d4c611a15cc0fff0b",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 20930,
"upload_time": "2024-01-18T06:12:01",
"upload_time_iso_8601": "2024-01-18T06:12:01.927475Z",
"url": "https://files.pythonhosted.org/packages/33/50/2dc15731a822f1a4c33f293980d206a8614ba2bf2f397d00014439d23fb4/fpds-1.3.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "aa753cbc5dea6a5cb0f8141c619c27441bca327d2ae04f37f5085b0f7c54c729",
"md5": "ad13dc4cff477a4cbaf358057d6bff63",
"sha256": "0f384b6f861a33e10637e4de64cc92421ce3454ff9b9a2e3a22c8e726998edfe"
},
"downloads": -1,
"filename": "fpds-1.3.0.tar.gz",
"has_sig": false,
"md5_digest": "ad13dc4cff477a4cbaf358057d6bff63",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 35047,
"upload_time": "2024-01-18T06:12:03",
"upload_time_iso_8601": "2024-01-18T06:12:03.414383Z",
"url": "https://files.pythonhosted.org/packages/aa/75/3cbc5dea6a5cb0f8141c619c27441bca327d2ae04f37f5085b0f7c54c729/fpds-1.3.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-18 06:12:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dherincx92",
"github_project": "fpds",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [
{
"name": "aiohttp",
"specs": [
[
"==",
"3.9.1"
]
]
},
{
"name": "click",
"specs": [
[
">=",
"8.1.3"
]
]
}
],
"lcname": "fpds"
}
