| Name | open-ksa JSON |
| Version |
0.1.5
JSON |
| download |
| home_page | None |
| Summary | Commands to download datasets from the Open Data portal of the Kingdom of Saudi Arabia (KSA). |
| upload_time | 2024-10-29 09:19:47 |
| maintainer | None |
| docs_url | None |
| author | None |
| requires_python | >=3.9.1 |
| license | MIT |
| keywords |
open data
ksa
datasets
api
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
## Open Data - Kingdom of Saudi Arabia
> This repository contains scripts to download datasets from the Open Data portal of the Kingdom of Saudi Arabia (KSA). The main script downloads all datasets for a given organization ID and saves them locally.
### Directory Structure
```{bash}
.
├── README.md
├── download_all_org.py
├── opendata (optional - parameter in file)
├── requirements.txt
├── open_ksa
│ ├── download_file.py
│ ├── organizations.py
│ ├── get_dataset_resources.py
│ └── get_org_resources.py
└── system-drawing.excalidraw
```
### Functions Overview
- `organizations()`: Get the organization information, including the option to write to a target file as a CSV or JSON.
- `get_org_resources(org_id)`: Retrieves the organization name, organization ID, and dataset IDs for the specified organization.
- `get_dataset_resources(dataset_ids, allowed_exts=['csv', 'xlsx', 'xls'], output_dir='opendata/org_resources', verbose=False)`: Downloads all data resources for the specified dataset IDs.
- `download_file(session, url, headers, file_path)`: Downloads a file from the specified URL using the provided session and headers.
### Process Flow
```mermaid
graph TD
subgraph Initialization
A[Start] --> B[Create SSL Adapter]
B --> C[Setup Session]
end
subgraph Resource Extraction
C --> D[Extract Organization ID]
D --> E[Extract Dataset IDs]
end
subgraph Directory Setup
E --> F[Create Directory for Organization]
end
subgraph Data Download
F --> G[Download Dataset Resources]
G --> H[Save Data Locally]
end
subgraph Example - 3_get_dataset_resources.py
H --> J[Get Organization ID]
J --> J1[Get All Dataset IDs]
J1 --> K[Loop Through Dataset IDs]
K --> L[Download Each Resource]
L --> M[Save Each Resource Locally]
M --> N[Run Jupyter Notebook 1_download_data.ipynb]
end
N --> I[End]
```
### Installation
To begin collecting data using the `open-ksa` package, simply install via `pip`:
```bash
pip install open-ksa
```
### Usage
For examples on how to use the package, see the following folders:
- [Scripts](/examples/scripts/)
- [Workbooks](/examples/workbooks/)
These illustrate how to use the package in a script or workbook and examples on how to collect information for one or multiple organizations.
NOTE: For a different organization, you need to update the parameter in the file for the `org_id` parameter in the function
### Release Plan / To DO
- [X] Create a set of functions to cover the entire API, including:
- [X] Create a function to get the list of organizations
- [X] Create a function to get the list of datasets for an organization
- [X] Create a function to get the list of resources for a dataset
- [X] Create a function to download a resource
- [X] Create a function to check the status of a download
- [X] Create a set of unit tests for the functions
- [X] Create a set of examples for the functions
- [ ] Create a set of documentation for the functions
- [X] Move the repository to a PyPi library
### Contribution
The contribution process is as follows:
1. Clone the repository and create a new branch
2. Make your changes, following the coding style guidelines
3. Create a pull request with a detailed description of your changes
4. Wait for your pull request to be reviewed and approved
5. Once approved, your changes will be merged and available in the main branch
When contributing to this repository, please first discuss the change you wish to make via issue,
email, or any other method with the owners of this repository before making a change.
Please note we have a code of conduct, please follow it in all your interactions with the project.
Raw data
{
"_id": null,
"home_page": null,
"name": "open-ksa",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9.1",
"maintainer_email": "Esteban Valencia <me@estebanvalencia.com>",
"keywords": "open data, KSA, datasets, API",
"author": null,
"author_email": "Esteban Valencia <me@estebanvalencia.com>",
"download_url": "https://files.pythonhosted.org/packages/4a/49/f438d9b5cb0a084f82e368944fdb65b306e18a519bc19df4c8647c24f2d6/open_ksa-0.1.5.tar.gz",
"platform": null,
"description": "## Open Data - Kingdom of Saudi Arabia \n \n> This repository contains scripts to download datasets from the Open Data portal of the Kingdom of Saudi Arabia (KSA). The main script downloads all datasets for a given organization ID and saves them locally. \n\n\n### Directory Structure \n\n```{bash}\n.\n\u251c\u2500\u2500 README.md\n\u251c\u2500\u2500 download_all_org.py\n\u251c\u2500\u2500 opendata (optional - parameter in file)\n\u251c\u2500\u2500 requirements.txt\n\u251c\u2500\u2500 open_ksa\n\u2502 \u251c\u2500\u2500 download_file.py\n\u2502 \u251c\u2500\u2500 organizations.py\n\u2502 \u251c\u2500\u2500 get_dataset_resources.py\n\u2502 \u2514\u2500\u2500 get_org_resources.py\n\u2514\u2500\u2500 system-drawing.excalidraw\n\n```\n### Functions Overview \n\n- `organizations()`: Get the organization information, including the option to write to a target file as a CSV or JSON.\n- `get_org_resources(org_id)`: Retrieves the organization name, organization ID, and dataset IDs for the specified organization.\n- `get_dataset_resources(dataset_ids, allowed_exts=['csv', 'xlsx', 'xls'], output_dir='opendata/org_resources', verbose=False)`: Downloads all data resources for the specified dataset IDs.\n- `download_file(session, url, headers, file_path)`: Downloads a file from the specified URL using the provided session and headers.\n\n### Process Flow\n\n```mermaid\ngraph TD\n subgraph Initialization\n A[Start] --> B[Create SSL Adapter]\n B --> C[Setup Session]\n end\n\n subgraph Resource Extraction\n C --> D[Extract Organization ID]\n D --> E[Extract Dataset IDs]\n end\n\n subgraph Directory Setup\n E --> F[Create Directory for Organization]\n end\n\n subgraph Data Download\n F --> G[Download Dataset Resources]\n G --> H[Save Data Locally]\n end\n\n subgraph Example - 3_get_dataset_resources.py\n H --> J[Get Organization ID]\n J --> J1[Get All Dataset IDs]\n J1 --> K[Loop Through Dataset IDs]\n K --> L[Download Each Resource]\n L --> M[Save Each Resource Locally]\n M --> N[Run Jupyter Notebook 1_download_data.ipynb]\n end\n\n N --> I[End]\n```\n\n### Installation \n\nTo begin collecting data using the `open-ksa` package, simply install via `pip`:\n\n```bash\npip install open-ksa\n```\n\n### Usage\n\nFor examples on how to use the package, see the following folders:\n- [Scripts](/examples/scripts/)\n- [Workbooks](/examples/workbooks/)\n\nThese illustrate how to use the package in a script or workbook and examples on how to collect information for one or multiple organizations.\n\nNOTE: For a different organization, you need to update the parameter in the file for the `org_id` parameter in the function\n\n\n### Release Plan / To DO\n\n- [X] Create a set of functions to cover the entire API, including:\n - [X] Create a function to get the list of organizations\n - [X] Create a function to get the list of datasets for an organization\n - [X] Create a function to get the list of resources for a dataset\n - [X] Create a function to download a resource\n - [X] Create a function to check the status of a download\n- [X] Create a set of unit tests for the functions\n- [X] Create a set of examples for the functions\n- [ ] Create a set of documentation for the functions\n- [X] Move the repository to a PyPi library\n\n### Contribution\n\nThe contribution process is as follows:\n\n1. Clone the repository and create a new branch\n2. Make your changes, following the coding style guidelines\n3. Create a pull request with a detailed description of your changes\n4. Wait for your pull request to be reviewed and approved\n5. Once approved, your changes will be merged and available in the main branch\n\nWhen contributing to this repository, please first discuss the change you wish to make via issue,\nemail, or any other method with the owners of this repository before making a change.\n\nPlease note we have a code of conduct, please follow it in all your interactions with the project.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Commands to download datasets from the Open Data portal of the Kingdom of Saudi Arabia (KSA).",
"version": "0.1.5",
"project_urls": null,
"split_keywords": [
"open data",
" ksa",
" datasets",
" api"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "39b9c9ac113528fa9741cdeb6acdbbb0ec4afbdbf9238af6a459782599776af1",
"md5": "51a95666793e40ef102185d342126459",
"sha256": "1ef1cce6153d2eb002873aabe3d141a1ab2f76b8af4425c87aa0e83768695424"
},
"downloads": -1,
"filename": "open_ksa-0.1.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "51a95666793e40ef102185d342126459",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9.1",
"size": 13631,
"upload_time": "2024-10-29T09:19:45",
"upload_time_iso_8601": "2024-10-29T09:19:45.475703Z",
"url": "https://files.pythonhosted.org/packages/39/b9/c9ac113528fa9741cdeb6acdbbb0ec4afbdbf9238af6a459782599776af1/open_ksa-0.1.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "4a49f438d9b5cb0a084f82e368944fdb65b306e18a519bc19df4c8647c24f2d6",
"md5": "d7dc20ceb4ffb34f04942744ede45cf7",
"sha256": "5ebc16572f77805921ceda5583b81400c6c40268885edbcb8efe916f94d31d9c"
},
"downloads": -1,
"filename": "open_ksa-0.1.5.tar.gz",
"has_sig": false,
"md5_digest": "d7dc20ceb4ffb34f04942744ede45cf7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9.1",
"size": 48299,
"upload_time": "2024-10-29T09:19:47",
"upload_time_iso_8601": "2024-10-29T09:19:47.087405Z",
"url": "https://files.pythonhosted.org/packages/4a/49/f438d9b5cb0a084f82e368944fdb65b306e18a519bc19df4c8647c24f2d6/open_ksa-0.1.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-29 09:19:47",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "open-ksa"
}
