# osm2pgsql-tuner
The osm2pgsql-tuner project recommends an osm2pgsql command based on
available system resources and the size of the input PBF file.
The recommendations made by this program are targeted for:
* osm2pgsql v1.5.0 and newer
* Flex output
* No stage 2 processing
Stage 2 processing has less predictable RAM consumption
[per this discussion on GitHub](https://github.com/openstreetmap/osm2pgsql/discussions/1536).
## Using the API
This project is hosted as a free API at https://osm2pgsql-tuner.com by RustProof Labs.
The following is an example of using this API from Python.
```python
import requests
system_ram_gb = 64
osm_pbf_gb = 10.4
pbf_filename = 'north-america-latest'
append = False
api_endpoint = 'https://osm2pgsql-tuner.com/api/v1'
api_endpoint += f'?system_ram_gb={system_ram_gb}&osm_pbf_gb={osm_pbf_gb}&append={append}&pbf_filename={pbf_filename}'
```
Query the endpoint, check the status.
```python
result = requests.get(api_endpoint)
print(f'Status code: {result.status_code}')
```
Get recommendation data.
```python
rec = result.json()['osm2pgsql']
```
Command is the most interesting part.
```python
print(f"\nCommand:\n{rec['cmd']} ")
```
Other details returned used in decision making to determine the command `cmd`.
```python
print(rec.keys())
```
## Using osm2pgsql via Python
To use the osm2pgsql recommendation without using the API/website, the
`osm2pgsql_tuner` package can be used.
Install `osm2pgsql-tuner` within an virtual environment.
```bash
pip install osm2pgsql-tuner
```
Import `osm2pgsql_tuner` and create an instance of the `recommendation` class.
```python
import osm2pgsql_tuner
rec = osm2pgsql_tuner.recommendation(system_ram_gb=8,
osm_pbf_gb=0.5,
pgosm_layer_set='run')
pbf_path = '~/pgosm-data/example_file.osm.pbf'
osm2pgsql_command = rec.get_osm2pgsql_command(out_format='api',
pbf_path=pbf_path)
print(osm2pgsql_command)
```
Returns.
```bash
osm2pgsql -d $PGOSM_CONN --output=flex --style=./run.lua ~/pgosm-data/example_file.osm.pbf
```
## Deployment Instructions
> Note: Need to update the sub-version of Python over time. Can use simply
`python3` but that can lead to using older unsupported versions based on distribution defaults.
```bash
cd ~/venv
python3.8 -m venv osm2pgsql-tuner
source ~/venv/osm2pgsql-tuner/bin/activate
```
Install requirements
```bash
source ~/venv/osm2pgsql-tuner/bin/activate
cd ~/git/osm2pgsql-tuner
pip install -r requirements.txt
```
Run web server w/ uWSGI.
```bash
source ~/venv/osm2pgsql-tuner/bin/activate
cd ~/git/osm2pgsql-tuner
python run_server.py
```
## Unit tests
Run unit tests.
```bash
python -m unittest tests/*.py
```
Or run unit tests with coverage.
```bash
coverage run -m unittest tests/*.py
```
Generate report.
```bash
coverage report -m osm2pgsql_tuner/*.py
```
Run pylint.
```bash
pylint --rcfile=./.pylintrc -f parseable \
./osm2pgsql_tuner/*.py
```
## Used by
This project is used by [PgOSM Flex](https://github.com/rustprooflabs/pgosm-flex)
to automate commands in the [PgOSM Flex Docker image](https://hub.docker.com/r/rustprooflabs/pgosm-flex).
## Development install
```
pip install -e .
```
Raw data
{
"_id": null,
"home_page": "https://github.com/rustprooflabs/osm2pgsql-tuner",
"name": "osm2pgsql-tuner",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "RustProof Labs",
"author_email": "RustProof Labs <support@rustprooflabs.com>",
"download_url": "https://files.pythonhosted.org/packages/64/3b/03f2c33a7fc0105bd5f81987b24dd175e06e847fdbea4ac91362ca440df8/osm2pgsql-tuner-0.1.0.tar.gz",
"platform": null,
"description": "# osm2pgsql-tuner\n\nThe osm2pgsql-tuner project recommends an osm2pgsql command based on\navailable system resources and the size of the input PBF file.\nThe recommendations made by this program are targeted for:\n\n* osm2pgsql v1.5.0 and newer\n* Flex output\n* No stage 2 processing\n\nStage 2 processing has less predictable RAM consumption\n[per this discussion on GitHub](https://github.com/openstreetmap/osm2pgsql/discussions/1536).\n\n\n\n## Using the API\n\nThis project is hosted as a free API at https://osm2pgsql-tuner.com by RustProof Labs.\nThe following is an example of using this API from Python.\n\n```python\nimport requests\n\nsystem_ram_gb = 64\nosm_pbf_gb = 10.4\npbf_filename = 'north-america-latest'\nappend = False\n\napi_endpoint = 'https://osm2pgsql-tuner.com/api/v1'\napi_endpoint += f'?system_ram_gb={system_ram_gb}&osm_pbf_gb={osm_pbf_gb}&append={append}&pbf_filename={pbf_filename}'\n```\n\nQuery the endpoint, check the status.\n\n```python\nresult = requests.get(api_endpoint)\nprint(f'Status code: {result.status_code}')\n```\n\nGet recommendation data.\n\n```python\nrec = result.json()['osm2pgsql']\n```\n\nCommand is the most interesting part.\n\n```python\nprint(f\"\\nCommand:\\n{rec['cmd']} \")\n```\n\nOther details returned used in decision making to determine the command `cmd`.\n\n```python\nprint(rec.keys())\n```\n\n## Using osm2pgsql via Python\n\nTo use the osm2pgsql recommendation without using the API/website, the\n`osm2pgsql_tuner` package can be used.\n\nInstall `osm2pgsql-tuner` within an virtual environment.\n\n```bash\npip install osm2pgsql-tuner\n```\n\nImport `osm2pgsql_tuner` and create an instance of the `recommendation` class.\n\n```python\nimport osm2pgsql_tuner\nrec = osm2pgsql_tuner.recommendation(system_ram_gb=8,\n osm_pbf_gb=0.5,\n pgosm_layer_set='run')\npbf_path = '~/pgosm-data/example_file.osm.pbf'\nosm2pgsql_command = rec.get_osm2pgsql_command(out_format='api',\n pbf_path=pbf_path)\nprint(osm2pgsql_command)\n```\n\nReturns.\n\n```bash\nosm2pgsql -d $PGOSM_CONN --output=flex --style=./run.lua ~/pgosm-data/example_file.osm.pbf\n```\n\n\n## Deployment Instructions\n\n> Note: Need to update the sub-version of Python over time. Can use simply\n`python3` but that can lead to using older unsupported versions based on distribution defaults.\n\n\n```bash\ncd ~/venv\npython3.8 -m venv osm2pgsql-tuner\nsource ~/venv/osm2pgsql-tuner/bin/activate\n```\n\nInstall requirements\n\n```bash\nsource ~/venv/osm2pgsql-tuner/bin/activate\ncd ~/git/osm2pgsql-tuner\npip install -r requirements.txt\n```\n\nRun web server w/ uWSGI.\n\n```bash\nsource ~/venv/osm2pgsql-tuner/bin/activate\ncd ~/git/osm2pgsql-tuner\npython run_server.py\n```\n\n\n## Unit tests\n\nRun unit tests.\n\n```bash\npython -m unittest tests/*.py\n```\n\nOr run unit tests with coverage.\n\n```bash\ncoverage run -m unittest tests/*.py\n```\n\nGenerate report.\n\n```bash\ncoverage report -m osm2pgsql_tuner/*.py\n```\n\n\nRun pylint.\n\n```bash\npylint --rcfile=./.pylintrc -f parseable \\\n ./osm2pgsql_tuner/*.py\n```\n\n## Used by\n\nThis project is used by [PgOSM Flex](https://github.com/rustprooflabs/pgosm-flex)\nto automate commands in the [PgOSM Flex Docker image](https://hub.docker.com/r/rustprooflabs/pgosm-flex).\n\n\n## Development install\n\n\n```\npip install -e .\n```\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "osm2pgsql-tuner project recommends an osm2pgsql command based on available system resources and the size of the input PBF file.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/rustprooflabs/osm2pgsql-tuner",
"Issues": "https://github.com/rustprooflabs/osm2pgsql-tuner/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "07c514b152bb791546c8592371e69412fe4d164233005d7a4e636acd063bc8d4",
"md5": "269a0feb9fc80f8e292576929b3275a9",
"sha256": "ba1591516ea1231c8833fa61705ac863f2a26cdb96c2358d141544ac55d6f72e"
},
"downloads": -1,
"filename": "osm2pgsql_tuner-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "269a0feb9fc80f8e292576929b3275a9",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 6457,
"upload_time": "2024-01-04T03:07:55",
"upload_time_iso_8601": "2024-01-04T03:07:55.968137Z",
"url": "https://files.pythonhosted.org/packages/07/c5/14b152bb791546c8592371e69412fe4d164233005d7a4e636acd063bc8d4/osm2pgsql_tuner-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "643b03f2c33a7fc0105bd5f81987b24dd175e06e847fdbea4ac91362ca440df8",
"md5": "cf35e06cfa598ccb46391f12c09e32ef",
"sha256": "9595fe0a16b94e76edc02dad3a1da5f34ff66f7c863f890ca272daebd1dfaca2"
},
"downloads": -1,
"filename": "osm2pgsql-tuner-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "cf35e06cfa598ccb46391f12c09e32ef",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 6235,
"upload_time": "2024-01-04T03:07:57",
"upload_time_iso_8601": "2024-01-04T03:07:57.594098Z",
"url": "https://files.pythonhosted.org/packages/64/3b/03f2c33a7fc0105bd5f81987b24dd175e06e847fdbea4ac91362ca440df8/osm2pgsql-tuner-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-01-04 03:07:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "rustprooflabs",
"github_project": "osm2pgsql-tuner",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "coverage",
"specs": [
[
"==",
"7.4.0"
]
]
},
{
"name": "itsdangerous",
"specs": [
[
"==",
"2.1.2"
]
]
},
{
"name": "MarkupSafe",
"specs": [
[
"==",
"2.1.1"
]
]
},
{
"name": "pylint",
"specs": [
[
"==",
"2.17.5"
]
]
},
{
"name": "pytest",
"specs": [
[
"==",
"7.4.2"
]
]
},
{
"name": "six",
"specs": [
[
"==",
"1.16.0"
]
]
},
{
"name": "Sphinx",
"specs": [
[
"==",
"7.2.6"
]
]
},
{
"name": "sphinx-autoapi",
"specs": [
[
"==",
"2.1.1"
]
]
},
{
"name": "sphinx-git",
"specs": [
[
"==",
"11.0.0"
]
]
},
{
"name": "sphinx-rtd-theme",
"specs": [
[
"==",
"1.3.0"
]
]
}
],
"lcname": "osm2pgsql-tuner"
}
JSON
