# pypi-to-sqlite
[](https://pypi.org/project/pypi-to-sqlite/)
[](https://github.com/simonw/pypi-to-sqlite/releases)
[](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest)
[](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE)
Load data about Python packages from PyPI into SQLite
## Installation
Install this tool using `pip`:
pip install pypi-to-sqlite
## Usage
To create a SQLite database with details of one or more packages, run:
pypi-to-sqlite pypi.db datasette sqlite-utils
You can also process JSON that you have previously saved to disk like so:
curl -o datasette.json https://pypi.org/pypi/datasette/json
pypi-to-sqlite pypi.db -f datasette.json
The tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below.
To create the tables with a prefix, use `--prefix prefix`. For example:
pypi-to-sqlite pypi.db datasette --prefix pypi_
This will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`.
## Demo
You can see examples of tables created using this tool running in [Datasette](https://datasette.io/) here:
- [packages](https://datasette.io/content/pypi_packages)
- [versions](https://datasette.io/content/pypi_versions)
- [releases](https://datasette.io/content/pypi_releases)
## Database schema
<!-- [[[cog
import cog, json
from pypi_to_sqlite import cli
from click.testing import CliRunner
import sqlite_utils
import tempfile, pathlib
tmpdir = pathlib.Path(tempfile.mkdtemp())
db_path = str(tmpdir / "pypi.db")
runner = CliRunner()
result = runner.invoke(cli.cli, [db_path, "-f", "tests/datasette-block.json"])
cog.out("```sql\n")
cog.out(sqlite_utils.Database(db_path).schema)
cog.out("\n```")
]]] -->
```sql
CREATE TABLE [packages] (
[name] TEXT PRIMARY KEY,
[summary] TEXT,
[classifiers] TEXT,
[description] TEXT,
[author] TEXT,
[author_email] TEXT,
[description_content_type] TEXT,
[home_page] TEXT,
[keywords] TEXT,
[license] TEXT,
[maintainer] TEXT,
[maintainer_email] TEXT,
[package_url] TEXT,
[platform] TEXT,
[project_url] TEXT,
[project_urls] TEXT,
[release_url] TEXT,
[requires_dist] TEXT,
[requires_python] TEXT,
[version] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
CREATE TABLE [versions] (
[id] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[name] TEXT
);
CREATE TABLE [releases] (
[md5_digest] TEXT PRIMARY KEY,
[package] TEXT REFERENCES [packages]([name]),
[version] TEXT REFERENCES [versions]([id]),
[packagetype] TEXT,
[filename] TEXT,
[comment_text] TEXT,
[digests] TEXT,
[has_sig] INTEGER,
[python_version] TEXT,
[requires_python] TEXT,
[size] INTEGER,
[upload_time] TEXT,
[upload_time_iso_8601] TEXT,
[url] TEXT,
[yanked] INTEGER,
[yanked_reason] TEXT
);
```
<!-- [[[end]]] -->
## pypi-to-sqlite --help
<!-- [[[cog
result = runner.invoke(cli.cli, ["--help"])
cog.out("```\n")
cog.out(result.output.replace("Usage: cli", "Usage: pypi-to-sqlite"))
cog.out("\n```")
]]] -->
```
Usage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...
Load data about Python packages from PyPI into SQLite
Usage example:
pypi-to-sqlite pypy.db datasette sqlite-utils
Use -f to load data from a JSON file instead:
pypi-to-sqlite pypy.db -f datasette.json
Created tables will be packages, versions and releases
To create tables called pypi_packages, pypi_versions, pypi_releases use
--prefix pypi_:
pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_
Options:
--version Show the version and exit.
-f, --file FILENAME Import JSON from this file
-d, --delay FLOAT Wait this many seconds between requests
--prefix TEXT Prefix to use for the created database tables
--help Show this message and exit.
```
<!-- [[[end]]] -->
## Development
To contribute to this tool, first checkout the code. Then create a new virtual environment:
cd pypi-to-sqlite
python -m venv venv
source venv/bin/activate
Now install the dependencies and test dependencies:
pip install -e '.[test]'
To run the tests:
pytest
Raw data
{
"_id": null,
"home_page": "https://github.com/simonw/pypi-to-sqlite",
"name": "pypi-to-sqlite",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "Simon Willison",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/1f/29/cf97b1c72cb1aab81917e44d52e34ba3ffbb7efdfa2125ccf7cedc852602/pypi-to-sqlite-0.2.3.tar.gz",
"platform": null,
"description": "# pypi-to-sqlite\n\n[](https://pypi.org/project/pypi-to-sqlite/)\n[](https://github.com/simonw/pypi-to-sqlite/releases)\n[](https://github.com/simonw/pypi-to-sqlite/actions?query=workflow%3ATest)\n[](https://github.com/simonw/pypi-to-sqlite/blob/master/LICENSE)\n\nLoad data about Python packages from PyPI into SQLite\n\n## Installation\n\nInstall this tool using `pip`:\n\n pip install pypi-to-sqlite\n\n## Usage\n\nTo create a SQLite database with details of one or more packages, run:\n\n pypi-to-sqlite pypi.db datasette sqlite-utils\n\nYou can also process JSON that you have previously saved to disk like so:\n\n curl -o datasette.json https://pypi.org/pypi/datasette/json\n pypi-to-sqlite pypi.db -f datasette.json\n\nThe tool will create three tables: `packages`, `versions` and `releases`. The full table schema is shown below.\n\nTo create the tables with a prefix, use `--prefix prefix`. For example:\n\n pypi-to-sqlite pypi.db datasette --prefix pypi_\n\nThis will create tables called `pypi_packages`, `pypi_versions` and `pypi_releases`.\n\n## Demo\n\nYou can see examples of tables created using this tool running in [Datasette](https://datasette.io/) here:\n\n- [packages](https://datasette.io/content/pypi_packages)\n- [versions](https://datasette.io/content/pypi_versions)\n- [releases](https://datasette.io/content/pypi_releases)\n\n## Database schema\n\n<!-- [[[cog\nimport cog, json\nfrom pypi_to_sqlite import cli\nfrom click.testing import CliRunner\nimport sqlite_utils\nimport tempfile, pathlib\ntmpdir = pathlib.Path(tempfile.mkdtemp())\ndb_path = str(tmpdir / \"pypi.db\")\nrunner = CliRunner()\nresult = runner.invoke(cli.cli, [db_path, \"-f\", \"tests/datasette-block.json\"])\ncog.out(\"```sql\\n\")\ncog.out(sqlite_utils.Database(db_path).schema)\ncog.out(\"\\n```\")\n]]] -->\n```sql\nCREATE TABLE [packages] (\n [name] TEXT PRIMARY KEY,\n [summary] TEXT,\n [classifiers] TEXT,\n [description] TEXT,\n [author] TEXT,\n [author_email] TEXT,\n [description_content_type] TEXT,\n [home_page] TEXT,\n [keywords] TEXT,\n [license] TEXT,\n [maintainer] TEXT,\n [maintainer_email] TEXT,\n [package_url] TEXT,\n [platform] TEXT,\n [project_url] TEXT,\n [project_urls] TEXT,\n [release_url] TEXT,\n [requires_dist] TEXT,\n [requires_python] TEXT,\n [version] TEXT,\n [yanked] INTEGER,\n [yanked_reason] TEXT\n);\nCREATE TABLE [versions] (\n [id] TEXT PRIMARY KEY,\n [package] TEXT REFERENCES [packages]([name]),\n [name] TEXT\n);\nCREATE TABLE [releases] (\n [md5_digest] TEXT PRIMARY KEY,\n [package] TEXT REFERENCES [packages]([name]),\n [version] TEXT REFERENCES [versions]([id]),\n [packagetype] TEXT,\n [filename] TEXT,\n [comment_text] TEXT,\n [digests] TEXT,\n [has_sig] INTEGER,\n [python_version] TEXT,\n [requires_python] TEXT,\n [size] INTEGER,\n [upload_time] TEXT,\n [upload_time_iso_8601] TEXT,\n [url] TEXT,\n [yanked] INTEGER,\n [yanked_reason] TEXT\n);\n```\n<!-- [[[end]]] -->\n\n## pypi-to-sqlite --help\n\n<!-- [[[cog\nresult = runner.invoke(cli.cli, [\"--help\"])\ncog.out(\"```\\n\")\ncog.out(result.output.replace(\"Usage: cli\", \"Usage: pypi-to-sqlite\"))\ncog.out(\"\\n```\")\n]]] -->\n```\nUsage: pypi-to-sqlite [OPTIONS] DB_PATH [PACKAGE]...\n\n Load data about Python packages from PyPI into SQLite\n\n Usage example:\n\n pypi-to-sqlite pypy.db datasette sqlite-utils\n\n Use -f to load data from a JSON file instead:\n\n pypi-to-sqlite pypy.db -f datasette.json\n\n Created tables will be packages, versions and releases\n\n To create tables called pypi_packages, pypi_versions, pypi_releases use\n --prefix pypi_:\n\n pypi-to-sqlite pypy.db datasette sqlite-utils --prefix pypi_\n\nOptions:\n --version Show the version and exit.\n -f, --file FILENAME Import JSON from this file\n -d, --delay FLOAT Wait this many seconds between requests\n --prefix TEXT Prefix to use for the created database tables\n --help Show this message and exit.\n\n```\n<!-- [[[end]]] -->\n\n## Development\n\nTo contribute to this tool, first checkout the code. Then create a new virtual environment:\n\n cd pypi-to-sqlite\n python -m venv venv\n source venv/bin/activate\n\nNow install the dependencies and test dependencies:\n\n pip install -e '.[test]'\n\nTo run the tests:\n\n pytest\n",
"bugtrack_url": null,
"license": "Apache License, Version 2.0",
"summary": "Load data about Python packages from PyPI into SQLite",
"version": "0.2.3",
"project_urls": {
"CI": "https://github.com/simonw/pypi-to-sqlite/actions",
"Changelog": "https://github.com/simonw/pypi-to-sqlite/releases",
"Homepage": "https://github.com/simonw/pypi-to-sqlite",
"Issues": "https://github.com/simonw/pypi-to-sqlite/issues"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "edb1b417733d3808d635e5b6aa29ec829df81eddfebaf2f7b37379c17a6a3106",
"md5": "9214205f884e9197a36072a7024d1d03",
"sha256": "20f720cecb4920b7456deb0c718cefb321fb5124f7058fe1b203840a64ec98c0"
},
"downloads": -1,
"filename": "pypi_to_sqlite-0.2.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9214205f884e9197a36072a7024d1d03",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 8719,
"upload_time": "2024-03-06T06:17:52",
"upload_time_iso_8601": "2024-03-06T06:17:52.168424Z",
"url": "https://files.pythonhosted.org/packages/ed/b1/b417733d3808d635e5b6aa29ec829df81eddfebaf2f7b37379c17a6a3106/pypi_to_sqlite-0.2.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "1f29cf97b1c72cb1aab81917e44d52e34ba3ffbb7efdfa2125ccf7cedc852602",
"md5": "4f27ea2f944de13c32853235156e4772",
"sha256": "d6866be63e347d5733d259e0708ffc6e69409744203c82e438ee3bbda44a4e20"
},
"downloads": -1,
"filename": "pypi-to-sqlite-0.2.3.tar.gz",
"has_sig": false,
"md5_digest": "4f27ea2f944de13c32853235156e4772",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 8999,
"upload_time": "2024-03-06T06:17:53",
"upload_time_iso_8601": "2024-03-06T06:17:53.604156Z",
"url": "https://files.pythonhosted.org/packages/1f/29/cf97b1c72cb1aab81917e44d52e34ba3ffbb7efdfa2125ccf7cedc852602/pypi-to-sqlite-0.2.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-06 06:17:53",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "simonw",
"github_project": "pypi-to-sqlite",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "pypi-to-sqlite"
}
JSON
