| Name | django-migrations-ci JSON |
| Version |
0.11
JSON |
| download |
| home_page | None |
| Summary | Django migrations CI optimization |
| upload_time | 2024-08-23 14:44:23 |
| maintainer | None |
| docs_url | None |
| author | Iuri de Silvio |
| requires_python | <4.0,>=3.8 |
| license | None |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# django-migrations-ci
Reuse database state on CI. Run migrations on CI tests only for changes.
Migrations are slow, but you have to run it on CI for testing reasons, so avoid
to run them when the database state was already tested.
## Install
Install the package with pip:
```shell
pip install django-migrations-ci
```
Add `django_migrations_ci` to Django settings `INSTALLED_APPS`.
```python
INSTALLED_APPS = [
..., # other packages
"django_migrations_ci",
]
```
## How to use
The command `migrateci` execute all migrations and save dump files `migrateci-*`.
If these files already exist on disk, they are used to prepare the database
without running all migrations again.
## Workflow
This is how the "run test" CI job should work.
```shell
./manage.py migrateci
./manage.py test --keepdb
```
It works with `pytest-django` too as a plugin:
```shell
pytest --migrateci --reuse-db
```
The recommended way to work with it is configuring default [pytest `addopts`](https://docs.pytest.org/en/7.1.x/example/simple.html#how-to-change-command-line-options-defaults) with `--migrateci --reuse-db` to run without recreating database. When you want to recreate, run pytest with `--create-db` that has precedence over `--reuse-db`.
## Parallel tests
```shell
./manage.py migrateci --parallel $(nproc)
./manage.py test --keepdb --parallel $(nproc)
```
### Parallel tests with pytest-django
```shell
pytest --migrateci --reuse-db --parallel $(nproc)
```
Also check [database names for parallel tests](#database-names-for-parallel-tests).
## Settings
##### `MIGRATECI_STORAGE="django.core.files.storage.FileSystemStorage"`
File storage class. The [django-storages](https://pypi.org/project/django-storages/) package has many backends implemented.
Saving cache files to an external storage allow the lib to reuse partial migrations.
When you write a new migration, it will try to get a cache without this
last migration and load from it, running only the new migrations.
An S3 example,, but it works with any custom backend:
```python
from storages.backends.s3boto3 import S3Boto3Storage
class MigrateCIStorage(S3Boto3Storage):
bucket_name = "mybucket-migrateci-cache"
region_name = "us-east-1"
object_parameters = {
"StorageClass": "REDUCED_REDUNDANCY",
}
```
##### `MIGRATECI_LOCATION=""`
[File storage API](https://docs.djangoproject.com/en/4.1/ref/files/storage/#the-filesystemstorage-class) has a location arg that all backend use in some way.
If no storage is defined, it defaults to `~/.migrateci` to make it easy to work local.
##### `MIGRATECI_PYTEST=False`
The [`pytest-django`](https://pypi.org/project/pytest-django) package use custom test database names.
If you use it and don´t change their default fixtures, just use `MIGRATECI_PYTEST=True`.
#### `MIGRATECI_PARALLEL=None`
Before tests, Django execute all migrations in one database and clone it to be able to run parallel tests.
Use `MIGRATECI_PARALLEL="auto"` to create one database per process or define the exact number of processes with `MIGRATECI_PARALLEL=4`.
It supports how Django test and how [pytest-xdist](https://pypi.org/project/pytest-xdist) works.
#### `MIGRATECI_DEPTH=1`
This is how we decide which migration cache to use.
First, it'll try to find a cache with all migration files, but in some cases it's not possible,
like when you just pushed a new migration.
For `MIGRATECI_DEPTH=1`, it'll remove one migration a time for each Django app installed and check if some cached migration exists. It support the most common use case and it's reasonably fast.
Bigger values cause a cost operation, it'll remove N migrations a time and check if some cached migration exists. It's a combination of every Django app. E.g. for 10 apps, it'll take at most 10^N checks, with some hashing operations.
### Command line settings
All below settings can be defined through command line args.
```
manage.py migrateci [-h] [-n PARALLEL] [--storage STORAGE_CLASS] [--location LOCATION]
[--pytest] [--depth DEPTH] [-v {0,1,2,3}]
options:
-h, --help show this help message and exit
-n PARALLEL, --parallel PARALLEL
--storage STORAGE_CLASS
--location LOCATION
--pytest
--depth DEPTH
--checksum Prints the current checksum and exits.
-v {0,1,2,3}
```
## Local migration caching
As a stretch of this package, it's possible to use the same strategy during local
development. It'll by default cache files at `~/.migrateci`.
```shell
./manage.py migrateci --parallel $(nproc)
./manage.py test --keepdb --parallel $(nproc)
```
## Why migrations are slow?
Django migrations are slow because of state recreation for every migration and other internal Django magic.
In the past, I tried to optimize that on Django core, but learnt it's a [running issue](https://code.djangoproject.com/ticket/29898).
## Supported databases
* mysql
* postgresql
* sqlite3
Django default run sqlite3 tests as in memory database and does not work because
`migrateci` runs in a different process. Add a test database name to settings,
like [sqlite test settings](django_migrations_ci/tests/testapp/settings_sqlite.py).
Django supports oracle, but the dump function is not implemented here.
## Database names for parallel tests
Django test framework has a `--parallel N` flag to test with N parallel processes,
naming databases from 1 to N.
* On sqlite3, a `db.sqlite3` generate `db_N.sqlite3` files.
* On other databases, a `db` generate `test_db_N`.
Pytest `pytest-django` use `pytest-xdist` for parallel support, naming databases
from 0 to N-1.
* On sqlite3, a `db.sqlite3` generate `db.sqlite3_gwN` files.
* On other databases, a `db` generate `test_db_gwN`.
Raw data
{
"_id": null,
"home_page": null,
"name": "django-migrations-ci",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": null,
"author": "Iuri de Silvio",
"author_email": "iurisilvio@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/08/2f/b96fb1ae28a83976a9650268504f8ed4100e43afff7eee896f90e749477c/django_migrations_ci-0.11.tar.gz",
"platform": null,
"description": "# django-migrations-ci\n\nReuse database state on CI. Run migrations on CI tests only for changes.\n\nMigrations are slow, but you have to run it on CI for testing reasons, so avoid\nto run them when the database state was already tested.\n\n## Install\n\nInstall the package with pip:\n\n```shell\npip install django-migrations-ci\n```\n\nAdd `django_migrations_ci` to Django settings `INSTALLED_APPS`.\n\n```python\nINSTALLED_APPS = [\n ..., # other packages\n \"django_migrations_ci\",\n]\n```\n\n## How to use\n\nThe command `migrateci` execute all migrations and save dump files `migrateci-*`.\n\nIf these files already exist on disk, they are used to prepare the database\nwithout running all migrations again.\n\n## Workflow\n\nThis is how the \"run test\" CI job should work.\n\n```shell\n./manage.py migrateci\n./manage.py test --keepdb\n```\n\nIt works with `pytest-django` too as a plugin:\n\n```shell\npytest --migrateci --reuse-db\n```\n\nThe recommended way to work with it is configuring default [pytest `addopts`](https://docs.pytest.org/en/7.1.x/example/simple.html#how-to-change-command-line-options-defaults) with `--migrateci --reuse-db` to run without recreating database. When you want to recreate, run pytest with `--create-db` that has precedence over `--reuse-db`.\n\n\n## Parallel tests\n\n```shell\n./manage.py migrateci --parallel $(nproc)\n./manage.py test --keepdb --parallel $(nproc)\n```\n\n### Parallel tests with pytest-django\n\n```shell\npytest --migrateci --reuse-db --parallel $(nproc)\n```\n\nAlso check [database names for parallel tests](#database-names-for-parallel-tests).\n\n## Settings\n\n##### `MIGRATECI_STORAGE=\"django.core.files.storage.FileSystemStorage\"`\n\nFile storage class. The [django-storages](https://pypi.org/project/django-storages/) package has many backends implemented.\n\nSaving cache files to an external storage allow the lib to reuse partial migrations.\nWhen you write a new migration, it will try to get a cache without this\nlast migration and load from it, running only the new migrations.\n\nAn S3 example,, but it works with any custom backend:\n\n```python\nfrom storages.backends.s3boto3 import S3Boto3Storage\n\nclass MigrateCIStorage(S3Boto3Storage):\n bucket_name = \"mybucket-migrateci-cache\"\n region_name = \"us-east-1\"\n object_parameters = {\n \"StorageClass\": \"REDUCED_REDUNDANCY\",\n }\n```\n\n##### `MIGRATECI_LOCATION=\"\"`\n\n[File storage API](https://docs.djangoproject.com/en/4.1/ref/files/storage/#the-filesystemstorage-class) has a location arg that all backend use in some way.\n\nIf no storage is defined, it defaults to `~/.migrateci` to make it easy to work local.\n\n##### `MIGRATECI_PYTEST=False`\n\nThe [`pytest-django`](https://pypi.org/project/pytest-django) package use custom test database names.\n\nIf you use it and don\u00b4t change their default fixtures, just use `MIGRATECI_PYTEST=True`.\n\n\n#### `MIGRATECI_PARALLEL=None`\n\nBefore tests, Django execute all migrations in one database and clone it to be able to run parallel tests.\nUse `MIGRATECI_PARALLEL=\"auto\"` to create one database per process or define the exact number of processes with `MIGRATECI_PARALLEL=4`.\n\nIt supports how Django test and how [pytest-xdist](https://pypi.org/project/pytest-xdist) works.\n\n#### `MIGRATECI_DEPTH=1`\n\nThis is how we decide which migration cache to use.\n\nFirst, it'll try to find a cache with all migration files, but in some cases it's not possible,\nlike when you just pushed a new migration.\n\nFor `MIGRATECI_DEPTH=1`, it'll remove one migration a time for each Django app installed and check if some cached migration exists. It support the most common use case and it's reasonably fast.\n\nBigger values cause a cost operation, it'll remove N migrations a time and check if some cached migration exists. It's a combination of every Django app. E.g. for 10 apps, it'll take at most 10^N checks, with some hashing operations.\n\n### Command line settings\n\nAll below settings can be defined through command line args.\n\n```\nmanage.py migrateci [-h] [-n PARALLEL] [--storage STORAGE_CLASS] [--location LOCATION]\n[--pytest] [--depth DEPTH] [-v {0,1,2,3}]\n\noptions:\n -h, --help show this help message and exit\n -n PARALLEL, --parallel PARALLEL\n --storage STORAGE_CLASS\n --location LOCATION\n --pytest\n --depth DEPTH\n --checksum Prints the current checksum and exits.\n -v {0,1,2,3}\n```\n\n## Local migration caching\n\nAs a stretch of this package, it's possible to use the same strategy during local\ndevelopment. It'll by default cache files at `~/.migrateci`.\n\n```shell\n./manage.py migrateci --parallel $(nproc)\n./manage.py test --keepdb --parallel $(nproc)\n```\n\n## Why migrations are slow?\n\nDjango migrations are slow because of state recreation for every migration and other internal Django magic.\n\nIn the past, I tried to optimize that on Django core, but learnt it's a [running issue](https://code.djangoproject.com/ticket/29898).\n\n## Supported databases\n\n* mysql\n* postgresql\n* sqlite3\n\nDjango default run sqlite3 tests as in memory database and does not work because\n`migrateci` runs in a different process. Add a test database name to settings,\nlike [sqlite test settings](django_migrations_ci/tests/testapp/settings_sqlite.py).\n\nDjango supports oracle, but the dump function is not implemented here.\n\n## Database names for parallel tests\n\nDjango test framework has a `--parallel N` flag to test with N parallel processes,\nnaming databases from 1 to N.\n\n* On sqlite3, a `db.sqlite3` generate `db_N.sqlite3` files.\n* On other databases, a `db` generate `test_db_N`.\n\nPytest `pytest-django` use `pytest-xdist` for parallel support, naming databases\nfrom 0 to N-1.\n\n* On sqlite3, a `db.sqlite3` generate `db.sqlite3_gwN` files.\n* On other databases, a `db` generate `test_db_gwN`.\n",
"bugtrack_url": null,
"license": null,
"summary": "Django migrations CI optimization",
"version": "0.11",
"project_urls": {
"Homepage": "https://github.com/businho/django-migrations-ci"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b2cc25fdda7535c1f6d345a1e5cf1ba4319841df9bf157659636737c71bf09ec",
"md5": "a5d9c969bfdaf507c116df4bec133358",
"sha256": "d959213ee7a8f9bf041ebd3b7c7719f97b2520d3dd8f6c385a3d14313ddeb376"
},
"downloads": -1,
"filename": "django_migrations_ci-0.11-py3-none-any.whl",
"has_sig": false,
"md5_digest": "a5d9c969bfdaf507c116df4bec133358",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 13814,
"upload_time": "2024-08-23T14:44:22",
"upload_time_iso_8601": "2024-08-23T14:44:22.026900Z",
"url": "https://files.pythonhosted.org/packages/b2/cc/25fdda7535c1f6d345a1e5cf1ba4319841df9bf157659636737c71bf09ec/django_migrations_ci-0.11-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "082fb96fb1ae28a83976a9650268504f8ed4100e43afff7eee896f90e749477c",
"md5": "86ac3d262270fc5fc4c2ca16ba89cf32",
"sha256": "31e922f9c91b98b819fca0585b03f080be7a39b90d0abc2b2ec40f88fc975d24"
},
"downloads": -1,
"filename": "django_migrations_ci-0.11.tar.gz",
"has_sig": false,
"md5_digest": "86ac3d262270fc5fc4c2ca16ba89cf32",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 13416,
"upload_time": "2024-08-23T14:44:23",
"upload_time_iso_8601": "2024-08-23T14:44:23.387427Z",
"url": "https://files.pythonhosted.org/packages/08/2f/b96fb1ae28a83976a9650268504f8ed4100e43afff7eee896f90e749477c/django_migrations_ci-0.11.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-23 14:44:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "businho",
"github_project": "django-migrations-ci",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"tox": true,
"lcname": "django-migrations-ci"
}
