# es-testbed
A way to create indices, datastreams, and snapshots to facilitate testing.
# Preliminary Documentation
## 1. Create a Preset
Create a preset directory. An example preset directory is in
`src/es_testbed/presets/searchable_test`.
Your preset directory must include the following files:
- A plan YAML file, e.g. `plan.yml`
- A buildlist YAML file, e.g. `buildlist.yml`
- A `functions.py` file (the actual python code), which must contain a
function named `doc_generator()`. This function must accept all kwargs from
the buildlist's `options`
- A `definitions.py` file, which is a Python variable file that helps find
the path to the module, etc., as well as import the plan, the buildlist,
the mappings and settings, etc. This must at least include a `get_plan()`
function that returns a dictionary of a built/configured plan.
- A `mappings.json` file (contains the index mappings your docs need)
- A `settings.json` file (contains the index settings)
Any other files can be included to help your doc_generator function, e.g.
Faker definitions and classes, etc. Once the preset module is imported,
relative imports should work.
**Note:** If `ilm['enabled'] == False`, the other subkeys will be ignored. In fact, `ilm: False` is also acceptable.
Acceptable values for `readonly` are the tier names where `readonly` is acceptable: `hot`, `warm`, or `cold`.
Save this for step 2.
## 2. Create a TestBed
**Must have an Elasticsearch client established to proceed**
```
from es_testbed import TestBed
tb = TestBed(client, **kwargs)
```
For a builtin preset, like `searchable_test`, this is:
```
tb = TestBed(client, builtin='searchable_test', scenario=None)
```
`scenario` can be one of many, if configured.
For using your own preset in a filesystem path:
```
tb = TestBed(client, path='/path/to/preset/dir')
```
`path` _must_ be a directory.
For importing from a Git repository:
```
tb = TestBed(client, url='user:token@https://github.com/GITUSER/reponame.git', ref='main', path='subpath/to/preset')
```
Note that `user:token@` is only necessary if it's a protected repository.
In this case `path` must be a subdirectory of the repository. `depth=1` is manually set, so only the most recent
bits will be pulled into a tmpdir, which will be destroyed as part of `teardown()`.
### 2.1 Index Template creation (behind the scenes)
Based on the settings in step 1, you will have 2 component templates and one index template that
references them:
#### Component Template 1
```
es-testbed-cmp-mytest-000001
```
Settings:
```
{'index.number_of_replicas': 0}
```
If using a `rollover_alias` or ILM Policy, then additional values will automatically be added.
#### Component Template 2
```
es-testbed-cmp-mytest-000002
```
Mappings:
```
{
'properties': {
'@timestamp': {'type': 'date'},
'message': {'type': 'keyword'},
'number': {'type': 'long'},
'nested': {'properties': {'key': {'type': 'keyword'}}},
'deep': {'properties': {'l1': {'properties': {'l2': {
'properties': {'l3': {'type': 'keyword'}}}}}}
}
}
}
```
#### Index Template
```
es-testbed-tmpl-mytest-000001
```
### 2.2 You have indices
Based on what was provided in step 1, you will have 3 indices with a basic mapping, and 10 documents each:
```
es-testbed-idx-mytest-000001
es-testbed-idx-mytest-000002
es-testbed-idx-mytest-000003
```
Documents will have been added per the TestPlan settings. The orthography for these documents is in
`es_testbed.helpers.utils.doc_gen()`. Counts are preserved and continue to grow from one index to
the next.
## 3. Perform your tests.
This is where the testing can be performed.
## 4. Teardown
```
tb.teardown()
```
Barring anything unusual happening, all indices, data_streams, ILM policies, index & component
templates, and snapshots (if an index is promoted to searchable snapshots) will be deleted as part
of the `teardown()` method.
Raw data
{
"_id": null,
"home_page": null,
"name": "es-testbed",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "datastream, elasticsearch, index, repository, snapshot, testing",
"author": null,
"author_email": "Elastic <info@elastic.co>",
"download_url": "https://files.pythonhosted.org/packages/98/a9/34a05825b90eb0584a09c55324b0b10ce99e3f5d9945897e99b9ae7dfe47/es_testbed-0.8.5.tar.gz",
"platform": null,
"description": "# es-testbed\nA way to create indices, datastreams, and snapshots to facilitate testing.\n\n# Preliminary Documentation\n\n## 1. Create a Preset\n\nCreate a preset directory. An example preset directory is in\n`src/es_testbed/presets/searchable_test`.\n\nYour preset directory must include the following files:\n\n- A plan YAML file, e.g. `plan.yml`\n- A buildlist YAML file, e.g. `buildlist.yml`\n- A `functions.py` file (the actual python code), which must contain a\n function named `doc_generator()`. This function must accept all kwargs from\n the buildlist's `options`\n- A `definitions.py` file, which is a Python variable file that helps find\n the path to the module, etc., as well as import the plan, the buildlist,\n the mappings and settings, etc. This must at least include a `get_plan()`\n function that returns a dictionary of a built/configured plan.\n- A `mappings.json` file (contains the index mappings your docs need)\n- A `settings.json` file (contains the index settings)\nAny other files can be included to help your doc_generator function, e.g.\nFaker definitions and classes, etc. Once the preset module is imported,\nrelative imports should work.\n\n**Note:** If `ilm['enabled'] == False`, the other subkeys will be ignored. In fact, `ilm: False` is also acceptable.\n\nAcceptable values for `readonly` are the tier names where `readonly` is acceptable: `hot`, `warm`, or `cold`.\n\nSave this for step 2.\n\n## 2. Create a TestBed\n\n**Must have an Elasticsearch client established to proceed**\n\n```\nfrom es_testbed import TestBed\n\ntb = TestBed(client, **kwargs)\n```\n\nFor a builtin preset, like `searchable_test`, this is:\n\n```\ntb = TestBed(client, builtin='searchable_test', scenario=None)\n```\n\n`scenario` can be one of many, if configured.\n\nFor using your own preset in a filesystem path:\n\n```\ntb = TestBed(client, path='/path/to/preset/dir')\n```\n\n`path` _must_ be a directory.\n\nFor importing from a Git repository:\n\n```\ntb = TestBed(client, url='user:token@https://github.com/GITUSER/reponame.git', ref='main', path='subpath/to/preset')\n```\n\nNote that `user:token@` is only necessary if it's a protected repository.\nIn this case `path` must be a subdirectory of the repository. `depth=1` is manually set, so only the most recent\nbits will be pulled into a tmpdir, which will be destroyed as part of `teardown()`.\n\n### 2.1 Index Template creation (behind the scenes)\n\nBased on the settings in step 1, you will have 2 component templates and one index template that\nreferences them:\n\n#### Component Template 1\n\n```\nes-testbed-cmp-mytest-000001\n```\n\nSettings:\n\n```\n{'index.number_of_replicas': 0}\n```\n\nIf using a `rollover_alias` or ILM Policy, then additional values will automatically be added.\n\n#### Component Template 2\n\n```\nes-testbed-cmp-mytest-000002\n```\n\nMappings:\n\n```\n{\n 'properties': {\n '@timestamp': {'type': 'date'},\n 'message': {'type': 'keyword'},\n 'number': {'type': 'long'},\n 'nested': {'properties': {'key': {'type': 'keyword'}}},\n 'deep': {'properties': {'l1': {'properties': {'l2': {\n 'properties': {'l3': {'type': 'keyword'}}}}}}\n }\n }\n}\n```\n\n#### Index Template\n```\nes-testbed-tmpl-mytest-000001\n```\n\n### 2.2 You have indices\n\nBased on what was provided in step 1, you will have 3 indices with a basic mapping, and 10 documents each:\n\n```\nes-testbed-idx-mytest-000001\nes-testbed-idx-mytest-000002\nes-testbed-idx-mytest-000003\n```\n\nDocuments will have been added per the TestPlan settings. The orthography for these documents is in\n`es_testbed.helpers.utils.doc_gen()`. Counts are preserved and continue to grow from one index to\nthe next.\n\n## 3. Perform your tests.\n\nThis is where the testing can be performed.\n\n## 4. Teardown\n\n```\ntb.teardown()\n```\n\nBarring anything unusual happening, all indices, data_streams, ILM policies, index & component\ntemplates, and snapshots (if an index is promoted to searchable snapshots) will be deleted as part\nof the `teardown()` method.",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Library to help with building and tearing down indices, data streams, repositories and snapshots",
"version": "0.8.5",
"project_urls": {
"Documentation": "https://github.com/untergeek/es-testbed#readme",
"Issues": "https://github.com/untergeek/es-testbed/issues",
"Source": "https://github.com/untergeek/es-testbed"
},
"split_keywords": [
"datastream",
" elasticsearch",
" index",
" repository",
" snapshot",
" testing"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f52b8bc817631c30d80f210fe5c1f84b2a0c76a2bc3c7f6e8610cd44f8f44a44",
"md5": "9b080a29926ea8a8bdb3c131bddf697d",
"sha256": "4ece3c6bf2024bd130fe2d11336c4d1687e19db835093035528ef2330a4732d3"
},
"downloads": -1,
"filename": "es_testbed-0.8.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9b080a29926ea8a8bdb3c131bddf697d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 44308,
"upload_time": "2024-08-31T15:43:02",
"upload_time_iso_8601": "2024-08-31T15:43:02.155799Z",
"url": "https://files.pythonhosted.org/packages/f5/2b/8bc817631c30d80f210fe5c1f84b2a0c76a2bc3c7f6e8610cd44f8f44a44/es_testbed-0.8.5-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "98a934a05825b90eb0584a09c55324b0b10ce99e3f5d9945897e99b9ae7dfe47",
"md5": "6e66bcca818e7695d95a9b0b622c11bb",
"sha256": "4d9bb7f7100499d0ea085e096ea35d2f10e78cbf7673f4df6c57a1abd7e805b9"
},
"downloads": -1,
"filename": "es_testbed-0.8.5.tar.gz",
"has_sig": false,
"md5_digest": "6e66bcca818e7695d95a9b0b622c11bb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 130967,
"upload_time": "2024-08-31T15:43:03",
"upload_time_iso_8601": "2024-08-31T15:43:03.656305Z",
"url": "https://files.pythonhosted.org/packages/98/a9/34a05825b90eb0584a09c55324b0b10ce99e3f5d9945897e99b9ae7dfe47/es_testbed-0.8.5.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-31 15:43:03",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "untergeek",
"github_project": "es-testbed#readme",
"travis_ci": false,
"coveralls": true,
"github_actions": false,
"lcname": "es-testbed"
}
JSON
