# meilisync
[](https://pypi.python.org/pypi/meilisync)
[](https://github.com/meilisync/meilisync)
[](https://github.com/meilisync/meilisync/actions?query=workflow:pypi)
[](https://github.com/meilisync/meilisync/actions?query=workflow:ci)
[](https://github.com/long2ice/meilisync)
## Introduction
Realtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch.
There is also a web admin dashboard for meilisync [meilisync-admin](https://github.com/long2ice/meilisync-admin).
## Install
Just install from pypi:
```shell
pip install meilisync
```
## Use docker (Recommended)
You can use docker to run `meilisync`:
```yaml
version: "3"
services:
meilisync:
image: long2ice/meilisync
volumes:
- ./config.yml:/meilisync/config.yml
restart: always
```
## Prerequisites
- `MySQL`: `binlog_format = ROW`, use binary log.
- `PostgreSQL`: `wal_level = logical` and install `wal2json` extension, use logical replication.
- `MongoDB`: enable replica set mode, use change stream.
## Quick Start
If you run `meilisync` without any arguments, it will try to load the configuration from `config.yml` in the current
directory.
```shell
❯ meilisync --help
Usage: meilisync [OPTIONS] COMMAND [ARGS]...
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --config -c TEXT Config file path [default: config.yml] │
│ --install-completion Install completion for the current shell. │
│ --show-completion Show completion for the current shell, to copy it or customize the installation. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ check Check whether the data in the database is consistent with the data in Meilisearch │
│ refresh Refresh all data by swap index │
│ start Start meilisync │
│ version Show meilisync version │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```
### Start sync
Start sync data from MySQL to Meilisearch:
```shell
❯ meilisync start
2023-03-07 08:37:25.656 | INFO | meilisync.main:_:86 - Start increment sync data from "mysql" to Meilisearch...
```
### Refresh sync
Refresh all data by swap index:
```shell
❯ meilisync refresh -t test
```
Before refresh, you need stop the sync process first to avoid data inconsistency.
### Check sync
Check whether the data count in the database is consistent with the data in Meilisearch:
```shell
❯ meilisync check -t test
```
## Configuration
Here is an example configuration file:
```yaml
debug: true
plugins:
- meilisync.plugin.Plugin
progress:
type: file
source:
type: mysql
host: 192.168.123.205
port: 3306
user: root
password: "123456"
database: beauty
meilisearch:
api_url: http://192.168.123.205:7700
api_key:
insert_size: 1000
insert_interval: 10
sync:
- table: collection
index: beauty-collections
plugins:
- meilisync.plugin.Plugin
full: true
fields:
id:
title:
description:
category:
- table: picture
index: beauty-pictures
full: true
fields:
id:
description:
category:
sentry:
dsn: ""
environment: "production"
```
### debug (optional)
Enable debug mode, default is `false`, if you want to see more logs, you can set it to `true`.
### plugins (optional)
The plugins are used to customize the data before or after insert to Meilisearch and the plugins is a list of python
modules.
Which is a python class with `pre_event` and `post_event` methods, the `pre_event` method is called before insert to
Meilisearch, the `post_event` method is called after insert to Meilisearch.
```python
class Plugin:
is_global = False
async def pre_event(self, event: Event):
logger.debug(f"pre_event: {event}, is_global: {self.is_global}")
return event
async def post_event(self, event: Event):
logger.debug(f"post_event: {event}, is_global: {self.is_global}")
return event
```
The `is_global` is used to indicate whether the plugin instance is global, if set to `True`, the plugin instance will be
created only once, otherwise, the plugin instance will be created for each event.
### progress
The progress is used to record the last sync position, such as binlog position for MySQL.
- `type`: `file` or `redis`, if set to file, another option `path` is required.
- `path`: the file path to store the progress, default is `progress.json`.
- `key`: the redis key to store the progress, default is `meilisync:progress`.
- `dsn`: the redis dsn, default is `redis://localhost:6379/0`.
### source
Source database configuration, currently only support MySQL and PostgreSQL and MongoDB.
- `type`: `mysql` or `postgres` or `mongo`.
- `server_id`: the server id for MySQL binlog, default is `1`.
- `database`: the database name.
- `other keys`: the database connection arguments, MySQL see [asyncmy](https://github.com/long2ice/asyncmy), PostgreSQL
see [psycopg2](https://www.psycopg.org/docs/usage.html), MongoDB see [motor](https://motor.readthedocs.io/en/stable/).
### meilisearch
Meilisearch configuration.
- `api_url`: the Meilisearch API URL.
- `api_key`: the Meilisearch API key.
- `insert_size`: insert after collecting this many documents, optional.
- `insert_interval`: insert after this many seconds have passed, optional.
If nether `insert_size` nor `insert_interval` is set, it will insert each document immediately.
If you prefer performance, just set and increase `insert_size` and `insert_interval`. The insert will be made as long as
one of the conditions is met.
### sync
The sync configuration, you can add multiple sync tasks.
- `table`: the database table name or collection name.
- `index`: the Meilisearch index name, if not set, it will use the table name.
- `full`: whether to do a full sync, default is `false`.
- `fields`: the fields to sync, if not set, it will sync all fields. The key is table field name, the value is the
Meilisearch field name, if not set, it will use the table field name.
- `plugins`: the table level plugins, optional.
### sentry (optional)
Sentry configuration.
- `dsn`: the sentry dsn.
- `environment`: the sentry environment, default is `production`.
## License
This project is licensed under the
[Apache-2.0](https://github.com/meilisync/meilisync/blob/main/LICENSE) License.
Raw data
{
"_id": null,
"home_page": "https://github.com/long2ice/meilisync.git",
"name": "meilisync",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.9,<4.0",
"maintainer_email": "",
"keywords": "meilisearch,postgres,mysql,mongodb,sync",
"author": "long2ice",
"author_email": "long2ice@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/aa/e3/2287613a6aaeb099b7234280c27ccae777a5169313bbf2647b949a0102df/meilisync-0.1.3.tar.gz",
"platform": null,
"description": "# meilisync\n\n[](https://pypi.python.org/pypi/meilisync)\n[](https://github.com/meilisync/meilisync)\n[](https://github.com/meilisync/meilisync/actions?query=workflow:pypi)\n[](https://github.com/meilisync/meilisync/actions?query=workflow:ci)\n[](https://github.com/long2ice/meilisync)\n\n## Introduction\n\nRealtime sync data from MySQL/PostgreSQL/MongoDB to Meilisearch.\n\nThere is also a web admin dashboard for meilisync [meilisync-admin](https://github.com/long2ice/meilisync-admin).\n\n## Install\n\nJust install from pypi:\n\n```shell\npip install meilisync\n```\n\n## Use docker (Recommended)\n\nYou can use docker to run `meilisync`:\n\n```yaml\nversion: \"3\"\nservices:\n meilisync:\n image: long2ice/meilisync\n volumes:\n - ./config.yml:/meilisync/config.yml\n restart: always\n```\n\n## Prerequisites\n\n- `MySQL`: `binlog_format = ROW`, use binary log.\n- `PostgreSQL`: `wal_level = logical` and install `wal2json` extension, use logical replication.\n- `MongoDB`: enable replica set mode, use change stream.\n\n## Quick Start\n\nIf you run `meilisync` without any arguments, it will try to load the configuration from `config.yml` in the current\ndirectory.\n\n```shell\n\u276f meilisync --help\n\n Usage: meilisync [OPTIONS] COMMAND [ARGS]...\n\n\u256d\u2500 Options \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 --config -c TEXT Config file path [default: config.yml] \u2502\n\u2502 --install-completion Install completion for the current shell. \u2502\n\u2502 --show-completion Show completion for the current shell, to copy it or customize the installation. \u2502\n\u2502 --help Show this message and exit. \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n\u256d\u2500 Commands \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256e\n\u2502 check Check whether the data in the database is consistent with the data in Meilisearch \u2502\n\u2502 refresh Refresh all data by swap index \u2502\n\u2502 start Start meilisync \u2502\n\u2502 version Show meilisync version \u2502\n\u2570\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u256f\n```\n\n### Start sync\n\nStart sync data from MySQL to Meilisearch:\n\n```shell\n\u276f meilisync start\n2023-03-07 08:37:25.656 | INFO | meilisync.main:_:86 - Start increment sync data from \"mysql\" to Meilisearch...\n```\n\n### Refresh sync\n\nRefresh all data by swap index:\n\n```shell\n\u276f meilisync refresh -t test\n```\n\nBefore refresh, you need stop the sync process first to avoid data inconsistency.\n\n### Check sync\n\nCheck whether the data count in the database is consistent with the data in Meilisearch:\n\n```shell\n\u276f meilisync check -t test\n\n```\n\n## Configuration\n\nHere is an example configuration file:\n\n```yaml\ndebug: true\nplugins:\n - meilisync.plugin.Plugin\nprogress:\n type: file\nsource:\n type: mysql\n host: 192.168.123.205\n port: 3306\n user: root\n password: \"123456\"\n database: beauty\nmeilisearch:\n api_url: http://192.168.123.205:7700\n api_key:\n insert_size: 1000\n insert_interval: 10\nsync:\n - table: collection\n index: beauty-collections\n plugins:\n - meilisync.plugin.Plugin\n full: true\n fields:\n id:\n title:\n description:\n category:\n - table: picture\n index: beauty-pictures\n full: true\n fields:\n id:\n description:\n category:\nsentry:\n dsn: \"\"\n environment: \"production\"\n```\n\n### debug (optional)\n\nEnable debug mode, default is `false`, if you want to see more logs, you can set it to `true`.\n\n### plugins (optional)\n\nThe plugins are used to customize the data before or after insert to Meilisearch and the plugins is a list of python\nmodules.\n\nWhich is a python class with `pre_event` and `post_event` methods, the `pre_event` method is called before insert to\nMeilisearch, the `post_event` method is called after insert to Meilisearch.\n\n```python\nclass Plugin:\n is_global = False\n\n async def pre_event(self, event: Event):\n logger.debug(f\"pre_event: {event}, is_global: {self.is_global}\")\n return event\n\n async def post_event(self, event: Event):\n logger.debug(f\"post_event: {event}, is_global: {self.is_global}\")\n return event\n```\n\nThe `is_global` is used to indicate whether the plugin instance is global, if set to `True`, the plugin instance will be\ncreated only once, otherwise, the plugin instance will be created for each event.\n\n### progress\n\nThe progress is used to record the last sync position, such as binlog position for MySQL.\n\n- `type`: `file` or `redis`, if set to file, another option `path` is required.\n- `path`: the file path to store the progress, default is `progress.json`.\n- `key`: the redis key to store the progress, default is `meilisync:progress`.\n- `dsn`: the redis dsn, default is `redis://localhost:6379/0`.\n\n### source\n\nSource database configuration, currently only support MySQL and PostgreSQL and MongoDB.\n\n- `type`: `mysql` or `postgres` or `mongo`.\n- `server_id`: the server id for MySQL binlog, default is `1`.\n- `database`: the database name.\n- `other keys`: the database connection arguments, MySQL see [asyncmy](https://github.com/long2ice/asyncmy), PostgreSQL\n see [psycopg2](https://www.psycopg.org/docs/usage.html), MongoDB see [motor](https://motor.readthedocs.io/en/stable/).\n\n### meilisearch\n\nMeilisearch configuration.\n\n- `api_url`: the Meilisearch API URL.\n- `api_key`: the Meilisearch API key.\n- `insert_size`: insert after collecting this many documents, optional.\n- `insert_interval`: insert after this many seconds have passed, optional.\n\nIf nether `insert_size` nor `insert_interval` is set, it will insert each document immediately.\n\nIf you prefer performance, just set and increase `insert_size` and `insert_interval`. The insert will be made as long as\none of the conditions is met.\n\n### sync\n\nThe sync configuration, you can add multiple sync tasks.\n\n- `table`: the database table name or collection name.\n- `index`: the Meilisearch index name, if not set, it will use the table name.\n- `full`: whether to do a full sync, default is `false`.\n- `fields`: the fields to sync, if not set, it will sync all fields. The key is table field name, the value is the\n Meilisearch field name, if not set, it will use the table field name.\n- `plugins`: the table level plugins, optional.\n\n### sentry (optional)\n\nSentry configuration.\n\n- `dsn`: the sentry dsn.\n- `environment`: the sentry environment, default is `production`.\n\n## License\n\nThis project is licensed under the\n[Apache-2.0](https://github.com/meilisync/meilisync/blob/main/LICENSE) License.\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Realtime sync data from MySQL/PostgreSQL/MongoDB to meilisearch",
"version": "0.1.3",
"project_urls": {
"Documentation": "https://github.com/meilisync/meilisync",
"Homepage": "https://github.com/long2ice/meilisync.git",
"Repository": "https://github.com/long2ice/meilisync.git"
},
"split_keywords": [
"meilisearch",
"postgres",
"mysql",
"mongodb",
"sync"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a09c277d8a4f21498a72a674776a119d6d9fdc58e05fc8d71feef5b7f9fb3b70",
"md5": "4c0cb6ea80e7af27d06cc281baaef731",
"sha256": "7c92e765d435d51423fc15d52ada8feb1a116b06d3131f688b0e299a31eeae3a"
},
"downloads": -1,
"filename": "meilisync-0.1.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4c0cb6ea80e7af27d06cc281baaef731",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9,<4.0",
"size": 27723,
"upload_time": "2024-02-13T12:45:36",
"upload_time_iso_8601": "2024-02-13T12:45:36.439668Z",
"url": "https://files.pythonhosted.org/packages/a0/9c/277d8a4f21498a72a674776a119d6d9fdc58e05fc8d71feef5b7f9fb3b70/meilisync-0.1.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "aae32287613a6aaeb099b7234280c27ccae777a5169313bbf2647b949a0102df",
"md5": "9438bbd65b8cb99e98c1f7d38776b05c",
"sha256": "64894f5992e2d672c90f773aefee0bc18e857903815a76317571b7bffdb3318a"
},
"downloads": -1,
"filename": "meilisync-0.1.3.tar.gz",
"has_sig": false,
"md5_digest": "9438bbd65b8cb99e98c1f7d38776b05c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9,<4.0",
"size": 17989,
"upload_time": "2024-02-13T12:45:38",
"upload_time_iso_8601": "2024-02-13T12:45:38.218540Z",
"url": "https://files.pythonhosted.org/packages/aa/e3/2287613a6aaeb099b7234280c27ccae777a5169313bbf2647b949a0102df/meilisync-0.1.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-13 12:45:38",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "long2ice",
"github_project": "meilisync",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "meilisync"
}
JSON
