# Synchronizer Framework
This framework provides a data synchronization pattern based on a declarative bi-directional definition. The definition
is a mapping of entity types with identity keys and non-key attributes which support CRUD operations and a
synchronization strategy. The data sources may be arbitrarily implemented as long as they inherit from `sync.base.Base`.
Common data sources like ORMs (django-orm, sqlalchemy) or REST APIs may be used as long as they support CRUD.
Data sources may mixin `sync.base.Timestamped` to apply a last-write wins synchronization strategy based on
the `updated_at` field.
The project targets audiences which have chosen data synchronization as a primary integration method for microservices.
Inspiration has been taken from projects Kubernetes and Chef.
Disclaimer: The demo and README mentions systems like Desire and Daphne. Those are no open source projects, but some
target audience will understand them. The definition is supposed to cover all use-cases of the Desire architecture.
### Features
* Adding missing objects
* Defer adding objects, which miss required foreign keys, because they do not exist or which are not synchronized yet
* Linking synchronized objects with each other to track them subsequently
* Many objects contributing to a single synchronized object on the remote side
* Updating non-key attributes
* Last write wins (entity needs to support `sync.base.Timestamped`)
* Deep-join attribute sourcing
* Abstract data access to make the pattern work with any CRUD data source (like Microsoft Dynamics, ldap, REST APIs,
netconf, gNMI, CSP SDKs)
* i.e. AWS SDK Create as `AllocateHostedConnection`, Read as `DescribeConnections`, ...
* Data sources need to implement crud operations as defined in `sync.datasource.DataSource`
* pre-load/include for deep joins
* Optional deletion based on orphaned entities
* Possibility to expedite single object synchronization based on an event stream
* Celery support
* SQLAlchemy backend support
* Redis linking backend support
* dockerized
More details about the pattern can be found in the `tests` path.
### Missing/TODO
* Deletion based on intent (with cryptographic verification)
### Known Limitations
1. Cannot synchronize three-way at once.
*Example:*
Assume entities `PA` required parent of `A`, `BA` required parent of `B`, `CA` required parent of `C`. Mappings are
defined as `M1: A -> B` and `M2: B <- C`. The pattern prevents each `M1` and `M2` to progress independently. For `M1`
, information about `PC` is missing and for `M2` information about `PA` is missing. Further, assume `x(e)` to denote
an entity `e`
transformed to the namespace of `x`.
*Solution:*
* Introduce an intermediary entity `I` which accepts optional parents `I(PA)` and `I(PC)`
* remove `M1` and `M2`
* create `M3: A -> I`, `M4: I <- C`
* create `M5: I -> B`
The sequence of `M3, M4, M5` leads to a synchronized `B` containing both parents `B(PA)` and `B(PC)`. The order of
events always keeps consistent states. For example, `M5` may never progress before `M4` even if it is executed
before.
The solution is *not* limited to uni-directional mappings, but the author chose demonstrating uni-directional
mappings for easier understanding of how data flows.
1. The orphan deletion may be counter-intuitive and will be changed in a future version. Refrain from using it, if you
do not understand the consequences.
## Define Entities
Entities below resemble the desire and daphne entity hierarchy and deviate to some extent. The intention is to
demonstrate the synchronization pattern instead of resembling existing models accurately. The details which are left out
of consideration seem to be straight-forward to implement.
### Base
```python
class DemoBase(InMemoryLinks, Hierarchical, InMemoryTimestamp):
...
```
### Daphne
```python
class VlanServiceProvider(DemoBase):
def __init__(self, name=None, children=set()):
super().__init__()
self.name = name
self.children = children
class VlanServiceConnection(DemoBase):
def __init__(self, name, region, pop):
super().__init__()
self.name = name
self.region = region
self.pop = pop
```
### Desire
```python
class CloudServiceProvider(DemoBase):
def __init__(self, name=None, children=None):
super().__init__()
self.name = name
self.children = children or set()
class CloudRegion(DemoBase):
def __init__(self, name=None, children=None):
super().__init__()
self.name = name
self.children = children or set()
class CloudHandover(DemoBase):
def __init__(self, name=None, children=None):
super().__init__()
self.name = name
self.children = children or set()
class NIC(DemoBase):
def __init__(self, name=None):
super().__init__()
self.name = name
```
## Define Mapping
```python
mappings = [
Mapping(
entity_types=(VlanServiceConnection, CloudRegion),
modes={Mode.LEFT_TO_RIGHT},
keys={
AttributeMap("region", "name"),
AttributeMap(
"parent", "parent", VlanServiceProvider, CloudServiceProvider
),
},
attributes={AttributeMap("externalRef", "external_ref")},
),
Mapping(
entity_types=(VlanServiceProvider, CloudServiceProvider),
modes={Mode.LEFT_TO_RIGHT, Mode.RIGHT_TO_LEFT},
keys={AttributeMap("name", "name")},
attributes={AttributeMap("canUpgrade", "upgrade_allowed")},
),
Mapping(
entity_types=(VlanServiceConnection, CloudHandover),
modes={Mode.LEFT_TO_RIGHT},
keys={
AttributeMap("pop", "name"),
AttributeMap("__self__", "parent", VlanServiceConnection, CloudRegion),
},
attributes={AttributeMap("externalRef", "external_ref")},
),
Mapping(
entity_types=(VlanServiceConnection, NIC),
modes={Mode.RIGHT_TO_LEFT},
keys={
AttributeMap("name", "name"),
AttributeMap("region", ("parent", "parent", "name")),
AttributeMap("pop", ("parent", "name")),
AttributeMap(
"parent",
("parent", "parent", "parent"),
VlanServiceProvider,
CloudServiceProvider,
),
},
attributes={AttributeMap("externalRef", "external_ref")},
),
Mapping(
entity_types=(VlanServiceConnection, NIC),
modes={Mode.LEFT_TO_RIGHT},
keys={
AttributeMap("name", "name"),
AttributeMap(
"__self__", "parent", VlanServiceConnection, CloudHandover
),
},
attributes={AttributeMap("externalRef", "external_ref")},
),
]
```
## Define some objects
```python
daphne_db = Db.from_collection(
{
VlanServiceProvider(
name="AWS",
children={
VlanServiceConnection("nic-1", "eu-central-1", "INX6"),
VlanServiceConnection("nic-2", "eu-central-1", "EqFA5"),
VlanServiceConnection("nic-3", "eu-west-2", "EqFA5"),
},
),
VlanServiceProvider(name="AZURE", children=set()),
}
)
desire_db = Db.from_collection(
{
CloudServiceProvider(
name="AWS",
children={
CloudRegion(
name="eu-central-1",
children={
CloudHandover(name="INX6", children={NIC("nic-1")}),
CloudHandover(name="EqFA5", children={NIC("nic-2a")}),
},
),
CloudRegion(
name="eu-west-1",
children={
CloudHandover(name="LON1", children={NIC(name="nic-3a")})
},
),
CloudRegion(
name="eu-west-2",
children={
CloudHandover(
name="EqFA5",
children={NIC(name="nic-3", external_ref="nic3-eref")},
)
},
),
},
),
CloudServiceProvider(
name="IBM",
children={
CloudRegion(
name="EU-Frankfurt",
children={CloudHandover(name="fra03", children={NIC("nic-4")})},
)
},
),
}
)
```
## Plug it together
```python
sync(daphne_db, desire_db, mappings)
```
## Demo Output
```shell
$ python src/demo/main.py
/home/sspies/git/sync-poc/venv/bin/python3 /home/sspies/git/sync-poc/src/demo/main.py
INFO:root:Initial DB states
INFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),
CloudServiceProvider(name=AWS)],
<class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),
CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),
CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),
CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],
<class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),
CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),
CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),
CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),
CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],
<class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),
NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),
NIC(name=nic-1, parent=CloudHandover(name=INX6)),
NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),
NIC(name=nic-3a, parent=CloudHandover(name=LON1))]}
INFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),
VlanServiceProvider(name=AWS)],
<class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),
VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),
VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)]}
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider
INFO:root:✨ Adding CloudServiceProvider(name=AZURE)
INFO:root:✨ Adding VlanServiceProvider(name=IBM)
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).__self__. VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6) not in CloudRegion
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).__self__. VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5) not in CloudRegion
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudRegion
INFO:root:✨ Adding VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)
INFO:root:✨ Adding VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)
INFO:root:✨ Adding VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).__self__. VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).__self__. VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03).__self__. VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudHandover
WARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1).__self__. VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1) not in CloudHandover
INFO:root:27 operation(s) made
INFO:root:Operations: [SkipMissingLink(VlanServiceConnection -> CloudRegion),
SkipMissingLink(VlanServiceConnection -> CloudRegion),
SkipMissingLink(VlanServiceConnection -> CloudRegion),
Addition(CloudServiceProvider(name=AZURE)),
Linking(frozenset({VlanServiceProvider(name=AZURE), CloudServiceProvider(name=AZURE)})),
Linking(frozenset({VlanServiceProvider(name=AWS), CloudServiceProvider(name=AWS)})),
AttributeUpdated(CloudServiceProvider(name=AWS).upgrade_allowed: True -> False from VlanServiceProvider(name=AWS)),
Addition(VlanServiceProvider(name=IBM)),
Linking(frozenset({VlanServiceProvider(name=IBM), CloudServiceProvider(name=IBM)})),
SkipMissingLink(VlanServiceConnection -> CloudHandover),
SkipMissingLink(VlanServiceConnection -> CloudHandover),
SkipMissingLink(VlanServiceConnection -> CloudHandover),
Addition(VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)),
Linking(frozenset({NIC(name=nic-4, parent=CloudHandover(name=fra03)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),
Linking(frozenset({NIC(name=nic-3, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),
AttributeUpdated(VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).externalRef: -> nic3-eref from NIC(name=nic-3, parent=CloudHandover(name=EqFA5))),
Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), NIC(name=nic-1, parent=CloudHandover(name=INX6))})),
Addition(VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)),
Linking(frozenset({NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),
Addition(VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)),
Linking(frozenset({NIC(name=nic-3a, parent=CloudHandover(name=LON1)), VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)})),
SkipMissingLink(VlanServiceConnection -> NIC),
SkipMissingLink(VlanServiceConnection -> NIC),
SkipMissingLink(VlanServiceConnection -> NIC),
SkipMissingLink(VlanServiceConnection -> NIC),
SkipMissingLink(VlanServiceConnection -> NIC),
SkipMissingLink(VlanServiceConnection -> NIC)]
INFO:root:DB states after first and before second run
INFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),
CloudServiceProvider(name=AWS),
CloudServiceProvider(name=AZURE)],
<class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),
CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),
CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),
CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],
<class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),
CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),
CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),
CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),
CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],
<class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),
NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),
NIC(name=nic-1, parent=CloudHandover(name=INX6)),
NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),
NIC(name=nic-3a, parent=CloudHandover(name=LON1))]}
INFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),
VlanServiceProvider(name=AWS),
VlanServiceProvider(name=IBM)],
<class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),
VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),
VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03),
VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)]}
INFO:root:✨ Adding NIC(name=nic-2, parent=CloudHandover(name=EqFA5))
INFO:root:Sync with 16 operation(s) made
INFO:root:Operations: [Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS))})),
Linking(frozenset({CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),
AttributeUpdated(CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)).external_ref: -> nic3-eref from VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)),
Linking(frozenset({CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),
Linking(frozenset({CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),
Linking(frozenset({VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5), CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS))})),
Linking(frozenset({VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1), CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))})),
Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1))})),
Linking(frozenset({CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),
AttributeUpdated(CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)).external_ref: -> nic3-eref from VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)),
Linking(frozenset({CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),
Linking(frozenset({CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),
Linking(frozenset({VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5), CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1))})),
Linking(frozenset({CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1)), VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)})),
Addition(NIC(name=nic-2, parent=CloudHandover(name=EqFA5))),
Linking(frozenset({NIC(name=nic-2, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)}))]
INFO:root:🙌 Final DB states
INFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),
CloudServiceProvider(name=AWS),
CloudServiceProvider(name=AZURE)],
<class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),
CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),
CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),
CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],
<class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),
CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),
CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),
CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),
CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],
<class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),
NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),
NIC(name=nic-1, parent=CloudHandover(name=INX6)),
NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),
NIC(name=nic-3a, parent=CloudHandover(name=LON1)),
NIC(name=nic-2, parent=CloudHandover(name=EqFA5))]}
INFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),
VlanServiceProvider(name=AWS),
VlanServiceProvider(name=IBM)],
<class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),
VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),
VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03),
VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),
VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)]}
INFO:root:🎉🥳 Sync with 0 operations made. Synchronization has converged
INFO:root:Operations: []
Process finished with exit code 0
```
# Celery Demo
## Start the compose setup
``` shell
❯ docker-compose up -d --build
Creating network "synchronizer-framework_default" with the default driver
Building worker
Step 1/11 : FROM python:3.8-slim
---> b281745b6df9
Step 2/11 : RUN apt-get update ; apt-get install -y sqlite3 vim ; apt-get clean
---> Using cache
---> 25b681e3e49c
Step 3/11 : WORKDIR /app/synchronizer-framework
---> Using cache
---> 20c652ac628d
Step 4/11 : COPY ./requirements.txt .
---> Using cache
---> 9a1f9aa5c9a4
Step 5/11 : RUN pip install -r requirements.txt
---> Using cache
---> ae9c400626e5
Step 6/11 : COPY src/ ./src
---> Using cache
---> be73849dfe6f
Step 7/11 : COPY tests ./tests
---> Using cache
---> 3e59bd21b326
Step 8/11 : COPY conftest.py .
---> Using cache
---> 3c06b0dcc7f9
Step 9/11 : ENV C_FORCE_ROOT=true
---> Using cache
---> 2a187de06fcd
Step 10/11 : ENV PYTHONPATH="src/"
---> Using cache
---> 027396f1f6f9
Step 11/11 : CMD [ "python", "-m", "demo.main"]
---> Using cache
---> b8b1c92054b7
Successfully built b8b1c92054b7
Successfully tagged synchronizer-framework_worker:latest
Building beat
Step 1/11 : FROM python:3.8-slim
---> b281745b6df9
Step 2/11 : RUN apt-get update ; apt-get install -y sqlite3 vim ; apt-get clean
---> Using cache
---> 25b681e3e49c
Step 3/11 : WORKDIR /app/synchronizer-framework
---> Using cache
---> 20c652ac628d
Step 4/11 : COPY ./requirements.txt .
---> Using cache
---> 9a1f9aa5c9a4
Step 5/11 : RUN pip install -r requirements.txt
---> Using cache
---> ae9c400626e5
Step 6/11 : COPY src/ ./src
---> Using cache
---> be73849dfe6f
Step 7/11 : COPY tests ./tests
---> Using cache
---> 3e59bd21b326
Step 8/11 : COPY conftest.py .
---> Using cache
---> 3c06b0dcc7f9
Step 9/11 : ENV C_FORCE_ROOT=true
---> Using cache
---> 2a187de06fcd
Step 10/11 : ENV PYTHONPATH="src/"
---> Using cache
---> 027396f1f6f9
Step 11/11 : CMD [ "python", "-m", "demo.main"]
---> Using cache
---> b8b1c92054b7
Successfully built b8b1c92054b7
Successfully tagged synchronizer-framework_beat:latest
Creating synchronizer-framework_worker_1 ... done
Creating synchronizer-framework_beat_1 ... done
Creating synchronizer-framework_redis_1 ... done
Creating synchronizer-framework_rabbitmq_1 ... done
```
## Show worker logs
```shell
worker_1 | [2021-04-21 18:16:06,166: INFO/MainProcess] Received task: sync.tasks.sync_all[281f9bc5-6ac8-4355-864e-692995abca5c]
worker_1 | [2021-04-21 18:16:06,341: INFO/ForkPoolWorker-2] Task sync.tasks.sync_all[281f9bc5-6ac8-4355-864e-692995abca5c] succeeded in 0.17173888799879933s: [Linking(frozenset({VlanServiceConnection(id=1, name=nic-1, region=eu-central-1, pop=INX6, externalRef=None, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False)), CloudRegion(id=2, name=eu-central-1, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False))})), Linking(frozenset({CloudRegion(id=2, name=eu-central-1, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False)), VlanServiceConnection(id=2, name=nic-2, region=eu-central-1, pop=EqFA5, externalRef=None, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False))})), Linking(frozenset({VlanServiceConnection(id=3, name=nic-3, region=eu-west-2, pop=EqFA5, externalRef=nic3-eref, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False)), CloudRegion(id=4, name=eu-west-2, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False))})), Linking(frozenset({CloudRegion(id=1, name=EU-Frankfurt, external_ref=None, parent=CloudServiceProvider(id=1, name=IBM,..., Li..., ...]
worker_1 | [2021-04-21 18:16:16,165: INFO/MainProcess] Received task: sync.tasks.sync_all[64b780e0-b5e3-4771-913d-de3f54b2ccec]
worker_1 | [2021-04-21 18:16:16,300: INFO/ForkPoolWorker-2] Task sync.tasks.sync_all[64b780e0-b5e3-4771-913d-de3f54b2ccec] succeeded in 0.13250631099799648s: []
```
## Issue a new object and synchronize it immediately
```python
from demo.daphne import VlanServiceProvider
from demo.fixtures import daphne_sqa_ds, desire_sqa_ds, make_mappings
from demo.celery import tasks
if __name__ == "__main__":
ds = daphne_sqa_ds()
vlsp = VlanServiceProvider(name="test")
ds.create(vlsp)
tasks.sync_one.delay(
daphne_sqa_ds, desire_sqa_ds, make_mappings(), vlsp.__class__, vlsp.id
)
```
```shell
docker-compose exec worker python src/demo/create_and_sync.py
```
## Worker log
```shell
worker_1 | [2021-04-21 18:22:12,938: INFO/MainProcess] Received task: sync.tasks.sync_one[11dfd53a-1198-47e8-b624-25218b463660]
worker_1 | [2021-04-21 18:22:12,957: INFO/ForkPoolWorker-2] ✨ Adding CloudServiceProvider(id=4, name=test, upgrade_allowed=None)
worker_1 | [2021-04-21 18:22:12,959: INFO/ForkPoolWorker-2] Task sync.tasks.sync_one[11dfd53a-1198-47e8-b624-25218b463660] succeeded in 0.02017242200236069s: [Addition(CloudServiceProvider(id=4, name=test, upgrade_allowed=None)), Linking(frozenset({CloudServiceProvider(id=4, name=test, upgrade_allowed=None), VlanServiceProvider(id=4, name=test, canUpgrade=None)}))]
```
## Show link and synchronized entry
```shell
❯ docker-compose exec redis redis-cli smembers VlanServiceProvider:4:CloudServiceProvider
1) "CloudServiceProvider:4"
❯ docker-compose exec worker sqlite3 /tmp/desire.sqlite "SELECT * FROM cloud_service_providers where id = 4;"
4|test|
```
Raw data
{
"_id": null,
"home_page": "https://gitlab.com/sebastianspies9/synchronizer-framework",
"name": "synchronizer-framework",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "",
"author": "Sebastian Spies",
"author_email": "s+gitlab@sloc.de",
"download_url": "https://files.pythonhosted.org/packages/46/18/35c343c34cd6046c0a23ed8eb49dfabe6a532479b3c4add7cd4e308558c5/synchronizer_framework-0.0.22.linux-x86_64.tar.gz",
"platform": null,
"description": "# Synchronizer Framework\n\nThis framework provides a data synchronization pattern based on a declarative bi-directional definition. The definition\nis a mapping of entity types with identity keys and non-key attributes which support CRUD operations and a\nsynchronization strategy. The data sources may be arbitrarily implemented as long as they inherit from `sync.base.Base`.\nCommon data sources like ORMs (django-orm, sqlalchemy) or REST APIs may be used as long as they support CRUD.\n\nData sources may mixin `sync.base.Timestamped` to apply a last-write wins synchronization strategy based on\nthe `updated_at` field.\n\nThe project targets audiences which have chosen data synchronization as a primary integration method for microservices.\nInspiration has been taken from projects Kubernetes and Chef.\n\nDisclaimer: The demo and README mentions systems like Desire and Daphne. Those are no open source projects, but some\ntarget audience will understand them. The definition is supposed to cover all use-cases of the Desire architecture.\n\n### Features\n\n* Adding missing objects\n* Defer adding objects, which miss required foreign keys, because they do not exist or which are not synchronized yet\n* Linking synchronized objects with each other to track them subsequently\n* Many objects contributing to a single synchronized object on the remote side\n* Updating non-key attributes\n* Last write wins (entity needs to support `sync.base.Timestamped`)\n* Deep-join attribute sourcing\n* Abstract data access to make the pattern work with any CRUD data source (like Microsoft Dynamics, ldap, REST APIs,\n netconf, gNMI, CSP SDKs)\n * i.e. AWS SDK Create as `AllocateHostedConnection`, Read as `DescribeConnections`, ...\n * Data sources need to implement crud operations as defined in `sync.datasource.DataSource`\n* pre-load/include for deep joins\n* Optional deletion based on orphaned entities\n* Possibility to expedite single object synchronization based on an event stream\n * Celery support\n* SQLAlchemy backend support\n* Redis linking backend support\n* dockerized\n\nMore details about the pattern can be found in the `tests` path.\n\n### Missing/TODO\n\n* Deletion based on intent (with cryptographic verification) \n\n### Known Limitations\n\n1. Cannot synchronize three-way at once.\n\n *Example:*\n\n Assume entities `PA` required parent of `A`, `BA` required parent of `B`, `CA` required parent of `C`. Mappings are\n defined as `M1: A -> B` and `M2: B <- C`. The pattern prevents each `M1` and `M2` to progress independently. For `M1`\n , information about `PC` is missing and for `M2` information about `PA` is missing. Further, assume `x(e)` to denote\n an entity `e`\n transformed to the namespace of `x`.\n\n *Solution:*\n * Introduce an intermediary entity `I` which accepts optional parents `I(PA)` and `I(PC)`\n * remove `M1` and `M2`\n * create `M3: A -> I`, `M4: I <- C`\n * create `M5: I -> B`\n\n The sequence of `M3, M4, M5` leads to a synchronized `B` containing both parents `B(PA)` and `B(PC)`. The order of\n events always keeps consistent states. For example, `M5` may never progress before `M4` even if it is executed\n before.\n\n The solution is *not* limited to uni-directional mappings, but the author chose demonstrating uni-directional\n mappings for easier understanding of how data flows.\n\n1. The orphan deletion may be counter-intuitive and will be changed in a future version. Refrain from using it, if you\n do not understand the consequences.\n\n## Define Entities\n\nEntities below resemble the desire and daphne entity hierarchy and deviate to some extent. The intention is to\ndemonstrate the synchronization pattern instead of resembling existing models accurately. The details which are left out\nof consideration seem to be straight-forward to implement.\n\n### Base\n\n\n```python\nclass DemoBase(InMemoryLinks, Hierarchical, InMemoryTimestamp):\n ...\n```\n\n### Daphne\n\n```python\nclass VlanServiceProvider(DemoBase):\n def __init__(self, name=None, children=set()):\n super().__init__()\n self.name = name\n self.children = children\n\n\nclass VlanServiceConnection(DemoBase):\n def __init__(self, name, region, pop):\n super().__init__()\n self.name = name\n self.region = region\n self.pop = pop\n```\n\n### Desire\n\n```python\nclass CloudServiceProvider(DemoBase):\n def __init__(self, name=None, children=None):\n super().__init__()\n self.name = name\n self.children = children or set()\n\n\nclass CloudRegion(DemoBase):\n def __init__(self, name=None, children=None):\n super().__init__()\n self.name = name\n self.children = children or set()\n\n\nclass CloudHandover(DemoBase):\n def __init__(self, name=None, children=None):\n super().__init__()\n self.name = name\n self.children = children or set()\n\n\nclass NIC(DemoBase):\n def __init__(self, name=None):\n super().__init__()\n self.name = name\n```\n\n## Define Mapping\n\n```python\nmappings = [\n Mapping(\n entity_types=(VlanServiceConnection, CloudRegion),\n modes={Mode.LEFT_TO_RIGHT},\n keys={\n AttributeMap(\"region\", \"name\"),\n AttributeMap(\n \"parent\", \"parent\", VlanServiceProvider, CloudServiceProvider\n ),\n },\n attributes={AttributeMap(\"externalRef\", \"external_ref\")},\n ),\n Mapping(\n entity_types=(VlanServiceProvider, CloudServiceProvider),\n modes={Mode.LEFT_TO_RIGHT, Mode.RIGHT_TO_LEFT},\n keys={AttributeMap(\"name\", \"name\")},\n attributes={AttributeMap(\"canUpgrade\", \"upgrade_allowed\")},\n ),\n Mapping(\n entity_types=(VlanServiceConnection, CloudHandover),\n modes={Mode.LEFT_TO_RIGHT},\n keys={\n AttributeMap(\"pop\", \"name\"),\n AttributeMap(\"__self__\", \"parent\", VlanServiceConnection, CloudRegion),\n },\n attributes={AttributeMap(\"externalRef\", \"external_ref\")},\n ),\n Mapping(\n entity_types=(VlanServiceConnection, NIC),\n modes={Mode.RIGHT_TO_LEFT},\n keys={\n AttributeMap(\"name\", \"name\"),\n AttributeMap(\"region\", (\"parent\", \"parent\", \"name\")),\n AttributeMap(\"pop\", (\"parent\", \"name\")),\n AttributeMap(\n \"parent\",\n (\"parent\", \"parent\", \"parent\"),\n VlanServiceProvider,\n CloudServiceProvider,\n ),\n },\n attributes={AttributeMap(\"externalRef\", \"external_ref\")},\n ),\n Mapping(\n entity_types=(VlanServiceConnection, NIC),\n modes={Mode.LEFT_TO_RIGHT},\n keys={\n AttributeMap(\"name\", \"name\"),\n AttributeMap(\n \"__self__\", \"parent\", VlanServiceConnection, CloudHandover\n ),\n },\n attributes={AttributeMap(\"externalRef\", \"external_ref\")},\n ),\n]\n\n```\n\n## Define some objects\n\n```python\ndaphne_db = Db.from_collection(\n {\n VlanServiceProvider(\n name=\"AWS\",\n children={\n VlanServiceConnection(\"nic-1\", \"eu-central-1\", \"INX6\"),\n VlanServiceConnection(\"nic-2\", \"eu-central-1\", \"EqFA5\"),\n VlanServiceConnection(\"nic-3\", \"eu-west-2\", \"EqFA5\"),\n },\n ),\n VlanServiceProvider(name=\"AZURE\", children=set()),\n }\n)\n\ndesire_db = Db.from_collection(\n {\n CloudServiceProvider(\n name=\"AWS\",\n children={\n CloudRegion(\n name=\"eu-central-1\",\n children={\n CloudHandover(name=\"INX6\", children={NIC(\"nic-1\")}),\n CloudHandover(name=\"EqFA5\", children={NIC(\"nic-2a\")}),\n },\n ),\n CloudRegion(\n name=\"eu-west-1\",\n children={\n CloudHandover(name=\"LON1\", children={NIC(name=\"nic-3a\")})\n },\n ),\n CloudRegion(\n name=\"eu-west-2\",\n children={\n CloudHandover(\n name=\"EqFA5\",\n children={NIC(name=\"nic-3\", external_ref=\"nic3-eref\")},\n )\n },\n ),\n },\n ),\n CloudServiceProvider(\n name=\"IBM\",\n children={\n CloudRegion(\n name=\"EU-Frankfurt\",\n children={CloudHandover(name=\"fra03\", children={NIC(\"nic-4\")})},\n )\n },\n ),\n }\n)\n\n```\n\n## Plug it together\n\n```python\nsync(daphne_db, desire_db, mappings)\n```\n\n## Demo Output\n\n```shell\n$ python src/demo/main.py\n/home/sspies/git/sync-poc/venv/bin/python3 /home/sspies/git/sync-poc/src/demo/main.py\nINFO:root:Initial DB states\nINFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),\n CloudServiceProvider(name=AWS)],\n <class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),\n CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),\n CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),\n CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],\n <class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),\n CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),\n CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),\n CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),\n CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],\n <class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),\n NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),\n NIC(name=nic-1, parent=CloudHandover(name=INX6)),\n NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),\n NIC(name=nic-3a, parent=CloudHandover(name=LON1))]}\nINFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),\n VlanServiceProvider(name=AWS)],\n <class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),\n VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),\n VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)]}\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).parent. VlanServiceProvider(name=AWS) not in CloudServiceProvider\nINFO:root:\u2728 Adding CloudServiceProvider(name=AZURE)\nINFO:root:\u2728 Adding VlanServiceProvider(name=IBM)\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).__self__. VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6) not in CloudRegion\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).__self__. VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5) not in CloudRegion\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudRegion\nINFO:root:\u2728 Adding VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)\nINFO:root:\u2728 Adding VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)\nINFO:root:\u2728 Adding VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6).__self__. VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6) not in CloudHandover\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).__self__. VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5) not in CloudHandover\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudHandover\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03).__self__. VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03) not in CloudHandover\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5).__self__. VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5) not in CloudHandover\nWARNING:root:Cannot find linked entity of VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1).__self__. VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1) not in CloudHandover\nINFO:root:27 operation(s) made\nINFO:root:Operations: [SkipMissingLink(VlanServiceConnection -> CloudRegion),\n SkipMissingLink(VlanServiceConnection -> CloudRegion),\n SkipMissingLink(VlanServiceConnection -> CloudRegion),\n Addition(CloudServiceProvider(name=AZURE)),\n Linking(frozenset({VlanServiceProvider(name=AZURE), CloudServiceProvider(name=AZURE)})),\n Linking(frozenset({VlanServiceProvider(name=AWS), CloudServiceProvider(name=AWS)})),\n AttributeUpdated(CloudServiceProvider(name=AWS).upgrade_allowed: True -> False from VlanServiceProvider(name=AWS)),\n Addition(VlanServiceProvider(name=IBM)),\n Linking(frozenset({VlanServiceProvider(name=IBM), CloudServiceProvider(name=IBM)})),\n SkipMissingLink(VlanServiceConnection -> CloudHandover),\n SkipMissingLink(VlanServiceConnection -> CloudHandover),\n SkipMissingLink(VlanServiceConnection -> CloudHandover),\n Addition(VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)),\n Linking(frozenset({NIC(name=nic-4, parent=CloudHandover(name=fra03)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),\n Linking(frozenset({NIC(name=nic-3, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),\n AttributeUpdated(VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5).externalRef: -> nic3-eref from NIC(name=nic-3, parent=CloudHandover(name=EqFA5))),\n Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), NIC(name=nic-1, parent=CloudHandover(name=INX6))})),\n Addition(VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)),\n Linking(frozenset({NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),\n Addition(VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)),\n Linking(frozenset({NIC(name=nic-3a, parent=CloudHandover(name=LON1)), VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)})),\n SkipMissingLink(VlanServiceConnection -> NIC),\n SkipMissingLink(VlanServiceConnection -> NIC),\n SkipMissingLink(VlanServiceConnection -> NIC),\n SkipMissingLink(VlanServiceConnection -> NIC),\n SkipMissingLink(VlanServiceConnection -> NIC),\n SkipMissingLink(VlanServiceConnection -> NIC)]\nINFO:root:DB states after first and before second run\nINFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),\n CloudServiceProvider(name=AWS),\n CloudServiceProvider(name=AZURE)],\n <class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),\n CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),\n CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),\n CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],\n <class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),\n CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),\n CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),\n CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),\n CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],\n <class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),\n NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),\n NIC(name=nic-1, parent=CloudHandover(name=INX6)),\n NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),\n NIC(name=nic-3a, parent=CloudHandover(name=LON1))]}\nINFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),\n VlanServiceProvider(name=AWS),\n VlanServiceProvider(name=IBM)],\n <class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),\n VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),\n VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),\n VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03),\n VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),\n VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)]}\nINFO:root:\u2728 Adding NIC(name=nic-2, parent=CloudHandover(name=EqFA5))\nINFO:root:Sync with 16 operation(s) made\nINFO:root:Operations: [Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS))})),\n Linking(frozenset({CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),\n AttributeUpdated(CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)).external_ref: -> nic3-eref from VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)),\n Linking(frozenset({CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),\n Linking(frozenset({CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),\n Linking(frozenset({VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5), CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS))})),\n Linking(frozenset({VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1), CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))})),\n Linking(frozenset({VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6), CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1))})),\n Linking(frozenset({CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)), VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)})),\n AttributeUpdated(CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)).external_ref: -> nic3-eref from VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5)),\n Linking(frozenset({CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)})),\n Linking(frozenset({CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)), VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03)})),\n Linking(frozenset({VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5), CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1))})),\n Linking(frozenset({CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1)), VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)})),\n Addition(NIC(name=nic-2, parent=CloudHandover(name=EqFA5))),\n Linking(frozenset({NIC(name=nic-2, parent=CloudHandover(name=EqFA5)), VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5)}))]\nINFO:root:\ud83d\ude4c Final DB states\nINFO:root:{<class 'demo.desire.CloudServiceProvider'>: [CloudServiceProvider(name=IBM),\n CloudServiceProvider(name=AWS),\n CloudServiceProvider(name=AZURE)],\n <class 'demo.desire.CloudRegion'>: [CloudRegion(name=EU-Frankfurt, parent=CloudServiceProvider(name=IBM)),\n CloudRegion(name=eu-west-2, parent=CloudServiceProvider(name=AWS)),\n CloudRegion(name=eu-central-1, parent=CloudServiceProvider(name=AWS)),\n CloudRegion(name=eu-west-1, parent=CloudServiceProvider(name=AWS))],\n <class 'demo.desire.CloudHandover'>: [CloudHandover(name=fra03, parent=CloudRegion(name=EU-Frankfurt)),\n CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-west-2)),\n CloudHandover(name=INX6, parent=CloudRegion(name=eu-central-1)),\n CloudHandover(name=EqFA5, parent=CloudRegion(name=eu-central-1)),\n CloudHandover(name=LON1, parent=CloudRegion(name=eu-west-1))],\n <class 'demo.desire.NIC'>: [NIC(name=nic-4, parent=CloudHandover(name=fra03)),\n NIC(name=nic-3, parent=CloudHandover(name=EqFA5)),\n NIC(name=nic-1, parent=CloudHandover(name=INX6)),\n NIC(name=nic-2a, parent=CloudHandover(name=EqFA5)),\n NIC(name=nic-3a, parent=CloudHandover(name=LON1)),\n NIC(name=nic-2, parent=CloudHandover(name=EqFA5))]}\nINFO:root:{<class 'demo.daphne.VlanServiceProvider'>: [VlanServiceProvider(name=AZURE),\n VlanServiceProvider(name=AWS),\n VlanServiceProvider(name=IBM)],\n <class 'demo.daphne.VlanServiceConnection'>: [VlanServiceConnection(name=nic-1, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=INX6),\n VlanServiceConnection(name=nic-3, parent=VlanServiceProvider(name=AWS), region=eu-west-2, pop=EqFA5),\n VlanServiceConnection(name=nic-2, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),\n VlanServiceConnection(name=nic-4, parent=VlanServiceProvider(name=IBM), region=EU-Frankfurt, pop=fra03),\n VlanServiceConnection(name=nic-2a, parent=VlanServiceProvider(name=AWS), region=eu-central-1, pop=EqFA5),\n VlanServiceConnection(name=nic-3a, parent=VlanServiceProvider(name=AWS), region=eu-west-1, pop=LON1)]}\nINFO:root:\ud83c\udf89\ud83e\udd73 Sync with 0 operations made. Synchronization has converged\nINFO:root:Operations: []\n\nProcess finished with exit code 0\n\n```\n\n# Celery Demo\n\n## Start the compose setup\n\n``` shell\n\u276f docker-compose up -d --build\nCreating network \"synchronizer-framework_default\" with the default driver\nBuilding worker\nStep 1/11 : FROM python:3.8-slim\n ---> b281745b6df9\nStep 2/11 : RUN apt-get update ; apt-get install -y sqlite3 vim ; apt-get clean\n ---> Using cache\n ---> 25b681e3e49c\nStep 3/11 : WORKDIR /app/synchronizer-framework\n ---> Using cache\n ---> 20c652ac628d\nStep 4/11 : COPY ./requirements.txt .\n ---> Using cache\n ---> 9a1f9aa5c9a4\nStep 5/11 : RUN pip install -r requirements.txt\n ---> Using cache\n ---> ae9c400626e5\nStep 6/11 : COPY src/ ./src\n ---> Using cache\n ---> be73849dfe6f\nStep 7/11 : COPY tests ./tests\n ---> Using cache\n ---> 3e59bd21b326\nStep 8/11 : COPY conftest.py .\n ---> Using cache\n ---> 3c06b0dcc7f9\nStep 9/11 : ENV C_FORCE_ROOT=true\n ---> Using cache\n ---> 2a187de06fcd\nStep 10/11 : ENV PYTHONPATH=\"src/\"\n ---> Using cache\n ---> 027396f1f6f9\nStep 11/11 : CMD [ \"python\", \"-m\", \"demo.main\"]\n ---> Using cache\n ---> b8b1c92054b7\nSuccessfully built b8b1c92054b7\nSuccessfully tagged synchronizer-framework_worker:latest\nBuilding beat\nStep 1/11 : FROM python:3.8-slim\n ---> b281745b6df9\nStep 2/11 : RUN apt-get update ; apt-get install -y sqlite3 vim ; apt-get clean\n ---> Using cache\n ---> 25b681e3e49c\nStep 3/11 : WORKDIR /app/synchronizer-framework\n ---> Using cache\n ---> 20c652ac628d\nStep 4/11 : COPY ./requirements.txt .\n ---> Using cache\n ---> 9a1f9aa5c9a4\nStep 5/11 : RUN pip install -r requirements.txt\n ---> Using cache\n ---> ae9c400626e5\nStep 6/11 : COPY src/ ./src\n ---> Using cache\n ---> be73849dfe6f\nStep 7/11 : COPY tests ./tests\n ---> Using cache\n ---> 3e59bd21b326\nStep 8/11 : COPY conftest.py .\n ---> Using cache\n ---> 3c06b0dcc7f9\nStep 9/11 : ENV C_FORCE_ROOT=true\n ---> Using cache\n ---> 2a187de06fcd\nStep 10/11 : ENV PYTHONPATH=\"src/\"\n ---> Using cache\n ---> 027396f1f6f9\nStep 11/11 : CMD [ \"python\", \"-m\", \"demo.main\"]\n ---> Using cache\n ---> b8b1c92054b7\nSuccessfully built b8b1c92054b7\nSuccessfully tagged synchronizer-framework_beat:latest\nCreating synchronizer-framework_worker_1 ... done\nCreating synchronizer-framework_beat_1 ... done\nCreating synchronizer-framework_redis_1 ... done\nCreating synchronizer-framework_rabbitmq_1 ... done\n```\n\n## Show worker logs\n```shell\nworker_1 | [2021-04-21 18:16:06,166: INFO/MainProcess] Received task: sync.tasks.sync_all[281f9bc5-6ac8-4355-864e-692995abca5c] \nworker_1 | [2021-04-21 18:16:06,341: INFO/ForkPoolWorker-2] Task sync.tasks.sync_all[281f9bc5-6ac8-4355-864e-692995abca5c] succeeded in 0.17173888799879933s: [Linking(frozenset({VlanServiceConnection(id=1, name=nic-1, region=eu-central-1, pop=INX6, externalRef=None, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False)), CloudRegion(id=2, name=eu-central-1, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False))})), Linking(frozenset({CloudRegion(id=2, name=eu-central-1, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False)), VlanServiceConnection(id=2, name=nic-2, region=eu-central-1, pop=EqFA5, externalRef=None, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False))})), Linking(frozenset({VlanServiceConnection(id=3, name=nic-3, region=eu-west-2, pop=EqFA5, externalRef=nic3-eref, parent=VlanServiceProvider(id=2, name=AWS, canUpgrade=False)), CloudRegion(id=4, name=eu-west-2, external_ref=None, parent=CloudServiceProvider(id=2, name=AWS, upgrade_allowed=False))})), Linking(frozenset({CloudRegion(id=1, name=EU-Frankfurt, external_ref=None, parent=CloudServiceProvider(id=1, name=IBM,..., Li..., ...]\nworker_1 | [2021-04-21 18:16:16,165: INFO/MainProcess] Received task: sync.tasks.sync_all[64b780e0-b5e3-4771-913d-de3f54b2ccec] \nworker_1 | [2021-04-21 18:16:16,300: INFO/ForkPoolWorker-2] Task sync.tasks.sync_all[64b780e0-b5e3-4771-913d-de3f54b2ccec] succeeded in 0.13250631099799648s: []\n```\n\n## Issue a new object and synchronize it immediately\n\n```python\nfrom demo.daphne import VlanServiceProvider\nfrom demo.fixtures import daphne_sqa_ds, desire_sqa_ds, make_mappings\nfrom demo.celery import tasks\n\nif __name__ == \"__main__\":\n ds = daphne_sqa_ds()\n vlsp = VlanServiceProvider(name=\"test\")\n ds.create(vlsp)\n\n tasks.sync_one.delay(\n daphne_sqa_ds, desire_sqa_ds, make_mappings(), vlsp.__class__, vlsp.id\n )\n```\n\n\n```shell\n docker-compose exec worker python src/demo/create_and_sync.py\n``` \n\n## Worker log\n\n```shell\nworker_1 | [2021-04-21 18:22:12,938: INFO/MainProcess] Received task: sync.tasks.sync_one[11dfd53a-1198-47e8-b624-25218b463660] \nworker_1 | [2021-04-21 18:22:12,957: INFO/ForkPoolWorker-2] \u2728 Adding CloudServiceProvider(id=4, name=test, upgrade_allowed=None)\nworker_1 | [2021-04-21 18:22:12,959: INFO/ForkPoolWorker-2] Task sync.tasks.sync_one[11dfd53a-1198-47e8-b624-25218b463660] succeeded in 0.02017242200236069s: [Addition(CloudServiceProvider(id=4, name=test, upgrade_allowed=None)), Linking(frozenset({CloudServiceProvider(id=4, name=test, upgrade_allowed=None), VlanServiceProvider(id=4, name=test, canUpgrade=None)}))]\n```\n\n## Show link and synchronized entry\n```shell\n\u276f docker-compose exec redis redis-cli smembers VlanServiceProvider:4:CloudServiceProvider\n1) \"CloudServiceProvider:4\"\n\u276f docker-compose exec worker sqlite3 /tmp/desire.sqlite \"SELECT * FROM cloud_service_providers where id = 4;\"\n4|test|\n``` \n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Bi-directional synchronization of CRUD data sources",
"version": "0.0.22",
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"md5": "9bc9cae7fe7583a6e72d8673ae49b605",
"sha256": "ebb4b514e4923d90811dca8ab2b9bae67fd6bf0a8ef83d598a0bfe9613f82111"
},
"downloads": -1,
"filename": "synchronizer_framework-0.0.22.linux-x86_64.tar.gz",
"has_sig": false,
"md5_digest": "9bc9cae7fe7583a6e72d8673ae49b605",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 40669,
"upload_time": "2022-12-23T09:50:49",
"upload_time_iso_8601": "2022-12-23T09:50:49.449837Z",
"url": "https://files.pythonhosted.org/packages/46/18/35c343c34cd6046c0a23ed8eb49dfabe6a532479b3c4add7cd4e308558c5/synchronizer_framework-0.0.22.linux-x86_64.tar.gz",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"md5": "9a35a49000dd964e9afbfddd832756e0",
"sha256": "c4d76fa4dab37c684a27548051a494f5ba0c1ec5af5f26d808e774e298600bbc"
},
"downloads": -1,
"filename": "synchronizer_framework-0.0.22-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9a35a49000dd964e9afbfddd832756e0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 23892,
"upload_time": "2022-12-23T09:50:47",
"upload_time_iso_8601": "2022-12-23T09:50:47.442627Z",
"url": "https://files.pythonhosted.org/packages/7b/5b/4271a3c6a4acd0fc8759e6612f06608601f30254d3297e22acaae01e85e3/synchronizer_framework-0.0.22-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-12-23 09:50:49",
"github": false,
"gitlab": true,
"bitbucket": false,
"gitlab_user": "sebastianspies9",
"gitlab_project": "synchronizer-framework",
"lcname": "synchronizer-framework"
}
JSON