==================
Connector Importer
==================
..
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! source digest: sha256:b7619b9395aed104a6a70844f2ada45200720c2107955fb3ae65628f0515f922
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png
:target: https://odoo-community.org/page/development-status
:alt: Beta
.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
:alt: License: AGPL-3
.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fconnector--interfaces-lightgray.png?logo=github
:target: https://github.com/OCA/connector-interfaces/tree/18.0/connector_importer
:alt: OCA/connector-interfaces
.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
:target: https://translation.odoo-community.org/projects/connector-interfaces-18-0/connector-interfaces-18-0-connector_importer
:alt: Translate me on Weblate
.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
:target: https://runboat.odoo-community.org/builds?repo=OCA/connector-interfaces&target_branch=18.0
:alt: Try me on Runboat
|badge1| |badge2| |badge3| |badge4| |badge5|
This module allows to import / update records from files using the
connector framework and job queue.
To run an import you need at least:
- a backend, hosts the global configuration of the import.
- a recordset, hosts the configuration of the import for specific
models and source
- a source, provides the data to import
- an import type, describes which models you want to import and how to
import them
**Table of contents**
.. contents::
:local:
Configuration
=============
Import type
-----------
Import types are the main configuration of the import. They describe
which models you want to import and how to import them.
Exaple of configuration:
::
<record id="import_type_product_product_all_in_one" model="import.type">
<field name="name">Import Product - all in one</field>
<field name="key">product_product_all_in_one</field>
<field name="options">
- model: product.product
options:
importer:
odoo_unique_key: barcode
mapper:
name: product.product.mapper
- model: res.partner
options:
importer:
odoo_unique_key: name
override_existing: false
mapper:
name: importer.mapper.dynamic
source_key_prefix: supplier.
source_key_whitelist: supplier.name
default_keys:
supplier_rank: 1
- model: product.supplierinfo
options:
importer:
odoo_unique_key: name
mapper:
name: product.supplierinfo.mapper
source_key_prefix: supplier.
</field>
</record>
In this example we have 3 models to import one after the other using the
same source file:
- product.product
- res.partner
- product.supplierinfo
The import will run in the order of the configuration: first
product.product, then res.partner and finally product.supplierinfo. For
each model we have a configuration that describes how to import the
data. With the ``options`` key we can define the configuration of the
import for each component: ``importer``, ``mapper``, ``record_handler``,
``tracking_handler``.
The are 4 main components in the import configuration:
- importer
- mapper
- record_handler
- tracking_handler
Each of them is responsible for a specific part of the import.
The importer
------------
``importer`` is the main component that will import the data. It will
use the ``mapper`` to map the data from the source to the destination
model. If no ``name`` is defined the importer will use the default
importer for the model which is capable of importing any model. Most of
the time you don't need a specific importer.
As the importer is the main component of the import if you want to
customize it you'll have to declare it at an higher level, next to the
``options`` key:
::
- model: product.product
importer:
name: product.product.importer
options:
mapper:
name: product.product.mapper
The importer accepts the following options:
- ``odoo_unique_key``: the field that will be used to find the record
in Odoo. If the record is found it will be updated, otherwise it will
be created.
NOTE: the value in the column declared as ``odoo_unique_key`` will
be treated as xid only if the name of the column is ``ìd`` or if
it starts with ``xid::``.
- ``break_on_error``: if set to True the import will stop if an error
occurs. Default is False.
- ``override_existing``: if set to True the existing records will be
updated. Default is True.
- ``translation_key_sep``: the separator used to split the translation
key. Default is ``:``. See below for information about translation
keys.
- ``translation_use_regional_lang``: if set to True the importer will
use the regional language, eg: fr_CH vs fr.
- ``ctx``: a dictionary of values to inject in the context of the
import.
- ``write_only``: if set to True the importer will not create new
records, it will only update existing ones. Default is False.
The mapper
----------
The mapper is the component that will map the data from the source to
the destination model.
The most flexible mapper is the ``importer.mapper.dynamic`` that will
map the data based on the model introspection and some options that you
can define. The dynamic mapper accepts the following options:
- ``name``: the name of the mapper to use. If no name is defined the
default mapper for the model will be used.
- ``source_key_prefix``: a prefix to add to the source key. This is
useful when you want to map the same source key to different
destination fields.
- ``source_key_whitelist``: a list of source keys to import. If not
defined all the keys will be imported.
- ``source_key_blacklist``: a list of source keys to exclude from the
import.
- ``source_key_rename``: a dictionary of source keys to rename. The key
is the source key and the value is the new key.
- ``default_keys``: a dictionary of default values to set on the
destination record. The key is the field name and the value is the
default value.
- ``translation_keys``: a list of keys that will be used to translate
the data. See below for information about translation keys.
- ``required_keys``: a list of keys that are required. If one of the
keys is missing the record will be skipped. Please refer to the
documentation of the mapper to see advanced options.
Considering the example above:
::
- model: product.product
options:
mapper:
name: importer.mapper.dynamic
source_key_prefix: supplier.
source_key_whitelist: supplier.name
default_keys:
supplier_rank: 1
The mapper will:
- import only keys starting with ``supplier.`` ignoring the rest
- import only the key ``supplier.name``
- set the default value of ``supplier_rank`` to 1
The record_handler
------------------
The record handler is the component that will handle the record create
or update in Odoo. This component is responsible for:
- finding the record in Odoo
- creating the record if not found
- updating the record if found
- handling the translations
If no ``name`` is defined the importer will use the default record
handler for the model which is capable of handling any model. If you
want to customize the record handler you'll have to declare it at an
higher level, next to the ``options`` key:
::
- model: product.product
options:
record_handler:
name: product.product.record_handler
To find the record in Odoo the record handler will use the
``odoo_unique_key`` if defined in the importer otherwise it will
fallback to the matching domain. See below.
The record handler accepts the following options:
- ``name``: the name of the record handler to use. If no name is
defined the default record handler for the model will be used.
- ``match_domain``: a domain to match the record in Odoo. When no
odoo_unique_key is provided by the importer you must provide a
match_domain.
This key accepts a snippet returning a domain. The snippet will be
evaluated in the context of the import and will receive:
- ``orig_values``: the values from the source
- ``values``: values computed by the mapper for the record
- ``env``
- ``user``
- ``datetime``
- ``dateutil``
- ``time``
- ``ref_id``: a function to get a record ID from a reference
- ``ref``: a function to get a record from a reference
Example:
::
match_domain: |
[('name', '=', values.get('name'))]
- ``must_generate_xmlid``: if set to True the importer will generate an
XML ID for the record. Default is True if the unique key is an xmlid.
- ``skip_fields_unchanged``: if set to True the importer will skip the
fields that are unchanged. Default is False.
Translations
------------
The importer can translate the data using the translation keys. The
translation keys are a list of keys (column) that will be handled as
translatable. Whenever a key is found in the translation keys the
importer will look for a column with the same name suffixed by the
language code (eg: name:fr_CH). If the column is found the importer will
translate the data using the language code as context.
Known issues / Roadmap
======================
- with the import of standard Odoo CSV files, a concurrency error
occurs when updating the report_data of import_recordset table (from
the importer: self._do_report() -> self.recordset.set_report(...)).
The job is automatically retried a second time (without concurrency
errors). For small files it's not a big issue, but for files with a
huge amount of lines it takes time to process them two times.
- move generic functions from utils.mapper_utils to the connector
module
- unit tests for record handler and tracker
- add more test coverage for mapper utils and dynamic mapper
- consider making dynamic mapper the default one
- control how to generate xid (eg: from a specicic field with key
must_generate_xmlid_from_key)
- add manual control for backend_to_rel mappers
- refactor source to be a specific m2o to ease mgmt instead of a
generic relation
Bug Tracker
===========
Bugs are tracked on `GitHub Issues <https://github.com/OCA/connector-interfaces/issues>`_.
In case of trouble, please check there if your issue has already been reported.
If you spotted it first, help us to smash it by providing a detailed and welcomed
`feedback <https://github.com/OCA/connector-interfaces/issues/new?body=module:%20connector_importer%0Aversion:%2018.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.
Do not contact contributors directly about support or help with technical issues.
Credits
=======
Authors
-------
* Camptocamp
Contributors
------------
Simone Orsi (Camptocamp) for the original implementation.
Other contributors include:
- Guewen Baconnier (Camptocamp)
- Mykhailo Panarin (Camptocamp)
- Sébastien Alix (Camptocamp)
- Thien Vo (Trobz)
Other credits
-------------
The migration of this module from 16.0 to 18.0 was financially supported
by Camptocamp.
Maintainers
-----------
This module is maintained by the OCA.
.. image:: https://odoo-community.org/logo.png
:alt: Odoo Community Association
:target: https://odoo-community.org
OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.
.. |maintainer-simahawk| image:: https://github.com/simahawk.png?size=40px
:target: https://github.com/simahawk
:alt: simahawk
Current `maintainer <https://odoo-community.org/page/maintainer-role>`__:
|maintainer-simahawk|
This module is part of the `OCA/connector-interfaces <https://github.com/OCA/connector-interfaces/tree/18.0/connector_importer>`_ project on GitHub.
You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.
Raw data
{
"_id": null,
"home_page": "https://github.com/OCA/connector-interfaces",
"name": "odoo-addon-connector-importer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Camptocamp, Odoo Community Association (OCA)",
"author_email": "support@odoo-community.org",
"download_url": null,
"platform": null,
"description": "==================\nConnector Importer\n==================\n\n.. \n !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n !! This file is generated by oca-gen-addon-readme !!\n !! changes will be overwritten. !!\n !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n !! source digest: sha256:b7619b9395aed104a6a70844f2ada45200720c2107955fb3ae65628f0515f922\n !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n\n.. |badge1| image:: https://img.shields.io/badge/maturity-Beta-yellow.png\n :target: https://odoo-community.org/page/development-status\n :alt: Beta\n.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png\n :target: http://www.gnu.org/licenses/agpl-3.0-standalone.html\n :alt: License: AGPL-3\n.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fconnector--interfaces-lightgray.png?logo=github\n :target: https://github.com/OCA/connector-interfaces/tree/18.0/connector_importer\n :alt: OCA/connector-interfaces\n.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png\n :target: https://translation.odoo-community.org/projects/connector-interfaces-18-0/connector-interfaces-18-0-connector_importer\n :alt: Translate me on Weblate\n.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png\n :target: https://runboat.odoo-community.org/builds?repo=OCA/connector-interfaces&target_branch=18.0\n :alt: Try me on Runboat\n\n|badge1| |badge2| |badge3| |badge4| |badge5|\n\nThis module allows to import / update records from files using the\nconnector framework and job queue.\n\nTo run an import you need at least:\n\n- a backend, hosts the global configuration of the import.\n- a recordset, hosts the configuration of the import for specific\n models and source\n- a source, provides the data to import\n- an import type, describes which models you want to import and how to\n import them\n\n**Table of contents**\n\n.. contents::\n :local:\n\nConfiguration\n=============\n\nImport type\n-----------\n\nImport types are the main configuration of the import. They describe\nwhich models you want to import and how to import them.\n\nExaple of configuration:\n\n::\n\n <record id=\"import_type_product_product_all_in_one\" model=\"import.type\">\n <field name=\"name\">Import Product - all in one</field>\n <field name=\"key\">product_product_all_in_one</field>\n <field name=\"options\">\n\n - model: product.product\n options:\n importer:\n odoo_unique_key: barcode\n mapper:\n name: product.product.mapper\n\n - model: res.partner\n options:\n importer:\n odoo_unique_key: name\n override_existing: false\n mapper:\n name: importer.mapper.dynamic\n source_key_prefix: supplier.\n source_key_whitelist: supplier.name\n default_keys:\n supplier_rank: 1\n\n - model: product.supplierinfo\n options:\n importer:\n odoo_unique_key: name\n mapper:\n name: product.supplierinfo.mapper\n source_key_prefix: supplier.\n\n </field>\n\n </record>\n\nIn this example we have 3 models to import one after the other using the\nsame source file:\n\n- product.product\n- res.partner\n- product.supplierinfo\n\nThe import will run in the order of the configuration: first\nproduct.product, then res.partner and finally product.supplierinfo. For\neach model we have a configuration that describes how to import the\ndata. With the ``options`` key we can define the configuration of the\nimport for each component: ``importer``, ``mapper``, ``record_handler``,\n``tracking_handler``.\n\nThe are 4 main components in the import configuration:\n\n- importer\n- mapper\n- record_handler\n- tracking_handler\n\nEach of them is responsible for a specific part of the import.\n\nThe importer\n------------\n\n``importer`` is the main component that will import the data. It will\nuse the ``mapper`` to map the data from the source to the destination\nmodel. If no ``name`` is defined the importer will use the default\nimporter for the model which is capable of importing any model. Most of\nthe time you don't need a specific importer.\n\nAs the importer is the main component of the import if you want to\ncustomize it you'll have to declare it at an higher level, next to the\n``options`` key:\n\n::\n\n - model: product.product\n importer:\n name: product.product.importer\n options:\n mapper:\n name: product.product.mapper\n\nThe importer accepts the following options:\n\n- ``odoo_unique_key``: the field that will be used to find the record\n in Odoo. If the record is found it will be updated, otherwise it will\n be created.\n\n NOTE: the value in the column declared as ``odoo_unique_key`` will\n be treated as xid only if the name of the column is ``\u00ecd`` or if\n it starts with ``xid::``.\n\n- ``break_on_error``: if set to True the import will stop if an error\n occurs. Default is False.\n\n- ``override_existing``: if set to True the existing records will be\n updated. Default is True.\n\n- ``translation_key_sep``: the separator used to split the translation\n key. Default is ``:``. See below for information about translation\n keys.\n\n- ``translation_use_regional_lang``: if set to True the importer will\n use the regional language, eg: fr_CH vs fr.\n\n- ``ctx``: a dictionary of values to inject in the context of the\n import.\n\n- ``write_only``: if set to True the importer will not create new\n records, it will only update existing ones. Default is False.\n\nThe mapper\n----------\n\nThe mapper is the component that will map the data from the source to\nthe destination model.\n\nThe most flexible mapper is the ``importer.mapper.dynamic`` that will\nmap the data based on the model introspection and some options that you\ncan define. The dynamic mapper accepts the following options:\n\n- ``name``: the name of the mapper to use. If no name is defined the\n default mapper for the model will be used.\n- ``source_key_prefix``: a prefix to add to the source key. This is\n useful when you want to map the same source key to different\n destination fields.\n- ``source_key_whitelist``: a list of source keys to import. If not\n defined all the keys will be imported.\n- ``source_key_blacklist``: a list of source keys to exclude from the\n import.\n- ``source_key_rename``: a dictionary of source keys to rename. The key\n is the source key and the value is the new key.\n- ``default_keys``: a dictionary of default values to set on the\n destination record. The key is the field name and the value is the\n default value.\n- ``translation_keys``: a list of keys that will be used to translate\n the data. See below for information about translation keys.\n- ``required_keys``: a list of keys that are required. If one of the\n keys is missing the record will be skipped. Please refer to the\n documentation of the mapper to see advanced options.\n\nConsidering the example above:\n\n::\n\n - model: product.product\n options:\n mapper:\n name: importer.mapper.dynamic\n source_key_prefix: supplier.\n source_key_whitelist: supplier.name\n default_keys:\n supplier_rank: 1\n\nThe mapper will:\n\n- import only keys starting with ``supplier.`` ignoring the rest\n- import only the key ``supplier.name``\n- set the default value of ``supplier_rank`` to 1\n\nThe record_handler\n------------------\n\nThe record handler is the component that will handle the record create\nor update in Odoo. This component is responsible for:\n\n- finding the record in Odoo\n- creating the record if not found\n- updating the record if found\n- handling the translations\n\nIf no ``name`` is defined the importer will use the default record\nhandler for the model which is capable of handling any model. If you\nwant to customize the record handler you'll have to declare it at an\nhigher level, next to the ``options`` key:\n\n::\n\n - model: product.product\n options:\n record_handler:\n name: product.product.record_handler\n\nTo find the record in Odoo the record handler will use the\n``odoo_unique_key`` if defined in the importer otherwise it will\nfallback to the matching domain. See below.\n\nThe record handler accepts the following options:\n\n- ``name``: the name of the record handler to use. If no name is\n defined the default record handler for the model will be used.\n\n- ``match_domain``: a domain to match the record in Odoo. When no\n odoo_unique_key is provided by the importer you must provide a\n match_domain.\n\n This key accepts a snippet returning a domain. The snippet will be\n evaluated in the context of the import and will receive:\n\n - ``orig_values``: the values from the source\n\n - ``values``: values computed by the mapper for the record\n\n - ``env``\n\n - ``user``\n\n - ``datetime``\n\n - ``dateutil``\n\n - ``time``\n\n - ``ref_id``: a function to get a record ID from a reference\n\n - ``ref``: a function to get a record from a reference\n\n Example:\n\n ::\n\n match_domain: |\n [('name', '=', values.get('name'))]\n\n- ``must_generate_xmlid``: if set to True the importer will generate an\n XML ID for the record. Default is True if the unique key is an xmlid.\n\n- ``skip_fields_unchanged``: if set to True the importer will skip the\n fields that are unchanged. Default is False.\n\nTranslations\n------------\n\nThe importer can translate the data using the translation keys. The\ntranslation keys are a list of keys (column) that will be handled as\ntranslatable. Whenever a key is found in the translation keys the\nimporter will look for a column with the same name suffixed by the\nlanguage code (eg: name:fr_CH). If the column is found the importer will\ntranslate the data using the language code as context.\n\nKnown issues / Roadmap\n======================\n\n- with the import of standard Odoo CSV files, a concurrency error\n occurs when updating the report_data of import_recordset table (from\n the importer: self._do_report() -> self.recordset.set_report(...)).\n The job is automatically retried a second time (without concurrency\n errors). For small files it's not a big issue, but for files with a\n huge amount of lines it takes time to process them two times.\n- move generic functions from utils.mapper_utils to the connector\n module\n- unit tests for record handler and tracker\n- add more test coverage for mapper utils and dynamic mapper\n- consider making dynamic mapper the default one\n- control how to generate xid (eg: from a specicic field with key\n must_generate_xmlid_from_key)\n- add manual control for backend_to_rel mappers\n- refactor source to be a specific m2o to ease mgmt instead of a\n generic relation\n\nBug Tracker\n===========\n\nBugs are tracked on `GitHub Issues <https://github.com/OCA/connector-interfaces/issues>`_.\nIn case of trouble, please check there if your issue has already been reported.\nIf you spotted it first, help us to smash it by providing a detailed and welcomed\n`feedback <https://github.com/OCA/connector-interfaces/issues/new?body=module:%20connector_importer%0Aversion:%2018.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.\n\nDo not contact contributors directly about support or help with technical issues.\n\nCredits\n=======\n\nAuthors\n-------\n\n* Camptocamp\n\nContributors\n------------\n\nSimone Orsi (Camptocamp) for the original implementation.\n\nOther contributors include:\n\n- Guewen Baconnier (Camptocamp)\n- Mykhailo Panarin (Camptocamp)\n- S\u00e9bastien Alix (Camptocamp)\n- Thien Vo (Trobz)\n\nOther credits\n-------------\n\nThe migration of this module from 16.0 to 18.0 was financially supported\nby Camptocamp.\n\nMaintainers\n-----------\n\nThis module is maintained by the OCA.\n\n.. image:: https://odoo-community.org/logo.png\n :alt: Odoo Community Association\n :target: https://odoo-community.org\n\nOCA, or the Odoo Community Association, is a nonprofit organization whose\nmission is to support the collaborative development of Odoo features and\npromote its widespread use.\n\n.. |maintainer-simahawk| image:: https://github.com/simahawk.png?size=40px\n :target: https://github.com/simahawk\n :alt: simahawk\n\nCurrent `maintainer <https://odoo-community.org/page/maintainer-role>`__:\n\n|maintainer-simahawk| \n\nThis module is part of the `OCA/connector-interfaces <https://github.com/OCA/connector-interfaces/tree/18.0/connector_importer>`_ project on GitHub.\n\nYou are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.\n",
"bugtrack_url": null,
"license": "AGPL-3",
"summary": "This module takes care of import sessions.",
"version": "18.0.1.0.0.9",
"project_urls": {
"Homepage": "https://github.com/OCA/connector-interfaces"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "f9182767b794f76e7ea464eed92c1f8b3731d9859f31a6a7d044253cab66341e",
"md5": "25a29418af9e0aa047e7e64886ac7aab",
"sha256": "19a5fbefea312963c68ac6e6616b56282c043bdc6546ee267deed25bdc12d157"
},
"downloads": -1,
"filename": "odoo_addon_connector_importer-18.0.1.0.0.9-py3-none-any.whl",
"has_sig": false,
"md5_digest": "25a29418af9e0aa047e7e64886ac7aab",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 121999,
"upload_time": "2024-10-23T02:49:26",
"upload_time_iso_8601": "2024-10-23T02:49:26.620569Z",
"url": "https://files.pythonhosted.org/packages/f9/18/2767b794f76e7ea464eed92c1f8b3731d9859f31a6a7d044253cab66341e/odoo_addon_connector_importer-18.0.1.0.0.9-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-10-23 02:49:26",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "OCA",
"github_project": "connector-interfaces",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "odoo-addon-connector-importer"
}
JSON
