Event Sink ClickHouse
#####################
Purpose
*******
This project acts as a plugin to the `Edx Platform`_, listens for
configured `Open edX events`_, and sends them to a `ClickHouse`_ database for
analytics or other processing. This is being maintained as part of the
`Aspects`_ project.
OARS consumes the data sent to ClickHouse by this plugin as part of data
enrichment for reporting, or capturing data that otherwise does not fit in
xAPI.
Sinks
*****
Currently the only sink is in the CMS. It listens for the ``COURSE_PUBLISHED``
signal and serializes a subset of the published course blocks into one table
in ClickHouse.
Commands
********
In addition to being an event listener, this package provides commands for
exporting the same data in bulk. This allows bootstrapping a new data platform
or backfilling lost or missing data. Currently the only command is the Django
command for the ``COURSE_PUBLISHED`` data:
``python manage.py cms dump_courses_to_clickhouse``
This command allows bulk export of all courses, or various limiting factors.
Please see the command help for details:
``python manage.py cms dump_courses_to_clickhouse -h``
.. _Open edX events: https://github.com/openedx/openedx-events
.. _Edx Platform: https://github.com/openedx/edx-platform
.. _ClickHouse: https://clickhouse.com
.. _Aspects: https://docs.openedx.org/projects/openedx-aspects/en/latest/index.html
Getting Started
***************
Developing
==========
One Time Setup
--------------
.. code-block::
# Clone the repository
git clone git@github.com:openedx/openedx-event-sink-clickhouse.git
cd openedx-event-sink-clickhouse
# Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it
mkvirtualenv -p python3.8 openedx-event-sink-clickhouse
Every time you develop something in this repo
---------------------------------------------
.. code-block::
# Activate the virtualenv
workon openedx-event-sink-clickhouse
# Grab the latest code
git checkout main
git pull
# Install/update the dev requirements
make requirements
# Run the tests and quality checks (to verify the status before you make any changes)
make validate
# Make a new branch for your changes
git checkout -b <your_github_username>/<short_description>
# Using your favorite editor, edit the code to make your change.
vim ...
# Run your new tests
pytest ./path/to/new/tests
# Run all the tests and quality checks
make validate
# Commit all your changes
git commit ...
git push
# Open a PR and ask for review.
Deploying
=========
The Open edX Event Sink Clickhouse component is a django plugin which doesn't
need independent deployment. Therefore, its setup is reasonably
straightforward. First, it needs to be added to your service
requirements, and then it will be installed alongside requirements
of the service.
This plugin will be deployed by default in an OARS Tutor environment. For other
deployments install the library or add it to private requirements of your
virtual environment ( ``requirements/private.txt`` ).
#. Run ``pip install openedx-event-sink-clickhouse``.
#. Run migrations:
- ``python manage.py lms migrate``
- ``python manage.py cms migrate``
#. Restart LMS service and celery workers of edx-platform.
Configuration
===============
Currently all events will be listened to by default (there is only one). So
the only necessary configuration is a ClickHouse connection:
.. code-block::
EVENT_SINK_CLICKHOUSE_BACKEND_CONFIG = {
# URL to a running ClickHouse server's HTTP interface. ex: https://foo.openedx.org:8443/ or
# http://foo.openedx.org:8123/ . Note that we only support the ClickHouse HTTP interface
# to avoid pulling in more dependencies to the platform than necessary.
"url": "http://clickhouse:8123",
"username": "changeme",
"password": "changeme",
"database": "event_sink",
"timeout_secs": 3,
}
Getting Help
************
Documentation
=============
See `documentation on Read the Docs <https://openedx-event-sink-clickhouse.readthedocs.io/en/latest/>`_.
More Help
=========
If you're having trouble, we have discussion forums at
https://discuss.openedx.org where you can connect with others in the
community.
Our real-time conversations are on Slack. You can request a `Slack
invitation`_, then join our `community Slack workspace`_.
For anything non-trivial, the best path is to open an issue in this
repository with as many details about the issue you are facing as you
can provide.
https://github.com/openedx/openedx-event-sink-clickhouse/issues
For more information about these options, see the `Getting Help`_ page.
.. _Slack invitation: https://openedx.org/slack
.. _community Slack workspace: https://openedx.slack.com/
.. _Getting Help: https://openedx.org/getting-help
License
*******
The code in this repository is licensed under the AGPL 3.0 unless
otherwise noted.
Please see `LICENSE.txt <LICENSE.txt>`_ for details.
Contributing
************
Contributions are very welcome.
Please read `How To Contribute <https://openedx.org/r/how-to-contribute>`_ for details.
This project is currently accepting all types of contributions, bug fixes,
security fixes, maintenance work, or new features. However, please make sure
to have a discussion about your new feature idea with the maintainers prior to
beginning development to maximize the chances of your change being accepted.
You can start a conversation by creating a new issue on this repo summarizing
your idea.
The Open edX Code of Conduct
****************************
All community members are expected to follow the `Open edX Code of Conduct`_.
.. _Open edX Code of Conduct: https://openedx.org/code-of-conduct/
People
******
The assigned maintainers for this component and other project details may be
found in `Backstage`_. Backstage pulls this data from the ``catalog-info.yaml``
file in this repo.
.. _Backstage: https://open-edx-backstage.herokuapp.com/catalog/default/component/openedx-event-sink-clickhouse
Reporting Security Issues
*************************
Please do not report security issues in public. Please email security@openedx.org.
Raw data
{
"_id": null,
"home_page": "https://github.com/openedx/openedx_event_sink_clickhouse",
"name": "openedx-event-sink-clickhouse",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "",
"keywords": "Python edx",
"author": "edX",
"author_email": "oscm@edx.org",
"download_url": "https://files.pythonhosted.org/packages/82/9f/5fd5032bff3dfcc442a083c05a7544c0ef1db3fccc6499057cb8f6ee4f88/openedx_event_sink_clickhouse-1.1.1.tar.gz",
"platform": null,
"description": "Event Sink ClickHouse\n#####################\n\nPurpose\n*******\n\nThis project acts as a plugin to the `Edx Platform`_, listens for\nconfigured `Open edX events`_, and sends them to a `ClickHouse`_ database for\nanalytics or other processing. This is being maintained as part of the\n`Aspects`_ project.\n\nOARS consumes the data sent to ClickHouse by this plugin as part of data\nenrichment for reporting, or capturing data that otherwise does not fit in\nxAPI.\n\nSinks\n*****\n\nCurrently the only sink is in the CMS. It listens for the ``COURSE_PUBLISHED``\nsignal and serializes a subset of the published course blocks into one table\nin ClickHouse.\n\nCommands\n********\n\nIn addition to being an event listener, this package provides commands for\nexporting the same data in bulk. This allows bootstrapping a new data platform\nor backfilling lost or missing data. Currently the only command is the Django\ncommand for the ``COURSE_PUBLISHED`` data:\n\n``python manage.py cms dump_courses_to_clickhouse``\n\nThis command allows bulk export of all courses, or various limiting factors.\nPlease see the command help for details:\n\n``python manage.py cms dump_courses_to_clickhouse -h``\n\n\n.. _Open edX events: https://github.com/openedx/openedx-events\n.. _Edx Platform: https://github.com/openedx/edx-platform\n.. _ClickHouse: https://clickhouse.com\n.. _Aspects: https://docs.openedx.org/projects/openedx-aspects/en/latest/index.html\n\nGetting Started\n***************\n\nDeveloping\n==========\n\nOne Time Setup\n--------------\n.. code-block::\n\n # Clone the repository\n git clone git@github.com:openedx/openedx-event-sink-clickhouse.git\n cd openedx-event-sink-clickhouse\n\n # Set up a virtualenv using virtualenvwrapper with the same name as the repo and activate it\n mkvirtualenv -p python3.8 openedx-event-sink-clickhouse\n\n\nEvery time you develop something in this repo\n---------------------------------------------\n.. code-block::\n\n # Activate the virtualenv\n workon openedx-event-sink-clickhouse\n\n # Grab the latest code\n git checkout main\n git pull\n\n # Install/update the dev requirements\n make requirements\n\n # Run the tests and quality checks (to verify the status before you make any changes)\n make validate\n\n # Make a new branch for your changes\n git checkout -b <your_github_username>/<short_description>\n\n # Using your favorite editor, edit the code to make your change.\n vim ...\n\n # Run your new tests\n pytest ./path/to/new/tests\n\n # Run all the tests and quality checks\n make validate\n\n # Commit all your changes\n git commit ...\n git push\n\n # Open a PR and ask for review.\n\nDeploying\n=========\n\nThe Open edX Event Sink Clickhouse component is a django plugin which doesn't\nneed independent deployment. Therefore, its setup is reasonably\nstraightforward. First, it needs to be added to your service\nrequirements, and then it will be installed alongside requirements\nof the service.\n\nThis plugin will be deployed by default in an OARS Tutor environment. For other\ndeployments install the library or add it to private requirements of your\nvirtual environment ( ``requirements/private.txt`` ).\n\n#. Run ``pip install openedx-event-sink-clickhouse``.\n\n#. Run migrations:\n\n- ``python manage.py lms migrate``\n\n- ``python manage.py cms migrate``\n\n#. Restart LMS service and celery workers of edx-platform.\n\nConfiguration\n===============\n\nCurrently all events will be listened to by default (there is only one). So\nthe only necessary configuration is a ClickHouse connection:\n\n.. code-block::\n\n EVENT_SINK_CLICKHOUSE_BACKEND_CONFIG = {\n # URL to a running ClickHouse server's HTTP interface. ex: https://foo.openedx.org:8443/ or\n # http://foo.openedx.org:8123/ . Note that we only support the ClickHouse HTTP interface\n # to avoid pulling in more dependencies to the platform than necessary.\n \"url\": \"http://clickhouse:8123\",\n \"username\": \"changeme\",\n \"password\": \"changeme\",\n \"database\": \"event_sink\",\n \"timeout_secs\": 3,\n }\n\nGetting Help\n************\n\nDocumentation\n=============\n\nSee `documentation on Read the Docs <https://openedx-event-sink-clickhouse.readthedocs.io/en/latest/>`_.\n\nMore Help\n=========\n\nIf you're having trouble, we have discussion forums at\nhttps://discuss.openedx.org where you can connect with others in the\ncommunity.\n\nOur real-time conversations are on Slack. You can request a `Slack\ninvitation`_, then join our `community Slack workspace`_.\n\nFor anything non-trivial, the best path is to open an issue in this\nrepository with as many details about the issue you are facing as you\ncan provide.\n\nhttps://github.com/openedx/openedx-event-sink-clickhouse/issues\n\nFor more information about these options, see the `Getting Help`_ page.\n\n.. _Slack invitation: https://openedx.org/slack\n.. _community Slack workspace: https://openedx.slack.com/\n.. _Getting Help: https://openedx.org/getting-help\n\nLicense\n*******\n\nThe code in this repository is licensed under the AGPL 3.0 unless\notherwise noted.\n\nPlease see `LICENSE.txt <LICENSE.txt>`_ for details.\n\nContributing\n************\n\nContributions are very welcome.\nPlease read `How To Contribute <https://openedx.org/r/how-to-contribute>`_ for details.\n\nThis project is currently accepting all types of contributions, bug fixes,\nsecurity fixes, maintenance work, or new features. However, please make sure\nto have a discussion about your new feature idea with the maintainers prior to\nbeginning development to maximize the chances of your change being accepted.\nYou can start a conversation by creating a new issue on this repo summarizing\nyour idea.\n\nThe Open edX Code of Conduct\n****************************\n\nAll community members are expected to follow the `Open edX Code of Conduct`_.\n\n.. _Open edX Code of Conduct: https://openedx.org/code-of-conduct/\n\nPeople\n******\n\nThe assigned maintainers for this component and other project details may be\nfound in `Backstage`_. Backstage pulls this data from the ``catalog-info.yaml``\nfile in this repo.\n\n.. _Backstage: https://open-edx-backstage.herokuapp.com/catalog/default/component/openedx-event-sink-clickhouse\n\nReporting Security Issues\n*************************\n\nPlease do not report security issues in public. Please email security@openedx.org.\n",
"bugtrack_url": null,
"license": "AGPL 3.0",
"summary": "A sink for Open edX events to send them to ClickHouse",
"version": "1.1.1",
"project_urls": {
"Homepage": "https://github.com/openedx/openedx_event_sink_clickhouse"
},
"split_keywords": [
"python",
"edx"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "7481aade0142f67698befc87feb418dfd9282382d1620e5c5ab9ae5080a93f65",
"md5": "98789d32866122abf93c2a91b77d3788",
"sha256": "62ec5d127b0835a4fc94fae257a5074572a30c51b7bc435d36f086152f01109c"
},
"downloads": -1,
"filename": "openedx_event_sink_clickhouse-1.1.1-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "98789d32866122abf93c2a91b77d3788",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 34826,
"upload_time": "2024-02-20T15:30:53",
"upload_time_iso_8601": "2024-02-20T15:30:53.944006Z",
"url": "https://files.pythonhosted.org/packages/74/81/aade0142f67698befc87feb418dfd9282382d1620e5c5ab9ae5080a93f65/openedx_event_sink_clickhouse-1.1.1-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "829f5fd5032bff3dfcc442a083c05a7544c0ef1db3fccc6499057cb8f6ee4f88",
"md5": "f6da3cef835968f52a374c11e1c6c593",
"sha256": "0e611bbf8f0687766fd1e2ea56f86e65c1f32e92e07fd9f59e6a9a7c7a006bbe"
},
"downloads": -1,
"filename": "openedx_event_sink_clickhouse-1.1.1.tar.gz",
"has_sig": false,
"md5_digest": "f6da3cef835968f52a374c11e1c6c593",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 40301,
"upload_time": "2024-02-20T15:30:55",
"upload_time_iso_8601": "2024-02-20T15:30:55.586856Z",
"url": "https://files.pythonhosted.org/packages/82/9f/5fd5032bff3dfcc442a083c05a7544c0ef1db3fccc6499057cb8f6ee4f88/openedx_event_sink_clickhouse-1.1.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-02-20 15:30:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "openedx",
"github_project": "openedx_event_sink_clickhouse",
"github_not_found": true,
"lcname": "openedx-event-sink-clickhouse"
}
JSON
