Platform Plugin Aspects
#######################
Purpose
*******
This repository holds various Aspects plugins for the Open edX platform.
Sinks
*****
Sinks are components that listen for events and store them in ClickHouse. The
events are emitted by the Open edX platform via `Open edX events`_ or Django signals.
Available Sinks
===============
- `CourseOverviewSink` - Listens for the `COURSE_PUBLISHED` event and stores the
course structure data through `XBlockSink` in ClickHouse.
- `ExternalIdSink` - Listens for the `post_save` Django signal on the `ExternalId`
model and stores the external id data in ClickHouse.
- `UserProfile` - Listens for the `post_save` Django signal on the `UserProfile`
model and stores the user profile data in ClickHouse.
- `UserRetirementSink` - Listen for the `USER_RETIRE_LMS_MISC` Django signal and
remove the user PII information from ClickHouse.
Commands
========
In addition to being an event listener, this package provides the following commands:
- `dump_data_to_clickhouse` - This command allows bulk export of the data from the Sinks.
Allows bootstrapping a new data platform or backfilling lost or missing data.
``python manage.py cms dump_data_to_clickhouse``
- `load_test_tracking_events` - This command allows loading test tracking events into
ClickHouse. This is useful for testing the ClickHouse connection to measure the performance of the
different data pipelines such as Vector, Event Bus (Redis and Kafka), and Celery.
Do not use this command in production as it will generate a large amount of data
and will slow down the system.
``python manage.py cms load_test_tracking_events``
- `monitor_load_test_tracking` - Monitors the load test tracking script and saves
output for later analysis.
``python manage.py cms monitor_load_test_tracking``
Instructor Dashboard Integration
================================
Dashboards from `Aspects`_ are integrated into the Instructor Dashboard via `Superset Embedded SDK`_.
See `Configuration`_ for more details.
Getting Started with Development
********************************
Please see the Open edX documentation for `guidance on Python development <https://docs.openedx.org/en/latest/developers/how-tos/get-ready-for-python-dev.html>`_ in this repo.
Deploying
*********
The `Platform Plugin Aspects` component is a django plugin which doesn't
need independent deployment. Therefore, its setup is reasonably straightforward.
First, it needs to be added to your service requirements, and then it will be
installed alongside requirements of the service.
Configuration
*************
Instructor Dashboard Configuration
==================================
The Instructor Dashboard integration uses the `Open edX Filters`_. To learn more about
the filters, see the `Open edX Filters`_ documentation. Make sure to configure the
superset pipeline into the filter as follows:
.. code-block:: python
OPEN_EDX_FILTERS_CONFIG = {
"org.openedx.learning.instructor.dashboard.render.started.v1": {
"fail_silently": False,
"pipeline": [
"platform_plugin_superset.extensions.filters.AddSupersetTab",
]
},
}
- `SUPERSET_CONFIG` - This setting is used to configure the Superset Embedded SDK.
The configuration is a dictionary that contains the following keys:
- `internal_service_url` - The URL of the Superset instance (useful in development, omit in production).
- `service_url` - The URL of the Superset instance.
- `username` - The username of the Superset user.
- `password` - The password of the Superset user.
- `ASPECTS_INSTRUCTOR_DASHBOARDS` - This setting is used to configure the dashboards
that will be displayed in the Instructor Dashboard. The configuration is a list of
dictionaries that contains the following keys:
- `name` - The name of the dashboard.
- `slug` - The slug of the dashboard.
- `uuid` - The UUID of the dashboard.
- `allow_translations` - A boolean value that determines if the dashboard
is translated in `Aspects`_.
- `SUPERSET_EXTRA_FILTERS_FORMAT` - This setting is used to configure the extra filters
that will be applied to the dashboards. The configuration is a list of strings that
can be formatted with the following variables:
- `user` - The user object.
- `course` - The course object.
- `SUPERSET_DASHBOARD_LOCALES` - This setting is used to configure the available locales
for the dashboards. The configuration is a list of supported locales by `Aspects`_.
Event Sink Configuration
========================
- `EVENT_SINK_CLICKHOUSE_BACKEND_CONFIG` - This setting is used to configure the ClickHouse
connection. The configuration is a dictionary that contains the following keys:
- `url` - The host of the ClickHouse instance.
- `database` - The database name.
- `username` - The username of the ClickHouse user.
- `password` - The password of the ClickHouse user.
- `timeout_secs` - The timeout in seconds for the ClickHouse connection.
- `EVENT_SINK_CLICKHOUSE_PII_MODELS` - This setting is used to configure the models that
contain PII information. The configuration is a list of strings that contain the
table names where the PII information is stored.
- `EVENT_SINK_CLICKHOUSE_MODEL_CONFIG` - This setting is used to provide compatibility
with multiple Open edX models. The configuration is a dictionary that contains the
following a key per model that contains a dictionary with the following keys:
- `module` - The module path of the model.
- `model` - The model class name.
Event Sinks are disabled by default. To enable them, you need to enable the following
waffle flag: `event_sink_clickhouse.{{model_name}}.enabled` where model name is the name
of the model that you want to enable. Or, you can enable them via settings by setting
`EVENT_SINK_CLICKHOUSE_{{model_name}}_ENABLED` to `True`.
Getting Help
************
Documentation
=============
PLACEHOLDER: Start by going through `the documentation`_. If you need more help see below.
.. _the documentation: https://docs.openedx.org/projects/platform-plugin-aspects
(TODO: `Set up documentation <https://openedx.atlassian.net/wiki/spaces/DOC/pages/21627535/Publish+Documentation+on+Read+the+Docs>`_)
More Help
=========
If you're having trouble, we have discussion forums at
https://discuss.openedx.org where you can connect with others in the
community.
Our real-time conversations are on Slack. You can request a `Slack
invitation`_, then join our `community Slack workspace`_.
For anything non-trivial, the best path is to open an issue in this
repository with as many details about the issue you are facing as you
can provide.
https://github.com/openedx/platform-plugin-aspects/issues
For more information about these options, see the `Getting Help <https://openedx.org/getting-help>`__ page.
.. _Slack invitation: https://openedx.org/slack
.. _community Slack workspace: https://openedx.slack.com/
License
*******
Please see `LICENSE.txt <LICENSE.txt>`_ for details.
Contributing
************
Contributions are very welcome.
Please read `How To Contribute <https://openedx.org/r/how-to-contribute>`_ for details.
This project is currently accepting all types of contributions, bug fixes,
security fixes, maintenance work, or new features. However, please make sure
to have a discussion about your new feature idea with the maintainers prior to
beginning development to maximize the chances of your change being accepted.
You can start a conversation by creating a new issue on this repo summarizing
your idea.
The Open edX Code of Conduct
****************************
All community members are expected to follow the `Open edX Code of Conduct`_.
.. _Open edX Code of Conduct: https://openedx.org/code-of-conduct/
People
******
The assigned maintainers for this component and other project details may be
found in `Backstage`_. Backstage pulls this data from the ``catalog-info.yaml``
file in this repo.
.. _Backstage: https://backstage.openedx.org/catalog/default/component/platform-plugin-aspects
Reporting Security Issues
*************************
Please do not report security issues in public. Please email security@openedx.org.
.. _Open edX events: https://github.com/openedx/openedx-events
.. _Edx Platform: https://github.com/openedx/edx-platform
.. _ClickHouse: https://clickhouse.com
.. _Aspects: https://docs.openedx.org/projects/openedx-aspects/en/latest/index.html
.. _Superset Embedded SDK: https://www.npmjs.com/package/@superset-ui/embedded-sdk
.. _Open edX Filters: https://docs.openedx.org/projects/openedx-filters/en/latest/
Change Log
##########
..
All enhancements and patches to platform_plugin_aspects will be documented
in this file. It adheres to the structure of https://keepachangelog.com/ ,
but in reStructuredText instead of Markdown (for ease of incorporation into
Sphinx documentation and the PyPI description).
This project adheres to Semantic Versioning (https://semver.org/).
.. There should always be an "Unreleased" section for changes pending release.
Unreleased
**********
*
0.7.3 - 2024-04-30
******************
Fixed
=====
* Fixed UUID generation for localized Superset assets, which caused embedded
dashboards to fail to load when localized.
0.7.2 - 2024-04-19
******************
Fixed
=====
* Fixed cms url configuration
0.7.1 - 2024-04-17
******************
Fixed
=====
* Fixed issue with embedded dashboards throwing javascript errors
* Fixed issues with translated embedded dashboards erroring in Superset
0.7.0 - 2024-04-12
******************
Added
=====
* Add endpoint for fetchGuestToken
0.6.0 - 2024-04-08
******************
Added
=====
* Allow to embed translated Superset Dashboards.
0.5.0 - 2024-04-01
******************
Added
=====
* Load testing and test monitoring scripts.
0.4.0 - 2024-03-18
******************
Added
=====
* Embed multiple Superset Dashboards.
0.3.1 - 2024-03-14
******************
Added
=====
* Fixed development server configuration.
0.3.0 – 2024-03-10
******************
Added
=====
* Imported XBlock code from platform-plugin-superset
0.2.0 – 2024-03-05
******************
Added
=====
* Imported code from event-sink-clickhouse.
0.1.0 – 2024-02-29
**********************************************
Added
=====
* First release on PyPI.
Raw data
{
"_id": null,
"home_page": "https://github.com/openedx/platform-plugin-aspects",
"name": "platform-plugin-aspects",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "Python edx",
"author": "Open edX Project",
"author_email": "oscm@openedx.org",
"download_url": "https://files.pythonhosted.org/packages/d3/4e/1c06fb8d8ff7045402b4e484dc36cf004104db1592cdbda4caf54351efe7/platform_plugin_aspects-0.7.3.tar.gz",
"platform": null,
"description": "Platform Plugin Aspects\n#######################\n\nPurpose\n*******\n\nThis repository holds various Aspects plugins for the Open edX platform.\n\nSinks\n*****\n\nSinks are components that listen for events and store them in ClickHouse. The\nevents are emitted by the Open edX platform via `Open edX events`_ or Django signals.\n\nAvailable Sinks\n===============\n\n- `CourseOverviewSink` - Listens for the `COURSE_PUBLISHED` event and stores the\n course structure data through `XBlockSink` in ClickHouse.\n- `ExternalIdSink` - Listens for the `post_save` Django signal on the `ExternalId`\n model and stores the external id data in ClickHouse.\n- `UserProfile` - Listens for the `post_save` Django signal on the `UserProfile`\n model and stores the user profile data in ClickHouse.\n- `UserRetirementSink` - Listen for the `USER_RETIRE_LMS_MISC` Django signal and\n remove the user PII information from ClickHouse.\n\nCommands\n========\n\nIn addition to being an event listener, this package provides the following commands:\n\n- `dump_data_to_clickhouse` - This command allows bulk export of the data from the Sinks.\n Allows bootstrapping a new data platform or backfilling lost or missing data.\n\n ``python manage.py cms dump_data_to_clickhouse``\n\n- `load_test_tracking_events` - This command allows loading test tracking events into\n ClickHouse. This is useful for testing the ClickHouse connection to measure the performance of the\n different data pipelines such as Vector, Event Bus (Redis and Kafka), and Celery.\n\n Do not use this command in production as it will generate a large amount of data\n and will slow down the system.\n\n ``python manage.py cms load_test_tracking_events``\n\n- `monitor_load_test_tracking` - Monitors the load test tracking script and saves\n output for later analysis.\n\n ``python manage.py cms monitor_load_test_tracking``\n\nInstructor Dashboard Integration\n================================\n\nDashboards from `Aspects`_ are integrated into the Instructor Dashboard via `Superset Embedded SDK`_.\nSee `Configuration`_ for more details.\n\nGetting Started with Development\n********************************\n\nPlease see the Open edX documentation for `guidance on Python development <https://docs.openedx.org/en/latest/developers/how-tos/get-ready-for-python-dev.html>`_ in this repo.\n\nDeploying\n*********\n\nThe `Platform Plugin Aspects` component is a django plugin which doesn't\nneed independent deployment. Therefore, its setup is reasonably straightforward.\nFirst, it needs to be added to your service requirements, and then it will be\ninstalled alongside requirements of the service.\n\nConfiguration\n*************\n\nInstructor Dashboard Configuration\n==================================\n\nThe Instructor Dashboard integration uses the `Open edX Filters`_. To learn more about\nthe filters, see the `Open edX Filters`_ documentation. Make sure to configure the\nsuperset pipeline into the filter as follows:\n\n .. code-block:: python\n\n OPEN_EDX_FILTERS_CONFIG = {\n \"org.openedx.learning.instructor.dashboard.render.started.v1\": {\n \"fail_silently\": False,\n \"pipeline\": [\n \"platform_plugin_superset.extensions.filters.AddSupersetTab\",\n ]\n },\n }\n\n- `SUPERSET_CONFIG` - This setting is used to configure the Superset Embedded SDK.\n The configuration is a dictionary that contains the following keys:\n\n - `internal_service_url` - The URL of the Superset instance (useful in development, omit in production).\n - `service_url` - The URL of the Superset instance.\n - `username` - The username of the Superset user.\n - `password` - The password of the Superset user.\n\n- `ASPECTS_INSTRUCTOR_DASHBOARDS` - This setting is used to configure the dashboards\n that will be displayed in the Instructor Dashboard. The configuration is a list of\n dictionaries that contains the following keys:\n\n - `name` - The name of the dashboard.\n - `slug` - The slug of the dashboard.\n - `uuid` - The UUID of the dashboard.\n - `allow_translations` - A boolean value that determines if the dashboard\n is translated in `Aspects`_.\n\n- `SUPERSET_EXTRA_FILTERS_FORMAT` - This setting is used to configure the extra filters\n that will be applied to the dashboards. The configuration is a list of strings that\n can be formatted with the following variables:\n\n - `user` - The user object.\n - `course` - The course object.\n\n- `SUPERSET_DASHBOARD_LOCALES` - This setting is used to configure the available locales\n for the dashboards. The configuration is a list of supported locales by `Aspects`_.\n\nEvent Sink Configuration\n========================\n\n- `EVENT_SINK_CLICKHOUSE_BACKEND_CONFIG` - This setting is used to configure the ClickHouse\n connection. The configuration is a dictionary that contains the following keys:\n\n - `url` - The host of the ClickHouse instance.\n - `database` - The database name.\n - `username` - The username of the ClickHouse user.\n - `password` - The password of the ClickHouse user.\n - `timeout_secs` - The timeout in seconds for the ClickHouse connection.\n\n- `EVENT_SINK_CLICKHOUSE_PII_MODELS` - This setting is used to configure the models that\n contain PII information. The configuration is a list of strings that contain the\n table names where the PII information is stored.\n\n- `EVENT_SINK_CLICKHOUSE_MODEL_CONFIG` - This setting is used to provide compatibility\n with multiple Open edX models. The configuration is a dictionary that contains the\n following a key per model that contains a dictionary with the following keys:\n\n - `module` - The module path of the model.\n - `model` - The model class name.\n\nEvent Sinks are disabled by default. To enable them, you need to enable the following\nwaffle flag: `event_sink_clickhouse.{{model_name}}.enabled` where model name is the name\nof the model that you want to enable. Or, you can enable them via settings by setting\n`EVENT_SINK_CLICKHOUSE_{{model_name}}_ENABLED` to `True`.\n\n\nGetting Help\n************\n\nDocumentation\n=============\n\nPLACEHOLDER: Start by going through `the documentation`_. If you need more help see below.\n\n.. _the documentation: https://docs.openedx.org/projects/platform-plugin-aspects\n\n(TODO: `Set up documentation <https://openedx.atlassian.net/wiki/spaces/DOC/pages/21627535/Publish+Documentation+on+Read+the+Docs>`_)\n\nMore Help\n=========\n\nIf you're having trouble, we have discussion forums at\nhttps://discuss.openedx.org where you can connect with others in the\ncommunity.\n\nOur real-time conversations are on Slack. You can request a `Slack\ninvitation`_, then join our `community Slack workspace`_.\n\nFor anything non-trivial, the best path is to open an issue in this\nrepository with as many details about the issue you are facing as you\ncan provide.\n\nhttps://github.com/openedx/platform-plugin-aspects/issues\n\nFor more information about these options, see the `Getting Help <https://openedx.org/getting-help>`__ page.\n\n.. _Slack invitation: https://openedx.org/slack\n.. _community Slack workspace: https://openedx.slack.com/\n\nLicense\n*******\n\nPlease see `LICENSE.txt <LICENSE.txt>`_ for details.\n\nContributing\n************\n\nContributions are very welcome.\nPlease read `How To Contribute <https://openedx.org/r/how-to-contribute>`_ for details.\n\nThis project is currently accepting all types of contributions, bug fixes,\nsecurity fixes, maintenance work, or new features. However, please make sure\nto have a discussion about your new feature idea with the maintainers prior to\nbeginning development to maximize the chances of your change being accepted.\nYou can start a conversation by creating a new issue on this repo summarizing\nyour idea.\n\nThe Open edX Code of Conduct\n****************************\n\nAll community members are expected to follow the `Open edX Code of Conduct`_.\n\n.. _Open edX Code of Conduct: https://openedx.org/code-of-conduct/\n\nPeople\n******\n\nThe assigned maintainers for this component and other project details may be\nfound in `Backstage`_. Backstage pulls this data from the ``catalog-info.yaml``\nfile in this repo.\n\n.. _Backstage: https://backstage.openedx.org/catalog/default/component/platform-plugin-aspects\n\nReporting Security Issues\n*************************\n\nPlease do not report security issues in public. Please email security@openedx.org.\n\n.. _Open edX events: https://github.com/openedx/openedx-events\n.. _Edx Platform: https://github.com/openedx/edx-platform\n.. _ClickHouse: https://clickhouse.com\n.. _Aspects: https://docs.openedx.org/projects/openedx-aspects/en/latest/index.html\n.. _Superset Embedded SDK: https://www.npmjs.com/package/@superset-ui/embedded-sdk\n.. _Open edX Filters: https://docs.openedx.org/projects/openedx-filters/en/latest/\n\n\nChange Log\n##########\n\n..\n All enhancements and patches to platform_plugin_aspects will be documented\n in this file. It adheres to the structure of https://keepachangelog.com/ ,\n but in reStructuredText instead of Markdown (for ease of incorporation into\n Sphinx documentation and the PyPI description).\n\n This project adheres to Semantic Versioning (https://semver.org/).\n\n.. There should always be an \"Unreleased\" section for changes pending release.\n\nUnreleased\n**********\n\n*\n\n0.7.3 - 2024-04-30\n******************\n\nFixed\n=====\n\n* Fixed UUID generation for localized Superset assets, which caused embedded\n dashboards to fail to load when localized.\n\n0.7.2 - 2024-04-19\n******************\n\nFixed\n=====\n\n* Fixed cms url configuration\n\n0.7.1 - 2024-04-17\n******************\n\nFixed\n=====\n\n* Fixed issue with embedded dashboards throwing javascript errors\n* Fixed issues with translated embedded dashboards erroring in Superset\n\n0.7.0 - 2024-04-12\n******************\n\nAdded\n=====\n\n* Add endpoint for fetchGuestToken\n\n0.6.0 - 2024-04-08\n******************\n\nAdded\n=====\n\n* Allow to embed translated Superset Dashboards.\n\n0.5.0 - 2024-04-01\n******************\n\nAdded\n=====\n\n* Load testing and test monitoring scripts.\n\n0.4.0 - 2024-03-18\n******************\n\nAdded\n=====\n\n* Embed multiple Superset Dashboards.\n\n0.3.1 - 2024-03-14\n******************\n\nAdded\n=====\n\n* Fixed development server configuration.\n\n0.3.0 \u2013 2024-03-10\n******************\n\nAdded\n=====\n\n* Imported XBlock code from platform-plugin-superset\n\n0.2.0 \u2013 2024-03-05\n******************\n\nAdded\n=====\n\n* Imported code from event-sink-clickhouse.\n\n0.1.0 \u2013 2024-02-29\n**********************************************\n\nAdded\n=====\n\n* First release on PyPI.\n",
"bugtrack_url": null,
"license": "Apache 2.0",
"summary": "Aspects plugins for edx-platform",
"version": "0.7.3",
"project_urls": {
"Homepage": "https://github.com/openedx/platform-plugin-aspects"
},
"split_keywords": [
"python",
"edx"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "26a24af782145604524694711adcf0f90500875bee1553e4e5f7af3a78f266ef",
"md5": "8fddd8d3c4d1b23e9b7ed813aa3ab3b4",
"sha256": "6914730607039866e5d5a36561a595277f3f2a7937a12a3c9e7385ada9c9fabe"
},
"downloads": -1,
"filename": "platform_plugin_aspects-0.7.3-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "8fddd8d3c4d1b23e9b7ed813aa3ab3b4",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": ">=3.8",
"size": 48900,
"upload_time": "2024-04-30T18:32:48",
"upload_time_iso_8601": "2024-04-30T18:32:48.475807Z",
"url": "https://files.pythonhosted.org/packages/26/a2/4af782145604524694711adcf0f90500875bee1553e4e5f7af3a78f266ef/platform_plugin_aspects-0.7.3-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "d34e1c06fb8d8ff7045402b4e484dc36cf004104db1592cdbda4caf54351efe7",
"md5": "c69c446fd83dc2e7f260d8f37c300643",
"sha256": "431703c2da5a790b6568ce190e59b05a00a41673bb2610f7d2f3976d8ec3f257"
},
"downloads": -1,
"filename": "platform_plugin_aspects-0.7.3.tar.gz",
"has_sig": false,
"md5_digest": "c69c446fd83dc2e7f260d8f37c300643",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 43797,
"upload_time": "2024-04-30T18:32:49",
"upload_time_iso_8601": "2024-04-30T18:32:49.827462Z",
"url": "https://files.pythonhosted.org/packages/d3/4e/1c06fb8d8ff7045402b4e484dc36cf004104db1592cdbda4caf54351efe7/platform_plugin_aspects-0.7.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-30 18:32:49",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "openedx",
"github_project": "platform-plugin-aspects",
"travis_ci": false,
"coveralls": true,
"github_actions": true,
"tox": true,
"lcname": "platform-plugin-aspects"
}
JSON
