==========
Annalist
==========
.. image:: https://img.shields.io/pypi/v/data-annalist.svg
:target: https://pypi.python.org/pypi/data-annalist
.. image:: https://readthedocs.org/projects/annalist/badge/?version=latest
:target: https://annalist.readthedocs.io/en/latest/?version=latest
:alt: Documentation Status
.. image:: https://results.pre-commit.ci/badge/github/nicmostert/annalist/main.svg
:target: https://results.pre-commit.ci/latest/github/nicmostert/annalist/main
:alt: pre-commit.ci status
Audit trail generator for data processing scripts.
* Free software: GNU General Public License v3
* Documentation: https://annalist.readthedocs.io.
==================
Usage
==================
Create an ``Annalist`` object at the base of the module you'd like to audit. use the ``@Annalist.annalize`` decorator on any function you would like to annalize
::
from annalist.annalist import Annalist
from annalist.decorators import function_logger
ann = Annalist()
ann.configure()
@function_logger
def example_function():
...
Annalist also works on class methods, with the help of the ``ClassLogger`` decorator.
::
from annalist.decorators import ClassLogger
class ExampleClass():
# Initializers can be annalized just fine
@ClassLogger # type: ignore
__init__(self, arg1, arg2):
self.arg1 = arg1
self._arg2 = arg2
...
# DO NOT put an annalizer on a property definition.
# The annalizer calls the property itself, creating infinite recursion.
@property
def arg2(self):
return self._arg2
# Putting an annalizer on a setter is fine though.
# Just make sure you put it after the setter decorator.
@ClassLogger # type: ignore
@arg2.setter
def arg2(self, value):
self._arg2 = value
# DO NOT put it on the __repr__ either.
# Same as before, this creates infinite recursion.
def __repr__(self):
return f"{str(arg1)}: {str(arg2)}"
In the main script, the Annalist object must be called again. This will point to the singleton object initialized in the dependency. The annalist must be configured before usage.
.. note:: Note the `# type: ignore` inline comments. These are only necessary when using static type checkers like MyPy and Pyright. They don't really seem to like decorators very much. They need to be supplied when decorating an `__init__` constructor method, or when adding multiple decorators to a method.
>>> ann = Annalist()
>>> ann.configure(logger_name="Example Logger", analyst_name="Speve")
Now the annalized code can be run like normal, and will be audited.
>>> example_function()
2023/11/2 09:42:13 | INFO | example_function called by Speve as part of Example Logger session
Formatters
-------------------
Annalist is built on the standard python *logging* library. Formatters can be specified in the same syntax as is documented in the `logging docs`. The available fields can be found in `Fields`.
Annalist supports two formatters. The *File formatters* formats the output to the logfile, and *Stream formatter* formats the console output.
::
from annalist.annalist import Annalist
ann = Annalist()
ann.configure(...)
ann.set_file_formatter(
"%(asctime)s, %(analyst_name)s, example_funtion "
"| %(message)s",
)
ann.set_stream_formatter(
"%(asctime)s, %(function_name)s "
)
In this example, the console output might be
>>> example_function()
2023/11/2 09:42:13, example_function
whereas the contents of the logfile might be:
::
2023/11/2 09:42:13, example_function, Speve | This is an example.
Fields
___________
Annalist collects information about a decorated function and makes those available as fields. Additionally, the fields from the logging library are also available, although they are generally less useful. Below are all the useful features that are available. See all the logging fields `here`_.The reason for their limited usefulness are that most of the code references made there point to the annalist library, and not the decorated code.
All the fields that we consider useful are listed below:
.. _here: https://docs.python.org/3/library/logging.html#logrecord-attributes
+--------------------+----------------------------------------+---------------------+
| Field | Description | Source |
+====================+========================================+=====================+
| ``analyst_name`` | Name of the analyst writing the script | User configured |
+--------------------+----------------------------------------+---------------------+
| ``function_name`` | Function Name | Function Inspection |
+--------------------+----------------------------------------+---------------------+
| ``function_doc`` | Function Docstring | Function Inspection |
+--------------------+----------------------------------------+---------------------+
| ``ret_val`` | Return value | Function Inspection |
+--------------------+----------------------------------------+---------------------+
| ``ret_val_type`` | Return value type | Function Inspection |
+--------------------+----------------------------------------+---------------------+
| ``ret_annotation`` | Annotation of return value | Function Inspection |
+--------------------+----------------------------------------+---------------------+
| ``params`` | Input parameters | Function Inspection |
+--------------------+----------------------------------------+---------------------+
| ``asctime`` | Time of function call | Logging Library |
+--------------------+----------------------------------------+---------------------+
| ``levelname`` | Logging level name | Logging Library |
+--------------------+----------------------------------------+---------------------+
| ``levelno`` | Logging level number | Logging Library |
+--------------------+----------------------------------------+---------------------+
| ``message``\* | Needs to be passed as extra param | Logging Library |
+--------------------+----------------------------------------+---------------------+
| ``name`` | Logger name | Logging Library |
+--------------------+----------------------------------------+---------------------+
The ``message`` field is an optional parameter that can be passed directly to the decorator. This is the simplest way to add more information to a function log.
::
@function_logger(message="this is a message")
def example_function():
...
You can also specify the level of the logger in the same way, as a decorator keword argument.
::
@function_logger(level="DEBUG")
def example_function():
...
Unfortunately, Annalist does not yet offer support of passing these fields into the ``@ClassLogger``. However, we can still get information to the logger by inspecting the method arguments, and the attributes on the class instance. Consider the following setup::
from annalist.decorators import ClassLogger
class MyClass:
@ClassLogger # type: ignore
def __init__(attr, prop):
self.attr = attr
self._prop = prop
@property
def prop(self):
return prop
@ClassLogger # type: ignore
@prop.setter
def prop(self, value):
self._prop = value
@ClassLogger
def square_attr(self):
return self.attr ** 2
@ClassLogger
def add_prop_to_attr(self):
return attr + prop
@ClassLogger # type: ignore
@staticmethod
def increment_value(attr):
return attr += 1
Note the two class attributes named ``attr`` and ``prop``. We can track these properties based on their variable names by passing it into the formatter:
>>> from annalist.annalist import Annalist
>>> ann = Annalist()
>>> ann.configure(...)
>>> ann.add_stream_formatter("%(function_name)s | prop: %(prop)s | attr: %(attr)s")
The ``ClassLogger`` decorator activates upon runtime and inspects the namespace. First, it looks for the attribute in the names of the input arguments of the decorated function. If found, it sends it to the formatter (See "Custom Fields"):
>>> mc = MyClass(7, 2)
>>> mc.prop = 3
prop | prop: 3 | attr: 7
Notice that the ``setter`` of ``prop`` caused ``ClassLogger`` to look for the values of ``prop`` and ``attr`` on the ``mc`` instance.
>>> mc.square_attr()
49
square_attr | prop: 3 | attr: 7
Notice how the function ``square_attr`` did not alter the value of ``attr``.
Because this logger is sensitive to the state of the logger, it is important to be weary of variable names.
>>> mc.increment_value(5)
6
square_attr | prop: 3 | attr: 5
Notice how, despite having no real reference to the attribute ``attr`` on the namespace, the logger found the input argument named ``attr``, and associated this with the attribute it is logging. I believe this to be a useful feature, but care should be taken when using it like this.
Custom Fields
--------------
Annalist accepts any number of arbitrary fields in the formatter. If these fields are not one of the fields available by default, the fields is dynamically added and processed. However, this field must then be passed to the decorator in the ``extra_info`` argument.
For example, you might set the formatter as follows. In this example, the fields ``site`` and ``hts_file`` are custom, and are not available by default.
::
from annalist.annalist import Annalist
ann = Annalist()
ann.configure(...)
ann.set_file_formatter(
"%(asctime)s, %(analyst_name)s, %(site)s, %(hts_file)s "
"| %(message)s",
)
Then, passing those parameters into the example function looks like this:
::
from annalist.decorators import function_logger
hts_file = "file.hts"
@function_logger(
level="INFO",
message="This decorator passes extra parameters",
extra_info={
"site_name": "Site one",
"hts_file": hts_file,
}
)
def example_function(arg1, arg2):
...
If the custom fields are not included in a function decorator, they will simply default to ``None``.
The ``function_logger`` can also be used in "wrapper" mode. This is useful when an undecorated function needs to be annalized at call-time::
function_logger(
example_function,
level="INFO",
message="This decorator passes extra parameters",
extra_info={
"site_name": "Site one",
"hts_file": hts_file,
}
)(arg1, arg2)
When using Annalist in a class method, you might want to log class attributes. Unfortunately, the following syntax will not work, since the decorator has no knowledge of the class instance (self).
::
class ExampleClass:
...
@function_logger(
level="INFO",
message="This decorator passes extra parameters",
extra_info={
"site_name": self.site_name, # THIS DOES NOT WORK!
"hts_file": self.hts_file, # THIS DOES NOT WORK!
}
)
def example_method(self):
...
The way to track class attributes is to use the ``ClassLogger`` decorator. This decorator activates at runtime. Any custom fields passed to the Annalist formatter are noted, and ``ClassLogger`` inspects the class instance for attributes with the same name.
::
from annalist.decorators import ClassLogger
class ExampleClass:
def __init__(self, attr):
self.attr = attr
@ClassLogger
def example_function(self):
...
See annalist.decorators.ClassLogger for more details.
Levels
--------
Annalist uses the levels as defined in the logging library. Upon configuration, the ``default level`` can be set, which is the level at which all logs are logged unless overridden. The default value for ``default level`` is "INFO".
::
ann.configure(
analyst_name="Speve",
stream_format_str=format_str,
level_filter="WARNING",
)
A annalized method can be logged at a raised or lowered level by specifying the logging level explicitely in the decorator:
::
@function_logger(level="DEBUG")
def untracked_function():
...
==================
Feature Roadmap
==================
This roadmap outlines the planned features and milestones for the development of our deterministic and reproducible process auditing system.
Milestone 1: Audit Logging Framework
------------------------------------
x Develop a custom audit logging framework or class.
x Capture function names, input parameters, return values, data types, and timestamps.
x Implement basic logging mechanisms for integration.
Milestone 1.5: Hilltop Auditing Parity
---------------------------------------
x Define custom fields and formatters
x Manage logger levels correctly
Milestone 2: Standardized Logging Format
-----------------------------------------
- Define a standardized logging format for comprehensive auditing.
- Ensure consistency and machine-readability of the logging format.
Milestone 3: Serialization and Deserialization
----------------------------------------------
- Implement serialization and deserialization mechanisms.
- Store and retrieve complex data structures and objects.
- Test serialization for data integrity.
Milestone 4: Versioning and Dependency Tracking
-----------------------------------------------
- Capture and log codebase version (Git commit hash) and dependencies.
- Ensure accurate logging of version and dependency information.
Milestone 5: Integration Testing
--------------------------------
- Create integration tests using the audit logging framework.
- Log information during the execution of key processes.
- Begin development of process recreation capability.
Milestone 6: Reproduction Tool (Partial)
----------------------------------------
- Develop a tool or script to read and reproduce processes from the audit trail.
- Focus on recreating the environment and loading serialized data.
Milestone 7: Documentation (Partial)
--------------------------------------
- Create initial documentation.
- Explain how to use the audit logging framework and the audit trail format.
- Document basic project functionalities.
Milestone 8: Error Handling
---------------------------
- Implement robust error handling for auditing and reproduction code.
- Gracefully handle potential issues.
- Provide informative and actionable error messages.
Milestone 9: MVP Testing
-------------------------
- Conduct testing of the MVP.
- Reproduce processes from the audit trail and verify correctness.
- Gather feedback from initial users within the organization.
Milestone 10: MVP Deployment
------------------------------
- Deploy the MVP within the organization.
- Make it available to relevant team members.
- Encourage usage and collect user feedback.
Milestone 11: Feedback and Iteration
--------------------------------------
- Gather feedback from MVP users.
- Identify shortcomings, usability issues, or missing features.
- Prioritize and plan improvements based on user feedback.
Milestone 12: Scaling and Extending
------------------------------------
- Explore scaling the solution to cover more processes.
- Add additional features and capabilities to enhance usability.
Please note that milestones may overlap, and the order can be adjusted based on project-specific needs. We aim to remain flexible and responsive to feedback during development.
=======
Credits
=======
This package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.
.. _Cookiecutter: https://github.com/audreyr/cookiecutter
.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage
Raw data
{
"_id": null,
"home_page": null,
"name": "data-annalist",
"maintainer": null,
"docs_url": null,
"requires_python": "==3.11.*",
"maintainer_email": null,
"keywords": "logging, auditing, audit trail, hydrology, automation, hilltop, hydrobot, HorizonsRC",
"author": null,
"author_email": "Nic Mostert <nicolas.mostert@horizons.govt.nz>",
"download_url": "https://files.pythonhosted.org/packages/b7/c5/4d6b7a365b7418c3d1687207d141aed0841e57bae23c739405a0590bfe2a/data-annalist-0.4.3.tar.gz",
"platform": null,
"description": "==========\nAnnalist\n==========\n\n.. image:: https://img.shields.io/pypi/v/data-annalist.svg\n :target: https://pypi.python.org/pypi/data-annalist\n\n.. image:: https://readthedocs.org/projects/annalist/badge/?version=latest\n :target: https://annalist.readthedocs.io/en/latest/?version=latest\n :alt: Documentation Status\n\n.. image:: https://results.pre-commit.ci/badge/github/nicmostert/annalist/main.svg\n :target: https://results.pre-commit.ci/latest/github/nicmostert/annalist/main\n :alt: pre-commit.ci status\n\nAudit trail generator for data processing scripts.\n\n\n* Free software: GNU General Public License v3\n* Documentation: https://annalist.readthedocs.io.\n\n==================\nUsage\n==================\n\nCreate an ``Annalist`` object at the base of the module you'd like to audit. use the ``@Annalist.annalize`` decorator on any function you would like to annalize\n\n::\n\n from annalist.annalist import Annalist\n from annalist.decorators import function_logger\n\n ann = Annalist()\n ann.configure()\n\n @function_logger\n def example_function():\n ...\n\nAnnalist also works on class methods, with the help of the ``ClassLogger`` decorator.\n\n::\n\n from annalist.decorators import ClassLogger\n\n class ExampleClass():\n\n # Initializers can be annalized just fine\n @ClassLogger # type: ignore\n __init__(self, arg1, arg2):\n self.arg1 = arg1\n self._arg2 = arg2\n ...\n\n # DO NOT put an annalizer on a property definition.\n # The annalizer calls the property itself, creating infinite recursion.\n @property\n def arg2(self):\n return self._arg2\n\n # Putting an annalizer on a setter is fine though.\n # Just make sure you put it after the setter decorator.\n @ClassLogger # type: ignore\n @arg2.setter\n def arg2(self, value):\n self._arg2 = value\n\n # DO NOT put it on the __repr__ either.\n # Same as before, this creates infinite recursion.\n def __repr__(self):\n return f\"{str(arg1)}: {str(arg2)}\"\n\nIn the main script, the Annalist object must be called again. This will point to the singleton object initialized in the dependency. The annalist must be configured before usage.\n\n.. note:: Note the `# type: ignore` inline comments. These are only necessary when using static type checkers like MyPy and Pyright. They don't really seem to like decorators very much. They need to be supplied when decorating an `__init__` constructor method, or when adding multiple decorators to a method.\n\n>>> ann = Annalist()\n>>> ann.configure(logger_name=\"Example Logger\", analyst_name=\"Speve\")\n\nNow the annalized code can be run like normal, and will be audited.\n\n>>> example_function()\n2023/11/2 09:42:13 | INFO | example_function called by Speve as part of Example Logger session\n\n\nFormatters\n-------------------\n\nAnnalist is built on the standard python *logging* library. Formatters can be specified in the same syntax as is documented in the `logging docs`. The available fields can be found in `Fields`.\n\nAnnalist supports two formatters. The *File formatters* formats the output to the logfile, and *Stream formatter* formats the console output.\n\n::\n\n from annalist.annalist import Annalist\n\n ann = Annalist()\n ann.configure(...)\n\n ann.set_file_formatter(\n \"%(asctime)s, %(analyst_name)s, example_funtion \"\n \"| %(message)s\",\n )\n\n ann.set_stream_formatter(\n \"%(asctime)s, %(function_name)s \"\n )\n\n\nIn this example, the console output might be\n\n>>> example_function()\n2023/11/2 09:42:13, example_function\n\nwhereas the contents of the logfile might be:\n\n::\n\n 2023/11/2 09:42:13, example_function, Speve | This is an example.\n\nFields\n___________\n\nAnnalist collects information about a decorated function and makes those available as fields. Additionally, the fields from the logging library are also available, although they are generally less useful. Below are all the useful features that are available. See all the logging fields `here`_.The reason for their limited usefulness are that most of the code references made there point to the annalist library, and not the decorated code.\n\nAll the fields that we consider useful are listed below:\n\n.. _here: https://docs.python.org/3/library/logging.html#logrecord-attributes\n\n+--------------------+----------------------------------------+---------------------+\n| Field | Description | Source |\n+====================+========================================+=====================+\n| ``analyst_name`` | Name of the analyst writing the script | User configured |\n+--------------------+----------------------------------------+---------------------+\n| ``function_name`` | Function Name | Function Inspection |\n+--------------------+----------------------------------------+---------------------+\n| ``function_doc`` | Function Docstring | Function Inspection |\n+--------------------+----------------------------------------+---------------------+\n| ``ret_val`` | Return value | Function Inspection |\n+--------------------+----------------------------------------+---------------------+\n| ``ret_val_type`` | Return value type | Function Inspection |\n+--------------------+----------------------------------------+---------------------+\n| ``ret_annotation`` | Annotation of return value | Function Inspection |\n+--------------------+----------------------------------------+---------------------+\n| ``params`` | Input parameters | Function Inspection |\n+--------------------+----------------------------------------+---------------------+\n| ``asctime`` | Time of function call | Logging Library |\n+--------------------+----------------------------------------+---------------------+\n| ``levelname`` | Logging level name | Logging Library |\n+--------------------+----------------------------------------+---------------------+\n| ``levelno`` | Logging level number | Logging Library |\n+--------------------+----------------------------------------+---------------------+\n| ``message``\\* | Needs to be passed as extra param | Logging Library |\n+--------------------+----------------------------------------+---------------------+\n| ``name`` | Logger name | Logging Library |\n+--------------------+----------------------------------------+---------------------+\n\nThe ``message`` field is an optional parameter that can be passed directly to the decorator. This is the simplest way to add more information to a function log.\n\n::\n\n @function_logger(message=\"this is a message\")\n def example_function():\n ...\n\n\nYou can also specify the level of the logger in the same way, as a decorator keword argument.\n\n::\n\n @function_logger(level=\"DEBUG\")\n def example_function():\n ...\n\nUnfortunately, Annalist does not yet offer support of passing these fields into the ``@ClassLogger``. However, we can still get information to the logger by inspecting the method arguments, and the attributes on the class instance. Consider the following setup::\n\n from annalist.decorators import ClassLogger\n\n class MyClass:\n @ClassLogger # type: ignore\n def __init__(attr, prop):\n self.attr = attr\n self._prop = prop\n\n @property\n def prop(self):\n return prop\n\n @ClassLogger # type: ignore\n @prop.setter\n def prop(self, value):\n self._prop = value\n\n @ClassLogger\n def square_attr(self):\n return self.attr ** 2\n\n @ClassLogger\n def add_prop_to_attr(self):\n return attr + prop\n\n @ClassLogger # type: ignore\n @staticmethod\n def increment_value(attr):\n return attr += 1\n\nNote the two class attributes named ``attr`` and ``prop``. We can track these properties based on their variable names by passing it into the formatter:\n\n>>> from annalist.annalist import Annalist\n>>> ann = Annalist()\n>>> ann.configure(...)\n>>> ann.add_stream_formatter(\"%(function_name)s | prop: %(prop)s | attr: %(attr)s\")\n\nThe ``ClassLogger`` decorator activates upon runtime and inspects the namespace. First, it looks for the attribute in the names of the input arguments of the decorated function. If found, it sends it to the formatter (See \"Custom Fields\"):\n\n>>> mc = MyClass(7, 2)\n>>> mc.prop = 3\nprop | prop: 3 | attr: 7\n\nNotice that the ``setter`` of ``prop`` caused ``ClassLogger`` to look for the values of ``prop`` and ``attr`` on the ``mc`` instance.\n\n>>> mc.square_attr()\n49\nsquare_attr | prop: 3 | attr: 7\n\nNotice how the function ``square_attr`` did not alter the value of ``attr``.\n\nBecause this logger is sensitive to the state of the logger, it is important to be weary of variable names.\n\n>>> mc.increment_value(5)\n6\nsquare_attr | prop: 3 | attr: 5\n\nNotice how, despite having no real reference to the attribute ``attr`` on the namespace, the logger found the input argument named ``attr``, and associated this with the attribute it is logging. I believe this to be a useful feature, but care should be taken when using it like this.\n\nCustom Fields\n--------------\n\nAnnalist accepts any number of arbitrary fields in the formatter. If these fields are not one of the fields available by default, the fields is dynamically added and processed. However, this field must then be passed to the decorator in the ``extra_info`` argument.\n\nFor example, you might set the formatter as follows. In this example, the fields ``site`` and ``hts_file`` are custom, and are not available by default.\n\n\n::\n\n from annalist.annalist import Annalist\n\n ann = Annalist()\n ann.configure(...)\n\n ann.set_file_formatter(\n \"%(asctime)s, %(analyst_name)s, %(site)s, %(hts_file)s \"\n \"| %(message)s\",\n )\n\nThen, passing those parameters into the example function looks like this:\n\n::\n\n from annalist.decorators import function_logger\n hts_file = \"file.hts\"\n\n @function_logger(\n level=\"INFO\",\n message=\"This decorator passes extra parameters\",\n extra_info={\n \"site_name\": \"Site one\",\n \"hts_file\": hts_file,\n }\n )\n def example_function(arg1, arg2):\n ...\n\n\nIf the custom fields are not included in a function decorator, they will simply default to ``None``.\n\nThe ``function_logger`` can also be used in \"wrapper\" mode. This is useful when an undecorated function needs to be annalized at call-time::\n\n function_logger(\n example_function,\n level=\"INFO\",\n message=\"This decorator passes extra parameters\",\n extra_info={\n \"site_name\": \"Site one\",\n \"hts_file\": hts_file,\n }\n )(arg1, arg2)\n\nWhen using Annalist in a class method, you might want to log class attributes. Unfortunately, the following syntax will not work, since the decorator has no knowledge of the class instance (self).\n\n::\n\n class ExampleClass:\n ...\n\n @function_logger(\n level=\"INFO\",\n message=\"This decorator passes extra parameters\",\n extra_info={\n \"site_name\": self.site_name, # THIS DOES NOT WORK!\n \"hts_file\": self.hts_file, # THIS DOES NOT WORK!\n }\n )\n def example_method(self):\n ...\n\n\nThe way to track class attributes is to use the ``ClassLogger`` decorator. This decorator activates at runtime. Any custom fields passed to the Annalist formatter are noted, and ``ClassLogger`` inspects the class instance for attributes with the same name.\n\n::\n\n from annalist.decorators import ClassLogger\n\n class ExampleClass:\n def __init__(self, attr):\n self.attr = attr\n\n @ClassLogger\n def example_function(self):\n ...\n\nSee annalist.decorators.ClassLogger for more details.\n\nLevels\n--------\n\nAnnalist uses the levels as defined in the logging library. Upon configuration, the ``default level`` can be set, which is the level at which all logs are logged unless overridden. The default value for ``default level`` is \"INFO\".\n\n::\n\n ann.configure(\n analyst_name=\"Speve\",\n stream_format_str=format_str,\n level_filter=\"WARNING\",\n )\n\nA annalized method can be logged at a raised or lowered level by specifying the logging level explicitely in the decorator:\n\n::\n\n @function_logger(level=\"DEBUG\")\n def untracked_function():\n ...\n\n==================\nFeature Roadmap\n==================\n\nThis roadmap outlines the planned features and milestones for the development of our deterministic and reproducible process auditing system.\n\nMilestone 1: Audit Logging Framework\n------------------------------------\n\nx Develop a custom audit logging framework or class.\nx Capture function names, input parameters, return values, data types, and timestamps.\nx Implement basic logging mechanisms for integration.\n\nMilestone 1.5: Hilltop Auditing Parity\n---------------------------------------\nx Define custom fields and formatters\nx Manage logger levels correctly\n\nMilestone 2: Standardized Logging Format\n-----------------------------------------\n- Define a standardized logging format for comprehensive auditing.\n- Ensure consistency and machine-readability of the logging format.\n\nMilestone 3: Serialization and Deserialization\n----------------------------------------------\n- Implement serialization and deserialization mechanisms.\n- Store and retrieve complex data structures and objects.\n- Test serialization for data integrity.\n\nMilestone 4: Versioning and Dependency Tracking\n-----------------------------------------------\n- Capture and log codebase version (Git commit hash) and dependencies.\n- Ensure accurate logging of version and dependency information.\n\nMilestone 5: Integration Testing\n--------------------------------\n- Create integration tests using the audit logging framework.\n- Log information during the execution of key processes.\n- Begin development of process recreation capability.\n\nMilestone 6: Reproduction Tool (Partial)\n----------------------------------------\n- Develop a tool or script to read and reproduce processes from the audit trail.\n- Focus on recreating the environment and loading serialized data.\n\nMilestone 7: Documentation (Partial)\n--------------------------------------\n- Create initial documentation.\n- Explain how to use the audit logging framework and the audit trail format.\n- Document basic project functionalities.\n\nMilestone 8: Error Handling\n---------------------------\n- Implement robust error handling for auditing and reproduction code.\n- Gracefully handle potential issues.\n- Provide informative and actionable error messages.\n\nMilestone 9: MVP Testing\n-------------------------\n- Conduct testing of the MVP.\n- Reproduce processes from the audit trail and verify correctness.\n- Gather feedback from initial users within the organization.\n\nMilestone 10: MVP Deployment\n------------------------------\n- Deploy the MVP within the organization.\n- Make it available to relevant team members.\n- Encourage usage and collect user feedback.\n\nMilestone 11: Feedback and Iteration\n--------------------------------------\n- Gather feedback from MVP users.\n- Identify shortcomings, usability issues, or missing features.\n- Prioritize and plan improvements based on user feedback.\n\nMilestone 12: Scaling and Extending\n------------------------------------\n- Explore scaling the solution to cover more processes.\n- Add additional features and capabilities to enhance usability.\n\nPlease note that milestones may overlap, and the order can be adjusted based on project-specific needs. We aim to remain flexible and responsive to feedback during development.\n\n=======\nCredits\n=======\n\nThis package was created with Cookiecutter_ and the `audreyr/cookiecutter-pypackage`_ project template.\n\n.. _Cookiecutter: https://github.com/audreyr/cookiecutter\n.. _`audreyr/cookiecutter-pypackage`: https://github.com/audreyr/cookiecutter-pypackage\n",
"bugtrack_url": null,
"license": "GNU General Public License v3",
"summary": "Audit trail generator for data processing scripts.",
"version": "0.4.3",
"project_urls": {
"Documentation": "https://annalist.readthedocs.io",
"Homepage": "https://github.com/nicmostert/annalist",
"Issues": "https://github.com/nicmostert/annalist/issues",
"Package": "https://pypi.org/project/data-annalist"
},
"split_keywords": [
"logging",
" auditing",
" audit trail",
" hydrology",
" automation",
" hilltop",
" hydrobot",
" horizonsrc"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "5dd5ee9715aea8ce54d1f577ef221e691b81481ba9dd5b63e509be0ac1965709",
"md5": "d79d2fe6250df863237ea5f8d4e8cc35",
"sha256": "e5a9489f963be98a723ee7d04e8bb28e0d6ffe2b977c5437ca16a3a01b184769"
},
"downloads": -1,
"filename": "data_annalist-0.4.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "d79d2fe6250df863237ea5f8d4e8cc35",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "==3.11.*",
"size": 14836,
"upload_time": "2024-04-03T02:52:47",
"upload_time_iso_8601": "2024-04-03T02:52:47.928962Z",
"url": "https://files.pythonhosted.org/packages/5d/d5/ee9715aea8ce54d1f577ef221e691b81481ba9dd5b63e509be0ac1965709/data_annalist-0.4.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "b7c54d6b7a365b7418c3d1687207d141aed0841e57bae23c739405a0590bfe2a",
"md5": "8ceefd7393ddf642f6cf5b65472b529f",
"sha256": "ebd81e3d18d11266a5fcc1d6424444274d3b2ef6663f7dbaac4dd50dc73d11da"
},
"downloads": -1,
"filename": "data-annalist-0.4.3.tar.gz",
"has_sig": false,
"md5_digest": "8ceefd7393ddf642f6cf5b65472b529f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "==3.11.*",
"size": 41482,
"upload_time": "2024-04-03T02:52:50",
"upload_time_iso_8601": "2024-04-03T02:52:50.250312Z",
"url": "https://files.pythonhosted.org/packages/b7/c5/4d6b7a365b7418c3d1687207d141aed0841e57bae23c739405a0590bfe2a/data-annalist-0.4.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-04-03 02:52:50",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nicmostert",
"github_project": "annalist",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "data-annalist"
}
JSON
