# logsteplib
- [Description](#package-description)
- [Usage](#usage)
- [Installation](#installation)
- [Docstring](#docstring)
- [License](#license)
## Package Description
Package containing a standard format for the logging module.
## Usage
### Stream Console Logs
From a script:
```python
# Initialise a logger for the process and log an informational message
from logsteplib.streamer import StreamLogger
logger = StreamLogger(name="my_process").logger
logger.info(msg="Something to log!")
# 2025-11-02 00:00:01 - my_process - INFO - Something to log!
```
### Lakehouse DQ Logs
From a SQL script:
```sql
-- Create the Delta lakehouse table for tracking SharePoint file uploads
-- Includes metadata such as file details, user info, and processing status
-- Table (example): workspace.default.sharepoint_uploader_monitoring_logs
DROP TABLE IF EXISTS workspace.default.sharepoint_uploader_monitoring_logs;
CREATE TABLE workspace.default.sharepoint_uploader_monitoring_logs (
target STRING,
key STRING,
input_file_name STRING,
file_name STRING,
user_name STRING,
user_email STRING,
modify_date STRING, -- Should be timestamp
file_size STRING, -- Should be INT
file_row_count STRING, -- Should be INT
status STRING,
rejection_reason STRING,
file_web_url STRING
)
USING DELTA;
```
From a Python script:
```python
from logsteplib.dq import DQStatusCode
print(DQStatusCode.get_description("SchemaMismatch")) # DQ FAIL: SCHEMA MISMATCH
print(DQStatusCode.get_description("UnknownCode")) # UNKNOWN STATUS CODE
```
Status Code Table
| Code | Description |
| -------------------------- | --------------------------------------- |
| NA | NOT APPLICABLE |
| EmptyFile | DQ FAIL: EMPTY FILE |
| SchemaMismatch | DQ FAIL: SCHEMA MISMATCH |
| SchemaMismatchAndEmptyFile | DQ FAIL: SCHEMA MISMATCH AND EMPTY FILE |
| InvalidNumericFormat | DQ FAIL: INVALID NUMERIC FORMAT |
| InvalidDateFormat | DQ FAIL: INVALID DATE FORMAT |
```python
from logsteplib.dq import DQWriter
from logsteplib.dq import DQMetadata
# Init DQWriter
monitoring_table = "workspace.default.sharepoint_uploader_monitoring_logs"
dq_writer = DQWriter(table_name=monitoring_table)
```python
# Create a DQMetadata instance containing metadata about a processed file
# This metadata can be used for logging, auditing, or writing to a lakehouse table
metadata = DQMetadata(
target="my_folder/my_system",
key="customer_20251031",
input_file_name="raw_customers.csv",
file_name="clean_customers.csv",
user_name="Parker, Peter",
user_email="peter.parker@example.com",
modify_date="2025-11-02",
file_size="204800",
file_row_count="15000",
status="FAIL",
rejection_reason=DQStatusCode.get_description("SchemaMismatch"),
file_web_url="https://lakehouse.company.com/files/clean_customers.parquet"
)
# Write the metadata (DQMetadata) to the lakehouse monitoring table
dq_writer.write_metadata(metadata=metadata)
```
### Email Notifications
```python
from logsteplib.notifier import EmailNotifier
# Init email notifications
notifier = EmailNotifier(
client_id=client_id,
client_secret=client_secret,
tenant_id=tenant_id,
client_email="sender@example.com"
)
# Send email notification
response = notifier.send_email(recipients=["peter.parker@example.com"],
subject="Notification X",
message="This is<br>a test...",
attachments=None)
if response.status_code == 200:
print("Email sent")
```
```python
# Using pre-defined templates
from email_templates import EmailTemplates
# Parameters
recipient_name = "Peter Parker"
error_message = "The web is crashing."
# Generate an email using the technical template
message = EmailTemplates.technical_error(recipient_name, error_message)
# Print result
print(message) # Dear Peter Parker...
```
## Installation
Install python and pip if you have not already.
Then run:
```bash
pip install pip --upgrade
```
For production:
```bash
pip install logsteplib
```
This will install the package and all of it's python dependencies.
If you want to install the project for development:
```bash
git clone https://github.com/aghuttun/logsteplib.git
cd logsteplib
pip install -e ".[dev]"
```
## Docstring
The script's docstrings follow the numpydoc style.
## License
BSD License (see license file)
[top](#logsteplib)
Raw data
{
"_id": null,
"home_page": null,
"name": "logsteplib",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "Paulo Portela <portela.paulo@gmail.com>",
"keywords": "Logging, Log, Python, Configuration, Lakehouse, DataBricks",
"author": null,
"author_email": "Paulo Portela <portela.paulo@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/5c/b7/ddbe9554eab49ce248bf040617fbb17d7b649251f8ea661473860f1109df/logsteplib-0.0.18.tar.gz",
"platform": null,
"description": "# logsteplib\r\n\r\n- [Description](#package-description)\r\n- [Usage](#usage)\r\n- [Installation](#installation)\r\n- [Docstring](#docstring)\r\n- [License](#license)\r\n\r\n## Package Description\r\n\r\nPackage containing a standard format for the logging module.\r\n\r\n## Usage\r\n\r\n### Stream Console Logs\r\n\r\nFrom a script:\r\n\r\n```python\r\n# Initialise a logger for the process and log an informational message\r\nfrom logsteplib.streamer import StreamLogger\r\n\r\nlogger = StreamLogger(name=\"my_process\").logger\r\n\r\nlogger.info(msg=\"Something to log!\")\r\n# 2025-11-02 00:00:01 - my_process - INFO - Something to log!\r\n```\r\n\r\n### Lakehouse DQ Logs\r\n\r\nFrom a SQL script:\r\n\r\n```sql\r\n-- Create the Delta lakehouse table for tracking SharePoint file uploads\r\n-- Includes metadata such as file details, user info, and processing status\r\n-- Table (example): workspace.default.sharepoint_uploader_monitoring_logs\r\nDROP TABLE IF EXISTS workspace.default.sharepoint_uploader_monitoring_logs;\r\nCREATE TABLE workspace.default.sharepoint_uploader_monitoring_logs (\r\n target STRING,\r\n key STRING,\r\n input_file_name STRING,\r\n file_name STRING,\r\n user_name STRING,\r\n user_email STRING,\r\n modify_date STRING, -- Should be timestamp\r\n file_size STRING, -- Should be INT\r\n file_row_count STRING, -- Should be INT\r\n status STRING,\r\n rejection_reason STRING,\r\n file_web_url STRING\r\n)\r\nUSING DELTA;\r\n```\r\n\r\nFrom a Python script:\r\n\r\n```python\r\nfrom logsteplib.dq import DQStatusCode\r\n\r\nprint(DQStatusCode.get_description(\"SchemaMismatch\")) # DQ FAIL: SCHEMA MISMATCH\r\nprint(DQStatusCode.get_description(\"UnknownCode\")) # UNKNOWN STATUS CODE\r\n```\r\n\r\nStatus Code Table\r\n\r\n| Code | Description |\r\n| -------------------------- | --------------------------------------- |\r\n| NA | NOT APPLICABLE |\r\n| EmptyFile | DQ FAIL: EMPTY FILE |\r\n| SchemaMismatch | DQ FAIL: SCHEMA MISMATCH |\r\n| SchemaMismatchAndEmptyFile | DQ FAIL: SCHEMA MISMATCH AND EMPTY FILE |\r\n| InvalidNumericFormat | DQ FAIL: INVALID NUMERIC FORMAT |\r\n| InvalidDateFormat | DQ FAIL: INVALID DATE FORMAT |\r\n\r\n```python\r\nfrom logsteplib.dq import DQWriter\r\nfrom logsteplib.dq import DQMetadata\r\n\r\n# Init DQWriter\r\nmonitoring_table = \"workspace.default.sharepoint_uploader_monitoring_logs\"\r\ndq_writer = DQWriter(table_name=monitoring_table)\r\n\r\n```python\r\n# Create a DQMetadata instance containing metadata about a processed file\r\n# This metadata can be used for logging, auditing, or writing to a lakehouse table\r\nmetadata = DQMetadata(\r\n target=\"my_folder/my_system\",\r\n key=\"customer_20251031\",\r\n input_file_name=\"raw_customers.csv\",\r\n file_name=\"clean_customers.csv\",\r\n user_name=\"Parker, Peter\",\r\n user_email=\"peter.parker@example.com\",\r\n modify_date=\"2025-11-02\",\r\n file_size=\"204800\",\r\n file_row_count=\"15000\",\r\n status=\"FAIL\",\r\n rejection_reason=DQStatusCode.get_description(\"SchemaMismatch\"),\r\n file_web_url=\"https://lakehouse.company.com/files/clean_customers.parquet\"\r\n)\r\n\r\n# Write the metadata (DQMetadata) to the lakehouse monitoring table\r\ndq_writer.write_metadata(metadata=metadata)\r\n```\r\n\r\n### Email Notifications\r\n\r\n```python\r\nfrom logsteplib.notifier import EmailNotifier\r\n\r\n# Init email notifications\r\nnotifier = EmailNotifier(\r\n client_id=client_id,\r\n client_secret=client_secret,\r\n tenant_id=tenant_id,\r\n client_email=\"sender@example.com\"\r\n)\r\n\r\n# Send email notification\r\nresponse = notifier.send_email(recipients=[\"peter.parker@example.com\"],\r\n subject=\"Notification X\",\r\n message=\"This is<br>a test...\",\r\n attachments=None)\r\nif response.status_code == 200:\r\n print(\"Email sent\")\r\n```\r\n\r\n```python\r\n# Using pre-defined templates\r\nfrom email_templates import EmailTemplates\r\n\r\n# Parameters\r\nrecipient_name = \"Peter Parker\"\r\nerror_message = \"The web is crashing.\"\r\n\r\n# Generate an email using the technical template\r\nmessage = EmailTemplates.technical_error(recipient_name, error_message)\r\n\r\n# Print result\r\nprint(message) # Dear Peter Parker...\r\n```\r\n\r\n## Installation\r\n\r\nInstall python and pip if you have not already.\r\n\r\nThen run:\r\n\r\n```bash\r\npip install pip --upgrade\r\n```\r\n\r\nFor production:\r\n\r\n```bash\r\npip install logsteplib\r\n```\r\n\r\nThis will install the package and all of it's python dependencies.\r\n\r\nIf you want to install the project for development:\r\n\r\n```bash\r\ngit clone https://github.com/aghuttun/logsteplib.git\r\ncd logsteplib\r\npip install -e \".[dev]\"\r\n```\r\n\r\n## Docstring\r\n\r\nThe script's docstrings follow the numpydoc style.\r\n\r\n## License\r\n\r\nBSD License (see license file)\r\n\r\n[top](#logsteplib)\r\n",
"bugtrack_url": null,
"license": null,
"summary": "Library for logging and data quality tracking (lakehouse).",
"version": "0.0.18",
"project_urls": {
"Documentation": "https://github.com/aghuttun/logsteplib/blob/main/README.md",
"Homepage": "https://github.com/aghuttun/logsteplib"
},
"split_keywords": [
"logging",
" log",
" python",
" configuration",
" lakehouse",
" databricks"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2b1e8c803f084b9b4d6a71ddf3736822b3d04896dba922b441bbf69b51aad23a",
"md5": "dad19563768c889d49e62a3b99e60389",
"sha256": "da8bcfdf074361daab71ed84785b0fa93721cf14ad57809e07c58134ae78bad9"
},
"downloads": -1,
"filename": "logsteplib-0.0.18-py3-none-any.whl",
"has_sig": false,
"md5_digest": "dad19563768c889d49e62a3b99e60389",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 12985,
"upload_time": "2025-11-04T00:29:03",
"upload_time_iso_8601": "2025-11-04T00:29:03.703678Z",
"url": "https://files.pythonhosted.org/packages/2b/1e/8c803f084b9b4d6a71ddf3736822b3d04896dba922b441bbf69b51aad23a/logsteplib-0.0.18-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "5cb7ddbe9554eab49ce248bf040617fbb17d7b649251f8ea661473860f1109df",
"md5": "c78d54a7fb75a45ca5216d762f8f60cb",
"sha256": "3767f01755176e6bd22d437bf64860bb4b657652976871ab61073f6846dfd940"
},
"downloads": -1,
"filename": "logsteplib-0.0.18.tar.gz",
"has_sig": false,
"md5_digest": "c78d54a7fb75a45ca5216d762f8f60cb",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 12813,
"upload_time": "2025-11-04T00:29:05",
"upload_time_iso_8601": "2025-11-04T00:29:05.012713Z",
"url": "https://files.pythonhosted.org/packages/5c/b7/ddbe9554eab49ce248bf040617fbb17d7b649251f8ea661473860f1109df/logsteplib-0.0.18.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-11-04 00:29:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "aghuttun",
"github_project": "logsteplib",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "logsteplib"
}
JSON
