# ScholarSparkObservability
A Python package for monitoring and observability in Apache Spark applications.
## Installation
You can install the package using pip:
```bash
pip install scholarSparkObservability
```
Or using Poetry:
```bash
poetry add scholarSparkObservability
```
## Usage
Here's a simple example of how to use the package:
```python
from scholarSparkObservability import ScholarSparkObservability
client = ScholarSparkObservability()
result = client.example_method()
print(result)
```
## Features
## Features
### OpenTelemetry Integration
- Full OpenTelemetry support for distributed tracing and metrics collection
- Configurable exporters for different observability backends (e.g., Jaeger, Zipkin)
- Automatic context propagation across Spark jobs and stages
### Comprehensive Monitoring
- Real-time metrics collection for Spark executors and tasks
- Custom span creation for detailed performance tracking
- Exception tracking and error reporting with detailed attributes
- Resource utilization metrics (CPU, memory, I/O)
### Flexible Configuration
- Singleton pattern for consistent telemetry setup across your application
- Environment-aware configuration (production, staging, development)
- Customizable export intervals and batch processing
- Debug mode for detailed logging and troubleshooting
### Enterprise-Ready
- Low-overhead implementation suitable for production workloads
- Batch span processing for efficient telemetry data export
- Support for multiple exporters and monitoring backends
- Robust error handling and logging capabilities
### Easy Integration
- Simple API for creating spans and recording metrics
- Automatic service name and version tracking
- Built-in support for custom attributes and tags
- Seamless integration with existing Spark applications
## Development
To contribute to this project:
1. Clone the repository:
```bash
git clone https://github.com/pouyaardehkhani/ScholarSparkObservability.git
cd ScholarSparkObservability
```
2. Install dependencies:
```bash
# Using poetry (recommended)
poetry install
# Using pip
pip install -r requirements.txt
```
3. Run tests:
```bash
# Using poetry
poetry run pytest
# Using pytest directly
pytest tests/
```
4. Set up pre-commit hooks:
```bash
pre-commit install
```
5. Create a new branch for your feature:
```bash
git checkout -b feature/your-feature-name
```
### Development Guidelines
- Follow PEP 8 style guidelines
- Write tests for new features
- Update documentation as needed
- Add type hints to all new functions
- Ensure all tests pass before submitting PR
### Building Documentation
```bash
# Generate documentation
poetry run sphinx-build -b html docs/source docs/build
```
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
## License
This project is licensed under the MIT License - see the LICENSE file for details.
# Authors
Pouya Ataei- Initial work
Raw data
{
"_id": null,
"home_page": "https://github.com/Polyhistor/ScholarSparkObservability",
"name": "scholar-spark-observability",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.8",
"maintainer_email": null,
"keywords": "spark, observability, monitoring, opentelemetry",
"author": "Pouya Ataei",
"author_email": "pouya.ataei.7@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/0e/5c/788bd10a9c63bfe1ee19ecef479ad4178c3e0483403759a81d2021683eb6/scholar_spark_observability-0.8.0.tar.gz",
"platform": null,
"description": "# ScholarSparkObservability\n\nA Python package for monitoring and observability in Apache Spark applications.\n\n\n## Installation\n\nYou can install the package using pip:\n\n```bash\npip install scholarSparkObservability\n```\n\nOr using Poetry:\n\n```bash\npoetry add scholarSparkObservability\n```\n\n## Usage\n\nHere's a simple example of how to use the package:\n\n```python\nfrom scholarSparkObservability import ScholarSparkObservability\n\nclient = ScholarSparkObservability()\nresult = client.example_method()\nprint(result)\n```\n\n## Features\n\n## Features\n\n### OpenTelemetry Integration\n- Full OpenTelemetry support for distributed tracing and metrics collection\n- Configurable exporters for different observability backends (e.g., Jaeger, Zipkin)\n- Automatic context propagation across Spark jobs and stages\n\n### Comprehensive Monitoring\n- Real-time metrics collection for Spark executors and tasks\n- Custom span creation for detailed performance tracking\n- Exception tracking and error reporting with detailed attributes\n- Resource utilization metrics (CPU, memory, I/O)\n\n### Flexible Configuration\n- Singleton pattern for consistent telemetry setup across your application\n- Environment-aware configuration (production, staging, development)\n- Customizable export intervals and batch processing\n- Debug mode for detailed logging and troubleshooting\n\n### Enterprise-Ready\n- Low-overhead implementation suitable for production workloads\n- Batch span processing for efficient telemetry data export\n- Support for multiple exporters and monitoring backends\n- Robust error handling and logging capabilities\n\n### Easy Integration\n- Simple API for creating spans and recording metrics\n- Automatic service name and version tracking\n- Built-in support for custom attributes and tags\n- Seamless integration with existing Spark applications\n\n## Development\n\nTo contribute to this project:\n\n1. Clone the repository:\n```bash\ngit clone https://github.com/pouyaardehkhani/ScholarSparkObservability.git\ncd ScholarSparkObservability\n```\n\n2. Install dependencies:\n```bash\n# Using poetry (recommended)\npoetry install\n\n# Using pip\npip install -r requirements.txt\n```\n\n3. Run tests:\n```bash\n# Using poetry\npoetry run pytest\n\n# Using pytest directly\npytest tests/\n```\n\n4. Set up pre-commit hooks:\n```bash\npre-commit install\n```\n\n5. Create a new branch for your feature:\n```bash\ngit checkout -b feature/your-feature-name\n```\n\n### Development Guidelines\n- Follow PEP 8 style guidelines\n- Write tests for new features\n- Update documentation as needed\n- Add type hints to all new functions\n- Ensure all tests pass before submitting PR\n\n### Building Documentation\n```bash\n# Generate documentation\npoetry run sphinx-build -b html docs/source docs/build\n```\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request.\n\n## License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n# Authors\nPouya Ataei- Initial work\n\n",
"bugtrack_url": null,
"license": null,
"summary": "A Python package for monitoring and observability in Apache Spark applications",
"version": "0.8.0",
"project_urls": {
"Documentation": "https://scholar-spark-observability.readthedocs.io",
"Homepage": "https://github.com/Polyhistor/ScholarSparkObservability",
"Repository": "https://github.com/Polyhistor/ScholarSparkObservability"
},
"split_keywords": [
"spark",
" observability",
" monitoring",
" opentelemetry"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8e3600027fadb97ee021a780b53c17ffd9ef1617a14fa4f931d5f3c701a5afbf",
"md5": "65c5804df405737df3d6d57e696cb4ba",
"sha256": "106b77dc5eae427b1a243183a1fa8746ba3af949292553b35addce3d9180e554"
},
"downloads": -1,
"filename": "scholar_spark_observability-0.8.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "65c5804df405737df3d6d57e696cb4ba",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.8",
"size": 5243,
"upload_time": "2024-12-30T07:50:22",
"upload_time_iso_8601": "2024-12-30T07:50:22.790160Z",
"url": "https://files.pythonhosted.org/packages/8e/36/00027fadb97ee021a780b53c17ffd9ef1617a14fa4f931d5f3c701a5afbf/scholar_spark_observability-0.8.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0e5c788bd10a9c63bfe1ee19ecef479ad4178c3e0483403759a81d2021683eb6",
"md5": "57caa3f0469b83fee5d854586285f66d",
"sha256": "5bfdf6db099fc79b6c947e247bc91601d1a5797189ebb0c57994eb37ecc0f757"
},
"downloads": -1,
"filename": "scholar_spark_observability-0.8.0.tar.gz",
"has_sig": false,
"md5_digest": "57caa3f0469b83fee5d854586285f66d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.8",
"size": 4519,
"upload_time": "2024-12-30T07:50:23",
"upload_time_iso_8601": "2024-12-30T07:50:23.935357Z",
"url": "https://files.pythonhosted.org/packages/0e/5c/788bd10a9c63bfe1ee19ecef479ad4178c3e0483403759a81d2021683eb6/scholar_spark_observability-0.8.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-12-30 07:50:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Polyhistor",
"github_project": "ScholarSparkObservability",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "scholar-spark-observability"
}
JSON
