# Spot Optimizer
[](https://pypi.org/project/spot-optimizer/)
[](https://github.com/amarlearning/spot-optimizer/actions/workflows/python-tests.yml)
[](https://codecov.io/gh/amarlearning/spot-optimizer)
[](https://pypi.org/project/spot-optimizer/)
[](https://www.apache.org/licenses/LICENSE-2.0.txt)
[](https://pepy.tech/projects/spot-optimizer)
🚀 Spot Optimizer is a Python library that helps users select the best AWS spot instances based on their resource requirements, including cores, RAM, storage type (SSD), instance architecture (x86 or ARM), AWS region, EMR version compatibility, and instance family preferences.
It replaces complex, in-house logic for finding the best spot instances with a simple and powerful abstraction. No more manual guesswork — just the right instances at the right time.
## Why Spot Optimizer?
Managing spot instance selection within your codebase can be tedious and error-prone. Spot Optimizer provides a clean, abstracted solution to intelligently select the most stable and cost-effective instances.
### Configuration Guarantee
It ensures that the selected configuration meets or exceeds the user's requirements. For example, if you request 20 cores and 100GB of RAM, the library will suggest a configuration with at least those resources, rounding up to the nearest available configuration.
---
## Key Benefits
- **💡 Informed Decisions**: Picks instances with the lowest interruption rates and the best fit for your workload.
- **🧠 Dynamic Reliability**: Smartly updates interruption rates every hour to ensure the most stable instance selection.
- **🛠️ Operational Efficiency**: No more homegrown scripts or complex logic — just plug and play.
- **⚡ High Flexibility**: Supports diverse use cases like Spark/EMR clusters, ML workloads, gaming servers, and more.
- **🏗️ Scalable and Reliable**: Automatically adjusts to changing resource needs while minimizing downtime.
---
## Installation
### For Users
```bash
pip install spot-optimizer
```
### For Development
```bash
# Clone the repository
git clone git@github.com:amarlearning/spot-optimizer.git
cd spot-optimizer
# Install dependencies and set up development environment
make install
```
---
## Usage
### API Usage
```python
from spot_optimizer import optimize
# Basic usage
result = optimize(cores=8, memory=32)
# Advanced usage with all options
result = optimize(
cores=8,
memory=32,
region="us-east-1",
ssd_only=True,
arm_instances=False,
instance_family=["m6i", "r6i"],
mode="balanced"
)
# output
{
"instances": {
"type": "m6i.2xlarge",
"count": 1
},
"mode": "balanced",
"total_cores": 8,
"total_ram": 32
}
```
### CLI Usage
```bash
# Basic usage
spot-optimizer --cores 8 --memory 32
# Advanced usage
spot-optimizer \
--cores 8 \
--memory 32 \
--region us-east-1 \
--ssd-only \
--no-arm \
--instance-family m6i r6i \
--mode balanced
# Get help
spot-optimizer --help
```
---
## Inputs
### Required Parameters
1. **cores (int)**: The total number of CPU cores required.
2. **memory (int)**: The total amount of memory required in GB.
### Optional Parameters
1. **region (str)**: AWS region for spot instance selection (default: "us-west-2").
2. **ssd_only (bool)**: If `True`, only suggest instances with SSD-backed storage (default: False).
3. **arm_instances (bool)**: If `True`, include ARM-based instances (default: True).
4. **instance_family (List[str])**: Filter by specific instance families (e.g., ['m6i', 'r6i']).
5. **emr_version (str)**: Optional EMR version to ensure instance compatibility.
6. **mode (str)**:
- **`latency`**: Optimize for fewer, larger nodes (lower latency).
- **`fault_tolerance`**: Optimize for more, smaller nodes (better fault tolerance).
- **`balanced`**: Aim for a middle ground between fewer and more nodes.
---
## Future Enhancements
1. **Cost Optimization**:
- Include estimated instance costs and recommend the most cost-effective configuration.
2. **Support for Other Cloud Providers**:
- Extend the library to support GCP and Azure instance types.
3. **Spot Interruption Rates**:
- Include interruption rates in the selection criteria for spot instances.
---
## Development
### Make Commands
```bash
# Install dependencies
make install
# Run tests
make test
# Check test coverage
make coverage
# Clean up build artifacts
make clean
```
---
# Performance Optimisations
- Efficiently updates the instance interruption table only every hour, avoiding unnecessary data fetches.
- Focuses on providing the most stable instances based on the latest interruption rate data.
---
## Performance Benchmarks
Performance tests were run on GitHub Actions runner (2 vCPU, 7GB RAM) with 64,295 different combinations of resource requirements and constraints.
### Quick Stats
- **Total Combinations Tested**: 64,295
- **Total Processing Time**: 338.88 seconds
- **Cache Preparation Time**: 19.30 seconds
### Query Performance
| Metric | Time (ms) |
|--------|-----------|
| Average | 5.4 |
| Minimum | 4.1 |
| Maximum | 21.5 |
| Median | 5.3 |
| 95th Percentile | 6.4 |
| Standard Deviation | 0.6 |
### Throughput
- **Average Processing Rate**: ~190 queries/second
- **Effective Throughput**: 64,295 combinations in 338.88 seconds
> Note: These benchmarks were run on GitHub Actions' standard runner (2 vCPU, 7GB RAM). Performance in production environments will likely be better with dedicated hardware and more resources.
---
## Issues
If you encounter any bugs, please report them on the [issue tracker](https://github.com/amarlearning/spark-cluster-optimiser/issues).
Alternatively, feel free to [tweet me](https://twitter.com/iamarpandey) if you're having trouble. In fact, you should tweet me anyway.
---
## License
Built with ♥ by Amar Prakash Pandey([@amarlearning](http://github.com/amarlearning)) under Apache License 2.0.
Raw data
{
"_id": null,
"home_page": "https://github.com/amarlearning/spot-optimizer",
"name": "spot-optimizer",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "aws, ec2, spot, optimizer",
"author": "Amar Prakash",
"author_email": "amar.om1994@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/eb/9d/332e196a92cea71441df2543325765b055116e2309b29269d1b2fc019ade/spot_optimizer-1.2.0.tar.gz",
"platform": null,
"description": "# Spot Optimizer\n\n[](https://pypi.org/project/spot-optimizer/)\n[](https://github.com/amarlearning/spot-optimizer/actions/workflows/python-tests.yml)\n[](https://codecov.io/gh/amarlearning/spot-optimizer)\n[](https://pypi.org/project/spot-optimizer/)\n[](https://www.apache.org/licenses/LICENSE-2.0.txt)\n[](https://pepy.tech/projects/spot-optimizer)\n\n\n\ud83d\ude80 Spot Optimizer is a Python library that helps users select the best AWS spot instances based on their resource requirements, including cores, RAM, storage type (SSD), instance architecture (x86 or ARM), AWS region, EMR version compatibility, and instance family preferences. \n\nIt replaces complex, in-house logic for finding the best spot instances with a simple and powerful abstraction. No more manual guesswork \u2014 just the right instances at the right time. \n\n## Why Spot Optimizer?\nManaging spot instance selection within your codebase can be tedious and error-prone. Spot Optimizer provides a clean, abstracted solution to intelligently select the most stable and cost-effective instances.\n\n### Configuration Guarantee\nIt ensures that the selected configuration meets or exceeds the user's requirements. For example, if you request 20 cores and 100GB of RAM, the library will suggest a configuration with at least those resources, rounding up to the nearest available configuration.\n\n---\n\n## Key Benefits\n- **\ud83d\udca1 Informed Decisions**: Picks instances with the lowest interruption rates and the best fit for your workload.\n- **\ud83e\udde0 Dynamic Reliability**: Smartly updates interruption rates every hour to ensure the most stable instance selection.\n- **\ud83d\udee0\ufe0f Operational Efficiency**: No more homegrown scripts or complex logic \u2014 just plug and play.\n- **\u26a1 High Flexibility**: Supports diverse use cases like Spark/EMR clusters, ML workloads, gaming servers, and more.\n- **\ud83c\udfd7\ufe0f Scalable and Reliable**: Automatically adjusts to changing resource needs while minimizing downtime.\n\n---\n\n## Installation\n\n### For Users\n```bash\npip install spot-optimizer\n```\n\n### For Development\n```bash\n# Clone the repository\ngit clone git@github.com:amarlearning/spot-optimizer.git\ncd spot-optimizer\n\n# Install dependencies and set up development environment\nmake install\n```\n\n---\n\n## Usage\n\n### API Usage\n\n```python\nfrom spot_optimizer import optimize\n\n# Basic usage\nresult = optimize(cores=8, memory=32)\n\n# Advanced usage with all options\nresult = optimize(\n cores=8,\n memory=32,\n region=\"us-east-1\",\n ssd_only=True,\n arm_instances=False,\n instance_family=[\"m6i\", \"r6i\"],\n mode=\"balanced\"\n)\n\n# output\n{\n \"instances\": {\n \"type\": \"m6i.2xlarge\",\n \"count\": 1\n },\n \"mode\": \"balanced\",\n \"total_cores\": 8,\n \"total_ram\": 32\n}\n\n```\n\n### CLI Usage\n\n```bash\n# Basic usage\nspot-optimizer --cores 8 --memory 32\n\n# Advanced usage\nspot-optimizer \\\n --cores 8 \\\n --memory 32 \\\n --region us-east-1 \\\n --ssd-only \\\n --no-arm \\\n --instance-family m6i r6i \\\n --mode balanced\n\n# Get help\nspot-optimizer --help\n```\n\n---\n\n## Inputs\n\n### Required Parameters\n\n1. **cores (int)**: The total number of CPU cores required.\n2. **memory (int)**: The total amount of memory required in GB.\n\n### Optional Parameters\n\n1. **region (str)**: AWS region for spot instance selection (default: \"us-west-2\").\n2. **ssd_only (bool)**: If `True`, only suggest instances with SSD-backed storage (default: False).\n3. **arm_instances (bool)**: If `True`, include ARM-based instances (default: True).\n4. **instance_family (List[str])**: Filter by specific instance families (e.g., ['m6i', 'r6i']).\n5. **emr_version (str)**: Optional EMR version to ensure instance compatibility.\n6. **mode (str)**:\n - **`latency`**: Optimize for fewer, larger nodes (lower latency).\n - **`fault_tolerance`**: Optimize for more, smaller nodes (better fault tolerance).\n - **`balanced`**: Aim for a middle ground between fewer and more nodes.\n\n---\n\n## Future Enhancements\n\n1. **Cost Optimization**:\n - Include estimated instance costs and recommend the most cost-effective configuration.\n2. **Support for Other Cloud Providers**:\n - Extend the library to support GCP and Azure instance types.\n3. **Spot Interruption Rates**:\n - Include interruption rates in the selection criteria for spot instances.\n\n---\n\n## Development\n\n### Make Commands\n\n```bash\n# Install dependencies\nmake install\n\n# Run tests\nmake test\n\n# Check test coverage\nmake coverage\n\n# Clean up build artifacts\nmake clean\n```\n\n---\n\n# Performance Optimisations\n\n- Efficiently updates the instance interruption table only every hour, avoiding unnecessary data fetches.\n- Focuses on providing the most stable instances based on the latest interruption rate data.\n\n---\n\n## Performance Benchmarks\n\nPerformance tests were run on GitHub Actions runner (2 vCPU, 7GB RAM) with 64,295 different combinations of resource requirements and constraints.\n\n### Quick Stats\n- **Total Combinations Tested**: 64,295\n- **Total Processing Time**: 338.88 seconds\n- **Cache Preparation Time**: 19.30 seconds\n\n### Query Performance\n| Metric | Time (ms) |\n|--------|-----------|\n| Average | 5.4 |\n| Minimum | 4.1 |\n| Maximum | 21.5 |\n| Median | 5.3 |\n| 95th Percentile | 6.4 |\n| Standard Deviation | 0.6 |\n\n### Throughput\n- **Average Processing Rate**: ~190 queries/second\n- **Effective Throughput**: 64,295 combinations in 338.88 seconds\n\n> Note: These benchmarks were run on GitHub Actions' standard runner (2 vCPU, 7GB RAM). Performance in production environments will likely be better with dedicated hardware and more resources.\n\n---\n\n## Issues\n\nIf you encounter any bugs, please report them on the [issue tracker](https://github.com/amarlearning/spark-cluster-optimiser/issues).\nAlternatively, feel free to [tweet me](https://twitter.com/iamarpandey) if you're having trouble. In fact, you should tweet me anyway.\n\n---\n\n## License\n\nBuilt with \u2665 by Amar Prakash Pandey([@amarlearning](http://github.com/amarlearning)) under Apache License 2.0. \n",
"bugtrack_url": null,
"license": null,
"summary": "AWS EC2 spot instance optimizer",
"version": "1.2.0",
"project_urls": {
"Homepage": "https://github.com/amarlearning/spot-optimizer",
"Repository": "https://github.com/amarlearning/spot-optimizer"
},
"split_keywords": [
"aws",
" ec2",
" spot",
" optimizer"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "cb7c32e004e76ef3ca8f54c4318d10e743adaf6acc93f8cbeb91a3802e4d227a",
"md5": "f1e184715b2e7411a4ea801c01ef876d",
"sha256": "3225e1a4ebe2b636804c9899ea88ce9775b86d01b233506f35c0612816173c84"
},
"downloads": -1,
"filename": "spot_optimizer-1.2.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f1e184715b2e7411a4ea801c01ef876d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 26486,
"upload_time": "2025-09-01T17:07:54",
"upload_time_iso_8601": "2025-09-01T17:07:54.939840Z",
"url": "https://files.pythonhosted.org/packages/cb/7c/32e004e76ef3ca8f54c4318d10e743adaf6acc93f8cbeb91a3802e4d227a/spot_optimizer-1.2.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "eb9d332e196a92cea71441df2543325765b055116e2309b29269d1b2fc019ade",
"md5": "3967cec810d4aac0e53c81839b9872ab",
"sha256": "f4b170c919625980dee51b451a68184339dfd150645c3b1b49e2d7d259a1dfae"
},
"downloads": -1,
"filename": "spot_optimizer-1.2.0.tar.gz",
"has_sig": false,
"md5_digest": "3967cec810d4aac0e53c81839b9872ab",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 24618,
"upload_time": "2025-09-01T17:07:55",
"upload_time_iso_8601": "2025-09-01T17:07:55.730162Z",
"url": "https://files.pythonhosted.org/packages/eb/9d/332e196a92cea71441df2543325765b055116e2309b29269d1b2fc019ade/spot_optimizer-1.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-01 17:07:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "amarlearning",
"github_project": "spot-optimizer",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "spot-optimizer"
}
JSON
