# ๐ฅ OSDU Performance Testing Framework
[](https://www.python.org/downloads/)
[](https://opensource.org/licenses/MIT)
[](https://pypi.org/project/osdu-perf/)
A comprehensive Python framework for performance testing OSDU (Open Subsurface Data Universe) services. Features automatic test discovery, Azure authentication, Locust integration, and both local and cloud-based load testing capabilities with intelligent service orchestration.
## ๐ Key Features
โ
**Service Orchestration** - Intelligent service discovery and execution management
โ
**Azure Authentication** - Seamless Azure AD token management with multiple credential flows
โ
**Dual Execution Modes** - Run locally with Locust or scale with Azure Load Testing
โ
**CLI Tools** - Comprehensive command-line interface with three main commands
โ
**Template System** - Pre-built templates for common OSDU services
โ
**Configuration Management** - YAML-based configuration with environment-aware settings
โ
**Metrics Collection** - Automated metrics push to Azure Data Explorer (Kusto)
โ
**Environment Detection** - Automatically adapts behavior for local vs Azure environments
## ๐๏ธ Framework Architecture
### Core Components
- **`PerformanceUser`**: Locust integration with automatic service discovery
- **`ServiceOrchestrator`**: Plugin architecture for test discovery and execution
- **`BaseService`**: Abstract base class for implementing performance tests
- **`InputHandler`**: Configuration management and environment detection
- **`AzureTokenManager`**: Multi-credential authentication system
## ๐ Quick Start
### Installation
```bash
# Install from PyPI
pip install osdu_perf
```
### Three Simple Commands
The framework provides three main commands for the complete performance testing workflow:
#### 1. Initialize Project (`init`)
```bash
# Create a new performance testing project
osdu_perf init <service_name>
# Examples:
osdu_perf init storage # Creates storage service performance tests
osdu_perf init search # Creates search service performance tests
osdu_perf init wellbore # Creates wellbore service performance tests
```
**What this creates:**
```
perf_tests/
โโโ config.yaml # Framework configuration
โโโ locustfile.py # Main test file with API calls
โโโ requirements.txt # Python dependencies
โโโ README.md # Project documentation
```
#### 2. Run Local Tests (`run local`)
```bash
# Run performance tests locally using Locust
osdu_perf run local --config config.yaml
```
**Features:**
- Uses Locust for load generation
- Azure CLI authentication for local development
- Real-time web UI at http://localhost:8089
- Automatic service discovery and execution
- Automatic metric collection and sends to Kusto
#### 3. Run Azure Load Tests (`run azure_load_test`)
```bash
# Deploy and run tests on Azure Load Testing service
osdu_perf run azure_load_test --config config.yaml
```
**Features:**
- Creates Azure Load Testing resources automatically
- Scales to hundreds/thousands of concurrent users
- Managed Identity authentication in Azure
- Comprehensive metrics and reporting
- Entitlement will be created on ADME for azure load tests
## ๐ ๏ธ Command Reference
### 1. Initialize Command
```bash
osdu_perf init <service_name> [OPTIONS]
```
**Parameters:**
- `service_name`: Name of the OSDU service to test (e.g., storage, search, wellbore)
- `--force`: Force overwrite existing files without prompting
**Examples:**
```bash
osdu_perf init storage # Initialize storage service tests
osdu_perf init search --force # Force overwrite existing search tests
osdu_perf init wellbore # Initialize wellbore service tests
```
**Generated Files:**
- `config.yaml` - Framework configuration with OSDU connection details
- `locustfile.py` - Main test file with API calls to your service
- `requirements.txt` - Python dependencies
- `README.md` - Project-specific documentation
### 2. Local Testing Command
```bash
osdu_perf run local [OPTIONS]
```
**Configuration:**
- Uses `config.yaml` for base configuration
- CLI arguments override config file settings
- Environment variables provide runtime values
**Key Options:**
- `--config`: Path to config.yaml file (required)
- `--host`: OSDU host URL (overrides config)
- `--partition`: OSDU data partition ID (overrides config)
- `--app-id`: Azure AD Application ID (overrides config)
- `--users` (`-u`): Number of concurrent users (default: from config)
- `--spawn-rate` (`-r`): User spawn rate per second (default: from config)
- `--run-time` (`-t`): Test duration (default: from config)
**Examples:**
```bash
# Basic run using config.yaml
osdu_perf run local --config config.yaml
# Override specific settings
osdu_perf run local --config config.yaml --users 50 --run-time 5m
# Full override
osdu_perf run local \
--config config.yaml \
--host https://api.example.com \
--partition dp1 \
--app-id 12345678-1234-1234-1234-123456789abc \
--users 25 --spawn-rate 5
```
### 3. Azure Load Testing Command
```bash
osdu_perf run azure_load_test [OPTIONS]
```
**Required Parameters:**
- `--config`: Path to config.yaml file
**Optional Parameters:**
- `--loadtest-name`: Azure Load Testing resource name (auto-generated)
- `--test-name`: Test name (auto-generated with timestamp)
- `--engine-instances`: Number of load generator instances (default: from config)
- `--users` (`-u`): Number of concurrent users per instance (default: from config)
- `--run-time` (`-t`): Test duration (default: from config)
**Examples:**
```bash
# Basic Azure Load Test using config
osdu_perf run azure \
--config config.yaml \
--subscription-id "12345678-1234-1234-1234-123456789012" \
--resource-group "myResourceGroup" \
--location "eastus"
# High-scale cloud test
osdu_perf run azure \
--config config.yaml \
--subscription-id "12345678-1234-1234-1234-123456789012" \
--resource-group "myResourceGroup" \
--location "eastus" \
--users 100 --engine-instances 5 --run-time 30m
```
## ๐ Configuration System
### config.yaml Structure
The framework uses a centralized configuration file that supports both local and Azure environments:
```yaml
# OSDU Environment Configuration
osdu_environment:
# OSDU instance details (required for run local command)
host: "https://your-osdu-host.com"
partition: "your-partition-id"
app_id: "your-azure-app-id"
# OSDU deployment details (optional - used for metrics collection)
sku: "Standard"
version: "25.2.35"
# Authentication (optional - uses automatic token generation if not provided)
auth:
# Manual token override (optional)
token: ""
# Metrics Collection Configuration
metrics_collector:
# Kusto (Azure Data Explorer) Configuration
kusto:
cluster: "https://your-kusto-cluster.eastus.kusto.windows.net"
database: "your-database"
ingest_uri: "https://ingest-your-kusto.eastus.kusto.windows.net"
# Test Configuration (Optional)
test_settings:
# Azure Load Test resource and test locations
subscription_id: "your-azure-subscription-id"
resource_group: "your-resource-group"
location: "eastus"
# Test-specific configurations
default_wait_time:
min: 1
max: 3
users: 10
spawn_rate: 2
run_time: "60s"
engine_instances: 1
test_name_prefix: "osdu_perf_test"
test_scenario: "health_check"
test_run_id_description: "Automated performance test"
```
### Configuration Hierarchy
The framework uses a layered configuration approach:
1. **config.yaml** (project-specific settings)
2. **CLI arguments** (highest priority)
## ๐๏ธ How It Works
### ๐ Simple API-Based Approach
The framework now uses a simplified API-based approach where developers write test methods directly in `locustfile.py`:
```
perf_tests/
โโโ locustfile.py โ OSDUUser class with @task methods for testing
โโโ config.yaml โ Configuration for host, partition, authentication
โโโ requirements.txt โ Dependencies (osdu_perf package)
```
**Simplified Process:**
1. `osdu_perf init <service>` generates `locustfile.py` template
2. Developers add `@task` methods with API calls (`self.get()`, `self.post()`, etc.)
3. `PerformanceUser` base class handles authentication, headers, tokens automatically
4. Run with `osdu_perf run local` or `osdu_perf run azure_load_test`
### ๐ฏ Smart Resource Naming
Based on detected services, Azure resources are automatically named:
- **Load Test Resource**: `osdu-{service}-loadtest-{timestamp}`
- **Test Name**: `osdu_{service}_test_{timestamp}`
- **Example**: `osdu-storage-loadtest-20241028` with test `osdu_storage_test_20241028_142250`
### ๐ Multi-Environment Authentication
**Local Development:**
- Azure CLI credentials (`az login`)
- Manual token via config or environment variables
- Automatic token refresh and caching
**Azure Load Testing:**
- Managed Identity authentication (no secrets needed)
- Environment variables injected by Azure Load Testing service
- Automatic credential detection and fallback
### ๐ Intelligent Metrics Collection
**Automatic Kusto Integration:**
- Detects environment (local vs Azure) automatically
- Uses appropriate authentication method
- Pushes detailed metrics to three tables:
- `LocustMetrics` - Per-endpoint statistics
- `LocustExceptions` - Error tracking
- `LocustTestSummary` - Overall test summaries
## ๐งช Writing Performance Tests
### Simple API-Based Approach
The framework generate your `locustfile.py`:
```python
"""
OSDU Performance Tests - Locust Configuration
Generated by OSDU Performance Testing Framework
"""
import os
from locust import events, task, tag
from osdu_perf import PerformanceUser
# STEP 1: Register custom CLI args with Locust
@events.init_command_line_parser.add_listener
def add_custom_args(parser):
"""Add OSDU-specific command line arguments"""
parser.add_argument("--partition", type=str, default=os.getenv("PARTITION"), help="OSDU Data Partition ID")
parser.add_argument("--appid", type=str, default=os.getenv("APPID"), help="Azure AD Application ID")
class OSDUUser(PerformanceUser):
"""
OSDU Performance Test User
This class automatically:
- Handles Azure authentication using --appid
- Manages HTTP headers and tokens
- Provides simple API methods for testing
- Manages Locust user simulation and load testing
"""
def on_start(self):
"""Called when a user starts - performs setup"""
super().on_start()
# Access OSDU parameters from Locust parsed options or environment variables
partition = getattr(self.environment.parsed_options, 'partition', None) or os.getenv('PARTITION')
host = getattr(self.environment.parsed_options, 'host', None) or self.host or os.getenv('HOST')
token = os.getenv('ADME_BEARER_TOKEN') # Token only from environment for security
appid = getattr(self.environment.parsed_options, 'appid', None) or os.getenv('APPID')
print(f"๏ฟฝ Started performance testing user")
print(f" ๐ Partition: {partition}")
print(f" ๐ Host: {host}")
print(f" ๐ Token: {'***' if token else 'Not provided'}")
print(f" ๐ App ID: {appid or 'Not provided'}")
@tag("storage", "health_check")
@task(1)
def check_service_health(self):
# Simple API call - framework handles headers, tokens, authentication
self.get("/api/storage/v2/health")
@tag("storage", "health_check")
@task(2)
def test_service_endpoints(self):
# More API calls for your service
self.get("/api/storage/v2/info")
self.post("/api/storage/v2/records", json={"test": "data"})
```
### Key Implementation Points
1. **Inherit from PerformanceUser**: Your class extends `PerformanceUser` which handles all authentication and setup
2. **Use @task decorators**: Mark methods with `@task(weight)` to define test scenarios
3. **Simple HTTP methods**: Use `self.get()`, `self.post()`, `self.put()`, `self.delete()` - framework handles headers/tokens
4. **No manual authentication**: Framework automatically handles Azure AD tokens and HTTP headers
5. **Environment awareness**: Automatically adapts for local vs Azure Load Testing environments
### Available HTTP Methods
The `PerformanceUser` base class provides these simple methods:
```python
# GET request
self.get("/api/storage/v2/records/12345")
# POST request with JSON data
self.post("/api/storage/v2/records", json={
"kind": "osdu:wks:partition:storage:1.0.0",
"data": {"test": "data"}
})
# PUT request
self.put("/api/storage/v2/records/12345", json=updated_data)
# DELETE request
self.delete("/api/storage/v2/records/12345")
# Custom headers (if needed)
self.get("/api/storage/v2/info", headers={"Custom-Header": "value"})
# Also locust client available
self.client.get("/api/storage/v2/records/12345")
```
### Authentication Handling
The framework automatically manages authentication:
- **Local Development**: Uses Azure CLI credentials (`az login`)
- **Azure Load Testing**: Uses Managed Identity
- **Manual Override**: Set `ADME_BEARER_TOKEN` environment variable
- **All requests**: Automatically include proper Authorization headers
## ๐ง Configuration & Environment Variables
### Configuration Hierarchy
The framework uses a layered configuration approach (highest priority first):
1. **CLI arguments** - Direct command-line overrides
2. **Environment variables** - Runtime values
3. **config.yaml** - Project-specific settings
4. **Default values** - Framework defaults
### Environment Variables
**Universal Variables:**
- `OSDU_HOST`: Base URL of OSDU instance
- `OSDU_PARTITION`: Data partition ID
- `OSDU_APP_ID`: Azure AD Application ID
- `ADME_BEARER_TOKEN`: Manual bearer token override
**Azure Load Testing Variables (auto-set):**
- `AZURE_LOAD_TEST=true`: Indicates Azure environment
- `PARTITION`: Data partition ID
- `LOCUST_HOST`: OSDU host URL
- `APPID`: Azure AD Application ID
**Metrics Collection:**
- `KUSTO_CLUSTER`: Azure Data Explorer cluster URL
- `KUSTO_DATABASE`: Database name for metrics
- `TEST_RUN_ID`: Unique identifier for test run
### Azure Authentication
The framework supports multiple Azure authentication methods with automatic detection:
**Local Development:**
- Azure CLI credentials (`az login`)
- Service Principal (via environment variables)
- DefaultAzureCredential chain
**Azure Environments:**
- Managed Identity (preferred for Azure-hosted resources)
- System-assigned or user-assigned identities
- Automatic credential detection and fallback
## ๐ Monitoring & Results
### Local Testing (Web UI)
- Open http://localhost:8089 after starting with `--web-ui`
- Real-time performance metrics
- Request statistics and response times
- Download results as CSV
### Azure Load Testing
- Monitor in Azure Portal under "Load Testing"
- Comprehensive dashboards and metrics
- Automated result retention
- Integration with Azure Monitor
### Key Metrics
- **Requests per second (RPS)**
- **Average response time**
- **95th percentile response time**
- **Error rate**
- **Failure count by endpoint**
## ๐ Advanced Usage
### Multiple Services
Test multiple services by adding more `@task` methods in your `locustfile.py`:
```python
class OSDUUser(PerformanceUser):
@task(3) # Higher weight = more frequent execution
def test_storage_apis(self):
self.get("/api/storage/v2/info")
self.post("/api/storage/v2/records", json={"data": "test"})
@task(2)
def test_search_apis(self):
self.get("/api/search/v2/query")
self.post("/api/search/v2/query", json={"query": "*"})
@task(1)
def test_schema_apis(self):
self.get("/api/schema-service/v1/schema")
```
All tests run in the same `locustfile.py` with automatic load balancing based on task weights.
### CI/CD Integration
```yaml
# Example GitHub Actions workflow
- name: Run OSDU Performance Tests
run: |
osdu_perf run local \
--host ${{ secrets.OSDU_HOST }} \
--partition ${{ secrets.OSDU_PARTITION }} \
--token ${{ secrets.OSDU_TOKEN }} \
--headless \
--users 5 \
--run-time 2m
```
## ๐ Troubleshooting
### Common Issues
**Authentication Errors**
```bash
# Ensure Azure CLI is logged in
az login
```
**Import Errors**
```bash
# Install dependencies
pip install -r requirements.txt
```
**Service Discovery Issues**
```bash
# Ensure locustfile.py exists and inherits from PerformanceUser
ls locustfile.py
# Check class inheritance
grep "PerformanceUser" locustfile.py
```
**Azure Load Testing Errors**
```bash
# Install Azure dependencies
pip install azure-cli azure-identity azure-mgmt-loadtesting azure-mgmt-resource requests
```
## ๐งฉ Project Structure (Generated)
```
perf_tests/
โโโ locustfile.py # Main test file with API calls and @task methods
โโโ config.yaml # Framework configuration (OSDU, metrics, test settings)
โโโ requirements.txt # Python dependencies (osdu_perf package)
โโโ README.md # Project documentation
```
## ๐งช Development
### Running Tests
```bash
pytest tests/
```
### Code Quality
```bash
# Formatting
black osdu_perf/
# Linting
flake8 osdu_perf/
```
### Building Package
```bash
# Build wheel and source distribution
python -m build
# Upload to TestPyPI
python -m twine upload --repository testpypi dist/*
```
## ๐ License
This project is licensed under the MIT License โ see the `LICENSE` file for details.
## ๐ Support
- **Issues**: [GitHub Issues](https://github.com/janraj/osdu_perf/issues)
- **Contact**: janrajcj@microsoft.com
- **Documentation**: This README and inline code documentation
## ๐ What's New in v1.0.24
- โ
**Three-Command Workflow**: `init`, `run local`, `run azure` - complete testing pipeline
- โ
**Configuration-Driven**: YAML-based configuration with environment-aware settings
- โ
**Service Orchestration**: Intelligent service discovery with lifecycle management
- โ
**Enhanced Authentication**: Multi-credential Azure authentication with automatic detection
- โ
**Metrics Integration**: Automated Kusto metrics collection with environment detection
- โ
**Template System**: Updated project templates with modern framework patterns
- โ
**Error Handling**: Improved error handling and defensive coding patterns
- โ
**CLI Improvements**: Better argument parsing and validation
---
**Generated by OSDU Performance Testing Framework v1.0.24**
Raw data
{
"_id": null,
"home_page": "https://github.com/janraj/osdu_perf",
"name": "osdu-perf",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Janraj CJ <janrajcj@microsoft.com>",
"keywords": "performance, testing, locust, azure, osdu, load-testing, azure-sdk, performance-testing",
"author": "Janraj CJ",
"author_email": "Janraj CJ <janrajcj@microsoft.com>",
"download_url": "https://files.pythonhosted.org/packages/2b/62/81564be84afd1b87191e9cfff42cf514a544923767d26bb08c1508737451/osdu_perf-1.0.28.2.tar.gz",
"platform": null,
"description": "# \ud83d\udd25 OSDU Performance Testing Framework\r\n\r\n[](https://www.python.org/downloads/)\r\n[](https://opensource.org/licenses/MIT)\r\n[](https://pypi.org/project/osdu-perf/)\r\n\r\nA comprehensive Python framework for performance testing OSDU (Open Subsurface Data Universe) services. Features automatic test discovery, Azure authentication, Locust integration, and both local and cloud-based load testing capabilities with intelligent service orchestration.\r\n\r\n## \ud83d\udccb Key Features\r\n\r\n\u2705 **Service Orchestration** - Intelligent service discovery and execution management \r\n\u2705 **Azure Authentication** - Seamless Azure AD token management with multiple credential flows \r\n\u2705 **Dual Execution Modes** - Run locally with Locust or scale with Azure Load Testing \r\n\u2705 **CLI Tools** - Comprehensive command-line interface with three main commands \r\n\u2705 **Template System** - Pre-built templates for common OSDU services \r\n\u2705 **Configuration Management** - YAML-based configuration with environment-aware settings \r\n\u2705 **Metrics Collection** - Automated metrics push to Azure Data Explorer (Kusto) \r\n\u2705 **Environment Detection** - Automatically adapts behavior for local vs Azure environments \r\n\r\n## \ud83c\udfd7\ufe0f Framework Architecture\r\n\r\n### Core Components\r\n- **`PerformanceUser`**: Locust integration with automatic service discovery\r\n- **`ServiceOrchestrator`**: Plugin architecture for test discovery and execution\r\n- **`BaseService`**: Abstract base class for implementing performance tests\r\n- **`InputHandler`**: Configuration management and environment detection\r\n- **`AzureTokenManager`**: Multi-credential authentication system \r\n\r\n## \ud83d\ude80 Quick Start\r\n\r\n### Installation\r\n\r\n```bash\r\n# Install from PyPI\r\npip install osdu_perf\r\n```\r\n\r\n### Three Simple Commands\r\n\r\nThe framework provides three main commands for the complete performance testing workflow:\r\n\r\n#### 1. Initialize Project (`init`)\r\n\r\n```bash\r\n# Create a new performance testing project\r\nosdu_perf init <service_name>\r\n\r\n# Examples:\r\nosdu_perf init storage # Creates storage service performance tests\r\nosdu_perf init search # Creates search service performance tests\r\nosdu_perf init wellbore # Creates wellbore service performance tests\r\n```\r\n\r\n**What this creates:**\r\n```\r\nperf_tests/\r\n\u251c\u2500\u2500 config.yaml # Framework configuration\r\n\u251c\u2500\u2500 locustfile.py # Main test file with API calls\r\n\u251c\u2500\u2500 requirements.txt # Python dependencies\r\n\u2514\u2500\u2500 README.md # Project documentation\r\n```\r\n\r\n#### 2. Run Local Tests (`run local`)\r\n\r\n```bash\r\n# Run performance tests locally using Locust\r\nosdu_perf run local --config config.yaml\r\n\r\n```\r\n\r\n**Features:**\r\n- Uses Locust for load generation\r\n- Azure CLI authentication for local development \r\n- Real-time web UI at http://localhost:8089\r\n- Automatic service discovery and execution\r\n- Automatic metric collection and sends to Kusto \r\n\r\n#### 3. Run Azure Load Tests (`run azure_load_test`)\r\n\r\n```bash\r\n# Deploy and run tests on Azure Load Testing service\r\nosdu_perf run azure_load_test --config config.yaml \r\n```\r\n\r\n**Features:**\r\n- Creates Azure Load Testing resources automatically\r\n- Scales to hundreds/thousands of concurrent users\r\n- Managed Identity authentication in Azure\r\n- Comprehensive metrics and reporting\r\n- Entitlement will be created on ADME for azure load tests \r\n\r\n## \ud83d\udee0\ufe0f Command Reference\r\n\r\n### 1. Initialize Command\r\n\r\n```bash\r\nosdu_perf init <service_name> [OPTIONS]\r\n```\r\n\r\n**Parameters:**\r\n- `service_name`: Name of the OSDU service to test (e.g., storage, search, wellbore)\r\n- `--force`: Force overwrite existing files without prompting\r\n\r\n**Examples:**\r\n```bash\r\nosdu_perf init storage # Initialize storage service tests\r\nosdu_perf init search --force # Force overwrite existing search tests\r\nosdu_perf init wellbore # Initialize wellbore service tests\r\n```\r\n\r\n**Generated Files:**\r\n- `config.yaml` - Framework configuration with OSDU connection details\r\n- `locustfile.py` - Main test file with API calls to your service\r\n- `requirements.txt` - Python dependencies\r\n- `README.md` - Project-specific documentation\r\n\r\n### 2. Local Testing Command\r\n\r\n```bash\r\nosdu_perf run local [OPTIONS]\r\n```\r\n\r\n**Configuration:**\r\n- Uses `config.yaml` for base configuration\r\n- CLI arguments override config file settings\r\n- Environment variables provide runtime values\r\n\r\n**Key Options:**\r\n- `--config`: Path to config.yaml file (required)\r\n- `--host`: OSDU host URL (overrides config)\r\n- `--partition`: OSDU data partition ID (overrides config) \r\n- `--app-id`: Azure AD Application ID (overrides config)\r\n- `--users` (`-u`): Number of concurrent users (default: from config)\r\n- `--spawn-rate` (`-r`): User spawn rate per second (default: from config)\r\n- `--run-time` (`-t`): Test duration (default: from config)\r\n\r\n**Examples:**\r\n```bash\r\n# Basic run using config.yaml\r\nosdu_perf run local --config config.yaml\r\n\r\n# Override specific settings\r\nosdu_perf run local --config config.yaml --users 50 --run-time 5m\r\n\r\n# Full override\r\nosdu_perf run local \\\r\n --config config.yaml \\\r\n --host https://api.example.com \\\r\n --partition dp1 \\\r\n --app-id 12345678-1234-1234-1234-123456789abc \\\r\n --users 25 --spawn-rate 5\r\n```\r\n\r\n### 3. Azure Load Testing Command\r\n\r\n```bash\r\nosdu_perf run azure_load_test [OPTIONS]\r\n```\r\n\r\n**Required Parameters:**\r\n- `--config`: Path to config.yaml file\r\n\r\n\r\n**Optional Parameters:**\r\n- `--loadtest-name`: Azure Load Testing resource name (auto-generated)\r\n- `--test-name`: Test name (auto-generated with timestamp)\r\n- `--engine-instances`: Number of load generator instances (default: from config)\r\n- `--users` (`-u`): Number of concurrent users per instance (default: from config)\r\n- `--run-time` (`-t`): Test duration (default: from config)\r\n\r\n**Examples:**\r\n```bash\r\n# Basic Azure Load Test using config\r\nosdu_perf run azure \\\r\n --config config.yaml \\\r\n --subscription-id \"12345678-1234-1234-1234-123456789012\" \\\r\n --resource-group \"myResourceGroup\" \\\r\n --location \"eastus\"\r\n\r\n# High-scale cloud test\r\nosdu_perf run azure \\\r\n --config config.yaml \\\r\n --subscription-id \"12345678-1234-1234-1234-123456789012\" \\\r\n --resource-group \"myResourceGroup\" \\\r\n --location \"eastus\" \\\r\n --users 100 --engine-instances 5 --run-time 30m\r\n```\r\n\r\n## \ud83d\udcdd Configuration System\r\n\r\n### config.yaml Structure\r\n\r\nThe framework uses a centralized configuration file that supports both local and Azure environments:\r\n\r\n```yaml\r\n# OSDU Environment Configuration\r\nosdu_environment:\r\n # OSDU instance details (required for run local command)\r\n host: \"https://your-osdu-host.com\"\r\n partition: \"your-partition-id\"\r\n app_id: \"your-azure-app-id\"\r\n \r\n # OSDU deployment details (optional - used for metrics collection)\r\n sku: \"Standard\"\r\n version: \"25.2.35\"\r\n \r\n # Authentication (optional - uses automatic token generation if not provided)\r\n auth:\r\n # Manual token override (optional)\r\n token: \"\"\r\n\r\n# Metrics Collection Configuration \r\nmetrics_collector:\r\n # Kusto (Azure Data Explorer) Configuration\r\n kusto:\r\n cluster: \"https://your-kusto-cluster.eastus.kusto.windows.net\"\r\n database: \"your-database\"\r\n ingest_uri: \"https://ingest-your-kusto.eastus.kusto.windows.net\"\r\n\r\n# Test Configuration (Optional)\r\ntest_settings:\r\n # Azure Load Test resource and test locations\r\n subscription_id: \"your-azure-subscription-id\"\r\n resource_group: \"your-resource-group\"\r\n location: \"eastus\"\r\n \r\n # Test-specific configurations\r\n default_wait_time: \r\n min: 1\r\n max: 3\r\n users: 10\r\n spawn_rate: 2\r\n run_time: \"60s\"\r\n engine_instances: 1\r\n test_name_prefix: \"osdu_perf_test\"\r\n test_scenario: \"health_check\"\r\n test_run_id_description: \"Automated performance test\"\r\n```\r\n\r\n### Configuration Hierarchy\r\n\r\nThe framework uses a layered configuration approach:\r\n\r\n1. **config.yaml** (project-specific settings)\r\n2. **CLI arguments** (highest priority)\r\n\r\n\r\n## \ud83c\udfd7\ufe0f How It Works\r\n\r\n### \ud83d\udd0d Simple API-Based Approach\r\n\r\nThe framework now uses a simplified API-based approach where developers write test methods directly in `locustfile.py`:\r\n\r\n```\r\nperf_tests/\r\n\u251c\u2500\u2500 locustfile.py \u2192 OSDUUser class with @task methods for testing\r\n\u251c\u2500\u2500 config.yaml \u2192 Configuration for host, partition, authentication \r\n\u251c\u2500\u2500 requirements.txt \u2192 Dependencies (osdu_perf package)\r\n```\r\n\r\n**Simplified Process:**\r\n1. `osdu_perf init <service>` generates `locustfile.py` template\r\n2. Developers add `@task` methods with API calls (`self.get()`, `self.post()`, etc.)\r\n3. `PerformanceUser` base class handles authentication, headers, tokens automatically\r\n4. Run with `osdu_perf run local` or `osdu_perf run azure_load_test`\r\n\r\n\r\n### \ud83c\udfaf Smart Resource Naming\r\n\r\nBased on detected services, Azure resources are automatically named:\r\n- **Load Test Resource**: `osdu-{service}-loadtest-{timestamp}`\r\n- **Test Name**: `osdu_{service}_test_{timestamp}`\r\n- **Example**: `osdu-storage-loadtest-20241028` with test `osdu_storage_test_20241028_142250`\r\n\r\n### \ud83d\udd10 Multi-Environment Authentication\r\n\r\n**Local Development:**\r\n- Azure CLI credentials (`az login`)\r\n- Manual token via config or environment variables\r\n- Automatic token refresh and caching\r\n\r\n**Azure Load Testing:**\r\n- Managed Identity authentication (no secrets needed)\r\n- Environment variables injected by Azure Load Testing service\r\n- Automatic credential detection and fallback\r\n\r\n### \ud83d\udcca Intelligent Metrics Collection\r\n\r\n**Automatic Kusto Integration:**\r\n- Detects environment (local vs Azure) automatically\r\n- Uses appropriate authentication method\r\n- Pushes detailed metrics to three tables:\r\n - `LocustMetrics` - Per-endpoint statistics\r\n - `LocustExceptions` - Error tracking\r\n - `LocustTestSummary` - Overall test summaries\r\n\r\n## \ud83e\uddea Writing Performance Tests\r\n\r\n### Simple API-Based Approach\r\n\r\nThe framework generate your `locustfile.py`:\r\n\r\n```python\r\n\"\"\"\r\nOSDU Performance Tests - Locust Configuration\r\nGenerated by OSDU Performance Testing Framework\r\n\"\"\"\r\n\r\nimport os\r\nfrom locust import events, task, tag\r\nfrom osdu_perf import PerformanceUser\r\n\r\n# STEP 1: Register custom CLI args with Locust\r\n@events.init_command_line_parser.add_listener\r\ndef add_custom_args(parser):\r\n \"\"\"Add OSDU-specific command line arguments\"\"\"\r\n parser.add_argument(\"--partition\", type=str, default=os.getenv(\"PARTITION\"), help=\"OSDU Data Partition ID\")\r\n parser.add_argument(\"--appid\", type=str, default=os.getenv(\"APPID\"), help=\"Azure AD Application ID\")\r\n\r\nclass OSDUUser(PerformanceUser):\r\n \"\"\"\r\n OSDU Performance Test User\r\n \r\n This class automatically:\r\n - Handles Azure authentication using --appid\r\n - Manages HTTP headers and tokens\r\n - Provides simple API methods for testing\r\n - Manages Locust user simulation and load testing\r\n \"\"\"\r\n \r\n def on_start(self):\r\n \"\"\"Called when a user starts - performs setup\"\"\"\r\n super().on_start()\r\n \r\n # Access OSDU parameters from Locust parsed options or environment variables\r\n partition = getattr(self.environment.parsed_options, 'partition', None) or os.getenv('PARTITION')\r\n host = getattr(self.environment.parsed_options, 'host', None) or self.host or os.getenv('HOST')\r\n token = os.getenv('ADME_BEARER_TOKEN') # Token only from environment for security\r\n appid = getattr(self.environment.parsed_options, 'appid', None) or os.getenv('APPID')\r\n \r\n print(f\"\ufffd Started performance testing user\")\r\n print(f\" \ud83d\udccd Partition: {partition}\")\r\n print(f\" \ud83c\udf10 Host: {host}\")\r\n print(f\" \ud83d\udd11 Token: {'***' if token else 'Not provided'}\")\r\n print(f\" \ud83c\udd94 App ID: {appid or 'Not provided'}\")\r\n \r\n @tag(\"storage\", \"health_check\")\r\n @task(1)\r\n def check_service_health(self):\r\n # Simple API call - framework handles headers, tokens, authentication\r\n self.get(\"/api/storage/v2/health\")\r\n \r\n @tag(\"storage\", \"health_check\")\r\n @task(2)\r\n def test_service_endpoints(self):\r\n # More API calls for your service\r\n self.get(\"/api/storage/v2/info\")\r\n self.post(\"/api/storage/v2/records\", json={\"test\": \"data\"})\r\n```\r\n\r\n### Key Implementation Points\r\n\r\n1. **Inherit from PerformanceUser**: Your class extends `PerformanceUser` which handles all authentication and setup\r\n2. **Use @task decorators**: Mark methods with `@task(weight)` to define test scenarios\r\n3. **Simple HTTP methods**: Use `self.get()`, `self.post()`, `self.put()`, `self.delete()` - framework handles headers/tokens\r\n4. **No manual authentication**: Framework automatically handles Azure AD tokens and HTTP headers\r\n5. **Environment awareness**: Automatically adapts for local vs Azure Load Testing environments\r\n\r\n### Available HTTP Methods\r\n\r\nThe `PerformanceUser` base class provides these simple methods:\r\n\r\n```python\r\n# GET request\r\nself.get(\"/api/storage/v2/records/12345\")\r\n\r\n# POST request with JSON data\r\nself.post(\"/api/storage/v2/records\", json={\r\n \"kind\": \"osdu:wks:partition:storage:1.0.0\",\r\n \"data\": {\"test\": \"data\"}\r\n})\r\n\r\n# PUT request\r\nself.put(\"/api/storage/v2/records/12345\", json=updated_data)\r\n\r\n# DELETE request \r\nself.delete(\"/api/storage/v2/records/12345\")\r\n\r\n# Custom headers (if needed)\r\nself.get(\"/api/storage/v2/info\", headers={\"Custom-Header\": \"value\"})\r\n\r\n# Also locust client available \r\n\r\nself.client.get(\"/api/storage/v2/records/12345\")\r\n```\r\n\r\n### Authentication Handling\r\n\r\nThe framework automatically manages authentication:\r\n\r\n- **Local Development**: Uses Azure CLI credentials (`az login`)\r\n- **Azure Load Testing**: Uses Managed Identity \r\n- **Manual Override**: Set `ADME_BEARER_TOKEN` environment variable\r\n- **All requests**: Automatically include proper Authorization headers\r\n\r\n## \ud83d\udd27 Configuration & Environment Variables\r\n\r\n### Configuration Hierarchy\r\n\r\nThe framework uses a layered configuration approach (highest priority first):\r\n\r\n1. **CLI arguments** - Direct command-line overrides\r\n2. **Environment variables** - Runtime values \r\n3. **config.yaml** - Project-specific settings\r\n4. **Default values** - Framework defaults\r\n\r\n### Environment Variables\r\n\r\n**Universal Variables:**\r\n- `OSDU_HOST`: Base URL of OSDU instance\r\n- `OSDU_PARTITION`: Data partition ID\r\n- `OSDU_APP_ID`: Azure AD Application ID\r\n- `ADME_BEARER_TOKEN`: Manual bearer token override\r\n\r\n**Azure Load Testing Variables (auto-set):**\r\n- `AZURE_LOAD_TEST=true`: Indicates Azure environment\r\n- `PARTITION`: Data partition ID\r\n- `LOCUST_HOST`: OSDU host URL\r\n- `APPID`: Azure AD Application ID\r\n\r\n**Metrics Collection:**\r\n- `KUSTO_CLUSTER`: Azure Data Explorer cluster URL\r\n- `KUSTO_DATABASE`: Database name for metrics\r\n- `TEST_RUN_ID`: Unique identifier for test run\r\n\r\n### Azure Authentication\r\n\r\nThe framework supports multiple Azure authentication methods with automatic detection:\r\n\r\n**Local Development:**\r\n- Azure CLI credentials (`az login`)\r\n- Service Principal (via environment variables)\r\n- DefaultAzureCredential chain\r\n\r\n**Azure Environments:**\r\n- Managed Identity (preferred for Azure-hosted resources)\r\n- System-assigned or user-assigned identities\r\n- Automatic credential detection and fallback\r\n\r\n## \ud83d\udcca Monitoring & Results\r\n\r\n### Local Testing (Web UI)\r\n- Open http://localhost:8089 after starting with `--web-ui`\r\n- Real-time performance metrics\r\n- Request statistics and response times\r\n- Download results as CSV\r\n\r\n### Azure Load Testing\r\n- Monitor in Azure Portal under \"Load Testing\"\r\n- Comprehensive dashboards and metrics\r\n- Automated result retention\r\n- Integration with Azure Monitor\r\n\r\n### Key Metrics\r\n- **Requests per second (RPS)**\r\n- **Average response time**\r\n- **95th percentile response time** \r\n- **Error rate**\r\n- **Failure count by endpoint**\r\n\r\n## \ud83d\ude80 Advanced Usage\r\n\r\n\r\n### Multiple Services\r\nTest multiple services by adding more `@task` methods in your `locustfile.py`:\r\n\r\n```python\r\nclass OSDUUser(PerformanceUser):\r\n \r\n @task(3) # Higher weight = more frequent execution\r\n def test_storage_apis(self):\r\n self.get(\"/api/storage/v2/info\")\r\n self.post(\"/api/storage/v2/records\", json={\"data\": \"test\"})\r\n \r\n @task(2) \r\n def test_search_apis(self):\r\n self.get(\"/api/search/v2/query\")\r\n self.post(\"/api/search/v2/query\", json={\"query\": \"*\"})\r\n \r\n @task(1)\r\n def test_schema_apis(self):\r\n self.get(\"/api/schema-service/v1/schema\")\r\n```\r\n\r\nAll tests run in the same `locustfile.py` with automatic load balancing based on task weights.\r\n\r\n### CI/CD Integration\r\n\r\n```yaml\r\n# Example GitHub Actions workflow\r\n- name: Run OSDU Performance Tests\r\n run: |\r\n osdu_perf run local \\\r\n --host ${{ secrets.OSDU_HOST }} \\\r\n --partition ${{ secrets.OSDU_PARTITION }} \\\r\n --token ${{ secrets.OSDU_TOKEN }} \\\r\n --headless \\\r\n --users 5 \\\r\n --run-time 2m\r\n```\r\n\r\n## \ud83d\udc1b Troubleshooting\r\n\r\n### Common Issues\r\n\r\n**Authentication Errors**\r\n```bash\r\n# Ensure Azure CLI is logged in\r\naz login\r\n\r\n```\r\n\r\n**Import Errors**\r\n```bash\r\n# Install dependencies\r\npip install -r requirements.txt\r\n```\r\n\r\n**Service Discovery Issues**\r\n```bash\r\n# Ensure locustfile.py exists and inherits from PerformanceUser\r\nls locustfile.py\r\n\r\n# Check class inheritance\r\ngrep \"PerformanceUser\" locustfile.py\r\n```\r\n\r\n**Azure Load Testing Errors**\r\n```bash\r\n# Install Azure dependencies\r\npip install azure-cli azure-identity azure-mgmt-loadtesting azure-mgmt-resource requests\r\n```\r\n\r\n## \ud83e\udde9 Project Structure (Generated)\r\n\r\n```\r\nperf_tests/\r\n\u251c\u2500\u2500 locustfile.py # Main test file with API calls and @task methods\r\n\u251c\u2500\u2500 config.yaml # Framework configuration (OSDU, metrics, test settings)\r\n\u251c\u2500\u2500 requirements.txt # Python dependencies (osdu_perf package) \r\n\u2514\u2500\u2500 README.md # Project documentation\r\n```\r\n\r\n## \ud83e\uddea Development\r\n\r\n### Running Tests\r\n```bash\r\npytest tests/\r\n```\r\n\r\n### Code Quality\r\n```bash\r\n# Formatting\r\nblack osdu_perf/\r\n\r\n# Linting \r\nflake8 osdu_perf/\r\n```\r\n\r\n### Building Package\r\n```bash\r\n# Build wheel and source distribution\r\npython -m build\r\n\r\n# Upload to TestPyPI\r\npython -m twine upload --repository testpypi dist/*\r\n```\r\n\r\n## \ud83d\udcc4 License\r\n\r\nThis project is licensed under the MIT License \u2014 see the `LICENSE` file for details.\r\n\r\n## \ud83c\udd98 Support\r\n\r\n- **Issues**: [GitHub Issues](https://github.com/janraj/osdu_perf/issues)\r\n- **Contact**: janrajcj@microsoft.com\r\n- **Documentation**: This README and inline code documentation\r\n\r\n## \ud83d\ude80 What's New in v1.0.24\r\n\r\n- \u2705 **Three-Command Workflow**: `init`, `run local`, `run azure` - complete testing pipeline\r\n- \u2705 **Configuration-Driven**: YAML-based configuration with environment-aware settings\r\n- \u2705 **Service Orchestration**: Intelligent service discovery with lifecycle management\r\n- \u2705 **Enhanced Authentication**: Multi-credential Azure authentication with automatic detection\r\n- \u2705 **Metrics Integration**: Automated Kusto metrics collection with environment detection\r\n- \u2705 **Template System**: Updated project templates with modern framework patterns\r\n- \u2705 **Error Handling**: Improved error handling and defensive coding patterns\r\n- \u2705 **CLI Improvements**: Better argument parsing and validation\r\n\r\n---\r\n\r\n**Generated by OSDU Performance Testing Framework v1.0.24**\r\n\r\n",
"bugtrack_url": null,
"license": null,
"summary": "Performance Testing Framework for OSDU Services - A comprehensive tool for testing OSDU APIs with Locust and Azure Load Testing SDK",
"version": "1.0.28.2",
"project_urls": {
"Documentation": "https://github.com/janraj/osdu_perf#readme",
"Homepage": "https://github.com/janraj/osdu_perf",
"Issues": "https://github.com/janraj/osdu_perf/issues",
"Repository": "https://github.com/janraj/osdu_perf"
},
"split_keywords": [
"performance",
" testing",
" locust",
" azure",
" osdu",
" load-testing",
" azure-sdk",
" performance-testing"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "379b5c07f1c65aa954c7f373beb770103e023ed752658853c2875589420fb3d4",
"md5": "94562918eb8e13b6b1ba86a5f86896e1",
"sha256": "601f4e949b693e5c1a66dfcc041d1d41c8b05b8baf0f533384dd543e92669e8c"
},
"downloads": -1,
"filename": "osdu_perf-1.0.28.2-py3-none-any.whl",
"has_sig": false,
"md5_digest": "94562918eb8e13b6b1ba86a5f86896e1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 75423,
"upload_time": "2025-11-12T02:36:46",
"upload_time_iso_8601": "2025-11-12T02:36:46.720986Z",
"url": "https://files.pythonhosted.org/packages/37/9b/5c07f1c65aa954c7f373beb770103e023ed752658853c2875589420fb3d4/osdu_perf-1.0.28.2-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "2b6281564be84afd1b87191e9cfff42cf514a544923767d26bb08c1508737451",
"md5": "d6ef1ca3a3ef08ac6de43f7c11d63d1c",
"sha256": "8cd934f87291b7868f4cc4280d68363224b40aef548d315797bac272014f3680"
},
"downloads": -1,
"filename": "osdu_perf-1.0.28.2.tar.gz",
"has_sig": false,
"md5_digest": "d6ef1ca3a3ef08ac6de43f7c11d63d1c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 70031,
"upload_time": "2025-11-12T02:36:48",
"upload_time_iso_8601": "2025-11-12T02:36:48.190021Z",
"url": "https://files.pythonhosted.org/packages/2b/62/81564be84afd1b87191e9cfff42cf514a544923767d26bb08c1508737451/osdu_perf-1.0.28.2.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-11-12 02:36:48",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "janraj",
"github_project": "osdu_perf",
"github_not_found": true,
"lcname": "osdu-perf"
}
JSON
