# OGC SensorThings API Schema Validator
[](https://github.com/janbeckert/ogc-sta-schema-validator/actions/workflows/ci.yml)
## Abstract
This tool provides automated validation of entities stored in [OGC SensorThings API](https://www.ogc.org/standards/sensorthings) implementations against organization-specific JSON Schema definitions. The OGC SensorThings API is an open geospatial standard for interconnecting Internet of Things (IoT) devices, data, and applications over the Web, providing a unified framework for managing sensor data and observations.
As SensorThings API implementations grow in scale and complexity, ensuring data quality and compliance with organizational requirements becomes critical. This validation tool addresses the need for systematic quality assurance by enabling configurable, schema-driven validation of entity properties beyond the base OGC specification requirements.
## Standards & Specifications
This implementation is designed to work with:
- **[OGC SensorThings API Part 1: Sensing (Version 1.1)](https://docs.ogc.org/is/18-088/18-088.html)** - The core specification defining the data model and REST API for IoT sensor observations (OGC Document 18-088)
- **[JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12/json-schema-core.html)** - The validation schema language used to define entity constraints
## Compatible Implementations
This tool has been tested with:
- **[FROST-Server](https://github.com/FraunhoferIOSB/FROST-Server)** - The Fraunhofer IOSB reference implementation of the OGC SensorThings API, widely used in production environments
- Any OGC SensorThings API 1.0/1.1 compliant server implementation
## Overview
### SensorThings API Data Model
The OGC SensorThings API defines a standard data model consisting of eight core entity types that capture the "who, what, when, where, and how" of sensor observations:
- **Thing** - The object of interest (e.g., a weather station, vehicle, building)
- **Location** - The geographic location of a Thing
- **HistoricalLocation** - Historical location information
- **Datastream** - A collection of observations grouped by sensor and observed property
- **Sensor** - The instrument or procedure that produced the observations
- **ObservedProperty** - The phenomenon being measured (e.g., temperature, humidity)
- **Observation** - The actual measurement or result
- **FeatureOfInterest** - The feature that is the subject of an observation
This validator enables organizations to enforce custom validation rules on any of these entity types through JSON Schema definitions.
## Features
- **Standards-Based Validation**: Implements JSON Schema Draft 2020-12 for validation rule definition
- **Comprehensive Coverage**: Supports all OGC SensorThings API entity types
- **Flexible Validation Scope**: Validate individual entities, entity types, or entire datasets
- **Multiple Output Formats**: Console, JSON, and CSV reporting for different use cases
- **Continuous Monitoring**: Scheduled validation runs for ongoing data quality assurance
- **Nested Property Support**: Full validation of nested object structures and custom properties
- **Custom Extensions**: Organization-specific error messages and severity levels
- **Configurable Architecture**: YAML-based configuration with entity-specific settings
- **Production Ready**: Docker images, Docker Compose, and CI/CD workflows
- **Authentication Support**: Multiple authentication methods including Basic Auth, Bearer tokens, and Keycloak OIDC
## Quick Start
### 🎯 Quick Demo (5 Minutes)
Want to see the validator in action right away? Try the fully automated demo:
```bash
# Clone the repository
git clone https://github.com/janbeckert/ogc-sta-schema-validator.git
cd ogc-sta-schema-validator
# Run the demo (requires Docker and Docker Compose)
./demo.sh
```
This will:
- Start a complete FROST-Server environment
- Provision sample entities (both valid and invalid)
- Run validation and show results with intentional errors
- Demonstrate the validator catching real data quality issues
**Or use Docker Compose directly:**
```bash
docker compose up --build
```
The demo shows validation catching common errors like missing required fields, invalid data types, out of range values, and malformed dates.
---
### Option 1: Docker (Recommended for Quick Testing)
The easiest way to get started is using Docker:
```bash
# Pull the latest image from GitHub Container Registry
docker pull ghcr.io/janbeckert/ogc-sta-schema-validator:latest
# Or build locally
docker build -t ogc-sta-schema-validator .
# Run a simple test (show help)
docker run --rm ogc-sta-schema-validator
# Test connection to a FROST-Server
docker run --rm \
-e VALIDATOR_SERVER__URL=http://your-server:8080/FROST-Server/v1.1 \
ogc-sta-schema-validator \
test-connection
# Validate with your own config and schemas
docker run --rm \
-v $(pwd)/config:/app/config:ro \
-v $(pwd)/schemas:/app/schemas:ro \
-v $(pwd)/output:/app/output \
ogc-sta-schema-validator \
validate --entity-type Things --output-file /app/output/report.json
```
### Option 2: Local Installation
This project uses [uv](https://docs.astral.sh/uv/) for dependency management.
```bash
# Install uv if you haven't already
curl -LsSf https://astral.sh/uv/install.sh | sh
# Clone and setup the project
cd ogc-sta-schema-validator
# Install dependencies
uv sync
# Activate the virtual environment
source .venv/bin/activate # On Unix/macOS
# or
.venv\Scripts\activate # On Windows
```
### Configuration
1. Copy the example configuration file:
```bash
cp config/config.example.yaml config/config.yaml
```
2. Edit `config/config.yaml` with your server details:
```yaml
server:
url: "http://your-frost-server:8080/FROST-Server/v1.1"
```
3. Configure authentication if required (see config file for examples)
4. Set up your validation schemas:
```bash
# Option 1: Copy example schemas
cp examples/schemas/*.json schemas/
# Option 2: Create custom schemas in schemas/ directory
```
### Basic Usage
```bash
# Test connection to your FROST-Server
uv run ogc-sta-validate test-connection
# List available entity types
uv run ogc-sta-validate list-entities
# Validate all Things
uv run ogc-sta-validate validate --entity-type Things
# Validate a specific Thing
uv run ogc-sta-validate validate --entity-type Things --entity-id 123
# Validate with custom schema
uv run ogc-sta-validate validate --entity-type Things --schema schemas/thing_schema.json
# Generate JSON report
uv run ogc-sta-validate validate --entity-type Things --output-format json --output-file report.json
# Validate all entity types
uv run ogc-sta-validate validate-all
# Run continuous validation (every 30 minutes)
uv run ogc-sta-validate continuous --interval 1800
```
## Configuration
The tool supports **multiple configuration sources** with the following precedence (highest to lowest):
1. **CLI arguments** (highest priority)
2. **Environment variables**
3. **YAML configuration file**
4. **Default values** (lowest priority)
This flexible approach allows you to:
- Store sensitive credentials in environment variables
- Override configuration temporarily via CLI arguments
- Maintain base configuration in YAML files
- Use different configurations across environments (dev/staging/prod)
### Method 1: YAML Configuration File
The primary configuration method uses YAML files. This is covered in the Quick Start section above.
```yaml
# config/config.yaml
server:
url: "http://localhost:8080/FROST-Server/v1.1"
timeout: 30
auth:
method: "basic"
username: "user"
password: "pass"
validation:
batch_size: 100
stop_on_error: false
```
See `config/config.example.yaml` for a complete configuration template with all available options.
### Method 2: Environment Variables
Environment variables can override YAML configuration values. This is especially useful for:
- **Sensitive data**: Store credentials securely outside of config files
- **CI/CD pipelines**: Configure differently across environments
- **Docker deployments**: Use Docker environment variables
#### Naming Convention
Environment variables use the prefix `VALIDATOR_` and double underscores (`__`) for nested values:
```bash
# Format: VALIDATOR_<SECTION>__<KEY>
VALIDATOR_SERVER__URL=http://example.com/v1.1
VALIDATOR_SERVER__TIMEOUT=60
VALIDATOR_VALIDATION__BATCH_SIZE=200
```
#### Quick Setup
```bash
# Copy the example environment file
cp .env.example .env
```
#### Example Environment Variables
```bash
# Server configuration
export VALIDATOR_SERVER__URL="http://example.com/FROST-Server/v1.1"
export VALIDATOR_SERVER__TIMEOUT=60
# Authentication (recommended for sensitive credentials)
export VALIDATOR_SERVER__AUTH__METHOD="basic"
export VALIDATOR_SERVER__AUTH__USERNAME="myuser"
export VALIDATOR_SERVER__AUTH__PASSWORD="mypassword"
# Validation settings
export VALIDATOR_VALIDATION__BATCH_SIZE=200
export VALIDATOR_OUTPUT__FORMAT="json"
# Now run the tool - environment variables will be used
uv run ogc-sta-validate test-connection
```
See `.env.example` for all available environment variables.
### Method 3: CLI Arguments
CLI arguments provide the highest precedence.
#### Common CLI Options
```bash
# Override server URL
uv run ogc-sta-validate --server-url http://example.com/v1.1 test-connection
# Override timeout
uv run ogc-sta-validate --timeout 60 test-connection
# Override batch size
uv run ogc-sta-validate --batch-size 200 validate --entity-type Things
# Combine multiple overrides
uv run ogc-sta-validate \
--server-url http://example.com/v1.1 \
--timeout 60 \
--batch-size 200 \
validate --entity-type Things
```
#### Command-Specific Options
Each command also has specific options:
```bash
# Validate command options
uv run ogc-sta-validate validate \
--entity-type Things \
--limit 100 \
--output-format json \
--output-file report.json \
--filter "properties/status eq 'active'"
# Continuous validation options
uv run ogc-sta-validate continuous \
--interval 3600 \
--entity-types Things \
--entity-types Sensors \
--max-entities 500
```
### Getting Help
View all available CLI options:
```bash
# Main help
uv run ogc-sta-validate --help
# Command-specific help
uv run ogc-sta-validate validate --help
uv run ogc-sta-validate continuous --help
```
## Validation Methodology
### Approach
This tool implements a **schema-driven validation approach** that operates independently of the OGC SensorThings API server's internal validation. While the SensorThings API specification defines mandatory properties and data types, organizations often have additional requirements such as:
- Custom property schemas for domain-specific metadata
- Enumerated value constraints for controlled vocabularies
- Pattern matching for identifiers following organizational conventions
- Range validation for sensor measurements
- Date/time format requirements beyond ISO 8601
The validation engine retrieves entities via the SensorThings API's standardized REST interface and applies JSON Schema rules without modifying the source data, making it suitable for both development and production environments.
### Validation Process
1. **Entity Retrieval**: Fetches entities from the SensorThings API server using HTTP GET requests with optional [OData filtering](https://www.odata.org/documentation/)
2. **Schema Loading**: Loads JSON Schema definitions from the local filesystem, matched by entity type
3. **Rule Application**: Applies JSON Schema validation rules using the `jsonschema` library
4. **Error Aggregation**: Collects validation errors with property paths and rule type information
5. **Report Generation**: Produces detailed reports in multiple formats (console, JSON, CSV)
### Architecture
The validator implements a modular architecture with separation of concerns:
```
┌─────────────────┐
│ CLI Interface │ Command-line interface (Click framework)
│ (cli.py) │
└────────┬────────┘
│
├──┬──────────────────────────────────────────────────┐
│ │ │
┌────────▼──────────┐ ┌──────────────┐ ┌──────────────────┐ │
│ API Client │ │Schema Loader │ │ Validator Engine │ │
│(api_client.py) │ │(loader.py) │ │ (validator.py) │ │
│ │ │ │ │ │ │
│ - HTTP requests │ │- JSON Schema │ │ - Rule engine │ │
│ - Pagination │ │ discovery │ │ - Error handling │ │
│ - Auth strategies │ │- Validation │ │ - Type checking │ │
│ - OData filters │ │ │ │ │ │
└────────┬──────────┘ └──────┬───────┘ └────────┬─────────┘ │
│ │ │ │
│ │ │ │
└────────────────────┴───────────────────┴────────────┤
│
┌──────────────────────────────────▼┐
│ Validation Runner │
│ (runner.py) │
│ │
│ - Orchestration & batch processing│
│ - Multi-format report generation │
│ - Error/warning categorization │
└───────────────────────────────────┘
```
## JSON Schema Format
Validation schemas use standard JSON Schema Draft 2020-12. This example demonstrates common validation features:
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://schemas.sensorthings.org/things.json",
"title": "Things Schema",
"type": "object",
"required": ["name", "description"],
"properties": {
"name": {
"type": "string",
"minLength": 1,
"maxLength": 100,
"description": "Thing name is mandatory",
"x-errorMessage": "Thing must have a non-empty name (1-100 characters)"
},
"description": {
"type": "string",
"minLength": 1
},
"properties": {
"type": "object",
"required": ["thingId"],
"properties": {
"thingId": {
"type": "string",
"minLength": 1
},
"status": {
"enum": ["active", "inactive", "maintenance"],
"description": "Status must be one of the allowed values"
},
"code": {
"type": "string",
"pattern": "^[A-Z]{2}[0-9]{4}$",
"description": "Building code format: two uppercase letters + four digits"
},
"temperature": {
"type": "number",
"minimum": -50,
"maximum": 100
},
"timestamp": {
"type": "string",
"format": "date-time"
},
"coordinates": {
"type": "array",
"minItems": 2,
"maxItems": 3,
"description": "Coordinates [longitude, latitude, altitude?]"
},
"tags": {
"type": "array",
"minItems": 1,
"maxItems": 10
}
}
}
}
}
```
See [JSON Schema specification](https://json-schema.org/draft/2020-12/json-schema-validation.html) for complete documentation.
### Custom Extensions
This validator supports custom JSON Schema extension fields (prefixed with `x-` per JSON Schema conventions) to enhance validation functionality:
#### Schema-Level Extensions
**`x-version`** (string, optional): Schema version identifier for tracking and documentation
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"x-version": "2.0.0",
"x-entityType": "Things",
"title": "Things Schema"
}
```
**`x-entityType`** (string, optional): Specifies the SensorThings API entity type this schema validates
```json
{
"x-entityType": "Things"
}
```
Used for entity type identification when the schema filename doesn't match the entity type.
#### Property-Level Extensions
**`x-errorMessage`** (string, optional): Custom error message shown when validation fails
```json
{
"type": "string",
"pattern": "^de\\.sn\\.stlp\\..+",
"x-errorMessage": "ID must start with Leipzig prefix (de.sn.stlp.)"
}
```
**`x-severity`** (string, optional): Validation severity level - either `"error"` or `"warning"`
```json
{
"type": "string",
"pattern": "^[A-Z]{2}[0-9]{4}$",
"x-errorMessage": "Recommended format: XX9999",
"x-severity": "warning"
}
```
- **`error`** (default): Validation failure causes entity to be marked as invalid
- **`warning`**: Validation failure is reported but entity remains valid
#### Complete Example
```json
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$id": "https://schemas.sensorthings.org/things.json",
"x-version": "2.0.0",
"x-entityType": "Things",
"title": "Things Schema",
"type": "object",
"required": ["name"],
"properties": {
"name": {
"type": "string",
"minLength": 1,
"x-errorMessage": "Thing must have a non-empty name"
},
"properties": {
"type": "object",
"properties": {
"thingId": {
"type": "string",
"pattern": "^de\\.sn\\.stlp\\..+",
"x-errorMessage": "Thing ID must use Leipzig prefix",
"x-severity": "error"
},
"installationDate": {
"type": "string",
"format": "date-time",
"x-errorMessage": "Installation date should be in ISO 8601 format",
"x-severity": "warning"
}
}
}
}
}
```
## CLI Commands
### test-connection
Test connection to the SensorThings API server.
### validate
Validate entities of a specific type.
Options:
- `--entity-type`, `-t`: Entity type to validate (required)
- `--entity-id`, `-i`: Specific entity ID to validate
- `--schema`, `-s`: Schema file to use
- `--limit`, `-l`: Maximum number of entities to validate
- `--filter`, `-f`: OData filter expression
- `--output-format`: Output format (console, json, csv)
- `--output-file`, `-o`: Output file path
- `--include-valid`: Include valid entities in output
### validate-all
Validate all entity types.
Options:
- `--entity-types`, `-t`: Specific entity types to validate
- `--limit-per-type`, `-l`: Maximum entities per type
- `--output-format`: Output format
- `--output-file`, `-o`: Output file path
- `--include-valid`: Include valid entities in output
### continuous
Run continuous validation.
Options:
- `--interval`, `-i`: Validation interval in seconds
- `--entity-types`, `-t`: Entity types to validate
- `--max-entities`, `-m`: Maximum entities per run
### validate-schema
Validate schema files.
Options:
- `--schema-file`, `-s`: Specific schema file to validate
### list-entities
List available entity types and their counts.
## Development
### Setup Development Environment
```bash
# Install with development dependencies
uv sync --dev
# Run tests
uv run pytest
# Format code and fix imports (replaces black + isort)
uv run ruff format .
# Lint code (replaces flake8)
uv run ruff check .
# Lint and auto-fix issues
uv run ruff check --fix .
```
### Adding Dependencies
```bash
# Add a new dependency
uv add requests
# Add a development dependency
uv add --dev pytest
# Update dependencies
uv sync
```
## Example Validation Schemas
The repository includes example validation schemas in `examples/schemas/` demonstrating validation patterns for common SensorThings API entity types:
- `examples/schemas/thing_schema.json` - Validation schema for Thing entities with custom property constraints
- `examples/schemas/sensor_schema.json` - Validation schema for Sensor entities with metadata requirements
- `examples/schemas/observation_schema.json` - Validation schema for Observation entities with result value constraints
- `examples/schemas/datastream_schema.json` - Validation schema for Datastream entities with unit of measurement validation
Copy these to your `schemas/` directory and customize them for your organization:
```bash
cp examples/schemas/*.json schemas/
```
See `examples/schemas/README.md` for customization guidance.
## Output Formats
### Console Output
Human-readable text format with validation summary and error details.
### JSON Output
Structured JSON format suitable for programmatic processing:
```json
{
"summary": {
"total_entities": 100,
"valid_entities": 95,
"invalid_entities": 5,
"total_errors": 8,
"total_warnings": 2
},
"results": [
{
"entityId": "123",
"entityType": "Things",
"isValid": false,
"errors": [
{
"property_path": "properties.building",
"message": "Required field is missing",
"validator": "required"
}
]
}
]
}
```
### CSV Output
Tabular format suitable for spreadsheet analysis with columns:
- EntityType, EntityId, IsValid, ErrorCount, WarningCount, PropertyPath, ErrorType, Message, Severity
## Troubleshooting
### Connection Issues
- Verify the SensorThings API server URL conforms to the specification (e.g., ends with `/v1.0` or `/v1.1`)
- Check network connectivity and firewall rules
- Ensure authentication credentials are correct for your server's authentication method
### Schema Issues
- Use `validate-schema` command to check JSON Schema validity against Draft 2020-12
- Verify JSON syntax and required meta-properties (`$schema`, `type`, etc.)
- Ensure custom extensions (`x-errorMessage`, `x-severity`) follow the expected format
### Performance
- Adjust `batch_size` based on entity complexity and network latency (typical range: 50-500)
- Use OData `$filter` expressions to limit validation scope to relevant entities
## License
See LICENSE file for details.
## Contributing
Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.
Raw data
{
"_id": null,
"home_page": null,
"name": "ogc-sta-schema-validator",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "data-quality, frost-server, iot, json-schema, ogc, sensor, sensorthings, validation",
"author": null,
"author_email": "Jan Beckert <jan@beckert-it.com>",
"download_url": "https://files.pythonhosted.org/packages/20/3c/3202479d119c4bf819be5769da22427cbcdfe29d7ddd97ef7ff678c97ec9/ogc_sta_schema_validator-0.1.0.tar.gz",
"platform": null,
"description": "# OGC SensorThings API Schema Validator\n\n[](https://github.com/janbeckert/ogc-sta-schema-validator/actions/workflows/ci.yml)\n\n## Abstract\n\nThis tool provides automated validation of entities stored in [OGC SensorThings API](https://www.ogc.org/standards/sensorthings) implementations against organization-specific JSON Schema definitions. The OGC SensorThings API is an open geospatial standard for interconnecting Internet of Things (IoT) devices, data, and applications over the Web, providing a unified framework for managing sensor data and observations.\n\nAs SensorThings API implementations grow in scale and complexity, ensuring data quality and compliance with organizational requirements becomes critical. This validation tool addresses the need for systematic quality assurance by enabling configurable, schema-driven validation of entity properties beyond the base OGC specification requirements.\n\n## Standards & Specifications\n\nThis implementation is designed to work with:\n\n- **[OGC SensorThings API Part 1: Sensing (Version 1.1)](https://docs.ogc.org/is/18-088/18-088.html)** - The core specification defining the data model and REST API for IoT sensor observations (OGC Document 18-088)\n- **[JSON Schema Draft 2020-12](https://json-schema.org/draft/2020-12/json-schema-core.html)** - The validation schema language used to define entity constraints\n\n## Compatible Implementations\n\nThis tool has been tested with:\n\n- **[FROST-Server](https://github.com/FraunhoferIOSB/FROST-Server)** - The Fraunhofer IOSB reference implementation of the OGC SensorThings API, widely used in production environments\n- Any OGC SensorThings API 1.0/1.1 compliant server implementation\n\n## Overview\n\n### SensorThings API Data Model\n\nThe OGC SensorThings API defines a standard data model consisting of eight core entity types that capture the \"who, what, when, where, and how\" of sensor observations:\n\n- **Thing** - The object of interest (e.g., a weather station, vehicle, building)\n- **Location** - The geographic location of a Thing\n- **HistoricalLocation** - Historical location information\n- **Datastream** - A collection of observations grouped by sensor and observed property\n- **Sensor** - The instrument or procedure that produced the observations\n- **ObservedProperty** - The phenomenon being measured (e.g., temperature, humidity)\n- **Observation** - The actual measurement or result\n- **FeatureOfInterest** - The feature that is the subject of an observation\n\nThis validator enables organizations to enforce custom validation rules on any of these entity types through JSON Schema definitions.\n\n## Features\n\n- **Standards-Based Validation**: Implements JSON Schema Draft 2020-12 for validation rule definition\n- **Comprehensive Coverage**: Supports all OGC SensorThings API entity types\n- **Flexible Validation Scope**: Validate individual entities, entity types, or entire datasets\n- **Multiple Output Formats**: Console, JSON, and CSV reporting for different use cases\n- **Continuous Monitoring**: Scheduled validation runs for ongoing data quality assurance\n- **Nested Property Support**: Full validation of nested object structures and custom properties\n- **Custom Extensions**: Organization-specific error messages and severity levels\n- **Configurable Architecture**: YAML-based configuration with entity-specific settings\n- **Production Ready**: Docker images, Docker Compose, and CI/CD workflows\n- **Authentication Support**: Multiple authentication methods including Basic Auth, Bearer tokens, and Keycloak OIDC\n\n## Quick Start\n\n### \ud83c\udfaf Quick Demo (5 Minutes)\n\nWant to see the validator in action right away? Try the fully automated demo:\n\n```bash\n# Clone the repository\ngit clone https://github.com/janbeckert/ogc-sta-schema-validator.git\ncd ogc-sta-schema-validator\n\n# Run the demo (requires Docker and Docker Compose)\n./demo.sh\n```\n\nThis will:\n- Start a complete FROST-Server environment\n- Provision sample entities (both valid and invalid)\n- Run validation and show results with intentional errors\n- Demonstrate the validator catching real data quality issues\n\n**Or use Docker Compose directly:**\n\n```bash\ndocker compose up --build\n```\n\nThe demo shows validation catching common errors like missing required fields, invalid data types, out of range values, and malformed dates.\n\n---\n\n### Option 1: Docker (Recommended for Quick Testing)\n\nThe easiest way to get started is using Docker:\n\n```bash\n# Pull the latest image from GitHub Container Registry\ndocker pull ghcr.io/janbeckert/ogc-sta-schema-validator:latest\n\n# Or build locally\ndocker build -t ogc-sta-schema-validator .\n\n# Run a simple test (show help)\ndocker run --rm ogc-sta-schema-validator\n\n# Test connection to a FROST-Server\ndocker run --rm \\\n -e VALIDATOR_SERVER__URL=http://your-server:8080/FROST-Server/v1.1 \\\n ogc-sta-schema-validator \\\n test-connection\n\n# Validate with your own config and schemas\ndocker run --rm \\\n -v $(pwd)/config:/app/config:ro \\\n -v $(pwd)/schemas:/app/schemas:ro \\\n -v $(pwd)/output:/app/output \\\n ogc-sta-schema-validator \\\n validate --entity-type Things --output-file /app/output/report.json\n```\n\n### Option 2: Local Installation\n\nThis project uses [uv](https://docs.astral.sh/uv/) for dependency management.\n\n```bash\n# Install uv if you haven't already\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n# Clone and setup the project\ncd ogc-sta-schema-validator\n\n# Install dependencies\nuv sync\n\n# Activate the virtual environment\nsource .venv/bin/activate # On Unix/macOS\n# or\n.venv\\Scripts\\activate # On Windows\n```\n\n### Configuration\n\n1. Copy the example configuration file:\n```bash\ncp config/config.example.yaml config/config.yaml\n```\n\n2. Edit `config/config.yaml` with your server details:\n```yaml\nserver:\n url: \"http://your-frost-server:8080/FROST-Server/v1.1\"\n```\n\n3. Configure authentication if required (see config file for examples)\n\n4. Set up your validation schemas:\n```bash\n# Option 1: Copy example schemas\ncp examples/schemas/*.json schemas/\n\n# Option 2: Create custom schemas in schemas/ directory\n```\n\n### Basic Usage\n\n```bash\n# Test connection to your FROST-Server\nuv run ogc-sta-validate test-connection\n\n# List available entity types\nuv run ogc-sta-validate list-entities\n\n# Validate all Things\nuv run ogc-sta-validate validate --entity-type Things\n\n# Validate a specific Thing\nuv run ogc-sta-validate validate --entity-type Things --entity-id 123\n\n# Validate with custom schema\nuv run ogc-sta-validate validate --entity-type Things --schema schemas/thing_schema.json\n\n# Generate JSON report\nuv run ogc-sta-validate validate --entity-type Things --output-format json --output-file report.json\n\n# Validate all entity types\nuv run ogc-sta-validate validate-all\n\n# Run continuous validation (every 30 minutes)\nuv run ogc-sta-validate continuous --interval 1800\n```\n\n## Configuration\n\nThe tool supports **multiple configuration sources** with the following precedence (highest to lowest):\n\n1. **CLI arguments** (highest priority)\n2. **Environment variables**\n3. **YAML configuration file**\n4. **Default values** (lowest priority)\n\nThis flexible approach allows you to:\n- Store sensitive credentials in environment variables\n- Override configuration temporarily via CLI arguments\n- Maintain base configuration in YAML files\n- Use different configurations across environments (dev/staging/prod)\n\n### Method 1: YAML Configuration File\n\nThe primary configuration method uses YAML files. This is covered in the Quick Start section above.\n\n```yaml\n# config/config.yaml\nserver:\n url: \"http://localhost:8080/FROST-Server/v1.1\"\n timeout: 30\n auth:\n method: \"basic\"\n username: \"user\"\n password: \"pass\"\n\nvalidation:\n batch_size: 100\n stop_on_error: false\n```\n\nSee `config/config.example.yaml` for a complete configuration template with all available options.\n\n### Method 2: Environment Variables\n\nEnvironment variables can override YAML configuration values. This is especially useful for:\n- **Sensitive data**: Store credentials securely outside of config files\n- **CI/CD pipelines**: Configure differently across environments\n- **Docker deployments**: Use Docker environment variables\n\n#### Naming Convention\n\nEnvironment variables use the prefix `VALIDATOR_` and double underscores (`__`) for nested values:\n\n```bash\n# Format: VALIDATOR_<SECTION>__<KEY>\nVALIDATOR_SERVER__URL=http://example.com/v1.1\nVALIDATOR_SERVER__TIMEOUT=60\nVALIDATOR_VALIDATION__BATCH_SIZE=200\n```\n\n#### Quick Setup\n\n```bash\n# Copy the example environment file\ncp .env.example .env\n```\n\n#### Example Environment Variables\n\n```bash\n# Server configuration\nexport VALIDATOR_SERVER__URL=\"http://example.com/FROST-Server/v1.1\"\nexport VALIDATOR_SERVER__TIMEOUT=60\n\n# Authentication (recommended for sensitive credentials)\nexport VALIDATOR_SERVER__AUTH__METHOD=\"basic\"\nexport VALIDATOR_SERVER__AUTH__USERNAME=\"myuser\"\nexport VALIDATOR_SERVER__AUTH__PASSWORD=\"mypassword\"\n\n# Validation settings\nexport VALIDATOR_VALIDATION__BATCH_SIZE=200\nexport VALIDATOR_OUTPUT__FORMAT=\"json\"\n\n# Now run the tool - environment variables will be used\nuv run ogc-sta-validate test-connection\n```\n\nSee `.env.example` for all available environment variables.\n\n### Method 3: CLI Arguments\n\nCLI arguments provide the highest precedence.\n\n#### Common CLI Options\n\n```bash\n# Override server URL\nuv run ogc-sta-validate --server-url http://example.com/v1.1 test-connection\n\n# Override timeout\nuv run ogc-sta-validate --timeout 60 test-connection\n\n# Override batch size\nuv run ogc-sta-validate --batch-size 200 validate --entity-type Things\n\n# Combine multiple overrides\nuv run ogc-sta-validate \\\n --server-url http://example.com/v1.1 \\\n --timeout 60 \\\n --batch-size 200 \\\n validate --entity-type Things\n```\n\n#### Command-Specific Options\n\nEach command also has specific options:\n\n```bash\n# Validate command options\nuv run ogc-sta-validate validate \\\n --entity-type Things \\\n --limit 100 \\\n --output-format json \\\n --output-file report.json \\\n --filter \"properties/status eq 'active'\"\n\n# Continuous validation options\nuv run ogc-sta-validate continuous \\\n --interval 3600 \\\n --entity-types Things \\\n --entity-types Sensors \\\n --max-entities 500\n```\n\n### Getting Help\n\nView all available CLI options:\n\n```bash\n# Main help\nuv run ogc-sta-validate --help\n\n# Command-specific help\nuv run ogc-sta-validate validate --help\nuv run ogc-sta-validate continuous --help\n```\n\n## Validation Methodology\n\n### Approach\n\nThis tool implements a **schema-driven validation approach** that operates independently of the OGC SensorThings API server's internal validation. While the SensorThings API specification defines mandatory properties and data types, organizations often have additional requirements such as:\n\n- Custom property schemas for domain-specific metadata\n- Enumerated value constraints for controlled vocabularies\n- Pattern matching for identifiers following organizational conventions\n- Range validation for sensor measurements\n- Date/time format requirements beyond ISO 8601\n\nThe validation engine retrieves entities via the SensorThings API's standardized REST interface and applies JSON Schema rules without modifying the source data, making it suitable for both development and production environments.\n\n### Validation Process\n\n1. **Entity Retrieval**: Fetches entities from the SensorThings API server using HTTP GET requests with optional [OData filtering](https://www.odata.org/documentation/)\n2. **Schema Loading**: Loads JSON Schema definitions from the local filesystem, matched by entity type\n3. **Rule Application**: Applies JSON Schema validation rules using the `jsonschema` library\n4. **Error Aggregation**: Collects validation errors with property paths and rule type information\n5. **Report Generation**: Produces detailed reports in multiple formats (console, JSON, CSV)\n\n### Architecture\n\nThe validator implements a modular architecture with separation of concerns:\n\n```\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 CLI Interface \u2502 Command-line interface (Click framework)\n\u2502 (cli.py) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502\n \u251c\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n \u2502 \u2502 \u2502\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u2502\n\u2502 API Client \u2502 \u2502Schema Loader \u2502 \u2502 Validator Engine \u2502 \u2502\n\u2502(api_client.py) \u2502 \u2502(loader.py) \u2502 \u2502 (validator.py) \u2502 \u2502\n\u2502 \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n\u2502 - HTTP requests \u2502 \u2502- JSON Schema \u2502 \u2502 - Rule engine \u2502 \u2502\n\u2502 - Pagination \u2502 \u2502 discovery \u2502 \u2502 - Error handling \u2502 \u2502\n\u2502 - Auth strategies \u2502 \u2502- Validation \u2502 \u2502 - Type checking \u2502 \u2502\n\u2502 - OData filters \u2502 \u2502 \u2502 \u2502 \u2502 \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 \u2502\n \u2502 \u2502 \u2502 \u2502\n \u2502 \u2502 \u2502 \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2524\n \u2502\n \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2510\n \u2502 Validation Runner \u2502\n \u2502 (runner.py) \u2502\n \u2502 \u2502\n \u2502 - Orchestration & batch processing\u2502\n \u2502 - Multi-format report generation \u2502\n \u2502 - Error/warning categorization \u2502\n \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n```\n\n## JSON Schema Format\n\nValidation schemas use standard JSON Schema Draft 2020-12. This example demonstrates common validation features:\n\n```json\n{\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://schemas.sensorthings.org/things.json\",\n \"title\": \"Things Schema\",\n \"type\": \"object\",\n \"required\": [\"name\", \"description\"],\n \"properties\": {\n \"name\": {\n \"type\": \"string\",\n \"minLength\": 1,\n \"maxLength\": 100,\n \"description\": \"Thing name is mandatory\",\n \"x-errorMessage\": \"Thing must have a non-empty name (1-100 characters)\"\n },\n \"description\": {\n \"type\": \"string\",\n \"minLength\": 1\n },\n \"properties\": {\n \"type\": \"object\",\n \"required\": [\"thingId\"],\n \"properties\": {\n \"thingId\": {\n \"type\": \"string\",\n \"minLength\": 1\n },\n \"status\": {\n \"enum\": [\"active\", \"inactive\", \"maintenance\"],\n \"description\": \"Status must be one of the allowed values\"\n },\n \"code\": {\n \"type\": \"string\",\n \"pattern\": \"^[A-Z]{2}[0-9]{4}$\",\n \"description\": \"Building code format: two uppercase letters + four digits\"\n },\n \"temperature\": {\n \"type\": \"number\",\n \"minimum\": -50,\n \"maximum\": 100\n },\n \"timestamp\": {\n \"type\": \"string\",\n \"format\": \"date-time\"\n },\n \"coordinates\": {\n \"type\": \"array\",\n \"minItems\": 2,\n \"maxItems\": 3,\n \"description\": \"Coordinates [longitude, latitude, altitude?]\"\n },\n \"tags\": {\n \"type\": \"array\",\n \"minItems\": 1,\n \"maxItems\": 10\n }\n }\n }\n }\n}\n```\n\nSee [JSON Schema specification](https://json-schema.org/draft/2020-12/json-schema-validation.html) for complete documentation.\n\n### Custom Extensions\n\nThis validator supports custom JSON Schema extension fields (prefixed with `x-` per JSON Schema conventions) to enhance validation functionality:\n\n#### Schema-Level Extensions\n\n**`x-version`** (string, optional): Schema version identifier for tracking and documentation\n```json\n{\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"x-version\": \"2.0.0\",\n \"x-entityType\": \"Things\",\n \"title\": \"Things Schema\"\n}\n```\n\n**`x-entityType`** (string, optional): Specifies the SensorThings API entity type this schema validates\n```json\n{\n \"x-entityType\": \"Things\"\n}\n```\nUsed for entity type identification when the schema filename doesn't match the entity type.\n\n#### Property-Level Extensions\n\n**`x-errorMessage`** (string, optional): Custom error message shown when validation fails\n```json\n{\n \"type\": \"string\",\n \"pattern\": \"^de\\\\.sn\\\\.stlp\\\\..+\",\n \"x-errorMessage\": \"ID must start with Leipzig prefix (de.sn.stlp.)\"\n}\n```\n\n**`x-severity`** (string, optional): Validation severity level - either `\"error\"` or `\"warning\"`\n```json\n{\n \"type\": \"string\",\n \"pattern\": \"^[A-Z]{2}[0-9]{4}$\",\n \"x-errorMessage\": \"Recommended format: XX9999\",\n \"x-severity\": \"warning\"\n}\n```\n\n- **`error`** (default): Validation failure causes entity to be marked as invalid\n- **`warning`**: Validation failure is reported but entity remains valid\n\n#### Complete Example\n\n```json\n{\n \"$schema\": \"https://json-schema.org/draft/2020-12/schema\",\n \"$id\": \"https://schemas.sensorthings.org/things.json\",\n \"x-version\": \"2.0.0\",\n \"x-entityType\": \"Things\",\n \"title\": \"Things Schema\",\n \"type\": \"object\",\n \"required\": [\"name\"],\n \"properties\": {\n \"name\": {\n \"type\": \"string\",\n \"minLength\": 1,\n \"x-errorMessage\": \"Thing must have a non-empty name\"\n },\n \"properties\": {\n \"type\": \"object\",\n \"properties\": {\n \"thingId\": {\n \"type\": \"string\",\n \"pattern\": \"^de\\\\.sn\\\\.stlp\\\\..+\",\n \"x-errorMessage\": \"Thing ID must use Leipzig prefix\",\n \"x-severity\": \"error\"\n },\n \"installationDate\": {\n \"type\": \"string\",\n \"format\": \"date-time\",\n \"x-errorMessage\": \"Installation date should be in ISO 8601 format\",\n \"x-severity\": \"warning\"\n }\n }\n }\n }\n}\n```\n\n## CLI Commands\n\n### test-connection\nTest connection to the SensorThings API server.\n\n### validate\nValidate entities of a specific type.\n\nOptions:\n- `--entity-type`, `-t`: Entity type to validate (required)\n- `--entity-id`, `-i`: Specific entity ID to validate\n- `--schema`, `-s`: Schema file to use\n- `--limit`, `-l`: Maximum number of entities to validate\n- `--filter`, `-f`: OData filter expression\n- `--output-format`: Output format (console, json, csv)\n- `--output-file`, `-o`: Output file path\n- `--include-valid`: Include valid entities in output\n\n### validate-all\nValidate all entity types.\n\nOptions:\n- `--entity-types`, `-t`: Specific entity types to validate\n- `--limit-per-type`, `-l`: Maximum entities per type\n- `--output-format`: Output format\n- `--output-file`, `-o`: Output file path\n- `--include-valid`: Include valid entities in output\n\n### continuous\nRun continuous validation.\n\nOptions:\n- `--interval`, `-i`: Validation interval in seconds\n- `--entity-types`, `-t`: Entity types to validate\n- `--max-entities`, `-m`: Maximum entities per run\n\n### validate-schema\nValidate schema files.\n\nOptions:\n- `--schema-file`, `-s`: Specific schema file to validate\n\n### list-entities\nList available entity types and their counts.\n\n## Development\n\n### Setup Development Environment\n```bash\n# Install with development dependencies\nuv sync --dev\n\n# Run tests\nuv run pytest\n\n# Format code and fix imports (replaces black + isort)\nuv run ruff format .\n\n# Lint code (replaces flake8)\nuv run ruff check .\n\n# Lint and auto-fix issues\nuv run ruff check --fix .\n```\n\n### Adding Dependencies\n```bash\n# Add a new dependency\nuv add requests\n\n# Add a development dependency\nuv add --dev pytest\n\n# Update dependencies\nuv sync\n```\n\n## Example Validation Schemas\n\nThe repository includes example validation schemas in `examples/schemas/` demonstrating validation patterns for common SensorThings API entity types:\n\n- `examples/schemas/thing_schema.json` - Validation schema for Thing entities with custom property constraints\n- `examples/schemas/sensor_schema.json` - Validation schema for Sensor entities with metadata requirements\n- `examples/schemas/observation_schema.json` - Validation schema for Observation entities with result value constraints\n- `examples/schemas/datastream_schema.json` - Validation schema for Datastream entities with unit of measurement validation\n\nCopy these to your `schemas/` directory and customize them for your organization:\n\n```bash\ncp examples/schemas/*.json schemas/\n```\n\nSee `examples/schemas/README.md` for customization guidance.\n\n## Output Formats\n\n### Console Output\nHuman-readable text format with validation summary and error details.\n\n### JSON Output\nStructured JSON format suitable for programmatic processing:\n```json\n{\n \"summary\": {\n \"total_entities\": 100,\n \"valid_entities\": 95,\n \"invalid_entities\": 5,\n \"total_errors\": 8,\n \"total_warnings\": 2\n },\n \"results\": [\n {\n \"entityId\": \"123\",\n \"entityType\": \"Things\",\n \"isValid\": false,\n \"errors\": [\n {\n \"property_path\": \"properties.building\",\n \"message\": \"Required field is missing\",\n \"validator\": \"required\"\n }\n ]\n }\n ]\n}\n```\n\n### CSV Output\nTabular format suitable for spreadsheet analysis with columns:\n- EntityType, EntityId, IsValid, ErrorCount, WarningCount, PropertyPath, ErrorType, Message, Severity\n\n## Troubleshooting\n\n### Connection Issues\n- Verify the SensorThings API server URL conforms to the specification (e.g., ends with `/v1.0` or `/v1.1`)\n- Check network connectivity and firewall rules\n- Ensure authentication credentials are correct for your server's authentication method\n\n### Schema Issues\n- Use `validate-schema` command to check JSON Schema validity against Draft 2020-12\n- Verify JSON syntax and required meta-properties (`$schema`, `type`, etc.)\n- Ensure custom extensions (`x-errorMessage`, `x-severity`) follow the expected format\n\n### Performance\n- Adjust `batch_size` based on entity complexity and network latency (typical range: 50-500)\n- Use OData `$filter` expressions to limit validation scope to relevant entities\n\n## License\n\nSee LICENSE file for details.\n\n## Contributing\n\nContributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.",
"bugtrack_url": null,
"license": "MIT",
"summary": "Validation tool for OGC SensorThings API entities against configurable JSON schemas",
"version": "0.1.0",
"project_urls": {
"Changelog": "https://github.com/janbeckert/ogc-sta-schema-validator/blob/main/CHANGELOG.md",
"Documentation": "https://github.com/janbeckert/ogc-sta-schema-validator#readme",
"Homepage": "https://github.com/janbeckert/ogc-sta-schema-validator",
"Issues": "https://github.com/janbeckert/ogc-sta-schema-validator/issues",
"Repository": "https://github.com/janbeckert/ogc-sta-schema-validator"
},
"split_keywords": [
"data-quality",
" frost-server",
" iot",
" json-schema",
" ogc",
" sensor",
" sensorthings",
" validation"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "429fa024b40da8cadbb147d66b65920f43b71cb7c614288e63576b1d580066a6",
"md5": "af54839b2e5393b91ea9ea1cc1e827c0",
"sha256": "b6cf405f0d59ba87508c26d98794b7115b75c6024b7d6d1887ec3b679e1df093"
},
"downloads": -1,
"filename": "ogc_sta_schema_validator-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "af54839b2e5393b91ea9ea1cc1e827c0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 32960,
"upload_time": "2025-10-13T19:21:16",
"upload_time_iso_8601": "2025-10-13T19:21:16.405788Z",
"url": "https://files.pythonhosted.org/packages/42/9f/a024b40da8cadbb147d66b65920f43b71cb7c614288e63576b1d580066a6/ogc_sta_schema_validator-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "203c3202479d119c4bf819be5769da22427cbcdfe29d7ddd97ef7ff678c97ec9",
"md5": "2358dd97fa3d9c9c9def69c62dc19544",
"sha256": "623b27eb5c7c57d255db4bb34a9a8539f8f622eee929c721d70e5bf212bda176"
},
"downloads": -1,
"filename": "ogc_sta_schema_validator-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "2358dd97fa3d9c9c9def69c62dc19544",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 121148,
"upload_time": "2025-10-13T19:21:17",
"upload_time_iso_8601": "2025-10-13T19:21:17.731352Z",
"url": "https://files.pythonhosted.org/packages/20/3c/3202479d119c4bf819be5769da22427cbcdfe29d7ddd97ef7ff678c97ec9/ogc_sta_schema_validator-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-13 19:21:17",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "janbeckert",
"github_project": "ogc-sta-schema-validator",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "ogc-sta-schema-validator"
}
JSON
