<!--
© 2025 KR-Labs. All rights reserved.
KR-Labs™ is a trademark of Quipu Research Labs, LLC, a subsidiary of Sudiata Giddasira, Inc.
SPDX-License-Identifier: Apache-2.0
-->
<div align="center">
<img src="docs/assets/KRLabs_WebLogo.png" alt="KR-Labs Logo" width="400"/>
</div>
# KRL Data Connectors
<div align="center">
<p>
<strong>Production-grade connectors for socioeconomic, demographic, and policy data infrastructure.</strong>
</p>
<p>
<a href="https://opensource.org/licenses/Apache-2.0"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="License"></a>
<a href="https://www.python.org/downloads/"><img src="https://img.shields.io/badge/python-3.9+-blue.svg" alt="Python 3.9+"></a>
<a href="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/build-and-sign.yml"><img src="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/build-and-sign.yml/badge.svg" alt="Build"></a>
<a href="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/tests.yml"><img src="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/tests.yml/badge.svg" alt="Tests"></a>
<a href="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/comprehensive-testing.yml"><img src="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/comprehensive-testing.yml/badge.svg" alt="Comprehensive Tests"></a>
<a href="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/lint.yml"><img src="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/lint.yml/badge.svg" alt="Lint"></a>
<a href="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/security-checks.yml"><img src="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/security-checks.yml/badge.svg" alt="Security"></a>
<a href="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/license-compliance.yml"><img src="https://github.com/KR-Labs/krl-data-connectors/actions/workflows/license-compliance.yml/badge.svg" alt="License Compliance"></a>
<a href="https://github.com/psf/black"><img src="https://img.shields.io/badge/code%20style-black-000000.svg" alt="Code style: black"></a>
</p>
<p>
<a href="#installation">Installation</a> •
<a href="#quick-start">Quick Start</a> •
<a href="https://krl-data-connectors.readthedocs.io">Documentation</a> •
<a href="./examples/">Examples</a> •
<a href="./CONTRIBUTING.md">Contributing</a>
</p>
</div>
---
## Overview
KRL Data Connectors provide a unified, production-grade interface for institutional access to socioeconomic, demographic, health, and environmental datasets. Designed for operational discipline and engineered reliability, these connectors form the backbone of the [KRL Analytics Suite](https://krlabs.dev), supporting robust economic analysis, causal inference, and policy evaluation at scale.
Built on a foundation of 73 source modules, 52 connector implementations across 14 domains, and validated by over 2,098 automated tests, this library represents a comprehensive data infrastructure solution for research and production environments.
### Infrastructure Guarantees
KRL Data Connectors guarantee:
- **Unified API Surface**: Consistent, type-safe interfaces across all 52 connector implementations spanning 14 domains.
- **Operational Reliability**: Structured logging, resilient error handling, and automated retry mechanisms backed by comprehensive integration testing.
- **Type Safety and Validation**: Complete type hints across 73 source modules with runtime input validation and contract testing.
- **Optimized Caching**: Intelligent caching to reduce redundant data access and improve throughput, with configurable TTL and multiple backend support.
- **Automated Metadata Extraction**: Standardized profiling of dataset metadata for downstream integration and data cataloging.
- **Comprehensive Testing**: 2,098 automated tests across unit, integration, and contract layers, maintaining sustained coverage above 78%.
- **Secure Credential Management**: Institutional-grade credential resolution with multiple configuration sources and environment isolation.
- **Production Documentation**: 16 quickstart notebooks, comprehensive API documentation, and operational runbooks for deployment scenarios.
### Supported Data Sources
KRL Data Connectors provide institutional-grade access to 52 production-ready connector implementations across 14 domains. Each domain is represented by connectors engineered for reliability, operational consistency, and seamless integration with analytical workflows. The library encompasses 73 source modules and is validated by 2,098 automated tests ensuring data integrity and API stability.
#### Economic & Financial Data (8 connectors)
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| FRED | Economics | Yes | Daily/Real-time | 800K+ series | Production |
| BLS | Labor | Recommended | Monthly | National/State | Production |
| BEA | Economics | Yes | Quarterly/Annual | National/Regional | Production |
| OECD | International | No | Varies | Country-level | Production |
| World Bank | International | No | Annual | Country-level | Production |
| SEC | Financial | No | Real-time | Public filings | Production |
| Treasury | Financial | No | Daily | Federal finances | Production |
| FDIC | Banking | No | Quarterly | Bank data | Production |
#### Demographic & Labor Data (3 connectors)
Institutional connectors for core demographic and labor datasets.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| Census ACS | Demographics | Optional | Annual | All US geographies | Production |
| Census CBP | Business | Optional | Annual | County-level | Production |
| Census LEHD | Employment | No | Quarterly | County-level | Production |
#### Health & Wellbeing Data (5 connectors)
Connectors for health outcomes, provider, and wellbeing datasets.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| HRSA | Health | No | Annual | HPSA/MUA/P | Production |
| CDC WONDER | Health | No | Varies | County-level | Production |
| County Health Rankings| Health | No | Annual | County-level | Production |
| FDA | Health | No | Real-time | Drugs/devices | Production |
| NIH | Research | No | Daily | Grants/projects | Production |
#### Environmental & Climate Data (5 connectors)
Production connectors for environmental, air, and climate data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| EPA EJScreen | Environment | No | Annual | Block group | Production |
| EPA Air Quality | Environment | No | Hourly/Real-time | Station-level | Production |
| EPA Superfund | Environment | No | Real-time | Site-level | Production |
| EPA Water Quality | Environment | No | Real-time | Facility-level | Production |
| NOAA Climate | Climate | No | Daily | Station-level | Production |
#### Education Data (3 connectors)
Institutional connectors for education and postsecondary data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| NCES | Education | No | Annual | School-level | Production |
| College Scorecard | Education | Yes | Annual | Institution | Production |
| IPEDS | Education | No | Annual | Institution | Production |
#### Housing & Urban Data (2 connectors)
Production-ready connectors for housing and urban indicators.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| HUD Fair Market Rent | Housing | Yes | Annual | Metro/County | Production |
| Zillow Research | Housing | No | Monthly | Metro/ZIP | Production |
#### Agricultural Data (2 connectors)
Connectors for agricultural and food environment data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| USDA Food Atlas | Agricultural | Yes | Annual | County-level | Production |
| USDA NASS | Agricultural | Yes | Varies | National/State | Production |
#### Crime & Justice Data (3 connectors)
Production connectors for crime and justice data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| FBI UCR | Crime | Recommended | Annual | Agency-level | Production |
| Bureau of Justice | Justice | No | Annual | National | Production |
| Victims of Crime | Justice | No | Annual | State-level | Production |
#### Energy Data (1 connector)
Institutional connector for energy datasets.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| EIA | Energy | Yes | Real-time | National/State | Production |
#### Science & Research Data (2 connectors)
Connectors for geoscience and research awards data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| USGS | Geoscience | No | Real-time | National | Production |
| NSF | Research | No | Daily | Awards/grants | Production |
#### Transportation Data (1 connector)
Production connector for aviation and transportation data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| FAA | Aviation | No | Real-time | Airport/flight | Production |
#### Labor Safety Data (1 connector)
Connector for labor safety and inspection data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| OSHA | Safety | No | Real-time | Inspections | Production |
#### Social Services Data (2 connectors)
Connectors for social services and benefits data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| Social Security Admin | Social | No | Annual | National | Production |
| ACF | Social | No | Annual | State/County | Production |
#### Veterans Services Data (1 connector)
Connector for veterans services and benefits data.
| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |
|----------------------|---------------|---------------|---------------------|---------------------|--------------|
| VA | Veterans | No | Real-time | Facilities/benefits | Production |
**Total: 52 Production-Ready Connector Implementations** | All connectors maintained with continuous integration, automated testing, and security scanning.
---
## Security & IP Protection
KRL Data Connectors are developed and maintained under a comprehensive, audit-grade security and intellectual property (IP) protection framework. The repository implements a 10-layer defense strategy with 6 automated CI/CD workflows, continuous security scanning, and zero-tolerance secret detection to ensure compliance, confidentiality, and operational integrity.
### Compliance and Protection Layers
| Layer | Control Area | Implementation | Coverage |
|-------|----------------|----------------|----------|
| 1 | Legal Protection (Copyright, Trademark, License) | Automated header validation | 100% of 136 total files |
| 2 | Technical Protection (Secret Scanning) | Gitleaks, GitHub Advanced Security, Pre-commit | Repository-wide, all commits |
| 3 | Build Verification (CI/CD Security) | 6 GitHub Actions workflows | All pull requests and pushes |
| 4 | License Compliance | Automated dependency scanning | Apache 2.0, GPL/AGPL exclusion |
| 5 | Continuous Monitoring & Response | Multi-scanner validation, SARIF reporting | Real-time detection and alerts |
| 6 | Static Application Security Testing (SAST) | Bandit, Safety, CodeQL | All Python code |
| 7 | Dependency Security | Trivy, pip-audit | All dependencies, daily scans |
| 8 | Type & Contract Verification | mypy strict mode, pydantic validation | All public APIs |
| 9 | Code Quality Gates | Black, isort, flake8, ruff | Pre-commit and CI enforcement |
| 10 | Documentation & Auditability | Structured logging, SBOM generation | All operations logged |
### Automated Security Controls
All code contributions and repository changes are subject to automated validation through 6 continuous integration workflows:
- **Secret Detection**: Gitleaks, GitHub Advanced Security, pre-commit hooks with zero false-negative tolerance
- **Vulnerability Assessment**: Trivy container scanning, CodeQL semantic analysis, SARIF reporting
- **Dependency Security**: CVE scanning, pip-audit, safety checks with automatic PR creation for updates
- **License Compliance**: GPL/AGPL blocklisting, Apache 2.0 compatibility verification, SBOM generation
- **Static Analysis**: Bandit security linting, Safety vulnerability scanning, mypy strict type checking
- **Code Quality**: Black formatting (100% compliance), isort import ordering, flake8 style enforcement
- **Copyright Validation**: Automated header verification across all source files with trademark protection
### Repository Validation Status
Continuous validation across 2,098 automated tests confirms:
- Zero secrets detected across 1,000+ commit history
- 100% of 136 files with compliant copyright headers
- Full adherence to Apache 2.0 licensing with no GPL/AGPL dependencies
- 78%+ sustained test coverage across all connectors
- All 6 CI/CD workflows passing with automated quality gates
### Contributor Compliance
All contributions undergo rigorous automated validation:
- Credentials and secrets automatically blocked at commit time via pre-commit hooks
- All pull requests subject to 6 CI/CD workflows including security, testing, and compliance validation
- Pre-commit hooks and automated tooling required for all contributors
- Signed Contributor License Agreement (CLA) required for code contributions
- Automated dependency updates via Dependabot with security patch prioritization
For detailed security practices, vulnerability reporting procedures, and compliance documentation, refer to [SECURITY.md](./SECURITY.md).
---
## Installation
To ensure seamless integration with institutional environments, KRL Data Connectors support multiple installation profiles tailored for production, development, and extended use cases.
```bash
# Basic installation
pip install krl-data-connectors
# With all optional dependencies
pip install krl-data-connectors[all]
# Development installation
pip install krl-data-connectors[dev]
```
---
## Quick Start
The following examples demonstrate reproducible initialization and usage of KRL Data Connectors for primary data sources. All connectors are designed for direct integration into institutional analytics pipelines.
### County Business Patterns (CBP)
```python
from krl_data_connectors import CountyBusinessPatternsConnector
# Initialize connector using environment variable for API key
cbp = CountyBusinessPatternsConnector()
# Retrieve retail trade data for Rhode Island (FIPS 44, NAICS 44)
retail_data = cbp.get_state_data(
year=2021,
state='44',
naics='44'
)
print(f"Records: {len(retail_data)}")
print(retail_data[['NAICS2017', 'ESTAB', 'EMP', 'PAYANN']].head())
```
### LEHD Origin-Destination
```python
from krl_data_connectors import LEHDConnector
# Initialize connector
lehd = LEHDConnector()
# Retrieve origin-destination employment flows for RI, 2021
od_data = lehd.get_od_data(
state='ri',
year=2021,
job_type='JT00',
segment='S000'
)
print(f"Origin-destination pairs: {len(od_data)}")
print(od_data[['w_geocode', 'h_geocode', 'S000', 'SA01']].head())
```
### FRED
```python
from krl_data_connectors import FREDConnector
# Initialize connector using FRED_API_KEY environment variable
fred = FREDConnector()
# Fetch unemployment rate time series
unemployment = fred.get_series(
series_id="UNRATE",
observation_start="2020-01-01",
observation_end="2023-12-31"
)
print(unemployment.head())
```
### BLS
```python
from krl_data_connectors import BLSConnector
# Initialize connector using BLS_API_KEY environment variable
bls = BLSConnector()
# Retrieve unemployment rate for selected states
unemployment = bls.get_series(
series_ids=['LASST060000000000003', 'LASST440000000000003'],
start_year=2020,
end_year=2023
)
print(unemployment.head())
```
### BEA
```python
from krl_data_connectors import BEAConnector
# Initialize connector using BEA_API_KEY environment variable
bea = BEAConnector()
# Retrieve GDP by state for 2021
gdp_data = bea.get_data(
dataset='Regional',
method='GetData',
TableName='SAGDP2N',
LineCode=1,
Year='2021',
GeoFips='STATE'
)
print(gdp_data.head())
```
### Caching and Base Connector
All connectors inherit from `BaseConnector`, which standardizes caching, configuration, and logging for operational consistency.
```python
from krl_data_connectors import FREDConnector
# Enable file-based caching with 1-hour TTL
fred = FREDConnector(
api_key="your_api_key",
cache_dir="/tmp/fred_cache",
cache_ttl=3600
)
# Cached responses are handled automatically
data1 = fred.get_series("UNRATE") # API call
data2 = fred.get_series("UNRATE") # Served from cache
# Retrieve cache statistics
stats = fred.cache.get_stats()
print(f"Cache hit rate: {stats['hit_rate']:.1f}%")
```
---
## Architecture
KRL Data Connectors are built on a modular, auditable architecture supporting institutional-scale deployment. The codebase spans 73 source modules organized across 14 domain-specific directories, with all 52 connector implementations deriving from a unified `BaseConnector` that enforces interface consistency, operational controls, and standardized behavior.
### BaseConnector Foundation
The `BaseConnector` class provides institutional-grade infrastructure for all 52 connector implementations:
- **Structured Logging**: JSON-formatted logs with complete request/response metadata and audit trails
- **Consistent Configuration**: Environment variable and YAML-based configuration for credentials, caching, and logging
- **Standardized Caching**: File-based and memory caching with configurable TTL across all connectors
- **Error Handling**: Uniform exception handling, retry logic, and rate limit management
- **Type Safety**: Full mypy strict mode compliance with pydantic validation across all public APIs
- **Request Management**: HTTP session pooling, connection reuse, and automated timeout handling
```python
from abc import ABC, abstractmethod
from krl_core import get_logger, ConfigManager, FileCache
class BaseConnector(ABC):
"""Abstract base class for all data connectors. Enforces interface and operational consistency."""
def __init__(self, api_key=None, cache_dir=None, cache_ttl=3600):
self.logger = get_logger(self.__class__.__name__)
self.config = ConfigManager()
self.cache = FileCache(
cache_dir=cache_dir,
default_ttl=cache_ttl,
namespace=self.__class__.__name__.lower()
)
# ... initialization
```
---
## API Key Management
KRL Data Connectors implement secure, automated credential resolution for institutional and development use. The credential management system ensures operational consistency and auditability. See [API_KEY_SETUP.md](./API_KEY_SETUP.md) for procedural details.
### Credential Resolution Order
API credentials are resolved in the following order:
1. **Environment Variables** (preferred for production)
2. **Configuration file** at `~/.krl/apikeys` (recommended for development/testing)
3. **Direct assignment in code** (permitted for testing only)
#### Example: Environment Variables
```bash
export BEA_API_KEY="your_bea_key"
export FRED_API_KEY="your_fred_key"
export BLS_API_KEY="your_bls_key"
export CENSUS_API_KEY="your_census_key"
```
#### Example: Configuration File
```bash
mkdir -p ~/.krl
cat > ~/.krl/apikeys << EOF
BEA API KEY: your_bea_key
FRED API KEY: your_fred_key
BLS API KEY: your_bls_key
CENSUS API: your_census_key
EOF
chmod 600 ~/.krl/apikeys
```
#### Obtaining API Keys
| Service | Required? | Registration URL |
|-------------------|--------------|-------------------------------------------------------|
| CBP/Census | Optional | https://api.census.gov/data/key_signup.html |
| FRED | Yes | https://fred.stlouisfed.org/docs/api/api_key.html |
| BLS | Recommended* | https://www.bls.gov/developers/home.htm |
| BEA | Yes | https://apps.bea.gov/api/signup/ |
| LEHD | No | N/A |
*BLS is accessible without a key but with reduced rate limits.
#### Configuration Utilities
Automatic discovery and management of configuration files are supported:
```python
from krl_data_connectors import find_config_file, BEAConnector
config_path = find_config_file('apikeys')
print(f"Config found at: {config_path}")
# Connectors resolve credentials from configuration or environment
bea = BEAConnector()
```
---
## Configuration
KRL Data Connectors support robust, institution-ready configuration via environment variables and YAML files. This enables precise management of credentials, caching, and logging for reproducible deployments.
### Environment Variables
```bash
# API Keys
export CENSUS_API_KEY="your_census_key"
export FRED_API_KEY="your_fred_key"
export BLS_API_KEY="your_bls_key"
export BEA_API_KEY="your_bea_key"
# Cache settings
export KRL_CACHE_DIR="~/.krl_cache"
export KRL_CACHE_TTL="3600"
# Logging
export KRL_LOG_LEVEL="INFO"
export KRL_LOG_FORMAT="json"
```
### YAML Configuration File
```yaml
fred:
api_key: "your_fred_key"
base_url: "https://api.stlouisfed.org/fred"
timeout: 30
census:
api_key: "your_census_key"
base_url: "https://api.census.gov/data"
cache:
directory: "~/.krl_cache"
ttl: 3600
logging:
level: "INFO"
format: "json"
```
Apply configuration in code:
```python
from krl_core import ConfigManager
config = ConfigManager("config.yaml")
fred = FREDConnector(api_key=config.get("fred.api_key"))
```
---
## Connector Catalog
KRL Data Connectors enable reliable, scalable integration with the following data sources. All connectors are engineered to institutional standards for reliability and seamless analytics integration.
### Production-Ready Connectors
- **County Business Patterns (CBP):** Establishment and employment statistics by industry and geography. [examples/cbp_quickstart.ipynb](examples/)
- **LEHD Origin-Destination:** Worker flows and employment demographics. [examples/lehd_quickstart.ipynb](examples/)
- **FRED:** Economic time series and metadata.
- **BLS:** Labor market and inflation statistics.
- **BEA:** GDP, regional accounts, and personal income.
- **EPA EJScreen:** Environmental justice indicators. [examples/ejscreen_quickstart.ipynb](examples/ejscreen_quickstart.ipynb)
- **HRSA:** Health Professional Shortage Areas, MUA/P, FQHC. [examples/hrsa_quickstart.ipynb](examples/hrsa_quickstart.ipynb)
- **County Health Rankings:** County-level health measures. [examples/chr_quickstart.ipynb](examples/chr_quickstart.ipynb)
- **EPA Air Quality / AirNow:** Real-time and historical AQI. [examples/air_quality_quickstart.ipynb](examples/air_quality_quickstart.ipynb)
- **Zillow Research:** Housing price and rent indices. [examples/zillow_quickstart.ipynb](examples/zillow_quickstart.ipynb)
- **HUD Fair Market Rents:** Rental affordability and income limits. [examples/hud_fmr_quickstart.ipynb](examples/hud_fmr_quickstart.ipynb)
- **FBI UCR:** Crime statistics and arrest data. [examples/fbi_ucr_quickstart.ipynb](examples/fbi_ucr_quickstart.ipynb)
- **NCES:** School and district education statistics. [examples/nces_quickstart.ipynb](examples/nces_quickstart.ipynb)
### In Development and Planned
- **CDC WONDER:** Mortality and natality data (API non-functional; web interface recommended).
- **USDA Food Environment Atlas:** Food access, insecurity, and local food systems.
- **OECD, World Bank, College Scorecard, IPEDS, Superfund Sites, and additional sources:** Refer to [ROADMAP.md](ROADMAP.md) for the complete development roadmap.
---
## Roadmap and Quality Standards
KRL Data Connectors are developed in accordance with a structured roadmap, with 52 connector implementations currently in production across major institutional domains. Prioritization is driven by institutional requirements, API stability, and comprehensive domain coverage.
**Quality Standards (Applied to All 52 Connectors):**
- ≥78% test coverage sustained across 2,098 automated tests
- Full type hints with mypy strict mode compliance on all public methods
- Pydantic validation for all API contracts and data models
- Robust error handling with structured diagnostics and retry logic
- Configurable caching (file-based and memory) with intelligent TTL management
- Structured JSON logging for operational transparency and audit trails
- Comprehensive documentation with 16 quickstart Jupyter notebooks
- Secure API key management with automated validation and credential resolution
For implementation milestones and API specifications, consult [ROADMAP.md](ROADMAP.md).
---
## Testing
KRL Data Connectors utilize a 10-layer, open-source testing architecture aligned with institutional best practices. All tests and validation tools are fully auditable and reproducible.
### Testing Model
| Layer | Purpose | Tools | Status |
|-------|--------|-------|--------|
| 1. Unit Tests | Function correctness | pytest, hypothesis | Implemented (2,098 tests, 78%+ coverage) |
| 2. Integration | Component interaction | pytest, requests-mock | Implemented |
| 3. E2E Tests | Workflow validation | playwright | Planned |
| 4. Performance | Load/stress testing | locust, pytest-benchmark | Planned |
| 5. SAST | Static security analysis | bandit, safety, mypy | Active in CI |
| 6. DAST | Runtime security testing | OWASP ZAP | Planned |
| 7. Mutation | Test quality measurement | mutmut, hypothesis | Planned |
| 8. Contract | Type/interface validation | pydantic, mypy strict | Active in CI |
| 9. Penetration | Security assessment | metasploit, burp | Annual |
| 10. Monitoring | Continuous validation | GitHub Actions (6 workflows), Snyk | Active |
### Test Execution
```bash
# Run all tests
make test
# Run with coverage
make coverage
# Run security scans
make security
# Run type checking
make type-check
# Simulate full CI pipeline
make ci
# List available commands
make help
```
### Coverage Objectives
- Current: 78%+ overall test coverage sustained across 2,098 automated tests
- Target: ≥90% line coverage, ≥85% branch coverage across all 52 connector implementations
- Mutation: ≥90% kill rate for test quality validation
For detailed procedures, see [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md).
---
## Development
All development must ensure reproducibility and full compliance with institutional contribution standards. Establish a local environment and follow the verified workflow:
```bash
# Clone the repository
git clone https://github.com/KR-Labs/krl-data-connectors.git
cd krl-data-connectors
# Create and activate a virtual environment
python -m venv .venv
source .venv/bin/activate # or `.venv\Scripts\activate` on Windows
# Install development and test dependencies
pip install -e ".[dev,test]"
# Run pre-commit hooks for code compliance
pre-commit install
pre-commit run --all-files
# Execute tests
pytest
# Build documentation
cd docs && make html
```
---
## Contributing
All contributions must maintain reproducibility, compliance, and operational reliability. Review [CONTRIBUTING.md](CONTRIBUTING.md) before submitting changes. A signed [Contributor License Agreement (CLA)](https://krlabs.dev/cla) is required for all code contributions.
---
## License
KRL Data Connectors are distributed under the **Apache License 2.0**. For the full license text, refer to the [LICENSE](LICENSE) file.
**Key License Terms:**
- Permits commercial use, modification, and redistribution
- Patent grant included
- Compatible with proprietary software
---
## Support
Use the following channels for verified support and communication:
- **Documentation:** https://docs.krlabs.dev/data-connectors
- **Issue Tracker:** https://github.com/KR-Labs/krl-data-connectors/issues
- **Discussions:** https://github.com/KR-Labs/krl-data-connectors/discussions
- **Email:** support@krlabs.dev
---
## Related Projects
KRL Data Connectors are a core component of the KR-Labs analytics infrastructure ecosystem:
- **[krl-open-core](https://github.com/KR-Labs/krl-open-core):** Logging, configuration, and caching utilities
- **[krl-model-zoo](https://github.com/KR-Labs/krl-model-zoo):** Causal inference and forecasting models
- **[krl-dashboard](https://github.com/KR-Labs/krl-dashboard):** Interactive analytics and visualization platform
- **[krl-tutorials](https://github.com/KR-Labs/krl-tutorials):** Reproducible example workflows and onboarding materials
---
## Citation
For institutional or research citation of KRL Data Connectors, use the following BibTeX entry:
```bibtex
@software{krl_data_connectors,
title = {KRL Data Connectors: Standardized Interfaces for Economic and Social Data},
author = {KR-Labs},
year = {2025},
url = {https://github.com/KR-Labs/krl-data-connectors},
license = {Apache-2.0}
}
```
---
**Engineered for reproducibility, scalability, and institutional trust by [KR-Labs](https://krlabs.dev)**
© 2025 KR-Labs. All rights reserved.
KR-Labs is a trademark of Quipu Research Labs, LLC, a subsidiary of Sudiata Giddasira, Inc.
Raw data
{
"_id": null,
"home_page": null,
"name": "krl-data-connectors",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": "KR-Labs Foundation <dev@krlabs.dev>",
"keywords": "economics, data, api, fred, census, bls, world-bank, oecd, connectors",
"author": null,
"author_email": "KR-Labs Foundation <dev@krlabs.dev>",
"download_url": "https://files.pythonhosted.org/packages/6a/ba/703a940a4927502a63143d6bbba752f38ad92912d5ed759814351dde5afb/krl_data_connectors-0.3.3.tar.gz",
"platform": null,
"description": "<!--\n\u00a9 2025 KR-Labs. All rights reserved. \nKR-Labs\u2122 is a trademark of Quipu Research Labs, LLC, a subsidiary of Sudiata Giddasira, Inc.\n\nSPDX-License-Identifier: Apache-2.0\n-->\n\n<div align=\"center\">\n <img src=\"docs/assets/KRLabs_WebLogo.png\" alt=\"KR-Labs Logo\" width=\"400\"/>\n</div>\n\n# KRL Data Connectors\n\n<div align=\"center\">\n\n<p>\n<strong>Production-grade connectors for socioeconomic, demographic, and policy data infrastructure.</strong>\n</p>\n\n<p>\n <a href=\"https://opensource.org/licenses/Apache-2.0\"><img src=\"https://img.shields.io/badge/License-Apache%202.0-blue.svg\" alt=\"License\"></a>\n <a href=\"https://www.python.org/downloads/\"><img src=\"https://img.shields.io/badge/python-3.9+-blue.svg\" alt=\"Python 3.9+\"></a>\n <a href=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/build-and-sign.yml\"><img src=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/build-and-sign.yml/badge.svg\" alt=\"Build\"></a>\n <a href=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/tests.yml\"><img src=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/tests.yml/badge.svg\" alt=\"Tests\"></a>\n <a href=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/comprehensive-testing.yml\"><img src=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/comprehensive-testing.yml/badge.svg\" alt=\"Comprehensive Tests\"></a>\n <a href=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/lint.yml\"><img src=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/lint.yml/badge.svg\" alt=\"Lint\"></a>\n <a href=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/security-checks.yml\"><img src=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/security-checks.yml/badge.svg\" alt=\"Security\"></a>\n <a href=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/license-compliance.yml\"><img src=\"https://github.com/KR-Labs/krl-data-connectors/actions/workflows/license-compliance.yml/badge.svg\" alt=\"License Compliance\"></a>\n <a href=\"https://github.com/psf/black\"><img src=\"https://img.shields.io/badge/code%20style-black-000000.svg\" alt=\"Code style: black\"></a>\n</p>\n\n<p>\n <a href=\"#installation\">Installation</a> \u2022\n <a href=\"#quick-start\">Quick Start</a> \u2022\n <a href=\"https://krl-data-connectors.readthedocs.io\">Documentation</a> \u2022\n <a href=\"./examples/\">Examples</a> \u2022\n <a href=\"./CONTRIBUTING.md\">Contributing</a>\n</p>\n\n</div>\n\n---\n\n## Overview\n\nKRL Data Connectors provide a unified, production-grade interface for institutional access to socioeconomic, demographic, health, and environmental datasets. Designed for operational discipline and engineered reliability, these connectors form the backbone of the [KRL Analytics Suite](https://krlabs.dev), supporting robust economic analysis, causal inference, and policy evaluation at scale.\n\nBuilt on a foundation of 73 source modules, 52 connector implementations across 14 domains, and validated by over 2,098 automated tests, this library represents a comprehensive data infrastructure solution for research and production environments.\n\n### Infrastructure Guarantees\n\nKRL Data Connectors guarantee:\n- **Unified API Surface**: Consistent, type-safe interfaces across all 52 connector implementations spanning 14 domains.\n- **Operational Reliability**: Structured logging, resilient error handling, and automated retry mechanisms backed by comprehensive integration testing.\n- **Type Safety and Validation**: Complete type hints across 73 source modules with runtime input validation and contract testing.\n- **Optimized Caching**: Intelligent caching to reduce redundant data access and improve throughput, with configurable TTL and multiple backend support.\n- **Automated Metadata Extraction**: Standardized profiling of dataset metadata for downstream integration and data cataloging.\n- **Comprehensive Testing**: 2,098 automated tests across unit, integration, and contract layers, maintaining sustained coverage above 78%.\n- **Secure Credential Management**: Institutional-grade credential resolution with multiple configuration sources and environment isolation.\n- **Production Documentation**: 16 quickstart notebooks, comprehensive API documentation, and operational runbooks for deployment scenarios.\n\n### Supported Data Sources\n\nKRL Data Connectors provide institutional-grade access to 52 production-ready connector implementations across 14 domains. Each domain is represented by connectors engineered for reliability, operational consistency, and seamless integration with analytical workflows. The library encompasses 73 source modules and is validated by 2,098 automated tests ensuring data integrity and API stability.\n\n#### Economic & Financial Data (8 connectors)\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| FRED | Economics | Yes | Daily/Real-time | 800K+ series | Production |\n| BLS | Labor | Recommended | Monthly | National/State | Production |\n| BEA | Economics | Yes | Quarterly/Annual | National/Regional | Production |\n| OECD | International | No | Varies | Country-level | Production |\n| World Bank | International | No | Annual | Country-level | Production |\n| SEC | Financial | No | Real-time | Public filings | Production |\n| Treasury | Financial | No | Daily | Federal finances | Production |\n| FDIC | Banking | No | Quarterly | Bank data | Production |\n\n#### Demographic & Labor Data (3 connectors)\nInstitutional connectors for core demographic and labor datasets.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| Census ACS | Demographics | Optional | Annual | All US geographies | Production |\n| Census CBP | Business | Optional | Annual | County-level | Production |\n| Census LEHD | Employment | No | Quarterly | County-level | Production |\n\n#### Health & Wellbeing Data (5 connectors)\nConnectors for health outcomes, provider, and wellbeing datasets.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| HRSA | Health | No | Annual | HPSA/MUA/P | Production |\n| CDC WONDER | Health | No | Varies | County-level | Production |\n| County Health Rankings| Health | No | Annual | County-level | Production |\n| FDA | Health | No | Real-time | Drugs/devices | Production |\n| NIH | Research | No | Daily | Grants/projects | Production |\n\n#### Environmental & Climate Data (5 connectors)\nProduction connectors for environmental, air, and climate data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| EPA EJScreen | Environment | No | Annual | Block group | Production |\n| EPA Air Quality | Environment | No | Hourly/Real-time | Station-level | Production |\n| EPA Superfund | Environment | No | Real-time | Site-level | Production |\n| EPA Water Quality | Environment | No | Real-time | Facility-level | Production |\n| NOAA Climate | Climate | No | Daily | Station-level | Production |\n\n#### Education Data (3 connectors)\nInstitutional connectors for education and postsecondary data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| NCES | Education | No | Annual | School-level | Production |\n| College Scorecard | Education | Yes | Annual | Institution | Production |\n| IPEDS | Education | No | Annual | Institution | Production |\n\n#### Housing & Urban Data (2 connectors)\nProduction-ready connectors for housing and urban indicators.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| HUD Fair Market Rent | Housing | Yes | Annual | Metro/County | Production |\n| Zillow Research | Housing | No | Monthly | Metro/ZIP | Production |\n\n#### Agricultural Data (2 connectors)\nConnectors for agricultural and food environment data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| USDA Food Atlas | Agricultural | Yes | Annual | County-level | Production |\n| USDA NASS | Agricultural | Yes | Varies | National/State | Production |\n\n#### Crime & Justice Data (3 connectors)\nProduction connectors for crime and justice data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| FBI UCR | Crime | Recommended | Annual | Agency-level | Production |\n| Bureau of Justice | Justice | No | Annual | National | Production |\n| Victims of Crime | Justice | No | Annual | State-level | Production |\n\n#### Energy Data (1 connector)\nInstitutional connector for energy datasets.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| EIA | Energy | Yes | Real-time | National/State | Production |\n\n#### Science & Research Data (2 connectors)\nConnectors for geoscience and research awards data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| USGS | Geoscience | No | Real-time | National | Production |\n| NSF | Research | No | Daily | Awards/grants | Production |\n\n#### Transportation Data (1 connector)\nProduction connector for aviation and transportation data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| FAA | Aviation | No | Real-time | Airport/flight | Production |\n\n#### Labor Safety Data (1 connector)\nConnector for labor safety and inspection data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| OSHA | Safety | No | Real-time | Inspections | Production |\n\n#### Social Services Data (2 connectors)\nConnectors for social services and benefits data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| Social Security Admin | Social | No | Annual | National | Production |\n| ACF | Social | No | Annual | State/County | Production |\n\n#### Veterans Services Data (1 connector)\nConnector for veterans services and benefits data.\n| Data Source | Domain | Auth Required | Update Frequency | Coverage | Status |\n|----------------------|---------------|---------------|---------------------|---------------------|--------------|\n| VA | Veterans | No | Real-time | Facilities/benefits | Production |\n\n**Total: 52 Production-Ready Connector Implementations** | All connectors maintained with continuous integration, automated testing, and security scanning.\n\n---\n\n## Security & IP Protection\n\nKRL Data Connectors are developed and maintained under a comprehensive, audit-grade security and intellectual property (IP) protection framework. The repository implements a 10-layer defense strategy with 6 automated CI/CD workflows, continuous security scanning, and zero-tolerance secret detection to ensure compliance, confidentiality, and operational integrity.\n\n### Compliance and Protection Layers\n\n| Layer | Control Area | Implementation | Coverage |\n|-------|----------------|----------------|----------|\n| 1 | Legal Protection (Copyright, Trademark, License) | Automated header validation | 100% of 136 total files |\n| 2 | Technical Protection (Secret Scanning) | Gitleaks, GitHub Advanced Security, Pre-commit | Repository-wide, all commits |\n| 3 | Build Verification (CI/CD Security) | 6 GitHub Actions workflows | All pull requests and pushes |\n| 4 | License Compliance | Automated dependency scanning | Apache 2.0, GPL/AGPL exclusion |\n| 5 | Continuous Monitoring & Response | Multi-scanner validation, SARIF reporting | Real-time detection and alerts |\n| 6 | Static Application Security Testing (SAST) | Bandit, Safety, CodeQL | All Python code |\n| 7 | Dependency Security | Trivy, pip-audit | All dependencies, daily scans |\n| 8 | Type & Contract Verification | mypy strict mode, pydantic validation | All public APIs |\n| 9 | Code Quality Gates | Black, isort, flake8, ruff | Pre-commit and CI enforcement |\n| 10 | Documentation & Auditability | Structured logging, SBOM generation | All operations logged |\n\n### Automated Security Controls\n\nAll code contributions and repository changes are subject to automated validation through 6 continuous integration workflows:\n- **Secret Detection**: Gitleaks, GitHub Advanced Security, pre-commit hooks with zero false-negative tolerance\n- **Vulnerability Assessment**: Trivy container scanning, CodeQL semantic analysis, SARIF reporting\n- **Dependency Security**: CVE scanning, pip-audit, safety checks with automatic PR creation for updates\n- **License Compliance**: GPL/AGPL blocklisting, Apache 2.0 compatibility verification, SBOM generation\n- **Static Analysis**: Bandit security linting, Safety vulnerability scanning, mypy strict type checking\n- **Code Quality**: Black formatting (100% compliance), isort import ordering, flake8 style enforcement\n- **Copyright Validation**: Automated header verification across all source files with trademark protection\n\n### Repository Validation Status\n\nContinuous validation across 2,098 automated tests confirms:\n- Zero secrets detected across 1,000+ commit history\n- 100% of 136 files with compliant copyright headers\n- Full adherence to Apache 2.0 licensing with no GPL/AGPL dependencies\n- 78%+ sustained test coverage across all connectors\n- All 6 CI/CD workflows passing with automated quality gates\n\n### Contributor Compliance\n\nAll contributions undergo rigorous automated validation:\n- Credentials and secrets automatically blocked at commit time via pre-commit hooks\n- All pull requests subject to 6 CI/CD workflows including security, testing, and compliance validation\n- Pre-commit hooks and automated tooling required for all contributors\n- Signed Contributor License Agreement (CLA) required for code contributions\n- Automated dependency updates via Dependabot with security patch prioritization\n\nFor detailed security practices, vulnerability reporting procedures, and compliance documentation, refer to [SECURITY.md](./SECURITY.md).\n\n---\n\n## Installation\n\nTo ensure seamless integration with institutional environments, KRL Data Connectors support multiple installation profiles tailored for production, development, and extended use cases.\n\n```bash\n# Basic installation\npip install krl-data-connectors\n\n# With all optional dependencies\npip install krl-data-connectors[all]\n\n# Development installation\npip install krl-data-connectors[dev]\n```\n\n---\n\n## Quick Start\n\nThe following examples demonstrate reproducible initialization and usage of KRL Data Connectors for primary data sources. All connectors are designed for direct integration into institutional analytics pipelines.\n\n### County Business Patterns (CBP)\n\n```python\nfrom krl_data_connectors import CountyBusinessPatternsConnector\n\n# Initialize connector using environment variable for API key\ncbp = CountyBusinessPatternsConnector()\n\n# Retrieve retail trade data for Rhode Island (FIPS 44, NAICS 44)\nretail_data = cbp.get_state_data(\n year=2021,\n state='44',\n naics='44'\n)\n\nprint(f\"Records: {len(retail_data)}\")\nprint(retail_data[['NAICS2017', 'ESTAB', 'EMP', 'PAYANN']].head())\n```\n\n### LEHD Origin-Destination\n\n```python\nfrom krl_data_connectors import LEHDConnector\n\n# Initialize connector\nlehd = LEHDConnector()\n\n# Retrieve origin-destination employment flows for RI, 2021\nod_data = lehd.get_od_data(\n state='ri',\n year=2021,\n job_type='JT00',\n segment='S000'\n)\n\nprint(f\"Origin-destination pairs: {len(od_data)}\")\nprint(od_data[['w_geocode', 'h_geocode', 'S000', 'SA01']].head())\n```\n\n### FRED\n\n```python\nfrom krl_data_connectors import FREDConnector\n\n# Initialize connector using FRED_API_KEY environment variable\nfred = FREDConnector()\n\n# Fetch unemployment rate time series\nunemployment = fred.get_series(\n series_id=\"UNRATE\",\n observation_start=\"2020-01-01\",\n observation_end=\"2023-12-31\"\n)\n\nprint(unemployment.head())\n```\n\n### BLS\n\n```python\nfrom krl_data_connectors import BLSConnector\n\n# Initialize connector using BLS_API_KEY environment variable\nbls = BLSConnector()\n\n# Retrieve unemployment rate for selected states\nunemployment = bls.get_series(\n series_ids=['LASST060000000000003', 'LASST440000000000003'],\n start_year=2020,\n end_year=2023\n)\n\nprint(unemployment.head())\n```\n\n### BEA\n\n```python\nfrom krl_data_connectors import BEAConnector\n\n# Initialize connector using BEA_API_KEY environment variable\nbea = BEAConnector()\n\n# Retrieve GDP by state for 2021\ngdp_data = bea.get_data(\n dataset='Regional',\n method='GetData',\n TableName='SAGDP2N',\n LineCode=1,\n Year='2021',\n GeoFips='STATE'\n)\n\nprint(gdp_data.head())\n```\n\n### Caching and Base Connector\n\nAll connectors inherit from `BaseConnector`, which standardizes caching, configuration, and logging for operational consistency.\n\n```python\nfrom krl_data_connectors import FREDConnector\n\n# Enable file-based caching with 1-hour TTL\nfred = FREDConnector(\n api_key=\"your_api_key\",\n cache_dir=\"/tmp/fred_cache\",\n cache_ttl=3600\n)\n\n# Cached responses are handled automatically\ndata1 = fred.get_series(\"UNRATE\") # API call\ndata2 = fred.get_series(\"UNRATE\") # Served from cache\n\n# Retrieve cache statistics\nstats = fred.cache.get_stats()\nprint(f\"Cache hit rate: {stats['hit_rate']:.1f}%\")\n```\n\n---\n\n## Architecture\n\nKRL Data Connectors are built on a modular, auditable architecture supporting institutional-scale deployment. The codebase spans 73 source modules organized across 14 domain-specific directories, with all 52 connector implementations deriving from a unified `BaseConnector` that enforces interface consistency, operational controls, and standardized behavior.\n\n### BaseConnector Foundation\n\nThe `BaseConnector` class provides institutional-grade infrastructure for all 52 connector implementations:\n- **Structured Logging**: JSON-formatted logs with complete request/response metadata and audit trails\n- **Consistent Configuration**: Environment variable and YAML-based configuration for credentials, caching, and logging\n- **Standardized Caching**: File-based and memory caching with configurable TTL across all connectors\n- **Error Handling**: Uniform exception handling, retry logic, and rate limit management\n- **Type Safety**: Full mypy strict mode compliance with pydantic validation across all public APIs\n- **Request Management**: HTTP session pooling, connection reuse, and automated timeout handling\n\n```python\nfrom abc import ABC, abstractmethod\nfrom krl_core import get_logger, ConfigManager, FileCache\n\nclass BaseConnector(ABC):\n \"\"\"Abstract base class for all data connectors. Enforces interface and operational consistency.\"\"\"\n def __init__(self, api_key=None, cache_dir=None, cache_ttl=3600):\n self.logger = get_logger(self.__class__.__name__)\n self.config = ConfigManager()\n self.cache = FileCache(\n cache_dir=cache_dir,\n default_ttl=cache_ttl,\n namespace=self.__class__.__name__.lower()\n )\n # ... initialization\n```\n\n---\n\n## API Key Management\n\nKRL Data Connectors implement secure, automated credential resolution for institutional and development use. The credential management system ensures operational consistency and auditability. See [API_KEY_SETUP.md](./API_KEY_SETUP.md) for procedural details.\n\n### Credential Resolution Order\n\nAPI credentials are resolved in the following order:\n1. **Environment Variables** (preferred for production)\n2. **Configuration file** at `~/.krl/apikeys` (recommended for development/testing)\n3. **Direct assignment in code** (permitted for testing only)\n\n#### Example: Environment Variables\n\n```bash\nexport BEA_API_KEY=\"your_bea_key\"\nexport FRED_API_KEY=\"your_fred_key\"\nexport BLS_API_KEY=\"your_bls_key\"\nexport CENSUS_API_KEY=\"your_census_key\"\n```\n\n#### Example: Configuration File\n\n```bash\nmkdir -p ~/.krl\ncat > ~/.krl/apikeys << EOF\nBEA API KEY: your_bea_key\nFRED API KEY: your_fred_key\nBLS API KEY: your_bls_key\nCENSUS API: your_census_key\nEOF\nchmod 600 ~/.krl/apikeys\n```\n\n#### Obtaining API Keys\n\n| Service | Required? | Registration URL |\n|-------------------|--------------|-------------------------------------------------------|\n| CBP/Census | Optional | https://api.census.gov/data/key_signup.html |\n| FRED | Yes | https://fred.stlouisfed.org/docs/api/api_key.html |\n| BLS | Recommended* | https://www.bls.gov/developers/home.htm |\n| BEA | Yes | https://apps.bea.gov/api/signup/ |\n| LEHD | No | N/A |\n\n*BLS is accessible without a key but with reduced rate limits.\n\n#### Configuration Utilities\n\nAutomatic discovery and management of configuration files are supported:\n\n```python\nfrom krl_data_connectors import find_config_file, BEAConnector\n\nconfig_path = find_config_file('apikeys')\nprint(f\"Config found at: {config_path}\")\n\n# Connectors resolve credentials from configuration or environment\nbea = BEAConnector()\n```\n\n---\n\n## Configuration\n\nKRL Data Connectors support robust, institution-ready configuration via environment variables and YAML files. This enables precise management of credentials, caching, and logging for reproducible deployments.\n\n### Environment Variables\n\n```bash\n# API Keys\nexport CENSUS_API_KEY=\"your_census_key\"\nexport FRED_API_KEY=\"your_fred_key\"\nexport BLS_API_KEY=\"your_bls_key\"\nexport BEA_API_KEY=\"your_bea_key\"\n\n# Cache settings\nexport KRL_CACHE_DIR=\"~/.krl_cache\"\nexport KRL_CACHE_TTL=\"3600\"\n\n# Logging\nexport KRL_LOG_LEVEL=\"INFO\"\nexport KRL_LOG_FORMAT=\"json\"\n```\n\n### YAML Configuration File\n\n```yaml\nfred:\n api_key: \"your_fred_key\"\n base_url: \"https://api.stlouisfed.org/fred\"\n timeout: 30\n\ncensus:\n api_key: \"your_census_key\"\n base_url: \"https://api.census.gov/data\"\n\ncache:\n directory: \"~/.krl_cache\"\n ttl: 3600\n\nlogging:\n level: \"INFO\"\n format: \"json\"\n```\n\nApply configuration in code:\n\n```python\nfrom krl_core import ConfigManager\n\nconfig = ConfigManager(\"config.yaml\")\nfred = FREDConnector(api_key=config.get(\"fred.api_key\"))\n```\n\n---\n\n## Connector Catalog\n\nKRL Data Connectors enable reliable, scalable integration with the following data sources. All connectors are engineered to institutional standards for reliability and seamless analytics integration.\n\n### Production-Ready Connectors\n\n- **County Business Patterns (CBP):** Establishment and employment statistics by industry and geography. [examples/cbp_quickstart.ipynb](examples/)\n- **LEHD Origin-Destination:** Worker flows and employment demographics. [examples/lehd_quickstart.ipynb](examples/)\n- **FRED:** Economic time series and metadata.\n- **BLS:** Labor market and inflation statistics.\n- **BEA:** GDP, regional accounts, and personal income.\n- **EPA EJScreen:** Environmental justice indicators. [examples/ejscreen_quickstart.ipynb](examples/ejscreen_quickstart.ipynb)\n- **HRSA:** Health Professional Shortage Areas, MUA/P, FQHC. [examples/hrsa_quickstart.ipynb](examples/hrsa_quickstart.ipynb)\n- **County Health Rankings:** County-level health measures. [examples/chr_quickstart.ipynb](examples/chr_quickstart.ipynb)\n- **EPA Air Quality / AirNow:** Real-time and historical AQI. [examples/air_quality_quickstart.ipynb](examples/air_quality_quickstart.ipynb)\n- **Zillow Research:** Housing price and rent indices. [examples/zillow_quickstart.ipynb](examples/zillow_quickstart.ipynb)\n- **HUD Fair Market Rents:** Rental affordability and income limits. [examples/hud_fmr_quickstart.ipynb](examples/hud_fmr_quickstart.ipynb)\n- **FBI UCR:** Crime statistics and arrest data. [examples/fbi_ucr_quickstart.ipynb](examples/fbi_ucr_quickstart.ipynb)\n- **NCES:** School and district education statistics. [examples/nces_quickstart.ipynb](examples/nces_quickstart.ipynb)\n\n### In Development and Planned\n\n- **CDC WONDER:** Mortality and natality data (API non-functional; web interface recommended).\n- **USDA Food Environment Atlas:** Food access, insecurity, and local food systems.\n- **OECD, World Bank, College Scorecard, IPEDS, Superfund Sites, and additional sources:** Refer to [ROADMAP.md](ROADMAP.md) for the complete development roadmap.\n\n---\n\n## Roadmap and Quality Standards\n\nKRL Data Connectors are developed in accordance with a structured roadmap, with 52 connector implementations currently in production across major institutional domains. Prioritization is driven by institutional requirements, API stability, and comprehensive domain coverage.\n\n**Quality Standards (Applied to All 52 Connectors):**\n- \u226578% test coverage sustained across 2,098 automated tests\n- Full type hints with mypy strict mode compliance on all public methods\n- Pydantic validation for all API contracts and data models\n- Robust error handling with structured diagnostics and retry logic\n- Configurable caching (file-based and memory) with intelligent TTL management\n- Structured JSON logging for operational transparency and audit trails\n- Comprehensive documentation with 16 quickstart Jupyter notebooks\n- Secure API key management with automated validation and credential resolution\n\nFor implementation milestones and API specifications, consult [ROADMAP.md](ROADMAP.md).\n\n---\n\n## Testing\n\nKRL Data Connectors utilize a 10-layer, open-source testing architecture aligned with institutional best practices. All tests and validation tools are fully auditable and reproducible.\n\n### Testing Model\n\n| Layer | Purpose | Tools | Status |\n|-------|--------|-------|--------|\n| 1. Unit Tests | Function correctness | pytest, hypothesis | Implemented (2,098 tests, 78%+ coverage) |\n| 2. Integration | Component interaction | pytest, requests-mock | Implemented |\n| 3. E2E Tests | Workflow validation | playwright | Planned |\n| 4. Performance | Load/stress testing | locust, pytest-benchmark | Planned |\n| 5. SAST | Static security analysis | bandit, safety, mypy | Active in CI |\n| 6. DAST | Runtime security testing | OWASP ZAP | Planned |\n| 7. Mutation | Test quality measurement | mutmut, hypothesis | Planned |\n| 8. Contract | Type/interface validation | pydantic, mypy strict | Active in CI |\n| 9. Penetration | Security assessment | metasploit, burp | Annual |\n| 10. Monitoring | Continuous validation | GitHub Actions (6 workflows), Snyk | Active |\n\n### Test Execution\n\n```bash\n# Run all tests\nmake test\n\n# Run with coverage\nmake coverage\n\n# Run security scans\nmake security\n\n# Run type checking\nmake type-check\n\n# Simulate full CI pipeline\nmake ci\n\n# List available commands\nmake help\n```\n\n### Coverage Objectives\n\n- Current: 78%+ overall test coverage sustained across 2,098 automated tests\n- Target: \u226590% line coverage, \u226585% branch coverage across all 52 connector implementations\n- Mutation: \u226590% kill rate for test quality validation\n\nFor detailed procedures, see [docs/TESTING_GUIDE.md](docs/TESTING_GUIDE.md).\n\n---\n\n## Development\n\nAll development must ensure reproducibility and full compliance with institutional contribution standards. Establish a local environment and follow the verified workflow:\n\n```bash\n# Clone the repository\ngit clone https://github.com/KR-Labs/krl-data-connectors.git\ncd krl-data-connectors\n\n# Create and activate a virtual environment\npython -m venv .venv\nsource .venv/bin/activate # or `.venv\\Scripts\\activate` on Windows\n\n# Install development and test dependencies\npip install -e \".[dev,test]\"\n\n# Run pre-commit hooks for code compliance\npre-commit install\npre-commit run --all-files\n\n# Execute tests\npytest\n\n# Build documentation\ncd docs && make html\n```\n\n---\n\n## Contributing\n\nAll contributions must maintain reproducibility, compliance, and operational reliability. Review [CONTRIBUTING.md](CONTRIBUTING.md) before submitting changes. A signed [Contributor License Agreement (CLA)](https://krlabs.dev/cla) is required for all code contributions.\n\n---\n\n## License\n\nKRL Data Connectors are distributed under the **Apache License 2.0**. For the full license text, refer to the [LICENSE](LICENSE) file.\n\n**Key License Terms:**\n- Permits commercial use, modification, and redistribution\n- Patent grant included\n- Compatible with proprietary software\n\n---\n\n## Support\n\nUse the following channels for verified support and communication:\n- **Documentation:** https://docs.krlabs.dev/data-connectors\n- **Issue Tracker:** https://github.com/KR-Labs/krl-data-connectors/issues\n- **Discussions:** https://github.com/KR-Labs/krl-data-connectors/discussions\n- **Email:** support@krlabs.dev\n\n---\n\n## Related Projects\n\nKRL Data Connectors are a core component of the KR-Labs analytics infrastructure ecosystem:\n- **[krl-open-core](https://github.com/KR-Labs/krl-open-core):** Logging, configuration, and caching utilities\n- **[krl-model-zoo](https://github.com/KR-Labs/krl-model-zoo):** Causal inference and forecasting models\n- **[krl-dashboard](https://github.com/KR-Labs/krl-dashboard):** Interactive analytics and visualization platform\n- **[krl-tutorials](https://github.com/KR-Labs/krl-tutorials):** Reproducible example workflows and onboarding materials\n\n---\n\n## Citation\n\nFor institutional or research citation of KRL Data Connectors, use the following BibTeX entry:\n\n```bibtex\n@software{krl_data_connectors,\n title = {KRL Data Connectors: Standardized Interfaces for Economic and Social Data},\n author = {KR-Labs},\n year = {2025},\n url = {https://github.com/KR-Labs/krl-data-connectors},\n license = {Apache-2.0}\n}\n```\n\n---\n\n**Engineered for reproducibility, scalability, and institutional trust by [KR-Labs](https://krlabs.dev)**\n\n\u00a9 2025 KR-Labs. All rights reserved. \nKR-Labs is a trademark of Quipu Research Labs, LLC, a subsidiary of Sudiata Giddasira, Inc.\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Production-ready data connectors for 52 government and research data sources with 2,098 automated tests",
"version": "0.3.3",
"project_urls": {
"Bug Tracker": "https://github.com/KR-Labs/krl-data-connectors/issues",
"Changelog": "https://github.com/KR-Labs/krl-data-connectors/blob/main/CHANGELOG.md",
"Documentation": "https://docs.krlabs.dev/data-connectors",
"Homepage": "https://krlabs.dev",
"Repository": "https://github.com/KR-Labs/krl-data-connectors"
},
"split_keywords": [
"economics",
" data",
" api",
" fred",
" census",
" bls",
" world-bank",
" oecd",
" connectors"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "2735672dde2afb9aabad8ecc209e2c90042e8e095d097b7ffb030e05d1ae36b6",
"md5": "88cba8387cd19725d3bd504b554ec00c",
"sha256": "e0c56cbceec059b650e13e950d7a5eee7b9bc13cfc26724e0c2574dbbd68db9d"
},
"downloads": -1,
"filename": "krl_data_connectors-0.3.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "88cba8387cd19725d3bd504b554ec00c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 262230,
"upload_time": "2025-10-23T04:14:03",
"upload_time_iso_8601": "2025-10-23T04:14:03.558113Z",
"url": "https://files.pythonhosted.org/packages/27/35/672dde2afb9aabad8ecc209e2c90042e8e095d097b7ffb030e05d1ae36b6/krl_data_connectors-0.3.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "6aba703a940a4927502a63143d6bbba752f38ad92912d5ed759814351dde5afb",
"md5": "233a245741d566c0ece0281901b0f74b",
"sha256": "9b2f22a6a0d022dcf20fb5c0fb56bdb4ed35d0a900685e1a328f2d0e7f07d2bf"
},
"downloads": -1,
"filename": "krl_data_connectors-0.3.3.tar.gz",
"has_sig": false,
"md5_digest": "233a245741d566c0ece0281901b0f74b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 237722,
"upload_time": "2025-10-23T04:14:05",
"upload_time_iso_8601": "2025-10-23T04:14:05.530122Z",
"url": "https://files.pythonhosted.org/packages/6a/ba/703a940a4927502a63143d6bbba752f38ad92912d5ed759814351dde5afb/krl_data_connectors-0.3.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-10-23 04:14:05",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "KR-Labs",
"github_project": "krl-data-connectors",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "krl-data-connectors"
}
JSON
