# wickdata
High-performance Python library for fetching, storing, and streaming historical cryptocurrency market data.
## Features
- 📊 **Historical Data Fetching** - Fetch OHLCV candle data from 100+ cryptocurrency exchanges
- 💾 **Efficient Storage** - SQLite database with automatic deduplication and indexing
- 🚀 **Streaming Capabilities** - Memory-efficient streaming for large datasets
- 🔍 **Intelligent Gap Detection** - Automatically identify and fill missing data periods
- 📈 **Progress Tracking** - Real-time progress updates for long-running operations
- 🔄 **Retry Logic** - Automatic retries with exponential backoff for failed requests
- 🏗️ **Builder Patterns** - Intuitive APIs for constructing queries and requests
- ⚡ **Async/Await** - Full async support for high-performance operations
- 🔌 **Exchange Support** - Built on CCXT for compatibility with 100+ exchanges
- ✅ **Comprehensive Testing** - 90%+ test coverage ensuring reliability and stability
## Installation
```bash
pip install wickdata
```
Or install from source:
```bash
git clone https://github.com/h2337/wickdata.git
cd wickdata
pip install -e .
```
## Quick Start
```python
import asyncio
from wickdata import WickData, DataRequestBuilder, create_binance_config
async def main():
# Configure exchanges
exchange_configs = {
'binance': create_binance_config() # API keys optional for public data
}
# Initialize WickData
async with WickData(exchange_configs) as wickdata:
# Get data manager
data_manager = wickdata.get_data_manager()
# Build request for last 7 days of BTC/USDT hourly data
request = (DataRequestBuilder.create()
.with_exchange('binance')
.with_symbol('BTC/USDT')
.with_timeframe('1h')
.with_last_days(7)
.build())
# Fetch with progress tracking
def on_progress(info):
print(f"{info.stage}: {info.percentage:.1f}%")
stats = await data_manager.fetch_historical_data(request, on_progress)
print(f"Fetched {stats.total_candles} candles")
asyncio.run(main())
```
Check examples/ directory for more examples.
## Core Components
### WickData
Main entry point for the library. Manages database, exchanges, and provides access to data operations.
### DataManager
Handles fetching, storing, and querying historical data with intelligent gap detection.
### DataStreamer
Provides memory-efficient streaming of large datasets with various output options.
### Builder Patterns
- **DataRequestBuilder** - Build data fetch requests with convenient methods
- **CandleQueryBuilder** - Construct database queries with fluent interface
## Supported Timeframes
- 1m, 3m, 5m, 15m, 30m (minutes)
- 1h, 2h, 4h, 6h, 8h, 12h (hours)
- 1d, 3d (days)
- 1w (week)
- 1M (month)
## Configuration
### Exchange Configuration
```python
from wickdata import create_binance_config, create_coinbase_config
# Binance
binance_config = create_binance_config(
api_key='your-api-key', # Optional for public data
secret='your-secret', # Optional for public data
testnet=False
)
# Coinbase
coinbase_config = create_coinbase_config(
api_key='your-api-key',
secret='your-secret',
passphrase='your-passphrase',
sandbox=False
)
```
### Database Configuration
```python
from wickdata.models.config import DatabaseConfig, WickDataConfig
config = WickDataConfig(
exchanges={'binance': binance_config},
database=DatabaseConfig(
provider='sqlite',
url='sqlite:///my_data.db'
),
log_level='INFO'
)
```
## Examples
### Fetching Historical Data
```python
# Using convenience methods
request = (DataRequestBuilder.create()
.with_exchange('binance')
.with_symbol('ETH/USDT')
.with_timeframe('4h')
.with_last_weeks(2) # Last 2 weeks
.build())
# Or specific date range
request = (DataRequestBuilder.create()
.with_exchange('binance')
.with_symbol('BTC/USDT')
.with_timeframe('1d')
.with_date_range('2024-01-01', '2024-01-31')
.build())
```
### Querying Stored Data
```python
# Create query builder
query = CandleQueryBuilder(repository)
# Get recent data with pagination
candles = await (query
.exchange('binance')
.symbol('BTC/USDT')
.timeframe(Timeframe.ONE_HOUR)
.date_range(start_date, end_date)
.limit(100)
.offset(0)
.execute())
# Get statistics
stats = await query.stats()
```
### Streaming Data
```python
# Stream with async generator
async for batch in data_streamer.stream_candles(
exchange='binance',
symbol='ETH/USDT',
timeframe=Timeframe.FIVE_MINUTES,
start_time=start_timestamp,
end_time=end_timestamp,
options=StreamOptions(batch_size=1000, delay_ms=100)
):
process_batch(batch)
# Stream to callback
await data_streamer.stream_to_callback(
exchange='binance',
symbol='BTC/USDT',
timeframe=Timeframe.ONE_HOUR,
start_date=start_date,
end_date=end_date,
callback=process_candles,
options=StreamOptions(batch_size=500)
)
```
### Gap Detection and Analysis
```python
# Find missing data
gaps = await data_manager.find_missing_data(
exchange='binance',
symbol='BTC/USDT',
timeframe=Timeframe.ONE_HOUR,
start_date=start_date,
end_date=end_date
)
print(f"Found {len(gaps)} gaps")
for gap in gaps:
print(f" Gap: {gap.get_start_datetime()} to {gap.get_end_datetime()}")
print(f" Missing candles: {gap.candle_count}")
```
## Error Handling
WickData provides comprehensive error handling with specific exception types:
```python
from wickdata import (
WickDataError, # Base error class
ExchangeError, # Exchange-specific errors
ValidationError, # Input validation errors
RateLimitError, # Rate limiting errors
NetworkError, # Network connectivity issues
DatabaseError, # Database operation errors
ConfigurationError, # Configuration problems
DataGapError # Gap-related errors
)
try:
await data_manager.fetch_historical_data(request)
except RateLimitError as e:
print(f"Rate limited. Retry after {e.retry_after}s")
except ExchangeError as e:
print(f"Exchange error: {e.message}")
```
## Performance
- **Batch Processing**: Insert 10,000+ candles per second (SQLite)
- **Concurrent Fetching**: Multiple concurrent fetchers per exchange
- **Memory Efficient**: Streaming prevents memory overflow for large datasets
- **Smart Caching**: Automatic deduplication and gap detection
- **Connection Pooling**: Efficient database connection management
## Development
### Setup Development Environment
```bash
# Clone repository
git clone https://github.com/h2337/wickdata.git
cd wickdata
# Install in development mode with dev dependencies
pip install -e ".[dev]"
```
### Running Tests
```bash
# Run all tests
pytest
# Run with coverage
pytest --cov=wickdata
# Run specific test file
pytest tests/test_data_manager.py
```
### Code Quality
```bash
# Format code
black wickdata
# Lint code
ruff check wickdata
# Type checking
mypy wickdata
```
## Architecture
WickData follows a modular architecture with clear separation of concerns:
- **Core Layer**: Main WickData class, DataManager, DataStreamer
- **Database Layer**: Repository pattern with SQLite implementation
- **Exchange Layer**: CCXT integration with adapter pattern
- **Service Layer**: Gap analysis, retry logic, validation
- **Models**: Data models with validation
- **Builders**: Fluent interfaces for complex object construction
## Contributing
Contributions are welcome!
1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request
## License
This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
Raw data
{
"_id": null,
"home_page": null,
"name": "wickdata",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "cryptocurrency, trading, market-data, backtesting, ohlcv, ccxt",
"author": null,
"author_email": "h2337 <hikmat2337@gmail.com>",
"download_url": "https://files.pythonhosted.org/packages/f0/d2/393eb23648d960903577f3605b6023b8b7bb6e0bc62d2261cd08a65f738f/wickdata-0.1.0.tar.gz",
"platform": null,
"description": "# wickdata\n\nHigh-performance Python library for fetching, storing, and streaming historical cryptocurrency market data.\n\n## Features\n\n- \ud83d\udcca **Historical Data Fetching** - Fetch OHLCV candle data from 100+ cryptocurrency exchanges\n- \ud83d\udcbe **Efficient Storage** - SQLite database with automatic deduplication and indexing\n- \ud83d\ude80 **Streaming Capabilities** - Memory-efficient streaming for large datasets\n- \ud83d\udd0d **Intelligent Gap Detection** - Automatically identify and fill missing data periods\n- \ud83d\udcc8 **Progress Tracking** - Real-time progress updates for long-running operations\n- \ud83d\udd04 **Retry Logic** - Automatic retries with exponential backoff for failed requests\n- \ud83c\udfd7\ufe0f **Builder Patterns** - Intuitive APIs for constructing queries and requests\n- \u26a1 **Async/Await** - Full async support for high-performance operations\n- \ud83d\udd0c **Exchange Support** - Built on CCXT for compatibility with 100+ exchanges\n- \u2705 **Comprehensive Testing** - 90%+ test coverage ensuring reliability and stability\n\n## Installation\n\n```bash\npip install wickdata\n```\n\nOr install from source:\n\n```bash\ngit clone https://github.com/h2337/wickdata.git\ncd wickdata\npip install -e .\n```\n\n## Quick Start\n\n```python\nimport asyncio\nfrom wickdata import WickData, DataRequestBuilder, create_binance_config\n\nasync def main():\n # Configure exchanges\n exchange_configs = {\n 'binance': create_binance_config() # API keys optional for public data\n }\n \n # Initialize WickData\n async with WickData(exchange_configs) as wickdata:\n # Get data manager\n data_manager = wickdata.get_data_manager()\n \n # Build request for last 7 days of BTC/USDT hourly data\n request = (DataRequestBuilder.create()\n .with_exchange('binance')\n .with_symbol('BTC/USDT')\n .with_timeframe('1h')\n .with_last_days(7)\n .build())\n \n # Fetch with progress tracking\n def on_progress(info):\n print(f\"{info.stage}: {info.percentage:.1f}%\")\n \n stats = await data_manager.fetch_historical_data(request, on_progress)\n print(f\"Fetched {stats.total_candles} candles\")\n\nasyncio.run(main())\n```\n\nCheck examples/ directory for more examples.\n\n## Core Components\n\n### WickData\nMain entry point for the library. Manages database, exchanges, and provides access to data operations.\n\n### DataManager\nHandles fetching, storing, and querying historical data with intelligent gap detection.\n\n### DataStreamer\nProvides memory-efficient streaming of large datasets with various output options.\n\n### Builder Patterns\n- **DataRequestBuilder** - Build data fetch requests with convenient methods\n- **CandleQueryBuilder** - Construct database queries with fluent interface\n\n## Supported Timeframes\n\n- 1m, 3m, 5m, 15m, 30m (minutes)\n- 1h, 2h, 4h, 6h, 8h, 12h (hours)\n- 1d, 3d (days)\n- 1w (week)\n- 1M (month)\n\n## Configuration\n\n### Exchange Configuration\n\n```python\nfrom wickdata import create_binance_config, create_coinbase_config\n\n# Binance\nbinance_config = create_binance_config(\n api_key='your-api-key', # Optional for public data\n secret='your-secret', # Optional for public data\n testnet=False\n)\n\n# Coinbase\ncoinbase_config = create_coinbase_config(\n api_key='your-api-key',\n secret='your-secret',\n passphrase='your-passphrase',\n sandbox=False\n)\n```\n\n### Database Configuration\n\n```python\nfrom wickdata.models.config import DatabaseConfig, WickDataConfig\n\nconfig = WickDataConfig(\n exchanges={'binance': binance_config},\n database=DatabaseConfig(\n provider='sqlite',\n url='sqlite:///my_data.db'\n ),\n log_level='INFO'\n)\n```\n\n## Examples\n\n### Fetching Historical Data\n\n```python\n# Using convenience methods\nrequest = (DataRequestBuilder.create()\n .with_exchange('binance')\n .with_symbol('ETH/USDT')\n .with_timeframe('4h')\n .with_last_weeks(2) # Last 2 weeks\n .build())\n\n# Or specific date range\nrequest = (DataRequestBuilder.create()\n .with_exchange('binance')\n .with_symbol('BTC/USDT')\n .with_timeframe('1d')\n .with_date_range('2024-01-01', '2024-01-31')\n .build())\n```\n\n### Querying Stored Data\n\n```python\n# Create query builder\nquery = CandleQueryBuilder(repository)\n\n# Get recent data with pagination\ncandles = await (query\n .exchange('binance')\n .symbol('BTC/USDT')\n .timeframe(Timeframe.ONE_HOUR)\n .date_range(start_date, end_date)\n .limit(100)\n .offset(0)\n .execute())\n\n# Get statistics\nstats = await query.stats()\n```\n\n### Streaming Data\n\n```python\n# Stream with async generator\nasync for batch in data_streamer.stream_candles(\n exchange='binance',\n symbol='ETH/USDT',\n timeframe=Timeframe.FIVE_MINUTES,\n start_time=start_timestamp,\n end_time=end_timestamp,\n options=StreamOptions(batch_size=1000, delay_ms=100)\n):\n process_batch(batch)\n\n# Stream to callback\nawait data_streamer.stream_to_callback(\n exchange='binance',\n symbol='BTC/USDT',\n timeframe=Timeframe.ONE_HOUR,\n start_date=start_date,\n end_date=end_date,\n callback=process_candles,\n options=StreamOptions(batch_size=500)\n)\n```\n\n### Gap Detection and Analysis\n\n```python\n# Find missing data\ngaps = await data_manager.find_missing_data(\n exchange='binance',\n symbol='BTC/USDT',\n timeframe=Timeframe.ONE_HOUR,\n start_date=start_date,\n end_date=end_date\n)\n\nprint(f\"Found {len(gaps)} gaps\")\nfor gap in gaps:\n print(f\" Gap: {gap.get_start_datetime()} to {gap.get_end_datetime()}\")\n print(f\" Missing candles: {gap.candle_count}\")\n```\n\n## Error Handling\n\nWickData provides comprehensive error handling with specific exception types:\n\n```python\nfrom wickdata import (\n WickDataError, # Base error class\n ExchangeError, # Exchange-specific errors\n ValidationError, # Input validation errors\n RateLimitError, # Rate limiting errors\n NetworkError, # Network connectivity issues\n DatabaseError, # Database operation errors\n ConfigurationError, # Configuration problems\n DataGapError # Gap-related errors\n)\n\ntry:\n await data_manager.fetch_historical_data(request)\nexcept RateLimitError as e:\n print(f\"Rate limited. Retry after {e.retry_after}s\")\nexcept ExchangeError as e:\n print(f\"Exchange error: {e.message}\")\n```\n\n## Performance\n\n- **Batch Processing**: Insert 10,000+ candles per second (SQLite)\n- **Concurrent Fetching**: Multiple concurrent fetchers per exchange\n- **Memory Efficient**: Streaming prevents memory overflow for large datasets\n- **Smart Caching**: Automatic deduplication and gap detection\n- **Connection Pooling**: Efficient database connection management\n\n## Development\n\n### Setup Development Environment\n\n```bash\n# Clone repository\ngit clone https://github.com/h2337/wickdata.git\ncd wickdata\n\n# Install in development mode with dev dependencies\npip install -e \".[dev]\"\n```\n\n### Running Tests\n\n```bash\n# Run all tests\npytest\n\n# Run with coverage\npytest --cov=wickdata\n\n# Run specific test file\npytest tests/test_data_manager.py\n```\n\n### Code Quality\n\n```bash\n# Format code\nblack wickdata\n\n# Lint code\nruff check wickdata\n\n# Type checking\nmypy wickdata\n```\n\n## Architecture\n\nWickData follows a modular architecture with clear separation of concerns:\n\n- **Core Layer**: Main WickData class, DataManager, DataStreamer\n- **Database Layer**: Repository pattern with SQLite implementation\n- **Exchange Layer**: CCXT integration with adapter pattern\n- **Service Layer**: Gap analysis, retry logic, validation\n- **Models**: Data models with validation\n- **Builders**: Fluent interfaces for complex object construction\n\n## Contributing\n\nContributions are welcome!\n\n1. Fork the repository\n2. Create your feature branch (`git checkout -b feature/amazing-feature`)\n3. Commit your changes (`git commit -m 'Add amazing feature'`)\n4. Push to the branch (`git push origin feature/amazing-feature`)\n5. Open a Pull Request\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "High-performance library for fetching, storing, and streaming historical cryptocurrency market data",
"version": "0.1.0",
"project_urls": {
"Bug Tracker": "https://github.com/h2337/wickdata/issues",
"Homepage": "https://github.com/h2337/wickdata",
"Repository": "https://github.com/h2337/wickdata"
},
"split_keywords": [
"cryptocurrency",
" trading",
" market-data",
" backtesting",
" ohlcv",
" ccxt"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "e64eba20e9417a186f13898e5b6ff668b0c4c7a3eee21b66df0b3aee6166326a",
"md5": "329ad63342ae08036abc21fb220a3c29",
"sha256": "6d02e808faf4b0c6905ad3131f3e99eee36cfc5812f458e5e2abaf77342ff6b8"
},
"downloads": -1,
"filename": "wickdata-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "329ad63342ae08036abc21fb220a3c29",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 49162,
"upload_time": "2025-08-23T16:16:03",
"upload_time_iso_8601": "2025-08-23T16:16:03.225750Z",
"url": "https://files.pythonhosted.org/packages/e6/4e/ba20e9417a186f13898e5b6ff668b0c4c7a3eee21b66df0b3aee6166326a/wickdata-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "f0d2393eb23648d960903577f3605b6023b8b7bb6e0bc62d2261cd08a65f738f",
"md5": "698fb3c6904d7cb75eecd5f04f059f03",
"sha256": "bfc9f2e1270d5cf4364e15bd2f7329b8f10f50c347c03e33a8bc334e40dac5ac"
},
"downloads": -1,
"filename": "wickdata-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "698fb3c6904d7cb75eecd5f04f059f03",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 68397,
"upload_time": "2025-08-23T16:16:04",
"upload_time_iso_8601": "2025-08-23T16:16:04.668607Z",
"url": "https://files.pythonhosted.org/packages/f0/d2/393eb23648d960903577f3605b6023b8b7bb6e0bc62d2261cd08a65f738f/wickdata-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-23 16:16:04",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "h2337",
"github_project": "wickdata",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "wickdata"
}
JSON
