# Bible Gateway Downloader - (ByGoD) True Async Edition
A comprehensive, truly asynchronous tool for downloading Bible translations from BibleGateway.com in multiple formats (JSON, CSV, YAML, XML) with genuine parallel downloads, retry mechanisms, and flexible output options.
## 🚀 Features
- **True Async HTTP Requests**: Uses `aiohttp` for genuine parallelism, not just threading
- **Direct HTML Parsing**: Bypasses synchronous libraries to directly parse BibleGateway HTML
- **Multiple Translations**: Support for 30+ Bible translations (NIV, KJV, ESV, etc.)
- **Multiple Formats**: Output in JSON, CSV, YAML, and XML formats
- **Intelligent Rate Limiting**: Configurable concurrency with automatic rate limiting
- **Retry Mechanisms**: Exponential backoff with configurable retry attempts
- **Organized Output**: Structured directory organization by translation and format
- **Comprehensive Logging**: Colored, detailed progress tracking
- **Flexible Output Modes**: Download individual books, full Bibles, or both
## 📦 Installation
### Option 1: Install from PyPI (Recommended)
```bash
pip install bygod
```
### Option 2: Install from Source (Using Pipenv)
1. **Clone the repository**:
```bash
git clone git@github.com:Christ-Is-The-King/bible-gateway-downloader.git
cd bible-gateway-downloader
```
2. **Install pipenv** (if not already installed):
```bash
pip install pipenv
```
3. **Install dependencies and activate virtual environment**:
```bash
pipenv install
pipenv shell
```
4. **Install in development mode**:
```bash
pip install -e .
```
### Option 3: Install from Source (Using pip)
1. **Clone the repository**:
```bash
git clone git@github.com:Christ-Is-The-King/bible-gateway-downloader.git
cd bible-gateway-downloader
```
2. **Create a virtual environment**:
```bash
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
```
3. **Install dependencies**:
```bash
pip install -r requirements.txt
```
4. **Install in development mode**:
```bash
pip install -e .
```
### Option 4: Build and Install Package
1. **Build the package**:
```bash
python build_package.py
```
2. **Install the built package**:
```bash
pip install dist/bygod-*.whl
```
## 🎯 Quick Start
### Basic Usage
Download a single translation in JSON format:
```bash
bygod --translations NIV --formats json
```
Download multiple translations in multiple formats:
```bash
bygod --translations NIV,KJV,ESV --formats json,csv,xml
```
Download specific books only:
```bash
bygod --translations NIV --books Genesis,Exodus,Psalms
```
### Advanced Usage
Download with custom rate limiting and retry settings:
```bash
bygod \
--translations NIV,KJV \
--formats json,csv \
--rate-limit 10 \
--retries 5 \
--retry-delay 3 \
--timeout 600
```
Download only individual books (no full Bible):
```bash
bygod --translations NIV --output-mode books
```
Download only full Bible (no individual books):
```bash
bygod --translations NIV --output-mode book
```
## 📋 Command Line Options
| Option | Description | Default |
|--------|-------------|---------|
| `--translations` | Comma-separated list of Bible translations | `NIV` |
| `--formats` | Output formats: json, csv, xml, yaml | `json` |
| `--output-mode` | Output mode: book, books, all | `all` |
| `--output-dir` | Directory to save downloaded Bibles | `./bibles` |
| `--rate-limit` | Maximum concurrent requests | `5` |
| `--retries` | Maximum retry attempts | `3` |
| `--retry-delay` | Delay between retries (seconds) | `2` |
| `--timeout` | Request timeout (seconds) | `300` |
| `--books` | Comma-separated list of specific books | All books |
## 📚 Supported Translations
The downloader supports 30+ Bible translations including:
- **NIV** - New International Version
- **KJV** - King James Version
- **ESV** - English Standard Version
- **NKJV** - New King James Version
- **NLT** - New Living Translation
- **CSB** - Christian Standard Bible
- **NASB** - New American Standard Bible
- **RSV** - Revised Standard Version
- **ASV** - American Standard Version
- **WEB** - World English Bible
- **YLT** - Young's Literal Translation
- **AMP** - Amplified Bible
- **MSG** - The Message
- **CEV** - Contemporary English Version
- **ERV** - Easy-to-Read Version
- **GW** - God's Word Translation
- **HCSB** - Holman Christian Standard Bible
- **ICB** - International Children's Bible
- **ISV** - International Standard Version
- **LEB** - Lexham English Bible
- **NCV** - New Century Version
- **NET** - New English Translation
- **NIRV** - New International Reader's Version
- **NRSV** - New Revised Standard Version
- **TLB** - The Living Bible
- **TLV** - Tree of Life Version
- **VOICE** - The Voice
- **WYC** - Wycliffe Bible
## 📁 Output Structure
The downloader creates a well-organized directory structure:
```
bibles/
├── NIV/
│ ├── NIV.json # Full Bible in JSON
│ ├── NIV.csv # Full Bible in CSV
│ ├── NIV.xml # Full Bible in XML
│ ├── NIV.yml # Full Bible in YAML
│ └── books/
│ ├── Genesis.json # Individual book in JSON
│ ├── Genesis.csv # Individual book in CSV
│ ├── Genesis.xml # Individual book in XML
│ ├── Genesis.yml # Individual book in YAML
│ └── ...
├── KJV/
│ ├── KJV.json
│ ├── KJV.csv
│ └── books/
│ └── ...
└── ...
```
## 🔧 Technical Details
### True Async Architecture
Unlike traditional threading approaches, this downloader uses:
- **`asyncio`**: Python's native async/await framework
- **`aiohttp`**: True async HTTP client for concurrent requests
- **Semaphores**: Rate limiting with configurable concurrency
- **`asyncio.gather()`**: Parallel execution of multiple downloads
### HTML Parsing
The downloader directly parses BibleGateway HTML using:
- **BeautifulSoup**: HTML parsing and navigation
- **CSS Selectors**: Multiple fallback selectors for verse extraction
- **Regex Patterns**: Chapter discovery and verse number detection
### Error Handling
- **Exponential Backoff**: Intelligent retry with increasing delays
- **Rate Limit Detection**: Automatic handling of 429 responses
- **Graceful Degradation**: Continues processing even if some downloads fail
- **Detailed Logging**: Comprehensive error reporting and progress tracking
## 🧪 Testing
Run the test suite to verify functionality:
```bash
# Using pipenv
pipenv run python tests.py
# Or using pip
python tests.py
```
The test suite includes:
- Single chapter downloads
- Single book downloads
- Multiple book parallel downloads
- True concurrency testing with multiple translations
## 📊 Performance
The true async architecture provides significant performance improvements:
- **Genuine Parallelism**: Multiple HTTP requests execute simultaneously
- **Efficient Resource Usage**: No thread overhead, uses event loop
- **Scalable Concurrency**: Configurable rate limits prevent server overload
- **Memory Efficient**: Streams responses without loading entire files into memory
## 🤝 Contributing
1. Fork the repository
2. Create a feature branch
3. Install dependencies using pipenv:
```bash
pipenv install
pipenv install --dev
```
4. Make your changes
5. Add tests for new functionality
6. Ensure all tests pass:
```bash
pipenv run python tests.py
```
7. Submit a pull request
## 📄 License
This project is licensed under the MIT License - see the LICENSE file for details.
## 🙏 Acknowledgments
- BibleGateway.com for providing Bible content
- The Python async community for excellent tools and documentation
- Contributors and users who provide feedback and improvements
## 🆘 Troubleshooting
### Common Issues
**Rate Limiting**: If you encounter 429 errors, reduce the `--rate-limit` value.
**Timeout Errors**: Increase the `--timeout` value for slower connections.
**Missing Verses**: Some translations may have different HTML structures. The parser includes multiple fallback methods.
**Memory Usage**: For large downloads, consider downloading fewer books at once or using a lower rate limit.
### Getting Help
- Check the logs for detailed error messages
- Try with a single translation and book first
- Ensure your internet connection is stable
- Verify that BibleGateway.com is accessible from your location
Raw data
{
"_id": null,
"home_page": null,
"name": "bygod",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": "Bible Gateway Downloader Team <ByGoD@rapdirabbit.software>",
"keywords": "bible, download, biblegateway, async, scripture, religion, json, csv, xml, yaml",
"author": null,
"author_email": "Bible Gateway Downloader Team <ByGoD@rapdirabbit.software>",
"download_url": "https://files.pythonhosted.org/packages/79/8f/35aed09a5caa1bd2e1f936e9dc10207d8290d0f525ad128b45b2f710f786/bygod-2.1.0.tar.gz",
"platform": null,
"description": "# Bible Gateway Downloader - (ByGoD) True Async Edition\n\nA comprehensive, truly asynchronous tool for downloading Bible translations from BibleGateway.com in multiple formats (JSON, CSV, YAML, XML) with genuine parallel downloads, retry mechanisms, and flexible output options.\n\n## \ud83d\ude80 Features\n\n- **True Async HTTP Requests**: Uses `aiohttp` for genuine parallelism, not just threading\n- **Direct HTML Parsing**: Bypasses synchronous libraries to directly parse BibleGateway HTML\n- **Multiple Translations**: Support for 30+ Bible translations (NIV, KJV, ESV, etc.)\n- **Multiple Formats**: Output in JSON, CSV, YAML, and XML formats\n- **Intelligent Rate Limiting**: Configurable concurrency with automatic rate limiting\n- **Retry Mechanisms**: Exponential backoff with configurable retry attempts\n- **Organized Output**: Structured directory organization by translation and format\n- **Comprehensive Logging**: Colored, detailed progress tracking\n- **Flexible Output Modes**: Download individual books, full Bibles, or both\n\n## \ud83d\udce6 Installation\n\n### Option 1: Install from PyPI (Recommended)\n\n```bash\npip install bygod\n```\n\n### Option 2: Install from Source (Using Pipenv)\n\n1. **Clone the repository**:\n ```bash\n git clone git@github.com:Christ-Is-The-King/bible-gateway-downloader.git\n cd bible-gateway-downloader\n ```\n\n2. **Install pipenv** (if not already installed):\n ```bash\n pip install pipenv\n ```\n\n3. **Install dependencies and activate virtual environment**:\n ```bash\n pipenv install\n pipenv shell\n ```\n\n4. **Install in development mode**:\n ```bash\n pip install -e .\n ```\n\n### Option 3: Install from Source (Using pip)\n\n1. **Clone the repository**:\n ```bash\n git clone git@github.com:Christ-Is-The-King/bible-gateway-downloader.git\n cd bible-gateway-downloader\n ```\n\n2. **Create a virtual environment**:\n ```bash\n python -m venv venv\n source venv/bin/activate # On Windows: venv\\Scripts\\activate\n ```\n\n3. **Install dependencies**:\n ```bash\n pip install -r requirements.txt\n ```\n\n4. **Install in development mode**:\n ```bash\n pip install -e .\n ```\n\n### Option 4: Build and Install Package\n\n1. **Build the package**:\n ```bash\n python build_package.py\n ```\n\n2. **Install the built package**:\n ```bash\n pip install dist/bygod-*.whl\n ```\n\n## \ud83c\udfaf Quick Start\n\n### Basic Usage\n\nDownload a single translation in JSON format:\n```bash\nbygod --translations NIV --formats json\n```\n\nDownload multiple translations in multiple formats:\n```bash\nbygod --translations NIV,KJV,ESV --formats json,csv,xml\n```\n\nDownload specific books only:\n```bash\nbygod --translations NIV --books Genesis,Exodus,Psalms\n```\n\n### Advanced Usage\n\nDownload with custom rate limiting and retry settings:\n```bash\nbygod \\\n --translations NIV,KJV \\\n --formats json,csv \\\n --rate-limit 10 \\\n --retries 5 \\\n --retry-delay 3 \\\n --timeout 600\n```\n\nDownload only individual books (no full Bible):\n```bash\nbygod --translations NIV --output-mode books\n```\n\nDownload only full Bible (no individual books):\n```bash\nbygod --translations NIV --output-mode book\n```\n\n## \ud83d\udccb Command Line Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `--translations` | Comma-separated list of Bible translations | `NIV` |\n| `--formats` | Output formats: json, csv, xml, yaml | `json` |\n| `--output-mode` | Output mode: book, books, all | `all` |\n| `--output-dir` | Directory to save downloaded Bibles | `./bibles` |\n| `--rate-limit` | Maximum concurrent requests | `5` |\n| `--retries` | Maximum retry attempts | `3` |\n| `--retry-delay` | Delay between retries (seconds) | `2` |\n| `--timeout` | Request timeout (seconds) | `300` |\n| `--books` | Comma-separated list of specific books | All books |\n\n## \ud83d\udcda Supported Translations\n\nThe downloader supports 30+ Bible translations including:\n\n- **NIV** - New International Version\n- **KJV** - King James Version\n- **ESV** - English Standard Version\n- **NKJV** - New King James Version\n- **NLT** - New Living Translation\n- **CSB** - Christian Standard Bible\n- **NASB** - New American Standard Bible\n- **RSV** - Revised Standard Version\n- **ASV** - American Standard Version\n- **WEB** - World English Bible\n- **YLT** - Young's Literal Translation\n- **AMP** - Amplified Bible\n- **MSG** - The Message\n- **CEV** - Contemporary English Version\n- **ERV** - Easy-to-Read Version\n- **GW** - God's Word Translation\n- **HCSB** - Holman Christian Standard Bible\n- **ICB** - International Children's Bible\n- **ISV** - International Standard Version\n- **LEB** - Lexham English Bible\n- **NCV** - New Century Version\n- **NET** - New English Translation\n- **NIRV** - New International Reader's Version\n- **NRSV** - New Revised Standard Version\n- **TLB** - The Living Bible\n- **TLV** - Tree of Life Version\n- **VOICE** - The Voice\n- **WYC** - Wycliffe Bible\n\n## \ud83d\udcc1 Output Structure\n\nThe downloader creates a well-organized directory structure:\n\n```\nbibles/\n\u251c\u2500\u2500 NIV/\n\u2502 \u251c\u2500\u2500 NIV.json # Full Bible in JSON\n\u2502 \u251c\u2500\u2500 NIV.csv # Full Bible in CSV\n\u2502 \u251c\u2500\u2500 NIV.xml # Full Bible in XML\n\u2502 \u251c\u2500\u2500 NIV.yml # Full Bible in YAML\n\u2502 \u2514\u2500\u2500 books/\n\u2502 \u251c\u2500\u2500 Genesis.json # Individual book in JSON\n\u2502 \u251c\u2500\u2500 Genesis.csv # Individual book in CSV\n\u2502 \u251c\u2500\u2500 Genesis.xml # Individual book in XML\n\u2502 \u251c\u2500\u2500 Genesis.yml # Individual book in YAML\n\u2502 \u2514\u2500\u2500 ...\n\u251c\u2500\u2500 KJV/\n\u2502 \u251c\u2500\u2500 KJV.json\n\u2502 \u251c\u2500\u2500 KJV.csv\n\u2502 \u2514\u2500\u2500 books/\n\u2502 \u2514\u2500\u2500 ...\n\u2514\u2500\u2500 ...\n```\n\n## \ud83d\udd27 Technical Details\n\n### True Async Architecture\n\nUnlike traditional threading approaches, this downloader uses:\n\n- **`asyncio`**: Python's native async/await framework\n- **`aiohttp`**: True async HTTP client for concurrent requests\n- **Semaphores**: Rate limiting with configurable concurrency\n- **`asyncio.gather()`**: Parallel execution of multiple downloads\n\n### HTML Parsing\n\nThe downloader directly parses BibleGateway HTML using:\n\n- **BeautifulSoup**: HTML parsing and navigation\n- **CSS Selectors**: Multiple fallback selectors for verse extraction\n- **Regex Patterns**: Chapter discovery and verse number detection\n\n### Error Handling\n\n- **Exponential Backoff**: Intelligent retry with increasing delays\n- **Rate Limit Detection**: Automatic handling of 429 responses\n- **Graceful Degradation**: Continues processing even if some downloads fail\n- **Detailed Logging**: Comprehensive error reporting and progress tracking\n\n## \ud83e\uddea Testing\n\nRun the test suite to verify functionality:\n\n```bash\n# Using pipenv\npipenv run python tests.py\n\n# Or using pip\npython tests.py\n```\n\nThe test suite includes:\n- Single chapter downloads\n- Single book downloads\n- Multiple book parallel downloads\n- True concurrency testing with multiple translations\n\n## \ud83d\udcca Performance\n\nThe true async architecture provides significant performance improvements:\n\n- **Genuine Parallelism**: Multiple HTTP requests execute simultaneously\n- **Efficient Resource Usage**: No thread overhead, uses event loop\n- **Scalable Concurrency**: Configurable rate limits prevent server overload\n- **Memory Efficient**: Streams responses without loading entire files into memory\n\n## \ud83e\udd1d Contributing\n\n1. Fork the repository\n2. Create a feature branch\n3. Install dependencies using pipenv:\n ```bash\n pipenv install\n pipenv install --dev\n ```\n4. Make your changes\n5. Add tests for new functionality\n6. Ensure all tests pass:\n ```bash\n pipenv run python tests.py\n ```\n7. Submit a pull request\n\n## \ud83d\udcc4 License\n\nThis project is licensed under the MIT License - see the LICENSE file for details.\n\n## \ud83d\ude4f Acknowledgments\n\n- BibleGateway.com for providing Bible content\n- The Python async community for excellent tools and documentation\n- Contributors and users who provide feedback and improvements\n\n## \ud83c\udd98 Troubleshooting\n\n### Common Issues\n\n**Rate Limiting**: If you encounter 429 errors, reduce the `--rate-limit` value.\n\n**Timeout Errors**: Increase the `--timeout` value for slower connections.\n\n**Missing Verses**: Some translations may have different HTML structures. The parser includes multiple fallback methods.\n\n**Memory Usage**: For large downloads, consider downloading fewer books at once or using a lower rate limit.\n\n### Getting Help\n\n- Check the logs for detailed error messages\n- Try with a single translation and book first\n- Ensure your internet connection is stable\n- Verify that BibleGateway.com is accessible from your location \n",
"bugtrack_url": null,
"license": null,
"summary": "A comprehensive, truly asynchronous tool for downloading Bible translations from BibleGateway.com",
"version": "2.1.0",
"project_urls": {
"Bug Tracker": "https://github.com/Christ-Is-The-King/bible-gateway-downloader/issues",
"Documentation": "https://github.com/Christ-Is-The-King/bible-gateway-downloader#readme",
"Homepage": "https://github.com/Christ-Is-The-King/bible-gateway-downloader",
"Repository": "https://github.com/Christ-Is-The-King/bible-gateway-downloader"
},
"split_keywords": [
"bible",
" download",
" biblegateway",
" async",
" scripture",
" religion",
" json",
" csv",
" xml",
" yaml"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "cab710a2fa1f0e9a613423f28354ea4ecbb8c671552906d679465a6b7e833860",
"md5": "1339539a999504c2b3f7ef65c49daff7",
"sha256": "ecca9b970b400b25e98906147c15a9aeb58ee6e96a383e7f7e6f5da0d27f876a"
},
"downloads": -1,
"filename": "bygod-2.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "1339539a999504c2b3f7ef65c49daff7",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 16312,
"upload_time": "2025-07-23T23:22:39",
"upload_time_iso_8601": "2025-07-23T23:22:39.329666Z",
"url": "https://files.pythonhosted.org/packages/ca/b7/10a2fa1f0e9a613423f28354ea4ecbb8c671552906d679465a6b7e833860/bygod-2.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "798f35aed09a5caa1bd2e1f936e9dc10207d8290d0f525ad128b45b2f710f786",
"md5": "f2461116565c92cf6af35ae83d2bd963",
"sha256": "cc417ca36119f53631c1c5eedbbcc60014ff5e4125e29175b52115939c5b6fe4"
},
"downloads": -1,
"filename": "bygod-2.1.0.tar.gz",
"has_sig": false,
"md5_digest": "f2461116565c92cf6af35ae83d2bd963",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 29920,
"upload_time": "2025-07-23T23:22:40",
"upload_time_iso_8601": "2025-07-23T23:22:40.593348Z",
"url": "https://files.pythonhosted.org/packages/79/8f/35aed09a5caa1bd2e1f936e9dc10207d8290d0f525ad128b45b2f710f786/bygod-2.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-07-23 23:22:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "Christ-Is-The-King",
"github_project": "bible-gateway-downloader",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "bygod"
}
JSON
