| Name | datasure JSON |
| Version |
0.5.6
JSON |
| download |
| home_page | None |
| Summary | IPA Data Management System Dashboard |
| upload_time | 2025-09-10 17:30:57 |
| maintainer | None |
| docs_url | None |
| author | Innovations for Poverty Action |
| requires_python | >=3.11 |
| license | MIT License Copyright (c) 2024 Innovations for Poverty Action Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. |
| keywords |
data-quality
survey-data
streamlit
monitoring
hfc
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# DataSure
**DataSure** is IPA's Data Management System Dashboard - a comprehensive tool for survey data quality monitoring and high-frequency checks (HFCs) in research projects.
Built for data managers, survey coordinators, and research teams, DataSure provides real-time monitoring of survey data quality with interactive dashboards, automated checks, and flexible reporting capabilities.
## Key Features
- **📊 Data Quality Monitoring**: Real-time dashboards for comprehensive survey data analysis
- **🔍 Automated Checks**: 10 specialized quality check modules including duplicates, outliers, GPS validation, and missing data analysis
- **📈 Interactive Visualizations**: Charts and maps for data exploration and quality assessment
- **🔗 Multi-Source Integration**: Direct SurveyCTO API connection plus CSV/Excel file support
- **⚙️ Flexible Configuration**: Project-based settings with customizable check parameters
- **📋 Comprehensive Reporting**: Export capabilities for different audiences and formats
- **🎯 Enumerator Performance**: Monitor data collection team productivity and quality metrics
## Installation
### Step 1: Install uv from terminal
```bash
# WINDOWS
winget install astral-sh.uv
# MACOS/LINUX
brew install uv
```
### Step 2: Install datasure with uv
```bash
# install
uv tool install datasure
# ON WINDOWS: update windows path after installation
uv tool update-shell
```
### Step 3: verify installation
```bash
datasure --version
```
## Getting the latest release
```bash
# if datasure is already install, get latest version with
uv tool upgrade datasure
```
## Quick Start
1. **Launch the application**:
```bash
datasure
```
2. **Create your first project** and configure data quality checks
3. **Import survey data**:
- Connect directly to your SurveyCTO server
- Upload CSV or Excel files from local storage
4. **Monitor data quality** with interactive dashboards organized into specialized check modules
5. **Generate reports** and export results for your research team
## System Requirements
- **Python**: Version 3.11 or higher
- **Operating System**: Windows, macOS, or Linux
- **Memory**: Minimum 4GB RAM (8GB recommended for large datasets)
- **Storage**: 1GB free space for application and data cache
- **Internet**: Required for SurveyCTO integration and updates
## Data Quality Check Modules
DataSure includes 10 specialized modules for comprehensive survey data quality monitoring:
| Module | Purpose |
|--------|---------|
| **Summary** | Overall project progress and completion tracking |
| **Missing Data** | Identify patterns in incomplete responses |
| **Duplicates** | Find and manage duplicate survey entries |
| **GPS Validation** | Verify location data accuracy with interactive maps |
| **Outliers** | Identify unusual responses requiring review |
| **Enumerator Performance** | Monitor data collection team productivity |
| **Progress Tracking** | Real-time survey completion monitoring |
| **Descriptive Statistics** | Data distribution analysis and summaries |
| **Back-checks** | Verification workflow support |
| **Custom Checks** | Configure additional quality checks per project |
## Core Capabilities
### Data Import and Management
- **SurveyCTO Integration**: Direct API connection with form metadata and authentication
- **Local File Support**: CSV and Excel upload with automatic type detection
- **Multi-Project Organization**: Manage multiple surveys simultaneously
- **Data Preparation**: Cleaning and transformation workflows
### Interactive Dashboards
- **Real-time Monitoring**: Live updates as new data arrives
- **Customizable Views**: Configure dashboards per project requirements
- **Export Options**: Generate reports in PDF, Excel, and other formats
- **Automated Alerts**: Notifications for quality issues requiring attention
### Performance and Scalability
- **High-Performance Processing**: DuckDB backend for fast analytical queries
- **Large Dataset Support**: Optimized for datasets with hundreds of thousands of records
- **Intelligent Caching**: Reduces processing time and API calls
- **Cross-Platform Compatibility**: Works on Windows, macOS, and Linux
## Getting Started - Application Usage
### Using DataSure
Once DataSure is installed, you can begin monitoring your survey data quality:
#### 1. Launch the Application
```bash
datasure
```
The web interface will open in your default browser (typically at `http://localhost:8501`).
#### 2. Import Data
- **Import Data Page**: Start here to connect your data sources
- **SurveyCTO Integration**: Connect directly to your SurveyCTO server with authentication
- **Local Files**: Upload CSV or Excel files from your computer
- **Multiple Datasets**: Import and manage up to 10 datasets per project
#### 3. Prepare and Configure
- **Prepare Data Page**: Preview your imported datasets in separate tabs
- **Configure Checks Page**: Set up High-Frequency Checks (HFCs)
- Enter a page name for your quality monitoring dashboard
- Select the dataset to analyze
- Configure check parameters and thresholds
- Save settings to create your HFC page
#### 4. Monitor Data Quality
- **HFC Dashboard**: Access your configured quality check page
- **Interactive Tabs**: Each check type has its own tab (Summary, Missing Data, Duplicates, etc.)
- **Settings Expanders**: Configure specific parameters for each check
- **Real-time Updates**: Dashboard refreshes as new data becomes available
#### 5. Export and Share
- Generate reports for different audiences
- Export findings in various formats
- Monitor trends over time
### Command Line Options
```bash
# Show version information
datasure --version
# Launch with custom host/port
datasure --host 0.0.0.0 --port 8080
# View all available options
datasure --help
```
## Data Storage and Cache
DataSure automatically manages data storage and caching for optimal performance:
### Cache Directory Locations
- **Development Mode**: `./cache/` (in project root)
- **Production Mode**:
- **Windows**: `%APPDATA%/datasure/cache/`
- **Linux/macOS**: `~/.local/share/datasure/cache/`
### What's Stored
- **Project configurations**: HFC page settings and form configurations
- **Database files**: DuckDB databases for processed survey data
- **SurveyCTO cache**: Cached form metadata and server connections
- **User settings**: Check configurations and preferences
Cache directories are created automatically - no manual setup required.
## Support and Resources
### Getting Help
- **GitHub Issues**: [Report bugs and request features](https://github.com/PovertyAction/datasure/issues)
- **Email Support**: <researchsupport@poverty-action.org>
- **Documentation**: See [RELEASENOTES.md](RELEASENOTES.md) for latest updates
### Version Information
- **Current Version**: See [RELEASENOTES.md](RELEASENOTES.md) for the latest release information
- **Version History**: Track all changes and improvements
- **Upgrade Instructions**: Follow installation commands above to get the latest version
## Contributing
We welcome contributions from the research community! DataSure is developed by Innovations for Poverty Action (IPA) with input from data managers and survey coordinators worldwide.
### Ways to Contribute
- **Report Issues**: Found a bug or have a feature request? [Open an issue](https://github.com/PovertyAction/datasure/issues)
- **Suggest Features**: Share ideas for new data quality checks or workflow improvements
- **Share Use Cases**: Help us understand how DataSure fits into different research workflows
- **Code Contributions**: Developers can contribute code improvements and new features
### For Developers
If you're interested in contributing code or setting up a development environment, see our comprehensive [CONTRIBUTING.md](CONTRIBUTING.md) guide which includes:
- Development environment setup
- Code quality standards and testing requirements
- Package building and distribution workflows
- Release process and documentation guidelines
- Technical architecture and development patterns
### Community Standards
- Use clear, descriptive language when reporting issues
- Follow our code of conduct and treat all contributors with respect
- Help create a welcoming environment for researchers and developers from all backgrounds
## Authors and Acknowledgments
DataSure is developed and maintained by the [**Global Research & Data Science (GRDS)**](https://poverty-action.org/research-support) team at [**Innovations for Poverty Action (IPA)**](https://poverty-action.org/). Contact GRDS at <researchsupport@poverty-action.org>.
### Core Development Team
- [Ishmail Azindoo Baako](https://poverty-action.org/people/ishmail-azindoo-baako)
- [Wesley Kirui](https://poverty-action.org/people/wesley-kirui)
- [Niall Keleher](https://poverty-action.org/people/niall-keleher)
- [Dania Ochoa](https://poverty-action.org/people/dania-ochoa)
- [Laura Lahoz](https://poverty-action.org/people/laura-lahoz)
## License and Contact
- **License**: MIT License - see [LICENSE](LICENSE) file for details
- **Repository**: [https://github.com/PovertyAction/datasure](https://github.com/PovertyAction/datasure)
- **Organization**: Innovations for Poverty Action (IPA)
- **Contact**: <researchsupport@poverty-action.org>
---
**DataSure** - Ensuring data quality for better research outcomes.
Raw data
{
"_id": null,
"home_page": null,
"name": "datasure",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.11",
"maintainer_email": null,
"keywords": "data-quality, survey-data, streamlit, monitoring, hfc",
"author": "Innovations for Poverty Action",
"author_email": "Innovations for Poverty Action <researchsupport@poverty-action.org>",
"download_url": "https://files.pythonhosted.org/packages/b2/4e/b53f275fa374a59e6a183ecf5bf1d8fd9b51196f893ca202a85796941b7e/datasure-0.5.6.tar.gz",
"platform": null,
"description": "# DataSure\n\n**DataSure** is IPA's Data Management System Dashboard - a comprehensive tool for survey data quality monitoring and high-frequency checks (HFCs) in research projects.\n\nBuilt for data managers, survey coordinators, and research teams, DataSure provides real-time monitoring of survey data quality with interactive dashboards, automated checks, and flexible reporting capabilities.\n\n## Key Features\n\n- **\ud83d\udcca Data Quality Monitoring**: Real-time dashboards for comprehensive survey data analysis\n- **\ud83d\udd0d Automated Checks**: 10 specialized quality check modules including duplicates, outliers, GPS validation, and missing data analysis\n- **\ud83d\udcc8 Interactive Visualizations**: Charts and maps for data exploration and quality assessment\n- **\ud83d\udd17 Multi-Source Integration**: Direct SurveyCTO API connection plus CSV/Excel file support\n- **\u2699\ufe0f Flexible Configuration**: Project-based settings with customizable check parameters\n- **\ud83d\udccb Comprehensive Reporting**: Export capabilities for different audiences and formats\n- **\ud83c\udfaf Enumerator Performance**: Monitor data collection team productivity and quality metrics\n\n## Installation\n\n### Step 1: Install uv from terminal\n\n```bash\n# WINDOWS\nwinget install astral-sh.uv\n\n# MACOS/LINUX\nbrew install uv\n```\n\n### Step 2: Install datasure with uv\n\n```bash\n# install\nuv tool install datasure\n\n# ON WINDOWS: update windows path after installation\nuv tool update-shell \n```\n\n### Step 3: verify installation\n\n```bash\ndatasure --version\n```\n\n## Getting the latest release\n\n```bash\n# if datasure is already install, get latest version with\nuv tool upgrade datasure\n```\n\n## Quick Start\n\n1. **Launch the application**:\n\n ```bash\n datasure\n ```\n\n2. **Create your first project** and configure data quality checks\n\n3. **Import survey data**:\n - Connect directly to your SurveyCTO server\n - Upload CSV or Excel files from local storage\n\n4. **Monitor data quality** with interactive dashboards organized into specialized check modules\n\n5. **Generate reports** and export results for your research team\n\n## System Requirements\n\n- **Python**: Version 3.11 or higher\n- **Operating System**: Windows, macOS, or Linux\n- **Memory**: Minimum 4GB RAM (8GB recommended for large datasets)\n- **Storage**: 1GB free space for application and data cache\n- **Internet**: Required for SurveyCTO integration and updates\n\n## Data Quality Check Modules\n\nDataSure includes 10 specialized modules for comprehensive survey data quality monitoring:\n\n| Module | Purpose |\n|--------|---------|\n| **Summary** | Overall project progress and completion tracking |\n| **Missing Data** | Identify patterns in incomplete responses |\n| **Duplicates** | Find and manage duplicate survey entries |\n| **GPS Validation** | Verify location data accuracy with interactive maps |\n| **Outliers** | Identify unusual responses requiring review |\n| **Enumerator Performance** | Monitor data collection team productivity |\n| **Progress Tracking** | Real-time survey completion monitoring |\n| **Descriptive Statistics** | Data distribution analysis and summaries |\n| **Back-checks** | Verification workflow support |\n| **Custom Checks** | Configure additional quality checks per project |\n\n## Core Capabilities\n\n### Data Import and Management\n\n- **SurveyCTO Integration**: Direct API connection with form metadata and authentication\n- **Local File Support**: CSV and Excel upload with automatic type detection \n- **Multi-Project Organization**: Manage multiple surveys simultaneously\n- **Data Preparation**: Cleaning and transformation workflows\n\n### Interactive Dashboards\n\n- **Real-time Monitoring**: Live updates as new data arrives\n- **Customizable Views**: Configure dashboards per project requirements\n- **Export Options**: Generate reports in PDF, Excel, and other formats\n- **Automated Alerts**: Notifications for quality issues requiring attention\n\n### Performance and Scalability\n\n- **High-Performance Processing**: DuckDB backend for fast analytical queries\n- **Large Dataset Support**: Optimized for datasets with hundreds of thousands of records\n- **Intelligent Caching**: Reduces processing time and API calls\n- **Cross-Platform Compatibility**: Works on Windows, macOS, and Linux\n\n## Getting Started - Application Usage\n\n### Using DataSure\n\nOnce DataSure is installed, you can begin monitoring your survey data quality:\n\n#### 1. Launch the Application\n\n```bash\ndatasure\n```\n\nThe web interface will open in your default browser (typically at `http://localhost:8501`).\n\n#### 2. Import Data\n\n- **Import Data Page**: Start here to connect your data sources\n- **SurveyCTO Integration**: Connect directly to your SurveyCTO server with authentication\n- **Local Files**: Upload CSV or Excel files from your computer\n- **Multiple Datasets**: Import and manage up to 10 datasets per project\n\n#### 3. Prepare and Configure\n\n- **Prepare Data Page**: Preview your imported datasets in separate tabs\n- **Configure Checks Page**: Set up High-Frequency Checks (HFCs)\n - Enter a page name for your quality monitoring dashboard\n - Select the dataset to analyze\n - Configure check parameters and thresholds\n - Save settings to create your HFC page\n\n#### 4. Monitor Data Quality\n\n- **HFC Dashboard**: Access your configured quality check page\n- **Interactive Tabs**: Each check type has its own tab (Summary, Missing Data, Duplicates, etc.)\n- **Settings Expanders**: Configure specific parameters for each check\n- **Real-time Updates**: Dashboard refreshes as new data becomes available\n\n#### 5. Export and Share\n\n- Generate reports for different audiences\n- Export findings in various formats\n- Monitor trends over time\n\n### Command Line Options\n\n```bash\n# Show version information\ndatasure --version\n\n# Launch with custom host/port \ndatasure --host 0.0.0.0 --port 8080\n\n# View all available options\ndatasure --help\n```\n\n## Data Storage and Cache\n\nDataSure automatically manages data storage and caching for optimal performance:\n\n### Cache Directory Locations\n\n- **Development Mode**: `./cache/` (in project root)\n- **Production Mode**:\n - **Windows**: `%APPDATA%/datasure/cache/`\n - **Linux/macOS**: `~/.local/share/datasure/cache/`\n\n### What's Stored\n\n- **Project configurations**: HFC page settings and form configurations\n- **Database files**: DuckDB databases for processed survey data\n- **SurveyCTO cache**: Cached form metadata and server connections\n- **User settings**: Check configurations and preferences\n\nCache directories are created automatically - no manual setup required.\n\n## Support and Resources\n\n### Getting Help\n\n- **GitHub Issues**: [Report bugs and request features](https://github.com/PovertyAction/datasure/issues)\n- **Email Support**: <researchsupport@poverty-action.org>\n- **Documentation**: See [RELEASENOTES.md](RELEASENOTES.md) for latest updates\n\n### Version Information\n\n- **Current Version**: See [RELEASENOTES.md](RELEASENOTES.md) for the latest release information\n- **Version History**: Track all changes and improvements\n- **Upgrade Instructions**: Follow installation commands above to get the latest version\n\n## Contributing\n\nWe welcome contributions from the research community! DataSure is developed by Innovations for Poverty Action (IPA) with input from data managers and survey coordinators worldwide.\n\n### Ways to Contribute\n\n- **Report Issues**: Found a bug or have a feature request? [Open an issue](https://github.com/PovertyAction/datasure/issues)\n- **Suggest Features**: Share ideas for new data quality checks or workflow improvements\n- **Share Use Cases**: Help us understand how DataSure fits into different research workflows\n- **Code Contributions**: Developers can contribute code improvements and new features\n\n### For Developers\n\nIf you're interested in contributing code or setting up a development environment, see our comprehensive [CONTRIBUTING.md](CONTRIBUTING.md) guide which includes:\n\n- Development environment setup\n- Code quality standards and testing requirements\n- Package building and distribution workflows \n- Release process and documentation guidelines\n- Technical architecture and development patterns\n\n### Community Standards\n\n- Use clear, descriptive language when reporting issues\n- Follow our code of conduct and treat all contributors with respect\n- Help create a welcoming environment for researchers and developers from all backgrounds\n\n## Authors and Acknowledgments\n\nDataSure is developed and maintained by the [**Global Research & Data Science (GRDS)**](https://poverty-action.org/research-support) team at [**Innovations for Poverty Action (IPA)**](https://poverty-action.org/). Contact GRDS at <researchsupport@poverty-action.org>.\n\n### Core Development Team\n\n- [Ishmail Azindoo Baako](https://poverty-action.org/people/ishmail-azindoo-baako)\n- [Wesley Kirui](https://poverty-action.org/people/wesley-kirui)\n- [Niall Keleher](https://poverty-action.org/people/niall-keleher)\n- [Dania Ochoa](https://poverty-action.org/people/dania-ochoa)\n- [Laura Lahoz](https://poverty-action.org/people/laura-lahoz)\n\n## License and Contact\n\n- **License**: MIT License - see [LICENSE](LICENSE) file for details\n- **Repository**: [https://github.com/PovertyAction/datasure](https://github.com/PovertyAction/datasure)\n- **Organization**: Innovations for Poverty Action (IPA)\n- **Contact**: <researchsupport@poverty-action.org>\n\n---\n\n**DataSure** - Ensuring data quality for better research outcomes.\n",
"bugtrack_url": null,
"license": "MIT License Copyright (c) 2024 Innovations for Poverty Action Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the \"Software\"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.",
"summary": "IPA Data Management System Dashboard",
"version": "0.5.6",
"project_urls": {
"Issues": "https://github.com/PovertyAction/datasure/issues",
"Source": "https://github.com/PovertyAction/datasure"
},
"split_keywords": [
"data-quality",
" survey-data",
" streamlit",
" monitoring",
" hfc"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "3c9e84844003083ba0264472fd4f8e75427fd825b89ddddb718da44ef1784abd",
"md5": "3bde355c516f9e296457dc79af4ffb2c",
"sha256": "923fc0cb7b982cd7c66aa8e057a3296d670767fa9d13bff3f146c06b82cdb159"
},
"downloads": -1,
"filename": "datasure-0.5.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "3bde355c516f9e296457dc79af4ffb2c",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.11",
"size": 1024695,
"upload_time": "2025-09-10T17:30:55",
"upload_time_iso_8601": "2025-09-10T17:30:55.910230Z",
"url": "https://files.pythonhosted.org/packages/3c/9e/84844003083ba0264472fd4f8e75427fd825b89ddddb718da44ef1784abd/datasure-0.5.6-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "b24eb53f275fa374a59e6a183ecf5bf1d8fd9b51196f893ca202a85796941b7e",
"md5": "4e14381b4fdfd9e6396780f3379b06a7",
"sha256": "c6a82b8fa05b835cfe298425fb57dcd485fb0d73a22b1173a01e456df297dce8"
},
"downloads": -1,
"filename": "datasure-0.5.6.tar.gz",
"has_sig": false,
"md5_digest": "4e14381b4fdfd9e6396780f3379b06a7",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.11",
"size": 1009522,
"upload_time": "2025-09-10T17:30:57",
"upload_time_iso_8601": "2025-09-10T17:30:57.428438Z",
"url": "https://files.pythonhosted.org/packages/b2/4e/b53f275fa374a59e6a183ecf5bf1d8fd9b51196f893ca202a85796941b7e/datasure-0.5.6.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-09-10 17:30:57",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "PovertyAction",
"github_project": "datasure",
"github_not_found": true,
"lcname": "datasure"
}
