# IndoxGen: Enterprise-Grade Synthetic Data Generation Framework
[](https://github.com/osllmai/indoxGen/blob/main/LICENSE)
[](https://pypi.org/project/indoxGen/0.0.3/)
[](https://pypi.org/project/indoxGen/0.0.3/)
[](https://pepy.tech/project/indoxGen)
[](https://discord.com/invite/ossllmai)
[](https://github.com/osllmai/indoxGen)
<p align="center">
<a href="https://osllm.ai">Official Website</a> • <a href="https://docs.osllm.ai/index.html">Documentation</a> • <a href="https://discord.gg/qrCc56ZR">Discord</a>
</p>
<p align="center">
<b>NEW:</b> <a href="https://docs.google.com/forms/d/1CQXJvxLUqLBSXnjqQmRpOyZqD6nrKubLz2WTcIJ37fU/prefill">Subscribe to our mailing list</a> for updates and news!
</p>
## Overview
IndoxGen is a state-of-the-art, enterprise-ready framework designed for generating high-fidelity synthetic data. Leveraging advanced AI technologies, including Large Language Models (LLMs) and incorporating human feedback loops, IndoxGen offers unparalleled flexibility and precision in synthetic data creation across various domains and use cases.
## Key Features
- **Multiple Generation Pipelines**:
- `SyntheticDataGenerator`: Standard LLM-powered generation pipeline for structured data with embedded quality control mechanisms.
- `SyntheticDataGeneratorHF`: Advanced pipeline integrating human feedback to improve generation.
- `DataFromPrompt`: Dynamic data generation based on natural language prompts, useful for rapid prototyping.
- **Customization & Control**: Fine-grained control over data attributes, structure, and diversity. Customize every aspect of the synthetic data generation process.
- **Human-in-the-Loop**: Seamlessly integrates expert feedback for continuous improvement of generated data, offering the highest quality assurance.
- **AI-Driven Diversity**: Algorithms ensure representative and varied datasets, providing data diversity for robust modeling.
- **Flexible I/O**: Supports various data sources and export formats (Excel, CSV, etc.) for easy integration into existing workflows.
- **Advanced Learning Techniques**: Incorporation of few-shot learning for rapid adaptation to new domains with minimal examples.
- **Scalability**: Designed to handle both small-scale experiments and large-scale data generation tasks with multi-LLM support.
## Installation
```bash
pip install indoxgen
```
## Quick Start Guide
### Basic Usage: SyntheticDataGenerator
```python
from indoxGen.synthCore import SyntheticDataGenerator
from indoxGen.llms import OpenAi
columns = ["name", "age", "occupation"]
example_data = [
{"name": "Alice Johnson", "age": 35, "occupation": "Manager"},
{"name": "Bob Williams", "age": 42, "occupation": "Accountant"}
]
openai = OpenAi(api_key=OPENAI_API_KEY, model="gpt-4o-mini")
nemotron = OpenAi(api_key=NVIDIA_API_KEY, model="nvidia/nemotron-4-340b-instruct",
base_url="https://integrate.api.nvidia.com/v1")
generator = SyntheticDataGenerator(
generator_llm=nemotron,
judge_llm=openai,
columns=columns,
example_data=example_data,
user_instruction="Generate diverse, realistic data including name, age, and occupation. Ensure variability in demographics and professions.",
verbose=1
)
generated_data = generator.generate_data(num_samples=100)
```
### Advanced Usage: SyntheticDataGeneratorHF with Human Feedback
```python
from indoxGen.synthCore import SyntheticDataGeneratorHF
from indoxGen.llms import OpenAi
openai = OpenAi(api_key=OPENAI_API_KEY, model="gpt-4-0613")
nemotron = OpenAi(api_key=NVIDIA_API_KEY, model="nvidia/nemotron-4-340b-instruct",
base_url="https://integrate.api.nvidia.com/v1")
generator = SyntheticDataGeneratorHF(
generator_llm=nemotron,
judge_llm=openai,
columns=columns,
example_data=example_data,
user_instruction="Generate diverse, realistic professional profiles with name, age, and occupation.",
verbose=1,
diversity_threshold=0.4,
feedback_range=feedback_range
)
# Implement human feedback loop
generator.user_review_and_regenerate(
regenerate_rows=[0],
accepted_rows=[],
regeneration_feedback='Diversify names and occupations further',
min_score=0.7
)
```
### Prompt-Based Generation: DataFromPrompt
```python
from indoxGen.synthCore import DataFromPrompt, DataGenerationPrompt
from indoxGen.llms import OpenAi
nemotron = OpenAi(api_key=NVIDIA_API_KEY, model="nvidia/nemotron-4-340b-instruct",
base_url="https://integrate.api.nvidia.com/v1")
user_prompt = "Generate a comprehensive dataset with 3 columns and 3 rows about exoplanets."
instruction = DataGenerationPrompt.get_instruction(user_prompt)
data_generator = DataFromPrompt(
prompt_name="Exoplanet Dataset Generation",
args={
"llm": nemotron,
"n": 1,
"instruction": instruction,
},
outputs={"generations": "generate"},
)
generated_df = data_generator.run()
data_generator.save_to_excel("exoplanet_data.xlsx")
```
## Advanced Techniques
### Few-Shot Learning for Specialized Domains
```python
from indoxGen.synthCore import FewShotPrompt
from indoxGen.llms import OpenAi
openai = OpenAi(api_key=OPENAI_API_KEY, model="gpt-4o-mini")
examples = [
{
"input": "Generate a dataset with 3 columns and 2 rows about quantum computing.",
"output": '[{"Qubit Type": "Superconducting", "Coherence Time": "100 μs", "Gate Fidelity": "0.9999"}, {"Qubit Type": "Trapped Ion", "Coherence Time": "10 ms", "Gate Fidelity": "0.99999"}]'
},
{
"input": "Generate a dataset with 3 columns and 2 rows about nanotechnology.",
"output": '[{"Material": "Graphene", "Thickness": "1 nm", "Conductivity": "1.0e6 S/m"}, {"Material": "Carbon Nanotube", "Thickness": "1-2 nm", "Conductivity": "1.0e7 S/m"}]'
}
]
user_prompt = "Generate a dataset with 3 columns and 2 rows about advanced AI architectures."
data_generator = FewShotPrompt(
prompt_name="Generate AI Architecture Dataset",
args={
"llm": openai,
"n": 1,
"instruction": user_prompt,
},
outputs={"generations": "generate"},
examples=examples
)
generated_df = data_generator.run()
data_generator.save_to_excel("ai_architectures.xlsx", generated_df)
```
### Attributed Prompts for Controlled Variation
```python
from indoxGen.synthCore import DataFromAttributedPrompt
from indoxGen.llms import OpenAi
openai = OpenAi(api_key=OPENAI_API_KEY, model="gpt-4o-mini")
args = {
"instruction": "Generate a {complexity} machine learning algorithm description that is {application_area} focused.",
"attributes": {
"complexity": ["basic", "advanced", "cutting-edge"],
"application_area": ["computer vision", "natural language processing", "reinforcement learning"]
},
"llm": openai
}
dataset = DataFromAttributedPrompt(
prompt_name="ML Algorithm Generator",
args=args,
outputs={}
)
df = dataset.run()
print(df)
```
## Configuration and Customization
Each generator class in IndoxGen is highly configurable to meet specific data generation requirements. Key parameters include:
- `generator_llm` and `judge_llm`: Specify the LLMs used for generation and quality assessment
- `columns` and `example_data`: Define the structure and provide examples for the generated data
- `user_instruction`: Customize the generation process with specific guidelines
- `diversity_threshold`: Control the level of variation in the generated data
- `verbose`: Adjust the level of feedback during the generation process
Refer to the API documentation for a comprehensive list of configuration options for each class.
## Best Practices
1. **Data Quality Assurance**: Regularly validate generated data against predefined quality metrics.
2. **Iterative Refinement**: Utilize the human feedback loop to continuously improve generation quality.
3. **Domain Expertise Integration**: Collaborate with domain experts to fine-tune generation parameters and validate outputs.
4. **Ethical Considerations**: Ensure generated data adheres to privacy standards and ethical guidelines.
5. **Performance Optimization**: Monitor and optimize generation pipeline for large-scale tasks.
## Roadmap
* [x] Implement basic synthetic data generation
* [x] Add LLM-based judge for quality control
* [x] Improve diversity checking mechanism
* [x] Integrate human feedback loop for continuous improvement
* [ ] Develop a web-based UI for easier interaction
* [ ] Support for more data types (images, time series, etc.)
* [ ] Implement differential privacy techniques
* [ ] Create plugin system for custom data generation rules
* [ ] Develop comprehensive documentation and tutorials
## Contributing
We welcome contributions! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) file for details on how to get started.
## License
IndoxGen is released under the MIT License. See [LICENSE.md](LICENSE.md) for more details.
---
IndoxGen - Empowering Data-Driven Innovation with Advanced Synthetic Data Generation
Raw data
{
"_id": null,
"home_page": "https://github.com/osllmai/IndoxGen",
"name": "IndoxGen",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.9",
"maintainer_email": null,
"keywords": "AI, deep learning, language models, synthetic data generation, machine learning, NLP",
"author": "nerdstudio",
"author_email": "ashkan@nematifamilyfundation.onmicrosoft.com",
"download_url": "https://files.pythonhosted.org/packages/e7/15/614201a8654d9c8d685069ab9cf7cd9618661a2fd3e27277649f6ddf98f1/indoxgen-0.0.3.tar.gz",
"platform": null,
"description": "\r\n# IndoxGen: Enterprise-Grade Synthetic Data Generation Framework\r\n\r\n\r\n[](https://github.com/osllmai/indoxGen/blob/main/LICENSE)\r\n[](https://pypi.org/project/indoxGen/0.0.3/)\r\n[](https://pypi.org/project/indoxGen/0.0.3/)\r\n[](https://pepy.tech/project/indoxGen)\r\n\r\n[](https://discord.com/invite/ossllmai)\r\n[](https://github.com/osllmai/indoxGen)\r\n\r\n<p align=\"center\">\r\n <a href=\"https://osllm.ai\">Official Website</a> • <a href=\"https://docs.osllm.ai/index.html\">Documentation</a> • <a href=\"https://discord.gg/qrCc56ZR\">Discord</a>\r\n</p>\r\n\r\n<p align=\"center\">\r\n <b>NEW:</b> <a href=\"https://docs.google.com/forms/d/1CQXJvxLUqLBSXnjqQmRpOyZqD6nrKubLz2WTcIJ37fU/prefill\">Subscribe to our mailing list</a> for updates and news!\r\n</p>\r\n\r\n\r\n## Overview\r\n\r\nIndoxGen is a state-of-the-art, enterprise-ready framework designed for generating high-fidelity synthetic data. Leveraging advanced AI technologies, including Large Language Models (LLMs) and incorporating human feedback loops, IndoxGen offers unparalleled flexibility and precision in synthetic data creation across various domains and use cases.\r\n\r\n## Key Features\r\n\r\n- **Multiple Generation Pipelines**:\r\n - `SyntheticDataGenerator`: Standard LLM-powered generation pipeline for structured data with embedded quality control mechanisms.\r\n - `SyntheticDataGeneratorHF`: Advanced pipeline integrating human feedback to improve generation.\r\n - `DataFromPrompt`: Dynamic data generation based on natural language prompts, useful for rapid prototyping.\r\n \r\n- **Customization & Control**: Fine-grained control over data attributes, structure, and diversity. Customize every aspect of the synthetic data generation process.\r\n \r\n- **Human-in-the-Loop**: Seamlessly integrates expert feedback for continuous improvement of generated data, offering the highest quality assurance.\r\n \r\n- **AI-Driven Diversity**: Algorithms ensure representative and varied datasets, providing data diversity for robust modeling.\r\n \r\n- **Flexible I/O**: Supports various data sources and export formats (Excel, CSV, etc.) for easy integration into existing workflows.\r\n \r\n- **Advanced Learning Techniques**: Incorporation of few-shot learning for rapid adaptation to new domains with minimal examples.\r\n \r\n- **Scalability**: Designed to handle both small-scale experiments and large-scale data generation tasks with multi-LLM support.\r\n\r\n## Installation\r\n\r\n```bash\r\npip install indoxgen\r\n```\r\n\r\n## Quick Start Guide\r\n\r\n### Basic Usage: SyntheticDataGenerator\r\n\r\n```python\r\nfrom indoxGen.synthCore import SyntheticDataGenerator\r\nfrom indoxGen.llms import OpenAi\r\n\r\ncolumns = [\"name\", \"age\", \"occupation\"]\r\nexample_data = [\r\n {\"name\": \"Alice Johnson\", \"age\": 35, \"occupation\": \"Manager\"},\r\n {\"name\": \"Bob Williams\", \"age\": 42, \"occupation\": \"Accountant\"}\r\n]\r\n\r\nopenai = OpenAi(api_key=OPENAI_API_KEY, model=\"gpt-4o-mini\")\r\nnemotron = OpenAi(api_key=NVIDIA_API_KEY, model=\"nvidia/nemotron-4-340b-instruct\",\r\n base_url=\"https://integrate.api.nvidia.com/v1\")\r\n\r\ngenerator = SyntheticDataGenerator(\r\n generator_llm=nemotron,\r\n judge_llm=openai,\r\n columns=columns,\r\n example_data=example_data,\r\n user_instruction=\"Generate diverse, realistic data including name, age, and occupation. Ensure variability in demographics and professions.\",\r\n verbose=1\r\n)\r\n\r\ngenerated_data = generator.generate_data(num_samples=100)\r\n```\r\n\r\n### Advanced Usage: SyntheticDataGeneratorHF with Human Feedback\r\n\r\n```python\r\nfrom indoxGen.synthCore import SyntheticDataGeneratorHF\r\nfrom indoxGen.llms import OpenAi\r\n\r\nopenai = OpenAi(api_key=OPENAI_API_KEY, model=\"gpt-4-0613\")\r\nnemotron = OpenAi(api_key=NVIDIA_API_KEY, model=\"nvidia/nemotron-4-340b-instruct\",\r\n base_url=\"https://integrate.api.nvidia.com/v1\")\r\n\r\ngenerator = SyntheticDataGeneratorHF(\r\n generator_llm=nemotron,\r\n judge_llm=openai,\r\n columns=columns,\r\n example_data=example_data,\r\n user_instruction=\"Generate diverse, realistic professional profiles with name, age, and occupation.\",\r\n verbose=1,\r\n diversity_threshold=0.4,\r\n feedback_range=feedback_range\r\n)\r\n\r\n# Implement human feedback loop\r\ngenerator.user_review_and_regenerate(\r\n regenerate_rows=[0],\r\n accepted_rows=[],\r\n regeneration_feedback='Diversify names and occupations further',\r\n min_score=0.7\r\n)\r\n```\r\n\r\n### Prompt-Based Generation: DataFromPrompt\r\n\r\n```python\r\nfrom indoxGen.synthCore import DataFromPrompt, DataGenerationPrompt\r\nfrom indoxGen.llms import OpenAi\r\n\r\nnemotron = OpenAi(api_key=NVIDIA_API_KEY, model=\"nvidia/nemotron-4-340b-instruct\",\r\n base_url=\"https://integrate.api.nvidia.com/v1\")\r\n\r\n\r\nuser_prompt = \"Generate a comprehensive dataset with 3 columns and 3 rows about exoplanets.\"\r\ninstruction = DataGenerationPrompt.get_instruction(user_prompt)\r\n\r\ndata_generator = DataFromPrompt(\r\n prompt_name=\"Exoplanet Dataset Generation\",\r\n args={\r\n \"llm\": nemotron,\r\n \"n\": 1,\r\n \"instruction\": instruction,\r\n },\r\n outputs={\"generations\": \"generate\"},\r\n)\r\n\r\ngenerated_df = data_generator.run()\r\ndata_generator.save_to_excel(\"exoplanet_data.xlsx\")\r\n```\r\n\r\n## Advanced Techniques\r\n\r\n### Few-Shot Learning for Specialized Domains\r\n\r\n```python\r\nfrom indoxGen.synthCore import FewShotPrompt\r\nfrom indoxGen.llms import OpenAi\r\n\r\nopenai = OpenAi(api_key=OPENAI_API_KEY, model=\"gpt-4o-mini\")\r\n\r\nexamples = [\r\n {\r\n \"input\": \"Generate a dataset with 3 columns and 2 rows about quantum computing.\",\r\n \"output\": '[{\"Qubit Type\": \"Superconducting\", \"Coherence Time\": \"100 \u03bcs\", \"Gate Fidelity\": \"0.9999\"}, {\"Qubit Type\": \"Trapped Ion\", \"Coherence Time\": \"10 ms\", \"Gate Fidelity\": \"0.99999\"}]'\r\n },\r\n {\r\n \"input\": \"Generate a dataset with 3 columns and 2 rows about nanotechnology.\",\r\n \"output\": '[{\"Material\": \"Graphene\", \"Thickness\": \"1 nm\", \"Conductivity\": \"1.0e6 S/m\"}, {\"Material\": \"Carbon Nanotube\", \"Thickness\": \"1-2 nm\", \"Conductivity\": \"1.0e7 S/m\"}]'\r\n }\r\n]\r\n\r\nuser_prompt = \"Generate a dataset with 3 columns and 2 rows about advanced AI architectures.\"\r\n\r\ndata_generator = FewShotPrompt(\r\n prompt_name=\"Generate AI Architecture Dataset\",\r\n args={\r\n \"llm\": openai,\r\n \"n\": 1, \r\n \"instruction\": user_prompt, \r\n },\r\n outputs={\"generations\": \"generate\"},\r\n examples=examples \r\n)\r\n\r\ngenerated_df = data_generator.run()\r\ndata_generator.save_to_excel(\"ai_architectures.xlsx\", generated_df)\r\n```\r\n\r\n### Attributed Prompts for Controlled Variation\r\n\r\n```python\r\nfrom indoxGen.synthCore import DataFromAttributedPrompt\r\nfrom indoxGen.llms import OpenAi\r\n\r\nopenai = OpenAi(api_key=OPENAI_API_KEY, model=\"gpt-4o-mini\")\r\n\r\nargs = {\r\n \"instruction\": \"Generate a {complexity} machine learning algorithm description that is {application_area} focused.\",\r\n \"attributes\": {\r\n \"complexity\": [\"basic\", \"advanced\", \"cutting-edge\"],\r\n \"application_area\": [\"computer vision\", \"natural language processing\", \"reinforcement learning\"]\r\n },\r\n \"llm\": openai\r\n}\r\n\r\ndataset = DataFromAttributedPrompt(\r\n prompt_name=\"ML Algorithm Generator\",\r\n args=args,\r\n outputs={}\r\n)\r\n\r\ndf = dataset.run()\r\nprint(df)\r\n```\r\n\r\n## Configuration and Customization\r\n\r\nEach generator class in IndoxGen is highly configurable to meet specific data generation requirements. Key parameters include:\r\n\r\n- `generator_llm` and `judge_llm`: Specify the LLMs used for generation and quality assessment\r\n- `columns` and `example_data`: Define the structure and provide examples for the generated data\r\n- `user_instruction`: Customize the generation process with specific guidelines\r\n- `diversity_threshold`: Control the level of variation in the generated data\r\n- `verbose`: Adjust the level of feedback during the generation process\r\n\r\nRefer to the API documentation for a comprehensive list of configuration options for each class.\r\n\r\n## Best Practices\r\n\r\n1. **Data Quality Assurance**: Regularly validate generated data against predefined quality metrics.\r\n2. **Iterative Refinement**: Utilize the human feedback loop to continuously improve generation quality.\r\n3. **Domain Expertise Integration**: Collaborate with domain experts to fine-tune generation parameters and validate outputs.\r\n4. **Ethical Considerations**: Ensure generated data adheres to privacy standards and ethical guidelines.\r\n5. **Performance Optimization**: Monitor and optimize generation pipeline for large-scale tasks.\r\n\r\n## Roadmap\r\n* [x] Implement basic synthetic data generation\r\n* [x] Add LLM-based judge for quality control\r\n* [x] Improve diversity checking mechanism\r\n* [x] Integrate human feedback loop for continuous improvement\r\n* [ ] Develop a web-based UI for easier interaction\r\n* [ ] Support for more data types (images, time series, etc.)\r\n* [ ] Implement differential privacy techniques\r\n* [ ] Create plugin system for custom data generation rules\r\n* [ ] Develop comprehensive documentation and tutorials\r\n\r\n## Contributing\r\n\r\nWe welcome contributions! Please see our [CONTRIBUTING.md](CONTRIBUTING.md) file for details on how to get started.\r\n\r\n## License\r\n\r\nIndoxGen is released under the MIT License. See [LICENSE.md](LICENSE.md) for more details.\r\n\r\n\r\n---\r\n\r\nIndoxGen - Empowering Data-Driven Innovation with Advanced Synthetic Data Generation\r\n",
"bugtrack_url": null,
"license": "AGPL-3.0",
"summary": "Indox Synthetic Data Generation",
"version": "0.0.3",
"project_urls": {
"Homepage": "https://github.com/osllmai/IndoxGen"
},
"split_keywords": [
"ai",
" deep learning",
" language models",
" synthetic data generation",
" machine learning",
" nlp"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "87353ecacedeaa49526f39d1581f93da78415794609b1754a7615280fa165bac",
"md5": "b1d5a0b241da139552d78d1cdb459fa0",
"sha256": "191d36431bb077e0f74e5dadff02961335842ff00acbc13bda27cf60f2250d90"
},
"downloads": -1,
"filename": "IndoxGen-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b1d5a0b241da139552d78d1cdb459fa0",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.9",
"size": 42294,
"upload_time": "2024-09-30T16:33:53",
"upload_time_iso_8601": "2024-09-30T16:33:53.554851Z",
"url": "https://files.pythonhosted.org/packages/87/35/3ecacedeaa49526f39d1581f93da78415794609b1754a7615280fa165bac/IndoxGen-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "e715614201a8654d9c8d685069ab9cf7cd9618661a2fd3e27277649f6ddf98f1",
"md5": "aa93be94118732d8142d63b7d99eb156",
"sha256": "db763127b46cf99b8de7311a199ba2b4a2eb2d6baa32d1132b398e6b71a271bf"
},
"downloads": -1,
"filename": "indoxgen-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "aa93be94118732d8142d63b7d99eb156",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.9",
"size": 35136,
"upload_time": "2024-09-30T16:33:55",
"upload_time_iso_8601": "2024-09-30T16:33:55.830770Z",
"url": "https://files.pythonhosted.org/packages/e7/15/614201a8654d9c8d685069ab9cf7cd9618661a2fd3e27277649f6ddf98f1/indoxgen-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-09-30 16:33:55",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "osllmai",
"github_project": "IndoxGen",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "loguru",
"specs": [
[
"==",
"0.7.2"
]
]
},
{
"name": "numpy",
"specs": [
[
"==",
"2.1.1"
]
]
},
{
"name": "pandas",
"specs": [
[
"==",
"2.2.3"
]
]
},
{
"name": "protobuf",
"specs": [
[
"==",
"3.20.3"
]
]
},
{
"name": "Requests",
"specs": [
[
"==",
"2.32.3"
]
]
},
{
"name": "scikit_learn",
"specs": [
[
"==",
"1.4.2"
]
]
},
{
"name": "setuptools",
"specs": [
[
"==",
"69.5.1"
]
]
},
{
"name": "tenacity",
"specs": [
[
"==",
"8.2.2"
]
]
}
],
"lcname": "indoxgen"
}
JSON
