| Name | dialektik JSON |
| Version |
0.0.3
JSON |
| download |
| home_page | https://github.com/JosefAlbers/Dialektik |
| Summary | Merge. Synthesize. Create. Dialektik generates new content by fusing ideas from diverse sources, revealing unexpected insights and perspectives. |
| upload_time | 2024-08-10 02:41:15 |
| maintainer | None |
| docs_url | None |
| author | Josef Albers |
| requires_python | >=3.12.3 |
| license | MIT |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# Dialektik
Merge. Synthesize. Create. Dialektik generates new content by fusing ideas from diverse sources, revealing unexpected insights and perspectives through a dialectical process.
## Features
- Loads and processes datasets from multiple sources
- Summarizes text into concise bullet points
- Performs semantic search for topic-based content selection
- Generates thesis, antithesis, and synthesis from summarized content
- Uses phi-3-vision-mlx for text generation and embeddings
## Requirements
- Python 3.8+
- `datasets`
- `huggingface_hub`
- `phi_3_vision_mlx`
- `mlx`
- `fire`
## Installation
To install Dialektik with all required dependencies:
```
pip install dialektik
```
Note: Make sure you have the necessary system requirements to run phi-3-vision-mlx and mlx.
## Usage
### Command Line Interface
Dialektik can be used from the command line after installation. Here are some example usages:
1. Generate a synthesis with default settings:
```
python dialektik.py
```
2. Specify a topic for semantic search:
```
python dialektik.py --topic "AI agents"
```
3. Specify sources and exclude certain terms:
```
python dialektik.py --list_source arxiv --list_exclude MIRI "Machine Intelligence Research Institute"
```
4. Set the number of books and bullet points per book:
```
python dialektik.py --num_book 5 --per_book 4
```
5. Use a different language model via API:
```
python dialektik.py --llm_model "mistralai/Mistral-Nemo-Instruct-2407"
```
6. For a full list of options, use:
```
python dialektik.py --help
```
### Python API
You can also use Dialektik in your Python scripts:
```python
from dialektik import synthesize
# Generate a synthesis with default settings
thesis, antithesis, synthesis = synthesize()
# Customize the synthesis process
output = synthesize(
topic="AI agents",
list_source=['arxiv'],
list_exclude=['MIRI', 'Machine Intelligence Research Institute'],
num_book=3,
per_book=3
)
```
### Accessing the Dataset
The default dataset at 'JosefAlbers/StampyAI-alignment-research-dataset' is publicly available. You don't need to set up any environment variables to use `dialektik` with this dataset.
### Using Custom Datasets
If you want to use your own dataset:
1. Prepare your dataset according to the required format.
2. Modify the `PATH_DS` variable in the code to point to your dataset.
3. If your dataset is private or requires authentication, set up the following environment variables:
- `HF_WRITE_TOKEN`: Hugging Face write token (for pushing datasets)
- `HF_READ_TOKEN`: Hugging Face read token (for accessing private datasets)
## Customizing the Model
Dialektik now uses phi-3-vision-mlx for text generation and embeddings. This model is not easily swappable, but you can modify the `pv.load()` and `pv.generate()` calls in the code if you need to use a different model.
## Output
The `synthesize()` function generates three outputs:
1. Thesis: An article exploring the main themes and insights from the selected sources.
2. Antithesis: A text presenting alternative perspectives and counterarguments to the thesis.
3. Synthesis: A reconciliation of the thesis and antithesis, presenting a new, unified viewpoint.
All outputs are saved in the 'syntheses' folder with timestamps for easy reference.
## License
This project is licensed under the [MIT License](LICENSE).
## Citation
<a href="https://zenodo.org/doi/10.5281/zenodo.11403221"><img src="https://zenodo.org/badge/806709541.svg" alt="DOI"></a>
## Contributing
Contributions to Dialektik are always welcome! Please feel free to submit a Pull Request.
Raw data
{
"_id": null,
"home_page": "https://github.com/JosefAlbers/Dialektik",
"name": "dialektik",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.12.3",
"maintainer_email": null,
"keywords": null,
"author": "Josef Albers",
"author_email": "albersj66@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/45/7b/f47de9d5e0d349b84eb672e04900bb5dcf47b341810740ca53f4f1227fde/dialektik-0.0.3.tar.gz",
"platform": null,
"description": "# Dialektik\n\nMerge. Synthesize. Create. Dialektik generates new content by fusing ideas from diverse sources, revealing unexpected insights and perspectives through a dialectical process.\n\n## Features\n\n- Loads and processes datasets from multiple sources\n- Summarizes text into concise bullet points\n- Performs semantic search for topic-based content selection\n- Generates thesis, antithesis, and synthesis from summarized content\n- Uses phi-3-vision-mlx for text generation and embeddings\n\n## Requirements\n\n- Python 3.8+\n- `datasets`\n- `huggingface_hub`\n- `phi_3_vision_mlx`\n- `mlx`\n- `fire`\n\n## Installation\n\nTo install Dialektik with all required dependencies:\n\n```\npip install dialektik\n```\n\nNote: Make sure you have the necessary system requirements to run phi-3-vision-mlx and mlx.\n\n## Usage\n\n### Command Line Interface\n\nDialektik can be used from the command line after installation. Here are some example usages:\n\n1. Generate a synthesis with default settings:\n\n ```\n python dialektik.py\n ```\n\n2. Specify a topic for semantic search:\n\n ```\n python dialektik.py --topic \"AI agents\"\n ```\n\n3. Specify sources and exclude certain terms:\n\n ```\n python dialektik.py --list_source arxiv --list_exclude MIRI \"Machine Intelligence Research Institute\"\n ```\n\n4. Set the number of books and bullet points per book:\n\n ```\n python dialektik.py --num_book 5 --per_book 4\n ```\n\n5. Use a different language model via API:\n\n ```\n python dialektik.py --llm_model \"mistralai/Mistral-Nemo-Instruct-2407\"\n ```\n\n6. For a full list of options, use:\n\n ```\n python dialektik.py --help\n ```\n\n### Python API\n\nYou can also use Dialektik in your Python scripts:\n\n```python\nfrom dialektik import synthesize\n\n# Generate a synthesis with default settings\nthesis, antithesis, synthesis = synthesize()\n\n# Customize the synthesis process\noutput = synthesize(\n topic=\"AI agents\",\n list_source=['arxiv'],\n list_exclude=['MIRI', 'Machine Intelligence Research Institute'],\n num_book=3,\n per_book=3\n)\n```\n\n### Accessing the Dataset\n\nThe default dataset at 'JosefAlbers/StampyAI-alignment-research-dataset' is publicly available. You don't need to set up any environment variables to use `dialektik` with this dataset.\n\n### Using Custom Datasets\n\nIf you want to use your own dataset:\n\n1. Prepare your dataset according to the required format.\n2. Modify the `PATH_DS` variable in the code to point to your dataset.\n3. If your dataset is private or requires authentication, set up the following environment variables:\n - `HF_WRITE_TOKEN`: Hugging Face write token (for pushing datasets)\n - `HF_READ_TOKEN`: Hugging Face read token (for accessing private datasets)\n\n## Customizing the Model\n\nDialektik now uses phi-3-vision-mlx for text generation and embeddings. This model is not easily swappable, but you can modify the `pv.load()` and `pv.generate()` calls in the code if you need to use a different model.\n\n## Output\n\nThe `synthesize()` function generates three outputs:\n\n1. Thesis: An article exploring the main themes and insights from the selected sources.\n2. Antithesis: A text presenting alternative perspectives and counterarguments to the thesis.\n3. Synthesis: A reconciliation of the thesis and antithesis, presenting a new, unified viewpoint.\n\nAll outputs are saved in the 'syntheses' folder with timestamps for easy reference.\n\n## License\n\nThis project is licensed under the [MIT License](LICENSE).\n\n## Citation\n\n<a href=\"https://zenodo.org/doi/10.5281/zenodo.11403221\"><img src=\"https://zenodo.org/badge/806709541.svg\" alt=\"DOI\"></a>\n\n## Contributing\n\nContributions to Dialektik are always welcome! Please feel free to submit a Pull Request.\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Merge. Synthesize. Create. Dialektik generates new content by fusing ideas from diverse sources, revealing unexpected insights and perspectives.",
"version": "0.0.3",
"project_urls": {
"Homepage": "https://github.com/JosefAlbers/Dialektik"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d8c967447482265bfe7d0bf582788015d774c839d04602c7d67799e88672f8dc",
"md5": "0573e73267cb55f62ca48d91d2746039",
"sha256": "566d91495e33c0ab5e7960e88aae72a32c66245540d57fd6bd52ad512a89b1e1"
},
"downloads": -1,
"filename": "dialektik-0.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "0573e73267cb55f62ca48d91d2746039",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.12.3",
"size": 5865,
"upload_time": "2024-08-10T02:41:14",
"upload_time_iso_8601": "2024-08-10T02:41:14.593712Z",
"url": "https://files.pythonhosted.org/packages/d8/c9/67447482265bfe7d0bf582788015d774c839d04602c7d67799e88672f8dc/dialektik-0.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "457bf47de9d5e0d349b84eb672e04900bb5dcf47b341810740ca53f4f1227fde",
"md5": "d800e5965f2e9cc111764dcfeb9f1d6c",
"sha256": "a56e23a0126c08ec37d9be9291b7f8577c996c957d90bfd19159fd289c750001"
},
"downloads": -1,
"filename": "dialektik-0.0.3.tar.gz",
"has_sig": false,
"md5_digest": "d800e5965f2e9cc111764dcfeb9f1d6c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.12.3",
"size": 5574,
"upload_time": "2024-08-10T02:41:15",
"upload_time_iso_8601": "2024-08-10T02:41:15.838291Z",
"url": "https://files.pythonhosted.org/packages/45/7b/f47de9d5e0d349b84eb672e04900bb5dcf47b341810740ca53f4f1227fde/dialektik-0.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-08-10 02:41:15",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "JosefAlbers",
"github_project": "Dialektik",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "dialektik"
}
