# Data Processing Library
This provides a python framework for developing _transforms_
on data stored in files - currently parquet files are supported -
and running them in a [ray](https://www.ray.io/) cluster.
Data files may be stored in the local file system or COS/S3.
For more details see the [documentation](../doc/overview.md).
### Virtual Environment
The project uses `pyproject.toml` and a Makefile for operations.
To do development you should establish the virtual environment
```shell
make venv
```
and then either activate
```shell
source venv/bin/activate
```
or set up your IDE to use the venv directory when developing in this project
## Library Artifact Build and Publish
To test, build and publish the library
```shell
make test build publish
```
To up the version number, edit the Makefile to change VERSION and rerun
the above. This will require committing both the `Makefile` and the
autotmatically updated `pyproject.toml` file.
Raw data
{
"_id": null,
"home_page": null,
"name": "data-prep-toolkit",
"maintainer": null,
"docs_url": null,
"requires_python": "<3.13,>=3.10",
"maintainer_email": null,
"keywords": "data, data preprocessing, data preparation, llm, generative, ai, fine-tuning, llmapps",
"author": null,
"author_email": "Maroun Touma <touma@us.ibm.com>",
"download_url": null,
"platform": null,
"description": "# Data Processing Library\nThis provides a python framework for developing _transforms_\non data stored in files - currently parquet files are supported -\nand running them in a [ray](https://www.ray.io/) cluster.\nData files may be stored in the local file system or COS/S3.\nFor more details see the [documentation](../doc/overview.md).\n\n### Virtual Environment\nThe project uses `pyproject.toml` and a Makefile for operations.\nTo do development you should establish the virtual environment\n```shell\nmake venv\n```\nand then either activate\n```shell\nsource venv/bin/activate\n```\nor set up your IDE to use the venv directory when developing in this project\n\n## Library Artifact Build and Publish\nTo test, build and publish the library \n```shell\nmake test build publish\n```\n\nTo up the version number, edit the Makefile to change VERSION and rerun\nthe above. This will require committing both the `Makefile` and the\nautotmatically updated `pyproject.toml` file.\n\n\n\n",
"bugtrack_url": null,
"license": "Apache-2.0",
"summary": "Data Preparation Toolkit Library for Ray and Python",
"version": "0.2.3.post1",
"project_urls": null,
"split_keywords": [
"data",
" data preprocessing",
" data preparation",
" llm",
" generative",
" ai",
" fine-tuning",
" llmapps"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "ab181ad8a352e768a2a508db44b775420e74875374bf3dd65579c3119ed6b571",
"md5": "337162b8eef782136b30589a8afe38d8",
"sha256": "6093ef4e12ff1930f7378a6ce461df5ab360160848284edb2e5f04dc8d729365"
},
"downloads": -1,
"filename": "data_prep_toolkit-0.2.3.post1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "337162b8eef782136b30589a8afe38d8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<3.13,>=3.10",
"size": 112773,
"upload_time": "2025-01-28T17:41:19",
"upload_time_iso_8601": "2025-01-28T17:41:19.553736Z",
"url": "https://files.pythonhosted.org/packages/ab/18/1ad8a352e768a2a508db44b775420e74875374bf3dd65579c3119ed6b571/data_prep_toolkit-0.2.3.post1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-01-28 17:41:19",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "data-prep-toolkit"
}
JSON