# Palimpzest (PZ)
[](https://discord.gg/dN85JJ6jaH)
[](https://palimpzest.org/)
[](https://colab.research.google.com/drive/1zqOxnh_G6eZ8_xax6PvDr-EjMt7hp4R5?usp=sharing)
[](https://pypi.org/project/palimpzest/)
[](https://pypi.org/project/palimpzest/)
<!-- [](https://arxiv.org/pdf/2405.14696) -->
<!-- [](https://youtu.be/T8VQfyBiki0?si=eiph57DSEkDNbEIu) -->
## Learn How to Use PZ
Our [full documentation](https://palimpzest.org) is the definitive resource for learning how to use PZ. It contains all of the installation and quickstart materials on this page, as well as user guides, full API documentation, and much more.
## Getting started
You can find a stable version of the PZ package on PyPI [here](https://pypi.org/project/palimpzest/). To install the package, run:
```bash
$ pip install palimpzest
```
Alternatively, to install the latest version of the package from this repository, you can clone this repository and run the following commands:
```bash
$ git clone git@github.com:mitdbg/palimpzest.git
$ cd palimpzest
$ pip install .
```
## Join the PZ Community
We are actively hacking on PZ and would love to have you join our community [](https://discord.gg/dN85JJ6jaH)
[Our Discord server](https://discord.gg/dN85JJ6jaH) is the best place to:
- Get help with your PZ program(s)
- Give feedback to the maintainers
- Discuss the future direction(s) of the project
- Discuss anything related to data processing with LLMs!
We are eager to learn more about your workloads and use cases, and will take them into consideration in planning our future roadmap.
## Quick Start
The easiest way to get started with Palimpzest is to run the `quickstart.ipynb` jupyter notebook. We demonstrate the full workflow of working with PZ, including registering a dataset, composing and executing a pipeline, and accessing the results.
To run the notebook, you can use the following command:
```bash
$ jupyter notebook
```
And then access the notebook from the jupyter interface in your browser at `localhost:8888`.
### Even Quicker Start
For eager readers, the code in the notebook can be found in the following condensed snippet. However, we do suggest reading the notebook as it contains more insight into each element of the program.
```python
import palimpzest as pz
# define the fields we wish to compute
email_cols = [
{"name": "sender", "type": str, "desc": "The email address of the sender"},
{"name": "subject", "type": str, "desc": "The subject of the email"},
{"name": "date", "type": str, "desc": "The date the email was sent"},
]
# lazily construct the computation to get emails about holidays sent in July
dataset = pz.Dataset("testdata/enron-tiny/")
dataset = dataset.sem_add_columns(email_cols)
dataset = dataset.sem_filter("The email was sent in July")
dataset = dataset.sem_filter("The email is about holidays")
# execute the computation w/the MinCost policy
config = pz.QueryProcessorConfig(policy=pz.MinCost(), verbose=True)
output = dataset.run(config)
# display output (if using Jupyter, otherwise use print(output_df))
output_df = output.to_df(cols=["date", "sender", "subject"])
display(output_df)
```
## Python Demos
Below are simple instructions to run PZ on a test data set of enron emails that is included with the system.
### Downloading test data
To run the provided demos, you will need to download the test data. Due to the size of the data, we are unable to include it in the repository. You can download the test data by running the following command from a unix terminal (requires `wget` and `tar`):
```
chmod +x testdata/download-testdata.sh
./testdata/download-testdata.sh
```
### Running the Demos
Set your OpenAI (or Together.ai) api key at the command line:
```bash
# set one (or both) of the following:
export OPENAI_API_KEY=<your-api-key>
export TOGETHER_API_KEY=<your-api-key>
```
Now you can run the simple test program with:
```bash
$ python demos/simple-demo.py --task enron --dataset testdata/enron-eval-tiny --verbose
```
Raw data
{
"_id": null,
"home_page": null,
"name": "palimpzest",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "relational, optimization, llm, AI programming, extraction, tools, document, search, integration",
"author": null,
"author_email": "MIT DSG Semantic Management Lab <michjc@csail.mit.edu>",
"download_url": "https://files.pythonhosted.org/packages/8b/61/6987698b3ff6d417b5ff311335aac03e6b76764aa9a5b452cdf4711ac511/palimpzest-0.6.4.tar.gz",
"platform": null,
"description": "\n\n# Palimpzest (PZ)\n[](https://discord.gg/dN85JJ6jaH)\n[](https://palimpzest.org/)\n[](https://colab.research.google.com/drive/1zqOxnh_G6eZ8_xax6PvDr-EjMt7hp4R5?usp=sharing)\n[](https://pypi.org/project/palimpzest/)\n[](https://pypi.org/project/palimpzest/)\n<!-- [](https://arxiv.org/pdf/2405.14696) -->\n<!-- [](https://youtu.be/T8VQfyBiki0?si=eiph57DSEkDNbEIu) -->\n\n## Learn How to Use PZ\nOur [full documentation](https://palimpzest.org) is the definitive resource for learning how to use PZ. It contains all of the installation and quickstart materials on this page, as well as user guides, full API documentation, and much more.\n\n## Getting started\nYou can find a stable version of the PZ package on PyPI [here](https://pypi.org/project/palimpzest/). To install the package, run:\n```bash\n$ pip install palimpzest\n```\n\nAlternatively, to install the latest version of the package from this repository, you can clone this repository and run the following commands:\n```bash\n$ git clone git@github.com:mitdbg/palimpzest.git\n$ cd palimpzest\n$ pip install .\n```\n\n## Join the PZ Community\nWe are actively hacking on PZ and would love to have you join our community [](https://discord.gg/dN85JJ6jaH)\n\n[Our Discord server](https://discord.gg/dN85JJ6jaH) is the best place to:\n- Get help with your PZ program(s)\n- Give feedback to the maintainers\n- Discuss the future direction(s) of the project\n- Discuss anything related to data processing with LLMs!\n\nWe are eager to learn more about your workloads and use cases, and will take them into consideration in planning our future roadmap.\n\n## Quick Start\nThe easiest way to get started with Palimpzest is to run the `quickstart.ipynb` jupyter notebook. We demonstrate the full workflow of working with PZ, including registering a dataset, composing and executing a pipeline, and accessing the results.\nTo run the notebook, you can use the following command:\n```bash\n$ jupyter notebook\n```\nAnd then access the notebook from the jupyter interface in your browser at `localhost:8888`.\n\n### Even Quicker Start\nFor eager readers, the code in the notebook can be found in the following condensed snippet. However, we do suggest reading the notebook as it contains more insight into each element of the program.\n```python\nimport palimpzest as pz\n\n# define the fields we wish to compute\nemail_cols = [\n {\"name\": \"sender\", \"type\": str, \"desc\": \"The email address of the sender\"},\n {\"name\": \"subject\", \"type\": str, \"desc\": \"The subject of the email\"},\n {\"name\": \"date\", \"type\": str, \"desc\": \"The date the email was sent\"},\n]\n\n# lazily construct the computation to get emails about holidays sent in July\ndataset = pz.Dataset(\"testdata/enron-tiny/\")\ndataset = dataset.sem_add_columns(email_cols)\ndataset = dataset.sem_filter(\"The email was sent in July\")\ndataset = dataset.sem_filter(\"The email is about holidays\")\n\n# execute the computation w/the MinCost policy\nconfig = pz.QueryProcessorConfig(policy=pz.MinCost(), verbose=True)\noutput = dataset.run(config)\n\n# display output (if using Jupyter, otherwise use print(output_df))\noutput_df = output.to_df(cols=[\"date\", \"sender\", \"subject\"])\ndisplay(output_df)\n```\n\n## Python Demos\nBelow are simple instructions to run PZ on a test data set of enron emails that is included with the system.\n\n### Downloading test data\nTo run the provided demos, you will need to download the test data. Due to the size of the data, we are unable to include it in the repository. You can download the test data by running the following command from a unix terminal (requires `wget` and `tar`):\n```\nchmod +x testdata/download-testdata.sh\n./testdata/download-testdata.sh\n```\n\n### Running the Demos\nSet your OpenAI (or Together.ai) api key at the command line:\n```bash\n# set one (or both) of the following:\nexport OPENAI_API_KEY=<your-api-key>\nexport TOGETHER_API_KEY=<your-api-key>\n```\n\nNow you can run the simple test program with:\n```bash\n$ python demos/simple-demo.py --task enron --dataset testdata/enron-eval-tiny --verbose\n```\n",
"bugtrack_url": null,
"license": null,
"summary": "Palimpzest is a system which enables anyone to process AI-powered analytical queries simply by defining them in a declarative language",
"version": "0.6.4",
"project_urls": {
"documentation": "https://palimpzest.org",
"homepage": "https://palimpzest.org",
"repository": "https://github.com/mitdbg/palimpzest/"
},
"split_keywords": [
"relational",
" optimization",
" llm",
" ai programming",
" extraction",
" tools",
" document",
" search",
" integration"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "87d5d5b1a88b2f069152a57bf74bf11f6d54481ab18a76c3b6834936a32e0d61",
"md5": "86dc8e8943ccf5500ac1dc413efae617",
"sha256": "9e24836b11e28c4da6099e70c248ce9dc179b9526aea9adc3c21f74ba80b89be"
},
"downloads": -1,
"filename": "palimpzest-0.6.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "86dc8e8943ccf5500ac1dc413efae617",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 183482,
"upload_time": "2025-02-20T15:44:00",
"upload_time_iso_8601": "2025-02-20T15:44:00.477272Z",
"url": "https://files.pythonhosted.org/packages/87/d5/d5b1a88b2f069152a57bf74bf11f6d54481ab18a76c3b6834936a32e0d61/palimpzest-0.6.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "8b616987698b3ff6d417b5ff311335aac03e6b76764aa9a5b452cdf4711ac511",
"md5": "6b5f846cb30065579acbdbfde932c0ac",
"sha256": "312ca63d8076431c711874f643823d197eb42f2f1714c12c1d58c49afd0afedb"
},
"downloads": -1,
"filename": "palimpzest-0.6.4.tar.gz",
"has_sig": false,
"md5_digest": "6b5f846cb30065579acbdbfde932c0ac",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 149051,
"upload_time": "2025-02-20T15:44:02",
"upload_time_iso_8601": "2025-02-20T15:44:02.425357Z",
"url": "https://files.pythonhosted.org/packages/8b/61/6987698b3ff6d417b5ff311335aac03e6b76764aa9a5b452cdf4711ac511/palimpzest-0.6.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-02-20 15:44:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mitdbg",
"github_project": "palimpzest",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "palimpzest"
}
JSON
