# freshspark
Create **truly fresh** local Spark sessions with isolated temp dirs and reliable teardown.
- Isolates `spark.sql.warehouse.dir` and (optionally) embedded Derby **metastore** in unique temp dirs
- Defaults to **in-memory catalog** (no Derby) to avoid classic local locks
- Randomizes Spark UI port to avoid collisions and prints the UI URL
- Aggressively shuts down Py4J so the JVM actually exits
- Simple API: context manager, `(spark, cleanup)` pair, decorator, and a tiny CLI
## Install
```bash
pip install freshspark
```
## Quick start
```python
from freshspark import fresh_local_spark, get_fresh_local_spark
# Context manager (fresh per `with`)
with fresh_local_spark(app_name="etl", preset="dev") as spark:
spark.range(10).show()
# Manual lifecycle
spark, cleanup = get_fresh_local_spark(app_name="demo", preset="fat")
try:
spark.range(1000).summary().show()
finally:
cleanup()
```
## Friendly features
- **Presets**: `preset="tiny" | "dev" | "fat"` set sane memory defaults.
- **No Hive by default**: in-memory catalog avoids Derby locks. Enable with `hive_metastore=True` if you need it.
- **Clean UI**: UI port auto-randomized; prints the URL once up.
- **Optional reuse (same process)**: `reuse_within_process=True` to keep one isolated session for repeated calls.
- **Decorator**: run any function inside a fresh session:
```python
from freshspark import ensure_fresh
@ensure_fresh(preset="dev")
def job(input_path: str, *, spark):
return spark.read.csv(input_path, header=True).count()
print(job("data.csv"))
```
## CLI
```bash
# Open a REPL with `spark` ready:
freshspark repl --preset fat
# Stop any sticky active session in this process:
freshspark reset
```
## Jupyter tip
To be extra safe in notebooks:
```python
from freshspark import get_fresh_local_spark
spark, cleanup = get_fresh_local_spark(app_name="nb", preset="dev")
# ... work ...
# On finish (or in a finally cell):
cleanup()
```
## Why?
Local PySpark sessions can "stick"—leaving JVMs, metastore locks, or port clashes behind.
**freshspark** guarantees a clean slate every run.
## License
Apache 2.0
Raw data
{
"_id": null,
"home_page": null,
"name": "freshspark",
"maintainer": null,
"docs_url": null,
"requires_python": ">=3.8",
"maintainer_email": null,
"keywords": "pyspark, spark, spark-session, spark local, spark cleanup",
"author": "Odos Matthews",
"author_email": null,
"download_url": "https://files.pythonhosted.org/packages/e9/53/71bd7db40ddb5b2084a7d1c587f8d36b86d59f431da19ffdc9a2eabcde68/freshspark-0.1.0.tar.gz",
"platform": null,
"description": "# freshspark\n\nCreate **truly fresh** local Spark sessions with isolated temp dirs and reliable teardown.\n\n- Isolates `spark.sql.warehouse.dir` and (optionally) embedded Derby **metastore** in unique temp dirs\n- Defaults to **in-memory catalog** (no Derby) to avoid classic local locks\n- Randomizes Spark UI port to avoid collisions and prints the UI URL\n- Aggressively shuts down Py4J so the JVM actually exits\n- Simple API: context manager, `(spark, cleanup)` pair, decorator, and a tiny CLI\n\n## Install\n```bash\npip install freshspark\n```\n\n## Quick start\n```python\nfrom freshspark import fresh_local_spark, get_fresh_local_spark\n\n# Context manager (fresh per `with`)\nwith fresh_local_spark(app_name=\"etl\", preset=\"dev\") as spark:\n spark.range(10).show()\n\n# Manual lifecycle\nspark, cleanup = get_fresh_local_spark(app_name=\"demo\", preset=\"fat\")\ntry:\n spark.range(1000).summary().show()\nfinally:\n cleanup()\n```\n\n## Friendly features\n\n- **Presets**: `preset=\"tiny\" | \"dev\" | \"fat\"` set sane memory defaults.\n- **No Hive by default**: in-memory catalog avoids Derby locks. Enable with `hive_metastore=True` if you need it.\n- **Clean UI**: UI port auto-randomized; prints the URL once up.\n- **Optional reuse (same process)**: `reuse_within_process=True` to keep one isolated session for repeated calls.\n- **Decorator**: run any function inside a fresh session:\n\n```python\nfrom freshspark import ensure_fresh\n\n@ensure_fresh(preset=\"dev\")\ndef job(input_path: str, *, spark):\n return spark.read.csv(input_path, header=True).count()\n\nprint(job(\"data.csv\"))\n```\n\n## CLI\n\n```bash\n# Open a REPL with `spark` ready:\nfreshspark repl --preset fat\n\n# Stop any sticky active session in this process:\nfreshspark reset\n```\n\n## Jupyter tip\n\nTo be extra safe in notebooks:\n\n```python\nfrom freshspark import get_fresh_local_spark\n\nspark, cleanup = get_fresh_local_spark(app_name=\"nb\", preset=\"dev\")\n# ... work ...\n# On finish (or in a finally cell):\ncleanup()\n```\n\n## Why?\nLocal PySpark sessions can \"stick\"\u2014leaving JVMs, metastore locks, or port clashes behind.\n**freshspark** guarantees a clean slate every run.\n\n## License\nApache 2.0\n",
"bugtrack_url": null,
"license": null,
"summary": "Create truly fresh local Spark sessions with isolated temp dirs and reliable teardown.",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/eddiethedean/freshspark",
"Issues": "https://github.com/eddiethedean/freshspark/issues"
},
"split_keywords": [
"pyspark",
" spark",
" spark-session",
" spark local",
" spark cleanup"
],
"urls": [
{
"comment_text": null,
"digests": {
"blake2b_256": "5fe6e3b8d4e5c368e3ce9e544e3fa87251c774250702b0d1b21980748bfe9b3c",
"md5": "e10e650e984df6d3d7383afd26816d64",
"sha256": "edd2ca59420d7a07e793495d7165f17f0997a06a8da21c04a649c0ef3651ff7b"
},
"downloads": -1,
"filename": "freshspark-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "e10e650e984df6d3d7383afd26816d64",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.8",
"size": 8554,
"upload_time": "2025-08-11T15:28:09",
"upload_time_iso_8601": "2025-08-11T15:28:09.252329Z",
"url": "https://files.pythonhosted.org/packages/5f/e6/e3b8d4e5c368e3ce9e544e3fa87251c774250702b0d1b21980748bfe9b3c/freshspark-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": null,
"digests": {
"blake2b_256": "e95371bd7db40ddb5b2084a7d1c587f8d36b86d59f431da19ffdc9a2eabcde68",
"md5": "5bda8bc5a18f82869f6639f41ae1ac94",
"sha256": "e8ac23ac820c2d2e85c6f7a41a672d5bfacd9a1978fac22d2cca84d2a8d5f52d"
},
"downloads": -1,
"filename": "freshspark-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "5bda8bc5a18f82869f6639f41ae1ac94",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.8",
"size": 10170,
"upload_time": "2025-08-11T15:28:10",
"upload_time_iso_8601": "2025-08-11T15:28:10.247998Z",
"url": "https://files.pythonhosted.org/packages/e9/53/71bd7db40ddb5b2084a7d1c587f8d36b86d59f431da19ffdc9a2eabcde68/freshspark-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2025-08-11 15:28:10",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "eddiethedean",
"github_project": "freshspark",
"github_not_found": true,
"lcname": "freshspark"
}
JSON
