| Name | scramjet-framework-py JSON |
| Version |
0.10.1
JSON |
| download |
| home_page | |
| Summary | Scramjet is a simple reactive stream programming framework. |
| upload_time | 2023-10-24 07:08:24 |
| maintainer | |
| docs_url | None |
| author | Scramjet.org |
| requires_python | |
| license | |
| keywords |
python
streams
|
| VCS |
|
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
Scramjet in Python
==================
<p align="center">
<a><img src="https://img.shields.io/github/license/scramjetorg/framework-python?color=green&style=plastic" alt="GitHub license" /></a>
<a><img src="https://img.shields.io/github/v/tag/scramjetorg/framework-python?label=version&color=blue&style=plastic" alt="version" /></a>
<a><img src="https://static.pepy.tech/personalized-badge/scramjet-framework-py?period=total&units=none&left_color=purple&right_color=darkgreen&left_text=Downloads" alt="downloads" /></a>
<a><img src="https://img.shields.io/github/stars/scramjetorg/framework-python?color=pink&style=plastic" alt="GitHub stars" /></a>
<a href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=7F7V65C43EBMW">
<img src="https://img.shields.io/badge/Donate-PayPal-green.svg?color=yellow&style=plastic" alt="Donate" />
</a>
</p>
<p align="center">⭐ Star us on GitHub — it motivates us a lot! 🚀 </p>
<p align="center">
<img src="https://assets.scramjet.org/images/framework-logo-256.svg" width="420" alt="Scramjet Framework">
</p>
Scramjet is a simple reactive stream programming framework. The code is written
by chaining functions that transform the streamed data, including well known
map, filter and reduce.
The main advantage of Scramjet is running asynchronous operations on your data
streams concurrently. It allows you to perform the transformations both
synchronously and asynchronously by using the same API - so now you can "map"
your stream from whatever source and call any number of API's consecutively.
[Originally written](https://github.com/scramjetorg/scramjet) on top of node.js
object streams, Scramjet is now being ported into Python. This is what is
happening in this repository.
>_Tested with Python 3.8.10 and Ubuntu 20.04._
## Table of contents
- [Installation](#installation)
- [Quick start](#quick-start)
- [Usage](#usage)
- [Requesting features](#requesting-features)
- [Reporting bugs](#reporting-bugs)
- [Contributing](#contributing)
- [Development Setup](#development-setup)
## Installation
Scramjet Framework is available on PyPI, You can install it with simple pip command:
```bash
pip install scramjet-framework-py
```
## Quick start
Let's say we have a `fruits.csv` file like this:
```csv
orange,sweet,1
lemon,sour,2
pigface,salty,5
banana,sweet,3
cranberries,bitter,6
```
and we want to write the names of the sweet fruits to a separate file.
To do this, write an async function like this:
```python
from scramjet import streams
import asyncio
async def sweet_stream():
with open("fruits.csv") as file_in, open("sweet.txt", "w") as file_out:
await (
streams.Stream
.read_from(file_in)
.map(lambda line: line.split(','))
.filter(lambda record: record[1] == "sweet")
.map(lambda record: f"{record[0]}\n")
.write_to(file_out)
)
asyncio.run(sweet_stream())
```
output saved in sweet.txt:
```
orange
banana
```
and that's it!
## Usage
Basic building block of Scramjet is the `Stream` class. It reads input in
chunks, performs operations on these chunks and produces an iterable output
that can be collected and written somewhere.
**Creating a stream** is done using `read_from` class method. It accepts
any iterable or an object implementing .read() method as the input, and returns
a `Stream` instance.
**Transforming a stream:**
* `map` - transform each chunk in a stream using specified function.
* `filter` - keep only chunks for which specified function evaluates to `True`.
* `flatmap` - run specified function on each chunk, and return all of its results as separate chunks.
* `batch` - convert a stream of chunks into a stream of lists of chunks.
Each of these methods return the modified stream, so they can be chained like
this: `some_stream.map(...).filter(...).batch(...)`
**Collecting data** from the stream (asynchronous):
* `write_to` - write all resulting stream chunks into a target.
* `to_list` - return a list with all stream chunks.
* `reduce` - combine all chunks using specified function.
Examples :books:
--------
You can find more examples in [`hello_datastream.py`](./hello_datastream.py)
file. They don't require any additional dependencies, just the standard library,
so you can run them simply with:
```bash
python hello_datastream.py
```
## Requesting Features
Anything missing? Or maybe there is something which would make using Scramjet Framework much easier or efficient? Don't hesitate to fill up a [new feature request](https://github.com/scramjetorg/framework-python/issues/new)! We really appreciate all feedback.
## Reporting bugs
If you have found a bug, inconsistent or confusing behavior please fill up a [new bug report](https://github.com/scramjetorg/framework-python/issues/new).
## Contributing
You can contribute to this project by giving us feedback ([reporting bugs](#reporting-bugs) and [requesting features](#reporting-features)) and also by writing code yourself!
The easiest way is to [create a fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) of this repository and then [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) with all your changes. In most cases, you should branch from and target `main` branch.
Please refer to [Development Setup](#development-setup) section on how to setup this project.
## Development Setup
1. Install Python3 interpreter on your computer. Refer to [official docs](https://wiki.python.org/moin/BeginnersGuide/Download).
2. Install `git` version control system. Refer to [official docs](https://git-scm.com/downloads).
3. Clone this repository:
```bash
git clone git@github.com:scramjetorg/framework-python.git
```
4. Create and activate a virtualenv:
```bash
sudo apt install python3-virtualenv
virtualenv -p python3 venv
.venv/bin/activate
```
5. Check Python version:
```bash
$ python --version
Python 3.8.10
```
6. Install dependencies:
```bash
pip install -r dev-requirements.txt
```
7. Run test cases (with activated virtualenv):
```bash
pytest
```
> :bulb: **HINT:** add a filename if you want to limit which tests are run
8. If you want to enable detailed debug logging, set one of the following env variables:
```bash
PYFCA_DEBUG=1 # debug pyfca
DATASTREAM_DEBUG=1 # debug datastream
SCRAMJET_DEBUG=1 # debug both
```
Raw data
{
"_id": null,
"home_page": "",
"name": "scramjet-framework-py",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "python,streams",
"author": "Scramjet.org",
"author_email": "<info@scramjet.org>",
"download_url": "https://files.pythonhosted.org/packages/be/9c/5ce4ca1292b855611db03ceae707ff890d643c8693f71d49c34ec4f3865a/scramjet-framework-py-0.10.1.tar.gz",
"platform": null,
"description": "Scramjet in Python\n==================\n\n<p align=\"center\">\n <a><img src=\"https://img.shields.io/github/license/scramjetorg/framework-python?color=green&style=plastic\" alt=\"GitHub license\" /></a>\n <a><img src=\"https://img.shields.io/github/v/tag/scramjetorg/framework-python?label=version&color=blue&style=plastic\" alt=\"version\" /></a>\n <a><img src=\"https://static.pepy.tech/personalized-badge/scramjet-framework-py?period=total&units=none&left_color=purple&right_color=darkgreen&left_text=Downloads\" alt=\"downloads\" /></a> \n <a><img src=\"https://img.shields.io/github/stars/scramjetorg/framework-python?color=pink&style=plastic\" alt=\"GitHub stars\" /></a>\n <a href=\"https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=7F7V65C43EBMW\">\n <img src=\"https://img.shields.io/badge/Donate-PayPal-green.svg?color=yellow&style=plastic\" alt=\"Donate\" />\n </a>\n</p>\n<p align=\"center\">\u2b50 Star us on GitHub \u2014 it motivates us a lot! \ud83d\ude80 </p>\n<p align=\"center\">\n <img src=\"https://assets.scramjet.org/images/framework-logo-256.svg\" width=\"420\" alt=\"Scramjet Framework\">\n</p>\n\nScramjet is a simple reactive stream programming framework. The code is written\nby chaining functions that transform the streamed data, including well known\nmap, filter and reduce.\n\nThe main advantage of Scramjet is running asynchronous operations on your data\nstreams concurrently. It allows you to perform the transformations both\nsynchronously and asynchronously by using the same API - so now you can \"map\"\nyour stream from whatever source and call any number of API's consecutively.\n\n[Originally written](https://github.com/scramjetorg/scramjet) on top of node.js\nobject streams, Scramjet is now being ported into Python. This is what is\nhappening in this repository.\n\n>_Tested with Python 3.8.10 and Ubuntu 20.04._\n## Table of contents\n\n- [Installation](#installation)\n- [Quick start](#quick-start)\n- [Usage](#usage)\n- [Requesting features](#requesting-features)\n- [Reporting bugs](#reporting-bugs)\n- [Contributing](#contributing)\n- [Development Setup](#development-setup)\n\n## Installation\n\nScramjet Framework is available on PyPI, You can install it with simple pip command:\n\n```bash\npip install scramjet-framework-py\n```\n## Quick start\n\nLet's say we have a `fruits.csv` file like this:\n\n```csv\norange,sweet,1\nlemon,sour,2\npigface,salty,5\nbanana,sweet,3\ncranberries,bitter,6\n```\n\nand we want to write the names of the sweet fruits to a separate file.\nTo do this, write an async function like this:\n\n\n```python\n\nfrom scramjet import streams\nimport asyncio\n\n\nasync def sweet_stream():\n with open(\"fruits.csv\") as file_in, open(\"sweet.txt\", \"w\") as file_out:\n await (\n streams.Stream\n .read_from(file_in)\n .map(lambda line: line.split(','))\n .filter(lambda record: record[1] == \"sweet\")\n .map(lambda record: f\"{record[0]}\\n\")\n .write_to(file_out)\n )\n\nasyncio.run(sweet_stream())\n```\n\noutput saved in sweet.txt:\n\n```\norange\nbanana\n```\n\nand that's it!\n\n## Usage\n\nBasic building block of Scramjet is the `Stream` class. It reads input in\nchunks, performs operations on these chunks and produces an iterable output\nthat can be collected and written somewhere.\n\n**Creating a stream** is done using `read_from` class method. It accepts\nany iterable or an object implementing .read() method as the input, and returns\na `Stream` instance.\n\n**Transforming a stream:**\n\n* `map` - transform each chunk in a stream using specified function.\n* `filter` - keep only chunks for which specified function evaluates to `True`.\n* `flatmap` - run specified function on each chunk, and return all of its results as separate chunks.\n* `batch` - convert a stream of chunks into a stream of lists of chunks.\n\nEach of these methods return the modified stream, so they can be chained like\nthis: `some_stream.map(...).filter(...).batch(...)`\n\n**Collecting data** from the stream (asynchronous):\n\n* `write_to` - write all resulting stream chunks into a target.\n* `to_list` - return a list with all stream chunks.\n* `reduce` - combine all chunks using specified function.\n\n\nExamples :books:\n--------\n\nYou can find more examples in [`hello_datastream.py`](./hello_datastream.py)\nfile. They don't require any additional dependencies, just the standard library,\nso you can run them simply with:\n\n```bash\npython hello_datastream.py\n```\n\n## Requesting Features\n\nAnything missing? Or maybe there is something which would make using Scramjet Framework much easier or efficient? Don't hesitate to fill up a [new feature request](https://github.com/scramjetorg/framework-python/issues/new)! We really appreciate all feedback.\n\n## Reporting bugs\n\nIf you have found a bug, inconsistent or confusing behavior please fill up a [new bug report](https://github.com/scramjetorg/framework-python/issues/new).\n\n## Contributing\n\nYou can contribute to this project by giving us feedback ([reporting bugs](#reporting-bugs) and [requesting features](#reporting-features)) and also by writing code yourself!\n\nThe easiest way is to [create a fork](https://docs.github.com/en/get-started/quickstart/fork-a-repo) of this repository and then [create a pull request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) with all your changes. In most cases, you should branch from and target `main` branch.\n\nPlease refer to [Development Setup](#development-setup) section on how to setup this project.\n\n## Development Setup\n\n1. Install Python3 interpreter on your computer. Refer to [official docs](https://wiki.python.org/moin/BeginnersGuide/Download).\n\n2. Install `git` version control system. Refer to [official docs](https://git-scm.com/downloads).\n\n3. Clone this repository:\n\n```bash\ngit clone git@github.com:scramjetorg/framework-python.git\n```\n4. Create and activate a virtualenv:\n\n```bash\nsudo apt install python3-virtualenv\nvirtualenv -p python3 venv\n.venv/bin/activate\n```\n\n5. Check Python version:\n\n```bash\n$ python --version\nPython 3.8.10\n```\n\n6. Install dependencies:\n\n```bash\npip install -r dev-requirements.txt\n```\n\n7. Run test cases (with activated virtualenv):\n\n```bash\npytest\n```\n\n> :bulb: **HINT:** add a filename if you want to limit which tests are run\n\n\n8. If you want to enable detailed debug logging, set one of the following env variables:\n\n```bash\nPYFCA_DEBUG=1 # debug pyfca\nDATASTREAM_DEBUG=1 # debug datastream\nSCRAMJET_DEBUG=1 # debug both\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "Scramjet is a simple reactive stream programming framework.",
"version": "0.10.1",
"project_urls": null,
"split_keywords": [
"python",
"streams"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b17864b0c0d3934946f89bbc38331162fab0ca416829f3f7d757f536faaf5fc4",
"md5": "b6721291f6e9e4af49d9ee946ef9b76f",
"sha256": "ff50a2144dac63c0f2aafbd7c7ad293e3e9f4ac4aa60fc2a44f1d0efff2691d5"
},
"downloads": -1,
"filename": "scramjet_framework_py-0.10.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b6721291f6e9e4af49d9ee946ef9b76f",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 12085,
"upload_time": "2023-10-24T07:08:22",
"upload_time_iso_8601": "2023-10-24T07:08:22.484983Z",
"url": "https://files.pythonhosted.org/packages/b1/78/64b0c0d3934946f89bbc38331162fab0ca416829f3f7d757f536faaf5fc4/scramjet_framework_py-0.10.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "be9c5ce4ca1292b855611db03ceae707ff890d643c8693f71d49c34ec4f3865a",
"md5": "f31aec54a78dc3e358f7d15e10f75337",
"sha256": "1844c49f4dfb0072b8208d3ed73c2076ab827f31909af0ed68964b116c1a01d7"
},
"downloads": -1,
"filename": "scramjet-framework-py-0.10.1.tar.gz",
"has_sig": false,
"md5_digest": "f31aec54a78dc3e358f7d15e10f75337",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 26278,
"upload_time": "2023-10-24T07:08:24",
"upload_time_iso_8601": "2023-10-24T07:08:24.034421Z",
"url": "https://files.pythonhosted.org/packages/be/9c/5ce4ca1292b855611db03ceae707ff890d643c8693f71d49c34ec4f3865a/scramjet-framework-py-0.10.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-24 07:08:24",
"github": false,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"lcname": "scramjet-framework-py"
}
