<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
**Table of Contents**
- [Generic Streaming Spouts](#generic-streaming-spouts)
- [Usage](#usage)
- [Webhooks](#webhooks)
- [Kafka](#kafka)
- [Websockets](#websockets)
- [UDP](#udp)
- [QUIC](#quic)
- [REST API polling](#rest-api-polling)
<!-- END doctoc generated TOC please keep comment here to allow auto update -->
# Generic Streaming Spouts
This is a collection of generic Spouts and (micro) Batch spouts.
Includes:
| Name | Description | Output Type | Input Type |
| ------------ | ----------------------------------------------- | ----------- | ---------- |
| Webhook | Cherrypy server that accepts all HTTP API calls | Streaming | HTTP |
| Webhook | Cherrypy server that accepts all HTTP API calls | Batch | HTTP |
| Kafka | Kafka client that listens to a topic | Streaming | Kafka |
| Kafka | Kafka client that listens to a topic | Batch | Kafka |
| Websocket | Websocket client that listens to a socket | Streaming | TCP |
| Websocket | Websocket client that listens to a socket | Batch | TCP |
| UDP | UDP server that listens to a given port | Streaming | UDP |
| UDP | UDP server that listens to a given port | Batch | UDP |
| QUIC | Aioquic server that listens to a given port | Streaming | UDP |
| QUIC | Aioquic server that listens to a given port | Batch | UDP |
| HTTP Polling | HTTP client that keeps polling an API | Streaming | HTTP |
| HTTP Polling | HTTP client that keeps polling an API | Batch | HTTP |
All of the above spouts support both streaming and batch usecases.
## Usage
### Webhooks
Run the spout:
```bash
genius Webhook rise \
streaming \
--output_kafka_topic webhook_test \
--output_kafka_cluster_connection_string localhost:9094 \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise \
--postgres_table state \
listen \
--args port=3001
```
Test:
```bash
curl -X POST \
-H "Content-Type: application/json" \
-d '{"lol": "teeeestss"}' \
http://localhost:3001
```
### Kafka
```bash
genius Kafka rise \
streaming \
--output_kafka_topic kafka_test \
--output_kafka_cluster_connection_string localhost:9094 \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise \
--postgres_table state \
listen \
--args topic=kafka_test_input group_id=kafka_test bootstrap_servers=localhost:9094
```
### Websockets
```bash
genius Websocket rise \
streaming \
--output_kafka_topic websocket_test \
--output_kafka_cluster_connection_string localhost:9094 \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise \
--postgres_table state \
listen \
--args host=localhost port=3002
```
Test:
```bash
cargo install websocat
echo '{"lol": "heheheheheheheh"}' | websocat ws://localhost:3002
```
### UDP
```bash
genius Udp rise \
streaming \
--output_kafka_topic udp_test \
--output_kafka_cluster_connection_string localhost:9094 \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise \
--postgres_table state \
listen \
--args host=localhost port=3003
```
Test:
```bash
yay -S netcat
echo -n '{"key": "value"}' | nc -u -w1 localhost 3003
```
### QUIC
```bash
openssl genpkey -algorithm RSA -out ~/.ssh/quic_key.pem
openssl req -new -x509 -key ~/.ssh/quic_key.pem -out ~/.ssh/quic_cert.pem -days 365 -subj "/CN=localhost"
genius Quic rise \
streaming \
--output_kafka_topic udp_test \
--output_kafka_cluster_connection_string localhost:9094 \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise \
--postgres_table state \
listen \
--args cert_path=/home/ixaxaar/.ssh/quic_cert.pem key_path=/home/ixaxaar/.ssh/quic_key.pem host=localhost port=3004
```
Test:
```
go install github.com/spacewander/quick@latest
quick -insecure https://localhost:3004
```
### REST API polling
```bash
genius RESTAPIPoll rise \
streaming \
--output_kafka_topic poll_test \
--output_kafka_cluster_connection_string localhost:9094 \
postgres \
--postgres_host 127.0.0.1 \
--postgres_port 5432 \
--postgres_user postgres \
--postgres_password postgres \
--postgres_database geniusrise \
--postgres_table state \
listen \
--args url=https://reqres.in/api/users method=GET interval=6 body="" headers='{"content-type": "application/json"}' params='{"page": 2}'
```
Raw data
{
"_id": null,
"home_page": "https://github.com/ixaxaar/streaming-spouts",
"name": "genius-streaming-spouts",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.10",
"maintainer_email": "",
"keywords": "",
"author": "ixaxaar",
"author_email": "ixaxaar@geniusrise.ai",
"download_url": "https://files.pythonhosted.org/packages/0f/98/e8ef370ccd86b4ceecae16c1a605fb4e466f8225e74f04f6adea459dd2b6/genius_streaming_spouts-0.1.0.tar.gz",
"platform": null,
"description": "\n\n<!-- START doctoc generated TOC please keep comment here to allow auto update -->\n<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->\n\n**Table of Contents**\n\n- [Generic Streaming Spouts](#generic-streaming-spouts)\n - [Usage](#usage)\n - [Webhooks](#webhooks)\n - [Kafka](#kafka)\n - [Websockets](#websockets)\n - [UDP](#udp)\n - [QUIC](#quic)\n - [REST API polling](#rest-api-polling)\n\n<!-- END doctoc generated TOC please keep comment here to allow auto update -->\n\n# Generic Streaming Spouts\n\nThis is a collection of generic Spouts and (micro) Batch spouts.\n\nIncludes:\n\n| Name | Description | Output Type | Input Type |\n| ------------ | ----------------------------------------------- | ----------- | ---------- |\n| Webhook | Cherrypy server that accepts all HTTP API calls | Streaming | HTTP |\n| Webhook | Cherrypy server that accepts all HTTP API calls | Batch | HTTP |\n| Kafka | Kafka client that listens to a topic | Streaming | Kafka |\n| Kafka | Kafka client that listens to a topic | Batch | Kafka |\n| Websocket | Websocket client that listens to a socket | Streaming | TCP |\n| Websocket | Websocket client that listens to a socket | Batch | TCP |\n| UDP | UDP server that listens to a given port | Streaming | UDP |\n| UDP | UDP server that listens to a given port | Batch | UDP |\n| QUIC | Aioquic server that listens to a given port | Streaming | UDP |\n| QUIC | Aioquic server that listens to a given port | Batch | UDP |\n| HTTP Polling | HTTP client that keeps polling an API | Streaming | HTTP |\n| HTTP Polling | HTTP client that keeps polling an API | Batch | HTTP |\n\nAll of the above spouts support both streaming and batch usecases.\n\n## Usage\n\n### Webhooks\n\nRun the spout:\n\n```bash\ngenius Webhook rise \\\n streaming \\\n --output_kafka_topic webhook_test \\\n --output_kafka_cluster_connection_string localhost:9094 \\\n postgres \\\n --postgres_host 127.0.0.1 \\\n --postgres_port 5432 \\\n --postgres_user postgres \\\n --postgres_password postgres \\\n --postgres_database geniusrise \\\n --postgres_table state \\\n listen \\\n --args port=3001\n```\n\nTest:\n\n```bash\ncurl -X POST \\\n -H \"Content-Type: application/json\" \\\n -d '{\"lol\": \"teeeestss\"}' \\\n http://localhost:3001\n```\n\n### Kafka\n\n```bash\ngenius Kafka rise \\\n streaming \\\n --output_kafka_topic kafka_test \\\n --output_kafka_cluster_connection_string localhost:9094 \\\n postgres \\\n --postgres_host 127.0.0.1 \\\n --postgres_port 5432 \\\n --postgres_user postgres \\\n --postgres_password postgres \\\n --postgres_database geniusrise \\\n --postgres_table state \\\n listen \\\n --args topic=kafka_test_input group_id=kafka_test bootstrap_servers=localhost:9094\n```\n\n### Websockets\n\n```bash\ngenius Websocket rise \\\n streaming \\\n --output_kafka_topic websocket_test \\\n --output_kafka_cluster_connection_string localhost:9094 \\\n postgres \\\n --postgres_host 127.0.0.1 \\\n --postgres_port 5432 \\\n --postgres_user postgres \\\n --postgres_password postgres \\\n --postgres_database geniusrise \\\n --postgres_table state \\\n listen \\\n --args host=localhost port=3002\n```\n\nTest:\n\n```bash\ncargo install websocat\necho '{\"lol\": \"heheheheheheheh\"}' | websocat ws://localhost:3002\n```\n\n### UDP\n\n```bash\ngenius Udp rise \\\n streaming \\\n --output_kafka_topic udp_test \\\n --output_kafka_cluster_connection_string localhost:9094 \\\n postgres \\\n --postgres_host 127.0.0.1 \\\n --postgres_port 5432 \\\n --postgres_user postgres \\\n --postgres_password postgres \\\n --postgres_database geniusrise \\\n --postgres_table state \\\n listen \\\n --args host=localhost port=3003\n```\n\nTest:\n\n```bash\nyay -S netcat\necho -n '{\"key\": \"value\"}' | nc -u -w1 localhost 3003\n```\n\n### QUIC\n\n```bash\nopenssl genpkey -algorithm RSA -out ~/.ssh/quic_key.pem\nopenssl req -new -x509 -key ~/.ssh/quic_key.pem -out ~/.ssh/quic_cert.pem -days 365 -subj \"/CN=localhost\"\n\ngenius Quic rise \\\n streaming \\\n --output_kafka_topic udp_test \\\n --output_kafka_cluster_connection_string localhost:9094 \\\n postgres \\\n --postgres_host 127.0.0.1 \\\n --postgres_port 5432 \\\n --postgres_user postgres \\\n --postgres_password postgres \\\n --postgres_database geniusrise \\\n --postgres_table state \\\n listen \\\n --args cert_path=/home/ixaxaar/.ssh/quic_cert.pem key_path=/home/ixaxaar/.ssh/quic_key.pem host=localhost port=3004\n```\n\nTest:\n\n```\ngo install github.com/spacewander/quick@latest\n\nquick -insecure https://localhost:3004\n```\n\n### REST API polling\n\n```bash\ngenius RESTAPIPoll rise \\\n streaming \\\n --output_kafka_topic poll_test \\\n --output_kafka_cluster_connection_string localhost:9094 \\\n postgres \\\n --postgres_host 127.0.0.1 \\\n --postgres_port 5432 \\\n --postgres_user postgres \\\n --postgres_password postgres \\\n --postgres_database geniusrise \\\n --postgres_table state \\\n listen \\\n --args url=https://reqres.in/api/users method=GET interval=6 body=\"\" headers='{\"content-type\": \"application/json\"}' params='{\"page\": 2}'\n```\n",
"bugtrack_url": null,
"license": "",
"summary": "Streaming spouts for geniusrise",
"version": "0.1.0",
"project_urls": {
"Homepage": "https://github.com/ixaxaar/streaming-spouts"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "8129976fe3417a497225f98672ba61d0c92efc1b24ba991133ef6ff742793016",
"md5": "4a3c9fb8cf17fc0702eb7a17c4a24ad5",
"sha256": "d90ce95252413262f1b87cf81fc2a557ca7feed0db57066d2f7cf0d628f6b86f"
},
"downloads": -1,
"filename": "genius_streaming_spouts-0.1.0-py3-none-any.whl",
"has_sig": false,
"md5_digest": "4a3c9fb8cf17fc0702eb7a17c4a24ad5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.10",
"size": 18248,
"upload_time": "2023-08-20T15:58:28",
"upload_time_iso_8601": "2023-08-20T15:58:28.325239Z",
"url": "https://files.pythonhosted.org/packages/81/29/976fe3417a497225f98672ba61d0c92efc1b24ba991133ef6ff742793016/genius_streaming_spouts-0.1.0-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "0f98e8ef370ccd86b4ceecae16c1a605fb4e466f8225e74f04f6adea459dd2b6",
"md5": "f9597e3bc0c65d6fb562f37b2a81e01e",
"sha256": "cc92538ce79e3ab0d5b4f87a7edb93d4cb4825d689ce3768c9c0efc09fb985d8"
},
"downloads": -1,
"filename": "genius_streaming_spouts-0.1.0.tar.gz",
"has_sig": false,
"md5_digest": "f9597e3bc0c65d6fb562f37b2a81e01e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.10",
"size": 8841,
"upload_time": "2023-08-20T15:58:29",
"upload_time_iso_8601": "2023-08-20T15:58:29.529713Z",
"url": "https://files.pythonhosted.org/packages/0f/98/e8ef370ccd86b4ceecae16c1a605fb4e466f8225e74f04f6adea459dd2b6/genius_streaming_spouts-0.1.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-08-20 15:58:29",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "ixaxaar",
"github_project": "streaming-spouts",
"github_not_found": true,
"lcname": "genius-streaming-spouts"
}
JSON
