# spider-nest
Tools to pull text data from github, slack, website, stackoverflow, etc.
## Components
### gitpump
Fetch github data, including repos, issues, discussions, trendings. Requires github account tokens.
### slackpump
Fetch slack messages of a workspace. Requires slack bot token.
### stackpump
Fetch questions/answers from stackoverflow by web parser
### web2markdown
Get web pages and parse text content to markdown content.
## From source code
```commandline
git clone git@github.com:yhmo/spider-nest.git
cd spider-nest
pip3 install -r ./requirements.txt
```
## Installation
```commandline
pip3 install spider-nest
```
## Requirement
python >= 3.7
## How to run examples
### To run the examples of gitpump, follow the steps:
1. create a github personal token by following [this guide](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
2. create a text file `token_github.txt` under the examples folder
3. put some github tokens into the `token_github.txt`, no delimiter, one line for each token
4. now you can run examples `example_github_xxx.py`
### To run the example of example_slack_xxx.py
1. create a slack bot, bind the bot to a workspace, get the token
2. create a text file `token_slack.txt` under the examples folder
3. put the bot token into the `token_slack.txt`
4. now you can run examples `example_slack_xxx.py`
### You can run other examples without specific setting
Raw data
{
"_id": null,
"home_page": "https://github.com/yhmo/spider-nest",
"name": "spider-nest",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "",
"keywords": "",
"author": "yihua.mo",
"author_email": "yihua.mo@zilliz.com",
"download_url": "https://files.pythonhosted.org/packages/d4/45/6c46bb2dee0d870e8524d3c6ecb71af81459bdb1cacee26d684ed5e17f67/spider-nest-0.2.0.tar.gz",
"platform": null,
"description": "# spider-nest\nTools to pull text data from github, slack, website, stackoverflow, etc.\n\n## Components\n### gitpump\nFetch github data, including repos, issues, discussions, trendings. Requires github account tokens.\n\n### slackpump\nFetch slack messages of a workspace. Requires slack bot token.\n\n### stackpump\nFetch questions/answers from stackoverflow by web parser\n\n### web2markdown\nGet web pages and parse text content to markdown content.\n\n## From source code\n```commandline\ngit clone git@github.com:yhmo/spider-nest.git\ncd spider-nest\npip3 install -r ./requirements.txt\n```\n\n## Installation\n```commandline\npip3 install spider-nest\n```\n\n## Requirement\npython >= 3.7\n\n## How to run examples\n### To run the examples of gitpump, follow the steps:\n1. create a github personal token by following [this guide](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)\n2. create a text file `token_github.txt` under the examples folder\n3. put some github tokens into the `token_github.txt`, no delimiter, one line for each token\n4. now you can run examples `example_github_xxx.py`\n\n### To run the example of example_slack_xxx.py\n1. create a slack bot, bind the bot to a workspace, get the token\n2. create a text file `token_slack.txt` under the examples folder\n3. put the bot token into the `token_slack.txt`\n4. now you can run examples `example_slack_xxx.py`\n\n### You can run other examples without specific setting\n\n\n",
"bugtrack_url": null,
"license": "GPL v3.0",
"summary": "Tools to pull text data from github, slack, website, stackoverflow, etc.",
"version": "0.2.0",
"project_urls": {
"Homepage": "https://github.com/yhmo/spider-nest"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "d4456c46bb2dee0d870e8524d3c6ecb71af81459bdb1cacee26d684ed5e17f67",
"md5": "3740e1004767b030fdb5b4a4fb07033b",
"sha256": "321fa675b652c678bb7140ea5e81893557f6ee92f4ac3dfc0d6ef5bf70bf8adc"
},
"downloads": -1,
"filename": "spider-nest-0.2.0.tar.gz",
"has_sig": false,
"md5_digest": "3740e1004767b030fdb5b4a4fb07033b",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 67331,
"upload_time": "2023-05-18T06:53:42",
"upload_time_iso_8601": "2023-05-18T06:53:42.497838Z",
"url": "https://files.pythonhosted.org/packages/d4/45/6c46bb2dee0d870e8524d3c6ecb71af81459bdb1cacee26d684ed5e17f67/spider-nest-0.2.0.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-05-18 06:53:42",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "yhmo",
"github_project": "spider-nest",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [],
"lcname": "spider-nest"
}
JSON
