openget

Name	openget JSON
Version	0.3.2 JSON
	download
home_page	https://github.com/dytttf/openget
Summary	A Spider FrameWork
upload_time	2024-07-31 09:38:22
maintainer	None
docs_url	None
author	Dytttf
requires_python	None
license	BSD
keywords	openget spider batch-spider
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # openget
A Spider FrameWork.

## Installation
You can install openget by simply doing:

    pip install openget
    pip install git+https://github.com/dytttf/openget.git@main
    pip install git+https://github.com/dytttf/openget.git@dev

    
## Usage
**Very Important Thing**
This line must be the first line in code
```
from openget.spiders import *
```
### 命令行支持
1. 生成配置文件样例
```shell
# 环境变量类型的配置文件
python -m openget --gen_env_example
```


## About Environment Variable
### 优先级
1. $(pwd)/.env
2. ~/.openget/.env


## docker build
```shell
cd docekr
bash build.sh
```

## TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/dytttf/openget",
    "name": "openget",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "openget, spider, batch-spider",
    "author": "Dytttf",
    "author_email": "dytttf@foxmail.com",
    "download_url": "https://files.pythonhosted.org/packages/cf/3d/b1b36530e2258a5892aa662e98c37011923591953661a5d2c8e7da361708/openget-0.3.2.tar.gz",
    "platform": null,
    "description": "# openget\r\nA Spider FrameWork.\r\n\r\n## Installation\r\nYou can install openget by simply doing:\r\n\r\n    pip install openget\r\n    pip install git+https://github.com/dytttf/openget.git@main\r\n    pip install git+https://github.com/dytttf/openget.git@dev\r\n\r\n    \r\n## Usage\r\n**Very Important Thing**\r\nThis line must be the first line in code\r\n```\r\nfrom openget.spiders import *\r\n```\r\n### \u547d\u4ee4\u884c\u652f\u6301\r\n1. \u751f\u6210\u914d\u7f6e\u6587\u4ef6\u6837\u4f8b\r\n```shell\r\n# \u73af\u5883\u53d8\u91cf\u7c7b\u578b\u7684\u914d\u7f6e\u6587\u4ef6\r\npython -m openget --gen_env_example\r\n```\r\n\r\n\r\n## About Environment Variable\r\n### \u4f18\u5148\u7ea7\r\n1. $(pwd)/.env\r\n2. ~/.openget/.env\r\n\r\n\r\n## docker build\r\n```shell\r\ncd docekr\r\nbash build.sh\r\n```\r\n\r\n## TODO\r\n- ftp download\r\n- auto add task\r\n- statis crawl speed\r\n- browser support: zhipin.com tmall.com\r\n- sqlite.db rename\r\n- use cmd to create spider template\r\n- \u91d1\u878d\u8bb8\u53ef\u8bc1 \u5361\u6b7b\u95ee\u9898\u5904\u7406\r\n- \u73af\u5883\u53d8\u91cf\u5b9a\u4e49\u4f18\u5316\r\n- \u65b0\u9879\u76ee\u521b\u5efa\u4f18\u5316\r\n- \u6781\u7b80\u6a21\u5f0f\u3001\u5b8c\u6574\u6a21\u5f0f \u51cf\u5c11\u5305\u7684\u4f9d\u8d56\r\n",
    "bugtrack_url": null,
    "license": "BSD",
    "summary": "A Spider FrameWork",
    "version": "0.3.2",
    "project_urls": {
        "Homepage": "https://github.com/dytttf/openget"
    },
    "split_keywords": [
        "openget",
        " spider",
        " batch-spider"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "7a903ceb30a8c986b3e8e547261d385cf3944e636cfdd7b49dd02576b3b4ac31",
                "md5": "4cac2673906a631f7b3997c51fa88303",
                "sha256": "6528b1910da204be4d25584019c2828c39504c546766c66b486597b82982f9f1"
            },
            "downloads": -1,
            "filename": "openget-0.3.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4cac2673906a631f7b3997c51fa88303",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 81392,
            "upload_time": "2024-07-31T09:38:20",
            "upload_time_iso_8601": "2024-07-31T09:38:20.598007Z",
            "url": "https://files.pythonhosted.org/packages/7a/90/3ceb30a8c986b3e8e547261d385cf3944e636cfdd7b49dd02576b3b4ac31/openget-0.3.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "cf3db1b36530e2258a5892aa662e98c37011923591953661a5d2c8e7da361708",
                "md5": "3efcd6da820e56be7879d5dda06802a8",
                "sha256": "f5983ca0938e82c664b0a4d8879773f68df6cba6628f8f56ec8f6f3ad92b50f7"
            },
            "downloads": -1,
            "filename": "openget-0.3.2.tar.gz",
            "has_sig": false,
            "md5_digest": "3efcd6da820e56be7879d5dda06802a8",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 72100,
            "upload_time": "2024-07-31T09:38:22",
            "upload_time_iso_8601": "2024-07-31T09:38:22.252337Z",
            "url": "https://files.pythonhosted.org/packages/cf/3d/b1b36530e2258a5892aa662e98c37011923591953661a5d2c8e7da361708/openget-0.3.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-07-31 09:38:22",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "dytttf",
    "github_project": "openget",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "openget"
}