openget

Name	openget JSON
Version	0.3.3 JSON
	download
home_page	https://github.com/dytttf/openget
Summary	A Spider FrameWork
upload_time	2024-11-15 05:55:14
maintainer	None
docs_url	None
author	Dytttf
requires_python	None
license	BSD
keywords	openget spider batch-spider
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # openget
A Spider FrameWork.

## Installation
You can install openget by simply doing:

    pip install openget
    pip install git+https://github.com/dytttf/openget.git@main
    pip install git+https://github.com/dytttf/openget.git@dev

    
## Usage
**Very Important Thing**
This line must be the first line in code
```
from openget.spiders import *
```
### 命令行支持
1. 生成配置文件样例
```shell
# 环境变量类型的配置文件
python -m openget --gen_env_example
```


## About Environment Variable
### 优先级
1. $(pwd)/.env
2. ~/.openget/.env


## docker build
```shell
cd docekr
bash build.sh
```

## TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/dytttf/openget",
    "name": "openget",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "openget, spider, batch-spider",
    "author": "Dytttf",
    "author_email": "dytttf@foxmail.com",
    "download_url": "https://files.pythonhosted.org/packages/09/b4/8572890a12c818969b13932368452126f51f613b86ba4493d48028a6cee9/openget-0.3.3.tar.gz",
    "platform": null,
    "description": "# openget\r\nA Spider FrameWork.\r\n\r\n## Installation\r\nYou can install openget by simply doing:\r\n\r\n    pip install openget\r\n    pip install git+https://github.com/dytttf/openget.git@main\r\n    pip install git+https://github.com/dytttf/openget.git@dev\r\n\r\n    \r\n## Usage\r\n**Very Important Thing**\r\nThis line must be the first line in code\r\n```\r\nfrom openget.spiders import *\r\n```\r\n### \u547d\u4ee4\u884c\u652f\u6301\r\n1. \u751f\u6210\u914d\u7f6e\u6587\u4ef6\u6837\u4f8b\r\n```shell\r\n# \u73af\u5883\u53d8\u91cf\u7c7b\u578b\u7684\u914d\u7f6e\u6587\u4ef6\r\npython -m openget --gen_env_example\r\n```\r\n\r\n\r\n## About Environment Variable\r\n### \u4f18\u5148\u7ea7\r\n1. $(pwd)/.env\r\n2. ~/.openget/.env\r\n\r\n\r\n## docker build\r\n```shell\r\ncd docekr\r\nbash build.sh\r\n```\r\n\r\n## TODO\r\n- ftp download\r\n- auto add task\r\n- statis crawl speed\r\n- browser support: zhipin.com tmall.com\r\n- sqlite.db rename\r\n- use cmd to create spider template\r\n- \u91d1\u878d\u8bb8\u53ef\u8bc1 \u5361\u6b7b\u95ee\u9898\u5904\u7406\r\n- \u73af\u5883\u53d8\u91cf\u5b9a\u4e49\u4f18\u5316\r\n- \u65b0\u9879\u76ee\u521b\u5efa\u4f18\u5316\r\n- \u6781\u7b80\u6a21\u5f0f\u3001\u5b8c\u6574\u6a21\u5f0f \u51cf\u5c11\u5305\u7684\u4f9d\u8d56\r\n",
    "bugtrack_url": null,
    "license": "BSD",
    "summary": "A Spider FrameWork",
    "version": "0.3.3",
    "project_urls": {
        "Homepage": "https://github.com/dytttf/openget"
    },
    "split_keywords": [
        "openget",
        " spider",
        " batch-spider"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "a4742766c34d4065bf412cce8ae8354d3b02b3b1f5169f2f70c1ceeea2767170",
                "md5": "53415ba177830f5bab6893e6e2076a84",
                "sha256": "cb3c80a9590367d6a2f3135756bc9990d696f148ea16a9e3c2c7a0b8c341ae75"
            },
            "downloads": -1,
            "filename": "openget-0.3.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "53415ba177830f5bab6893e6e2076a84",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 81664,
            "upload_time": "2024-11-15T05:55:12",
            "upload_time_iso_8601": "2024-11-15T05:55:12.706965Z",
            "url": "https://files.pythonhosted.org/packages/a4/74/2766c34d4065bf412cce8ae8354d3b02b3b1f5169f2f70c1ceeea2767170/openget-0.3.3-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "09b48572890a12c818969b13932368452126f51f613b86ba4493d48028a6cee9",
                "md5": "a4f532375e6f799a84bbd69374866bcb",
                "sha256": "1bcf9f090d4d7d007e96ac1108332b931ceba10b4a0897b6a4c5b3638810b34e"
            },
            "downloads": -1,
            "filename": "openget-0.3.3.tar.gz",
            "has_sig": false,
            "md5_digest": "a4f532375e6f799a84bbd69374866bcb",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 72392,
            "upload_time": "2024-11-15T05:55:14",
            "upload_time_iso_8601": "2024-11-15T05:55:14.129126Z",
            "url": "https://files.pythonhosted.org/packages/09/b4/8572890a12c818969b13932368452126f51f613b86ba4493d48028a6cee9/openget-0.3.3.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-11-15 05:55:14",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "dytttf",
    "github_project": "openget",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "openget"
}