openget

Name	openget JSON
Version	0.3.1 JSON
	download
home_page	https://github.com/dytttf/openget
Summary	A Spider FrameWork
upload_time	2024-04-10 15:43:32
maintainer	None
docs_url	None
author	Dytttf
requires_python	None
license	BSD
keywords	openget spider batch-spider
VCS
bugtrack_url
requirements	No requirements were recorded.
Travis-CI	No Travis.
coveralls test coverage	No coveralls.

            # openget
A Spider FrameWork.

## Installation
You can install openget by simply doing:

    pip install openget
    pip install git+https://github.com/dytttf/openget.git@main
    pip install git+https://github.com/dytttf/openget.git@dev

    
## Usage
**Very Important Thing**
This line must be the first line in code
```
from openget.spiders import *
```
### 命令行支持
1. 生成配置文件样例
```shell
# 环境变量类型的配置文件
python -m openget --gen_env_example
```


## About Environment Variable
### 优先级
1. $(pwd)/.env
2. ~/.openget/.env


## docker build
```shell
cd docekr
bash build.sh
```

## TODO
- ftp download
- auto add task
- statis crawl speed
- browser support: zhipin.com tmall.com
- sqlite.db rename
- use cmd to create spider template
- 金融许可证 卡死问题处理
- 环境变量定义优化
- 新项目创建优化
- 极简模式、完整模式 减少包的依赖

            

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/dytttf/openget",
    "name": "openget",
    "maintainer": null,
    "docs_url": null,
    "requires_python": null,
    "maintainer_email": null,
    "keywords": "openget, spider, batch-spider",
    "author": "Dytttf",
    "author_email": "dytttf@foxmail.com",
    "download_url": "https://files.pythonhosted.org/packages/bd/14/bb868d16e080946f1af729b8feddecf22e8ba01ac7b5639caa93d72776dd/openget-0.3.1.tar.gz",
    "platform": null,
    "description": "# openget\nA Spider FrameWork.\n\n## Installation\nYou can install openget by simply doing:\n\n    pip install openget\n    pip install git+https://github.com/dytttf/openget.git@main\n    pip install git+https://github.com/dytttf/openget.git@dev\n\n    \n## Usage\n**Very Important Thing**\nThis line must be the first line in code\n```\nfrom openget.spiders import *\n```\n### \u547d\u4ee4\u884c\u652f\u6301\n1. \u751f\u6210\u914d\u7f6e\u6587\u4ef6\u6837\u4f8b\n```shell\n# \u73af\u5883\u53d8\u91cf\u7c7b\u578b\u7684\u914d\u7f6e\u6587\u4ef6\npython -m openget --gen_env_example\n```\n\n\n## About Environment Variable\n### \u4f18\u5148\u7ea7\n1. $(pwd)/.env\n2. ~/.openget/.env\n\n\n## docker build\n```shell\ncd docekr\nbash build.sh\n```\n\n## TODO\n- ftp download\n- auto add task\n- statis crawl speed\n- browser support: zhipin.com tmall.com\n- sqlite.db rename\n- use cmd to create spider template\n- \u91d1\u878d\u8bb8\u53ef\u8bc1 \u5361\u6b7b\u95ee\u9898\u5904\u7406\n- \u73af\u5883\u53d8\u91cf\u5b9a\u4e49\u4f18\u5316\n- \u65b0\u9879\u76ee\u521b\u5efa\u4f18\u5316\n- \u6781\u7b80\u6a21\u5f0f\u3001\u5b8c\u6574\u6a21\u5f0f \u51cf\u5c11\u5305\u7684\u4f9d\u8d56\n",
    "bugtrack_url": null,
    "license": "BSD",
    "summary": "A Spider FrameWork",
    "version": "0.3.1",
    "project_urls": {
        "Homepage": "https://github.com/dytttf/openget"
    },
    "split_keywords": [
        "openget",
        " spider",
        " batch-spider"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "0dbca653f50bb9f3b6801b6cc82c97f6449e8c87d6bb55efe7387ceb53da25e2",
                "md5": "83627f0d428e574fc3f84b88dd567e64",
                "sha256": "eaeb20cd11ce10a2178115251968dca171b026c23d54d160a3b9a9873ad3ea82"
            },
            "downloads": -1,
            "filename": "openget-0.3.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "83627f0d428e574fc3f84b88dd567e64",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 80868,
            "upload_time": "2024-04-10T15:43:28",
            "upload_time_iso_8601": "2024-04-10T15:43:28.890233Z",
            "url": "https://files.pythonhosted.org/packages/0d/bc/a653f50bb9f3b6801b6cc82c97f6449e8c87d6bb55efe7387ceb53da25e2/openget-0.3.1-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "bd14bb868d16e080946f1af729b8feddecf22e8ba01ac7b5639caa93d72776dd",
                "md5": "9d87f7f2bef4691061a2e488dfa3f0c0",
                "sha256": "48501ac563e99efec13bd8ab8dfec274e2ebd3106aa85c39b555a36aac006ded"
            },
            "downloads": -1,
            "filename": "openget-0.3.1.tar.gz",
            "has_sig": false,
            "md5_digest": "9d87f7f2bef4691061a2e488dfa3f0c0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 69859,
            "upload_time": "2024-04-10T15:43:32",
            "upload_time_iso_8601": "2024-04-10T15:43:32.376738Z",
            "url": "https://files.pythonhosted.org/packages/bd/14/bb868d16e080946f1af729b8feddecf22e8ba01ac7b5639caa93d72776dd/openget-0.3.1.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2024-04-10 15:43:32",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "codeberg": false,
    "github_user": "dytttf",
    "github_project": "openget",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "lcname": "openget"
}