# Spider Admin Pro
[](https://pypi.org/project/spider-admin-pro)
[](https://pypi.org/project/spider-admin-pro)
[](https://pypi.org/project/spider-admin-pro)
[](https://hub.docker.com/r/mouday/spider-admin-pro)
[](https://app.travis-ci.com/mouday/spider-admin-pro)
[](https://app.travis-ci.com/mouday/spider-admin-pro)
[](https://github.com/mouday/spider-admin-pro/blob/master/LICENSE)
## 简介
Spider Admin Pro 是[Spider Admin](https://github.com/mouday/SpiderAdmin)的升级版，一个可视化的Scrapy爬虫管理平台，依赖于Scrapyd
- Github: [https://github.com/mouday/spider-admin-pro](https://github.com/mouday/spider-admin-pro)
- Gitee: [https://gitee.com/mouday/spider-admin-pro](https://gitee.com/mouday/spider-admin-pro)
- Pypi: [https://pypi.org/project/spider-admin-pro](https://pypi.org/project/spider-admin-pro)
- Docker: [https://hub.docker.com/r/mouday/spider-admin-pro](https://hub.docker.com/r/mouday/spider-admin-pro)
- releases: [https://github.com/mouday/spider-admin-pro/releases](https://github.com/mouday/spider-admin-pro/releases)
## v3.0 和 v2.0
> 注意：v3.0 和 v2.0版本不兼容，无法平滑迁移
- v3.0 版本支持多Scrapyd
- v2.0 版本仅支持一个Scrapyd
- [Spider Admin Pro V2版本文档看这里](README-v2.md)
- [Spider Admin Pro V1版本文档看这里](README-v1.md)
## 安装启动
本项目基于Python3.7.0 开发，所以推荐使用Python3.7.0及其以上版本
运行项目前，请先确保[scrapyd](https://pengshiyu.blog.csdn.net/article/details/79842514)服务已经启动
方式一：
```bash
$ python3 --version
Python 3.7.0
# 创建名为 venv 的虚拟环境
$ python3 -m venv venv
# 激活虚拟环境
$ source venv/bin/activate
# 安装spider-admin-pro
$ pip3 install spider-admin-pro
# 可选
$ pip3 install -U spider-admin-pro -i https://pypi.org/simple
# Linux macOS 运行启动
$ gunicorn 'spider_admin_pro:app'
# windows 环境使用waitress 替换 gunicorn
$ pip install waitress
$ waitress-serve --listen=127.0.0.1:8000 'spider_admin_pro:app'
```
方式二：
```bash
$ git clone https://github.com/mouday/spider-admin-pro.git
$ cd spider-admin-pro
# 安装依赖（建议：最好新建一个虚拟环境）
$ pip3 install -r requirements.txt
# Linux/macOS 以开发模式运行
$ make dev
# windows 以开发模式运行
$ python3 dev.py
# 以生产模式运行
$ make pro
```
安装 scrapy 全家桶`[可选]`
```bash
pip install scrapy scrapyd scrapyd-client
```
方式三：
```bash
vim config.yaml # 配置文件文件内容见⬇️：配置参数
docker run -e TZ=Asia/Shanghai -p 8000:8000 -v ./config.yml:/app/config.yml mouday/spider-admin-pro
```
## 配置参数
自定义配置
在运行目录下新建`config.yml` 文件，也就是执行启动命令的目录，运行时会自动读取该配置文件
例如
```bash
$ ls
config.yml
$ gunicorn 'spider_admin_pro:app'
```
> 强烈建议：修改密码和秘钥项
eg:
```yaml
# 登录账号密码
USERNAME: admin
PASSWORD: "123456"
# 【可选】是否开开启前端静态文件预压缩以提升页面加载速度，默认为True
COMPRESS_STATIC: True
# 以下配置在v3.0不支持
# scrapyd地址, 结尾不要加斜杆
# SCRAPYD_SERVER: "http://127.0.0.1:6800"
# 【可选】支持 basic auth @since 2.0.8
# SCRAPYD_USERNAME: ''
# SCRAPYD_PASSWORD: ''
```
## 使用扩展
收集运行日志：[scrapy-util](https://github.com/mouday/scrapy-util) 可以帮助你收集到程序运行的统计数据
## 技术栈：
1、前端技术：
| 功能 | 第三方库及文档 |
| - | - |
| 基本框架 | [vue2.js](https://cn.vuejs.org/) |
| 仪表盘图表 | [echarts](https://echarts.apache.org/) |
| 网络请求 | [axios](https://www.npmjs.com/package/axios) |
| 界面样式 | [Element-UI](https://element.eleme.cn/) |
2、后端技术
| 功能 | 第三方库及文档 |
| - | - |
| 接口服务 | [Flask](https://dormousehole.readthedocs.io/) |
| 任务调度 | [apscheduler](https://apscheduler.readthedocs.io/) |
| scrapyd接口 | [scrapyd-api](https://github.com/mouday/scrapyd-api) |
| 网络请求 | [session-request](https://github.com/mouday/session-request) |
| ORM | [peewee](http://docs.peewee-orm.com/) |
| jwt | [jwt](https://pyjwt.readthedocs.io/) |
| 系统信息 | [psutil](https://psutil.readthedocs.io/) |
## 项目结构
【公开仓库】基于Flask的后端项目spider-admin-pro: [https://github.com/mouday/spider-admin-pro](https://github.com/mouday/spider-admin-pro)
【私有仓库】基于Vue的前端项目spider-admin-pro-web: 获取方式往下看
> 备注：前端Vue项目，可入QQ群发送github用户名获取权限
获取前端源码的几个方式：
- 方式一：提供github的用户名
- 方式二：先赞助项目￥10，再提供github的用户名
- 方式三：关注微信公众号：
<img src="https://gitee.com/mouday/domain-admin/raw/master/image/coding-big-tree.jpg" width="300">
回复：`spider-admin-web`，获取完整的前端代码
回复：`spider-admin`，可加入Python技术交流群，和更多开发者学习交流遇到的问题
spider-admin-pro项目主要目录结构：
```bash
.
├── run.py # 程序入口
├── api # Controller层
├── service # Sevice层
├── model # Model层
├── exceptions # 异常
├── utils # 工具类
└── web # 静态web页
```
## 经验总结
Scrapyd 不能直接暴露在外网
1. 其他人通过deploy部署可以将代码部署到你的机器上，如果是root用户运行，还会在你机器上做其他的事情
2. 还有运行日志中会出现配置文件中的信息，存在信息泄露的危险
## TODO
~~1. 补全开发文档~~
~~2. 支持命令行安装可用~~
~~3. 优化代码布局，提取公共库~~
~~4. 日志自动刷新~~
~~5. scrapy项目数据收集~~
[ok]6. 定时任务spider列左对齐，支持本地排序
[x]7. 调度器控制移除停止开启开关，只保留暂停继续
[x]8. 添加任务，默认项目名，关闭弹框取消form校验结果
[x]9. 统计的日志量太大，增加一个一个定时清理的功能
[x]10. 定时任务备份，不小心把任务清空
[x]11. 希望能加入更好的定时方式,类似 scrapyd_web那种定时
[x]12. 简单的爬虫不用非要去打包，比如我自己上传一个py文件，可以定时任务，脚本的方式运行
[x]13. 爬虫能配置带参数运行
## 交流沟通
关注本项目的小伙伴越来越多，为了更好地交流沟通，可以加入群聊
- 一群: 1074075691（已满）
- 二群: 864983297
问题：邀请码 答案：SpiderAdmin
<img src="https://github.com/mouday/spider-admin-pro/raw/master/doc/img/qq-2.jpg" width="300"/>
## 联系作者
微信扫码二维码
<img src="https://raw.githubusercontent.com/mouday/domain-admin/master/image/coding-big-tree.jpg" width="300">
扫描二维码后，回复：Python，即可进入Python技术交流群，和技术大佬们学习交流
## 项目赞助
| 日期 | 姓名 | 金额 |
|------------| - | - |
| 2022-04-16 | [@realhellosunsun](https://github.com/realhellosunsun) | ￥188.00
| 2022-08-30 | [@yangxiaozhe13](https://github.com/yangxiaozhe13) | ￥88.00
| 2022-09-01 | [@robot-2233](https://github.com/robot-2233) | ￥88.00
| 2023-05-09 | 埃菲尔没有塔尖 | ￥68.80
| 2023-09-21 | [@burujiuzheyang](https://github.com/burujiuzheyang) | ￥50.00
| 2023-10-07 | [@Lnine9](https://github.com/Lnine9) | ￥20.00
| 2023-10-09 | [@xiaoran-xr](https://github.com/xiaoran-xr) | ￥20.00
| 2023-10-10 | [@hsdanbai](https://github.com/hsdanbai) | ￥20.00
| 2023-10-19 | [@shuiniu86](https://github.com/shuiniu86) | ￥50.00
| 2023-10-24 | [@yuzhou6](https://github.com/yuzhou6) | ￥50.00
| 2023-11-13 | [@xuedipiaofei](https://github.com/xuedipiaofei) | ￥50.00
| 2024-01-06 | [@if-always](https://github.com/if-always) | ￥10.00
| 2024-01-21 | [@dydwgmcnl4241](https://github.com/dydwgmcnl4241) | ￥10.00
| 2024-02-18 | [@zhaolipo](https://github.com/zhaolipo) | ￥10.00
| 2024-02-19 | [@qianwangali](https://github.com/qianwangali) | ￥10.00
| 2024-02-20 | 没事干的小伙子 | ￥8.88
| 2024-02-20 | [@ning-0217](https://github.com/ning-0217) | ￥50.00
| 2024-03-08 | [@zhaolipo](https://github.com/zhaolipo) | ￥20.00
| 2024-03-18 | [@magiceric](https://github.com/magiceric) | ￥88.00
| 2024-09-30 | [@dream2333](https://github.com/dream2333) | ￥100.00
<img src="https://github.com/mouday/spider-admin-pro/raw/master/doc/img/alipay.jpg" width="300">
## 项目截图
## 安装升级
```
pip3 install -U spider-admin-pro -i https://pypi.org/simple
```
## Stargazers over time
[](https://starchart.cc/mouday/spider-admin-pro)
## 其他问题
1、windows系统 scrapyd 启动失败，可能缺少依赖pywin32
```
pip install pywin32
```
感谢[@whobywind](https://github.com/whobywind)，提供的解决方案
2、网站有ip校验，刚访问几个请求就被禁止访问？
同一个ip可能有被封的风险，可以使用代理ip去请求，有免费和付费。
如果是个人使用，可以找一些免费的ip临时使用
如果是企业项目，可以使用付费代理ip
某爬虫大佬也推荐过一个不错的动态代理 [云立方](http://www.yunlifang.cn?u=mouday)
<a href="http://www.yunlifang.cn?u=mouday" target="_blank" style="display: inline-block; background-color: #000;">
<img src="https://www.yunlifang.cn/user/img/720X90.png">
</a>
找客服发送暗号：【爬虫推广】可以获取打折优惠
具体搭建方法在大佬的博客中有详尽说明：
[使用 Tornado+Redis 维护 ADSL 拨号服务器代理池](https://cuiqingcai.com/4596.html)
如果有问题，可以加QQ群，群里的小伙伴会积极解答喔
3、为什么外网访问不到，如何修改端口号
增加`--bind` 参数
格式
```bash
--bind 监听地址:监听端口号
```
例如
```bash
# 启动运行
$ gunicorn 'spider_admin_pro:app'
# 支持外网可访问，云服务器（阿里云或腾讯云）需要设置安全组
# 默认内网访问 --bind 127.0.0.1:8000
$ gunicorn --bind '0.0.0.0:8000' 'spider_admin_pro:app'
```
更多设置，可参考[gunicorn](https://docs.gunicorn.org/en/stable/index.html)
4、提示缺少libfile
群友 `@Yuan、红尘美` 提供的解决方法
安装依赖
```bash
yum install libffi-devel -y
```
## 更新日志
[CHANGELOG.md](./CHANGELOG.md)
## 社区其他优秀工具推荐
- https://github.com/DormyMo/SpiderKeeper
- https://github.com/my8100/scrapydweb
- https://github.com/ouqiang/gocron 使用Go语言开发的轻量级定时任务集中调度和管理系统, 用于替代Linux-crontab
## Spider Admin Pro vs. Spider Admin
1. 简化了一些功能；
2. 优化了前端界面，基于Vue的组件化开发；
3. 优化了后端接口，对后端项目进行了目录划分；
4. 整体代码利于升级维护。
5. 目前仅对Python3进行了支持
6. 路由统一管理
7. 全局异常捕获
8. 接口统一返回
9. 前后端分离
10. 可视化参数配置
Raw data
{
"_id": null,
"home_page": "https://github.com/mouday/spider-admin-pro",
"name": "spider-admin-pro",
"maintainer": null,
"docs_url": null,
"requires_python": null,
"maintainer_email": null,
"keywords": "spider admin scrapy scrapyd scheduler",
"author": "Peng Shiyu",
"author_email": "pengshiyuyx@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/3f/98/c06092f83049c4aee13bb58318b54317783a0bc3bfd4307cb927e3e45832/spider_admin_pro-3.0.9.tar.gz",
"platform": null,
"description": "# Spider Admin Pro\n\n[](https://pypi.org/project/spider-admin-pro)\n[](https://pypi.org/project/spider-admin-pro)\n[](https://pypi.org/project/spider-admin-pro)\n[](https://hub.docker.com/r/mouday/spider-admin-pro)\n[](https://app.travis-ci.com/mouday/spider-admin-pro)\n[](https://app.travis-ci.com/mouday/spider-admin-pro)\n[](https://github.com/mouday/spider-admin-pro/blob/master/LICENSE)\n\n\n\n## \u7b80\u4ecb\n\nSpider Admin Pro \u662f[Spider Admin](https://github.com/mouday/SpiderAdmin)\u7684\u5347\u7ea7\u7248\uff0c\u4e00\u4e2a\u53ef\u89c6\u5316\u7684Scrapy\u722c\u866b\u7ba1\u7406\u5e73\u53f0\uff0c\u4f9d\u8d56\u4e8eScrapyd\n\n- Github: [https://github.com/mouday/spider-admin-pro](https://github.com/mouday/spider-admin-pro)\n- Gitee: [https://gitee.com/mouday/spider-admin-pro](https://gitee.com/mouday/spider-admin-pro)\n\n- Pypi: [https://pypi.org/project/spider-admin-pro](https://pypi.org/project/spider-admin-pro)\n- Docker: [https://hub.docker.com/r/mouday/spider-admin-pro](https://hub.docker.com/r/mouday/spider-admin-pro)\n- releases: [https://github.com/mouday/spider-admin-pro/releases](https://github.com/mouday/spider-admin-pro/releases)\n\n\n\n## v3.0 \u548c v2.0\n\n> \u6ce8\u610f\uff1av3.0 \u548c v2.0\u7248\u672c\u4e0d\u517c\u5bb9\uff0c\u65e0\u6cd5\u5e73\u6ed1\u8fc1\u79fb\n\n- v3.0 \u7248\u672c\u652f\u6301\u591aScrapyd\n- v2.0 \u7248\u672c\u4ec5\u652f\u6301\u4e00\u4e2aScrapyd\n\n- [Spider Admin Pro V2\u7248\u672c\u6587\u6863\u770b\u8fd9\u91cc](README-v2.md)\n- [Spider Admin Pro V1\u7248\u672c\u6587\u6863\u770b\u8fd9\u91cc](README-v1.md)\n\n## \u5b89\u88c5\u542f\u52a8\n\n\u672c\u9879\u76ee\u57fa\u4e8ePython3.7.0 \u5f00\u53d1\uff0c\u6240\u4ee5\u63a8\u8350\u4f7f\u7528Python3.7.0\u53ca\u5176\u4ee5\u4e0a\u7248\u672c\n\n\u8fd0\u884c\u9879\u76ee\u524d\uff0c\u8bf7\u5148\u786e\u4fdd[scrapyd](https://pengshiyu.blog.csdn.net/article/details/79842514)\u670d\u52a1\u5df2\u7ecf\u542f\u52a8\n\n\u65b9\u5f0f\u4e00\uff1a\n\n```bash\n$ python3 --version\nPython 3.7.0\n\n# \u521b\u5efa\u540d\u4e3a venv \u7684\u865a\u62df\u73af\u5883\n$ python3 -m venv venv\n\n# \u6fc0\u6d3b\u865a\u62df\u73af\u5883\n$ source venv/bin/activate\n\n# \u5b89\u88c5spider-admin-pro\n$ pip3 install spider-admin-pro\n\n# \u53ef\u9009\n$ pip3 install -U spider-admin-pro -i https://pypi.org/simple\n\n# Linux macOS \u8fd0\u884c\u542f\u52a8\n$ gunicorn 'spider_admin_pro:app'\n\n# windows \u73af\u5883\u4f7f\u7528waitress \u66ff\u6362 gunicorn\n$ pip install waitress\n\n$ waitress-serve --listen=127.0.0.1:8000 'spider_admin_pro:app'\n```\n\n\u65b9\u5f0f\u4e8c\uff1a\n\n```bash\n$ git clone https://github.com/mouday/spider-admin-pro.git\n\n$ cd spider-admin-pro\n\n# \u5b89\u88c5\u4f9d\u8d56\uff08\u5efa\u8bae\uff1a\u6700\u597d\u65b0\u5efa\u4e00\u4e2a\u865a\u62df\u73af\u5883\uff09\n$ pip3 install -r requirements.txt \n\n# Linux/macOS \u4ee5\u5f00\u53d1\u6a21\u5f0f\u8fd0\u884c\n$ make dev\n\n# windows \u4ee5\u5f00\u53d1\u6a21\u5f0f\u8fd0\u884c\n$ python3 dev.py\n\n# \u4ee5\u751f\u4ea7\u6a21\u5f0f\u8fd0\u884c\n$ make pro\n```\n\n\u5b89\u88c5 scrapy \u5168\u5bb6\u6876`[\u53ef\u9009]`\n\n```bash\npip install scrapy scrapyd scrapyd-client\n```\n\n\u65b9\u5f0f\u4e09\uff1a\n\n```bash\nvim config.yaml # \u914d\u7f6e\u6587\u4ef6\u6587\u4ef6\u5185\u5bb9\u89c1\u2b07\ufe0f\uff1a\u914d\u7f6e\u53c2\u6570\ndocker run -e TZ=Asia/Shanghai -p 8000:8000 -v ./config.yml:/app/config.yml mouday/spider-admin-pro\n```\n\n## \u914d\u7f6e\u53c2\u6570\n\n\u81ea\u5b9a\u4e49\u914d\u7f6e\n\n\u5728\u8fd0\u884c\u76ee\u5f55\u4e0b\u65b0\u5efa`config.yml` \u6587\u4ef6\uff0c\u4e5f\u5c31\u662f\u6267\u884c\u542f\u52a8\u547d\u4ee4\u7684\u76ee\u5f55\uff0c\u8fd0\u884c\u65f6\u4f1a\u81ea\u52a8\u8bfb\u53d6\u8be5\u914d\u7f6e\u6587\u4ef6\n\n\u4f8b\u5982\n```bash\n$ ls\nconfig.yml\n\n$ gunicorn 'spider_admin_pro:app'\n```\n> \u5f3a\u70c8\u5efa\u8bae\uff1a\u4fee\u6539\u5bc6\u7801\u548c\u79d8\u94a5\u9879\n\neg:\n\n```yaml\n# \u767b\u5f55\u8d26\u53f7\u5bc6\u7801\nUSERNAME: admin\nPASSWORD: \"123456\"\n\n# \u3010\u53ef\u9009\u3011\u662f\u5426\u5f00\u5f00\u542f\u524d\u7aef\u9759\u6001\u6587\u4ef6\u9884\u538b\u7f29\u4ee5\u63d0\u5347\u9875\u9762\u52a0\u8f7d\u901f\u5ea6\uff0c\u9ed8\u8ba4\u4e3aTrue\nCOMPRESS_STATIC: True\n\n# \u4ee5\u4e0b\u914d\u7f6e\u5728v3.0\u4e0d\u652f\u6301\n# scrapyd\u5730\u5740, \u7ed3\u5c3e\u4e0d\u8981\u52a0\u659c\u6746\n# SCRAPYD_SERVER: \"http://127.0.0.1:6800\"\n\n# \u3010\u53ef\u9009\u3011\u652f\u6301 basic auth @since 2.0.8 \n# SCRAPYD_USERNAME: ''\n# SCRAPYD_PASSWORD: ''\n\n```\n\n## \u4f7f\u7528\u6269\u5c55\n\n\u6536\u96c6\u8fd0\u884c\u65e5\u5fd7\uff1a[scrapy-util](https://github.com/mouday/scrapy-util) \u53ef\u4ee5\u5e2e\u52a9\u4f60\u6536\u96c6\u5230\u7a0b\u5e8f\u8fd0\u884c\u7684\u7edf\u8ba1\u6570\u636e\n\n## \u6280\u672f\u6808\uff1a\n\n1\u3001\u524d\u7aef\u6280\u672f\uff1a\n\n| \u529f\u80fd | \u7b2c\u4e09\u65b9\u5e93\u53ca\u6587\u6863 | \n| - | - | \n| \u57fa\u672c\u6846\u67b6 | [vue2.js](https://cn.vuejs.org/) |\n| \u4eea\u8868\u76d8\u56fe\u8868 | [echarts](https://echarts.apache.org/) |\n| \u7f51\u7edc\u8bf7\u6c42 | [axios](https://www.npmjs.com/package/axios) |\n| \u754c\u9762\u6837\u5f0f | [Element-UI](https://element.eleme.cn/) |\n\n2\u3001\u540e\u7aef\u6280\u672f\n\n| \u529f\u80fd | \u7b2c\u4e09\u65b9\u5e93\u53ca\u6587\u6863 |\n| - | - |\n| \u63a5\u53e3\u670d\u52a1 | [Flask](https://dormousehole.readthedocs.io/) |\n| \u4efb\u52a1\u8c03\u5ea6 | [apscheduler](https://apscheduler.readthedocs.io/) |\n| scrapyd\u63a5\u53e3 | [scrapyd-api](https://github.com/mouday/scrapyd-api) |\n| \u7f51\u7edc\u8bf7\u6c42 | [session-request](https://github.com/mouday/session-request) |\n| ORM | [peewee](http://docs.peewee-orm.com/) |\n| jwt | [jwt](https://pyjwt.readthedocs.io/) |\n| \u7cfb\u7edf\u4fe1\u606f | [psutil](https://psutil.readthedocs.io/) |\n\n## \u9879\u76ee\u7ed3\u6784\n\n\u3010\u516c\u5f00\u4ed3\u5e93\u3011\u57fa\u4e8eFlask\u7684\u540e\u7aef\u9879\u76eespider-admin-pro: [https://github.com/mouday/spider-admin-pro](https://github.com/mouday/spider-admin-pro)\n\n\u3010\u79c1\u6709\u4ed3\u5e93\u3011\u57fa\u4e8eVue\u7684\u524d\u7aef\u9879\u76eespider-admin-pro-web: \u83b7\u53d6\u65b9\u5f0f\u5f80\u4e0b\u770b\n\n> \u5907\u6ce8\uff1a\u524d\u7aefVue\u9879\u76ee\uff0c\u53ef\u5165QQ\u7fa4\u53d1\u9001github\u7528\u6237\u540d\u83b7\u53d6\u6743\u9650\n\n\u83b7\u53d6\u524d\u7aef\u6e90\u7801\u7684\u51e0\u4e2a\u65b9\u5f0f\uff1a\n\n- \u65b9\u5f0f\u4e00\uff1a\u63d0\u4f9bgithub\u7684\u7528\u6237\u540d\n\n- \u65b9\u5f0f\u4e8c\uff1a\u5148\u8d5e\u52a9\u9879\u76ee\uffe510\uff0c\u518d\u63d0\u4f9bgithub\u7684\u7528\u6237\u540d\n\n- \u65b9\u5f0f\u4e09\uff1a\u5173\u6ce8\u5fae\u4fe1\u516c\u4f17\u53f7\uff1a\n\n<img src=\"https://gitee.com/mouday/domain-admin/raw/master/image/coding-big-tree.jpg\" width=\"300\">\n\n\u56de\u590d\uff1a`spider-admin-web`\uff0c\u83b7\u53d6\u5b8c\u6574\u7684\u524d\u7aef\u4ee3\u7801\n\n\u56de\u590d\uff1a`spider-admin`\uff0c\u53ef\u52a0\u5165Python\u6280\u672f\u4ea4\u6d41\u7fa4\uff0c\u548c\u66f4\u591a\u5f00\u53d1\u8005\u5b66\u4e60\u4ea4\u6d41\u9047\u5230\u7684\u95ee\u9898\n\nspider-admin-pro\u9879\u76ee\u4e3b\u8981\u76ee\u5f55\u7ed3\u6784\uff1a\n\n```bash\n.\n\u251c\u2500\u2500 run.py # \u7a0b\u5e8f\u5165\u53e3\n\u251c\u2500\u2500 api # Controller\u5c42\n\u251c\u2500\u2500 service # Sevice\u5c42\n\u251c\u2500\u2500 model # Model\u5c42\n\u251c\u2500\u2500 exceptions # \u5f02\u5e38 \n\u251c\u2500\u2500 utils # \u5de5\u5177\u7c7b\n\u2514\u2500\u2500 web # \u9759\u6001web\u9875\n\n```\n\n## \u7ecf\u9a8c\u603b\u7ed3\n\nScrapyd \u4e0d\u80fd\u76f4\u63a5\u66b4\u9732\u5728\u5916\u7f51\n\n1. \u5176\u4ed6\u4eba\u901a\u8fc7deploy\u90e8\u7f72\u53ef\u4ee5\u5c06\u4ee3\u7801\u90e8\u7f72\u5230\u4f60\u7684\u673a\u5668\u4e0a\uff0c\u5982\u679c\u662froot\u7528\u6237\u8fd0\u884c\uff0c\u8fd8\u4f1a\u5728\u4f60\u673a\u5668\u4e0a\u505a\u5176\u4ed6\u7684\u4e8b\u60c5\n2. \u8fd8\u6709\u8fd0\u884c\u65e5\u5fd7\u4e2d\u4f1a\u51fa\u73b0\u914d\u7f6e\u6587\u4ef6\u4e2d\u7684\u4fe1\u606f\uff0c\u5b58\u5728\u4fe1\u606f\u6cc4\u9732\u7684\u5371\u9669\n\n\n## TODO\n\n~~1. \u8865\u5168\u5f00\u53d1\u6587\u6863~~\n\n~~2. \u652f\u6301\u547d\u4ee4\u884c\u5b89\u88c5\u53ef\u7528~~\n\n~~3. \u4f18\u5316\u4ee3\u7801\u5e03\u5c40\uff0c\u63d0\u53d6\u516c\u5171\u5e93~~\n\n~~4. \u65e5\u5fd7\u81ea\u52a8\u5237\u65b0~~\n\n~~5. scrapy\u9879\u76ee\u6570\u636e\u6536\u96c6~~\n\n[ok]6. \u5b9a\u65f6\u4efb\u52a1spider\u5217\u5de6\u5bf9\u9f50\uff0c\u652f\u6301\u672c\u5730\u6392\u5e8f\n\n[x]7. \u8c03\u5ea6\u5668\u63a7\u5236\u79fb\u9664\u505c\u6b62\u5f00\u542f\u5f00\u5173\uff0c\u53ea\u4fdd\u7559\u6682\u505c\u7ee7\u7eed\n\n[x]8. \u6dfb\u52a0\u4efb\u52a1\uff0c\u9ed8\u8ba4\u9879\u76ee\u540d\uff0c\u5173\u95ed\u5f39\u6846\u53d6\u6d88form\u6821\u9a8c\u7ed3\u679c\n\n[x]9. \u7edf\u8ba1\u7684\u65e5\u5fd7\u91cf\u592a\u5927\uff0c\u589e\u52a0\u4e00\u4e2a\u4e00\u4e2a\u5b9a\u65f6\u6e05\u7406\u7684\u529f\u80fd\n\n[x]10. \u5b9a\u65f6\u4efb\u52a1\u5907\u4efd\uff0c\u4e0d\u5c0f\u5fc3\u628a\u4efb\u52a1\u6e05\u7a7a\n\n[x]11. \u5e0c\u671b\u80fd\u52a0\u5165\u66f4\u597d\u7684\u5b9a\u65f6\u65b9\u5f0f,\u7c7b\u4f3c scrapyd_web\u90a3\u79cd\u5b9a\u65f6\n\n[x]12. \u7b80\u5355\u7684\u722c\u866b\u4e0d\u7528\u975e\u8981\u53bb\u6253\u5305\uff0c\u6bd4\u5982\u6211\u81ea\u5df1\u4e0a\u4f20\u4e00\u4e2apy\u6587\u4ef6\uff0c\u53ef\u4ee5\u5b9a\u65f6\u4efb\u52a1\uff0c\u811a\u672c\u7684\u65b9\u5f0f\u8fd0\u884c\n\n[x]13. \u722c\u866b\u80fd\u914d\u7f6e\u5e26\u53c2\u6570\u8fd0\u884c\n\n## \u4ea4\u6d41\u6c9f\u901a\n\n\u5173\u6ce8\u672c\u9879\u76ee\u7684\u5c0f\u4f19\u4f34\u8d8a\u6765\u8d8a\u591a\uff0c\u4e3a\u4e86\u66f4\u597d\u5730\u4ea4\u6d41\u6c9f\u901a\uff0c\u53ef\u4ee5\u52a0\u5165\u7fa4\u804a\n\n- \u4e00\u7fa4: 1074075691\uff08\u5df2\u6ee1\uff09\n- \u4e8c\u7fa4: 864983297\n\n\u95ee\u9898\uff1a\u9080\u8bf7\u7801 \u7b54\u6848\uff1aSpiderAdmin\n\n<img src=\"https://github.com/mouday/spider-admin-pro/raw/master/doc/img/qq-2.jpg\" width=\"300\"/>\n\n## \u8054\u7cfb\u4f5c\u8005\n\n\u5fae\u4fe1\u626b\u7801\u4e8c\u7ef4\u7801\n\n<img src=\"https://raw.githubusercontent.com/mouday/domain-admin/master/image/coding-big-tree.jpg\" width=\"300\">\n\n\u626b\u63cf\u4e8c\u7ef4\u7801\u540e\uff0c\u56de\u590d\uff1aPython\uff0c\u5373\u53ef\u8fdb\u5165Python\u6280\u672f\u4ea4\u6d41\u7fa4\uff0c\u548c\u6280\u672f\u5927\u4f6c\u4eec\u5b66\u4e60\u4ea4\u6d41\n\n## \u9879\u76ee\u8d5e\u52a9\n\n| \u65e5\u671f | \u59d3\u540d | \u91d1\u989d | \n|------------| - | - |\n| 2022-04-16 | [@realhellosunsun](https://github.com/realhellosunsun) | \uffe5188.00\n| 2022-08-30 | [@yangxiaozhe13](https://github.com/yangxiaozhe13) | \uffe588.00\n| 2022-09-01 | [@robot-2233](https://github.com/robot-2233) | \uffe588.00\n| 2023-05-09 | \u57c3\u83f2\u5c14\u6ca1\u6709\u5854\u5c16 | \uffe568.80\n| 2023-09-21 | [@burujiuzheyang](https://github.com/burujiuzheyang) | \uffe550.00\n| 2023-10-07 | [@Lnine9](https://github.com/Lnine9) | \uffe520.00\n| 2023-10-09 | [@xiaoran-xr](https://github.com/xiaoran-xr) | \uffe520.00\n| 2023-10-10 | [@hsdanbai](https://github.com/hsdanbai) | \uffe520.00\n| 2023-10-19 | [@shuiniu86](https://github.com/shuiniu86) | \uffe550.00\n| 2023-10-24 | [@yuzhou6](https://github.com/yuzhou6) | \uffe550.00\n| 2023-11-13 | [@xuedipiaofei](https://github.com/xuedipiaofei) | \uffe550.00\n| 2024-01-06 | [@if-always](https://github.com/if-always) | \uffe510.00\n| 2024-01-21 | [@dydwgmcnl4241](https://github.com/dydwgmcnl4241) | \uffe510.00\n| 2024-02-18 | [@zhaolipo](https://github.com/zhaolipo) | \uffe510.00\n| 2024-02-19 | [@qianwangali](https://github.com/qianwangali) | \uffe510.00\n| 2024-02-20 | \u6ca1\u4e8b\u5e72\u7684\u5c0f\u4f19\u5b50 | \uffe58.88\n| 2024-02-20 | [@ning-0217](https://github.com/ning-0217) | \uffe550.00\n| 2024-03-08 | [@zhaolipo](https://github.com/zhaolipo) | \uffe520.00\n| 2024-03-18 | [@magiceric](https://github.com/magiceric) | \uffe588.00\n| 2024-09-30 | [@dream2333](https://github.com/dream2333) | \uffe5100.00\n\n<img src=\"https://github.com/mouday/spider-admin-pro/raw/master/doc/img/alipay.jpg\" width=\"300\">\n\n## \u9879\u76ee\u622a\u56fe\n\n\n\n\n\n\n\n\n\n## \u5b89\u88c5\u5347\u7ea7\n```\npip3 install -U spider-admin-pro -i https://pypi.org/simple\n```\n\n\n## Stargazers over time\n\n[](https://starchart.cc/mouday/spider-admin-pro)\n\n\n\n## \u5176\u4ed6\u95ee\u9898\n\n1\u3001windows\u7cfb\u7edf scrapyd \u542f\u52a8\u5931\u8d25\uff0c\u53ef\u80fd\u7f3a\u5c11\u4f9d\u8d56pywin32\n\n```\npip install pywin32\n```\n\n\u611f\u8c22[@whobywind](https://github.com/whobywind)\uff0c\u63d0\u4f9b\u7684\u89e3\u51b3\u65b9\u6848\n\n2\u3001\u7f51\u7ad9\u6709ip\u6821\u9a8c\uff0c\u521a\u8bbf\u95ee\u51e0\u4e2a\u8bf7\u6c42\u5c31\u88ab\u7981\u6b62\u8bbf\u95ee\uff1f\n\n\u540c\u4e00\u4e2aip\u53ef\u80fd\u6709\u88ab\u5c01\u7684\u98ce\u9669\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ee3\u7406ip\u53bb\u8bf7\u6c42\uff0c\u6709\u514d\u8d39\u548c\u4ed8\u8d39\u3002\n\n\u5982\u679c\u662f\u4e2a\u4eba\u4f7f\u7528\uff0c\u53ef\u4ee5\u627e\u4e00\u4e9b\u514d\u8d39\u7684ip\u4e34\u65f6\u4f7f\u7528\n\n\u5982\u679c\u662f\u4f01\u4e1a\u9879\u76ee\uff0c\u53ef\u4ee5\u4f7f\u7528\u4ed8\u8d39\u4ee3\u7406ip\n\n\u67d0\u722c\u866b\u5927\u4f6c\u4e5f\u63a8\u8350\u8fc7\u4e00\u4e2a\u4e0d\u9519\u7684\u52a8\u6001\u4ee3\u7406 [\u4e91\u7acb\u65b9](http://www.yunlifang.cn?u=mouday)\n\n<a href=\"http://www.yunlifang.cn?u=mouday\" target=\"_blank\" style=\"display: inline-block; background-color: #000;\">\n<img src=\"https://www.yunlifang.cn/user/img/720X90.png\">\n</a>\n\n\u627e\u5ba2\u670d\u53d1\u9001\u6697\u53f7\uff1a\u3010\u722c\u866b\u63a8\u5e7f\u3011\u53ef\u4ee5\u83b7\u53d6\u6253\u6298\u4f18\u60e0\n\n\u5177\u4f53\u642d\u5efa\u65b9\u6cd5\u5728\u5927\u4f6c\u7684\u535a\u5ba2\u4e2d\u6709\u8be6\u5c3d\u8bf4\u660e\uff1a\n\n[\u4f7f\u7528 Tornado+Redis \u7ef4\u62a4 ADSL \u62e8\u53f7\u670d\u52a1\u5668\u4ee3\u7406\u6c60](https://cuiqingcai.com/4596.html)\n\n\u5982\u679c\u6709\u95ee\u9898\uff0c\u53ef\u4ee5\u52a0QQ\u7fa4\uff0c\u7fa4\u91cc\u7684\u5c0f\u4f19\u4f34\u4f1a\u79ef\u6781\u89e3\u7b54\u5594\n\n3\u3001\u4e3a\u4ec0\u4e48\u5916\u7f51\u8bbf\u95ee\u4e0d\u5230\uff0c\u5982\u4f55\u4fee\u6539\u7aef\u53e3\u53f7\n\n\u589e\u52a0`--bind` \u53c2\u6570\n\n\u683c\u5f0f\n\n```bash\n--bind \u76d1\u542c\u5730\u5740:\u76d1\u542c\u7aef\u53e3\u53f7\n```\n\n\u4f8b\u5982\n\n```bash\n# \u542f\u52a8\u8fd0\u884c\n$ gunicorn 'spider_admin_pro:app'\n\n# \u652f\u6301\u5916\u7f51\u53ef\u8bbf\u95ee\uff0c\u4e91\u670d\u52a1\u5668\uff08\u963f\u91cc\u4e91\u6216\u817e\u8baf\u4e91\uff09\u9700\u8981\u8bbe\u7f6e\u5b89\u5168\u7ec4 \n# \u9ed8\u8ba4\u5185\u7f51\u8bbf\u95ee --bind 127.0.0.1:8000\n$ gunicorn --bind '0.0.0.0:8000' 'spider_admin_pro:app'\n```\n\n\u66f4\u591a\u8bbe\u7f6e\uff0c\u53ef\u53c2\u8003[gunicorn](https://docs.gunicorn.org/en/stable/index.html)\n\n4\u3001\u63d0\u793a\u7f3a\u5c11libfile\n\n\u7fa4\u53cb `@Yuan\u3001\u7ea2\u5c18\u7f8e` \u63d0\u4f9b\u7684\u89e3\u51b3\u65b9\u6cd5\n\n\u5b89\u88c5\u4f9d\u8d56\n\n```bash\nyum install libffi-devel -y\n```\n\n## \u66f4\u65b0\u65e5\u5fd7\n\n[CHANGELOG.md](./CHANGELOG.md)\n\n## \u793e\u533a\u5176\u4ed6\u4f18\u79c0\u5de5\u5177\u63a8\u8350\n\n- https://github.com/DormyMo/SpiderKeeper\n- https://github.com/my8100/scrapydweb\n- https://github.com/ouqiang/gocron \u4f7f\u7528Go\u8bed\u8a00\u5f00\u53d1\u7684\u8f7b\u91cf\u7ea7\u5b9a\u65f6\u4efb\u52a1\u96c6\u4e2d\u8c03\u5ea6\u548c\u7ba1\u7406\u7cfb\u7edf, \u7528\u4e8e\u66ff\u4ee3Linux-crontab\n\n## Spider Admin Pro vs. Spider Admin\n\n1. \u7b80\u5316\u4e86\u4e00\u4e9b\u529f\u80fd\uff1b\n2. \u4f18\u5316\u4e86\u524d\u7aef\u754c\u9762\uff0c\u57fa\u4e8eVue\u7684\u7ec4\u4ef6\u5316\u5f00\u53d1\uff1b\n3. \u4f18\u5316\u4e86\u540e\u7aef\u63a5\u53e3\uff0c\u5bf9\u540e\u7aef\u9879\u76ee\u8fdb\u884c\u4e86\u76ee\u5f55\u5212\u5206\uff1b\n4. \u6574\u4f53\u4ee3\u7801\u5229\u4e8e\u5347\u7ea7\u7ef4\u62a4\u3002\n5. \u76ee\u524d\u4ec5\u5bf9Python3\u8fdb\u884c\u4e86\u652f\u6301\n6. \u8def\u7531\u7edf\u4e00\u7ba1\u7406\n7. \u5168\u5c40\u5f02\u5e38\u6355\u83b7\n8. \u63a5\u53e3\u7edf\u4e00\u8fd4\u56de\n9. \u524d\u540e\u7aef\u5206\u79bb\n10. \u53ef\u89c6\u5316\u53c2\u6570\u914d\u7f6e\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "a spider admin based vue, scrapyd api and APScheduler",
"version": "3.0.9",
"project_urls": {
"Homepage": "https://github.com/mouday/spider-admin-pro"
},
"split_keywords": [
"spider",
"admin",
"scrapy",
"scrapyd",
"scheduler"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a0f45d3ef0dbee1573dc86760b001a9b673dc3887f320c10a4d3aad28575ceeb",
"md5": "7b2e009bc2c5471b9f8538545645fda9",
"sha256": "6e2b1eeabd35e141b28b26cabe475f87a29aed3608fee5860acfd3232ebaaa61"
},
"downloads": -1,
"filename": "spider_admin_pro-3.0.9-py2.py3-none-any.whl",
"has_sig": false,
"md5_digest": "7b2e009bc2c5471b9f8538545645fda9",
"packagetype": "bdist_wheel",
"python_version": "py2.py3",
"requires_python": null,
"size": 1039162,
"upload_time": "2024-11-01T14:07:37",
"upload_time_iso_8601": "2024-11-01T14:07:37.239471Z",
"url": "https://files.pythonhosted.org/packages/a0/f4/5d3ef0dbee1573dc86760b001a9b673dc3887f320c10a4d3aad28575ceeb/spider_admin_pro-3.0.9-py2.py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "3f98c06092f83049c4aee13bb58318b54317783a0bc3bfd4307cb927e3e45832",
"md5": "62d7e5d72af5cad7c30d54a648c340e3",
"sha256": "57a5ba0482b2b2be6e9493a84437e3ac5b3db4c29f350a0c93da6c91dd9ee13e"
},
"downloads": -1,
"filename": "spider_admin_pro-3.0.9.tar.gz",
"has_sig": false,
"md5_digest": "62d7e5d72af5cad7c30d54a648c340e3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1005016,
"upload_time": "2024-11-01T14:07:39",
"upload_time_iso_8601": "2024-11-01T14:07:39.867552Z",
"url": "https://files.pythonhosted.org/packages/3f/98/c06092f83049c4aee13bb58318b54317783a0bc3bfd4307cb927e3e45832/spider_admin_pro-3.0.9.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-01 14:07:39",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "mouday",
"github_project": "spider-admin-pro",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"requirements": [],
"lcname": "spider-admin-pro"
}
JSON
