| Name | ao3scraper JSON |
| Version |
1.0.3
JSON |
| download |
| home_page | https://github.com/EthanLeitch/ao3scraper |
| Summary | ao3scraper is a python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated. |
| upload_time | 2024-11-07 21:51:40 |
| maintainer | None |
| docs_url | None |
| author | Ethan |
| requires_python | <4.0,>=3.10 |
| license | GPL-3.0-or-later |
| keywords |
|
| VCS |
 |
| bugtrack_url |
|
| requirements |
No requirements were recorded.
|
| Travis-CI |
No Travis.
|
| coveralls test coverage |
No coveralls.
|
# ao3scraper
A python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated.
*Table with an updated entry highlighted.*
## Installation
You can easily install the latest version from pip:
pip3 install ao3scraper
### Development Installation
Create a python virtual environment with `python3 -m venv dev_venv` and activate it.
Then, install required packages with:
poetry install
This will also install ao3scraper into the virtual environment.
## Usage
Usage: ao3scraper [OPTIONS]
Options:
-s, --scrape Launches scraping mode.
-c, --cache Prints the last scraped table.
-l, --list Lists all entries in the database.
-a, --add TEXT Adds a single url to the database.
--add-urls Opens a text file to add multiple urls to the database.
-d, --delete INTEGER Deletes an entry from the database.
-v, --version Display version of ao3scraper and other info.
--help Show this message and exit.
## Configuration
ao3scraper is ridiculously customisable, and most aspects of the program can be modified from here.
To find the configuration file location, run `python3 ao3scraper -v`.
ao3scraper uses [rich](https://rich.readthedocs.io/en/stable/style.html)'s styling. To disable any styling options, replace the styling value with 'none'.
Fics have many attributes that are not displayed by default. To add these columns, create a new option under table_template, like so:
```yaml
table_template:
- column: characters # The specified attribute
name: Characters :) # This is what the column will be labelled as
styles: none # Rich styling
```
A complete list of attributes can be found [on the wiki](https://github.com/EthanLeitch/ao3scraper/wiki/Fic-Attributes/).
## Migrating the database
If you're updating from a legacy version of ao3scraper (before 1.0.0), move `fics.db` to the data location.
This can be found by running `python3 ao3scraper -v`.
The migration wizard will then prompt you to upgrade your database.
If you accept, a backup of the current `fics.db` will be created in `/backups`, and migration will proceed.
## Contributing
Contributions are always appreciated. Submit a pull request with your suggested changes!
## Acknowledgements
ao3scraper would not be possible without the existence of [ao3_api](https://github.com/ArmindoFlores/ao3_api/) and the work of its [contributors](https://github.com/ArmindoFlores/ao3_api/graphs/contributors).
Raw data
{
"_id": null,
"home_page": "https://github.com/EthanLeitch/ao3scraper",
"name": "ao3scraper",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0,>=3.10",
"maintainer_email": null,
"keywords": null,
"author": "Ethan",
"author_email": "ethanjohnleitch@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/9f/de/4ee6739bae1823eba55be40f12ecaeff329bb7a28d54acf99f12efb60ea7/ao3scraper-1.0.3.tar.gz",
"platform": null,
"description": "# ao3scraper\nA python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated.\n\n\n\n*Table with an updated entry highlighted.*\n\n## Installation\nYou can easily install the latest version from pip:\n\n pip3 install ao3scraper\n\n### Development Installation\nCreate a python virtual environment with `python3 -m venv dev_venv` and activate it.\nThen, install required packages with:\n\n poetry install\n\nThis will also install ao3scraper into the virtual environment.\n\n## Usage\n Usage: ao3scraper [OPTIONS]\n\n Options:\n -s, --scrape Launches scraping mode.\n -c, --cache Prints the last scraped table.\n -l, --list Lists all entries in the database.\n -a, --add TEXT Adds a single url to the database.\n --add-urls Opens a text file to add multiple urls to the database.\n -d, --delete INTEGER Deletes an entry from the database.\n -v, --version Display version of ao3scraper and other info.\n --help Show this message and exit.\n\n## Configuration\nao3scraper is ridiculously customisable, and most aspects of the program can be modified from here.\nTo find the configuration file location, run `python3 ao3scraper -v`.\n\nao3scraper uses [rich](https://rich.readthedocs.io/en/stable/style.html)'s styling. To disable any styling options, replace the styling value with 'none'.\n\nFics have many attributes that are not displayed by default. To add these columns, create a new option under table_template, like so:\n```yaml\ntable_template:\n- column: characters # The specified attribute\n name: Characters :) # This is what the column will be labelled as\n styles: none # Rich styling\n```\nA complete list of attributes can be found [on the wiki](https://github.com/EthanLeitch/ao3scraper/wiki/Fic-Attributes/).\n\n## Migrating the database\nIf you're updating from a legacy version of ao3scraper (before 1.0.0), move `fics.db` to the data location. \nThis can be found by running `python3 ao3scraper -v`.\nThe migration wizard will then prompt you to upgrade your database. \nIf you accept, a backup of the current `fics.db` will be created in `/backups`, and migration will proceed.\n\n## Contributing\nContributions are always appreciated. Submit a pull request with your suggested changes!\n\n## Acknowledgements\nao3scraper would not be possible without the existence of [ao3_api](https://github.com/ArmindoFlores/ao3_api/) and the work of its [contributors](https://github.com/ArmindoFlores/ao3_api/graphs/contributors).",
"bugtrack_url": null,
"license": "GPL-3.0-or-later",
"summary": "ao3scraper is a python webscraper that scrapes AO3 for fanfiction data, stores it in a database, and highlights entries when they are updated.",
"version": "1.0.3",
"project_urls": {
"Homepage": "https://github.com/EthanLeitch/ao3scraper",
"Repository": "https://github.com/EthanLeitch/ao3scraper"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "a56cb8710ab16d22e09346ffa29f3a77e2d0356420f85e3e53c2fef00ce2884b",
"md5": "36eb179bf7384db83d6292f63fdb9f22",
"sha256": "6e86bebdf51ee8f9c513c053a40d5f4aecd22e349640d302d2ad47ea9cf96187"
},
"downloads": -1,
"filename": "ao3scraper-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "36eb179bf7384db83d6292f63fdb9f22",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0,>=3.10",
"size": 24153,
"upload_time": "2024-11-07T21:51:39",
"upload_time_iso_8601": "2024-11-07T21:51:39.473673Z",
"url": "https://files.pythonhosted.org/packages/a5/6c/b8710ab16d22e09346ffa29f3a77e2d0356420f85e3e53c2fef00ce2884b/ao3scraper-1.0.3-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9fde4ee6739bae1823eba55be40f12ecaeff329bb7a28d54acf99f12efb60ea7",
"md5": "68f30931ac474be13108b0fc56552fb8",
"sha256": "b3eb4d12a89c5a15f955190b6634e1a1fade6a7d769f5fc0e3ba0052d358ebfd"
},
"downloads": -1,
"filename": "ao3scraper-1.0.3.tar.gz",
"has_sig": false,
"md5_digest": "68f30931ac474be13108b0fc56552fb8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0,>=3.10",
"size": 22682,
"upload_time": "2024-11-07T21:51:40",
"upload_time_iso_8601": "2024-11-07T21:51:40.662314Z",
"url": "https://files.pythonhosted.org/packages/9f/de/4ee6739bae1823eba55be40f12ecaeff329bb7a28d54acf99f12efb60ea7/ao3scraper-1.0.3.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-11-07 21:51:40",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "EthanLeitch",
"github_project": "ao3scraper",
"travis_ci": false,
"coveralls": false,
"github_actions": true,
"lcname": "ao3scraper"
}
