# Scrapy Processors
[](https://www.python.org/)
<!-- [](https://codecov.io/gh/nicholas-mischke/scrapy-processors) -->
Scrapy Processors is a collection of Processor classes meant to work with
the [itemloaders](https://pypi.org/project/itemloaders/) package, commonly used with the [scrapy](https://pypi.org/project/Scrapy/) webscraping framework.
These processors are meant to extend / replace the provided processors in the [itemloaders](https://pypi.org/project/itemloaders/) package.
Additionally the provided Processor and ProcessorCollection classes can be extended to create custom processors.
## Installation
To install Scrapy Processors, simply use pip:
```bash
$ pip install scrapy-processors
```
## Built-in Processors
- [CharWhitespacePadding](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#charwhitespacepadding)
- [Coalesce](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#coalesce)
- [Date](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#date)
- [DateTime](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#datetime)
- [DateTimeExtrordinaire](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#datetimeextraordinaire)
- [Demojize](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#demojize)
- [Emails](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#emails)
- [ExtractDigits](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#extractdigits)
- [Flatten](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#flatten)
- [Join](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#join)
- [NormalizeNumericString](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#normalizenumericstring)
- [NormalizeWhitespace](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#normalizewhitespace)
- [PhoneNumbers](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#phonenumbers)
- [PriceParser](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#priceparser)
- [RemoveEmojis](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#removeemojis)
- [RemoveHTMLTags](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#removehtmltags)
- [SelectJmes](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#selectjmes)
- [Socials](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#socials)
- [StripQuotes](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#stripquotes)
- [TakeAll](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takeall)
- [TakeAllTruthy](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takealltruthy)
- [TakeFirst](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takefirst)
- [TakeFirstTruthy](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takefirsttruthy)
- [Time](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#time)
- [ToFloat](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#tofloat)
- [UnicodeEscape](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#unicodeescape)
## Built-in Processor Collections
- [Compose](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-collections.md#compose)
- [MapCompose](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-collections.md#mapcompose)
## Table of Contents
- [What's a Processor?](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/whats-a-processor.md)
- [What's Context?](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/whats-context.md)
- [Subclassing Processor and ProcessorCollection](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/subclassing-processor-and-processorcollection.md)
- [Built-in ProcessorCollection Subclasses](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-collections.md)
- [Built-in value-by-value processors](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md)
- [Built-in iterable processors](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md)
<!-- # Local Links, work locally and on github, not PyPI -->
<!-- ## Built-in Processors
- [CharWhitespacePadding](README/built-in-process-value.md#charwhitespacepadding)
- [Coalesce](README/built-in-dunder-call.md#coalesce)
- [Date](README/built-in-process-value.md#date)
- [DateTime](README/built-in-process-value.md#datetime)
- [DateTimeExtrordinaire](README/built-in-process-value.md#datetimeextraordinaire)
- [Demojize](README/built-in-process-value.md#demojize)
- [Emails](README/built-in-process-value.md#emails)
- [ExtractDigits](README/built-in-process-value.md#extractdigits)
- [Flatten](README/built-in-dunder-call.md#flatten)
- [Join](README/built-in-dunder-call.md#join)
- [NormalizeNumericString](README/built-in-process-value.md#normalizenumericstring)
- [NormalizeWhitespace](README/built-in-process-value.md#normalizewhitespace)
- [PhoneNumbers](README/built-in-process-value.md#phonenumbers)
- [PriceParser](README/built-in-process-value.md#priceparser)
- [RemoveEmojis](README/built-in-process-value.md#removeemojis)
- [RemoveHTMLTags](README/built-in-process-value.md#removehtmltags)
- [SelectJmes](README/built-in-process-value.md#selectjmes)
- [Socials](README/built-in-process-value.md#socials)
- [StripQuotes](README/built-in-process-value.md#stripquotes)
- [TakeAll](README/built-in-dunder-call.md#takeall)
- [TakeAllTruthy](README/built-in-dunder-call.md#takealltruthy)
- [TakeFirst](README/built-in-dunder-call.md#takefirst)
- [TakeFirstTruthy](README/built-in-dunder-call.md#takefirsttruthy)
- [Time](README/built-in-process-value.md#time)
- [ToFloat](README/built-in-process-value.md#tofloat)
- [UnicodeEscape](README/built-in-process-value.md#unicodeescape)
## Built-in Processor Collections
- [Compose](README/built-in-collections.md#compose)
- [MapCompose](README/built-in-collections.md#mapcompose)
## Table of Contents
- [What's a Processor?](README/whats-a-processor.md)
- [What's Context?](README/whats-context.md)
- [Subclassing Processor and ProcessorCollection](README/subclassing-processor-and-processorcollection.md)
- [Built-in ProcessorCollection Subclasses](README/built-in-collections.md)
- [Built-in value-by-value processors](README/built-in-process-value.md)
- [Built-in iterable processors](README/built-in-dunder-call.md) -->
## Opening an Issue
If you encounter a problem with the project or have a feature request, you can open an issue to let us know.
To open an issue, please follow these steps:
1. Go to the [Issues](https://github.com/nicholas-mischke/scrapy-processors/issues) tab on the github repository page.
2. Click on the "New Issue" button.
3. Provide a descriptive title for the issue.
4. In the issue description, provide detailed information about the problem you are experiencing or the feature you are requesting.
5. If applicable, include steps to reproduce the problem or any relevant code examples.
6. Add appropriate labels to categorize the issue (e.g., bug, enhancement, documentation).
7. Click on the "Submit new issue" button to create the issue.
Once you have opened an issue, our team will review it and provide assistance or discuss the requested feature.
Note: Before opening a new issue, please search the existing issues to see if a similar issue has already been reported. This helps avoid duplicates and allows us to focus on resolving existing problems.
## Contributing
Thank you for considering contributing to this project! We welcome your contributions to help make this project better.
To contribute to this project, please follow these steps:
1. Fork the repository by clicking on the "Fork" button at the top of the repository page. This will create a copy of the repository in your GitHub account.
2. Clone the forked repository to your local machine using Git:
```
$ git clone https://github.com/your-username/scrapy-processors.git
```
3. Create a new branch for your changes:
```
$ git checkout -b feature
```
4. Make your desired changes to the codebase.
5. Commit your changes with descriptive commit messages:
```
$ git commit -m "Add new feature"
```
6. Push your changes to your forked repository:
```
$ git push origin feature
```
7. Open a pull request (PR) from your forked repository to the original repository's `master` branch.
8. Provide a clear and descriptive title for your PR and explain the changes you have made.
9. Wait for the project maintainers to review your PR. You may need to make additional changes based on their feedback.
10. Once your PR is approved, it will be merged into the main codebase. Congratulations on your contribution!
If you have any questions or need further assistance, feel free to open an issue or reach out to the project maintainers.
Happy contributing!
## License
This project is licensed under the MIT License. See the LICENSE file for more details.
Raw data
{
"_id": null,
"home_page": "https://github.com/nicholas-mischke/scrapy-processors",
"name": "scrapy-processors",
"maintainer": null,
"docs_url": null,
"requires_python": "<4.0.0,>=3.9.0",
"maintainer_email": null,
"keywords": "scrapy, itemloaders",
"author": "Nicholas Mischke",
"author_email": "nmischkework@proton.me",
"download_url": "https://files.pythonhosted.org/packages/91/da/bb876a76bd0322aec67e3e9572296218f65bf1454f1455a54f4cf11528a6/scrapy_processors-2.0.4.tar.gz",
"platform": null,
"description": "\n# Scrapy Processors\n\n\n[](https://www.python.org/)\n<!-- [](https://codecov.io/gh/nicholas-mischke/scrapy-processors) -->\n\nScrapy Processors is a collection of Processor classes meant to work with\nthe [itemloaders](https://pypi.org/project/itemloaders/) package, commonly used with the [scrapy](https://pypi.org/project/Scrapy/) webscraping framework.\n\nThese processors are meant to extend / replace the provided processors in the [itemloaders](https://pypi.org/project/itemloaders/) package.\n\nAdditionally the provided Processor and ProcessorCollection classes can be extended to create custom processors.\n\n\n## Installation\n\nTo install Scrapy Processors, simply use pip:\n\n```bash\n$ pip install scrapy-processors\n```\n\n## Built-in Processors\n- [CharWhitespacePadding](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#charwhitespacepadding)\n- [Coalesce](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#coalesce)\n- [Date](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#date)\n- [DateTime](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#datetime)\n- [DateTimeExtrordinaire](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#datetimeextraordinaire)\n- [Demojize](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#demojize)\n- [Emails](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#emails)\n- [ExtractDigits](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#extractdigits)\n- [Flatten](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#flatten)\n- [Join](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#join)\n- [NormalizeNumericString](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#normalizenumericstring)\n- [NormalizeWhitespace](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#normalizewhitespace)\n- [PhoneNumbers](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#phonenumbers)\n- [PriceParser](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#priceparser)\n- [RemoveEmojis](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#removeemojis)\n- [RemoveHTMLTags](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#removehtmltags)\n- [SelectJmes](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#selectjmes)\n- [Socials](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#socials)\n- [StripQuotes](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#stripquotes)\n- [TakeAll](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takeall)\n- [TakeAllTruthy](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takealltruthy)\n- [TakeFirst](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takefirst)\n- [TakeFirstTruthy](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md#takefirsttruthy)\n- [Time](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#time)\n- [ToFloat](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#tofloat)\n- [UnicodeEscape](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md#unicodeescape)\n\n## Built-in Processor Collections\n- [Compose](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-collections.md#compose)\n- [MapCompose](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-collections.md#mapcompose)\n\n## Table of Contents\n\n- [What's a Processor?](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/whats-a-processor.md)\n- [What's Context?](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/whats-context.md)\n- [Subclassing Processor and ProcessorCollection](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/subclassing-processor-and-processorcollection.md)\n- [Built-in ProcessorCollection Subclasses](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-collections.md)\n- [Built-in value-by-value processors](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-process-value.md)\n- [Built-in iterable processors](https://github.com/nicholas-mischke/scrapy-processors/blob/main/README/built-in-dunder-call.md)\n\n\n<!-- # Local Links, work locally and on github, not PyPI -->\n<!-- ## Built-in Processors\n- [CharWhitespacePadding](README/built-in-process-value.md#charwhitespacepadding)\n- [Coalesce](README/built-in-dunder-call.md#coalesce)\n- [Date](README/built-in-process-value.md#date)\n- [DateTime](README/built-in-process-value.md#datetime)\n- [DateTimeExtrordinaire](README/built-in-process-value.md#datetimeextraordinaire)\n- [Demojize](README/built-in-process-value.md#demojize)\n- [Emails](README/built-in-process-value.md#emails)\n- [ExtractDigits](README/built-in-process-value.md#extractdigits)\n- [Flatten](README/built-in-dunder-call.md#flatten)\n- [Join](README/built-in-dunder-call.md#join)\n- [NormalizeNumericString](README/built-in-process-value.md#normalizenumericstring)\n- [NormalizeWhitespace](README/built-in-process-value.md#normalizewhitespace)\n- [PhoneNumbers](README/built-in-process-value.md#phonenumbers)\n- [PriceParser](README/built-in-process-value.md#priceparser)\n- [RemoveEmojis](README/built-in-process-value.md#removeemojis)\n- [RemoveHTMLTags](README/built-in-process-value.md#removehtmltags)\n- [SelectJmes](README/built-in-process-value.md#selectjmes)\n- [Socials](README/built-in-process-value.md#socials)\n- [StripQuotes](README/built-in-process-value.md#stripquotes)\n- [TakeAll](README/built-in-dunder-call.md#takeall)\n- [TakeAllTruthy](README/built-in-dunder-call.md#takealltruthy)\n- [TakeFirst](README/built-in-dunder-call.md#takefirst)\n- [TakeFirstTruthy](README/built-in-dunder-call.md#takefirsttruthy)\n- [Time](README/built-in-process-value.md#time)\n- [ToFloat](README/built-in-process-value.md#tofloat)\n- [UnicodeEscape](README/built-in-process-value.md#unicodeescape)\n\n## Built-in Processor Collections\n- [Compose](README/built-in-collections.md#compose)\n- [MapCompose](README/built-in-collections.md#mapcompose)\n\n## Table of Contents\n\n- [What's a Processor?](README/whats-a-processor.md)\n- [What's Context?](README/whats-context.md)\n- [Subclassing Processor and ProcessorCollection](README/subclassing-processor-and-processorcollection.md)\n- [Built-in ProcessorCollection Subclasses](README/built-in-collections.md)\n- [Built-in value-by-value processors](README/built-in-process-value.md)\n- [Built-in iterable processors](README/built-in-dunder-call.md) -->\n\n## Opening an Issue\n\nIf you encounter a problem with the project or have a feature request, you can open an issue to let us know.\n\nTo open an issue, please follow these steps:\n\n1. Go to the [Issues](https://github.com/nicholas-mischke/scrapy-processors/issues) tab on the github repository page.\n2. Click on the \"New Issue\" button.\n3. Provide a descriptive title for the issue.\n4. In the issue description, provide detailed information about the problem you are experiencing or the feature you are requesting.\n5. If applicable, include steps to reproduce the problem or any relevant code examples.\n6. Add appropriate labels to categorize the issue (e.g., bug, enhancement, documentation).\n7. Click on the \"Submit new issue\" button to create the issue.\n\nOnce you have opened an issue, our team will review it and provide assistance or discuss the requested feature.\n\nNote: Before opening a new issue, please search the existing issues to see if a similar issue has already been reported. This helps avoid duplicates and allows us to focus on resolving existing problems.\n\n## Contributing\n\nThank you for considering contributing to this project! We welcome your contributions to help make this project better.\n\nTo contribute to this project, please follow these steps:\n\n1. Fork the repository by clicking on the \"Fork\" button at the top of the repository page. This will create a copy of the repository in your GitHub account.\n2. Clone the forked repository to your local machine using Git:\n\n ```\n $ git clone https://github.com/your-username/scrapy-processors.git\n ```\n\n3. Create a new branch for your changes:\n\n ```\n $ git checkout -b feature\n ```\n\n4. Make your desired changes to the codebase.\n5. Commit your changes with descriptive commit messages:\n\n ```\n $ git commit -m \"Add new feature\"\n ```\n\n6. Push your changes to your forked repository:\n\n ```\n $ git push origin feature\n ```\n\n7. Open a pull request (PR) from your forked repository to the original repository's `master` branch.\n8. Provide a clear and descriptive title for your PR and explain the changes you have made.\n9. Wait for the project maintainers to review your PR. You may need to make additional changes based on their feedback.\n10. Once your PR is approved, it will be merged into the main codebase. Congratulations on your contribution!\n\nIf you have any questions or need further assistance, feel free to open an issue or reach out to the project maintainers.\n\nHappy contributing!\n\n## License\nThis project is licensed under the MIT License. See the LICENSE file for more details.",
"bugtrack_url": null,
"license": "MIT",
"summary": "Provides processors for the itemloaders package, commonly used with scrapy.",
"version": "2.0.4",
"project_urls": {
"Homepage": "https://github.com/nicholas-mischke/scrapy-processors",
"Repository": "https://github.com/nicholas-mischke/scrapy-processors"
},
"split_keywords": [
"scrapy",
" itemloaders"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "3e1ef946a3dfdd5c03b96b3d991f273d599a6a6bc96e5d72838beb771dc0664b",
"md5": "b9c9252c9ca62ba63bbede1bd04964e4",
"sha256": "01393efe5c9f1375d59e88ca019069a84a6c12dbf310c5b4d40db31ec2e37520"
},
"downloads": -1,
"filename": "scrapy_processors-2.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b9c9252c9ca62ba63bbede1bd04964e4",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": "<4.0.0,>=3.9.0",
"size": 28504,
"upload_time": "2024-03-21T03:27:50",
"upload_time_iso_8601": "2024-03-21T03:27:50.439826Z",
"url": "https://files.pythonhosted.org/packages/3e/1e/f946a3dfdd5c03b96b3d991f273d599a6a6bc96e5d72838beb771dc0664b/scrapy_processors-2.0.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "91dabb876a76bd0322aec67e3e9572296218f65bf1454f1455a54f4cf11528a6",
"md5": "3e43833c1b7291c526523a648c119f1d",
"sha256": "ebb68991cd1f023a2e8762913bfc81df401dc230c6eacf6fa1b5b457562125ca"
},
"downloads": -1,
"filename": "scrapy_processors-2.0.4.tar.gz",
"has_sig": false,
"md5_digest": "3e43833c1b7291c526523a648c119f1d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": "<4.0.0,>=3.9.0",
"size": 28202,
"upload_time": "2024-03-21T03:27:52",
"upload_time_iso_8601": "2024-03-21T03:27:52.283044Z",
"url": "https://files.pythonhosted.org/packages/91/da/bb876a76bd0322aec67e3e9572296218f65bf1454f1455a54f4cf11528a6/scrapy_processors-2.0.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2024-03-21 03:27:52",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "nicholas-mischke",
"github_project": "scrapy-processors",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"tox": true,
"lcname": "scrapy-processors"
}
JSON
