# Scraping Engine for Competitive Programming Accelerated Retriever (Secpar)
[](https://pypi.python.org/pypi/Secpar)
[](https://pepy.tech/project/Secpar)
[](https://github.com/MostafaGalal1/Secpar/blob/master/LICENSE)
## Overview
Secpar is a Python command-line tool designed to scrape code submissions from various online programming platforms and store them in a GitHub repository. It supports platforms such as Codeforces, CSES (University of Helsinki), and Vjudge. This documentation provides a detailed overview of the Scraper's functionalities and how to use them.
#### Demo Repo: [CP-Submissions](https://github.com/MostafaGalal1/CP-Submissions)
## Table of Contents
1. [Features](#features)
2. [Installation](#installation)
3. [Usage](#usage)
- [Initialization](#initialization)
- [Scraping](#scraping)
4. [Command-Line Interface](#command-line-interface)
5. [Scraper Configuration](#scraper-configuration)
6. [Customization](#customization)
7. [Data Storage](#data-storage)
8. [FAQs](#faqs)
9. [Contributing](#contributing)
10. [License](#license)
11. [Upcoming Features](#upcoming-features)
## 1. Features <a name="features"></a>
- **Supported Platforms**: Secpar supports the following programming platforms:
- Codeforces
- CSES (University of Helsinki)
- Vjudge
- **GitHub Integration**: Code submissions are stored in a GitHub repository, making it easy to manage and share your solutions.
- **Automatic README Generation**: Secpar automatically generates a README file for your GitHub repository, listing all your code submissions with problem details, links, and tags.
- **Incremental Scraping**: Secpar keeps track of previously scraped submissions, ensuring that only new submissions are added to your repository.
- **Multi-accounts Scraping**: Scrape the same platform more than once in case of having multiple accounts on that platform without worry of redundancy.
- **Authentication**: Securely authenticate with the supported platforms to access your submissions.
- **Customization**: Customize the formatting of your README and configure other scraper options.
## 2. Installation <a name="installation"></a>
To use Secpar, follow these installation steps:
**Install Secpar**: Install Secpar package on your local machine:
```shell
pip install Secpar
```
## 3. Usage <a name="usage"></a>
Secpar has two primary modes of operation: initialization and scraping.
### Initialization <a name="initialization"></a>
Initialization is the first step to configure your scraper for a GitHub repository.
1. Run the initialization command:
```shell
python secpar -c init
```
2. Follow the prompts to enter your GitHub username, repository name, and access token.
### Scraping <a name="scraping"></a>
Scraping allows you to retrieve code submissions from supported platforms and store them in your GitHub repository.
1. To scrape submissions, use the following command:
```shell
secpar -s PLATFORM_NAME
```
Replace `PLATFORM_NAME` with one of the supported platforms: `codeforces`, `cses`, or `vjudge`.
2. Depending on the platform, you may need to provide additional information such as your platform password.
3. Secpar will fetch new submissions and update your GitHub repository.
### Note:
To upload submissions codes for codeforces you need to have `Tor` installed and inside your torrc file place these two line:
```shell
SocksPort 9050
ControlPort 9051
```
Open tor tab and make sure you can browse using it and keep it open till you scrape using the terminal.
## 4. Command-Line Interface <a name="command-line-interface"></a>
Secpar provides a command-line interface with the following options:
- `-c`, `--command`: Specify the command (`init` for initialization or `update` for scraping).
- `-s`, `--scrape`: Specify the platform to scrape data from (`codeforces`, `cses`, or `vjudge`).
- `-h`, `--help`: Display usage instructions and available options.
Example usage:
```shell
python secpar -c init
python secpar -s codeforces
```
## 5. Scraper Configuration <a name="scraper-configuration"></a>
Secpar can be configured in several ways:
- **GitHub Configuration**: Set up your GitHub repository details and access token during initialization.
- **Platform Authentication**: Authenticate with your platform credentials (e.g., Codeforces username and password) for scraping.
- **Customization**: Customize the formatting of your README and configure scraper options in the code (e.g., maximum requests, submission per update).
## 6. Customization <a name="customization"></a>
You can customize Secpar's behavior by modifying the source code. Here are some customization options:
- **Formatting**: Customize the formatting of the generated README for each platform. You can modify the formatting in the corresponding `Formatter` class.
- **Configuration**: Adjust Secpar's settings, such as maximum requests, submissions per update, or other platform-specific parameters.
## 7. Data Storage <a name="data-storage"></a>
Secpar stores code submissions and related information in your GitHub repository. Each submission is listed in the README with details such as problem name, language, solution link, tags, and submission date.
## 8. FAQs <a name="faqs"></a>
- **What platforms does Secpar support?**
- Secpar currently supports Codeforces, CSES (University of Helsinki), and Vjudge.
- **Is it safe to store my GitHub access token?**
- Access tokens should be stored securely. Secpar stores them in a configuration file, and it's essential to protect this file.
- **How can I customize the README format?**
- You can customize the README format by modifying the corresponding `Formatter` class for each platform.
## 9. Contributing <a name="contributing"></a>
Contributions to Secpar are welcome! Feel free to fork the repository, make improvements, and create pull requests.
## 10. License <a name="license"></a>
Secpar is released under the MIT License. See the [LICENSE](https://github.com/MostafaGalal1/Secpar/blob/main/LICENSE) file for details.
## 11. Upcoming Features <a name="upcoming-features">
- More platforms: platforms such as Atcoder and CodeChef are currently being worked on.
---
*Note: This documentation provides an overview of Secpar's functionality and usage. For detailed code explanations, refer to the source code and comments in Secpar's repository.*
Raw data
{
"_id": null,
"home_page": "https://github.com/MostafaGalal1/Secpar",
"name": "Secpar",
"maintainer": "",
"docs_url": null,
"requires_python": ">=3.5",
"maintainer_email": "",
"keywords": "CP,scraper,secpar,problem solving,python,tools",
"author": "Mostafa Galal <mostafam.galal82@gmail.com>, Mahmoud Goda <mahmoden17@gmail.com>",
"author_email": "",
"download_url": "https://files.pythonhosted.org/packages/85/18/4147806b141fc8f798fa47416118bcebcc21b0dbb9484facc9ed10a2e426/Secpar-1.1.4.tar.gz",
"platform": null,
"description": "# Scraping Engine for Competitive Programming Accelerated Retriever (Secpar)\r\n\r\n[](https://pypi.python.org/pypi/Secpar)\r\n\r\n[](https://pepy.tech/project/Secpar)\r\n\r\n[](https://github.com/MostafaGalal1/Secpar/blob/master/LICENSE)\r\n\r\n\r\n\r\n\r\n\r\n## Overview\r\n\r\n\r\n\r\nSecpar is a Python command-line tool designed to scrape code submissions from various online programming platforms and store them in a GitHub repository. It supports platforms such as Codeforces, CSES (University of Helsinki), and Vjudge. This documentation provides a detailed overview of the Scraper's functionalities and how to use them.\r\n\r\n\r\n\r\n#### Demo Repo: [CP-Submissions](https://github.com/MostafaGalal1/CP-Submissions)\r\n\r\n\r\n\r\n## Table of Contents\r\n\r\n\r\n\r\n1. [Features](#features)\r\n\r\n2. [Installation](#installation)\r\n\r\n3. [Usage](#usage)\r\n\r\n - [Initialization](#initialization)\r\n\r\n - [Scraping](#scraping)\r\n\r\n4. [Command-Line Interface](#command-line-interface)\r\n\r\n5. [Scraper Configuration](#scraper-configuration)\r\n\r\n6. [Customization](#customization)\r\n\r\n7. [Data Storage](#data-storage)\r\n\r\n8. [FAQs](#faqs)\r\n\r\n9. [Contributing](#contributing)\r\n\r\n10. [License](#license)\r\n\r\n11. [Upcoming Features](#upcoming-features) \r\n\r\n\r\n\r\n## 1. Features <a name=\"features\"></a>\r\n\r\n\r\n\r\n- **Supported Platforms**: Secpar supports the following programming platforms:\r\n\r\n - Codeforces\r\n\r\n - CSES (University of Helsinki)\r\n\r\n - Vjudge\r\n\r\n\r\n\r\n- **GitHub Integration**: Code submissions are stored in a GitHub repository, making it easy to manage and share your solutions.\r\n\r\n\r\n\r\n- **Automatic README Generation**: Secpar automatically generates a README file for your GitHub repository, listing all your code submissions with problem details, links, and tags.\r\n\r\n\r\n\r\n- **Incremental Scraping**: Secpar keeps track of previously scraped submissions, ensuring that only new submissions are added to your repository.\r\n\r\n \r\n\r\n- **Multi-accounts Scraping**: Scrape the same platform more than once in case of having multiple accounts on that platform without worry of redundancy.\r\n\r\n\r\n\r\n- **Authentication**: Securely authenticate with the supported platforms to access your submissions.\r\n\r\n\r\n\r\n- **Customization**: Customize the formatting of your README and configure other scraper options.\r\n\r\n\r\n\r\n## 2. Installation <a name=\"installation\"></a>\r\n\r\n\r\n\r\nTo use Secpar, follow these installation steps:\r\n\r\n\r\n\r\n**Install Secpar**: Install Secpar package on your local machine:\r\n\r\n\r\n\r\n```shell\r\n\r\npip install Secpar\r\n\r\n```\r\n\r\n\r\n\r\n## 3. Usage <a name=\"usage\"></a>\r\n\r\n\r\n\r\nSecpar has two primary modes of operation: initialization and scraping.\r\n\r\n\r\n\r\n### Initialization <a name=\"initialization\"></a>\r\n\r\n\r\n\r\nInitialization is the first step to configure your scraper for a GitHub repository.\r\n\r\n\r\n\r\n1. Run the initialization command:\r\n\r\n\r\n\r\n ```shell\r\n\r\n python secpar -c init\r\n\r\n ```\r\n\r\n\r\n\r\n2. Follow the prompts to enter your GitHub username, repository name, and access token.\r\n\r\n\r\n\r\n### Scraping <a name=\"scraping\"></a>\r\n\r\n\r\n\r\nScraping allows you to retrieve code submissions from supported platforms and store them in your GitHub repository.\r\n\r\n\r\n\r\n1. To scrape submissions, use the following command:\r\n\r\n\r\n\r\n ```shell\r\n\r\n secpar -s PLATFORM_NAME\r\n\r\n ```\r\n\r\n\r\n\r\n Replace `PLATFORM_NAME` with one of the supported platforms: `codeforces`, `cses`, or `vjudge`.\r\n\r\n\r\n\r\n2. Depending on the platform, you may need to provide additional information such as your platform password.\r\n\r\n\r\n\r\n3. Secpar will fetch new submissions and update your GitHub repository.\r\n\r\n\r\n\r\n### Note:\r\n\r\nTo upload submissions codes for codeforces you need to have `Tor` installed and inside your torrc file place these two line:\r\n\r\n\r\n\r\n```shell\r\n\r\nSocksPort 9050\r\n\r\nControlPort 9051\r\n\r\n```\r\n\r\n\r\n\r\nOpen tor tab and make sure you can browse using it and keep it open till you scrape using the terminal.\r\n\r\n\r\n\r\n## 4. Command-Line Interface <a name=\"command-line-interface\"></a>\r\n\r\n\r\n\r\nSecpar provides a command-line interface with the following options:\r\n\r\n\r\n\r\n- `-c`, `--command`: Specify the command (`init` for initialization or `update` for scraping).\r\n\r\n\r\n\r\n- `-s`, `--scrape`: Specify the platform to scrape data from (`codeforces`, `cses`, or `vjudge`).\r\n\r\n\r\n\r\n- `-h`, `--help`: Display usage instructions and available options.\r\n\r\n\r\n\r\nExample usage:\r\n\r\n\r\n\r\n```shell\r\n\r\npython secpar -c init\r\n\r\npython secpar -s codeforces\r\n\r\n```\r\n\r\n\r\n\r\n## 5. Scraper Configuration <a name=\"scraper-configuration\"></a>\r\n\r\n\r\n\r\nSecpar can be configured in several ways:\r\n\r\n\r\n\r\n- **GitHub Configuration**: Set up your GitHub repository details and access token during initialization.\r\n\r\n\r\n\r\n- **Platform Authentication**: Authenticate with your platform credentials (e.g., Codeforces username and password) for scraping.\r\n\r\n\r\n\r\n- **Customization**: Customize the formatting of your README and configure scraper options in the code (e.g., maximum requests, submission per update).\r\n\r\n\r\n\r\n## 6. Customization <a name=\"customization\"></a>\r\n\r\n\r\n\r\nYou can customize Secpar's behavior by modifying the source code. Here are some customization options:\r\n\r\n\r\n\r\n- **Formatting**: Customize the formatting of the generated README for each platform. You can modify the formatting in the corresponding `Formatter` class.\r\n\r\n\r\n\r\n- **Configuration**: Adjust Secpar's settings, such as maximum requests, submissions per update, or other platform-specific parameters.\r\n\r\n\r\n\r\n## 7. Data Storage <a name=\"data-storage\"></a>\r\n\r\n\r\n\r\nSecpar stores code submissions and related information in your GitHub repository. Each submission is listed in the README with details such as problem name, language, solution link, tags, and submission date.\r\n\r\n\r\n\r\n## 8. FAQs <a name=\"faqs\"></a>\r\n\r\n\r\n\r\n- **What platforms does Secpar support?**\r\n\r\n - Secpar currently supports Codeforces, CSES (University of Helsinki), and Vjudge.\r\n\r\n\r\n\r\n- **Is it safe to store my GitHub access token?**\r\n\r\n - Access tokens should be stored securely. Secpar stores them in a configuration file, and it's essential to protect this file.\r\n\r\n\r\n\r\n- **How can I customize the README format?**\r\n\r\n - You can customize the README format by modifying the corresponding `Formatter` class for each platform.\r\n\r\n\r\n\r\n## 9. Contributing <a name=\"contributing\"></a>\r\n\r\n\r\n\r\nContributions to Secpar are welcome! Feel free to fork the repository, make improvements, and create pull requests.\r\n\r\n\r\n\r\n## 10. License <a name=\"license\"></a>\r\n\r\n\r\n\r\nSecpar is released under the MIT License. See the [LICENSE](https://github.com/MostafaGalal1/Secpar/blob/main/LICENSE) file for details.\r\n\r\n\r\n\r\n## 11. Upcoming Features <a name=\"upcoming-features\">\r\n\r\n- More platforms: platforms such as Atcoder and CodeChef are currently being worked on.\r\n\r\n\r\n\r\n---\r\n\r\n\r\n\r\n*Note: This documentation provides an overview of Secpar's functionality and usage. For detailed code explanations, refer to the source code and comments in Secpar's repository.*\r\n\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "Scrape problems submissions from different platforms",
"version": "1.1.4",
"project_urls": {
"Homepage": "https://github.com/MostafaGalal1/Secpar"
},
"split_keywords": [
"cp",
"scraper",
"secpar",
"problem solving",
"python",
"tools"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "b8d85fd6d977f1e5425bfb530bf0d072ef54f80905d9ce70761dd96b2a95ff6d",
"md5": "446812ab96b9b40fccb5c029ff2955e8",
"sha256": "4404a6f3ba6983ad12a0c9830c28ed8871c26e3711f8bc919b5265048fe2f896"
},
"downloads": -1,
"filename": "Secpar-1.1.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "446812ab96b9b40fccb5c029ff2955e8",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.5",
"size": 47213,
"upload_time": "2023-10-11T20:17:21",
"upload_time_iso_8601": "2023-10-11T20:17:21.699254Z",
"url": "https://files.pythonhosted.org/packages/b8/d8/5fd6d977f1e5425bfb530bf0d072ef54f80905d9ce70761dd96b2a95ff6d/Secpar-1.1.4-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "85184147806b141fc8f798fa47416118bcebcc21b0dbb9484facc9ed10a2e426",
"md5": "93de43f6739c8e22475714ee76712e35",
"sha256": "dd2da6b1ee8fac1a8330656999ef05f4a9582b4c6372ad469ff868dd92e5dc2d"
},
"downloads": -1,
"filename": "Secpar-1.1.4.tar.gz",
"has_sig": false,
"md5_digest": "93de43f6739c8e22475714ee76712e35",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.5",
"size": 32096,
"upload_time": "2023-10-11T20:17:23",
"upload_time_iso_8601": "2023-10-11T20:17:23.963828Z",
"url": "https://files.pythonhosted.org/packages/85/18/4147806b141fc8f798fa47416118bcebcc21b0dbb9484facc9ed10a2e426/Secpar-1.1.4.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-10-11 20:17:23",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "MostafaGalal1",
"github_project": "Secpar",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "secpar"
}
JSON
