reposcraping


Namereposcraping JSON
Version 1.1.7 PyPI version JSON
download
home_pagehttps://github.com/emresvd/reposcraping
SummaryScraping GitHub repository
upload_time2023-01-29 14:08:13
maintainer
docs_urlNone
authoremresvd
requires_python
licenseMIT
keywords github scraping repository clone
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # [reposcraping](https://pypi.org/project/reposcraping/)

### Scraping GitHub repository

This library allows you to access the names of files and folders of any GitHub repository. Cloner class allows you to clone the file types you want to the path you want.<br>

## downloads

[![Downloads](https://static.pepy.tech/badge/reposcraping)](https://pepy.tech/project/reposcraping) [![Downloads](https://static.pepy.tech/badge/reposcraping/month)](https://pepy.tech/project/reposcraping) [![Downloads](https://static.pepy.tech/badge/reposcraping/week)](https://pepy.tech/project/reposcraping)

## setup

```bash

pip install reposcraping

```

## usage

```python

from reposcraping import RepoScraping

from reposcraping.cloner import Cloner



scraping = RepoScraping(

    "https://github.com/emresvd/random-video",

    p=True,

)



print(scraping.tree_urls)

print(scraping.file_urls)



cloner = Cloner(scraping)

cloner.clone(

    paths={

        ".py": "files/python_files",

        ".txt": "files/text_files",

        ".md": "files/markdown_files",

        ".html": "files/html_files",

        "": "files/other_files",

    },

    only_file_name=True,

    p=True,

)

```

## output

```python

['https://github.com/emresvd/random-video', 'https://github.com/emresvd/random-video/tree/master/random_video', 'https://github.com/emresvd/random-video/tree/master/special_search', 'https://github.com/emresvd/random-video/tree/master/static', 'https://github.com/emresvd/random-video/tree/master/templates', 'https://github.com/emresvd/random-video/tree/master/video', 'https://github.com/emresvd/random-video/tree/master/random_video/__pycache__', 'https://github.com/emresvd/random-video/tree/master/video/__pycache__', 'https://github.com/emresvd/random-video/tree/master/video/migrations', 'https://github.com/emresvd/random-video/tree/master/video/migrations/__pycache__']

['https://github.com/emresvd/random-video/blob/master/LICENSE.md', 'https://github.com/emresvd/random-video/blob/master/.gitignore', 'https://github.com/emresvd/random-video/blob/master/README.md', 'https://github.com/emresvd/random-video/blob/master/db.sqlite3', 'https://github.com/emresvd/random-video/blob/master/manage.py', 'https://github.com/emresvd/random-video/blob/master/requirements.txt', 'https://github.com/emresvd/random-video/blob/master/words.txt', 'https://github.com/emresvd/random-video/blob/master/static/ic_launcher-playstore.png', 'https://github.com/emresvd/random-video/blob/master/random_video/__init__.py', 'https://github.com/emresvd/random-video/blob/master/random_video/asgi.py', 'https://github.com/emresvd/random-video/blob/master/random_video/settings.py', 'https://github.com/emresvd/random-video/blob/master/random_video/urls.py', 'https://github.com/emresvd/random-video/blob/master/random_video/wsgi.py', 'https://github.com/emresvd/random-video/blob/master/special_search/car.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/food.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/rocket.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/space.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/travel.txt', 'https://github.com/emresvd/random-video/blob/master/static/favicon.ico', 'https://github.com/emresvd/random-video/blob/master/templates/download.html', 'https://github.com/emresvd/random-video/blob/master/templates/index.html', 'https://github.com/emresvd/random-video/blob/master/video/__init__.py', 'https://github.com/emresvd/random-video/blob/master/video/admin.py', 'https://github.com/emresvd/random-video/blob/master/video/apps.py', 'https://github.com/emresvd/random-video/blob/master/video/models.py', 'https://github.com/emresvd/random-video/blob/master/video/random_video.py', 'https://github.com/emresvd/random-video/blob/master/video/tests.py', 'https://github.com/emresvd/random-video/blob/master/video/views.py', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/__init__.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/settings.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/urls.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/wsgi.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/__init__.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/admin.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/models.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/random_video.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/views.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/migrations/__init__.py', 'https://github.com/emresvd/random-video/blob/master/video/migrations/__pycache__/__init__.cpython-37.pyc']

```

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/emresvd/reposcraping",
    "name": "reposcraping",
    "maintainer": "",
    "docs_url": null,
    "requires_python": "",
    "maintainer_email": "",
    "keywords": "github,scraping,repository,clone",
    "author": "emresvd",
    "author_email": "",
    "download_url": "https://files.pythonhosted.org/packages/3e/1c/11df304708e5be2ab6612dfbd85610f53e60cabc32ad6b0376f01d67b390/reposcraping-1.1.7.tar.gz",
    "platform": null,
    "description": "# [reposcraping](https://pypi.org/project/reposcraping/)\r\n\r\n### Scraping GitHub repository\r\n\r\nThis library allows you to access the names of files and folders of any GitHub repository. Cloner class allows you to clone the file types you want to the path you want.<br>\r\n\r\n## downloads\r\n\r\n[![Downloads](https://static.pepy.tech/badge/reposcraping)](https://pepy.tech/project/reposcraping) [![Downloads](https://static.pepy.tech/badge/reposcraping/month)](https://pepy.tech/project/reposcraping) [![Downloads](https://static.pepy.tech/badge/reposcraping/week)](https://pepy.tech/project/reposcraping)\r\n\r\n## setup\r\n\r\n```bash\r\n\r\npip install reposcraping\r\n\r\n```\r\n\r\n## usage\r\n\r\n```python\r\n\r\nfrom reposcraping import RepoScraping\r\n\r\nfrom reposcraping.cloner import Cloner\r\n\r\n\r\n\r\nscraping = RepoScraping(\r\n\r\n    \"https://github.com/emresvd/random-video\",\r\n\r\n    p=True,\r\n\r\n)\r\n\r\n\r\n\r\nprint(scraping.tree_urls)\r\n\r\nprint(scraping.file_urls)\r\n\r\n\r\n\r\ncloner = Cloner(scraping)\r\n\r\ncloner.clone(\r\n\r\n    paths={\r\n\r\n        \".py\": \"files/python_files\",\r\n\r\n        \".txt\": \"files/text_files\",\r\n\r\n        \".md\": \"files/markdown_files\",\r\n\r\n        \".html\": \"files/html_files\",\r\n\r\n        \"\": \"files/other_files\",\r\n\r\n    },\r\n\r\n    only_file_name=True,\r\n\r\n    p=True,\r\n\r\n)\r\n\r\n```\r\n\r\n## output\r\n\r\n```python\r\n\r\n['https://github.com/emresvd/random-video', 'https://github.com/emresvd/random-video/tree/master/random_video', 'https://github.com/emresvd/random-video/tree/master/special_search', 'https://github.com/emresvd/random-video/tree/master/static', 'https://github.com/emresvd/random-video/tree/master/templates', 'https://github.com/emresvd/random-video/tree/master/video', 'https://github.com/emresvd/random-video/tree/master/random_video/__pycache__', 'https://github.com/emresvd/random-video/tree/master/video/__pycache__', 'https://github.com/emresvd/random-video/tree/master/video/migrations', 'https://github.com/emresvd/random-video/tree/master/video/migrations/__pycache__']\r\n\r\n['https://github.com/emresvd/random-video/blob/master/LICENSE.md', 'https://github.com/emresvd/random-video/blob/master/.gitignore', 'https://github.com/emresvd/random-video/blob/master/README.md', 'https://github.com/emresvd/random-video/blob/master/db.sqlite3', 'https://github.com/emresvd/random-video/blob/master/manage.py', 'https://github.com/emresvd/random-video/blob/master/requirements.txt', 'https://github.com/emresvd/random-video/blob/master/words.txt', 'https://github.com/emresvd/random-video/blob/master/static/ic_launcher-playstore.png', 'https://github.com/emresvd/random-video/blob/master/random_video/__init__.py', 'https://github.com/emresvd/random-video/blob/master/random_video/asgi.py', 'https://github.com/emresvd/random-video/blob/master/random_video/settings.py', 'https://github.com/emresvd/random-video/blob/master/random_video/urls.py', 'https://github.com/emresvd/random-video/blob/master/random_video/wsgi.py', 'https://github.com/emresvd/random-video/blob/master/special_search/car.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/food.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/rocket.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/space.txt', 'https://github.com/emresvd/random-video/blob/master/special_search/travel.txt', 'https://github.com/emresvd/random-video/blob/master/static/favicon.ico', 'https://github.com/emresvd/random-video/blob/master/templates/download.html', 'https://github.com/emresvd/random-video/blob/master/templates/index.html', 'https://github.com/emresvd/random-video/blob/master/video/__init__.py', 'https://github.com/emresvd/random-video/blob/master/video/admin.py', 'https://github.com/emresvd/random-video/blob/master/video/apps.py', 'https://github.com/emresvd/random-video/blob/master/video/models.py', 'https://github.com/emresvd/random-video/blob/master/video/random_video.py', 'https://github.com/emresvd/random-video/blob/master/video/tests.py', 'https://github.com/emresvd/random-video/blob/master/video/views.py', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/__init__.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/settings.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/urls.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/random_video/__pycache__/wsgi.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/__init__.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/admin.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/models.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/random_video.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/__pycache__/views.cpython-37.pyc', 'https://github.com/emresvd/random-video/blob/master/video/migrations/__init__.py', 'https://github.com/emresvd/random-video/blob/master/video/migrations/__pycache__/__init__.cpython-37.pyc']\r\n\r\n```\r\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Scraping GitHub repository",
    "version": "1.1.7",
    "split_keywords": [
        "github",
        "scraping",
        "repository",
        "clone"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "6074e020a8637e84c2336db3d4bf63e9dbe19170ae809362a1d99515a7c6e210",
                "md5": "f9bb93394d2d7221e08c8e791db65328",
                "sha256": "4f515b6c2a9673fd87f6ab64501c5ed6a63359e2692c486947715cfe997042ed"
            },
            "downloads": -1,
            "filename": "reposcraping-1.1.7-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "f9bb93394d2d7221e08c8e791db65328",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 4342,
            "upload_time": "2023-01-29T14:08:10",
            "upload_time_iso_8601": "2023-01-29T14:08:10.968285Z",
            "url": "https://files.pythonhosted.org/packages/60/74/e020a8637e84c2336db3d4bf63e9dbe19170ae809362a1d99515a7c6e210/reposcraping-1.1.7-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "3e1c11df304708e5be2ab6612dfbd85610f53e60cabc32ad6b0376f01d67b390",
                "md5": "51b63589a63a394da38400f8660faf98",
                "sha256": "3ef3ba2675594ec9ad0f0a6748f027edf016eae75c95d1407306aee173675a48"
            },
            "downloads": -1,
            "filename": "reposcraping-1.1.7.tar.gz",
            "has_sig": false,
            "md5_digest": "51b63589a63a394da38400f8660faf98",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 3795,
            "upload_time": "2023-01-29T14:08:13",
            "upload_time_iso_8601": "2023-01-29T14:08:13.054074Z",
            "url": "https://files.pythonhosted.org/packages/3e/1c/11df304708e5be2ab6612dfbd85610f53e60cabc32ad6b0376f01d67b390/reposcraping-1.1.7.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-01-29 14:08:13",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "emresvd",
    "github_project": "reposcraping",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": false,
    "requirements": [],
    "lcname": "reposcraping"
}
emresvd
Elapsed time: 0.10119s