š Introducing āØ Bose Framework - The Swiss Army Knife for Bot Developers š¤
=============================================================================
Bot Development is Tough.
Bot Detectors like Cloudflare are ready to defend websites from our
Bots. Configuring Selenium with ChromeOptions to specify the driver
path, profile, user agent, and window size is Cumbersome and a nightmare
in windows. Debugging Bot Crashes via logs is hard. How do you solve
these pain points without sacrificing speed and handy development?
Enter Bose. Bose is the first bot development framework in the Developer
Community that is specifically designed to provide the best developer
experience for bot developers. Powered by Selenium, it offers a range of
features and functionalities to simplify the process of bot development.
As far as our knowledge goes, Bose is the first bot development
framework of its kind in the town.
Getting Started
---------------
Clone Starter Template
::
git clone https://github.com/omkarcloud/bose-starter my-bose-project
Then change into that directory, install dependencies, and start the
project:
::
cd my-bose-project
python -m pip install -r requirements.txt
python main.py
The first run will take some time as it downloads the chrome driver
executable, subsequent runs will be fast.
Core features
-------------
1. Adds Powerful Methods to make working with Selenium a lot easier.
2. Follows best practices to avoid Bot Detection by Cloudflare and
PerimeterX.
3. Saves the HTML, Screenshot, and the run details for each task run to
enable easy debugging.
4. Utility components to write scraped data as JSON, CSV, and Excel
files.
5. Automatically downloads and initializes the correct Chrome driver.
6. Fast and Developer friendly.
Usage
-----
Say you want to start scraping a website. If you were using bare
Selenium, you would have to handle the imperative tasks of opening and
closing the driver like this:
.. code:: python
from selenium import webdriver
driver_path = 'path/to/chromedriver'
driver = webdriver.Chrome(executable_path=driver_path)
driver.get('https://www.example.com')
driver.quit()
However, with Bose Framework, you can take a declarative and structured
approach. You only need to write the following code, and Bose driver
will take care of creating the driver, passing it to the **``run``**
method of the Task, and closing the driver:
.. code:: python
from bose import *
class Task(BaseTask):
def run(self, driver):
driver.get('https://www.example.com')
Configuration
-------------
In bare Selenium, if you want to configure options such as the profile,
user agent, or window size, it requires writing a lot of code, as shown
below:
.. code:: python
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
driver_path = 'path/to/chromedriver.exe'
options = Options()
profile_path = '1'
options.add_argument(f'--user-data-dir={profile_path}')
user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37")'
options.add_argument(f'--user-agent={user_agent}')
window_width = 1200
window_height = 720
options.add_argument(f'--window-size={window_width},{window_height}')
driver = webdriver.Chrome(executable_path=driver_path, options=options)
On the other hand, Bose Framework simplifies these complexities by
encapsulating the browser configuration within the **``BrowserConfig``**
property of the Task, as shown below:
.. code:: python
from bose import BaseTask, BrowserConfig, UserAgent, WindowSize
class Task(BaseTask):
browser_config = BrowserConfig(user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1)
Exception handling
------------------
Exceptions are common when using Selenium. In bare Selenium, if an
exception occurs, the driver automatically closes, leaving you with only
logs to debug.
In Bose, when an exception occurs in a scraping task, the browser
remains open instead of immediately closing. This allows you to see the
live browser state at the moment the exception occurred, which greatly
helps in debugging.
.. figure:: https://www.omkar.cloud/bose/assets/images/error-prompt-83de79e560f129197afb9f831d388383.png
Debugging
---------
Web scraping can often be fraught with errors, such as incorrect
selectors or pages that fail to load. When debugging with raw Selenium,
you may have to sift through logs to identify the issue. Fortunately,
Bose makes it simple for you to debug by storing information about each
run.
After each run a directory is created in tasks which contains three
files, which are listed below:
``task_info.json``
~~~~~~~~~~~~~~~~~~
It contains information about the task run such as duration for which
the task run, the ip details of task, the user agent, window_size and
profile which used to execute the task.
.. figure:: https://www.omkar.cloud/bose/assets/images/task-info-1ad8d89552138e2edc900434144dfbe0.png
``final.png``
~~~~~~~~~~~~~
This is the screenshot captured before driver was closed.
.. figure:: https://www.omkar.cloud/bose/assets/images/final-d2ca24d2717d17576eb8233ad0cd2b10.png
``page.html``
~~~~~~~~~~~~~
This is the html source captured before driver was closed. Very useful
to know in case your selectors failed to select elements.
.. figure:: https://www.omkar.cloud/bose/assets/images/page-cffce10976b4bf201b49a479c2340075.png
``error.log``
~~~~~~~~~~~~~
In case your task crashed due to exception we also store error.log which
contains the error due to which the task crashed. This is very helful in
debugging.
.. figure:: https://www.omkar.cloud/bose/assets/images/error-log-9ebb09dca133b2d7df1ae6cfc67df909.png
Outputting Data
---------------
After performing web scraping, we need to store the data in either JSON
or CSV format. Typically, this process involves writing a significant
amount of imperative code which looks like this:
.. code:: python
import csv
import json
def write_json(data, filename):
with open(filename, 'w') as fp:
json.dump(data, fp, indent=4)
def write_csv(data, filename):
with open(filename, 'w', newline='', encoding='utf-8') as csvfile:
fieldnames = data[0].keys() # get the fieldnames from the first dictionary
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writeheader() # write the header row
writer.writerows(data) # write each row of data
data = [
{
"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
"author": "Albert Einstein"
},
{
"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
"author": "J.K. Rowling"
}
]
write_json(data, "data.json")
write_csv(data, "data.csv")
Bose simplifies these complexities by encapsulating them in Output
Module for reading and writing Data.
To use Output Method, call theĀ ``write``Ā method for the type of file you
want to save.
All data will be saved in theĀ ``output/``Ā folder:
See following Code for Reference
.. code:: python
from bose import Output
data = [
{
"text": "\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\u201d",
"author": "Albert Einstein"
},
{
"text": "\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\u201d",
"author": "J.K. Rowling"
}
]
Output.write_json(data, "data.json")
Output.write_csv(data, "data.csv")
Undetected Driver
-----------------
`Ultrafunkamsterdam <https://github.com/ultrafunkamsterdam>`__Ā created
aĀ `ChromeDriver <https://github.com/ultrafunkamsterdam/undetected-chromedriver>`__Ā that
has excellent support for bypassing **all major bot detection
systems**Ā such as Distil, Datadome, Cloudflare, and others.
Bose recognized the importance of bypassing bot detections and provides
in built support for
`Ultrafunkamsterdamās <https://github.com/ultrafunkamsterdam>`__
`Undetected
Driver <https://github.com/ultrafunkamsterdam/undetected-chromedriver>`__
Using the Undetected Driver in Bose Framework is as simple as passing
the **``use_undetected_driver``** option to the **``BrowserConfig``**,
like so:
.. code:: python
from bose import BaseTask, BrowserConfig
class Task(BaseTask):
browser_config = BrowserConfig(use_undetected_driver=True)
LocalStorage
------------
Just like how modern browsers have a local storage module, Bose has also
incorporated the same concept in its framework.
You can import the LocalStorage object from Bose to persist data across
browser runs, which is extremely useful when scraping large amounts of
data.
The data is stored in a file namedĀ ``local_storage.json``Ā in the root
directory of your project. Hereās how you can use it:
.. code:: python
from bose import LocalStorage
LocalStorage.set_item("pages", 5)
print(LocalStorage.get_item("pages"))
Learn More
----------
To learn about Bose Bot Development Framework in detail, read the Bose
docs at https://www.omkar.cloud/bose/
--------------
If Bose Framework helped in Bot Development, please take a moment to `star the repository <https://github.com/omkarcloud/bose>`__. Your act of starring will help developers in discovering our Repository and contribute towards helping fellow developers in Bot Development. Dhanyawad š! Vande Mataram!
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Raw data
{
"_id": null,
"home_page": "",
"name": "bose",
"maintainer": "Chetan Jain",
"docs_url": null,
"requires_python": ">=3.7",
"maintainer_email": "chetan@omkar.cloud",
"keywords": "crawler,framework,scraping,crawling,web-scraping,web-scraping-python,cloudflare-bypass,anti-detection,bot-detection,automation,webdriver,browser",
"author": "Chetan Jain",
"author_email": "chetan@omkar.cloud",
"download_url": "https://files.pythonhosted.org/packages/0b/13/28b706a99e0696ef6347e3d9b5dbb384d0e5e45a52519e462665b4f9ad1f/bose-2.0.22.tar.gz",
"platform": null,
"description": "\ud83d\ude80 Introducing \u2728 Bose Framework - The Swiss Army Knife for Bot Developers \ud83e\udd16\r\n=============================================================================\r\n\r\nBot Development is Tough.\r\n\r\nBot Detectors like Cloudflare are ready to defend websites from our\r\nBots. Configuring Selenium with ChromeOptions to specify the driver\r\npath, profile, user agent, and window size is Cumbersome and a nightmare\r\nin windows. Debugging Bot Crashes via logs is hard. How do you solve\r\nthese pain points without sacrificing speed and handy development?\r\n\r\nEnter Bose. Bose is the first bot development framework in the Developer\r\nCommunity that is specifically designed to provide the best developer\r\nexperience for bot developers. Powered by Selenium, it offers a range of\r\nfeatures and functionalities to simplify the process of bot development.\r\nAs far as our knowledge goes, Bose is the first bot development\r\nframework of its kind in the town.\r\n\r\nGetting Started\r\n---------------\r\n\r\nClone Starter Template\r\n\r\n::\r\n\r\n git clone https://github.com/omkarcloud/bose-starter my-bose-project\r\n\r\nThen change into that directory, install dependencies, and start the\r\nproject:\r\n\r\n::\r\n\r\n cd my-bose-project\r\n python -m pip install -r requirements.txt\r\n python main.py\r\n\r\nThe first run will take some time as it downloads the chrome driver\r\nexecutable, subsequent runs will be fast.\r\n\r\nCore features\r\n-------------\r\n\r\n1. Adds Powerful Methods to make working with Selenium a lot easier.\r\n2. Follows best practices to avoid Bot Detection by Cloudflare and\r\n PerimeterX.\r\n3. Saves the HTML, Screenshot, and the run details for each task run to\r\n enable easy debugging.\r\n4. Utility components to write scraped data as JSON, CSV, and Excel\r\n files.\r\n5. Automatically downloads and initializes the correct Chrome driver.\r\n6. Fast and Developer friendly.\r\n\r\nUsage\r\n-----\r\n\r\nSay you want to start scraping a website. If you were using bare\r\nSelenium, you would have to handle the imperative tasks of opening and\r\nclosing the driver like this:\r\n\r\n.. code:: python\r\n\r\n from selenium import webdriver\r\n\r\n driver_path = 'path/to/chromedriver'\r\n\r\n driver = webdriver.Chrome(executable_path=driver_path)\r\n\r\n driver.get('https://www.example.com')\r\n\r\n driver.quit()\r\n\r\nHowever, with Bose Framework, you can take a declarative and structured\r\napproach. You only need to write the following code, and Bose driver\r\nwill take care of creating the driver, passing it to the **``run``**\r\nmethod of the Task, and closing the driver:\r\n\r\n.. code:: python\r\n\r\n from bose import *\r\n \r\n class Task(BaseTask):\r\n def run(self, driver):\r\n driver.get('https://www.example.com')\r\n\r\nConfiguration\r\n-------------\r\n\r\nIn bare Selenium, if you want to configure options such as the profile,\r\nuser agent, or window size, it requires writing a lot of code, as shown\r\nbelow:\r\n\r\n.. code:: python\r\n\r\n from selenium.webdriver.chrome.options import Options\r\n from selenium import webdriver\r\n\r\n driver_path = 'path/to/chromedriver.exe'\r\n\r\n options = Options()\r\n\r\n profile_path = '1'\r\n\r\n options.add_argument(f'--user-data-dir={profile_path}')\r\n\r\n user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.37\")'\r\n options.add_argument(f'--user-agent={user_agent}')\r\n\r\n window_width = 1200\r\n window_height = 720\r\n options.add_argument(f'--window-size={window_width},{window_height}')\r\n\r\n driver = webdriver.Chrome(executable_path=driver_path, options=options)\r\n\r\nOn the other hand, Bose Framework simplifies these complexities by\r\nencapsulating the browser configuration within the **``BrowserConfig``**\r\nproperty of the Task, as shown below:\r\n\r\n.. code:: python\r\n\r\n from bose import BaseTask, BrowserConfig, UserAgent, WindowSize\r\n\r\n class Task(BaseTask):\r\n browser_config = BrowserConfig(user_agent=UserAgent.user_agent_106, window_size=WindowSize.window_size_1280_720, profile=1)\r\n\r\nException handling\r\n------------------\r\n\r\nExceptions are common when using Selenium. In bare Selenium, if an\r\nexception occurs, the driver automatically closes, leaving you with only\r\nlogs to debug.\r\n\r\nIn Bose, when an exception occurs in a scraping task, the browser\r\nremains open instead of immediately closing. This allows you to see the\r\nlive browser state at the moment the exception occurred, which greatly\r\nhelps in debugging.\r\n\r\n.. figure:: https://www.omkar.cloud/bose/assets/images/error-prompt-83de79e560f129197afb9f831d388383.png\r\n\r\nDebugging\r\n---------\r\n\r\nWeb scraping can often be fraught with errors, such as incorrect\r\nselectors or pages that fail to load. When debugging with raw Selenium,\r\nyou may have to sift through logs to identify the issue. Fortunately,\r\nBose makes it simple for you to debug by storing information about each\r\nrun.\r\n\r\nAfter each run a directory is created in tasks which contains three\r\nfiles, which are listed below:\r\n\r\n``task_info.json``\r\n~~~~~~~~~~~~~~~~~~\r\n\r\nIt contains information about the task run such as duration for which\r\nthe task run, the ip details of task, the user agent, window_size and\r\nprofile which used to execute the task.\r\n\r\n.. figure:: https://www.omkar.cloud/bose/assets/images/task-info-1ad8d89552138e2edc900434144dfbe0.png\r\n\r\n``final.png``\r\n~~~~~~~~~~~~~\r\n\r\nThis is the screenshot captured before driver was closed.\r\n\r\n.. figure:: https://www.omkar.cloud/bose/assets/images/final-d2ca24d2717d17576eb8233ad0cd2b10.png\r\n\r\n``page.html``\r\n~~~~~~~~~~~~~\r\n\r\nThis is the html source captured before driver was closed. Very useful\r\nto know in case your selectors failed to select elements.\r\n\r\n.. figure:: https://www.omkar.cloud/bose/assets/images/page-cffce10976b4bf201b49a479c2340075.png\r\n\r\n``error.log``\r\n~~~~~~~~~~~~~\r\n\r\nIn case your task crashed due to exception we also store error.log which\r\ncontains the error due to which the task crashed. This is very helful in\r\ndebugging.\r\n\r\n.. figure:: https://www.omkar.cloud/bose/assets/images/error-log-9ebb09dca133b2d7df1ae6cfc67df909.png\r\n\r\nOutputting Data\r\n---------------\r\n\r\nAfter performing web scraping, we need to store the data in either JSON\r\nor CSV format. Typically, this process involves writing a significant\r\namount of imperative code which looks like this:\r\n\r\n.. code:: python\r\n\r\n import csv\r\n import json\r\n\r\n def write_json(data, filename):\r\n with open(filename, 'w') as fp:\r\n json.dump(data, fp, indent=4)\r\n\r\n def write_csv(data, filename):\r\n with open(filename, 'w', newline='', encoding='utf-8') as csvfile:\r\n fieldnames = data[0].keys() # get the fieldnames from the first dictionary\r\n writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\r\n writer.writeheader() # write the header row\r\n writer.writerows(data) # write each row of data\r\n\r\n data = [\r\n {\r\n \"text\": \"\\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\\u201d\",\r\n \"author\": \"Albert Einstein\"\r\n },\r\n {\r\n \"text\": \"\\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\\u201d\",\r\n \"author\": \"J.K. Rowling\"\r\n }\r\n ]\r\n\r\n write_json(data, \"data.json\")\r\n write_csv(data, \"data.csv\")\r\n\r\nBose simplifies these complexities by encapsulating them in Output\r\nModule for reading and writing Data.\r\n\r\nTo use Output Method, call the\u00a0``write``\u00a0method for the type of file you\r\nwant to save.\r\n\r\nAll data will be saved in the\u00a0``output/``\u00a0folder:\r\n\r\nSee following Code for Reference\r\n\r\n.. code:: python\r\n\r\n from bose import Output\r\n\r\n data = [\r\n {\r\n \"text\": \"\\u201cThe world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.\\u201d\",\r\n \"author\": \"Albert Einstein\"\r\n },\r\n {\r\n \"text\": \"\\u201cIt is our choices, Harry, that show what we truly are, far more than our abilities.\\u201d\",\r\n \"author\": \"J.K. Rowling\"\r\n }\r\n ]\r\n\r\n Output.write_json(data, \"data.json\")\r\n Output.write_csv(data, \"data.csv\")\r\n\r\nUndetected Driver\r\n-----------------\r\n\r\n`Ultrafunkamsterdam <https://github.com/ultrafunkamsterdam>`__\u00a0created\r\na\u00a0`ChromeDriver <https://github.com/ultrafunkamsterdam/undetected-chromedriver>`__\u00a0that\r\nhas excellent support for bypassing **all major bot detection\r\nsystems**\u00a0such as Distil, Datadome, Cloudflare, and others.\r\n\r\nBose recognized the importance of bypassing bot detections and provides\r\nin built support for\r\n`Ultrafunkamsterdam\u2019s <https://github.com/ultrafunkamsterdam>`__\r\n`Undetected\r\nDriver <https://github.com/ultrafunkamsterdam/undetected-chromedriver>`__\r\n\r\nUsing the Undetected Driver in Bose Framework is as simple as passing\r\nthe **``use_undetected_driver``** option to the **``BrowserConfig``**,\r\nlike so:\r\n\r\n.. code:: python\r\n\r\n from bose import BaseTask, BrowserConfig\r\n\r\n class Task(BaseTask):\r\n browser_config = BrowserConfig(use_undetected_driver=True)\r\n\r\nLocalStorage\r\n------------\r\n\r\nJust like how modern browsers have a local storage module, Bose has also\r\nincorporated the same concept in its framework.\r\n\r\nYou can import the LocalStorage object from Bose to persist data across\r\nbrowser runs, which is extremely useful when scraping large amounts of\r\ndata.\r\n\r\nThe data is stored in a file named\u00a0``local_storage.json``\u00a0in the root\r\ndirectory of your project. Here\u2019s how you can use it:\r\n\r\n.. code:: python\r\n\r\n from bose import LocalStorage\r\n\r\n LocalStorage.set_item(\"pages\", 5)\r\n print(LocalStorage.get_item(\"pages\"))\r\n\r\nLearn More\r\n----------\r\n\r\nTo learn about Bose Bot Development Framework in detail, read the Bose\r\ndocs at https://www.omkar.cloud/bose/\r\n\r\n--------------\r\n\r\nIf Bose Framework helped in Bot Development, please take a moment to `star the repository <https://github.com/omkarcloud/bose>`__. Your act of starring will help developers in discovering our Repository and contribute towards helping fellow developers in Bot Development. Dhanyawad \ud83d\ude4f! Vande Mataram!\r\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n",
"bugtrack_url": null,
"license": "MIT",
"summary": "The Ultimate Web Scraping Framework",
"version": "2.0.22",
"project_urls": {
"Documentation": "https://omkar.cloud/bose/",
"Source": "https://github.com/omkarcloud/bose",
"Tracker": "https://github.com/omkarcloud/bose/issues"
},
"split_keywords": [
"crawler",
"framework",
"scraping",
"crawling",
"web-scraping",
"web-scraping-python",
"cloudflare-bypass",
"anti-detection",
"bot-detection",
"automation",
"webdriver",
"browser"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "0b1328b706a99e0696ef6347e3d9b5dbb384d0e5e45a52519e462665b4f9ad1f",
"md5": "07dd7168fe6c0aa543e0efbee316450c",
"sha256": "ce2a8e0e1564c6998feca3d6a7a0211e53712d6ce8473b461761f1ceb342f2a0"
},
"downloads": -1,
"filename": "bose-2.0.22.tar.gz",
"has_sig": false,
"md5_digest": "07dd7168fe6c0aa543e0efbee316450c",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 31651,
"upload_time": "2023-12-11T16:19:44",
"upload_time_iso_8601": "2023-12-11T16:19:44.094049Z",
"url": "https://files.pythonhosted.org/packages/0b/13/28b706a99e0696ef6347e3d9b5dbb384d0e5e45a52519e462665b4f9ad1f/bose-2.0.22.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2023-12-11 16:19:44",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "omkarcloud",
"github_project": "bose",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "bose"
}
JSON
