google-search-results


Namegoogle-search-results JSON
Version 2.4.2 PyPI version JSON
download
home_pagehttps://github.com/serpapi/google-search-results-python
SummaryScrape and search localized results from Google, Bing, Baidu, Yahoo, Yandex, Ebay, Homedepot, youtube at scale using SerpApi.com
upload_time2023-03-10 11:13:09
maintainer
docs_urlNone
authorvikoky
requires_python>=3.5
licenseMIT
keywords scrape serp api json search localized rank google bing baidu yandex yahoo ebay scale datamining training machine ml youtube naver walmart apple store app
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
coveralls test coverage No coveralls. 
            # Google Search Results in Python

[![Package](https://badge.fury.io/py/google-search-results.svg)](https://badge.fury.io/py/google-search-results)
[![Build](https://github.com/serpapi/google-search-results-python/actions/workflows/python-package.yml/badge.svg)](https://github.com/serpapi/google-search-results-python/actions/workflows/python-package.yml)

This Python package is meant to scrape and parse search results from Google, Bing, Baidu, Yandex, Yahoo, Home Depot, eBay and more, using [SerpApi](https://serpapi.com). 

The following services are provided:
- [Search API](https://serpapi.com/search-api)
- [Search Archive API](https://serpapi.com/search-archive-api)
- [Account API](https://serpapi.com/account-api)
- [Location API](https://serpapi.com/locations-api) (Google Only)

SerpApi provides a [script builder](https://serpapi.com/demo) to get you started quickly.

## Installation

Python 3.7+
```bash
pip install google-search-results
```

[Link to the python package page](https://pypi.org/project/google-search-results/)

## Quick start

```python
from serpapi import GoogleSearch
search = GoogleSearch({
    "q": "coffee", 
    "location": "Austin,Texas",
    "api_key": "<your secret api key>"
  })
result = search.get_dict()
```

This example runs a search for "coffee" using your secret API key.

The SerpApi service (backend)
- Searches Google using the search: q = "coffee"
- Parses the messy HTML responses
- Returns a standardized JSON response
The GoogleSearch class
- Formats the request
- Executes a GET http request against SerpApi service
- Parses the JSON response into a dictionary

Et voilà...

Alternatively, you can search:
- Bing using BingSearch class
- Baidu using BaiduSearch class
- Yahoo using YahooSearch class
- DuckDuckGo using DuckDuckGoSearch class
- eBay using EbaySearch class
- Yandex using YandexSearch class
- HomeDepot using HomeDepotSearch class
- GoogleScholar using GoogleScholarSearch class
- Youtube using YoutubeSearch class
- Walmart using WalmartSearch
- Apple App Store using AppleAppStoreSearch class
- Naver using NaverSearch class


See the [playground to generate your code.](https://serpapi.com/playground)

## Summary
- [Google Search Results in Python](#google-search-results-in-python)
  - [Installation](#installation)
  - [Quick start](#quick-start)
  - [Summary](#summary)
    - [Google Search API capability](#google-search-api-capability)
    - [How to set SERP API key](#how-to-set-serp-api-key)
    - [Example by specification](#example-by-specification)
    - [Location API](#location-api)
    - [Search Archive API](#search-archive-api)
    - [Account API](#account-api)
    - [Search Bing](#search-bing)
    - [Search Baidu](#search-baidu)
    - [Search Yandex](#search-yandex)
    - [Search Yahoo](#search-yahoo)
    - [Search Ebay](#search-ebay)
    - [Search Home depot](#search-home-depot)
    - [Search Youtube](#search-youtube)
    - [Search Google Scholar](#search-google-scholar)
    - [Generic search with SerpApiClient](#generic-search-with-serpapiclient)
    - [Search Google Images](#search-google-images)
    - [Search Google News](#search-google-news)
    - [Search Google Shopping](#search-google-shopping)
    - [Google Search By Location](#google-search-by-location)
    - [Batch Asynchronous Searches](#batch-asynchronous-searches)
    - [Python object as a result](#python-object-as-a-result)
    - [Python paginate using iterator](#pagination-using-iterator)
    - [Error management](#error-management)
  - [Change log](#change-log)
  - [Conclusion](#conclusion)

### Google Search API capability
Source code.
```python
params = {
  "q": "coffee",
  "location": "Location Requested", 
  "device": "desktop|mobile|tablet",
  "hl": "Google UI Language",
  "gl": "Google Country",
  "safe": "Safe Search Flag",
  "num": "Number of Results",
  "start": "Pagination Offset",
  "api_key": "Your SERP API Key", 
  # To be match
  "tbm": "nws|isch|shop", 
  # To be search
  "tbs": "custom to be search criteria",
  # allow async request
  "async": "true|false",
  # output format
  "output": "json|html"
}

# define the search search
search = GoogleSearch(params)
# override an existing parameter
search.params_dict["location"] = "Portland"
# search format return as raw html
html_results = search.get_html()
# parse results
#  as python Dictionary
dict_results = search.get_dict()
#  as JSON using json package
json_results = search.get_json()
#  as dynamic Python object
object_result = search.get_object()
```
[Link to the full documentation](https://serpapi.com/search-api)

See below for more hands-on examples.

### How to set SERP API key

You can get an API key here if you don't already have one: https://serpapi.com/users/sign_up

The SerpApi `api_key` can be set globally:
```python
GoogleSearch.SERP_API_KEY = "Your Private Key"
```
The SerpApi `api_key` can be provided for each search:
```python
query = GoogleSearch({"q": "coffee", "serp_api_key": "Your Private Key"})
```

### Example by specification

We love true open source, continuous integration and Test Driven Development (TDD). 
 We are using RSpec to test [our infrastructure around the clock](https://travis-ci.org/serpapi/google-search-results-python) to achieve the best Quality of Service (QoS).
 
The directory test/ includes specification/examples.

Set your API key.
```bash
export API_KEY="your secret key"
```

Run test
```python
make test
```

### Location API

```python
from serpapi import GoogleSearch
search = GoogleSearch({})
location_list = search.get_location("Austin", 3)
print(location_list)
```

This prints the first 3 locations matching Austin (Texas, Texas, Rochester).
```python
[   {   'canonical_name': 'Austin,TX,Texas,United States',
        'country_code': 'US',
        'google_id': 200635,
        'google_parent_id': 21176,
        'gps': [-97.7430608, 30.267153],
        'id': '585069bdee19ad271e9bc072',
        'keys': ['austin', 'tx', 'texas', 'united', 'states'],
        'name': 'Austin, TX',
        'reach': 5560000,
        'target_type': 'DMA Region'},
        ...]
```

### Search Archive API

The search results are stored in a temporary cache.
The previous search can be retrieved from the cache for free.

```python
from serpapi import GoogleSearch
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas"})
search_result = search.get_dictionary()
assert search_result.get("error") == None
search_id = search_result.get("search_metadata").get("id")
print(search_id)
```

Now let's retrieve the previous search from the archive.

```python
archived_search_result = GoogleSearch({}).get_search_archive(search_id, 'json')
print(archived_search_result.get("search_metadata").get("id"))
```
This prints the search result from the archive.

### Account API
```python
from serpapi import GoogleSearch
search = GoogleSearch({})
account = search.get_account()
```
This prints your account information.

### Search Bing
```python
from serpapi import BingSearch
search = BingSearch({"q": "Coffee", "location": "Austin,Texas"})
data = search.get_dict()
```
This code prints Bing search results for coffee as a Dictionary. 

https://serpapi.com/bing-search-api

### Search Baidu
```python
from serpapi import BaiduSearch
search = BaiduSearch({"q": "Coffee"})
data = search.get_dict()
```
This code prints Baidu search results for coffee as a Dictionary. 
https://serpapi.com/baidu-search-api

### Search Yandex
```python
from serpapi import YandexSearch
search = YandexSearch({"text": "Coffee"})
data = search.get_dict()
```
This code prints Yandex search results for coffee as a Dictionary. 

https://serpapi.com/yandex-search-api

### Search Yahoo
```python
from serpapi import YahooSearch
search = YahooSearch({"p": "Coffee"})
data = search.get_dict()
```
This code prints Yahoo search results for coffee as a Dictionary. 

https://serpapi.com/yahoo-search-api


### Search eBay
```python
from serpapi import EbaySearch
search = EbaySearch({"_nkw": "Coffee"})
data = search.get_dict()
```
This code prints eBay search results for coffee as a Dictionary. 

https://serpapi.com/ebay-search-api

### Search Home Depot
```python
from serpapi import HomeDepotSearch
search = HomeDepotSearch({"q": "chair"})
data = search.get_dict()
```
This code prints Home Depot search results for chair as Dictionary. 

https://serpapi.com/home-depot-search-api

### Search Youtube
```python
from serpapi import HomeDepotSearch
search = YoutubeSearch({"q": "chair"})
data = search.get_dict()
```
This code prints Youtube search results for chair as Dictionary. 

https://serpapi.com/youtube-search-api

### Search Google Scholar
```python
from serpapi import GoogleScholarSearch
search = GoogleScholarSearch({"q": "Coffee"})
data = search.get_dict()
```
This code prints Google Scholar search results.

### Search Walmart
```python
from serpapi import WalmartSearch
search = WalmartSearch({"query": "chair"})
data = search.get_dict()
```
This code prints Walmart search results.

### Search Youtube
```python
from serpapi import YoutubeSearch
search = YoutubeSearch({"search_query": "chair"})
data = search.get_dict()
```
This code prints Youtube search results.

### Search Apple App Store
```python
from serpapi import AppleAppStoreSearch
search = AppleAppStoreSearch({"term": "Coffee"})
data = search.get_dict()
```
This code prints Apple App Store search results.

### Search Naver
```python
from serpapi import NaverSearch
search = NaverSearch({"query": "chair"})
data = search.get_dict()
```
This code prints Naver search results.

### Generic search with SerpApiClient
```python
from serpapi import SerpApiClient
query = {"q": "Coffee", "location": "Austin,Texas", "engine": "google"}
search = SerpApiClient(query)
data = search.get_dict()
```
This class enables interaction with any search engine supported by SerpApi.com 

### Search Google Images

```python
from serpapi import GoogleSearch
search = GoogleSearch({"q": "coffe", "tbm": "isch"})
for image_result in search.get_dict()['images_results']:
    link = image_result["original"]
    try:
        print("link: " + link)
        # wget.download(link, '.')
    except:
        pass
```

This code prints all the image links, 
 and downloads the images if you un-comment the line with wget (Linux/OS X tool to download files).

This tutorial covers more ground on this topic.
https://github.com/serpapi/showcase-serpapi-tensorflow-keras-image-training

### Search Google News

```python
from serpapi import GoogleSearch
search = GoogleSearch({
    "q": "coffe",   # search search
    "tbm": "nws",  # news
    "tbs": "qdr:d", # last 24h
    "num": 10
})
for offset in [0,1,2]:
    search.params_dict["start"] = offset * 10
    data = search.get_dict()
    for news_result in data['news_results']:
        print(str(news_result['position'] + offset * 10) + " - " + news_result['title'])
```

This script prints the first 3 pages of the news headlines for the last 24 hours.

### Search Google Shopping

```python
from serpapi import GoogleSearch
search = GoogleSearch({
    "q": "coffe",   # search search
    "tbm": "shop",  # news
    "tbs": "p_ord:rv", # last 24h
    "num": 100
})
data = search.get_dict()
for shopping_result in data['shopping_results']:
    print(shopping_result['position']) + " - " + shopping_result['title'])

```

This script prints all the shopping results, ordered by review order.

### Google Search By Location

With SerpApi, we can build a Google search from anywhere in the world.
This code looks for the best coffee shop for the given cities.

```python
from serpapi import GoogleSearch
for city in ["new york", "paris", "berlin"]:
  location = GoogleSearch({}).get_location(city, 1)[0]["canonical_name"]
  search = GoogleSearch({
      "q": "best coffee shop",   # search search
      "location": location,
      "num": 1,
      "start": 0
  })
  data = search.get_dict()
  top_result = data["organic_results"][0]["title"]
```

### Batch Asynchronous Searches

We offer two ways to boost your searches thanks to the`async` parameter.
 - Blocking - async=false - more compute intensive because the search needs to maintain many connections. (default) 
- Non-blocking - async=true - the way to go for large batches of queries  (recommended)

```python
# Operating system
import os

# regular expression library
import re

# safe queue (named Queue in python2)
from queue import Queue

# Time utility
import time

# SerpApi search
from serpapi import GoogleSearch

# store searches
search_queue = Queue()

# SerpApi search
search = GoogleSearch({
    "location": "Austin,Texas",
    "async": True,
    "api_key": os.getenv("API_KEY")
})

# loop through a list of companies
for company in ['amd', 'nvidia', 'intel']:
    print("execute async search: q = " + company)
    search.params_dict["q"] = company
    result = search.get_dict()
    if "error" in result:
        print("oops error: ", result["error"])
        continue
    print("add search to the queue where id: ", result['search_metadata'])
    # add search to the search_queue
    search_queue.put(result)

print("wait until all search statuses are cached or success")

# Create regular search
while not search_queue.empty():
    result = search_queue.get()
    search_id = result['search_metadata']['id']

    # retrieve search from the archive - blocker
    print(search_id + ": get search from archive")
    search_archived = search.get_search_archive(search_id)
    print(search_id + ": status = " +
          search_archived['search_metadata']['status'])

    # check status
    if re.search('Cached|Success',
                 search_archived['search_metadata']['status']):
        print(search_id + ": search done with q = " +
              search_archived['search_parameters']['q'])
    else:
        # requeue search_queue
        print(search_id + ": requeue search")
        search_queue.put(result)

        # wait 1s
        time.sleep(1)

print('all searches completed')
```

This code shows how to run searches asynchronously.
The search parameters must have {async: True}. This indicates that the client shouldn't wait for the search to be completed.
The current thread that executes the search is now non-blocking, which allows it to execute thousands of searches in seconds. The SerpApi backend will do the processing work.
The actual search result is deferred to a later call from the search archive using get_search_archive(search_id).
In this example the non-blocking searches are persisted in a queue: search_queue.
A loop through the search_queue allows it to fetch individual search results.
This process can easily be multithreaded to allow a large number of concurrent search requests.
To keep things simple, this example only explores search results one at a time (single threaded).

[See example.](https://github.com/serpapi/google-search-results-python/blob/master/tests/test_example.py)

### Python object as a result

The search results can be automatically wrapped in dynamically generated Python object.
This solution offers a more dynamic, fully Oriented Object Programming approach over the regular Dictionary / JSON data structure.

```python
from serpapi import GoogleSearch
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas"})
r = search.get_object()
assert type(r.organic_results), list
assert r.organic_results[0].title
assert r.search_metadata.id
assert r.search_metadata.google_url
assert r.search_parameters.q, "Coffee"
assert r.search_parameters.engine, "google"
```

### Pagination using iterator
Let's collect links across multiple search results pages.
```python
# to get 2 pages
start = 0
end = 40
page_size = 10

# basic search parameters
parameter = {
  "q": "coca cola",
  "tbm": "nws",
  "api_key": os.getenv("API_KEY"),
  # optional pagination parameter
  #  the pagination method can take argument directly
  "start": start,
  "end": end,
  "num": page_size
}

# as proof of concept 
# urls collects
urls = []

# initialize a search
search = GoogleSearch(parameter)

# create a python generator using parameter
pages = search.pagination()
# or set custom parameter
pages = search.pagination(start, end, page_size)

# fetch one search result per iteration 
# using a basic python for loop 
# which invokes python iterator under the hood.
for page in pages:
  print(f"Current page: {page['serpapi_pagination']['current']}")
  for news_result in page["news_results"]:
    print(f"Title: {news_result['title']}\nLink: {news_result['link']}\n")
    urls.append(news_result['link'])
  
# check if the total number pages is as expected
# note: the exact number if variable depending on the search engine backend
if len(urls) == (end - start):
  print("all search results count match!")
if len(urls) == len(set(urls)):
  print("all search results are unique!")
```

Examples to fetch links with pagination: [test file](https://github.com/serpapi/google-search-results-python/blob/master/tests/test_example_paginate.py), [online IDE](https://replit.com/@DimitryZub1/Scrape-Google-News-with-Pagination-python-serpapi)

### Error management

SerpApi keeps error management simple.
 - backend service error or search fail
 - client error

If it's a backend error, a simple error message is returned as string in the server response.
```python
from serpapi import GoogleSearch
search = GoogleSearch({"q": "Coffee", "location": "Austin,Texas", "api_key": "<secret_key>"})
data = search.get_json()
assert data["error"] == None
```
In some cases, there are more details available in the data object.

If it's a client error, then a SerpApiClientException is raised.

## Change log
2023-03-10 @ 2.4.2
 - Change long description to README.md

2021-12-22 @ 2.4.1
 - add more search engine 
   - youtube
   - walmart
   - apple_app_store
   - naver 
 - raise SerpApiClientException instead of raw string in order to follow Python guideline 3.5+
 - add more unit error tests for serp_api_client

2021-07-26 @ 2.4.0
 - add page size support using num parameter
 - add youtube search engine

2021-06-05 @ 2.3.0
 - add pagination support

2021-04-28 @ 2.2.0
 - add get_response method to provide raw requests.Response object

2021-04-04 @ 2.1.0
 - Add home depot search engine
 - get_object() returns dynamic Python object
 
2020-10-26 @ 2.0.0
 - Reduce class name to <engine>Search
 - Add get_raw_json

2020-06-30 @ 1.8.3
 - simplify import
 - improve package for python 3.5+
 - add support for python 3.5 and 3.6

2020-03-25 @ 1.8
 - add support for Yandex, Yahoo, Ebay
 - clean-up test

2019-11-10 @ 1.7.1
 - increase engine parameter priority over engine value set in the class

2019-09-12 @ 1.7
 - Change  namespace "from lib." instead: "from serpapi import GoogleSearch"
 - Support for Bing and Baidu

2019-06-25 @ 1.6
 - New search engine supported: Baidu and Bing

## Conclusion
SerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more..
To enable a type of search, the field tbm (to be matched) must be set to:

 * isch: Google Images API.
 * nws: Google News API.
 * shop: Google Shopping API.
 * any other Google service should work out of the box.
 * (no tbm parameter): regular Google search.

The field `tbs` allows to customize the search even more.

[The full documentation is available here.](https://serpapi.com/search-api)

Raw data

            {
    "_id": null,
    "home_page": "https://github.com/serpapi/google-search-results-python",
    "name": "google-search-results",
    "maintainer": "",
    "docs_url": null,
    "requires_python": ">=3.5",
    "maintainer_email": "",
    "keywords": "scrape,serp,api,json,search,localized,rank,google,bing,baidu,yandex,yahoo,ebay,scale,datamining,training,machine,ml,youtube,naver,walmart,apple,store,app",
    "author": "vikoky",
    "author_email": "victor@serpapi.com",
    "download_url": "https://files.pythonhosted.org/packages/77/30/b3a6f6a2e00f8153549c2fa345c58ae1ce8e5f3153c2fe0484d444c3abcb/google_search_results-2.4.2.tar.gz",
    "platform": null,
    "description": "# Google Search Results in Python\n\n[![Package](https://badge.fury.io/py/google-search-results.svg)](https://badge.fury.io/py/google-search-results)\n[![Build](https://github.com/serpapi/google-search-results-python/actions/workflows/python-package.yml/badge.svg)](https://github.com/serpapi/google-search-results-python/actions/workflows/python-package.yml)\n\nThis Python package is meant to scrape and parse search results from Google, Bing, Baidu, Yandex, Yahoo, Home Depot, eBay and more, using [SerpApi](https://serpapi.com). \n\nThe following services are provided:\n- [Search API](https://serpapi.com/search-api)\n- [Search Archive API](https://serpapi.com/search-archive-api)\n- [Account API](https://serpapi.com/account-api)\n- [Location API](https://serpapi.com/locations-api) (Google Only)\n\nSerpApi provides a [script builder](https://serpapi.com/demo) to get you started quickly.\n\n## Installation\n\nPython 3.7+\n```bash\npip install google-search-results\n```\n\n[Link to the python package page](https://pypi.org/project/google-search-results/)\n\n## Quick start\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\n    \"q\": \"coffee\", \n    \"location\": \"Austin,Texas\",\n    \"api_key\": \"<your secret api key>\"\n  })\nresult = search.get_dict()\n```\n\nThis example runs a search for \"coffee\" using your secret API key.\n\nThe SerpApi service (backend)\n- Searches Google using the search: q = \"coffee\"\n- Parses the messy HTML responses\n- Returns a standardized JSON response\nThe GoogleSearch class\n- Formats the request\n- Executes a GET http request against SerpApi service\n- Parses the JSON response into a dictionary\n\nEt voil\u00e0...\n\nAlternatively, you can search:\n- Bing using BingSearch class\n- Baidu using BaiduSearch class\n- Yahoo using YahooSearch class\n- DuckDuckGo using DuckDuckGoSearch class\n- eBay using EbaySearch class\n- Yandex using YandexSearch class\n- HomeDepot using HomeDepotSearch class\n- GoogleScholar using GoogleScholarSearch class\n- Youtube using YoutubeSearch class\n- Walmart using WalmartSearch\n- Apple App Store using AppleAppStoreSearch class\n- Naver using NaverSearch class\n\n\nSee the [playground to generate your code.](https://serpapi.com/playground)\n\n## Summary\n- [Google Search Results in Python](#google-search-results-in-python)\n  - [Installation](#installation)\n  - [Quick start](#quick-start)\n  - [Summary](#summary)\n    - [Google Search API capability](#google-search-api-capability)\n    - [How to set SERP API key](#how-to-set-serp-api-key)\n    - [Example by specification](#example-by-specification)\n    - [Location API](#location-api)\n    - [Search Archive API](#search-archive-api)\n    - [Account API](#account-api)\n    - [Search Bing](#search-bing)\n    - [Search Baidu](#search-baidu)\n    - [Search Yandex](#search-yandex)\n    - [Search Yahoo](#search-yahoo)\n    - [Search Ebay](#search-ebay)\n    - [Search Home depot](#search-home-depot)\n    - [Search Youtube](#search-youtube)\n    - [Search Google Scholar](#search-google-scholar)\n    - [Generic search with SerpApiClient](#generic-search-with-serpapiclient)\n    - [Search Google Images](#search-google-images)\n    - [Search Google News](#search-google-news)\n    - [Search Google Shopping](#search-google-shopping)\n    - [Google Search By Location](#google-search-by-location)\n    - [Batch Asynchronous Searches](#batch-asynchronous-searches)\n    - [Python object as a result](#python-object-as-a-result)\n    - [Python paginate using iterator](#pagination-using-iterator)\n    - [Error management](#error-management)\n  - [Change log](#change-log)\n  - [Conclusion](#conclusion)\n\n### Google Search API capability\nSource code.\n```python\nparams = {\n  \"q\": \"coffee\",\n  \"location\": \"Location Requested\", \n  \"device\": \"desktop|mobile|tablet\",\n  \"hl\": \"Google UI Language\",\n  \"gl\": \"Google Country\",\n  \"safe\": \"Safe Search Flag\",\n  \"num\": \"Number of Results\",\n  \"start\": \"Pagination Offset\",\n  \"api_key\": \"Your SERP API Key\", \n  # To be match\n  \"tbm\": \"nws|isch|shop\", \n  # To be search\n  \"tbs\": \"custom to be search criteria\",\n  # allow async request\n  \"async\": \"true|false\",\n  # output format\n  \"output\": \"json|html\"\n}\n\n# define the search search\nsearch = GoogleSearch(params)\n# override an existing parameter\nsearch.params_dict[\"location\"] = \"Portland\"\n# search format return as raw html\nhtml_results = search.get_html()\n# parse results\n#  as python Dictionary\ndict_results = search.get_dict()\n#  as JSON using json package\njson_results = search.get_json()\n#  as dynamic Python object\nobject_result = search.get_object()\n```\n[Link to the full documentation](https://serpapi.com/search-api)\n\nSee below for more hands-on examples.\n\n### How to set SERP API key\n\nYou can get an API key here if you don't already have one: https://serpapi.com/users/sign_up\n\nThe SerpApi `api_key` can be set globally:\n```python\nGoogleSearch.SERP_API_KEY = \"Your Private Key\"\n```\nThe SerpApi `api_key` can be provided for each search:\n```python\nquery = GoogleSearch({\"q\": \"coffee\", \"serp_api_key\": \"Your Private Key\"})\n```\n\n### Example by specification\n\nWe love true open source, continuous integration and Test Driven Development (TDD). \n We are using RSpec to test [our infrastructure around the clock](https://travis-ci.org/serpapi/google-search-results-python) to achieve the best Quality of Service (QoS).\n \nThe directory test/ includes specification/examples.\n\nSet your API key.\n```bash\nexport API_KEY=\"your secret key\"\n```\n\nRun test\n```python\nmake test\n```\n\n### Location API\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({})\nlocation_list = search.get_location(\"Austin\", 3)\nprint(location_list)\n```\n\nThis prints the first 3 locations matching Austin (Texas, Texas, Rochester).\n```python\n[   {   'canonical_name': 'Austin,TX,Texas,United States',\n        'country_code': 'US',\n        'google_id': 200635,\n        'google_parent_id': 21176,\n        'gps': [-97.7430608, 30.267153],\n        'id': '585069bdee19ad271e9bc072',\n        'keys': ['austin', 'tx', 'texas', 'united', 'states'],\n        'name': 'Austin, TX',\n        'reach': 5560000,\n        'target_type': 'DMA Region'},\n        ...]\n```\n\n### Search Archive API\n\nThe search results are stored in a temporary cache.\nThe previous search can be retrieved from the cache for free.\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\"q\": \"Coffee\", \"location\": \"Austin,Texas\"})\nsearch_result = search.get_dictionary()\nassert search_result.get(\"error\") == None\nsearch_id = search_result.get(\"search_metadata\").get(\"id\")\nprint(search_id)\n```\n\nNow let's retrieve the previous search from the archive.\n\n```python\narchived_search_result = GoogleSearch({}).get_search_archive(search_id, 'json')\nprint(archived_search_result.get(\"search_metadata\").get(\"id\"))\n```\nThis prints the search result from the archive.\n\n### Account API\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({})\naccount = search.get_account()\n```\nThis prints your account information.\n\n### Search Bing\n```python\nfrom serpapi import BingSearch\nsearch = BingSearch({\"q\": \"Coffee\", \"location\": \"Austin,Texas\"})\ndata = search.get_dict()\n```\nThis code prints Bing search results for coffee as a Dictionary. \n\nhttps://serpapi.com/bing-search-api\n\n### Search Baidu\n```python\nfrom serpapi import BaiduSearch\nsearch = BaiduSearch({\"q\": \"Coffee\"})\ndata = search.get_dict()\n```\nThis code prints Baidu search results for coffee as a Dictionary. \nhttps://serpapi.com/baidu-search-api\n\n### Search Yandex\n```python\nfrom serpapi import YandexSearch\nsearch = YandexSearch({\"text\": \"Coffee\"})\ndata = search.get_dict()\n```\nThis code prints Yandex search results for coffee as a Dictionary. \n\nhttps://serpapi.com/yandex-search-api\n\n### Search Yahoo\n```python\nfrom serpapi import YahooSearch\nsearch = YahooSearch({\"p\": \"Coffee\"})\ndata = search.get_dict()\n```\nThis code prints Yahoo search results for coffee as a Dictionary. \n\nhttps://serpapi.com/yahoo-search-api\n\n\n### Search eBay\n```python\nfrom serpapi import EbaySearch\nsearch = EbaySearch({\"_nkw\": \"Coffee\"})\ndata = search.get_dict()\n```\nThis code prints eBay search results for coffee as a Dictionary. \n\nhttps://serpapi.com/ebay-search-api\n\n### Search Home Depot\n```python\nfrom serpapi import HomeDepotSearch\nsearch = HomeDepotSearch({\"q\": \"chair\"})\ndata = search.get_dict()\n```\nThis code prints Home Depot search results for chair as Dictionary. \n\nhttps://serpapi.com/home-depot-search-api\n\n### Search Youtube\n```python\nfrom serpapi import HomeDepotSearch\nsearch = YoutubeSearch({\"q\": \"chair\"})\ndata = search.get_dict()\n```\nThis code prints Youtube search results for chair as Dictionary. \n\nhttps://serpapi.com/youtube-search-api\n\n### Search Google Scholar\n```python\nfrom serpapi import GoogleScholarSearch\nsearch = GoogleScholarSearch({\"q\": \"Coffee\"})\ndata = search.get_dict()\n```\nThis code prints Google Scholar search results.\n\n### Search Walmart\n```python\nfrom serpapi import WalmartSearch\nsearch = WalmartSearch({\"query\": \"chair\"})\ndata = search.get_dict()\n```\nThis code prints Walmart search results.\n\n### Search Youtube\n```python\nfrom serpapi import YoutubeSearch\nsearch = YoutubeSearch({\"search_query\": \"chair\"})\ndata = search.get_dict()\n```\nThis code prints Youtube search results.\n\n### Search Apple App Store\n```python\nfrom serpapi import AppleAppStoreSearch\nsearch = AppleAppStoreSearch({\"term\": \"Coffee\"})\ndata = search.get_dict()\n```\nThis code prints Apple App Store search results.\n\n### Search Naver\n```python\nfrom serpapi import NaverSearch\nsearch = NaverSearch({\"query\": \"chair\"})\ndata = search.get_dict()\n```\nThis code prints Naver search results.\n\n### Generic search with SerpApiClient\n```python\nfrom serpapi import SerpApiClient\nquery = {\"q\": \"Coffee\", \"location\": \"Austin,Texas\", \"engine\": \"google\"}\nsearch = SerpApiClient(query)\ndata = search.get_dict()\n```\nThis class enables interaction with any search engine supported by SerpApi.com \n\n### Search Google Images\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\"q\": \"coffe\", \"tbm\": \"isch\"})\nfor image_result in search.get_dict()['images_results']:\n    link = image_result[\"original\"]\n    try:\n        print(\"link: \" + link)\n        # wget.download(link, '.')\n    except:\n        pass\n```\n\nThis code prints all the image links, \n and downloads the images if you un-comment the line with wget (Linux/OS X tool to download files).\n\nThis tutorial covers more ground on this topic.\nhttps://github.com/serpapi/showcase-serpapi-tensorflow-keras-image-training\n\n### Search Google News\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\n    \"q\": \"coffe\",   # search search\n    \"tbm\": \"nws\",  # news\n    \"tbs\": \"qdr:d\", # last 24h\n    \"num\": 10\n})\nfor offset in [0,1,2]:\n    search.params_dict[\"start\"] = offset * 10\n    data = search.get_dict()\n    for news_result in data['news_results']:\n        print(str(news_result['position'] + offset * 10) + \" - \" + news_result['title'])\n```\n\nThis script prints the first 3 pages of the news headlines for the last 24 hours.\n\n### Search Google Shopping\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\n    \"q\": \"coffe\",   # search search\n    \"tbm\": \"shop\",  # news\n    \"tbs\": \"p_ord:rv\", # last 24h\n    \"num\": 100\n})\ndata = search.get_dict()\nfor shopping_result in data['shopping_results']:\n    print(shopping_result['position']) + \" - \" + shopping_result['title'])\n\n```\n\nThis script prints all the shopping results, ordered by review order.\n\n### Google Search By Location\n\nWith SerpApi, we can build a Google search from anywhere in the world.\nThis code looks for the best coffee shop for the given cities.\n\n```python\nfrom serpapi import GoogleSearch\nfor city in [\"new york\", \"paris\", \"berlin\"]:\n  location = GoogleSearch({}).get_location(city, 1)[0][\"canonical_name\"]\n  search = GoogleSearch({\n      \"q\": \"best coffee shop\",   # search search\n      \"location\": location,\n      \"num\": 1,\n      \"start\": 0\n  })\n  data = search.get_dict()\n  top_result = data[\"organic_results\"][0][\"title\"]\n```\n\n### Batch Asynchronous Searches\n\nWe offer two ways to boost your searches thanks to the`async` parameter.\n - Blocking - async=false - more compute intensive because the search needs to maintain many connections. (default) \n- Non-blocking - async=true - the way to go for large batches of queries  (recommended)\n\n```python\n# Operating system\nimport os\n\n# regular expression library\nimport re\n\n# safe queue (named Queue in python2)\nfrom queue import Queue\n\n# Time utility\nimport time\n\n# SerpApi search\nfrom serpapi import GoogleSearch\n\n# store searches\nsearch_queue = Queue()\n\n# SerpApi search\nsearch = GoogleSearch({\n    \"location\": \"Austin,Texas\",\n    \"async\": True,\n    \"api_key\": os.getenv(\"API_KEY\")\n})\n\n# loop through a list of companies\nfor company in ['amd', 'nvidia', 'intel']:\n    print(\"execute async search: q = \" + company)\n    search.params_dict[\"q\"] = company\n    result = search.get_dict()\n    if \"error\" in result:\n        print(\"oops error: \", result[\"error\"])\n        continue\n    print(\"add search to the queue where id: \", result['search_metadata'])\n    # add search to the search_queue\n    search_queue.put(result)\n\nprint(\"wait until all search statuses are cached or success\")\n\n# Create regular search\nwhile not search_queue.empty():\n    result = search_queue.get()\n    search_id = result['search_metadata']['id']\n\n    # retrieve search from the archive - blocker\n    print(search_id + \": get search from archive\")\n    search_archived = search.get_search_archive(search_id)\n    print(search_id + \": status = \" +\n          search_archived['search_metadata']['status'])\n\n    # check status\n    if re.search('Cached|Success',\n                 search_archived['search_metadata']['status']):\n        print(search_id + \": search done with q = \" +\n              search_archived['search_parameters']['q'])\n    else:\n        # requeue search_queue\n        print(search_id + \": requeue search\")\n        search_queue.put(result)\n\n        # wait 1s\n        time.sleep(1)\n\nprint('all searches completed')\n```\n\nThis code shows how to run searches asynchronously.\nThe search parameters must have {async: True}. This indicates that the client shouldn't wait for the search to be completed.\nThe current thread that executes the search is now non-blocking, which allows it to execute thousands of searches in seconds. The SerpApi backend will do the processing work.\nThe actual search result is deferred to a later call from the search archive using get_search_archive(search_id).\nIn this example the non-blocking searches are persisted in a queue: search_queue.\nA loop through the search_queue allows it to fetch individual search results.\nThis process can easily be multithreaded to allow a large number of concurrent search requests.\nTo keep things simple, this example only explores search results one at a time (single threaded).\n\n[See example.](https://github.com/serpapi/google-search-results-python/blob/master/tests/test_example.py)\n\n### Python object as a result\n\nThe search results can be automatically wrapped in dynamically generated Python object.\nThis solution offers a more dynamic, fully Oriented Object Programming approach over the regular Dictionary / JSON data structure.\n\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\"q\": \"Coffee\", \"location\": \"Austin,Texas\"})\nr = search.get_object()\nassert type(r.organic_results), list\nassert r.organic_results[0].title\nassert r.search_metadata.id\nassert r.search_metadata.google_url\nassert r.search_parameters.q, \"Coffee\"\nassert r.search_parameters.engine, \"google\"\n```\n\n### Pagination using iterator\nLet's collect links across multiple search results pages.\n```python\n# to get 2 pages\nstart = 0\nend = 40\npage_size = 10\n\n# basic search parameters\nparameter = {\n  \"q\": \"coca cola\",\n  \"tbm\": \"nws\",\n  \"api_key\": os.getenv(\"API_KEY\"),\n  # optional pagination parameter\n  #  the pagination method can take argument directly\n  \"start\": start,\n  \"end\": end,\n  \"num\": page_size\n}\n\n# as proof of concept \n# urls collects\nurls = []\n\n# initialize a search\nsearch = GoogleSearch(parameter)\n\n# create a python generator using parameter\npages = search.pagination()\n# or set custom parameter\npages = search.pagination(start, end, page_size)\n\n# fetch one search result per iteration \n# using a basic python for loop \n# which invokes python iterator under the hood.\nfor page in pages:\n  print(f\"Current page: {page['serpapi_pagination']['current']}\")\n  for news_result in page[\"news_results\"]:\n    print(f\"Title: {news_result['title']}\\nLink: {news_result['link']}\\n\")\n    urls.append(news_result['link'])\n  \n# check if the total number pages is as expected\n# note: the exact number if variable depending on the search engine backend\nif len(urls) == (end - start):\n  print(\"all search results count match!\")\nif len(urls) == len(set(urls)):\n  print(\"all search results are unique!\")\n```\n\nExamples to fetch links with pagination: [test file](https://github.com/serpapi/google-search-results-python/blob/master/tests/test_example_paginate.py), [online IDE](https://replit.com/@DimitryZub1/Scrape-Google-News-with-Pagination-python-serpapi)\n\n### Error management\n\nSerpApi keeps error management simple.\n - backend service error or search fail\n - client error\n\nIf it's a backend error, a simple error message is returned as string in the server response.\n```python\nfrom serpapi import GoogleSearch\nsearch = GoogleSearch({\"q\": \"Coffee\", \"location\": \"Austin,Texas\", \"api_key\": \"<secret_key>\"})\ndata = search.get_json()\nassert data[\"error\"] == None\n```\nIn some cases, there are more details available in the data object.\n\nIf it's a client error, then a SerpApiClientException is raised.\n\n## Change log\n2023-03-10 @ 2.4.2\n - Change long description to README.md\n\n2021-12-22 @ 2.4.1\n - add more search engine \n   - youtube\n   - walmart\n   - apple_app_store\n   - naver \n - raise SerpApiClientException instead of raw string in order to follow Python guideline 3.5+\n - add more unit error tests for serp_api_client\n\n2021-07-26 @ 2.4.0\n - add page size support using num parameter\n - add youtube search engine\n\n2021-06-05 @ 2.3.0\n - add pagination support\n\n2021-04-28 @ 2.2.0\n - add get_response method to provide raw requests.Response object\n\n2021-04-04 @ 2.1.0\n - Add home depot search engine\n - get_object() returns dynamic Python object\n \n2020-10-26 @ 2.0.0\n - Reduce class name to <engine>Search\n - Add get_raw_json\n\n2020-06-30 @ 1.8.3\n - simplify import\n - improve package for python 3.5+\n - add support for python 3.5 and 3.6\n\n2020-03-25 @ 1.8\n - add support for Yandex, Yahoo, Ebay\n - clean-up test\n\n2019-11-10 @ 1.7.1\n - increase engine parameter priority over engine value set in the class\n\n2019-09-12 @ 1.7\n - Change  namespace \"from lib.\" instead: \"from serpapi import GoogleSearch\"\n - Support for Bing and Baidu\n\n2019-06-25 @ 1.6\n - New search engine supported: Baidu and Bing\n\n## Conclusion\nSerpApi supports all the major search engines. Google has the more advance support with all the major services available: Images, News, Shopping and more..\nTo enable a type of search, the field tbm (to be matched) must be set to:\n\n * isch: Google Images API.\n * nws: Google News API.\n * shop: Google Shopping API.\n * any other Google service should work out of the box.\n * (no tbm parameter): regular Google search.\n\nThe field `tbs` allows to customize the search even more.\n\n[The full documentation is available here.](https://serpapi.com/search-api)\n",
    "bugtrack_url": null,
    "license": "MIT",
    "summary": "Scrape and search localized results from Google, Bing, Baidu, Yahoo, Yandex, Ebay, Homedepot, youtube at scale using SerpApi.com",
    "version": "2.4.2",
    "split_keywords": [
        "scrape",
        "serp",
        "api",
        "json",
        "search",
        "localized",
        "rank",
        "google",
        "bing",
        "baidu",
        "yandex",
        "yahoo",
        "ebay",
        "scale",
        "datamining",
        "training",
        "machine",
        "ml",
        "youtube",
        "naver",
        "walmart",
        "apple",
        "store",
        "app"
    ],
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "blake2b_256": "7730b3a6f6a2e00f8153549c2fa345c58ae1ce8e5f3153c2fe0484d444c3abcb",
                "md5": "4e72e0d5af065af613f1d8bda10e84c1",
                "sha256": "603a30ecae2af8e600b22635757a6df275dad4b934f975e67878ccd640b78245"
            },
            "downloads": -1,
            "filename": "google_search_results-2.4.2.tar.gz",
            "has_sig": false,
            "md5_digest": "4e72e0d5af065af613f1d8bda10e84c1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.5",
            "size": 18818,
            "upload_time": "2023-03-10T11:13:09",
            "upload_time_iso_8601": "2023-03-10T11:13:09.953387Z",
            "url": "https://files.pythonhosted.org/packages/77/30/b3a6f6a2e00f8153549c2fa345c58ae1ce8e5f3153c2fe0484d444c3abcb/google_search_results-2.4.2.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "upload_time": "2023-03-10 11:13:09",
    "github": true,
    "gitlab": false,
    "bitbucket": false,
    "github_user": "serpapi",
    "github_project": "google-search-results-python",
    "travis_ci": false,
    "coveralls": false,
    "github_actions": true,
    "requirements": [],
    "lcname": "google-search-results"
}
vikoky
Elapsed time: 0.08356s