aquests


Nameaquests JSON
Version 0.6.7.1 PyPI version JSON
home_pagehttps://gitlab.com/hansroh/aquests
SummaryAsynchronous Multiplexing HTTP2/DBO Requests
upload_time2017-02-14 11:45:40
maintainerNone
docs_urlNone
authorHans Roh
requires_pythonNone
licenseMIT
keywords
VCS
bugtrack_url
requirements No requirements were recorded.
Travis-CI No Travis.
Coveralis test coverage No Coveralis.
            ====================================
Asynchronous Multiplexing Requests
====================================

Aquests is generating asynchronous requests and fetching data from HTTP2, REST API, Websocket, RPCs and several Database engines. This project was originally started for testing `Skitai App Engine`_ and seperated for efficient developing eaches on Jan 2017.

Supported requests are:

- HTTP/1.1
- HTTP/2.0 if target server provides
- Websocket
- XML-RPC
- gRPC
- PostgreSQL
- MongoDB
- Redis

.. _`Skitai App Engine`: https://pypi.python.org/pypi/skitaid

.. contents:: Table of Contents


Quickstart
=============

For fetching single web page:

.. code-block:: python

  import aquests
  
  aquests.get ("http://127.0.0.1:5000/")
  aquests.fetchall ()

Its result is:

.. code-block:: bash

  user$ REQID 0. HTTP/2.0 200 OK 4210 bytes received
  
Let's add more pages:

.. code-block:: python
  
  for i in range (3):
    aquests.get ("http://127.0.0.1:5000/")     
  aquests.fetchall ()
  
Its result is:

.. code-block:: bash

  REQID 0. HTTP/2.0 200 OK 4210 bytes received
  REQID 1. HTTP/2.0 200 OK 4210 bytes received
  REQID 2. HTTP/2.0 200 OK 4210 bytes received

Now increase fetching workers,

.. code-block:: python

  aquests.configure (3)  # 3 workers
  for i in range (3):
    aquests.get ("http://127.0.0.1:5000/")     
  aquests.fetchall ()

Result is same as above but somewhat faster and REQID is not ordered.

Now increase workers and pages for making load,

.. code-block:: python
  
  aquests.configure (100) # 100 workers
  for i in range (10000):
    aquests.get ("http://127.0.0.1:5000/")      
  aquests.fetchall ()
  
Now result is,

.. code-block:: bash

  REQID 3635. HTTP/2.0 200 OK 4210 bytes received
  REQID 3627. HTTP/2.0 200 OK 4210 bytes received
  REQID 3594. HTTP/2.0 200 OK 4210 bytes received
  REQID 3702. HTTP/2.0 200 OK 4210 bytes received
  REQID 3685. HTTP/2.0 200 OK 4210 bytes received
  REQID 3637. HTTP/2.0 200 OK 4210 bytes received
  REQID 3591. HTTP/2.0 200 OK 4210 bytes received
  REQID 3586. HTTP/2.0 200 OK 4210 bytes received
  (and scrolled fast...)

Installation
===============

.. code-block:: bash

  pip install aquests


Usage
======

Binding Callback
-------------------

.. code-block:: python
  
  def finish_request (response):
    print (response.status_code)
    print (response.content)
  	
  aquests.configure (workers = 10, callback = finish_request)
  for i in range (10):
    aquests.get ("http://127.0.0.1:5000/")    
  aquests.fetchall ()


Making Traffic Load With Generator Style
------------------------------------------

.. code-block:: python
  
  numreq = 0
  limit = 1000000
  workers = 100
  
  def finish_request (response):
    global numreq, limit     
    if numreq < limit:
      aquests.get ("http://127.0.0.1:5000/")
      numreq += 1
  	
  aquests.configure (workers, callback = finish_request)
  for i in range (workers):
    aquests.get ("http://127.0.0.1:5000/")  
    numreq += 1
  aquests.fetchall ()  


Set/Get Request Meta Information
------------------------------------

.. code-block:: python
  
  def finish_request (response):
    print (response.meta ['req_id'])
    print (response.meta ['req_method'])
    print (response.meta ['job_name'])
  	
  aquests.configure (workers = 10, callback = finish_request)
  aquests.get ("http://127.0.0.1:5000/", meta = {'job_name': 'test1'})  
  aquests.get ("http://127.0.0.1:5000/", meta = {'job_name': 'test2'})

Note: meta ['req_id'] and meta ['req_method'] are automatically added by aquests.


Timeout Setting
----------------

.. code-block:: python
  
  aquests.configure (20, timeout = 10) # 10 seconds
  aquests.get ("https://www.google.co.kr/?gfe_rd=cr&ei=3y14WPCTG4XR8gfSjoK4DQ")  
  aquests.fetchall ()

If timeout occured, response status_code will be 702. Also note above 700 codes mostly indicates network related error.


**Caution**

1. You can't specify timout for each task
2. Cause of aquests' single thread coroutine feature, timeout will not work with exactly timeout seconds.


Mixed Requests
----------------

.. code-block:: python

  dbo = aquests.mongodb ("127.0.0.1:27017", "test_database")
  aquests.configure (20)
  for i in range (1000): 
    aquests.get ("http://127.0.0.1:5000/")
    dbo.findone ("posts", {"author": "James Milton"})
  aquests.fetchall ()


Authorization
-----------------

For requesting with basic/digest authorization:

.. code:: python

  stub = aquests.rpc (url, auth = (username, password))
  stub.get_prime_number_gt (10000)
  aquests.fetchall ()
  
If you provide both (username, password), aquests try basic/digest authorization. But if just (username,) aquests handle username as bearer token like API Key.


Redirection
------------

For automatically redireting by http status 301, 302, 307, 308:

.. code:: python
  
  def finish_request (response):
    print (response.history)
    
  aquests.configure (callback = finish_request)
  aquests.get ('http://pypi.python.org')
  aquests.fetchall ()
  
response.history is like,

.. code:: python

  [<Response [301]>, <Response [302]>]

Also for disabling redirect,

.. code:: python

  aquests.configure (callback = finish_request, allow_redirects = False)
  

Enabling Cookie
------------------

.. code-block:: python
  
  def finish_request (response):
    print (response.cookies)
    
  aquests.configure (20, callback = finish_request, cookie = True)
  aquests.get ("https://www.google.co.kr/?gfe_rd=cr&ei=3y14WPCTG4XR8gfSjoK4DQ")  
  aquests.fetchall ()

**Caution**

This cookie feature shouldn't handle as different sessions per worker. All workers (connections) of aquests share same cookie values per domain. It means a worker sign in a website, so are the others. Imagine lots of FireFox windows on a desktop computer. If you really need session control, use requests_.


Change Logger
--------------

.. code-block:: python
  
  from aquests.lib import logger
  
  aquests.configure (
    workers = 10, 
    logger = logger.file_logger ('/tmp/logs', 'aquests')
  )


Response
---------

I make similar naming with requests_' attribute and method names as possible.

Response has these attributes and method:

- meta: user added meta data including 'req_id' and 'req_method'
- status_code: HTTP status code or DBO query success (200) or failure (500) code
- reason: status text like OK, Not Found...
- content: bytes content or original db result
- data: usally same as content but on RPC, DB query or json response situation, it returns result object.
- logger: logger.log (msg, type ='info'), logger.trace ()
- method: POST, GET, PUT etc for HTTP/RPC and execute, get, set or lrange etc for DBO
- raise_for_status (): raise exception when HTTP status code >= 300 or DBO command excution failure
- reraise (): shortcut for raise_for_status ()

Below thing is available only on Websocket response.

- opcode: websocket opcode of received message

Below things are available only on DBO responses.

- server: database server address
- dbname: database object name
- params: database command parameters

Below things aren't available on DBO and Websocket responses.

- url: requested url
- history: redirected history by http code 301, 302, 307 and 308 like *[<Response [302]>, <Response [302]>]*
- version: HTTP protocol version
- headers: Response headers
- text: charset encoded string (unicode)
- raw: file like object for bytes stream has raw.read (), raw.readline (),... methods
- cookies: if configure (cookie = True), returns dictionary
- encoding: extracted from content-type header
- request.headers
- request.payload: request body bytes, not available at upload and grpc
- json (): load JSON data, but if response content-type is application/json, automatically loaded into response.data then you can just use it.
- get_header (key, default = None): returns header value, if not exists return default
- get_header_with_attr (key, default = None): returns header value and attr dict like 'text/html', {'charset': 'utf-8'}
- set_cookie (key, val, domain = None, path = "/")
- get_cookie (key)

.. _requests: https://pypi.python.org/pypi/requests


Configuration Parameters
==========================

.. code-block:: python

  import aquests
  
  aquests.configure (
    workers = 1, 
    logger = None, 
    callback = None, 
    timeout = 10, 
    cookie = False,
    force_http1 = False, 
    http2_constreams = 1,
    allow_redirects = True
  )
  
- workers: number of fetching workers, it'not threads
- logger: logger shoukd have 2 method - log (msg, type = 'info') and trace () for exception logging. if not provided, aquests uses aquests.ib.logger.screen_logger
- callback: function has receiving response arg
- timeout: request timeout seconds
- cookie: enable/disable using cookie for request
- force_http1: enforce http 1.1 not 2.0
- http2_constreams: if you making requests to single http2 server, how many concurrent streams per channel. BE CAREFUL, it might be useful for generating traffic load for testing your http2 web servers. and if your server doesn't provide http2, your workers will be increased to number of http2_constreams times than you really want.
- allow_redirects: if set True, in case HTTP status code is in 301, 302, 307, 308 then redirect automatically


List of Methods
==================

GET, DELETE and etc.
---------------------

.. code-block:: python

  aquests.get ("http://127.0.0.1:5000/")
  aquests.delete ("http://127.0.0.1:5000/models/ak-40")
  aquests.get ("https://www.google.co.kr/search?q=aquests")

Also aquests.head (), options () and trace () are available.


POST, PUT
---------------

.. code-block:: python

  aquests.post (
    "http://127.0.0.1:5000/", 
    {'author': 'James Milton'}, 
    {'Content-Type': 'application/x-www-form-urlencoded'}
   )
   
  # is equal to:
   
  aquests.postform (
    "http://127.0.0.1:5000/", 
    {'author': 'James Milton'}    
  )

Put example,

.. code-block:: python
  
  aquest.put (
    "http://127.0.0.1:5000/users/jamesmilton",
    {'fullnamer': 'James Milton'},
    {'Content-Type': 'application/json'}
    )
  )
  
  # is equal to:
   
  aquests.putjson (
    "http://127.0.0.1:5000/users/jamesmilton",
    {'fullnamer': 'James Milton'}
  )
  
There're some shorter ways ratehr than specifing content type:

- postform: application/x-www-form-urlencoded, data value should be dictionary
- postjson: application/json, data value should be json dumpable
- postxml: text/xml, data value should be xml string or utf-8 encoded bytes
- postnvp: text/namevalue, data value should be dictionary 

And putform (), putjson ()... is also available.

  
File Upload
------------

.. code-block:: python

  aquests.upload (
    "http://127.0.0.1:5000/", 
    {
      'author': 'James Milton',
      'file': open ('/tmp/mycar.jpg', 'rb')
    }
  )

You should open file with 'rb' mode.

Websocket
-----------

.. code-block:: python

  aquests.ws ("ws://127.0.0.1:5000/websocket/echo", "Hello World")
  # secure websocket channel, use wss
  aquests.ws ("wss://127.0.0.1:5000/websocket/echo", "Hello World")
  aquests.fetchall ()

Response is like this,
  
- response.status_code: 200
- response.reason: "OK"
- response.content: (1, "Hello World") # (opcode, message)
- response.opcode: 1 # OPCODE_TEXT
- response.data: "Hello World"

Note: Sometimes status_code is 200, opcode is -1. It is NOT official websocket spec. but means websocket is successfully connected but disconnected before receving a message by some reasons.

If you want to send specify message type.

.. code-block:: python
  
  from aquests.protocols.ws import OPCODE_TEXT, OPCODE_BINARY
    
  aquests.ws ("ws://127.0.0.1:5000/websocket/echo", (OPCODE_BINARY, b"Hello World"))
  aquests.fetchall ()


XML-RPC
----------

.. code-block:: python

  stub = aquests.rpc ("https://pypi.python.org/pypi")
  stub.package_releases('roundup')
  stub.prelease_urls('roundup', '1.4.10')
  aquests.fetchall ()

Returns,

.. code-block:: bash

  ['1.5.1']
  <class 'xmlrpc.client.Fault'> <Fault 1:...>


gRPC
----------

.. code-block:: python
  
  import route_guide_pb2
  
  stub = aquests.grpc ("http://127.0.0.1:5000/routeguide.RouteGuide")
  point = route_guide_pb2.Point (latitude=409146138, longitude=-746188906)
  for i in range (3):
    stub.GetFeature (point)
  aquests.fetchall ()


Returns,

.. code-block:: python

  name: "Berkshire Valley Management Area Trail, Jefferson, NJ, USA"
  location {
    latitude: 409146138
    longitude: -746188906
  }

For more about gRPC and route_guide_pb2, go to `gRPC Basics - Python`_.

.. _`gRPC Basics - Python`: http://www.grpc.io/docs/tutorials/basic/python.html


PostgreSQL
-------------

.. code-block:: python
  
  def finish_request (response):
    print (response.data)  	
  
  aquests.configure (3, callback = finish_request)	
  dbo = aquests.postgresql ("127.0.0.1:5432", "mydb", ("test", "1111"))
  for i in range (10):
    dbo.execute ("SELECT city, prcp, temp_hi, temp_low FROM weather;")

Returns,

.. code-block:: bash

  [
    {'prcp': 0.25, 'temp_hi': 50, 'city': 'San  Francisco', 'temp_lo': 46}, 
    {'prcp': 0.0, 'temp_hi': 54, 'city': 'Hayward', 'temp_lo': 37}
  ]  

MongoDB
---------

.. code-block:: python

  dbo = aquests.mongodb ("127.0.0.1:27017", "test_database")
  for i in range (3):
    dbo.findone ("posts", {"author": "Steve Newman"})  
    dbo.findall ("posts", {"author": "Hans Roh"})
  aquests.fetchall ()

Returns,

.. code-block:: bash

  {
    'starting_from': 0, 
    'number_returned': 1, 
    'cursor_id': 0, 
    'data': [
      {
        '_id': ObjectId('586a11f80d23915c7ec76f01'), 
        'author': 'Steve Newman', 
        'title': 'How to swim'
      }
    ]
  }
  

**Available Functions**

- find (colname, spec, offset = 0, limit = 1)
- findone (colname, spec): equivalent with find (colname, spec, 0, 1)
- findall (colname, spec): equivalent with find (colname, spec, 0, -1)
- insert (colname, docs, continue_on_error = 0)
- update (colname, spec, doc)
- updateone (colname, spec, doc)
- upsert (colname, spec, doc)
- upsertone (colname, spec, doc)
- delete (colname, spec, flag = 0)
- findkc (colname, spec, offset = 0, limit = 1): after finidhing search, it keeps cursor alive. then you can use 'get_more()'
- get_more (colname, cursor_id, num_to_return): cursor_id can be got from (findkc()'s result).data ["cursor_id"]
- kill_cursors (cursor_ids): if you use findkc() and stop fetching documents, you should mannually call this.

Note: User authorization is not supported yet.


Redis
---------

.. code-block:: python

  dbo = aquests.redis ("127.0.0.1:6379")
  dbo.get ("session-5ae675bc")
  dbo.lrange ("user-saved-docs", 0, 3)
  aquests.fetchall ()

Returns,

.. code-block:: bash
  
  response-of-session-5ae675bc
  
  [32534, 3453, 6786]

Possibly you can use all `Redis commands`_.

.. _`Redis commands`: https://redis.io/commands


Note: User authorization is not supported yet.


SQLite3 For Fast App Prototyping
---------------------------------

Usage is almost same with PostgreSQL. This service IS NOT asynchronous BUT just emulating.

.. code:: python
  
  dbo = aquests.sqlite3 ("sqlite3.db")
  dbo.execute ("""
    drop table if exists people;
    create table people (name_last, age);
    insert into people values ('Cho', 42);
  """)
  aquests.fetchall ()
  
  
Requests Parameters
========================

For get, post*, put*, upload, delete, options, trace parameters are the same.

.. code-block:: python

  aquests.get (url, params = None, headers = None, auth = None, meta = {})
  
- url: request url string
- params: None or dictionary, if it provide with get method, it will be attached on tail of url with '?'
- headers: None or dictionary
- auth: None or tuple (username, password)
- meta: dictionary

For Websocket,

.. code-block:: python

  aquests.ws (url, params = None, headers = None, auth = None, meta = {})
  
- url: request url string, should start with 'ws://' or 'wss://'(SSL Websocket)
- params: string, bytes or tuple. if messages is not string you specify message type code using tuple like (ws.OPCODE_PING, b"hello"), you can find OPCODE list, 'from aquests.protocols import ws'. CAUTION. if your params type is bytes type, opcode will automatically be OPCODE_BINARY and string type, be OPCODE_TEXT. and opcode is inffluent to receiver. if you avoid auto opcode, specify opcode with tuple.
  
    * ws.OPCODE_TEXT
    * ws.OPCODE_BINARY
    * ws.OPCODE_CONTINUATION
    * ws.OPCODE_PING
    * ws.OPCODE_PONG
    * ws.OPCODE_CLOSE
    
- headers: None or dictionary
- auth: None or tuple (username, password)
- meta: dictionary

For rpc, grpc stub creation:

.. code-block:: python

  stub = aquests.rpc (url, headers = None, auth = None, meta = {})
  stub = aquests.grpc (url, headers = None, auth = None, meta = {})
  
- url: request url string
- headers: None or dictionary
- auth: None or tuple (username, password)
- meta: dictionary

Note: stub's methods and parameters are defined by RPC service providers

For postgresql, mongodb, redis dbo creation:

.. code-block:: python

  dbo = aquests.postgresql (server, dbname = None, auth = None, meta = {})
  dbo = aquests.mongodb (server, dbname = None, auth = None, meta = {})  
  dbo = aquests.redis (server, dbname = None, auth = None, meta = {})
  
- server: address:port formated string
- dbname: None or string
- auth: None or tuple (username, password)
- meta: dictionary

Note: stub's methods and parameters are defined by database engines. Please read above related chapters But SQL based postgresql has only 1 method and parameters - execute(sql) or do(sql) just for your convinience.


History
=========

- 0.6.7: change socket closing log message

- 0.6.6: fix asyncon active

- 0.6.4.2: license changed from BSD to MIT

- 0.6.4.1: fix await_fifo bug

- 0.6.3: fix lifetime,  tmap

- 0.6.2: change queue list -> deque

- 0.6.1: fix websocket text data encoding

- 0.6: 
  
  * add configure option: allow_redirects
  * new response.history
  * fix 30x redirection
  * fix 401 unauthorized
  
- 0.5.2: remove ready_producer_fifo, this will be used only serverside
- 0.5.1: change from list to deque on producer_fifo
- 0.4.33: force_http1 applied to https
- 0.4.32: fix http.buffer.list_buffer class
- 0.4.30: add websocket message type detection
- 0.4.28: remove aquests.wss, use aquests.ws with url wss://...
- 0.4.25: fix select.select () divide and conquer
- 0.4.22: fix http2_constreams
- 0.4.21: fix http2 flow control window
- 0.4.20: add configure options: force_http1, http2_constreams
- 0.4.18: url / trailing
- 0.4.17: fix finding end of data on http2
- 0.4.16: fix http2 disconnecting behavior
- 0.4.10: fix xmlrpc stub url / trailing
- 0.4.9: changed response properties - request.method -> method, request.server -> server, request.dbname -> dbname and request.params -> params
- 0.4.4: add lib.athreads
- 0.4.2: fix http2 large content download
- 0.4.1: add a few examples
- 0.4: add timeout feature
- 0.3.10: fix http2 frame length validation, add cookie feature
- 0.3.8: fix dbo request shutdown behavior
- 0.3.1: add HEAD, OPTIONS, TRACE
- 0.3: fix installation error
- 0.2.13: change default display callback
- 0.2.10: fix xmlrpc


            

Raw data

            {
    "maintainer": null, 
    "docs_url": null, 
    "requires_python": null, 
    "maintainer_email": null, 
    "cheesecake_code_kwalitee_id": null, 
    "keywords": null, 
    "upload_time": "2017-02-14 11:45:40", 
    "author": "Hans Roh", 
    "home_page": "https://gitlab.com/hansroh/aquests", 
    "download_url": "https://pypi.python.org/packages/62/41/1404f08acad90f69ed8bdebab89adb3e4415a1099cc9c0d4623cdc0463b4/aquests-0.6.7.1.tar.gz", 
    "platform": "posix,nt", 
    "version": "0.6.7.1", 
    "cheesecake_documentation_id": null, 
    "description": "====================================\r\nAsynchronous Multiplexing Requests\r\n====================================\r\n\r\nAquests is generating asynchronous requests and fetching data from HTTP2, REST API, Websocket, RPCs and several Database engines. This project was originally started for testing `Skitai App Engine`_ and seperated for efficient developing eaches on Jan 2017.\r\n\r\nSupported requests are:\r\n\r\n- HTTP/1.1\r\n- HTTP/2.0 if target server provides\r\n- Websocket\r\n- XML-RPC\r\n- gRPC\r\n- PostgreSQL\r\n- MongoDB\r\n- Redis\r\n\r\n.. _`Skitai App Engine`: https://pypi.python.org/pypi/skitaid\r\n\r\n.. contents:: Table of Contents\r\n\r\n\r\nQuickstart\r\n=============\r\n\r\nFor fetching single web page:\r\n\r\n.. code-block:: python\r\n\r\n  import aquests\r\n  \r\n  aquests.get (\"http://127.0.0.1:5000/\")\r\n  aquests.fetchall ()\r\n\r\nIts result is:\r\n\r\n.. code-block:: bash\r\n\r\n  user$ REQID 0. HTTP/2.0 200 OK 4210 bytes received\r\n  \r\nLet's add more pages:\r\n\r\n.. code-block:: python\r\n  \r\n  for i in range (3):\r\n    aquests.get (\"http://127.0.0.1:5000/\")     \r\n  aquests.fetchall ()\r\n  \r\nIts result is:\r\n\r\n.. code-block:: bash\r\n\r\n  REQID 0. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 1. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 2. HTTP/2.0 200 OK 4210 bytes received\r\n\r\nNow increase fetching workers,\r\n\r\n.. code-block:: python\r\n\r\n  aquests.configure (3)  # 3 workers\r\n  for i in range (3):\r\n    aquests.get (\"http://127.0.0.1:5000/\")     \r\n  aquests.fetchall ()\r\n\r\nResult is same as above but somewhat faster and REQID is not ordered.\r\n\r\nNow increase workers and pages for making load,\r\n\r\n.. code-block:: python\r\n  \r\n  aquests.configure (100) # 100 workers\r\n  for i in range (10000):\r\n    aquests.get (\"http://127.0.0.1:5000/\")      \r\n  aquests.fetchall ()\r\n  \r\nNow result is,\r\n\r\n.. code-block:: bash\r\n\r\n  REQID 3635. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3627. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3594. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3702. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3685. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3637. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3591. HTTP/2.0 200 OK 4210 bytes received\r\n  REQID 3586. HTTP/2.0 200 OK 4210 bytes received\r\n  (and scrolled fast...)\r\n\r\nInstallation\r\n===============\r\n\r\n.. code-block:: bash\r\n\r\n  pip install aquests\r\n\r\n\r\nUsage\r\n======\r\n\r\nBinding Callback\r\n-------------------\r\n\r\n.. code-block:: python\r\n  \r\n  def finish_request (response):\r\n    print (response.status_code)\r\n    print (response.content)\r\n  \t\r\n  aquests.configure (workers = 10, callback = finish_request)\r\n  for i in range (10):\r\n    aquests.get (\"http://127.0.0.1:5000/\")    \r\n  aquests.fetchall ()\r\n\r\n\r\nMaking Traffic Load With Generator Style\r\n------------------------------------------\r\n\r\n.. code-block:: python\r\n  \r\n  numreq = 0\r\n  limit = 1000000\r\n  workers = 100\r\n  \r\n  def finish_request (response):\r\n    global numreq, limit     \r\n    if numreq < limit:\r\n      aquests.get (\"http://127.0.0.1:5000/\")\r\n      numreq += 1\r\n  \t\r\n  aquests.configure (workers, callback = finish_request)\r\n  for i in range (workers):\r\n    aquests.get (\"http://127.0.0.1:5000/\")  \r\n    numreq += 1\r\n  aquests.fetchall ()  \r\n\r\n\r\nSet/Get Request Meta Information\r\n------------------------------------\r\n\r\n.. code-block:: python\r\n  \r\n  def finish_request (response):\r\n    print (response.meta ['req_id'])\r\n    print (response.meta ['req_method'])\r\n    print (response.meta ['job_name'])\r\n  \t\r\n  aquests.configure (workers = 10, callback = finish_request)\r\n  aquests.get (\"http://127.0.0.1:5000/\", meta = {'job_name': 'test1'})  \r\n  aquests.get (\"http://127.0.0.1:5000/\", meta = {'job_name': 'test2'})\r\n\r\nNote: meta ['req_id'] and meta ['req_method'] are automatically added by aquests.\r\n\r\n\r\nTimeout Setting\r\n----------------\r\n\r\n.. code-block:: python\r\n  \r\n  aquests.configure (20, timeout = 10) # 10 seconds\r\n  aquests.get (\"https://www.google.co.kr/?gfe_rd=cr&ei=3y14WPCTG4XR8gfSjoK4DQ\")  \r\n  aquests.fetchall ()\r\n\r\nIf timeout occured, response status_code will be 702. Also note above 700 codes mostly indicates network related error.\r\n\r\n\r\n**Caution**\r\n\r\n1. You can't specify timout for each task\r\n2. Cause of aquests' single thread coroutine feature, timeout will not work with exactly timeout seconds.\r\n\r\n\r\nMixed Requests\r\n----------------\r\n\r\n.. code-block:: python\r\n\r\n  dbo = aquests.mongodb (\"127.0.0.1:27017\", \"test_database\")\r\n  aquests.configure (20)\r\n  for i in range (1000): \r\n    aquests.get (\"http://127.0.0.1:5000/\")\r\n    dbo.findone (\"posts\", {\"author\": \"James Milton\"})\r\n  aquests.fetchall ()\r\n\r\n\r\nAuthorization\r\n-----------------\r\n\r\nFor requesting with basic/digest authorization:\r\n\r\n.. code:: python\r\n\r\n  stub = aquests.rpc (url, auth = (username, password))\r\n  stub.get_prime_number_gt (10000)\r\n  aquests.fetchall ()\r\n  \r\nIf you provide both (username, password), aquests try basic/digest authorization. But if just (username,) aquests handle username as bearer token like API Key.\r\n\r\n\r\nRedirection\r\n------------\r\n\r\nFor automatically redireting by http status 301, 302, 307, 308:\r\n\r\n.. code:: python\r\n  \r\n  def finish_request (response):\r\n    print (response.history)\r\n    \r\n  aquests.configure (callback = finish_request)\r\n  aquests.get ('http://pypi.python.org')\r\n  aquests.fetchall ()\r\n  \r\nresponse.history is like,\r\n\r\n.. code:: python\r\n\r\n  [<Response [301]>, <Response [302]>]\r\n\r\nAlso for disabling redirect,\r\n\r\n.. code:: python\r\n\r\n  aquests.configure (callback = finish_request, allow_redirects = False)\r\n  \r\n\r\nEnabling Cookie\r\n------------------\r\n\r\n.. code-block:: python\r\n  \r\n  def finish_request (response):\r\n    print (response.cookies)\r\n    \r\n  aquests.configure (20, callback = finish_request, cookie = True)\r\n  aquests.get (\"https://www.google.co.kr/?gfe_rd=cr&ei=3y14WPCTG4XR8gfSjoK4DQ\")  \r\n  aquests.fetchall ()\r\n\r\n**Caution**\r\n\r\nThis cookie feature shouldn't handle as different sessions per worker. All workers (connections) of aquests share same cookie values per domain. It means a worker sign in a website, so are the others. Imagine lots of FireFox windows on a desktop computer. If you really need session control, use requests_.\r\n\r\n\r\nChange Logger\r\n--------------\r\n\r\n.. code-block:: python\r\n  \r\n  from aquests.lib import logger\r\n  \r\n  aquests.configure (\r\n    workers = 10, \r\n    logger = logger.file_logger ('/tmp/logs', 'aquests')\r\n  )\r\n\r\n\r\nResponse\r\n---------\r\n\r\nI make similar naming with requests_' attribute and method names as possible.\r\n\r\nResponse has these attributes and method:\r\n\r\n- meta: user added meta data including 'req_id' and 'req_method'\r\n- status_code: HTTP status code or DBO query success (200) or failure (500) code\r\n- reason: status text like OK, Not Found...\r\n- content: bytes content or original db result\r\n- data: usally same as content but on RPC, DB query or json response situation, it returns result object.\r\n- logger: logger.log (msg, type ='info'), logger.trace ()\r\n- method: POST, GET, PUT etc for HTTP/RPC and execute, get, set or lrange etc for DBO\r\n- raise_for_status (): raise exception when HTTP status code >= 300 or DBO command excution failure\r\n- reraise (): shortcut for raise_for_status ()\r\n\r\nBelow thing is available only on Websocket response.\r\n\r\n- opcode: websocket opcode of received message\r\n\r\nBelow things are available only on DBO responses.\r\n\r\n- server: database server address\r\n- dbname: database object name\r\n- params: database command parameters\r\n\r\nBelow things aren't available on DBO and Websocket responses.\r\n\r\n- url: requested url\r\n- history: redirected history by http code 301, 302, 307 and 308 like *[<Response [302]>, <Response [302]>]*\r\n- version: HTTP protocol version\r\n- headers: Response headers\r\n- text: charset encoded string (unicode)\r\n- raw: file like object for bytes stream has raw.read (), raw.readline (),... methods\r\n- cookies: if configure (cookie = True), returns dictionary\r\n- encoding: extracted from content-type header\r\n- request.headers\r\n- request.payload: request body bytes, not available at upload and grpc\r\n- json (): load JSON data, but if response content-type is application/json, automatically loaded into response.data then you can just use it.\r\n- get_header (key, default = None): returns header value, if not exists return default\r\n- get_header_with_attr (key, default = None): returns header value and attr dict like 'text/html', {'charset': 'utf-8'}\r\n- set_cookie (key, val, domain = None, path = \"/\")\r\n- get_cookie (key)\r\n\r\n.. _requests: https://pypi.python.org/pypi/requests\r\n\r\n\r\nConfiguration Parameters\r\n==========================\r\n\r\n.. code-block:: python\r\n\r\n  import aquests\r\n  \r\n  aquests.configure (\r\n    workers = 1, \r\n    logger = None, \r\n    callback = None, \r\n    timeout = 10, \r\n    cookie = False,\r\n    force_http1 = False, \r\n    http2_constreams = 1,\r\n    allow_redirects = True\r\n  )\r\n  \r\n- workers: number of fetching workers, it'not threads\r\n- logger: logger shoukd have 2 method - log (msg, type = 'info') and trace () for exception logging. if not provided, aquests uses aquests.ib.logger.screen_logger\r\n- callback: function has receiving response arg\r\n- timeout: request timeout seconds\r\n- cookie: enable/disable using cookie for request\r\n- force_http1: enforce http 1.1 not 2.0\r\n- http2_constreams: if you making requests to single http2 server, how many concurrent streams per channel. BE CAREFUL, it might be useful for generating traffic load for testing your http2 web servers. and if your server doesn't provide http2, your workers will be increased to number of http2_constreams times than you really want.\r\n- allow_redirects: if set True, in case HTTP status code is in 301, 302, 307, 308 then redirect automatically\r\n\r\n\r\nList of Methods\r\n==================\r\n\r\nGET, DELETE and etc.\r\n---------------------\r\n\r\n.. code-block:: python\r\n\r\n  aquests.get (\"http://127.0.0.1:5000/\")\r\n  aquests.delete (\"http://127.0.0.1:5000/models/ak-40\")\r\n  aquests.get (\"https://www.google.co.kr/search?q=aquests\")\r\n\r\nAlso aquests.head (), options () and trace () are available.\r\n\r\n\r\nPOST, PUT\r\n---------------\r\n\r\n.. code-block:: python\r\n\r\n  aquests.post (\r\n    \"http://127.0.0.1:5000/\", \r\n    {'author': 'James Milton'}, \r\n    {'Content-Type': 'application/x-www-form-urlencoded'}\r\n   )\r\n   \r\n  # is equal to:\r\n   \r\n  aquests.postform (\r\n    \"http://127.0.0.1:5000/\", \r\n    {'author': 'James Milton'}    \r\n  )\r\n\r\nPut example,\r\n\r\n.. code-block:: python\r\n  \r\n  aquest.put (\r\n    \"http://127.0.0.1:5000/users/jamesmilton\",\r\n    {'fullnamer': 'James Milton'},\r\n    {'Content-Type': 'application/json'}\r\n    )\r\n  )\r\n  \r\n  # is equal to:\r\n   \r\n  aquests.putjson (\r\n    \"http://127.0.0.1:5000/users/jamesmilton\",\r\n    {'fullnamer': 'James Milton'}\r\n  )\r\n  \r\nThere're some shorter ways ratehr than specifing content type:\r\n\r\n- postform: application/x-www-form-urlencoded, data value should be dictionary\r\n- postjson: application/json, data value should be json dumpable\r\n- postxml: text/xml, data value should be xml string or utf-8 encoded bytes\r\n- postnvp: text/namevalue, data value should be dictionary \r\n\r\nAnd putform (), putjson ()... is also available.\r\n\r\n  \r\nFile Upload\r\n------------\r\n\r\n.. code-block:: python\r\n\r\n  aquests.upload (\r\n    \"http://127.0.0.1:5000/\", \r\n    {\r\n      'author': 'James Milton',\r\n      'file': open ('/tmp/mycar.jpg', 'rb')\r\n    }\r\n  )\r\n\r\nYou should open file with 'rb' mode.\r\n\r\nWebsocket\r\n-----------\r\n\r\n.. code-block:: python\r\n\r\n  aquests.ws (\"ws://127.0.0.1:5000/websocket/echo\", \"Hello World\")\r\n  # secure websocket channel, use wss\r\n  aquests.ws (\"wss://127.0.0.1:5000/websocket/echo\", \"Hello World\")\r\n  aquests.fetchall ()\r\n\r\nResponse is like this,\r\n  \r\n- response.status_code: 200\r\n- response.reason: \"OK\"\r\n- response.content: (1, \"Hello World\") # (opcode, message)\r\n- response.opcode: 1 # OPCODE_TEXT\r\n- response.data: \"Hello World\"\r\n\r\nNote: Sometimes status_code is 200, opcode is -1. It is NOT official websocket spec. but means websocket is successfully connected but disconnected before receving a message by some reasons.\r\n\r\nIf you want to send specify message type.\r\n\r\n.. code-block:: python\r\n  \r\n  from aquests.protocols.ws import OPCODE_TEXT, OPCODE_BINARY\r\n    \r\n  aquests.ws (\"ws://127.0.0.1:5000/websocket/echo\", (OPCODE_BINARY, b\"Hello World\"))\r\n  aquests.fetchall ()\r\n\r\n\r\nXML-RPC\r\n----------\r\n\r\n.. code-block:: python\r\n\r\n  stub = aquests.rpc (\"https://pypi.python.org/pypi\")\r\n  stub.package_releases('roundup')\r\n  stub.prelease_urls('roundup', '1.4.10')\r\n  aquests.fetchall ()\r\n\r\nReturns,\r\n\r\n.. code-block:: bash\r\n\r\n  ['1.5.1']\r\n  <class 'xmlrpc.client.Fault'> <Fault 1:...>\r\n\r\n\r\ngRPC\r\n----------\r\n\r\n.. code-block:: python\r\n  \r\n  import route_guide_pb2\r\n  \r\n  stub = aquests.grpc (\"http://127.0.0.1:5000/routeguide.RouteGuide\")\r\n  point = route_guide_pb2.Point (latitude=409146138, longitude=-746188906)\r\n  for i in range (3):\r\n    stub.GetFeature (point)\r\n  aquests.fetchall ()\r\n\r\n\r\nReturns,\r\n\r\n.. code-block:: python\r\n\r\n  name: \"Berkshire Valley Management Area Trail, Jefferson, NJ, USA\"\r\n  location {\r\n    latitude: 409146138\r\n    longitude: -746188906\r\n  }\r\n\r\nFor more about gRPC and route_guide_pb2, go to `gRPC Basics - Python`_.\r\n\r\n.. _`gRPC Basics - Python`: http://www.grpc.io/docs/tutorials/basic/python.html\r\n\r\n\r\nPostgreSQL\r\n-------------\r\n\r\n.. code-block:: python\r\n  \r\n  def finish_request (response):\r\n    print (response.data)  \t\r\n  \r\n  aquests.configure (3, callback = finish_request)\t\r\n  dbo = aquests.postgresql (\"127.0.0.1:5432\", \"mydb\", (\"test\", \"1111\"))\r\n  for i in range (10):\r\n    dbo.execute (\"SELECT city, prcp, temp_hi, temp_low FROM weather;\")\r\n\r\nReturns,\r\n\r\n.. code-block:: bash\r\n\r\n  [\r\n    {'prcp': 0.25, 'temp_hi': 50, 'city': 'San  Francisco', 'temp_lo': 46}, \r\n    {'prcp': 0.0, 'temp_hi': 54, 'city': 'Hayward', 'temp_lo': 37}\r\n  ]  \r\n\r\nMongoDB\r\n---------\r\n\r\n.. code-block:: python\r\n\r\n  dbo = aquests.mongodb (\"127.0.0.1:27017\", \"test_database\")\r\n  for i in range (3):\r\n    dbo.findone (\"posts\", {\"author\": \"Steve Newman\"})  \r\n    dbo.findall (\"posts\", {\"author\": \"Hans Roh\"})\r\n  aquests.fetchall ()\r\n\r\nReturns,\r\n\r\n.. code-block:: bash\r\n\r\n  {\r\n    'starting_from': 0, \r\n    'number_returned': 1, \r\n    'cursor_id': 0, \r\n    'data': [\r\n      {\r\n        '_id': ObjectId('586a11f80d23915c7ec76f01'), \r\n        'author': 'Steve Newman', \r\n        'title': 'How to swim'\r\n      }\r\n    ]\r\n  }\r\n  \r\n\r\n**Available Functions**\r\n\r\n- find (colname, spec, offset = 0, limit = 1)\r\n- findone (colname, spec): equivalent with find (colname, spec, 0, 1)\r\n- findall (colname, spec): equivalent with find (colname, spec, 0, -1)\r\n- insert (colname, docs, continue_on_error = 0)\r\n- update (colname, spec, doc)\r\n- updateone (colname, spec, doc)\r\n- upsert (colname, spec, doc)\r\n- upsertone (colname, spec, doc)\r\n- delete (colname, spec, flag = 0)\r\n- findkc (colname, spec, offset = 0, limit = 1): after finidhing search, it keeps cursor alive. then you can use 'get_more()'\r\n- get_more (colname, cursor_id, num_to_return): cursor_id can be got from (findkc()'s result).data [\"cursor_id\"]\r\n- kill_cursors (cursor_ids): if you use findkc() and stop fetching documents, you should mannually call this.\r\n\r\nNote: User authorization is not supported yet.\r\n\r\n\r\nRedis\r\n---------\r\n\r\n.. code-block:: python\r\n\r\n  dbo = aquests.redis (\"127.0.0.1:6379\")\r\n  dbo.get (\"session-5ae675bc\")\r\n  dbo.lrange (\"user-saved-docs\", 0, 3)\r\n  aquests.fetchall ()\r\n\r\nReturns,\r\n\r\n.. code-block:: bash\r\n  \r\n  response-of-session-5ae675bc\r\n  \r\n  [32534, 3453, 6786]\r\n\r\nPossibly you can use all `Redis commands`_.\r\n\r\n.. _`Redis commands`: https://redis.io/commands\r\n\r\n\r\nNote: User authorization is not supported yet.\r\n\r\n\r\nSQLite3 For Fast App Prototyping\r\n---------------------------------\r\n\r\nUsage is almost same with PostgreSQL. This service IS NOT asynchronous BUT just emulating.\r\n\r\n.. code:: python\r\n  \r\n  dbo = aquests.sqlite3 (\"sqlite3.db\")\r\n  dbo.execute (\"\"\"\r\n    drop table if exists people;\r\n    create table people (name_last, age);\r\n    insert into people values ('Cho', 42);\r\n  \"\"\")\r\n  aquests.fetchall ()\r\n  \r\n  \r\nRequests Parameters\r\n========================\r\n\r\nFor get, post*, put*, upload, delete, options, trace parameters are the same.\r\n\r\n.. code-block:: python\r\n\r\n  aquests.get (url, params = None, headers = None, auth = None, meta = {})\r\n  \r\n- url: request url string\r\n- params: None or dictionary, if it provide with get method, it will be attached on tail of url with '?'\r\n- headers: None or dictionary\r\n- auth: None or tuple (username, password)\r\n- meta: dictionary\r\n\r\nFor Websocket,\r\n\r\n.. code-block:: python\r\n\r\n  aquests.ws (url, params = None, headers = None, auth = None, meta = {})\r\n  \r\n- url: request url string, should start with 'ws://' or 'wss://'(SSL Websocket)\r\n- params: string, bytes or tuple. if messages is not string you specify message type code using tuple like (ws.OPCODE_PING, b\"hello\"), you can find OPCODE list, 'from aquests.protocols import ws'. CAUTION. if your params type is bytes type, opcode will automatically be OPCODE_BINARY and string type, be OPCODE_TEXT. and opcode is inffluent to receiver. if you avoid auto opcode, specify opcode with tuple.\r\n  \r\n    * ws.OPCODE_TEXT\r\n    * ws.OPCODE_BINARY\r\n    * ws.OPCODE_CONTINUATION\r\n    * ws.OPCODE_PING\r\n    * ws.OPCODE_PONG\r\n    * ws.OPCODE_CLOSE\r\n    \r\n- headers: None or dictionary\r\n- auth: None or tuple (username, password)\r\n- meta: dictionary\r\n\r\nFor rpc, grpc stub creation:\r\n\r\n.. code-block:: python\r\n\r\n  stub = aquests.rpc (url, headers = None, auth = None, meta = {})\r\n  stub = aquests.grpc (url, headers = None, auth = None, meta = {})\r\n  \r\n- url: request url string\r\n- headers: None or dictionary\r\n- auth: None or tuple (username, password)\r\n- meta: dictionary\r\n\r\nNote: stub's methods and parameters are defined by RPC service providers\r\n\r\nFor postgresql, mongodb, redis dbo creation:\r\n\r\n.. code-block:: python\r\n\r\n  dbo = aquests.postgresql (server, dbname = None, auth = None, meta = {})\r\n  dbo = aquests.mongodb (server, dbname = None, auth = None, meta = {})  \r\n  dbo = aquests.redis (server, dbname = None, auth = None, meta = {})\r\n  \r\n- server: address:port formated string\r\n- dbname: None or string\r\n- auth: None or tuple (username, password)\r\n- meta: dictionary\r\n\r\nNote: stub's methods and parameters are defined by database engines. Please read above related chapters But SQL based postgresql has only 1 method and parameters - execute(sql) or do(sql) just for your convinience.\r\n\r\n\r\nHistory\r\n=========\r\n\r\n- 0.6.7: change socket closing log message\r\n\r\n- 0.6.6: fix asyncon active\r\n\r\n- 0.6.4.2: license changed from BSD to MIT\r\n\r\n- 0.6.4.1: fix await_fifo bug\r\n\r\n- 0.6.3: fix lifetime,  tmap\r\n\r\n- 0.6.2: change queue list -> deque\r\n\r\n- 0.6.1: fix websocket text data encoding\r\n\r\n- 0.6: \r\n  \r\n  * add configure option: allow_redirects\r\n  * new response.history\r\n  * fix 30x redirection\r\n  * fix 401 unauthorized\r\n  \r\n- 0.5.2: remove ready_producer_fifo, this will be used only serverside\r\n- 0.5.1: change from list to deque on producer_fifo\r\n- 0.4.33: force_http1 applied to https\r\n- 0.4.32: fix http.buffer.list_buffer class\r\n- 0.4.30: add websocket message type detection\r\n- 0.4.28: remove aquests.wss, use aquests.ws with url wss://...\r\n- 0.4.25: fix select.select () divide and conquer\r\n- 0.4.22: fix http2_constreams\r\n- 0.4.21: fix http2 flow control window\r\n- 0.4.20: add configure options: force_http1, http2_constreams\r\n- 0.4.18: url / trailing\r\n- 0.4.17: fix finding end of data on http2\r\n- 0.4.16: fix http2 disconnecting behavior\r\n- 0.4.10: fix xmlrpc stub url / trailing\r\n- 0.4.9: changed response properties - request.method -> method, request.server -> server, request.dbname -> dbname and request.params -> params\r\n- 0.4.4: add lib.athreads\r\n- 0.4.2: fix http2 large content download\r\n- 0.4.1: add a few examples\r\n- 0.4: add timeout feature\r\n- 0.3.10: fix http2 frame length validation, add cookie feature\r\n- 0.3.8: fix dbo request shutdown behavior\r\n- 0.3.1: add HEAD, OPTIONS, TRACE\r\n- 0.3: fix installation error\r\n- 0.2.13: change default display callback\r\n- 0.2.10: fix xmlrpc\r\n\r\n", 
    "lcname": "aquests", 
    "bugtrack_url": null, 
    "github": false, 
    "name": "aquests", 
    "license": "MIT", 
    "summary": "Asynchronous Multiplexing HTTP2/DBO Requests", 
    "split_keywords": [], 
    "author_email": "hansroh@gmail.com", 
    "urls": [
        {
            "has_sig": false, 
            "upload_time": "2017-02-14T11:45:40", 
            "comment_text": "", 
            "python_version": "source", 
            "url": "https://pypi.python.org/packages/62/41/1404f08acad90f69ed8bdebab89adb3e4415a1099cc9c0d4623cdc0463b4/aquests-0.6.7.1.tar.gz", 
            "md5_digest": "65292d19a6a1aca73844eae9240fc927", 
            "downloads": 0, 
            "filename": "aquests-0.6.7.1.tar.gz", 
            "packagetype": "sdist", 
            "path": "62/41/1404f08acad90f69ed8bdebab89adb3e4415a1099cc9c0d4623cdc0463b4/aquests-0.6.7.1.tar.gz", 
            "size": 109002
        }
    ], 
    "_id": null, 
    "cheesecake_installability_id": null
}