# boto3_type_annotations
A programmatically created package that defines `boto3` services as stand in classes with type annotations. `boto3` is
an incredibly useful, well designed interface to the AWS API. However, we live in an age where even free IDEs like
PyCharm CE have full code completion (IntelliSense). Because `boto3`'s services are created at runtime, IDEs aren't
able to index its code in order to provide code completion or infer the type of these services or of the objects created
by them. Even if it was able to do so, clients and service resources are created using a service agnostic factory method
and are only identified by a string argument of that method. IDEs don't parse arguments to infer the return type of a
method, and they probably shouldn't. Meaning that the only way for an IDE to know the type of a client created by
`boto3.client('<service>')` is for it to be explicitly declared in type annotations, type comments, or docstrings, which
brings us back to the original problem of services being defined at runtime. All of that to say that working with
`boto3` can be very frustrating at times.
To reduce this frustration, `boto3_type_annotations` defines stand in classes for the clients, service resources,
paginators, and waiters provided by `boto3`'s services. Even though these services are created by `boto3` are created at
runtime, they are still full fledged Python objects, and AWS has been nice enough to include documentation in the
docstrings of these objects' methods. By parsing those docstrings, we can retrieve the types of method
arguments--also, which arguments are required and which may be omitted--and the types of their return
values. With that, we have everything we need to create objects which mimic the class structure of `boto3`'s objects.
And with Python's `typing` module, we can annotate the methods of the stand in objects with the types which we've
parsed. What this means is that we can use these stand in objects to declare the type of `boto3` service objects in our
own code.
## With or Without Docstrings
This package is available both with docstrings, named `boto3_type_annotations_with_docs` on PyPi (which contains the
same documentation you'll find online), and without, named `boto3_type_annotations` on PyPi. The reason for this is
that, for a python package, `boto3_type_annotations_with_docs` is HUGE. `boto3_type_annotations` is pretty large itself
at 2.2 MB, but `boto3_type_annotations_with_docs` dwarfs it at 41 MB. Being that `boto3` and `botocore` add up to be 34
MB, this is likely not ideal for many use cases. However, there are use cases in which you may want documentation in
your IDE, during development for example. A possible workflow for this use case is detailed below.
## Installation
Without docs:
```
pip install boto3_type_annotations
```
With docs:
```
pip install boto3_type_annotations_with_docs
```
## Usage
Regardless of which deployment package you install, you'll still import the same package, `boto3_type_annotations`.
Its constituent packages and modules can be used to declare the type of `boto3` objects. For instance, everybody's
favorite, S3:
```python
import boto3
from boto3_type_annotations.s3 import Client, ServiceResource
from boto3_type_annotations.s3.waiter import BucketExists
from boto3_type_annotations.s3.paginator import ListObjectsV2
# With type annotations
client: Client = boto3.client('s3')
client.create_bucket(Bucket='foo') # Not only does your IDE knows the name of this method,
# it knows the type of the `Bucket` argument too!
# It also, knows that `Bucket` is required, but `ACL` isn't!
# Waiters and paginators and defined also...
waiter: BucketExists = client.get_waiter('bucket_exists')
waiter.wait('foo')
paginator: ListObjectsV2 = client.get_paginator('list_objects_v2')
response = paginator.paginate(Bucket='foo')
# Along with service resources.
resource: ServiceResource = boto3.resource('s3')
bucket = resource.Bucket('bar')
bucket.create()
# With type comments
client = boto3.client('s3') # type: Client
response = client.get_object(Bucket='foo', Key='bar')
# In docstrings
class Foo:
def __init__(self, client):
"""
:param client: It's an S3 Client and the IDE is gonna know what it is!
:type client: Client
"""
self.client = client
def bar(self):
"""
:rtype: Client
"""
self.client.delete_object(Bucket='foo', Key='bar')
return self.client
```
## How Is This Package Different From `pyboto3`?
`pyboto3` has been a useful package which was created for the same purpose and using the same methodology as this
package. It does have its shortcomings, though. For one, it only defines clients, no service resources, waiters, or
paginators. Two, it defines2 clients as modules when the objects created by `boto3` are classes. This seems
nitpicky until you realize that modules can't be used to declare type with type annotations. Even a variable in the
outermost scope of a module would require rst docstring to declare its type. Also, and this is actually is nitpicky,
the package structure doesn't mimic that of `boto3`--which you can see in the documentation i.e. `sqs.ServiceResource`,
`s3.Bucket`, `ec2.waiter.InstanceExists`. Though I don't want to purport that this is perfectly one to one with what is
in the docs. For instance, there's not much consistency in the docs as far as casing. You'll sometimes see
`S3.Waiter.BucketExists` and in other places `sqs.Bucket`. I chose to go with the pep8 guidelines where module names are
in snake case and classes are in Pascal case.
## Development Workflow With Docstring
As mentioned above, there may be scenarios in which you would want to have docstrings in development, but not want
to package a 41MB dependency with your production code. To accommodate this and similar scenarios, I decided to provide
two deployment packages, each containing a `boto3_type_annotations` package. So, one workflow may be to have two
requirements files: requirements.txt and requirements-dev.txt (`boto3` does something similar in that they have
requirements.txt for the API resource and requirements-docs.txt for building documentation.). These two files would
look like this:
**requirements.txt**
```
boto3_type_annotations
# other dependencies
```
**requirements-dev.txt**
```
boto3_type_annotations_with_docs
# other dependencies
```
You would then install `requirements.txt` in production and `requirements-dev.txt` in development. Because both
deployment packages define the `boto3_type_annotations` package, you won't have to change your code. You just need to
install the appropriate deployment package.
## Custom Builds
In cases when you're only using a small number of `boto3` services, you may not want to depend on a package containing
every service available. To provide a bit more flexibility, this package provides a way to create a custom build of
the `boto3_type_annotations` package. The `configs/` directory contains configurations for `boto3_type_annotations` and
`boto3_type_annnotations_with_docs`, along with a couple example configurations.
```yaml
services: # A list of services. Use `boto3.session.Session.get_available_services()` to view services.
- ec2
- rds
- sqs
- sns
- lambda
- s3
with_docs: true # Include docstrings.
with_clients: true # Include client classes
with_service_resources: true # Include service resources.
with_paginators: true # Include paginators
with_waiters: true # Include waiters
package_name: boto3_type_annotations_essentials # The name of the package.
module_name: boto3_type_annotations # The name of the module.
version: 0.2.4 # Version of the package.
readme: README.md # Path to readme file.
license: LICENSE # Path to file containing license.
```
The preceding configuration is the contents of `config/example.essentials.yaml`. When `build.py` is run with this
config, it will parse and write the ec2, rds, sqs, sns, lambda, and s3 services with docstrings and including all
clients, service resources, paginators, and waiters.
```bash
$ python build_scripts/build.py ../configs/example.essentials.yaml
```
It will create a directory named `boto3_type_annotations_essentials`
in the root directory of the repository. That directory will contain a python module named `boto3_type_annotations`,
a license file, and a `setup.py` file. Now all you need to do is package everything up and install it.
```bash
$ python setup.py sdist bdist_wheel
$ pip3 install dist/boto3_type_annotations_essentials-0.2.4-py3-none-any.whl --user
```
## TODO
- Create an "essentials" deployment package only containing often used services like Lambda, S3, SQS, and CloudFormation
- Package related services into separate deployment packages, to create smaller packages containing only services
which are essential to a certain use case, group EC2 and RDS for instance.
- ~~Create custom builds. If a project only uses S3's service resource, provide a way to build a deployment package
containing just that package. This would require some sort of configuration and more mature build script.~~
- Reduce the size of `boto3_type_annotations_with_docs`. I'm already cutting out extraneous new lines and some
whitespaces which reduced the size by 10 MB(!), but I'd like to see it closer to the 34 MB of `boto3` + `botocore`.
Raw data
{
"_id": null,
"home_page": "https://github.com/alliefitter/boto3_type_annotations",
"name": "boto3-type-annotations",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "",
"author": "Allie Fitter",
"author_email": "fitterj@gmail.com",
"download_url": "https://files.pythonhosted.org/packages/9f/f1/fd99a0d0e52fe26ffaf601cce72757e740a45e0d08d942cdedf4935dcd0f/boto3_type_annotations-0.3.1.tar.gz",
"platform": "",
"description": "# boto3_type_annotations\n\nA programmatically created package that defines `boto3` services as stand in classes with type annotations. `boto3` is \nan incredibly useful, well designed interface to the AWS API. However, we live in an age where even free IDEs like \nPyCharm CE have full code completion (IntelliSense). Because `boto3`'s services are created at runtime, IDEs aren't \nable to index its code in order to provide code completion or infer the type of these services or of the objects created \nby them. Even if it was able to do so, clients and service resources are created using a service agnostic factory method\nand are only identified by a string argument of that method. IDEs don't parse arguments to infer the return type of a\nmethod, and they probably shouldn't. Meaning that the only way for an IDE to know the type of a client created by \n`boto3.client('<service>')` is for it to be explicitly declared in type annotations, type comments, or docstrings, which\nbrings us back to the original problem of services being defined at runtime. All of that to say that working with \n`boto3` can be very frustrating at times.\n\nTo reduce this frustration, `boto3_type_annotations` defines stand in classes for the clients, service resources, \npaginators, and waiters provided by `boto3`'s services. Even though these services are created by `boto3` are created at\nruntime, they are still full fledged Python objects, and AWS has been nice enough to include documentation in the \ndocstrings of these objects' methods. By parsing those docstrings, we can retrieve the types of method \narguments--also, which arguments are required and which may be omitted--and the types of their return \nvalues. With that, we have everything we need to create objects which mimic the class structure of `boto3`'s objects. \nAnd with Python's `typing` module, we can annotate the methods of the stand in objects with the types which we've \nparsed. What this means is that we can use these stand in objects to declare the type of `boto3` service objects in our \nown code.\n\n\n\n## With or Without Docstrings\n\nThis package is available both with docstrings, named `boto3_type_annotations_with_docs` on PyPi (which contains the \nsame documentation you'll find online), and without, named `boto3_type_annotations` on PyPi. The reason for this is \nthat, for a python package, `boto3_type_annotations_with_docs` is HUGE. `boto3_type_annotations` is pretty large itself \nat 2.2 MB, but `boto3_type_annotations_with_docs` dwarfs it at 41 MB. Being that `boto3` and `botocore` add up to be 34 \nMB, this is likely not ideal for many use cases. However, there are use cases in which you may want documentation in \nyour IDE, during development for example. A possible workflow for this use case is detailed below.\n\n## Installation\n\nWithout docs:\n```\npip install boto3_type_annotations\n```\n\nWith docs:\n```\npip install boto3_type_annotations_with_docs\n```\n\n## Usage\n\nRegardless of which deployment package you install, you'll still import the same package, `boto3_type_annotations`.\nIts constituent packages and modules can be used to declare the type of `boto3` objects. For instance, everybody's \nfavorite, S3:\n\n```python\nimport boto3\nfrom boto3_type_annotations.s3 import Client, ServiceResource\nfrom boto3_type_annotations.s3.waiter import BucketExists\nfrom boto3_type_annotations.s3.paginator import ListObjectsV2\n\n# With type annotations\n\nclient: Client = boto3.client('s3')\nclient.create_bucket(Bucket='foo') # Not only does your IDE knows the name of this method, \n # it knows the type of the `Bucket` argument too!\n # It also, knows that `Bucket` is required, but `ACL` isn't!\n\n# Waiters and paginators and defined also...\n\nwaiter: BucketExists = client.get_waiter('bucket_exists')\nwaiter.wait('foo')\n\npaginator: ListObjectsV2 = client.get_paginator('list_objects_v2')\nresponse = paginator.paginate(Bucket='foo')\n\n# Along with service resources.\n\nresource: ServiceResource = boto3.resource('s3')\nbucket = resource.Bucket('bar')\nbucket.create()\n\n# With type comments\n\nclient = boto3.client('s3') # type: Client\nresponse = client.get_object(Bucket='foo', Key='bar')\n\n# In docstrings\n\nclass Foo:\n def __init__(self, client):\n \"\"\"\n :param client: It's an S3 Client and the IDE is gonna know what it is!\n :type client: Client\n \"\"\"\n self.client = client\n\n def bar(self):\n \"\"\"\n :rtype: Client\n \"\"\"\n self.client.delete_object(Bucket='foo', Key='bar')\n return self.client\n```\n\n## How Is This Package Different From `pyboto3`?\n\n`pyboto3` has been a useful package which was created for the same purpose and using the same methodology as this \npackage. It does have its shortcomings, though. For one, it only defines clients, no service resources, waiters, or \npaginators. Two, it defines2 clients as modules when the objects created by `boto3` are classes. This seems \nnitpicky until you realize that modules can't be used to declare type with type annotations. Even a variable in the \noutermost scope of a module would require rst docstring to declare its type. Also, and this is actually is nitpicky, \nthe package structure doesn't mimic that of `boto3`--which you can see in the documentation i.e. `sqs.ServiceResource`, \n`s3.Bucket`, `ec2.waiter.InstanceExists`. Though I don't want to purport that this is perfectly one to one with what is\nin the docs. For instance, there's not much consistency in the docs as far as casing. You'll sometimes see \n`S3.Waiter.BucketExists` and in other places `sqs.Bucket`. I chose to go with the pep8 guidelines where module names are\nin snake case and classes are in Pascal case.\n\n## Development Workflow With Docstring\n\nAs mentioned above, there may be scenarios in which you would want to have docstrings in development, but not want\nto package a 41MB dependency with your production code. To accommodate this and similar scenarios, I decided to provide \ntwo deployment packages, each containing a `boto3_type_annotations` package. So, one workflow may be to have two \nrequirements files: requirements.txt and requirements-dev.txt (`boto3` does something similar in that they have \nrequirements.txt for the API resource and requirements-docs.txt for building documentation.). These two files would \nlook like this:\n\n**requirements.txt**\n```\nboto3_type_annotations\n# other dependencies\n```\n\n**requirements-dev.txt**\n```\nboto3_type_annotations_with_docs\n# other dependencies\n```\n\nYou would then install `requirements.txt` in production and `requirements-dev.txt` in development. Because both \ndeployment packages define the `boto3_type_annotations` package, you won't have to change your code. You just need to \ninstall the appropriate deployment package.\n\n## Custom Builds\n\nIn cases when you're only using a small number of `boto3` services, you may not want to depend on a package containing \nevery service available. To provide a bit more flexibility, this package provides a way to create a custom build of\nthe `boto3_type_annotations` package. The `configs/` directory contains configurations for `boto3_type_annotations` and\n`boto3_type_annnotations_with_docs`, along with a couple example configurations.\n\n```yaml\nservices: # A list of services. Use `boto3.session.Session.get_available_services()` to view services. \n - ec2\n - rds\n - sqs\n - sns\n - lambda\n - s3\nwith_docs: true # Include docstrings.\nwith_clients: true # Include client classes\nwith_service_resources: true # Include service resources.\nwith_paginators: true # Include paginators\nwith_waiters: true # Include waiters\npackage_name: boto3_type_annotations_essentials # The name of the package.\nmodule_name: boto3_type_annotations # The name of the module.\nversion: 0.2.4 # Version of the package.\nreadme: README.md # Path to readme file.\nlicense: LICENSE # Path to file containing license.\n```\n\nThe preceding configuration is the contents of `config/example.essentials.yaml`. When `build.py` is run with this\nconfig, it will parse and write the ec2, rds, sqs, sns, lambda, and s3 services with docstrings and including all\nclients, service resources, paginators, and waiters. \n\n```bash\n$ python build_scripts/build.py ../configs/example.essentials.yaml\n```\n\nIt will create a directory named `boto3_type_annotations_essentials`\nin the root directory of the repository. That directory will contain a python module named `boto3_type_annotations`,\na license file, and a `setup.py` file. Now all you need to do is package everything up and install it.\n\n```bash\n$ python setup.py sdist bdist_wheel\n\n$ pip3 install dist/boto3_type_annotations_essentials-0.2.4-py3-none-any.whl --user\n\n```\n\n## TODO\n\n- Create an \"essentials\" deployment package only containing often used services like Lambda, S3, SQS, and CloudFormation\n\n- Package related services into separate deployment packages, to create smaller packages containing only services\n which are essential to a certain use case, group EC2 and RDS for instance.\n\n- ~~Create custom builds. If a project only uses S3's service resource, provide a way to build a deployment package \n containing just that package. This would require some sort of configuration and more mature build script.~~\n\n- Reduce the size of `boto3_type_annotations_with_docs`. I'm already cutting out extraneous new lines and some\n whitespaces which reduced the size by 10 MB(!), but I'd like to see it closer to the 34 MB of `boto3` + `botocore`.\n\n\n",
"bugtrack_url": null,
"license": "MIT License",
"summary": "Type annotations for boto3. Adds code completion in IDEs such as PyCharm.",
"version": "0.3.1",
"project_urls": {
"Homepage": "https://github.com/alliefitter/boto3_type_annotations"
},
"split_keywords": [],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "967f9477016e1bfd5b87ce9fcbadd3363ca3b7eb07ed2fe43137588a44dfc06c",
"md5": "797cbd73d7ce71c6739e43ea21fc1f18",
"sha256": "52e007ac2ab792abee42aaac3019c2acc9781a84e79e1283a5bc64164225435e"
},
"downloads": -1,
"filename": "boto3_type_annotations-0.3.1-py3-none-any.whl",
"has_sig": false,
"md5_digest": "797cbd73d7ce71c6739e43ea21fc1f18",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 338432,
"upload_time": "2019-05-02T06:19:11",
"upload_time_iso_8601": "2019-05-02T06:19:11.695214Z",
"url": "https://files.pythonhosted.org/packages/96/7f/9477016e1bfd5b87ce9fcbadd3363ca3b7eb07ed2fe43137588a44dfc06c/boto3_type_annotations-0.3.1-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
},
{
"comment_text": "",
"digests": {
"blake2b_256": "9ff1fd99a0d0e52fe26ffaf601cce72757e740a45e0d08d942cdedf4935dcd0f",
"md5": "6ad9eafe72a78246aa08d73b6bdbdca1",
"sha256": "e5682093d035e935c8189f24c601ef6eaccedfcd1e642ba7922429d093b1b885"
},
"downloads": -1,
"filename": "boto3_type_annotations-0.3.1.tar.gz",
"has_sig": false,
"md5_digest": "6ad9eafe72a78246aa08d73b6bdbdca1",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 151166,
"upload_time": "2019-05-02T06:19:13",
"upload_time_iso_8601": "2019-05-02T06:19:13.605596Z",
"url": "https://files.pythonhosted.org/packages/9f/f1/fd99a0d0e52fe26ffaf601cce72757e740a45e0d08d942cdedf4935dcd0f/boto3_type_annotations-0.3.1.tar.gz",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2019-05-02 06:19:13",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "alliefitter",
"github_project": "boto3_type_annotations",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"requirements": [
{
"name": "boto3",
"specs": []
},
{
"name": "botocore",
"specs": []
},
{
"name": "docstring",
"specs": []
},
{
"name": "docutils",
"specs": []
},
{
"name": "jmespath",
"specs": []
},
{
"name": "python-dateutil",
"specs": []
},
{
"name": "PyYAML",
"specs": []
},
{
"name": "s3transfer",
"specs": []
},
{
"name": "six",
"specs": []
},
{
"name": "urllib3",
"specs": []
}
],
"lcname": "boto3-type-annotations"
}
JSON
