# Larry
Larry is a library of utilities for common data tasks using AWS for data science and data engineering projects.
While boto3 is a great interface for interacting with AWS services, it can be overly complex for data scientists and
others who want to perform straightforward operations on data. Rather than spend time
worrying about API-specific interactions and parameters. Larry makes it easy to use services like S3, MTurk,
and other data-oriented AWS services in a far more **functional** manner to let you focus on the data rather than
syntax. This library is designed to make getting tasks completed in Jupyter Notebooks or AWS Lambda functions as
easy as possible by providing simplified interfaces while still giving you access to the underlying boto3 libraries
when you need them.
## Installation
```
pip install larry
```
In addition, you can add Larry to your AWS Lambda functions by adding one of the following public Layers:
* us-east-1:
* Larry: `arn:aws:lambda:us-east-1:981332165467:layer:Larry:7`
* Larry with Jinja2: `arn:aws:lambda:us-east-1:981332165467:layer:LarryWithJinja:7`
* Larry with Jinja2 and Pillow: `arn:aws:lambda:us-east-1:981332165467:layer:LarryWithJinjaPillow:7`
* us-west-2:
* Larry: `arn:aws:lambda:us-west-2:981332165467:layer:Larry:7`
* Larry with Jinja2: `arn:aws:lambda:us-west-2:981332165467:layer:LarryWithJinja:7`
* Larry with Jinja2 and Pillow: `arn:aws:lambda:us-west-2:981332165467:layer:LarryWithJinjaPillow:7`
## Configuring your AWS session
By default Larry creates a boto3 session using your default AWS credentials that can be configured using the
[AWS CLI](https://aws.amazon.com/cli/). To use a different profile, you can change using the following:
```python
import larry as lry
lry.set_session(profile_name='my_profile')
```
Alternatively, you can pass AWS credentials directly
```python
import larry as lry
lry.set_session(aws_access_key_id='XXXXXXXXXX', aws_secret_access_key='XXXXXXXXXXXXX')
```
## Functional S3 interactions
When using boto3 alone, the following is how you would read in a JSON formatted object from S3 into a dict:
```python
import boto3
import json
resource = boto3.resource('s3')
obj = resource.Bucket('mybucket').Object(key='myfile.json').get()
contents = obj['Body'].read()
my_dict = json.loads(contents.decode('utf-8'))
```
In contrast, Larry takes care of all those steps for you and let's you simply call one function to get your data.
```python
import larry as lry
my_dict = lry.s3.read_dict(bucket='mybucket', key='myfile.json')
```
In addition to accessing data using bucket/key pairs, you can S3 URIs like those commonly used in SageMaker.
```python
my_dict2 = lry.s3.read_dict(uri='s3://mybucket/myfile.json')
```
To write files to S3, simply call `write_object` to write your object out in the appropriate format:
```python
# Write json to S3
my_dict = {'key': 'val'}
lry.s3.write_object(my_dict, bucket='mybucket', key='myfile.json')
# Write a list of strings to S3 as rows
my_list = ['a','b','c','d']
lry.s3.write_object(my_list, bucket='mybucket', key='myfile.txt')
# Write a JSON lines file to S3 (commonly used for SageMaker manifest files)
my_dictlist = [{'a': 1}, {'b': 2}, {'c': 3}]
lry.s3.write_object(my_dictlist, bucket='mybucket', key='myfile.jsonl')
```
## Powerful MTurk extensions
Larry is especially useful for services like MTurk which have more complex interaction patterns and legacy aspects
of their APIs. The MTurk module includes a number of features to make using MTurk much easier:
* Easy toggling between sandbox and production clients (no more copy/pasting in endpoint urls)
* Worker answers are converted into easy to access dict objects (no more QuestionFormAnswer XML!)
* Potentially expensive operations such as list_hits and list_assignments_for_hit return generators
* Utilities are included to easily generate HTMLQuestion and ExternalQuestion XML objects with integrated Jinja2 templating
* Helpers to enable state data in the RequesterAnnotation field
The combination of these features means that creating a HIT in MTurk is as easy as the following:
```python
import larry as lry
# Indicate we want to use the production environment
lry.mturk.use_production()
# Identify the task template we want to use and the parameters we'll populate in the template
template_uri = 's3://mybucket/templates/imageCat.html'
task_data = {'image_url': 'https://mywebsite.com/images/233.jpg'}
# Add some tracking information we can pass through in our RequesterAnnotation
task_data['request_id'] = 'MY_TRACKING_ID'
# Create a HIT
hit = lry.mturk.create_hit(title='Test task', description='Categorize images',
reward_cents=5, max_assignments=5, lifetime=86400, assignment_duration=600,
question_template_uri=template_uri, template_context=task_data, annotation=task_data)
# Display where the HIT can be viewed on the Worker website
print('HIT {} created, preview at {}'.format(hit.hit_id, hit.preview))
```
Getting the results from that task is as simple as the following:
```python
import larry as lry
hit_id = 'HIT_ID_FROM_EARLIER'
# Indicate we want to use the production environment
lry.mturk.use_production()
# retrieve the HIT
hit = lry.mturk.get_hit(hit_id)
# retrieve the requester annotation data
task_data = hit.annotation
# get the results
hit.retrieve_assignments()
for assignment in hit.assignments:
print('Worker {} responded with {}'.format(assignment.worker_id, assignment.answer['category']))
```
More features will be added over time, feel free to submit your feature suggestions.
Raw data
{
"_id": null,
"home_page": "https://github.com/dschultz0/larry",
"name": "larry",
"maintainer": "",
"docs_url": null,
"requires_python": "",
"maintainer_email": "",
"keywords": "larry data aws boto3 mturk s3",
"author": "Dave Schultz",
"author_email": "djschult@gmail.com",
"download_url": "",
"platform": null,
"description": "# Larry\nLarry is a library of utilities for common data tasks using AWS for data science and data engineering projects. \nWhile boto3 is a great interface for interacting with AWS services, it can be overly complex for data scientists and \nothers who want to perform straightforward operations on data. Rather than spend time\nworrying about API-specific interactions and parameters. Larry makes it easy to use services like S3, MTurk, \nand other data-oriented AWS services in a far more **functional** manner to let you focus on the data rather than \nsyntax. This library is designed to make getting tasks completed in Jupyter Notebooks or AWS Lambda functions as \neasy as possible by providing simplified interfaces while still giving you access to the underlying boto3 libraries\nwhen you need them.\n\n## Installation\n```\npip install larry\n```\nIn addition, you can add Larry to your AWS Lambda functions by adding one of the following public Layers:\n* us-east-1:\n * Larry: `arn:aws:lambda:us-east-1:981332165467:layer:Larry:7`\n * Larry with Jinja2: `arn:aws:lambda:us-east-1:981332165467:layer:LarryWithJinja:7`\n * Larry with Jinja2 and Pillow: `arn:aws:lambda:us-east-1:981332165467:layer:LarryWithJinjaPillow:7`\n* us-west-2:\n * Larry: `arn:aws:lambda:us-west-2:981332165467:layer:Larry:7`\n * Larry with Jinja2: `arn:aws:lambda:us-west-2:981332165467:layer:LarryWithJinja:7`\n * Larry with Jinja2 and Pillow: `arn:aws:lambda:us-west-2:981332165467:layer:LarryWithJinjaPillow:7`\n\n## Configuring your AWS session\nBy default Larry creates a boto3 session using your default AWS credentials that can be configured using the \n[AWS CLI](https://aws.amazon.com/cli/). To use a different profile, you can change using the following:\n```python\nimport larry as lry\nlry.set_session(profile_name='my_profile')\n```\nAlternatively, you can pass AWS credentials directly\n```python\nimport larry as lry\nlry.set_session(aws_access_key_id='XXXXXXXXXX', aws_secret_access_key='XXXXXXXXXXXXX')\n```\n\n## Functional S3 interactions\nWhen using boto3 alone, the following is how you would read in a JSON formatted object from S3 into a dict:\n```python\nimport boto3\nimport json\n\nresource = boto3.resource('s3')\nobj = resource.Bucket('mybucket').Object(key='myfile.json').get()\ncontents = obj['Body'].read()\nmy_dict = json.loads(contents.decode('utf-8'))\n```\n\nIn contrast, Larry takes care of all those steps for you and let's you simply call one function to get your data.\n```python\nimport larry as lry\n\nmy_dict = lry.s3.read_dict(bucket='mybucket', key='myfile.json')\n```\nIn addition to accessing data using bucket/key pairs, you can S3 URIs like those commonly used in SageMaker.\n```python\nmy_dict2 = lry.s3.read_dict(uri='s3://mybucket/myfile.json')\n```\n\nTo write files to S3, simply call `write_object` to write your object out in the appropriate format:\n```python\n# Write json to S3\nmy_dict = {'key': 'val'}\nlry.s3.write_object(my_dict, bucket='mybucket', key='myfile.json')\n\n# Write a list of strings to S3 as rows\nmy_list = ['a','b','c','d']\nlry.s3.write_object(my_list, bucket='mybucket', key='myfile.txt')\n\n# Write a JSON lines file to S3 (commonly used for SageMaker manifest files)\nmy_dictlist = [{'a': 1}, {'b': 2}, {'c': 3}]\nlry.s3.write_object(my_dictlist, bucket='mybucket', key='myfile.jsonl')\n```\n\n## Powerful MTurk extensions\nLarry is especially useful for services like MTurk which have more complex interaction patterns and legacy aspects\nof their APIs. The MTurk module includes a number of features to make using MTurk much easier:\n* Easy toggling between sandbox and production clients (no more copy/pasting in endpoint urls)\n* Worker answers are converted into easy to access dict objects (no more QuestionFormAnswer XML!)\n* Potentially expensive operations such as list_hits and list_assignments_for_hit return generators\n* Utilities are included to easily generate HTMLQuestion and ExternalQuestion XML objects with integrated Jinja2 templating\n* Helpers to enable state data in the RequesterAnnotation field\n\nThe combination of these features means that creating a HIT in MTurk is as easy as the following:\n```python\nimport larry as lry\n\n# Indicate we want to use the production environment\nlry.mturk.use_production()\n\n# Identify the task template we want to use and the parameters we'll populate in the template\ntemplate_uri = 's3://mybucket/templates/imageCat.html'\ntask_data = {'image_url': 'https://mywebsite.com/images/233.jpg'}\n\n# Add some tracking information we can pass through in our RequesterAnnotation\ntask_data['request_id'] = 'MY_TRACKING_ID'\n\n# Create a HIT\nhit = lry.mturk.create_hit(title='Test task', description='Categorize images', \n reward_cents=5, max_assignments=5, lifetime=86400, assignment_duration=600, \n question_template_uri=template_uri, template_context=task_data, annotation=task_data)\n\n# Display where the HIT can be viewed on the Worker website\nprint('HIT {} created, preview at {}'.format(hit.hit_id, hit.preview))\n```\nGetting the results from that task is as simple as the following:\n```python\nimport larry as lry\n\nhit_id = 'HIT_ID_FROM_EARLIER'\n\n# Indicate we want to use the production environment\nlry.mturk.use_production()\n\n# retrieve the HIT\nhit = lry.mturk.get_hit(hit_id)\n\n# retrieve the requester annotation data\ntask_data = hit.annotation\n\n# get the results\nhit.retrieve_assignments()\nfor assignment in hit.assignments:\n print('Worker {} responded with {}'.format(assignment.worker_id, assignment.answer['category']))\n```\n\nMore features will be added over time, feel free to submit your feature suggestions.\n\n\n",
"bugtrack_url": null,
"license": "",
"summary": "Library of helper reference for common data tasks using AWS resources such as S3, MTurk and others",
"version": "0.2.12",
"project_urls": {
"Homepage": "https://github.com/dschultz0/larry"
},
"split_keywords": [
"larry",
"data",
"aws",
"boto3",
"mturk",
"s3"
],
"urls": [
{
"comment_text": "",
"digests": {
"blake2b_256": "fb645a9cf9fef74e9ae29e3366ecee7845c1c669a61d6d10aa23cab6795ccd7e",
"md5": "449acdeef394792a7e408c9e27a56357",
"sha256": "c4d6544d22dae9fcc68fff6ac44c48f5e05e8fdd34f425b5895e52cea6057dbd"
},
"downloads": -1,
"filename": "larry-0.2.12-py3-none-any.whl",
"has_sig": false,
"md5_digest": "449acdeef394792a7e408c9e27a56357",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": null,
"size": 74021,
"upload_time": "2022-04-09T17:27:02",
"upload_time_iso_8601": "2022-04-09T17:27:02.706033Z",
"url": "https://files.pythonhosted.org/packages/fb/64/5a9cf9fef74e9ae29e3366ecee7845c1c669a61d6d10aa23cab6795ccd7e/larry-0.2.12-py3-none-any.whl",
"yanked": false,
"yanked_reason": null
}
],
"upload_time": "2022-04-09 17:27:02",
"github": true,
"gitlab": false,
"bitbucket": false,
"codeberg": false,
"github_user": "dschultz0",
"github_project": "larry",
"travis_ci": false,
"coveralls": false,
"github_actions": false,
"lcname": "larry"
}
JSON
