Boto3 concepts you need to know before you start working on it : Meta, Waiters, Collections, Paginators

Hi all, it's great to see you again. As you all know this article is in continuation with gayatri.hashnode.dev/boto3-concepts-you-nee..

If you have not gone through the above article then do visit the Boto3 concepts you need to know before you start working on it : Session, Client, Resource to get the hold on basics of Boto3 and to know how to do the basic setup.

In this article we will be covering below points:

Meta
Collections
Waiters
Paginators

Meta:

As we all know that resource offers limited options to work on the AWS services, that is all operations cannot be performed through resource object. In such situations 'meta' is handy workaround. It helps us to enter into client object from resource. For better understanding refer the below code snippet.

import boto3
my_session = boto3.session.Session(profile_name='s3_user')
s3_resource = my_session.resource('s3')
print(dir(s3_resource))

You can see 'meta' in above screenshot. We need to use that to enter into client object, we can accomplish that as shown below:

import boto3
my_session = boto3.session.Session(profile_name='s3_user')
s3_resource = my_session.resource('s3')
print(dir(s3_resource.meta.client))

dir(s3_resource.meta.client) will list down the methods available for that particular object and as shown in screenshot above we can see all the methods available for client through resource object. You can use any method you want. I find this feature pretty cool. Now let's move to the next feature.

Waiters:

Waiters are used to block the code execution until the certain state is achieved. A number of requests in AWS using boto3 are not instant. In many cases, we want to wait for the request to complete before we move on to the next parts of the script that may depend on a long running process to have been completed. Here we are taking s3 bucket example for understanding purpose. Below example demonstrates the waiters concepts from resource.

import boto3
my_session = boto3.session.Session(profile_name='s3_user')
s3_resource = my_session.resource('s3')
print(dir(s3_resource.Bucket('s3-gk-demo')))

You will see the 'wait_until_exists', 'wait_until_not_exists', these are called 'Waiters'

So now the question is how these waiters work, you will get that from the code below:

import boto3
my_session = boto3.session.Session(profile_name='s3_user')
s3_resource = my_session.resource('s3')

buck_obj = s3_resource.Bucket('s3-gk-demo')
print("Now bucket does exists.....")
buck_obj.wait_until_not_exists()
print("Unnecessary bucket is deleted")

So above code will wait until the particular bucket get's deleted. You can try this code yourself. If the bucket is not deleted in particular time the script will throw an error. This particular time is 100 seconds which is mentioned in official documentation :

Traceback (most recent call last):
  File "s3_demo.py", line 7, in <module>
    buck_obj.wait_until_not_exists()
  File "/Users/gayatrikale/gayatri_scripts/boto3_course_notes/env/lib/python3.7/site-packages/boto3/resources/factory.py", line 369, in do_waiter
    waiter(self, *args, **kwargs)
  File "/Users/gayatrikale/gayatri_scripts/boto3_course_notes/env/lib/python3.7/site-packages/boto3/resources/action.py", line 202, in __call__
    response = waiter.wait(**params)
  File "/Users/gayatrikale/gayatri_scripts/boto3_course_notes/env/lib/python3.7/site-packages/botocore/waiter.py", line 53, in wait
    Waiter.wait(self, **kwargs)
  File "/Users/gayatrikale/gayatri_scripts/boto3_course_notes/env/lib/python3.7/site-packages/botocore/waiter.py", line 358, in wait
    last_response=response,
botocore.exceptions.WaiterError: Waiter BucketNotExists failed: Max attempts exceeded

Below example will demonstrate the waiters concepts from client:

import boto3
my_session = boto3.session.Session(profile_name='s3_user')
s3_client = my_session.client('s3')
waiter = s3_client.get_waiter('bucket_not_exists')
print("Now bucket does exists.....")
waiter.wait(Bucket='s3-gk-demo')
print("Unnecessary bucket is deleted")

Basically both waiters will have same behaviour. You need to decide which to use based on the use case you have.

Collections:

A collection provides an iterable interface to a group of resources. Following are some of the basic features of collections:

Filtering : We can filter the returned data set
Chainability : They return copies of the collection rather than modifying the collection, including a deep copy of any associated operation parameters
Limiting Results : It is possible to limit the number of items returned from a collection by using either the limit() method
Controlling page size : Collections automatically handle paging through results, but you may want to control the number of items returned from a single service operation call. You can do so using the page_size() method
Batch Actions : Some collections support batch actions, which are actions that operate on an entire page of results at a time.

In case of resources you will find the collections in the third set of methods as shown in figure:

The following code demonstrates the use of collections, you can explore the above listed features by visiting the official documentation of Boto3:

Paginators:

According to official documentation :

Some AWS operations return results that are incomplete and require subsequent requests in order to attain the entire result set. The process of sending subsequent requests to continue where a previous request left off is called pagination. Paginators are a feature of boto3 that act as an abstraction over the process of iterating over an entire result set of a truncated API operation.

I will try to explain above definition with example. Let's take example of IAM users. If you have hundreds of IAM users in your account and you want to list them down using list_users() method of client then you will get only one page of result which contains maximum of 50 to 100 users. But you want all the users present in your account so for that you need to leverage paginators as shown below:

import boto3
my_session = boto3.session.Session(profile_name='root')
s3_client = my_session.client('iam')
paginator = s3_client.get_paginator('list_users')
for page in paginator.paginate():
    for user in page['Users']:
        print(user['UserName'])

Above code will list down the usernames for all the users present in your account using pagination.

These are all the basics you need to know to start working with Boto3. By referring both of these articles you will surely get a hold on Boto3 concepts and will be able to write some amazing scripts. Do let me know in the comments section if you have any suggestions or queries.