List S3 buckets easily using Python and CLI

In this blog, we will learn how to list all buckets in our AWS account using Python and AWS CLI. We will learn different ways to list buckets and filter them using tags.

Using CLI to list S3 buckets

Listing all buckets

We can list buckets with CLI in one single command.

aws s3api list-buckets

If you have many buckets, this output will become difficult to follow. However, AWS CLI now supports query parameters. Using query parameters, we can extract the required information from the output.

aws s3api list-buckets --query "Buckets[].Name"

We can also use jq (a lightweight command-line JSON parser) to do advanced things. The following code will print bucket names along with tags associated with them.

for bucket in `aws s3api list-buckets --profile admin-analyticshut | jq .Buckets[].Name | tr -d \"`; do
    echo $bucket
    tags=$(aws s3api get-bucket-tagging --bucket elasticbeanstalk-ap-south-1-195556345987 --profile admin-analyticshut | jq -c '.[][] | {(.Key): .Value}' | tr '\n' '\t')
    echo $tags
done

Listing S3 bucktes using python

We can also easily list down all buckets in the AWS account using Python.

import boto3
from botocore.exceptions import ClientError

#
# Option 1: S3 client list of buckets with name and is creation date
#
s3 = boto3.client('s3')
response = s3.list_buckets()['Buckets']
for bucket in response:
    print('Bucket name: {}, Created on: {}'.format(bucket['Name'], bucket['CreationDate']))

Boto3 also provides us with Bucket resources. We can use its all() function to list all buckets in the AWS account.

import boto3
from botocore.exceptions import ClientError

#
# option 2: S3 resource object will return list of all bucket resources.
# This is useful if we want to further process each bucket resource.
#
s3 = boto3.resource('s3')
buckets = s3.buckets.all()
for bucket in buckets:
    print(bucket)

I also tried bucket filtering based on tags. The account can have hundreds if not thousands, of buckets; the best way to filter them is by using tags. Boto3 does provide a filter method for bucket resources, but I did not find out how to use it. So, I tried a workaround to filter buckets using tag values in Python.

#
# Option 3: Filtering buckets
# This is not working as i have expected. There is filter function for bucket resource.
# But there is no mention of how to use it.
# https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.ServiceResource.buckets
# If anyone know how to use it please let us all know
# buckets1 = s3.buckets.filter(Filters=[{'Name': 'tag:Status', 'Values': ['Logs']}])
# Filtering all buckets with specific tag value
# Same method can be sued to filter buckets with specific string in its name.
for bucket in buckets:
    try:
        tag_set = s3.BucketTagging(bucket.name).tag_set
        for tag in tag_set:
            tag_values = list(tag.values())
            if tag_values[0] == 'Status' and tag_values[1] == 'Logs':
                print(bucket.name)
    except ClientError as e:
        pass
        # print('No Tags')

Please let me know if you find out how to use the filter method for this approach. Here is the actual function given by boto3.

Conclusion

We have learned how to list buckets in the AWS account using CLI and Python. This series will teach us more about performing S3 operations using CLI and Python. If you are interested, please subscribe to the newsletter. See you in the next blog.