S3Uploader: Simple S3 upload script

For a project of mine I needed a simple script or application to upload files to an Amazon S3 bucket. I couldn’t quickly find something that scratched my itch so I wrote  a simple application called S3Uploader.

Amazon S3 Logo

Using S3Uploader is very simple, once you have your S3 bucket and credentials configured, simply install the application using

and you can start uploading files to your bucket, for example

The code is quite simple and is actually very suitable or a beginner to learn from. You can check it out the code from the source repository on BitBucket and the PyPi package page.

Diving into the code

The most interesting bit, the actual uploading of a file happens in the s3uploader.uploader module:

import os

import boto3

class S3Uploader():
 def __init__(self, bucket_name):
 self.s3 = boto3.resource('s3')
 self.bucket_name = bucket_name
 self.count = 0

def store_file(self, filename, store_paths=False, verbose=False):
 with open(filename, 'rb') as file:
 if store_paths:
 s3_object_name = filename
 s3_object_name = os.path.basename(filename)

if verbose:
 print(' Uploading file ' + filename + ' to object ' + s3_object_name)

 obj = self.s3.Object(self.bucket_name, s3_object_name).put(Body=file)
 except Exception as e:
 print('!! Got an error while uploading file {file} to the S3 bucket {bucket}. Error: {error}'.format(
 self.count += 1

It’s a very simple class that takes a path to a file and uploads it to the S3 bucket. The interesting bit here is at the beginning of the store_file method where we extract the filename from the path using the os.path.basename function.

Another interesting piece of code can be found in the s3uploader.main module where we use the argparse module to process arguments.

Creating an ArgumentParser is quite easy using the following code (bits have een removed from the original source for clarity)

import argparse

parser = argparse.ArgumentParser(description='Upload files to an Amazon S3 bucket. version 0.1')
parser.add_argument('--verbose', action='store_true', help='Give verbose output while uploading files')
parser.add_argument('--store-paths', action='store_true', help='Store the file path in the S3 object name')
parser.add_argument('bucket', help='Name of the S3 bucket to use')
parser.add_argument('file', nargs='+', help='File to upload to the S3 bucket')

The  action=‘store_true’ flag changes our argument parsing so the parameter acts as a flag instead of variable. The bucket parameter requires a variable and the –verbose parameter gives a true or false value.

The  nargs=‘+’  on the file parameter enables the user to pass multiple values for our file parameter.

We can process the parameters from argparse like so

args = parser.parse_args()

uploader = S3Uploader(args.bucket)
verbose = args.verbose
store_paths = args.store_paths

for filename in args.file:

Interesting parts here is the change from the parameter name store-paths to store_paths and the fact that the file parameter is an iterable due to the nargs=‘+’  option.

This little application was a quick and dirty job and can be improved. I’ll save that for a later time and try and write more about it if it’s interesting.

Leave a Reply