Nov 1, 2019

Upload files to Amazon S3 with Go

Connecting your Go application with Amazon S3 is pretty easy and requires only a few lines of code. But if you have no experience with Amazon Web Services, it can get pretty tricky at the beginning to figure out even the most straightforward task.

We are going to see here how to connect to S3 with Go, upload a file from a form to an AWS S3 bucket, download it, and list all items saved on this bucket.

I am going to suppose that you have already:

created an AWS account
created an Amazon Simple Storage (S3) Bucket
generated the credentials to access it (Access key ID and Secret access key)

I am not going to explain those things here since there are plenty of good tutorials on the Internet about it.

Set up your credentials

After having created the S3 Bucket, the very first thing that we are going to do is to save the credentials.

There are a few ways of doing this as described here https://docs.aws.amazon.com/es_es/sdk-for-go/v1/developer-guide/configuring-sdk.html

We are going to use the easiest one using a shared credential file. For this, create a file like described below on ~/.aws/credentials on Linux, macOS or Unix. If using windows, create it on C:\Users\USERNAME\.aws\credential.

[default] 
aws_access_key_id = your_access_key_id 
aws_secret_access_key = your_secret_access_key

Install the SDK

After this, we need to install de AWS SDK for Go:

go get -u github.com/aws/aws-sdk-go/...

and import the AWS packages into your Go application:

import ( 
    "github.com/aws/aws-sdk-go/aws" 
    "github.com/aws/aws-sdk-go/aws/session" 
    "github.com/aws/aws-sdk-go/service/s3"
)

The Code

You can skip all explanations below and check the code directly on https://github.com/antsanchez/goS3example.

I have tried to keep the code as minimal as possible.

Connect to AWS S3

I like to use this code on my main.go file or whatever other file where I can share the session (sess variable) to reuse it.

const (
	AWS_S3_REGION = ""
	AWS_S3_BUCKET = ""
)

var sess = connectAWS()

func connectAWS() *session.Session {
	sess, err := session.NewSession(
            &aws.Config{
                Region: aws.String(AWS_S3_REGION)
            }
        )
	if err != nil {
		panic(err)
	}
	return sess
}

Upload a file from a form to Amazon S3

First, get the file from the Form:

file, header, err := r.FormFile("file")
if err != nil {
    // Do your error handling here
    return
}
defer file.Close()

filename := header.Filename

Then, upload it to S3:

uploader := s3manager.NewUploader(sess)

_, err = uploader.Upload(&s3manager.UploadInput{
    Bucket: aws.String(AWS_S3_BUCKET), // Bucket to be used
    Key:    aws.String(filename),      // Name of the file to be saved
    Body:   file,                      // File
})
if err != nil {
    // Do your error handling here
    return
}

Download a file from Amazon S3

First, create a temporary file where to store the downloaded file:

f, err := os.Create(filename)
if err != nil {
    // Do your error handling here
    return
}

Now, download the file and do whatever you want with it (f)

downloader := s3manager.NewDownloader(sess)
_, err = downloader.Download(f, &s3.GetObjectInput{
    Bucket: aws.String(AWS_S3_BUCKET),
    Key:    aws.String(filename),
})
if err != nil {
    // Do your error handling here
    return
}

List the items of an AWS S3 Bucket

svc := s3.New(sess)
input := &s3.ListObjectsInput{
    Bucket: aws.String(AWS_S3_BUCKET),
}

result, err := svc.ListObjects(input)
if err != nil {
    if aerr, ok := err.(awserr.Error); ok {
        switch aerr.Code() {
        case s3.ErrCodeNoSuchBucket:
            fmt.Println(s3.ErrCodeNoSuchBucket, aerr.Error())
        default:
            fmt.Println(aerr.Error())
        }
    } else {
        // Print the error, cast err to awserr.Error to get the Code and
        // Message from an error.
        fmt.Println(err.Error())
    }
    // Do your error handling here
    return
}

You could now list the items showing the name like this:

w.Header().Set("Content-Type", "text/html")

for _, item := range result.Contents {
    fmt.Fprintf(w, "<li>File %s</li>", *item.Key)
}

More information

There are plenty of tutorials on internet, but not all of them are updated or following the best practices. I would recommend you to check those links:

And that’s all.