Sep 26, 2021

Using the Amazon S3 SDK v2 for Go

Two years ago, I published the article “Upload files to Amazon S3 with Golang” explaining very briefly how to do some basic operations with Go and Amazon S3. Surprisingly, the article was very successful and two years later it still gets over one thousand visits monthly.

Since Amazon upgraded the SDK for Go to version 2 some time ago, I have decided to create a new article adapting the examples to the new SDK. You can find here all information about the AWS SDK v2 for Go.

As well as on the first article, I am going to show here how to connect to S3 using Go, upload a file from a form to an AWS S3 bucket, download it, and list all items saved on this bucket.

I am going to suppose that you have already:

created an AWS account
created an Amazon Simple Storage (S3) Bucket
generated the credentials to access it (Access key ID and Secret access key)

I am not going to explain those things here since there are plenty of good tutorials on the Internet about it.

Set up your credentials

If you haven’t done it yet, remember to save your credentials using the configure command of the AWS CLI or just creating a file like described below on ~/.aws/credentials on Linux, macOS, or Unix. If using windows, create it on C:\Users\USERNAME.aws\credential.

Contents of the file:

[default] 
aws_access_key_id = your_access_key_id 
aws_secret_access_key = your_secret_access_key

Install the SDK

Create your Go project by executing:

go mod init example

And retrieve the standard set of SDK modules by doing:

go get github.com/aws/aws-sdk-go-v2
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/service/s3

The Code

You can skip all explanations below and check the code directly on https://github.com/antsanchez/goS3example/.

I have tried to keep the code as minimal as possible.

Connect to AWS S3

Connecting Go with Amazon S3 is easy.

package main

import (
	"context"
	"log"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

const (
	AWS_S3_REGION = "" // Region
	AWS_S3_BUCKET = "" // Bucket
)

// We will be using this client everywhere in our code
var awsS3Client *s3.Client

func main() {
    configS3()
}

// configS3 creates the S3 client
func configS3() {

	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(AWS_S3_REGION))
	if err != nil {
		log.Fatal(err)
	}

	awsS3Client = s3.NewFromConfig(cfg)
}

If for some reason you want to programmatically introduce the credentials (if for instance you have to easily switch credentials, or you need to use in your code different buckets from different accounts), you can do it like this:

import "github.com/aws/aws-sdk-go-v2/credentials"

// ...

// configS3 creates the S3 client
func configS3() {

	creds := credentials.NewStaticCredentialsProvider( your_access_key, your_secret_key, "")

	cfg, err := config.LoadDefaultConfig(context.TODO(), config.WithCredentialsProvider(creds), config.WithRegion(AWS_S3_BUCKET))
	if err != nil {
		log.Printf("error: %v", err)
		return
	}

	awsS3Client = s3.NewFromConfig(cfg)
}

Upload a file from a form to Amazon S3

Let’s see how upload a file form a form to Amazon S3 using Go and the http package.

First, get the file from the Form:

file, header, err := r.FormFile("file")
if err != nil {
    // Do your error handling here
    return
}
defer file.Close()

filename := header.Filename

Then, upload it to S3:

import "github.com/aws/aws-sdk-go-v2/service/s3"
import "github.com/aws/aws-sdk-go-v2/feature/s3/manager"

// ...

uploader := manager.NewUploader(awsS3Client)
result, err := uploader.Upload(context.TODO(), &s3.PutObjectInput{
	Bucket: aws.String(AWS_S3_BUCKET),
	Key:    aws.String(filename),
	Body:   file,
})

Download a file from Amazon S3

Downloading a file from AWS S3 using Go requires only a few lines of code:

import "github.com/aws/aws-sdk-go-v2/service/s3"
import "github.com/aws/aws-sdk-go-v2/feature/s3/manager"

// ...

// Name of the file where you want to save the downloaded file
var filename string

// Key to the file to be downloaded
var key string 

// Create the file
newFile, err := os.Create( filename )
if err != nil {
    log.Println(err)
}
defer newFile.Close()

downloader := manager.NewDownloader(awsS3Client)
numBytes, err := downloader.Download(context.TODO(), newFile, &s3.GetObjectInput{
	Bucket: aws.String(AWS_S3_BUCKET), 
	Key:    aws.String(key),
})

List the items of an AWS S3 Bucket

Listing an entire S3 bucket or the contents of a folder is very easy using Go.

About folders, there aren’t really any folders in S3, but we can emulate them by using “/” in the key names of the objects. In case we want to list the contents of a “folder” in S3, what we really need to do is to list all objects which have a certain prefix.

import "github.com/aws/aws-sdk-go-v2/service/s3"

// ...

var prefix string       // For instance, list all ojbects under /bar/foo/ ...
var delimeter string    // ... using "/" as delimeter

paginator := s3.NewListObjectsV2Paginator(awsS3Client, &s3.ListObjectsV2Input{
    Bucket:     aws.String(AWS_S3_BUCKET),
    Prefix:     aws.String(prefix),
    Delimiter:  aws.String(delimeter),
})

for paginator.HasMorePages() {
    page, err := paginator.NextPage(context.TODO())
    if err != nil {
        // Error handling goes here
    }
    for _, obj := range page.Contents {
        // Do whatever you need with each object "obj"
        fmt.Println(obj.Key)
    }
}

And that’s all.