Skip to content

Commit

Permalink
update imports
Browse files Browse the repository at this point in the history
  • Loading branch information
ChrisSchinnerl committed Sep 11, 2023
1 parent 7c0cbc1 commit c68b81a
Show file tree
Hide file tree
Showing 19 changed files with 37 additions and 235 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ jobs:
GO111MODULE=on
docker:
- image: circleci/golang:1.16
working_directory: /go/src/github.com/johannesboyne/gofakes3
working_directory: /go/src/github.com/SiaFoundation/gofakes3
steps:
- checkout
- run: go get -v -t -d ./...
Expand Down
201 changes: 1 addition & 200 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,202 +1,3 @@
[![CircleCI](https://circleci.com/gh/johannesboyne/gofakes3.svg?style=svg)](https://circleci.com/gh/johannesboyne/gofakes3)
[![Codecov](https://codecov.io/gh/johannesboyne/gofakes3/branch/master/graph/badge.svg)](https://codecov.io/gh/johannesboyne/gofakes3)

![Logo](/GoFakeS3.png)
# AWS (GOFAKE)S3

AWS S3 fake server and testing library for extensive S3 test integrations.
Either by running a test-server, e.g. for testing of AWS Lambda functions
accessing S3. Or, to have a simple and convencience S3 mock- and test-server.

## What to use it for?

We're using it for the local development of S3 dependent Lambda functions,
to test AWS S3 golang implementations and access, and
to test browser based direct uploads to S3 locally.


## What not to use it for?

Please don't use gofakes3 as a production service. The intended use case for
gofakes3 is currently to facilitate testing. It's not meant to be used for
safe, persistent access to production data at the moment.

There's no reason we couldn't set that as a stretch goal at a later date, but
it's a long way down the road, especially while we have so much of the API left
to implement; breaking changes are inevitable.

In the meantime, there are more battle-hardened solutions for production
workloads out there, some of which are listed in the "Similar Notable Projects"
section below.


## How to use it?

### Example (aws-sdk-go version 1)

```golang
// fake s3
backend := s3mem.New()
faker := gofakes3.New(backend)
ts := httptest.NewServer(faker.Server())
defer ts.Close()

// configure S3 client
s3Config := &aws.Config{
Credentials: credentials.NewStaticCredentials("YOUR-ACCESSKEYID", "YOUR-SECRETACCESSKEY", ""),
Endpoint: aws.String(ts.URL),
Region: aws.String("eu-central-1"),
DisableSSL: aws.Bool(true),
S3ForcePathStyle: aws.Bool(true),
}
newSession := session.New(s3Config)

s3Client := s3.New(newSession)
cparams := &s3.CreateBucketInput{
Bucket: aws.String("newbucket"),
}

// Create a new bucket using the CreateBucket call.
_, err := s3Client.CreateBucket(cparams)
if err != nil {
// Message from an error.
fmt.Println(err.Error())
return
}

// Upload a new object "testobject" with the string "Hello World!" to our "newbucket".
_, err = s3Client.PutObject(&s3.PutObjectInput{
Body: strings.NewReader(`{"configuration": {"main_color": "#333"}, "screens": []}`),
Bucket: aws.String("newbucket"),
Key: aws.String("test.txt"),
})

// ... accessing of test.txt through any S3 client would now be possible
```

### Example for V2 (aws-sdk-go-v2)

```golang
backend := s3mem.New()
faker := gofakes3.New(backend)
ts := httptest.NewServer(faker.Server())
defer ts.Close()

// Difference in configuring the client

// Setup a new config
cfg, _ := config.LoadDefaultConfig(
context.TODO(),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider("KEY", "SECRET", "SESSION")),
config.WithHTTPClient(&http.Client{
Transport: &http.Transport{
TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
},
}),
config.WithEndpointResolverWithOptions(
aws.EndpointResolverWithOptionsFunc(func(_, _ string, _ ...interface{}) (aws.Endpoint, error) {
return aws.Endpoint{URL: ts.URL}, nil
}),
),
)

// Create an Amazon S3 v2 client, important to use o.UsePathStyle
// alternatively change local DNS settings, e.g., in /etc/hosts
// to support requests to http://<bucketname>.127.0.0.1:32947/...
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.UsePathStyle = true
})

```


Please feel free to check it out and to provide useful feedback (using github
issues), but be aware, this software is used internally and for the local
development only. Thus, it has no demand for correctness, performance or
security.

There are two ways to run locally: using DNS, or using S3 path mode.

S3 path mode is the most flexible and least restrictive, but it does require that you
are able to modify your client code.In Go, the modification would look like so:

config := aws.Config{}
config.WithS3ForcePathStyle(true)

S3 path mode works over the network by default for all bucket names.

If you are unable to modify the code, DNS mode can be used, but it comes with further
restrictions and requires you to be able to modify your local DNS resolution.

If using `localhost` as your endpoint, you will need to add the following to
`/etc/hosts` for *every bucket you want to fake*:

127.0.0.1 <bucket-name>.localhost

It is trickier if you want other machines to be able to use your fake S3 server
as you need to be able to modify their DNS resolution as well.


## Exemplary usage

### Lambda Example

```javascript
var AWS = require('aws-sdk')

var ep = new AWS.Endpoint('http://localhost:9000');
var s3 = new AWS.S3({endpoint: ep});

exports.handle = function (e, ctx) {
s3.createBucket({
Bucket: '<bucket-name>',
}, function(err, data) {
if (err) return console.log(err, err.stack);
ctx.succeed(data)
});
}
```

### Upload Example

```html
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>

<form action="http://localhost:9000/<bucket-name>/" method="post" enctype="multipart/form-data">
Key to upload:
<input type="input" name="key" value="user/user1/test/<filename>" /><br />
<input type="hidden" name="acl" value="public-read" />
<input type="hidden" name="x-amz-meta-uuid" value="14365123651274" />
<input type="hidden" name="x-amz-server-side-encryption" value="AES256" />
<input type="text" name="X-Amz-Credential" value="AKIAIOSFODNN7EXAMPLE/20151229/us-east-1/s3/aws4_request" />
<input type="text" name="X-Amz-Algorithm" value="AWS4-HMAC-SHA256" />
<input type="text" name="X-Amz-Date" value="20151229T000000Z" />

Tags for File:
<input type="input" name="x-amz-meta-tag" value="" /><br />
<input type="hidden" name="Policy" value='<Base64-encoded policy string>' />
<input type="hidden" name="X-Amz-Signature" value="<signature-value>" />
File:
<input type="file" name="file" /> <br />
<!-- The elements after this will be ignored -->
<input type="submit" name="submit" value="Upload to Amazon S3" />
</form>
</html>
```

## Similar notable projects

- https://github.com/minio/minio **not similar but powerfull ;-)**
- https://github.com/andrewgaul/s3proxy by @andrewgaul

## Contributors

A big thank you to all the [contributors](https://github.com/johannesboyne/gofakes3/graphs/contributors),
especially [Blake @shabbyrobe](https://github.com/shabbyrobe) who pushed this
little project to the next level!

**Help wanted**
This is a fork of [GoFakeS3](https://github.com/SiaFoundation/gofakes3) by [johannesboyne](https://github.com/johannesboyne) used to make [renterd](https://github.com/SiaFoundation/renterd) S3 compatible.
2 changes: 1 addition & 1 deletion awscli_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import (
"testing"
"time"

"github.com/johannesboyne/gofakes3"
"github.com/SiaFoundation/gofakes3"
)

func TestCLILsBuckets(t *testing.T) {
Expand Down
2 changes: 1 addition & 1 deletion backend/s3afero/backend_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ import (
"reflect"
"testing"

"github.com/johannesboyne/gofakes3"
"github.com/SiaFoundation/gofakes3"
"github.com/spf13/afero"
)

Expand Down
4 changes: 2 additions & 2 deletions backend/s3afero/multi.go
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,8 @@ import (
"strings"
"sync"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/internal/s3io"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/internal/s3io"
"github.com/spf13/afero"
)

Expand Down
4 changes: 2 additions & 2 deletions backend/s3afero/single.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ import (
"sync"
"time"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/internal/s3io"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/internal/s3io"
"github.com/spf13/afero"
)

Expand Down
2 changes: 1 addition & 1 deletion backend/s3afero/util.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import (
"path/filepath"
"strings"

"github.com/johannesboyne/gofakes3"
"github.com/SiaFoundation/gofakes3"
"github.com/spf13/afero"
)

Expand Down
4 changes: 2 additions & 2 deletions backend/s3bolt/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import (
"io"
"log"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/internal/s3io"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/internal/s3io"
bolt "go.etcd.io/bbolt"
"gopkg.in/mgo.v2/bson"
)
Expand Down
4 changes: 2 additions & 2 deletions backend/s3bolt/schema.go
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@ import (
"bytes"
"time"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/internal/s3io"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/internal/s3io"
bolt "go.etcd.io/bbolt"
"gopkg.in/mgo.v2/bson"
)
Expand Down
4 changes: 2 additions & 2 deletions backend/s3mem/backend.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
"io"
"sync"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/internal/goskipiter"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/internal/goskipiter"
)

var (
Expand Down
4 changes: 2 additions & 2 deletions backend/s3mem/bucket.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import (
"io"
"time"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/internal/s3io"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/internal/s3io"
"github.com/ryszard/goskiplist/skiplist"
)

Expand Down
2 changes: 1 addition & 1 deletion backend/s3mem/versionid.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ import (
"math/big"
"sync"

"github.com/johannesboyne/gofakes3"
"github.com/SiaFoundation/gofakes3"
)

var add1 = new(big.Int).SetInt64(1)
Expand Down
2 changes: 1 addition & 1 deletion backend/s3mem/versionid_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import (
"strings"
"testing"

"github.com/johannesboyne/gofakes3"
"github.com/SiaFoundation/gofakes3"
)

func TestVersionID(t *testing.T) {
Expand Down
8 changes: 4 additions & 4 deletions cmd/gofakes3/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,10 @@ import (
"strings"
"time"

"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3afero"
"github.com/johannesboyne/gofakes3/backend/s3bolt"
"github.com/johannesboyne/gofakes3/backend/s3mem"
"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/backend/s3afero"
"github.com/SiaFoundation/gofakes3/backend/s3bolt"
"github.com/SiaFoundation/gofakes3/backend/s3mem"
"github.com/spf13/afero"
)

Expand Down
4 changes: 2 additions & 2 deletions gofakes3_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ import (
"testing"
"time"

"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/backend/s3mem"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
)

func TestCreateBucket(t *testing.T) {
Expand Down
4 changes: 2 additions & 2 deletions init_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,14 @@ import (
"testing"
"time"

"github.com/SiaFoundation/gofakes3"
"github.com/SiaFoundation/gofakes3/backend/s3mem"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/aws/awserr"
"github.com/aws/aws-sdk-go/aws/credentials"
"github.com/aws/aws-sdk-go/aws/session"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/aws/aws-sdk-go/service/s3/s3manager"
"github.com/johannesboyne/gofakes3"
"github.com/johannesboyne/gofakes3/backend/s3mem"
)

const (
Expand Down
2 changes: 1 addition & 1 deletion option_autobucket_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ import (
"bytes"
"testing"

"github.com/SiaFoundation/gofakes3"
"github.com/aws/aws-sdk-go/aws"
"github.com/aws/aws-sdk-go/service/s3"
"github.com/johannesboyne/gofakes3"
)

const autoBucket = "autobucket"
Expand Down
Loading

0 comments on commit c68b81a

Please sign in to comment.