Getting lots of errors while using tusd with s3 store #1033
Comments
|
It seems you have installed the AWS SDK for Go V1, but tusd uses V2 now (github.com/aws/aws-sdk-go-v2). Please make sure that you provide a V2 instance to s3store. I see that we have an outdated link to V1 on the s3store docs and I will update that one. |
|
@Acconut I am already using the aws-sdk v2, here is my go mod require (
github.com/aws/aws-sdk-go-v2/config v1.25.5
github.com/aws/aws-sdk-go-v2/service/s3 v1.44.0
github.com/casbin/casbin/v2 v2.77.2
github.com/casbin/gorm-adapter/v3 v3.20.0
github.com/go-playground/validator/v10 v10.16.0
github.com/gofiber/fiber/v2 v2.51.0
github.com/golang-jwt/jwt/v5 v5.1.0
github.com/joho/godotenv v1.5.1
github.com/tus/tusd/v2 v2.1.0
golang.org/x/crypto v0.15.0
golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa
gorm.io/driver/postgres v1.5.4
gorm.io/gorm v1.25.5
) |
|
How does your code look? Can you provide me with an example to reproduce this? |
|
@Acconut Sorry for the late reply, it was a health emergency // /drive/s3.go
package drive
import (
"context"
"log"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/service/s3"
)
func s3Client() *s3.Client {
config, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
log.Fatal(err)
}
return s3.NewFromConfig(config)
}// /drive/tus.go
package drive
package drive
import (
"fmt"
"os"
"path/filepath"
"github.com/gofiber/fiber/v2"
tusd "github.com/tus/tusd/v2/pkg/handler"
"github.com/tus/tusd/v2/pkg/s3store"
)
func GetTusHandler(app *fiber.App) *tusd.UnroutedHandler {
uploadPath, err := filepath.Abs("public/uploads")
if err != nil {
panic(fmt.Errorf("unable to get absolute path for uploads directory: %s", err))
}
store := s3store.S3Store{
Bucket: os.Getenv("AWS_S3_BUCKET_NAME"),
TemporaryDirectory: uploadPath,
Service: s3Client(),
MaxObjectSize: 500 * 1024 * 1024, // 500MB
}
composer := tusd.NewStoreComposer()
store.UseIn(composer)
tusdHandler, err := tusd.NewUnroutedHandler(tusd.Config{
BasePath: "/api/drive/",
StoreComposer: composer,
NotifyCompleteUploads: true,
})
if err != nil {
panic(fmt.Errorf("unable to create handler: %s", err))
}
go func() {
for {
event := <-tusdHandler.CompleteUploads
fmt.Printf("Upload %s finished\n", event.Upload.ID)
}
}()
return tusdHandler
}// /drive/controller.go
package drive
import (
"github.com/gofiber/fiber/v2"
"github.com/gofiber/fiber/v2/middleware/adaptor"
)
func RegisterDriveRoutes(app *fiber.App, checkAuth func() fiber.Handler) *fiber.App {
tusHandler := GetTusHandler(app)
app.Post("/api/files/stop", checkAuth(), adaptor.HTTPHandlerFunc(tusHandler.DelFile))
app.Get("/api/files/download", checkAuth(), adaptor.HTTPHandlerFunc(tusHandler.GetFile))
app.Head("/api/files/head", checkAuth(), adaptor.HTTPHandlerFunc(tusHandler.HeadFile))
app.Post("/api/files/patch", checkAuth(), adaptor.HTTPHandlerFunc(tusHandler.PatchFile))
app.Post("/api/files/upload", checkAuth(), adaptor.HTTPHandlerFunc(tusHandler.PostFile))
app.Post("/api/files/upload-v2", checkAuth(), adaptor.HTTPHandlerFunc(tusHandler.PostFileV2))
return app
}// /main.go
// ...other app configs and routes setups
drive.RegisterDriveRoutes(app, helpers.CheckAuth)
// ... listening for requests to port 4000 (locally, it is not deployed even once yet) |
|
There is no problem in the server, the problem occurs when i actually upload the file, (the whole server crashes down) |
I am confused now. In your original statement, you provided errors from build times when the Go code is compiled. But in your latest comment you talk about runtime errors. Can you elaborate what issues you are seeing? |
|
i am seeing issues just when the route is hit. // go.mod
require (
// ... other packages
github.com/aws/aws-sdk-go-v2/config v1.20.0
github.com/aws/aws-sdk-go-v2/service/s3 v1.41.0
github.com/gofiber/fiber/v2 v2.51.0
github.com/tus/tusd/v2 v2.1.0
) |
|
I think you have to use Lines 231 to 245 in 7970961 |
|
That worked I guess, because I am now getting a new error |
|
Two things:
|
|
As it turns out, you might be running into issues that are caused by recent breaking changes in the AWS Go SDK. The minor release v1.14.0 of the S3 package introduced a breaking change (https://github.com/aws/aws-sdk-go-v2/blob/release-2023-11-17/feature/s3/manager/CHANGELOG.md#v1140-2023-11-17) to fix issues related to default values (aws/aws-sdk-go-v2#2162). This breaking change seems to be the cause for the compile issues you faced originally. The runtime error with I am not sure if and how we can fix them in tusd. Maybe upgrading the AWS SDK version helps, but I am a bit hesitant since this would also be a breaking change for people using tusd as a package. |
|
tusd is now upgraded to the most recent AWS SDK version, so these problems should be fixed in v2.2.0. See #1039 and https://github.com/tus/tusd/releases/tag/v2.2.0 for more details. |
github.com/tus/tusd/v2/pkg/s3store
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:394:18: cannot use int64(len(infoJson)) (value of type int64) as *int64 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:510:18: cannot use part.number (variable of type int32) as *int32 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:746:13: cannot use 0 (untyped int constant) as *int32 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:803:12: cannot use true (untyped bool constant) as *bool value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:845:16: cannot use 1 (untyped int constant) as *int32 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:869:16: cannot use part.number (variable of type int32) as *int32 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:995:17: cannot use partNumber (variable of type int32) as *int32 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:1047:13: cannot use part.PartNumber (variable of type *int32) as int32 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:1048:13: cannot use part.Size (variable of type *int64) as int64 value in struct literal
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:1053:6: non-boolean condition in if statement
../../../go/pkg/mod/github.com/tus/tusd/[email protected]/pkg/s3store/s3store.go:1053:6: too many errors
Describe the bug
I just installed the package (go get -u ...) and follow the docs for s3 store on pkg.go.dev
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Setup details
Please provide following details, if applicable to your situation:
The text was updated successfully, but these errors were encountered: