chore: adopt testcontainers-go for Postgres, MySQL, MSSQL and MongoDB #1515
Conversation
mdelapenya
requested review from
gaby,
sixcolors,
ReneWerner87 and
efectn
and removed request for
a team
|
Important Review skippedReview was skipped due to path filters ⛔ Files ignored due to path filters (14)
CodeRabbit blocks several paths by default. You can override this behavior by explicitly including those paths in the path filters. For example, including You can disable this status message by setting the WalkthroughThe changes involve significant modifications to the GitHub Actions workflow files and associated test files for various database services, including MongoDB, MSSQL, MySQL, PostgreSQL, ClickHouse, Couchbase, and MinIO. The updates include the removal of service configurations and the introduction of environment variables for updated Docker images. Additionally, resource management improvements have been made across multiple test files to enhance test isolation and reliability. Changes
Possibly related PRs
Suggested labels
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (8)
mongodb/go.modis excluded by!**/*.modmongodb/go.sumis excluded by!**/*.sum,!**/*.summssql/go.modis excluded by!**/*.modmssql/go.sumis excluded by!**/*.sum,!**/*.summysql/go.modis excluded by!**/*.modmysql/go.sumis excluded by!**/*.sum,!**/*.sumpostgres/go.modis excluded by!**/*.modpostgres/go.sumis excluded by!**/*.sum,!**/*.sum
Files selected for processing (9)
- .github/workflows/benchmark.yml (2 hunks)
- .github/workflows/test-mongodb.yml (2 hunks)
- .github/workflows/test-mssql.yml (2 hunks)
- .github/workflows/test-mysql.yml (2 hunks)
- .github/workflows/test-postgres.yml (2 hunks)
- mongodb/mongodb_test.go (11 hunks)
- mssql/mssql_test.go (12 hunks)
- mysql/mysql_test.go (13 hunks)
- postgres/postgres_test.go (12 hunks)
Additional comments not posted (32)
.github/workflows/test-mysql.yml (1)
30-31: LGTM!The introduction of the
TEST_MYSQL_IMAGEenvironment variable is a good practice to ensure consistency and reliability in the testing environment..github/workflows/test-mongodb.yml (2)
19-20: LGTM!Updating the Go version matrix to include newer versions is a good practice to ensure compatibility and performance with the latest Go versions.
30-30: LGTM!The introduction of the
TEST_MONGODB_IMAGEenvironment variable is a good practice to ensure consistency and reliability in the testing environment..github/workflows/test-postgres.yml (1)
30-31: LGTM!The introduction of the
TEST_POSTGRES_IMAGEenvironment variable is a good practice to ensure consistency and reliability in the testing environment..github/workflows/test-mssql.yml (1)
30-31: LGTM!The addition of the environment variable
TEST_MSSQL_IMAGEis correct and aligns with the PR objective..github/workflows/benchmark.yml (4)
118-118: LGTM!The addition of the environment variable
TEST_MONGODB_IMAGEis correct and aligns with the PR objective.
119-119: LGTM!The addition of the environment variable
TEST_MSSQL_IMAGEis correct and aligns with the PR objective.
120-120: LGTM!The addition of the environment variable
TEST_MYSQL_IMAGEis correct and aligns with the PR objective.
121-121: LGTM!The addition of the environment variable
TEST_POSTGRES_IMAGEis correct and aligns with the PR objective.mongodb/mongodb_test.go (10)
10-10: LGTM!The import statement for
testcontainers-go/modules/mongodbis correct and necessary for using the testcontainers-go library for MongoDB.
13-19: LGTM!The constant block for MongoDB image and credentials is correctly defined and aligns with the PR objective.
21-44: LGTM!The
newTestStorefunction is correctly implemented and improves the isolation of tests by ensuring that each test starts with a fresh instance of the MongoDB store.
53-55: LGTM!The modification of the
Test_MongoDB_Setfunction to usenewTestStoreis correct and aligns with the PR objective.
66-68: LGTM!The modification of the
Test_MongoDB_Set_Overridefunction to usenewTestStoreis correct and aligns with the PR objective.
82-84: LGTM!The modification of the
Test_MongoDB_Getfunction to usenewTestStoreis correct and aligns with the PR objective.
100-102: LGTM!The modification of the
Test_MongoDB_Set_Expirationfunction to usenewTestStoreis correct and aligns with the PR objective.
112-114: LGTM!The modification of the
Test_MongoDB_Get_Expiredfunction to usenewTestStoreis correct and aligns with the PR objective.
121-123: LGTM!The modification of the
Test_MongoDB_Get_NotExistfunction to usenewTestStoreis correct and aligns with the PR objective.
135-137: LGTM!The modification of the
Test_MongoDB_Deletefunction to usenewTestStoreis correct and aligns with the PR objective.mssql/mssql_test.go (4)
4-11: LGTM!The import statements are correct and necessary for the functionality.
14-21: LGTM!The constants improve configurability and readability.
23-51: LGTM!The function is correctly implemented and ensures each test runs in isolation with a fresh database connection.
Line range hint
59-264: LGTM!The updates ensure that each test retrieves its own instance of the storage, improving test isolation and reliability.
mysql/mysql_test.go (5)
4-11: LGTM!The import statements are correct and necessary for the functionality.
14-21: LGTM!The constants improve configurability and readability.
23-39: LGTM!The function is correctly implemented and ensures each test runs in isolation with a fresh database connection.
41-57: LGTM!The function is correctly implemented and ensures the MySQL container is properly configured and started.
Line range hint
59-294: LGTM!The updates ensure that each test retrieves its own instance of the storage, improving test isolation and reliability.
postgres/postgres_test.go (4)
Line range hint
4-13: LGTM!The import statements are correct and necessary for the functionality.
16-23: LGTM!The constants improve configurability and readability.
25-56: LGTM!The function is correctly implemented and ensures each test runs in isolation with a fresh database connection.
Line range hint
64-282: LGTM!The updates ensure that each test retrieves its own instance of the storage, improving test isolation and reliability.
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- .github/workflows/test-mongodb.yml (2 hunks)
Files skipped from review as they are similar to previous changes (1)
- .github/workflows/test-mongodb.yml
There was a problem hiding this comment.
Actionable comments posted: 0
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (2)
mongodb/go.modis excluded by!**/*.modpostgres/go.modis excluded by!**/*.mod
Files selected for processing (3)
- .github/workflows/test-mssql.yml (2 hunks)
- .github/workflows/test-mysql.yml (2 hunks)
- .github/workflows/test-postgres.yml (2 hunks)
Files skipped from review as they are similar to previous changes (2)
- .github/workflows/test-mssql.yml
- .github/workflows/test-postgres.yml
Additional comments not posted (4)
.github/workflows/test-mysql.yml (4)
19-19: LGTM!The addition of Go version 1.22.x to the matrix is appropriate.
20-20: LGTM!The addition of Go version 1.23.x to the matrix is appropriate.
30-30: LGTM!Setting the
TEST_MYSQL_IMAGEenvironment variable todocker.io/mysql:9ensures consistency and reliability in the testing environment.
31-31: LGTM!Running the tests with the specified MySQL image is appropriate.
|
For some reason, the memcache service is failing in the benchmark job. Any guess? |
|
sorry overlooked, that's really strange, I'll have to take a look at it in the next few days |
|
@ReneWerner87 any update on the benchmark thing? 🙏 Once merge, I can send a new PR with more stores |
|
unfortunately haven't had time to look yet, will have a look on sunday and fix it then, but feel free to make pull requests for the other stores if you like in this case we only have to apply the fix to the other PR |
|
@mdelapenya now the reason you removed the other services 
now the options from postgres for the health check belongs to memcached 
but there is no postgres started 
other question |
The tests are failing because the container is terminated with error code 127
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (8)
mongodb/go.modis excluded by!**/*.modmongodb/go.sumis excluded by!**/*.sum,!**/*.summssql/go.modis excluded by!**/*.modmssql/go.sumis excluded by!**/*.sum,!**/*.summysql/go.modis excluded by!**/*.modmysql/go.sumis excluded by!**/*.sum,!**/*.sumpostgres/go.modis excluded by!**/*.modpostgres/go.sumis excluded by!**/*.sum,!**/*.sum
Files selected for processing (3)
- .github/workflows/benchmark.yml (1 hunks)
- .github/workflows/test-mssql.yml (2 hunks)
- mssql/mssql_test.go (12 hunks)
Additional comments not posted (4)
.github/workflows/test-mssql.yml (2)
20-20: Go version update approved.Adding Go version
1.22.xto the test matrix ensures compatibility with the latest Go release.
30-31: Environment variableTEST_MSSQL_IMAGEset appropriately.Defining
TEST_MSSQL_IMAGEwith the specified image ensures consistency across testing environments..github/workflows/benchmark.yml (1)
115-118: Database image environment variables configured correctly.The addition of
TEST_MONGODB_IMAGE,TEST_MSSQL_IMAGE,TEST_MYSQL_IMAGE, andTEST_POSTGRES_IMAGEenvironment variables specifies the database images for testing, enhancing consistency and reproducibility.mssql/mssql_test.go (1)
23-51: Well-implementednewTestStorefunction enhances test isolation.The
newTestStorefunction provides each test with an independent storage instance, improving test reliability and avoiding shared state.
mssql/mssql_test.go
Outdated
| testStore, err := newTestStore(t) | ||
| require.NoError(t, err) | ||
|
|
There was a problem hiding this comment.
Ensure proper cleanup by closing the test store in tests and benchmarks.
In the test and benchmark functions, after initializing testStore, consider adding defer testStore.Close() to ensure that resources are properly released and to prevent potential resource leaks.
Apply the following diff to add the defer statements:
For tests:
func Test_FunctionName(t *testing.T) {
testStore, err := newTestStore(t)
require.NoError(t, err)
+ defer testStore.Close()
// rest of the test code
}For benchmarks:
func Benchmark_FunctionName(b *testing.B) {
testStore, err := newTestStore(b)
require.NoError(b, err)
+ defer testStore.Close()
// rest of the benchmark code
}Also applies to: 72-74, 88-90, 106-108, 118-120, 127-129, 141-143, 158-160, 182-184, 207-209, 234-236, 241-243, 248-250, 262-264, 279-281
There was a problem hiding this comment.
Actionable comments posted: 1
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (7)
- clickhouse/clickhouse_test.go (1 hunks)
- couchbase/couchbase_test.go (10 hunks)
- minio/minio_test.go (15 hunks)
- mongodb/mongodb_test.go (11 hunks)
- mssql/mssql_test.go (12 hunks)
- mysql/mysql_test.go (13 hunks)
- postgres/postgres_test.go (12 hunks)
Additional comments not posted (75)
couchbase/couchbase_test.go (11)
59-59: Good practice: Ensuring resource cleanup withdefer testStore.Close()Adding
defer testStore.Close()ensures that the test store is properly closed after the testTestSetCouchbase_ShouldReturnNoErrorexecutes, preventing potential resource leaks.
69-69: Good practice: Ensuring resource cleanup inTestGetCouchbase_ShouldReturnNil_WhenDocumentNotFoundIncluding
defer testStore.Close()guarantees that resources are released after the test, enhancing test reliability.
80-80: Good practice: Resource cleanup inTestSetAndGet_GetShouldReturn_SetValueWithoutErrorThe addition of
defer testStore.Close()ensures that the test store is closed properly, preventing resource leaks.
94-94: Good practice: Proper resource management inTestSetAndGet_GetShouldReturnNil_WhenTTLExpiredUsing
defer testStore.Close()aids in cleaning up resources after the test execution.
110-110: Good practice: Ensuring resource cleanup inTestSetAndDelete_DeleteShouldReturn_NoErrorAdding
defer testStore.Close()ensures that the test store is properly closed, enhancing resource management.
125-125: Good practice: Resource cleanup inTestSetAndReset_ResetShouldReturn_NoErrorIncluding
defer testStore.Close()helps prevent resource leaks by ensuring the test store is closed after the test.
140-140: Good practice: Proper closure inTestClose_CloseShouldReturn_NoErrorAdding
defer testStore.Close()ensures that resources are released appropriately after the test execution.
149-149: Good practice: Ensuring resource cleanup inTestGetConn_ReturnsNotNilUsing
defer testStore.Close()enhances resource management by closing the test store after the test.
157-157: Good practice: Resource cleanup in benchmarkBenchmark_Couchbase_SetIncluding
defer testStore.Close()ensures that resources are properly released after the benchmark execution.
172-172: Good practice: Proper resource management in benchmarkBenchmark_Couchbase_GetAdding
defer testStore.Close()helps prevent resource leaks during benchmark execution.
190-190: Good practice: Resource cleanup in benchmarkBenchmark_Couchbase_SetAndDeleteUsing
defer testStore.Close()ensures that the test store is closed after the benchmark completes.mongodb/mongodb_test.go (14)
21-44: Well-designednewTestStorefunction for test isolationThe
newTestStorefunction effectively initializes a MongoDB test container using Testcontainers, enhancing test isolation and reliability by providing a fresh instance for each test.
53-56: Proper initialization and cleanup inTest_MongoDB_SetInitializing the test store with
newTestStore(t)and addingdefer testStore.Close()ensures each test is isolated and that resources are properly released.
67-70: Consistent test setup inTest_MongoDB_Set_OverrideUsing the new test store initialization pattern enhances test reliability and resource management.
84-87: Improved resource management inTest_MongoDB_GetIncluding
defer testStore.Close()ensures that resources are cleaned up after the test execution.
103-106: Proper test isolation inTest_MongoDB_Set_ExpirationInitializing a fresh test store and deferring its closure helps prevent interference between tests.
116-119: Resource cleanup inTest_MongoDB_Get_ExpiredAdding
defer testStore.Close()ensures that the test store is properly closed after the test.
126-129: Consistent resource management inTest_MongoDB_Get_NotExistIncluding
defer testStore.Close()enhances resource cleanup post-test execution.
141-144: Proper closure of test store inTest_MongoDB_DeleteUsing
defer testStore.Close()ensures resources are released after the test runs.
159-162: Resource cleanup inTest_MongoDB_ResetAdding
defer testStore.Close()aids in preventing resource leaks.
182-185: Proper resource management inTest_MongoDB_CloseDeferring
testStore.Close()ensures that resources are released appropriately.
190-193: Consistent cleanup inTest_MongoDB_ConnIncluding
defer testStore.Close()enhances resource management after the test execution.
198-201: Resource cleanup in benchmarkBenchmark_MongoDB_SetAdding
defer testStore.Close()ensures that resources are properly released after the benchmark.
213-216: Proper resource management in benchmarkBenchmark_MongoDB_GetUsing
defer testStore.Close()helps prevent resource leaks during benchmark execution.
231-234: Resource cleanup in benchmarkBenchmark_MongoDB_SetAndDeleteIncluding
defer testStore.Close()ensures that the test store is closed after the benchmark completes.clickhouse/clickhouse_test.go (1)
70-75: Proper assignment and resource cleanup inTest_ConnectionAssigning the result of
getTestConnectiontoclientand deferringclient.Close()ensures that the connection is utilized and properly closed after the test, preventing resource leaks.minio/minio_test.go (15)
67-67: Good practice: Addingdefer testStore.Close()inTest_GetIncluding
defer testStore.Close()ensures that resources are properly released after the test execution, preventing potential leaks.
88-88: Resource cleanup inTest_Get_Empty_KeyAdding
defer testStore.Close()enhances resource management by closing the test store after the test.
102-102: Proper resource management inTest_Get_Not_Exists_KeyUsing
defer testStore.Close()ensures that resources are released appropriately after the test.
116-116: Consistent cleanup inTest_Get_Not_Exists_BucketAdding
defer testStore.Close()helps prevent resource leaks by closing the test store.
135-135: Good practice: Ensuring resource cleanup inTest_SetIncluding
defer testStore.Close()ensures proper resource management.
149-149: Resource management inTest_Set_Empty_KeyAdding
defer testStore.Close()ensures that resources are released after the test execution.
166-166: Proper closure inTest_Set_Not_Exists_BucketUsing
defer testStore.Close()enhances resource cleanup post-test.
184-184: Resource cleanup inTest_DeleteIncluding
defer testStore.Close()ensures that the test store is properly closed after the test.
201-201: Consistent resource management inTest_Delete_Empty_KeyAdding
defer testStore.Close()helps prevent resource leaks.
215-215: Proper resource cleanup inTest_Delete_Not_Exists_BucketUsing
defer testStore.Close()ensures resources are released appropriately.
233-233: Resource management inTest_ResetIncluding
defer testStore.Close()enhances test reliability by closing resources after execution.
252-252: Good practice: Ensuring resource cleanup inTest_CloseAdding
defer testStore.Close()ensures that resources are properly released.
263-263: Resource cleanup in benchmarkBenchmark_Minio_SetIncluding
defer testStore.Close()ensures proper resource management during benchmarks.
275-275: Proper resource management in benchmarkBenchmark_Minio_GetAdding
defer testStore.Close()helps prevent resource leaks during benchmark execution.
293-293: Resource cleanup in benchmarkBenchmark_Minio_SetAndDeleteUsing
defer testStore.Close()ensures that the test store is closed after the benchmark completes.mssql/mssql_test.go (16)
23-51: Well-implementednewTestStorefunction for MSSQL test setupThe
newTestStorefunction correctly initializes a MSSQL test container using Testcontainers, enhancing test isolation and ensuring each test operates with a fresh database instance.
59-62: Proper initialization and cleanup inTest_MSSQL_SetUsing
newTestStore(t)and deferringtestStore.Close()ensures tests are isolated and resources are properly managed.
73-76: Consistent test setup inTest_MSSQL_Set_OverrideAdding
defer testStore.Close()enhances resource management, consistent with previous test functions.
90-93: Resource cleanup inTest_MSSQL_GetIncluding
defer testStore.Close()ensures proper resource release after test execution.
109-112: Proper resource management inTest_MSSQL_Set_ExpirationDeferring
testStore.Close()aids in resource cleanup.
122-125: Resource cleanup inTest_MSSQL_Get_ExpiredAdding
defer testStore.Close()ensures proper closure of the test store.
132-135: Consistent resource management inTest_MSSQL_Get_NotExistUsing
defer testStore.Close()enhances test reliability.
147-150: Resource cleanup inTest_MSSQL_DeleteIncluding
defer testStore.Close()ensures resources are released appropriately.
165-168: Proper resource management inTest_MSSQL_ResetDeferring
testStore.Close()aids in preventing resource leaks.
190-193: Resource cleanup inTest_MSSQL_GCAdding
defer testStore.Close()ensures resources are properly released after the test.
216-219: Proper closure inTest_MSSQL_Non_UTF8Using
defer testStore.Close()enhances resource management.
244-247: Resource cleanup inTest_MSSQL_CloseIncluding
defer testStore.Close()ensures proper resource release.
252-255: Consistent resource management inTest_MSSQL_ConnAdding
defer testStore.Close()ensures resources are properly closed.
260-263: Resource cleanup in benchmarkBenchmark_MSSQL_SetDeferring
testStore.Close()prevents resource leaks during benchmark execution.
275-278: Proper resource management in benchmarkBenchmark_MSSQL_GetUsing
defer testStore.Close()ensures resources are released after the benchmark.
293-296: Resource cleanup in benchmarkBenchmark_MSSQL_SetAndDeleteIncluding
defer testStore.Close()helps prevent resource leaks in benchmarks.mysql/mysql_test.go (18)
23-39: Well-structurednewTestStorefunction for MySQL test setupThe
newTestStorefunction effectively initializes a MySQL test container using Testcontainers, improving test isolation by providing a fresh database instance for each test.
41-57: Proper container management inmustStartMySQLfunctionThe
mustStartMySQLfunction correctly starts a MySQL container with specified configurations, enhancing the reliability of the testing environment.
59-70: Improved test initialization inTest_MYSQL_NewInitializing the test store using the new pattern and handling errors appropriately enhances test reliability.
88-91: Proper resource management inTest_MYSQL_SetUsing
newTestStore(t)and deferringtestStore.Close()ensures resources are managed correctly.
102-105: Consistent test setup inTest_MYSQL_Set_OverrideIncluding
defer testStore.Close()aids in resource cleanup after the test execution.
119-122: Resource cleanup inTest_MYSQL_GetAdding
defer testStore.Close()ensures that the test store is properly closed.
138-141: Proper resource management inTest_MYSQL_Set_ExpirationDeferring
testStore.Close()helps in resource cleanup post-test execution.
151-154: Resource cleanup inTest_MYSQL_Get_ExpiredIncluding
defer testStore.Close()ensures resources are properly released.
161-164: Consistent resource management inTest_MYSQL_Get_NotExistAdding
defer testStore.Close()enhances resource cleanup.
176-179: Proper resource management inTest_MYSQL_DeleteUsing
defer testStore.Close()ensures the test store is closed after the test.
194-197: Resource cleanup inTest_MYSQL_ResetDeferring
testStore.Close()aids in preventing resource leaks.
220-223: Resource management inTest_MYSQL_GCIncluding
defer testStore.Close()ensures resources are released appropriately after the test.
245-248: Proper closure inTest_MYSQL_Non_UTF8Using
defer testStore.Close()ensures proper resource cleanup.
258-261: Resource cleanup inTest_MYSQL_CloseAdding
defer testStore.Close()enhances resource management post-test execution.
266-269: Consistent resource management inTest_MYSQL_ConnDeferring
testStore.Close()ensures that resources are properly released.
274-277: Resource cleanup in benchmarkBenchmark_MYSQL_SetUsing
defer testStore.Close()helps prevent resource leaks during benchmark execution.
289-292: Proper resource management in benchmarkBenchmark_MYSQL_GetIncluding
defer testStore.Close()ensures resources are released after the benchmark.
307-310: Resource cleanup in benchmarkBenchmark_MYSQL_SetAndDeleteAdding
defer testStore.Close()ensures the test store is closed after the benchmark completes.
postgres/postgres_test.go
Outdated
| func newTestStore(t testing.TB) (*Storage, error) { | ||
| t.Helper() | ||
|
|
||
| ctx := context.Background() | ||
|
|
||
| img := postgresImage | ||
| if imgFromEnv := os.Getenv(postgresImageEnvVar); imgFromEnv != "" { | ||
| img = imgFromEnv | ||
| } | ||
|
|
||
| c, err := postgres.Run(ctx, img, | ||
| postgres.WithUsername(postgresUser), | ||
| postgres.WithPassword(postgresPass), | ||
| postgres.WithDatabase(postgresDatabase), | ||
| testcontainers.WithWaitStrategy( | ||
| // First, we wait for the container to log readiness twice. | ||
| // This is because it will restart itself after the first startup. | ||
| wait.ForLog("database system is ready to accept connections").WithOccurrence(2), | ||
| ), | ||
| ) | ||
| require.NoError(t, err) | ||
|
|
||
| conn, err := c.ConnectionString(ctx, "sslmode=disable") | ||
| if err != nil { | ||
| return nil, err | ||
| } | ||
|
|
||
| return New(Config{ | ||
| ConnectionURI: conn, | ||
| Reset: true, | ||
| }), nil | ||
| } |
There was a problem hiding this comment.
Ensure proper cleanup of test containers to prevent resource leaks
The newTestStore function creates a new PostgreSQL container for each test but does not terminate the container after the test completes. This could lead to resource leaks and accumulation of unused Docker containers during testing.
Apply this diff to ensure the container is properly terminated:
func newTestStore(t testing.TB) (*Storage, error) {
t.Helper()
ctx := context.Background()
img := postgresImage
if imgFromEnv := os.Getenv(postgresImageEnvVar); imgFromEnv != "" {
img = imgFromEnv
}
c, err := postgres.Run(ctx, img,
postgres.WithUsername(postgresUser),
postgres.WithPassword(postgresPass),
postgres.WithDatabase(postgresDatabase),
testcontainers.WithWaitStrategy(
// First, we wait for the container to log readiness twice.
// This is because it will restart itself after the first startup.
wait.ForLog("database system is ready to accept connections").WithOccurrence(2),
),
)
require.NoError(t, err)
conn, err := c.ConnectionString(ctx, "sslmode=disable")
if err != nil {
return nil, err
}
- return New(Config{
+ store := New(Config{
ConnectionURI: conn,
Reset: true,
- }), nil
+ })
+ cleanup := func() {
+ err := c.Terminate(ctx)
+ require.NoError(t, err)
+ }
- return store, nil
+ return store, cleanup, nil
}Then, update your test functions to handle the cleanup function:
-func Test_Postgres_Set(t *testing.T) {
+func Test_Postgres_Set(t *testing.T) {
var (
key = "john"
val = []byte("doe")
)
- testStore, err := newTestStore(t)
+ testStore, cleanup, err := newTestStore(t)
require.NoError(t, err)
defer testStore.Close()
+ defer cleanup()
err = testStore.Set(key, val, 0)
require.NoError(t, err)
}Repeat this change for all other test functions using newTestStore.
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func newTestStore(t testing.TB) (*Storage, error) { | |
| t.Helper() | |
| ctx := context.Background() | |
| img := postgresImage | |
| if imgFromEnv := os.Getenv(postgresImageEnvVar); imgFromEnv != "" { | |
| img = imgFromEnv | |
| } | |
| c, err := postgres.Run(ctx, img, | |
| postgres.WithUsername(postgresUser), | |
| postgres.WithPassword(postgresPass), | |
| postgres.WithDatabase(postgresDatabase), | |
| testcontainers.WithWaitStrategy( | |
| // First, we wait for the container to log readiness twice. | |
| // This is because it will restart itself after the first startup. | |
| wait.ForLog("database system is ready to accept connections").WithOccurrence(2), | |
| ), | |
| ) | |
| require.NoError(t, err) | |
| conn, err := c.ConnectionString(ctx, "sslmode=disable") | |
| if err != nil { | |
| return nil, err | |
| } | |
| return New(Config{ | |
| ConnectionURI: conn, | |
| Reset: true, | |
| }), nil | |
| } | |
| func newTestStore(t testing.TB) (*Storage, func(), error) { | |
| t.Helper() | |
| ctx := context.Background() | |
| img := postgresImage | |
| if imgFromEnv := os.Getenv(postgresImageEnvVar); imgFromEnv != "" { | |
| img = imgFromEnv | |
| } | |
| c, err := postgres.Run(ctx, img, | |
| postgres.WithUsername(postgresUser), | |
| postgres.WithPassword(postgresPass), | |
| postgres.WithDatabase(postgresDatabase), | |
| testcontainers.WithWaitStrategy( | |
| // First, we wait for the container to log readiness twice. | |
| // This is because it will restart itself after the first startup. | |
| wait.ForLog("database system is ready to accept connections").WithOccurrence(2), | |
| ), | |
| ) | |
| require.NoError(t, err) | |
| conn, err := c.ConnectionString(ctx, "sslmode=disable") | |
| if err != nil { | |
| return nil, nil, err | |
| } | |
| store := New(Config{ | |
| ConnectionURI: conn, | |
| Reset: true, | |
| }) | |
| cleanup := func() { | |
| err := c.Terminate(ctx) | |
| require.NoError(t, err) | |
| } | |
| return store, cleanup, nil | |
| } |
This is not relevant for this PR, as Ryuk is responsible for terminating containers after the test session finishes
There was a problem hiding this comment.
Actionable comments posted: 2
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (7)
- clickhouse/clickhouse_test.go (2 hunks)
- couchbase/couchbase_test.go (11 hunks)
- minio/minio_test.go (16 hunks)
- mongodb/mongodb_test.go (11 hunks)
- mssql/mssql_test.go (12 hunks)
- mysql/mysql_test.go (13 hunks)
- postgres/postgres_test.go (12 hunks)
Files skipped from review due to trivial changes (1)
- minio/minio_test.go
Files skipped from review as they are similar to previous changes (3)
- clickhouse/clickhouse_test.go
- couchbase/couchbase_test.go
- mssql/mssql_test.go
Additional comments not posted (2)
mongodb/mongodb_test.go (1)
21-51: Great job on enhancing test isolation withnewTestStore!The
newTestStorefunction effectively encapsulates the setup and teardown of the MongoDB test store using Testcontainers. The use oft.Cleanupensures proper resource cleanup, preventing resource leaks. This change improves test reliability and maintains clean test environments.postgres/postgres_test.go (1)
35-50: Excellent resource cleanup witht.CleanupThe implementation of
t.Cleanupto terminate the PostgreSQL container ensures that resources are properly cleaned up after tests, preventing potential resource leaks.
mysql/mysql_test.go
Outdated
| mysqlUser string = "password" | ||
| mysqlPass string = "username" |
There was a problem hiding this comment.
Swap mysqlUser and mysqlPass values to correct credentials
The constants for mysqlUser and mysqlPass appear to be swapped. Currently, mysqlUser is set to "password" and mysqlPass is set to "username", which is likely unintended.
Apply this diff to correct the username and password assignments:
const (
// mysqlImage is the default image used for running MySQL in tests.
mysqlImage = "docker.io/mysql:9"
mysqlImageEnvVar string = "TEST_MYSQL_IMAGE"
- mysqlUser string = "password"
- mysqlPass string = "username"
+ mysqlUser string = "username"
+ mysqlPass string = "password"
mysqlDatabase string = "fiber"
)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| mysqlUser string = "password" | |
| mysqlPass string = "username" | |
| mysqlUser string = "username" | |
| mysqlPass string = "password" |
| c, err := mysql.Run(ctx, img, | ||
| mysql.WithPassword(mysqlPass), | ||
| mysql.WithUsername(mysqlUser), | ||
| mysql.WithDatabase(mysqlDatabase), | ||
| ) |
There was a problem hiding this comment.
Tip
Codebase Verification
Swap mysqlUser and mysqlPass constant definitions
The mysql.Run function call in mustStartMySQL is correct, using mysqlUser with WithUsername and mysqlPass with WithPassword. However, the constant definitions are incorrect:
- mysqlUser is defined as "password"
- mysqlPass is defined as "username"
These definitions should be swapped to correctly represent their intended values.
Analysis chain
Verify MySQL container credentials in mustStartMySQL
Ensure that the credentials passed to mysql.Run match the corrected mysqlUser and mysqlPass constants after swapping them.
Run the following script to confirm that the username and password are correctly used:
Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash
# Description: Verify that the username and password in mustStartMySQL match the constants.
# Expected result: The literals "mysqlUser" and "mysqlPass" should be used with the correct methods.
ast-grep --lang go --pattern $'mysql.Run($_, $_,\n mysql.WithUsername(mysqlUser),\n mysql.WithPassword(mysqlPass),\n $$$\n)'
Length of output: 130
Script:
#!/bin/bash
# Display the content of mysql/mysql_test.go
cat mysql/mysql_test.go
# Search for mysqlUser and mysqlPass constants
rg "mysqlUser|mysqlPass" mysql/mysql_test.go
# Find mysql.Run function call with a more flexible pattern
ast-grep --lang go --pattern 'mysql.Run($_, $_, $$$)'
Length of output: 6894
|
@ReneWerner87 I noticed a regression in the recent update (Azure DevOps agent / GitHub Actions runner): actions/runner-images#10649 I think it could be related 🤔 |
This reverts commit 3cd89db.
|
@ReneWerner87 I resolved the MSSQL errors caused by the recent GH workers upgrade, and I'm currently debugging the couchbase failures (https://github.com/gofiber/storage/actions/runs/10950265038/job/30405176292?pr=1515) where it only fails for Go 1.20 🤷 Do you think it could be caused by the number of running containers so the worker could eventually be non-responsive? I'd say that the same happens for the benchmarks. Will update this thread after my research. Sorry for the inconvenience 🙏 |
could be possible thanks for the effort so far |
|
@mdelapenya Did you add "wait_for" to each service? A lot of these services take a bit to start, so it could be that. |
Indeed, we always recommend adding wait strategies for container, so our modules usually come with a wait strategy. For couchbase we have this: https://github.com/testcontainers/testcontainers-go/blob/main/modules/couchbase/couchbase.go#L189 |
|
@mdelapenya in mssql I saw that you changed the version to 1.22 in the go.mod, but the test workflow still tests down to 1.20 |
|
same for couchbase |
|
@ReneWerner87 thanks for going back to this. I was focused on other topics and left this behind. I can jump on this now. Before I add more (probably wrong) code, what is the Go version policy for the modules? We collected the values in #1508 (comment). This will be important because in older Go versions, we'll find some issues with missing types required by packages using a more modern version. Could you help me out defining the right version? 🙏 |
|
We should leave the versions as they are, otherwise we would have to increase the major version of the respective package and update other things if possible, so that this breaking change was worth it |
Ok, makes sense. There would be modules we won't be able to adopt the integration tests as in others. Will take a look, and would close this PR in case it's not possible to do it. Thanks for your work here! |
|
It would be possible if we upgrade the major version and no longer support the old golang version |
What does this PR do?
It uses testcontainers-go in the integration tests for the given stores, refactoring the tests to always consume a new test store, backed by a Docker container.
WHy is it important?
Reproducibility: any developer will be able to run the tests locally exactly equals as in the Github Actions.
Related issues
For reference, please check out previous work for Couchbase and Minio (#1508) and ClickHouse (#1506)
There is more ongoing work, as defined in #1507
Summary by CodeRabbit
New Features
Bug Fixes
Refactor