Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Start export s3 and fixes for demo #28

Merged
merged 5 commits into from
Jul 30, 2024
Merged

Start export s3 and fixes for demo #28

merged 5 commits into from
Jul 30, 2024

Conversation

bma13
Copy link
Collaborator

@bma13 bma13 commented Jul 26, 2024

No description provided.

@github-actions GitHub Actions
Copy link

📝 Test results

ydbcp

15 tests (15 passed) 
%%{init: {"theme":"base","themeVariables":{"fontFamily":"monospace","pieSectionTextSize":"24px","darkMode":true,"pie1":"#2da44e"}}}%%
pie showData
"Passed" : 15
Loading
📦 Package🟢 Passed🔴 Failed🟡 Skipped⏳ Duration
🟡 ydbcp/cmd/ydbcp0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/config0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/connectors/client0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/connectors/db0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟢 ydbcp/internal/connectors/db/yql/queries20016ms
🧪 Tests
  • 🟢TestQueryBuilder_Read
  • 🟢TestQueryBuilder_Write
🖨️ Output
=== RUN   TestQueryBuilder_Read
2024-07-26T09:00:45.064Z	DEBUG	queries/read.go:146	read query	{"yql": "DECLARE $param0 AS String;\nDECLARE $param1 AS String;\nDECLARE $param2 AS String;\nDECLARE $param3 AS String;\nSELECT column1, column2, column3 FROM table1 WHERE (column1 = $param0 OR column1 = $param1) AND (column2 = $param2 OR column2 = $param3)"}
--- PASS: TestQueryBuilder_Read (0.00s)
=== RUN   TestQueryBuilder_Write
2024-07-26T09:00:45.064Z	DEBUG	queries/write.go:193	write query	{"yql": "DECLARE $id_0 AS Uuid;\nDECLARE $status_0 AS String;\nUPSERT INTO Backups (id, status) VALUES ($id_0, $status_0);\nDECLARE $id_1 AS Uuid;\nDECLARE $status_1 AS String;\nDECLARE $message_1 AS String;\nUPSERT INTO Operations (id, status, message) VALUES ($id_1, $status_1, $message_1)"}
--- PASS: TestQueryBuilder_Write (0.00s)
🟢 ydbcp/internal/handlers110010ms
🧪 Tests
  • 🟢TestRBOperationHandlerCancellingOperationCancelled
  • 🟢TestRBOperationHandlerCancellingOperationCompletedSuccessfully
  • 🟢TestRBOperationHandlerCancellingOperationInProgress
  • 🟢TestRBOperationHandlerDeadlineExceededForCancellingOperation
  • 🟢TestRBOperationHandlerDeadlineExceededForPendingOperation
  • 🟢TestRBOperationHandlerInvalidOperationResponse
  • 🟢TestRBOperationHandlerPendingOperationCancelled
  • 🟢TestRBOperationHandlerPendingOperationCompletedSuccessfully
  • 🟢TestRBOperationHandlerPendingOperationInProgress
  • 🟢TestRBOperationHandlerRetriableErrorForPendingOperation
  • 🟢TestTBOperationHandler
🖨️ Output
=== RUN   TestRBOperationHandlerInvalidOperationResponse
2024-07-26T09:00:45.661Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "1723beda-ba52-4242-8e71-81bcef03f47b", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:00:45.661Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "1723beda-ba52-4242-8e71-81bcef03f47b", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerInvalidOperationResponse (0.00s)
=== RUN   TestRBOperationHandlerDeadlineExceededForPendingOperation
2024-07-26T09:00:45.661Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "9a10a4e8-3b0b-4bda-a14a-6dc9fdf938e7", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:00:45.661Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "9a10a4e8-3b0b-4bda-a14a-6dc9fdf938e7", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:00:45.661Z	INFO	handlers/restore_backup.go:110	cancelling operation due to ttl	{"id": "9a10a4e8-3b0b-4bda-a14a-6dc9fdf938e7", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerDeadlineExceededForPendingOperation (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationInProgress
2024-07-26T09:00:45.661Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "2b1d021e-89a5-45b5-ad3d-22cac2cc6b11", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "2b1d021e-89a5-45b5-ad3d-22cac2cc6b11", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationInProgress (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationCompletedSuccessfully
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "564d7b6f-9a27-499a-aecf-e4e151a142d5", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "564d7b6f-9a27-499a-aecf-e4e151a142d5", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "564d7b6f-9a27-499a-aecf-e4e151a142d5", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationCompletedSuccessfully (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationCancelled
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "f0a94a0f-f704-481d-9ab5-28423844867c", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "f0a94a0f-f704-481d-9ab5-28423844867c", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "f0a94a0f-f704-481d-9ab5-28423844867c", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationCancelled (0.00s)
=== RUN   TestRBOperationHandlerDeadlineExceededForCancellingOperation
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "f2f2708e-2429-4338-8e63-c2f81760064e", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:00:45.662Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "f2f2708e-2429-4338-8e63-c2f81760064e", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerDeadlineExceededForCancellingOperation (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationInProgress
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "e5d42dd0-366c-4b84-8259-f062785d42dd", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "e5d42dd0-366c-4b84-8259-f062785d42dd", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationInProgress (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationCompletedSuccessfully
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "57a9d2c6-6627-4d4e-b174-f17554329192", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "57a9d2c6-6627-4d4e-b174-f17554329192", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "57a9d2c6-6627-4d4e-b174-f17554329192", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationCompletedSuccessfully (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationCancelled
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "f9610848-9c3f-443b-a8a7-04f07337c2b8", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "f9610848-9c3f-443b-a8a7-04f07337c2b8", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "f9610848-9c3f-443b-a8a7-04f07337c2b8", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationCancelled (0.00s)
=== RUN   TestRBOperationHandlerRetriableErrorForPendingOperation
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "1436bf7c-428d-46b9-a9f8-7e3ed8d834b3", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "1436bf7c-428d-46b9-a9f8-7e3ed8d834b3", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:00:45.663Z	INFO	handlers/restore_backup.go:87	received retriable error	{"id": "1436bf7c-428d-46b9-a9f8-7e3ed8d834b3", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerRetriableErrorForPendingOperation (0.00s)
=== RUN   TestTBOperationHandler
--- PASS: TestTBOperationHandler (0.00s)
🟢 ydbcp/internal/processor2009ms
🧪 Tests
  • 🟢TestOperationHandlerRegistry
  • 🟢TestProcessor
🖨️ Output
=== RUN   TestProcessor
2024-07-26T09:00:45.662Z	DEBUG	processor/processor.go:76	Operation Processor started	{"period": "10s"}
2024-07-26T09:00:45.662Z	DEBUG	processor/processor.go:108	operation already running	{"operation": "Operation, id 71160693-26ea-4d2c-a2ef-c283d39ae84f, type TB, state PENDING"}
2024-07-26T09:00:45.662Z	DEBUG	processor/processor.go:108	operation already running	{"operation": "Operation, id 71160693-26ea-4d2c-a2ef-c283d39ae84f, type TB, state PENDING"}
2024-07-26T09:00:45.662Z	DEBUG	processor/processor.go:120	start operation handler	{"operation": "Operation, id 71160693-26ea-4d2c-a2ef-c283d39ae84f, type TB, state PENDING"}
2024-07-26T09:00:45.662Z	DEBUG	processor/processor_test.go:41	TB handler called for operation	{"operation": "Operation, id 71160693-26ea-4d2c-a2ef-c283d39ae84f, type TB, state PENDING"}
2024-07-26T09:00:45.662Z	DEBUG	processor/processor.go:137	operation handler is finished	{"operationID": "71160693-26ea-4d2c-a2ef-c283d39ae84f"}
2024-07-26T09:00:45.662Z	DEBUG	processor/processor.go:83	Operation Processor stopped
--- PASS: TestProcessor (0.00s)
=== RUN   TestOperationHandlerRegistry
--- PASS: TestOperationHandlerRegistry (0.00s)
🟡 ydbcp/internal/types0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/ticker0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/timer0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/xlog0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/pkg/proto0000ms
🧪 Tests(none)
🖨️ Output
(none)

@github-actions GitHub Actions
Copy link

📝 Test results

ydbcp

15 tests (15 passed) 
%%{init: {"theme":"base","themeVariables":{"fontFamily":"monospace","pieSectionTextSize":"24px","darkMode":true,"pie1":"#2da44e"}}}%%
pie showData
"Passed" : 15
Loading
📦 Package🟢 Passed🔴 Failed🟡 Skipped⏳ Duration
🟡 ydbcp/cmd/ydbcp0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/config0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/connectors/client0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/connectors/db0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟢 ydbcp/internal/connectors/db/yql/queries2000ms
🧪 Tests
  • 🟢TestQueryBuilder_Read
  • 🟢TestQueryBuilder_Write
🖨️ Output
=== RUN   TestQueryBuilder_Read
2024-07-26T09:00:45.064Z	DEBUG	queries/read.go:146	read query	{"yql": "DECLARE $param0 AS String;\nDECLARE $param1 AS String;\nDECLARE $param2 AS String;\nDECLARE $param3 AS String;\nSELECT column1, column2, column3 FROM table1 WHERE (column1 = $param0 OR column1 = $param1) AND (column2 = $param2 OR column2 = $param3)"}
--- PASS: TestQueryBuilder_Read (0.00s)
=== RUN   TestQueryBuilder_Write
2024-07-26T09:00:45.064Z	DEBUG	queries/write.go:193	write query	{"yql": "DECLARE $id_0 AS Uuid;\nDECLARE $status_0 AS String;\nUPSERT INTO Backups (id, status) VALUES ($id_0, $status_0);\nDECLARE $id_1 AS Uuid;\nDECLARE $status_1 AS String;\nDECLARE $message_1 AS String;\nUPSERT INTO Operations (id, status, message) VALUES ($id_1, $status_1, $message_1)"}
--- PASS: TestQueryBuilder_Write (0.00s)
🟢 ydbcp/internal/handlers110011ms
🧪 Tests
  • 🟢TestRBOperationHandlerCancellingOperationCancelled
  • 🟢TestRBOperationHandlerCancellingOperationCompletedSuccessfully
  • 🟢TestRBOperationHandlerCancellingOperationInProgress
  • 🟢TestRBOperationHandlerDeadlineExceededForCancellingOperation
  • 🟢TestRBOperationHandlerDeadlineExceededForPendingOperation
  • 🟢TestRBOperationHandlerInvalidOperationResponse
  • 🟢TestRBOperationHandlerPendingOperationCancelled
  • 🟢TestRBOperationHandlerPendingOperationCompletedSuccessfully
  • 🟢TestRBOperationHandlerPendingOperationInProgress
  • 🟢TestRBOperationHandlerRetriableErrorForPendingOperation
  • 🟢TestTBOperationHandler
🖨️ Output
=== RUN   TestRBOperationHandlerInvalidOperationResponse
2024-07-26T09:59:13.747Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "60b05646-fbdd-4cff-8851-687d03b2da14", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:59:13.747Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "60b05646-fbdd-4cff-8851-687d03b2da14", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerInvalidOperationResponse (0.00s)
=== RUN   TestRBOperationHandlerDeadlineExceededForPendingOperation
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "49336ac7-81cc-412a-bf08-bef4db6a2159", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "49336ac7-81cc-412a-bf08-bef4db6a2159", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:110	cancelling operation due to ttl	{"id": "49336ac7-81cc-412a-bf08-bef4db6a2159", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerDeadlineExceededForPendingOperation (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationInProgress
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "8133d57a-c406-4f3f-8b26-6b0ed088f417", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "8133d57a-c406-4f3f-8b26-6b0ed088f417", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationInProgress (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationCompletedSuccessfully
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "926b2982-f4ba-42c3-af22-02311a0bc05d", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "926b2982-f4ba-42c3-af22-02311a0bc05d", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "926b2982-f4ba-42c3-af22-02311a0bc05d", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationCompletedSuccessfully (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationCancelled
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "10f19cc8-489f-4f4e-8930-7abe7e54bfcb", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "10f19cc8-489f-4f4e-8930-7abe7e54bfcb", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:59:13.748Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "10f19cc8-489f-4f4e-8930-7abe7e54bfcb", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationCancelled (0.00s)
=== RUN   TestRBOperationHandlerDeadlineExceededForCancellingOperation
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "483076a3-0f74-4a33-814e-ff0ff49983c7", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "483076a3-0f74-4a33-814e-ff0ff49983c7", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerDeadlineExceededForCancellingOperation (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationInProgress
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "1171e73e-88b0-4a91-b09a-123fe82032c9", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "1171e73e-88b0-4a91-b09a-123fe82032c9", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationInProgress (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationCompletedSuccessfully
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "80b3fc7b-db0c-4263-b25b-41bc9cf387a6", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "80b3fc7b-db0c-4263-b25b-41bc9cf387a6", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "80b3fc7b-db0c-4263-b25b-41bc9cf387a6", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationCompletedSuccessfully (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationCancelled
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "1071df81-b317-4c3c-b8a2-a51c853d61d9", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "1071df81-b317-4c3c-b8a2-a51c853d61d9", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:59:13.749Z	INFO	handlers/restore_backup.go:173	forgetting operation	{"id": "1071df81-b317-4c3c-b8a2-a51c853d61d9", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationCancelled (0.00s)
=== RUN   TestRBOperationHandlerRetriableErrorForPendingOperation
2024-07-26T09:59:13.750Z	INFO	handlers/restore_backup.go:37	received operation	{"id": "b8347de5-6de5-4cdc-8c16-8b34f02ad480", "type": "RB", "state": "PENDING", "message": ""}
2024-07-26T09:59:13.750Z	INFO	handlers/restore_backup.go:64	getting operation status	{"id": "b8347de5-6de5-4cdc-8c16-8b34f02ad480", "type": "RB", "ydb_operation_id": "1"}
2024-07-26T09:59:13.750Z	INFO	handlers/restore_backup.go:87	received retriable error	{"id": "b8347de5-6de5-4cdc-8c16-8b34f02ad480", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerRetriableErrorForPendingOperation (0.00s)
=== RUN   TestTBOperationHandler
--- PASS: TestTBOperationHandler (0.00s)
🟢 ydbcp/internal/processor2007ms
🧪 Tests
  • 🟢TestOperationHandlerRegistry
  • 🟢TestProcessor
🖨️ Output
=== RUN   TestProcessor
2024-07-26T09:59:13.770Z	DEBUG	processor/processor.go:76	Operation Processor started	{"period": "10s"}
2024-07-26T09:59:13.770Z	DEBUG	processor/processor.go:108	operation already running	{"operation": "Operation, id 774ed0c1-0d51-4b86-98ba-10ce18e73752, type TB, state PENDING"}
2024-07-26T09:59:13.770Z	DEBUG	processor/processor.go:108	operation already running	{"operation": "Operation, id 774ed0c1-0d51-4b86-98ba-10ce18e73752, type TB, state PENDING"}
2024-07-26T09:59:13.770Z	DEBUG	processor/processor.go:120	start operation handler	{"operation": "Operation, id 774ed0c1-0d51-4b86-98ba-10ce18e73752, type TB, state PENDING"}
2024-07-26T09:59:13.770Z	DEBUG	processor/processor_test.go:41	TB handler called for operation	{"operation": "Operation, id 774ed0c1-0d51-4b86-98ba-10ce18e73752, type TB, state PENDING"}
2024-07-26T09:59:13.770Z	DEBUG	processor/processor.go:137	operation handler is finished	{"operationID": "774ed0c1-0d51-4b86-98ba-10ce18e73752"}
2024-07-26T09:59:13.770Z	DEBUG	processor/processor.go:83	Operation Processor stopped
--- PASS: TestProcessor (0.00s)
=== RUN   TestOperationHandlerRegistry
--- PASS: TestOperationHandlerRegistry (0.00s)
🟡 ydbcp/internal/types0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/ticker0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/timer0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/xlog0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/pkg/proto0000ms
🧪 Tests(none)
🖨️ Output
(none)

qrort
qrort approved these changes Jul 26, 2024
View reviewed changes
internal/connectors/db/process_result_set.go Outdated Show resolved Hide resolved
qrort
qrort approved these changes Jul 29, 2024
View reviewed changes
Copy link
Collaborator

@qrort qrort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase and merge

@github-actions GitHub Actions
Copy link

📝 Test results

ydbcp

26 tests (26 passed) 
%%{init: {"theme":"base","themeVariables":{"fontFamily":"monospace","pieSectionTextSize":"24px","darkMode":true,"pie1":"#2da44e"}}}%%
pie showData
"Passed" : 26
Loading
📦 Package🟢 Passed🔴 Failed🟡 Skipped⏳ Duration
🟡 ydbcp/cmd/ydbcp0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/config0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/connectors/client0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/connectors/db0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟢 ydbcp/internal/connectors/db/yql/queries4008ms
🧪 Tests
  • 🟢TestQueryBuilder_CreateCreate
  • 🟢TestQueryBuilder_Read
  • 🟢TestQueryBuilder_UpdateCreate
  • 🟢TestQueryBuilder_UpdateUpdate
🖨️ Output
=== RUN   TestQueryBuilder_Read
2024-07-30T13:24:28.894Z	DEBUG	queries/read.go:146	read query	{"yql": "DECLARE $param0 AS String;\nDECLARE $param1 AS String;\nDECLARE $param2 AS String;\nDECLARE $param3 AS String;\nSELECT column1, column2, column3 FROM table1 WHERE (column1 = $param0 OR column1 = $param1) AND (column2 = $param2 OR column2 = $param3)"}
--- PASS: TestQueryBuilder_Read (0.00s)
=== RUN   TestQueryBuilder_UpdateUpdate
2024-07-30T13:24:28.895Z	DEBUG	queries/write.go:258	write query	{"yql": "DECLARE $id_0 AS Uuid;\nDECLARE $status_0 AS String;\nUPDATE Backups SET status = $status_0 WHERE id = $id_0;\nDECLARE $id_1 AS Uuid;\nDECLARE $status_1 AS String;\nDECLARE $message_1 AS String;\nUPDATE Operations SET status = $status_1, message = $message_1 WHERE id = $id_1"}
--- PASS: TestQueryBuilder_UpdateUpdate (0.00s)
=== RUN   TestQueryBuilder_CreateCreate
2024-07-30T13:24:28.895Z	DEBUG	queries/write.go:258	write query	{"yql": "DECLARE $id_0 AS Uuid;\nDECLARE $container_id_0 AS String;\nDECLARE $database_0 AS String;\nDECLARE $endpoint_0 AS String;\nDECLARE $initiated_0 AS String;\nDECLARE $s3_endpoint_0 AS String;\nDECLARE $s3_region_0 AS String;\nDECLARE $s3_bucket_0 AS String;\nDECLARE $s3_path_prefix_0 AS String;\nDECLARE $status_0 AS String;\nDECLARE $message_0 AS String;\nUPSERT INTO Backups (id, container_id, database, endpoint, initiated, s3_endpoint, s3_region, s3_bucket, s3_path_prefix, status, message) VALUES ($id_0, $container_id_0, $database_0, $endpoint_0, $initiated_0, $s3_endpoint_0, $s3_region_0, $s3_bucket_0, $s3_path_prefix_0, $status_0, $message_0);\nDECLARE $id_1 AS Uuid;\nDECLARE $type_1 AS String;\nDECLARE $status_1 AS String;\nDECLARE $container_id_1 AS String;\nDECLARE $database_1 AS String;\nDECLARE $endpoint_1 AS String;\nDECLARE $backup_id_1 AS Uuid;\nDECLARE $initiated_1 AS String;\nDECLARE $created_at_1 AS Timestamp;\nDECLARE $operation_id_1 AS String;\nDECLARE $message_1 AS String;\nUPSERT INTO Operations (id, type, status, container_id, database, endpoint, backup_id, initiated, created_at, operation_id, message) VALUES ($id_1, $type_1, $status_1, $container_id_1, $database_1, $endpoint_1, $backup_id_1, $initiated_1, $created_at_1, $operation_id_1, $message_1)"}
--- PASS: TestQueryBuilder_CreateCreate (0.00s)
=== RUN   TestQueryBuilder_UpdateCreate
2024-07-30T13:24:28.895Z	DEBUG	queries/write.go:258	write query	{"yql": "DECLARE $id_0 AS Uuid;\nDECLARE $status_0 AS String;\nUPDATE Backups SET status = $status_0 WHERE id = $id_0;\nDECLARE $id_1 AS Uuid;\nDECLARE $type_1 AS String;\nDECLARE $status_1 AS String;\nDECLARE $container_id_1 AS String;\nDECLARE $database_1 AS String;\nDECLARE $endpoint_1 AS String;\nDECLARE $backup_id_1 AS Uuid;\nDECLARE $initiated_1 AS String;\nDECLARE $created_at_1 AS Timestamp;\nDECLARE $operation_id_1 AS String;\nDECLARE $message_1 AS String;\nUPSERT INTO Operations (id, type, status, container_id, database, endpoint, backup_id, initiated, created_at, operation_id, message) VALUES ($id_1, $type_1, $status_1, $container_id_1, $database_1, $endpoint_1, $backup_id_1, $initiated_1, $created_at_1, $operation_id_1, $message_1)"}
--- PASS: TestQueryBuilder_UpdateCreate (0.00s)
🟢 ydbcp/internal/handlers200010ms
🧪 Tests
  • 🟢TestRBOperationHandlerCancellingOperationCancelled
  • 🟢TestRBOperationHandlerCancellingOperationCompletedSuccessfully
  • 🟢TestRBOperationHandlerCancellingOperationInProgress
  • 🟢TestRBOperationHandlerDeadlineExceededForCancellingOperation
  • 🟢TestRBOperationHandlerDeadlineExceededForPendingOperation
  • 🟢TestRBOperationHandlerInvalidOperationResponse
  • 🟢TestRBOperationHandlerPendingOperationCancelled
  • 🟢TestRBOperationHandlerPendingOperationCompletedSuccessfully
  • 🟢TestRBOperationHandlerPendingOperationInProgress
  • 🟢TestRBOperationHandlerRetriableErrorForPendingOperation
  • 🟢TestTBOperationHandlerCancellingOperationCancelled
  • 🟢TestTBOperationHandlerCancellingOperationCompletedSuccessfully
  • 🟢TestTBOperationHandlerCancellingOperationInProgress
  • 🟢TestTBOperationHandlerDeadlineExceededForCancellingOperation
  • 🟢TestTBOperationHandlerDeadlineExceededForPendingOperation
  • 🟢TestTBOperationHandlerInvalidOperationResponse
  • 🟢TestTBOperationHandlerPendingOperationCancelled
  • 🟢TestTBOperationHandlerPendingOperationCompletedSuccessfully
  • 🟢TestTBOperationHandlerPendingOperationInProgress
  • 🟢TestTBOperationHandlerRetriableErrorForPendingOperation
🖨️ Output
=== RUN   TestRBOperationHandlerInvalidOperationResponse
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "68c1c834-928c-477a-a284-94799ec6c0d4", "type": "RB", "state": "PENDING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "68c1c834-928c-477a-a284-94799ec6c0d4", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerInvalidOperationResponse (0.00s)
=== RUN   TestRBOperationHandlerDeadlineExceededForPendingOperation
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "967a6c38-1851-4b38-94d9-0ccb90d4ef24", "type": "RB", "state": "PENDING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "967a6c38-1851-4b38-94d9-0ccb90d4ef24", "type": "RB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:98	cancelling operation	{"reason": "TTL", "id": "967a6c38-1851-4b38-94d9-0ccb90d4ef24", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerDeadlineExceededForPendingOperation (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationInProgress
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "72bd1f77-02cc-4687-9f85-0c0dfabb3827", "type": "RB", "state": "PENDING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "72bd1f77-02cc-4687-9f85-0c0dfabb3827", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationInProgress (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationCompletedSuccessfully
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "9184af5d-1aae-4e52-8983-dc9e78ef4ff0", "type": "RB", "state": "PENDING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "9184af5d-1aae-4e52-8983-dc9e78ef4ff0", "type": "RB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:119	forgetting operation	{"id": "9184af5d-1aae-4e52-8983-dc9e78ef4ff0", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationCompletedSuccessfully (0.00s)
=== RUN   TestRBOperationHandlerPendingOperationCancelled
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "952b2d8f-6461-4e82-98fb-c7325e2661bf", "type": "RB", "state": "PENDING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "952b2d8f-6461-4e82-98fb-c7325e2661bf", "type": "RB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:119	forgetting operation	{"id": "952b2d8f-6461-4e82-98fb-c7325e2661bf", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerPendingOperationCancelled (0.00s)
=== RUN   TestRBOperationHandlerDeadlineExceededForCancellingOperation
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "4c2e261a-f11a-4f2e-8755-5f6059641696", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "4c2e261a-f11a-4f2e-8755-5f6059641696", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerDeadlineExceededForCancellingOperation (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationInProgress
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "fc3bb876-1343-4df6-9e89-0762c813e797", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "fc3bb876-1343-4df6-9e89-0762c813e797", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationInProgress (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationCompletedSuccessfully
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "ffc51aa1-8a1c-484f-818f-075a94459f63", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "ffc51aa1-8a1c-484f-818f-075a94459f63", "type": "RB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:119	forgetting operation	{"id": "ffc51aa1-8a1c-484f-818f-075a94459f63", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationCompletedSuccessfully (0.00s)
=== RUN   TestRBOperationHandlerCancellingOperationCancelled
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "197d51ff-2452-4c07-b1a7-cb1a3230f413", "type": "RB", "state": "CANCELLING", "message": ""}
2024-07-30T13:24:29.475Z	INFO	handlers/utils.go:45	getting operation status	{"id": "197d51ff-2452-4c07-b1a7-cb1a3230f413", "type": "RB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.475Z	INFO	handlers/restore_backup.go:119	forgetting operation	{"id": "197d51ff-2452-4c07-b1a7-cb1a3230f413", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerCancellingOperationCancelled (0.00s)
=== RUN   TestRBOperationHandlerRetriableErrorForPendingOperation
2024-07-30T13:24:29.476Z	INFO	handlers/restore_backup.go:30	received operation	{"id": "971a9998-c028-41c6-bca2-07cca9951740", "type": "RB", "state": "PENDING", "message": ""}
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "971a9998-c028-41c6-bca2-07cca9951740", "type": "RB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:68	received retriable error	{"id": "971a9998-c028-41c6-bca2-07cca9951740", "type": "RB", "ydb_operation_id": "1"}
--- PASS: TestRBOperationHandlerRetriableErrorForPendingOperation (0.00s)
=== RUN   TestTBOperationHandlerInvalidOperationResponse
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "253b8a29-09fb-4295-95bf-7ea5ac9f286b", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerInvalidOperationResponse (0.00s)
=== RUN   TestTBOperationHandlerDeadlineExceededForPendingOperation
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "d469d0da-4eb2-4062-8d5c-e1e4ab076d4b", "type": "TB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:98	cancelling operation	{"reason": "TTL", "id": "d469d0da-4eb2-4062-8d5c-e1e4ab076d4b", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerDeadlineExceededForPendingOperation (0.00s)
=== RUN   TestTBOperationHandlerPendingOperationInProgress
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "5d342052-b1b4-4420-b2bc-654d91790caf", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerPendingOperationInProgress (0.00s)
=== RUN   TestTBOperationHandlerPendingOperationCompletedSuccessfully
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "d1cf57e4-b0d0-48a0-8c2b-80e966006004", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerPendingOperationCompletedSuccessfully (0.00s)
=== RUN   TestTBOperationHandlerPendingOperationCancelled
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "c25b5850-7dd2-4b7e-a573-78b6b633f0ba", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerPendingOperationCancelled (0.00s)
=== RUN   TestTBOperationHandlerDeadlineExceededForCancellingOperation
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "d8546731-6fad-41b3-a5c5-4f9569405754", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerDeadlineExceededForCancellingOperation (0.00s)
=== RUN   TestTBOperationHandlerCancellingOperationInProgress
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "c71de877-83e2-401d-b3fb-95d11994002c", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerCancellingOperationInProgress (0.00s)
=== RUN   TestTBOperationHandlerCancellingOperationCompletedSuccessfully
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "09bf36e2-6c1d-486b-a7a3-dff146f1bfaa", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerCancellingOperationCompletedSuccessfully (0.00s)
=== RUN   TestTBOperationHandlerCancellingOperationCancelled
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "3245871b-cf0c-43b7-b085-fb424532d501", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerCancellingOperationCancelled (0.00s)
=== RUN   TestTBOperationHandlerRetriableErrorForPendingOperation
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:45	getting operation status	{"id": "720a32e2-74f9-4ae1-a23a-7bf062bd514c", "type": "TB", "ydb_operation_id": "1"}
2024-07-30T13:24:29.476Z	INFO	handlers/utils.go:68	received retriable error	{"id": "720a32e2-74f9-4ae1-a23a-7bf062bd514c", "type": "TB", "ydb_operation_id": "1"}
--- PASS: TestTBOperationHandlerRetriableErrorForPendingOperation (0.00s)
🟢 ydbcp/internal/processor2008ms
🧪 Tests
  • 🟢TestOperationHandlerRegistry
  • 🟢TestProcessor
🖨️ Output
=== RUN   TestProcessor
2024-07-30T13:24:29.476Z	DEBUG	processor/processor.go:76	Operation Processor started	{"period": "10s"}
2024-07-30T13:24:29.476Z	DEBUG	processor/processor.go:108	operation already running	{"operation": "Operation, id edeb114c-2af4-4617-a7b4-7f70994d32c8, type TB, state PENDING"}
2024-07-30T13:24:29.476Z	DEBUG	processor/processor.go:108	operation already running	{"operation": "Operation, id edeb114c-2af4-4617-a7b4-7f70994d32c8, type TB, state PENDING"}
2024-07-30T13:24:29.476Z	DEBUG	processor/processor.go:120	start operation handler	{"operation": "Operation, id edeb114c-2af4-4617-a7b4-7f70994d32c8, type TB, state PENDING"}
2024-07-30T13:24:29.476Z	DEBUG	processor/processor_test.go:41	TB handler called for operation	{"operation": "Operation, id edeb114c-2af4-4617-a7b4-7f70994d32c8, type TB, state PENDING"}
2024-07-30T13:24:29.476Z	DEBUG	processor/processor.go:137	operation handler is finished	{"operationID": "edeb114c-2af4-4617-a7b4-7f70994d32c8"}
2024-07-30T13:24:29.476Z	DEBUG	processor/processor.go:83	Operation Processor stopped
--- PASS: TestProcessor (0.00s)
=== RUN   TestOperationHandlerRegistry
--- PASS: TestOperationHandlerRegistry (0.00s)
🟡 ydbcp/internal/types0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/ticker0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/timer0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/internal/util/xlog0000ms
🧪 Tests(none)
🖨️ Output
(none)
🟡 ydbcp/pkg/proto/ydbcp/v1alpha10000ms
🧪 Tests(none)
🖨️ Output
(none)

@bma13 bma13 merged commit 255f79f into main Jul 30, 2024
1 check passed
@bma13 bma13 deleted the export-s3 branch July 30, 2024 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qrort qrort qrort approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@bma13 @qrort