[stdout] Reboot Test === RUN TestReboot reboot_test.go:162: time: 2023-10-13T15:48:33.509405626Z out: Wait for state of 2140eee4-132c-4c5a-b17b-aa5d905a84aa testContext.go:380: done waiting for State reboot_test.go:166: time: 2023-10-13T15:48:33.509427106Z out: timewait: 20m0s reboot_test.go:167: time: 2023-10-13T15:48:33.509432316Z out: reboot: true reboot_test.go:168: time: 2023-10-13T15:48:33.509436733Z out: count: 1 reboot_test.go:172: time: 2023-10-13T15:48:33.509442614Z out: LastRebootTime: 2023-10-13 15:27:37.239123613 +0000 UTC reboot_test.go:174: time: 2023-10-13T15:48:33.509447574Z out: LastRebootReason: NORMAL: First boot of device - at 2023-10-13T15:27:37.239123613Z config changed, to see config run 'eden controller edge-node get-config' rebooted with reason Reboot reason - kernel crash (no kdump) - at 2023-10-13T15:56:44.421651599Z at 2023-10-13 15:56:44.421651599 +0000 UTC/n reboot_test.go:64: abnormal reboot: Reboot reason - kernel crash (no kdump) - at 2023-10-13T15:56:44.421651599Z testContext.go:302: WaitForProc terminated by timeout 20m0s reboot_test.go:186: time: 2023-10-13T15:57:24.524786933Z out: Number of reboots: 0 --- FAIL: TestReboot (531.02s) FAIL FAIL: ../eclient/testdata/shutdown_test.txt:74: command failure

Process completed with exit code 1.

[stdout] time="2023-10-13T16:41:42Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:41:56Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:42:10Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:42:23Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:42:37Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:42:51Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:43:04Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:43:18Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:43:31Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:43:45Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:43:59Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:44:12Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:44:26Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:44:40Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:44:53Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:45:07Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:45:20Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:45:34Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:45:48Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:46:01Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:46:15Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:46:29Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:46:42Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:46:56Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:47:10Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:47:23Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:47:37Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:47:50Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:48:04Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:48:18Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:48:32Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:48:45Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:48:59Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:49:13Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:49:26Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:49:40Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:49:54Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:50:07Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:50:21Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:50:35Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:50:48Z" level=fatal msg="command ssh failed: exit status 1" time="2023-10-13T16:51:02Z" level=fatal msg="command ssh failed: exit status 1" [stderr] ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3' (115) ERROR 2002 (HY000): Can't connect to server on '10.11.12.3

Process completed with exit code 1.

[stdout] Docker app's state test === RUN TestAppStatus apps: '[t1 t2]' state: 'RUNNING' secs: 600 time: 2023-10-13T16:52:02.003815197Z out: appName t1 state changed to UNKNOWN time: 2023-10-13T16:52:02.003825416Z out: appName t2 state changed to UNKNOWN time: 2023-10-13T16:52:05.663942003Z out: appName t1 state changed to INITIAL time: 2023-10-13T16:52:07.664539157Z out: appName t2 state changed to INITIAL time: 2023-10-13T16:52:07.664556619Z out: appName t1 state changed to RESOLVING_TAG time: 2023-10-13T16:52:10.666380575Z out: appName t1 state changed to DOWNLOAD_STARTED time: 2023-10-13T16:52:12.668457418Z out: appName t1 state changed to DOWNLOAD_STARTED (0%) time: 2023-10-13T16:52:12.668510558Z out: appName t2 state changed to DOWNLOAD_STARTED time: 2023-10-13T16:52:13.66900731Z out: appName t2 state changed to DOWNLOAD_STARTED (0%) time: 2023-10-13T16:52:21.674360781Z out: appName t2 state changed to DOWNLOAD_STARTED (100%) time: 2023-10-13T16:52:47.69467664Z out: appName t1 state changed to DOWNLOAD_STARTED (100%) time: 2023-10-13T16:53:03.704508609Z out: appName t1 state changed to DOWNLOAD_STARTED (0%) time: 2023-10-13T16:53:05.705093197Z out: appName t2 state changed to DOWNLOAD_STARTED (0%) time: 2023-10-13T16:53:08.706879952Z out: appName t2 state changed to DOWNLOAD_STARTED (1%) time: 2023-10-13T16:53:12.709295286Z out: appName t1 state changed to DOWNLOAD_STARTED (1%) time: 2023-10-13T16:53:13.710056054Z out: appName t1 state changed to DOWNLOAD_STARTED (4%) time: 2023-10-13T16:53:14.710917647Z out: appName t2 state changed to DOWNLOAD_STARTED (4%) time: 2023-10-13T16:53:15.711628499Z out: appName t2 state changed to DOWNLOAD_STARTED (6%) time: 2023-10-13T16:53:16.712393387Z out: appName t1 state changed to DOWNLOAD_STARTED (5%) time: 2023-10-13T16:53:17.713133821Z out: appName t1 state changed to DOWNLOAD_STARTED (6%) time: 2023-10-13T16:53:22.716713514Z out: appName t2 state changed to DOWNLOAD_STARTED (7%) time: 2023-10-13T16:53:24.718075208Z out: appName t1 state changed to DOWNLOAD_STARTED (7%) time: 2023-10-13T16:53:25.718621757Z out: appName t2 state changed to DOWNLOAD_STARTED (8%) time: 2023-10-13T16:53:26.718896288Z out: appName t1 state changed to DOWNLOAD_STARTED (10%) time: 2023-10-13T16:53:28.720744087Z out: appName t2 state changed to DOWNLOAD_STARTED (9%) time: 2023-10-13T16:53:29.721141289Z out: appName t2 state changed to DOWNLOAD_STARTED (44%) time: 2023-10-13T16:53:32.723641144Z out: appName t1 state changed to DOWNLOAD_STARTED (44%) time: 2023-10-13T16:53:33.724035787Z out: appName t2 state changed to DOWNLOAD_STARTED (82%) time: 2023-10-13T16:53:35.725420995Z out: appName t1 state changed to DOWNLOAD_STARTED (82%) time: 2023-10-13T16:53:35.725444109Z out: appName t2 state changed to DOWNLOAD_STARTED (83%) time: 2023-10-13T16:53:35.725467754Z out: appName t1 state changed to DOWNLOAD_STARTED (83%) time: 2023-10-13T16:53:36.726170537Z out: appName t2 state changed to DOWNLOAD_STARTED (84%) time: 2023-10-13T16:53:37.727188428Z out: appName t1 state changed to DOWNLOAD_STARTED (84%) time: 2023-10-13T16:53:39.72893148Z out: appName t2 state changed to DOWNLOAD_STARTED (85%) time: 2023-10-13T16:53:40.72999091Z out: appName t1 state changed to DOWNLOAD_STARTED (85%) time: 2023-10-13T16:53:43.732779205Z out: appName t1 state changed to DOWNLOAD_STARTED (87%) time: 2023-10-13T16:53:43.732827998Z out: appName t2 state changed to DOWNLOAD_STARTED (87%) time: 2023-10-13T16:53:44.733968395Z out: appName t1 state changed to DOWNLOAD_STARTED (88%) time: 2023-10-13T16:53:44.734004994Z out: appName t2 state changed to DOWNLOAD_STARTED (91%) time: 2023-10-13T16:53:44.73401358Z out: appName t1 state changed to DOWNLOAD_STARTED (91%) time: 2023-10-13T16:53:45.734606509Z out: appName t2 state changed to DOWNLOAD_STARTED (92%) time: 2023-10-13T16:53:46.735753786Z out: appName t1 state changed to DOWNLOAD_STARTED (92%) time: 2023-10-13T16:53:47.736302047Z out: appName t2 state changed to DOWNLOAD_STARTED (94%) time: 2023-10-13T16:53:47.736327635Z out: appName t1

Process completed with exit code 1.

[stdout] Reboot Test === RUN TestReboot reboot_test.go:162: time: 2023-10-13T16:41:06.457057736Z out: Wait for state of 0be3b074-cec5-4dbc-8193-d1f639bcf5b1 testContext.go:380: done waiting for State reboot_test.go:166: time: 2023-10-13T16:41:06.457093844Z out: timewait: 1h40m0s reboot_test.go:167: time: 2023-10-13T16:41:06.457098322Z out: reboot: false reboot_test.go:168: time: 2023-10-13T16:41:06.457102289Z out: count: 3 reboot_test.go:172: time: 2023-10-13T16:41:06.457108Z out: LastRebootTime: 2023-10-13 16:04:21.661681863 +0000 UTC reboot_test.go:174: time: 2023-10-13T16:41:06.45711347Z out: LastRebootReason: NORMAL: First boot of device - at 2023-10-13T16:04:21.661681863Z rebooted with reason NORMAL: controller reboot at EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T16:49:28.534891086Z at 2023-10-13 16:49:28.793660281 +0000 UTC/nrebooted with reason Reboot from agent zedbox[1369] in partition IMGA at EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T17:01:04.276551487Z: fatal: agent zedbox[1369]: couldn't initialize containerd (this should not happen): initContainerdClient: could not create containerd client. failed to dial "/run/containerd/containerd.sock": write unix @->/run/containerd/containerd.sock: use of closed network connection. Exiting. Watchdog report for IMGA EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T17:05:57+00:00: 2 /run/zedrouter.touch at 2023-10-13 17:01:04.748133879 +0000 UTC/n testProc.go:84: time: 2023-10-13T17:09:19.524794669Z out: projects.ProcInfoFunc done with return: abnormal reboot: Reboot from agent zedbox[1369] in partition IMGA at EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T17:01:04.276551487Z: fatal: agent zedbox[1369]: couldn't initialize containerd (this should not happen): initContainerdClient: could not create containerd client. failed to dial "/run/containerd/containerd.sock": write unix @->/run/containerd/containerd.sock: use of closed network connection. Exiting. Watchdog report for IMGA EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T17:05:57+00:00: 2 /run/zedrouter.touch testContext.go:271: done for device 0be3b074-cec5-4dbc-8193-d1f639bcf5b1 reboot_test.go:186: time: 2023-10-13T17:09:19.524855843Z out: Number of reboots: 1 --- PASS: TestReboot (1693.07s) PASS [stderr] time="2023-10-13T17:09:19Z" level=info msg="time: 2023-10-13T17:09:19.524794669Z out: projects.ProcInfoFunc done with return: abnormal reboot: Reboot from agent zedbox[1369] in partition IMGA at EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T17:01:04.276551487Z: fatal: agent zedbox[1369]: couldn't initialize containerd (this should not happen): initContainerdClient: could not create containerd client. failed to dial \"/run/containerd/containerd.sock\": write unix @->/run/containerd/containerd.sock: use of closed network connection. Exiting.\nWatchdog report for IMGA EVE version 0.0.0-pr3496-df490a94-kvm-amd64 at 2023-10-13T17:05:57+00:00: 2 /run/zedrouter.touch\n" FAIL: ../eclient/testdata/dev_local_info.txt:128: unexpected command success

[stdout] Docker app's state test === RUN TestAppStatus apps: '[app1]' state: 'RUNNING' secs: 1200 time: 2023-10-13T16:59:12.560334455Z out: appName app1 state changed to HALTED time: 2023-10-13T17:09:19.018035889Z out: appName app1 state changed to UNKNOWN [context canceled] FAIL: ../eclient/testdata/dev_local_info.txt:128: test interrupted while running command

Process completed with exit code 1.

[stdout] VNC access to app Test === RUN TestVNCVMStart vnc_test.go:245: Add app to list config changed, to see config run 'eden controller edge-node get-config' vnc_test.go:251: Add processing of app running messages vnc_test.go:101: state: DOWNLOAD_STARTED received in: 2023-10-13T17:04:17.69431969Z vnc_test.go:101: state: LOADING received in: 2023-10-13T17:09:31.948574557Z vnc_test.go:101: state: CREATING_VOLUME received in: 2023-10-13T17:12:44.073816445Z vnc_test.go:101: state: INSTALLED received in: 2023-10-13T17:15:18.197246801Z --- app vnc-app logs --- ------ testContext.go:285: WaitForProcWithErrorCallback terminated by timeout 15m0s --- FAIL: TestVNCVMStart (901.04s) FAIL [stderr] time="2023-10-13T17:04:03Z" level=info msg="Starting download of image from https://cloud-images.ubuntu.com/releases/groovy/release-20210108/ubuntu-20.10-server-cloudimg-amd64.img" [exit status 1] FAIL: ../vnc/testdata/vnc_test.txt:7: command failure