WIP Add a `k6 cloud run --local-execution` mode of execution #3818

oleiade · 2024-06-27T15:55:36Z

What?

This PR adds support for a new --local-execution flag to the k6 cloud run command. Using it, users can run their test script or archive on their local machine, while streaming the results to the cloud.

It is built upon the latter, and ⚠️ cannot be merged before #3813 ⚠️.

In the context of this PR, the goal is to reproduce the behavior of the k6 run -o cloud behavior, and depending on the advancement of internal work, to also add support for adding archive upload. In a future PR, at a later point in time, we will also add support for logs and traces streaming to the cloud but are dependent on internal work to be able to do so.

⚠️ This is currently a work-in-progress, and non-functional (yet) PR ⚠️

Why?

We aim to integrate the experience of running tests locally, whilst streaming results to the cloud more intuitive, and integrated in the k6 experience. We also aim to take this opportunity to align some of the logic and behavior involved in running tests in the cloud and running tests locally better.

Task list

We're gonna introduce a couple of options specific to --local-execution. One of the ideas we've been having in the k6 v1 UX work stream revolved around the idea of scoping options with a dot separator like in Prometheus: --local.execution, --local.send-archive, --local.send-logs, etc. cc @dgzlopes
Once we're at that point we'd like sending the archive to be opt-in (via an option that users need to explicitly pass). Same goes for logs. The reason being that they might contain sensitive info. cc @dgzlopes
When using k6 cloud run --local-execution, we should make sure the outputs field of the terminal output of the command displays a sensible value (like the URL of the job in the grafana cloud app?). cc @dgzlopes
Should the k6 cloud run --local-execution show the same terminal output as k6 run or k6 cloud run?

Checklist

I have performed a self-review of my code.
I have added tests for my changes.
I have run linter locally (make lint) and all checks pass.
I have run tests locally (make tests) and all tests pass.
I have commented on my code, particularly in hard-to-understand areas.

Related PR(s)/Issue(s)

codecov-commenter · 2024-06-27T16:05:32Z

Codecov Report

Attention: Patch coverage is 15.29412% with 216 lines in your changes missing coverage. Please review.

Please upload report for BASE (cloud_subcommands@1468a97). Learn more about missing BASE report.

❗ Current head 0226b54 differs from pull request most recent head 812efc5

Please upload reports for the commit 812efc5 to get more accurate results.

Files	Patch %	Lines
cmd/cloud_run.go	9.32%	212 Missing and 2 partials ⚠️
js/modules/k6/html/html.go	90.90%	0 Missing and 1 partial ⚠️
js/runner.go	50.00%	1 Missing ⚠️

Additional details and impacted files

@@                 Coverage Diff                  @@
##             cloud_subcommands    #3818   +/-   ##
====================================================
  Coverage                     ?   70.98%           
====================================================
  Files                        ?      293           
  Lines                        ?    21827           
  Branches                     ?        0           
====================================================
  Hits                         ?    15494           
  Misses                       ?     5244           
  Partials                     ?     1089

Flag	Coverage Δ
ubuntu	`70.92% <15.29%> (?)`
windows	`70.85% <15.29%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

joanlopez · 2024-07-15T16:14:14Z

One of the ideas we've been having in the k6 v1 UX work stream revolved around the idea of scoping options with a dot separator like in Prometheus: --local.execution, --local.send-archive, --local.send-logs, etc

I don't know how sensible that would be, but as far as I know, using the dot as separator is not very Windows friendly (it requires quoting, see PowerShell/PowerShell#19640), just to be considered when evaluating it.

joanlopez · 2024-07-15T16:16:44Z

cmd/cloud_run.go

+	// TODO: Figure out a better way to handle the CLI flags
+	flags.BoolVar(&c.exitOnRunning, "exit-on-running", c.exitOnRunning,
+		"exits when test reaches the running status")


Should we add some kind of validation for incompatible flags? For instance, --exit-on-running & --local-execution, which I think don't have sense together.

Great idea, let's look into it 👍🏻

joanlopez · 2024-07-15T16:19:06Z

cmd/cloud_run.go

+	flags.BoolVar(&c.uploadOnly, "upload-only", c.uploadOnly,
+		"only upload the test to the cloud without actually starting a test run")


Dropping this 👍

Important notice: this commit declare a cobra sub-command holding the logic for the `k6 cloud run` sub-command, but does not register it. In this commit, we duplicate the logic from the existing `k6 cloud` logic, with very little adjustments, to support the later registry of the `k6 cloud run` command. To simplify the collaboration on this and further reviews, we delegate any refactoring of the original cloud command's logic, to a further commit or Pull Request.

Important notice: this commit declare a cobra sub-command holding the logic for the `k6 cloud login` sub-command, but does not register it. In this commit, we duplicate the logic from the existing `k6 login` logic, with very little adjustments, to support the later registry of the `k6 cloud login` command. To simplify the collaboration on this and further reviews, we delegate any refactoring of the original cloud command's logic, to a further commit or Pull Request. This new `k6 cloud login` command is notably focusing solely on authenticating with the Grafana Cloud k6, and by design does not aim to support InfluxDB authentication.

Co-authored-by: Joan López de la Franca Beltran <[email protected]>

codebien · 2024-07-30T14:22:23Z

One of the ideas we've been having in the k6 v1 UX work stream revolved around the idea of scoping options with a dot separator like in Prometheus

If we already envision the need for it, isn't better to have a dedicated subcommand? I mean something like k6 cloud local-run --send-logs script.js

oleiade · 2024-08-21T08:22:10Z

Closing in favor of #3904

oleiade added the enhancement label Jun 27, 2024

oleiade assigned oleiade and joanlopez Jun 27, 2024

oleiade changed the base branch from master to cloud_subcommands June 27, 2024 15:56

joanlopez reviewed Jul 15, 2024

View reviewed changes

oleiade and others added 6 commits July 18, 2024 16:22

Register run and login subcommands of the cloud command

3c6a22e

Add deprecation warning to k6 login and k6 login cloud commands

199b63b

FIXME add tests assert k6 cloud run command's arguments handling

eff4f04

Apply Pull-Request suggestions

cf7e0b9

Co-authored-by: Joan López de la Franca Beltran <[email protected]>

oleiade force-pushed the cloud_subcommands branch from ddce8cb to cf7e0b9 Compare July 18, 2024 14:22

oleiade force-pushed the cloud_run_local_execution branch from 812efc5 to 9957dc9 Compare July 18, 2024 14:33

WIP

25dd426

oleiade force-pushed the cloud_run_local_execution branch from 9957dc9 to 25dd426 Compare July 19, 2024 12:45

oleiade force-pushed the cloud_subcommands branch from aad9c76 to 2b0bc1f Compare July 23, 2024 15:47

Base automatically changed from cloud_subcommands to master July 24, 2024 10:53

joanlopez mentioned this pull request Jul 30, 2024

Release v0.53.0 #3872

Merged

5 tasks

oleiade mentioned this pull request Aug 21, 2024

[1/2] Implement the --local-execution mode for k6 cloud run #3904

Merged

5 tasks

oleiade closed this Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP Add a `k6 cloud run --local-execution` mode of execution #3818

WIP Add a `k6 cloud run --local-execution` mode of execution #3818

oleiade commented Jun 27, 2024 •

edited

Loading

codecov-commenter commented Jun 27, 2024 •

edited

Loading

joanlopez commented Jul 15, 2024 •

edited

Loading

joanlopez Jul 15, 2024 •

edited

Loading

oleiade Jul 16, 2024

joanlopez Jul 15, 2024

oleiade Jul 19, 2024

codebien commented Jul 30, 2024 •

edited

Loading

oleiade commented Aug 21, 2024

		flags.BoolVar(&c.uploadOnly, "upload-only", c.uploadOnly,
		"only upload the test to the cloud without actually starting a test run")

WIP Add a k6 cloud run --local-execution mode of execution #3818

WIP Add a k6 cloud run --local-execution mode of execution #3818

Conversation

oleiade commented Jun 27, 2024 • edited Loading

What?

Why?

Task list

Checklist

Related PR(s)/Issue(s)

codecov-commenter commented Jun 27, 2024 • edited Loading

Codecov Report

joanlopez commented Jul 15, 2024 • edited Loading

joanlopez Jul 15, 2024 • edited Loading

Choose a reason for hiding this comment

oleiade Jul 16, 2024

Choose a reason for hiding this comment

joanlopez Jul 15, 2024

Choose a reason for hiding this comment

oleiade Jul 19, 2024

Choose a reason for hiding this comment

codebien commented Jul 30, 2024 • edited Loading

oleiade commented Aug 21, 2024

WIP Add a `k6 cloud run --local-execution` mode of execution #3818

WIP Add a `k6 cloud run --local-execution` mode of execution #3818

oleiade commented Jun 27, 2024 •

edited

Loading

codecov-commenter commented Jun 27, 2024 •

edited

Loading

joanlopez commented Jul 15, 2024 •

edited

Loading

joanlopez Jul 15, 2024 •

edited

Loading

codebien commented Jul 30, 2024 •

edited

Loading