From 6b17f75d94568ef0ba55b91e8930974fab04426f Mon Sep 17 00:00:00 2001 From: Julianne Fermi Date: Fri, 25 Aug 2023 09:45:18 -0500 Subject: [PATCH] Add Dashboard and CLI Tabs To scheduling tests and test triggers pages. --- docs/docs/articles/scheduling-tests.mdx | 196 ++++++++++++++++++++++++ docs/docs/articles/test-triggers.mdx | 190 +++++++++++++++++++++++ 2 files changed, 386 insertions(+) create mode 100644 docs/docs/articles/scheduling-tests.mdx create mode 100644 docs/docs/articles/test-triggers.mdx diff --git a/docs/docs/articles/scheduling-tests.mdx b/docs/docs/articles/scheduling-tests.mdx new file mode 100644 index 00000000000..2b3e91e6693 --- /dev/null +++ b/docs/docs/articles/scheduling-tests.mdx @@ -0,0 +1,196 @@ +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; + +# Scheduling Tests + + + + +## Create a Scheduled Test or Test Suite from the Testkube Dashboard + +In order to run Tests and Test Suites on a regular basis, we support a scheduling mechanism for these objects. +CRDs both for Tests and Test Suites contain a **schedule** field used to define rules for launching them in time. +Testkube's schedule data format is the same that is used to define Kubernetes Cron jobs (check Wikipedia Cron format for details ). + +## Scheduling Architecture + +Testkube uses the scheduling engine from Kubernetes Cron jobs. +In fact, for each scheduled Test or Test Suite, a special cron job is created from this [template](ttps://github.com/kubeshop/helm-charts/tree/main/charts/testkube-operator/cronjob-template.yml). + +Technically, it is a callback to the Testkube API server method, launching either Test or Test Suite execution. + +This works similarly to scheduled Test and Test Suite executions done by external scheduling platforms. + +In the Testkube Dashboard, when you have selected a Test or Test Suite, in the **Settings** tab, click on **Scheduling** to create a schedule for the Test or Test Suite. + +![Schedule Tests](../img/schedule-tests.png) + +![Schedule Tests Test Suites](../img/schedule-test-suites.png) + + + + + +In order to run Tests and Test Suites on a regular basis, we support a scheduling mechanism for these objects. +CRDs both for Tests and Test Suites contain a **schedule** field used to define rules for launching them in time. +Testkube's schedule data format is the same that is used to define Kubernetes Cron jobs (check [Wikipedia Cron format](https://en.wikipedia.org/wiki/Cron) for details. + +## Scheduling Architecture + +Testkube uses the scheduling engine from Kubernetes Cron jobs. +In fact, for each scheduled Test or Test Suite, a special cron job is created from this [template](https://github.com/kubeshop/helm-charts/tree/main/charts/testkube-operator/cronjob-template.yml). + +Technically, it is a callback to the Testkube API server method, launching either Test or Test Suite execution. +This works similarly to scheduled Test and Test Suite executions done by external scheduling platforms. + +## Create a Test with a Schedule + +Let's create a test with a required schedule using Testkube CLI command: + +```sh +kubectl testkube create test --file test/postman/TODO.postman_collection.json --name scheduled-test --schedule="*/1 * * * *" +``` + +```sh title="Expected output:" +Detected test type postman/collection +Test created / scheduled-test 🥇 +``` + +We successfully created a scheduled test and can check the list of the available tests: + +```sh +kubectl testkube get tests +``` + +```sh title="Expected output:" + NAME | TYPE | CREATED | LABELS | SCHEDULE | STATUS | EXECUTION ID ++-------------------+--------------------+-------------------------------+--------+-------------+--------+--------------------------+ + scheduled-test | postman/collection | 2022-04-13 12:37:40 +0000 UTC | | */1 * * * * | | +``` + +The scheduled test was created and successfully scheduled for execution. + +Let's check a Cron job connected to this test: + +```sh +kubectl get cronjobs -A +``` + +```sh title="Expected output:" +NAMESPACE NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE +testkube scheduled-test-tests */1 * * * * False 1 42s 3m22s +``` + +The Cron job for this test was successfully created and the test was executed. + +To the check Cron job details: + +```sh +kubectl describe cronjob scheduled-test-tests -n testkube +``` + +```sh title="Expected output:" +Name: scheduled-test-tests +Namespace: testkube +Labels: testkube=tests +Annotations: +Schedule: */1 * * * * +Concurrency Policy: Forbid +Suspend: False +Successful Job History Limit: 3 +Failed Job History Limit: 1 +Starting Deadline Seconds: +Selector: +Parallelism: +Completions: +Pod Template: + Labels: + Containers: + curlimage: + Image: curlimages/curl + Port: + Host Port: + Command: + sh + -c + Args: + curl -X POST -H "Content-Type: application/json" -d '{}' "http://testkube-api-server:8088/v1/tests/scheduled-test/executions?callback=true" + Environment: + Mounts: + Volumes: +Last Schedule Time: Wed, 13 Apr 2022 15:50:00 +0300 +Active Jobs: scheduled-test-tests-27497570 +Events: + Type Reason Age From Message + ---- ------ ---- ---- ------- + Normal SuccessfulCreate 5m41s cronjob-controller Created job scheduled-test-tests-2749757 +``` + +As mentioned above, we have a scheduled callback for launching our test. + +## Getting Scheduled Test Results + +Now we can check if the test is executed every minute per the schedule we provided. + +```sh +kubectl testkube get execution +``` + +```sh title="Expected output:" + ID | NAME | TYPE | STATUS | LABELS ++--------------------------+---------------------+--------------------+---------+--------+ + 6256c98f418062706814e1fc | scheduled-test | postman/collection | passed | + 6256c953418062706814e1fa | scheduled-test | postman/collection | passed | + 6256c91e418062706814e1f8 | scheduled-test | postman/collection | passed | + 6256c8db418062706814e1f6 | scheduled-test | postman/collection | passed | + 6256c89f418062706814e1f4 | scheduled-test | postman/collection | passed | + 6256c885418062706814e1f2 | scheduled-test | postman/collection | passed | + 6256c87e418062706814e1f0 | scheduled-test | postman/collection | passed | +``` + +The test is successfully executed on the stated schedule. + +## Create a Test Suite with a Schedule + +Let's create a Test Suite with a required schedule using the Testkube CLI command: + +```sh +cat test/suites/testsuite.json | kubectl testkube create testsuite --name scheduled-testsuite --schedule="*/1 * * * *" +``` + +```sh title="Expected output:" +TestSuite created scheduled-testsuite 🥇 +``` + +We successfully created a scheduled Test Suite and can view a list of the available Test Suites: + +```sh +kubectl testkube get testsuites +``` + +```sh title="Expected output:" + NAME | DESCRIPTION | STEPS | LABELS | SCHEDULE | STATUS | EXECUTION ID ++---------------------+------------------------+-------+--------+-------------+--------+--------------+ + scheduled-testsuite | Run test several times | 2 | | */1 * * * * | | +``` + +The scheduled Test Suite was created and successfully scheduled for execution. +We will skip the Cron job details, they are fully similar to test one described above. + +## Getting Scheduled Test Suite Results + +The Test Suite is executed every minute per the schedule we provided. + +```sh +kubectl testkube get tse +``` + +```sh title="Expected output:" + ID | TEST SUITE NAME | EXECUTION NAME | STATUS | STEPS | LABELS ++--------------------------+---------------------+--------------------------------------------+--------+-------+--------+ + 6256ce3f418062706814e210 | scheduled-testsuite | scheduled-testsuite.abnormally-in-lark | passed | 2 | + 6256ce04418062706814e20c | scheduled-testsuite | scheduled-testsuite.kindly-evolved-primate | passed | 2 | + 6256cdcc418062706814e208 | scheduled-testsuite | scheduled-testsuite.formerly-champion-dodo | passed | 2 | +``` + +The Test Suite is successfully executed on the stated schedule. \ No newline at end of file diff --git a/docs/docs/articles/test-triggers.mdx b/docs/docs/articles/test-triggers.mdx new file mode 100644 index 00000000000..2d08d7184c6 --- /dev/null +++ b/docs/docs/articles/test-triggers.mdx @@ -0,0 +1,190 @@ +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; + +# Triggers + + + + +Testkube allows you to automate running tests and test suites by defining triggers on certain events for various Kubernetes resources. + +## What is a Testkube Test Trigger? + +In generic terms, a _Trigger_ defines an _action_ which will be executed for a given _execution_ when a certain _event_ on a specific _resource_ occurs. For example, we could define a _TestTrigger_ which _runs_ a _Test_ when a _ConfigMap_ gets _modified_. + +Watch our [video guide](#video-tutorial) on using Testkube Test Triggers to perform **Asynchronous Testing in Kubernetes**. + +## Creating Test Triggers in the Testkube Dashboard + +Click on the lightening bolt icon on the left of the Testkube IDE to open the dialog to create test triggers. Any current test triggers will be listed and the `Create a new trigger` button is at the top right of the screen. + +![Trigger Screen](../img/trigger-screen.png) + +The `Create new trigger` dialog opens: + +![Create Trigger](../img/create-trigger.png) + +Input the condition that will cause the trigger and click `Next`. + +Input the action that will be the result of the trigger condition happening and click `Create`. + +![Create Trigger Action](../img/create-trigger-action.png) + +Get all the details in the video below! + +## Video Tutorial + + + + + + + + + +Testkube allows you to automate running tests and test suites by defining triggers on certain events for various Kubernetes resources. + +## What is a Testkube Test Trigger? + +In generic terms, a _Trigger_ defines an _action_ which will be executed for a given _execution_ when a certain _event_ on a specific _resource_ occurs. For example, we could define a _TestTrigger_ which _runs_ a _Test_ when a _ConfigMap_ gets _modified_. + +Watch our [video guide](#video-tutorial) on using Testkube Test Triggers to perform **Asynchronous Testing in Kubernetes**. + +## Custom Resource Definition Model +### Selectors + +The `resourceSelector` and `testSelector` fields support selecting resources either by name or using +the Kubernetes [Label Selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements). + +Each selector should specify the `namespace` of the object, otherwise the namespace defaults to `testkube`. + +``` +selector := resourceSelector | testSelector +``` + +#### Name Selector + +Name selectors are used when we want to select a specific resource in a specific namespace. + +```yaml +selector: + name: Kubernetes object name + namespace: Kubernetes object namespace (default is **testkube**) +``` + +#### Label Selector + +Label selectors are used when we want to select a group of resources in a specific namespace. + +```yaml +selector: + namespace: Kubernetes object namespace (default is **testkube**) + labelSelector: + matchLabels: map of key-value pairs + matchExpressions: + - key: label name + operator: [In | NotIn | Exists | DoesNotExist + values: list of values +``` + +### Resource Conditions + +Resource Conditions allows triggers to be defined based on the status conditions for a specific resource. + +```yaml +conditionSpec: + timeout: duration in seconds the test trigger waits for conditions, until its stopped + delay: duration in seconds the test trigger waits between condition checks + conditions: + - type: test trigger condition type + status: test trigger condition status, supported values - True, False, Unknown + reason: test trigger condition reason + ttl: test trigger condition ttl +``` + +### Resource Probes + +Resource Probes allows triggers to be defined based on the probe status. + +```yaml +probeSpec: + timeout: duration in seconds the test trigger waits for probes, until its stopped + delay: duration in seconds the test trigger waits between probes + probes: + - scheme: test trigger condition probe scheme to connect to host, default is http + host: test trigger condition probe host, default is pod ip or service name + path: test trigger condition probe path to check, default is / + port: test trigger condition probe port to connect + headers: test trigger condition probe headers to submit +``` + +### Supported Values +* **Resource** - pod, deployment, statefulset, daemonset, service, ingress, event, configmap +* **Action** - run +* **Event** - created, modified, deleted +* **Execution** - test, testsuite + +**NOTE**: All resources support the above-mentioned events, a list of finer-grained events is in the works, stay tuned... + +## Example + +Here is an example for a **Test Trigger** *default/testtrigger-example* which runs the **TestSuite** *frontend/sanity-test* +when a **deployment** containing the label **testkube.io/tier: backend** gets **modified** and also has the conditions **Progressing: True: NewReplicaSetAvailable** and **Available: True**. + +```yaml +apiVersion: tests.testkube.io/v1 +kind: TestTrigger +metadata: + name: testtrigger-example + namespace: default +spec: + resource: deployment + resourceSelector: + labelSelector: + matchLabels: + testkube.io/tier: backend + event: modified + conditionSpec: + timeout: 100 + delay: 2 + conditions: + - type: Progressing + status: "True" + reason: "NewReplicaSetAvailable" + ttl: 60 + - type: Available + status: "True" + probeSpec: + timeout: 50 + delay: 1 + probes: + - scheme: http + host: testkube-api-server + path: /health + port: 8088 + headers: + X-Token: "12345" + - host: testkube-dashboard + port: 8080 + action: run + execution: testsuite + testSelector: + name: sanity-test + namespace: frontend +``` + +## Architecture + +Testkube uses [Informers](https://pkg.go.dev/k8s.io/client-go/informers) to watch Kubernetes resources and register handlers +on certain actions on the watched Kubernetes resources. + +Informers are a reliable, scalable and fault-tolerant Kubernetes concept where each informer registers handlers with the +Kubernetes API and gets notified by Kubernetes on each event on the watched resources. + +## API + +Testkube exposes CRUD operations on test triggers in the REST API. Check out the [Open API](../openapi.md) docs for more info. + +## Video Tutorial + +