Skip to content

Commit

Permalink
Add Dashboard and CLI Tabs
Browse files Browse the repository at this point in the history 
To scheduling tests and test triggers pages.
  • Loading branch information
@jfermi
jfermi committed Aug 25, 2023
1 parent 4b482cf commit 6b17f75
Show file tree
Hide file tree
Showing 2 changed files with 386 additions and 0 deletions.
196 changes: 196 additions & 0 deletions docs/docs/articles/scheduling-tests.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";

# Scheduling Tests

<Tabs>
<TabItem value="dashboard" label="Dashboard" default>

## Create a Scheduled Test or Test Suite from the Testkube Dashboard

In order to run Tests and Test Suites on a regular basis, we support a scheduling mechanism for these objects.
CRDs both for Tests and Test Suites contain a **schedule** field used to define rules for launching them in time.
Testkube's schedule data format is the same that is used to define Kubernetes Cron jobs (check Wikipedia Cron format for details <https://en.wikipedia.org/wiki/Cron>).

## Scheduling Architecture

Testkube uses the scheduling engine from Kubernetes Cron jobs.
In fact, for each scheduled Test or Test Suite, a special cron job is created from this [template](ttps://github.com/kubeshop/helm-charts/tree/main/charts/testkube-operator/cronjob-template.yml).

Technically, it is a callback to the Testkube API server method, launching either Test or Test Suite execution.

This works similarly to scheduled Test and Test Suite executions done by external scheduling platforms.

In the Testkube Dashboard, when you have selected a Test or Test Suite, in the **Settings** tab, click on **Scheduling** to create a schedule for the Test or Test Suite.

![Schedule Tests](../img/schedule-tests.png)

![Schedule Tests Test Suites](../img/schedule-test-suites.png)

</TabItem>

<TabItem value="cli" label="CLI">

In order to run Tests and Test Suites on a regular basis, we support a scheduling mechanism for these objects.
CRDs both for Tests and Test Suites contain a **schedule** field used to define rules for launching them in time.
Testkube's schedule data format is the same that is used to define Kubernetes Cron jobs (check [Wikipedia Cron format](https://en.wikipedia.org/wiki/Cron) for details.

## Scheduling Architecture

Testkube uses the scheduling engine from Kubernetes Cron jobs.
In fact, for each scheduled Test or Test Suite, a special cron job is created from this [template](https://github.com/kubeshop/helm-charts/tree/main/charts/testkube-operator/cronjob-template.yml).

Technically, it is a callback to the Testkube API server method, launching either Test or Test Suite execution.
This works similarly to scheduled Test and Test Suite executions done by external scheduling platforms.

## Create a Test with a Schedule

Let's create a test with a required schedule using Testkube CLI command:

```sh
kubectl testkube create test --file test/postman/TODO.postman_collection.json --name scheduled-test --schedule="*/1 * * * *"
```

```sh title="Expected output:"
Detected test type postman/collection
Test created / scheduled-test 🥇
```

We successfully created a scheduled test and can check the list of the available tests:

```sh
kubectl testkube get tests
```

```sh title="Expected output:"
NAME | TYPE | CREATED | LABELS | SCHEDULE | STATUS | EXECUTION ID
+-------------------+--------------------+-------------------------------+--------+-------------+--------+--------------------------+
scheduled-test | postman/collection | 2022-04-13 12:37:40 +0000 UTC | | */1 * * * * | |
```

The scheduled test was created and successfully scheduled for execution.

Let's check a Cron job connected to this test:

```sh
kubectl get cronjobs -A
```

```sh title="Expected output:"
NAMESPACE NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
testkube scheduled-test-tests */1 * * * * False 1 42s 3m22s
```

The Cron job for this test was successfully created and the test was executed.

To the check Cron job details:

```sh
kubectl describe cronjob scheduled-test-tests -n testkube
```

```sh title="Expected output:"
Name: scheduled-test-tests
Namespace: testkube
Labels: testkube=tests
Annotations: <none>
Schedule: */1 * * * *
Concurrency Policy: Forbid
Suspend: False
Successful Job History Limit: 3
Failed Job History Limit: 1
Starting Deadline Seconds: <unset>
Selector: <unset>
Parallelism: <unset>
Completions: <unset>
Pod Template:
Labels: <none>
Containers:
curlimage:
Image: curlimages/curl
Port: <none>
Host Port: <none>
Command:
sh
-c
Args:
curl -X POST -H "Content-Type: application/json" -d '{}' "http://testkube-api-server:8088/v1/tests/scheduled-test/executions?callback=true"
Environment: <none>
Mounts: <none>
Volumes: <none>
Last Schedule Time: Wed, 13 Apr 2022 15:50:00 +0300
Active Jobs: scheduled-test-tests-27497570
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 5m41s cronjob-controller Created job scheduled-test-tests-2749757
```

As mentioned above, we have a scheduled callback for launching our test.

## Getting Scheduled Test Results

Now we can check if the test is executed every minute per the schedule we provided.

```sh
kubectl testkube get execution
```

```sh title="Expected output:"
ID | NAME | TYPE | STATUS | LABELS
+--------------------------+---------------------+--------------------+---------+--------+
6256c98f418062706814e1fc | scheduled-test | postman/collection | passed |
6256c953418062706814e1fa | scheduled-test | postman/collection | passed |
6256c91e418062706814e1f8 | scheduled-test | postman/collection | passed |
6256c8db418062706814e1f6 | scheduled-test | postman/collection | passed |
6256c89f418062706814e1f4 | scheduled-test | postman/collection | passed |
6256c885418062706814e1f2 | scheduled-test | postman/collection | passed |
6256c87e418062706814e1f0 | scheduled-test | postman/collection | passed |
```

The test is successfully executed on the stated schedule.

## Create a Test Suite with a Schedule

Let's create a Test Suite with a required schedule using the Testkube CLI command:

```sh
cat test/suites/testsuite.json | kubectl testkube create testsuite --name scheduled-testsuite --schedule="*/1 * * * *"
```

```sh title="Expected output:"
TestSuite created scheduled-testsuite 🥇
```

We successfully created a scheduled Test Suite and can view a list of the available Test Suites:

```sh
kubectl testkube get testsuites
```

```sh title="Expected output:"
NAME | DESCRIPTION | STEPS | LABELS | SCHEDULE | STATUS | EXECUTION ID
+---------------------+------------------------+-------+--------+-------------+--------+--------------+
scheduled-testsuite | Run test several times | 2 | | */1 * * * * | |
```

The scheduled Test Suite was created and successfully scheduled for execution.
We will skip the Cron job details, they are fully similar to test one described above.

## Getting Scheduled Test Suite Results

The Test Suite is executed every minute per the schedule we provided.

```sh
kubectl testkube get tse
```

```sh title="Expected output:"
ID | TEST SUITE NAME | EXECUTION NAME | STATUS | STEPS | LABELS
+--------------------------+---------------------+--------------------------------------------+--------+-------+--------+
6256ce3f418062706814e210 | scheduled-testsuite | scheduled-testsuite.abnormally-in-lark | passed | 2 |
6256ce04418062706814e20c | scheduled-testsuite | scheduled-testsuite.kindly-evolved-primate | passed | 2 |
6256cdcc418062706814e208 | scheduled-testsuite | scheduled-testsuite.formerly-champion-dodo | passed | 2 |
```

The Test Suite is successfully executed on the stated schedule.
190 changes: 190 additions & 0 deletions docs/docs/articles/test-triggers.mdx
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";

# Triggers

<Tabs>
<TabItem value="dashboard" label="Dashboard" default>

Testkube allows you to automate running tests and test suites by defining triggers on certain events for various Kubernetes resources.

## What is a Testkube Test Trigger?

In generic terms, a _Trigger_ defines an _action_ which will be executed for a given _execution_ when a certain _event_ on a specific _resource_ occurs. For example, we could define a _TestTrigger_ which _runs_ a _Test_ when a _ConfigMap_ gets _modified_.

Watch our [video guide](#video-tutorial) on using Testkube Test Triggers to perform **Asynchronous Testing in Kubernetes**.

## Creating Test Triggers in the Testkube Dashboard

Click on the lightening bolt icon on the left of the Testkube IDE to open the dialog to create test triggers. Any current test triggers will be listed and the `Create a new trigger` button is at the top right of the screen.

![Trigger Screen](../img/trigger-screen.png)

The `Create new trigger` dialog opens:

![Create Trigger](../img/create-trigger.png)

Input the condition that will cause the trigger and click `Next`.

Input the action that will be the result of the trigger condition happening and click `Create`.

![Create Trigger Action](../img/create-trigger-action.png)

Get all the details in the video below!

## Video Tutorial

<iframe width="100%" height="350px" src="https://www.youtube.com/embed/t4V6E9rQ5W4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>



</TabItem>

<TabItem value="cli" label="CLI">

Testkube allows you to automate running tests and test suites by defining triggers on certain events for various Kubernetes resources.

## What is a Testkube Test Trigger?

In generic terms, a _Trigger_ defines an _action_ which will be executed for a given _execution_ when a certain _event_ on a specific _resource_ occurs. For example, we could define a _TestTrigger_ which _runs_ a _Test_ when a _ConfigMap_ gets _modified_.

Watch our [video guide](#video-tutorial) on using Testkube Test Triggers to perform **Asynchronous Testing in Kubernetes**.

## Custom Resource Definition Model
### Selectors

The `resourceSelector` and `testSelector` fields support selecting resources either by name or using
the Kubernetes [Label Selector](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/#resources-that-support-set-based-requirements).

Each selector should specify the `namespace` of the object, otherwise the namespace defaults to `testkube`.

```
selector := resourceSelector | testSelector
```

#### Name Selector

Name selectors are used when we want to select a specific resource in a specific namespace.

```yaml
selector:
name: Kubernetes object name
namespace: Kubernetes object namespace (default is **testkube**)
```
#### Label Selector
Label selectors are used when we want to select a group of resources in a specific namespace.
```yaml
selector:
namespace: Kubernetes object namespace (default is **testkube**)
labelSelector:
matchLabels: map of key-value pairs
matchExpressions:
- key: label name
operator: [In | NotIn | Exists | DoesNotExist
values: list of values
```
### Resource Conditions
Resource Conditions allows triggers to be defined based on the status conditions for a specific resource.
```yaml
conditionSpec:
timeout: duration in seconds the test trigger waits for conditions, until its stopped
delay: duration in seconds the test trigger waits between condition checks
conditions:
- type: test trigger condition type
status: test trigger condition status, supported values - True, False, Unknown
reason: test trigger condition reason
ttl: test trigger condition ttl
```
### Resource Probes
Resource Probes allows triggers to be defined based on the probe status.
```yaml
probeSpec:
timeout: duration in seconds the test trigger waits for probes, until its stopped
delay: duration in seconds the test trigger waits between probes
probes:
- scheme: test trigger condition probe scheme to connect to host, default is http
host: test trigger condition probe host, default is pod ip or service name
path: test trigger condition probe path to check, default is /
port: test trigger condition probe port to connect
headers: test trigger condition probe headers to submit
```
### Supported Values
* **Resource** - pod, deployment, statefulset, daemonset, service, ingress, event, configmap
* **Action** - run
* **Event** - created, modified, deleted
* **Execution** - test, testsuite
**NOTE**: All resources support the above-mentioned events, a list of finer-grained events is in the works, stay tuned...
## Example
Here is an example for a **Test Trigger** *default/testtrigger-example* which runs the **TestSuite** *frontend/sanity-test*
when a **deployment** containing the label **testkube.io/tier: backend** gets **modified** and also has the conditions **Progressing: True: NewReplicaSetAvailable** and **Available: True**.
```yaml
apiVersion: tests.testkube.io/v1
kind: TestTrigger
metadata:
name: testtrigger-example
namespace: default
spec:
resource: deployment
resourceSelector:
labelSelector:
matchLabels:
testkube.io/tier: backend
event: modified
conditionSpec:
timeout: 100
delay: 2
conditions:
- type: Progressing
status: "True"
reason: "NewReplicaSetAvailable"
ttl: 60
- type: Available
status: "True"
probeSpec:
timeout: 50
delay: 1
probes:
- scheme: http
host: testkube-api-server
path: /health
port: 8088
headers:
X-Token: "12345"
- host: testkube-dashboard
port: 8080
action: run
execution: testsuite
testSelector:
name: sanity-test
namespace: frontend
```
## Architecture
Testkube uses [Informers](https://pkg.go.dev/k8s.io/client-go/informers) to watch Kubernetes resources and register handlers
on certain actions on the watched Kubernetes resources.
Informers are a reliable, scalable and fault-tolerant Kubernetes concept where each informer registers handlers with the
Kubernetes API and gets notified by Kubernetes on each event on the watched resources.
## API
Testkube exposes CRUD operations on test triggers in the REST API. Check out the [Open API](../openapi.md) docs for more info.
## Video Tutorial
<iframe width="100%" height="350px" src="https://www.youtube.com/embed/t4V6E9rQ5W4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen></iframe>

0 comments on commit 6b17f75

Please sign in to comment.