Introduce k6/testing package for assertions and expectations

[oleiade]

Background

k6 aims to be a versatile tool for both load and functional testing. However, it currently lacks intuitive and robust assertion capabilities needed for effective functional testing. Existing constructs like check(), fail(), and k6/execution.abort() have limitations:

• check(): Records pass/fail metrics but does not affect the test’s pass/fail status or exit code unless thresholds are explicitly set. It also lacks native support for asynchronous code and provides limited context on failures.
• fail(): Aborts the current iteration but does not mark the test as failed or exit with a non-zero exit code.
• k6/execution.abort(): Immediately halts the test run but does not provide detailed failure context.

These limitations make it challenging to perform functional testing where immediate feedback and detailed failure information are crucial.

Objectives

• Enable Assertions That Fail Tests: Allow users to write assertions that can mark a test as failed, optionally aborting it immediately, and cause k6 to exit with a non-zero exit code.
• Provide Detailed Failure Context: Offer clear, human-readable error messages with details about what was expected, what was received, and where in the code the failure occurred.
• Support Asynchronous Assertions: Allow assertions over asynchronous code, particularly important for browser testing and scenarios involving dynamic content.
• Introduce Playwright-Compatible API: Facilitate a smooth transition for users familiar with Playwright by providing a compatible assertion library, especially beneficial for k6’s browser module.
• Maintain Backward Compatibility: Avoid breaking existing scripts and workflows by introducing new capabilities in a way that does not disrupt current functionalities.

Suggested Solution (optional)

Note: following solution is dependent on #4062, and may also depend on #4065 depending on how much we rely on checks internally for soft expectations.

Introduce k6/testing package

Proof of concept

Add a new k6/testing package written in JavaScript/TypeScript, replicating the Playwright assertions API, to provide robust and intuitive assertion capabilities.

Expect API

• Function: expect(value, [message])
• Offers expressive expectations using matchers (e.g., toBe(), toContain(), toBeGreaterThan()).
• Supports both hard expectations (fail test immediately) and soft expectations (mark test as failed but continue execution).
• Supports both synchronous, non-retrying expectations, as well as asynchronous, retrying expectations.

Non-retrying expectations

Example

import { expect } from 'k6/testing`

export default function() {
  const otherValue = getOtherValue();
  expect.soft(otherValue).toContain('hello'); // Soft expectation
  
  const value = getValue();
  expect(value).toBeGreaterThan(10); // Hard expectation
}

Target (for illustration) output

1) Soft Expectation unmet.

    Expected: otherValue to contain 'hello'.
    Received: 'world'
        File: script.js
        Line: 15

32   |   // some statement
33 >|   expect(otherValue).toContain('hello');


2) Expectation unmet.

    Expected: value to be greater than 10.
    Received: 8
        File: script.js
        Line: 15

32   |   // some statement
33 >|   expect(value).toBeGreatherThan(10);

Error code 108

API

Here is the target proposed API, aligned with playwright. We might want to implement only a subset of matchers, depending on k6's specific needs.

Expectation	Description
expect(value: unknown).toBe()	Value is the same
expect(value: unknown).toBeCloseTo()	Number is approximately equal
expect(value: unknown).toBeDefined()	Value is not undefined
expect(value: unknown).toBeFalsy()	Value is falsy, e.g., false, 0, null, etc.
expect(value: unknown).toBeGreaterThan()	Number is more than
expect(value: unknown).toBeGreaterThanOrEqual()	Number is more than or equal
expect(value: unknown).toBeInstanceOf()	Object is an instance of a class
expect(value: unknown).toBeLessThan()	Number is less than
expect(value: unknown).toBeLessThanOrEqual()	Number is less than or equal
expect(value: unknown).toBeNaN()	Value is NaN
expect(value: unknown).toBeNull()	Value is null
expect(value: unknown).toBeTruthy()	Value is truthy, i.e., not false, 0, null, etc.
expect(value: unknown).toBeUndefined()	Value is undefined
expect(value: unknown).toContain()	String contains a substring
expect(value: unknown).toContain()	Array or set contains an element
expect(value: unknown).toContainEqual()	Array or set contains a similar element
expect(value: unknown).toEqual()	Value is similar—deep equality and pattern matching
expect(value: unknown).toHaveLength()	Array or string has length
expect(value: unknown).toHaveProperty()	Object has a property
expect(value: unknown).toMatch()	String matches a regular expression
expect(value: unknown).toMatchObject()	Object contains specified properties
expect(value: unknown).toStrictEqual()	Value is similar, including property types
expect(value: unknown).toThrow()	Function throws an error
expect(value: unknown).any()	Matches any instance of a class/primitive
expect(value: unknown).anything()	Matches anything
expect(value: unknown).arrayContaining()	Array contains specific elements
expect(value: unknown).closeTo()	Number is approximately equal
expect(value: unknown).objectContaining()	Object contains specific properties
expect(value: unknown).stringContaining()	String contains a substring
expect(value: unknown).stringMatching()	String matches a regular expression

Retrying (Async) Expectations

Designed for scenarios where the condition may not be immediately met (e.g., waiting for a DOM element). Use await with retry logic until the condition is met or a timeout is reached.

Example

import { browser } from 'k6/browser';
import { expect } from 'k6/testing';

export default async function () {
  const page = browser.newPage();
  await page.goto('https://example.com');

  const locator = page.locator('#submit-button');
  await expect(locator).toBeVisible(); // Retrying expectation
}

Target (for illustration) output

Soft Expectation unmet:

            Locator: locator('input[name="login"]')
    Expected string: "foo"
    Received string: "test"
        retry count: 9
    timed out after: 5secs

    32  |        // We're expecting this to fail as we have typed 'test' into the input
   33 >|        await expect.soft(page.locator('input[name="login"]')).toHaveValue("foo");

API

Expectation	Description
await expect(locator: browser.Locator).toBeAttached()	Element is attached
await expect(locator: browser.Locator).toBeChecked()	Checkbox is checked
await expect(locator: browser.Locator).toBeDisabled()	Element is disabled
await expect(locator: browser.Locator).toBeEditable()	Element is editable
await expect(locator: browser.Locator).toBeEmpty()	Container is empty
await expect(locator: browser.Locator).toBeEnabled()	Element is enabled
await expect(locator: browser.Locator).toBeFocused()	Element is focused
await expect(locator: browser.Locator).toBeHidden()	Element is not visible
await expect(locator: browser.Locator).toBeInViewport()	Element intersects viewport
await expect(locator: browser.Locator).toBeVisible()	Element is visible
await expect(locator: browser.Locator).toContainText()	Element contains text
await expect(locator: browser.Locator).toHaveAccessibleDescription()	Element has a matching accessible description
await expect(locator: browser.Locator).toHaveAccessibleName()	Element has a matching accessible name
await expect(locator: browser.Locator).toHaveAttribute()	Element has a DOM attribute
await expect(locator: browser.Locator).toHaveClass()	Element has a class property
await expect(locator: browser.Locator).toHaveCount()	List has exact number of children
await expect(locator: browser.Locator).toHaveCSS()	Element has CSS property
await expect(locator: browser.Locator).toHaveId()	Element has an ID
await expect(locator: browser.Locator).toHaveJSProperty()	Element has a JavaScript property
await expect(locator: browser.Locator).toHaveRole()	Element has a specific ARIA role
await expect(locator: browser.Locator).toHaveScreenshot()	Element has a screenshot
await expect(locator: browser.Locator).toHaveText()	Element matches text
await expect(locator: browser.Locator).toHaveValue()	Input has a value
await expect(locator: browser.Locator).toHaveValues()	Select has options selected

Negation

Expectations, regardless of their retrying fashion can be negated using the .not helper:

expect(value).not.toEqual(0);
await expect(locator).not.toContainText('some text');

Soft assertions

By default, failed expectations will terminate test execution with a non-zero exit code. Our testing package also support soft assertions: failed soft assertions do not terminate test execution, but mark the test as failed. An expectation can be made soft using the .soft() helper:

// Test continues even if assertions fail
expect.soft(response.status).toBe(200);
expect.soft(data.items).toHaveLength(5);

Configuration options

• Use expect.configure(options) to create a customized expect instance.
• Options include timeout settings, default expectation type (hard or soft), and output formatting.

Example

const customExpect = expect.configure({ timeout: '30s', soft: true });
await customExpect(locator).toHaveText('Submit');

PS: internal stakeholders expressed the desire to be able to also configure the behavior of the expect function from outside the script, to apply it in a systematic way.

Ensure first-class integration

• The k6/testing package should be an official part of k6, ensuring it feels like a first-class citizen. Regardless of where and how it is implemented, its import path should feel "native", as opposed to a jslib library which we don't judge desirable.
• Users import it using the familiar k6/ prefix, maintaining consistency with other k6 modules.

import { expect } from 'k6/testing';

Already existing or connected issues / PRs (optional)

Direct

#4062
#4065

Indirect

#3406