k6/browser does not support true client performance testing

[jsaujla]

Feature Description

k6/browser does not provide feature to capture individual transaction response time.

Below are some examples of true client (browser based) performance tests:

1. The time taken to launch a website: I should be able to start a timer before launching the website then stop the timer when I would verify an element on the page visible/displayed.
2. The time taken to redirect from one page to another page: I should be able to start a timer before clicking on a link then stop the timer when I would verify an element on the next page visible/displayed.
3. Time taken to login: I should be able to start a timer before clicking on login button then stop the timer when I would verify an element on the next page (user dashboard page) visible/displayed.

Suggested Solution (optional)

Below is an example of JMeter WebDriver Plugin script (I executed the script with loop count 10):

//##### Config | Test Data #####
var maxWaitTime = 30;
var k6TestBaseUrl = "https://test.k6.io/";
var loginUser = "test_user";
var loginPassword = "1234";

//##### Selenium WebDriver Imports #####
var selenium = JavaImporter(org.openqa.selenium, org.openqa.selenium.support.ui);
var wait = new selenium.WebDriverWait(WDS.browser, java.time.Duration.ofSeconds(maxWaitTime));
var select = JavaImporter(org.openqa.selenium.support.ui.Select);
var actions = new org.openqa.selenium.interactions.Actions(WDS.browser);


//##### JMeter/WebDriver Script #####

WDS.sampleResult.sampleStart();
WDS.browser.manage().deleteAllCookies();
WDS.log.info("Sample Start - True_Client_Example_Scenario");

try {	
     // Launch test.k6.io
     WDS.sampleResult.subSampleStart('Launch_' + k6TestBaseUrl);
     WDS.browser.get(k6TestBaseUrl);
     wait.until(selenium.ExpectedConditions.elementToBeClickable(selenium.By.xpath("//a[@href='/my_messages.php']"))).click();     
     WDS.sampleResult.subSampleEnd(true);
	
     // Log in
     wait.until(selenium.ExpectedConditions.visibilityOfElementLocated(selenium.By.xpath("//input[@name='login']"))).sendKeys(loginUser);
     wait.until(selenium.ExpectedConditions.visibilityOfElementLocated(selenium.By.xpath("//input[@name='password']"))).sendKeys(loginPassword);

     WDS.sampleResult.subSampleStart('Testk6_Login');
     wait.until(selenium.ExpectedConditions.elementToBeClickable(selenium.By.xpath("//input[@value='Go!']"))).click();
     wait.until(selenium.ExpectedConditions.visibilityOfElementLocated(selenium.By.xpath("//h2[text()='Welcome, test_user!']")));  
     WDS.sampleResult.subSampleEnd(true);

     // Logout
     WDS.sampleResult.subSampleStart('Testk6_Logout');
     wait.until(selenium.ExpectedConditions.elementToBeClickable(selenium.By.xpath("//input[@value='Logout']"))).click();
     wait.until(selenium.ExpectedConditions.visibilityOfElementLocated(selenium.By.xpath("//input[@value='Go!']"))); 
     WDS.sampleResult.subSampleEnd(true);
}
catch (error) {
	WDS.log.error(error);
	WDS.sampleResult.subSampleEnd(false);
	throw error;
}
finally {
	WDS.sampleResult.sampleEnd();
}

Below is an overview of html output report:

Already existing or connected issues / PRs (optional)

No response

[oleiade]

For your eyes @inancgumus @ankur22 👀

[ankur22]

Hi @jsaujla,

Thanks for opening an issue.

please correct me if i've missed anything, but it sounds like you would like to time the individual navigations within a single test iteration, correct?

Would something like this suffice when working with custom metrics?:

import { browser } from 'k6/x/browser/async';
import { Trend } from 'k6/metrics';

export const options = {
  scenarios: {
    ui: {
      executor: 'shared-iterations',
      options: {
        browser: {
            type: 'chromium',
        },
      },
    },
  },
}

const launchTrend = new Trend('Launch');
const loginTrend = new Trend('Testk6_Login');
const logoutTrend = new Trend('Testk6_Logout');

export default async function() {
  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    var start = Date.now();
    await page.goto('https://test.k6.io/', { waitUntil: 'networkidle' });
    await Promise.all([
      page.waitForNavigation(),
      page.locator('a[href="/my_messages.php"]').click(),
    ]);
    var end = Date.now();
    launchTrend.add(end - start);

    await page.locator('input[name="login"]').type('admin');
    await page.locator('input[name="password"]').type("123");

    start = Date.now();
    await Promise.all([
      page.waitForNavigation(),
      page.locator('input[type="submit"]').click(),
    ]);
    var end = Date.now();
    loginTrend.add(end - start);

    start = Date.now();
    await Promise.all([
      page.waitForNavigation(),
      page.locator('input[type="submit"]').click(),
    ]);
    var end = Date.now();
    logoutTrend.add(end - start);
  } finally {
    await page.close();
  }
}

When running it locally, the summary will show the custom metrics like so:

     Launch......................: avg=2375     min=2375    med=2375     max=2375     p(90)=2375     p(95)=2375    
     Testk6_Login................: avg=479      min=479     med=479      max=479      p(90)=479      p(95)=479     
     Testk6_Logout...............: avg=448      min=448     med=448      max=448      p(90)=448      p(95)=448

[jsaujla]

Thanks for the advise. Yes, I am able to implement true client performance testing checks.

import { browser } from 'k6/browser';
import { check, sleep } from 'k6';
import { Trend } from 'k6/metrics';
// @ts-ignore
import { htmlReport } from "https://raw.githubusercontent.com/benc-uk/k6-reporter/main/dist/bundle.js";

export const options = {
  scenarios: {
    sequential: {
      executor: 'constant-vus',
      vus: 1,
      duration: '30s',
      gracefulStop: '15s',
      options: {
        browser: {
          type: 'chromium',
        },
      },
    },
  },
};

const expectedResponseTime = 3000;

const TREND_NAMES = {
  launch: 'Testk6_Launch',
  loginForm: 'Testk6_LoginForm',
  login: 'Testk6_Login',
  logout: 'Testk6_Logout',
};

const launchTrend = new Trend(TREND_NAMES.launch);
const loginFormTrend = new Trend(TREND_NAMES.loginForm);
const loginTrend = new Trend(TREND_NAMES.login);
const logoutTrend = new Trend(TREND_NAMES.logout);

export default async function () {

  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    let start, end;

    start = Date.now();
    await page.goto('https://test.k6.io/');
    await page.waitForSelector('//h2[text()="Public pages"]', { state: 'visible' });
    end = Date.now();
    launchTrend.add(end - start);
    check(page, {
        [`Duration Less Than ${expectedResponseTime} ms | ${TREND_NAMES.launch}`]: () => end - start < expectedResponseTime,
    });

    start = Date.now();
    await page.locator('a[href="/my_messages.php"]').click(),
    await page.waitForSelector('//input[@name="login"]', { state: 'visible' });
    end = Date.now();
    loginFormTrend.add(end - start);
    check(page, {
        [`Duration Less Than ${expectedResponseTime} ms | ${TREND_NAMES.loginForm}`]: () => end - start < expectedResponseTime,
    });

    await page.locator('input[name="login"]').type('test_user');
    await page.locator('input[name="password"]').type("1234");
    start = Date.now();
    await page.locator('input[value="Go!"]').click();
    await page.waitForSelector('//h2[text()="Welcome, test_user!"]', { state: 'visible' });
    end = Date.now();
    loginTrend.add(end - start);
    check(page, {
        [`Duration Less Than ${expectedResponseTime} ms | ${TREND_NAMES.login}`]: () => end - start < expectedResponseTime,
    });

    start = Date.now();
    await page.locator('input[value="Logout"]').click();
    await page.waitForSelector('input[value="Go!"]', { state: 'visible' });
    end = Date.now();
    logoutTrend.add(end - start);
    check(page, {
        [`Duration Less Than ${expectedResponseTime} ms | ${TREND_NAMES.logout}`]: () => end - start < expectedResponseTime,
    });
  } 
  catch (error) {
    await page.screenshot({ path: 'screenshots/test-k6-true-client-example_failure.png' });
    console.error('***** An error occurred:', error.message);
    console.error('***** Error Stack trace:', error.stack);
    throw error;
  } 
  finally {
    await page.close();
  }
}

export function handleSummary(data) {
  return {
    'summary-test-k6.html': htmlReport(data, {debugger: true}),
  };
}

Below is an output I wanted to see. It's good:

Would you please advise how to remove (or separate) other Custom Metrics (browser_data_received, browser_web_vital_ttfb, etc.) from the 4 I added? I don't think these are relevant in terms of individual navigation checks.

[ankur22]

Hi @jsaujla,

Happy to hear that the solution worked for you.

Would you please advise how to remove (or separate) other Custom Metrics (browser_data_received, browser_web_vital_ttfb, etc.) from the 4 I added? I don't think these are relevant in terms of individual navigation checks.

There isn't a way to not emit the build in metrics. In the tool that you are working with to store and display the results, is it not possible to ignore the metrics that you are not interested in?