Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thresholds using URL may not correlate in browser based tests #4042

Open
ankur22 opened this issue Nov 7, 2024 · 1 comment
Open

Thresholds using URL may not correlate in browser based tests #4042

ankur22 opened this issue Nov 7, 2024 · 1 comment
Assignees

Comments

@ankur22
Copy link
Contributor

ankur22 commented Nov 7, 2024

Brief summary

When running a browser test, and navigating to a site (e.g. https://test.k6.io) if the trailing "/" is missing, chrome will add it. All metrics that are associated to that navigation will work with the amended address with the trailing "/" (e.g. https://test.k6.io/). This can be problematic when working with thresholds, since the user may not add the trailing "/" (e.g. thresholds: { 'browser_web_vital_lcp{url:https://test.k6.io}': ['p(95) < 2000'] }) which will mean that k6 will not be able to correlate the metrics with the thresholds. The end result will mean that the threshold value will be incorrect.

k6 version

NA

OS

NA

Docker version and image (if applicable)

No response

Steps to reproduce the problem

  1. Run the following:
import { browser } from 'k6/browser';
import { check, sleep } from 'k6';

export const options = {
  scenarios: {
    FsFacility: {
      executor: 'per-vu-iterations',
      vus: 1,
      iterations: 1,
      options: {
        browser: {
          type: 'chromium',
        },
      },
    },
  },
  thresholds: {
    'browser_web_vital_lcp{url:https://test.k6.io}': ['p(95) < 2000'],
  },
};

export default async function () {
  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    await page.goto('https://test.k6.io');
    sleep(10); // Allows time to capture LCP and other web vitals
  } finally {
    await page.close();
  }
}

Which will result in:

browser_web_vital_lcp..........: avg=204.4ms min=204.4ms med=204.4ms max=204.4ms  p(90)=204.4ms  p(95)=204.4ms

     ✓ { url:https://test.k6.io }...: avg=0s      min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s

Changing the threshold to:

thresholds: {
    'browser_web_vital_lcp{url:https://test.k6.io/}': ['p(95) < 2000'],
  },

Fixes the issue.

Expected behaviour

The threshold correlates with the metric regardless of the missing trailing "/".

Actual behaviour

The threshold doesn't correlate with the metric when it is missing the trailing "/".

@ankur22
Copy link
Contributor Author

ankur22 commented Nov 7, 2024

It's possible to hit this issue when the page redirects to another page. E.g. when navigating to your grafana stack (some-name.grafana.com), it will redirect to the login page first. In some cases it may not redirect back to the initial page. The thresholds will not correlate due to the redirects. It currently requires the user to know up front all the possible urls that the test will navigate to and adding the thresholds in for each url, e.g.:

    thresholds: {
        'browser_web_vital_lcp{url:https://some-name.grafana.net/login}': ['p(95) < 2000'],
        'browser_web_vital_lcp{url:https://some-name.grafana.net/}': ['p(95) < 2000']
      },

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants