Merge branch 'develop' of github.com:smartcontractkit/chainlink into …

…fixCCIPChaos

.changeset/flat-horses-argue.md

@@ -0,0 +1,5 @@
+---
+"chainlink": minor
+---
+
+#added LogPoller MaxLogsKept feature: recency count-based instead of time based log retention

.changeset/red-jobs-marry.md

@@ -0,0 +1,5 @@
+---
+"chainlink": patch
+---
+
+Rename DA oracle consts to be more descriptive #wip

.changeset/rude-spies-serve.md

@@ -0,0 +1,5 @@
+---
+"chainlink": patch
+---
+
+#added new config field to FeeQuoter

.github/E2E_TESTS_ON_GITHUB_CI.md

@@ -1,58 +1,90 @@
 # E2E Tests on GitHub CI
 
-E2E tests are executed on GitHub CI using the [E2E Tests Reusable Workflow](#about-the-reusable-workflow) or dedicated workflows.
+- [E2E Tests on GitHub CI](#e2e-tests-on-github-ci)
+  - [Scheduled test workflows](#scheduled-test-workflows)
+    - [PR E2E Tests](#pr-e2e-tests)
+    - [Nightly E2E Tests](#nightly-e2e-tests)
+    - [Release E2E Tests](#release-e2e-tests)
+      - [Integration (smoke) Tests](#integration-smoke-tests)
+      - [Client Compatibility Tests](#client-compatibility-tests)
+  - [On-Demand Workflows](#on-demand-workflows)
+    - [Test workflows setup in CI](#test-workflows-setup-in-ci)
+      - [Configuration Overrides](#configuration-overrides)
+      - [Test Secrets](#test-secrets)
 
-## Automatic workflows
+E2E tests are executed on GitHub CI using the [E2E Tests Reusable Workflow](https://github.com/smartcontractkit/.github/blob/main/.github/workflows/README.md) or dedicated workflows.
 
-These workflows are designed to run automatically at crucial stages of the software development process, such as on every commit in a PR, nightly or before release.
+## Scheduled test workflows
+
+These workflows are designed to run on every commit in a PR, nightly or before release (see `triggers` in [e2e-tests.yaml](./e2e-tests.yml)).
 
 ### PR E2E Tests
 
-Run on every commit in a PR to ensure changes do not introduce regressions.
+Tests triggered on every commit in a PR to ensure changes do not introduce regressions.
 
-[Link](https://github.com/smartcontractkit/chainlink/blob/develop/.github/workflows/integration-tests.yml)
+**Workflow:** [integration-tests.yml](https://github.com/smartcontractkit/chainlink/blob/develop/.github/workflows/integration-tests.yml)
 
 ### Nightly E2E Tests
 
-Conducted nightly to catch issues that may develop over time or with accumulated changes.
+Nightly E2E test runs.
 
-[Link](https://github.com/smartcontractkit/chainlink/blob/develop/.github/workflows/run-nightly-e2e-tests.yml)
+**Workflow:** [nightly-e2e-tests.yml](https://github.com/smartcontractkit/chainlink/blob/develop/.github/workflows/run-nightly-e2e-tests.yml)
 
 ### Release E2E Tests
 
-This section contains automatic workflows triggering E2E tests at release.
+E2E tests triggered on a release tag.
+
+#### Integration (smoke) Tests
+
+**Workflow:** [integration-tests.yml](https://github.com/smartcontractkit/chainlink/blob/develop/.github/workflows/integration-tests.yml)
 
 #### Client Compatibility Tests
 
-[Link](https://github.com/smartcontractkit/chainlink/actions/workflows/client-compatibility-tests.yml)
+**Workflow:** [client-compatibility-tests.yml](https://github.com/smartcontractkit/chainlink/actions/workflows/client-compatibility-tests.yml)
 
 ## On-Demand Workflows
 
-Triggered manually by QA for specific testing needs.
+These are dispatched parametrized workflows, that may be triggered manually for specific testing needs. For more details refer [integration-tests README](../integration-tests/README.md) and [per-product test run books](../integration-tests/run-books/).
 
 **Examples:**
 
+- [Selected E2E Tests Workflow](https://github.com/smartcontractkit/chainlink/actions/workflows/run-selected-e2e-tests.yml)
+- [Client Compatibility Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/client-compatibility-tests.yml)
+- [Chaos Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/integration-chaos-tests.yml)
+- [OCR Soak Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/on-demand-ocr-soak-test.yml)
 - [On-Demand Automation Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/automation-ondemand-tests.yml)
 - [CCIP Chaos Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/ccip-chaos-tests.yml)
-- [OCR Soak Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/on-demand-ocr-soak-test.yml)
+- [CCIP Load Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/ccip-load-tests.yml)
 - [VRFv2Plus Smoke Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/on-demand-vrfv2plus-smoke-tests.yml)
+- [VRFv2Plus Performance Tests](https://github.com/smartcontractkit/chainlink/actions/workflows/on-demand-vrfv2plus-performance-test.yml)
+
+### Test workflows setup in CI
 
-## Test Configs
+Most workflows may be triggered with default configs. Some, nevertheless, may be overridden.
 
-E2E tests utilize TOML files to define their parameters. Each test is equipped with a default TOML config, which can be overridden by specifying an alternative TOML config. This allows for running tests with varied parameters, such as on a non-default blockchain network. For tests executed on GitHub CI, both the default configs and any override configs must reside within the git repository. The `test_config_override_path` workflow input is used to provide a path to an override config.
+> [!TIP]
+> Use `gh` CLI commands to run workflows from local machine.
 
-Config overrides should be stored in `testconfig/*/overrides/*.toml`. Placing files here will not trigger a rebuild of the test runner image.
+#### Configuration Overrides
 
-**Important Note:** The use of `base64Config` input is deprecated in favor of `test_config_override_path`. For more details, refer to [the decision log](https://smartcontract-it.atlassian.net/wiki/spaces/TT/pages/927596563/Storing+All+Test+Configs+In+Git).
+> [!CAUTION]
+> Test configurations should not keep any [sensitive data or secrets](#test-secrets).
 
-To learn more about test configs see [CTF Test Config](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md).
+1. Reference sources:
+   1. [Integration-Tests configurations](../integration-tests/testconfig/README.md);
+   2. [CTF Test Config](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md).
+2. Defaults and overrides should be stored (committed) in repository under `../integration-tests/testconfig/<product>/overrides/<override-name>.toml` (see example [here](../integration-tests/testconfig/ocr2/overrides/base_sepolia.toml)).
+3. Use `test_config_override_path` to point to an override config. For example: `test_config_override_path="testconfig/ocr2/overrides/base_sepolia.toml"`
 
-## Test Secrets
+#### Test Secrets
 
-For security reasons, test secrets and sensitive information are not stored directly within the test config TOML files. Instead, these secrets are securely injected into tests using environment variables. For a detailed explanation on managing test secrets, refer to our [Test Secrets documentation](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets).
+> [!CAUTION]
+> Pay attention to never store/expose/commit your test secrets in repository.
 
-If you need to run a GitHub workflow using custom secrets, please refer to the [guide on running GitHub workflows with your test secrets](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#run-github-workflow-with-your-test-secrets).
+Test secrets allow provisioning and override the sensitive data such as EOA's private key, RPCs, Docker registry links, etc.
 
-## About the E2E Test Reusable Workflow
+Reference sources:
 
-For information on the E2E Test Reusable Workflow, visit the documentation in the [smartcontractkit/.github repository](https://github.com/smartcontractkit/.github/blob/main/.github/workflows/README.md).
+1. [BASE64_CONFIG_OVERRIDE](../integration-tests/testconfig/README.md#base64_config_override).
+2. [CTF Test Secrets documentation](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#test-secrets).
+3. [Guide on running GitHub workflows with your test secrets](https://github.com/smartcontractkit/chainlink-testing-framework/blob/main/lib/config/README.md#run-github-workflow-with-your-test-secrets).

.github/actions/golangci-lint/action.yml

@@ -70,3 +70,4 @@ runs:
       with:
         name: golangci-lint-report
         path: ${{ inputs.go-directory }}/golangci-lint-report.xml
+        retention-days: 7

.github/actions/goreleaser-build-sign-publish/release.js

@@ -1,25 +1,146 @@
 #!/usr/bin/env node
 const { execSync } = require("child_process");
+const fs = require("fs");
+const path = require("path");
 
 function main() {
-  const goreleaserConfig = mustGetEnv("GORELEASER_CONFIG");
-  const releaseType = mustGetEnv("RELEASE_TYPE");
-  const command = constructGoreleaserCommand(releaseType, goreleaserConfig);
-
-  if (process.env.DRY_RUN) {
-    console.log(`Generated command: ${command}`);
-    console.log("Dry run enabled. Exiting without executing the command.");
-    return;
+  const args = process.argv.slice(2);
+  const useExistingDist = args.includes("--use-existing-dist");
+  const chainlinkVersion = getVersion();
+
+  if (!useExistingDist) {
+    const goreleaserConfig = mustGetEnv("GORELEASER_CONFIG");
+    const releaseType = mustGetEnv("RELEASE_TYPE");
+    const command = constructGoreleaserCommand(
+      releaseType,
+      chainlinkVersion,
+      goreleaserConfig
+    );
+
+    if (process.env.DRY_RUN) {
+      console.log(`Generated command: ${command}`);
+      console.log("Dry run enabled. Exiting without executing the command.");
+      return;
+    } else {
+      console.log(`Executing command: ${command}`);
+      execSync(command, { stdio: "inherit" });
+    }
   } else {
-    console.log(`Executing command: ${command}`);
-    execSync(command, { stdio: "inherit" });
+    console.log(
+      "Skipping Goreleaser command execution as '--use-existing-dist' is set."
+    );
+  }
+
+  const artifactsJsonPath = findArtifactsJson();
+  const dockerImages = extractDockerImages(artifactsJsonPath);
+  const repoSha = execSync("git rev-parse HEAD", { encoding: "utf-8" }).trim();
+
+  const results = dockerImages.map((image) => {
+    try {
+      console.log(`Checking version for image: ${image}, expected version: ${chainlinkVersion}, expected SHA: ${repoSha}`);
+      const versionOutput = execSync(`docker run --rm ${image} --version`, {
+        encoding: "utf-8",
+      });
+      console.log(`Output from image ${image}: ${versionOutput}`);
+
+      const cleanedOutput = versionOutput.replace("chainlink version ", "").trim();
+      const [version, sha] = cleanedOutput.split("@");
+      if (!version || !sha) {
+        throw new Error("Version or SHA not found in output.");
+      }
+
+      if (sha.trim() !== repoSha) {
+        throw new Error(`SHA mismatch: Expected ${repoSha}, got ${sha.trim()}`);
+      }
+      if (version.trim() !== chainlinkVersion) {
+        throw new Error(
+          `Version mismatch: Expected ${chainlinkVersion}, got ${version.trim()}`
+        );
+      }
+
+      return { image, success: true, message: "Version check passed." };
+    } catch (error) {
+      return { image, success: false, message: error.message };
+    }
+  });
+
+  printSummary(results);
+  if (results.some((result) => !result.success)) {
+    process.exit(1);
   }
 }
 
-main();
+function printSummary(results) {
+  const passed = results.filter((result) => result.success);
+  const failed = results.filter((result) => !result.success);
+
+  console.log("\nSummary:");
+  console.log(`Total images checked: ${results.length}`);
+  console.log(`Passed: ${passed.length}`);
+  console.log(`Failed: ${failed.length}`);
+
+  if (passed.length > 0) {
+    console.log("\nPassed images:");
+    passed.forEach((result) =>
+      console.log(`${result.image}:\n${result.message}`)
+    );
+  }
+
+  if (failed.length > 0) {
+    console.log("\nFailed images:");
+    failed.forEach((result) =>
+      console.log(`${result.image}:\n${result.message}`)
+    );
+  }
+}
+
+function findArtifactsJson() {
+  const distDir = path.resolve(process.cwd(), "dist");
+  const files = [];
+
+  function findJsonFiles(dir) {
+    const items = fs.readdirSync(dir, { withFileTypes: true });
+    for (const item of items) {
+      const fullPath = path.join(dir, item.name);
+      if (item.isDirectory()) {
+        findJsonFiles(fullPath);
+      } else if (item.isFile() && item.name === "artifacts.json") {
+        files.push(fullPath);
+      }
+    }
+  }
+
+  findJsonFiles(distDir);
+
+  if (files.length === 0) {
+    console.error("Error: No artifacts.json found in /dist.");
+    process.exit(1);
+  } else if (files.length > 1) {
+    console.error("Error: Multiple artifacts.json files found.");
+    process.exit(1);
+  }
+
+  return files[0];
+}
+
+function extractDockerImages(artifactsJsonPath) {
+  console.log(`Reading artifacts.json from: ${artifactsJsonPath}`);
+  const artifactsJson = JSON.parse(fs.readFileSync(artifactsJsonPath, "utf-8"));
+
+  const dockerImages = artifactsJson
+    .filter((artifact) => artifact.type === "Docker Image")
+    .map((artifact) => artifact.name);
+
+  if (dockerImages.length === 0) {
+    console.error("Error: No Docker images found in artifacts.json.");
+    process.exit(1);
+  }
 
-function constructGoreleaserCommand(releaseType, goreleaserConfig) {
-  const version = getVersion();
+  console.log(`Found Docker images:\n  - ${dockerImages.join("\n  - ")}`);
+  return dockerImages;
+}
+
+function constructGoreleaserCommand(releaseType, version, goreleaserConfig) {
   const flags = [];
 
   checkReleaseType(releaseType);
@@ -59,6 +180,7 @@ function checkReleaseType(releaseType) {
     console.error(
       `Error: Invalid release type: ${releaseType}. Must be one of: ${validReleaseTypesStr}`
     );
+    process.exit(1);
   }
 }
 
@@ -74,7 +196,7 @@ function mustGetEnv(key) {
 
 function getVersion() {
   try {
-    const pkgPath = process.cwd() + "/package.json";
+    const pkgPath = path.resolve(process.cwd(), "package.json");
     console.log("Looking for chainlink version in package.json at: ", pkgPath);
     const packageJson = require(pkgPath);
     if (!packageJson.version) {
@@ -91,3 +213,5 @@ function getVersion() {
     process.exit(1);
   }
 }
+
+main();

.github/workflows/automation-benchmark-tests.yml

@@ -44,7 +44,7 @@ jobs:
       GRAFANA_INTERNAL_HOST: ${{ secrets.GRAFANA_INTERNAL_HOST }}
       GRAFANA_INTERNAL_URL_SHORTENER_TOKEN: ${{ secrets.GRAFANA_INTERNAL_URL_SHORTENER_TOKEN }}
       LOKI_TENANT_ID: ${{ secrets.LOKI_TENANT_ID }}
-      LOKI_URL: https://${{ secrets.GRAFANA_INTERNAL_HOST }}/loki/api/v1/push
+      LOKI_URL: ${{ secrets.LOKI_URL }}
       LOKI_BASIC_AUTH: ${{ secrets.LOKI_BASIC_AUTH }}
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
       AWS_REGION: ${{ secrets.QA_AWS_REGION }}

.github/workflows/automation-load-tests.yml

@@ -39,7 +39,7 @@ jobs:
       GRAFANA_INTERNAL_HOST: ${{ secrets.GRAFANA_INTERNAL_HOST }}
       GRAFANA_INTERNAL_URL_SHORTENER_TOKEN: ${{ secrets.GRAFANA_INTERNAL_URL_SHORTENER_TOKEN }}
       LOKI_TENANT_ID: ${{ secrets.LOKI_TENANT_ID }}
-      LOKI_URL: https://${{ secrets.GRAFANA_INTERNAL_HOST }}/loki/api/v1/push
+      LOKI_URL: ${{ secrets.LOKI_URL }}
       LOKI_BASIC_AUTH: ${{ secrets.LOKI_BASIC_AUTH }}
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
       AWS_REGION: ${{ secrets.QA_AWS_REGION }}

.github/workflows/automation-nightly-tests.yml

@@ -32,7 +32,7 @@ jobs:
       GRAFANA_INTERNAL_HOST: ${{ secrets.GRAFANA_INTERNAL_HOST }}
       GRAFANA_INTERNAL_URL_SHORTENER_TOKEN: ${{ secrets.GRAFANA_INTERNAL_URL_SHORTENER_TOKEN }}
       LOKI_TENANT_ID: ${{ secrets.LOKI_TENANT_ID }}
-      LOKI_URL: https://${{ secrets.GRAFANA_INTERNAL_HOST }}/loki/api/v1/push
+      LOKI_URL: ${{ secrets.LOKI_URL }}
       LOKI_BASIC_AUTH: ${{ secrets.LOKI_BASIC_AUTH }}
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
       AWS_REGION: ${{ secrets.QA_AWS_REGION }}

.github/workflows/automation-ondemand-tests.yml

@@ -174,7 +174,7 @@ jobs:
       GRAFANA_INTERNAL_HOST: ${{ secrets.GRAFANA_INTERNAL_HOST }}
       GRAFANA_INTERNAL_URL_SHORTENER_TOKEN: ${{ secrets.GRAFANA_INTERNAL_URL_SHORTENER_TOKEN }}
       LOKI_TENANT_ID: ${{ secrets.LOKI_TENANT_ID }}
-      LOKI_URL: https://${{ secrets.GRAFANA_INTERNAL_HOST }}/loki/api/v1/push
+      LOKI_URL: ${{ secrets.LOKI_URL }}
       LOKI_BASIC_AUTH: ${{ secrets.LOKI_BASIC_AUTH }}
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
       AWS_REGION: ${{ secrets.QA_AWS_REGION }}

.github/workflows/ccip-chaos-tests.yml

@@ -38,7 +38,7 @@ jobs:
       GRAFANA_INTERNAL_HOST: ${{ secrets.GRAFANA_INTERNAL_HOST }}
       GRAFANA_INTERNAL_URL_SHORTENER_TOKEN: ${{ secrets.GRAFANA_INTERNAL_URL_SHORTENER_TOKEN }}
       LOKI_TENANT_ID: ${{ secrets.LOKI_TENANT_ID }}
-      LOKI_URL: https://${{ secrets.GRAFANA_INTERNAL_HOST }}/loki/api/v1/push
+      LOKI_URL: ${{ secrets.LOKI_URL }}
       LOKI_BASIC_AUTH: ${{ secrets.LOKI_BASIC_AUTH }}
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
       AWS_REGION: ${{ secrets.QA_AWS_REGION }}

.github/workflows/ccip-load-tests.yml

@@ -54,7 +54,7 @@ jobs:
       GRAFANA_INTERNAL_HOST: ${{ secrets.GRAFANA_INTERNAL_HOST }}
       GRAFANA_INTERNAL_URL_SHORTENER_TOKEN: ${{ secrets.GRAFANA_INTERNAL_URL_SHORTENER_TOKEN }}
       LOKI_TENANT_ID: ${{ secrets.LOKI_TENANT_ID }}
-      LOKI_URL: ${{ secrets.LOKI_URL }} # CCIP has a different one for some reason
+      LOKI_URL: ${{ secrets.LOKI_URL }}
       LOKI_BASIC_AUTH: ${{ secrets.LOKI_BASIC_AUTH }}
       GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}  
       AWS_REGION: ${{ secrets.QA_AWS_REGION }}