Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integrate with CRIC to fetch list of Datalake endpoints to test #3

Open
Tracked by #2
andreaceccanti opened this issue Sep 22, 2021 · 2 comments
Open
Tracked by #2

Comments

@andreaceccanti
Copy link
Contributor

andreaceccanti commented Sep 22, 2021

This JSON document provides a list of active RSEs in the Datalake.
We can fetch the list of endpoints to test starting from the list of RSEs, however we should probably use a different prefix for each endpoint to not mess with RUCIO data.

@federicaagostini
Copy link
Contributor

federicaagostini commented Sep 25, 2021

The fetch-rses-from-cric.sh script provides the list of datalake endpoints in yaml syntax compliant with the variables.yaml file.

  • Updated variables.yaml file with CRIC datalake endpoints
  • Added endpoint.<endpoint_alias>.path.prefix key to variables.yaml file, which match the RUCIO path (i.e. rses.<endpoint_alias>.protocols[0].prefix of JSON document)
  • Added Get ESCAPE path keywork to get correct ESCAPE path from prefix
  • Added warning message in CI if there are new CRIC RSEs (here), filtering by .rses.<endpoint_alias>.name
  • Updated README accordingly

@federicaagostini
Copy link
Contributor

Integrated dynamic fetch of CRIC endpoints in CI (this commit):

  • locally (i.e. using run-testsuite.sh): run endpoints listed in variables.yaml
  • CI (i.e. using ci/run.sh): run endpoints listed in variables.yaml and new CRIC endpoints

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants