Code

Add a local cache module#884

Code

Merged

haoming29 merged 48 commits intoPelicanPlatform:mainPelicanPlatform/pelican:mainfrom bbockelm:simple_cachebbockelm/pelican:simple_cacheCopy head branch name to clipboardMar 11, 2024

+4,165-1,363

Commits

Commits on Mar 2, 2024

• Refactor the HTTP test module

  Show description for 8477db4
  bbockelm

  committed
  8477db4

• Create a new file_cache module (WIP)

  Show description for 334fb4e
  bbockelm

  committed
  334fb4e

• Start integration between the file cache and client/launchers

  Show description for 05388e8
  bbockelm

  committed
  05388e8

• Revise MakeRequest to take a context

  Show description for 171f6c5
  bbockelm

  committed
  171f6c5

• Add support for fetching JWKS URLs by issuer name

  Show description for 9224cd8
  bbockelm

  committed
  9224cd8

• Add support for "resource scopes"

  Show description for 19724d6
  bbockelm

  committed
  19724d6

• Add support for authorization in the local cache

  Show description for 3c8ef0c
  bbockelm

  committed
  3c8ef0c

Commits on Mar 3, 2024

• Remove unreachable code

  bbockelm

  committed
  fda3ea0

• Do not export JobID from TransferResults

  Show description for 5f9419c
  bbockelm

  committed
  5f9419c

• Add initial tests for the local cache functionality

  Show description for 60f7510
  bbockelm

  committed
  60f7510

• Simplify use of AddScopes

  Show description for 377f879
  bbockelm

  committed
  377f879

• Update jwx library

  bbockelm

  committed
  33a5b80

• Add tests for authenticated LocalCache and access via HTTP

  Show description for 01a4dda
  bbockelm

  committed
  01a4dda

• Fix linter issue in LocalCache test

  bbockelm

  committed
  83ffc21

• Add unit test for invoking the Pelican client to download

  Show description for 60c7071
  bbockelm

  committed
  60c7071

Commits on Mar 4, 2024

• Add support for Stat in the local cache

  Show description for e36ae1a
  bbockelm

  committed
  e36ae1a

• Add support for MaximumDownloadSpeed

  Show description for ecefa09
  bbockelm

  committed
  ecefa09

• Add unit tests for large file support and stat

  Show description for 9727bc4
  bbockelm

  committed
  9727bc4

• Add integration test to ensure the LocalCache works with OSDF.

  bbockelm

  committed
  eb8d2e0

Commits on Mar 5, 2024

• Mass rename to ensure we consistently use 'localcache'

  bbockelm

  committed
  459b7a6

• Fix compilation on Windows

  bbockelm

  committed
  9abbdd5

• Fix error handling around the client

  Show description for d08c46b
  bbockelm

  committed
  d08c46b

• Add support for on-demand purging and corresponding unit tests

  Show description for f68e940
  bbockelm

  committed
  f68e940

• Disable caches for all the unit test platforms

  Show description for 242851e
  bbockelm

  committed
  242851e

Commits on Mar 6, 2024

• Merge remote-tracking branch 'origin/main' into simple_cache

  bbockelm

  committed
  1726bb5

• For consistency, accept stash protocol for DoStat

  Show description for 5edb5cd
  bbockelm

  committed
  5edb5cd

• Tweak cache test

  Show description for 5383e21
  bbockelm

  committed
  5383e21

• Correct missed file permission

  bbockelm

  committed
  a2159ac

• Bugfix: recursive downloads were not incorporating directory structure

  Show description for fb467a5
  bbockelm

  committed
  fb467a5

• Merge remote-tracking branch 'origin/main' into simple_cache

  bbockelm

  committed
  4a9f2ec

• Add directory template characters; fails on Linux otherwise

  bbockelm

  committed
  59a4ef8

Commits on Mar 7, 2024

• Make integration test more robust

  Show description for dceedad
  bbockelm

  committed
  dceedad

• Fix overwrite of URL path via pointer

  Show description for aa8d10c
  bbockelm

  committed
  aa8d10c

Commits on Mar 9, 2024

• Refactor response object to be in common between cache and broker

  bbockelm

  committed
  b5ae631

• Various fixups from code review

  Show description for 3572efc
  bbockelm

  committed
  3572efc

• Refactor TokenConfig so setting audience is type-safe

  Show description for 5850086
  bbockelm

  committed
  5850086

• Add unit test for calculating resource scopes

  bbockelm

  committed
  3ab8389

• Refactor FedTest to be shared across client and local_cache

  Show description for 224b4f7
  bbockelm

  committed
  224b4f7

• Merge remote-tracking branch 'origin/main' into simple_cache

  bbockelm

  committed
  9af4ba7

• Switch to utility-based verification routine

  bbockelm

  committed
  b1dc96c

Commits on Mar 10, 2024

• Fix tests to have audience

  bbockelm

  committed
  fe957ea

• Ensure registry tests can run without a network connection

  bbockelm

  committed
  d2b775f

• Fix SetVersion to use correct version

  bbockelm

  committed
  d98f5ad

• Correct ownership of the origin directory when run as root

  bbockelm

  committed
  cd3d906

Commits on Mar 11, 2024

• Sort the caches based on responsiveness

  Show description for 316cc2e
  bbockelm

  committed
  316cc2e

• Perform HEAD against multiple caches if provided

  Show description for 223125e
  bbockelm

  committed
  223125e

• Add comment to token audience function

  haoming29

  committed
  f7d47c9

• Improve sorting logic for HEAD requests

  Show description for fffac65
  bbockelm

  committed
  fffac65