Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a local cache module #884

Merged
merged 48 commits into from
Mar 11, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
8477db4
Refactor the HTTP test module
bbockelm Mar 2, 2024
334fb4e
Create a new `file_cache` module (WIP)
bbockelm Feb 29, 2024
05388e8
Start integration between the file cache and client/launchers
bbockelm Feb 29, 2024
171f6c5
Revise `MakeRequest` to take a context
bbockelm Mar 2, 2024
9224cd8
Add support for fetching JWKS URLs by issuer name
bbockelm Mar 2, 2024
19724d6
Add support for "resource scopes"
bbockelm Mar 2, 2024
3c8ef0c
Add support for authorization in the local cache
bbockelm Mar 2, 2024
fda3ea0
Remove unreachable code
bbockelm Mar 3, 2024
5f9419c
Do not export JobID from TransferResults
bbockelm Mar 3, 2024
60f7510
Add initial tests for the local cache functionality
bbockelm Mar 3, 2024
377f879
Simplify use of `AddScopes`
bbockelm Mar 3, 2024
33a5b80
Update jwx library
bbockelm Mar 3, 2024
01a4dda
Add tests for authenticated LocalCache and access via HTTP
bbockelm Mar 3, 2024
83ffc21
Fix linter issue in LocalCache test
bbockelm Mar 3, 2024
60c7071
Add unit test for invoking the Pelican client to download
bbockelm Mar 3, 2024
e36ae1a
Add support for `Stat` in the local cache
bbockelm Mar 4, 2024
ecefa09
Add support for MaximumDownloadSpeed
bbockelm Mar 4, 2024
9727bc4
Add unit tests for large file support and stat
bbockelm Mar 4, 2024
eb8d2e0
Add integration test to ensure the LocalCache works with OSDF.
bbockelm Mar 4, 2024
459b7a6
Mass rename to ensure we consistently use 'localcache'
bbockelm Mar 5, 2024
9abbdd5
Fix compilation on Windows
bbockelm Mar 5, 2024
d08c46b
Fix error handling around the client
bbockelm Mar 5, 2024
f68e940
Add support for on-demand purging and corresponding unit tests
bbockelm Mar 5, 2024
242851e
Disable caches for all the unit test platforms
bbockelm Mar 5, 2024
1726bb5
Merge remote-tracking branch 'origin/main' into simple_cache
bbockelm Mar 6, 2024
5edb5cd
For consistency, accept `stash` protocol for `DoStat`
bbockelm Mar 6, 2024
5383e21
Tweak cache test
bbockelm Mar 6, 2024
a2159ac
Correct missed file permission
bbockelm Mar 6, 2024
fb467a5
Bugfix: recursive downloads were not incorporating directory structure
bbockelm Mar 6, 2024
4a9f2ec
Merge remote-tracking branch 'origin/main' into simple_cache
bbockelm Mar 6, 2024
59a4ef8
Add directory template characters; fails on Linux otherwise
bbockelm Mar 6, 2024
dceedad
Make integration test more robust
bbockelm Mar 7, 2024
aa8d10c
Fix overwrite of URL path via pointer
bbockelm Mar 7, 2024
b5ae631
Refactor response object to be in common between cache and broker
bbockelm Mar 9, 2024
3572efc
Various fixups from code review
bbockelm Mar 9, 2024
5850086
Refactor TokenConfig so setting audience is type-safe
bbockelm Mar 9, 2024
3ab8389
Add unit test for calculating resource scopes
bbockelm Mar 9, 2024
224b4f7
Refactor FedTest to be shared across client and local_cache
bbockelm Mar 9, 2024
9af4ba7
Merge remote-tracking branch 'origin/main' into simple_cache
bbockelm Mar 9, 2024
b1dc96c
Switch to utility-based verification routine
bbockelm Mar 9, 2024
fe957ea
Fix tests to have audience
bbockelm Mar 10, 2024
d2b775f
Ensure registry tests can run without a network connection
bbockelm Mar 10, 2024
d98f5ad
Fix SetVersion to use correct version
bbockelm Mar 10, 2024
cd3d906
Correct ownership of the origin directory when run as root
bbockelm Mar 10, 2024
316cc2e
Sort the caches based on responsiveness
bbockelm Mar 11, 2024
223125e
Perform HEAD against multiple caches if provided
bbockelm Mar 11, 2024
f7d47c9
Add comment to token audience function
haoming29 Mar 11, 2024
fffac65
Improve sorting logic for HEAD requests
bbockelm Mar 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions cache_ui/advertise.go
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
package cache_ui

import (
"context"
"encoding/json"
"net/url"
"strings"
Expand Down Expand Up @@ -76,14 +77,14 @@ func (server *CacheServer) GetNamespaceAdsFromDirector() error {
// Attempt to get data from the 2.0 endpoint, if that returns a 404 error, then attempt to get data
// from the 1.0 endpoint and convert from V1 to V2

respData, err := utils.MakeRequest(directorNSListEndpointURL, "GET", nil, nil)
respData, err := utils.MakeRequest(context.Background(), directorNSListEndpointURL, "GET", nil, nil)
if err != nil {
if strings.Contains(err.Error(), "404") {
directorNSListEndpointURL, err = url.JoinPath(directorEndpoint, "api", "v1.0", "director", "listNamespaces")
if err != nil {
return err
}
respData, err = utils.MakeRequest(directorNSListEndpointURL, "GET", nil, nil)
respData, err = utils.MakeRequest(context.Background(), directorNSListEndpointURL, "GET", nil, nil)
var respNSV1 []common.NamespaceAdV1
if err != nil {
return errors.Wrap(err, "Failed to make request")
Expand Down
44 changes: 30 additions & 14 deletions client/director.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ import (
"io"
"net/http"
"net/url"
"path"
"sort"
"strconv"
"strings"
Expand Down Expand Up @@ -244,7 +245,9 @@ func NewTransferDetailsUsingDirector(cache namespaces.DirectorCache, opts transf
log.Errorln("Failed to parse cache:", cache, "error:", err)
return nil
}
if cacheURL.Host == "" {
if cacheURL.Scheme == "unix" && cacheURL.Host != "" {
cacheURL.Path = path.Clean("/" + path.Join(cacheURL.Host, cacheURL.Path))
} else if cacheURL.Host == "" {
// Assume the cache is just a hostname
cacheURL.Host = cacheEndpoint
cacheURL.Path = ""
Expand All @@ -253,26 +256,31 @@ func NewTransferDetailsUsingDirector(cache namespaces.DirectorCache, opts transf
}
log.Debugf("Parsed Cache: %s", cacheURL.String())
if opts.NeedsToken {
cacheURL.Scheme = "https"
if !hasPort(cacheURL.Host) {
// Add port 8444 and 8443
urlCopy := *cacheURL
urlCopy.Host += ":8444"
details = append(details, transferAttemptDetails{
Url: &urlCopy,
Proxy: false,
PackOption: opts.PackOption,
})
// Strip the port off and add 8443
cacheURL.Host = cacheURL.Host + ":8443"
// Unless we're using the local Unix domain socket cache, force HTTPS
if cacheURL.Scheme != "unix" {
cacheURL.Scheme = "https"
if !hasPort(cacheURL.Host) {
// Add port 8444 and 8443
urlCopy := *cacheURL
urlCopy.Host += ":8444"
details = append(details, transferAttemptDetails{
Url: &urlCopy,
Proxy: false,
PackOption: opts.PackOption,
})
// Strip the port off and add 8443
cacheURL.Host = cacheURL.Host + ":8443"
}
}
// Whether port is specified or not, add a transfer without proxy
details = append(details, transferAttemptDetails{
Url: cacheURL,
Proxy: false,
PackOption: opts.PackOption,
})
} else {
} else if cacheURL.Scheme == "" || cacheURL.Scheme == "http" {
// Assume a transfer not needing a token and not specifying a scheme is HTTP
// WARNING: This is legacy code; we should always specify a scheme
cacheURL.Scheme = "http"
if !hasPort(cacheURL.Host) {
cacheURL.Host += ":8000"
Expand All @@ -290,6 +298,14 @@ func NewTransferDetailsUsingDirector(cache namespaces.DirectorCache, opts transf
PackOption: opts.PackOption,
})
}
} else {
// A non-HTTP scheme is specified and a token is not needed; this wasn't possible
// in the legacy cases. Simply use the provided config
details = append(details, transferAttemptDetails{
Url: cacheURL,
Proxy: false,
PackOption: opts.PackOption,
})
}

return details
Expand Down
Loading
Loading