Skip to content

Commit

Permalink
Merge branch 'master' into itb
Browse files Browse the repository at this point in the history
* master: (100 commits)
  Create Rice_Li.yaml
  fixed bug where osg-notify prompts user indefinitely
  Add downtime for SDSC_NRP_OSDF_CACHE due to network issue
  Update topology/Montana State University/Montana State RCI/MTState-Tempest.yaml
  Montana State: Added local contacts; FD #74884
  put sc-cache back in downtime
  Add downtime for CARDIFF_UK_OSDF_CACHE due to overloaded
  Add downtime for PIC_STASHCACHE_CACHE due to overloaded
  Add downtime for Stashcache-UofA due to overloaded
  Fix 'syntax errors' in README.md code samples (i.e. put ellipses in comments)
  Hawaii: Enabled Koa resource; FD #75095
  Renamed Hawaii Facility and ITS Site per internal thread
  Remove OASIS repository captmnv.opensciencegrid.org
  Update Cache DN in SyracuseCachingInfrastructure.yaml
  End sc-cache.chtc.wisc.edu downtime early
  Create SITE.yaml
  Update CHTC-OSDF_downtime.yaml
  Create CNU_Henry.yaml
  Drop site contact, add local contacts (SOFTWARE-5815)
  Setting back the downtime changing the end date
  initial commit for new Hawaii Hosted CE
  Update UERJ_downtime.yaml
  Create UCSD_Rao.yaml
  Create UNL_Ramamurthy.yaml
  Amsterdam cache is back
  Revert "Removing last downtime"
  Updated UMontana name per Janet and ROR
  osg-notify: add support for Local Executive Contact (SOFTWARE-5814)
  Added local executive contact per blin; FD #75346
  Added local contacts; FD #75346
  Removing last downtime
  Removing downtime
  make Langston indefinite dt until we hear back
  put in Langston downtime
  changing description
  Add downtime for PIC_STASHCACHE_CACHE due to Xrootd issues
  Add downtime for FZU_STASH due to Firewall issue
  Fix docstring
  Set sc-origin to port 1095; this is the port running the Pelican-based origin instance
  Took Alabama-CHPC out of downtime early; FD #75128
  Adds new cvmfs repo for scisoft for fermilab https://support.opensciencegrid.org/a/tickets/75339
  Add downtime for Stashcache-UofA due to HW problem
  Extend downtime for CHTC_STASHCACHE_CACHE and CHTC_STASHCACHE_ORIGIN
  enable MTState-Tempest CE
  remove extraneous + chars
  start with MTState CE disabled
  register Montana State Hosted CE
  Make Description a multiline string
  Create MSState_2024_Chen.yaml
  Add downtime for Stashcache-UofA due to  Network issues
  ...
  • Loading branch information
matyasselmeci committed Feb 20, 2024
2 parents 15a36fd + ebda6a0 commit 20b5b24
Show file tree
Hide file tree
Showing 65 changed files with 1,814 additions and 144 deletions.
13 changes: 9 additions & 4 deletions bin/osg-notify
Original file line number Diff line number Diff line change
Expand Up @@ -116,10 +116,9 @@ def parseargs():
oparser.add_argument("--oim-owner-filter", dest="owner_vo",
help="Filter on resources that list VO(s) as a partial owner")

oparser.add_argument("--oim-contact-type", default="all", dest="contact_type",
choices=["all", "administrative", "miscellaneous", "security", "submitter", "site"],
help="Filter on contact type e.g. administrative, miscellaneous, security, submitter or site"
"(default: all)", )
oparser.add_argument("--oim-contact-type", action="append", dest="contact_type",
choices=["all"] + topology_utils.CONTACT_TYPES,
help="Filter on contact type (default: all)")
oparser.add_argument("--bypass-dns-check", action="store_true", dest="bypass_dns_check",
help="Bypass checking that one of the host's IP addresses matches with the hostanme resolution")
oparser.add_argument("--allow-non-ascii", action="store_true", dest="allow_non_ascii",
Expand All @@ -132,6 +131,10 @@ def parseargs():
if args.name_filter and args.fqdn_filter:
oparser.error("Can't specify both --oim-name-filter and --oim-fqdn-filter")

if not args.contact_type or ("all" in args.contact_type):
args.contact_type = ["all"]
args.contact_type = set(args.contact_type) # remove dupes

if args.from_name == 'security':
args.from_name = 'OSG Security Team'
args.from_addr = '[email protected]'
Expand Down Expand Up @@ -186,6 +189,7 @@ def main():
while attempts > 0:
try:
results = topology_utils.get_vo_contacts(args)
break
except topology_utils.InvalidPathError as exc:
print(exc)
exit(1)
Expand All @@ -211,6 +215,7 @@ def main():
results = topology_utils.get_resource_contacts_by_fqdn(args)
else:
results = topology_utils.get_resource_contacts(args)
break
except topology_utils.InvalidPathError as exc:
exit(str(exc))
except topology_utils.IncorrectPasswordError as exc:
Expand Down
7 changes: 7 additions & 0 deletions projects/CNU_Henry.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Department: Department of Physics, Computer Science, and Engineering
Description: >
Research related to biomedical and clinical natural language processing, including information retrieval,
information extraction, summarization, and question answering.
FieldOfScience: Computer and Information Sciences
Organization: Christopher Newport University
PIName: Samuel Henry
5 changes: 5 additions & 0 deletions projects/LBNL_Jensen.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Department: ATAP, BELLA Center
Description: Spectral reconstruction of laser-driven secondary light sources
FieldOfScience: Physics
Organization: Lawrence Berkeley National Laboratory
PIName: Kyle Jensen
8 changes: 8 additions & 0 deletions projects/MSState_2024_Chen.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
Department: Computer Science and Engineering
Description: >
This project is dedicated to creating a comprehensive multimodal platform, specifically designed to enhance graph dynamics analytics in professional settings.Our goal is to encompass the complete life cycle of graph dynamics with particular emphasis on spatial trajectory mining, including location prediction, optimal routing inference, and identity recognition.
Central to this initiative is utilization the capabilities of large transformer models (e.g., Large Language Models, Vision Transformers, or Graph Transformers). These will be tailored to effectively capture the complexities inherent in trajectory over spatial graphs. By leveraging natural language, we can unlock the potential of advanced graph analytics to those without programming expertise, and enable seamless collaboration between experts across domains by speaking the universal language.
FieldOfScience: Computer and Information Services
Organization: Mississippi State University
PIName: Zhiqian Chen

6 changes: 6 additions & 0 deletions projects/NOAA_Bell.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Department: Cooperative Institute for Research in Environmental Sciences
Description: >-
Water column sonar data, the acoustic back-scatter from the near surface to the seafloor, are used to assess physical and biological characteristics of the ocean including the spatial distribution of plankton, fish, methane seeps, and underwater oil plumes. Currently our catalog includes 270 TB of data that we are working to convert from a proprietary industry format into a cloud native Zarr format. Further documentation: <https://cires.gitbook.io/ncei-wcsd-archive/>
FieldOfScience: Ocean Sciences
Organization: National Oceanic and Atmospheric Administration
PIName: Carrie Bell
7 changes: 7 additions & 0 deletions projects/Princeton_Jamieson.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Description: >
studying channel modeling using AI/ML techniques.
Ray tracing and improving upon 3GPP standardized channel models are our goals.
Department: Princeton Advanced Wireless Systems Lab
FieldOfScience: Electrical, Electronic, and Communications
Organization: Princeton University
PIName: Kyle Jamieson
7 changes: 7 additions & 0 deletions projects/Rice_Li.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Department: Physics and Astronomy Department
Description: We do research on experimental nuclear and particle physics at the LHC.
Our main purpose of using OSG is to perform phenomenological model simulations to
compare with and understand the experimental data. https://lilab.rice.edu/
FieldOfScience: Physics
Organization: Rice University
PIName: Wei Li
5 changes: 5 additions & 0 deletions projects/Syracuse_Nitz.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Description: Looking for sub-solar mass neutron stars.
Department: Physics
FieldOfScience: Astronomy and Astrophysics
Organization: Syracuse University
PIName: Alex Nitz
13 changes: 13 additions & 0 deletions projects/UA_OIT.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
Description: Our goal is to educate researchers on how to use OSG to
perform their research. We have researchers working on medical
technology, biology, chemistry, data science, machine learning, and
artificial intelligence.
Department: Office of Information Technology
FieldOfScience: Computer and Information Services
Organization: University of Alabama
PIName: Donald Jay Cervino


Sponsor:
CampusGrid:
Name: OSG Connect
7 changes: 7 additions & 0 deletions projects/UCSD_Rao.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
Department: Electrical and Computer Engineering
Description: >
Machine learning algorithms for signal processing and communications,
particularly mmWave and medical imaging.
FieldOfScience: Computer and Information Sciences
Organization: University of California, San Diego
PIName: Bhaskar Rao
5 changes: 5 additions & 0 deletions projects/UNL_Ramamurthy.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
Department: School of Computing
Description: Research on optimizing large data transfers for science experiments
FieldOfScience: Computer and Information Services
Organization: University of Nebraska-Lincoln
PIName: Byrav Ramamurthy
14 changes: 7 additions & 7 deletions src/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,7 @@ DataFederation:
Namespaces:
- <NAMESPACE 1>
- <NAMESPACE 2>
...
# ...
- <NAMESPACE n>
```
Expand All @@ -250,7 +250,7 @@ Alternatively:
```yaml
Authorizations:
- <DN/FQAN/SCITOKENS AUTH 1>
...
# ...
- <DN/FQAN/SCITOKENS AUTH n>
```
These denote an authenticated namespace, which requires authentication for read access.
Expand Down Expand Up @@ -302,7 +302,7 @@ There are three kinds of authorization types:
```yaml
AllowedOrigins:
- ORIGIN_RESOURCE1
...
# ...
- ORIGIN_RESOURCEn
```
AllowedOrigins is a list of resource names of origins that will serve data for this namespace.
Expand All @@ -316,7 +316,7 @@ or
```yaml
AllowedCaches:
- CACHE_RESOURCE1
...
# ...
- CACHE_RESOURCEn
```
AllowedCaches is a list of resource names of caches that will serve data for this namespace.
Expand Down Expand Up @@ -389,14 +389,14 @@ For example:
```yaml
Resources:
Stashcache-Chicago:
...
# ...
Services:
XRootD cache server:
Description: Internet2 Chicago Cache
Details:
endpoint_override: osg-chicago-stashcache.nrp.internet2.edu:8443
auth_endpoint_override: osg-chicago-stashcache.nrp.internet2.edu:8444
...
# ...
```

### Supporting a Namespace
Expand Down Expand Up @@ -577,7 +577,7 @@ The final result looks like
},
{
"caches": [
(a whole bunch)
// (a whole bunch)
],
"credential_generation": {
"issuer": "https://osg-htc.org/ospool",
Expand Down
2 changes: 1 addition & 1 deletion src/stashcache.py
Original file line number Diff line number Diff line change
Expand Up @@ -541,7 +541,7 @@ def get_scitokens_list_for_namespace(ns: Namespace) -> List[Dict]:
def get_namespaces_info(global_data: GlobalData) -> PreJSON:
"""Return data for the /stashcache/namespaces JSON endpoint.
This includes a list of caches, with some data about cache endpoints,
This includes a list of caches and origins, with some data about their endpoints,
and a list of namespaces with some data about each namespace; see README.md for details.
"""
Expand Down
18 changes: 15 additions & 3 deletions src/topology_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,17 @@

import requests

# List of contact types stored in Topology data
# At time of writing, there isn't anything that restricts a contact to one of these types
CONTACT_TYPES = ["administrative",
"miscellaneous",
"security",
"submitter",
"site",
"local executive",
"local operational",
"local security"]

class Error(Exception):
pass

Expand Down Expand Up @@ -365,14 +376,15 @@ def filter_contacts(args, results):
args.fqdn_filter not in fqdn:
del results[fqdn]

if args.contact_type != 'all':
if 'all' not in args.contact_type:
# filter out undesired contact types
for name in list(results):
contact_list = []
for contact in results[name]:
contact_type = contact['ContactType']
if contact_type.startswith(args.contact_type):
contact_list.append(contact)
for args_contact_type in args.contact_type:
if contact_type.startswith(args_contact_type):
contact_list.append(contact)
if contact_list == []:
del results[name]
else:
Expand Down
43 changes: 38 additions & 5 deletions template-resourcegroup.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -52,11 +52,44 @@ Resources:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>

# Site contact (optional) are persons or groups of people (i.e.,
# mailing lists) that are generally responsible for a site's
# relationship with the OSG (e.g., principal investigators,
# local administrator contact for OSG Hosted CEs)
# Site Contact:
# Local contacts are an optional set of persons or groups of
# people (i.e., mailing lists) if they are a different from the
# persons responsible for the resouce. For example, OSG Hosted
# CEs are operated by OSG Staff, who are separate from the local
# contacts that are responsible for running the site's scheduler

# (OPTIONAL) Local Executive Contacts are persons or groups of
# people (i.e., mailing lists) are responsible for making policy
# decisions regarding the site's integration with the OSG resource
# Local Executive Contact:
# Primary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>
# Secondary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>
# Tertiary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>

# (OPTIONAL) Local Operaitonal Contacts are persons or groups of
# people (i.e., mailing lists) that are directly responsible for the
# maintenance of the site's integration with the OSG resource
# Local Operational Contact:
# Primary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>
# Secondary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>
# Tertiary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>

# (OPTIONAL) Local Security Contacts are persons or groups of
# mailing lists) that are responsible for handling security
# issues related to the site's integration with the OSG resource
# Local Security Contact:
# Primary:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>
Expand Down
23 changes: 23 additions & 0 deletions topology/Albert Einstein Institute/GEO 600/SITE.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# LongName is the expanded name of the site
LongName: Albert Einstein Institute, Hannover
# Description is a brief description of your site
Description: GEO 600 data server at AEI Hannover

# AddressLine1 is the street address of the site
AddressLine1: Callinstrasse 38
### AddressLine2 (optional) is the second line of the street address
# AddressLine2: Room 101
# City is the city the site is located in
City: Hannover

# Country is the country the site is located in
Country: Germany
### State (optional) is the state or province the site is located in
# State: Wisconsin
### Zipcode (optional) is the zipcode/postal code of the site
# Zipcode: "01234"

# Latitude is the latitude of the site (positive values are in the northern hemisphere)
Latitude: 52.388
# Longitude is the longitude of the site (positive values are in the eastern hemisphere)
Longitude: 9.711
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,21 @@ Resources:
# Name: <FIRSTNAME> <LASTNAME>
# ID: <EMAIL HASH>

Local Operational Contact:
Primary:
Name: Alan Chapman
ID: OSG1000646
Secondary:
Name: Johnathan Lee
ID: 0cc59bba93ee546d91d24a2c46fe52bbeebd4c63
Local Security Contact:
Primary:
Name: Alan Chapman
ID: OSG1000646
Secondary:
Name: Johnathan Lee
ID: 0cc59bba93ee546d91d24a2c46fe52bbeebd4c63

# Site contact (optional) are persons or groups of people (i.e.,
# mailing lists) that are generally responsible for a site's
# relationship with the OSG (e.g., principal investigators,
Expand Down
Loading

0 comments on commit 20b5b24

Please sign in to comment.