From edcac932464a6a47a030a59f4edfc3bbde8bf740 Mon Sep 17 00:00:00 2001 From: Derek Weitzel Date: Fri, 28 Jun 2024 12:34:42 -0500 Subject: [PATCH 1/3] Adding external cvmfs repos docs --- docs/services/adding-external-cvmfs-repos.md | 53 ++++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 docs/services/adding-external-cvmfs-repos.md diff --git a/docs/services/adding-external-cvmfs-repos.md b/docs/services/adding-external-cvmfs-repos.md new file mode 100644 index 00000000..44b449cd --- /dev/null +++ b/docs/services/adding-external-cvmfs-repos.md @@ -0,0 +1,53 @@ +Adding External CVMFS Repos +=========================== + +This document describes how to add an external repo like public-uc.osgstorage.org or ligo.storage.igwn.org. + +## Use OSG Docs +Authoritative source for adding new oasis repo is [Install an OASIS Repository](https://osg-htc.org/docs/data/external-oasis-repos/). Follow instructions starting at heading "Creating a Repository", your first command should be `cvmfs_server mkfs` + +When it asks you to open a ticket, if it's an osgstorage.org or opensciencegrid.org domain, then all you need to do is add the CVMFS repo to topology like: https://github.com/opensciencegrid/topology/pull/3986 + +Once you have completed adding the fetch-cvmfs-whitelist line to cron, you are done with the OSG documentation. + +## Configure the external CVMFS repo + +In `/etc/cvmfs/repositories.d//server.conf`, add the lines: +``` +CVMFS_COMPRESSION_ALGORITHM=none +CVMFS_GARBAGE_COLLECTION=true +CVMFS_AUTO_GC=true +CVMFS_AUTO_GC_TIMESPAN="2 days ago" +CVMFS_EXTERNAL_DATA=true +CVMFS_AUTO_TAG_TIMESPAN="2 weeks ago" +``` +Check in the file for duplicate lines of the above with different settings, comment (`#`) out those lines. + +The imporatant line is `CVMFS_COMPRESSION_ALGORITHM`. If it is set to the default, then CVMFS clients will expect the data to be delivered in compressed format, while the caches will deliver the file in un-compressed format. + +## Configure CVMFS-sync + +`cvmfs-sync` synchonizes the data from an XRootD server (origin) to a CVMFS repo. + +Create a new config file (by copying another existing config) in `/etc/cvmfs-sync`. The name of the configuration should be `.config` + +In the config, you will need to modify the repo, source and destination. This is where cvmfs-sync will scan for new files to add to the CVMFS repo. + +### Make the systemd timer +Copy an existing timer like: +``` +cp -r /etc/systemd/system/cvmfs-data-update@gwosc.osgstorage.org.service.d /etc/systemd/system/cvmfs-data-update@.service.d +``` +You may need to edit the override file in the directory above to change the user. + +Enable the timer: +``` +systemctl enable cvmfs-data-update@public.uc.osgstorage.org.timer +systemctl start cvmfs-data-update@public.uc.osgstorage.org.timer +``` + +## Checking cvmfs-sync +``` +journalctl -u cvmfs-data-update@public.uc.osgstorage.org +``` + From 28b3ca7cd66929b8797ae77ac0c22196794e7873 Mon Sep 17 00:00:00 2001 From: Derek Weitzel Date: Fri, 28 Jun 2024 12:35:32 -0500 Subject: [PATCH 2/3] Add generalized --- docs/services/adding-external-cvmfs-repos.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/services/adding-external-cvmfs-repos.md b/docs/services/adding-external-cvmfs-repos.md index 44b449cd..c1549697 100644 --- a/docs/services/adding-external-cvmfs-repos.md +++ b/docs/services/adding-external-cvmfs-repos.md @@ -42,12 +42,12 @@ You may need to edit the override file in the directory above to change the user Enable the timer: ``` -systemctl enable cvmfs-data-update@public.uc.osgstorage.org.timer -systemctl start cvmfs-data-update@public.uc.osgstorage.org.timer +systemctl enable cvmfs-data-update@.timer +systemctl start cvmfs-data-update@.timer ``` ## Checking cvmfs-sync ``` -journalctl -u cvmfs-data-update@public.uc.osgstorage.org +journalctl -u cvmfs-data-update@ ``` From 589f49c36bb3566b5a3de38e6d3c8498740fab77 Mon Sep 17 00:00:00 2001 From: Derek Weitzel Date: Fri, 28 Jun 2024 12:41:09 -0500 Subject: [PATCH 3/3] Adding the cvmfs repos to index --- mkdocs.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/mkdocs.yml b/mkdocs.yml index c3161e43..ff3e6f36 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -23,6 +23,7 @@ nav: - CE Monitoring Dashboards: 'services/ce-monitoring-dashboards.md' - Yum Repository: - Troubleshooting Guide for Yum Repository Scripts: 'troubleshooting/repository-scripts.md' + - Adding External CVMFS repos: 'services/adding-external-cvmfs-repos.md' - Service Level Agreements: - General: 'SLA/general.md' - Access Point: 'SLA/access-point.md'