Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding external CVMFS repo docs #7

Merged
merged 3 commits into from
Jun 28, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions docs/services/adding-external-cvmfs-repos.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
Adding External CVMFS Repos
===========================

This document describes how to add an external repo like public-uc.osgstorage.org or ligo.storage.igwn.org.

## Use OSG Docs
Authoritative source for adding new oasis repo is [Install an OASIS Repository](https://osg-htc.org/docs/data/external-oasis-repos/). Follow instructions starting at heading "Creating a Repository", your first command should be `cvmfs_server mkfs`

When it asks you to open a ticket, if it's an osgstorage.org or opensciencegrid.org domain, then all you need to do is add the CVMFS repo to topology like: https://github.com/opensciencegrid/topology/pull/3986

Once you have completed adding the fetch-cvmfs-whitelist line to cron, you are done with the OSG documentation.

## Configure the external CVMFS repo

In `/etc/cvmfs/repositories.d/<reponame>/server.conf`, add the lines:
```
CVMFS_COMPRESSION_ALGORITHM=none
CVMFS_GARBAGE_COLLECTION=true
CVMFS_AUTO_GC=true
CVMFS_AUTO_GC_TIMESPAN="2 days ago"
CVMFS_EXTERNAL_DATA=true
CVMFS_AUTO_TAG_TIMESPAN="2 weeks ago"
```
Check in the file for duplicate lines of the above with different settings, comment (`#`) out those lines.

The imporatant line is `CVMFS_COMPRESSION_ALGORITHM`. If it is set to the default, then CVMFS clients will expect the data to be delivered in compressed format, while the caches will deliver the file in un-compressed format.

## Configure CVMFS-sync

`cvmfs-sync` synchonizes the data from an XRootD server (origin) to a CVMFS repo.

Create a new config file (by copying another existing config) in `/etc/cvmfs-sync`. The name of the configuration should be `<reponame>.config`

In the config, you will need to modify the repo, source and destination. This is where cvmfs-sync will scan for new files to add to the CVMFS repo.

### Make the systemd timer
Copy an existing timer like:
```
cp -r /etc/systemd/system/[email protected] /etc/systemd/system/cvmfs-data-update@<reponame>.service.d
```
You may need to edit the override file in the directory above to change the user.

Enable the timer:
```
systemctl enable cvmfs-data-update@<reponame>.timer
systemctl start cvmfs-data-update@<reponame>.timer
```

## Checking cvmfs-sync
```
journalctl -u cvmfs-data-update@<reponame>
```

1 change: 1 addition & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ nav:
- CE Monitoring Dashboards: 'services/ce-monitoring-dashboards.md'
- Yum Repository:
- Troubleshooting Guide for Yum Repository Scripts: 'troubleshooting/repository-scripts.md'
- Adding External CVMFS repos: 'services/adding-external-cvmfs-repos.md'
- Service Level Agreements:
- General: 'SLA/general.md'
- Access Point: 'SLA/access-point.md'
Expand Down
Loading