Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(tools): Instructions for cleaning up the repo #3250

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions dev_docs/REPO_CLEANUP.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Repo Cleanup

This document is a guide on cleaning old blobs from the repo if it get's too big.

## Steps

You can see how big the repo is by running the following command:

```bash
git count-objects -vH
```

If we start getting north of 200MB, we should likely clean the repo.

### Install BFG Repo-Cleaner

[Download BFG Repo-Cleaner](https://rtyley.github.io/bfg-repo-cleaner/) and install java if you don't have it.

```bash
brew install java
```

### Clone the repo

Checkout a bare version of the repo with the following command:

```bash
git clone --mirror https://github.com/wandb/weave.git weave-cleanup.git
```

### Backup the repo

```bash
git bundle create weave-$(date +%Y-%m-%d).bundle --all
```

### Clean the repo

The below command will remove all files larger than 500K and delete the files with the following extensions:

```bash
java -jar bfg.jar --strip-blobs-bigger-than 500K --delete-files "*.{so,pdb,pyx,whl,dat,dylib}" weave-cleanup.git
```

Then gc the repo and see how much space it saved:

```bash
git reflog expire --expire=now --all && git gc --prune=now --aggressive
git count-objects -vH
```

### YOLO

```bash
git push --force --all
```
Loading