Run linux repository mirrors with docker/kubernetes
- This implementation is intended to be used in a small organization and cannot be deployed as a public mirror due to the performance reasons.
- Furthermore, because current configuration has been adjusted for @yamaoka-kitaguchi-lab local environment, if you want to deploy this elsewhere, you'll need to read through all the files and modify them accordingly.
As of August 10th, 2020, following seven linux mirrors are available - see Mirroring details for further information.
- http://mirror.intra.net.ict.e.titech.ac.jp/pacman/archlinux
- http://pacman.mirror.intra.net.ict.e.titech.ac.jp/archlinux
Add the below lines to the beginning of your /etc/pacman.d/mirrorlist and then run pacman -Syy for force syncing.
## Tokyo Tech
Server = http://pacman.mirror.intra.net.ict.e.titech.ac.jp/archlinux/$repo/os/$arch
- http://mirror.intra.net.ict.e.titech.ac.jp/pacman/manjaro
- http://pacman.mirror.intra.net.ict.e.titech.ac.jp/manjaro
Add the below lines to the beginning of your /etc/pacman.d/mirrorlist and then run pacman -Syy for force syncing.
## Tokyo Tech
Server = http://pacman.mirror.intra.net.ict.e.titech.ac.jp/manjaro/$repo/os/$arch
For both server and desktop editions of AMD64 and i386 architecture. Only available in 20.04 LTS (focal) and 18.04 LTS (bionic).
- http://mirror.intra.net.ict.e.titech.ac.jp/apt/ubuntu
- http://apt.mirror.intra.net.ict.e.titech.ac.jp/ubuntu
Use the following command to replace the repository URLs from Japanese official archive to the local mirror.
% sed -i.bak -e "s%jp.archive.ubuntu.com%apt.mirror.intra.net.ict.e.titech.ac.jp%g" /etc/apt/sources.list
Testing and stable release of 11 (bullseye), 10 (buster), and 9 (strech) are available.
- http://mirror.intra.net.ict.e.titech.ac.jp/apt/debian
- http://apt.mirror.intra.net.ict.e.titech.ac.jp/debian
Use the following command to replace the repository URLs from Japanese official archive to the local mirror.
% sed -i.bak -e "s%ftp.jp.debian.org%apt.mirror.intra.net.ict.e.titech.ac.jp%g" /etc/apt/sources.list
- http://mirror.intra.net.ict.e.titech.ac.jp/apt/raspbian
- http://apt.mirror.intra.net.ict.e.titech.ac.jp/raspbian
Use the following command to replace the repository URLs from the official archive to the local mirror.
% sed -i.bak -e "s%raspbian.raspberrypi.org%apt.mirror.intra.net.ict.e.titech.ac.jp%g" /etc/apt/sources.list
This repository is for lab's external router FS N5850-48S6Q and is not intended to be used on any other machine.
- http://mirror.intra.net.ict.e.titech.ac.jp/apt/cumulus
- http://apt.mirror.intra.net.ict.e.titech.ac.jp/cumulus
Use the following command to replace the repository URLs from the official archive to the local mirror - also refer to the official documentation as appropriate.
% sed -i.bak -e "s%repo3.cumulusnetworks.com/repo%apt.mirror.intra.net.ict.e.titech.ac.jp/cumulus%g" /etc/apt/sources.list
- http://mirror.intra.net.ict.e.titech.ac.jp/portage/gentoo
- http://portage.mirror.intra.net.ict.e.titech.ac.jp/gentoo
Check and follow the instructions of official handbook.
There are two ways to deploy; both ways are very simple and easy, but if you're not familiar with kubernetes, deploying with docker-compose is recommended.
NOTE: You need at least 2.5TB of free disk space to deploy all mirrors. The initial synchronization may take a few days because of the bandwidth limitation on upstream servers.
If you don't have docker-compose (1.26.0 or higher), get it first. All commands must be executed as root (use sudo).
% make install-compose
Then deploy mirrors on your system with:
% make install
It's done. All mirrors are automatically synchronized and kept up to date by systemd/Timers. Of course, you can initiate git updates or mirror synchronization anytime you want. The following commands may help you:
% make update # Update git repository and rebuild docker images
% make sync # Sync all mirrors with their upstream servers
% make publish # Publish mirrors (start Web server)
% make log # Show syncing logs
Type make uninstall to unregister automatic mirror syncing, and type make destroy to purge all of local repositories.
Prior to the deployment, install and setup microk8s, then prepare PVs. Check hint if necessary.
Add chart repository.
% microk8s helm3 repo add linux-repository-mirror https://yamaoka-kitaguchi-lab.github.io/linux-repository-mirror/chart
% microk8s helm3 repo list
NAME URL
linux-repository-mirror https://yamaoka-kitaguchi-lab.github.io/linux-repository-mirror/chart
Instead of the above commands, you can clone this repository and move directory to chart.
% git clone --depth 1 https://github.com/yamaoka-kitaguchi-lab/linux-repository-mirror
% cd linux-repository-mirror/chart
Deploy mirrors. All mirrors are automatically synchronized and kept up to date by CronJob.
% microk8s helm3 install linux-repository-mirror linux-repository-mirror
% microk8s kubectl get all,pv,pvc --namespace default
Each repository size is as of June 20, 2020. ALL means that the local mirror is a complete replication of the upper server.
| Distribution | Upstream Server | Syncing Repositories | Update Interval | Size |
|---|---|---|---|---|
| ArchLinux | jpn.mirror.pkgbuild.com | ALL (only 64bit binaries) | every 20 min. | 96GB |
| Manjaro | ftp.tsukuba.wide.ad.jp | ALL (only 64bit binaries) | every 20 min. | 322GB |
| Ubuntu | • jp.archive.ubuntu.com • ftp.jaist.ac.jp (for i18n support) |
20.04 LTS, 18.04 LTS | every 20 min. | 419GB |
| Debian | • ftp.jp.debian.org • hanzubon.jp (for i18n support) |
testing, 11, 10, 9 | every 20 min. | 365GB |
| Raspbian | archive.raspbian.org | ALL | every 20 min. | 228GB |
| Cumulus | repo3.cumulusnetworks.com | w/o early-access | every 20 min. | - |
| Gentoo | • rsync.jp.gentoo.org • ftp.iij.ad.jp (for source) |
ALL (w/ distfiles and packages) |
every day | 819GB |
Why don't you set the lab mirror as your upstream server. You can download packages without annoying bandwidth limitation. This is especially useful for the initial synchronization, which requires a lot of data transfer.
- Add FreeBSD repository mirror (Of course BSD is not Linux, though) (#1)
- Allow rsync client to pull packages from this local mirror (#2)
- ArchLinux repository has not been updated due to the upper mirror failure (#4)
- I18n support of Debian has been freezed due to the upper mirror failure (#5)
- Horizontal Pod Autoscaler doesn't work with microk8s (#3)
Releases are managed in CalVer scheme and it is independent of chart versioning. For the versions available, see the tags on this repository.
This project is licensed under the MIT License - see the LICENSE file for details.