Docker images: Are integrations still building?

[symgryph]

My own thought is either host the docker binary images here, or simply let users build them on their own. Its not too hard to do a 'docker build -t shiori:latest' from one's workstation.

The problem with hosting the images is 'keeping them up to date' since old ones tend to go stale fast. If we just edit the docker file appropriately, people can build their own versions, saving on transfer/whatnot cost.

Automated pipelines, however, might be an answer to this, as well as possibly a nice docker-compose file. I could work on this with someone if they were interested.

Thomas

[fmartingr]

I have been playing around with this. It seems that dockerhub team plans is now a paid service, so that's out of the question here. I'm also reluctant to create a "go-shiori" user and have control. The easiest way is to host the images here on github's container registry.

I have pushed the current master build and it's pretty easy and supposedly anyone can download from here. The problem is storage, teams come with a 500MB limit on storage (and 1GB bandwitdh) for packages (and currently the image size is about 300MB+, but I've tagged it latest so we can overwrite it with a lighter one) so that's the first issue to solve here, and even with that I'm not sure if we should publish every version or just the n latests to save on space. And then there's the problem with bandwidth.

I believe the best PR to tackle this is #278 by @n8225. Is very complete and have a lot of thought into it (and will help closing other related issues and PRs) but it will require some changes if we decide to use github container registry. From that I'm also worried it we need so many CI steps, ideally we should test with the current version in the go.mod and the +1. The builds could also be done using GoReleaser.

I'm going to comment on that PR if that's something @n8225 is willing to update (or remove) and we could take it from there.

[n8225]

I just created a pull request #341 this will slim down the resulting image quite a bit. The previous version was using a golang image as the final base, which isn't needed. This PR is also using ghcr.io images as well. The resulting image is only about ~25 MB as the alpine image it is using as a base is already cached on ghcr.io. I couldn't find a way to check storage usage on ghcr.io to verify, so I could be wrong on that. Alternatives for container storage: gitlab (5GB total storage), quay.io (unlimited for public repo).

I think every merge should be published as dev(overwriting previous) as well as every release(latest and version, keeping 1-2 most recent versions). All releases could be archived on quay.io or another registry. Depending on the number of supported architectures the overhead shouldn't be to bad as long as the base image is the same(each version of that architecture essentially only adding the size of the binary due to layering).

[symgryph]

I was able to get the docker image to binary size+6or so MB. I could ressurect this. We could even do a 'from scratch' image and get it down to ONLY binary and some minimum things to function. I suspect we could get it to binary size (with UPX its compresses nicely) 30 or 40MB.

[symgryph]

https://github.com/go-shiori/shiori/pull/278/files looks like the other fellow and I already did this (from scratch that is). With the addition of a 'docker-compose' we could have 1 stop shopping. Personally, I don't think its such a horror for people to build themselves. Amazon also offers public repos for free afaik as well.

[symgryph]

https://aws.amazon.com/blogs/aws/amazon-ecr-public-a-new-public-container-registry/

[symgryph]

Crazy idea: We simply make a container 'tarball' that people can reconstitute. Then we can use 'best' compression with zstandard or somesuch. Might be a bit of overkill, but would be interesting. I think that with minimal upx of the binary with --best' would yield a reasonably non-smallish container that would resist compression anyways.....Might be fun to see what our sizes look like.

[n8225]

Not that crazy of an idea, a little non standard though.
https://docs.podman.io/en/latest/markdown/podman-save.1.html
https://docs.docker.com/engine/reference/commandline/save/

[n8225]

Here are my thoughts on CI/CD implementation. Please provide suggestions/input on this.

• Comment out or remove tests until project has unit tests(zero currently)

• Every PR

  • golangci-lint(a lot of options here)

• Every push to master/main:

  • golangci-lint
  • build binary
  • Build and release dev docker image(replacing previous dev release)

• Every tagged release:

  • Build and release binaries
  • Build and release docker image tagged latest and ver
  • Remove oldest tagged docker image, keeping x previous versions

• Supported Platforms

  • Linux: amd64, aarch64, armv7
  • Win: amd64
  • OSX: amd64, arm64

• Docker images:

  • Utilize docker manifest for supported platforms
  • Scratch and alpine (root or non-root)

Using goreleaser will be more involved due to cgo.
Look into https://gitlab.com/cznic/sqlite (compatibility, binary size, performance...)

[fmartingr]

Hey, again, thanks a lot for this! I agree with most of your point and made some notes in some of them:

Here are my thoughts on CI/CD implementation. Please provide suggestions/input on this.

• Comment out or remove tests until project has unit tests(zero currently)

This makes sense.

• Every PR

  • golangci-lint(a lot of options here)

👍

• Every push to master/main:

  • golangci-lint
  • build binary
  • Build and release dev docker image(replacing previous dev release)

Linting shouldn't be necessary since we are already linting PRs (and I shouldnt push directly), the rest 👍

• Every tagged release:

  • Build and release binaries
  • Build and release docker image tagged latest and ver
  • Remove oldest tagged docker image, keeping x previous versions

👍

• Supported Platforms

  • Linux: amd64, aarch64, armv7
  • Win: amd64
  • OSX: amd64, arm64

👍 This are the already provided builds + OSX + armv7, so if it's doable it's fine by me.

As a note on this, we need to have in mind that the organization has limited action minutes (2000).

• Docker images:

  • Utilize docker manifest for supported platforms
  • Scratch and alpine (root or non-root)

👍 non-root if it isn't much work.

Using goreleaser will be more involved due to cgo. Look into https://gitlab.com/cznic/sqlite (compatibility, binary size, performance...)

I believe goreleaser has some helpers to builds project with CGO, but I can also take a look at replacing the current sqlite implementation if it speeds up working with builds. I mentioned goreleaser because it can also easily pack OS packages (deb, rpm, apk), AUR packages, etc.