Skip to content

Commit

Permalink
Remove direct calls to "soci create/push" in docs
Browse files Browse the repository at this point in the history 
Signed-off-by: David Son <[email protected]>
  • Loading branch information
@sondavidb
sondavidb committed Oct 6, 2023
1 parent 62d5510 commit a75a377
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 53 deletions.
61 changes: 16 additions & 45 deletions docs/getting-started.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,8 @@ with the SOCI snapshotter.
- [Dependencies](#dependencies)
- [Install the SOCI snapshotter](#install-the-soci-snapshotter)
- [Push an image to your registry](#push-an-image-to-your-registry)
- [Create and push SOCI index](#create-and-push-soci-index)
- [Create SOCI index](#create-soci-index)
- [About the SOCI index](#about-the-soci-index)
- [(Optional) Inspect SOCI index and zTOC](#optional-inspect-soci-index-and-ztoc)
- [Push SOCI index to registry](#push-soci-index-to-registry)
- [Run container with the SOCI snapshotter](#run-container-with-the-soci-snapshotter)
- [Configure containerd](#configure-containerd)
- [Start the SOCI snapshotter](#start-the-soci-snapshotter)
Expand All @@ -34,7 +32,7 @@ to install them on your machine:
- **[containerd](https://github.com/containerd/containerd/blob/main/docs/getting-started.md) >= 1.4** -
required to run the SOCI snapshotter; to confirm please check with `sudo nerdctl system info`.
- **[nerdctl](https://github.com/containerd/nerdctl#install) >= v1.5.0** - required for this doc to interact with containerd/registry. You do not need any of the additional components mentioned in the install documentation for this getting started, but you might if you want complex networking in the future. Please note that SOCI will not work with rootless nerdctl.
- **[nerdctl](https://github.com/containerd/nerdctl#install) >= v1.6.0** - required for this doc to interact with containerd/registry. You do not need any of the additional components mentioned in the install documentation for this getting started, but you might if you want complex networking in the future. Please note that SOCI will not work with rootless nerdctl.
- **fuse** - used for mounting without root access (`sudo yum install fuse` or
other Linux package manager like `apt-get`, depending on your Linux distro).

Expand All @@ -45,6 +43,8 @@ The SOCI project produces 2 binaries:
- `soci`: the CLI tool used to build/manage SOCI indices.
- `soci-snapshotter-grpc`: the daemon (a containerd snapshotter plugin) used for lazy loading.

Note that while the SOCI CLI is never explicitly used, nerdctl (used in this doc) uses it under the hood when given the flag `--snapshotter soci`.

You can download prebuilt binaries from our [release page](https://github.com/awslabs/soci-snapshotter/releases)
or [build them from source](./build.md).

Expand Down Expand Up @@ -98,47 +98,29 @@ echo $REGISTRY_PASSWORD | nerdctl login -u $REGISTRY_USER --password-stdin $REGI
echo $REGISTRY_PASSWORD | sudo nerdctl login -u $REGISTRY_USER --password-stdin $REGISTRY
sudo nerdctl pull docker.io/library/rabbitmq:latest
sudo nerdctl image tag docker.io/library/rabbitmq:latest $REGISTRY/rabbitmq:latest
sudo nerdctl push --platform linux/amd64 $REGISTRY/rabbitmq:latest
sudo nerdctl push --platform linux/amd64 --snapshotter soci $REGISTRY/rabbitmq:latest
```

After this step, please check your registry to confirm the image is present.
You can go to your registry console or use your registry's CLI (e.g. for ECR, you
can use `aws ecr describe-images --repository-name rabbitmq --region $AWS_REGION`).

## Create and push SOCI index

Instead of converting the image format, the SOCI snapshotter uses the SOCI index
associated with an image to implement its lazy loading. For more details
please see [README](../README.md#no-image-conversion).
associated with an image to implement its lazy loading. (For more details
please see [README](../README.md#no-image-conversion).)
Upon pushing with nerdctl, the `--snapshotter soci` flag causes it to
create a SOCI index and manifest before pushing all associated files to the registry
(the original image, the SOCI index, and manifest).

### Create SOCI index

Let's create a SOCI index, which later will be pushed to your registry:

```shell
sudo soci create $REGISTRY/rabbitmq:latest
After this step, please check your registry to confirm the image and SOCI index are present.
You can go to your registry console or use your registry's CLI (e.g. for ECR, you
can use `aws ecr describe-images --repository-name rabbitmq --region $AWS_REGION`).

# output
layer sha256:57315aaee690b22265ebb83b5443587443398a7cd99dd2a43985c28868d34053 -> ztoc skipped
layer sha256:ed46dea0429646ca97e7a90d273159154ab8c28e631f2582d32713e584d98ace -> ztoc skipped
layer sha256:3f0e404c1d688448c1c3947d91d6e0926c67212f4d647369518077513ebdfd91 -> ztoc skipped
layer sha256:626e07084b41a102f8bcedf05172676423d1c37b8391be76eee2d7bbf56ec31e -> ztoc skipped
layer sha256:b49348aba7cfd44d33b07730fd8d3b44ac97d16a268f2d74f7bfb78c4c9d1ff7 -> ztoc skipped
layer sha256:ec66df5c883fd24406c6ef53864970f628b51216e8e1f3f5981c439ed6e4ed41 -> ztoc skipped
layer sha256:8147f1b064ec70039aad0068f71b316b42cf515d2ba87e6668cb66de4f042f5a -> ztoc skipped
layer sha256:f63218e95551afe34f3107b1769a556a3c9a39279cb66979914215e03f4e2754 -> ztoc sha256:ccae6b7217b73ae9caf80bff4c5411dada341739c8b443791fba227b226c61d0
layer sha256:7608715873ec5c02d370e963aa9b19a149023ce218887221d93fe671b3abbf58 -> ztoc sha256:740374aa7cac1764593430843d428a73a30d4a6a0d45fb171c369f3914a638eb
layer sha256:96fb4c28b2c1fc1528bf053e2938d5173990eb12097d51f66c2bb3d01a2c9a39 -> ztoc sha256:dc9a2ca27d2b680279fc8052228772b9c03a779d0b7cc61012d2ad833ad1ff5e
...
```
### About the SOCI index

Behind the scene SOCI created two kinds of objects. One is a series of ztocs
(one per layer). A ztoc is a table of contents for compressed data. The other is
a manifest that relates the ztocs to their corresponding image layers and relates
the entire SOCI index to a particular image manifest (i.e. a particular image for a particular platform).

> We skip building ztocs for smaller layers (controlled by `--min-layer-size` of
> `soci create`) because small layers don't benefit much from lazy loading.)
> We skip building ztocs for smaller layers (controlled by `--soci-min-layer-size` in
> `nerdctl push`) because small layers don't benefit much from lazy loading.)
From the above output, we can see that SOCI creates ztocs for 3 layers and skips
7 layers, which means only the 3 layers with ztocs will be lazily pulled.
Expand Down Expand Up @@ -166,17 +148,6 @@ out the index manifest in json:
sudo soci index info sha256:f5f2a8558d0036c0a316638c5575607c01d1fa1588dbe56c6a5a7253e30ce107
```

### Push SOCI index to registry

Next we need to push the manifest to the registry with the following command.
This will push all of the SOCI related artifacts (index manifest, ztoc):

```shell
sudo soci push --user $REGISTRY_USER:$REGISTRY_PASSWORD $REGISTRY/rabbitmq:latest
```

Credentials here can be omitted if `nerdctl login` has stored credentials for this registry.

## Run container with the SOCI snapshotter

### Configure containerd
Expand Down
11 changes: 6 additions & 5 deletions docs/pull-modes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,9 @@ Overall, lazily pulling a container image with the SOCI snapshotter
<!-- START doctoc generated TOC please keep comment here to allow auto update -->
<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->

- [Pull modes of the SOCI snapshotter](#pull-modes-of-the-soci-snapshotter)
- [Step 1: specify SOCI index digest](#step-1-specify-soci-index-digest)
- [Step 2: fetch SOCI artifacts](#step-2-fetch-soci-artifacts)
- [Step 3: fetch image layers](#step-3-fetch-image-layers)
- [Step 1: specify SOCI index digest](#step-1-specify-soci-index-digest)
- [Step 2: fetch SOCI artifacts](#step-2-fetch-soci-artifacts)
- [Step 3: fetch image layers](#step-3-fetch-image-layers)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand Down Expand Up @@ -68,7 +67,9 @@ as a fuse mountpoint, and will be lazily loaded while a container is running.
> Whether a layer belongs to 1 or 2 depends on its size. When creating a SOCI
> index, SOCI only creates zTOC for layers larger than a given size which is
> specified by the `--min-layer-size` flag of
[`soci create` command](https://github.com/awslabs/soci-snapshotter/blob/9ff88817f3f2635b926f9fd32f6f05f389f7ecee/cmd/soci/commands/create.go#L56).
> [`soci create` command](https://github.com/awslabs/soci-snapshotter/blob/9ff88817f3f2635b926f9fd32f6f05f389f7ecee/cmd/soci/commands/create.go#L56),
> which is used under the hood with the value of `--soci-min-layer-size` flag in
> [`nerdctl push` command](https://github.com/containerd/nerdctl/blob/ff684d5a8c2bd40c459d555b395ded7c80e3f23c/pkg/snapshotterutil/sociutil.go#L65C31-L65C31)
With debug logging enabled, you can see an entry in logs for each layer.
`checking mount point` indicates that the layer will be lazily loaded.
Expand Down
6 changes: 3 additions & 3 deletions docs/registry.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ Below are some slightly redacted examples from the services that don't support t
**Docker Hub**

```
$ sudo ./out/soci push --user $USERNAME:$PASSWORD docker.io/####/busybox:latest
$ $ sudo ./out/soci push --user $USERNAME:$PASSWORD docker.io/####/busybox:latest
checking if a soci index already exists in remote repository...
pushing soci index with digest: sha256:d6ebffd218ead37e4862172b4f19491341e72aebc3cc6d9cf1a22297c40fb3c2
pushing artifact with digest: sha256:cce4c7e12e01b32151d69348fcf52e0db7b44f6df6c23c511fa5c52eaf272c28
Expand All @@ -105,7 +105,7 @@ The index manifest can't be pushed at all.
**Google Cloud Artifact Registry**

```
$ sudo ./out/soci push --user $USERNAME:$PASSWORD us-east1-docker.pkg.dev/####/busybox:latest
$ $ sudo ./out/soci push --user $USERNAME:$PASSWORD us-east1-docker.pkg.dev/####/busybox:latest
checking if a soci index already exists in remote repository...
soci: failed to fetch list of referrers: GET "https://accounts.google.com/v3/signin/identifier?dsh=###&continue=https%3A%2F%2Fconsole.cloud.google.com%2Fartifacts%2Ftags%2Fv2%2Fus-east1%2F####%252Fbusybox%252Freferrers%252Fsha256%2Facaddd9ed544f7baf3373064064a51250b14cfe3ec604d65765a53da5958e5f5...&service=cloudconsole&flowName=WebLiteSignIn&flowEntry=ServiceLogin": failed to decode response: invalid character '<' looking for beginning of value
```
Expand All @@ -117,7 +117,7 @@ Pushing a SOCI index works as long as the SOCI CLI receives the `--existing-inde
**Quay.io**

```
$ sudo ./out/soci push --user $USERNAME:$PASSWORD quay.io/####/busybox:latest
$ $ sudo ./out/soci push --user $USERNAME:$PASSWORD quay.io/####/busybox:latest
checking if a soci index already exists in remote repository...
pushing soci index with digest: sha256:3f2f40d12b70b94e43f17b3840cd0dd850d6ce497f80cee9515fe4f7253d176d
skipped artifact with digest: sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
Expand Down

0 comments on commit a75a377

Please sign in to comment.