CHANGELOG.md

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

Unreleased

Security

• A security flaw was found in the OCI image-spec, where it is possible to cause a blob with one media-type to be interpreted as a different media-type. As umoci is not a registry nor does it handle signatures, this vulnerability had no real impact on umoci but for safety we implemented the now-recommended media-type embedding and verification. CVE-2021-41190

Changes

• In this release, the primary development branch was renamed to main.
• The runtime-spec version of the config.json version we generate is no longer hard-coded to 1.0.0. We now use the version of the spec we have imported (with any -dev suffix stripped, as such a prefix causes havoc with verification tools -- ideally we would only ever use released versions of the spec but that's not always possible). #452
• Add the cgroup namespace to the default configuration generated by umoci unpack to make sure that our configuration plays nicely with runc when on cgroupv2 systems.

Fixed

• In 0.4.7, a performance regression was introduced as part of the VerifiedReadCloser hardening work (to read all trailing bytes) which would cause walk operations on images to hash every blob in the image (even blobs which we couldn't parse and thus couldn't recurse into). To resolve this, we no longer recurse into unparseable blobs. #373 #375 #394
• Handle EINTR on io.Copy operations. Newer Go versions have added more opportunistic pre-emption which can cause EINTR errors in io paths that didn't occur before. #437
• Quite a few changes were made to CI to try to avoid issues with fragility. #452
• umoci will now return an explicit error if you pass invalid uid or gid values to --uid-map and --gid-map rather than silently truncating the value.

0.4.7 - 2021-04-05

Security

• A security flaw was found in umoci, and has been fixed in this release. If umoci was used to unpack a malicious image (using either umoci unpack or umoci raw unpack) that contained a symlink entry for /., umoci would apply subsequent layers to the target of the symlink (resolved on the host filesystem). This means that if you ran umoci as root, a malicious image could overwrite any file on the system (assuming you didn't have any other access control restrictions). CVE-2021-29136

Added

• umoci now compiles on FreeBSD and appears to work, with the notable limitation that it currently refuses to extract non-Linux images on any platform (this will be fixed in a future release -- see #364). #357
• Initial fuzzer implementations for oss-fuzz. #365

Changed

• umoci will now read all trailing data from image layers, to combat the existence of some image generators that appear to append NUL bytes to the end of the gzip stream (which would previously cause checksum failures because we didn't read nor checksum the trailing junk bytes). However, umoci will still not read past the descriptor length. #360
• umoci now ignores all overlayfs xattrs during unpack and repack operations, to avoid causing issues when packing a raw overlayfs directory. #354
• Changes to the (still-internal) APIs to allow for users to use umoci more effectively as a library.
  • The garbage collection API now supports custom GC policies. #338
  • The mutate API now returns information about what layers were added by the operation. #344
  • The mutate API now supports custom compression, and has in-tree support for zstd. #348 #350
  • Support overlayfs-style whiteouts during unpack and repack. #342

0.4.6 - 2020-06-24

umoci has been adopted by the Open Container Initative as a reference implementation of the OCI Image Specification. This will have little impact on the roadmap or scope of umoci, but it does further solidify umoci as a useful piece of "boring container infrastructure" that can be used to build larger systems.

Changed

• As part of the adoption procedure, the import path and module name of umoci has changed from github.com/openSUSE/umoci to github.com/opencontainers/umoci. This means that users of our (still unstable) Go API will have to change their import paths in order to update to newer versions of umoci.

  The old GitHub project will contain a snapshot of v0.4.5 with a few minor changes to the readme that explain the situation. Go projects which import the archived project will receive build warnings that explain the need to update their import paths.

Added

• umoci now builds on MacOS, and we currently run the unit tests on MacOS to hopefully catch core regressions (in the future we will get the integration tests running to catch more possible regressions). #318

Fixed

• Suppress repeated xattr warnings on destination filesystems that do not support xattrs. #311
• Work around a long-standing issue in our command-line parsing library (see urfave/cli#1152) by disabling argument re-ordering for umoci config, which often takes --prefixed flag arguments. #328

0.4.5 - 2019-12-04

Added

• Expose umoci subcommands as part of the API, so they can be used by other Go projects. #289
• Add extensible hooking to the core libraries in umoci, to allow for third-party media-types to be treated just like first-party ones (the key difference is the introspection and parsing logic). #299 #307

Fixed

• Use type: bind for generated config.json bind-mounts. While this doesn't make too much sense (see opencontainers/runc#2035), it does mean that rootless containers work properly with newer runc releases (which appear to have regressed when handling file-based bind-mounts with a "bad" type). #294 #295
• Don't insert a new layer if there is no diff. #293
• Only output a warning if forbidden extended attributes are present inside the tar archive -- otherwise we fail on certain (completely broken) Docker images. #304

0.4.4 - 2019-01-30

Added

• Full-stack verification of blob hashes and descriptor sizes is now done on all operations, improving our hardening against bad blobs (we already did some verification of layer DiffIDs but this is far more thorough). #278 #280 #282

0.4.3 - 2018-11-11

Added

• All umoci commands that had --history.* options can now decide to omit a history entry with --no-history. Note that while this is supported for commands that create layers (umoci repack, umoci insert, and umoci raw add-layer) it is not recommended to use it for those commands since it can cause other tools to become confused when inspecting the image history. The primary usecase is to allow umoci config --no-history to leave no traces in the history. See OSInside/kiwi#871. #270
• umoci insert now has a --tag option that allows you to non-destructively insert files into an image. The semantics match umoci config --tag. #273

0.4.2 - 2018-09-11

Added

• umoci now has an exposed Go API. At the moment it's unclear whether it will be changed significantly, but at the least now users can use umoci-as-a-library in a fairly sane way. #245
• Added umoci unpack --keep-dirlinks (in the same vein as rsync's flag with the same name) which allows layers that contain entries which have a symlink as a path component. #246
• umoci insert now supports whiteouts in two significant ways. You can use --whiteout to "insert" a deletion of a given path, while you can use --opaque to replace a directory by adding an opaque whiteout (the default behaviour causes the old and new directories to be merged). #257

Fixed

• Docker has changed how they handle whiteouts for non-existent files. The specification is loose on this (and in umoci we've always been liberal with whiteout generation -- to avoid cases where someone was confused we didn't have a whiteout for every entry). But now that they have deviated from the spec, in the interest of playing nice, we can just follow their new restriction (even though it is not supported by the spec). This also makes our layers slightly smaller. #254
• umoci unpack now no longer erases system.nfs4_acl and also has some more sophisticated handling of forbidden xattrs. #252 #248
• umoci unpack now appears to work correctly on SELinux-enabled systems (previously we had various issues where umoci wouldn't like it when it was trying to ensure the filesystem was reproducibly generated and SELinux xattrs would act strangely). To fix this, now umoci unpack will only cause errors if it has been asked to change a forbidden xattr to a value different than it's current on-disk value. #235 #259

0.4.1 - 2018-08-16

Added

• The number of possible tags that are now valid with umoci subcommands has increased significantly due to an expansion in the specification of the format of the ref.name annotation. To quote the specification, the following is the EBNF of valid refname values. #234

  refname   ::= component ("/" component)*
  component ::= alphanum (separator alphanum)*
  alphanum  ::= [A-Za-z0-9]+
  separator ::= [-._:@+] | "--"
  

• A new umoci insert subcommand which adds a given file to a path inside the container. #237
• A new umoci raw unpack subcommand in order to allow users to unpack images without needing a configuration or any of the manifest generation. #239
• umoci how has a logo. Thanks to Max Bailey for contributing this to the project. #165 #249

Fixed

• umoci unpack now handles out-of-order regular whiteouts correctly (though this ordering is not recommended by the spec -- nor is it required). This is an extension of #229 that was missed during review. #232
• umoci unpack and umoci repack now make use of a far more optimised gzip compression library. In some benchmarks this has resulted in umoci repack speedups of up to 3x (though of course, you should do your own benchmarks). umoci unpack unfortunately doesn't have as significant of a performance improvement, due to the nature of gzip decompression (in future we may switch to zlib wrappers). #225 #233

0.4.0 - 2018-03-10

Added

• umoci repack now supports --refresh-bundle which will update the OCI bundle's metadata (mtree and umoci-specific manifests) after packing the image tag. This means that the bundle can be used as a base layer for future diffs without needing to unpack the image again. #196
• Added a website, and reworked the documentation to be better structured. You can visit the website at umo.ci. #188
• Added support for the user.rootlesscontainers specification, which allows for persistent on-disk emulation of chown(2) inside rootless containers. This implementation is interoperable with @AkihiroSuda's PRoot fork (though we do not test its interoperability at the moment) as both tools use the same protobuf specification. #227
• umoci unpack now has support for opaque whiteouts (whiteouts which remove all children of a directory in the lower layer), though umoci repack does not currently have support for generating them. While this is technically a spec requirement, through testing we've never encountered an actual user of these whiteouts. #224 #229
• umoci unpack will now use some rootless tricks inside user namespaces for operations that are known to fail (such as mknod(2)) while other operations will be carried out as normal (such as lchown(2)). It should be noted that the /proc/self/uid_map checking we do can be tricked into not detecting user namespaces, but you would need to be trying to break it on purpose. #171 #230

Fixed

• Fix a bug in our "parent directory restore" code, which is responsible for ensuring that the mtime and other similar properties of a directory are not modified by extraction inside said directory. The bug would manifest as xattrs not being restored properly in certain edge-cases (which we incidentally hit in a test-case). #161 #162
• umoci unpack will now "clean up" the bundle generated if an error occurs during unpacking. Previously this didn't happen, which made cleaning up the responsibility of the caller (which was quite difficult if you were unprivileged). This is a breaking change, but is in the error path so it's not critical. #174 #187
• umoci gc now will no longer remove unknown files and directories that aren't flock(2)ed, thus ensuring that any possible OCI image-spec extensions or other users of an image being operated on will no longer break. #198
• umoci unpack --rootless will now correctly handle regular file unpacking when overwriting a file that umoci doesn't have write access to. In addition, the semantics of pre-existing hardlinks to a clobbered file are clarified (the hard-links will not refer to the new layer's inode). #222 #223

0.3.1 - 2017-10-04

Fixed

• Fix several minor bugs in hack/release.sh that caused the release artefacts to not match the intended style, as well as making it more generic so other projects can use it. #155 #163
• A recent configuration issue caused go vet and go lint to not run as part of our CI jobs. This means that some of the information submitted as part of CII best practices badging was not accurate. This has been corrected, and after review we concluded that only stylistic issues were discovered by static analysis. #158
• 32-bit unit test builds were broken in a refactor in 0.3.0. This has been fixed, and we've added tests to our CI to ensure that something like this won't go unnoticed in the future. #157
• umoci unpack would not correctly preserve set{uid,gid} bits. While this would not cause issues when building an image (as we only create a manifest of the final extracted rootfs), it would cause issues for other users of umoci. #166 #169
• Updated to v0.4.1 of go-mtree, which fixes several minor bugs with manifest generation. #176
• umoci unpack would not handle "weird" tar archive layers previously (it would error out with DiffID errors). While this wouldn't cause issues for layers generated using Go's archive/tar implementation, it would cause issues for GNU gzip and other such tools. #178 #179

Changed

• umoci unpack's mapping options (--uid-map and --gid-map) have had an interface change, to better match the user_namespaces(7) interfaces. Note that this is a breaking change, but the workaround is to switch to the trivially different (but now more consistent) format. #167

Security

• umoci unpack used to create the bundle and rootfs with world read-and-execute permissions by default. This could potentially result in an unsafe rootfs (containing dangerous setuid binaries for instance) being accessible by an unprivileged user. This has been fixed by always setting the mode of the bundle to 0700, which requires a user to explicitly work around this basic protection. This scenario was documented in our security documentation previously, but has now been fixed. #181 #182

0.3.0 - 2017-07-20

Added

• umoci now passes all of the requirements for the CII best practices bading program. #134
• umoci also now has more extensive architecture, quick-start and roadmap documentation. #134
• umoci now supports 1.0.0 of the OCI image specification and 1.0.0 of the OCI runtime specification, which are the first milestone release. Note that there are still some remaining UX issues with --image and other parts of umoci which may be subject to change in future versions. In particular, this update of the specification now means that images may have ambiguous tags. umoci will warn you if an operation may have an ambiguous result, but we plan to improve this functionality far more in the future. #133 #142
• umoci also now supports more complicated descriptor walk structures, and also handles mutation of such structures more sanely. At the moment, this functionality has not been used "in the wild" and umoci doesn't have the UX to create such structures (yet) but these will be implemented in future versions. #145
• umoci repack now supports --mask-path to ignore changes in the rootfs that are in a child of at least one of the provided masks when generating new layers. #127

Changed

• Error messages from github.com/opencontainers/umoci/oci/cas/drivers/dir actually make sense now. #121
• umoci unpack now generates config.json blobs according to the still proposed OCI image specification conversion document. #120
• umoci repack also now automatically adding Config.Volumes from the image configuration to the set of masked paths. This matches recently added recommendations by the spec, but is a backwards-incompatible change because the new default is that Config.Volumes will be masked. If you wish to retain the old semantics, use --no-mask-volumes (though make sure to be aware of the reasoning behind Config.Volume masking). #127
• umoci now uses SecureJoin rather than a patched version of FollowSymlinkInScope. The two implementations are roughly equivalent, but SecureJoin has a nicer API and is maintained as a separate project.
• Switched to using golang.org/x/sys/unix over syscall where possible, which makes the codebase significantly cleaner. #141

0.2.1 - 2017-04-12

Added

• hack/release.sh automates the process of generating all of the published artefacts for releases. The new script also generates signed source code archives. #116

Changed

• umoci now outputs configurations that are compliant with v1.0.0-rc5 of the OCI runtime-spec. This means that now you can use runc v1.0.0-rc3 with umoci (and rootless containers should work out of the box if you use a development build of runc). #114
• umoci unpack no longer adds a dummy linux.seccomp entry, and instead just sets it to null. #114

0.2.0 - 2017-04-11

Added

• umoci now has some automated scripts for generated RPMs that are used in openSUSE to automatically submit packages to OBS. #101
• --clear=config.{cmd,entrypoint} is now supported. While this interface is a bit weird (cmd and entrypoint aren't treated atomically) this makes the UX more consistent while we come up with a better cmd and entrypoint UX. #107
• New subcommand: umoci raw runtime-config. It generates the runtime-spec config.json for a particular image without also unpacking the root filesystem, allowing for users of umoci that are regularly parsing config.json without caring about the root filesystem to be more efficient. However, a downside of this approach is that some image-spec fields (Config.User) require a root filesystem in order to make sense, which is why this command is hidden under the umoci-raw(1) subcommand (to make sure only users that understand what they're doing use it). #110

Changed

• umoci's oci/cas and oci/config libraries have been massively refactored and rewritten, to allow for third-parties to use the OCI libraries. The plan is for these to eventually become part of an OCI project. #90
• The oci/cas interface has been modifed to switch from *ispec.Descriptor to ispec.Descriptor. This is a breaking, but fairly insignificant, change. #89

Fixed

• umoci now uses an updated version of go-mtree, which has a complete rewrite of Vis and Unvis. The rewrite ensures that unicode handling is handled in a far more consistent and sane way. #88
• umoci used to set process.user.additionalGids to the "normal value" when unpacking an image in rootless mode, causing issues when trying to actually run said bundle with runC. #109

0.1.0 - 2017-02-11

Added

• CHANGELOG.md has now been added. #76

Changed

• umoci now supports v1.0.0-rc4 images, which has made fairly minimal changes to the schema (mainly related to mediaTypes). While this change is backwards compatible (several fields were removed from the schema, but the specification allows for "additional fields"), tools using older versions of the specification may fail to operate on newer OCI images. There was no UX change associated with this update.

Fixed

• umoci tag would fail to clobber existing tags, which was in contrast to how the rest of the tag clobbering commands operated. This has been fixed and is now consistent with the other commands. #78
• umoci repack now can correctly handle unicode-encoded filenames, allowing the creation of containers that have oddly named files. This required fixes to go-mtree (where the issue was). #80

0.0.0 - 2017-02-07

Added

• Unit tests are massively expanded, as well as the integration tests. #68 #69
• Full coverage profiles (unit+integration) are generated to get all information about how much code is tested. #68 #69

Fixed

• Static compilation now works properly. #64
• 32-bit architecture builds are fixed. #70

Changed

• Unit tests can now be run inside %check of an rpmbuild script, allowing for proper testing. #65.
• The logging output has been cleaned up to be much nicer for end-users to read. #73
• Project has been moved to an openSUSE project. #75

0.0.0-rc3 - 2016-12-19

Added

• unpack, repack: xattr support which also handles security.selinux.* difficulties. #49 #52
• config, unpack: Ensure that environment variables are not duplicated in the extracted or stored configurations. #30
• Add support for read-only CAS operations for read-only filesystems. #47
• Add some helpful output about --rootless if umoci fails with EPERM.
• Enable stack traces with errors if the --debug flag was given to umoci. This requires a patch to pkg/errors.

Changed

• gc: Garbage collection now also garbage collects temporary directories. #17
• Clean-ups to vendoring of go-mtree so that it's much more upstream-friendly.

0.0.0-rc2 - 2016-12-12

Added

• unpack, repack: Support for rootless unpacking and repacking. #26
• unpack, repack: UID and GID mapping when unpacking and repacking. #26
• tag, rm, ls: Tag modification commands such as umoci tag, umoci rm and umoci ls. #6 #27
• stat: Output information about an image. Currently only shows the history information. Only the JSON output is stable. #38
• init, new: New commands have been created to allow for image creation from scratch. #5 #42
• gc: Garbage collection of images. #6
• Full integration and unit testing, with OCI validation to ensure that we always create valid images. #12

Changed

• unpack, repack: Create history entries automatically (with options to modify the entries). #36
• unpack: Store information about its source to ensure consistency when doing a repack. #14
• The --image and --from arguments have been combined into a single <path>[:<tag>] argument for --image. #39
• unpack: Configuration annotations are now extracted, though there are still some discussions happening upstream about the correct way of doing this. #43

Fixed

• repack: Errors encountered during generation of delta layers are now correctly propagated. #33
• unpack: Hardlinks are now extracted as real hardlinks. #25

Security

• unpack, repack: Symlinks are now correctly resolved inside the unpacked rootfs. #27

0.0.0-rc1 - 2016-11-10

Added

• Proof of concept with major functionality implemented.
  • unpack
  • repack
  • config