Skip to content

Latest commit

 

History

History
171 lines (145 loc) · 7.97 KB

0004-icas.md

File metadata and controls

171 lines (145 loc) · 7.97 KB
Raw

Buildbarn Architecture Decision Record #4: Indirect Content Addressable Storage

Author: Ed Schouten
Date: 2020-06-17

Context

In an ideal build environment, build actions scheduled through the REv2 protocol are fully isolated from the outer world. They either rely on data that is provided by the client (source files and resources declared in Bazel's WORKSPACE file), or use files that are outputs of previous build actions. In addition to improving reproducibility of work, it makes this easier for Buildbarn and the build client to distinguish actual build failures (compiler errors) from infrastructure failures.

One pain point of this model has always been network bandwidth consumption. Buildbarn clusters can be run in data centers that have excellent network connectivity, while build clients may run on a laptop that uses public WiFi connectivity. These clients are still responsible for downloading external artifacts from their upstream location, followed by uploading them into Buildbarn's Content Addressable Storage (CAS).

People within the Bazel community are working on solving this problem by adding a new Remote Asset API. When used, Bazel is capable of sending RPCs to a dedicated service to download external artifacts, optionally extract them and upload them into the CAS. Bazel is then capable of passing the resulting objects by reference to build actions that consume them.

Though it makes a lot of sense to fully ingest remote assets into the CAS in the general case, there are certain workloads where doing this is undesirable. If objects are already stored on a file share or cloud storage bucket close to a Buildbarn cluster, copying these objects into Buildbarn's centralized storage based on LocalBlobAccess may be wasteful. It may be smarter to let workers fetch these assets remotely only as part of the input root population. This ADR proposes changes to Buildbarn to facilitate this.

Changes proposed at the architectural level

Ideally, we would like to extend the CAS to be able to store references to remote assets. A CAS object would then either contain a blob or a reference describing where the asset is actually located. When a worker attempts to read an object from the CAS and receives a reference, it retries the operation against the remote service. This is unfortunately hard to realise for a couple of reasons:

  • None of the Buildbarn code was designed with this in mind. The Buffer interface assumes that every data store is homogeneous, in that all objects stored within have the same type. The CAS may only hold content addressed blobs, while the AC may only contain ActionResult messages.
  • The ByteStream and ContentAddressableStorage protocols have no mechanisms for returning references. This is problematic, because we also use these protocols inside of our clusters.

This is why we propose adding a separate data store that contains these references, called the Indirect Content Addressable Storage (ICAS). When attempting to load an object, a worker will first query the CAS, falling back to loading a reference from the ICAS and following it. References are stored in the form of Protobuf messages.

In terms of semantics, the ICAS can be seen as a mixture between the CAS and the Action Cache (AC). Like the AC, it holds Protobuf messages. Like the CAS, references expand to data that is content addressed and needs checksum validation. It also needs to support efficient bulk existence queries (i.e., FindMissingBlobs()).

Changes proposed at the implementation level

Because the ICAS is similar to both the CAS and AC, we should integrate support for it into bb-storage's pkg/blobstore. This would allow us to reuse most of the existing storage backends, including LocalBlobAccess, MirroredBlobAccess and ShardingBlobAccess. This requires two major changes:

  • The Buffer interface has only been designed with the CAS and the AC as data sources and sinks in mind. In order to support ICAS references as sources, the NewACBuffer*() functions will be replaced with generic NewProtoBuffer*() functions that take arbitrary Protobuf messages. To support ICAS references as sinks, the Buffer.ToActionResult() method will be replaced with generic method Buffer.ToProto().
  • All of the code in pkg/blobstore/configuration is already fairly messy because it needs to distinguish between the CAS and AC. This is only going to get worse once ICAS support is added. Work will be performed to let the generic configuration code call into Blob{Access,Replicator}Creator interfaces. All of the existing backend specific code will be moved into {AC,CAS}Blob{Access,Replicator}Creator implementations.

With those changes in place, it should become easier to add support for arbitrary storage backends; not just the ICAS. To add support for the ICAS, the following classes will be added:

  • ICASStorageType, which will be used by most generic BlobAccess implementations to construct ICAS specific Buffer objects.
  • ICASBlob{Access,Replicator}Creator, which will be used to construct ICAS specific BlobAccess instances from Jsonnet configuration.
  • ICASBlobAccess and IndirectContentAddressableStorageServer, which act as gRPC clients and servers for the ICAS protocol. This protocol will be similar to REv2's ActionCache and ContentAddressableStorage.

The above will only bring in support for the ICAS, but won't add any logic to let workers fall back to reading ICAS entries when objects are absent from the CAS. To realise that, two new implementations of BlobAccess are added:

  • ReferenceExpandingBlobAccess, which effectively converts an ICAS BlobAccess to a CAS BlobAccess. It loads references from the ICAS, followed by loading the referenced object from its target location.
  • ReadFallbackBlobAccess, which which forwards all requests to a primary backend, followed by sending requests to a secondary backend on absence.

Below is an example blobstore configuration that can be used to let workers access both the CAS and ICAS:

{
  contentAddressableStorage: {
    readFallback: {
      primary: {
        // CAS backend goes here.
        grpc: ...,
      },
      secondary: {
        referenceExpanding: {
          // ICAS backend goes here.
          grpc: ...,
        },
      },
    },
  },
  actionCache: {
    // AC backend goes here.
    grpc: ...,
  },
}

How do I integrate this into my environment?

Though care has been taken to make this entire implementation as generic as possible, the inherent problem is that the process of expanding references stored in the ICAS is specific to the environment in which Buildbarn is deployed. We will make sure that the ICAS protocol and ReferenceExpandingBlobAccess have support for some commonly used protocols (e.g., HTTP), but it is not unlikely that operators of Buildbarn clusters need to maintain forks if they want to integrate this with their own infrastructure. This is likely unavoidable.

The question is also how entries are written into the ICAS. This is again specific to the environment. Users will likely need to implement their own gRPC service that implements the Remote Asset API that parses environment specific URL schemes and stores them as ICAS entries.

Unfortunately, Bazel does not implement the Remote Asset API at this point in time. As a workaround, one may create a custom bb_worker-like process that captures certain types of build actions. Instead of going through the regular execution process, this worker directly interprets these requests, treating them as a remote asset request. This can be achieved by implementing a custom BuildExecutor.

Future work

When we added Buffer.ToActionResult(), we discovered that we no longer needed the ActionCache interface. Now that we have a general purpose Buffer.ToProto() function, it looks like the ContentAddressableStorage interface also becomes less useful. It may make sense to investigate what it takes to remove that interface as well.