drone-cache

A Drone plugin for caching current workspace files between builds to reduce your build times. drone-cache is a small CLI program, written in Go without any external OS dependencies (such as tar, etc).

With drone-cache, you can provide your own cache key templates, specify archive format (tar, tar.gz, etc) and you can use popular object storage as storage for your cached files, even better you can implement your custom storage backend to cover your use case.

For detailed usage information and a list of available options please take a look at usage and examples. If you want to learn more about custom cache keys, see cache key templates.

If you want to learn more about the story behind drone-cache, you can read our blogpost Making Drone Builds 10 Times Faster!!

Supported Storage Backends

• AWS S3
  • Configuration
  • Example
  • Other AWS API compatible stores:
    • Minio
    • Red Hat Ceph
    • IBM Object Store
    • and many many others
• Azure Storage
  • Configuration
  • Example
• Google Cloud Storage
  • Configuration
  • Example
• or any mounted local volume
  • Configuration
  • Example

How does it work

drone-cache stores mounted directories and files under a key at the specified backend (by default S3).

Use this plugin to cache data that makes your builds faster. In the case of a cache miss or an empty cache restore it will fail silently in won't break your running pipeline.

The best example would be to use this with your package managers such as Mix, Bundler or Maven. After your initial download, you can build a cache and then you can restore that cache in your next build.

With restored dependencies from a cache, commands like mix deps.get will only need to download new dependencies, rather than re-download every package on each build.

Example Usage of drone-cache

The following example configuration file (.drone.yml) shows the most common use of drone-cache.

Simple (with AWS S3 backend)

kind: pipeline
name: default

steps:
  - name: restore-cache
    image: meltwater/drone-cache
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: aws_access_key_id
      AWS_SECRET_ACCESS_KEY:
        from_secret: aws_secret_access_key
    pull: true
    settings:
      restore: true
      cache_key: {{ .Commit.Branch }}-{{ checksum "go.mod" }} # default if ommitted is {{ .Commit.Branch }}
      bucket: drone-cache-bucket
      region: eu-west-1
      mount:
        - 'vendor'

  - name: build
    image: golang:1.14.4-alpine3.12
    pull: true
    commands:
      - apk add --update make git
      - make drone-cache

  - name: rebuild-cache
    image: meltwater/drone-cache
    pull: true
    environment:
      AWS_ACCESS_KEY_ID:
        from_secret: aws_access_key_id
      AWS_SECRET_ACCESS_KEY:
        from_secret: aws_secret_access_key
    settings:
      rebuild: true
      cache_key: {{ .Commit.Branch }}-{{ checksum "go.mod" }} # default if ommitted is {{ .Commit.Branch }}
      bucket: drone-cache-bucket
      region: eu-west-1
      mount:
        - 'vendor'

More Examples

• examples for Drone, see docs/examples/drone-1.0.md

Usage

Using executable (with CLI args)

NAME:
   Drone cache plugin - Drone cache plugin

USAGE:
   drone-cache [global options] command [command options] [arguments...]

VERSION:
   v1.1.0

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --log.level value                     log filtering level. ('error', 'warn', 'info', 'debug') (default: "info") [$PLUGIN_LOG_LEVEL, $LOG_LEVEL]
   --log.format value                    log format to use. ('logfmt', 'json') (default: "logfmt") [$PLUGIN_LOG_FORMAT, $LOG_FORMAT]
   --repo.fullname value                 repository full name [$DRONE_REPO]
   --repo.namespace value                repository namespace [$DRONE_REPO_NAMESPACE]
   --repo.owner value                    repository owner (for Drone version < 1.0) [$DRONE_REPO_OWNER]
   --repo.name value                     repository name [$DRONE_REPO_NAME]
   --repo.link value                     repository link [$DRONE_REPO_LINK]
   --repo.avatar value                   repository avatar [$DRONE_REPO_AVATAR]
   --repo.branch value                   repository default branch [$DRONE_REPO_BRANCH]
   --repo.private                        repository is private (default: false) [$DRONE_REPO_PRIVATE]
   --repo.trusted                        repository is trusted (default: false) [$DRONE_REPO_TRUSTED]
   --remote.url value                    git remote url [$DRONE_REMOTE_URL]
   --commit.sha value                    git commit sha [$DRONE_COMMIT_SHA]
   --commit.ref value                    git commit ref (default: "refs/heads/master") [$DRONE_COMMIT_REF]
   --commit.branch value                 git commit branch (default: "master") [$DRONE_COMMIT_BRANCH]
   --commit.message value                git commit message [$DRONE_COMMIT_MESSAGE]
   --commit.link value                   git commit link [$DRONE_COMMIT_LINK]
   --commit.author.name value            git author name [$DRONE_COMMIT_AUTHOR]
   --commit.author.email value           git author email [$DRONE_COMMIT_AUTHOR_EMAIL]
   --commit.author.avatar value          git author avatar [$DRONE_COMMIT_AUTHOR_AVATAR]
   --build.event value                   build event (default: "push") [$DRONE_BUILD_EVENT]
   --build.number value                  build number (default: 0) [$DRONE_BUILD_NUMBER]
   --build.created value                 build created (default: 0) [$DRONE_BUILD_CREATED]
   --build.started value                 build started (default: 0) [$DRONE_BUILD_STARTED]
   --build.finished value                build finished (default: 0) [$DRONE_BUILD_FINISHED]
   --build.status value                  build status (default: "success") [$DRONE_BUILD_STATUS]
   --build.link value                    build link [$DRONE_BUILD_LINK]
   --build.deploy value                  build deployment target [$DRONE_DEPLOY_TO]
   --yaml.verified                       build yaml is verified (default: false) [$DRONE_YAML_VERIFIED]
   --yaml.signed                         build yaml is signed (default: false) [$DRONE_YAML_SIGNED]
   --prev.build.number value             previous build number (default: 0) [$DRONE_PREV_BUILD_NUMBER]
   --prev.build.status value             previous build status [$DRONE_PREV_BUILD_STATUS]
   --prev.commit.sha value               previous build sha [$DRONE_PREV_COMMIT_SHA]
   --backend value                       cache backend to use in plugin (s3, filesystem, sftp, azure, gcs) (default: "s3") [$PLUGIN_BACKEND]
   --mount value                         cache directories, an array of folders to cache [$PLUGIN_MOUNT]
   --rebuild                             rebuild the cache directories (default: false) [$PLUGIN_REBUILD]
   --restore                             restore the cache directories (default: false) [$PLUGIN_RESTORE]
   --cache-key value                     cache key to use for the cache directories [$PLUGIN_CACHE_KEY]
   --remote-root value                   remote root directory to contain all the cache files created (default repo.name) [$PLUGIN_REMOTE_ROOT]
   --local-root value                    local root directory to base given mount paths (default pwd [present working directory]) [$PLUGIN_LOCAL_ROOT]
   --override                            override even if cache key already exists in backend (default: true) [$PLUGIN_OVERRIDE]
   --archive-format value                archive format to use to store the cache directories (tar, gzip, zstd) (default: "tar") [$PLUGIN_ARCHIVE_FORMAT]
   --compression-level value             compression level to use for gzip/zstd compression when archive-format specified as gzip/zstd
                                             (check https://godoc.org/compress/flate#pkg-constants for available options for gzip
                                             and https://pkg.go.dev/github.com/klauspost/compress/zstd#EncoderLevelFromZstd for zstd) (default: -1) 
   --skip-symlinks                       skip symbolic links in archive (default: false) [$PLUGIN_SKIP_SYMLINKS, $SKIP_SYMLINKS]
   --debug                               debug (default: false) [$PLUGIN_DEBUG, $DEBUG]
   --backend.operation-timeout value     timeout value to use for each storage operations (default: 3m0s) [$PLUGIN_BACKEND_OPERATION_TIMEOUT, $BACKEND_OPERATION_TIMEOUT]
   --endpoint value                      endpoint for the s3/cloud storage connection [$PLUGIN_ENDPOINT, $S3_ENDPOINT, $GCS_ENDPOINT]
   --bucket value                        AWS bucket name [$PLUGIN_BUCKET, $S3_BUCKET, $GCS_BUCKET]
   --filesystem.cache-root value         local filesystem root directory for the filesystem cache (default: "/tmp/cache") [$PLUGIN_FILESYSTEM_CACHE_ROOT, $FILESYSTEM_CACHE_ROOT]
   --access-key value                    AWS access key [$PLUGIN_ACCESS_KEY, $AWS_ACCESS_KEY_ID, $CACHE_AWS_ACCESS_KEY_ID]
   --secret-key value                    AWS secret key [$PLUGIN_SECRET_KEY, $AWS_SECRET_ACCESS_KEY, $CACHE_AWS_SECRET_ACCESS_KEY]
   --region value                        AWS bucket region. (us-east-1, eu-west-1, ...) [$PLUGIN_REGION, $S3_REGION]
   --path-style                          AWS path style to use for bucket paths. (true for minio, false for aws) (default: false) [$PLUGIN_PATH_STYLE, $AWS_PLUGIN_PATH_STYLE]
   --acl value                           upload files with acl (private, public-read, ...) [$PLUGIN_ACL, $AWS_ACL]
   --encryption value                    server-side encryption algorithm, defaults to none. (AES256, aws:kms) [$PLUGIN_ENCRYPTION, $AWS_ENCRYPTION]
   --sts-endpoint value                  Custom STS endpoint for IAM role assumption [$PLUGIN_STS_ENDPOINT, $AWS_STS_ENDPOINT]
   --role-arn value                      AWS IAM role ARN to assume [$PLUGIN_ASSUME_ROLE_ARN, $AWS_ASSUME_ROLE_ARN]
   --gcs.api-key value                   Google service account API key [$PLUGIN_API_KEY, $GCP_API_KEY]
   --gcs.json-key value                  Google service account JSON key [$PLUGIN_JSON_KEY, $GCS_CACHE_JSON_KEY]
   --gcs.acl value                       upload files with acl (private, public-read, ...) (default: "private") [$PLUGIN_GCS_ACL, $GCS_ACL]
   --gcs.encryption-key value            server-side encryption key, must be a 32-byte AES-256 key, defaults to none
                                             (See https://cloud.google.com/storage/docs/encryption for details.) [$PLUGIN_GCS_ENCRYPTION_KEY, $GCS_ENCRYPTION_KEY]
   --azure.account-name value            Azure Blob Storage Account Name [$PLUGIN_ACCOUNT_NAME, $AZURE_ACCOUNT_NAME]
   --azure.account-key value             Azure Blob Storage Account Key [$PLUGIN_ACCOUNT_KEY, $AZURE_ACCOUNT_KEY]
   --azure.container-name value          Azure Blob Storage container name [$PLUGIN_CONTAINER, $AZURE_CONTAINER_NAME]
   --azure.blob-storage-url value        Azure Blob Storage URL (default: "blob.core.windows.net") [$AZURE_BLOB_STORAGE_URL]
   --azure.blob-max-retry-requets value  Azure Blob Storage Max Retry Requests (default: 4) [$AZURE_BLOB_MAX_RETRY_REQUESTS]
   --sftp.cache-root value               sftp root directory [$SFTP_CACHE_ROOT]
   --sftp.username value                 sftp username [$PLUGIN_USERNAME, $SFTP_USERNAME]
   --sftp.password value                 sftp password [$PLUGIN_PASSWORD, $SFTP_PASSWORD]
   --sftp.public-key-file value          sftp public key file path [$PLUGIN_PUBLIC_KEY_FILE, $SFTP_PUBLIC_KEY_FILE]
   --sftp.auth-method value              sftp auth method, defaults to none. (PASSWORD, PUBLIC_KEY_FILE) [$SFTP_AUTH_METHOD]
   --sftp.host value                     sftp host [$SFTP_HOST]
   --sftp.port value                     sftp port [$SFTP_PORT]
   --help, -h                            show help (default: false)
   --version, -v                         print the version (default: false)

Using Docker (with Environment variables)

$ docker run --rm \
      -v "$(pwd)":/app \
      -e DRONE_REPO=octocat/hello-world \
      -e DRONE_REPO_BRANCH=master \
      -e DRONE_COMMIT_BRANCH=master \
      -e PLUGIN_MOUNT=/app/node_modules \
      -e PLUGIN_RESTORE=false \
      -e PLUGIN_REBUILD=true \
      -e PLUGIN_BUCKET=<bucket> \
      -e AWS_ACCESS_KEY_ID=<token> \
      -e AWS_SECRET_ACCESS_KEY=<secret> \
      meltwater/drone-cache

Development

Usage:
  make <target>

Targets:
  setup          	  Setups dev environment
  drone-cache    	  Runs drone-cache target
  clean          	  Cleans build resourcess
  docs           	  Generates docs
  generate       	  Generate documentation, website and yaml files,
  vendor         	  Updates vendored copy of dependencies
  compress       	  Creates compressed binary
  container      	  Builds drone-cache docker image with latest tag
  container-push 	  Pushes latest $(CONTAINER_REPO) image to repository
  test           	  Runs tests
  test-integration	  Runs integration tests
  test-unit      	  Runs unit tests
  lint           	  Runs golangci-lint analysis
  fix            	  Runs golangci-lint fix
  format         	  Runs gofmt
  help           	  Shows this help message

Releases

Release management handled by the CI pipeline. When you create a tag on master branch, CI handles the rest.

You can find released artifacts (binaries, code, archives) under releases.

You can find released images at DockerHub.

PLEASE DO NOT INTRODUCE BREAKING CHANGES

Keep in mind that users usually use the image tagged with latest in their pipeline, please make sure you do not interfere with their working workflow. Latest stable releases will be tagged with the latest.

Versioning

drone-cache uses SemVer for versioning. For the versions available, see the tags on this repository.

As the versioning scheme dictates, drone-cache respects backward compatibility within the major versions. However, the project only offers guarantees regarding the command-line interface (flags and environment variables). Any exported public package can change its API.

Authors and Acknowledgement

See the list of all contributors.

• @dim - Thanks for original work!
• @AdamGlazerMW - Special thanks to Adam for the amazing artwork!

Inspiration

• github.com/bsm/drone-s3-cache (original work)
• github.com/Drillster/drone-volume-cache
• github.com/drone/drone-cache-lib

Contributing

Please read CONTRIBUTING.md to understand how to submit pull requests to us, and also see our code of conduct.

Future work

All ideas for new features and bug reports will be kept in github.com/meltwater/drone-cache/issues.

One bigger area of future investment is to add a couple of new storage backends for caching the workspace files.

License and Copyright

This project is licensed under the Apache License 2.0.